Compare commits
No commits in common. "main" and "manifest" have entirely different histories.
103
CLAUDE.md
103
CLAUDE.md
@ -11,11 +11,9 @@ This is a Go-based HLS (HTTP Live Streaming) recorder that monitors M3U8 playlis
|
||||
The project follows a modular architecture with clear separation of concerns:
|
||||
|
||||
- **cmd/**: Entry points for different execution modes
|
||||
- **main/main.go**: Primary CLI entry point with URL input, event naming, and mode selection
|
||||
- **main/main.go**: Primary CLI entry point with URL input and event naming
|
||||
- **downloader/download.go**: Core download orchestration logic with transfer service integration
|
||||
- **processor/process.go**: Alternative processing entry point
|
||||
- **transfer/transfer.go**: Transfer-only mode entry point
|
||||
- **check/check.go**: Connection-check mode entry point (stream host + NAS reachability, no downloads)
|
||||
- **pkg/**: Core packages containing the application logic
|
||||
- **media/**: HLS streaming and download logic
|
||||
- **stream.go**: Stream variant parsing and downloading orchestration (`GetAllVariants`, `VariantDownloader`)
|
||||
@ -29,24 +27,7 @@ The project follows a modular architecture with clear separation of concerns:
|
||||
- **nas.go**: NAS file transfer with retry logic
|
||||
- **cleanup.go**: Local file cleanup after successful transfer
|
||||
- **types.go**: Transfer system data structures
|
||||
- **processing/**: Video processing and concatenation system
|
||||
- **service.go**: Processing service orchestration with FFmpeg integration
|
||||
- **segment.go**: Individual segment processing logic
|
||||
- **types.go**: Processing system data structures
|
||||
- **nas/**: NAS connection and file operations (platform-agnostic: Windows and Linux)
|
||||
- **config.go**: NAS configuration structure
|
||||
- **nas.go**: `NASService` — resolves the destination to a host-native path, then copies/verifies files against it
|
||||
- **location.go**: Parses UNC / `//server/share` / `smb://` / local paths into a host-neutral `Location`
|
||||
- **connector.go**: `Connector` interface + runtime selection by GOOS (no build tags, so both platforms' code is compiled and tested everywhere)
|
||||
- **connector_netuse.go**: Windows strategy — `net use`, UNC path used directly
|
||||
- **connector_mount.go**: Linux strategy — resolve the share against `/proc/mounts`, optional root-only `mount.cifs`
|
||||
- **connector_local.go**: Already-mounted/plain directory on either platform
|
||||
- **mounts.go**: `/proc/mounts` parsing and share-to-mount-point matching
|
||||
- **config/**: Centralized configuration management with validation
|
||||
- **config.go**: Configuration loading, validation, and path resolution
|
||||
- **utils/**: Utility functions for cross-platform compatibility
|
||||
- **paths.go**: Path manipulation and validation utilities
|
||||
- **constants/constants.go**: Configuration constants and singleton access
|
||||
- **constants/constants.go**: Configuration constants (paths, timeouts, NAS settings)
|
||||
- **httpClient/error.go**: HTTP error handling utilities
|
||||
|
||||
## Core Functionality
|
||||
@ -66,13 +47,6 @@ The project follows a modular architecture with clear separation of concerns:
|
||||
4. **Local Cleanup**: Successfully transferred files are automatically cleaned up locally
|
||||
5. **State Persistence**: Queue state is persisted to survive crashes and restarts
|
||||
|
||||
### Video Processing Workflow (Optional)
|
||||
1. **Segment Collection**: Processing service reads downloaded segments from NAS storage
|
||||
2. **Quality Selection**: Automatically selects the highest quality variant available
|
||||
3. **FFmpeg Processing**: Uses FFmpeg to concatenate segments into a single MP4 file
|
||||
4. **Output Management**: Processed videos are saved to the configured output directory
|
||||
5. **Concurrent Processing**: Multiple events can be processed simultaneously with worker pools
|
||||
|
||||
## Key Data Structures
|
||||
|
||||
- `StreamVariant`: Represents a stream quality variant with URL, bandwidth, resolution, output directory, and manifest writer
|
||||
@ -81,60 +55,35 @@ The project follows a modular architecture with clear separation of concerns:
|
||||
- `ManifestItem`: Individual segment record with sequence number and resolution
|
||||
- `TransferItem`: Transfer queue item with source, destination, retry count, and status
|
||||
- `TransferService`: Orchestrates file watching, queuing, transfer, and cleanup
|
||||
- `ProcessingService`: Manages video processing operations with FFmpeg integration
|
||||
- `ProcessConfig`: Configuration for processing operations including worker count and paths
|
||||
- `NASService`: Handles NAS connection, authentication, and file operations
|
||||
- `NASConfig`: Configuration structure for NAS connection parameters
|
||||
|
||||
## Configuration
|
||||
|
||||
Configuration is managed through a centralized system in `pkg/config/config.go` with environment variable support for deployment flexibility. The system provides validation, cross-platform path resolution, and sensible defaults:
|
||||
Key configuration is managed in `pkg/constants/constants.go`:
|
||||
|
||||
### Core Settings
|
||||
- `Core.WorkerCount`: Number of concurrent segment downloaders per variant (4) - ENV: `WORKER_COUNT`
|
||||
- `Core.RefreshDelay`: How often to check for playlist updates (3 seconds) - ENV: `REFRESH_DELAY_SECONDS`
|
||||
|
||||
### Path Configuration
|
||||
- `Paths.LocalOutput`: Base directory for local downloads (`data/`) - ENV: `LOCAL_OUTPUT_DIR`
|
||||
- `Paths.ProcessOutput`: Directory for processed videos (`out/`) - ENV: `PROCESS_OUTPUT_DIR`
|
||||
- `Paths.ManifestDir`: Directory for manifest JSON files (`data/`)
|
||||
- `Paths.PersistenceFile`: Transfer queue state file location
|
||||
- `WorkerCount`: Number of concurrent segment downloaders per variant (4)
|
||||
- `RefreshDelay`: How often to check for playlist updates (3 seconds)
|
||||
- `LocalOutputDirPath`: Base directory for local downloads (`./data/`)
|
||||
- `ManifestPath`: Directory for manifest JSON files (`./data`)
|
||||
|
||||
### HTTP Settings
|
||||
- `HTTPUserAgent`: User agent string for HTTP requests
|
||||
- `REFERRER`: Referer header for HTTP requests (`https://www.flomarching.com`)
|
||||
- `REFERRER`: Referer header for HTTP requests
|
||||
|
||||
### NAS Transfer Settings
|
||||
- `NAS.EnableTransfer`: Enable/disable automatic NAS transfer (true) - ENV: `ENABLE_NAS_TRANSFER`
|
||||
- `NAS.OutputPath`: NAS destination (`\\HomeLabNAS\dci\streams`) - ENV: `NAS_OUTPUT_PATH`. Accepts UNC, `//server/share`, `smb://`, or a local/mounted directory; Windows connects it with `net use`, Linux resolves it against `/proc/mounts`
|
||||
- `NAS.Username`/`NAS.Password`: NAS credentials for authentication - ENV: `NAS_USERNAME`/`NAS_PASSWORD`
|
||||
- `NAS.AutoMount`: Linux-only, root-required auto-mount of the SMB share (false) - ENV: `NAS_AUTO_MOUNT`
|
||||
- `NAS.MountPoint`: Override for the auto-mount location - ENV: `NAS_MOUNT_POINT`
|
||||
- `Transfer.WorkerCount`: Concurrent transfer workers (2)
|
||||
- `Transfer.RetryLimit`: Max retry attempts per file (3)
|
||||
- `Transfer.Timeout`: Timeout per file transfer (30 seconds)
|
||||
- `Transfer.FileSettlingDelay`: Wait before queuing new files (5 seconds)
|
||||
- `Transfer.QueueSize`: Maximum queue size (100000)
|
||||
- `Transfer.BatchSize`: Batch processing size (1000)
|
||||
|
||||
### Processing Settings
|
||||
- `Processing.AutoProcess`: Enable automatic processing after download (true)
|
||||
- `Processing.Enabled`: Enable processing functionality (true)
|
||||
- `Processing.WorkerCount`: Concurrent processing workers (2)
|
||||
- `Processing.FFmpegPath`: Path to FFmpeg executable (`ffmpeg`) - ENV: `FFMPEG_PATH`
|
||||
- `EnableNASTransfer`: Enable/disable automatic NAS transfer (true)
|
||||
- `NASOutputPath`: UNC path to NAS storage (`\\HomeLabNAS\dci\streams`)
|
||||
- `NASUsername`/`NASPassword`: NAS credentials (empty for current user)
|
||||
- `TransferWorkerCount`: Concurrent transfer workers (2)
|
||||
- `TransferRetryLimit`: Max retry attempts per file (3)
|
||||
- `TransferTimeout`: Timeout per file transfer (30 seconds)
|
||||
- `FileSettlingDelay`: Wait before queuing new files (5 seconds)
|
||||
- `PersistencePath`: Transfer queue state file (`./data/transfer_queue.json`)
|
||||
|
||||
### Cleanup Settings
|
||||
- `Cleanup.AfterTransfer`: Delete local files after NAS transfer (true)
|
||||
- `Cleanup.BatchSize`: Files processed per cleanup batch (1000)
|
||||
- `Cleanup.RetainHours`: Hours to keep local files (0 = immediate cleanup)
|
||||
|
||||
### Configuration Access
|
||||
```go
|
||||
cfg := constants.MustGetConfig() // Get validated config singleton
|
||||
eventPath := cfg.GetEventPath("my-event") // Get cross-platform paths
|
||||
```
|
||||
|
||||
See `DEPLOYMENT.md` for detailed environment variable configuration and deployment examples.
|
||||
- `CleanupAfterTransfer`: Delete local files after NAS transfer (true)
|
||||
- `CleanupBatchSize`: Files processed per cleanup batch (10)
|
||||
- `RetainLocalHours`: Hours to keep local files (0 = immediate cleanup)
|
||||
|
||||
## Common Development Commands
|
||||
|
||||
@ -163,9 +112,6 @@ go fmt ./...
|
||||
- `-url`: M3U8 playlist URL (if not provided, prompts for input)
|
||||
- `-event`: Event name for organizing downloads (defaults to current date)
|
||||
- `-debug`: Debug mode (only downloads 1080p variant for easier testing)
|
||||
- `-transfer`: Transfer-only mode (transfer existing files without downloading)
|
||||
- `-process`: Process-only mode (process existing files without downloading)
|
||||
- `-check`: Connection-check mode (test stream URL + NAS connectivity and exit; exit code 1 on any failure)
|
||||
|
||||
## Monitoring and Downloads
|
||||
|
||||
@ -216,9 +162,7 @@ Downloaded files are organized as:
|
||||
│ ├── 1080p/ # High quality segments
|
||||
│ ├── 720p/ # Medium quality segments
|
||||
│ └── 480p/ # Lower quality segments
|
||||
├── transfer_queue.json # Transfer queue state
|
||||
├── refresh_token.txt # Authentication tokens
|
||||
└── tokens.txt # Session tokens
|
||||
└── transfer_queue.json # Transfer queue state
|
||||
```
|
||||
|
||||
NAS files mirror the local structure:
|
||||
@ -229,10 +173,3 @@ NAS files mirror the local structure:
|
||||
├── 720p/
|
||||
└── 480p/
|
||||
```
|
||||
|
||||
Processed files are output to:
|
||||
```
|
||||
./out/
|
||||
└── {event-name}/
|
||||
└── concatenated_segments.mp4 # Final processed video
|
||||
```
|
||||
205
DEPLOYMENT.md
205
DEPLOYMENT.md
@ -1,205 +0,0 @@
|
||||
# Deployment Guide
|
||||
|
||||
This document outlines how to deploy and configure the StreamRecorder application in different environments.
|
||||
|
||||
## Environment Variables
|
||||
|
||||
The application supports configuration through environment variables for flexible deployment:
|
||||
|
||||
### Core Settings
|
||||
- `WORKER_COUNT`: Number of concurrent segment downloaders per variant (default: 4)
|
||||
- `REFRESH_DELAY_SECONDS`: How often to check for playlist updates in seconds (default: 3)
|
||||
|
||||
### NAS Transfer Settings
|
||||
- `NAS_OUTPUT_PATH`: NAS destination (default: "\\\\HomeLabNAS\\dci\\streams"). Accepts a UNC path, `//server/share/dir`, `smb://server/share/dir`, or a local/mounted directory
|
||||
- `NAS_USERNAME`: NAS authentication username
|
||||
- `NAS_PASSWORD`: NAS authentication password
|
||||
- `ENABLE_NAS_TRANSFER`: Enable/disable automatic NAS transfer (default: true)
|
||||
- `NAS_AUTO_MOUNT`: Linux only — mount the SMB share if the host hasn't already (default: false, requires root)
|
||||
- `NAS_MOUNT_POINT`: Where `NAS_AUTO_MOUNT` attaches the share (default: `/mnt/streamrecorder/<server>/<share>`)
|
||||
|
||||
The same `NAS_OUTPUT_PATH` works on Windows and Linux. Windows authenticates the
|
||||
UNC path with `net use`; Linux resolves the share against `/proc/mounts` and
|
||||
rewrites the path onto the mount point, so `\\\\HomeLabNAS\\dci\\streams` with
|
||||
`//homelabnas/dci` mounted at `/mnt/dci` becomes `/mnt/dci/streams`. Mounting the
|
||||
share on the host (fstab/autofs/systemd, or `docker run -v`) is the recommended
|
||||
Linux setup; `NAS_AUTO_MOUNT` is the root-privileged fallback. Verify either with:
|
||||
|
||||
```bash
|
||||
./stream-recorder -check
|
||||
```
|
||||
|
||||
### Path Configuration
|
||||
- `LOCAL_OUTPUT_DIR`: Base directory for local downloads (default: "data")
|
||||
- `PROCESS_OUTPUT_DIR`: Output directory for processed videos (default: "out")
|
||||
|
||||
### Processing Settings
|
||||
- `FFMPEG_PATH`: Path to FFmpeg executable (default: "ffmpeg")
|
||||
|
||||
## Docker Deployment
|
||||
|
||||
### Dockerfile Example
|
||||
|
||||
```dockerfile
|
||||
FROM golang:1.23-alpine AS builder
|
||||
WORKDIR /app
|
||||
COPY go.mod go.sum ./
|
||||
RUN go mod download
|
||||
COPY . .
|
||||
RUN go build -o stream-recorder ./cmd/main
|
||||
|
||||
FROM alpine:latest
|
||||
RUN apk --no-cache add ca-certificates ffmpeg
|
||||
WORKDIR /root/
|
||||
COPY --from=builder /app/stream-recorder .
|
||||
CMD ["./stream-recorder"]
|
||||
```
|
||||
|
||||
### Docker Compose Example
|
||||
|
||||
```yaml
|
||||
version: '3.8'
|
||||
services:
|
||||
stream-recorder:
|
||||
build: .
|
||||
environment:
|
||||
- NAS_OUTPUT_PATH=/mnt/nas/streams
|
||||
- NAS_USERNAME=${NAS_USERNAME}
|
||||
- NAS_PASSWORD=${NAS_PASSWORD}
|
||||
- LOCAL_OUTPUT_DIR=/app/data
|
||||
- PROCESS_OUTPUT_DIR=/app/out
|
||||
- FFMPEG_PATH=ffmpeg
|
||||
volumes:
|
||||
- ./data:/app/data
|
||||
- ./out:/app/out
|
||||
- nas_mount:/mnt/nas
|
||||
networks:
|
||||
- stream_network
|
||||
|
||||
volumes:
|
||||
nas_mount:
|
||||
driver: local
|
||||
driver_opts:
|
||||
type: cifs
|
||||
device: "//HomeLabNAS/dci"
|
||||
o: username=${NAS_USERNAME},password=${NAS_PASSWORD},iocharset=utf8
|
||||
|
||||
networks:
|
||||
stream_network:
|
||||
driver: bridge
|
||||
```
|
||||
|
||||
## Kubernetes Deployment
|
||||
|
||||
### ConfigMap Example
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: stream-recorder-config
|
||||
data:
|
||||
WORKER_COUNT: "4"
|
||||
REFRESH_DELAY_SECONDS: "3"
|
||||
ENABLE_NAS_TRANSFER: "true"
|
||||
LOCAL_OUTPUT_DIR: "/app/data"
|
||||
PROCESS_OUTPUT_DIR: "/app/out"
|
||||
FFMPEG_PATH: "ffmpeg"
|
||||
```
|
||||
|
||||
### Secret Example
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: stream-recorder-secrets
|
||||
type: Opaque
|
||||
data:
|
||||
NAS_USERNAME: <base64-encoded-username>
|
||||
NAS_PASSWORD: <base64-encoded-password>
|
||||
```
|
||||
|
||||
### Deployment Example
|
||||
|
||||
```yaml
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: stream-recorder
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app: stream-recorder
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: stream-recorder
|
||||
spec:
|
||||
containers:
|
||||
- name: stream-recorder
|
||||
image: stream-recorder:latest
|
||||
envFrom:
|
||||
- configMapRef:
|
||||
name: stream-recorder-config
|
||||
- secretRef:
|
||||
name: stream-recorder-secrets
|
||||
volumeMounts:
|
||||
- name: data-storage
|
||||
mountPath: /app/data
|
||||
- name: output-storage
|
||||
mountPath: /app/out
|
||||
volumes:
|
||||
- name: data-storage
|
||||
persistentVolumeClaim:
|
||||
claimName: stream-data-pvc
|
||||
- name: output-storage
|
||||
persistentVolumeClaim:
|
||||
claimName: stream-output-pvc
|
||||
```
|
||||
|
||||
## Production Considerations
|
||||
|
||||
### Security
|
||||
- Never commit credentials to version control
|
||||
- Use environment variables or secret management systems for sensitive data
|
||||
- Consider using service accounts or IAM roles for cloud deployments
|
||||
- Rotate credentials regularly
|
||||
|
||||
### Monitoring
|
||||
- Implement health checks for the application
|
||||
- Monitor disk space for download directories
|
||||
- Set up alerts for failed transfers or processing
|
||||
- Log to centralized logging systems
|
||||
|
||||
### Scaling
|
||||
- Use horizontal scaling for multiple concurrent streams
|
||||
- Consider using message queues for segment processing
|
||||
- Implement distributed storage for high availability
|
||||
- Use load balancers for multiple instances
|
||||
|
||||
### Backup and Recovery
|
||||
- Regular backups of configuration and state files
|
||||
- Test recovery procedures
|
||||
- Document rollback processes
|
||||
- Maintain disaster recovery plans
|
||||
|
||||
## Configuration Validation
|
||||
|
||||
The application validates configuration at startup and will fail fast if:
|
||||
- Required directories cannot be created
|
||||
- NAS paths are invalid when transfer is enabled
|
||||
- FFmpeg is not found when processing is enabled
|
||||
- Critical environment variables are malformed
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
1. **Path Permission Errors**: Ensure the application has write access to configured directories
|
||||
2. **NAS Connection Failures**: Verify network connectivity and credentials
|
||||
3. **FFmpeg Not Found**: Install FFmpeg or set correct FFMPEG_PATH
|
||||
4. **Environment Variable Format**: Check for typos and correct boolean values
|
||||
|
||||
### Debug Mode
|
||||
Run with `-debug=true` to enable debug logging and download only 1080p variants for testing.
|
||||
128
Makefile
128
Makefile
@ -1,128 +0,0 @@
|
||||
# StreamRecorder Makefile
|
||||
|
||||
.PHONY: test test-verbose test-coverage build clean help
|
||||
|
||||
# Default target
|
||||
all: build
|
||||
|
||||
# Build the application
|
||||
build:
|
||||
@echo "🔨 Building StreamRecorder..."
|
||||
go build -o stream-recorder.exe ./cmd/main
|
||||
|
||||
# Build for different platforms
|
||||
build-windows:
|
||||
@echo "🔨 Building for Windows..."
|
||||
GOOS=windows GOARCH=amd64 go build -o stream-recorder-windows.exe ./cmd/main
|
||||
|
||||
build-linux:
|
||||
@echo "🔨 Building for Linux..."
|
||||
GOOS=linux GOARCH=amd64 go build -o stream-recorder-linux ./cmd/main
|
||||
|
||||
build-all: build-windows build-linux
|
||||
|
||||
# Run unit tests
|
||||
test:
|
||||
@echo "🧪 Running unit tests..."
|
||||
go run test_runner.go
|
||||
|
||||
# Run tests with verbose output
|
||||
test-verbose:
|
||||
@echo "🧪 Running unit tests (verbose)..."
|
||||
go test -v ./pkg/...
|
||||
|
||||
# Run tests with coverage
|
||||
test-coverage:
|
||||
@echo "🧪 Running tests with coverage..."
|
||||
go test -coverprofile=coverage.out ./pkg/...
|
||||
go tool cover -html=coverage.out -o coverage.html
|
||||
@echo "📊 Coverage report generated: coverage.html"
|
||||
|
||||
# Run tests for a specific package
|
||||
test-pkg:
|
||||
@if [ -z "$(PKG)" ]; then \
|
||||
echo "❌ Please specify package: make test-pkg PKG=./pkg/config"; \
|
||||
exit 1; \
|
||||
fi
|
||||
@echo "🧪 Testing package: $(PKG)"
|
||||
go test -v $(PKG)
|
||||
|
||||
# Run benchmarks
|
||||
benchmark:
|
||||
@echo "🏃 Running benchmarks..."
|
||||
go test -bench=. -benchmem ./pkg/...
|
||||
|
||||
# Clean build artifacts
|
||||
clean:
|
||||
@echo "🧹 Cleaning build artifacts..."
|
||||
rm -f stream-recorder.exe stream-recorder-windows.exe stream-recorder-linux
|
||||
rm -f coverage.out coverage.html
|
||||
rm -rf data/ out/ *.json
|
||||
|
||||
# Format code
|
||||
fmt:
|
||||
@echo "🎨 Formatting code..."
|
||||
go fmt ./...
|
||||
|
||||
# Lint code (requires golangci-lint)
|
||||
lint:
|
||||
@echo "🔍 Linting code..."
|
||||
golangci-lint run
|
||||
|
||||
# Tidy dependencies
|
||||
tidy:
|
||||
@echo "📦 Tidying dependencies..."
|
||||
go mod tidy
|
||||
|
||||
# Run security check (requires gosec)
|
||||
security:
|
||||
@echo "🔒 Running security check..."
|
||||
gosec ./...
|
||||
|
||||
# Install development tools
|
||||
install-tools:
|
||||
@echo "🛠️ Installing development tools..."
|
||||
go install github.com/golangci/golangci-lint/cmd/golangci-lint@latest
|
||||
go install github.com/securecodewarrior/gosec/v2/cmd/gosec@latest
|
||||
|
||||
# Quick development cycle: format, tidy, build, test
|
||||
dev: fmt tidy build test
|
||||
|
||||
# CI pipeline: format check, lint, security, test, build
|
||||
ci: fmt tidy lint security test build
|
||||
|
||||
# Help
|
||||
help:
|
||||
@echo "StreamRecorder Build Commands"
|
||||
@echo "============================="
|
||||
@echo ""
|
||||
@echo "Build Commands:"
|
||||
@echo " build - Build the main application"
|
||||
@echo " build-windows - Build for Windows (x64)"
|
||||
@echo " build-linux - Build for Linux (x64)"
|
||||
@echo " build-all - Build for all platforms"
|
||||
@echo ""
|
||||
@echo "Test Commands:"
|
||||
@echo " test - Run unit tests with custom runner"
|
||||
@echo " test-verbose - Run tests with verbose output"
|
||||
@echo " test-coverage - Run tests with coverage report"
|
||||
@echo " test-pkg PKG=<pkg> - Test specific package"
|
||||
@echo " benchmark - Run benchmarks"
|
||||
@echo ""
|
||||
@echo "Quality Commands:"
|
||||
@echo " fmt - Format code"
|
||||
@echo " lint - Lint code (requires golangci-lint)"
|
||||
@echo " security - Security analysis (requires gosec)"
|
||||
@echo " tidy - Tidy dependencies"
|
||||
@echo ""
|
||||
@echo "Development Commands:"
|
||||
@echo " dev - Quick dev cycle (fmt, tidy, build, test)"
|
||||
@echo " ci - Full CI pipeline"
|
||||
@echo " clean - Clean build artifacts"
|
||||
@echo " install-tools - Install development tools"
|
||||
@echo ""
|
||||
@echo "Examples:"
|
||||
@echo " make test"
|
||||
@echo " make test-pkg PKG=./pkg/config"
|
||||
@echo " make build-all"
|
||||
@echo " make dev"
|
||||
217
README.md
217
README.md
@ -1,217 +0,0 @@
|
||||
# m3u8-downloader
|
||||
|
||||
A Go command-line tool that records live HLS (HTTP Live Streaming) broadcasts. Point it at a master `.m3u8` playlist URL and it will:
|
||||
|
||||
1. **Download** every quality variant (1080p, 720p, ...) in real time as new segments appear.
|
||||
2. **Transfer** finished segments to NAS storage for archival, with retry logic and local cleanup.
|
||||
3. **Process** the archived segments into a single playable `.mp4` per event via FFmpeg.
|
||||
|
||||
**Download and transfer run together, in one invocation.** When NAS transfer is enabled (the default), starting a download also starts the transfer service in the same process: as soon as a segment is written to disk, a file watcher picks it up, transfers it to NAS, and deletes the local copy — all while the download keeps running. This keeps local disk usage bounded during a long recording instead of accumulating the whole event locally.
|
||||
|
||||
**Processing is a separate, later invocation** (see [Usage](#usage) below), because it reads finished segments *from NAS*, after transfer has fully drained — it can't run concurrently with the other two in any useful way.
|
||||
|
||||
There's also a standalone `-transfer` mode, but it's a catch-up/recovery tool, not the normal path: use it when a download session already ended with some files not yet confirmed on NAS (killed before the queue drained, or transfer was disabled during the recording and you want to sync afterward).
|
||||
|
||||
The project also ships structured JSON logging per subsystem and an HTTP status endpoint with a small web dashboard, so you can tell what's happening (and especially what's failing) without staring at a terminal.
|
||||
|
||||
> Project docs also refer to this as "StreamRecorder" (see `DEPLOYMENT.md`/`TESTING.md`) — same project, the Go module is named `m3u8-downloader`.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
|
||||
.m3u8 URL ──▶ │ DOWNLOAD │──────▶ │ TRANSFER │──────▶ │ PROCESSING │──▶ event.mp4
|
||||
│ (pkg/media) │ .ts │(pkg/transfer,│ NAS │(pkg/processing)
|
||||
└──────────────┘ files │ pkg/nas) │ files └──────────────┘
|
||||
│ └──────────────┘ │
|
||||
└──────────────┬─────────────────────────────────┘
|
||||
▼
|
||||
pkg/logging (JSON files) + pkg/health (HTTP /status, dashboard)
|
||||
```
|
||||
|
||||
### Download — `pkg/media`
|
||||
|
||||
- `stream.go`: `GetAllVariants()` fetches and parses the master playlist, extracting every quality variant. `VariantDownloader()` runs one per-variant polling loop (every `Core.RefreshDelay`, default 3s) that watches the variant's chunklist for new segments and dispatches downloads through a worker-limited semaphore (`Core.WorkerCount` per variant, default 4).
|
||||
- `playlist.go`: `LoadMediaPlaylist()` fetches and parses an individual variant's chunklist.
|
||||
- `segment.go`: `DownloadSegment()` downloads one `.ts` file, with a couple of retries (including a dedicated retry for transient HTTP 403s some CDNs return).
|
||||
- `manifest.go`: `ManifestWriter` tracks every segment downloaded per event and writes a JSON manifest (sequence number + resolution) at the end of a run.
|
||||
|
||||
Segments land on disk under `Paths.LocalOutput/<event>/<resolution>/`.
|
||||
|
||||
### Transfer — `pkg/transfer` + `pkg/nas`
|
||||
|
||||
- `service.go`: `TransferService` wires together the watcher, queue, and cleanup service, and owns the connection to NAS for the run.
|
||||
- `watcher.go`: `FileWatcher` (via `fsnotify`) watches the local event directory for new `.ts` files and queues them after a short settling delay (`Transfer.FileSettlingDelay`, default 5s), so a file isn't queued while still being written.
|
||||
- `queue.go`: `TransferQueue` is a priority-queued worker pool (`Transfer.WorkerCount`, default 2) that transfers queued files to NAS with retry + exponential backoff, and persists its state to `transfer_queue.json` so it survives a restart.
|
||||
- `nas.go` / `pkg/nas`: `NASService` attaches the NAS destination for whatever host it's running on and does the actual file copy + size verification. See [NAS connectivity](#nas-connectivity-windows-and-linux).
|
||||
- `cleanup.go`: `CleanupService` deletes local copies after a successful, verified transfer (`Cleanup.AfterTransfer`, default true).
|
||||
|
||||
During a normal download run, this whole stage starts automatically in the same process as the downloader (see the note above) — you don't invoke transfer separately for a live recording. The standalone `-transfer` mode exists only to catch up files that a prior download session ended without fully transferring.
|
||||
|
||||
### Processing — `pkg/processing`
|
||||
|
||||
- `ProcessingService` reads segments **from NAS** (not the local disk) for a given event, picks the highest-resolution variant available for each sequence number, writes an FFmpeg concat list, and shells out to FFmpeg to produce one `.mp4`.
|
||||
- This is why processing is normally the *last* step, run after transfer has fully drained — it only sees what's actually on NAS.
|
||||
- Requires a working `ffmpeg` on `PATH` (or `Processing.FFmpegPath` / `FFMPEG_PATH`), and connects to NAS on startup regardless of whether `Processing.Enabled` is set.
|
||||
|
||||
### Configuration — `pkg/config` + `pkg/constants`
|
||||
|
||||
`pkg/config/config.go` defines defaults and environment-variable overrides for every setting below. `pkg/constants.MustGetConfig()` lazily loads and caches this config as a process-wide singleton — most of the codebase reads config this way rather than passing it around explicitly.
|
||||
|
||||
### Logging — `pkg/logging`
|
||||
|
||||
A small factory (modeled after a structured-logging pattern used in another project) that hands out one `log/slog` JSON logger per pipeline stage: `App`, `Download`, `Transfer`, `Processing`. Each writes to its own file under `Logging.Dir` (default `data/logs/{app,download,transfer,processing}.log`). This is **additive** — every existing human-readable console log line is untouched, so you can still watch progress live in the terminal; the structured logs are there for after-the-fact debugging.
|
||||
|
||||
### Health / status — `pkg/health`
|
||||
|
||||
An in-memory registry that every subsystem reports into (active variants, segment success/failure counts, NAS connectivity, transfer queue stats, processing pipeline state), served over HTTP:
|
||||
|
||||
- `GET /status` — full JSON snapshot, `status: "healthy"` or `"degraded"` (degraded the moment anything has a recent error, NAS is disconnected while in use, or processing has failed).
|
||||
- `GET /healthz` — plain liveness check.
|
||||
- `GET /` — a small auto-refreshing dashboard (polls `/status` every 3s) showing the same information visually. No external dependencies (fonts/JS/CSS), so it works on a venue LAN with no internet access.
|
||||
|
||||
**Important scope note:** the health server only reflects whatever is live in the *current* process. During a normal download run, `/status` shows real-time download + transfer state together (they're the same process) with processing reported as `idle`. Running `-process` afterward is a separate process with its own fresh health server showing real-time processing state (and idle download/transfer). There's no single dashboard spanning download/transfer and processing at the same time, since processing only ever runs after the fact.
|
||||
|
||||
By default the health server binds all network interfaces (`:8090`) so it can be checked from another device on the same LAN (e.g. a phone) during an event. **There is no authentication or TLS** — this is fine on a trusted local network only, never expose it to the open internet.
|
||||
|
||||
## Configuration
|
||||
|
||||
All settings have sane defaults; override via environment variables as needed.
|
||||
|
||||
| Env var | Default | Meaning |
|
||||
|---|---|---|
|
||||
| `WORKER_COUNT` | `4` | Concurrent segment downloads per variant |
|
||||
| `REFRESH_DELAY_SECONDS` | `3` | How often each variant's chunklist is polled |
|
||||
| `ENABLE_NAS_TRANSFER` | `true` | Whether to run the transfer service alongside download |
|
||||
| `NAS_OUTPUT_PATH` | `\\HomeLabNAS\dci\streams` | NAS root segments are copied to. UNC (`\\server\share\dir`), `//server/share/dir`, `smb://server/share/dir`, or a local/mounted directory — see [NAS connectivity](#nas-connectivity-windows-and-linux) |
|
||||
| `NAS_USERNAME` / `NAS_PASSWORD` | *(see note below)* | NAS auth credentials. Used by `net use` on Windows and by `NAS_AUTO_MOUNT` on Linux; ignored for an already-mounted path |
|
||||
| `NAS_AUTO_MOUNT` | `false` | Linux only: mount the SMB share if the host hasn't. Requires root |
|
||||
| `NAS_MOUNT_POINT` | *(derived)* | Where `NAS_AUTO_MOUNT` attaches the share (default `/mnt/streamrecorder/<server>/<share>`) |
|
||||
| `LOCAL_OUTPUT_DIR` | `data` | Local base directory for downloaded segments |
|
||||
| `PROCESS_OUTPUT_DIR` | `out` | Output directory for the final concatenated `.mp4` |
|
||||
| `FFMPEG_PATH` | `ffmpeg` | Path to the FFmpeg binary used by the processing stage |
|
||||
| `LOG_DIR` | `data/logs` | Directory for the per-subsystem JSON log files |
|
||||
| `LOG_LEVEL` | `info` | `debug` \| `info` \| `warn` \| `error` (also forced to `debug` by the `-debug` flag) |
|
||||
| `HEALTH_ENABLED` | `true` | Whether to start the health/status HTTP server |
|
||||
| `HEALTH_ADDR` | `:8090` | Bind address for the health server + dashboard |
|
||||
|
||||
> **Security note:** the default NAS credentials currently ship as hardcoded defaults in `pkg/config/config.go` for this home-lab setup. If you fork or redeploy this, set `NAS_USERNAME`/`NAS_PASSWORD` explicitly and rotate the shipped default — see `TODO.md` for the full flagged list of housekeeping items like this one.
|
||||
|
||||
## NAS connectivity (Windows and Linux)
|
||||
|
||||
The same `NAS_OUTPUT_PATH` works on both platforms; only the mechanism for reaching it differs, and the recorder picks it at runtime.
|
||||
|
||||
| Host | Destination | What happens |
|
||||
|---|---|---|
|
||||
| Windows | UNC / `//server/share` / `smb://` | `net use \\server\share` with `NAS_USERNAME`/`NAS_PASSWORD`, then the UNC path is used directly |
|
||||
| Linux | UNC / `//server/share` / `smb://` | The share is looked up in `/proc/mounts` and the UNC path is rewritten onto the mount point (`\\nas\dci\streams` + `//nas/dci` mounted at `/mnt/dci` → `/mnt/dci/streams`). Set `NAS_AUTO_MOUNT=true` to mount it instead |
|
||||
| Either | A local or already-mounted directory | Used as-is, no network step |
|
||||
|
||||
Notes:
|
||||
|
||||
- **Server and share names match case-insensitively**, and both `cifs`/`smb3` and `nfs`/`nfs4` mounts are recognised. If the share is mounted at a deeper level than configured (`//nas/dci/streams` at `/mnt/streams`), the deepest matching mount wins.
|
||||
- **Nothing the host already set up is torn down.** A share found already mounted (or already connected via `net use`) is left alone at shutdown; only a mount or connection this process created is undone.
|
||||
- **`NAS_AUTO_MOUNT` requires root**, so the usual Linux setup is to mount the share yourself via fstab/autofs/systemd and either point `NAS_OUTPUT_PATH` at the mount point or leave it as the UNC path and let it resolve. Credentials are passed to `mount.cifs` through a `0600` credentials file, never on the command line where the process table would expose the password.
|
||||
- **A Windows-shaped path on Linux is rejected, not silently created.** Backslashes and drive letters are legal filename characters on Linux, so `C:\data` or an unresolved `\\nas\dci` would otherwise be created as a local directory named exactly that — transfers would report success, cleanup would delete the local originals, and nothing would reach the NAS.
|
||||
|
||||
Run `-check` to see which strategy applies and where the destination resolved to.
|
||||
|
||||
## Building
|
||||
|
||||
```bash
|
||||
go build -o m3u8-downloader ./cmd/main
|
||||
# or, via the Makefile (produces stream-recorder(.exe)):
|
||||
make build
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
All modes are invoked through the single `cmd/main` binary:
|
||||
|
||||
```
|
||||
m3u8-downloader [-url <m3u8-url>] [-event <name>] [-debug] [-transfer] [-process] [-check]
|
||||
```
|
||||
|
||||
| Flag | Applies to | Meaning |
|
||||
|---|---|---|
|
||||
| `-url` | download mode | Master `.m3u8` playlist URL. If omitted (and neither `-transfer` nor `-process` is set), you'll be prompted for it interactively. |
|
||||
| `-event` | all modes | Event name — used to namespace local output, NAS destination, and the manifest file. **Always set this explicitly**; if left blank in download mode, segments land directly under the local output root instead of a per-event subfolder. In `-transfer`/`-process` mode, omitting it shows an interactive picker over existing event directories. |
|
||||
| `-debug` | download mode | Restricts download to the 1080p variant only (for faster iteration) and forces log level to `debug`. |
|
||||
| `-transfer` | transfer-only mode | Scans an already-downloaded event's local directory and transfers/re-transfers anything not yet confirmed on NAS, then runs the normal transfer service until shutdown. Requires `ENABLE_NAS_TRANSFER=true`. |
|
||||
| `-process` | process-only mode | Reads an event's segments from NAS and concatenates the highest-quality ones into a single `.mp4` via FFmpeg. Always attempts a NAS connection on startup, even if processing is otherwise disabled in config. |
|
||||
| `-check` | connection-check mode | Tests connectivity to everything a recording depends on, then exits without downloading or transferring anything. Exit code is 0 only if no check failed. |
|
||||
|
||||
### Which command for which situation
|
||||
|
||||
**Verifying the setup before an event** (in the spirit of `dbt debug` — proves the stream host and NAS are reachable without recording anything):
|
||||
```bash
|
||||
m3u8-downloader -check -url "https://example.com/live/master.m3u8"
|
||||
```
|
||||
It runs three checks and prints pass/fail for each:
|
||||
|
||||
- **configuration** — config loads, and the local/process/log directories resolve and are creatable.
|
||||
- **stream playlist** — only when `-url` is given (skipped otherwise). Fetches the master playlist with the same User-Agent and Referer the downloader uses, requires HTTP 200, decodes it, and lists the variants (and the resolution labels that would become output directories) it found.
|
||||
- **nas storage** — only when `ENABLE_NAS_TRANSFER=true` (skipped otherwise). Runs the same connect → mkdir → write-probe sequence the transfer service does at startup, bounded by the NAS timeout so an unresponsive share reports a timeout instead of hanging.
|
||||
|
||||
Skipped checks don't fail the run; the exit code is 1 if any check actually failed, so this can gate a scripted start:
|
||||
```bash
|
||||
m3u8-downloader -check -url "$URL" && m3u8-downloader -url "$URL" -event "2026-regionals"
|
||||
```
|
||||
|
||||
**Recording a live event** (the normal case — downloads and transfers to NAS concurrently, in one process, so local disk usage stays bounded rather than accumulating the whole event):
|
||||
```bash
|
||||
m3u8-downloader -url "https://example.com/live/master.m3u8" -event "2026-regionals"
|
||||
```
|
||||
Stop it cleanly with Ctrl+C (SIGINT/SIGTERM); it finishes in-flight downloads, saves transfer queue state, and writes the manifest before exiting.
|
||||
|
||||
**Quick test/dry run** (single variant, verbose logs, to confirm a playlist URL actually works before committing to a full multi-quality recording):
|
||||
```bash
|
||||
m3u8-downloader -url "https://example.com/live/master.m3u8" -event "test-run" -debug
|
||||
```
|
||||
|
||||
**Catching up NAS transfer after the fact** (recovery only — the recording process above already transfers live; use this if a session ended before every file made it to NAS, or transfer was disabled during the recording and you want to sync afterward):
|
||||
```bash
|
||||
m3u8-downloader -event "2026-regionals" -transfer
|
||||
```
|
||||
|
||||
**Producing the final video** (after transfer has fully drained — check `/status` or the NAS directory to confirm nothing's still pending):
|
||||
```bash
|
||||
m3u8-downloader -event "2026-regionals" -process
|
||||
```
|
||||
|
||||
**Checking on a running instance** (from any device on the same network as the recorder):
|
||||
```
|
||||
http://<recorder-host>:8090/ # dashboard
|
||||
http://<recorder-host>:8090/status # raw JSON
|
||||
```
|
||||
|
||||
## Data layout
|
||||
|
||||
```
|
||||
data/
|
||||
├── {event}.json # download manifest (sequence numbers + resolutions)
|
||||
├── {event}/
|
||||
│ ├── 1080p/ # downloaded segments, before NAS transfer + cleanup
|
||||
│ ├── 720p/
|
||||
│ └── ...
|
||||
├── logs/
|
||||
│ ├── app.log # per-subsystem structured JSON logs
|
||||
│ ├── download.log
|
||||
│ ├── transfer.log
|
||||
│ └── processing.log
|
||||
└── transfer_queue.json # persisted transfer queue state
|
||||
|
||||
\\<NAS>\...\{event}\ # mirrors the local structure once transferred
|
||||
├── 1080p/
|
||||
└── ...
|
||||
|
||||
out/
|
||||
└── {event}/
|
||||
└── {event}.mp4 # final FFmpeg output (process-only mode)
|
||||
```
|
||||
|
||||
## More documentation
|
||||
|
||||
- `DEPLOYMENT.md` — Docker/Kubernetes deployment examples and production considerations.
|
||||
- `TESTING.md` — test suite layout and how to run it (`make test`, `make test-coverage`, etc.).
|
||||
- `TODO.md` — known issues, pre-existing bugs, and roadmap items found during review, not yet addressed.
|
||||
253
TESTING.md
253
TESTING.md
@ -1,253 +0,0 @@
|
||||
# Testing Guide
|
||||
|
||||
This document describes the test suite for the StreamRecorder application.
|
||||
|
||||
## Overview
|
||||
|
||||
The test suite provides comprehensive coverage of core application components without requiring external dependencies like video files, NAS connectivity, or FFmpeg. All tests are self-contained and clean up after themselves.
|
||||
|
||||
## Test Structure
|
||||
|
||||
### Unit Tests by Package
|
||||
|
||||
#### `pkg/config`
|
||||
- **File**: `config_test.go`
|
||||
- **Coverage**: Configuration loading, environment variable override, path validation, validation errors
|
||||
- **Key Tests**:
|
||||
- Default config loading
|
||||
- Environment variable overrides
|
||||
- Path resolution and creation
|
||||
- Validation error scenarios
|
||||
|
||||
#### `pkg/utils`
|
||||
- **File**: `paths_test.go`
|
||||
- **Coverage**: Cross-platform path utilities, directory operations, validation
|
||||
- **Key Tests**:
|
||||
- Safe path joining
|
||||
- Directory creation
|
||||
- Path existence checking
|
||||
- Path validation
|
||||
- Write permission testing
|
||||
|
||||
#### `pkg/constants`
|
||||
- **File**: `constants_test.go`
|
||||
- **Coverage**: Constants values, configuration singleton, integration
|
||||
- **Key Tests**:
|
||||
- Constant value verification
|
||||
- Singleton pattern testing
|
||||
- Config integration
|
||||
- Concurrent access safety
|
||||
|
||||
#### `pkg/httpClient`
|
||||
- **File**: `error_test.go`
|
||||
- **Coverage**: HTTP error handling, status code management
|
||||
- **Key Tests**:
|
||||
- HTTP error creation and formatting
|
||||
- Error comparison and detection
|
||||
- Status code extraction
|
||||
- Error wrapping support
|
||||
|
||||
#### `pkg/media`
|
||||
- **File**: `manifest_test.go`
|
||||
- **Coverage**: Manifest generation, segment tracking, JSON serialization
|
||||
- **Key Tests**:
|
||||
- Manifest writer initialization
|
||||
- Segment addition and updates
|
||||
- Quality resolution logic
|
||||
- JSON file generation
|
||||
- Sorting and deduplication
|
||||
|
||||
#### `pkg/processing`
|
||||
- **File**: `service_test.go`
|
||||
- **Coverage**: Processing service logic, path resolution, FFmpeg handling
|
||||
- **Key Tests**:
|
||||
- Service initialization
|
||||
- Event directory scanning
|
||||
- Resolution detection
|
||||
- Segment aggregation
|
||||
- File concatenation list generation
|
||||
- FFmpeg path resolution
|
||||
|
||||
## Running Tests
|
||||
|
||||
### Quick Test Run
|
||||
```bash
|
||||
make test
|
||||
```
|
||||
|
||||
### Verbose Output
|
||||
```bash
|
||||
make test-verbose
|
||||
```
|
||||
|
||||
### Coverage Report
|
||||
```bash
|
||||
make test-coverage
|
||||
```
|
||||
Generates `coverage.html` with detailed coverage report.
|
||||
|
||||
### Test Specific Package
|
||||
```bash
|
||||
make test-pkg PKG=./pkg/config
|
||||
```
|
||||
|
||||
### Manual Test Execution
|
||||
```bash
|
||||
# Run custom test runner
|
||||
go run test_runner.go
|
||||
|
||||
# Run standard go test
|
||||
go test ./pkg/...
|
||||
|
||||
# Run with coverage
|
||||
go test -coverprofile=coverage.out ./pkg/...
|
||||
go tool cover -html=coverage.out
|
||||
```
|
||||
|
||||
## Test Features
|
||||
|
||||
### ✅ Self-Contained
|
||||
- No external file dependencies
|
||||
- No network connections required
|
||||
- No NAS or FFmpeg installation needed
|
||||
|
||||
### ✅ Automatic Cleanup
|
||||
- All temporary files/directories removed after tests
|
||||
- Original environment variables restored
|
||||
- No side effects on host system
|
||||
|
||||
### ✅ Isolated Environment
|
||||
- Tests use temporary directories
|
||||
- Environment variables safely overridden
|
||||
- Configuration isolated from production settings
|
||||
|
||||
### ✅ Cross-Platform
|
||||
- Path handling tested on Windows/Unix
|
||||
- Platform-specific behavior validated
|
||||
- Cross-platform compatibility verified
|
||||
|
||||
### ✅ Comprehensive Coverage
|
||||
- Configuration management
|
||||
- Path utilities and validation
|
||||
- Error handling patterns
|
||||
- Data structures and serialization
|
||||
- Business logic without external dependencies
|
||||
|
||||
## Test Environment
|
||||
|
||||
The test suite automatically:
|
||||
|
||||
1. **Creates Temporary Workspace**: Each test run uses a fresh temporary directory
|
||||
2. **Sets Test Environment**: Overrides environment variables to use test settings
|
||||
3. **Disables External Dependencies**: Sets flags to disable NAS transfer and processing
|
||||
4. **Cleans Up Completely**: Removes all test artifacts and restores environment
|
||||
|
||||
### Environment Variables Set During Tests
|
||||
- `LOCAL_OUTPUT_DIR`: Points to temp directory
|
||||
- `PROCESS_OUTPUT_DIR`: Points to temp directory
|
||||
- `ENABLE_NAS_TRANSFER`: Set to `false`
|
||||
- `PROCESSING_ENABLED`: Set to `false`
|
||||
|
||||
## Extending Tests
|
||||
|
||||
### Adding New Test Cases
|
||||
|
||||
1. **Create test file**: `pkg/yourpackage/yourfile_test.go`
|
||||
2. **Follow naming convention**: `TestFunctionName`
|
||||
3. **Use temp directories**: Always clean up created files
|
||||
4. **Mock external dependencies**: Avoid real file operations where possible
|
||||
|
||||
### Test Template
|
||||
```go
|
||||
package yourpackage
|
||||
|
||||
import (
|
||||
"os"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestYourFunction(t *testing.T) {
|
||||
// Setup
|
||||
tempDir, err := os.MkdirTemp("", "test_*")
|
||||
if err != nil {
|
||||
t.Fatalf("Setup failed: %v", err)
|
||||
}
|
||||
defer os.RemoveAll(tempDir)
|
||||
|
||||
// Test logic
|
||||
result := YourFunction()
|
||||
|
||||
// Assertions
|
||||
if result != expected {
|
||||
t.Errorf("Expected %v, got %v", expected, result)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Best Practices
|
||||
|
||||
- **Always clean up**: Use `defer os.RemoveAll()` for temp directories
|
||||
- **Test error cases**: Don't just test happy paths
|
||||
- **Use table-driven tests**: For multiple similar test cases
|
||||
- **Mock external dependencies**: Use echo/dummy commands instead of real tools
|
||||
- **Validate cleanup**: Ensure tests don't leave artifacts
|
||||
|
||||
## CI/CD Integration
|
||||
|
||||
The test suite is designed for automated environments:
|
||||
|
||||
```bash
|
||||
# Complete CI pipeline
|
||||
make ci
|
||||
|
||||
# Just run tests in CI
|
||||
make test
|
||||
```
|
||||
|
||||
The custom test runner provides:
|
||||
- ✅ Colored output for easy reading
|
||||
- ✅ Test count and timing statistics
|
||||
- ✅ Failure details and summaries
|
||||
- ✅ Automatic environment management
|
||||
- ✅ Exit codes for CI integration
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
**Tests fail with permission errors**
|
||||
- Ensure write permissions in temp directory
|
||||
- Check antivirus software isn't blocking file operations
|
||||
|
||||
**Config tests fail**
|
||||
- Verify no conflicting environment variables are set
|
||||
- Check that temp directories can be created
|
||||
|
||||
**Path tests fail on Windows**
|
||||
- Confirm path separator handling is correct
|
||||
- Verify Windows path validation logic
|
||||
|
||||
### Debug Mode
|
||||
```bash
|
||||
# Run with verbose output to see detailed failures
|
||||
go test -v ./pkg/...
|
||||
|
||||
# Run specific failing test
|
||||
go test -v -run TestSpecificFunction ./pkg/config
|
||||
```
|
||||
|
||||
## Coverage Goals
|
||||
|
||||
Current test coverage targets:
|
||||
- **Configuration**: 95%+ (critical for startup validation)
|
||||
- **Path utilities**: 90%+ (cross-platform compatibility critical)
|
||||
- **Constants**: 85%+ (verify all values and singleton behavior)
|
||||
- **HTTP client**: 90%+ (error handling is critical)
|
||||
- **Media handling**: 85%+ (core business logic)
|
||||
- **Processing**: 70%+ (limited by external FFmpeg dependency)
|
||||
|
||||
Generate coverage report to verify:
|
||||
```bash
|
||||
make test-coverage
|
||||
open coverage.html
|
||||
```
|
||||
111
TODO.md
111
TODO.md
@ -1,111 +0,0 @@
|
||||
# Return-from-a-year-away Audit — 2026-08-05
|
||||
|
||||
Scope reviewed: `main` (checked out), `manifest`, `cleanup` (all local + `origin` equivalents).
|
||||
Remote: `https://git.aaronic.cc/kacarmichael/StreamRecorder.git`
|
||||
|
||||
## Branch situation (read this first)
|
||||
|
||||
The three branches are **not parallel efforts** — they're one linear line of development that `main` never caught up to:
|
||||
|
||||
```
|
||||
main (0f4a39f, 2025-07-26) ──┐ stale, single-file-era downloader only
|
||||
│
|
||||
manifest (718d69c, 2025-08-10) ── +restructured cmd/pkg layout, config-less
|
||||
│
|
||||
cleanup (7d67cec, 2025-08-11) ── +pkg/config, tests, CI, Makefile, deployment docs
|
||||
```
|
||||
|
||||
`manifest` is literally an ancestor of `cleanup` (same commit `718d69c`). `cleanup` is the most complete state that ever existed and was the last thing touched (2025-08-11) before the year gap. **None of this reached `main`.** If you pick this project back up, decide this before anything else:
|
||||
|
||||
- [ ] **Merge `cleanup` → `main`** (recommended — it's a strict superset: config validation, `pkg/nas`, `pkg/processing`, unit tests, CI workflow, `DEPLOYMENT.md`/`TESTING.md`). Then delete `manifest` (fully subsumed) and re-evaluate `cleanup` for deletion once merged.
|
||||
- [ ] Or, if `cleanup`'s direction (FFmpeg concatenation, NAS-mirroring processing pipeline) is no longer where you want to go, explicitly archive it and restart `main` from a clean design — but do this as a decision, not by default.
|
||||
|
||||
---
|
||||
|
||||
## 🔴 Critical — fix/rotate before running this again
|
||||
|
||||
1. **Leaked NAS credentials, committed in plaintext.** `pkg/config/config.go` (branch `cleanup`) hardcodes a real default:
|
||||
```go
|
||||
Username: "NASAdmin",
|
||||
Password: "s3tkY6tzA&KN6M",
|
||||
```
|
||||
This is in git history on a remote (`git.aaronic.cc`) regardless of whether you later change the default. **Rotate this NAS account's password now** — changing the code doesn't un-leak a password that's sat in a repo for a year. Then remove the credential entirely (fail closed: require `NAS_USERNAME`/`NAS_PASSWORD` env vars with no default, error out if unset when transfer is enabled) rather than shipping a working default.
|
||||
|
||||
2. **Guaranteed crash: double mutex unlock in `pkg/transfer/cleanup.go`.** In `ExecuteCleanup` (present on **`main` too**, not just `cleanup`):
|
||||
```go
|
||||
cs.mu.Lock()
|
||||
...
|
||||
cs.mu.Unlock() // line 72 — releases the lock
|
||||
...
|
||||
batch := make([]string, batchSize)
|
||||
copy(batch, cs.pendingFiles[:batchSize])
|
||||
cs.pendingFiles = cs.pendingFiles[batchSize:] // mutated WITHOUT the lock held
|
||||
cs.mu.Unlock() // line 79 — unlocking an already-unlocked mutex
|
||||
```
|
||||
An unlock of an unlocked `sync.Mutex` is a **fatal, unrecoverable runtime error** (`fatal error: sync: unlock of unlocked mutex`) — it kills the whole process, not just a goroutine. Since `Cleanup.AfterTransfer` defaults to `true` and `RetainHours: 0` (immediate eligibility), this fires the first time any file finishes transferring and cleanup runs. This is very likely why any end-to-end run with NAS transfer enabled never survived long. Fix: hold the lock for the whole slice mutation, or copy-then-release-then-mutate-under-a-second-short-lock, but don't unlock twice.
|
||||
|
||||
3. **Nil-pointer crash in `pkg/media/stream.go` (`VariantDownloader`, `main` branch):**
|
||||
```go
|
||||
playlist, err := LoadMediaPlaylist(variant.URL)
|
||||
seq := playlist.SeqNo // <-- dereferenced before the err check
|
||||
if err != nil {
|
||||
...
|
||||
goto waitTick
|
||||
}
|
||||
```
|
||||
`LoadMediaPlaylist` returns `nil, err` on any HTTP/parse failure, so `playlist` is a typed-nil pointer here — `playlist.SeqNo` panics. A panic in a goroutine takes down the entire process (Go doesn't isolate goroutine panics). Practically: any single transient network hiccup on any variant's playlist fetch kills the whole download session, mid-recording. Move the `err != nil` check above the `SeqNo` read.
|
||||
|
||||
---
|
||||
|
||||
## 🟠 High-priority bugs / correctness
|
||||
|
||||
4. **`PriorityQueue.Less` in `pkg/transfer/queue.go` sorts newest-first** (`pq[i].Timestamp.After(pq[j].Timestamp)`), making the transfer queue LIFO. For a live-recording pipeline that keeps producing new segments, this means the *oldest* queued segments can be starved indefinitely as long as new ones keep arriving — the opposite of what you'd want (drain oldest first so nothing waits forever). Worth confirming this was intentional; if not, flip the comparison.
|
||||
|
||||
5. **Retry backoff runs before the *first* attempt, not just retries** — `pkg/transfer/queue.go`, `processItem`:
|
||||
```go
|
||||
for attempt := 1; attempt <= maxRetries; attempt++ {
|
||||
if attempt > 0 { // always true, since attempt starts at 1
|
||||
... backoff ...
|
||||
```
|
||||
Every single transfer sleeps `1s` before even trying once. Should be `attempt > 1`.
|
||||
|
||||
6. **`pkg/constants/constants.go` (main) hardcodes a live FloSports auth token** directly in source (`MasterURL` with `token=...`, `hmac2=...`). It's expired now (epoch ~2025-07), so no live exposure, but it's the same pattern as #1 — secrets/tokens don't belong in source. Worth a rule going forward: URLs/tokens come from CLI args or env, never constants.
|
||||
|
||||
7. **`http.Client{}` created with no timeout** in `pkg/media/stream.go`, `playlist.go`, and `segment.go`. A stalled TCP connection (dead CDN edge, network partition) can hang a fetch forever — nothing in the surrounding code will time it out except the 10s context wrapped specifically around segment downloads in `VariantDownloader`. Playlist-fetch calls (`GetAllVariants`, `LoadMediaPlaylist`) have no such wrapper. Give the shared client a sane `Timeout` (e.g. 15–30s) as a backstop.
|
||||
|
||||
---
|
||||
|
||||
## 🟡 Cleanup / dead weight
|
||||
|
||||
8. **`cmd/proc/main.go` is a one-line stub** (`package proc`) on `main` — dead entry point, no `func main()`. Either finish it or delete it; right now `go build ./...` would fail on this if it were ever wired as a real `main` package (it currently silently isn't, which is its own trap for future-you).
|
||||
9. **Built binaries committed to git** (`bin/flo_download`, `bin/flo_download.exe`, ~18MB, on `cleanup`/`manifest`). The `.gitea/workflows/build.yml` already cross-compiles and uploads artifacts — there's no reason binaries should also live in the repo/history. Remove and add `bin/` to `.gitignore`.
|
||||
10. **`scratch.txt` is tracked despite being in `.gitignore`.** Adding a path to `.gitignore` after it's already tracked does nothing — it's still in the repo (and it's just a raw debug log dump). `git rm --cached scratch.txt`.
|
||||
11. `.idea/*` (JetBrains project files) are committed. Low stakes solo-repo, but standard practice is to gitignore the whole `.idea/` directory rather than cherry-pick.
|
||||
|
||||
---
|
||||
|
||||
## 🔵 Program / architecture — bigger picture
|
||||
|
||||
12. **The project's own `CLAUDE.md` states the goal is "a listening REST API [that] take[s] in m3u8 urls, parse[s] them, and eventually send[s] to a conversion service."** None of the three branches implement any HTTP server — all three are CLI-only, one-shot-per-invocation tools. This is the largest gap between stated intent and actual code. Worth deciding: is the REST-API framing still the goal, or has the project's actual direction (CLI recorder + NAS mirror + FFmpeg concat, per `cleanup`'s `CLAUDE.md`) superseded it? Right now the docs and the code disagree with each other.
|
||||
13. **`pkg/nas/nas.go`'s "connection" is a Windows-only shell-out to `net use`** (`EstablishConnection`), while CI (`build.yml`) cross-compiles for `linux`/`darwin`/`windows`. On non-Windows the UNC-path logic (`\\server\share`) and `net use` simply don't apply — there's no SMB client abstraction, so the Linux/macOS builds can produce a binary that can't actually do NAS transfer. If cross-platform deployment is real (not just "it happens to compile"), this needs a real cross-platform SMB/CIFS story (e.g., assume the NAS is pre-mounted by the OS and just treat it as a local path, which is actually simpler and already half-true for the Linux CIFS example in `DEPLOYMENT.md`).
|
||||
14. **Zero tests on `main`.** `cleanup` has real coverage (`pkg/config`, `pkg/utils`, `pkg/constants`, `pkg/httpClient`, `pkg/media/manifest`, `pkg/processing`) plus a `Makefile` (`test`, `test-coverage`, `lint`, `security` via `gosec`) and a `TESTING.md`. All of this is stuck on a branch that was never merged — it's not a "write tests" TODO so much as a "land the tests you already wrote" TODO.
|
||||
15. **CI never actually runs tests, lint, or `gosec`** despite the `Makefile` defining a full `ci` target (`fmt tidy lint security test build`). `.gitea/workflows/build.yml` only cross-compiles. Wire `make ci` (or its steps) into the pipeline so a broken build/test doesn't silently ship a binary artifact.
|
||||
16. **Unbounded `seen` map** in `VariantDownloader` (`pkg/media/stream.go`) grows for the lifetime of a variant's polling loop with no eviction — fine for a few-hour event, worth a note if this is ever meant to run continuously/24-7.
|
||||
|
||||
---
|
||||
|
||||
## 🟢 Deployment
|
||||
|
||||
17. `DEPLOYMENT.md` (on `cleanup`) documents Docker/Compose/Kubernetes deployment in detail, but there's no actual `Dockerfile` committed anywhere in the repo — it's a doc-only example. If containerized deployment is the real target, commit the Dockerfile and at least smoke-test the build.
|
||||
18. No secrets-management story beyond "use env vars" — fine for a home lab, but the doc's own "Security" section says "use secret management systems," which isn't backed by anything concrete (no `.env.example`, no sealed-secrets/vault reference). Given finding #1, this is worth actually doing, not just documenting.
|
||||
19. Health checks, metrics/log shipping are listed as "considerations" in `DEPLOYMENT.md` but nothing is implemented — the app is a fire-and-forget CLI process with `log.Printf` to stdout only. If this runs unattended (e.g., in a container watching for the next event), you have no way to know it's alive short of watching logs.
|
||||
|
||||
---
|
||||
|
||||
## Suggested order of attack
|
||||
|
||||
1. Rotate the NAS credential (#1) — this is time-sensitive regardless of anything else.
|
||||
2. Fix the two crash bugs (#2, #3) — nothing else is worth doing until a full recording session can survive.
|
||||
3. Decide the branch-merge question and land `cleanup` → `main` (or consciously don't).
|
||||
4. Wire CI to actually test/lint (#15) so regressions get caught going forward.
|
||||
5. Then tackle the architecture question (#12: REST API vs. CLI-recorder-as-is) before investing more in either direction.
|
||||
BIN
bin/flo_download
Executable file → Normal file
BIN
bin/flo_download
Executable file → Normal file
Binary file not shown.
BIN
bin/flo_download.exe
Normal file
BIN
bin/flo_download.exe
Normal file
Binary file not shown.
@ -1,284 +0,0 @@
|
||||
// Package check implements connection-check mode: it exercises the external
|
||||
// dependencies a recording needs — the stream host and the NAS share — and
|
||||
// reports pass/fail for each without downloading or transferring anything.
|
||||
// Same spirit as `dbt debug`: prove the wiring before starting real work.
|
||||
package check
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"log"
|
||||
"m3u8-downloader/pkg/config"
|
||||
"m3u8-downloader/pkg/constants"
|
||||
"m3u8-downloader/pkg/httpClient"
|
||||
"m3u8-downloader/pkg/logging"
|
||||
"m3u8-downloader/pkg/media"
|
||||
"m3u8-downloader/pkg/nas"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/grafov/m3u8"
|
||||
)
|
||||
|
||||
// playlistTimeout bounds the master playlist fetch. Deliberately short: a
|
||||
// stream host that takes longer than this is not usable for live recording.
|
||||
const playlistTimeout = 15 * time.Second
|
||||
|
||||
// Run executes every applicable check and returns a process exit code, 0 only
|
||||
// when nothing failed, so the command can gate a scripted recording start.
|
||||
func Run(masterURL string) int {
|
||||
fmt.Println("Running connection checks...")
|
||||
fmt.Println()
|
||||
|
||||
cfg, err := constants.GetConfig()
|
||||
if err != nil {
|
||||
reportFail("configuration", nil, err)
|
||||
fmt.Println()
|
||||
fmt.Println("1 check failed.")
|
||||
return 1
|
||||
}
|
||||
|
||||
appLog := logging.MustGetFactory().App
|
||||
appLog.Info("connection check starting", "url_provided", masterURL != "")
|
||||
|
||||
reportPass("configuration", []string{
|
||||
"local output: " + cfg.Paths.LocalOutput,
|
||||
"process output: " + cfg.Paths.ProcessOutput,
|
||||
"log dir: " + cfg.Logging.Dir,
|
||||
fmt.Sprintf("nas transfer: %s", enabledLabel(cfg.NAS.EnableTransfer)),
|
||||
})
|
||||
|
||||
failures := 0
|
||||
|
||||
if masterURL == "" {
|
||||
reportSkip("stream playlist", "no -url provided")
|
||||
} else if details, err := checkMasterPlaylist(masterURL); err != nil {
|
||||
reportFail("stream playlist", details, err)
|
||||
appLog.Error("stream playlist check failed", logging.KeyError, err.Error())
|
||||
failures++
|
||||
} else {
|
||||
reportPass("stream playlist", details)
|
||||
appLog.Info("stream playlist check passed")
|
||||
}
|
||||
|
||||
if !cfg.NAS.EnableTransfer {
|
||||
reportSkip("nas storage", "transfer disabled in configuration")
|
||||
} else if details, err := checkNAS(cfg); err != nil {
|
||||
reportFail("nas storage", details, err)
|
||||
appLog.Error("nas check failed", logging.KeyError, err.Error())
|
||||
failures++
|
||||
} else {
|
||||
reportPass("nas storage", details)
|
||||
appLog.Info("nas check passed", "path", cfg.NAS.OutputPath)
|
||||
}
|
||||
|
||||
if failures > 0 {
|
||||
fmt.Printf("%d check(s) failed.\n", failures)
|
||||
appLog.Error("connection check finished with failures", "failures", failures)
|
||||
return 1
|
||||
}
|
||||
|
||||
fmt.Println("All checks passed.")
|
||||
appLog.Info("connection check passed")
|
||||
return 0
|
||||
}
|
||||
|
||||
// checkMasterPlaylist fetches the master playlist exactly as the downloader
|
||||
// would — same headers, same decoder — and summarizes what it found. Details
|
||||
// gathered before the failure point are returned alongside the error so a
|
||||
// failed check still shows how far it got.
|
||||
func checkMasterPlaylist(masterURL string) ([]string, error) {
|
||||
parsed, err := url.Parse(masterURL)
|
||||
if err != nil || parsed.Scheme == "" || parsed.Host == "" {
|
||||
return nil, fmt.Errorf("not a valid absolute URL: %q", masterURL)
|
||||
}
|
||||
|
||||
details := []string{"host: " + parsed.Host}
|
||||
|
||||
ctx, cancel := context.WithTimeout(context.Background(), playlistTimeout)
|
||||
defer cancel()
|
||||
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodGet, masterURL, nil)
|
||||
if err != nil {
|
||||
return details, fmt.Errorf("failed to build request: %w", err)
|
||||
}
|
||||
req.Header.Set("User-Agent", constants.HTTPUserAgent)
|
||||
req.Header.Set("Referer", constants.REFERRER)
|
||||
|
||||
client := &http.Client{Timeout: playlistTimeout}
|
||||
start := time.Now()
|
||||
resp, err := client.Do(req)
|
||||
if err != nil {
|
||||
return details, fmt.Errorf("request failed: %w", err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
|
||||
details = append(details, fmt.Sprintf("HTTP %d in %s", resp.StatusCode, formatDuration(time.Since(start))))
|
||||
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return details, httpClient.NewHTTPError(resp.StatusCode, "unexpected status fetching master playlist")
|
||||
}
|
||||
|
||||
playlist, listType, err := m3u8.DecodeFrom(resp.Body, true)
|
||||
if err != nil {
|
||||
return details, fmt.Errorf("response is not a valid m3u8 playlist: %w", err)
|
||||
}
|
||||
|
||||
switch listType {
|
||||
case m3u8.MASTER:
|
||||
master, ok := playlist.(*m3u8.MasterPlaylist)
|
||||
if !ok {
|
||||
return details, fmt.Errorf("decoder returned an unexpected playlist type")
|
||||
}
|
||||
if len(master.Variants) == 0 {
|
||||
return details, fmt.Errorf("master playlist contains no variants")
|
||||
}
|
||||
details = append(details, fmt.Sprintf("master playlist with %d variant(s):", len(master.Variants)))
|
||||
for _, v := range master.Variants {
|
||||
if v == nil {
|
||||
continue
|
||||
}
|
||||
details = append(details, fmt.Sprintf(" %s (%d bps)", media.ExtractResolution(v), v.Bandwidth))
|
||||
}
|
||||
case m3u8.MEDIA:
|
||||
mediaPlaylist, ok := playlist.(*m3u8.MediaPlaylist)
|
||||
if !ok {
|
||||
return details, fmt.Errorf("decoder returned an unexpected playlist type")
|
||||
}
|
||||
details = append(details,
|
||||
fmt.Sprintf("media playlist with %d segment(s), recorded as a single \"unknown\" variant", mediaPlaylist.Count()))
|
||||
default:
|
||||
return details, fmt.Errorf("unrecognized playlist type")
|
||||
}
|
||||
|
||||
return details, nil
|
||||
}
|
||||
|
||||
// checkNAS runs the same attach/mkdir/write sequence the transfer service
|
||||
// performs at startup, so the check exercises the real code path for this host
|
||||
// — net use on Windows, mount resolution on Linux — rather than a simulation of
|
||||
// it. It disconnects afterwards so the check leaves no mount behind.
|
||||
func checkNAS(cfg *config.Config) ([]string, error) {
|
||||
svc, err := nas.New(nas.NASConfig{
|
||||
Path: cfg.NAS.OutputPath,
|
||||
Username: cfg.NAS.Username,
|
||||
Password: cfg.NAS.Password,
|
||||
Timeout: cfg.NAS.Timeout,
|
||||
RetryLimit: cfg.NAS.RetryLimit,
|
||||
VerifySize: true,
|
||||
AutoMount: cfg.NAS.AutoMount,
|
||||
MountPoint: cfg.NAS.MountPoint,
|
||||
})
|
||||
if err != nil {
|
||||
return []string{"path: " + cfg.NAS.OutputPath}, err
|
||||
}
|
||||
|
||||
start := time.Now()
|
||||
result := <-probeAsync(cfg.NAS.Timeout, func() ([]string, error) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), cfg.NAS.Timeout)
|
||||
defer cancel()
|
||||
|
||||
if err := svc.Connect(ctx); err != nil {
|
||||
return svc.ConnectionInfo(), err
|
||||
}
|
||||
defer func() {
|
||||
if err := svc.Disconnect(context.Background()); err != nil {
|
||||
log.Printf("Warning: failed to release NAS connection after check: %v", err)
|
||||
}
|
||||
}()
|
||||
|
||||
return append(svc.ConnectionInfo(), "attach, mkdir and write probe succeeded"), nil
|
||||
})
|
||||
|
||||
details := result.details
|
||||
if len(details) == 0 {
|
||||
details = svc.ConnectionInfo()
|
||||
}
|
||||
if result.err != nil {
|
||||
return details, result.err
|
||||
}
|
||||
|
||||
return append(details, "completed in "+formatDuration(time.Since(start))), nil
|
||||
}
|
||||
|
||||
func formatDuration(d time.Duration) string {
|
||||
if d < time.Millisecond {
|
||||
return "<1ms"
|
||||
}
|
||||
return d.Round(time.Millisecond).String()
|
||||
}
|
||||
|
||||
type probeResult struct {
|
||||
details []string
|
||||
err error
|
||||
}
|
||||
|
||||
// probeAsync bounds a filesystem probe with a timeout. An unresponsive SMB
|
||||
// mount can block indefinitely in the kernel, which would otherwise hang the
|
||||
// check with no output; the orphaned goroutine dies with the process.
|
||||
func probeAsync(timeout time.Duration, probe func() ([]string, error)) <-chan probeResult {
|
||||
out := make(chan probeResult, 1)
|
||||
done := make(chan probeResult, 1)
|
||||
|
||||
go func() {
|
||||
details, err := probe()
|
||||
done <- probeResult{details: details, err: err}
|
||||
}()
|
||||
|
||||
go func() {
|
||||
select {
|
||||
case res := <-done:
|
||||
out <- res
|
||||
case <-time.After(timeout):
|
||||
out <- probeResult{err: fmt.Errorf("timed out after %s", timeout)}
|
||||
}
|
||||
}()
|
||||
|
||||
return out
|
||||
}
|
||||
|
||||
func reportPass(name string, details []string) {
|
||||
fmt.Printf("✓ %s\n", name)
|
||||
printDetails(details)
|
||||
}
|
||||
|
||||
func reportFail(name string, details []string, err error) {
|
||||
fmt.Printf("✗ %s\n", name)
|
||||
printDetails(append(details, "error: "+err.Error()))
|
||||
}
|
||||
|
||||
func reportSkip(name, reason string) {
|
||||
fmt.Printf("- %s (skipped: %s)\n", name, reason)
|
||||
fmt.Println()
|
||||
}
|
||||
|
||||
// maxDetailLines caps a single detail entry. Failures from `net use` and
|
||||
// friends can carry a full usage dump, which buries the actual diagnosis.
|
||||
const maxDetailLines = 8
|
||||
|
||||
func printDetails(details []string) {
|
||||
for _, d := range details {
|
||||
lines := strings.Split(strings.TrimRight(d, "\n"), "\n")
|
||||
truncated := 0
|
||||
if len(lines) > maxDetailLines {
|
||||
truncated = len(lines) - maxDetailLines
|
||||
lines = lines[:maxDetailLines]
|
||||
}
|
||||
for _, line := range lines {
|
||||
fmt.Printf(" %s\n", line)
|
||||
}
|
||||
if truncated > 0 {
|
||||
fmt.Printf(" ... (%d more line(s))\n", truncated)
|
||||
}
|
||||
}
|
||||
fmt.Println()
|
||||
}
|
||||
|
||||
func enabledLabel(enabled bool) string {
|
||||
if enabled {
|
||||
return "enabled"
|
||||
}
|
||||
return "disabled"
|
||||
}
|
||||
@ -1,220 +0,0 @@
|
||||
package check
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"m3u8-downloader/pkg/config"
|
||||
"m3u8-downloader/pkg/httpClient"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
const masterPlaylistBody = `#EXTM3U
|
||||
#EXT-X-STREAM-INF:BANDWIDTH=5500000,RESOLUTION=1920x1080
|
||||
1080p/chunk.m3u8
|
||||
#EXT-X-STREAM-INF:BANDWIDTH=3200000,RESOLUTION=1280x720
|
||||
720p/chunk.m3u8
|
||||
`
|
||||
|
||||
const mediaPlaylistBody = `#EXTM3U
|
||||
#EXT-X-TARGETDURATION:4
|
||||
#EXT-X-MEDIA-SEQUENCE:10
|
||||
#EXTINF:4.0,
|
||||
seg1.ts
|
||||
#EXTINF:4.0,
|
||||
seg2.ts
|
||||
`
|
||||
|
||||
// playlistServer serves body at /playlist.m3u8 with the given status, and
|
||||
// records the request headers so we can assert the check sends the same ones
|
||||
// the real downloader does.
|
||||
func playlistServer(t *testing.T, status int, body string) (*httptest.Server, *http.Header) {
|
||||
t.Helper()
|
||||
var got http.Header
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
got = r.Header.Clone()
|
||||
w.WriteHeader(status)
|
||||
fmt.Fprint(w, body)
|
||||
}))
|
||||
t.Cleanup(srv.Close)
|
||||
return srv, &got
|
||||
}
|
||||
|
||||
func joinDetails(details []string) string {
|
||||
return strings.Join(details, "\n")
|
||||
}
|
||||
|
||||
func TestCheckMasterPlaylistMaster(t *testing.T) {
|
||||
srv, headers := playlistServer(t, http.StatusOK, masterPlaylistBody)
|
||||
|
||||
details, err := checkMasterPlaylist(srv.URL + "/playlist.m3u8")
|
||||
if err != nil {
|
||||
t.Fatalf("expected master playlist check to pass, got: %v", err)
|
||||
}
|
||||
|
||||
joined := joinDetails(details)
|
||||
for _, want := range []string{"master playlist with 2 variant(s)", "1080p (5500000 bps)", "720p (3200000 bps)", "HTTP 200"} {
|
||||
if !strings.Contains(joined, want) {
|
||||
t.Errorf("details missing %q:\n%s", want, joined)
|
||||
}
|
||||
}
|
||||
|
||||
if ua := headers.Get("User-Agent"); !strings.Contains(ua, "Mozilla/5.0") {
|
||||
t.Errorf("expected downloader User-Agent to be sent, got %q", ua)
|
||||
}
|
||||
if ref := headers.Get("Referer"); ref == "" {
|
||||
t.Error("expected Referer header to be sent")
|
||||
}
|
||||
}
|
||||
|
||||
func TestCheckMasterPlaylistMedia(t *testing.T) {
|
||||
srv, _ := playlistServer(t, http.StatusOK, mediaPlaylistBody)
|
||||
|
||||
details, err := checkMasterPlaylist(srv.URL + "/playlist.m3u8")
|
||||
if err != nil {
|
||||
t.Fatalf("expected media playlist check to pass, got: %v", err)
|
||||
}
|
||||
|
||||
if joined := joinDetails(details); !strings.Contains(joined, "media playlist with 2 segment(s)") {
|
||||
t.Errorf("expected media playlist segment count in details:\n%s", joined)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCheckMasterPlaylistNon200(t *testing.T) {
|
||||
srv, _ := playlistServer(t, http.StatusForbidden, "denied")
|
||||
|
||||
details, err := checkMasterPlaylist(srv.URL + "/playlist.m3u8")
|
||||
if err == nil {
|
||||
t.Fatal("expected a 403 response to fail the check")
|
||||
}
|
||||
if code := httpClient.GetHTTPStatusCode(err); code != http.StatusForbidden {
|
||||
t.Errorf("expected a structured HTTP 403 error, got code %d from %v", code, err)
|
||||
}
|
||||
// Details collected before the failure should still be reported.
|
||||
if joined := joinDetails(details); !strings.Contains(joined, "HTTP 403") {
|
||||
t.Errorf("expected partial details on failure:\n%s", joined)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCheckMasterPlaylistNotAPlaylist(t *testing.T) {
|
||||
srv, _ := playlistServer(t, http.StatusOK, "this is not a playlist")
|
||||
|
||||
if _, err := checkMasterPlaylist(srv.URL + "/playlist.m3u8"); err == nil {
|
||||
t.Fatal("expected a non-m3u8 body to fail the check")
|
||||
}
|
||||
}
|
||||
|
||||
func TestCheckMasterPlaylistEmptyMaster(t *testing.T) {
|
||||
srv, _ := playlistServer(t, http.StatusOK, "#EXTM3U\n")
|
||||
|
||||
if _, err := checkMasterPlaylist(srv.URL + "/playlist.m3u8"); err == nil {
|
||||
t.Fatal("expected a playlist with no variants or segments to fail the check")
|
||||
}
|
||||
}
|
||||
|
||||
func TestCheckMasterPlaylistInvalidURL(t *testing.T) {
|
||||
for _, bad := range []string{"", "notaurl", "/relative/path.m3u8", "://missing-scheme"} {
|
||||
if _, err := checkMasterPlaylist(bad); err == nil {
|
||||
t.Errorf("expected %q to be rejected as an invalid URL", bad)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestCheckMasterPlaylistUnreachable(t *testing.T) {
|
||||
// Port 9 (discard) refuses connections without a listener.
|
||||
if _, err := checkMasterPlaylist("http://127.0.0.1:9/playlist.m3u8"); err == nil {
|
||||
t.Fatal("expected an unreachable host to fail the check")
|
||||
}
|
||||
}
|
||||
|
||||
func writeFile(path string) error {
|
||||
return os.WriteFile(path, []byte("blocker"), 0644)
|
||||
}
|
||||
|
||||
func nasConfig(path string) *config.Config {
|
||||
cfg := &config.Config{}
|
||||
cfg.NAS.EnableTransfer = true
|
||||
cfg.NAS.OutputPath = path
|
||||
cfg.NAS.Timeout = 10 * time.Second
|
||||
cfg.NAS.RetryLimit = 3
|
||||
return cfg
|
||||
}
|
||||
|
||||
func TestCheckNASLocalPath(t *testing.T) {
|
||||
target := filepath.Join(t.TempDir(), "streams")
|
||||
|
||||
details, err := checkNAS(nasConfig(target))
|
||||
if err != nil {
|
||||
t.Fatalf("expected a writable local path to pass the NAS check, got: %v", err)
|
||||
}
|
||||
|
||||
joined := joinDetails(details)
|
||||
for _, want := range []string{"configured: " + target, "strategy: local", "write probe succeeded"} {
|
||||
if !strings.Contains(joined, want) {
|
||||
t.Errorf("details missing %q:\n%s", want, joined)
|
||||
}
|
||||
}
|
||||
// The probe must not leave its test file behind.
|
||||
if matches, _ := filepath.Glob(filepath.Join(target, ".connection_test")); len(matches) != 0 {
|
||||
t.Errorf("connection test file was not cleaned up: %v", matches)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCheckNASUnusableTarget(t *testing.T) {
|
||||
// A regular file cannot be turned into the destination directory.
|
||||
blocker := filepath.Join(t.TempDir(), "not-a-dir")
|
||||
if err := writeFile(blocker); err != nil {
|
||||
t.Fatalf("failed to set up test file: %v", err)
|
||||
}
|
||||
|
||||
details, err := checkNAS(nasConfig(filepath.Join(blocker, "streams")))
|
||||
if err == nil {
|
||||
t.Fatal("expected an unusable NAS target to fail the check")
|
||||
}
|
||||
if joined := joinDetails(details); !strings.Contains(joined, "configured: ") {
|
||||
t.Errorf("expected path detail on failure:\n%s", joined)
|
||||
}
|
||||
}
|
||||
|
||||
func TestProbeAsyncTimesOut(t *testing.T) {
|
||||
start := time.Now()
|
||||
result := <-probeAsync(50*time.Millisecond, func() ([]string, error) {
|
||||
time.Sleep(5 * time.Second)
|
||||
return nil, nil
|
||||
})
|
||||
|
||||
if result.err == nil {
|
||||
t.Fatal("expected a hung probe to report a timeout")
|
||||
}
|
||||
if !strings.Contains(result.err.Error(), "timed out") {
|
||||
t.Errorf("expected a timeout error, got: %v", result.err)
|
||||
}
|
||||
if elapsed := time.Since(start); elapsed > time.Second {
|
||||
t.Errorf("probe took %s, expected it to abandon the hung call promptly", elapsed)
|
||||
}
|
||||
}
|
||||
|
||||
func TestProbeAsyncPassesThroughResults(t *testing.T) {
|
||||
sentinel := errors.New("probe failed")
|
||||
|
||||
if result := <-probeAsync(time.Second, func() ([]string, error) { return []string{"ok"}, nil }); result.err != nil || len(result.details) != 1 {
|
||||
t.Errorf("expected success to pass through, got details=%v err=%v", result.details, result.err)
|
||||
}
|
||||
if result := <-probeAsync(time.Second, func() ([]string, error) { return nil, sentinel }); !errors.Is(result.err, sentinel) {
|
||||
t.Errorf("expected probe error to pass through, got: %v", result.err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestFormatDuration(t *testing.T) {
|
||||
if got := formatDuration(100 * time.Microsecond); got != "<1ms" {
|
||||
t.Errorf("sub-millisecond duration = %q, want \"<1ms\"", got)
|
||||
}
|
||||
if got := formatDuration(1500 * time.Millisecond); got != "1.5s" {
|
||||
t.Errorf("1.5s duration = %q", got)
|
||||
}
|
||||
}
|
||||
@ -4,11 +4,8 @@ import (
|
||||
"context"
|
||||
"log"
|
||||
"m3u8-downloader/pkg/constants"
|
||||
"m3u8-downloader/pkg/health"
|
||||
"m3u8-downloader/pkg/logging"
|
||||
"m3u8-downloader/pkg/media"
|
||||
"m3u8-downloader/pkg/transfer"
|
||||
"m3u8-downloader/pkg/utils"
|
||||
"os"
|
||||
"os/signal"
|
||||
"sync"
|
||||
@ -17,8 +14,6 @@ import (
|
||||
)
|
||||
|
||||
func Download(masterURL string, eventName string, debug bool) {
|
||||
appLog := logging.MustGetFactory().App
|
||||
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
defer cancel()
|
||||
|
||||
@ -28,34 +23,16 @@ func Download(masterURL string, eventName string, debug bool) {
|
||||
go func() {
|
||||
<-sigChan
|
||||
log.Println("Shutting down...")
|
||||
appLog.Info("shutdown signal received")
|
||||
cancel()
|
||||
}()
|
||||
|
||||
cfg := constants.MustGetConfig()
|
||||
|
||||
if cfg.Health.Enabled {
|
||||
healthSrv, err := health.Start(ctx, &cfg.Health, health.MustGetRegistry())
|
||||
if err != nil {
|
||||
appLog.Error("failed to start health server", logging.KeyError, err.Error())
|
||||
} else {
|
||||
appLog.Info("health server listening", "addr", cfg.Health.Addr)
|
||||
defer func() {
|
||||
shutdownCtx, shutdownCancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer shutdownCancel()
|
||||
_ = healthSrv.Shutdown(shutdownCtx)
|
||||
}()
|
||||
}
|
||||
}
|
||||
|
||||
var wg sync.WaitGroup
|
||||
var transferService *transfer.TransferService
|
||||
if cfg.NAS.EnableTransfer {
|
||||
ts, err := transfer.NewTrasferService(cfg.NAS.OutputPath, eventName)
|
||||
if constants.EnableNASTransfer {
|
||||
ts, err := transfer.NewTrasferService(constants.NASOutputPath, eventName)
|
||||
if err != nil {
|
||||
log.Printf("Failed to create transfer service: %v", err)
|
||||
log.Println("Continuing without transfer service...")
|
||||
appLog.Error("failed to create transfer service, continuing without it", logging.KeyError, err.Error())
|
||||
} else {
|
||||
transferService = ts
|
||||
wg.Add(1)
|
||||
@ -63,33 +40,24 @@ func Download(masterURL string, eventName string, debug bool) {
|
||||
defer wg.Done()
|
||||
if err := transferService.Start(ctx); err != nil && err != context.Canceled {
|
||||
log.Printf("Transfer service error: %v", err)
|
||||
appLog.Error("transfer service error", logging.KeyError, err.Error())
|
||||
}
|
||||
}()
|
||||
log.Println("Transfer service started.")
|
||||
appLog.Info("transfer service started")
|
||||
}
|
||||
}
|
||||
|
||||
// One writer for the whole run: every variant records into it, and it is
|
||||
// the one written to disk at the end.
|
||||
manifest := media.NewManifestWriter(eventName)
|
||||
manifestWriter := media.NewManifestWriter(eventName)
|
||||
|
||||
eventPath := cfg.GetEventPath(eventName)
|
||||
if err := utils.EnsureDir(eventPath); err != nil {
|
||||
appLog.Error("failed to create event directory", logging.KeyError, err.Error())
|
||||
log.Fatalf("Failed to create event directory: %v", err)
|
||||
}
|
||||
|
||||
variants, err := media.GetAllVariants(masterURL, eventPath)
|
||||
variants, err := media.GetAllVariants(masterURL, constants.LocalOutputDirPath+"/"+eventName, manifestWriter)
|
||||
if err != nil {
|
||||
appLog.Error("failed to get variants", logging.KeyError, err.Error())
|
||||
log.Fatalf("Failed to get variants: %v", err)
|
||||
}
|
||||
log.Printf("Found %d variants", len(variants))
|
||||
|
||||
sem := make(chan struct{}, constants.WorkerCount*len(variants))
|
||||
|
||||
manifest := media.NewManifestWriter(eventName)
|
||||
|
||||
for _, variant := range variants {
|
||||
// Debug mode only tracks one variant for easier debugging
|
||||
if debug {
|
||||
@ -106,7 +74,6 @@ func Download(masterURL string, eventName string, debug bool) {
|
||||
|
||||
wg.Wait()
|
||||
log.Println("All variant downloaders finished.")
|
||||
appLog.Info("all variant downloaders finished")
|
||||
|
||||
if transferService != nil {
|
||||
shutdownCtx, shutdownCancel := context.WithTimeout(context.Background(), 30*time.Second)
|
||||
@ -115,9 +82,7 @@ func Download(masterURL string, eventName string, debug bool) {
|
||||
}
|
||||
|
||||
log.Println("All Services shut down.")
|
||||
appLog.Info("all services shut down")
|
||||
|
||||
manifest.WriteManifest()
|
||||
manifestWriter.WriteManifest()
|
||||
log.Println("Manifest written.")
|
||||
appLog.Info("manifest written", logging.KeyEvent, eventName, "segments", manifest.Len())
|
||||
}
|
||||
|
||||
@ -4,38 +4,23 @@ import (
|
||||
"bufio"
|
||||
"flag"
|
||||
"fmt"
|
||||
"m3u8-downloader/cmd/check"
|
||||
"m3u8-downloader/cmd/downloader"
|
||||
"m3u8-downloader/cmd/processor"
|
||||
"m3u8-downloader/cmd/transfer"
|
||||
"m3u8-downloader/pkg/constants"
|
||||
"m3u8-downloader/pkg/logging"
|
||||
"os"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
func main() {
|
||||
url := flag.String("url", "", "M3U8 playlist URL")
|
||||
eventName := flag.String("event", "", "Event name")
|
||||
eventName := flag.String("event", time.Now().Format("2006-01-02"), "Event name")
|
||||
debug := flag.Bool("debug", false, "Enable debug mode")
|
||||
transferOnly := flag.Bool("transfer", false, "Transfer-only mode: transfer existing files without downloading")
|
||||
processOnly := flag.Bool("process", false, "Process-only mode: process existing files without downloading")
|
||||
checkOnly := flag.Bool("check", false, "Connection-check mode: test the stream URL and NAS connectivity, then exit")
|
||||
|
||||
flag.Parse()
|
||||
|
||||
// Must happen before the first logging.MustGetFactory() call anywhere in
|
||||
// the program, since the logging factory reads its level from this
|
||||
// config singleton once and caches it.
|
||||
if *debug {
|
||||
constants.MustGetConfig().Logging.Level = "debug"
|
||||
}
|
||||
logging.MustGetFactory().App.Info("m3u8-downloader starting", "debug", *debug)
|
||||
|
||||
if *checkOnly {
|
||||
os.Exit(check.Run(*url))
|
||||
}
|
||||
|
||||
if *transferOnly {
|
||||
transfer.RunTransferOnly(*eventName)
|
||||
return
|
||||
|
||||
@ -3,40 +3,16 @@ package processor
|
||||
import (
|
||||
"context"
|
||||
"log"
|
||||
"m3u8-downloader/pkg/constants"
|
||||
"m3u8-downloader/pkg/health"
|
||||
"m3u8-downloader/pkg/logging"
|
||||
"m3u8-downloader/pkg/processing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func Process(eventName string) {
|
||||
appLog := logging.MustGetFactory().App
|
||||
log.Printf("Starting processing for event: %s", eventName)
|
||||
appLog.Info("starting processing", logging.KeyEvent, eventName)
|
||||
cfg := constants.MustGetConfig()
|
||||
|
||||
if cfg.Health.Enabled {
|
||||
healthSrv, err := health.Start(context.Background(), &cfg.Health, health.MustGetRegistry())
|
||||
if err != nil {
|
||||
appLog.Error("failed to start health server", logging.KeyError, err.Error())
|
||||
} else {
|
||||
appLog.Info("health server listening", "addr", cfg.Health.Addr)
|
||||
defer func() {
|
||||
shutdownCtx, shutdownCancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer shutdownCancel()
|
||||
_ = healthSrv.Shutdown(shutdownCtx)
|
||||
}()
|
||||
}
|
||||
}
|
||||
|
||||
ps, err := processing.NewProcessingService(eventName, cfg)
|
||||
//log.Printf("Starting processing for event: %s", eventName)
|
||||
ps, err := processing.NewProcessingService(eventName)
|
||||
if err != nil {
|
||||
appLog.Error("failed to create processing service", logging.KeyError, err.Error())
|
||||
log.Fatalf("Failed to create processing service: %v", err)
|
||||
}
|
||||
if err := ps.Start(context.Background()); err != nil {
|
||||
appLog.Error("processing service failed", logging.KeyError, err.Error())
|
||||
log.Fatalf("Failed to run processing service: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
@ -1,77 +1,17 @@
|
||||
package transfer
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"context"
|
||||
"fmt"
|
||||
"log"
|
||||
"m3u8-downloader/pkg/config"
|
||||
"m3u8-downloader/pkg/constants"
|
||||
"m3u8-downloader/pkg/health"
|
||||
"m3u8-downloader/pkg/logging"
|
||||
"m3u8-downloader/pkg/transfer"
|
||||
"m3u8-downloader/pkg/utils"
|
||||
"os"
|
||||
"os/signal"
|
||||
"strconv"
|
||||
"strings"
|
||||
"syscall"
|
||||
"time"
|
||||
)
|
||||
|
||||
func getEventDirs(cfg *config.Config) ([]string, error) {
|
||||
dirs, err := os.ReadDir(cfg.Paths.LocalOutput)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to read directory: %w", err)
|
||||
}
|
||||
var eventDirs []string
|
||||
for _, dir := range dirs {
|
||||
if dir.IsDir() {
|
||||
eventDirs = append(eventDirs, dir.Name())
|
||||
}
|
||||
}
|
||||
return eventDirs, nil
|
||||
}
|
||||
|
||||
func RunTransferOnly(eventName string) {
|
||||
appLog := logging.MustGetFactory().App
|
||||
cfg := constants.MustGetConfig()
|
||||
|
||||
// Check if NAS transfer is enabled
|
||||
if !cfg.NAS.EnableTransfer {
|
||||
appLog.Error("NAS transfer disabled, cannot run transfer-only mode")
|
||||
log.Fatal("NAS transfer is disabled in configuration. Please enable it to use transfer-only mode.")
|
||||
}
|
||||
|
||||
if eventName == "" {
|
||||
events, err := getEventDirs(cfg)
|
||||
if err != nil {
|
||||
log.Fatalf("Failed to get event directories: %v", err)
|
||||
}
|
||||
if len(events) == 0 {
|
||||
log.Fatal("No events found")
|
||||
}
|
||||
if len(events) > 1 {
|
||||
fmt.Println("Multiple events found, please select one:")
|
||||
for i, event := range events {
|
||||
fmt.Printf("%d. %s\n", i+1, event)
|
||||
}
|
||||
reader := bufio.NewReader(os.Stdin)
|
||||
input, _ := reader.ReadString('\n')
|
||||
input = strings.TrimSpace(input)
|
||||
index, err := strconv.Atoi(input)
|
||||
if err != nil {
|
||||
log.Fatalf("Failed to parse input: %v", err)
|
||||
}
|
||||
if index < 1 || index > len(events) {
|
||||
log.Fatal("Invalid input")
|
||||
}
|
||||
eventName = events[index-1]
|
||||
} else {
|
||||
eventName = events[0]
|
||||
}
|
||||
}
|
||||
|
||||
log.Printf("Starting transfer-only mode for event: %s", eventName)
|
||||
|
||||
// Setup context and signal handling
|
||||
@ -83,50 +23,35 @@ func RunTransferOnly(eventName string) {
|
||||
go func() {
|
||||
<-sigChan
|
||||
log.Println("Shutting down transfer service...")
|
||||
appLog.Info("shutdown signal received")
|
||||
cancel()
|
||||
}()
|
||||
|
||||
if cfg.Health.Enabled {
|
||||
healthSrv, err := health.Start(ctx, &cfg.Health, health.MustGetRegistry())
|
||||
if err != nil {
|
||||
appLog.Error("failed to start health server", logging.KeyError, err.Error())
|
||||
} else {
|
||||
appLog.Info("health server listening", "addr", cfg.Health.Addr)
|
||||
defer func() {
|
||||
shutdownCtx, shutdownCancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer shutdownCancel()
|
||||
_ = healthSrv.Shutdown(shutdownCtx)
|
||||
}()
|
||||
}
|
||||
// Check if NAS transfer is enabled
|
||||
if !constants.EnableNASTransfer {
|
||||
log.Fatal("NAS transfer is disabled in constants. Please enable it to use transfer-only mode.")
|
||||
}
|
||||
|
||||
// Verify local event directory exists
|
||||
localEventPath := cfg.GetEventPath(eventName)
|
||||
if !utils.PathExists(localEventPath) {
|
||||
appLog.Error("local event directory does not exist", logging.KeyEvent, eventName)
|
||||
localEventPath := constants.LocalOutputDirPath + "/" + eventName
|
||||
if _, err := os.Stat(localEventPath); os.IsNotExist(err) {
|
||||
log.Fatalf("Local event directory does not exist: %s", localEventPath)
|
||||
}
|
||||
|
||||
// Create transfer service
|
||||
transferService, err := transfer.NewTrasferService(cfg.NAS.OutputPath, eventName)
|
||||
transferService, err := transfer.NewTrasferService(constants.NASOutputPath, eventName)
|
||||
if err != nil {
|
||||
appLog.Error("failed to create transfer service", logging.KeyError, err.Error())
|
||||
log.Fatalf("Failed to create transfer service: %v", err)
|
||||
}
|
||||
|
||||
// Find and queue existing files
|
||||
if err := transferService.QueueExistingFiles(localEventPath); err != nil {
|
||||
appLog.Error("failed to queue existing files", logging.KeyError, err.Error())
|
||||
log.Fatalf("Failed to queue existing files: %v", err)
|
||||
}
|
||||
|
||||
// Start transfer service
|
||||
log.Println("Starting transfer service...")
|
||||
appLog.Info("starting transfer service", logging.KeyEvent, eventName)
|
||||
if err := transferService.Start(ctx); err != nil && err != context.Canceled {
|
||||
log.Printf("Transfer service error: %v", err)
|
||||
appLog.Error("transfer service error", logging.KeyError, err.Error())
|
||||
}
|
||||
|
||||
// Graceful shutdown
|
||||
@ -135,5 +60,4 @@ func RunTransferOnly(eventName string) {
|
||||
transferService.Shutdown(shutdownCtx)
|
||||
|
||||
log.Println("Transfer-only mode completed.")
|
||||
appLog.Info("transfer-only mode completed", logging.KeyEvent, eventName)
|
||||
}
|
||||
|
||||
@ -1,305 +0,0 @@
|
||||
package config
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"time"
|
||||
)
|
||||
|
||||
type Config struct {
|
||||
Core CoreConfig
|
||||
HTTP HTTPConfig
|
||||
NAS NASConfig
|
||||
Processing ProcessingConfig
|
||||
Transfer TransferConfig
|
||||
Cleanup CleanupConfig
|
||||
Paths PathsConfig
|
||||
Logging LoggingConfig
|
||||
Health HealthConfig
|
||||
}
|
||||
|
||||
type CoreConfig struct {
|
||||
WorkerCount int
|
||||
RefreshDelay time.Duration
|
||||
}
|
||||
|
||||
type HTTPConfig struct {
|
||||
UserAgent string
|
||||
Referer string
|
||||
}
|
||||
|
||||
type NASConfig struct {
|
||||
EnableTransfer bool
|
||||
// OutputPath is the destination as configured: a UNC path, an smb:// URL,
|
||||
// or a directory on this host. How it is reached depends on the platform;
|
||||
// see pkg/nas.
|
||||
OutputPath string
|
||||
Username string
|
||||
Password string
|
||||
Timeout time.Duration
|
||||
RetryLimit int
|
||||
// AutoMount lets the recorder mount an SMB share itself when the host has
|
||||
// not already mounted it. Linux only, and requires root.
|
||||
AutoMount bool
|
||||
// MountPoint overrides where AutoMount attaches the share.
|
||||
MountPoint string
|
||||
}
|
||||
|
||||
type ProcessingConfig struct {
|
||||
Enabled bool
|
||||
AutoProcess bool
|
||||
WorkerCount int
|
||||
FFmpegPath string
|
||||
}
|
||||
|
||||
type TransferConfig struct {
|
||||
WorkerCount int
|
||||
RetryLimit int
|
||||
Timeout time.Duration
|
||||
FileSettlingDelay time.Duration
|
||||
QueueSize int
|
||||
BatchSize int
|
||||
}
|
||||
|
||||
type CleanupConfig struct {
|
||||
AfterTransfer bool
|
||||
BatchSize int
|
||||
RetainHours int
|
||||
}
|
||||
|
||||
type PathsConfig struct {
|
||||
BaseDir string
|
||||
LocalOutput string
|
||||
ProcessOutput string
|
||||
ManifestDir string
|
||||
PersistenceFile string
|
||||
}
|
||||
|
||||
type LoggingConfig struct {
|
||||
Dir string
|
||||
Level string
|
||||
}
|
||||
|
||||
type HealthConfig struct {
|
||||
Enabled bool
|
||||
Addr string
|
||||
}
|
||||
|
||||
var defaultConfig = Config{
|
||||
Core: CoreConfig{
|
||||
WorkerCount: 4,
|
||||
RefreshDelay: 3 * time.Second,
|
||||
},
|
||||
HTTP: HTTPConfig{
|
||||
UserAgent: "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36",
|
||||
Referer: "https://www.flomarching.com",
|
||||
},
|
||||
NAS: NASConfig{
|
||||
EnableTransfer: true,
|
||||
OutputPath: "\\\\HomeLabNAS\\dci\\streams",
|
||||
Username: "NASAdmin",
|
||||
Password: "s3tkY6tzA&KN6M",
|
||||
Timeout: 30 * time.Second,
|
||||
RetryLimit: 3,
|
||||
AutoMount: false,
|
||||
MountPoint: "",
|
||||
},
|
||||
Processing: ProcessingConfig{
|
||||
Enabled: true,
|
||||
AutoProcess: true,
|
||||
WorkerCount: 2,
|
||||
FFmpegPath: "ffmpeg",
|
||||
},
|
||||
Transfer: TransferConfig{
|
||||
WorkerCount: 2,
|
||||
RetryLimit: 3,
|
||||
Timeout: 30 * time.Second,
|
||||
FileSettlingDelay: 5 * time.Second,
|
||||
QueueSize: 100000,
|
||||
BatchSize: 1000,
|
||||
},
|
||||
Cleanup: CleanupConfig{
|
||||
AfterTransfer: true,
|
||||
BatchSize: 1000,
|
||||
RetainHours: 0,
|
||||
},
|
||||
Paths: PathsConfig{
|
||||
BaseDir: "data",
|
||||
LocalOutput: "data",
|
||||
ProcessOutput: "out",
|
||||
ManifestDir: "data",
|
||||
PersistenceFile: "transfer_queue.json",
|
||||
},
|
||||
Logging: LoggingConfig{
|
||||
Dir: "data/logs",
|
||||
Level: "info",
|
||||
},
|
||||
Health: HealthConfig{
|
||||
Enabled: true,
|
||||
Addr: ":8090",
|
||||
},
|
||||
}
|
||||
|
||||
func Load() (*Config, error) {
|
||||
cfg := defaultConfig
|
||||
|
||||
if err := cfg.loadFromEnvironment(); err != nil {
|
||||
return nil, fmt.Errorf("failed to load environment config: %w", err)
|
||||
}
|
||||
|
||||
if err := cfg.resolveAndValidatePaths(); err != nil {
|
||||
return nil, fmt.Errorf("path validation failed: %w", err)
|
||||
}
|
||||
|
||||
return &cfg, nil
|
||||
}
|
||||
|
||||
func (c *Config) loadFromEnvironment() error {
|
||||
if val := os.Getenv("WORKER_COUNT"); val != "" {
|
||||
if parsed, err := strconv.Atoi(val); err == nil {
|
||||
c.Core.WorkerCount = parsed
|
||||
}
|
||||
}
|
||||
|
||||
if val := os.Getenv("REFRESH_DELAY_SECONDS"); val != "" {
|
||||
if parsed, err := strconv.Atoi(val); err == nil {
|
||||
c.Core.RefreshDelay = time.Duration(parsed) * time.Second
|
||||
}
|
||||
}
|
||||
|
||||
if val := os.Getenv("NAS_OUTPUT_PATH"); val != "" {
|
||||
c.NAS.OutputPath = val
|
||||
}
|
||||
|
||||
if val := os.Getenv("NAS_USERNAME"); val != "" {
|
||||
c.NAS.Username = val
|
||||
}
|
||||
|
||||
if val := os.Getenv("NAS_PASSWORD"); val != "" {
|
||||
c.NAS.Password = val
|
||||
}
|
||||
|
||||
if val := os.Getenv("ENABLE_NAS_TRANSFER"); val != "" {
|
||||
c.NAS.EnableTransfer = val == "true"
|
||||
}
|
||||
|
||||
if val := os.Getenv("NAS_AUTO_MOUNT"); val != "" {
|
||||
c.NAS.AutoMount = val == "true"
|
||||
}
|
||||
|
||||
if val := os.Getenv("NAS_MOUNT_POINT"); val != "" {
|
||||
c.NAS.MountPoint = val
|
||||
}
|
||||
|
||||
if val := os.Getenv("LOCAL_OUTPUT_DIR"); val != "" {
|
||||
c.Paths.LocalOutput = val
|
||||
// The manifest describes the event directory and is documented as
|
||||
// sitting beside it, so it follows the output directory. Without this
|
||||
// it stays in the default data/ while the segments move elsewhere.
|
||||
c.Paths.ManifestDir = val
|
||||
}
|
||||
|
||||
// Applied after LOCAL_OUTPUT_DIR so an explicit setting wins.
|
||||
if val := os.Getenv("MANIFEST_DIR"); val != "" {
|
||||
c.Paths.ManifestDir = val
|
||||
}
|
||||
|
||||
if val := os.Getenv("PROCESS_OUTPUT_DIR"); val != "" {
|
||||
c.Paths.ProcessOutput = val
|
||||
}
|
||||
|
||||
if val := os.Getenv("FFMPEG_PATH"); val != "" {
|
||||
c.Processing.FFmpegPath = val
|
||||
}
|
||||
|
||||
if val := os.Getenv("LOG_DIR"); val != "" {
|
||||
c.Logging.Dir = val
|
||||
}
|
||||
|
||||
if val := os.Getenv("LOG_LEVEL"); val != "" {
|
||||
c.Logging.Level = val
|
||||
}
|
||||
|
||||
if val := os.Getenv("HEALTH_ENABLED"); val != "" {
|
||||
c.Health.Enabled = val == "true"
|
||||
}
|
||||
|
||||
if val := os.Getenv("HEALTH_ADDR"); val != "" {
|
||||
c.Health.Addr = val
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func (c *Config) resolveAndValidatePaths() error {
|
||||
cwd, err := os.Getwd()
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to get working directory: %w", err)
|
||||
}
|
||||
|
||||
// Only join with cwd if path is not already absolute
|
||||
if !filepath.IsAbs(c.Paths.BaseDir) {
|
||||
c.Paths.BaseDir = filepath.Join(cwd, c.Paths.BaseDir)
|
||||
}
|
||||
if !filepath.IsAbs(c.Paths.LocalOutput) {
|
||||
c.Paths.LocalOutput = filepath.Join(cwd, c.Paths.LocalOutput)
|
||||
}
|
||||
if !filepath.IsAbs(c.Paths.ProcessOutput) {
|
||||
c.Paths.ProcessOutput = filepath.Join(cwd, c.Paths.ProcessOutput)
|
||||
}
|
||||
if !filepath.IsAbs(c.Paths.ManifestDir) {
|
||||
c.Paths.ManifestDir = filepath.Join(cwd, c.Paths.ManifestDir)
|
||||
}
|
||||
if !filepath.IsAbs(c.Paths.PersistenceFile) {
|
||||
c.Paths.PersistenceFile = filepath.Join(c.Paths.BaseDir, c.Paths.PersistenceFile)
|
||||
}
|
||||
if !filepath.IsAbs(c.Logging.Dir) {
|
||||
c.Logging.Dir = filepath.Join(cwd, c.Logging.Dir)
|
||||
}
|
||||
|
||||
requiredDirs := []string{
|
||||
c.Paths.BaseDir,
|
||||
c.Paths.LocalOutput,
|
||||
c.Paths.ProcessOutput,
|
||||
c.Paths.ManifestDir,
|
||||
c.Logging.Dir,
|
||||
}
|
||||
|
||||
for _, dir := range requiredDirs {
|
||||
if err := os.MkdirAll(dir, 0755); err != nil {
|
||||
return fmt.Errorf("failed to create directory %s: %w", dir, err)
|
||||
}
|
||||
}
|
||||
|
||||
if c.NAS.EnableTransfer && c.NAS.OutputPath == "" {
|
||||
return fmt.Errorf("NAS output path is required when transfer is enabled")
|
||||
}
|
||||
|
||||
if c.Processing.Enabled && c.Processing.FFmpegPath == "" {
|
||||
return fmt.Errorf("FFmpeg path is required when processing is enabled")
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func (c *Config) GetEventPath(eventName string) string {
|
||||
return filepath.Join(c.Paths.LocalOutput, eventName)
|
||||
}
|
||||
|
||||
func (c *Config) GetManifestPath(eventName string) string {
|
||||
return filepath.Join(c.Paths.ManifestDir, eventName+".json")
|
||||
}
|
||||
|
||||
func (c *Config) GetNASEventPath(eventName string) string {
|
||||
return filepath.Join(c.NAS.OutputPath, eventName)
|
||||
}
|
||||
|
||||
func (c *Config) GetProcessOutputPath(eventName string) string {
|
||||
return filepath.Join(c.Paths.ProcessOutput, eventName)
|
||||
}
|
||||
|
||||
func (c *Config) GetQualityPath(eventName, quality string) string {
|
||||
return filepath.Join(c.GetEventPath(eventName), quality)
|
||||
}
|
||||
@ -1,181 +0,0 @@
|
||||
package config
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestConfig_Load(t *testing.T) {
|
||||
// Save original env vars
|
||||
originalVars := map[string]string{
|
||||
"WORKER_COUNT": os.Getenv("WORKER_COUNT"),
|
||||
"NAS_USERNAME": os.Getenv("NAS_USERNAME"),
|
||||
"LOCAL_OUTPUT_DIR": os.Getenv("LOCAL_OUTPUT_DIR"),
|
||||
"ENABLE_NAS_TRANSFER": os.Getenv("ENABLE_NAS_TRANSFER"),
|
||||
}
|
||||
defer func() {
|
||||
// Restore original env vars
|
||||
for key, value := range originalVars {
|
||||
if value == "" {
|
||||
os.Unsetenv(key)
|
||||
} else {
|
||||
os.Setenv(key, value)
|
||||
}
|
||||
}
|
||||
}()
|
||||
|
||||
// Test default config load
|
||||
cfg, err := Load()
|
||||
if err != nil {
|
||||
t.Fatalf("Load() failed: %v", err)
|
||||
}
|
||||
|
||||
// Verify defaults
|
||||
if cfg.Core.WorkerCount != 4 {
|
||||
t.Errorf("Expected WorkerCount=4, got %d", cfg.Core.WorkerCount)
|
||||
}
|
||||
if cfg.Core.RefreshDelay != 3*time.Second {
|
||||
t.Errorf("Expected RefreshDelay=3s, got %v", cfg.Core.RefreshDelay)
|
||||
}
|
||||
if !cfg.NAS.EnableTransfer {
|
||||
t.Errorf("Expected NAS.EnableTransfer=true, got false")
|
||||
}
|
||||
|
||||
// Test environment variable override
|
||||
os.Setenv("WORKER_COUNT", "8")
|
||||
os.Setenv("NAS_USERNAME", "testuser")
|
||||
os.Setenv("ENABLE_NAS_TRANSFER", "false")
|
||||
os.Setenv("LOCAL_OUTPUT_DIR", "custom_data")
|
||||
|
||||
cfg2, err := Load()
|
||||
if err != nil {
|
||||
t.Fatalf("Load() with env vars failed: %v", err)
|
||||
}
|
||||
|
||||
if cfg2.Core.WorkerCount != 8 {
|
||||
t.Errorf("Expected WorkerCount=8 from env, got %d", cfg2.Core.WorkerCount)
|
||||
}
|
||||
if cfg2.NAS.Username != "testuser" {
|
||||
t.Errorf("Expected NAS.Username='testuser' from env, got %s", cfg2.NAS.Username)
|
||||
}
|
||||
if cfg2.NAS.EnableTransfer {
|
||||
t.Errorf("Expected NAS.EnableTransfer=false from env, got true")
|
||||
}
|
||||
if !strings.Contains(cfg2.Paths.LocalOutput, "custom_data") {
|
||||
t.Errorf("Expected LocalOutput to contain 'custom_data', got %s", cfg2.Paths.LocalOutput)
|
||||
}
|
||||
}
|
||||
|
||||
func TestConfig_PathMethods(t *testing.T) {
|
||||
cfg, err := Load()
|
||||
if err != nil {
|
||||
t.Fatalf("Load() failed: %v", err)
|
||||
}
|
||||
|
||||
testEvent := "test-event"
|
||||
testQuality := "1080p"
|
||||
|
||||
// Test GetEventPath
|
||||
eventPath := cfg.GetEventPath(testEvent)
|
||||
if !strings.Contains(eventPath, testEvent) {
|
||||
t.Errorf("GetEventPath should contain event name, got %s", eventPath)
|
||||
}
|
||||
|
||||
// Test GetManifestPath
|
||||
manifestPath := cfg.GetManifestPath(testEvent)
|
||||
if !strings.Contains(manifestPath, testEvent) {
|
||||
t.Errorf("GetManifestPath should contain event name, got %s", manifestPath)
|
||||
}
|
||||
if !strings.HasSuffix(manifestPath, ".json") {
|
||||
t.Errorf("GetManifestPath should end with .json, got %s", manifestPath)
|
||||
}
|
||||
|
||||
// Test GetNASEventPath
|
||||
nasPath := cfg.GetNASEventPath(testEvent)
|
||||
if !strings.Contains(nasPath, testEvent) {
|
||||
t.Errorf("GetNASEventPath should contain event name, got %s", nasPath)
|
||||
}
|
||||
|
||||
// Test GetProcessOutputPath
|
||||
processPath := cfg.GetProcessOutputPath(testEvent)
|
||||
if !strings.Contains(processPath, testEvent) {
|
||||
t.Errorf("GetProcessOutputPath should contain event name, got %s", processPath)
|
||||
}
|
||||
|
||||
// Test GetQualityPath
|
||||
qualityPath := cfg.GetQualityPath(testEvent, testQuality)
|
||||
if !strings.Contains(qualityPath, testEvent) {
|
||||
t.Errorf("GetQualityPath should contain event name, got %s", qualityPath)
|
||||
}
|
||||
if !strings.Contains(qualityPath, testQuality) {
|
||||
t.Errorf("GetQualityPath should contain quality, got %s", qualityPath)
|
||||
}
|
||||
}
|
||||
|
||||
func TestConfig_PathValidation(t *testing.T) {
|
||||
// Create a temporary directory for testing
|
||||
tempDir, err := os.MkdirTemp("", "config_test_*")
|
||||
if err != nil {
|
||||
t.Fatalf("Failed to create temp dir: %v", err)
|
||||
}
|
||||
defer os.RemoveAll(tempDir)
|
||||
|
||||
// Set environment variables to use temp directory
|
||||
os.Setenv("LOCAL_OUTPUT_DIR", filepath.Join(tempDir, "data"))
|
||||
defer os.Unsetenv("LOCAL_OUTPUT_DIR")
|
||||
|
||||
cfg, err := Load()
|
||||
if err != nil {
|
||||
t.Fatalf("Load() failed: %v", err)
|
||||
}
|
||||
|
||||
// Verify directories were created
|
||||
if _, err := os.Stat(cfg.Paths.LocalOutput); os.IsNotExist(err) {
|
||||
t.Errorf("LocalOutput directory should have been created: %s", cfg.Paths.LocalOutput)
|
||||
}
|
||||
if _, err := os.Stat(cfg.Paths.ProcessOutput); os.IsNotExist(err) {
|
||||
t.Errorf("ProcessOutput directory should have been created: %s", cfg.Paths.ProcessOutput)
|
||||
}
|
||||
}
|
||||
|
||||
func TestConfig_ValidationErrors(t *testing.T) {
|
||||
// Save original env vars
|
||||
originalNASPath := os.Getenv("NAS_OUTPUT_PATH")
|
||||
originalFFmpegPath := os.Getenv("FFMPEG_PATH")
|
||||
defer func() {
|
||||
if originalNASPath == "" {
|
||||
os.Unsetenv("NAS_OUTPUT_PATH")
|
||||
} else {
|
||||
os.Setenv("NAS_OUTPUT_PATH", originalNASPath)
|
||||
}
|
||||
if originalFFmpegPath == "" {
|
||||
os.Unsetenv("FFMPEG_PATH")
|
||||
} else {
|
||||
os.Setenv("FFMPEG_PATH", originalFFmpegPath)
|
||||
}
|
||||
}()
|
||||
|
||||
// Note: Validation tests are limited because the default config
|
||||
// has working defaults. We can test that Load() works with valid configs.
|
||||
|
||||
// Test that Load works with proper paths set
|
||||
tempDir2, err := os.MkdirTemp("", "config_validation_test_*")
|
||||
if err != nil {
|
||||
t.Fatalf("Failed to create temp dir: %v", err)
|
||||
}
|
||||
defer os.RemoveAll(tempDir2)
|
||||
|
||||
os.Setenv("NAS_OUTPUT_PATH", "\\\\test\\path")
|
||||
os.Setenv("LOCAL_OUTPUT_DIR", tempDir2)
|
||||
|
||||
cfg, err := Load()
|
||||
if err != nil {
|
||||
t.Errorf("Load() should work with valid config: %v", err)
|
||||
}
|
||||
if cfg == nil {
|
||||
t.Error("Config should not be nil")
|
||||
}
|
||||
}
|
||||
@ -1,51 +1,34 @@
|
||||
package constants
|
||||
|
||||
import (
|
||||
"m3u8-downloader/pkg/config"
|
||||
"sync"
|
||||
)
|
||||
|
||||
var (
|
||||
globalConfig *config.Config
|
||||
configOnce sync.Once
|
||||
configError error
|
||||
)
|
||||
|
||||
func GetConfig() (*config.Config, error) {
|
||||
configOnce.Do(func() {
|
||||
globalConfig, configError = config.Load()
|
||||
})
|
||||
return globalConfig, configError
|
||||
}
|
||||
|
||||
func MustGetConfig() *config.Config {
|
||||
cfg, err := GetConfig()
|
||||
if err != nil {
|
||||
panic("Failed to load configuration: " + err.Error())
|
||||
}
|
||||
return cfg
|
||||
}
|
||||
import "time"
|
||||
|
||||
const (
|
||||
WorkerCount = 4
|
||||
RefreshDelay = 3
|
||||
RefreshDelay = 3 * time.Second
|
||||
|
||||
HTTPUserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36"
|
||||
REFERRER = "https://www.flomarching.com"
|
||||
HTTPUserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36"
|
||||
REFERRER = "https://www.flomarching.com"
|
||||
LocalOutputDirPath = "../data/"
|
||||
|
||||
DefaultNASOutputPath = "\\\\HomeLabNAS\\dci\\streams"
|
||||
DefaultNASUsername = "NASAdmin"
|
||||
EnableNASTransfer = false
|
||||
NASOutputPath = "\\\\HomeLabNAS\\dci\\streams"
|
||||
NASUsername = "NASAdmin"
|
||||
NASPassword = "s3tkY6tzA&KN6M"
|
||||
TransferWorkerCount = 2
|
||||
TransferRetryLimit = 3
|
||||
TransferTimeout = 30 * time.Second
|
||||
FileSettlingDelay = 5 * time.Second
|
||||
PersistencePath = "../data/transfer_queue.json"
|
||||
TransferQueueSize = 100000
|
||||
BatchSize = 1000
|
||||
ManifestPath = "../data"
|
||||
|
||||
DefaultTransferWorkerCount = 2
|
||||
DefaultTransferRetryLimit = 3
|
||||
DefaultTransferTimeout = 30
|
||||
DefaultFileSettlingDelay = 5
|
||||
DefaultTransferQueueSize = 100000
|
||||
DefaultBatchSize = 1000
|
||||
CleanupAfterTransfer = false
|
||||
CleanupBatchSize = 1000
|
||||
RetainLocalHours = 0
|
||||
|
||||
DefaultCleanupBatchSize = 1000
|
||||
DefaultRetainLocalHours = 0
|
||||
|
||||
DefaultProcessWorkerCount = 2
|
||||
DefaultFFmpegPath = "ffmpeg"
|
||||
ProcessOutputPath = "../out"
|
||||
AutoProcess = true
|
||||
ProcessingEnabled = true
|
||||
ProcessWorkerCount = 2
|
||||
)
|
||||
|
||||
@ -1,240 +0,0 @@
|
||||
package constants
|
||||
|
||||
import (
|
||||
"os"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestGetConfig(t *testing.T) {
|
||||
// Test successful config loading
|
||||
cfg, err := GetConfig()
|
||||
if err != nil {
|
||||
t.Fatalf("GetConfig() failed: %v", err)
|
||||
}
|
||||
if cfg == nil {
|
||||
t.Fatal("GetConfig() returned nil config")
|
||||
}
|
||||
|
||||
// Test that subsequent calls return the same instance (singleton)
|
||||
cfg2, err := GetConfig()
|
||||
if err != nil {
|
||||
t.Fatalf("Second GetConfig() call failed: %v", err)
|
||||
}
|
||||
|
||||
// Both should be the same instance due to sync.Once
|
||||
if cfg != cfg2 {
|
||||
t.Error("GetConfig() should return the same instance (singleton)")
|
||||
}
|
||||
}
|
||||
|
||||
func TestMustGetConfig(t *testing.T) {
|
||||
// This should not panic with valid environment
|
||||
cfg := MustGetConfig()
|
||||
if cfg == nil {
|
||||
t.Fatal("MustGetConfig() returned nil")
|
||||
}
|
||||
|
||||
// Verify it returns a properly initialized config
|
||||
if cfg.Core.WorkerCount <= 0 {
|
||||
t.Errorf("Expected positive WorkerCount, got %d", cfg.Core.WorkerCount)
|
||||
}
|
||||
if cfg.Core.RefreshDelay <= 0 {
|
||||
t.Errorf("Expected positive RefreshDelay, got %v", cfg.Core.RefreshDelay)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMustGetConfig_Panic(t *testing.T) {
|
||||
// We can't easily test the panic scenario without breaking the singleton,
|
||||
// but we can test that MustGetConfig works normally
|
||||
defer func() {
|
||||
if r := recover(); r != nil {
|
||||
t.Errorf("MustGetConfig() panicked unexpectedly: %v", r)
|
||||
}
|
||||
}()
|
||||
|
||||
cfg := MustGetConfig()
|
||||
if cfg == nil {
|
||||
t.Fatal("MustGetConfig() returned nil without panicking")
|
||||
}
|
||||
}
|
||||
|
||||
func TestConfigSingleton(t *testing.T) {
|
||||
// Reset the singleton for this test (this is a bit hacky but necessary for testing)
|
||||
// We'll create multiple goroutines to test concurrent access
|
||||
|
||||
configs := make(chan interface{}, 10)
|
||||
|
||||
// Launch multiple goroutines to call GetConfig concurrently
|
||||
for i := 0; i < 10; i++ {
|
||||
go func() {
|
||||
cfg, _ := GetConfig()
|
||||
configs <- cfg
|
||||
}()
|
||||
}
|
||||
|
||||
// Collect all configs
|
||||
var allConfigs []interface{}
|
||||
for i := 0; i < 10; i++ {
|
||||
allConfigs = append(allConfigs, <-configs)
|
||||
}
|
||||
|
||||
// All should be the same instance
|
||||
firstConfig := allConfigs[0]
|
||||
for i, cfg := range allConfigs {
|
||||
if cfg != firstConfig {
|
||||
t.Errorf("Config %d is different from first config", i)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestConstants_Values(t *testing.T) {
|
||||
// Test that constants have expected values
|
||||
if WorkerCount != 4 {
|
||||
t.Errorf("Expected WorkerCount=4, got %d", WorkerCount)
|
||||
}
|
||||
if RefreshDelay != 3 {
|
||||
t.Errorf("Expected RefreshDelay=3, got %d", RefreshDelay)
|
||||
}
|
||||
|
||||
// Test HTTP constants
|
||||
if HTTPUserAgent == "" {
|
||||
t.Error("HTTPUserAgent should not be empty")
|
||||
}
|
||||
if !strings.Contains(HTTPUserAgent, "Mozilla") {
|
||||
t.Error("HTTPUserAgent should contain 'Mozilla'")
|
||||
}
|
||||
if REFERRER != "https://www.flomarching.com" {
|
||||
t.Errorf("Expected REFERRER='https://www.flomarching.com', got '%s'", REFERRER)
|
||||
}
|
||||
|
||||
// Test default NAS constants
|
||||
if DefaultNASOutputPath != "\\\\HomeLabNAS\\dci\\streams" {
|
||||
t.Errorf("Expected DefaultNASOutputPath='\\\\HomeLabNAS\\dci\\streams', got '%s'", DefaultNASOutputPath)
|
||||
}
|
||||
if DefaultNASUsername != "NASAdmin" {
|
||||
t.Errorf("Expected DefaultNASUsername='NASAdmin', got '%s'", DefaultNASUsername)
|
||||
}
|
||||
|
||||
// Test transfer constants
|
||||
if DefaultTransferWorkerCount != 2 {
|
||||
t.Errorf("Expected DefaultTransferWorkerCount=2, got %d", DefaultTransferWorkerCount)
|
||||
}
|
||||
if DefaultTransferRetryLimit != 3 {
|
||||
t.Errorf("Expected DefaultTransferRetryLimit=3, got %d", DefaultTransferRetryLimit)
|
||||
}
|
||||
if DefaultTransferTimeout != 30 {
|
||||
t.Errorf("Expected DefaultTransferTimeout=30, got %d", DefaultTransferTimeout)
|
||||
}
|
||||
if DefaultFileSettlingDelay != 5 {
|
||||
t.Errorf("Expected DefaultFileSettlingDelay=5, got %d", DefaultFileSettlingDelay)
|
||||
}
|
||||
if DefaultTransferQueueSize != 100000 {
|
||||
t.Errorf("Expected DefaultTransferQueueSize=100000, got %d", DefaultTransferQueueSize)
|
||||
}
|
||||
if DefaultBatchSize != 1000 {
|
||||
t.Errorf("Expected DefaultBatchSize=1000, got %d", DefaultBatchSize)
|
||||
}
|
||||
|
||||
// Test cleanup constants
|
||||
if DefaultCleanupBatchSize != 1000 {
|
||||
t.Errorf("Expected DefaultCleanupBatchSize=1000, got %d", DefaultCleanupBatchSize)
|
||||
}
|
||||
if DefaultRetainLocalHours != 0 {
|
||||
t.Errorf("Expected DefaultRetainLocalHours=0, got %d", DefaultRetainLocalHours)
|
||||
}
|
||||
|
||||
// Test processing constants
|
||||
if DefaultProcessWorkerCount != 2 {
|
||||
t.Errorf("Expected DefaultProcessWorkerCount=2, got %d", DefaultProcessWorkerCount)
|
||||
}
|
||||
if DefaultFFmpegPath != "ffmpeg" {
|
||||
t.Errorf("Expected DefaultFFmpegPath='ffmpeg', got '%s'", DefaultFFmpegPath)
|
||||
}
|
||||
}
|
||||
|
||||
func TestConfig_Integration(t *testing.T) {
|
||||
cfg := MustGetConfig()
|
||||
|
||||
// Test that config values match or override constants appropriately
|
||||
if cfg.Core.WorkerCount != WorkerCount && os.Getenv("WORKER_COUNT") == "" {
|
||||
t.Errorf("Config WorkerCount (%d) should match constant (%d) when no env override", cfg.Core.WorkerCount, WorkerCount)
|
||||
}
|
||||
|
||||
if cfg.Core.RefreshDelay != time.Duration(RefreshDelay)*time.Second && os.Getenv("REFRESH_DELAY_SECONDS") == "" {
|
||||
t.Errorf("Config RefreshDelay (%v) should match constant (%v) when no env override", cfg.Core.RefreshDelay, time.Duration(RefreshDelay)*time.Second)
|
||||
}
|
||||
|
||||
// Test HTTP settings
|
||||
if cfg.HTTP.UserAgent != HTTPUserAgent {
|
||||
t.Errorf("Config UserAgent (%s) should match constant (%s)", cfg.HTTP.UserAgent, HTTPUserAgent)
|
||||
}
|
||||
if cfg.HTTP.Referer != REFERRER {
|
||||
t.Errorf("Config Referer (%s) should match constant (%s)", cfg.HTTP.Referer, REFERRER)
|
||||
}
|
||||
}
|
||||
|
||||
func TestConfig_PathMethods(t *testing.T) {
|
||||
cfg := MustGetConfig()
|
||||
|
||||
testEvent := "test-event-123"
|
||||
testQuality := "1080p"
|
||||
|
||||
// Test GetEventPath
|
||||
eventPath := cfg.GetEventPath(testEvent)
|
||||
if !strings.Contains(eventPath, testEvent) {
|
||||
t.Errorf("GetEventPath should contain event name '%s', got: %s", testEvent, eventPath)
|
||||
}
|
||||
|
||||
// Test GetManifestPath
|
||||
manifestPath := cfg.GetManifestPath(testEvent)
|
||||
if !strings.Contains(manifestPath, testEvent) {
|
||||
t.Errorf("GetManifestPath should contain event name '%s', got: %s", testEvent, manifestPath)
|
||||
}
|
||||
if !strings.HasSuffix(manifestPath, ".json") {
|
||||
t.Errorf("GetManifestPath should end with '.json', got: %s", manifestPath)
|
||||
}
|
||||
|
||||
// Test GetNASEventPath
|
||||
nasPath := cfg.GetNASEventPath(testEvent)
|
||||
if !strings.Contains(nasPath, testEvent) {
|
||||
t.Errorf("GetNASEventPath should contain event name '%s', got: %s", testEvent, nasPath)
|
||||
}
|
||||
|
||||
// Test GetProcessOutputPath
|
||||
processPath := cfg.GetProcessOutputPath(testEvent)
|
||||
if !strings.Contains(processPath, testEvent) {
|
||||
t.Errorf("GetProcessOutputPath should contain event name '%s', got: %s", testEvent, processPath)
|
||||
}
|
||||
|
||||
// Test GetQualityPath
|
||||
qualityPath := cfg.GetQualityPath(testEvent, testQuality)
|
||||
if !strings.Contains(qualityPath, testEvent) {
|
||||
t.Errorf("GetQualityPath should contain event name '%s', got: %s", testEvent, qualityPath)
|
||||
}
|
||||
if !strings.Contains(qualityPath, testQuality) {
|
||||
t.Errorf("GetQualityPath should contain quality '%s', got: %s", testQuality, qualityPath)
|
||||
}
|
||||
}
|
||||
|
||||
func TestConfig_DefaultValues(t *testing.T) {
|
||||
cfg := MustGetConfig()
|
||||
|
||||
// Test that default values are reasonable
|
||||
if cfg.Transfer.QueueSize != DefaultTransferQueueSize {
|
||||
t.Errorf("Expected transfer queue size %d, got %d", DefaultTransferQueueSize, cfg.Transfer.QueueSize)
|
||||
}
|
||||
|
||||
if cfg.Transfer.BatchSize != DefaultBatchSize {
|
||||
t.Errorf("Expected transfer batch size %d, got %d", DefaultBatchSize, cfg.Transfer.BatchSize)
|
||||
}
|
||||
|
||||
if cfg.Processing.WorkerCount != DefaultProcessWorkerCount {
|
||||
t.Errorf("Expected processing worker count %d, got %d", DefaultProcessWorkerCount, cfg.Processing.WorkerCount)
|
||||
}
|
||||
|
||||
if cfg.Cleanup.BatchSize != DefaultCleanupBatchSize {
|
||||
t.Errorf("Expected cleanup batch size %d, got %d", DefaultCleanupBatchSize, cfg.Cleanup.BatchSize)
|
||||
}
|
||||
}
|
||||
@ -1,323 +0,0 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<title>m3u8-downloader status</title>
|
||||
<style>
|
||||
:root {
|
||||
--bg: #0f1115;
|
||||
--panel: #161922;
|
||||
--border: #262b38;
|
||||
--text: #e6e8ee;
|
||||
--muted: #8b909e;
|
||||
--green: #3ecf8e;
|
||||
--red: #f2545b;
|
||||
--amber: #f2b705;
|
||||
--blue: #4f8ef7;
|
||||
}
|
||||
* { box-sizing: border-box; }
|
||||
body {
|
||||
margin: 0;
|
||||
font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif;
|
||||
background: var(--bg);
|
||||
color: var(--text);
|
||||
}
|
||||
header {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: space-between;
|
||||
flex-wrap: wrap;
|
||||
gap: 12px;
|
||||
padding: 16px 24px;
|
||||
border-bottom: 1px solid var(--border);
|
||||
}
|
||||
header h1 {
|
||||
font-size: 18px;
|
||||
margin: 0;
|
||||
font-weight: 600;
|
||||
}
|
||||
.header-meta {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 16px;
|
||||
color: var(--muted);
|
||||
font-size: 13px;
|
||||
}
|
||||
.badge {
|
||||
display: inline-flex;
|
||||
align-items: center;
|
||||
gap: 6px;
|
||||
padding: 4px 12px;
|
||||
border-radius: 999px;
|
||||
font-size: 13px;
|
||||
font-weight: 600;
|
||||
text-transform: uppercase;
|
||||
letter-spacing: 0.03em;
|
||||
}
|
||||
.badge-dot { width: 8px; height: 8px; border-radius: 50%; }
|
||||
.badge.healthy { background: rgba(62,207,142,0.15); color: var(--green); }
|
||||
.badge.healthy .badge-dot { background: var(--green); }
|
||||
.badge.degraded { background: rgba(242,84,91,0.15); color: var(--red); }
|
||||
.badge.degraded .badge-dot { background: var(--red); }
|
||||
.badge.unknown { background: rgba(139,144,158,0.15); color: var(--muted); }
|
||||
.badge.unknown .badge-dot { background: var(--muted); }
|
||||
|
||||
#conn-banner {
|
||||
display: none;
|
||||
background: var(--red);
|
||||
color: #fff;
|
||||
text-align: center;
|
||||
padding: 8px;
|
||||
font-size: 13px;
|
||||
font-weight: 600;
|
||||
}
|
||||
#conn-banner.show { display: block; }
|
||||
|
||||
main {
|
||||
padding: 24px;
|
||||
display: grid;
|
||||
grid-template-columns: repeat(auto-fit, minmax(340px, 1fr));
|
||||
gap: 20px;
|
||||
max-width: 1400px;
|
||||
margin: 0 auto;
|
||||
}
|
||||
.card {
|
||||
background: var(--panel);
|
||||
border: 1px solid var(--border);
|
||||
border-radius: 10px;
|
||||
padding: 18px 20px;
|
||||
}
|
||||
.card h2 {
|
||||
font-size: 14px;
|
||||
text-transform: uppercase;
|
||||
letter-spacing: 0.05em;
|
||||
color: var(--muted);
|
||||
margin: 0 0 14px 0;
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: space-between;
|
||||
}
|
||||
table { width: 100%; border-collapse: collapse; font-size: 13px; }
|
||||
th, td { text-align: left; padding: 6px 8px; border-bottom: 1px solid var(--border); }
|
||||
th { color: var(--muted); font-weight: 500; font-size: 12px; }
|
||||
tr.failed-row { background: rgba(242,84,91,0.08); }
|
||||
tr.inactive-row { opacity: 0.5; }
|
||||
.dot { display: inline-block; width: 8px; height: 8px; border-radius: 50%; margin-right: 6px; }
|
||||
.dot.on { background: var(--green); }
|
||||
.dot.off { background: var(--red); }
|
||||
|
||||
.stat-grid {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(3, 1fr);
|
||||
gap: 10px;
|
||||
margin-bottom: 14px;
|
||||
}
|
||||
.stat {
|
||||
background: rgba(255,255,255,0.03);
|
||||
border-radius: 8px;
|
||||
padding: 10px 12px;
|
||||
}
|
||||
.stat .value { font-size: 20px; font-weight: 600; }
|
||||
.stat .label { font-size: 11px; color: var(--muted); text-transform: uppercase; letter-spacing: 0.03em; }
|
||||
.stat.failed .value { color: var(--red); }
|
||||
|
||||
.state-pill {
|
||||
display: inline-block;
|
||||
padding: 3px 10px;
|
||||
border-radius: 999px;
|
||||
font-size: 12px;
|
||||
font-weight: 600;
|
||||
text-transform: uppercase;
|
||||
}
|
||||
.state-idle { background: rgba(139,144,158,0.15); color: var(--muted); }
|
||||
.state-scanning, .state-running { background: rgba(79,142,247,0.15); color: var(--blue); }
|
||||
.state-completed { background: rgba(62,207,142,0.15); color: var(--green); }
|
||||
.state-failed { background: rgba(242,84,91,0.15); color: var(--red); }
|
||||
|
||||
.kv { display: flex; justify-content: space-between; padding: 4px 0; font-size: 13px; border-bottom: 1px solid var(--border); }
|
||||
.kv:last-child { border-bottom: none; }
|
||||
.kv .k { color: var(--muted); }
|
||||
|
||||
.error-list { margin: 10px 0 0 0; padding: 0; list-style: none; }
|
||||
.error-list li {
|
||||
font-size: 12px;
|
||||
padding: 6px 8px;
|
||||
margin-bottom: 4px;
|
||||
background: rgba(242,84,91,0.08);
|
||||
border-left: 3px solid var(--red);
|
||||
border-radius: 4px;
|
||||
color: var(--text);
|
||||
}
|
||||
.error-list .err-time { color: var(--muted); margin-right: 6px; }
|
||||
.empty-note { color: var(--muted); font-size: 13px; font-style: italic; }
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
|
||||
<div id="conn-banner">Lost connection to status endpoint - retrying...</div>
|
||||
|
||||
<header>
|
||||
<h1>m3u8-downloader</h1>
|
||||
<div class="header-meta">
|
||||
<span id="overall-badge" class="badge unknown"><span class="badge-dot"></span>unknown</span>
|
||||
<span id="uptime">uptime: -</span>
|
||||
<span id="last-updated">last updated: -</span>
|
||||
</div>
|
||||
</header>
|
||||
|
||||
<main>
|
||||
<section class="card">
|
||||
<h2>Download</h2>
|
||||
<table>
|
||||
<thead>
|
||||
<tr><th>Resolution</th><th>Active</th><th>OK</th><th>Failed</th><th>Last activity</th></tr>
|
||||
</thead>
|
||||
<tbody id="variant-rows"></tbody>
|
||||
</table>
|
||||
<div id="download-empty" class="empty-note" style="display:none;">No variants reporting yet.</div>
|
||||
<ul id="download-errors" class="error-list"></ul>
|
||||
</section>
|
||||
|
||||
<section class="card">
|
||||
<h2>Transfer</h2>
|
||||
<div class="kv"><span class="k">NAS connection</span><span><span id="nas-dot" class="dot off"></span><span id="nas-text">unknown</span></span></div>
|
||||
<div class="stat-grid" style="margin-top:12px;">
|
||||
<div class="stat"><div class="value" id="t-added">-</div><div class="label">Added</div></div>
|
||||
<div class="stat"><div class="value" id="t-completed">-</div><div class="label">Completed</div></div>
|
||||
<div class="stat failed"><div class="value" id="t-failed">-</div><div class="label">Failed</div></div>
|
||||
<div class="stat"><div class="value" id="t-pending">-</div><div class="label">Pending</div></div>
|
||||
<div class="stat"><div class="value" id="t-queue">-</div><div class="label">Queue size</div></div>
|
||||
<div class="stat"><div class="value" id="t-cleanup">-</div><div class="label">Cleanup pending</div></div>
|
||||
</div>
|
||||
<div class="kv"><span class="k">Transferred</span><span id="t-bytes">-</span></div>
|
||||
<ul id="transfer-errors" class="error-list"></ul>
|
||||
</section>
|
||||
|
||||
<section class="card">
|
||||
<h2>Processing</h2>
|
||||
<div class="kv"><span class="k">State</span><span id="p-state" class="state-pill state-idle">idle</span></div>
|
||||
<div class="kv"><span class="k">Event</span><span id="p-event">-</span></div>
|
||||
<div class="kv"><span class="k">Segments found</span><span id="p-segments">-</span></div>
|
||||
<div class="kv"><span class="k">Started</span><span id="p-started">-</span></div>
|
||||
<div class="kv"><span class="k">Finished</span><span id="p-finished">-</span></div>
|
||||
<ul id="processing-errors" class="error-list"></ul>
|
||||
</section>
|
||||
</main>
|
||||
|
||||
<script>
|
||||
const POLL_INTERVAL_MS = 3000;
|
||||
const connBanner = document.getElementById('conn-banner');
|
||||
|
||||
function fmtBytes(n) {
|
||||
if (!n || n <= 0) return '0 B';
|
||||
const units = ['B', 'KB', 'MB', 'GB', 'TB'];
|
||||
let i = 0;
|
||||
while (n >= 1024 && i < units.length - 1) { n /= 1024; i++; }
|
||||
return n.toFixed(i === 0 ? 0 : 1) + ' ' + units[i];
|
||||
}
|
||||
|
||||
function fmtDuration(sec) {
|
||||
sec = Math.floor(sec || 0);
|
||||
const h = Math.floor(sec / 3600);
|
||||
const m = Math.floor((sec % 3600) / 60);
|
||||
const s = sec % 60;
|
||||
const parts = [];
|
||||
if (h) parts.push(h + 'h');
|
||||
if (m || h) parts.push(m + 'm');
|
||||
parts.push(s + 's');
|
||||
return parts.join(' ');
|
||||
}
|
||||
|
||||
function fmtTime(iso) {
|
||||
if (!iso) return '-';
|
||||
const d = new Date(iso);
|
||||
if (isNaN(d.getTime()) || d.getFullYear() < 1971) return '-';
|
||||
return d.toLocaleTimeString();
|
||||
}
|
||||
|
||||
function renderErrors(listEl, errors) {
|
||||
listEl.innerHTML = '';
|
||||
(errors || []).slice().reverse().forEach(e => {
|
||||
const li = document.createElement('li');
|
||||
const t = document.createElement('span');
|
||||
t.className = 'err-time';
|
||||
t.textContent = fmtTime(e.time);
|
||||
li.appendChild(t);
|
||||
li.appendChild(document.createTextNode(e.message));
|
||||
listEl.appendChild(li);
|
||||
});
|
||||
}
|
||||
|
||||
function render(data) {
|
||||
const badge = document.getElementById('overall-badge');
|
||||
badge.className = 'badge ' + (data.status === 'healthy' ? 'healthy' : 'degraded');
|
||||
badge.innerHTML = '<span class="badge-dot"></span>' + data.status;
|
||||
|
||||
document.getElementById('uptime').textContent = 'uptime: ' + fmtDuration(data.uptime_seconds);
|
||||
document.getElementById('last-updated').textContent = 'last updated: ' + new Date().toLocaleTimeString();
|
||||
|
||||
// Download
|
||||
const variants = (data.download && data.download.variants) || [];
|
||||
const rows = document.getElementById('variant-rows');
|
||||
rows.innerHTML = '';
|
||||
document.getElementById('download-empty').style.display = variants.length ? 'none' : 'block';
|
||||
variants.forEach(v => {
|
||||
const tr = document.createElement('tr');
|
||||
if (v.segments_failed > 0) tr.classList.add('failed-row');
|
||||
if (!v.active) tr.classList.add('inactive-row');
|
||||
tr.innerHTML =
|
||||
'<td>' + v.resolution + '</td>' +
|
||||
'<td><span class="dot ' + (v.active ? 'on' : 'off') + '"></span>' + (v.active ? 'active' : 'stopped') + '</td>' +
|
||||
'<td>' + v.segments_ok + '</td>' +
|
||||
'<td>' + v.segments_failed + '</td>' +
|
||||
'<td>' + fmtTime(v.last_activity) + '</td>';
|
||||
rows.appendChild(tr);
|
||||
});
|
||||
renderErrors(document.getElementById('download-errors'), data.download && data.download.recent_errors);
|
||||
|
||||
// Transfer
|
||||
const t = data.transfer || {};
|
||||
const nasDot = document.getElementById('nas-dot');
|
||||
nasDot.className = 'dot ' + (t.nas_connected ? 'on' : 'off');
|
||||
document.getElementById('nas-text').textContent = t.nas_connected ? 'connected' : 'disconnected';
|
||||
document.getElementById('t-added').textContent = t.added ?? '-';
|
||||
document.getElementById('t-completed').textContent = t.completed ?? '-';
|
||||
document.getElementById('t-failed').textContent = t.failed ?? '-';
|
||||
document.getElementById('t-pending').textContent = t.pending ?? '-';
|
||||
document.getElementById('t-queue').textContent = t.queue_size ?? '-';
|
||||
document.getElementById('t-cleanup').textContent = t.cleanup_pending ?? '-';
|
||||
document.getElementById('t-bytes').textContent = fmtBytes(t.bytes_transferred);
|
||||
renderErrors(document.getElementById('transfer-errors'), t.recent_errors);
|
||||
|
||||
// Processing
|
||||
const p = data.processing || {};
|
||||
const stateEl = document.getElementById('p-state');
|
||||
const state = p.state || 'idle';
|
||||
stateEl.textContent = state;
|
||||
stateEl.className = 'state-pill state-' + state;
|
||||
document.getElementById('p-event').textContent = p.event_name || '-';
|
||||
document.getElementById('p-segments').textContent = p.segments_found ?? '-';
|
||||
document.getElementById('p-started').textContent = fmtTime(p.started_at);
|
||||
document.getElementById('p-finished').textContent = fmtTime(p.finished_at);
|
||||
renderErrors(document.getElementById('processing-errors'), p.recent_errors);
|
||||
}
|
||||
|
||||
async function poll() {
|
||||
try {
|
||||
const resp = await fetch('/status', { cache: 'no-store' });
|
||||
if (!resp.ok) throw new Error('status ' + resp.status);
|
||||
const data = await resp.json();
|
||||
connBanner.classList.remove('show');
|
||||
render(data);
|
||||
} catch (e) {
|
||||
connBanner.classList.add('show');
|
||||
}
|
||||
}
|
||||
|
||||
poll();
|
||||
setInterval(poll, POLL_INTERVAL_MS);
|
||||
</script>
|
||||
|
||||
</body>
|
||||
</html>
|
||||
@ -1,352 +0,0 @@
|
||||
package health
|
||||
|
||||
import (
|
||||
"sort"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
const maxRecentErrors = 10
|
||||
|
||||
type ErrorEntry struct {
|
||||
Time time.Time `json:"time"`
|
||||
Message string `json:"message"`
|
||||
}
|
||||
|
||||
type VariantStatus struct {
|
||||
Resolution string `json:"resolution"`
|
||||
Bandwidth uint32 `json:"bandwidth"`
|
||||
Active bool `json:"active"`
|
||||
SegmentsOK int `json:"segments_ok"`
|
||||
SegmentsFailed int `json:"segments_failed"`
|
||||
LastActivity time.Time `json:"last_activity"`
|
||||
LastError string `json:"last_error,omitempty"`
|
||||
}
|
||||
|
||||
// StatsProvider is satisfied by *transfer.QueueStats without pkg/health
|
||||
// needing to import pkg/transfer (which would create an import cycle, since
|
||||
// pkg/transfer calls into pkg/health to report status).
|
||||
type StatsProvider interface {
|
||||
GetStats() (added, completed, failed, pending int, bytes int64)
|
||||
}
|
||||
|
||||
type downloadState struct {
|
||||
variants map[string]*VariantStatus
|
||||
recentErrors []ErrorEntry
|
||||
}
|
||||
|
||||
type transferState struct {
|
||||
statsProvider StatsProvider
|
||||
queueSizeFn func() int
|
||||
cleanupPendingFn func() int
|
||||
nasConnectedFn func() bool
|
||||
nasConnected bool
|
||||
recentErrors []ErrorEntry
|
||||
}
|
||||
|
||||
type processingState struct {
|
||||
state string // idle | scanning | running | completed | failed
|
||||
eventName string
|
||||
segmentsFound int
|
||||
startedAt time.Time
|
||||
finishedAt time.Time
|
||||
lastError string
|
||||
recentErrors []ErrorEntry
|
||||
}
|
||||
|
||||
// Registry is the process-wide health/metrics store. All subsystems report
|
||||
// into it via the Record* methods; the health HTTP server reads it via
|
||||
// Snapshot(). Safe for concurrent use.
|
||||
type Registry struct {
|
||||
mu sync.RWMutex
|
||||
startedAt time.Time
|
||||
download downloadState
|
||||
transfer transferState
|
||||
processing processingState
|
||||
}
|
||||
|
||||
var (
|
||||
globalRegistry *Registry
|
||||
registryOnce sync.Once
|
||||
)
|
||||
|
||||
// MustGetRegistry returns the process-wide registry, mirroring the
|
||||
// constants.MustGetConfig() / logging.MustGetFactory() singleton style.
|
||||
func MustGetRegistry() *Registry {
|
||||
registryOnce.Do(func() {
|
||||
globalRegistry = New()
|
||||
})
|
||||
return globalRegistry
|
||||
}
|
||||
|
||||
func New() *Registry {
|
||||
return &Registry{
|
||||
startedAt: time.Now(),
|
||||
download: downloadState{
|
||||
variants: make(map[string]*VariantStatus),
|
||||
},
|
||||
processing: processingState{
|
||||
state: "idle",
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func appendBounded(errs []ErrorEntry, e ErrorEntry) []ErrorEntry {
|
||||
errs = append(errs, e)
|
||||
if len(errs) > maxRecentErrors {
|
||||
errs = errs[len(errs)-maxRecentErrors:]
|
||||
}
|
||||
return errs
|
||||
}
|
||||
|
||||
func errString(err error) string {
|
||||
if err == nil {
|
||||
return ""
|
||||
}
|
||||
return err.Error()
|
||||
}
|
||||
|
||||
// --- Download ---
|
||||
|
||||
func (r *Registry) RecordVariantStarted(resolution string, bandwidth uint32) {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
r.download.variants[resolution] = &VariantStatus{
|
||||
Resolution: resolution,
|
||||
Bandwidth: bandwidth,
|
||||
Active: true,
|
||||
LastActivity: time.Now(),
|
||||
}
|
||||
}
|
||||
|
||||
func (r *Registry) RecordVariantStopped(resolution string) {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
if v, ok := r.download.variants[resolution]; ok {
|
||||
v.Active = false
|
||||
}
|
||||
}
|
||||
|
||||
func (r *Registry) RecordSegmentDownloaded(resolution string) {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
v := r.ensureVariantLocked(resolution)
|
||||
v.SegmentsOK++
|
||||
v.LastActivity = time.Now()
|
||||
}
|
||||
|
||||
func (r *Registry) RecordSegmentFailed(resolution string, err error) {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
v := r.ensureVariantLocked(resolution)
|
||||
v.SegmentsFailed++
|
||||
v.LastActivity = time.Now()
|
||||
v.LastError = errString(err)
|
||||
r.download.recentErrors = appendBounded(r.download.recentErrors, ErrorEntry{
|
||||
Time: time.Now(),
|
||||
Message: resolution + ": " + errString(err),
|
||||
})
|
||||
}
|
||||
|
||||
func (r *Registry) RecordPlaylistError(resolution string, err error) {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
v := r.ensureVariantLocked(resolution)
|
||||
v.LastError = errString(err)
|
||||
r.download.recentErrors = appendBounded(r.download.recentErrors, ErrorEntry{
|
||||
Time: time.Now(),
|
||||
Message: resolution + " playlist: " + errString(err),
|
||||
})
|
||||
}
|
||||
|
||||
func (r *Registry) ensureVariantLocked(resolution string) *VariantStatus {
|
||||
v, ok := r.download.variants[resolution]
|
||||
if !ok {
|
||||
v = &VariantStatus{Resolution: resolution, Active: true}
|
||||
r.download.variants[resolution] = v
|
||||
}
|
||||
return v
|
||||
}
|
||||
|
||||
// --- Transfer ---
|
||||
|
||||
// AttachTransfer wires the registry to the live counters/state already
|
||||
// maintained by pkg/transfer instead of duplicating them. Called once from
|
||||
// TransferService construction.
|
||||
func (r *Registry) AttachTransfer(stats StatsProvider, queueSizeFn, cleanupPendingFn func() int, nasConnectedFn func() bool) {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
r.transfer.statsProvider = stats
|
||||
r.transfer.queueSizeFn = queueSizeFn
|
||||
r.transfer.cleanupPendingFn = cleanupPendingFn
|
||||
r.transfer.nasConnectedFn = nasConnectedFn
|
||||
}
|
||||
|
||||
func (r *Registry) RecordNASConnectivity(connected bool) {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
r.transfer.nasConnected = connected
|
||||
}
|
||||
|
||||
// RecordTransferError captures failures QueueStats' cumulative counters
|
||||
// don't individually retain (pre-queue failures, cleanup failures) so they
|
||||
// still surface in the bounded recent-errors list.
|
||||
func (r *Registry) RecordTransferError(sourcePath string, err error) {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
r.transfer.recentErrors = appendBounded(r.transfer.recentErrors, ErrorEntry{
|
||||
Time: time.Now(),
|
||||
Message: sourcePath + ": " + errString(err),
|
||||
})
|
||||
}
|
||||
|
||||
// --- Processing ---
|
||||
|
||||
func (r *Registry) RecordProcessingStageChange(state, eventName string) {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
r.processing.state = state
|
||||
if eventName != "" {
|
||||
r.processing.eventName = eventName
|
||||
}
|
||||
switch state {
|
||||
case "scanning":
|
||||
r.processing.startedAt = time.Now()
|
||||
r.processing.finishedAt = time.Time{}
|
||||
r.processing.lastError = ""
|
||||
case "completed", "failed":
|
||||
r.processing.finishedAt = time.Now()
|
||||
}
|
||||
}
|
||||
|
||||
func (r *Registry) RecordProcessingSegmentsFound(count int) {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
r.processing.segmentsFound = count
|
||||
}
|
||||
|
||||
func (r *Registry) RecordProcessingError(err error) {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
msg := errString(err)
|
||||
r.processing.lastError = msg
|
||||
r.processing.recentErrors = appendBounded(r.processing.recentErrors, ErrorEntry{
|
||||
Time: time.Now(),
|
||||
Message: msg,
|
||||
})
|
||||
}
|
||||
|
||||
// --- Snapshot ---
|
||||
|
||||
type DownloadSnapshot struct {
|
||||
Variants []*VariantStatus `json:"variants"`
|
||||
RecentErrors []ErrorEntry `json:"recent_errors"`
|
||||
}
|
||||
|
||||
type TransferSnapshot struct {
|
||||
NASConnected bool `json:"nas_connected"`
|
||||
QueueSize int `json:"queue_size"`
|
||||
CleanupPending int `json:"cleanup_pending"`
|
||||
Added int `json:"added"`
|
||||
Completed int `json:"completed"`
|
||||
Failed int `json:"failed"`
|
||||
Pending int `json:"pending"`
|
||||
BytesTransferred int64 `json:"bytes_transferred"`
|
||||
RecentErrors []ErrorEntry `json:"recent_errors"`
|
||||
}
|
||||
|
||||
type ProcessingSnapshot struct {
|
||||
State string `json:"state"`
|
||||
EventName string `json:"event_name,omitempty"`
|
||||
SegmentsFound int `json:"segments_found"`
|
||||
StartedAt *time.Time `json:"started_at,omitempty"`
|
||||
FinishedAt *time.Time `json:"finished_at,omitempty"`
|
||||
LastError string `json:"last_error,omitempty"`
|
||||
RecentErrors []ErrorEntry `json:"recent_errors"`
|
||||
}
|
||||
|
||||
type Snapshot struct {
|
||||
Status string `json:"status"`
|
||||
UptimeSec float64 `json:"uptime_seconds"`
|
||||
Download DownloadSnapshot `json:"download"`
|
||||
Transfer TransferSnapshot `json:"transfer"`
|
||||
Processing ProcessingSnapshot `json:"processing"`
|
||||
}
|
||||
|
||||
func (r *Registry) Snapshot() Snapshot {
|
||||
r.mu.RLock()
|
||||
defer r.mu.RUnlock()
|
||||
|
||||
variants := make([]*VariantStatus, 0, len(r.download.variants))
|
||||
for _, v := range r.download.variants {
|
||||
vCopy := *v
|
||||
variants = append(variants, &vCopy)
|
||||
}
|
||||
sort.Slice(variants, func(i, j int) bool { return variants[i].Resolution < variants[j].Resolution })
|
||||
|
||||
dl := DownloadSnapshot{
|
||||
Variants: variants,
|
||||
RecentErrors: append([]ErrorEntry(nil), r.download.recentErrors...),
|
||||
}
|
||||
|
||||
tr := TransferSnapshot{
|
||||
NASConnected: r.transfer.nasConnected,
|
||||
RecentErrors: append([]ErrorEntry(nil), r.transfer.recentErrors...),
|
||||
}
|
||||
if r.transfer.statsProvider != nil {
|
||||
added, completed, failed, pending, bytes := r.transfer.statsProvider.GetStats()
|
||||
tr.Added, tr.Completed, tr.Failed, tr.Pending, tr.BytesTransferred = added, completed, failed, pending, bytes
|
||||
}
|
||||
if r.transfer.queueSizeFn != nil {
|
||||
tr.QueueSize = r.transfer.queueSizeFn()
|
||||
}
|
||||
if r.transfer.cleanupPendingFn != nil {
|
||||
tr.CleanupPending = r.transfer.cleanupPendingFn()
|
||||
}
|
||||
if r.transfer.nasConnectedFn != nil {
|
||||
tr.NASConnected = r.transfer.nasConnectedFn()
|
||||
}
|
||||
|
||||
proc := ProcessingSnapshot{
|
||||
State: r.processing.state,
|
||||
EventName: r.processing.eventName,
|
||||
SegmentsFound: r.processing.segmentsFound,
|
||||
LastError: r.processing.lastError,
|
||||
RecentErrors: append([]ErrorEntry(nil), r.processing.recentErrors...),
|
||||
}
|
||||
if !r.processing.startedAt.IsZero() {
|
||||
t := r.processing.startedAt
|
||||
proc.StartedAt = &t
|
||||
}
|
||||
if !r.processing.finishedAt.IsZero() {
|
||||
t := r.processing.finishedAt
|
||||
proc.FinishedAt = &t
|
||||
}
|
||||
|
||||
snap := Snapshot{
|
||||
UptimeSec: time.Since(r.startedAt).Seconds(),
|
||||
Download: dl,
|
||||
Transfer: tr,
|
||||
Processing: proc,
|
||||
}
|
||||
snap.Status = computeStatus(dl, tr, proc)
|
||||
return snap
|
||||
}
|
||||
|
||||
// computeStatus gives one glanceable field: "degraded" the moment anything
|
||||
// worth investigating shows up, "healthy" otherwise. Transfer is only
|
||||
// considered for degradation once it's actually in use (Attached + has seen
|
||||
// activity), so a download-only run with NAS transfer disabled doesn't
|
||||
// falsely report degraded.
|
||||
func computeStatus(dl DownloadSnapshot, tr TransferSnapshot, proc ProcessingSnapshot) string {
|
||||
if len(dl.RecentErrors) > 0 || len(tr.RecentErrors) > 0 || len(proc.RecentErrors) > 0 {
|
||||
return "degraded"
|
||||
}
|
||||
if !tr.NASConnected && (tr.Added > 0 || tr.QueueSize > 0) {
|
||||
return "degraded"
|
||||
}
|
||||
if proc.State == "failed" {
|
||||
return "degraded"
|
||||
}
|
||||
return "healthy"
|
||||
}
|
||||
@ -1,60 +0,0 @@
|
||||
package health
|
||||
|
||||
import (
|
||||
"context"
|
||||
_ "embed"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"m3u8-downloader/pkg/config"
|
||||
"m3u8-downloader/pkg/logging"
|
||||
"net"
|
||||
"net/http"
|
||||
)
|
||||
|
||||
//go:embed dashboard.html
|
||||
var dashboardHTML []byte
|
||||
|
||||
// Start binds the health/status HTTP server and returns immediately once the
|
||||
// listener is bound, so callers see bind failures (e.g. port already in use)
|
||||
// synchronously rather than losing them in a background goroutine. The
|
||||
// returned *http.Server should be shut down by the caller via Shutdown(ctx).
|
||||
//
|
||||
// Security note: cfg.Addr is expected to be a bind-all address (e.g. ":8090")
|
||||
// so status can be checked from other devices on the local network during a
|
||||
// live event. There is no authentication or TLS on these endpoints - this is
|
||||
// only safe on a trusted local network, never exposed to the open internet.
|
||||
func Start(ctx context.Context, cfg *config.HealthConfig, reg *Registry) (*http.Server, error) {
|
||||
mux := http.NewServeMux()
|
||||
|
||||
mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
|
||||
w.Header().Set("Content-Type", "text/html; charset=utf-8")
|
||||
w.Write(dashboardHTML)
|
||||
})
|
||||
|
||||
mux.HandleFunc("/status", func(w http.ResponseWriter, r *http.Request) {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
if err := json.NewEncoder(w).Encode(reg.Snapshot()); err != nil {
|
||||
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||
}
|
||||
})
|
||||
|
||||
mux.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {
|
||||
w.WriteHeader(http.StatusOK)
|
||||
_, _ = w.Write([]byte("ok"))
|
||||
})
|
||||
|
||||
ln, err := net.Listen("tcp", cfg.Addr)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to bind health server to %s: %w", cfg.Addr, err)
|
||||
}
|
||||
|
||||
srv := &http.Server{Handler: mux}
|
||||
|
||||
go func() {
|
||||
if err := srv.Serve(ln); err != nil && err != http.ErrServerClosed {
|
||||
logging.MustGetFactory().App.Error("health server stopped unexpectedly", logging.KeyError, err.Error())
|
||||
}
|
||||
}()
|
||||
|
||||
return srv, nil
|
||||
}
|
||||
@ -5,50 +5,6 @@ import (
|
||||
"fmt"
|
||||
)
|
||||
|
||||
// HTTPError represents an HTTP error with status code and message
|
||||
type HTTPError struct {
|
||||
StatusCode int
|
||||
Message string
|
||||
}
|
||||
|
||||
// Error returns the string representation of the HTTP error
|
||||
func (e *HTTPError) Error() string {
|
||||
return fmt.Sprintf("HTTP %d: %s", e.StatusCode, e.Message)
|
||||
}
|
||||
|
||||
// Is implements error comparison for errors.Is
|
||||
func (e *HTTPError) Is(target error) bool {
|
||||
var httpErr *HTTPError
|
||||
if errors.As(target, &httpErr) {
|
||||
return e.StatusCode == httpErr.StatusCode
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// NewHTTPError creates a new HTTP error
|
||||
func NewHTTPError(statusCode int, message string) error {
|
||||
return &HTTPError{
|
||||
StatusCode: statusCode,
|
||||
Message: message,
|
||||
}
|
||||
}
|
||||
|
||||
// IsHTTPError checks if an error is an HTTP error
|
||||
func IsHTTPError(err error) bool {
|
||||
var httpErr *HTTPError
|
||||
return errors.As(err, &httpErr)
|
||||
}
|
||||
|
||||
// GetHTTPStatusCode extracts the status code from an HTTP error
|
||||
func GetHTTPStatusCode(err error) int {
|
||||
var httpErr *HTTPError
|
||||
if errors.As(err, &httpErr) {
|
||||
return httpErr.StatusCode
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
// Legacy support for existing code
|
||||
type HttpError struct {
|
||||
Code int
|
||||
}
|
||||
|
||||
@ -1,318 +0,0 @@
|
||||
package httpClient
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"net/http"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestHTTPError_Error(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
statusCode int
|
||||
message string
|
||||
want string
|
||||
}{
|
||||
{
|
||||
name: "basic http error",
|
||||
statusCode: 404,
|
||||
message: "Not Found",
|
||||
want: "HTTP 404: Not Found",
|
||||
},
|
||||
{
|
||||
name: "server error",
|
||||
statusCode: 500,
|
||||
message: "Internal Server Error",
|
||||
want: "HTTP 500: Internal Server Error",
|
||||
},
|
||||
{
|
||||
name: "unauthorized error",
|
||||
statusCode: 401,
|
||||
message: "Unauthorized",
|
||||
want: "HTTP 401: Unauthorized",
|
||||
},
|
||||
{
|
||||
name: "empty message",
|
||||
statusCode: 400,
|
||||
message: "",
|
||||
want: "HTTP 400: ",
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
err := &HTTPError{
|
||||
StatusCode: tt.statusCode,
|
||||
Message: tt.message,
|
||||
}
|
||||
|
||||
got := err.Error()
|
||||
if got != tt.want {
|
||||
t.Errorf("HTTPError.Error() = %q, want %q", got, tt.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestHTTPError_Is(t *testing.T) {
|
||||
err404 := &HTTPError{StatusCode: 404, Message: "Not Found"}
|
||||
err500 := &HTTPError{StatusCode: 500, Message: "Server Error"}
|
||||
otherErr404 := &HTTPError{StatusCode: 404, Message: "Different message"}
|
||||
regularError := fmt.Errorf("regular error")
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
err error
|
||||
target error
|
||||
want bool
|
||||
}{
|
||||
{
|
||||
name: "same error instance",
|
||||
err: err404,
|
||||
target: err404,
|
||||
want: true,
|
||||
},
|
||||
{
|
||||
name: "different HTTP errors with same status",
|
||||
err: err404,
|
||||
target: otherErr404,
|
||||
want: true,
|
||||
},
|
||||
{
|
||||
name: "different HTTP errors with different status",
|
||||
err: err404,
|
||||
target: err500,
|
||||
want: false,
|
||||
},
|
||||
{
|
||||
name: "HTTP error vs regular error",
|
||||
err: err404,
|
||||
target: regularError,
|
||||
want: false,
|
||||
},
|
||||
{
|
||||
name: "regular error vs HTTP error",
|
||||
err: regularError,
|
||||
target: err404,
|
||||
want: false,
|
||||
},
|
||||
{
|
||||
name: "nil target",
|
||||
err: err404,
|
||||
target: nil,
|
||||
want: false,
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
var got bool
|
||||
if httpErr, ok := tt.err.(*HTTPError); ok {
|
||||
got = httpErr.Is(tt.target)
|
||||
} else {
|
||||
got = false // Non-HTTP errors return false
|
||||
}
|
||||
if got != tt.want {
|
||||
t.Errorf("HTTPError.Is() = %v, want %v", got, tt.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestIsHTTPError(t *testing.T) {
|
||||
httpErr := &HTTPError{StatusCode: 404, Message: "Not Found"}
|
||||
regularErr := fmt.Errorf("regular error")
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
err error
|
||||
want bool
|
||||
}{
|
||||
{
|
||||
name: "http error",
|
||||
err: httpErr,
|
||||
want: true,
|
||||
},
|
||||
{
|
||||
name: "regular error",
|
||||
err: regularErr,
|
||||
want: false,
|
||||
},
|
||||
{
|
||||
name: "nil error",
|
||||
err: nil,
|
||||
want: false,
|
||||
},
|
||||
{
|
||||
name: "wrapped http error",
|
||||
err: fmt.Errorf("wrapped: %w", httpErr),
|
||||
want: true,
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
got := IsHTTPError(tt.err)
|
||||
if got != tt.want {
|
||||
t.Errorf("IsHTTPError() = %v, want %v", got, tt.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestGetHTTPStatusCode(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
err error
|
||||
want int
|
||||
}{
|
||||
{
|
||||
name: "http error 404",
|
||||
err: &HTTPError{StatusCode: 404, Message: "Not Found"},
|
||||
want: 404,
|
||||
},
|
||||
{
|
||||
name: "http error 500",
|
||||
err: &HTTPError{StatusCode: 500, Message: "Server Error"},
|
||||
want: 500,
|
||||
},
|
||||
{
|
||||
name: "wrapped http error",
|
||||
err: fmt.Errorf("wrapped: %w", &HTTPError{StatusCode: 403, Message: "Forbidden"}),
|
||||
want: 403,
|
||||
},
|
||||
{
|
||||
name: "regular error",
|
||||
err: fmt.Errorf("regular error"),
|
||||
want: 0,
|
||||
},
|
||||
{
|
||||
name: "nil error",
|
||||
err: nil,
|
||||
want: 0,
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
got := GetHTTPStatusCode(tt.err)
|
||||
if got != tt.want {
|
||||
t.Errorf("GetHTTPStatusCode() = %v, want %v", got, tt.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestNewHTTPError(t *testing.T) {
|
||||
statusCode := 404
|
||||
message := "Page not found"
|
||||
|
||||
err := NewHTTPError(statusCode, message)
|
||||
|
||||
// Check type
|
||||
httpErr, ok := err.(*HTTPError)
|
||||
if !ok {
|
||||
t.Fatalf("NewHTTPError should return *HTTPError, got %T", err)
|
||||
}
|
||||
|
||||
// Check fields
|
||||
if httpErr.StatusCode != statusCode {
|
||||
t.Errorf("Expected StatusCode=%d, got %d", statusCode, httpErr.StatusCode)
|
||||
}
|
||||
if httpErr.Message != message {
|
||||
t.Errorf("Expected Message=%q, got %q", message, httpErr.Message)
|
||||
}
|
||||
|
||||
// Check error string
|
||||
expectedErrorString := fmt.Sprintf("HTTP %d: %s", statusCode, message)
|
||||
if httpErr.Error() != expectedErrorString {
|
||||
t.Errorf("Expected error string=%q, got %q", expectedErrorString, httpErr.Error())
|
||||
}
|
||||
}
|
||||
|
||||
func TestHTTPError_StatusCodeChecks(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
statusCode int
|
||||
isClient bool
|
||||
isServer bool
|
||||
}{
|
||||
{"200 OK", 200, false, false},
|
||||
{"400 Bad Request", 400, true, false},
|
||||
{"401 Unauthorized", 401, true, false},
|
||||
{"404 Not Found", 404, true, false},
|
||||
{"499 Client Error", 499, true, false},
|
||||
{"500 Server Error", 500, false, true},
|
||||
{"502 Bad Gateway", 502, false, true},
|
||||
{"599 Server Error", 599, false, true},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
err := &HTTPError{StatusCode: tt.statusCode, Message: "test"}
|
||||
|
||||
isClient := err.StatusCode >= 400 && err.StatusCode < 500
|
||||
isServer := err.StatusCode >= 500 && err.StatusCode < 600
|
||||
|
||||
if isClient != tt.isClient {
|
||||
t.Errorf("Status %d: expected isClient=%v, got %v", tt.statusCode, tt.isClient, isClient)
|
||||
}
|
||||
if isServer != tt.isServer {
|
||||
t.Errorf("Status %d: expected isServer=%v, got %v", tt.statusCode, tt.isServer, isServer)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestHTTPError_Integration(t *testing.T) {
|
||||
// Test that HTTPError integrates well with standard error handling
|
||||
err := NewHTTPError(http.StatusNotFound, "Resource not found")
|
||||
|
||||
// Should be able to use with errors.Is
|
||||
target := &HTTPError{StatusCode: http.StatusNotFound}
|
||||
if !err.(*HTTPError).Is(target) {
|
||||
t.Error("HTTPError should match target with same status code")
|
||||
}
|
||||
|
||||
// Should be detectable as HTTPError
|
||||
if !IsHTTPError(err) {
|
||||
t.Error("Should be detectable as HTTPError")
|
||||
}
|
||||
|
||||
// Should return correct status code
|
||||
if GetHTTPStatusCode(err) != http.StatusNotFound {
|
||||
t.Error("Should return correct status code")
|
||||
}
|
||||
|
||||
// Should have meaningful string representation
|
||||
errorString := err.Error()
|
||||
if !strings.Contains(errorString, "404") {
|
||||
t.Error("Error string should contain status code")
|
||||
}
|
||||
if !strings.Contains(errorString, "Resource not found") {
|
||||
t.Error("Error string should contain message")
|
||||
}
|
||||
}
|
||||
|
||||
func TestHTTPError_EdgeCases(t *testing.T) {
|
||||
// Test with zero status code
|
||||
err := NewHTTPError(0, "Zero status")
|
||||
if err.Error() != "HTTP 0: Zero status" {
|
||||
t.Errorf("Unexpected error string for zero status: %s", err.Error())
|
||||
}
|
||||
|
||||
// Test with very long message
|
||||
longMessage := strings.Repeat("a", 1000)
|
||||
err = NewHTTPError(500, longMessage)
|
||||
if !strings.Contains(err.Error(), longMessage) {
|
||||
t.Error("Long message should be preserved")
|
||||
}
|
||||
|
||||
// Test status code boundaries
|
||||
for _, code := range []int{399, 400, 499, 500, 599, 600} {
|
||||
err := &HTTPError{StatusCode: code}
|
||||
// Should not panic
|
||||
_ = err.Error()
|
||||
}
|
||||
}
|
||||
@ -1,72 +0,0 @@
|
||||
package logging
|
||||
|
||||
import (
|
||||
"log/slog"
|
||||
"m3u8-downloader/pkg/config"
|
||||
"m3u8-downloader/pkg/constants"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync"
|
||||
)
|
||||
|
||||
// Factory holds one structured logger per pipeline stage, each writing JSON
|
||||
// lines to its own file under LoggingConfig.Dir. These are additive to the
|
||||
// existing stdlib log.* calls, not a replacement for them.
|
||||
type Factory struct {
|
||||
App *slog.Logger
|
||||
Download *slog.Logger
|
||||
Transfer *slog.Logger
|
||||
Processing *slog.Logger
|
||||
}
|
||||
|
||||
var (
|
||||
globalFactory *Factory
|
||||
factoryOnce sync.Once
|
||||
)
|
||||
|
||||
// MustGetFactory returns the process-wide logging factory, building it from
|
||||
// the config singleton on first use. Mirrors constants.MustGetConfig().
|
||||
func MustGetFactory() *Factory {
|
||||
factoryOnce.Do(func() {
|
||||
globalFactory = New(&constants.MustGetConfig().Logging)
|
||||
})
|
||||
return globalFactory
|
||||
}
|
||||
|
||||
func New(cfg *config.LoggingConfig) *Factory {
|
||||
level := parseLevel(cfg.Level)
|
||||
return &Factory{
|
||||
App: newSubsystemLogger(cfg.Dir, "app.log", "app", level),
|
||||
Download: newSubsystemLogger(cfg.Dir, "download.log", "download", level),
|
||||
Transfer: newSubsystemLogger(cfg.Dir, "transfer.log", "transfer", level),
|
||||
Processing: newSubsystemLogger(cfg.Dir, "processing.log", "processing", level),
|
||||
}
|
||||
}
|
||||
|
||||
// newSubsystemLogger opens (or creates) the subsystem's log file in append
|
||||
// mode. A logging misconfiguration must never take down an active recording,
|
||||
// so failure to open the file falls back to stderr instead of panicking.
|
||||
func newSubsystemLogger(dir, filename, component string, level slog.Level) *slog.Logger {
|
||||
path := filepath.Join(dir, filename)
|
||||
f, err := os.OpenFile(path, os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0666)
|
||||
if err != nil {
|
||||
fallback := slog.New(slog.NewJSONHandler(os.Stderr, &slog.HandlerOptions{Level: level})).With("component", component)
|
||||
fallback.Error("failed to open log file, falling back to stderr", KeyFile, path, KeyError, err.Error())
|
||||
return fallback
|
||||
}
|
||||
return slog.New(slog.NewJSONHandler(f, &slog.HandlerOptions{Level: level})).With("component", component)
|
||||
}
|
||||
|
||||
func parseLevel(level string) slog.Level {
|
||||
switch strings.ToLower(strings.TrimSpace(level)) {
|
||||
case "debug":
|
||||
return slog.LevelDebug
|
||||
case "warn", "warning":
|
||||
return slog.LevelWarn
|
||||
case "error":
|
||||
return slog.LevelError
|
||||
default:
|
||||
return slog.LevelInfo
|
||||
}
|
||||
}
|
||||
@ -1,13 +0,0 @@
|
||||
package logging
|
||||
|
||||
const (
|
||||
KeyResolution = "resolution"
|
||||
KeyBandwidth = "bandwidth"
|
||||
KeySegment = "segment"
|
||||
KeySeq = "seq"
|
||||
KeyFile = "file"
|
||||
KeyBytes = "bytes"
|
||||
KeyEvent = "event"
|
||||
KeyError = "error"
|
||||
KeyAttempt = "attempt"
|
||||
)
|
||||
@ -4,26 +4,14 @@ import (
|
||||
"encoding/json"
|
||||
"log"
|
||||
"m3u8-downloader/pkg/constants"
|
||||
"m3u8-downloader/pkg/logging"
|
||||
"m3u8-downloader/pkg/utils"
|
||||
"os"
|
||||
"sort"
|
||||
"strconv"
|
||||
"sync"
|
||||
)
|
||||
|
||||
type ManifestWriter struct {
|
||||
ManifestPath string
|
||||
Segments []ManifestItem
|
||||
// Index maps a sequence number to its position in Segments. It holds
|
||||
// positions rather than pointers because append reallocates the backing
|
||||
// array, which would leave previously handed-out element pointers aimed at
|
||||
// a stale copy.
|
||||
Index map[string]int
|
||||
|
||||
// mu guards the fields above: every variant downloads concurrently, and
|
||||
// each reports its segments as they land.
|
||||
mu sync.Mutex
|
||||
Index map[string]*ManifestItem
|
||||
}
|
||||
|
||||
type ManifestItem struct {
|
||||
@ -32,75 +20,51 @@ type ManifestItem struct {
|
||||
}
|
||||
|
||||
func NewManifestWriter(eventName string) *ManifestWriter {
|
||||
cfg := constants.MustGetConfig()
|
||||
return &ManifestWriter{
|
||||
ManifestPath: cfg.GetManifestPath(eventName),
|
||||
ManifestPath: constants.ManifestPath + "/" + eventName + ".json",
|
||||
Segments: make([]ManifestItem, 0),
|
||||
Index: make(map[string]int),
|
||||
Index: make(map[string]*ManifestItem),
|
||||
}
|
||||
}
|
||||
|
||||
// AddOrUpdateSegment records that a segment was downloaded, keeping the
|
||||
// highest-resolution variant seen for each sequence number.
|
||||
func (m *ManifestWriter) AddOrUpdateSegment(seqNo string, resolution string) {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
|
||||
if m.Index == nil {
|
||||
m.Index = make(map[string]int)
|
||||
m.Index = make(map[string]*ManifestItem)
|
||||
}
|
||||
|
||||
if m.Segments == nil {
|
||||
m.Segments = make([]ManifestItem, 0)
|
||||
}
|
||||
|
||||
if pos, ok := m.Index[seqNo]; ok {
|
||||
if ResolutionHeight(resolution) > ResolutionHeight(m.Segments[pos].Resolution) {
|
||||
m.Segments[pos].Resolution = resolution
|
||||
if existing, ok := m.Index[seqNo]; ok {
|
||||
if resolution > existing.Resolution {
|
||||
existing.Resolution = resolution
|
||||
}
|
||||
return
|
||||
} else {
|
||||
item := ManifestItem{
|
||||
SeqNo: seqNo,
|
||||
Resolution: resolution,
|
||||
}
|
||||
m.Segments = append(m.Segments, item)
|
||||
m.Index[seqNo] = &item
|
||||
}
|
||||
|
||||
m.Segments = append(m.Segments, ManifestItem{SeqNo: seqNo, Resolution: resolution})
|
||||
m.Index[seqNo] = len(m.Segments) - 1
|
||||
}
|
||||
|
||||
// Len reports how many distinct sequence numbers have been recorded.
|
||||
func (m *ManifestWriter) Len() int {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
return len(m.Segments)
|
||||
}
|
||||
|
||||
func (m *ManifestWriter) WriteManifest() {
|
||||
// Sort a copy: reordering Segments in place would invalidate Index, and a
|
||||
// recording may still be writing into it.
|
||||
m.mu.Lock()
|
||||
segments := make([]ManifestItem, len(m.Segments))
|
||||
copy(segments, m.Segments)
|
||||
m.mu.Unlock()
|
||||
|
||||
sort.Slice(segments, func(i, j int) bool {
|
||||
return lessSeqNo(segments[i].SeqNo, segments[j].SeqNo)
|
||||
sort.Slice(m.Segments, func(i, j int) bool {
|
||||
return m.Segments[i].SeqNo < m.Segments[j].SeqNo
|
||||
})
|
||||
|
||||
data, err := json.MarshalIndent(segments, "", " ")
|
||||
data, err := json.MarshalIndent(m.Segments, "", " ")
|
||||
if err != nil {
|
||||
log.Printf("Failed to marshal manifest: %v", err)
|
||||
logging.MustGetFactory().Download.Error("failed to marshal manifest", logging.KeyError, err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
if err := utils.ValidateWritablePath(m.ManifestPath); err != nil {
|
||||
log.Printf("Manifest path validation failed: %v", err)
|
||||
logging.MustGetFactory().Download.Error("manifest path validation failed", logging.KeyError, err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
file, err := os.Create(m.ManifestPath)
|
||||
if err != nil {
|
||||
log.Printf("Failed to create manifest file: %v", err)
|
||||
logging.MustGetFactory().Download.Error("failed to create manifest file", logging.KeyError, err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
@ -109,19 +73,6 @@ func (m *ManifestWriter) WriteManifest() {
|
||||
_, err = file.Write(data)
|
||||
if err != nil {
|
||||
log.Printf("Failed to write manifest file: %v", err)
|
||||
logging.MustGetFactory().Download.Error("failed to write manifest file", logging.KeyError, err.Error())
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
// lessSeqNo orders sequence numbers numerically, so segment 9 precedes segment
|
||||
// 10 instead of sorting after it as the strings would. Values that are not
|
||||
// numeric fall back to lexicographic order.
|
||||
func lessSeqNo(a, b string) bool {
|
||||
ai, aErr := strconv.ParseUint(a, 10, 64)
|
||||
bi, bErr := strconv.ParseUint(b, 10, 64)
|
||||
if aErr == nil && bErr == nil {
|
||||
return ai < bi
|
||||
}
|
||||
return a < b
|
||||
}
|
||||
|
||||
@ -1,409 +0,0 @@
|
||||
package media
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"sync"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestManifestWriter_NewManifestWriter(t *testing.T) {
|
||||
// Set up temporary environment for testing
|
||||
tempDir, err := os.MkdirTemp("", "manifest_test_*")
|
||||
if err != nil {
|
||||
t.Fatalf("Failed to create temp dir: %v", err)
|
||||
}
|
||||
defer os.RemoveAll(tempDir)
|
||||
|
||||
// Set environment variable to use temp directory
|
||||
os.Setenv("LOCAL_OUTPUT_DIR", tempDir)
|
||||
defer os.Unsetenv("LOCAL_OUTPUT_DIR")
|
||||
|
||||
eventName := "test-event"
|
||||
writer := NewManifestWriter(eventName)
|
||||
|
||||
if writer == nil {
|
||||
t.Fatal("NewManifestWriter() returned nil")
|
||||
}
|
||||
|
||||
if writer.Segments == nil {
|
||||
t.Error("Segments should be initialized")
|
||||
}
|
||||
if writer.Index == nil {
|
||||
t.Error("Index should be initialized")
|
||||
}
|
||||
if len(writer.Segments) != 0 {
|
||||
t.Errorf("Segments should be empty, got %d items", len(writer.Segments))
|
||||
}
|
||||
}
|
||||
|
||||
func TestManifestWriter_AddOrUpdateSegment(t *testing.T) {
|
||||
writer := &ManifestWriter{
|
||||
ManifestPath: "test.json",
|
||||
Segments: make([]ManifestItem, 0),
|
||||
Index: make(map[string]int),
|
||||
}
|
||||
|
||||
// Test adding new segment
|
||||
writer.AddOrUpdateSegment("1001", "1080p")
|
||||
|
||||
if len(writer.Segments) != 1 {
|
||||
t.Errorf("Expected 1 segment, got %d", len(writer.Segments))
|
||||
}
|
||||
if writer.Segments[0].SeqNo != "1001" {
|
||||
t.Errorf("Expected SeqNo '1001', got '%s'", writer.Segments[0].SeqNo)
|
||||
}
|
||||
if writer.Segments[0].Resolution != "1080p" {
|
||||
t.Errorf("Expected Resolution '1080p', got '%s'", writer.Segments[0].Resolution)
|
||||
}
|
||||
|
||||
// Test updating existing segment with higher resolution
|
||||
writer.AddOrUpdateSegment("1001", "1440p")
|
||||
|
||||
if len(writer.Segments) != 1 {
|
||||
t.Errorf("Segments count should remain 1 after update, got %d", len(writer.Segments))
|
||||
}
|
||||
if writer.Segments[0].Resolution != "1440p" {
|
||||
t.Errorf("Expected updated resolution '1440p', got '%s'", writer.Segments[0].Resolution)
|
||||
}
|
||||
|
||||
// Test updating existing segment with lower resolution (should not change)
|
||||
writer.AddOrUpdateSegment("1001", "720p")
|
||||
|
||||
if writer.Segments[0].Resolution != "1440p" {
|
||||
t.Errorf("Resolution should remain '1440p', got '%s'", writer.Segments[0].Resolution)
|
||||
}
|
||||
|
||||
// Test adding different segment
|
||||
writer.AddOrUpdateSegment("1002", "720p")
|
||||
|
||||
if len(writer.Segments) != 2 {
|
||||
t.Errorf("Expected 2 segments, got %d", len(writer.Segments))
|
||||
}
|
||||
}
|
||||
|
||||
func TestManifestWriter_AddOrUpdateSegment_NilFields(t *testing.T) {
|
||||
writer := &ManifestWriter{
|
||||
ManifestPath: "test.json",
|
||||
}
|
||||
|
||||
// Test with nil fields (should initialize them)
|
||||
writer.AddOrUpdateSegment("1001", "1080p")
|
||||
|
||||
if writer.Segments == nil {
|
||||
t.Error("Segments should be initialized")
|
||||
}
|
||||
if writer.Index == nil {
|
||||
t.Error("Index should be initialized")
|
||||
}
|
||||
if len(writer.Segments) != 1 {
|
||||
t.Errorf("Expected 1 segment, got %d", len(writer.Segments))
|
||||
}
|
||||
}
|
||||
|
||||
func TestManifestWriter_WriteManifest(t *testing.T) {
|
||||
tempDir, err := os.MkdirTemp("", "manifest_test_*")
|
||||
if err != nil {
|
||||
t.Fatalf("Failed to create temp dir: %v", err)
|
||||
}
|
||||
defer os.RemoveAll(tempDir)
|
||||
|
||||
manifestPath := filepath.Join(tempDir, "test-manifest.json")
|
||||
writer := &ManifestWriter{
|
||||
ManifestPath: manifestPath,
|
||||
Segments: make([]ManifestItem, 0),
|
||||
Index: make(map[string]int),
|
||||
}
|
||||
|
||||
// Add some test segments out of order
|
||||
writer.AddOrUpdateSegment("1003", "1080p")
|
||||
writer.AddOrUpdateSegment("1001", "720p")
|
||||
writer.AddOrUpdateSegment("1002", "1080p")
|
||||
|
||||
// Write manifest
|
||||
writer.WriteManifest()
|
||||
|
||||
// Verify file was created
|
||||
if _, err := os.Stat(manifestPath); os.IsNotExist(err) {
|
||||
t.Fatalf("Manifest file was not created: %s", manifestPath)
|
||||
}
|
||||
|
||||
// Read and verify content
|
||||
content, err := os.ReadFile(manifestPath)
|
||||
if err != nil {
|
||||
t.Fatalf("Failed to read manifest file: %v", err)
|
||||
}
|
||||
|
||||
var segments []ManifestItem
|
||||
err = json.Unmarshal(content, &segments)
|
||||
if err != nil {
|
||||
t.Fatalf("Failed to unmarshal manifest JSON: %v", err)
|
||||
}
|
||||
|
||||
// Verify segments are sorted by sequence number
|
||||
if len(segments) != 3 {
|
||||
t.Errorf("Expected 3 segments in manifest, got %d", len(segments))
|
||||
}
|
||||
|
||||
expectedOrder := []string{"1001", "1002", "1003"}
|
||||
for i, segment := range segments {
|
||||
if segment.SeqNo != expectedOrder[i] {
|
||||
t.Errorf("Segment %d: expected SeqNo '%s', got '%s'", i, expectedOrder[i], segment.SeqNo)
|
||||
}
|
||||
}
|
||||
|
||||
// Verify content structure
|
||||
if segments[0].Resolution != "720p" {
|
||||
t.Errorf("Expected first segment resolution '720p', got '%s'", segments[0].Resolution)
|
||||
}
|
||||
if segments[1].Resolution != "1080p" {
|
||||
t.Errorf("Expected second segment resolution '1080p', got '%s'", segments[1].Resolution)
|
||||
}
|
||||
}
|
||||
|
||||
func TestManifestWriter_WriteManifest_EmptySegments(t *testing.T) {
|
||||
tempDir, err := os.MkdirTemp("", "manifest_test_*")
|
||||
if err != nil {
|
||||
t.Fatalf("Failed to create temp dir: %v", err)
|
||||
}
|
||||
defer os.RemoveAll(tempDir)
|
||||
|
||||
manifestPath := filepath.Join(tempDir, "empty-manifest.json")
|
||||
writer := &ManifestWriter{
|
||||
ManifestPath: manifestPath,
|
||||
Segments: make([]ManifestItem, 0),
|
||||
Index: make(map[string]int),
|
||||
}
|
||||
|
||||
// Write empty manifest
|
||||
writer.WriteManifest()
|
||||
|
||||
// Verify file was created
|
||||
if _, err := os.Stat(manifestPath); os.IsNotExist(err) {
|
||||
t.Fatalf("Manifest file was not created: %s", manifestPath)
|
||||
}
|
||||
|
||||
// Read and verify content is empty array
|
||||
content, err := os.ReadFile(manifestPath)
|
||||
if err != nil {
|
||||
t.Fatalf("Failed to read manifest file: %v", err)
|
||||
}
|
||||
|
||||
var segments []ManifestItem
|
||||
err = json.Unmarshal(content, &segments)
|
||||
if err != nil {
|
||||
t.Fatalf("Failed to unmarshal manifest JSON: %v", err)
|
||||
}
|
||||
|
||||
if len(segments) != 0 {
|
||||
t.Errorf("Expected empty segments array, got %d items", len(segments))
|
||||
}
|
||||
}
|
||||
|
||||
func TestManifestWriter_WriteManifest_InvalidPath(t *testing.T) {
|
||||
writer := &ManifestWriter{
|
||||
ManifestPath: "/invalid/path/that/does/not/exist/manifest.json",
|
||||
Segments: []ManifestItem{{SeqNo: "1001", Resolution: "1080p"}},
|
||||
Index: make(map[string]int),
|
||||
}
|
||||
|
||||
// This should not panic, just fail gracefully
|
||||
writer.WriteManifest()
|
||||
|
||||
// Test passes if no panic occurs
|
||||
}
|
||||
|
||||
func TestManifestItem_JSONSerialization(t *testing.T) {
|
||||
item := ManifestItem{
|
||||
SeqNo: "1001",
|
||||
Resolution: "1080p",
|
||||
}
|
||||
|
||||
// Test marshaling
|
||||
data, err := json.Marshal(item)
|
||||
if err != nil {
|
||||
t.Fatalf("Failed to marshal ManifestItem: %v", err)
|
||||
}
|
||||
|
||||
// Test unmarshaling
|
||||
var unmarshaled ManifestItem
|
||||
err = json.Unmarshal(data, &unmarshaled)
|
||||
if err != nil {
|
||||
t.Fatalf("Failed to unmarshal ManifestItem: %v", err)
|
||||
}
|
||||
|
||||
if unmarshaled.SeqNo != item.SeqNo {
|
||||
t.Errorf("SeqNo mismatch: expected '%s', got '%s'", item.SeqNo, unmarshaled.SeqNo)
|
||||
}
|
||||
if unmarshaled.Resolution != item.Resolution {
|
||||
t.Errorf("Resolution mismatch: expected '%s', got '%s'", item.Resolution, unmarshaled.Resolution)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolutionHeight(t *testing.T) {
|
||||
cases := map[string]int{
|
||||
"1080p": 1080,
|
||||
"720p": 720,
|
||||
"2160p": 2160,
|
||||
"1440p": 1440,
|
||||
"240p": 240,
|
||||
// Not a <height>p label: must rank below every real resolution rather
|
||||
// than accidentally outranking them.
|
||||
"unknown": 0,
|
||||
"": 0,
|
||||
"p": 0,
|
||||
}
|
||||
|
||||
for label, want := range cases {
|
||||
if got := ResolutionHeight(label); got != want {
|
||||
t.Errorf("ResolutionHeight(%q) = %d, want %d", label, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestAddOrUpdateSegmentKeepsHighestResolution pins the comparison used to
|
||||
// choose between variants. Comparing the labels as strings — as this once did —
|
||||
// ranks "720p" above "1080p", quietly downgrading the manifest.
|
||||
func TestAddOrUpdateSegmentKeepsHighestResolution(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
arriving []string
|
||||
want string
|
||||
}{
|
||||
{"higher replaces lower", []string{"720p", "1080p"}, "1080p"},
|
||||
{"lower does not replace higher", []string{"1080p", "720p"}, "1080p"},
|
||||
{"fewer digits must not win", []string{"1080p", "480p"}, "1080p"},
|
||||
{"4k beats 1080p", []string{"1080p", "2160p"}, "2160p"},
|
||||
{"unknown loses to a real resolution", []string{"unknown", "480p"}, "480p"},
|
||||
{"unknown does not displace a real resolution", []string{"1080p", "unknown"}, "1080p"},
|
||||
{"repeated identical resolution", []string{"720p", "720p"}, "720p"},
|
||||
}
|
||||
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
w := &ManifestWriter{}
|
||||
for _, res := range tc.arriving {
|
||||
w.AddOrUpdateSegment("1001", res)
|
||||
}
|
||||
|
||||
if len(w.Segments) != 1 {
|
||||
t.Fatalf("expected a single entry, got %d", len(w.Segments))
|
||||
}
|
||||
if got := w.Segments[0].Resolution; got != tc.want {
|
||||
t.Errorf("kept %q, want %q (arrival order %v)", got, tc.want, tc.arriving)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestAddOrUpdateSegmentUpdatesTheStoredEntry guards the aliasing bug: the
|
||||
// index once held a pointer to a local copy rather than to the stored element,
|
||||
// so updates were applied to an orphan and never reached the written manifest.
|
||||
func TestAddOrUpdateSegmentUpdatesTheStoredEntry(t *testing.T) {
|
||||
w := &ManifestWriter{}
|
||||
|
||||
// Enough entries to force at least one append reallocation, which would
|
||||
// invalidate any pointer held into the backing array.
|
||||
for i := 0; i < 64; i++ {
|
||||
w.AddOrUpdateSegment(strconv.Itoa(1000+i), "480p")
|
||||
}
|
||||
for i := 0; i < 64; i++ {
|
||||
w.AddOrUpdateSegment(strconv.Itoa(1000+i), "1080p")
|
||||
}
|
||||
|
||||
if len(w.Segments) != 64 {
|
||||
t.Fatalf("expected 64 entries, got %d", len(w.Segments))
|
||||
}
|
||||
for _, seg := range w.Segments {
|
||||
if seg.Resolution != "1080p" {
|
||||
t.Fatalf("segment %s kept %q; the upgrade never reached the stored entry", seg.SeqNo, seg.Resolution)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestAddOrUpdateSegmentIsConcurrencySafe reflects real use: every variant
|
||||
// downloads in its own goroutine and reports segments as they land.
|
||||
func TestAddOrUpdateSegmentIsConcurrencySafe(t *testing.T) {
|
||||
w := &ManifestWriter{}
|
||||
resolutions := []string{"240p", "480p", "720p", "1080p"}
|
||||
|
||||
var wg sync.WaitGroup
|
||||
for _, res := range resolutions {
|
||||
wg.Add(1)
|
||||
go func(res string) {
|
||||
defer wg.Done()
|
||||
for seq := 0; seq < 200; seq++ {
|
||||
w.AddOrUpdateSegment(strconv.Itoa(seq), res)
|
||||
}
|
||||
}(res)
|
||||
}
|
||||
wg.Wait()
|
||||
|
||||
if got := w.Len(); got != 200 {
|
||||
t.Errorf("expected 200 distinct sequence numbers, got %d", got)
|
||||
}
|
||||
for _, seg := range w.Segments {
|
||||
if seg.Resolution != "1080p" {
|
||||
t.Errorf("segment %s = %q, want the highest variant 1080p", seg.SeqNo, seg.Resolution)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestWriteManifestSortsNumerically checks that segment 9 precedes segment 10,
|
||||
// which string ordering would reverse.
|
||||
func TestWriteManifestSortsNumerically(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
w := &ManifestWriter{ManifestPath: filepath.Join(dir, "event.json")}
|
||||
|
||||
for _, seq := range []string{"100", "9", "1000", "10", "1"} {
|
||||
w.AddOrUpdateSegment(seq, "1080p")
|
||||
}
|
||||
w.WriteManifest()
|
||||
|
||||
data, err := os.ReadFile(w.ManifestPath)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to read manifest: %v", err)
|
||||
}
|
||||
|
||||
var items []ManifestItem
|
||||
if err := json.Unmarshal(data, &items); err != nil {
|
||||
t.Fatalf("manifest is not valid JSON: %v", err)
|
||||
}
|
||||
|
||||
want := []string{"1", "9", "10", "100", "1000"}
|
||||
if len(items) != len(want) {
|
||||
t.Fatalf("expected %d entries, got %d", len(want), len(items))
|
||||
}
|
||||
for i, seq := range want {
|
||||
if items[i].SeqNo != seq {
|
||||
t.Errorf("position %d = %q, want %q (full order: %v)", i, items[i].SeqNo, seq, items)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestWriteManifestLeavesIndexUsable ensures writing mid-recording does not
|
||||
// corrupt the writer: sorting in place would desynchronise Index from Segments.
|
||||
func TestWriteManifestLeavesIndexUsable(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
w := &ManifestWriter{ManifestPath: filepath.Join(dir, "event.json")}
|
||||
|
||||
w.AddOrUpdateSegment("20", "480p")
|
||||
w.AddOrUpdateSegment("10", "480p")
|
||||
w.WriteManifest()
|
||||
|
||||
// After the write, an upgrade must still land on the right entry.
|
||||
w.AddOrUpdateSegment("20", "1080p")
|
||||
|
||||
pos, ok := w.Index["20"]
|
||||
if !ok {
|
||||
t.Fatal("sequence 20 missing from the index after WriteManifest")
|
||||
}
|
||||
if got := w.Segments[pos].Resolution; got != "1080p" {
|
||||
t.Errorf("sequence 20 = %q, want 1080p; the index no longer matches the slice", got)
|
||||
}
|
||||
if w.Segments[w.Index["10"]].SeqNo != "10" {
|
||||
t.Error("index and slice disagree after WriteManifest")
|
||||
}
|
||||
}
|
||||
@ -6,7 +6,6 @@ import (
|
||||
"io"
|
||||
"m3u8-downloader/pkg/constants"
|
||||
"m3u8-downloader/pkg/httpClient"
|
||||
"m3u8-downloader/pkg/logging"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"os"
|
||||
@ -34,7 +33,6 @@ func (j SegmentJob) Key() string {
|
||||
func DownloadSegment(ctx context.Context, client *http.Client, segmentURL string, outputDir string) error {
|
||||
for attempt := 0; attempt < 2; attempt++ {
|
||||
if attempt > 0 {
|
||||
logging.MustGetFactory().Download.Debug("retrying segment download", logging.KeyAttempt, attempt, "url", segmentURL)
|
||||
time.Sleep(300 * time.Millisecond)
|
||||
}
|
||||
req, err := http.NewRequestWithContext(ctx, "GET", segmentURL, nil)
|
||||
|
||||
@ -7,13 +7,10 @@ import (
|
||||
"github.com/grafov/m3u8"
|
||||
"log"
|
||||
"m3u8-downloader/pkg/constants"
|
||||
"m3u8-downloader/pkg/health"
|
||||
"m3u8-downloader/pkg/httpClient"
|
||||
"m3u8-downloader/pkg/logging"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"path"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
@ -25,32 +22,10 @@ type StreamVariant struct {
|
||||
ID int
|
||||
Resolution string
|
||||
OutputDir string
|
||||
Writer *ManifestWriter
|
||||
}
|
||||
|
||||
// ResolutionHeight converts a label produced by ExtractResolution back into
|
||||
// the pixel height it stands for, so variants can be ranked by quality.
|
||||
//
|
||||
// The height is parsed rather than looked up in a table: labels come from the
|
||||
// playlist's own RESOLUTION attribute, so a stream can legitimately offer
|
||||
// 1440p or 2160p that no fixed table would anticipate. Anything not in
|
||||
// <height>p form — "unknown", from a playlist with no variants — returns 0 and
|
||||
// therefore ranks below every recognised resolution.
|
||||
//
|
||||
// Comparing the labels as strings instead is a trap worth naming: "720p" sorts
|
||||
// after "1080p", so lexicographic comparison silently prefers lower quality.
|
||||
func ResolutionHeight(resolution string) int {
|
||||
trimmed := strings.TrimSuffix(strings.ToLower(strings.TrimSpace(resolution)), "p")
|
||||
|
||||
height, err := strconv.Atoi(trimmed)
|
||||
if err != nil || height < 0 {
|
||||
return 0
|
||||
}
|
||||
return height
|
||||
}
|
||||
|
||||
// ExtractResolution maps a variant to the label used for its output directory,
|
||||
// falling back to a bandwidth heuristic when the playlist omits RESOLUTION.
|
||||
func ExtractResolution(variant *m3u8.Variant) string {
|
||||
func extractResolution(variant *m3u8.Variant) string {
|
||||
if variant.Resolution != "" {
|
||||
parts := strings.Split(variant.Resolution, "x")
|
||||
if len(parts) == 2 {
|
||||
@ -71,24 +46,19 @@ func ExtractResolution(variant *m3u8.Variant) string {
|
||||
}
|
||||
}
|
||||
|
||||
// GetAllVariants fetches the master playlist and returns one entry per stream
|
||||
// variant. Segments are recorded into the manifest by VariantDownloader, which
|
||||
// receives it directly, so no writer is threaded through here.
|
||||
func GetAllVariants(masterURL string, outputDir string) ([]*StreamVariant, error) {
|
||||
func GetAllVariants(masterURL string, outputDir string, writer *ManifestWriter) ([]*StreamVariant, error) {
|
||||
client := &http.Client{}
|
||||
req, _ := http.NewRequest("GET", masterURL, nil)
|
||||
req.Header.Set("User-Agent", constants.HTTPUserAgent)
|
||||
req.Header.Set("Referer", constants.REFERRER)
|
||||
resp, err := client.Do(req)
|
||||
if err != nil {
|
||||
logging.MustGetFactory().Download.Error("failed to fetch master playlist", logging.KeyError, err.Error())
|
||||
return nil, err
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
|
||||
playlist, listType, err := m3u8.DecodeFrom(resp.Body, true)
|
||||
if err != nil {
|
||||
logging.MustGetFactory().Download.Error("failed to decode master playlist", logging.KeyError, err.Error())
|
||||
return nil, err
|
||||
}
|
||||
|
||||
@ -102,6 +72,7 @@ func GetAllVariants(masterURL string, outputDir string) ([]*StreamVariant, error
|
||||
ID: 0,
|
||||
Resolution: "unknown",
|
||||
OutputDir: path.Join(outputDir, "unknown"),
|
||||
Writer: writer,
|
||||
}}, nil
|
||||
}
|
||||
|
||||
@ -114,7 +85,7 @@ func GetAllVariants(masterURL string, outputDir string) ([]*StreamVariant, error
|
||||
for i, v := range master.Variants {
|
||||
vURL, _ := url.Parse(v.URI)
|
||||
fullURL := base.ResolveReference(vURL).String()
|
||||
resolution := ExtractResolution(v)
|
||||
resolution := extractResolution(v)
|
||||
outputDir := path.Join(outputDir, resolution)
|
||||
variants = append(variants, &StreamVariant{
|
||||
URL: fullURL,
|
||||
@ -125,17 +96,11 @@ func GetAllVariants(masterURL string, outputDir string) ([]*StreamVariant, error
|
||||
OutputDir: outputDir,
|
||||
})
|
||||
}
|
||||
logging.MustGetFactory().Download.Info("discovered variants", "count", len(variants))
|
||||
return variants, nil
|
||||
}
|
||||
|
||||
func VariantDownloader(ctx context.Context, variant *StreamVariant, sem chan struct{}, manifest *ManifestWriter) {
|
||||
log.Printf("Starting %s variant downloader (bandwidth: %d)", variant.Resolution, variant.Bandwidth)
|
||||
dlLog := logging.MustGetFactory().Download.With(logging.KeyResolution, variant.Resolution)
|
||||
registry := health.MustGetRegistry()
|
||||
dlLog.Info("variant downloader started", logging.KeyBandwidth, variant.Bandwidth)
|
||||
registry.RecordVariantStarted(variant.Resolution, variant.Bandwidth)
|
||||
|
||||
ticker := time.NewTicker(constants.RefreshDelay)
|
||||
defer ticker.Stop()
|
||||
client := &http.Client{}
|
||||
@ -144,20 +109,16 @@ func VariantDownloader(ctx context.Context, variant *StreamVariant, sem chan str
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
registry.RecordVariantStopped(variant.Resolution)
|
||||
return
|
||||
default:
|
||||
}
|
||||
|
||||
var seq uint64
|
||||
playlist, err := LoadMediaPlaylist(variant.URL)
|
||||
seq := playlist.SeqNo
|
||||
if err != nil {
|
||||
log.Printf("%s: Error loading playlist playlist: %v", variant.Resolution, err)
|
||||
dlLog.Warn("failed to load media playlist", logging.KeyError, err.Error())
|
||||
registry.RecordPlaylistError(variant.Resolution, err)
|
||||
goto waitTick
|
||||
}
|
||||
seq = playlist.SeqNo
|
||||
|
||||
for _, seg := range playlist.Segments {
|
||||
if seg == nil {
|
||||
@ -187,11 +148,6 @@ func VariantDownloader(ctx context.Context, variant *StreamVariant, sem chan str
|
||||
|
||||
if err == nil {
|
||||
log.Printf("✓ %s downloaded segment %s", j.Variant.Resolution, name)
|
||||
dlLog.Debug("segment downloaded", logging.KeySegment, name)
|
||||
registry.RecordSegmentDownloaded(variant.Resolution)
|
||||
if manifest != nil {
|
||||
manifest.AddOrUpdateSegment(strconv.FormatUint(j.Seq, 10), variant.Resolution)
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
@ -202,27 +158,21 @@ func VariantDownloader(ctx context.Context, variant *StreamVariant, sem chan str
|
||||
|
||||
if httpClient.IsHTTPStatus(err, 403) {
|
||||
log.Printf("✗ %s failed to download segment %s (403)", j.Variant.Resolution, name)
|
||||
dlLog.Warn("segment download failed (403)", logging.KeySegment, name)
|
||||
} else {
|
||||
log.Printf("✗ %s failed to download segment %s: %v", j.Variant.Resolution, name, err)
|
||||
dlLog.Error("segment download failed", logging.KeySegment, name, logging.KeyError, err.Error())
|
||||
}
|
||||
registry.RecordSegmentFailed(variant.Resolution, err)
|
||||
}(job)
|
||||
seq++
|
||||
}
|
||||
|
||||
if playlist.Closed {
|
||||
log.Printf("%s: Playlist closed (#EXT-X-ENDLIST)", variant.Resolution)
|
||||
dlLog.Info("playlist closed")
|
||||
registry.RecordVariantStopped(variant.Resolution)
|
||||
return
|
||||
}
|
||||
|
||||
waitTick:
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
registry.RecordVariantStopped(variant.Resolution)
|
||||
return
|
||||
case <-ticker.C:
|
||||
}
|
||||
|
||||
@ -3,18 +3,10 @@ package nas
|
||||
import "time"
|
||||
|
||||
type NASConfig struct {
|
||||
// Path is the destination as configured. It may be a UNC path, an
|
||||
// smb:// URL, or a directory on this host; see ParseLocation.
|
||||
Path string
|
||||
Username string
|
||||
Password string
|
||||
Timeout time.Duration
|
||||
RetryLimit int
|
||||
VerifySize bool
|
||||
// AutoMount allows the recorder to mount an SMB share itself when the
|
||||
// host has not already mounted it. Linux only, and requires root, so it
|
||||
// is off by default in favour of a clear "mount it first" error.
|
||||
AutoMount bool
|
||||
// MountPoint overrides where AutoMount attaches the share.
|
||||
MountPoint string
|
||||
}
|
||||
|
||||
@ -1,98 +0,0 @@
|
||||
package nas
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os/exec"
|
||||
"runtime"
|
||||
)
|
||||
|
||||
// Connector attaches a configured Location to the host filesystem so that
|
||||
// ordinary file operations reach the NAS.
|
||||
//
|
||||
// Windows and Linux disagree about what "connecting to a share" means —
|
||||
// Windows authenticates a UNC path in place, Linux needs the share mounted
|
||||
// somewhere in its single filesystem tree — so each host gets its own
|
||||
// implementation and NASService stays platform-neutral. Selection happens at
|
||||
// runtime rather than through build tags so that every implementation is
|
||||
// compiled, vetted and unit-testable on every platform; only the final exec
|
||||
// call is genuinely host-specific.
|
||||
type Connector interface {
|
||||
// Name identifies the strategy in logs and check output.
|
||||
Name() string
|
||||
// Attach makes loc reachable and reports where to write.
|
||||
Attach(ctx context.Context, loc Location, cfg NASConfig) (Attachment, error)
|
||||
// Detach undoes whatever Attach set up. Implementations must undo only
|
||||
// what they created: a share this process found already mounted belongs
|
||||
// to the host, and unmounting it could break unrelated software.
|
||||
Detach(ctx context.Context, att Attachment) error
|
||||
}
|
||||
|
||||
// Attachment is the result of a successful Attach.
|
||||
type Attachment struct {
|
||||
// BasePath is the host-native directory that maps to the configured
|
||||
// destination. Every file operation is resolved against it.
|
||||
BasePath string
|
||||
// Notes explain how the host reached the destination, for -check output
|
||||
// and startup logs.
|
||||
Notes []string
|
||||
// Owned records that this process created the connection or mount, and
|
||||
// so may tear it down on shutdown.
|
||||
Owned bool
|
||||
// mountPoint is the directory this process mounted, when Owned is set by
|
||||
// the Linux connector.
|
||||
mountPoint string
|
||||
// share is the network path this process connected, when Owned is set by
|
||||
// the Windows connector.
|
||||
share string
|
||||
}
|
||||
|
||||
// commandRunner executes an external command. Implementations keep argument
|
||||
// construction in a pure function and call through this, so tests can assert
|
||||
// exactly what would run without running it.
|
||||
type commandRunner func(ctx context.Context, name string, args ...string) ([]byte, error)
|
||||
|
||||
// execRunner is the production runner: it combines stdout and stderr because
|
||||
// mount and net both report the useful diagnosis on one or the other.
|
||||
func execRunner(ctx context.Context, name string, args ...string) ([]byte, error) {
|
||||
return exec.CommandContext(ctx, name, args...).CombinedOutput()
|
||||
}
|
||||
|
||||
// connectorFor picks the strategy for a destination on a given OS. A
|
||||
// destination that is already an ordinary directory needs no network step on
|
||||
// any platform, so it short-circuits before the OS switch.
|
||||
func connectorFor(loc Location, goos string) Connector {
|
||||
if !loc.IsRemote() {
|
||||
return &localConnector{goos: goos}
|
||||
}
|
||||
|
||||
switch goos {
|
||||
case "windows":
|
||||
return &windowsConnector{run: execRunner}
|
||||
case "linux":
|
||||
return &linuxConnector{run: execRunner, mountsPath: procMountsPath}
|
||||
default:
|
||||
return &unsupportedConnector{goos: goos}
|
||||
}
|
||||
}
|
||||
|
||||
// defaultConnector selects the strategy for the running host.
|
||||
func defaultConnector(loc Location) Connector {
|
||||
return connectorFor(loc, runtime.GOOS)
|
||||
}
|
||||
|
||||
// unsupportedConnector fails clearly on platforms this project does not target,
|
||||
// rather than silently writing a share-shaped directory to local disk.
|
||||
type unsupportedConnector struct {
|
||||
goos string
|
||||
}
|
||||
|
||||
func (c *unsupportedConnector) Name() string { return "unsupported" }
|
||||
|
||||
func (c *unsupportedConnector) Attach(_ context.Context, loc Location, _ NASConfig) (Attachment, error) {
|
||||
return Attachment{}, fmt.Errorf(
|
||||
"connecting to network share %s is not supported on %s; mount the share and set NAS_OUTPUT_PATH to the mount point",
|
||||
loc.SMBSource(), c.goos)
|
||||
}
|
||||
|
||||
func (c *unsupportedConnector) Detach(context.Context, Attachment) error { return nil }
|
||||
@ -1,57 +0,0 @@
|
||||
package nas
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// localConnector handles destinations that are already part of the host
|
||||
// filesystem: a plain output directory, or — the normal arrangement on Linux —
|
||||
// an SMB/NFS share mounted by fstab, autofs or systemd before the recorder
|
||||
// starts. There is nothing to connect, but there is something to check.
|
||||
type localConnector struct {
|
||||
goos string
|
||||
}
|
||||
|
||||
func (c *localConnector) Name() string { return "local" }
|
||||
|
||||
func (c *localConnector) Attach(_ context.Context, loc Location, _ NASConfig) (Attachment, error) {
|
||||
if err := c.validate(loc.Raw); err != nil {
|
||||
return Attachment{}, err
|
||||
}
|
||||
|
||||
return Attachment{
|
||||
BasePath: filepath.Clean(loc.Raw),
|
||||
Notes: []string{"local or already-mounted path (no network authentication)"},
|
||||
}, nil
|
||||
}
|
||||
|
||||
func (c *localConnector) Detach(context.Context, Attachment) error { return nil }
|
||||
|
||||
// validate rejects a Windows-shaped path on a non-Windows host. This looks
|
||||
// pedantic and is not: backslashes and colons are legal characters in Linux
|
||||
// filenames, so a stray C:\data or \\server\share would be created verbatim as
|
||||
// a local directory. Transfers would report success, cleanup would delete the
|
||||
// local originals, and nothing would ever reach the NAS.
|
||||
func (c *localConnector) validate(path string) error {
|
||||
if c.goos == "windows" {
|
||||
return nil
|
||||
}
|
||||
|
||||
if driveLetter.MatchString(path) {
|
||||
return fmt.Errorf(
|
||||
"%q is a Windows drive path but this host is %s; set NAS_OUTPUT_PATH to a mount point such as /mnt/nas/streams",
|
||||
path, c.goos)
|
||||
}
|
||||
|
||||
if strings.Contains(path, `\`) {
|
||||
return fmt.Errorf(
|
||||
"%q contains backslashes but this host is %s, where they are ordinary filename characters rather than separators; "+
|
||||
"set NAS_OUTPUT_PATH to a mount point such as /mnt/nas/streams, or to \\\\server\\share\\folder to have the share resolved automatically",
|
||||
path, c.goos)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
@ -1,178 +0,0 @@
|
||||
package nas
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// linuxConnector reaches a share the way Linux does: through a mount point in
|
||||
// the single filesystem tree. There is no per-path authentication, so the
|
||||
// question is not "can we connect" but "where is this share mounted".
|
||||
//
|
||||
// The default and by far the safest answer is that the host already mounted it
|
||||
// (fstab, autofs, systemd, or `docker run -v`), so the connector looks the
|
||||
// share up in the mount table. Mounting it here is opt-in because it requires
|
||||
// root and changes system state that outlives a failed run.
|
||||
type linuxConnector struct {
|
||||
run commandRunner
|
||||
mountsPath string
|
||||
// geteuid is indirected for tests; production uses os.Geteuid.
|
||||
geteuid func() int
|
||||
}
|
||||
|
||||
func (c *linuxConnector) Name() string { return "mount" }
|
||||
|
||||
func (c *linuxConnector) Attach(ctx context.Context, loc Location, cfg NASConfig) (Attachment, error) {
|
||||
entries, err := readMountTable(c.mountsPath)
|
||||
if err != nil {
|
||||
return Attachment{}, err
|
||||
}
|
||||
|
||||
if point, rest, ok := findMount(entries, loc); ok {
|
||||
return Attachment{
|
||||
BasePath: filepath.Join(append([]string{point}, rest...)...),
|
||||
Notes: []string{
|
||||
"share: " + loc.SMBSource(),
|
||||
fmt.Sprintf("resolved via existing mount at %s", point),
|
||||
},
|
||||
}, nil
|
||||
}
|
||||
|
||||
if !cfg.AutoMount {
|
||||
return Attachment{}, fmt.Errorf("%s", notMountedMessage(loc, cfg))
|
||||
}
|
||||
|
||||
return c.mount(ctx, loc, cfg)
|
||||
}
|
||||
|
||||
func (c *linuxConnector) Detach(ctx context.Context, att Attachment) error {
|
||||
// Never unmount a share the host had already mounted; other software may
|
||||
// be using it, and we did not put it there.
|
||||
if !att.Owned || att.mountPoint == "" {
|
||||
return nil
|
||||
}
|
||||
|
||||
if out, err := c.run(ctx, "umount", att.mountPoint); err != nil {
|
||||
return fmt.Errorf("failed to unmount %s: %v", att.mountPoint, summarize(err, out))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// mount performs an opt-in mount.cifs of the share.
|
||||
func (c *linuxConnector) mount(ctx context.Context, loc Location, cfg NASConfig) (Attachment, error) {
|
||||
geteuid := c.geteuid
|
||||
if geteuid == nil {
|
||||
geteuid = os.Geteuid
|
||||
}
|
||||
if geteuid() != 0 {
|
||||
return Attachment{}, fmt.Errorf(
|
||||
"NAS_AUTO_MOUNT is set but mounting %s requires root; run as root, or mount the share beforehand and %s",
|
||||
loc.SMBSource(), suggestMountPointConfig(loc))
|
||||
}
|
||||
|
||||
point := autoMountPoint(loc, cfg)
|
||||
if err := os.MkdirAll(point, 0755); err != nil {
|
||||
return Attachment{}, fmt.Errorf("failed to create mount point %s: %w", point, err)
|
||||
}
|
||||
|
||||
credentialsPath, cleanup, err := writeCredentialsFile(cfg)
|
||||
if err != nil {
|
||||
return Attachment{}, err
|
||||
}
|
||||
defer cleanup()
|
||||
|
||||
if cfg.Timeout > 0 {
|
||||
var cancel context.CancelFunc
|
||||
ctx, cancel = context.WithTimeout(ctx, cfg.Timeout)
|
||||
defer cancel()
|
||||
}
|
||||
|
||||
if out, err := c.run(ctx, "mount", mountArgs(loc, point, credentialsPath)...); err != nil {
|
||||
return Attachment{}, fmt.Errorf("failed to mount %s at %s: %v", loc.SMBSource(), point, summarize(err, out))
|
||||
}
|
||||
|
||||
return Attachment{
|
||||
BasePath: filepath.Join(append([]string{point}, loc.Sub...)...),
|
||||
Owned: true,
|
||||
mountPoint: point,
|
||||
Notes: []string{
|
||||
"share: " + loc.SMBSource(),
|
||||
"user: " + usernameNote(cfg.Username),
|
||||
fmt.Sprintf("mounted at %s by this process (will unmount on shutdown)", point),
|
||||
},
|
||||
}, nil
|
||||
}
|
||||
|
||||
// mountArgs builds the mount.cifs invocation. Kept pure so it can be asserted
|
||||
// without mounting anything.
|
||||
func mountArgs(loc Location, mountPoint, credentialsPath string) []string {
|
||||
options := "rw"
|
||||
if credentialsPath != "" {
|
||||
options += ",credentials=" + credentialsPath
|
||||
} else {
|
||||
options += ",guest"
|
||||
}
|
||||
return []string{"-t", "cifs", loc.SMBSource(), mountPoint, "-o", options}
|
||||
}
|
||||
|
||||
// writeCredentialsFile stages the NAS credentials in a 0600 file. mount.cifs
|
||||
// also accepts user=/pass= options, but those land in the process table where
|
||||
// any local user can read the password; a credentials file does not.
|
||||
func writeCredentialsFile(cfg NASConfig) (path string, cleanup func(), err error) {
|
||||
if cfg.Username == "" || cfg.Password == "" {
|
||||
return "", func() {}, nil
|
||||
}
|
||||
|
||||
dir, err := os.MkdirTemp("", "streamrecorder-cifs-")
|
||||
if err != nil {
|
||||
return "", func() {}, fmt.Errorf("failed to stage NAS credentials: %w", err)
|
||||
}
|
||||
cleanup = func() { os.RemoveAll(dir) }
|
||||
|
||||
path = filepath.Join(dir, "credentials")
|
||||
content := fmt.Sprintf("username=%s\npassword=%s\n", cfg.Username, cfg.Password)
|
||||
if err := os.WriteFile(path, []byte(content), 0600); err != nil {
|
||||
cleanup()
|
||||
return "", func() {}, fmt.Errorf("failed to write NAS credentials file: %w", err)
|
||||
}
|
||||
|
||||
return path, cleanup, nil
|
||||
}
|
||||
|
||||
// autoMountPoint chooses where to mount the share when the operator has not
|
||||
// named a location. Everything lands under one predictable directory so a
|
||||
// leftover mount is easy to find and clean up.
|
||||
func autoMountPoint(loc Location, cfg NASConfig) string {
|
||||
if cfg.MountPoint != "" {
|
||||
return cfg.MountPoint
|
||||
}
|
||||
return filepath.Join("/mnt", "streamrecorder", strings.ToLower(loc.Server), strings.ToLower(loc.Share))
|
||||
}
|
||||
|
||||
// notMountedMessage explains the failure in terms of what the operator should
|
||||
// actually type. A bare "no such mount" would leave them guessing.
|
||||
func notMountedMessage(loc Location, cfg NASConfig) string {
|
||||
source := loc.SMBSource()
|
||||
example := filepath.Join("/mnt", strings.ToLower(loc.Share))
|
||||
|
||||
credentials := ""
|
||||
if cfg.Username != "" {
|
||||
credentials = fmt.Sprintf(",username=%s,password=***", cfg.Username)
|
||||
}
|
||||
|
||||
return fmt.Sprintf(
|
||||
"%s is not mounted on this host. Either mount it and %s, for example:\n"+
|
||||
" sudo mkdir -p %s && sudo mount -t cifs %s %s -o rw%s\n"+
|
||||
"or set NAS_AUTO_MOUNT=true to have the recorder mount it (requires root)",
|
||||
source, suggestMountPointConfig(loc), example, source, example, credentials)
|
||||
}
|
||||
|
||||
func suggestMountPointConfig(loc Location) string {
|
||||
if len(loc.Sub) == 0 {
|
||||
return "set NAS_OUTPUT_PATH to the mount point"
|
||||
}
|
||||
return fmt.Sprintf("set NAS_OUTPUT_PATH to the mount point plus %s", loc.SubPath())
|
||||
}
|
||||
@ -1,107 +0,0 @@
|
||||
package nas
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// windowsConnector reaches a share the way Windows does: authenticate the UNC
|
||||
// path with `net use`, then use the UNC path directly as a filesystem path.
|
||||
// Nothing is mounted into a tree, so the resolved base path is the configured
|
||||
// UNC path itself.
|
||||
type windowsConnector struct {
|
||||
run commandRunner
|
||||
}
|
||||
|
||||
func (c *windowsConnector) Name() string { return "net use" }
|
||||
|
||||
func (c *windowsConnector) Attach(ctx context.Context, loc Location, cfg NASConfig) (Attachment, error) {
|
||||
share := loc.ShareUNC()
|
||||
att := Attachment{
|
||||
BasePath: loc.UNC(),
|
||||
share: share,
|
||||
Notes: []string{"share: " + share, "user: " + usernameNote(cfg.Username)},
|
||||
}
|
||||
|
||||
if cfg.Timeout > 0 {
|
||||
var cancel context.CancelFunc
|
||||
ctx, cancel = context.WithTimeout(ctx, cfg.Timeout)
|
||||
defer cancel()
|
||||
}
|
||||
|
||||
out, err := c.run(ctx, "net", netUseArgs(loc, cfg)...)
|
||||
if err != nil {
|
||||
// A failure here is not decisive. `net use` also fails when the share
|
||||
// is already connected — commonly because Explorer or another process
|
||||
// got there first — in which case the path is perfectly usable. Record
|
||||
// the reason and let the caller's write probe deliver the verdict,
|
||||
// rather than refusing to run against a working share.
|
||||
att.Notes = append(att.Notes,
|
||||
fmt.Sprintf("net use did not complete (%v); continuing, the share may already be connected", summarize(err, out)))
|
||||
return att, nil
|
||||
}
|
||||
|
||||
att.Owned = true
|
||||
att.Notes = append(att.Notes, "connected via net use")
|
||||
return att, nil
|
||||
}
|
||||
|
||||
func (c *windowsConnector) Detach(ctx context.Context, att Attachment) error {
|
||||
// Only drop connections this process made. Deleting a mapping the user or
|
||||
// another application established would break them.
|
||||
if !att.Owned || att.share == "" {
|
||||
return nil
|
||||
}
|
||||
|
||||
if out, err := c.run(ctx, "net", "use", att.share, "/delete"); err != nil {
|
||||
return fmt.Errorf("failed to disconnect %s: %v", att.share, summarize(err, out))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// netUseArgs builds the `net use` invocation. Kept pure so the Windows code
|
||||
// path can be asserted from any host, including the Linux box this is
|
||||
// developed on, without executing anything.
|
||||
func netUseArgs(loc Location, cfg NASConfig) []string {
|
||||
args := []string{"use", loc.ShareUNC()}
|
||||
if cfg.Username != "" && cfg.Password != "" {
|
||||
args = append(args, "/user:"+cfg.Username, cfg.Password)
|
||||
}
|
||||
return append(args, "/persistent:no")
|
||||
}
|
||||
|
||||
// summarize pairs an exec error with the first line of command output, which is
|
||||
// where net and mount put the actual diagnosis.
|
||||
func summarize(err error, out []byte) string {
|
||||
text := strings.TrimSpace(string(out))
|
||||
if text == "" {
|
||||
return err.Error()
|
||||
}
|
||||
if idx := strings.IndexByte(text, '\n'); idx >= 0 {
|
||||
text = text[:idx]
|
||||
}
|
||||
return fmt.Sprintf("%v: %s", err, text)
|
||||
}
|
||||
|
||||
func usernameNote(username string) string {
|
||||
if username == "" {
|
||||
return "(none, using ambient credentials)"
|
||||
}
|
||||
return username
|
||||
}
|
||||
|
||||
// joinBase resolves a relative path against a base that may be a UNC path.
|
||||
// filepath.Join is correct on Windows and would mangle the leading \\ anywhere
|
||||
// else, so UNC bases are joined with backslashes explicitly.
|
||||
func joinBase(base string, elem ...string) string {
|
||||
if len(elem) == 0 {
|
||||
return base
|
||||
}
|
||||
if strings.HasPrefix(base, `\\`) {
|
||||
parts := append([]string{strings.TrimRight(base, `\`)}, elem...)
|
||||
return strings.Join(parts, `\`)
|
||||
}
|
||||
return filepath.Join(append([]string{base}, elem...)...)
|
||||
}
|
||||
@ -1,353 +0,0 @@
|
||||
package nas
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// recordingRunner captures the commands a connector would run instead of
|
||||
// running them. This is what lets the Windows path be tested from a Linux
|
||||
// development machine: the decision logic and the exact argv are asserted
|
||||
// without ever invoking net.exe.
|
||||
type recordingRunner struct {
|
||||
calls [][]string
|
||||
output []byte
|
||||
err error
|
||||
}
|
||||
|
||||
func (r *recordingRunner) run(_ context.Context, name string, args ...string) ([]byte, error) {
|
||||
r.calls = append(r.calls, append([]string{name}, args...))
|
||||
return r.output, r.err
|
||||
}
|
||||
|
||||
func (r *recordingRunner) lastCall() string {
|
||||
if len(r.calls) == 0 {
|
||||
return ""
|
||||
}
|
||||
return strings.Join(r.calls[len(r.calls)-1], " ")
|
||||
}
|
||||
|
||||
func remoteLocation(t *testing.T, raw string) Location {
|
||||
t.Helper()
|
||||
loc, err := ParseLocation(raw)
|
||||
if err != nil {
|
||||
t.Fatalf("ParseLocation(%q) failed: %v", raw, err)
|
||||
}
|
||||
return loc
|
||||
}
|
||||
|
||||
func TestConnectorSelection(t *testing.T) {
|
||||
remote := remoteLocation(t, `\\nas\dci\streams`)
|
||||
local := remoteLocation(t, "/mnt/dci/streams")
|
||||
|
||||
cases := []struct {
|
||||
name string
|
||||
loc Location
|
||||
goos string
|
||||
want string
|
||||
}{
|
||||
{"windows share", remote, "windows", "net use"},
|
||||
{"linux share", remote, "linux", "mount"},
|
||||
{"darwin share", remote, "darwin", "unsupported"},
|
||||
{"local path on windows", local, "windows", "local"},
|
||||
{"local path on linux", local, "linux", "local"},
|
||||
// A already-mounted path needs no network step even where shares are
|
||||
// unsupported, so it must not fall through to the error connector.
|
||||
{"local path on darwin", local, "darwin", "local"},
|
||||
}
|
||||
|
||||
for _, tc := range cases {
|
||||
if got := connectorFor(tc.loc, tc.goos).Name(); got != tc.want {
|
||||
t.Errorf("%s: connectorFor = %q, want %q", tc.name, got, tc.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestWindowsConnectorAttach(t *testing.T) {
|
||||
runner := &recordingRunner{}
|
||||
c := &windowsConnector{run: runner.run}
|
||||
loc := remoteLocation(t, `\\HomeLabNAS\dci\streams`)
|
||||
|
||||
att, err := c.Attach(context.Background(), loc, NASConfig{Username: "NASAdmin", Password: "secret"})
|
||||
if err != nil {
|
||||
t.Fatalf("Attach failed: %v", err)
|
||||
}
|
||||
|
||||
// The base path is the full UNC path; net use authenticates only the share.
|
||||
if want := `\\HomeLabNAS\dci\streams`; att.BasePath != want {
|
||||
t.Errorf("BasePath = %q, want %q", att.BasePath, want)
|
||||
}
|
||||
if want := `net use \\HomeLabNAS\dci /user:NASAdmin secret /persistent:no`; runner.lastCall() != want {
|
||||
t.Errorf("command = %q, want %q", runner.lastCall(), want)
|
||||
}
|
||||
if !att.Owned {
|
||||
t.Error("a successful net use should be owned so it can be released on shutdown")
|
||||
}
|
||||
}
|
||||
|
||||
func TestWindowsConnectorOmitsCredentialsWhenUnset(t *testing.T) {
|
||||
runner := &recordingRunner{}
|
||||
c := &windowsConnector{run: runner.run}
|
||||
|
||||
if _, err := c.Attach(context.Background(), remoteLocation(t, `\\nas\dci`), NASConfig{}); err != nil {
|
||||
t.Fatalf("Attach failed: %v", err)
|
||||
}
|
||||
|
||||
if want := `net use \\nas\dci /persistent:no`; runner.lastCall() != want {
|
||||
t.Errorf("command = %q, want %q", runner.lastCall(), want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestWindowsConnectorContinuesWhenAlreadyConnected(t *testing.T) {
|
||||
// net use fails when a connection already exists, but the share is usable;
|
||||
// refusing to run would be wrong. The write probe is the real verdict.
|
||||
runner := &recordingRunner{
|
||||
err: fmt.Errorf("exit status 2"),
|
||||
output: []byte("The network connection could not be found.\nMore help is available..."),
|
||||
}
|
||||
c := &windowsConnector{run: runner.run}
|
||||
|
||||
att, err := c.Attach(context.Background(), remoteLocation(t, `\\nas\dci`), NASConfig{})
|
||||
if err != nil {
|
||||
t.Fatalf("Attach should tolerate a net use failure, got: %v", err)
|
||||
}
|
||||
if att.Owned {
|
||||
t.Error("a failed net use must not be marked owned, or Detach would drop someone else's connection")
|
||||
}
|
||||
|
||||
notes := strings.Join(att.Notes, "\n")
|
||||
if !strings.Contains(notes, "net use did not complete") {
|
||||
t.Errorf("expected the failure to be recorded in notes:\n%s", notes)
|
||||
}
|
||||
// Only the first line of output is worth surfacing.
|
||||
if strings.Contains(notes, "More help is available") {
|
||||
t.Errorf("notes should not carry the full usage dump:\n%s", notes)
|
||||
}
|
||||
}
|
||||
|
||||
func TestWindowsConnectorDetachOnlyReleasesOwned(t *testing.T) {
|
||||
runner := &recordingRunner{}
|
||||
c := &windowsConnector{run: runner.run}
|
||||
|
||||
if err := c.Detach(context.Background(), Attachment{share: `\\nas\dci`, Owned: false}); err != nil {
|
||||
t.Fatalf("Detach failed: %v", err)
|
||||
}
|
||||
if len(runner.calls) != 0 {
|
||||
t.Errorf("must not disconnect a share this process did not connect, ran: %v", runner.calls)
|
||||
}
|
||||
|
||||
if err := c.Detach(context.Background(), Attachment{share: `\\nas\dci`, Owned: true}); err != nil {
|
||||
t.Fatalf("Detach failed: %v", err)
|
||||
}
|
||||
if want := `net use \\nas\dci /delete`; runner.lastCall() != want {
|
||||
t.Errorf("command = %q, want %q", runner.lastCall(), want)
|
||||
}
|
||||
}
|
||||
|
||||
// writeMountTable stages a /proc/mounts-format file.
|
||||
func writeMountTable(t *testing.T, lines ...string) string {
|
||||
t.Helper()
|
||||
path := filepath.Join(t.TempDir(), "mounts")
|
||||
body := strings.Join(lines, "\n") + "\n"
|
||||
if err := os.WriteFile(path, []byte(body), 0644); err != nil {
|
||||
t.Fatalf("failed to write mount table: %v", err)
|
||||
}
|
||||
return path
|
||||
}
|
||||
|
||||
func TestLinuxConnectorResolvesExistingMount(t *testing.T) {
|
||||
mounts := writeMountTable(t,
|
||||
"sysfs /sys sysfs rw,nosuid 0 0",
|
||||
"//homelabnas/dci /mnt/dci cifs rw,relatime,username=nasadmin 0 0",
|
||||
)
|
||||
c := &linuxConnector{run: (&recordingRunner{}).run, mountsPath: mounts}
|
||||
|
||||
att, err := c.Attach(context.Background(), remoteLocation(t, `\\HomeLabNAS\dci\streams`), NASConfig{})
|
||||
if err != nil {
|
||||
t.Fatalf("Attach failed: %v", err)
|
||||
}
|
||||
|
||||
// Server and share names are case-insensitive on SMB, so the configured
|
||||
// \\HomeLabNAS\dci must match a //homelabnas/dci mount.
|
||||
if want := "/mnt/dci/streams"; att.BasePath != want {
|
||||
t.Errorf("BasePath = %q, want %q", att.BasePath, want)
|
||||
}
|
||||
if att.Owned {
|
||||
t.Error("a pre-existing mount must not be owned, or shutdown would unmount it")
|
||||
}
|
||||
}
|
||||
|
||||
func TestLinuxConnectorPrefersDeepestMount(t *testing.T) {
|
||||
// Both mounts could serve the destination; the more specific one is the
|
||||
// only one guaranteed to actually contain it.
|
||||
mounts := writeMountTable(t,
|
||||
"//nas/dci /mnt/dci cifs rw 0 0",
|
||||
"//nas/dci/streams /mnt/streams cifs rw 0 0",
|
||||
)
|
||||
c := &linuxConnector{run: (&recordingRunner{}).run, mountsPath: mounts}
|
||||
|
||||
att, err := c.Attach(context.Background(), remoteLocation(t, `\\nas\dci\streams\2026`), NASConfig{})
|
||||
if err != nil {
|
||||
t.Fatalf("Attach failed: %v", err)
|
||||
}
|
||||
if want := "/mnt/streams/2026"; att.BasePath != want {
|
||||
t.Errorf("BasePath = %q, want %q", att.BasePath, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLinuxConnectorMatchesNFSAndEscapedPoints(t *testing.T) {
|
||||
mounts := writeMountTable(t,
|
||||
`nas:/dci /mnt/my\040share nfs4 rw 0 0`,
|
||||
)
|
||||
c := &linuxConnector{run: (&recordingRunner{}).run, mountsPath: mounts}
|
||||
|
||||
att, err := c.Attach(context.Background(), remoteLocation(t, `\\nas\dci\streams`), NASConfig{})
|
||||
if err != nil {
|
||||
t.Fatalf("Attach failed: %v", err)
|
||||
}
|
||||
if want := "/mnt/my share/streams"; att.BasePath != want {
|
||||
t.Errorf("BasePath = %q, want %q", att.BasePath, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLinuxConnectorIgnoresNonNetworkMounts(t *testing.T) {
|
||||
// A local directory that merely looks like the share must not match.
|
||||
mounts := writeMountTable(t, "//nas/dci /mnt/dci ext4 rw 0 0")
|
||||
c := &linuxConnector{run: (&recordingRunner{}).run, mountsPath: mounts}
|
||||
|
||||
if _, err := c.Attach(context.Background(), remoteLocation(t, `\\nas\dci`), NASConfig{}); err == nil {
|
||||
t.Fatal("expected an ext4 mount not to satisfy a network share")
|
||||
}
|
||||
}
|
||||
|
||||
func TestLinuxConnectorUnmountedGivesActionableError(t *testing.T) {
|
||||
mounts := writeMountTable(t, "sysfs /sys sysfs rw 0 0")
|
||||
c := &linuxConnector{run: (&recordingRunner{}).run, mountsPath: mounts}
|
||||
|
||||
_, err := c.Attach(context.Background(), remoteLocation(t, `\\nas\dci\streams`), NASConfig{Username: "admin"})
|
||||
if err == nil {
|
||||
t.Fatal("expected an unmounted share to fail")
|
||||
}
|
||||
|
||||
msg := err.Error()
|
||||
for _, want := range []string{"//nas/dci", "mount -t cifs", "NAS_AUTO_MOUNT", "NAS_OUTPUT_PATH"} {
|
||||
if !strings.Contains(msg, want) {
|
||||
t.Errorf("error should mention %q:\n%s", want, msg)
|
||||
}
|
||||
}
|
||||
// The remedy must not echo the real password.
|
||||
if strings.Contains(msg, "hunter2") {
|
||||
t.Error("error message leaked a credential")
|
||||
}
|
||||
}
|
||||
|
||||
func TestLinuxConnectorAutoMountRequiresRoot(t *testing.T) {
|
||||
mounts := writeMountTable(t, "sysfs /sys sysfs rw 0 0")
|
||||
runner := &recordingRunner{}
|
||||
c := &linuxConnector{run: runner.run, mountsPath: mounts, geteuid: func() int { return 1000 }}
|
||||
|
||||
_, err := c.Attach(context.Background(), remoteLocation(t, `\\nas\dci`), NASConfig{AutoMount: true})
|
||||
if err == nil || !strings.Contains(err.Error(), "requires root") {
|
||||
t.Fatalf("expected a root requirement error, got: %v", err)
|
||||
}
|
||||
if len(runner.calls) != 0 {
|
||||
t.Errorf("must not attempt mount without privileges, ran: %v", runner.calls)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLinuxConnectorAutoMount(t *testing.T) {
|
||||
mounts := writeMountTable(t, "sysfs /sys sysfs rw 0 0")
|
||||
runner := &recordingRunner{}
|
||||
point := filepath.Join(t.TempDir(), "mnt", "dci")
|
||||
c := &linuxConnector{run: runner.run, mountsPath: mounts, geteuid: func() int { return 0 }}
|
||||
|
||||
cfg := NASConfig{AutoMount: true, MountPoint: point, Username: "admin", Password: "hunter2", Timeout: 5 * time.Second}
|
||||
att, err := c.Attach(context.Background(), remoteLocation(t, `\\nas\dci\streams`), cfg)
|
||||
if err != nil {
|
||||
t.Fatalf("Attach failed: %v", err)
|
||||
}
|
||||
|
||||
if want := filepath.Join(point, "streams"); att.BasePath != want {
|
||||
t.Errorf("BasePath = %q, want %q", att.BasePath, want)
|
||||
}
|
||||
if !att.Owned || att.mountPoint != point {
|
||||
t.Errorf("a self-created mount must be owned and recorded, got owned=%v point=%q", att.Owned, att.mountPoint)
|
||||
}
|
||||
|
||||
call := runner.lastCall()
|
||||
if !strings.HasPrefix(call, "mount -t cifs //nas/dci "+point+" -o rw,credentials=") {
|
||||
t.Errorf("unexpected mount command: %q", call)
|
||||
}
|
||||
// The password must reach mount.cifs through a credentials file, never
|
||||
// through argv where any local user could read it from the process table.
|
||||
if strings.Contains(call, "hunter2") {
|
||||
t.Errorf("password leaked into the mount command line: %q", call)
|
||||
}
|
||||
// The staged credentials file must not outlive the mount call.
|
||||
credPath := call[strings.Index(call, "credentials=")+len("credentials="):]
|
||||
if _, err := os.Stat(credPath); !os.IsNotExist(err) {
|
||||
t.Errorf("credentials file %s should have been removed, stat err = %v", credPath, err)
|
||||
}
|
||||
|
||||
if err := c.Detach(context.Background(), att); err != nil {
|
||||
t.Fatalf("Detach failed: %v", err)
|
||||
}
|
||||
if want := "umount " + point; runner.lastCall() != want {
|
||||
t.Errorf("detach command = %q, want %q", runner.lastCall(), want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLinuxConnectorDetachLeavesForeignMounts(t *testing.T) {
|
||||
runner := &recordingRunner{}
|
||||
c := &linuxConnector{run: runner.run, mountsPath: writeMountTable(t)}
|
||||
|
||||
if err := c.Detach(context.Background(), Attachment{BasePath: "/mnt/dci", Owned: false}); err != nil {
|
||||
t.Fatalf("Detach failed: %v", err)
|
||||
}
|
||||
if len(runner.calls) != 0 {
|
||||
t.Errorf("must not unmount a share the host mounted, ran: %v", runner.calls)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLocalConnectorRejectsWindowsPathsOffWindows(t *testing.T) {
|
||||
c := &localConnector{goos: "linux"}
|
||||
|
||||
for _, raw := range []string{`C:\data\streams`, `data\streams`} {
|
||||
loc, err := ParseLocation(raw)
|
||||
if err != nil {
|
||||
t.Fatalf("ParseLocation(%q) failed: %v", raw, err)
|
||||
}
|
||||
if _, err := c.Attach(context.Background(), loc, NASConfig{}); err == nil {
|
||||
t.Errorf("%q should be rejected on Linux: backslashes are legal filename characters there, "+
|
||||
"so it would silently create a local directory instead of writing to the NAS", raw)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestLocalConnectorAcceptsWindowsPathsOnWindows(t *testing.T) {
|
||||
c := &localConnector{goos: "windows"}
|
||||
loc, err := ParseLocation(`C:\data\streams`)
|
||||
if err != nil {
|
||||
t.Fatalf("ParseLocation failed: %v", err)
|
||||
}
|
||||
|
||||
if _, err := c.Attach(context.Background(), loc, NASConfig{}); err != nil {
|
||||
t.Errorf("a drive path is valid on Windows: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestUnsupportedConnectorExplainsItself(t *testing.T) {
|
||||
c := &unsupportedConnector{goos: "plan9"}
|
||||
_, err := c.Attach(context.Background(), remoteLocation(t, `\\nas\dci`), NASConfig{})
|
||||
if err == nil {
|
||||
t.Fatal("expected an error on an unsupported platform")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "plan9") || !strings.Contains(err.Error(), "NAS_OUTPUT_PATH") {
|
||||
t.Errorf("error should name the platform and the remedy: %v", err)
|
||||
}
|
||||
}
|
||||
@ -1,144 +0,0 @@
|
||||
package nas
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Location is a configured NAS destination parsed into a host-neutral form.
|
||||
//
|
||||
// The same destination is spelled differently depending on where you're
|
||||
// standing: Windows wants \\HomeLabNAS\dci\streams, a Linux host reaches the
|
||||
// same bytes through a mount point like /mnt/dci/streams, and either host may
|
||||
// simply be handed a plain local directory. Parsing happens once, up front,
|
||||
// and platform-specific rendering is left to the Connector implementations.
|
||||
type Location struct {
|
||||
// Raw is the destination exactly as configured, for error messages.
|
||||
Raw string
|
||||
// Server is the host serving the share; empty for a plain local path.
|
||||
Server string
|
||||
// Share is the top-level export name; empty for a plain local path.
|
||||
Share string
|
||||
// Sub holds the path components below the share, if any.
|
||||
Sub []string
|
||||
}
|
||||
|
||||
// driveLetter matches a Windows drive-qualified path such as C:\data.
|
||||
var driveLetter = regexp.MustCompile(`^[A-Za-z]:[\\/]`)
|
||||
|
||||
// ParseLocation accepts every spelling of a destination this project supports:
|
||||
// Windows UNC (\\server\share\dir), the SMB source form (//server/share/dir),
|
||||
// a smb:// URL, or a plain local/mounted directory. It does no I/O and makes no
|
||||
// assumptions about the host OS, so the result is safe to compute anywhere.
|
||||
func ParseLocation(raw string) (Location, error) {
|
||||
trimmed := strings.TrimSpace(raw)
|
||||
if trimmed == "" {
|
||||
return Location{}, fmt.Errorf("NAS path is empty")
|
||||
}
|
||||
|
||||
loc := Location{Raw: trimmed}
|
||||
|
||||
remainder, isRemote := stripRemotePrefix(trimmed)
|
||||
if !isRemote {
|
||||
return loc, nil
|
||||
}
|
||||
|
||||
parts := splitPathComponents(remainder)
|
||||
if len(parts) < 2 {
|
||||
return Location{}, fmt.Errorf(
|
||||
"%q names a server but no share; expected a form like \\\\server\\share\\folder", trimmed)
|
||||
}
|
||||
|
||||
loc.Server = parts[0]
|
||||
loc.Share = parts[1]
|
||||
loc.Sub = parts[2:]
|
||||
return loc, nil
|
||||
}
|
||||
|
||||
// stripRemotePrefix reports whether raw names a network share and returns the
|
||||
// server/share/subpath portion when it does. A leading // is treated as a share
|
||||
// rather than a local path: POSIX leaves paths beginning with exactly two
|
||||
// slashes implementation-defined, and nobody configures a NAS destination that
|
||||
// way by accident.
|
||||
func stripRemotePrefix(raw string) (string, bool) {
|
||||
lower := strings.ToLower(raw)
|
||||
switch {
|
||||
case strings.HasPrefix(lower, "smb://"):
|
||||
return raw[len("smb://"):], true
|
||||
case strings.HasPrefix(lower, "cifs://"):
|
||||
return raw[len("cifs://"):], true
|
||||
case strings.HasPrefix(raw, `\\`):
|
||||
return raw[2:], true
|
||||
case strings.HasPrefix(raw, "//"):
|
||||
return raw[2:], true
|
||||
default:
|
||||
return raw, false
|
||||
}
|
||||
}
|
||||
|
||||
// splitPathComponents splits on either separator and drops empty components, so
|
||||
// that \\server\share\\folder\ and //server/share/folder agree.
|
||||
func splitPathComponents(path string) []string {
|
||||
fields := strings.FieldsFunc(path, func(r rune) bool { return r == '/' || r == '\\' })
|
||||
out := make([]string, 0, len(fields))
|
||||
for _, f := range fields {
|
||||
if f != "" {
|
||||
out = append(out, f)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// IsRemote reports whether reaching this destination requires attaching a
|
||||
// network share first, as opposed to writing to a directory that is already
|
||||
// part of the local filesystem.
|
||||
func (l Location) IsRemote() bool {
|
||||
return l.Server != "" && l.Share != ""
|
||||
}
|
||||
|
||||
// UNC renders the destination in Windows form: \\server\share\sub.
|
||||
func (l Location) UNC() string {
|
||||
if !l.IsRemote() {
|
||||
return l.Raw
|
||||
}
|
||||
return `\\` + strings.Join(append([]string{l.Server, l.Share}, l.Sub...), `\`)
|
||||
}
|
||||
|
||||
// ShareUNC renders just the share in Windows form: \\server\share. This is the
|
||||
// unit `net use` authenticates against; the subpath below it is an ordinary
|
||||
// directory on the connected share.
|
||||
func (l Location) ShareUNC() string {
|
||||
if !l.IsRemote() {
|
||||
return ""
|
||||
}
|
||||
return `\\` + l.Server + `\` + l.Share
|
||||
}
|
||||
|
||||
// SMBSource renders the share in the //server/share form used by mount tables
|
||||
// and by mount.cifs. It deliberately omits Sub: that part is a path *within*
|
||||
// the mounted share, not part of the share's identity.
|
||||
func (l Location) SMBSource() string {
|
||||
if !l.IsRemote() {
|
||||
return ""
|
||||
}
|
||||
return "//" + l.Server + "/" + l.Share
|
||||
}
|
||||
|
||||
// SubPath renders the components below the share using the host's separator, or
|
||||
// "" when the destination is the share root.
|
||||
func (l Location) SubPath() string {
|
||||
if len(l.Sub) == 0 {
|
||||
return ""
|
||||
}
|
||||
return filepath.Join(l.Sub...)
|
||||
}
|
||||
|
||||
// String describes the destination for logs and check output.
|
||||
func (l Location) String() string {
|
||||
if l.IsRemote() {
|
||||
return l.UNC()
|
||||
}
|
||||
return l.Raw
|
||||
}
|
||||
@ -1,123 +0,0 @@
|
||||
package nas
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestParseLocationRemoteForms(t *testing.T) {
|
||||
// Every spelling of the same destination must parse identically, so that
|
||||
// an operator can configure it the way their platform's documentation
|
||||
// describes without changing behaviour.
|
||||
equivalent := []string{
|
||||
`\\HomeLabNAS\dci\streams`,
|
||||
`//HomeLabNAS/dci/streams`,
|
||||
`smb://HomeLabNAS/dci/streams`,
|
||||
`SMB://HomeLabNAS/dci/streams`,
|
||||
`cifs://HomeLabNAS/dci/streams`,
|
||||
`\\HomeLabNAS\dci\streams\`,
|
||||
`\\HomeLabNAS\\dci\\streams`,
|
||||
}
|
||||
|
||||
for _, raw := range equivalent {
|
||||
loc, err := ParseLocation(raw)
|
||||
if err != nil {
|
||||
t.Errorf("ParseLocation(%q) failed: %v", raw, err)
|
||||
continue
|
||||
}
|
||||
if !loc.IsRemote() {
|
||||
t.Errorf("ParseLocation(%q) should be remote", raw)
|
||||
}
|
||||
if loc.Server != "HomeLabNAS" || loc.Share != "dci" {
|
||||
t.Errorf("ParseLocation(%q) = server %q share %q, want HomeLabNAS/dci", raw, loc.Server, loc.Share)
|
||||
}
|
||||
if len(loc.Sub) != 1 || loc.Sub[0] != "streams" {
|
||||
t.Errorf("ParseLocation(%q) sub = %v, want [streams]", raw, loc.Sub)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseLocationRendering(t *testing.T) {
|
||||
loc, err := ParseLocation(`//HomeLabNAS/dci/streams/2026`)
|
||||
if err != nil {
|
||||
t.Fatalf("unexpected error: %v", err)
|
||||
}
|
||||
|
||||
if got, want := loc.UNC(), `\\HomeLabNAS\dci\streams\2026`; got != want {
|
||||
t.Errorf("UNC() = %q, want %q", got, want)
|
||||
}
|
||||
if got, want := loc.ShareUNC(), `\\HomeLabNAS\dci`; got != want {
|
||||
t.Errorf("ShareUNC() = %q, want %q", got, want)
|
||||
}
|
||||
if got, want := loc.SMBSource(), "//HomeLabNAS/dci"; got != want {
|
||||
t.Errorf("SMBSource() = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseLocationShareRoot(t *testing.T) {
|
||||
loc, err := ParseLocation(`\\nas\media`)
|
||||
if err != nil {
|
||||
t.Fatalf("unexpected error: %v", err)
|
||||
}
|
||||
if len(loc.Sub) != 0 {
|
||||
t.Errorf("expected no subpath, got %v", loc.Sub)
|
||||
}
|
||||
if got, want := loc.UNC(), `\\nas\media`; got != want {
|
||||
t.Errorf("UNC() = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseLocationLocalPaths(t *testing.T) {
|
||||
for _, raw := range []string{"/mnt/nas/streams", "data/streams", `C:\data\streams`, "/mnt"} {
|
||||
loc, err := ParseLocation(raw)
|
||||
if err != nil {
|
||||
t.Errorf("ParseLocation(%q) failed: %v", raw, err)
|
||||
continue
|
||||
}
|
||||
if loc.IsRemote() {
|
||||
t.Errorf("ParseLocation(%q) should not be treated as a network share", raw)
|
||||
}
|
||||
if loc.Raw != raw {
|
||||
t.Errorf("ParseLocation(%q).Raw = %q", raw, loc.Raw)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseLocationRejectsIncomplete(t *testing.T) {
|
||||
for _, raw := range []string{"", " ", `\\`, `\\server`, "//server", "smb://server"} {
|
||||
if _, err := ParseLocation(raw); err == nil {
|
||||
t.Errorf("ParseLocation(%q) should have failed", raw)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseLocationTrimsWhitespace(t *testing.T) {
|
||||
loc, err := ParseLocation(" \\\\nas\\dci ")
|
||||
if err != nil {
|
||||
t.Fatalf("unexpected error: %v", err)
|
||||
}
|
||||
if loc.Server != "nas" || loc.Share != "dci" {
|
||||
t.Errorf("whitespace not trimmed: server %q share %q", loc.Server, loc.Share)
|
||||
}
|
||||
}
|
||||
|
||||
func TestJoinBaseKeepsUNCSeparators(t *testing.T) {
|
||||
// filepath.Join would produce \\nas\dci/event on Linux, which is not a
|
||||
// path Windows would accept if the value were ever written to state.
|
||||
got := joinBase(`\\nas\dci`, "event", "1080p")
|
||||
if want := `\\nas\dci\event\1080p`; got != want {
|
||||
t.Errorf("joinBase UNC = %q, want %q", got, want)
|
||||
}
|
||||
|
||||
if got, want := joinBase(`\\nas\dci\`, "event"), `\\nas\dci\event`; got != want {
|
||||
t.Errorf("joinBase trailing separator = %q, want %q", got, want)
|
||||
}
|
||||
|
||||
if got := joinBase("/mnt/dci"); got != "/mnt/dci" {
|
||||
t.Errorf("joinBase with no elements = %q", got)
|
||||
}
|
||||
|
||||
if got, want := joinBase("/mnt/dci", "event"), "/mnt/dci/event"; !strings.HasSuffix(got, want) {
|
||||
t.Errorf("joinBase local = %q, want suffix %q", got, want)
|
||||
}
|
||||
}
|
||||
@ -1,135 +0,0 @@
|
||||
package nas
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"fmt"
|
||||
"os"
|
||||
"strconv"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// procMountsPath is the kernel's view of what is mounted in this process's
|
||||
// mount namespace, which is the right question to ask inside a container too.
|
||||
const procMountsPath = "/proc/mounts"
|
||||
|
||||
// mountEntry is one line of the mount table.
|
||||
type mountEntry struct {
|
||||
Source string
|
||||
Point string
|
||||
FSType string
|
||||
}
|
||||
|
||||
// networkFilesystems are the mount types that can plausibly be serving a NAS
|
||||
// share. Restricting the search avoids matching, say, a local directory that
|
||||
// merely happens to be named like a server.
|
||||
var networkFilesystems = map[string]bool{
|
||||
"cifs": true,
|
||||
"smb3": true,
|
||||
"smbfs": true,
|
||||
"nfs": true,
|
||||
"nfs4": true,
|
||||
"fuse.smbnetfs": true,
|
||||
}
|
||||
|
||||
// readMountTable parses /proc/mounts (or any file in its format). Fields are
|
||||
// whitespace-separated: source, mount point, type, options, dump, pass.
|
||||
func readMountTable(path string) ([]mountEntry, error) {
|
||||
f, err := os.Open(path)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to read mount table %s: %w", path, err)
|
||||
}
|
||||
defer f.Close()
|
||||
|
||||
var entries []mountEntry
|
||||
scanner := bufio.NewScanner(f)
|
||||
for scanner.Scan() {
|
||||
fields := strings.Fields(scanner.Text())
|
||||
if len(fields) < 3 {
|
||||
continue
|
||||
}
|
||||
entries = append(entries, mountEntry{
|
||||
Source: unescapeMountField(fields[0]),
|
||||
Point: unescapeMountField(fields[1]),
|
||||
FSType: fields[2],
|
||||
})
|
||||
}
|
||||
if err := scanner.Err(); err != nil {
|
||||
return nil, fmt.Errorf("failed to scan mount table %s: %w", path, err)
|
||||
}
|
||||
|
||||
return entries, nil
|
||||
}
|
||||
|
||||
// unescapeMountField decodes the octal escapes the kernel uses for characters
|
||||
// that would otherwise break the whitespace-separated format — most often \040
|
||||
// for a space in a mount point.
|
||||
func unescapeMountField(field string) string {
|
||||
if !strings.Contains(field, `\`) {
|
||||
return field
|
||||
}
|
||||
|
||||
var b strings.Builder
|
||||
for i := 0; i < len(field); i++ {
|
||||
if field[i] == '\\' && i+3 < len(field) {
|
||||
if v, err := strconv.ParseUint(field[i+1:i+4], 8, 8); err == nil {
|
||||
b.WriteByte(byte(v))
|
||||
i += 3
|
||||
continue
|
||||
}
|
||||
}
|
||||
b.WriteByte(field[i])
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
|
||||
// findMount locates the mount serving loc and returns its mount point plus the
|
||||
// path components still to be appended below it.
|
||||
//
|
||||
// The search runs deepest-first because a share may be mounted at any level:
|
||||
// //nas/dci might be mounted at /mnt/dci, or //nas/dci/streams mounted directly
|
||||
// at /mnt/streams. Preferring the longest matching source keeps both working
|
||||
// and avoids pointing at a parent that the mount does not actually cover.
|
||||
func findMount(entries []mountEntry, loc Location) (point string, rest []string, ok bool) {
|
||||
if !loc.IsRemote() {
|
||||
return "", nil, false
|
||||
}
|
||||
|
||||
for depth := len(loc.Sub); depth >= 0; depth-- {
|
||||
for _, candidate := range mountSourceForms(loc, depth) {
|
||||
for _, entry := range entries {
|
||||
if !networkFilesystems[entry.FSType] {
|
||||
continue
|
||||
}
|
||||
if normalizeMountSource(entry.Source) == candidate {
|
||||
return entry.Point, loc.Sub[depth:], true
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return "", nil, false
|
||||
}
|
||||
|
||||
// mountSourceForms renders the share, plus its first `depth` subdirectories, in
|
||||
// the spellings a mount table might use for it.
|
||||
func mountSourceForms(loc Location, depth int) []string {
|
||||
segments := append([]string{loc.Share}, loc.Sub[:depth]...)
|
||||
tail := strings.Join(segments, "/")
|
||||
|
||||
return []string{
|
||||
normalizeMountSource("//" + loc.Server + "/" + tail), // cifs/smb3
|
||||
normalizeMountSource(loc.Server + ":/" + tail), // nfs
|
||||
}
|
||||
}
|
||||
|
||||
// normalizeMountSource makes sources comparable: SMB servers and shares are
|
||||
// case-insensitive, separators may be either slash, and a trailing slash is not
|
||||
// meaningful.
|
||||
func normalizeMountSource(source string) string {
|
||||
s := strings.ReplaceAll(source, `\`, "/")
|
||||
s = strings.ToLower(strings.TrimSpace(s))
|
||||
if len(s) > 1 {
|
||||
s = strings.TrimRight(s, "/")
|
||||
}
|
||||
return s
|
||||
}
|
||||
@ -1,96 +0,0 @@
|
||||
package nas
|
||||
|
||||
import (
|
||||
"os"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestUnescapeMountField(t *testing.T) {
|
||||
cases := map[string]string{
|
||||
`/mnt/plain`: "/mnt/plain",
|
||||
`/mnt/my\040share`: "/mnt/my share",
|
||||
`/mnt/tab\011here`: "/mnt/tab\there",
|
||||
`/mnt/back\134slash`: `/mnt/back\slash`,
|
||||
`//nas/dci`: "//nas/dci",
|
||||
`/mnt/not\9octal`: `/mnt/not\9octal`,
|
||||
`/mnt/trailing\`: `/mnt/trailing\`,
|
||||
}
|
||||
|
||||
for in, want := range cases {
|
||||
if got := unescapeMountField(in); got != want {
|
||||
t.Errorf("unescapeMountField(%q) = %q, want %q", in, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestNormalizeMountSource(t *testing.T) {
|
||||
cases := map[string]string{
|
||||
"//NAS/DCI": "//nas/dci",
|
||||
"//nas/dci/": "//nas/dci",
|
||||
`\\nas\dci`: "//nas/dci",
|
||||
" //nas/dci ": "//nas/dci",
|
||||
"/": "/",
|
||||
}
|
||||
|
||||
for in, want := range cases {
|
||||
if got := normalizeMountSource(in); got != want {
|
||||
t.Errorf("normalizeMountSource(%q) = %q, want %q", in, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestReadMountTableSkipsMalformedLines(t *testing.T) {
|
||||
path := writeMountTable(t,
|
||||
"source-only",
|
||||
"source /point-but-no-type",
|
||||
"",
|
||||
"//nas/dci /mnt/dci cifs rw 0 0",
|
||||
)
|
||||
|
||||
entries, err := readMountTable(path)
|
||||
if err != nil {
|
||||
t.Fatalf("readMountTable failed: %v", err)
|
||||
}
|
||||
if len(entries) != 1 {
|
||||
t.Fatalf("expected 1 usable entry, got %d: %v", len(entries), entries)
|
||||
}
|
||||
if entries[0].Source != "//nas/dci" || entries[0].Point != "/mnt/dci" || entries[0].FSType != "cifs" {
|
||||
t.Errorf("unexpected entry: %+v", entries[0])
|
||||
}
|
||||
}
|
||||
|
||||
func TestReadMountTableMissingFile(t *testing.T) {
|
||||
if _, err := readMountTable("/definitely/not/a/mount/table"); err == nil {
|
||||
t.Error("expected an error for a missing mount table")
|
||||
}
|
||||
}
|
||||
|
||||
// TestReadRealMountTable closes the loop between the fixtures above and the
|
||||
// kernel's actual output: the parser must cope with whatever this host has
|
||||
// mounted, not just hand-written examples.
|
||||
func TestReadRealMountTable(t *testing.T) {
|
||||
if _, err := os.Stat(procMountsPath); err != nil {
|
||||
t.Skipf("%s is not available on this platform", procMountsPath)
|
||||
}
|
||||
|
||||
entries, err := readMountTable(procMountsPath)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to parse the host's %s: %v", procMountsPath, err)
|
||||
}
|
||||
if len(entries) == 0 {
|
||||
t.Fatalf("%s parsed to zero entries, which cannot be right", procMountsPath)
|
||||
}
|
||||
|
||||
foundRoot := false
|
||||
for _, e := range entries {
|
||||
if e.Point == "" || e.FSType == "" {
|
||||
t.Errorf("entry parsed with empty fields: %+v", e)
|
||||
}
|
||||
if e.Point == "/" {
|
||||
foundRoot = true
|
||||
}
|
||||
}
|
||||
if !foundRoot {
|
||||
t.Error("expected the root filesystem in the host's mount table")
|
||||
}
|
||||
}
|
||||
261
pkg/nas/nas.go
261
pkg/nas/nas.go
@ -6,168 +6,31 @@ import (
|
||||
"io"
|
||||
"log"
|
||||
"os"
|
||||
"sync"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// NASService writes to a NAS destination without caring how this host reaches
|
||||
// it. Parsing the destination and attaching it are delegated to a Location and
|
||||
// a Connector; everything below operates on the resolved base path, which is
|
||||
// always an ordinary path this host's filesystem understands.
|
||||
type NASService struct {
|
||||
Config NASConfig
|
||||
|
||||
location Location
|
||||
connector Connector
|
||||
|
||||
mu sync.RWMutex
|
||||
attached Attachment
|
||||
Config NASConfig
|
||||
connected bool
|
||||
}
|
||||
|
||||
// New parses the configured destination and selects a connection strategy for
|
||||
// the running host. It performs no I/O and does not touch the network, so it is
|
||||
// safe to call before deciding whether a transfer will actually happen; call
|
||||
// Connect to attach the destination.
|
||||
func New(config NASConfig) (*NASService, error) {
|
||||
loc, err := ParseLocation(config.Path)
|
||||
func NewNASService(config NASConfig) *NASService {
|
||||
nt := &NASService{
|
||||
Config: config,
|
||||
}
|
||||
|
||||
// Establish network connection with credentials before accessing the path
|
||||
if err := nt.EstablishConnection(); err != nil {
|
||||
log.Fatalf("Failed to establish network connection to %s: %v", nt.Config.Path, err)
|
||||
}
|
||||
|
||||
err := nt.EnsureDirectoryExists(nt.Config.Path)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
log.Fatalf("Failed to create directory %s: %v", nt.Config.Path, err)
|
||||
}
|
||||
|
||||
return &NASService{
|
||||
Config: config,
|
||||
location: loc,
|
||||
connector: defaultConnector(loc),
|
||||
}, nil
|
||||
}
|
||||
|
||||
// Connect attaches the destination, ensures the target directory exists, and
|
||||
// proves it is writable. The write probe matters: on both platforms a share can
|
||||
// be reachable and still reject writes, and finding that out at startup beats
|
||||
// discovering it mid-recording.
|
||||
func (nt *NASService) Connect(ctx context.Context) error {
|
||||
att, err := nt.connector.Attach(ctx, nt.location, nt.Config)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
nt.mu.Lock()
|
||||
nt.attached = att
|
||||
nt.mu.Unlock()
|
||||
|
||||
// From here on the attachment exists, so any failure has to undo it —
|
||||
// otherwise a mount this process created would outlive the failed startup
|
||||
// and be left behind on the host.
|
||||
if err := nt.EnsureDirectoryExists(att.BasePath); err != nil {
|
||||
nt.abandon(ctx, att)
|
||||
return fmt.Errorf("%s is reachable but %s is unusable: %w", nt.location, att.BasePath, err)
|
||||
}
|
||||
|
||||
if err := nt.writeProbe(att.BasePath); err != nil {
|
||||
nt.abandon(ctx, att)
|
||||
return err
|
||||
}
|
||||
|
||||
nt.mu.Lock()
|
||||
nt.connected = true
|
||||
nt.mu.Unlock()
|
||||
|
||||
log.Printf("Connected to NAS at %s", att.BasePath)
|
||||
return nil
|
||||
}
|
||||
|
||||
// abandon releases a half-finished attachment after Connect fails. The
|
||||
// original error is the one worth reporting, so a teardown failure is only
|
||||
// logged.
|
||||
func (nt *NASService) abandon(ctx context.Context, att Attachment) {
|
||||
// Keep the notes: they explain how far the host got, which is exactly what
|
||||
// the caller needs to report. Everything else is cleared so the service
|
||||
// cannot be mistaken for usable and Detach cannot run twice.
|
||||
nt.mu.Lock()
|
||||
nt.attached = Attachment{Notes: att.Notes}
|
||||
nt.connected = false
|
||||
nt.mu.Unlock()
|
||||
|
||||
if err := nt.connector.Detach(ctx, att); err != nil {
|
||||
log.Printf("Warning: failed to release NAS attachment after a failed connect: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// writeProbe creates and removes a file to confirm the destination accepts
|
||||
// writes.
|
||||
func (nt *NASService) writeProbe(basePath string) error {
|
||||
testFile := joinBase(basePath, ".connection_test")
|
||||
|
||||
f, err := os.Create(testFile)
|
||||
if err != nil {
|
||||
return fmt.Errorf("%s is not writable: %w", basePath, err)
|
||||
}
|
||||
f.Close()
|
||||
|
||||
if err := os.Remove(testFile); err != nil {
|
||||
log.Printf("Warning: failed to remove connection test file %s: %v", testFile, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Disconnect releases only what Connect established. A share that was already
|
||||
// mounted or already connected when the recorder started is left exactly as it
|
||||
// was found.
|
||||
func (nt *NASService) Disconnect(ctx context.Context) error {
|
||||
nt.mu.Lock()
|
||||
att := nt.attached
|
||||
nt.attached = Attachment{}
|
||||
nt.connected = false
|
||||
nt.mu.Unlock()
|
||||
|
||||
if err := nt.connector.Detach(ctx, att); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if att.Owned {
|
||||
log.Printf("Released NAS connection to %s", nt.location)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// LocalPath is the resolved base directory on this host, valid after Connect.
|
||||
func (nt *NASService) LocalPath() string {
|
||||
nt.mu.RLock()
|
||||
defer nt.mu.RUnlock()
|
||||
return nt.attached.BasePath
|
||||
}
|
||||
|
||||
// Join resolves elements against the NAS base path using whatever separator
|
||||
// this host's base path requires.
|
||||
func (nt *NASService) Join(elem ...string) string {
|
||||
return joinBase(nt.LocalPath(), elem...)
|
||||
}
|
||||
|
||||
// Location returns the parsed destination.
|
||||
func (nt *NASService) Location() Location { return nt.location }
|
||||
|
||||
// ConnectionInfo describes how this host reaches the destination, for check
|
||||
// output and startup logs. Safe to call before Connect, when it reports only
|
||||
// what is known from configuration.
|
||||
func (nt *NASService) ConnectionInfo() []string {
|
||||
nt.mu.RLock()
|
||||
defer nt.mu.RUnlock()
|
||||
|
||||
info := []string{
|
||||
"configured: " + nt.location.String(),
|
||||
"strategy: " + nt.connector.Name(),
|
||||
}
|
||||
info = append(info, nt.attached.Notes...)
|
||||
if nt.attached.BasePath != "" && nt.attached.BasePath != nt.location.Raw {
|
||||
info = append(info, "resolved path: "+nt.attached.BasePath)
|
||||
}
|
||||
return info
|
||||
}
|
||||
|
||||
func (nt *NASService) IsConnected() bool {
|
||||
nt.mu.RLock()
|
||||
defer nt.mu.RUnlock()
|
||||
return nt.connected
|
||||
return nt
|
||||
}
|
||||
|
||||
func (nt *NASService) CopyFile(ctx context.Context, srcPath, destPath string) error {
|
||||
@ -226,10 +89,90 @@ func (nt *NASService) EnsureDirectoryExists(path string) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
// FileExists checks whether a file already exists on the NAS, optionally
|
||||
// verifying its size. destinationPath is relative to the NAS base path.
|
||||
func (nt *NASService) EstablishConnection() error {
|
||||
// Extract the network path (\\server\share) from the full path
|
||||
networkPath := nt.ExtractNetworkPath(nt.Config.Path)
|
||||
if networkPath == "" {
|
||||
// Local path, no authentication needed
|
||||
return nil
|
||||
}
|
||||
|
||||
log.Printf("Establishing network connection to %s with user %s", networkPath, nt.Config.Username)
|
||||
|
||||
// Use Windows net use command to establish authenticated connection
|
||||
var cmd *exec.Cmd
|
||||
if nt.Config.Username != "" && nt.Config.Password != "" {
|
||||
cmd = exec.Command("net", "use", networkPath, nt.Config.Password, "/user:"+nt.Config.Username, "/persistent:no")
|
||||
} else {
|
||||
cmd = exec.Command("net", "use", networkPath, "/persistent:no")
|
||||
}
|
||||
|
||||
output, err := cmd.CombinedOutput()
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to establish network connection: %w\nOutput: %s", err, string(output))
|
||||
}
|
||||
|
||||
log.Printf("Network connection established successfully")
|
||||
return nil
|
||||
}
|
||||
|
||||
func (nt *NASService) ExtractNetworkPath(fullPath string) string {
|
||||
// Extract \\server\share from paths like \\server\share\folder\subfolder
|
||||
if !strings.HasPrefix(fullPath, "\\\\") {
|
||||
return "" // Not a UNC path
|
||||
}
|
||||
|
||||
parts := strings.Split(fullPath[2:], "\\") // Remove leading \\
|
||||
if len(parts) < 2 {
|
||||
return "" // Invalid UNC path
|
||||
}
|
||||
|
||||
return "\\\\" + parts[0] + "\\" + parts[1]
|
||||
}
|
||||
|
||||
func (nt *NASService) TestConnection() error {
|
||||
testFile := filepath.Join(nt.Config.Path, ".connection_test")
|
||||
|
||||
f, err := os.Create(testFile)
|
||||
if err != nil {
|
||||
return fmt.Errorf("Failed to create test file: %w", err)
|
||||
}
|
||||
f.Close()
|
||||
|
||||
os.Remove(testFile)
|
||||
|
||||
nt.connected = true
|
||||
log.Printf("Connected to NAS at %s", nt.Config.Path)
|
||||
return nil
|
||||
}
|
||||
|
||||
func (nt *NASService) IsConnected() bool {
|
||||
return nt.connected
|
||||
}
|
||||
|
||||
// Disconnect removes the network connection
|
||||
func (nt *NASService) Disconnect() error {
|
||||
networkPath := nt.ExtractNetworkPath(nt.Config.Path)
|
||||
if networkPath == "" {
|
||||
return nil // Local path, nothing to disconnect
|
||||
}
|
||||
|
||||
cmd := exec.Command("net", "use", networkPath, "/delete")
|
||||
output, err := cmd.CombinedOutput()
|
||||
if err != nil {
|
||||
log.Printf("Warning: failed to disconnect from %s: %v\nOutput: %s", networkPath, err, string(output))
|
||||
// Don't return error since this is cleanup
|
||||
} else {
|
||||
log.Printf("Disconnected from network path: %s", networkPath)
|
||||
}
|
||||
|
||||
nt.connected = false
|
||||
return nil
|
||||
}
|
||||
|
||||
// FileExists checks if a file already exists on the NAS and optionally verifies size
|
||||
func (nt *NASService) FileExists(destinationPath string, expectedSize int64) (bool, error) {
|
||||
fullDestPath := nt.Join(destinationPath)
|
||||
fullDestPath := filepath.Join(nt.Config.Path, destinationPath)
|
||||
|
||||
destInfo, err := os.Stat(fullDestPath)
|
||||
if err != nil {
|
||||
@ -249,9 +192,9 @@ func (nt *NASService) FileExists(destinationPath string, expectedSize int64) (bo
|
||||
return true, nil
|
||||
}
|
||||
|
||||
// GetFileSize returns the size of a file on the NAS, relative to the base path.
|
||||
// GetFileSize returns the size of a file on the NAS
|
||||
func (nt *NASService) GetFileSize(destinationPath string) (int64, error) {
|
||||
fullDestPath := nt.Join(destinationPath)
|
||||
fullDestPath := filepath.Join(nt.Config.Path, destinationPath)
|
||||
|
||||
destInfo, err := os.Stat(fullDestPath)
|
||||
if err != nil {
|
||||
|
||||
@ -1,308 +0,0 @@
|
||||
package nas
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// fakeConnector stands in for a platform strategy so the service's own
|
||||
// behaviour — probing, path resolution, teardown — can be tested without a NAS.
|
||||
type fakeConnector struct {
|
||||
basePath string
|
||||
attachErr error
|
||||
owned bool
|
||||
attachCount int
|
||||
detachCount int
|
||||
detached Attachment
|
||||
}
|
||||
|
||||
func (f *fakeConnector) Name() string { return "fake" }
|
||||
|
||||
func (f *fakeConnector) Attach(context.Context, Location, NASConfig) (Attachment, error) {
|
||||
f.attachCount++
|
||||
if f.attachErr != nil {
|
||||
return Attachment{}, f.attachErr
|
||||
}
|
||||
return Attachment{BasePath: f.basePath, Owned: f.owned, Notes: []string{"fake attachment"}}, nil
|
||||
}
|
||||
|
||||
func (f *fakeConnector) Detach(_ context.Context, att Attachment) error {
|
||||
f.detachCount++
|
||||
f.detached = att
|
||||
return nil
|
||||
}
|
||||
|
||||
func newTestService(t *testing.T, configuredPath string, conn Connector) *NASService {
|
||||
t.Helper()
|
||||
svc, err := New(NASConfig{Path: configuredPath})
|
||||
if err != nil {
|
||||
t.Fatalf("New(%q) failed: %v", configuredPath, err)
|
||||
}
|
||||
svc.connector = conn
|
||||
return svc
|
||||
}
|
||||
|
||||
func TestNewRejectsUnusableDestination(t *testing.T) {
|
||||
if _, err := New(NASConfig{Path: ""}); err == nil {
|
||||
t.Error("an empty NAS path should be rejected")
|
||||
}
|
||||
if _, err := New(NASConfig{Path: `\\server`}); err == nil {
|
||||
t.Error("a server without a share should be rejected")
|
||||
}
|
||||
}
|
||||
|
||||
func TestNewDoesNoIO(t *testing.T) {
|
||||
// Constructing the service must not touch the network or filesystem, so
|
||||
// that modes which never transfer can build one harmlessly.
|
||||
svc, err := New(NASConfig{Path: `\\nas\dci\streams`})
|
||||
if err != nil {
|
||||
t.Fatalf("New failed: %v", err)
|
||||
}
|
||||
if svc.IsConnected() {
|
||||
t.Error("service should not report connected before Connect")
|
||||
}
|
||||
if svc.LocalPath() != "" {
|
||||
t.Errorf("no path should be resolved before Connect, got %q", svc.LocalPath())
|
||||
}
|
||||
}
|
||||
|
||||
func TestConnectResolvesAndProbes(t *testing.T) {
|
||||
base := filepath.Join(t.TempDir(), "streams")
|
||||
conn := &fakeConnector{basePath: base}
|
||||
svc := newTestService(t, `\\nas\dci\streams`, conn)
|
||||
|
||||
if err := svc.Connect(context.Background()); err != nil {
|
||||
t.Fatalf("Connect failed: %v", err)
|
||||
}
|
||||
|
||||
if !svc.IsConnected() {
|
||||
t.Error("service should report connected")
|
||||
}
|
||||
if svc.LocalPath() != base {
|
||||
t.Errorf("LocalPath = %q, want %q", svc.LocalPath(), base)
|
||||
}
|
||||
// Connect creates the destination directory if it is missing.
|
||||
if info, err := os.Stat(base); err != nil || !info.IsDir() {
|
||||
t.Errorf("Connect should have created %s: %v", base, err)
|
||||
}
|
||||
// The write probe must clean up after itself.
|
||||
if _, err := os.Stat(filepath.Join(base, ".connection_test")); !os.IsNotExist(err) {
|
||||
t.Errorf("probe file should have been removed, stat err = %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestConnectSurfacesAttachFailure(t *testing.T) {
|
||||
conn := &fakeConnector{attachErr: fmt.Errorf("share is not mounted")}
|
||||
svc := newTestService(t, `\\nas\dci`, conn)
|
||||
|
||||
err := svc.Connect(context.Background())
|
||||
if err == nil || !strings.Contains(err.Error(), "not mounted") {
|
||||
t.Fatalf("expected the connector's error to surface, got: %v", err)
|
||||
}
|
||||
if svc.IsConnected() {
|
||||
t.Error("a failed attach must not leave the service marked connected")
|
||||
}
|
||||
}
|
||||
|
||||
func TestConnectFailsWhenDestinationUnwritable(t *testing.T) {
|
||||
// A file where the directory should be: reachable, but unusable.
|
||||
blocker := filepath.Join(t.TempDir(), "blocker")
|
||||
if err := os.WriteFile(blocker, []byte("x"), 0644); err != nil {
|
||||
t.Fatalf("setup failed: %v", err)
|
||||
}
|
||||
|
||||
svc := newTestService(t, "/tmp/whatever", &fakeConnector{basePath: filepath.Join(blocker, "streams")})
|
||||
if err := svc.Connect(context.Background()); err == nil {
|
||||
t.Fatal("expected an unusable destination to fail the connect")
|
||||
}
|
||||
if svc.IsConnected() {
|
||||
t.Error("service should not report connected after a failed probe")
|
||||
}
|
||||
}
|
||||
|
||||
func TestFailedProbeReleasesTheAttachment(t *testing.T) {
|
||||
// A mount created by Attach must not outlive a failed Connect, or a failed
|
||||
// startup leaves a mount behind on the host.
|
||||
blocker := filepath.Join(t.TempDir(), "blocker")
|
||||
if err := os.WriteFile(blocker, []byte("x"), 0644); err != nil {
|
||||
t.Fatalf("setup failed: %v", err)
|
||||
}
|
||||
|
||||
conn := &fakeConnector{basePath: filepath.Join(blocker, "streams"), owned: true}
|
||||
svc := newTestService(t, `\\nas\dci`, conn)
|
||||
|
||||
if err := svc.Connect(context.Background()); err == nil {
|
||||
t.Fatal("expected Connect to fail")
|
||||
}
|
||||
|
||||
if conn.detachCount != 1 {
|
||||
t.Errorf("Detach called %d times after a failed connect, want 1", conn.detachCount)
|
||||
}
|
||||
if !conn.detached.Owned {
|
||||
t.Error("Detach should have been told the attachment was owned")
|
||||
}
|
||||
if svc.LocalPath() != "" {
|
||||
t.Errorf("a released attachment should expose no path, got %q", svc.LocalPath())
|
||||
}
|
||||
// The diagnosis of how far it got must survive for the caller to report.
|
||||
if info := strings.Join(svc.ConnectionInfo(), "\n"); !strings.Contains(info, "fake attachment") {
|
||||
t.Errorf("connection notes should survive a failed connect:\n%s", info)
|
||||
}
|
||||
|
||||
// A later Disconnect must not detach a second time.
|
||||
if err := svc.Disconnect(context.Background()); err != nil {
|
||||
t.Fatalf("Disconnect failed: %v", err)
|
||||
}
|
||||
if conn.detachCount != 2 || conn.detached.Owned {
|
||||
t.Errorf("Disconnect after release should be a no-op teardown, detachCount=%d owned=%v",
|
||||
conn.detachCount, conn.detached.Owned)
|
||||
}
|
||||
}
|
||||
|
||||
func TestJoinUsesResolvedPath(t *testing.T) {
|
||||
base := t.TempDir()
|
||||
svc := newTestService(t, `\\nas\dci\streams`, &fakeConnector{basePath: base})
|
||||
if err := svc.Connect(context.Background()); err != nil {
|
||||
t.Fatalf("Connect failed: %v", err)
|
||||
}
|
||||
|
||||
// Crucially the join is against the resolved path, not the configured UNC
|
||||
// path — that is the whole point of the indirection.
|
||||
if got, want := svc.Join("event", "1080p"), filepath.Join(base, "event", "1080p"); got != want {
|
||||
t.Errorf("Join = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestFileExistsAndGetFileSizeUseResolvedPath(t *testing.T) {
|
||||
base := t.TempDir()
|
||||
svc := newTestService(t, `\\nas\dci`, &fakeConnector{basePath: base})
|
||||
if err := svc.Connect(context.Background()); err != nil {
|
||||
t.Fatalf("Connect failed: %v", err)
|
||||
}
|
||||
|
||||
rel := filepath.Join("event", "seg.ts")
|
||||
if err := os.MkdirAll(filepath.Join(base, "event"), 0755); err != nil {
|
||||
t.Fatalf("setup failed: %v", err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(base, rel), []byte("0123456789"), 0644); err != nil {
|
||||
t.Fatalf("setup failed: %v", err)
|
||||
}
|
||||
|
||||
exists, err := svc.FileExists(rel, 10)
|
||||
if err != nil || !exists {
|
||||
t.Errorf("FileExists = %v, %v; want true, nil", exists, err)
|
||||
}
|
||||
|
||||
// A size mismatch means a partial transfer, which should not count.
|
||||
exists, err = svc.FileExists(rel, 99)
|
||||
if err != nil || exists {
|
||||
t.Errorf("FileExists with wrong size = %v, %v; want false, nil", exists, err)
|
||||
}
|
||||
|
||||
exists, err = svc.FileExists(filepath.Join("event", "missing.ts"), 0)
|
||||
if err != nil || exists {
|
||||
t.Errorf("FileExists for missing file = %v, %v; want false, nil", exists, err)
|
||||
}
|
||||
|
||||
size, err := svc.GetFileSize(rel)
|
||||
if err != nil || size != 10 {
|
||||
t.Errorf("GetFileSize = %d, %v; want 10, nil", size, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDisconnectDelegatesAndResets(t *testing.T) {
|
||||
conn := &fakeConnector{basePath: t.TempDir(), owned: true}
|
||||
svc := newTestService(t, `\\nas\dci`, conn)
|
||||
if err := svc.Connect(context.Background()); err != nil {
|
||||
t.Fatalf("Connect failed: %v", err)
|
||||
}
|
||||
|
||||
if err := svc.Disconnect(context.Background()); err != nil {
|
||||
t.Fatalf("Disconnect failed: %v", err)
|
||||
}
|
||||
|
||||
if conn.detachCount != 1 {
|
||||
t.Errorf("Detach called %d times, want 1", conn.detachCount)
|
||||
}
|
||||
if !conn.detached.Owned {
|
||||
t.Error("Detach should receive the attachment so it knows what it owns")
|
||||
}
|
||||
if svc.IsConnected() {
|
||||
t.Error("service should not report connected after Disconnect")
|
||||
}
|
||||
}
|
||||
|
||||
func TestConnectionInfoDescribesResolution(t *testing.T) {
|
||||
base := t.TempDir()
|
||||
svc := newTestService(t, `\\nas\dci\streams`, &fakeConnector{basePath: base})
|
||||
|
||||
// Before connecting, only configuration is known.
|
||||
before := strings.Join(svc.ConnectionInfo(), "\n")
|
||||
if !strings.Contains(before, `\\nas\dci\streams`) || !strings.Contains(before, "fake") {
|
||||
t.Errorf("pre-connect info should name the destination and strategy:\n%s", before)
|
||||
}
|
||||
|
||||
if err := svc.Connect(context.Background()); err != nil {
|
||||
t.Fatalf("Connect failed: %v", err)
|
||||
}
|
||||
|
||||
after := strings.Join(svc.ConnectionInfo(), "\n")
|
||||
if !strings.Contains(after, "resolved path: "+base) {
|
||||
t.Errorf("post-connect info should report where the destination resolved to:\n%s", after)
|
||||
}
|
||||
}
|
||||
|
||||
func TestVerifyTransferDetectsSizeMismatch(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
src := filepath.Join(dir, "src.ts")
|
||||
dst := filepath.Join(dir, "dst.ts")
|
||||
if err := os.WriteFile(src, []byte("0123456789"), 0644); err != nil {
|
||||
t.Fatalf("setup failed: %v", err)
|
||||
}
|
||||
if err := os.WriteFile(dst, []byte("012"), 0644); err != nil {
|
||||
t.Fatalf("setup failed: %v", err)
|
||||
}
|
||||
|
||||
svc := newTestService(t, dir, &fakeConnector{basePath: dir})
|
||||
if err := svc.VerifyTransfer(src, dst); err == nil {
|
||||
t.Error("a truncated destination should fail verification")
|
||||
}
|
||||
|
||||
if err := os.WriteFile(dst, []byte("0123456789"), 0644); err != nil {
|
||||
t.Fatalf("setup failed: %v", err)
|
||||
}
|
||||
if err := svc.VerifyTransfer(src, dst); err != nil {
|
||||
t.Errorf("matching sizes should verify: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCopyFile(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
src := filepath.Join(dir, "src.ts")
|
||||
dst := filepath.Join(dir, "dst.ts")
|
||||
payload := []byte("segment payload")
|
||||
if err := os.WriteFile(src, payload, 0644); err != nil {
|
||||
t.Fatalf("setup failed: %v", err)
|
||||
}
|
||||
|
||||
svc := newTestService(t, dir, &fakeConnector{basePath: dir})
|
||||
if err := svc.CopyFile(context.Background(), src, dst); err != nil {
|
||||
t.Fatalf("CopyFile failed: %v", err)
|
||||
}
|
||||
|
||||
got, err := os.ReadFile(dst)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to read copy: %v", err)
|
||||
}
|
||||
if string(got) != string(payload) {
|
||||
t.Errorf("copied contents = %q, want %q", got, payload)
|
||||
}
|
||||
|
||||
if err := svc.CopyFile(context.Background(), filepath.Join(dir, "missing.ts"), dst); err == nil {
|
||||
t.Error("copying a missing source should fail")
|
||||
}
|
||||
}
|
||||
@ -5,61 +5,48 @@ import (
|
||||
"context"
|
||||
"fmt"
|
||||
"log"
|
||||
"m3u8-downloader/pkg/config"
|
||||
"m3u8-downloader/pkg/health"
|
||||
"m3u8-downloader/pkg/logging"
|
||||
"m3u8-downloader/pkg/constants"
|
||||
"m3u8-downloader/pkg/nas"
|
||||
"m3u8-downloader/pkg/utils"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"runtime"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
)
|
||||
|
||||
type ProcessingService struct {
|
||||
config *config.Config
|
||||
eventName string
|
||||
nas *nas.NASService
|
||||
config *ProcessConfig
|
||||
nas *nas.NASService
|
||||
}
|
||||
|
||||
func NewProcessingService(eventName string, cfg *config.Config) (*ProcessingService, error) {
|
||||
if cfg == nil {
|
||||
return nil, fmt.Errorf("configuration is required")
|
||||
func NewProcessingService(eventName string) (*ProcessingService, error) {
|
||||
config := &ProcessConfig{
|
||||
WorkerCount: constants.ProcessWorkerCount,
|
||||
SourcePath: constants.NASOutputPath + "/" + eventName,
|
||||
DestinationPath: constants.ProcessOutputPath,
|
||||
Enabled: constants.ProcessingEnabled,
|
||||
EventName: eventName,
|
||||
}
|
||||
|
||||
nasConfig := nas.NASConfig{
|
||||
Path: cfg.NAS.OutputPath,
|
||||
Username: cfg.NAS.Username,
|
||||
Password: cfg.NAS.Password,
|
||||
Timeout: cfg.NAS.Timeout,
|
||||
RetryLimit: cfg.NAS.RetryLimit,
|
||||
nasConfig := &nas.NASConfig{
|
||||
Path: constants.NASOutputPath,
|
||||
Username: constants.NASUsername,
|
||||
Password: constants.NASPassword,
|
||||
Timeout: constants.TransferTimeout,
|
||||
RetryLimit: constants.TransferRetryLimit,
|
||||
VerifySize: true,
|
||||
AutoMount: cfg.NAS.AutoMount,
|
||||
MountPoint: cfg.NAS.MountPoint,
|
||||
}
|
||||
|
||||
nasService, err := nas.New(nasConfig)
|
||||
if err != nil {
|
||||
logging.MustGetFactory().Processing.Error("invalid NAS destination", logging.KeyError, err.Error())
|
||||
return nil, fmt.Errorf("invalid NAS destination: %w", err)
|
||||
}
|
||||
nasService := nas.NewNASService(*nasConfig)
|
||||
|
||||
connectCtx, cancel := context.WithTimeout(context.Background(), cfg.NAS.Timeout)
|
||||
defer cancel()
|
||||
if err := nasService.Connect(connectCtx); err != nil {
|
||||
logging.MustGetFactory().Processing.Error("failed to connect to NAS", logging.KeyError, err.Error())
|
||||
return nil, fmt.Errorf("failed to connect to NAS: %w", err)
|
||||
if err := nasService.TestConnection(); err != nil {
|
||||
return nil, fmt.Errorf("Failed to connect to NAS: %w", err)
|
||||
}
|
||||
|
||||
return &ProcessingService{
|
||||
config: cfg,
|
||||
eventName: eventName,
|
||||
nas: nasService,
|
||||
config: config,
|
||||
nas: nasService,
|
||||
}, nil
|
||||
}
|
||||
|
||||
@ -67,20 +54,11 @@ func (ps *ProcessingService) GetSegmentInfo() (map[int]string, error) {
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
// nasEventPath is the event's directory on the NAS as reached from this host.
|
||||
// It goes through the NAS service rather than config.GetNASEventPath because
|
||||
// the configured path may be a UNC path that this host reaches via a mount
|
||||
// point instead.
|
||||
func (ps *ProcessingService) nasEventPath() string {
|
||||
return ps.nas.Join(ps.eventName)
|
||||
}
|
||||
|
||||
func (ps *ProcessingService) GetEventDirs() ([]string, error) {
|
||||
if ps.eventName == "" {
|
||||
sourcePath := ps.nas.LocalPath()
|
||||
dirs, err := os.ReadDir(sourcePath)
|
||||
if ps.config.EventName == "" {
|
||||
dirs, err := os.ReadDir(ps.config.SourcePath)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to read directory %s: %w", sourcePath, err)
|
||||
return nil, fmt.Errorf("Failed to read directory: %w", err)
|
||||
}
|
||||
var eventDirs []string
|
||||
for _, dir := range dirs {
|
||||
@ -90,37 +68,23 @@ func (ps *ProcessingService) GetEventDirs() ([]string, error) {
|
||||
}
|
||||
return eventDirs, nil
|
||||
} else {
|
||||
return []string{ps.eventName}, nil
|
||||
return []string{ps.config.EventName}, nil
|
||||
}
|
||||
}
|
||||
|
||||
func (ps *ProcessingService) Start(ctx context.Context) error {
|
||||
procLog := logging.MustGetFactory().Processing
|
||||
registry := health.MustGetRegistry()
|
||||
|
||||
if !ps.config.Processing.Enabled {
|
||||
if !ps.config.Enabled {
|
||||
log.Println("Processing service disabled")
|
||||
return nil
|
||||
}
|
||||
|
||||
registry.RecordProcessingStageChange("scanning", ps.eventName)
|
||||
|
||||
// fail centralizes the log/registry bookkeeping for every error return
|
||||
// site below instead of repeating it at each one.
|
||||
fail := func(err error) error {
|
||||
procLog.Error("processing failed", logging.KeyEvent, ps.eventName, logging.KeyError, err.Error())
|
||||
registry.RecordProcessingStageChange("failed", ps.eventName)
|
||||
registry.RecordProcessingError(err)
|
||||
return err
|
||||
}
|
||||
|
||||
if ps.eventName == "" {
|
||||
if ps.config.EventName == "" {
|
||||
events, err := ps.GetEventDirs()
|
||||
if err != nil {
|
||||
return fail(fmt.Errorf("failed to get event directories: %w", err))
|
||||
return fmt.Errorf("Failed to get event directories: %w", err)
|
||||
}
|
||||
if len(events) == 0 {
|
||||
return fail(fmt.Errorf("no events found"))
|
||||
return fmt.Errorf("No events found")
|
||||
}
|
||||
if len(events) > 1 {
|
||||
fmt.Println("Multiple events found, please select one:")
|
||||
@ -132,23 +96,21 @@ func (ps *ProcessingService) Start(ctx context.Context) error {
|
||||
input = strings.TrimSpace(input)
|
||||
index, err := strconv.Atoi(input)
|
||||
if err != nil {
|
||||
return fail(fmt.Errorf("failed to parse input: %w", err))
|
||||
return fmt.Errorf("Failed to parse input: %w", err)
|
||||
}
|
||||
if index < 1 || index > len(events) {
|
||||
return fail(fmt.Errorf("invalid input"))
|
||||
return fmt.Errorf("Invalid input")
|
||||
}
|
||||
ps.eventName = events[index-1]
|
||||
ps.config.EventName = events[index-1]
|
||||
} else {
|
||||
ps.eventName = events[0]
|
||||
ps.config.EventName = events[0]
|
||||
}
|
||||
registry.RecordProcessingStageChange("scanning", ps.eventName)
|
||||
}
|
||||
procLog.Info("processing started", logging.KeyEvent, ps.eventName)
|
||||
|
||||
//Get all present resolutions
|
||||
dirs, err := ps.GetResolutions()
|
||||
if err != nil {
|
||||
return fail(fmt.Errorf("Failed to get resolutions: %w", err))
|
||||
return fmt.Errorf("Failed to get resolutions: %w", err)
|
||||
}
|
||||
|
||||
//Spawn a worker per resolution
|
||||
@ -166,37 +128,24 @@ func (ps *ProcessingService) Start(ctx context.Context) error {
|
||||
|
||||
segments, err := ps.AggregateSegmentInfo(ch)
|
||||
if err != nil {
|
||||
return fail(fmt.Errorf("Failed to aggregate segment info: %w", err))
|
||||
}
|
||||
registry.RecordProcessingSegmentsFound(len(segments))
|
||||
|
||||
aggFile, err := ps.WriteConcatFile(segments)
|
||||
if err != nil {
|
||||
return fail(fmt.Errorf("Failed to write concat file: %w", err))
|
||||
return fmt.Errorf("Failed to aggregate segment info: %w", err)
|
||||
}
|
||||
|
||||
// Feed info to ffmpeg to stitch files together
|
||||
outPath := ps.config.GetProcessOutputPath(ps.eventName)
|
||||
if err := utils.EnsureDir(outPath); err != nil {
|
||||
return fail(fmt.Errorf("failed to create output directory: %w", err))
|
||||
}
|
||||
ps.WriteConcatFile(segments)
|
||||
|
||||
registry.RecordProcessingStageChange("running", ps.eventName)
|
||||
concatErr := ps.RunFFmpeg(aggFile, outPath)
|
||||
//Feed info to ffmpeg to stitch files together
|
||||
concatErr := ps.RunFFmpeg(ps.config.SourcePath, ps.config.DestinationPath)
|
||||
if concatErr != nil {
|
||||
return fail(concatErr)
|
||||
return concatErr
|
||||
}
|
||||
|
||||
procLog.Info("processing completed", logging.KeyEvent, ps.eventName, "segments", len(segments))
|
||||
registry.RecordProcessingStageChange("completed", ps.eventName)
|
||||
return nil
|
||||
}
|
||||
|
||||
func (ps *ProcessingService) GetResolutions() ([]string, error) {
|
||||
eventPath := ps.nasEventPath()
|
||||
dirs, err := os.ReadDir(eventPath)
|
||||
dirs, err := os.ReadDir(ps.config.SourcePath)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to read source directory %s: %w", eventPath, err)
|
||||
return nil, fmt.Errorf("Failed to read source directory: %w", err)
|
||||
}
|
||||
|
||||
re := regexp.MustCompile(`^\d+p$`)
|
||||
@ -214,19 +163,15 @@ func (ps *ProcessingService) GetResolutions() ([]string, error) {
|
||||
func (ps *ProcessingService) ParseResolutionDirectory(resolution string, ch chan<- SegmentInfo, wg *sync.WaitGroup) {
|
||||
defer wg.Done()
|
||||
|
||||
resolutionPath := utils.SafeJoin(ps.nasEventPath(), resolution)
|
||||
files, err := os.ReadDir(resolutionPath)
|
||||
files, err := os.ReadDir(ps.config.SourcePath + "/" + resolution)
|
||||
if err != nil {
|
||||
log.Printf("Failed to read resolution directory %s: %v", resolutionPath, err)
|
||||
log.Printf("Failed to read resolution directory: %v", err)
|
||||
return
|
||||
}
|
||||
|
||||
for _, file := range files {
|
||||
if !file.IsDir() {
|
||||
if !strings.HasSuffix(strings.ToLower(file.Name()), ".ts") {
|
||||
continue
|
||||
}
|
||||
no, err := strconv.Atoi(file.Name()[6:10])
|
||||
no, err := strconv.Atoi(file.Name()[7:11])
|
||||
if err != nil {
|
||||
log.Printf("Failed to parse segment number: %v", err)
|
||||
continue
|
||||
@ -243,14 +188,20 @@ func (ps *ProcessingService) ParseResolutionDirectory(resolution string, ch chan
|
||||
func (ps *ProcessingService) AggregateSegmentInfo(ch <-chan SegmentInfo) (map[int]SegmentInfo, error) {
|
||||
segmentMap := make(map[int]SegmentInfo)
|
||||
|
||||
// Each sequence number arrives once per downloaded variant; keep the
|
||||
// highest-resolution copy of each, since that is the one worth stitching
|
||||
// into the final video.
|
||||
rank := map[string]int{
|
||||
"1080p": 1,
|
||||
"720p": 2,
|
||||
"540p": 3,
|
||||
"480p": 4,
|
||||
"450p": 5,
|
||||
"360p": 6,
|
||||
"270p": 7,
|
||||
"240p": 8,
|
||||
}
|
||||
|
||||
for segment := range ch {
|
||||
fmt.Printf("Received segment %s in resolution %s \n", segment.Name, segment.Resolution)
|
||||
logging.MustGetFactory().Processing.Debug("segment received", logging.KeySegment, segment.Name, logging.KeyResolution, segment.Resolution)
|
||||
current, exists := segmentMap[segment.SeqNo]
|
||||
if !exists || resolutionHeight(segment.Resolution) > resolutionHeight(current.Resolution) {
|
||||
if !exists || rank[segment.Resolution] > rank[current.Resolution] {
|
||||
segmentMap[segment.SeqNo] = segment
|
||||
}
|
||||
}
|
||||
@ -258,134 +209,30 @@ func (ps *ProcessingService) AggregateSegmentInfo(ch <-chan SegmentInfo) (map[in
|
||||
return segmentMap, nil
|
||||
}
|
||||
|
||||
// resolutionHeight converts a resolution label such as "1080p" into the pixel
|
||||
// height it stands for, so variants can be ranked by quality.
|
||||
//
|
||||
// The height is parsed rather than looked up in a table: labels come from the
|
||||
// playlist's own RESOLUTION attribute, so a stream can legitimately offer
|
||||
// 1440p or 2160p that no fixed table would anticipate. Anything not in
|
||||
// <height>p form — "unknown", from a playlist with no variants — returns 0 and
|
||||
// therefore loses to every recognised resolution.
|
||||
func resolutionHeight(resolution string) int {
|
||||
trimmed := strings.TrimSuffix(strings.ToLower(strings.TrimSpace(resolution)), "p")
|
||||
|
||||
height, err := strconv.Atoi(trimmed)
|
||||
if err != nil || height < 0 {
|
||||
return 0
|
||||
}
|
||||
return height
|
||||
}
|
||||
|
||||
func (ps *ProcessingService) WriteConcatFile(segmentMap map[int]SegmentInfo) (string, error) {
|
||||
concatPath := ps.config.GetProcessOutputPath(ps.eventName)
|
||||
|
||||
if err := utils.EnsureDir(concatPath); err != nil {
|
||||
return "", fmt.Errorf("failed to create directories for concat path: %w", err)
|
||||
}
|
||||
|
||||
concatFilePath := utils.SafeJoin(concatPath, ps.eventName+".txt")
|
||||
f, err := os.Create(concatFilePath)
|
||||
func (ps *ProcessingService) WriteConcatFile(segmentMap map[int]SegmentInfo) error {
|
||||
f, err := os.Create(ps.config.DestinationPath + "/concat.txt")
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("failed to create concat file: %w", err)
|
||||
return fmt.Errorf("Failed to create concat file: %w", err)
|
||||
}
|
||||
defer f.Close()
|
||||
|
||||
// Sort keys to preserve order
|
||||
keys := make([]int, 0, len(segmentMap))
|
||||
for k := range segmentMap {
|
||||
keys = append(keys, k)
|
||||
}
|
||||
sort.Ints(keys)
|
||||
|
||||
for _, seq := range keys {
|
||||
segment := segmentMap[seq]
|
||||
filePath := utils.SafeJoin(ps.nasEventPath(), segment.Resolution, segment.Name)
|
||||
line := fmt.Sprintf("file '%s'\n", filePath)
|
||||
if _, err := f.WriteString(line); err != nil {
|
||||
return "", fmt.Errorf("failed to write to concat file: %w", err)
|
||||
}
|
||||
}
|
||||
|
||||
return concatFilePath, nil
|
||||
}
|
||||
|
||||
func (ps *ProcessingService) getFFmpegPath() (string, error) {
|
||||
// First try the configured path
|
||||
configuredPath := ps.config.Processing.FFmpegPath
|
||||
if configuredPath != "" {
|
||||
// Check if it's just the command name or a full path
|
||||
if filepath.IsAbs(configuredPath) {
|
||||
return configuredPath, nil
|
||||
}
|
||||
|
||||
// Try to find it in PATH
|
||||
if fullPath, err := exec.LookPath(configuredPath); err == nil {
|
||||
return fullPath, nil
|
||||
}
|
||||
}
|
||||
|
||||
// Fallback: try local bin directory
|
||||
var baseDir string
|
||||
exePath, err := os.Executable()
|
||||
if err == nil {
|
||||
baseDir = filepath.Dir(exePath)
|
||||
} else {
|
||||
baseDir, err = os.Getwd()
|
||||
for _, segment := range segmentMap {
|
||||
_, err = f.WriteString("file '" + ps.config.SourcePath + "/" + segment.Resolution + "/" + segment.Name + "'\n")
|
||||
if err != nil {
|
||||
return "", err
|
||||
return fmt.Errorf("Failed to write to concat file: %w", err)
|
||||
}
|
||||
}
|
||||
|
||||
ffmpeg := utils.SafeJoin(baseDir, "bin", "ffmpeg")
|
||||
if runtime.GOOS == "windows" {
|
||||
ffmpeg += ".exe"
|
||||
}
|
||||
|
||||
if utils.PathExists(ffmpeg) {
|
||||
return ffmpeg, nil
|
||||
}
|
||||
|
||||
// Try current working directory
|
||||
cwd, err := os.Getwd()
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
ffmpeg = utils.SafeJoin(cwd, "bin", "ffmpeg")
|
||||
if runtime.GOOS == "windows" {
|
||||
ffmpeg += ".exe"
|
||||
}
|
||||
|
||||
if utils.PathExists(ffmpeg) {
|
||||
return ffmpeg, nil
|
||||
}
|
||||
|
||||
return "", fmt.Errorf("FFmpeg not found. Please install FFmpeg or set FFMPEG_PATH environment variable")
|
||||
return nil
|
||||
}
|
||||
|
||||
func (ps *ProcessingService) RunFFmpeg(inputPath, outputPath string) error {
|
||||
fmt.Println("Running ffmpeg...")
|
||||
|
||||
fileOutPath := utils.SafeJoin(outputPath, ps.eventName+".mp4")
|
||||
fmt.Println("Input path:", inputPath)
|
||||
fmt.Println("Output path:", fileOutPath)
|
||||
logging.MustGetFactory().Processing.Info("running ffmpeg", "input", inputPath, "output", fileOutPath)
|
||||
|
||||
path, err := ps.getFFmpegPath()
|
||||
if err != nil {
|
||||
logging.MustGetFactory().Processing.Error("failed to find ffmpeg", logging.KeyError, err.Error())
|
||||
return fmt.Errorf("failed to find FFmpeg: %w", err)
|
||||
}
|
||||
|
||||
cmd := exec.Command(path, "-f", "concat", "-safe", "0", "-i", inputPath, "-c", "copy", fileOutPath)
|
||||
cmd := exec.Command("ffmpeg", "-f", "concat", "-safe", "0", "-i", inputPath, "-c", "copy", outputPath)
|
||||
cmd.Stdout = os.Stdout
|
||||
cmd.Stderr = os.Stderr
|
||||
|
||||
if err := cmd.Run(); err != nil {
|
||||
logging.MustGetFactory().Processing.Error("ffmpeg run failed", logging.KeyError, err.Error())
|
||||
return fmt.Errorf("failed to run ffmpeg: %w", err)
|
||||
err := cmd.Run()
|
||||
if err != nil {
|
||||
return fmt.Errorf("Failed to run ffmpeg: %w", err)
|
||||
}
|
||||
|
||||
fmt.Println("FFmpeg completed successfully")
|
||||
logging.MustGetFactory().Processing.Info("ffmpeg completed successfully", "output", fileOutPath)
|
||||
return nil
|
||||
}
|
||||
|
||||
@ -1,472 +0,0 @@
|
||||
package processing
|
||||
|
||||
import (
|
||||
"context"
|
||||
"m3u8-downloader/pkg/config"
|
||||
"m3u8-downloader/pkg/nas"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"runtime"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// connectedNAS attaches a NAS service to a local directory. Reaching the NAS
|
||||
// is platform-specific, but an already-mounted or plain directory resolves the
|
||||
// same way everywhere, which is what makes this usable as a test double.
|
||||
func connectedNAS(t *testing.T, path string) *nas.NASService {
|
||||
t.Helper()
|
||||
|
||||
svc, err := nas.New(nas.NASConfig{Path: path, Timeout: 5 * time.Second})
|
||||
if err != nil {
|
||||
t.Fatalf("Failed to create NAS service for %s: %v", path, err)
|
||||
}
|
||||
if err := svc.Connect(context.Background()); err != nil {
|
||||
t.Fatalf("Failed to connect NAS service to %s: %v", path, err)
|
||||
}
|
||||
return svc
|
||||
}
|
||||
|
||||
func createTestConfig(tempDir string) *config.Config {
|
||||
return &config.Config{
|
||||
Core: config.CoreConfig{
|
||||
WorkerCount: 2,
|
||||
RefreshDelay: 1 * time.Second,
|
||||
},
|
||||
NAS: config.NASConfig{
|
||||
OutputPath: filepath.Join(tempDir, "nas"),
|
||||
Username: "testuser",
|
||||
Password: "testpass",
|
||||
Timeout: 10 * time.Second,
|
||||
RetryLimit: 2,
|
||||
EnableTransfer: false, // Disable to avoid NAS connection
|
||||
},
|
||||
Processing: config.ProcessingConfig{
|
||||
Enabled: true,
|
||||
AutoProcess: true,
|
||||
WorkerCount: 1,
|
||||
FFmpegPath: "echo", // Use echo command for testing
|
||||
},
|
||||
Paths: config.PathsConfig{
|
||||
LocalOutput: filepath.Join(tempDir, "data"),
|
||||
ProcessOutput: filepath.Join(tempDir, "out"),
|
||||
ManifestDir: filepath.Join(tempDir, "data"),
|
||||
PersistenceFile: filepath.Join(tempDir, "queue.json"),
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func TestNewProcessingService_Success(t *testing.T) {
|
||||
tempDir, err := os.MkdirTemp("", "processing_test_*")
|
||||
if err != nil {
|
||||
t.Fatalf("Failed to create temp dir: %v", err)
|
||||
}
|
||||
defer os.RemoveAll(tempDir)
|
||||
|
||||
cfg := createTestConfig(tempDir)
|
||||
cfg.NAS.EnableTransfer = false // Disable NAS to avoid connection
|
||||
|
||||
// We can't test actual NAS connection, so we'll skip the constructor test
|
||||
// that requires NAS connectivity. Instead, test the configuration handling.
|
||||
|
||||
if cfg.Processing.FFmpegPath != "echo" {
|
||||
t.Errorf("Expected FFmpegPath='echo', got '%s'", cfg.Processing.FFmpegPath)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNewProcessingService_NilConfig(t *testing.T) {
|
||||
_, err := NewProcessingService("test-event", nil)
|
||||
if err == nil {
|
||||
t.Error("Expected error for nil config")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "configuration is required") {
|
||||
t.Errorf("Expected 'configuration is required' error, got: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestProcessingService_GetEventDirs(t *testing.T) {
|
||||
tempDir, err := os.MkdirTemp("", "processing_test_*")
|
||||
if err != nil {
|
||||
t.Fatalf("Failed to create temp dir: %v", err)
|
||||
}
|
||||
defer os.RemoveAll(tempDir)
|
||||
|
||||
cfg := createTestConfig(tempDir)
|
||||
|
||||
// Create mock NAS directory structure
|
||||
nasDir := cfg.NAS.OutputPath
|
||||
os.MkdirAll(filepath.Join(nasDir, "event1"), 0755)
|
||||
os.MkdirAll(filepath.Join(nasDir, "event2"), 0755)
|
||||
os.MkdirAll(filepath.Join(nasDir, "event3"), 0755)
|
||||
// Create a file (should be ignored)
|
||||
os.WriteFile(filepath.Join(nasDir, "not_a_dir.txt"), []byte("test"), 0644)
|
||||
|
||||
ps := &ProcessingService{
|
||||
config: cfg,
|
||||
eventName: "", // Empty to test directory discovery
|
||||
nas: connectedNAS(t, cfg.NAS.OutputPath),
|
||||
}
|
||||
|
||||
dirs, err := ps.GetEventDirs()
|
||||
if err != nil {
|
||||
t.Fatalf("GetEventDirs() failed: %v", err)
|
||||
}
|
||||
|
||||
if len(dirs) != 3 {
|
||||
t.Errorf("Expected 3 event directories, got %d", len(dirs))
|
||||
}
|
||||
|
||||
expectedDirs := []string{"event1", "event2", "event3"}
|
||||
for _, expected := range expectedDirs {
|
||||
found := false
|
||||
for _, actual := range dirs {
|
||||
if actual == expected {
|
||||
found = true
|
||||
break
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
t.Errorf("Expected to find directory '%s' in results: %v", expected, dirs)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestProcessingService_GetEventDirs_WithEventName(t *testing.T) {
|
||||
// Scoped to a temp dir rather than /tmp directly: attaching the NAS
|
||||
// service creates the destination, and tests should not leave one behind.
|
||||
cfg := createTestConfig(t.TempDir())
|
||||
eventName := "specific-event"
|
||||
|
||||
ps := &ProcessingService{
|
||||
config: cfg,
|
||||
eventName: eventName,
|
||||
nas: connectedNAS(t, cfg.NAS.OutputPath),
|
||||
}
|
||||
|
||||
dirs, err := ps.GetEventDirs()
|
||||
if err != nil {
|
||||
t.Fatalf("GetEventDirs() failed: %v", err)
|
||||
}
|
||||
|
||||
if len(dirs) != 1 {
|
||||
t.Errorf("Expected 1 directory, got %d", len(dirs))
|
||||
}
|
||||
if dirs[0] != eventName {
|
||||
t.Errorf("Expected directory '%s', got '%s'", eventName, dirs[0])
|
||||
}
|
||||
}
|
||||
|
||||
func TestProcessingService_GetResolutions(t *testing.T) {
|
||||
tempDir, err := os.MkdirTemp("", "processing_test_*")
|
||||
if err != nil {
|
||||
t.Fatalf("Failed to create temp dir: %v", err)
|
||||
}
|
||||
defer os.RemoveAll(tempDir)
|
||||
|
||||
cfg := createTestConfig(tempDir)
|
||||
eventName := "test-event"
|
||||
|
||||
// Create mock event directory with quality subdirectories
|
||||
eventPath := filepath.Join(cfg.NAS.OutputPath, eventName)
|
||||
os.MkdirAll(filepath.Join(eventPath, "1080p"), 0755)
|
||||
os.MkdirAll(filepath.Join(eventPath, "720p"), 0755)
|
||||
os.MkdirAll(filepath.Join(eventPath, "480p"), 0755)
|
||||
os.MkdirAll(filepath.Join(eventPath, "not_resolution"), 0755) // Should be ignored
|
||||
os.WriteFile(filepath.Join(eventPath, "file.txt"), []byte("test"), 0644) // Should be ignored
|
||||
|
||||
ps := &ProcessingService{
|
||||
config: cfg,
|
||||
eventName: eventName,
|
||||
nas: connectedNAS(t, cfg.NAS.OutputPath),
|
||||
}
|
||||
|
||||
resolutions, err := ps.GetResolutions()
|
||||
if err != nil {
|
||||
t.Fatalf("GetResolutions() failed: %v", err)
|
||||
}
|
||||
|
||||
expectedResolutions := []string{"1080p", "720p", "480p"}
|
||||
if len(resolutions) != len(expectedResolutions) {
|
||||
t.Errorf("Expected %d resolutions, got %d: %v", len(expectedResolutions), len(resolutions), resolutions)
|
||||
}
|
||||
|
||||
for _, expected := range expectedResolutions {
|
||||
found := false
|
||||
for _, actual := range resolutions {
|
||||
if actual == expected {
|
||||
found = true
|
||||
break
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
t.Errorf("Expected to find resolution '%s' in results: %v", expected, resolutions)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestProcessingService_AggregateSegmentInfo(t *testing.T) {
|
||||
ps := &ProcessingService{}
|
||||
|
||||
// Create test channel with segments
|
||||
ch := make(chan SegmentInfo, 5)
|
||||
|
||||
// Add segments with different qualities for same sequence
|
||||
ch <- SegmentInfo{Name: "seg_1001.ts", SeqNo: 1001, Resolution: "720p"}
|
||||
ch <- SegmentInfo{Name: "seg_1001.ts", SeqNo: 1001, Resolution: "1080p"} // Higher quality, should win
|
||||
ch <- SegmentInfo{Name: "seg_1002.ts", SeqNo: 1002, Resolution: "480p"}
|
||||
ch <- SegmentInfo{Name: "seg_1003.ts", SeqNo: 1003, Resolution: "1080p"}
|
||||
ch <- SegmentInfo{Name: "seg_1001.ts", SeqNo: 1001, Resolution: "540p"} // Lower than 1080p, should not replace
|
||||
|
||||
close(ch)
|
||||
|
||||
segmentMap, err := ps.AggregateSegmentInfo(ch)
|
||||
if err != nil {
|
||||
t.Fatalf("AggregateSegmentInfo() failed: %v", err)
|
||||
}
|
||||
|
||||
// Should have 3 unique sequence numbers
|
||||
if len(segmentMap) != 3 {
|
||||
t.Errorf("Expected 3 unique segments, got %d", len(segmentMap))
|
||||
}
|
||||
|
||||
// Check sequence 1001 has the highest quality (1080p)
|
||||
seg1001, exists := segmentMap[1001]
|
||||
if !exists {
|
||||
t.Fatal("Segment 1001 should exist")
|
||||
}
|
||||
if seg1001.Resolution != "1080p" {
|
||||
t.Errorf("Expected segment 1001 to have resolution '1080p', got '%s'", seg1001.Resolution)
|
||||
}
|
||||
|
||||
// Check sequence 1002 has 480p
|
||||
seg1002, exists := segmentMap[1002]
|
||||
if !exists {
|
||||
t.Fatal("Segment 1002 should exist")
|
||||
}
|
||||
if seg1002.Resolution != "480p" {
|
||||
t.Errorf("Expected segment 1002 to have resolution '480p', got '%s'", seg1002.Resolution)
|
||||
}
|
||||
|
||||
// Check sequence 1003 has 1080p
|
||||
seg1003, exists := segmentMap[1003]
|
||||
if !exists {
|
||||
t.Fatal("Segment 1003 should exist")
|
||||
}
|
||||
if seg1003.Resolution != "1080p" {
|
||||
t.Errorf("Expected segment 1003 to have resolution '1080p', got '%s'", seg1003.Resolution)
|
||||
}
|
||||
}
|
||||
|
||||
func TestProcessingService_WriteConcatFile(t *testing.T) {
|
||||
tempDir, err := os.MkdirTemp("", "processing_test_*")
|
||||
if err != nil {
|
||||
t.Fatalf("Failed to create temp dir: %v", err)
|
||||
}
|
||||
defer os.RemoveAll(tempDir)
|
||||
|
||||
cfg := createTestConfig(tempDir)
|
||||
eventName := "test-event"
|
||||
|
||||
ps := &ProcessingService{
|
||||
config: cfg,
|
||||
eventName: eventName,
|
||||
nas: connectedNAS(t, cfg.NAS.OutputPath),
|
||||
}
|
||||
|
||||
// Create test segment map
|
||||
segmentMap := map[int]SegmentInfo{
|
||||
1003: {Name: "seg_1003.ts", SeqNo: 1003, Resolution: "1080p"},
|
||||
1001: {Name: "seg_1001.ts", SeqNo: 1001, Resolution: "720p"},
|
||||
1002: {Name: "seg_1002.ts", SeqNo: 1002, Resolution: "1080p"},
|
||||
}
|
||||
|
||||
concatFilePath, err := ps.WriteConcatFile(segmentMap)
|
||||
if err != nil {
|
||||
t.Fatalf("WriteConcatFile() failed: %v", err)
|
||||
}
|
||||
|
||||
// Verify file was created
|
||||
if _, err := os.Stat(concatFilePath); os.IsNotExist(err) {
|
||||
t.Fatalf("Concat file was not created: %s", concatFilePath)
|
||||
}
|
||||
|
||||
// Read and verify content
|
||||
content, err := os.ReadFile(concatFilePath)
|
||||
if err != nil {
|
||||
t.Fatalf("Failed to read concat file: %v", err)
|
||||
}
|
||||
|
||||
contentStr := string(content)
|
||||
lines := strings.Split(strings.TrimSpace(contentStr), "\n")
|
||||
|
||||
if len(lines) != 3 {
|
||||
t.Errorf("Expected 3 lines in concat file, got %d", len(lines))
|
||||
}
|
||||
|
||||
// Verify segments are sorted by sequence number
|
||||
expectedOrder := []string{"seg_1001.ts", "seg_1002.ts", "seg_1003.ts"}
|
||||
for i, line := range lines {
|
||||
if !strings.Contains(line, expectedOrder[i]) {
|
||||
t.Errorf("Line %d should contain '%s', got: %s", i, expectedOrder[i], line)
|
||||
}
|
||||
if !strings.HasPrefix(line, "file '") {
|
||||
t.Errorf("Line %d should start with 'file ', got: %s", i, line)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestProcessingService_getFFmpegPath(t *testing.T) {
|
||||
cfg := createTestConfig("/tmp")
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
ffmpegPath string
|
||||
shouldFind bool
|
||||
expectedError string
|
||||
}{
|
||||
{
|
||||
name: "echo command (should be found in PATH)",
|
||||
ffmpegPath: "echo",
|
||||
shouldFind: true,
|
||||
},
|
||||
{
|
||||
name: "absolute path test",
|
||||
ffmpegPath: func() string {
|
||||
if runtime.GOOS == "windows" {
|
||||
return "C:\\Windows\\System32\\cmd.exe"
|
||||
}
|
||||
return "/bin/echo"
|
||||
}(),
|
||||
shouldFind: true,
|
||||
},
|
||||
{
|
||||
name: "nonexistent command",
|
||||
ffmpegPath: "nonexistent_ffmpeg_command_12345",
|
||||
shouldFind: false,
|
||||
expectedError: "FFmpeg not found",
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
testCfg := *cfg
|
||||
testCfg.Processing.FFmpegPath = tt.ffmpegPath
|
||||
|
||||
ps := &ProcessingService{
|
||||
config: &testCfg,
|
||||
eventName: "test",
|
||||
}
|
||||
|
||||
path, err := ps.getFFmpegPath()
|
||||
|
||||
if tt.shouldFind {
|
||||
if err != nil {
|
||||
t.Errorf("Expected to find FFmpeg, but got error: %v", err)
|
||||
}
|
||||
if path == "" {
|
||||
t.Error("Expected non-empty path")
|
||||
}
|
||||
} else {
|
||||
if err == nil {
|
||||
t.Error("Expected error for nonexistent FFmpeg")
|
||||
}
|
||||
if tt.expectedError != "" && !strings.Contains(err.Error(), tt.expectedError) {
|
||||
t.Errorf("Expected error containing '%s', got: %v", tt.expectedError, err)
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestSegmentInfo_Structure(t *testing.T) {
|
||||
segment := SegmentInfo{
|
||||
Name: "test_segment.ts",
|
||||
SeqNo: 1001,
|
||||
Resolution: "1080p",
|
||||
}
|
||||
|
||||
if segment.Name != "test_segment.ts" {
|
||||
t.Errorf("Expected Name='test_segment.ts', got '%s'", segment.Name)
|
||||
}
|
||||
if segment.SeqNo != 1001 {
|
||||
t.Errorf("Expected SeqNo=1001, got %d", segment.SeqNo)
|
||||
}
|
||||
if segment.Resolution != "1080p" {
|
||||
t.Errorf("Expected Resolution='1080p', got '%s'", segment.Resolution)
|
||||
}
|
||||
}
|
||||
|
||||
func TestProcessJob_Structure(t *testing.T) {
|
||||
job := ProcessJob{
|
||||
EventName: "test-event",
|
||||
}
|
||||
|
||||
if job.EventName != "test-event" {
|
||||
t.Errorf("Expected EventName='test-event', got '%s'", job.EventName)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolutionHeight(t *testing.T) {
|
||||
cases := map[string]int{
|
||||
"1080p": 1080,
|
||||
"720p": 720,
|
||||
"2160p": 2160, // beyond anything a fixed quality table would list
|
||||
"1440p": 1440,
|
||||
"240p": 240,
|
||||
// Not in <height>p form: must rank below every real resolution rather
|
||||
// than accidentally outranking them.
|
||||
"unknown": 0,
|
||||
"": 0,
|
||||
"p": 0,
|
||||
"-720p": 0,
|
||||
}
|
||||
|
||||
for label, want := range cases {
|
||||
if got := resolutionHeight(label); got != want {
|
||||
t.Errorf("resolutionHeight(%q) = %d, want %d", label, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestAggregateSegmentInfoPrefersHighestResolution covers the variants a
|
||||
// hardcoded rank table could not express: resolutions above 1080p, and the
|
||||
// "unknown" label produced for a playlist with no variants.
|
||||
func TestAggregateSegmentInfoPrefersHighestResolution(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
arriving []string
|
||||
want string
|
||||
}{
|
||||
{"1080p beats 720p regardless of order", []string{"720p", "1080p"}, "1080p"},
|
||||
{"1080p beats 720p when it arrives first", []string{"1080p", "720p"}, "1080p"},
|
||||
{"4k beats 1080p", []string{"1080p", "2160p"}, "2160p"},
|
||||
{"1440p beats 1080p", []string{"1080p", "1440p"}, "1440p"},
|
||||
{"a known resolution beats unknown", []string{"unknown", "480p"}, "480p"},
|
||||
{"unknown does not displace a known resolution", []string{"1080p", "unknown"}, "1080p"},
|
||||
{"unknown alone is still kept", []string{"unknown"}, "unknown"},
|
||||
{"lowest only", []string{"240p"}, "240p"},
|
||||
}
|
||||
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
ps := &ProcessingService{}
|
||||
ch := make(chan SegmentInfo, len(tc.arriving))
|
||||
for _, res := range tc.arriving {
|
||||
ch <- SegmentInfo{Name: "seg_1001.ts", SeqNo: 1001, Resolution: res}
|
||||
}
|
||||
close(ch)
|
||||
|
||||
segments, err := ps.AggregateSegmentInfo(ch)
|
||||
if err != nil {
|
||||
t.Fatalf("AggregateSegmentInfo() failed: %v", err)
|
||||
}
|
||||
if len(segments) != 1 {
|
||||
t.Fatalf("expected a single sequence number, got %d", len(segments))
|
||||
}
|
||||
if got := segments[1001].Resolution; got != tc.want {
|
||||
t.Errorf("kept %q, want %q (arrival order: %v)", got, tc.want, tc.arriving)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@ -3,3 +3,11 @@ package processing
|
||||
type ProcessJob struct {
|
||||
EventName string
|
||||
}
|
||||
|
||||
type ProcessConfig struct {
|
||||
WorkerCount int
|
||||
SourcePath string
|
||||
DestinationPath string
|
||||
Enabled bool
|
||||
EventName string
|
||||
}
|
||||
|
||||
@ -2,20 +2,13 @@ package transfer
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"log"
|
||||
"m3u8-downloader/pkg/logging"
|
||||
"os"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
// errRetained reports that a file is still inside its retention window. It is
|
||||
// not a failure: the file simply has to be reconsidered on a later pass rather
|
||||
// than being discarded from the pending list.
|
||||
var errRetained = errors.New("file is within the retention period")
|
||||
|
||||
type CleanupService struct {
|
||||
config CleanupConfig
|
||||
pendingFiles []string
|
||||
@ -48,7 +41,6 @@ func (cs *CleanupService) Start(ctx context.Context) error {
|
||||
}
|
||||
|
||||
log.Printf("Cleanup service started (retention: %v, batch: %d)", cs.config.RetentionPeriod, cs.config.BatchSize)
|
||||
logging.MustGetFactory().Transfer.Info("cleanup service started", "retention", cs.config.RetentionPeriod.String(), "batch_size", cs.config.BatchSize)
|
||||
|
||||
ticker := time.NewTicker(cs.config.CheckInterval)
|
||||
defer ticker.Stop()
|
||||
@ -88,62 +80,36 @@ func (cs *CleanupService) ExecuteCleanup(ctx context.Context) error {
|
||||
log.Printf("Processing %d files for cleanup", len(batch))
|
||||
|
||||
var cleanedCount int
|
||||
var failures []error
|
||||
// requeue collects files this pass must not drop: ones still inside their
|
||||
// retention window, and — if the context is cancelled mid-batch — the
|
||||
// remainder that was never looked at. They were already removed from
|
||||
// pendingFiles above, so without this they would never be cleaned at all.
|
||||
var requeue []string
|
||||
var errors []error
|
||||
|
||||
for i, filePath := range batch {
|
||||
for _, filePath := range batch {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
requeue = append(requeue, batch[i:]...)
|
||||
cs.requeue(requeue)
|
||||
return ctx.Err()
|
||||
default:
|
||||
}
|
||||
|
||||
err := cs.cleanupFile(filePath)
|
||||
switch {
|
||||
case errors.Is(err, errRetained):
|
||||
requeue = append(requeue, filePath)
|
||||
case err != nil:
|
||||
failures = append(failures, fmt.Errorf("Failed to cleanup file %s: %w", filePath, err))
|
||||
default:
|
||||
if err := cs.cleanupFile(filePath); err != nil {
|
||||
errors = append(errors, fmt.Errorf("Failed to cleanup file %s: %w", filePath, err))
|
||||
} else {
|
||||
cleanedCount++
|
||||
}
|
||||
}
|
||||
|
||||
cs.requeue(requeue)
|
||||
log.Printf("Cleanup batch completed (cleaned: %d, errors: %d)", cleanedCount, len(errors))
|
||||
|
||||
log.Printf("Cleanup batch completed (cleaned: %d, retained: %d, errors: %d)", cleanedCount, len(requeue), len(failures))
|
||||
logging.MustGetFactory().Transfer.Info("cleanup batch completed",
|
||||
"cleaned", cleanedCount, "retained", len(requeue), "errors", len(failures))
|
||||
|
||||
if len(failures) > 0 {
|
||||
for i, err := range failures {
|
||||
if len(errors) > 0 {
|
||||
for i, err := range errors {
|
||||
if i >= 3 {
|
||||
log.Printf("... and %d more errors", len(failures)-3)
|
||||
log.Printf("... and %d more errors", len(errors)-3)
|
||||
break
|
||||
}
|
||||
log.Printf("Error: %v", err)
|
||||
logging.MustGetFactory().Transfer.Warn("cleanup error", logging.KeyError, err.Error())
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// requeue returns files to the pending list for a later pass.
|
||||
func (cs *CleanupService) requeue(files []string) {
|
||||
if len(files) == 0 {
|
||||
return
|
||||
}
|
||||
|
||||
cs.mu.Lock()
|
||||
defer cs.mu.Unlock()
|
||||
cs.pendingFiles = append(cs.pendingFiles, files...)
|
||||
}
|
||||
|
||||
func (cs *CleanupService) cleanupFile(filePath string) error {
|
||||
@ -158,7 +124,7 @@ func (cs *CleanupService) cleanupFile(filePath string) error {
|
||||
if cs.config.RetentionPeriod > 0 {
|
||||
if time.Since(info.ModTime()) < cs.config.RetentionPeriod {
|
||||
log.Printf("File too new to cleanup: %s", filePath)
|
||||
return errRetained
|
||||
return nil
|
||||
}
|
||||
}
|
||||
|
||||
@ -192,18 +158,6 @@ func (cs *CleanupService) ForceCleanupAll(ctx context.Context) error {
|
||||
return err
|
||||
}
|
||||
|
||||
// Stop when a pass makes no progress. Files still inside their
|
||||
// retention window are requeued by design, so requiring the list to
|
||||
// reach zero would spin here until the shutdown timeout expired.
|
||||
cs.mu.Lock()
|
||||
remaining := len(cs.pendingFiles)
|
||||
cs.mu.Unlock()
|
||||
|
||||
if remaining >= pendingCount {
|
||||
log.Printf("Force cleanup stopping with %d file(s) still pending (within retention period)", remaining)
|
||||
break
|
||||
}
|
||||
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return ctx.Err()
|
||||
|
||||
@ -1,190 +0,0 @@
|
||||
package transfer
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func writeAgedFile(t *testing.T, dir, name string, age time.Duration) string {
|
||||
t.Helper()
|
||||
|
||||
path := filepath.Join(dir, name)
|
||||
if err := os.WriteFile(path, []byte("segment"), 0644); err != nil {
|
||||
t.Fatalf("failed to write %s: %v", path, err)
|
||||
}
|
||||
|
||||
modTime := time.Now().Add(-age)
|
||||
if err := os.Chtimes(path, modTime, modTime); err != nil {
|
||||
t.Fatalf("failed to age %s: %v", path, err)
|
||||
}
|
||||
return path
|
||||
}
|
||||
|
||||
func TestCleanupRemovesTransferredFiles(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
cs := NewCleanupService(CleanupConfig{Enabled: true, BatchSize: 10, CheckInterval: time.Second})
|
||||
|
||||
path := writeAgedFile(t, dir, "seg_1001.ts", 0)
|
||||
if err := cs.ScheduleCleanup(path); err != nil {
|
||||
t.Fatalf("ScheduleCleanup failed: %v", err)
|
||||
}
|
||||
|
||||
if err := cs.ExecuteCleanup(context.Background()); err != nil {
|
||||
t.Fatalf("ExecuteCleanup failed: %v", err)
|
||||
}
|
||||
|
||||
if _, err := os.Stat(path); !os.IsNotExist(err) {
|
||||
t.Errorf("file should have been removed, stat err = %v", err)
|
||||
}
|
||||
if got := cs.GetPendingCount(); got != 0 {
|
||||
t.Errorf("pending count = %d, want 0", got)
|
||||
}
|
||||
}
|
||||
|
||||
// TestCleanupRetainsThenRemoves is the regression test for a file inside its
|
||||
// retention window being dropped from the pending list: it was reported as
|
||||
// cleaned, never deleted, and never reconsidered, so local files accumulated
|
||||
// forever whenever RetainHours was set.
|
||||
func TestCleanupRetainsThenRemoves(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
cs := NewCleanupService(CleanupConfig{
|
||||
Enabled: true,
|
||||
RetentionPeriod: time.Hour,
|
||||
BatchSize: 10,
|
||||
CheckInterval: time.Second,
|
||||
})
|
||||
|
||||
fresh := writeAgedFile(t, dir, "fresh.ts", time.Minute)
|
||||
old := writeAgedFile(t, dir, "old.ts", 2*time.Hour)
|
||||
|
||||
for _, p := range []string{fresh, old} {
|
||||
if err := cs.ScheduleCleanup(p); err != nil {
|
||||
t.Fatalf("ScheduleCleanup failed: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
if err := cs.ExecuteCleanup(context.Background()); err != nil {
|
||||
t.Fatalf("ExecuteCleanup failed: %v", err)
|
||||
}
|
||||
|
||||
if _, err := os.Stat(old); !os.IsNotExist(err) {
|
||||
t.Errorf("a file past its retention period should be removed, stat err = %v", err)
|
||||
}
|
||||
if _, err := os.Stat(fresh); err != nil {
|
||||
t.Errorf("a file inside its retention period should survive: %v", err)
|
||||
}
|
||||
if got := cs.GetPendingCount(); got != 1 {
|
||||
t.Fatalf("the retained file must stay queued for a later pass, pending = %d", got)
|
||||
}
|
||||
|
||||
// Once it ages past the window, the next pass must actually remove it.
|
||||
past := time.Now().Add(-2 * time.Hour)
|
||||
if err := os.Chtimes(fresh, past, past); err != nil {
|
||||
t.Fatalf("failed to age file: %v", err)
|
||||
}
|
||||
if err := cs.ExecuteCleanup(context.Background()); err != nil {
|
||||
t.Fatalf("ExecuteCleanup failed: %v", err)
|
||||
}
|
||||
if _, err := os.Stat(fresh); !os.IsNotExist(err) {
|
||||
t.Errorf("file should have been removed once past retention, stat err = %v", err)
|
||||
}
|
||||
if got := cs.GetPendingCount(); got != 0 {
|
||||
t.Errorf("pending count = %d, want 0", got)
|
||||
}
|
||||
}
|
||||
|
||||
// TestForceCleanupAllTerminatesWithRetainedFiles guards shutdown: retained
|
||||
// files are requeued by design, so a loop demanding an empty list would spin
|
||||
// until the shutdown timeout expired.
|
||||
func TestForceCleanupAllTerminatesWithRetainedFiles(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
cs := NewCleanupService(CleanupConfig{
|
||||
Enabled: true,
|
||||
RetentionPeriod: time.Hour,
|
||||
BatchSize: 10,
|
||||
CheckInterval: time.Second,
|
||||
})
|
||||
|
||||
if err := cs.ScheduleCleanup(writeAgedFile(t, dir, "fresh.ts", time.Minute)); err != nil {
|
||||
t.Fatalf("ScheduleCleanup failed: %v", err)
|
||||
}
|
||||
|
||||
done := make(chan error, 1)
|
||||
go func() { done <- cs.ForceCleanupAll(context.Background()) }()
|
||||
|
||||
select {
|
||||
case err := <-done:
|
||||
if err != nil {
|
||||
t.Fatalf("ForceCleanupAll failed: %v", err)
|
||||
}
|
||||
case <-time.After(3 * time.Second):
|
||||
t.Fatal("ForceCleanupAll did not terminate with a retained file pending")
|
||||
}
|
||||
|
||||
if got := cs.GetPendingCount(); got != 1 {
|
||||
t.Errorf("the retained file should still be pending, got %d", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCleanupMissingFileIsNotAnError(t *testing.T) {
|
||||
cs := NewCleanupService(CleanupConfig{Enabled: true, BatchSize: 10, CheckInterval: time.Second})
|
||||
|
||||
// Already gone — a file transferred and removed by something else.
|
||||
if err := cs.ScheduleCleanup(filepath.Join(t.TempDir(), "gone.ts")); err != nil {
|
||||
t.Fatalf("ScheduleCleanup failed: %v", err)
|
||||
}
|
||||
if err := cs.ExecuteCleanup(context.Background()); err != nil {
|
||||
t.Fatalf("ExecuteCleanup failed: %v", err)
|
||||
}
|
||||
if got := cs.GetPendingCount(); got != 0 {
|
||||
t.Errorf("a missing file should not be requeued, pending = %d", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCleanupDisabledSchedulesNothing(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
cs := NewCleanupService(CleanupConfig{Enabled: false, BatchSize: 10, CheckInterval: time.Second})
|
||||
|
||||
path := writeAgedFile(t, dir, "seg.ts", time.Hour)
|
||||
if err := cs.ScheduleCleanup(path); err != nil {
|
||||
t.Fatalf("ScheduleCleanup failed: %v", err)
|
||||
}
|
||||
|
||||
if got := cs.GetPendingCount(); got != 0 {
|
||||
t.Errorf("cleanup is disabled, nothing should be queued, got %d", got)
|
||||
}
|
||||
if err := cs.ExecuteCleanup(context.Background()); err != nil {
|
||||
t.Fatalf("ExecuteCleanup failed: %v", err)
|
||||
}
|
||||
if _, err := os.Stat(path); err != nil {
|
||||
t.Errorf("file must survive when cleanup is disabled: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestCleanupRequeuesOnCancellation covers shutdown arriving mid-batch: the
|
||||
// files already pulled off the pending list must go back, not vanish.
|
||||
func TestCleanupRequeuesOnCancellation(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
cs := NewCleanupService(CleanupConfig{Enabled: true, BatchSize: 10, CheckInterval: time.Second})
|
||||
|
||||
for i := 0; i < 5; i++ {
|
||||
path := writeAgedFile(t, dir, fmt.Sprintf("seg_%d.ts", i), 0)
|
||||
if err := cs.ScheduleCleanup(path); err != nil {
|
||||
t.Fatalf("ScheduleCleanup failed: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
cancel()
|
||||
|
||||
if err := cs.ExecuteCleanup(ctx); err == nil {
|
||||
t.Fatal("expected the cancelled context to be reported")
|
||||
}
|
||||
if got := cs.GetPendingCount(); got != 5 {
|
||||
t.Errorf("all files should be requeued after cancellation, pending = %d, want 5", got)
|
||||
}
|
||||
}
|
||||
@ -4,14 +4,13 @@ import (
|
||||
"context"
|
||||
"fmt"
|
||||
"log"
|
||||
"m3u8-downloader/pkg/logging"
|
||||
"m3u8-downloader/pkg/nas"
|
||||
"os"
|
||||
"path/filepath"
|
||||
)
|
||||
|
||||
func TransferFile(nt *nas.NASService, ctx context.Context, item *TransferItem) error {
|
||||
destPath := nt.Join(item.DestinationPath)
|
||||
destPath := filepath.Join(nt.Config.Path, item.DestinationPath)
|
||||
|
||||
destDir := filepath.Dir(destPath)
|
||||
if err := nt.EnsureDirectoryExists(destDir); err != nil {
|
||||
@ -33,7 +32,6 @@ func TransferFile(nt *nas.NASService, ctx context.Context, item *TransferItem) e
|
||||
}
|
||||
|
||||
log.Printf("File transfer completed: %s -> %s", item.SourcePath, destPath)
|
||||
logging.MustGetFactory().Transfer.Debug("nas copy completed", logging.KeyFile, item.SourcePath, "dest", destPath)
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
@ -6,8 +6,6 @@ import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"log"
|
||||
"m3u8-downloader/pkg/health"
|
||||
"m3u8-downloader/pkg/logging"
|
||||
"m3u8-downloader/pkg/nas"
|
||||
"os"
|
||||
"sync"
|
||||
@ -76,10 +74,7 @@ func (tq *TransferQueue) Add(item TransferItem) error {
|
||||
defer tq.mu.Unlock()
|
||||
|
||||
if tq.items.Len() >= tq.config.MaxQueueSize {
|
||||
err := fmt.Errorf("Queue is full (max size: %d)", tq.config.MaxQueueSize)
|
||||
logging.MustGetFactory().Transfer.Error("queue full, dropping file", logging.KeyFile, item.SourcePath, logging.KeyError, err.Error())
|
||||
health.MustGetRegistry().RecordTransferError(item.SourcePath, err)
|
||||
return err
|
||||
return fmt.Errorf("Queue is full (max size: %d)", tq.config.MaxQueueSize)
|
||||
}
|
||||
|
||||
heap.Push(tq.items, &item)
|
||||
@ -169,20 +164,13 @@ func (tq *TransferQueue) processItem(ctx context.Context, item TransferItem) {
|
||||
return
|
||||
}
|
||||
|
||||
maxRetries := tq.config.RetryLimit
|
||||
if maxRetries < 1 {
|
||||
maxRetries = 1
|
||||
}
|
||||
maxRetries := 3
|
||||
|
||||
for attempt := 1; attempt <= maxRetries; attempt++ {
|
||||
// Back off before a retry, never before the first attempt. The guard
|
||||
// was `attempt > 0`, which is always true, so every single transfer —
|
||||
// including healthy ones — paid a one-second delay before it began,
|
||||
// throttling the whole pipeline to the worker count per second.
|
||||
if attempt > 1 {
|
||||
if attempt > 0 {
|
||||
item.Status = StatusRetrying
|
||||
backoff := time.Duration(attempt*attempt) * time.Second
|
||||
log.Printf("Backing off for %s before retrying (attempt %d/%d)", backoff, attempt, maxRetries)
|
||||
log.Printf("Backing off for %d seconds before retrying (attempt %d/%d)", backoff, attempt, maxRetries)
|
||||
|
||||
select {
|
||||
case <-time.After(backoff):
|
||||
@ -199,11 +187,9 @@ func (tq *TransferQueue) processItem(ctx context.Context, item TransferItem) {
|
||||
if tq.cleanup != nil {
|
||||
if err := tq.cleanup.ScheduleCleanup(item.SourcePath); err != nil {
|
||||
log.Printf("Failed to add file to cleanup list: %v", err)
|
||||
logging.MustGetFactory().Transfer.Warn("failed to schedule cleanup", logging.KeyFile, item.SourcePath, logging.KeyError, err.Error())
|
||||
}
|
||||
}
|
||||
log.Printf("File transfer completed: %s", item.SourcePath)
|
||||
logging.MustGetFactory().Transfer.Info("file transfer completed", logging.KeyFile, item.SourcePath, logging.KeyBytes, item.FileSize)
|
||||
return
|
||||
}
|
||||
|
||||
@ -211,14 +197,11 @@ func (tq *TransferQueue) processItem(ctx context.Context, item TransferItem) {
|
||||
item.RetryCount++
|
||||
|
||||
log.Printf("File transfer failed: %s (attempt %d/%d): %v", item.SourcePath, item.RetryCount, maxRetries, err)
|
||||
logging.MustGetFactory().Transfer.Warn("file transfer attempt failed", logging.KeyFile, item.SourcePath, logging.KeyAttempt, item.RetryCount, logging.KeyError, err.Error())
|
||||
|
||||
if attempt == maxRetries {
|
||||
item.Status = StatusFailed
|
||||
tq.stats.IncrementFailed()
|
||||
log.Printf("Transfer permanently failed for file: %s", item.SourcePath)
|
||||
logging.MustGetFactory().Transfer.Error("transfer permanently failed", logging.KeyFile, item.SourcePath, logging.KeyError, err.Error())
|
||||
health.MustGetRegistry().RecordTransferError(item.SourcePath, err)
|
||||
return
|
||||
}
|
||||
}
|
||||
@ -242,12 +225,10 @@ func (tq *TransferQueue) SaveState() error {
|
||||
"timestamp": time.Now(),
|
||||
}, "", " ")
|
||||
if err != nil {
|
||||
logging.MustGetFactory().Transfer.Error("failed to marshal queue state", logging.KeyError, err.Error())
|
||||
return fmt.Errorf("Failed to marshal queue state: %w", err)
|
||||
}
|
||||
|
||||
if err := os.WriteFile(tq.config.PersistencePath, data, 0644); err != nil {
|
||||
logging.MustGetFactory().Transfer.Error("failed to save queue state", logging.KeyError, err.Error())
|
||||
return fmt.Errorf("Failed to save queue state: %w", err)
|
||||
}
|
||||
|
||||
@ -260,7 +241,6 @@ func (tq *TransferQueue) LoadState() error {
|
||||
if os.IsNotExist(err) {
|
||||
return nil
|
||||
}
|
||||
logging.MustGetFactory().Transfer.Error("failed to load queue state", logging.KeyError, err.Error())
|
||||
return fmt.Errorf("Failed to load queue state: %w", err)
|
||||
}
|
||||
|
||||
|
||||
@ -5,10 +5,7 @@ import (
|
||||
"fmt"
|
||||
"log"
|
||||
"m3u8-downloader/pkg/constants"
|
||||
"m3u8-downloader/pkg/health"
|
||||
"m3u8-downloader/pkg/logging"
|
||||
nas2 "m3u8-downloader/pkg/nas"
|
||||
"m3u8-downloader/pkg/utils"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
@ -25,64 +22,47 @@ type TransferService struct {
|
||||
}
|
||||
|
||||
func NewTrasferService(outputDir string, eventName string) (*TransferService, error) {
|
||||
cfg := constants.MustGetConfig()
|
||||
|
||||
nasConfig := nas2.NASConfig{
|
||||
Path: outputDir,
|
||||
Username: cfg.NAS.Username,
|
||||
Password: cfg.NAS.Password,
|
||||
Timeout: cfg.NAS.Timeout,
|
||||
RetryLimit: cfg.NAS.RetryLimit,
|
||||
Username: constants.NASUsername,
|
||||
Password: constants.NASPassword,
|
||||
Timeout: constants.TransferTimeout,
|
||||
RetryLimit: constants.TransferRetryLimit,
|
||||
VerifySize: true,
|
||||
AutoMount: cfg.NAS.AutoMount,
|
||||
MountPoint: cfg.NAS.MountPoint,
|
||||
}
|
||||
nas, err := nas2.New(nasConfig)
|
||||
if err != nil {
|
||||
logging.MustGetFactory().Transfer.Error("invalid NAS destination", logging.KeyError, err.Error())
|
||||
return nil, fmt.Errorf("invalid NAS destination: %w", err)
|
||||
}
|
||||
nas := nas2.NewNASService(nasConfig)
|
||||
|
||||
connectCtx, cancelConnect := context.WithTimeout(context.Background(), cfg.NAS.Timeout)
|
||||
defer cancelConnect()
|
||||
if err := nas.Connect(connectCtx); err != nil {
|
||||
logging.MustGetFactory().Transfer.Error("failed to connect to NAS", logging.KeyError, err.Error())
|
||||
return nil, fmt.Errorf("failed to connect to NAS: %w", err)
|
||||
if err := nas.TestConnection(); err != nil {
|
||||
return nil, fmt.Errorf("Failed to connect to NAS: %w", err)
|
||||
}
|
||||
logging.MustGetFactory().Transfer.Info("connected to NAS",
|
||||
"path", nas.LocalPath(), "configured", outputDir, "how", strings.Join(nas.ConnectionInfo(), "; "))
|
||||
health.MustGetRegistry().RecordNASConnectivity(true)
|
||||
|
||||
cleanupConfig := CleanupConfig{
|
||||
Enabled: cfg.Cleanup.AfterTransfer,
|
||||
RetentionPeriod: time.Duration(cfg.Cleanup.RetainHours) * time.Hour,
|
||||
BatchSize: cfg.Cleanup.BatchSize,
|
||||
CheckInterval: cfg.Transfer.FileSettlingDelay,
|
||||
Enabled: constants.CleanupAfterTransfer,
|
||||
RetentionPeriod: time.Duration(constants.RetainLocalHours) * time.Hour,
|
||||
BatchSize: constants.CleanupBatchSize,
|
||||
CheckInterval: 1 * time.Minute,
|
||||
}
|
||||
cleanup := NewCleanupService(cleanupConfig)
|
||||
|
||||
queueConfig := QueueConfig{
|
||||
WorkerCount: cfg.Transfer.WorkerCount,
|
||||
PersistencePath: cfg.Paths.PersistenceFile,
|
||||
MaxQueueSize: cfg.Transfer.QueueSize,
|
||||
BatchSize: cfg.Transfer.BatchSize,
|
||||
RetryLimit: cfg.Transfer.RetryLimit,
|
||||
WorkerCount: constants.TransferWorkerCount,
|
||||
PersistencePath: constants.PersistencePath,
|
||||
MaxQueueSize: constants.TransferQueueSize,
|
||||
BatchSize: constants.BatchSize,
|
||||
}
|
||||
queue := NewTransferQueue(queueConfig, nas, cleanup)
|
||||
|
||||
// Create local output directory if it doesn't exist
|
||||
localOutputPath := cfg.GetEventPath(eventName)
|
||||
if err := utils.EnsureDir(localOutputPath); err != nil {
|
||||
return nil, fmt.Errorf("failed to create local output directory: %w", err)
|
||||
localOutputPath := constants.LocalOutputDirPath + "/" + eventName
|
||||
if err := os.MkdirAll(localOutputPath, 0755); err != nil {
|
||||
return nil, fmt.Errorf("Failed to create local output directory: %w", err)
|
||||
}
|
||||
|
||||
watcher, err := NewFileWatcher(localOutputPath, eventName, queue, cfg.Transfer.FileSettlingDelay)
|
||||
watcher, err := NewFileWatcher(localOutputPath, queue)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to create file watcher: %w", err)
|
||||
return nil, fmt.Errorf("Failed to create file watcher: %w", err)
|
||||
}
|
||||
|
||||
health.MustGetRegistry().AttachTransfer(queue.stats, queue.GetQueueSize, cleanup.GetPendingCount, nas.IsConnected)
|
||||
|
||||
return &TransferService{
|
||||
watcher: watcher,
|
||||
queue: queue,
|
||||
@ -126,7 +106,6 @@ func (ts *TransferService) Start(ctx context.Context) error {
|
||||
}()
|
||||
|
||||
log.Println("Transfer service started")
|
||||
logging.MustGetFactory().Transfer.Info("transfer service started")
|
||||
wg.Wait()
|
||||
|
||||
return nil
|
||||
@ -146,9 +125,6 @@ func (ts *TransferService) reportStats(ctx context.Context) {
|
||||
cleanupPending := ts.cleanup.GetPendingCount()
|
||||
|
||||
log.Printf("Transfer Stats: Added: %d, Completed: %d, Failed: %d, Pending: %d, Bytes: %d, Queue Size: %d, Cleanup Pending: %d", added, completed, failed, pending, bytes, queueSize, cleanupPending)
|
||||
logging.MustGetFactory().Transfer.Info("transfer stats",
|
||||
"added", added, "completed", completed, "failed", failed, "pending", pending,
|
||||
logging.KeyBytes, bytes, "queue_size", queueSize, "cleanup_pending", cleanupPending)
|
||||
}
|
||||
}
|
||||
}
|
||||
@ -165,21 +141,17 @@ func (ts *TransferService) Shutdown(ctx context.Context) error {
|
||||
}
|
||||
|
||||
// Disconnect from NAS
|
||||
if err := ts.nas.Disconnect(ctx); err != nil {
|
||||
if err := ts.nas.Disconnect(); err != nil {
|
||||
log.Printf("Warning: failed to disconnect from NAS: %v", err)
|
||||
logging.MustGetFactory().Transfer.Warn("failed to disconnect from NAS", logging.KeyError, err.Error())
|
||||
}
|
||||
health.MustGetRegistry().RecordNASConnectivity(false)
|
||||
|
||||
log.Println("Transfer service shut down")
|
||||
logging.MustGetFactory().Transfer.Info("transfer service shut down")
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// QueueExistingFiles scans a directory for .ts files and queues them for transfer
|
||||
func (ts *TransferService) QueueExistingFiles(localEventPath string) error {
|
||||
cfg := constants.MustGetConfig()
|
||||
log.Printf("Scanning for existing files in: %s", localEventPath)
|
||||
|
||||
var fileCount, alreadyTransferred, scheduledForCleanup int
|
||||
@ -202,8 +174,6 @@ func (ts *TransferService) QueueExistingFiles(localEventPath string) error {
|
||||
relPath, err := filepath.Rel(localEventPath, path)
|
||||
if err != nil {
|
||||
log.Printf("Failed to get relative path for %s: %v", path, err)
|
||||
logging.MustGetFactory().Transfer.Warn("failed to get relative path", logging.KeyFile, path, logging.KeyError, err.Error())
|
||||
health.MustGetRegistry().RecordTransferError(path, err)
|
||||
return nil
|
||||
}
|
||||
|
||||
@ -220,7 +190,7 @@ func (ts *TransferService) QueueExistingFiles(localEventPath string) error {
|
||||
alreadyTransferred++
|
||||
|
||||
// Schedule for cleanup if cleanup is enabled
|
||||
if cfg.Cleanup.AfterTransfer {
|
||||
if constants.CleanupAfterTransfer {
|
||||
if err := ts.cleanup.ScheduleCleanup(path); err != nil {
|
||||
log.Printf("Failed to schedule cleanup for already-transferred file %s: %v", path, err)
|
||||
} else {
|
||||
@ -244,8 +214,6 @@ func (ts *TransferService) QueueExistingFiles(localEventPath string) error {
|
||||
// Add to queue
|
||||
if err := ts.queue.Add(item); err != nil {
|
||||
log.Printf("Failed to queue file %s: %v", path, err)
|
||||
logging.MustGetFactory().Transfer.Error("failed to queue file", logging.KeyFile, path, logging.KeyError, err.Error())
|
||||
health.MustGetRegistry().RecordTransferError(path, err)
|
||||
} else {
|
||||
log.Printf("Queued file: %s (%s, %d bytes)", path, resolution, info.Size())
|
||||
fileCount++
|
||||
|
||||
@ -49,9 +49,6 @@ type QueueConfig struct {
|
||||
PersistencePath string
|
||||
MaxQueueSize int
|
||||
BatchSize int
|
||||
// RetryLimit is the total number of attempts made per file before it is
|
||||
// marked failed.
|
||||
RetryLimit int
|
||||
}
|
||||
|
||||
type CleanupConfig struct {
|
||||
|
||||
@ -4,8 +4,7 @@ import (
|
||||
"context"
|
||||
"fmt"
|
||||
"log"
|
||||
"m3u8-downloader/pkg/health"
|
||||
"m3u8-downloader/pkg/logging"
|
||||
"m3u8-downloader/pkg/constants"
|
||||
"math/rand"
|
||||
"os"
|
||||
"path/filepath"
|
||||
@ -17,12 +16,7 @@ import (
|
||||
)
|
||||
|
||||
type FileWatcher struct {
|
||||
outputDir string
|
||||
// eventName namespaces the NAS destination. It is passed in rather than
|
||||
// derived from outputDir so that an unnamed event resolves to the NAS root
|
||||
// — matching where the downloader put the files — instead of picking up
|
||||
// the local output directory's own name.
|
||||
eventName string
|
||||
outputDir string
|
||||
queue *TransferQueue
|
||||
watcher *fsnotify.Watcher
|
||||
settingDelay time.Duration
|
||||
@ -30,17 +24,16 @@ type FileWatcher struct {
|
||||
mu sync.Mutex
|
||||
}
|
||||
|
||||
func NewFileWatcher(outputDir, eventName string, queue *TransferQueue, settlingDelay time.Duration) (*FileWatcher, error) {
|
||||
func NewFileWatcher(outputDir string, queue *TransferQueue) (*FileWatcher, error) {
|
||||
watcher, err := fsnotify.NewWatcher()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &FileWatcher{
|
||||
outputDir: outputDir,
|
||||
eventName: eventName,
|
||||
queue: queue,
|
||||
watcher: watcher,
|
||||
settingDelay: settlingDelay,
|
||||
settingDelay: constants.FileSettlingDelay,
|
||||
pendingFiles: make(map[string]*time.Timer),
|
||||
}, nil
|
||||
}
|
||||
@ -53,13 +46,11 @@ func (fw *FileWatcher) Start(ctx context.Context) error {
|
||||
}
|
||||
|
||||
log.Printf("Starting file watcher on %s", fw.outputDir)
|
||||
logging.MustGetFactory().Transfer.Info("file watcher started", "dir", fw.outputDir)
|
||||
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
log.Println("File watcher shutting down...")
|
||||
logging.MustGetFactory().Transfer.Info("file watcher shutting down")
|
||||
return ctx.Err()
|
||||
|
||||
case event, ok := <-fw.watcher.Events:
|
||||
@ -73,7 +64,6 @@ func (fw *FileWatcher) Start(ctx context.Context) error {
|
||||
return fmt.Errorf("Watcher errors channel closed")
|
||||
}
|
||||
log.Printf("Watcher error: %v", err)
|
||||
logging.MustGetFactory().Transfer.Error("watcher error", logging.KeyError, err.Error())
|
||||
}
|
||||
}
|
||||
}
|
||||
@ -98,18 +88,6 @@ func (fw *FileWatcher) addWatchRecursive(root string) error {
|
||||
}
|
||||
|
||||
func (fw *FileWatcher) handleFileEvent(event fsnotify.Event) {
|
||||
// Directories must be handled before the .ts filter below. fsnotify is not
|
||||
// recursive, so a resolution directory created after the watcher starts —
|
||||
// which is the normal case, since the transfer service starts before the
|
||||
// downloader creates 1080p/ and friends — has to be picked up here or
|
||||
// nothing inside it is ever seen.
|
||||
if event.Op&fsnotify.Create == fsnotify.Create {
|
||||
if info, err := os.Stat(event.Name); err == nil && info.IsDir() {
|
||||
fw.watchNewDirectory(event.Name)
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
if !strings.HasSuffix(strings.ToLower(event.Name), ".ts") {
|
||||
return
|
||||
}
|
||||
@ -122,33 +100,15 @@ func (fw *FileWatcher) handleFileEvent(event fsnotify.Event) {
|
||||
case event.Op&fsnotify.Remove == fsnotify.Remove:
|
||||
fw.cancelPendingTransfer(event.Name)
|
||||
}
|
||||
}
|
||||
|
||||
// watchNewDirectory starts watching a directory that appeared after startup,
|
||||
// and sweeps whatever is already inside it.
|
||||
//
|
||||
// The sweep is not redundant: the downloader creates a resolution directory and
|
||||
// writes segments into it immediately, so files can land between the mkdir and
|
||||
// the watch being registered. Those writes generate no event we would see, and
|
||||
// without the sweep the first segments of every recording would be lost.
|
||||
func (fw *FileWatcher) watchNewDirectory(dir string) {
|
||||
if err := fw.addWatchRecursive(dir); err != nil {
|
||||
log.Printf("Failed to watch new directory %s: %v", dir, err)
|
||||
logging.MustGetFactory().Transfer.Warn("failed to watch new directory", "dir", dir, logging.KeyError, err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
err := filepath.Walk(dir, func(path string, info os.FileInfo, err error) error {
|
||||
if err != nil {
|
||||
return nil // keep sweeping the rest
|
||||
if event.Op&fsnotify.Create == fsnotify.Create {
|
||||
if info, err := os.Stat(event.Name); err == nil && info.IsDir() {
|
||||
if err := fw.watcher.Add(event.Name); err != nil {
|
||||
log.Printf("Failed to watch directory %s: %v", event.Name, err)
|
||||
} else {
|
||||
log.Printf("Watching directory %s", event.Name)
|
||||
}
|
||||
}
|
||||
if !info.IsDir() && strings.HasSuffix(strings.ToLower(info.Name()), ".ts") {
|
||||
fw.scheduleTransfer(path)
|
||||
}
|
||||
return nil
|
||||
})
|
||||
if err != nil {
|
||||
log.Printf("Failed to sweep new directory %s: %v", dir, err)
|
||||
}
|
||||
}
|
||||
|
||||
@ -185,8 +145,6 @@ func (fw *FileWatcher) processFile(filePath string) {
|
||||
info, err := os.Stat(filePath)
|
||||
if err != nil {
|
||||
log.Printf("Failed to stat file %s: %v", filePath, err)
|
||||
logging.MustGetFactory().Transfer.Warn("failed to stat file", logging.KeyFile, filePath, logging.KeyError, err.Error())
|
||||
health.MustGetRegistry().RecordTransferError(filePath, err)
|
||||
return
|
||||
}
|
||||
|
||||
@ -195,18 +153,13 @@ func (fw *FileWatcher) processFile(filePath string) {
|
||||
relPath, err := filepath.Rel(fw.outputDir, filePath)
|
||||
if err != nil {
|
||||
log.Printf("Failed to get relative path for file %s: %v", filePath, err)
|
||||
logging.MustGetFactory().Transfer.Warn("failed to get relative path", logging.KeyFile, filePath, logging.KeyError, err.Error())
|
||||
health.MustGetRegistry().RecordTransferError(filePath, err)
|
||||
return
|
||||
}
|
||||
|
||||
// The destination is namespaced by event, mirroring the local layout and
|
||||
// matching what QueueExistingFiles produces for the same file — otherwise
|
||||
// live transfers and catch-up transfers would write to different places.
|
||||
item := TransferItem{
|
||||
ID: generateID(),
|
||||
SourcePath: filePath,
|
||||
DestinationPath: filepath.Join(fw.eventName, relPath),
|
||||
DestinationPath: relPath,
|
||||
Resolution: resolution,
|
||||
Timestamp: time.Now(),
|
||||
Status: StatusPending,
|
||||
@ -215,11 +168,8 @@ func (fw *FileWatcher) processFile(filePath string) {
|
||||
|
||||
if err := fw.queue.Add(item); err != nil {
|
||||
log.Printf("Failed to add file to queue: %v", err)
|
||||
logging.MustGetFactory().Transfer.Error("failed to add file to queue", logging.KeyFile, filePath, logging.KeyError, err.Error())
|
||||
health.MustGetRegistry().RecordTransferError(filePath, err)
|
||||
} else {
|
||||
log.Printf("Added file to queue: %s", filePath)
|
||||
logging.MustGetFactory().Transfer.Debug("file queued", logging.KeyFile, filePath)
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@ -1,223 +0,0 @@
|
||||
package transfer
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// settlingDelay is short so tests exercise the real timer path without
|
||||
// waiting the production five seconds.
|
||||
const testSettlingDelay = 40 * time.Millisecond
|
||||
|
||||
// newTestQueue builds a queue that accepts items but never transfers them:
|
||||
// ProcessQueue is not started, so Add is a pure recording of intent.
|
||||
func newTestQueue(t *testing.T) *TransferQueue {
|
||||
t.Helper()
|
||||
return NewTransferQueue(QueueConfig{
|
||||
WorkerCount: 1,
|
||||
PersistencePath: filepath.Join(t.TempDir(), "queue.json"),
|
||||
MaxQueueSize: 100,
|
||||
BatchSize: 10,
|
||||
}, nil, nil)
|
||||
}
|
||||
|
||||
// queuedDestinations returns the destination paths currently queued, sorted.
|
||||
func queuedDestinations(q *TransferQueue) []string {
|
||||
q.mu.RLock()
|
||||
defer q.mu.RUnlock()
|
||||
|
||||
out := make([]string, 0, q.items.Len())
|
||||
for _, item := range *q.items {
|
||||
out = append(out, filepath.ToSlash(item.DestinationPath))
|
||||
}
|
||||
sort.Strings(out)
|
||||
return out
|
||||
}
|
||||
|
||||
// waitForQueue polls until the queue holds want items or the deadline passes.
|
||||
// Polling beats a fixed sleep: it keeps the test fast when the watcher is
|
||||
// prompt and tolerant when a loaded CI machine is not.
|
||||
func waitForQueue(t *testing.T, q *TransferQueue, want int) []string {
|
||||
t.Helper()
|
||||
|
||||
deadline := time.Now().Add(3 * time.Second)
|
||||
var got []string
|
||||
for time.Now().Before(deadline) {
|
||||
got = queuedDestinations(q)
|
||||
if len(got) >= want {
|
||||
return got
|
||||
}
|
||||
time.Sleep(10 * time.Millisecond)
|
||||
}
|
||||
return got
|
||||
}
|
||||
|
||||
func startWatcher(t *testing.T, dir, eventName string, q *TransferQueue) {
|
||||
t.Helper()
|
||||
|
||||
fw, err := NewFileWatcher(dir, eventName, q, testSettlingDelay)
|
||||
if err != nil {
|
||||
t.Fatalf("NewFileWatcher failed: %v", err)
|
||||
}
|
||||
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
t.Cleanup(cancel)
|
||||
|
||||
started := make(chan struct{})
|
||||
go func() {
|
||||
close(started)
|
||||
_ = fw.Start(ctx)
|
||||
}()
|
||||
<-started
|
||||
|
||||
// Give Start a moment to register its initial watches before the test
|
||||
// begins creating files under them.
|
||||
time.Sleep(100 * time.Millisecond)
|
||||
}
|
||||
|
||||
// TestWatcherPicksUpDirectoriesCreatedAfterStart covers the normal recording
|
||||
// sequence: the transfer service starts watching the event directory, and only
|
||||
// then does the downloader create the per-resolution directories inside it.
|
||||
// fsnotify is not recursive, so those directories have to be picked up from
|
||||
// the create event or nothing inside them is ever transferred.
|
||||
func TestWatcherPicksUpDirectoriesCreatedAfterStart(t *testing.T) {
|
||||
eventDir := t.TempDir()
|
||||
queue := newTestQueue(t)
|
||||
startWatcher(t, eventDir, "2026-regionals", queue)
|
||||
|
||||
resolutionDir := filepath.Join(eventDir, "1080p")
|
||||
if err := os.MkdirAll(resolutionDir, 0755); err != nil {
|
||||
t.Fatalf("failed to create resolution dir: %v", err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(resolutionDir, "seg_1001.ts"), []byte("payload"), 0644); err != nil {
|
||||
t.Fatalf("failed to write segment: %v", err)
|
||||
}
|
||||
|
||||
got := waitForQueue(t, queue, 1)
|
||||
if len(got) != 1 {
|
||||
t.Fatalf("segment written to a new resolution directory was never queued, queue = %v", got)
|
||||
}
|
||||
if want := "2026-regionals/1080p/seg_1001.ts"; got[0] != want {
|
||||
t.Errorf("destination = %q, want %q", got[0], want)
|
||||
}
|
||||
}
|
||||
|
||||
// TestWatcherSweepsFilesRacingDirectoryCreation covers the tighter race where
|
||||
// segments are written immediately after the mkdir, before the new watch can
|
||||
// be registered. Those writes produce no event the watcher would receive, so
|
||||
// the directory has to be swept when it is first seen.
|
||||
func TestWatcherSweepsFilesRacingDirectoryCreation(t *testing.T) {
|
||||
eventDir := t.TempDir()
|
||||
queue := newTestQueue(t)
|
||||
startWatcher(t, eventDir, "event", queue)
|
||||
|
||||
// Stage the tree elsewhere and move it into place, so the files are
|
||||
// guaranteed to already exist the instant the directory appears.
|
||||
staging := filepath.Join(t.TempDir(), "1080p")
|
||||
if err := os.MkdirAll(staging, 0755); err != nil {
|
||||
t.Fatalf("failed to stage: %v", err)
|
||||
}
|
||||
for _, name := range []string{"seg_1001.ts", "seg_1002.ts"} {
|
||||
if err := os.WriteFile(filepath.Join(staging, name), []byte("payload"), 0644); err != nil {
|
||||
t.Fatalf("failed to stage segment: %v", err)
|
||||
}
|
||||
}
|
||||
if err := os.Rename(staging, filepath.Join(eventDir, "1080p")); err != nil {
|
||||
t.Fatalf("failed to move staged directory into place: %v", err)
|
||||
}
|
||||
|
||||
got := waitForQueue(t, queue, 2)
|
||||
want := []string{"event/1080p/seg_1001.ts", "event/1080p/seg_1002.ts"}
|
||||
if len(got) != 2 || got[0] != want[0] || got[1] != want[1] {
|
||||
t.Errorf("queued %v, want %v", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
// TestWatcherDestinationIncludesEventName pins the layout agreement between
|
||||
// live transfers and the catch-up path in QueueExistingFiles: both must write
|
||||
// to <nas>/<event>/<resolution>/<file>.
|
||||
func TestWatcherDestinationIncludesEventName(t *testing.T) {
|
||||
eventDir := t.TempDir()
|
||||
resolutionDir := filepath.Join(eventDir, "720p")
|
||||
if err := os.MkdirAll(resolutionDir, 0755); err != nil {
|
||||
t.Fatalf("setup failed: %v", err)
|
||||
}
|
||||
|
||||
queue := newTestQueue(t)
|
||||
startWatcher(t, eventDir, "my-event", queue)
|
||||
|
||||
if err := os.WriteFile(filepath.Join(resolutionDir, "seg_1001.ts"), []byte("payload"), 0644); err != nil {
|
||||
t.Fatalf("failed to write segment: %v", err)
|
||||
}
|
||||
|
||||
got := waitForQueue(t, queue, 1)
|
||||
if len(got) != 1 {
|
||||
t.Fatalf("expected one queued item, got %v", got)
|
||||
}
|
||||
if want := "my-event/720p/seg_1001.ts"; got[0] != want {
|
||||
t.Errorf("destination = %q, want %q", got[0], want)
|
||||
}
|
||||
}
|
||||
|
||||
// TestWatcherWithoutEventNameUsesRoot mirrors a download run with no -event:
|
||||
// segments land under the local output root, so the NAS destination must not
|
||||
// invent a directory level for them.
|
||||
func TestWatcherWithoutEventNameUsesRoot(t *testing.T) {
|
||||
eventDir := t.TempDir()
|
||||
resolutionDir := filepath.Join(eventDir, "1080p")
|
||||
if err := os.MkdirAll(resolutionDir, 0755); err != nil {
|
||||
t.Fatalf("setup failed: %v", err)
|
||||
}
|
||||
|
||||
queue := newTestQueue(t)
|
||||
startWatcher(t, eventDir, "", queue)
|
||||
|
||||
if err := os.WriteFile(filepath.Join(resolutionDir, "seg_1001.ts"), []byte("payload"), 0644); err != nil {
|
||||
t.Fatalf("failed to write segment: %v", err)
|
||||
}
|
||||
|
||||
got := waitForQueue(t, queue, 1)
|
||||
if len(got) != 1 {
|
||||
t.Fatalf("expected one queued item, got %v", got)
|
||||
}
|
||||
if want := "1080p/seg_1001.ts"; got[0] != want {
|
||||
t.Errorf("destination = %q, want %q", got[0], want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestWatcherIgnoresNonSegmentFiles(t *testing.T) {
|
||||
eventDir := t.TempDir()
|
||||
queue := newTestQueue(t)
|
||||
startWatcher(t, eventDir, "event", queue)
|
||||
|
||||
for _, name := range []string{"notes.txt", "manifest.json", "segment.ts.part"} {
|
||||
if err := os.WriteFile(filepath.Join(eventDir, name), []byte("x"), 0644); err != nil {
|
||||
t.Fatalf("failed to write %s: %v", name, err)
|
||||
}
|
||||
}
|
||||
|
||||
time.Sleep(testSettlingDelay + 300*time.Millisecond)
|
||||
if got := queuedDestinations(queue); len(got) != 0 {
|
||||
t.Errorf("non-segment files should not be queued, got %v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExtractResolution(t *testing.T) {
|
||||
fw := &FileWatcher{}
|
||||
|
||||
cases := map[string]string{
|
||||
filepath.Join("data", "event", "1080p", "seg.ts"): "1080p",
|
||||
filepath.Join("data", "event", "720p", "seg.ts"): "720p",
|
||||
filepath.Join("data", "event", "seg.ts"): "",
|
||||
}
|
||||
|
||||
for path, want := range cases {
|
||||
if got := fw.extractResolution(path); got != want {
|
||||
t.Errorf("extractResolution(%q) = %q, want %q", path, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
@ -1,85 +0,0 @@
|
||||
package utils
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
)
|
||||
|
||||
func SafeJoin(base string, elements ...string) string {
|
||||
path := filepath.Join(append([]string{base}, elements...)...)
|
||||
return filepath.Clean(path)
|
||||
}
|
||||
|
||||
func EnsureDir(path string) error {
|
||||
if err := os.MkdirAll(path, 0755); err != nil {
|
||||
return fmt.Errorf("failed to create directory %s: %w", path, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func PathExists(path string) bool {
|
||||
_, err := os.Stat(path)
|
||||
return !os.IsNotExist(err)
|
||||
}
|
||||
|
||||
// IsValidPath reports whether path is usable on Windows, the stricter of the
|
||||
// two platforms this project targets, which reserves <>:"|?* in names.
|
||||
//
|
||||
// A leading drive letter is checked separately: its colon is legal there and
|
||||
// nowhere else in the path. Without that carve-out the function rejects every
|
||||
// absolute Windows path, including the drive-qualified ones it exists to
|
||||
// validate.
|
||||
func IsValidPath(path string) bool {
|
||||
if path == "" {
|
||||
return false
|
||||
}
|
||||
|
||||
rest := path
|
||||
if len(rest) >= 2 && rest[1] == ':' && isDriveLetter(rest[0]) {
|
||||
rest = rest[2:]
|
||||
}
|
||||
|
||||
return !strings.ContainsAny(rest, `<>:"|?*`)
|
||||
}
|
||||
|
||||
func isDriveLetter(c byte) bool {
|
||||
return (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z')
|
||||
}
|
||||
|
||||
func NormalizePath(path string) string {
|
||||
return filepath.Clean(strings.ReplaceAll(path, "\\", string(filepath.Separator)))
|
||||
}
|
||||
|
||||
func GetRelativePath(basePath, targetPath string) (string, error) {
|
||||
// filepath.Rel("", "") succeeds and returns ".", which is a meaningless
|
||||
// answer to a meaningless question; callers passing an empty path have a
|
||||
// bug worth surfacing rather than papering over.
|
||||
if basePath == "" || targetPath == "" {
|
||||
return "", fmt.Errorf("both base and target paths are required (base %q, target %q)", basePath, targetPath)
|
||||
}
|
||||
|
||||
rel, err := filepath.Rel(basePath, targetPath)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("failed to get relative path: %w", err)
|
||||
}
|
||||
return rel, nil
|
||||
}
|
||||
|
||||
func ValidateWritablePath(path string) error {
|
||||
dir := filepath.Dir(path)
|
||||
if err := EnsureDir(dir); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
testFile := filepath.Join(dir, ".write_test")
|
||||
file, err := os.Create(testFile)
|
||||
if err != nil {
|
||||
return fmt.Errorf("path %s is not writable: %w", dir, err)
|
||||
}
|
||||
file.Close()
|
||||
os.Remove(testFile)
|
||||
|
||||
return nil
|
||||
}
|
||||
@ -1,235 +0,0 @@
|
||||
package utils
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"runtime"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestSafeJoin(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
base string
|
||||
elements []string
|
||||
want string
|
||||
}{
|
||||
{
|
||||
name: "basic join",
|
||||
base: "data",
|
||||
elements: []string{"events", "test-event"},
|
||||
want: filepath.Join("data", "events", "test-event"),
|
||||
},
|
||||
{
|
||||
name: "empty elements",
|
||||
base: "data",
|
||||
elements: []string{},
|
||||
want: "data",
|
||||
},
|
||||
{
|
||||
name: "with path separators",
|
||||
base: "data/events",
|
||||
elements: []string{"test-event", "1080p"},
|
||||
want: filepath.Join("data", "events", "test-event", "1080p"),
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
got := SafeJoin(tt.base, tt.elements...)
|
||||
if got != tt.want {
|
||||
t.Errorf("SafeJoin() = %v, want %v", got, tt.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestEnsureDir(t *testing.T) {
|
||||
tempDir, err := os.MkdirTemp("", "utils_test_*")
|
||||
if err != nil {
|
||||
t.Fatalf("Failed to create temp dir: %v", err)
|
||||
}
|
||||
defer os.RemoveAll(tempDir)
|
||||
|
||||
testPath := filepath.Join(tempDir, "test", "nested", "directory")
|
||||
|
||||
// Test creating nested directories
|
||||
err = EnsureDir(testPath)
|
||||
if err != nil {
|
||||
t.Errorf("EnsureDir() failed: %v", err)
|
||||
}
|
||||
|
||||
// Verify directory was created
|
||||
if _, err := os.Stat(testPath); os.IsNotExist(err) {
|
||||
t.Errorf("Directory was not created: %s", testPath)
|
||||
}
|
||||
|
||||
// Test with existing directory (should not fail)
|
||||
err = EnsureDir(testPath)
|
||||
if err != nil {
|
||||
t.Errorf("EnsureDir() failed on existing directory: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPathExists(t *testing.T) {
|
||||
tempDir, err := os.MkdirTemp("", "utils_test_*")
|
||||
if err != nil {
|
||||
t.Fatalf("Failed to create temp dir: %v", err)
|
||||
}
|
||||
defer os.RemoveAll(tempDir)
|
||||
|
||||
// Test existing path
|
||||
if !PathExists(tempDir) {
|
||||
t.Errorf("PathExists() should return true for existing path: %s", tempDir)
|
||||
}
|
||||
|
||||
// Test non-existing path
|
||||
nonExistentPath := filepath.Join(tempDir, "does-not-exist")
|
||||
if PathExists(nonExistentPath) {
|
||||
t.Errorf("PathExists() should return false for non-existent path: %s", nonExistentPath)
|
||||
}
|
||||
|
||||
// Test with file
|
||||
testFile := filepath.Join(tempDir, "test.txt")
|
||||
f, err := os.Create(testFile)
|
||||
if err != nil {
|
||||
t.Fatalf("Failed to create test file: %v", err)
|
||||
}
|
||||
f.Close()
|
||||
|
||||
if !PathExists(testFile) {
|
||||
t.Errorf("PathExists() should return true for existing file: %s", testFile)
|
||||
}
|
||||
}
|
||||
|
||||
func TestIsValidPath(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
path string
|
||||
want bool
|
||||
}{
|
||||
{"empty path", "", false},
|
||||
{"valid path", "data/events/test", true},
|
||||
{"path with colon", "data:events", false},
|
||||
{"path with pipe", "data|events", false},
|
||||
{"path with question mark", "data?events", false},
|
||||
{"path with asterisk", "data*events", false},
|
||||
{"path with quotes", "data\"events", false},
|
||||
{"path with angle brackets", "data<events>", false},
|
||||
{"normal windows path", "C:\\data\\events", true}, // Windows path separators are actually OK
|
||||
{"unix path", "/home/user/data", true},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
got := IsValidPath(tt.path)
|
||||
if got != tt.want {
|
||||
t.Errorf("IsValidPath(%q) = %v, want %v", tt.path, got, tt.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestNormalizePath(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
path string
|
||||
want string
|
||||
}{
|
||||
{
|
||||
name: "windows backslashes",
|
||||
path: "data\\events\\test",
|
||||
want: filepath.Join("data", "events", "test"),
|
||||
},
|
||||
{
|
||||
name: "unix forward slashes",
|
||||
path: "data/events/test",
|
||||
want: filepath.Join("data", "events", "test"),
|
||||
},
|
||||
{
|
||||
name: "mixed slashes",
|
||||
path: "data\\events/test\\file",
|
||||
want: filepath.Join("data", "events", "test", "file"),
|
||||
},
|
||||
{
|
||||
name: "redundant separators",
|
||||
path: "data//events\\\\test",
|
||||
want: filepath.Join("data", "events", "test"),
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
got := NormalizePath(tt.path)
|
||||
if got != tt.want {
|
||||
t.Errorf("NormalizePath(%q) = %q, want %q", tt.path, got, tt.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestGetRelativePath(t *testing.T) {
|
||||
tempDir, err := os.MkdirTemp("", "utils_test_*")
|
||||
if err != nil {
|
||||
t.Fatalf("Failed to create temp dir: %v", err)
|
||||
}
|
||||
defer os.RemoveAll(tempDir)
|
||||
|
||||
basePath := tempDir
|
||||
targetPath := filepath.Join(tempDir, "subdir", "file.txt")
|
||||
|
||||
rel, err := GetRelativePath(basePath, targetPath)
|
||||
if err != nil {
|
||||
t.Errorf("GetRelativePath() failed: %v", err)
|
||||
}
|
||||
|
||||
expected := filepath.Join("subdir", "file.txt")
|
||||
if rel != expected {
|
||||
t.Errorf("GetRelativePath() = %q, want %q", rel, expected)
|
||||
}
|
||||
|
||||
// Test with invalid paths
|
||||
_, err = GetRelativePath("", "")
|
||||
if err == nil {
|
||||
t.Error("GetRelativePath() should fail with empty paths")
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidateWritablePath(t *testing.T) {
|
||||
tempDir, err := os.MkdirTemp("", "utils_test_*")
|
||||
if err != nil {
|
||||
t.Fatalf("Failed to create temp dir: %v", err)
|
||||
}
|
||||
defer os.RemoveAll(tempDir)
|
||||
|
||||
// Test writable path
|
||||
writablePath := filepath.Join(tempDir, "test", "file.txt")
|
||||
err = ValidateWritablePath(writablePath)
|
||||
if err != nil {
|
||||
t.Errorf("ValidateWritablePath() failed for writable path: %v", err)
|
||||
}
|
||||
|
||||
// Verify directory was created
|
||||
dir := filepath.Dir(writablePath)
|
||||
if _, err := os.Stat(dir); os.IsNotExist(err) {
|
||||
t.Errorf("Directory should have been created: %s", dir)
|
||||
}
|
||||
|
||||
// Test with read-only directory (if supported by OS)
|
||||
if runtime.GOOS != "windows" { // Skip on Windows as it's more complex
|
||||
readOnlyDir := filepath.Join(tempDir, "readonly")
|
||||
os.MkdirAll(readOnlyDir, 0755)
|
||||
os.Chmod(readOnlyDir, 0444) // Read-only
|
||||
defer os.Chmod(readOnlyDir, 0755) // Restore permissions for cleanup
|
||||
|
||||
readOnlyPath := filepath.Join(readOnlyDir, "file.txt")
|
||||
err = ValidateWritablePath(readOnlyPath)
|
||||
if err == nil {
|
||||
t.Error("ValidateWritablePath() should fail for read-only directory")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "not writable") {
|
||||
t.Errorf("Expected 'not writable' error, got: %v", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
146
test_runner.go
146
test_runner.go
@ -1,146 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
func main() {
|
||||
fmt.Println("🧪 StreamRecorder Test Suite")
|
||||
fmt.Println("============================")
|
||||
|
||||
startTime := time.Now()
|
||||
|
||||
// Set test environment to avoid interfering with real data
|
||||
originalEnv := setupTestEnvironment()
|
||||
defer restoreEnvironment(originalEnv)
|
||||
|
||||
// Create temporary directory for test data
|
||||
tempDir, err := os.MkdirTemp("", "streamrecorder_test_*")
|
||||
if err != nil {
|
||||
fmt.Printf("❌ Failed to create temp directory: %v\n", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
defer func() {
|
||||
fmt.Printf("🧹 Cleaning up test directory: %s\n", tempDir)
|
||||
os.RemoveAll(tempDir)
|
||||
}()
|
||||
|
||||
// Set test-specific environment variables
|
||||
os.Setenv("LOCAL_OUTPUT_DIR", filepath.Join(tempDir, "data"))
|
||||
os.Setenv("PROCESS_OUTPUT_DIR", filepath.Join(tempDir, "out"))
|
||||
os.Setenv("ENABLE_NAS_TRANSFER", "false") // Disable NAS for tests
|
||||
os.Setenv("PROCESSING_ENABLED", "false") // Disable processing that needs FFmpeg
|
||||
|
||||
fmt.Printf("📁 Using temporary directory: %s\n", tempDir)
|
||||
fmt.Println()
|
||||
|
||||
// Run tests for each package
|
||||
packages := []string{
|
||||
"./pkg/config",
|
||||
"./pkg/utils",
|
||||
"./pkg/constants",
|
||||
"./pkg/httpClient",
|
||||
"./pkg/media",
|
||||
"./pkg/processing",
|
||||
}
|
||||
|
||||
var failedPackages []string
|
||||
totalTests := 0
|
||||
passedTests := 0
|
||||
|
||||
for _, pkg := range packages {
|
||||
fmt.Printf("🔍 Testing package: %s\n", pkg)
|
||||
|
||||
cmd := exec.Command("go", "test", "-v", pkg)
|
||||
cmd.Dir = "."
|
||||
|
||||
output, err := cmd.CombinedOutput()
|
||||
outputStr := string(output)
|
||||
|
||||
// Count tests
|
||||
testCount := strings.Count(outputStr, "=== RUN")
|
||||
passCount := strings.Count(outputStr, "--- PASS:")
|
||||
|
||||
totalTests += testCount
|
||||
passedTests += passCount
|
||||
|
||||
if err != nil {
|
||||
fmt.Printf("❌ FAILED: %s (%d/%d tests passed)\n", pkg, passCount, testCount)
|
||||
failedPackages = append(failedPackages, pkg)
|
||||
|
||||
// Show failure details
|
||||
lines := strings.Split(outputStr, "\n")
|
||||
for _, line := range lines {
|
||||
if strings.Contains(line, "FAIL:") ||
|
||||
strings.Contains(line, "Error:") ||
|
||||
strings.Contains(line, "panic:") {
|
||||
fmt.Printf(" %s\n", line)
|
||||
}
|
||||
}
|
||||
} else {
|
||||
fmt.Printf("✅ PASSED: %s (%d tests)\n", pkg, testCount)
|
||||
}
|
||||
fmt.Println()
|
||||
}
|
||||
|
||||
// Print summary
|
||||
duration := time.Since(startTime)
|
||||
fmt.Println("📊 Test Summary")
|
||||
fmt.Println("===============")
|
||||
fmt.Printf("Total packages: %d\n", len(packages))
|
||||
fmt.Printf("Passed packages: %d\n", len(packages)-len(failedPackages))
|
||||
fmt.Printf("Failed packages: %d\n", len(failedPackages))
|
||||
fmt.Printf("Total tests: %d\n", totalTests)
|
||||
fmt.Printf("Passed tests: %d\n", passedTests)
|
||||
fmt.Printf("Failed tests: %d\n", totalTests-passedTests)
|
||||
fmt.Printf("Duration: %v\n", duration.Round(time.Millisecond))
|
||||
|
||||
if len(failedPackages) > 0 {
|
||||
fmt.Println()
|
||||
fmt.Println("❌ Failed packages:")
|
||||
for _, pkg := range failedPackages {
|
||||
fmt.Printf(" - %s\n", pkg)
|
||||
}
|
||||
os.Exit(1)
|
||||
} else {
|
||||
fmt.Println()
|
||||
fmt.Println("🎉 All tests passed!")
|
||||
}
|
||||
}
|
||||
|
||||
func setupTestEnvironment() map[string]string {
|
||||
// Save original environment variables that we'll modify
|
||||
originalEnv := make(map[string]string)
|
||||
|
||||
envVars := []string{
|
||||
"LOCAL_OUTPUT_DIR",
|
||||
"PROCESS_OUTPUT_DIR",
|
||||
"ENABLE_NAS_TRANSFER",
|
||||
"PROCESSING_ENABLED",
|
||||
"NAS_OUTPUT_PATH",
|
||||
"FFMPEG_PATH",
|
||||
"WORKER_COUNT",
|
||||
}
|
||||
|
||||
for _, envVar := range envVars {
|
||||
originalEnv[envVar] = os.Getenv(envVar)
|
||||
}
|
||||
|
||||
return originalEnv
|
||||
}
|
||||
|
||||
func restoreEnvironment(originalEnv map[string]string) {
|
||||
fmt.Println("🔄 Restoring original environment...")
|
||||
for envVar, originalValue := range originalEnv {
|
||||
if originalValue == "" {
|
||||
os.Unsetenv(envVar)
|
||||
} else {
|
||||
os.Setenv(envVar, originalValue)
|
||||
}
|
||||
}
|
||||
}
|
||||
Loading…
x
Reference in New Issue
Block a user