Compare commits
14 Commits
0f4a39f426
...
309eeef59a
| Author | SHA1 | Date | |
|---|---|---|---|
| 309eeef59a | |||
| ce2c5f3181 | |||
| 11b2ba8a8b | |||
|
|
7d67cec2b7 | ||
|
|
20df800715 | ||
|
|
3c49a3fa9f | ||
|
|
718d69c12a | ||
|
|
0a349cc406 | ||
| a10688279f | |||
| 4d73ce25c2 | |||
| 899ca31bb3 | |||
|
|
c9d6900d48 | ||
|
|
8223a3ac99 | ||
|
|
99594597db |
37
.gitea/workflows/build.yml
Normal file
37
.gitea/workflows/build.yml
Normal file
@ -0,0 +1,37 @@
|
|||||||
|
name: Build Go Binaries
|
||||||
|
on:
|
||||||
|
push:
|
||||||
|
branches:
|
||||||
|
- main
|
||||||
|
|
||||||
|
jobs:
|
||||||
|
build:
|
||||||
|
runs-on: docker
|
||||||
|
steps:
|
||||||
|
- name: Checkout
|
||||||
|
uses: actions/checkout@v2
|
||||||
|
|
||||||
|
- name: Set up Go
|
||||||
|
uses: actions/setup-go@v5
|
||||||
|
with:
|
||||||
|
go-version: 1.23
|
||||||
|
|
||||||
|
- name: Cross-Compile
|
||||||
|
run: |
|
||||||
|
mkdir -p dist
|
||||||
|
for GOOS in linux windows darwin; do
|
||||||
|
for GOARCH in arm64 amd64; do
|
||||||
|
OUTPUT="dist/myapp_${GOOS}_${GOARCH}"
|
||||||
|
echo "Building ${OUTPUT}"
|
||||||
|
if [ "$GOOS" = "windows" ]; then
|
||||||
|
OUTPUT="${OUTPUT}.exe"
|
||||||
|
fi
|
||||||
|
GOOS=$GOOS GOARCH=$GOARCH go build -o $OUTPUT ./cmd/main
|
||||||
|
done
|
||||||
|
done
|
||||||
|
|
||||||
|
- name: Upload Artifact
|
||||||
|
uses: actions/upload-artifact@v4
|
||||||
|
with:
|
||||||
|
name: flo_download-binaries
|
||||||
|
path: dist
|
||||||
217
CLAUDE.md
217
CLAUDE.md
@ -4,31 +4,140 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
|
|||||||
|
|
||||||
## Project Overview
|
## Project Overview
|
||||||
|
|
||||||
This is a Go-based M3U8 downloader that parses HLS (HTTP Live Streaming) playlists to extract video and audio stream metadata. The end goal of this project is to have a listening REST API take in m3u8 urls, parse them, and eventually send to a conversion service.
|
This is a Go-based HLS (HTTP Live Streaming) recorder that monitors M3U8 playlists and downloads video segments in real-time with automatic NAS transfer capabilities. The program takes a master M3U8 playlist URL, parses all available stream variants (different qualities/bitrates), continuously monitors each variant's chunklist for new segments, downloads them locally, and optionally transfers them to network storage for long-term archival.
|
||||||
|
|
||||||
## Architecture
|
## Architecture
|
||||||
|
|
||||||
The project follows a clean separation of concerns:
|
The project follows a modular architecture with clear separation of concerns:
|
||||||
|
|
||||||
- **main.go**: Entry point that demonstrates usage of the media package
|
- **cmd/**: Entry points for different execution modes
|
||||||
- **media/**: Core package containing M3U8 parsing logic
|
- **main/main.go**: Primary CLI entry point with URL input, event naming, and mode selection
|
||||||
- **types.go**: Contains the main parsing logic and data structures (`StreamSet`, `VideoURL`, `AudioURL`)
|
- **downloader/download.go**: Core download orchestration logic with transfer service integration
|
||||||
- **utils.go**: Utility functions for parsing attributes and resolution calculations
|
- **processor/process.go**: Alternative processing entry point
|
||||||
|
- **transfer/transfer.go**: Transfer-only mode entry point
|
||||||
|
- **pkg/**: Core packages containing the application logic
|
||||||
|
- **media/**: HLS streaming and download logic
|
||||||
|
- **stream.go**: Stream variant parsing and downloading orchestration (`GetAllVariants`, `VariantDownloader`)
|
||||||
|
- **playlist.go**: M3U8 playlist loading and parsing (`LoadMediaPlaylist`)
|
||||||
|
- **segment.go**: Individual segment downloading logic (`DownloadSegment`, `SegmentJob`)
|
||||||
|
- **manifest.go**: Manifest generation and segment tracking (`ManifestWriter`, `ManifestItem`)
|
||||||
|
- **transfer/**: NAS transfer system (complete implementation available)
|
||||||
|
- **service.go**: Transfer service orchestration
|
||||||
|
- **watcher.go**: File system monitoring for new downloads
|
||||||
|
- **queue.go**: Priority queue with worker pool management
|
||||||
|
- **nas.go**: NAS file transfer with retry logic
|
||||||
|
- **cleanup.go**: Local file cleanup after successful transfer
|
||||||
|
- **types.go**: Transfer system data structures
|
||||||
|
- **processing/**: Video processing and concatenation system
|
||||||
|
- **service.go**: Processing service orchestration with FFmpeg integration
|
||||||
|
- **segment.go**: Individual segment processing logic
|
||||||
|
- **types.go**: Processing system data structures
|
||||||
|
- **nas/**: NAS connection and file operations
|
||||||
|
- **config.go**: NAS configuration structure
|
||||||
|
- **nas.go**: NAS service with connection management and file operations
|
||||||
|
- **config/**: Centralized configuration management with validation
|
||||||
|
- **config.go**: Configuration loading, validation, and path resolution
|
||||||
|
- **utils/**: Utility functions for cross-platform compatibility
|
||||||
|
- **paths.go**: Path manipulation and validation utilities
|
||||||
|
- **constants/constants.go**: Configuration constants and singleton access
|
||||||
|
- **httpClient/error.go**: HTTP error handling utilities
|
||||||
|
|
||||||
The `GetStreamMetadata()` function is the main entry point that:
|
## Core Functionality
|
||||||
1. Fetches the M3U8 master playlist via HTTP
|
|
||||||
2. Parses the content line by line
|
### Download Workflow
|
||||||
3. Extracts video streams (`#EXT-X-STREAM-INF`) and audio streams (`#EXT-X-MEDIA`)
|
1. **Parse Master Playlist**: `GetAllVariants()` fetches and parses the master M3U8 to extract all stream variants with different qualities/bitrates
|
||||||
4. Returns a `StreamSet` containing all parsed metadata
|
2. **Concurrent Monitoring**: Each variant gets its own goroutine running `VariantDownloader()` that continuously polls for playlist updates
|
||||||
|
3. **Segment Detection**: When new segments appear in a variant's playlist, they are queued for download
|
||||||
|
4. **Parallel Downloads**: Segments are downloaded concurrently with configurable worker pools and retry logic
|
||||||
|
5. **Quality Organization**: Downloaded segments are organized by resolution (1080p, 720p, etc.) in separate directories
|
||||||
|
6. **Manifest Generation**: `ManifestWriter` tracks all downloaded segments with sequence numbers and resolutions
|
||||||
|
|
||||||
|
### NAS Transfer Workflow (Optional)
|
||||||
|
1. **File Watching**: `FileWatcher` monitors download directories for new `.ts` files
|
||||||
|
2. **Transfer Queuing**: New files are added to a priority queue after a settling delay
|
||||||
|
3. **Background Transfer**: Worker pool transfers files to NAS with retry logic and verification
|
||||||
|
4. **Local Cleanup**: Successfully transferred files are automatically cleaned up locally
|
||||||
|
5. **State Persistence**: Queue state is persisted to survive crashes and restarts
|
||||||
|
|
||||||
|
### Video Processing Workflow (Optional)
|
||||||
|
1. **Segment Collection**: Processing service reads downloaded segments from NAS storage
|
||||||
|
2. **Quality Selection**: Automatically selects the highest quality variant available
|
||||||
|
3. **FFmpeg Processing**: Uses FFmpeg to concatenate segments into a single MP4 file
|
||||||
|
4. **Output Management**: Processed videos are saved to the configured output directory
|
||||||
|
5. **Concurrent Processing**: Multiple events can be processed simultaneously with worker pools
|
||||||
|
|
||||||
|
## Key Data Structures
|
||||||
|
|
||||||
|
- `StreamVariant`: Represents a stream quality variant with URL, bandwidth, resolution, output directory, and manifest writer
|
||||||
|
- `SegmentJob`: Represents a segment download task with URI, sequence number, and variant info
|
||||||
|
- `ManifestWriter`: Tracks downloaded segments and generates JSON manifests
|
||||||
|
- `ManifestItem`: Individual segment record with sequence number and resolution
|
||||||
|
- `TransferItem`: Transfer queue item with source, destination, retry count, and status
|
||||||
|
- `TransferService`: Orchestrates file watching, queuing, transfer, and cleanup
|
||||||
|
- `ProcessingService`: Manages video processing operations with FFmpeg integration
|
||||||
|
- `ProcessConfig`: Configuration for processing operations including worker count and paths
|
||||||
|
- `NASService`: Handles NAS connection, authentication, and file operations
|
||||||
|
- `NASConfig`: Configuration structure for NAS connection parameters
|
||||||
|
|
||||||
|
## Configuration
|
||||||
|
|
||||||
|
Configuration is managed through a centralized system in `pkg/config/config.go` with environment variable support for deployment flexibility. The system provides validation, cross-platform path resolution, and sensible defaults:
|
||||||
|
|
||||||
|
### Core Settings
|
||||||
|
- `Core.WorkerCount`: Number of concurrent segment downloaders per variant (4) - ENV: `WORKER_COUNT`
|
||||||
|
- `Core.RefreshDelay`: How often to check for playlist updates (3 seconds) - ENV: `REFRESH_DELAY_SECONDS`
|
||||||
|
|
||||||
|
### Path Configuration
|
||||||
|
- `Paths.LocalOutput`: Base directory for local downloads (`data/`) - ENV: `LOCAL_OUTPUT_DIR`
|
||||||
|
- `Paths.ProcessOutput`: Directory for processed videos (`out/`) - ENV: `PROCESS_OUTPUT_DIR`
|
||||||
|
- `Paths.ManifestDir`: Directory for manifest JSON files (`data/`)
|
||||||
|
- `Paths.PersistenceFile`: Transfer queue state file location
|
||||||
|
|
||||||
|
### HTTP Settings
|
||||||
|
- `HTTPUserAgent`: User agent string for HTTP requests
|
||||||
|
- `REFERRER`: Referer header for HTTP requests (`https://www.flomarching.com`)
|
||||||
|
|
||||||
|
### NAS Transfer Settings
|
||||||
|
- `NAS.EnableTransfer`: Enable/disable automatic NAS transfer (true) - ENV: `ENABLE_NAS_TRANSFER`
|
||||||
|
- `NAS.OutputPath`: UNC path to NAS storage (`\\HomeLabNAS\dci\streams`) - ENV: `NAS_OUTPUT_PATH`
|
||||||
|
- `NAS.Username`/`NAS.Password`: NAS credentials for authentication - ENV: `NAS_USERNAME`/`NAS_PASSWORD`
|
||||||
|
- `Transfer.WorkerCount`: Concurrent transfer workers (2)
|
||||||
|
- `Transfer.RetryLimit`: Max retry attempts per file (3)
|
||||||
|
- `Transfer.Timeout`: Timeout per file transfer (30 seconds)
|
||||||
|
- `Transfer.FileSettlingDelay`: Wait before queuing new files (5 seconds)
|
||||||
|
- `Transfer.QueueSize`: Maximum queue size (100000)
|
||||||
|
- `Transfer.BatchSize`: Batch processing size (1000)
|
||||||
|
|
||||||
|
### Processing Settings
|
||||||
|
- `Processing.AutoProcess`: Enable automatic processing after download (true)
|
||||||
|
- `Processing.Enabled`: Enable processing functionality (true)
|
||||||
|
- `Processing.WorkerCount`: Concurrent processing workers (2)
|
||||||
|
- `Processing.FFmpegPath`: Path to FFmpeg executable (`ffmpeg`) - ENV: `FFMPEG_PATH`
|
||||||
|
|
||||||
|
### Cleanup Settings
|
||||||
|
- `Cleanup.AfterTransfer`: Delete local files after NAS transfer (true)
|
||||||
|
- `Cleanup.BatchSize`: Files processed per cleanup batch (1000)
|
||||||
|
- `Cleanup.RetainHours`: Hours to keep local files (0 = immediate cleanup)
|
||||||
|
|
||||||
|
### Configuration Access
|
||||||
|
```go
|
||||||
|
cfg := constants.MustGetConfig() // Get validated config singleton
|
||||||
|
eventPath := cfg.GetEventPath("my-event") // Get cross-platform paths
|
||||||
|
```
|
||||||
|
|
||||||
|
See `DEPLOYMENT.md` for detailed environment variable configuration and deployment examples.
|
||||||
|
|
||||||
## Common Development Commands
|
## Common Development Commands
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Build the project
|
# Build the main application
|
||||||
go build -o m3u8-downloader
|
go build -o stream-recorder ./cmd/main
|
||||||
|
|
||||||
# Run the project
|
# Run with URL prompt
|
||||||
go run main.go
|
go run ./cmd/main/main.go
|
||||||
|
|
||||||
|
# Run with command line arguments
|
||||||
|
go run ./cmd/main/main.go -url="https://example.com/playlist.m3u8" -event="my-event" -debug=true
|
||||||
|
|
||||||
# Run with module support
|
# Run with module support
|
||||||
go mod tidy
|
go mod tidy
|
||||||
@ -40,12 +149,80 @@ go test ./...
|
|||||||
go fmt ./...
|
go fmt ./...
|
||||||
```
|
```
|
||||||
|
|
||||||
## Key Data Structures
|
## Command Line Options
|
||||||
|
|
||||||
- `StreamSet`: Root structure containing playlist URL and all streams
|
- `-url`: M3U8 playlist URL (if not provided, prompts for input)
|
||||||
- `VideoURL`: Represents video stream with bandwidth, codecs, resolution, frame rate
|
- `-event`: Event name for organizing downloads (defaults to current date)
|
||||||
- `AudioURL`: Represents audio stream with media type, group ID, name, and selection flags
|
- `-debug`: Debug mode (only downloads 1080p variant for easier testing)
|
||||||
|
- `-transfer`: Transfer-only mode (transfer existing files without downloading)
|
||||||
|
- `-process`: Process-only mode (process existing files without downloading)
|
||||||
|
|
||||||
|
## Monitoring and Downloads
|
||||||
|
|
||||||
|
The application implements comprehensive real-time stream monitoring:
|
||||||
|
|
||||||
|
### Download Features
|
||||||
|
- **Continuous Polling**: Each variant playlist is checked every 3 seconds for new segments
|
||||||
|
- **Deduplication**: Uses segment URIs and sequence numbers to avoid re-downloading
|
||||||
|
- **Graceful Shutdown**: Responds to SIGINT/SIGTERM signals for clean exit
|
||||||
|
- **Error Resilience**: Retries failed downloads and handles HTTP 403 errors specially
|
||||||
|
- **Quality Detection**: Automatically determines resolution from bandwidth or explicit resolution data
|
||||||
|
- **Context Cancellation**: Proper timeout and cancellation handling for clean shutdowns
|
||||||
|
|
||||||
|
### Transfer Features (when enabled)
|
||||||
|
- **Real-time Transfer**: Files are transferred to NAS as soon as they're downloaded
|
||||||
|
- **Queue Persistence**: Transfer queue survives application restarts
|
||||||
|
- **Retry Logic**: Failed transfers are retried with exponential backoff
|
||||||
|
- **Verification**: File sizes are verified after transfer
|
||||||
|
- **Automatic Cleanup**: Local files are removed after successful NAS transfer
|
||||||
|
- **Statistics Reporting**: Transfer progress and statistics are logged regularly
|
||||||
|
|
||||||
|
### Manifest Generation
|
||||||
|
- **Segment Tracking**: All downloaded segments are tracked with sequence numbers
|
||||||
|
- **Resolution Mapping**: Segments are associated with their quality variants
|
||||||
|
- **JSON Output**: Manifest files are generated as sorted JSON arrays for easy processing
|
||||||
|
|
||||||
## Error Handling
|
## Error Handling
|
||||||
|
|
||||||
The current implementation uses `panic()` for error handling. When extending functionality, consider implementing proper error handling with returned error values following Go conventions.
|
The implementation uses proper Go error handling patterns:
|
||||||
|
- **Custom HTTP Errors**: Structured error types for HTTP failures
|
||||||
|
- **Context-Aware Cancellation**: Proper handling of shutdown scenarios
|
||||||
|
- **Retry Logic**: Exponential backoff for transient failures
|
||||||
|
- **Logging**: Clear status indicators (✓ for success, ✗ for failure)
|
||||||
|
- **Graceful Degradation**: Transfer service failures don't stop downloads
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
|
||||||
|
- `github.com/grafov/m3u8`: M3U8 playlist parsing
|
||||||
|
- `github.com/fsnotify/fsnotify`: File system event monitoring for NAS transfers
|
||||||
|
|
||||||
|
## Data Organization
|
||||||
|
|
||||||
|
Downloaded files are organized as:
|
||||||
|
```
|
||||||
|
./data/
|
||||||
|
├── {event-name}.json # Manifest file
|
||||||
|
├── {event-name}/ # Event-specific directory
|
||||||
|
│ ├── 1080p/ # High quality segments
|
||||||
|
│ ├── 720p/ # Medium quality segments
|
||||||
|
│ └── 480p/ # Lower quality segments
|
||||||
|
├── transfer_queue.json # Transfer queue state
|
||||||
|
├── refresh_token.txt # Authentication tokens
|
||||||
|
└── tokens.txt # Session tokens
|
||||||
|
```
|
||||||
|
|
||||||
|
NAS files mirror the local structure:
|
||||||
|
```
|
||||||
|
\\HomeLabNAS\dci\streams\
|
||||||
|
└── {event-name}/
|
||||||
|
├── 1080p/
|
||||||
|
├── 720p/
|
||||||
|
└── 480p/
|
||||||
|
```
|
||||||
|
|
||||||
|
Processed files are output to:
|
||||||
|
```
|
||||||
|
./out/
|
||||||
|
└── {event-name}/
|
||||||
|
└── concatenated_segments.mp4 # Final processed video
|
||||||
|
```
|
||||||
192
DEPLOYMENT.md
Normal file
192
DEPLOYMENT.md
Normal file
@ -0,0 +1,192 @@
|
|||||||
|
# Deployment Guide
|
||||||
|
|
||||||
|
This document outlines how to deploy and configure the StreamRecorder application in different environments.
|
||||||
|
|
||||||
|
## Environment Variables
|
||||||
|
|
||||||
|
The application supports configuration through environment variables for flexible deployment:
|
||||||
|
|
||||||
|
### Core Settings
|
||||||
|
- `WORKER_COUNT`: Number of concurrent segment downloaders per variant (default: 4)
|
||||||
|
- `REFRESH_DELAY_SECONDS`: How often to check for playlist updates in seconds (default: 3)
|
||||||
|
|
||||||
|
### NAS Transfer Settings
|
||||||
|
- `NAS_OUTPUT_PATH`: UNC path to NAS storage (default: "\\\\HomeLabNAS\\dci\\streams")
|
||||||
|
- `NAS_USERNAME`: NAS authentication username
|
||||||
|
- `NAS_PASSWORD`: NAS authentication password
|
||||||
|
- `ENABLE_NAS_TRANSFER`: Enable/disable automatic NAS transfer (default: true)
|
||||||
|
|
||||||
|
### Path Configuration
|
||||||
|
- `LOCAL_OUTPUT_DIR`: Base directory for local downloads (default: "data")
|
||||||
|
- `PROCESS_OUTPUT_DIR`: Output directory for processed videos (default: "out")
|
||||||
|
|
||||||
|
### Processing Settings
|
||||||
|
- `FFMPEG_PATH`: Path to FFmpeg executable (default: "ffmpeg")
|
||||||
|
|
||||||
|
## Docker Deployment
|
||||||
|
|
||||||
|
### Dockerfile Example
|
||||||
|
|
||||||
|
```dockerfile
|
||||||
|
FROM golang:1.23-alpine AS builder
|
||||||
|
WORKDIR /app
|
||||||
|
COPY go.mod go.sum ./
|
||||||
|
RUN go mod download
|
||||||
|
COPY . .
|
||||||
|
RUN go build -o stream-recorder ./cmd/main
|
||||||
|
|
||||||
|
FROM alpine:latest
|
||||||
|
RUN apk --no-cache add ca-certificates ffmpeg
|
||||||
|
WORKDIR /root/
|
||||||
|
COPY --from=builder /app/stream-recorder .
|
||||||
|
CMD ["./stream-recorder"]
|
||||||
|
```
|
||||||
|
|
||||||
|
### Docker Compose Example
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
version: '3.8'
|
||||||
|
services:
|
||||||
|
stream-recorder:
|
||||||
|
build: .
|
||||||
|
environment:
|
||||||
|
- NAS_OUTPUT_PATH=/mnt/nas/streams
|
||||||
|
- NAS_USERNAME=${NAS_USERNAME}
|
||||||
|
- NAS_PASSWORD=${NAS_PASSWORD}
|
||||||
|
- LOCAL_OUTPUT_DIR=/app/data
|
||||||
|
- PROCESS_OUTPUT_DIR=/app/out
|
||||||
|
- FFMPEG_PATH=ffmpeg
|
||||||
|
volumes:
|
||||||
|
- ./data:/app/data
|
||||||
|
- ./out:/app/out
|
||||||
|
- nas_mount:/mnt/nas
|
||||||
|
networks:
|
||||||
|
- stream_network
|
||||||
|
|
||||||
|
volumes:
|
||||||
|
nas_mount:
|
||||||
|
driver: local
|
||||||
|
driver_opts:
|
||||||
|
type: cifs
|
||||||
|
device: "//HomeLabNAS/dci"
|
||||||
|
o: username=${NAS_USERNAME},password=${NAS_PASSWORD},iocharset=utf8
|
||||||
|
|
||||||
|
networks:
|
||||||
|
stream_network:
|
||||||
|
driver: bridge
|
||||||
|
```
|
||||||
|
|
||||||
|
## Kubernetes Deployment
|
||||||
|
|
||||||
|
### ConfigMap Example
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
apiVersion: v1
|
||||||
|
kind: ConfigMap
|
||||||
|
metadata:
|
||||||
|
name: stream-recorder-config
|
||||||
|
data:
|
||||||
|
WORKER_COUNT: "4"
|
||||||
|
REFRESH_DELAY_SECONDS: "3"
|
||||||
|
ENABLE_NAS_TRANSFER: "true"
|
||||||
|
LOCAL_OUTPUT_DIR: "/app/data"
|
||||||
|
PROCESS_OUTPUT_DIR: "/app/out"
|
||||||
|
FFMPEG_PATH: "ffmpeg"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Secret Example
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
apiVersion: v1
|
||||||
|
kind: Secret
|
||||||
|
metadata:
|
||||||
|
name: stream-recorder-secrets
|
||||||
|
type: Opaque
|
||||||
|
data:
|
||||||
|
NAS_USERNAME: <base64-encoded-username>
|
||||||
|
NAS_PASSWORD: <base64-encoded-password>
|
||||||
|
```
|
||||||
|
|
||||||
|
### Deployment Example
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
apiVersion: apps/v1
|
||||||
|
kind: Deployment
|
||||||
|
metadata:
|
||||||
|
name: stream-recorder
|
||||||
|
spec:
|
||||||
|
replicas: 1
|
||||||
|
selector:
|
||||||
|
matchLabels:
|
||||||
|
app: stream-recorder
|
||||||
|
template:
|
||||||
|
metadata:
|
||||||
|
labels:
|
||||||
|
app: stream-recorder
|
||||||
|
spec:
|
||||||
|
containers:
|
||||||
|
- name: stream-recorder
|
||||||
|
image: stream-recorder:latest
|
||||||
|
envFrom:
|
||||||
|
- configMapRef:
|
||||||
|
name: stream-recorder-config
|
||||||
|
- secretRef:
|
||||||
|
name: stream-recorder-secrets
|
||||||
|
volumeMounts:
|
||||||
|
- name: data-storage
|
||||||
|
mountPath: /app/data
|
||||||
|
- name: output-storage
|
||||||
|
mountPath: /app/out
|
||||||
|
volumes:
|
||||||
|
- name: data-storage
|
||||||
|
persistentVolumeClaim:
|
||||||
|
claimName: stream-data-pvc
|
||||||
|
- name: output-storage
|
||||||
|
persistentVolumeClaim:
|
||||||
|
claimName: stream-output-pvc
|
||||||
|
```
|
||||||
|
|
||||||
|
## Production Considerations
|
||||||
|
|
||||||
|
### Security
|
||||||
|
- Never commit credentials to version control
|
||||||
|
- Use environment variables or secret management systems for sensitive data
|
||||||
|
- Consider using service accounts or IAM roles for cloud deployments
|
||||||
|
- Rotate credentials regularly
|
||||||
|
|
||||||
|
### Monitoring
|
||||||
|
- Implement health checks for the application
|
||||||
|
- Monitor disk space for download directories
|
||||||
|
- Set up alerts for failed transfers or processing
|
||||||
|
- Log to centralized logging systems
|
||||||
|
|
||||||
|
### Scaling
|
||||||
|
- Use horizontal scaling for multiple concurrent streams
|
||||||
|
- Consider using message queues for segment processing
|
||||||
|
- Implement distributed storage for high availability
|
||||||
|
- Use load balancers for multiple instances
|
||||||
|
|
||||||
|
### Backup and Recovery
|
||||||
|
- Regular backups of configuration and state files
|
||||||
|
- Test recovery procedures
|
||||||
|
- Document rollback processes
|
||||||
|
- Maintain disaster recovery plans
|
||||||
|
|
||||||
|
## Configuration Validation
|
||||||
|
|
||||||
|
The application validates configuration at startup and will fail fast if:
|
||||||
|
- Required directories cannot be created
|
||||||
|
- NAS paths are invalid when transfer is enabled
|
||||||
|
- FFmpeg is not found when processing is enabled
|
||||||
|
- Critical environment variables are malformed
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
### Common Issues
|
||||||
|
1. **Path Permission Errors**: Ensure the application has write access to configured directories
|
||||||
|
2. **NAS Connection Failures**: Verify network connectivity and credentials
|
||||||
|
3. **FFmpeg Not Found**: Install FFmpeg or set correct FFMPEG_PATH
|
||||||
|
4. **Environment Variable Format**: Check for typos and correct boolean values
|
||||||
|
|
||||||
|
### Debug Mode
|
||||||
|
Run with `-debug=true` to enable debug logging and download only 1080p variants for testing.
|
||||||
128
Makefile
Normal file
128
Makefile
Normal file
@ -0,0 +1,128 @@
|
|||||||
|
# StreamRecorder Makefile
|
||||||
|
|
||||||
|
.PHONY: test test-verbose test-coverage build clean help
|
||||||
|
|
||||||
|
# Default target
|
||||||
|
all: build
|
||||||
|
|
||||||
|
# Build the application
|
||||||
|
build:
|
||||||
|
@echo "🔨 Building StreamRecorder..."
|
||||||
|
go build -o stream-recorder.exe ./cmd/main
|
||||||
|
|
||||||
|
# Build for different platforms
|
||||||
|
build-windows:
|
||||||
|
@echo "🔨 Building for Windows..."
|
||||||
|
GOOS=windows GOARCH=amd64 go build -o stream-recorder-windows.exe ./cmd/main
|
||||||
|
|
||||||
|
build-linux:
|
||||||
|
@echo "🔨 Building for Linux..."
|
||||||
|
GOOS=linux GOARCH=amd64 go build -o stream-recorder-linux ./cmd/main
|
||||||
|
|
||||||
|
build-all: build-windows build-linux
|
||||||
|
|
||||||
|
# Run unit tests
|
||||||
|
test:
|
||||||
|
@echo "🧪 Running unit tests..."
|
||||||
|
go run test_runner.go
|
||||||
|
|
||||||
|
# Run tests with verbose output
|
||||||
|
test-verbose:
|
||||||
|
@echo "🧪 Running unit tests (verbose)..."
|
||||||
|
go test -v ./pkg/...
|
||||||
|
|
||||||
|
# Run tests with coverage
|
||||||
|
test-coverage:
|
||||||
|
@echo "🧪 Running tests with coverage..."
|
||||||
|
go test -coverprofile=coverage.out ./pkg/...
|
||||||
|
go tool cover -html=coverage.out -o coverage.html
|
||||||
|
@echo "📊 Coverage report generated: coverage.html"
|
||||||
|
|
||||||
|
# Run tests for a specific package
|
||||||
|
test-pkg:
|
||||||
|
@if [ -z "$(PKG)" ]; then \
|
||||||
|
echo "❌ Please specify package: make test-pkg PKG=./pkg/config"; \
|
||||||
|
exit 1; \
|
||||||
|
fi
|
||||||
|
@echo "🧪 Testing package: $(PKG)"
|
||||||
|
go test -v $(PKG)
|
||||||
|
|
||||||
|
# Run benchmarks
|
||||||
|
benchmark:
|
||||||
|
@echo "🏃 Running benchmarks..."
|
||||||
|
go test -bench=. -benchmem ./pkg/...
|
||||||
|
|
||||||
|
# Clean build artifacts
|
||||||
|
clean:
|
||||||
|
@echo "🧹 Cleaning build artifacts..."
|
||||||
|
rm -f stream-recorder.exe stream-recorder-windows.exe stream-recorder-linux
|
||||||
|
rm -f coverage.out coverage.html
|
||||||
|
rm -rf data/ out/ *.json
|
||||||
|
|
||||||
|
# Format code
|
||||||
|
fmt:
|
||||||
|
@echo "🎨 Formatting code..."
|
||||||
|
go fmt ./...
|
||||||
|
|
||||||
|
# Lint code (requires golangci-lint)
|
||||||
|
lint:
|
||||||
|
@echo "🔍 Linting code..."
|
||||||
|
golangci-lint run
|
||||||
|
|
||||||
|
# Tidy dependencies
|
||||||
|
tidy:
|
||||||
|
@echo "📦 Tidying dependencies..."
|
||||||
|
go mod tidy
|
||||||
|
|
||||||
|
# Run security check (requires gosec)
|
||||||
|
security:
|
||||||
|
@echo "🔒 Running security check..."
|
||||||
|
gosec ./...
|
||||||
|
|
||||||
|
# Install development tools
|
||||||
|
install-tools:
|
||||||
|
@echo "🛠️ Installing development tools..."
|
||||||
|
go install github.com/golangci/golangci-lint/cmd/golangci-lint@latest
|
||||||
|
go install github.com/securecodewarrior/gosec/v2/cmd/gosec@latest
|
||||||
|
|
||||||
|
# Quick development cycle: format, tidy, build, test
|
||||||
|
dev: fmt tidy build test
|
||||||
|
|
||||||
|
# CI pipeline: format check, lint, security, test, build
|
||||||
|
ci: fmt tidy lint security test build
|
||||||
|
|
||||||
|
# Help
|
||||||
|
help:
|
||||||
|
@echo "StreamRecorder Build Commands"
|
||||||
|
@echo "============================="
|
||||||
|
@echo ""
|
||||||
|
@echo "Build Commands:"
|
||||||
|
@echo " build - Build the main application"
|
||||||
|
@echo " build-windows - Build for Windows (x64)"
|
||||||
|
@echo " build-linux - Build for Linux (x64)"
|
||||||
|
@echo " build-all - Build for all platforms"
|
||||||
|
@echo ""
|
||||||
|
@echo "Test Commands:"
|
||||||
|
@echo " test - Run unit tests with custom runner"
|
||||||
|
@echo " test-verbose - Run tests with verbose output"
|
||||||
|
@echo " test-coverage - Run tests with coverage report"
|
||||||
|
@echo " test-pkg PKG=<pkg> - Test specific package"
|
||||||
|
@echo " benchmark - Run benchmarks"
|
||||||
|
@echo ""
|
||||||
|
@echo "Quality Commands:"
|
||||||
|
@echo " fmt - Format code"
|
||||||
|
@echo " lint - Lint code (requires golangci-lint)"
|
||||||
|
@echo " security - Security analysis (requires gosec)"
|
||||||
|
@echo " tidy - Tidy dependencies"
|
||||||
|
@echo ""
|
||||||
|
@echo "Development Commands:"
|
||||||
|
@echo " dev - Quick dev cycle (fmt, tidy, build, test)"
|
||||||
|
@echo " ci - Full CI pipeline"
|
||||||
|
@echo " clean - Clean build artifacts"
|
||||||
|
@echo " install-tools - Install development tools"
|
||||||
|
@echo ""
|
||||||
|
@echo "Examples:"
|
||||||
|
@echo " make test"
|
||||||
|
@echo " make test-pkg PKG=./pkg/config"
|
||||||
|
@echo " make build-all"
|
||||||
|
@echo " make dev"
|
||||||
180
README.md
Normal file
180
README.md
Normal file
@ -0,0 +1,180 @@
|
|||||||
|
# m3u8-downloader
|
||||||
|
|
||||||
|
A Go command-line tool that records live HLS (HTTP Live Streaming) broadcasts. Point it at a master `.m3u8` playlist URL and it will:
|
||||||
|
|
||||||
|
1. **Download** every quality variant (1080p, 720p, ...) in real time as new segments appear.
|
||||||
|
2. **Transfer** finished segments to NAS storage for archival, with retry logic and local cleanup.
|
||||||
|
3. **Process** the archived segments into a single playable `.mp4` per event via FFmpeg.
|
||||||
|
|
||||||
|
**Download and transfer run together, in one invocation.** When NAS transfer is enabled (the default), starting a download also starts the transfer service in the same process: as soon as a segment is written to disk, a file watcher picks it up, transfers it to NAS, and deletes the local copy — all while the download keeps running. This keeps local disk usage bounded during a long recording instead of accumulating the whole event locally.
|
||||||
|
|
||||||
|
**Processing is a separate, later invocation** (see [Usage](#usage) below), because it reads finished segments *from NAS*, after transfer has fully drained — it can't run concurrently with the other two in any useful way.
|
||||||
|
|
||||||
|
There's also a standalone `-transfer` mode, but it's a catch-up/recovery tool, not the normal path: use it when a download session already ended with some files not yet confirmed on NAS (killed before the queue drained, or transfer was disabled during the recording and you want to sync afterward).
|
||||||
|
|
||||||
|
The project also ships structured JSON logging per subsystem and an HTTP status endpoint with a small web dashboard, so you can tell what's happening (and especially what's failing) without staring at a terminal.
|
||||||
|
|
||||||
|
> Project docs also refer to this as "StreamRecorder" (see `DEPLOYMENT.md`/`TESTING.md`) — same project, the Go module is named `m3u8-downloader`.
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
```
|
||||||
|
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
|
||||||
|
.m3u8 URL ──▶ │ DOWNLOAD │──────▶ │ TRANSFER │──────▶ │ PROCESSING │──▶ event.mp4
|
||||||
|
│ (pkg/media) │ .ts │(pkg/transfer,│ NAS │(pkg/processing)
|
||||||
|
└──────────────┘ files │ pkg/nas) │ files └──────────────┘
|
||||||
|
│ └──────────────┘ │
|
||||||
|
└──────────────┬─────────────────────────────────┘
|
||||||
|
▼
|
||||||
|
pkg/logging (JSON files) + pkg/health (HTTP /status, dashboard)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Download — `pkg/media`
|
||||||
|
|
||||||
|
- `stream.go`: `GetAllVariants()` fetches and parses the master playlist, extracting every quality variant. `VariantDownloader()` runs one per-variant polling loop (every `Core.RefreshDelay`, default 3s) that watches the variant's chunklist for new segments and dispatches downloads through a worker-limited semaphore (`Core.WorkerCount` per variant, default 4).
|
||||||
|
- `playlist.go`: `LoadMediaPlaylist()` fetches and parses an individual variant's chunklist.
|
||||||
|
- `segment.go`: `DownloadSegment()` downloads one `.ts` file, with a couple of retries (including a dedicated retry for transient HTTP 403s some CDNs return).
|
||||||
|
- `manifest.go`: `ManifestWriter` tracks every segment downloaded per event and writes a JSON manifest (sequence number + resolution) at the end of a run.
|
||||||
|
|
||||||
|
Segments land on disk under `Paths.LocalOutput/<event>/<resolution>/`.
|
||||||
|
|
||||||
|
### Transfer — `pkg/transfer` + `pkg/nas`
|
||||||
|
|
||||||
|
- `service.go`: `TransferService` wires together the watcher, queue, and cleanup service, and owns the connection to NAS for the run.
|
||||||
|
- `watcher.go`: `FileWatcher` (via `fsnotify`) watches the local event directory for new `.ts` files and queues them after a short settling delay (`Transfer.FileSettlingDelay`, default 5s), so a file isn't queued while still being written.
|
||||||
|
- `queue.go`: `TransferQueue` is a priority-queued worker pool (`Transfer.WorkerCount`, default 2) that transfers queued files to NAS with retry + exponential backoff, and persists its state to `transfer_queue.json` so it survives a restart.
|
||||||
|
- `nas.go` / `pkg/nas`: `NASService` establishes the NAS connection (on Windows, via `net use` for UNC paths) and does the actual file copy + size verification.
|
||||||
|
- `cleanup.go`: `CleanupService` deletes local copies after a successful, verified transfer (`Cleanup.AfterTransfer`, default true).
|
||||||
|
|
||||||
|
During a normal download run, this whole stage starts automatically in the same process as the downloader (see the note above) — you don't invoke transfer separately for a live recording. The standalone `-transfer` mode exists only to catch up files that a prior download session ended without fully transferring.
|
||||||
|
|
||||||
|
### Processing — `pkg/processing`
|
||||||
|
|
||||||
|
- `ProcessingService` reads segments **from NAS** (not the local disk) for a given event, picks the highest-resolution variant available for each sequence number, writes an FFmpeg concat list, and shells out to FFmpeg to produce one `.mp4`.
|
||||||
|
- This is why processing is normally the *last* step, run after transfer has fully drained — it only sees what's actually on NAS.
|
||||||
|
- Requires a working `ffmpeg` on `PATH` (or `Processing.FFmpegPath` / `FFMPEG_PATH`), and connects to NAS on startup regardless of whether `Processing.Enabled` is set.
|
||||||
|
|
||||||
|
### Configuration — `pkg/config` + `pkg/constants`
|
||||||
|
|
||||||
|
`pkg/config/config.go` defines defaults and environment-variable overrides for every setting below. `pkg/constants.MustGetConfig()` lazily loads and caches this config as a process-wide singleton — most of the codebase reads config this way rather than passing it around explicitly.
|
||||||
|
|
||||||
|
### Logging — `pkg/logging`
|
||||||
|
|
||||||
|
A small factory (modeled after a structured-logging pattern used in another project) that hands out one `log/slog` JSON logger per pipeline stage: `App`, `Download`, `Transfer`, `Processing`. Each writes to its own file under `Logging.Dir` (default `data/logs/{app,download,transfer,processing}.log`). This is **additive** — every existing human-readable console log line is untouched, so you can still watch progress live in the terminal; the structured logs are there for after-the-fact debugging.
|
||||||
|
|
||||||
|
### Health / status — `pkg/health`
|
||||||
|
|
||||||
|
An in-memory registry that every subsystem reports into (active variants, segment success/failure counts, NAS connectivity, transfer queue stats, processing pipeline state), served over HTTP:
|
||||||
|
|
||||||
|
- `GET /status` — full JSON snapshot, `status: "healthy"` or `"degraded"` (degraded the moment anything has a recent error, NAS is disconnected while in use, or processing has failed).
|
||||||
|
- `GET /healthz` — plain liveness check.
|
||||||
|
- `GET /` — a small auto-refreshing dashboard (polls `/status` every 3s) showing the same information visually. No external dependencies (fonts/JS/CSS), so it works on a venue LAN with no internet access.
|
||||||
|
|
||||||
|
**Important scope note:** the health server only reflects whatever is live in the *current* process. During a normal download run, `/status` shows real-time download + transfer state together (they're the same process) with processing reported as `idle`. Running `-process` afterward is a separate process with its own fresh health server showing real-time processing state (and idle download/transfer). There's no single dashboard spanning download/transfer and processing at the same time, since processing only ever runs after the fact.
|
||||||
|
|
||||||
|
By default the health server binds all network interfaces (`:8090`) so it can be checked from another device on the same LAN (e.g. a phone) during an event. **There is no authentication or TLS** — this is fine on a trusted local network only, never expose it to the open internet.
|
||||||
|
|
||||||
|
## Configuration
|
||||||
|
|
||||||
|
All settings have sane defaults; override via environment variables as needed.
|
||||||
|
|
||||||
|
| Env var | Default | Meaning |
|
||||||
|
|---|---|---|
|
||||||
|
| `WORKER_COUNT` | `4` | Concurrent segment downloads per variant |
|
||||||
|
| `REFRESH_DELAY_SECONDS` | `3` | How often each variant's chunklist is polled |
|
||||||
|
| `ENABLE_NAS_TRANSFER` | `true` | Whether to run the transfer service alongside download |
|
||||||
|
| `NAS_OUTPUT_PATH` | `\\HomeLabNAS\dci\streams` | UNC/NAS root path segments are copied to |
|
||||||
|
| `NAS_USERNAME` / `NAS_PASSWORD` | *(see note below)* | NAS auth credentials |
|
||||||
|
| `LOCAL_OUTPUT_DIR` | `data` | Local base directory for downloaded segments |
|
||||||
|
| `PROCESS_OUTPUT_DIR` | `out` | Output directory for the final concatenated `.mp4` |
|
||||||
|
| `FFMPEG_PATH` | `ffmpeg` | Path to the FFmpeg binary used by the processing stage |
|
||||||
|
| `LOG_DIR` | `data/logs` | Directory for the per-subsystem JSON log files |
|
||||||
|
| `LOG_LEVEL` | `info` | `debug` \| `info` \| `warn` \| `error` (also forced to `debug` by the `-debug` flag) |
|
||||||
|
| `HEALTH_ENABLED` | `true` | Whether to start the health/status HTTP server |
|
||||||
|
| `HEALTH_ADDR` | `:8090` | Bind address for the health server + dashboard |
|
||||||
|
|
||||||
|
> **Security note:** the default NAS credentials currently ship as hardcoded defaults in `pkg/config/config.go` for this home-lab setup. If you fork or redeploy this, set `NAS_USERNAME`/`NAS_PASSWORD` explicitly and rotate the shipped default — see `TODO.md` for the full flagged list of housekeeping items like this one.
|
||||||
|
|
||||||
|
## Building
|
||||||
|
|
||||||
|
```bash
|
||||||
|
go build -o m3u8-downloader ./cmd/main
|
||||||
|
# or, via the Makefile (produces stream-recorder(.exe)):
|
||||||
|
make build
|
||||||
|
```
|
||||||
|
|
||||||
|
## Usage
|
||||||
|
|
||||||
|
All modes are invoked through the single `cmd/main` binary:
|
||||||
|
|
||||||
|
```
|
||||||
|
m3u8-downloader [-url <m3u8-url>] [-event <name>] [-debug] [-transfer] [-process]
|
||||||
|
```
|
||||||
|
|
||||||
|
| Flag | Applies to | Meaning |
|
||||||
|
|---|---|---|
|
||||||
|
| `-url` | download mode | Master `.m3u8` playlist URL. If omitted (and neither `-transfer` nor `-process` is set), you'll be prompted for it interactively. |
|
||||||
|
| `-event` | all modes | Event name — used to namespace local output, NAS destination, and the manifest file. **Always set this explicitly**; if left blank in download mode, segments land directly under the local output root instead of a per-event subfolder. In `-transfer`/`-process` mode, omitting it shows an interactive picker over existing event directories. |
|
||||||
|
| `-debug` | download mode | Restricts download to the 1080p variant only (for faster iteration) and forces log level to `debug`. |
|
||||||
|
| `-transfer` | transfer-only mode | Scans an already-downloaded event's local directory and transfers/re-transfers anything not yet confirmed on NAS, then runs the normal transfer service until shutdown. Requires `ENABLE_NAS_TRANSFER=true`. |
|
||||||
|
| `-process` | process-only mode | Reads an event's segments from NAS and concatenates the highest-quality ones into a single `.mp4` via FFmpeg. Always attempts a NAS connection on startup, even if processing is otherwise disabled in config. |
|
||||||
|
|
||||||
|
### Which command for which situation
|
||||||
|
|
||||||
|
**Recording a live event** (the normal case — downloads and transfers to NAS concurrently, in one process, so local disk usage stays bounded rather than accumulating the whole event):
|
||||||
|
```bash
|
||||||
|
m3u8-downloader -url "https://example.com/live/master.m3u8" -event "2026-regionals"
|
||||||
|
```
|
||||||
|
Stop it cleanly with Ctrl+C (SIGINT/SIGTERM); it finishes in-flight downloads, saves transfer queue state, and writes the manifest before exiting.
|
||||||
|
|
||||||
|
**Quick test/dry run** (single variant, verbose logs, to confirm a playlist URL actually works before committing to a full multi-quality recording):
|
||||||
|
```bash
|
||||||
|
m3u8-downloader -url "https://example.com/live/master.m3u8" -event "test-run" -debug
|
||||||
|
```
|
||||||
|
|
||||||
|
**Catching up NAS transfer after the fact** (recovery only — the recording process above already transfers live; use this if a session ended before every file made it to NAS, or transfer was disabled during the recording and you want to sync afterward):
|
||||||
|
```bash
|
||||||
|
m3u8-downloader -event "2026-regionals" -transfer
|
||||||
|
```
|
||||||
|
|
||||||
|
**Producing the final video** (after transfer has fully drained — check `/status` or the NAS directory to confirm nothing's still pending):
|
||||||
|
```bash
|
||||||
|
m3u8-downloader -event "2026-regionals" -process
|
||||||
|
```
|
||||||
|
|
||||||
|
**Checking on a running instance** (from any device on the same network as the recorder):
|
||||||
|
```
|
||||||
|
http://<recorder-host>:8090/ # dashboard
|
||||||
|
http://<recorder-host>:8090/status # raw JSON
|
||||||
|
```
|
||||||
|
|
||||||
|
## Data layout
|
||||||
|
|
||||||
|
```
|
||||||
|
data/
|
||||||
|
├── {event}.json # download manifest (sequence numbers + resolutions)
|
||||||
|
├── {event}/
|
||||||
|
│ ├── 1080p/ # downloaded segments, before NAS transfer + cleanup
|
||||||
|
│ ├── 720p/
|
||||||
|
│ └── ...
|
||||||
|
├── logs/
|
||||||
|
│ ├── app.log # per-subsystem structured JSON logs
|
||||||
|
│ ├── download.log
|
||||||
|
│ ├── transfer.log
|
||||||
|
│ └── processing.log
|
||||||
|
└── transfer_queue.json # persisted transfer queue state
|
||||||
|
|
||||||
|
\\<NAS>\...\{event}\ # mirrors the local structure once transferred
|
||||||
|
├── 1080p/
|
||||||
|
└── ...
|
||||||
|
|
||||||
|
out/
|
||||||
|
└── {event}/
|
||||||
|
└── {event}.mp4 # final FFmpeg output (process-only mode)
|
||||||
|
```
|
||||||
|
|
||||||
|
## More documentation
|
||||||
|
|
||||||
|
- `DEPLOYMENT.md` — Docker/Kubernetes deployment examples and production considerations.
|
||||||
|
- `TESTING.md` — test suite layout and how to run it (`make test`, `make test-coverage`, etc.).
|
||||||
|
- `TODO.md` — known issues, pre-existing bugs, and roadmap items found during review, not yet addressed.
|
||||||
253
TESTING.md
Normal file
253
TESTING.md
Normal file
@ -0,0 +1,253 @@
|
|||||||
|
# Testing Guide
|
||||||
|
|
||||||
|
This document describes the test suite for the StreamRecorder application.
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
The test suite provides comprehensive coverage of core application components without requiring external dependencies like video files, NAS connectivity, or FFmpeg. All tests are self-contained and clean up after themselves.
|
||||||
|
|
||||||
|
## Test Structure
|
||||||
|
|
||||||
|
### Unit Tests by Package
|
||||||
|
|
||||||
|
#### `pkg/config`
|
||||||
|
- **File**: `config_test.go`
|
||||||
|
- **Coverage**: Configuration loading, environment variable override, path validation, validation errors
|
||||||
|
- **Key Tests**:
|
||||||
|
- Default config loading
|
||||||
|
- Environment variable overrides
|
||||||
|
- Path resolution and creation
|
||||||
|
- Validation error scenarios
|
||||||
|
|
||||||
|
#### `pkg/utils`
|
||||||
|
- **File**: `paths_test.go`
|
||||||
|
- **Coverage**: Cross-platform path utilities, directory operations, validation
|
||||||
|
- **Key Tests**:
|
||||||
|
- Safe path joining
|
||||||
|
- Directory creation
|
||||||
|
- Path existence checking
|
||||||
|
- Path validation
|
||||||
|
- Write permission testing
|
||||||
|
|
||||||
|
#### `pkg/constants`
|
||||||
|
- **File**: `constants_test.go`
|
||||||
|
- **Coverage**: Constants values, configuration singleton, integration
|
||||||
|
- **Key Tests**:
|
||||||
|
- Constant value verification
|
||||||
|
- Singleton pattern testing
|
||||||
|
- Config integration
|
||||||
|
- Concurrent access safety
|
||||||
|
|
||||||
|
#### `pkg/httpClient`
|
||||||
|
- **File**: `error_test.go`
|
||||||
|
- **Coverage**: HTTP error handling, status code management
|
||||||
|
- **Key Tests**:
|
||||||
|
- HTTP error creation and formatting
|
||||||
|
- Error comparison and detection
|
||||||
|
- Status code extraction
|
||||||
|
- Error wrapping support
|
||||||
|
|
||||||
|
#### `pkg/media`
|
||||||
|
- **File**: `manifest_test.go`
|
||||||
|
- **Coverage**: Manifest generation, segment tracking, JSON serialization
|
||||||
|
- **Key Tests**:
|
||||||
|
- Manifest writer initialization
|
||||||
|
- Segment addition and updates
|
||||||
|
- Quality resolution logic
|
||||||
|
- JSON file generation
|
||||||
|
- Sorting and deduplication
|
||||||
|
|
||||||
|
#### `pkg/processing`
|
||||||
|
- **File**: `service_test.go`
|
||||||
|
- **Coverage**: Processing service logic, path resolution, FFmpeg handling
|
||||||
|
- **Key Tests**:
|
||||||
|
- Service initialization
|
||||||
|
- Event directory scanning
|
||||||
|
- Resolution detection
|
||||||
|
- Segment aggregation
|
||||||
|
- File concatenation list generation
|
||||||
|
- FFmpeg path resolution
|
||||||
|
|
||||||
|
## Running Tests
|
||||||
|
|
||||||
|
### Quick Test Run
|
||||||
|
```bash
|
||||||
|
make test
|
||||||
|
```
|
||||||
|
|
||||||
|
### Verbose Output
|
||||||
|
```bash
|
||||||
|
make test-verbose
|
||||||
|
```
|
||||||
|
|
||||||
|
### Coverage Report
|
||||||
|
```bash
|
||||||
|
make test-coverage
|
||||||
|
```
|
||||||
|
Generates `coverage.html` with detailed coverage report.
|
||||||
|
|
||||||
|
### Test Specific Package
|
||||||
|
```bash
|
||||||
|
make test-pkg PKG=./pkg/config
|
||||||
|
```
|
||||||
|
|
||||||
|
### Manual Test Execution
|
||||||
|
```bash
|
||||||
|
# Run custom test runner
|
||||||
|
go run test_runner.go
|
||||||
|
|
||||||
|
# Run standard go test
|
||||||
|
go test ./pkg/...
|
||||||
|
|
||||||
|
# Run with coverage
|
||||||
|
go test -coverprofile=coverage.out ./pkg/...
|
||||||
|
go tool cover -html=coverage.out
|
||||||
|
```
|
||||||
|
|
||||||
|
## Test Features
|
||||||
|
|
||||||
|
### ✅ Self-Contained
|
||||||
|
- No external file dependencies
|
||||||
|
- No network connections required
|
||||||
|
- No NAS or FFmpeg installation needed
|
||||||
|
|
||||||
|
### ✅ Automatic Cleanup
|
||||||
|
- All temporary files/directories removed after tests
|
||||||
|
- Original environment variables restored
|
||||||
|
- No side effects on host system
|
||||||
|
|
||||||
|
### ✅ Isolated Environment
|
||||||
|
- Tests use temporary directories
|
||||||
|
- Environment variables safely overridden
|
||||||
|
- Configuration isolated from production settings
|
||||||
|
|
||||||
|
### ✅ Cross-Platform
|
||||||
|
- Path handling tested on Windows/Unix
|
||||||
|
- Platform-specific behavior validated
|
||||||
|
- Cross-platform compatibility verified
|
||||||
|
|
||||||
|
### ✅ Comprehensive Coverage
|
||||||
|
- Configuration management
|
||||||
|
- Path utilities and validation
|
||||||
|
- Error handling patterns
|
||||||
|
- Data structures and serialization
|
||||||
|
- Business logic without external dependencies
|
||||||
|
|
||||||
|
## Test Environment
|
||||||
|
|
||||||
|
The test suite automatically:
|
||||||
|
|
||||||
|
1. **Creates Temporary Workspace**: Each test run uses a fresh temporary directory
|
||||||
|
2. **Sets Test Environment**: Overrides environment variables to use test settings
|
||||||
|
3. **Disables External Dependencies**: Sets flags to disable NAS transfer and processing
|
||||||
|
4. **Cleans Up Completely**: Removes all test artifacts and restores environment
|
||||||
|
|
||||||
|
### Environment Variables Set During Tests
|
||||||
|
- `LOCAL_OUTPUT_DIR`: Points to temp directory
|
||||||
|
- `PROCESS_OUTPUT_DIR`: Points to temp directory
|
||||||
|
- `ENABLE_NAS_TRANSFER`: Set to `false`
|
||||||
|
- `PROCESSING_ENABLED`: Set to `false`
|
||||||
|
|
||||||
|
## Extending Tests
|
||||||
|
|
||||||
|
### Adding New Test Cases
|
||||||
|
|
||||||
|
1. **Create test file**: `pkg/yourpackage/yourfile_test.go`
|
||||||
|
2. **Follow naming convention**: `TestFunctionName`
|
||||||
|
3. **Use temp directories**: Always clean up created files
|
||||||
|
4. **Mock external dependencies**: Avoid real file operations where possible
|
||||||
|
|
||||||
|
### Test Template
|
||||||
|
```go
|
||||||
|
package yourpackage
|
||||||
|
|
||||||
|
import (
|
||||||
|
"os"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestYourFunction(t *testing.T) {
|
||||||
|
// Setup
|
||||||
|
tempDir, err := os.MkdirTemp("", "test_*")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Setup failed: %v", err)
|
||||||
|
}
|
||||||
|
defer os.RemoveAll(tempDir)
|
||||||
|
|
||||||
|
// Test logic
|
||||||
|
result := YourFunction()
|
||||||
|
|
||||||
|
// Assertions
|
||||||
|
if result != expected {
|
||||||
|
t.Errorf("Expected %v, got %v", expected, result)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Best Practices
|
||||||
|
|
||||||
|
- **Always clean up**: Use `defer os.RemoveAll()` for temp directories
|
||||||
|
- **Test error cases**: Don't just test happy paths
|
||||||
|
- **Use table-driven tests**: For multiple similar test cases
|
||||||
|
- **Mock external dependencies**: Use echo/dummy commands instead of real tools
|
||||||
|
- **Validate cleanup**: Ensure tests don't leave artifacts
|
||||||
|
|
||||||
|
## CI/CD Integration
|
||||||
|
|
||||||
|
The test suite is designed for automated environments:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Complete CI pipeline
|
||||||
|
make ci
|
||||||
|
|
||||||
|
# Just run tests in CI
|
||||||
|
make test
|
||||||
|
```
|
||||||
|
|
||||||
|
The custom test runner provides:
|
||||||
|
- ✅ Colored output for easy reading
|
||||||
|
- ✅ Test count and timing statistics
|
||||||
|
- ✅ Failure details and summaries
|
||||||
|
- ✅ Automatic environment management
|
||||||
|
- ✅ Exit codes for CI integration
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
### Common Issues
|
||||||
|
|
||||||
|
**Tests fail with permission errors**
|
||||||
|
- Ensure write permissions in temp directory
|
||||||
|
- Check antivirus software isn't blocking file operations
|
||||||
|
|
||||||
|
**Config tests fail**
|
||||||
|
- Verify no conflicting environment variables are set
|
||||||
|
- Check that temp directories can be created
|
||||||
|
|
||||||
|
**Path tests fail on Windows**
|
||||||
|
- Confirm path separator handling is correct
|
||||||
|
- Verify Windows path validation logic
|
||||||
|
|
||||||
|
### Debug Mode
|
||||||
|
```bash
|
||||||
|
# Run with verbose output to see detailed failures
|
||||||
|
go test -v ./pkg/...
|
||||||
|
|
||||||
|
# Run specific failing test
|
||||||
|
go test -v -run TestSpecificFunction ./pkg/config
|
||||||
|
```
|
||||||
|
|
||||||
|
## Coverage Goals
|
||||||
|
|
||||||
|
Current test coverage targets:
|
||||||
|
- **Configuration**: 95%+ (critical for startup validation)
|
||||||
|
- **Path utilities**: 90%+ (cross-platform compatibility critical)
|
||||||
|
- **Constants**: 85%+ (verify all values and singleton behavior)
|
||||||
|
- **HTTP client**: 90%+ (error handling is critical)
|
||||||
|
- **Media handling**: 85%+ (core business logic)
|
||||||
|
- **Processing**: 70%+ (limited by external FFmpeg dependency)
|
||||||
|
|
||||||
|
Generate coverage report to verify:
|
||||||
|
```bash
|
||||||
|
make test-coverage
|
||||||
|
open coverage.html
|
||||||
|
```
|
||||||
111
TODO.md
Normal file
111
TODO.md
Normal file
@ -0,0 +1,111 @@
|
|||||||
|
# Return-from-a-year-away Audit — 2026-08-05
|
||||||
|
|
||||||
|
Scope reviewed: `main` (checked out), `manifest`, `cleanup` (all local + `origin` equivalents).
|
||||||
|
Remote: `https://git.aaronic.cc/kacarmichael/StreamRecorder.git`
|
||||||
|
|
||||||
|
## Branch situation (read this first)
|
||||||
|
|
||||||
|
The three branches are **not parallel efforts** — they're one linear line of development that `main` never caught up to:
|
||||||
|
|
||||||
|
```
|
||||||
|
main (0f4a39f, 2025-07-26) ──┐ stale, single-file-era downloader only
|
||||||
|
│
|
||||||
|
manifest (718d69c, 2025-08-10) ── +restructured cmd/pkg layout, config-less
|
||||||
|
│
|
||||||
|
cleanup (7d67cec, 2025-08-11) ── +pkg/config, tests, CI, Makefile, deployment docs
|
||||||
|
```
|
||||||
|
|
||||||
|
`manifest` is literally an ancestor of `cleanup` (same commit `718d69c`). `cleanup` is the most complete state that ever existed and was the last thing touched (2025-08-11) before the year gap. **None of this reached `main`.** If you pick this project back up, decide this before anything else:
|
||||||
|
|
||||||
|
- [ ] **Merge `cleanup` → `main`** (recommended — it's a strict superset: config validation, `pkg/nas`, `pkg/processing`, unit tests, CI workflow, `DEPLOYMENT.md`/`TESTING.md`). Then delete `manifest` (fully subsumed) and re-evaluate `cleanup` for deletion once merged.
|
||||||
|
- [ ] Or, if `cleanup`'s direction (FFmpeg concatenation, NAS-mirroring processing pipeline) is no longer where you want to go, explicitly archive it and restart `main` from a clean design — but do this as a decision, not by default.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🔴 Critical — fix/rotate before running this again
|
||||||
|
|
||||||
|
1. **Leaked NAS credentials, committed in plaintext.** `pkg/config/config.go` (branch `cleanup`) hardcodes a real default:
|
||||||
|
```go
|
||||||
|
Username: "NASAdmin",
|
||||||
|
Password: "s3tkY6tzA&KN6M",
|
||||||
|
```
|
||||||
|
This is in git history on a remote (`git.aaronic.cc`) regardless of whether you later change the default. **Rotate this NAS account's password now** — changing the code doesn't un-leak a password that's sat in a repo for a year. Then remove the credential entirely (fail closed: require `NAS_USERNAME`/`NAS_PASSWORD` env vars with no default, error out if unset when transfer is enabled) rather than shipping a working default.
|
||||||
|
|
||||||
|
2. **Guaranteed crash: double mutex unlock in `pkg/transfer/cleanup.go`.** In `ExecuteCleanup` (present on **`main` too**, not just `cleanup`):
|
||||||
|
```go
|
||||||
|
cs.mu.Lock()
|
||||||
|
...
|
||||||
|
cs.mu.Unlock() // line 72 — releases the lock
|
||||||
|
...
|
||||||
|
batch := make([]string, batchSize)
|
||||||
|
copy(batch, cs.pendingFiles[:batchSize])
|
||||||
|
cs.pendingFiles = cs.pendingFiles[batchSize:] // mutated WITHOUT the lock held
|
||||||
|
cs.mu.Unlock() // line 79 — unlocking an already-unlocked mutex
|
||||||
|
```
|
||||||
|
An unlock of an unlocked `sync.Mutex` is a **fatal, unrecoverable runtime error** (`fatal error: sync: unlock of unlocked mutex`) — it kills the whole process, not just a goroutine. Since `Cleanup.AfterTransfer` defaults to `true` and `RetainHours: 0` (immediate eligibility), this fires the first time any file finishes transferring and cleanup runs. This is very likely why any end-to-end run with NAS transfer enabled never survived long. Fix: hold the lock for the whole slice mutation, or copy-then-release-then-mutate-under-a-second-short-lock, but don't unlock twice.
|
||||||
|
|
||||||
|
3. **Nil-pointer crash in `pkg/media/stream.go` (`VariantDownloader`, `main` branch):**
|
||||||
|
```go
|
||||||
|
playlist, err := LoadMediaPlaylist(variant.URL)
|
||||||
|
seq := playlist.SeqNo // <-- dereferenced before the err check
|
||||||
|
if err != nil {
|
||||||
|
...
|
||||||
|
goto waitTick
|
||||||
|
}
|
||||||
|
```
|
||||||
|
`LoadMediaPlaylist` returns `nil, err` on any HTTP/parse failure, so `playlist` is a typed-nil pointer here — `playlist.SeqNo` panics. A panic in a goroutine takes down the entire process (Go doesn't isolate goroutine panics). Practically: any single transient network hiccup on any variant's playlist fetch kills the whole download session, mid-recording. Move the `err != nil` check above the `SeqNo` read.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🟠 High-priority bugs / correctness
|
||||||
|
|
||||||
|
4. **`PriorityQueue.Less` in `pkg/transfer/queue.go` sorts newest-first** (`pq[i].Timestamp.After(pq[j].Timestamp)`), making the transfer queue LIFO. For a live-recording pipeline that keeps producing new segments, this means the *oldest* queued segments can be starved indefinitely as long as new ones keep arriving — the opposite of what you'd want (drain oldest first so nothing waits forever). Worth confirming this was intentional; if not, flip the comparison.
|
||||||
|
|
||||||
|
5. **Retry backoff runs before the *first* attempt, not just retries** — `pkg/transfer/queue.go`, `processItem`:
|
||||||
|
```go
|
||||||
|
for attempt := 1; attempt <= maxRetries; attempt++ {
|
||||||
|
if attempt > 0 { // always true, since attempt starts at 1
|
||||||
|
... backoff ...
|
||||||
|
```
|
||||||
|
Every single transfer sleeps `1s` before even trying once. Should be `attempt > 1`.
|
||||||
|
|
||||||
|
6. **`pkg/constants/constants.go` (main) hardcodes a live FloSports auth token** directly in source (`MasterURL` with `token=...`, `hmac2=...`). It's expired now (epoch ~2025-07), so no live exposure, but it's the same pattern as #1 — secrets/tokens don't belong in source. Worth a rule going forward: URLs/tokens come from CLI args or env, never constants.
|
||||||
|
|
||||||
|
7. **`http.Client{}` created with no timeout** in `pkg/media/stream.go`, `playlist.go`, and `segment.go`. A stalled TCP connection (dead CDN edge, network partition) can hang a fetch forever — nothing in the surrounding code will time it out except the 10s context wrapped specifically around segment downloads in `VariantDownloader`. Playlist-fetch calls (`GetAllVariants`, `LoadMediaPlaylist`) have no such wrapper. Give the shared client a sane `Timeout` (e.g. 15–30s) as a backstop.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🟡 Cleanup / dead weight
|
||||||
|
|
||||||
|
8. **`cmd/proc/main.go` is a one-line stub** (`package proc`) on `main` — dead entry point, no `func main()`. Either finish it or delete it; right now `go build ./...` would fail on this if it were ever wired as a real `main` package (it currently silently isn't, which is its own trap for future-you).
|
||||||
|
9. **Built binaries committed to git** (`bin/flo_download`, `bin/flo_download.exe`, ~18MB, on `cleanup`/`manifest`). The `.gitea/workflows/build.yml` already cross-compiles and uploads artifacts — there's no reason binaries should also live in the repo/history. Remove and add `bin/` to `.gitignore`.
|
||||||
|
10. **`scratch.txt` is tracked despite being in `.gitignore`.** Adding a path to `.gitignore` after it's already tracked does nothing — it's still in the repo (and it's just a raw debug log dump). `git rm --cached scratch.txt`.
|
||||||
|
11. `.idea/*` (JetBrains project files) are committed. Low stakes solo-repo, but standard practice is to gitignore the whole `.idea/` directory rather than cherry-pick.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🔵 Program / architecture — bigger picture
|
||||||
|
|
||||||
|
12. **The project's own `CLAUDE.md` states the goal is "a listening REST API [that] take[s] in m3u8 urls, parse[s] them, and eventually send[s] to a conversion service."** None of the three branches implement any HTTP server — all three are CLI-only, one-shot-per-invocation tools. This is the largest gap between stated intent and actual code. Worth deciding: is the REST-API framing still the goal, or has the project's actual direction (CLI recorder + NAS mirror + FFmpeg concat, per `cleanup`'s `CLAUDE.md`) superseded it? Right now the docs and the code disagree with each other.
|
||||||
|
13. **`pkg/nas/nas.go`'s "connection" is a Windows-only shell-out to `net use`** (`EstablishConnection`), while CI (`build.yml`) cross-compiles for `linux`/`darwin`/`windows`. On non-Windows the UNC-path logic (`\\server\share`) and `net use` simply don't apply — there's no SMB client abstraction, so the Linux/macOS builds can produce a binary that can't actually do NAS transfer. If cross-platform deployment is real (not just "it happens to compile"), this needs a real cross-platform SMB/CIFS story (e.g., assume the NAS is pre-mounted by the OS and just treat it as a local path, which is actually simpler and already half-true for the Linux CIFS example in `DEPLOYMENT.md`).
|
||||||
|
14. **Zero tests on `main`.** `cleanup` has real coverage (`pkg/config`, `pkg/utils`, `pkg/constants`, `pkg/httpClient`, `pkg/media/manifest`, `pkg/processing`) plus a `Makefile` (`test`, `test-coverage`, `lint`, `security` via `gosec`) and a `TESTING.md`. All of this is stuck on a branch that was never merged — it's not a "write tests" TODO so much as a "land the tests you already wrote" TODO.
|
||||||
|
15. **CI never actually runs tests, lint, or `gosec`** despite the `Makefile` defining a full `ci` target (`fmt tidy lint security test build`). `.gitea/workflows/build.yml` only cross-compiles. Wire `make ci` (or its steps) into the pipeline so a broken build/test doesn't silently ship a binary artifact.
|
||||||
|
16. **Unbounded `seen` map** in `VariantDownloader` (`pkg/media/stream.go`) grows for the lifetime of a variant's polling loop with no eviction — fine for a few-hour event, worth a note if this is ever meant to run continuously/24-7.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🟢 Deployment
|
||||||
|
|
||||||
|
17. `DEPLOYMENT.md` (on `cleanup`) documents Docker/Compose/Kubernetes deployment in detail, but there's no actual `Dockerfile` committed anywhere in the repo — it's a doc-only example. If containerized deployment is the real target, commit the Dockerfile and at least smoke-test the build.
|
||||||
|
18. No secrets-management story beyond "use env vars" — fine for a home lab, but the doc's own "Security" section says "use secret management systems," which isn't backed by anything concrete (no `.env.example`, no sealed-secrets/vault reference). Given finding #1, this is worth actually doing, not just documenting.
|
||||||
|
19. Health checks, metrics/log shipping are listed as "considerations" in `DEPLOYMENT.md` but nothing is implemented — the app is a fire-and-forget CLI process with `log.Printf` to stdout only. If this runs unattended (e.g., in a container watching for the next event), you have no way to know it's alive short of watching logs.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Suggested order of attack
|
||||||
|
|
||||||
|
1. Rotate the NAS credential (#1) — this is time-sensitive regardless of anything else.
|
||||||
|
2. Fix the two crash bugs (#2, #3) — nothing else is worth doing until a full recording session can survive.
|
||||||
|
3. Decide the branch-merge question and land `cleanup` → `main` (or consciously don't).
|
||||||
|
4. Wire CI to actually test/lint (#15) so regressions get caught going forward.
|
||||||
|
5. Then tackle the architecture question (#12: REST API vs. CLI-recorder-as-is) before investing more in either direction.
|
||||||
BIN
bin/flo_download
Normal file
BIN
bin/flo_download
Normal file
Binary file not shown.
BIN
bin/flo_download.exe
Normal file
BIN
bin/flo_download.exe
Normal file
Binary file not shown.
123
cmd/downloader/download.go
Normal file
123
cmd/downloader/download.go
Normal file
@ -0,0 +1,123 @@
|
|||||||
|
package downloader
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"log"
|
||||||
|
"m3u8-downloader/pkg/constants"
|
||||||
|
"m3u8-downloader/pkg/health"
|
||||||
|
"m3u8-downloader/pkg/logging"
|
||||||
|
"m3u8-downloader/pkg/media"
|
||||||
|
"m3u8-downloader/pkg/transfer"
|
||||||
|
"m3u8-downloader/pkg/utils"
|
||||||
|
"os"
|
||||||
|
"os/signal"
|
||||||
|
"sync"
|
||||||
|
"syscall"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
func Download(masterURL string, eventName string, debug bool) {
|
||||||
|
appLog := logging.MustGetFactory().App
|
||||||
|
|
||||||
|
ctx, cancel := context.WithCancel(context.Background())
|
||||||
|
defer cancel()
|
||||||
|
|
||||||
|
// Goroutine to listen for shutdown signals
|
||||||
|
sigChan := make(chan os.Signal, 1)
|
||||||
|
signal.Notify(sigChan, syscall.SIGINT, syscall.SIGTERM)
|
||||||
|
go func() {
|
||||||
|
<-sigChan
|
||||||
|
log.Println("Shutting down...")
|
||||||
|
appLog.Info("shutdown signal received")
|
||||||
|
cancel()
|
||||||
|
}()
|
||||||
|
|
||||||
|
cfg := constants.MustGetConfig()
|
||||||
|
|
||||||
|
if cfg.Health.Enabled {
|
||||||
|
healthSrv, err := health.Start(ctx, &cfg.Health, health.MustGetRegistry())
|
||||||
|
if err != nil {
|
||||||
|
appLog.Error("failed to start health server", logging.KeyError, err.Error())
|
||||||
|
} else {
|
||||||
|
appLog.Info("health server listening", "addr", cfg.Health.Addr)
|
||||||
|
defer func() {
|
||||||
|
shutdownCtx, shutdownCancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||||
|
defer shutdownCancel()
|
||||||
|
_ = healthSrv.Shutdown(shutdownCtx)
|
||||||
|
}()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
var wg sync.WaitGroup
|
||||||
|
var transferService *transfer.TransferService
|
||||||
|
if cfg.NAS.EnableTransfer {
|
||||||
|
ts, err := transfer.NewTrasferService(cfg.NAS.OutputPath, eventName)
|
||||||
|
if err != nil {
|
||||||
|
log.Printf("Failed to create transfer service: %v", err)
|
||||||
|
log.Println("Continuing without transfer service...")
|
||||||
|
appLog.Error("failed to create transfer service, continuing without it", logging.KeyError, err.Error())
|
||||||
|
} else {
|
||||||
|
transferService = ts
|
||||||
|
wg.Add(1)
|
||||||
|
go func() {
|
||||||
|
defer wg.Done()
|
||||||
|
if err := transferService.Start(ctx); err != nil && err != context.Canceled {
|
||||||
|
log.Printf("Transfer service error: %v", err)
|
||||||
|
appLog.Error("transfer service error", logging.KeyError, err.Error())
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
log.Println("Transfer service started.")
|
||||||
|
appLog.Info("transfer service started")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
manifestWriter := media.NewManifestWriter(eventName)
|
||||||
|
|
||||||
|
eventPath := cfg.GetEventPath(eventName)
|
||||||
|
if err := utils.EnsureDir(eventPath); err != nil {
|
||||||
|
appLog.Error("failed to create event directory", logging.KeyError, err.Error())
|
||||||
|
log.Fatalf("Failed to create event directory: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
variants, err := media.GetAllVariants(masterURL, eventPath, manifestWriter)
|
||||||
|
if err != nil {
|
||||||
|
appLog.Error("failed to get variants", logging.KeyError, err.Error())
|
||||||
|
log.Fatalf("Failed to get variants: %v", err)
|
||||||
|
}
|
||||||
|
log.Printf("Found %d variants", len(variants))
|
||||||
|
|
||||||
|
sem := make(chan struct{}, constants.WorkerCount*len(variants))
|
||||||
|
|
||||||
|
manifest := media.NewManifestWriter(eventName)
|
||||||
|
|
||||||
|
for _, variant := range variants {
|
||||||
|
// Debug mode only tracks one variant for easier debugging
|
||||||
|
if debug {
|
||||||
|
if variant.Resolution != "1080p" {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
}
|
||||||
|
wg.Add(1)
|
||||||
|
go func(v *media.StreamVariant) {
|
||||||
|
defer wg.Done()
|
||||||
|
media.VariantDownloader(ctx, v, sem, manifest)
|
||||||
|
}(variant)
|
||||||
|
}
|
||||||
|
|
||||||
|
wg.Wait()
|
||||||
|
log.Println("All variant downloaders finished.")
|
||||||
|
appLog.Info("all variant downloaders finished")
|
||||||
|
|
||||||
|
if transferService != nil {
|
||||||
|
shutdownCtx, shutdownCancel := context.WithTimeout(context.Background(), 30*time.Second)
|
||||||
|
defer shutdownCancel()
|
||||||
|
transferService.Shutdown(shutdownCtx)
|
||||||
|
}
|
||||||
|
|
||||||
|
log.Println("All Services shut down.")
|
||||||
|
appLog.Info("all services shut down")
|
||||||
|
|
||||||
|
manifestWriter.WriteManifest()
|
||||||
|
log.Println("Manifest written.")
|
||||||
|
appLog.Info("manifest written", logging.KeyEvent, eventName)
|
||||||
|
}
|
||||||
@ -1,75 +0,0 @@
|
|||||||
package main
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"log"
|
|
||||||
"m3u8-downloader/pkg/constants"
|
|
||||||
"m3u8-downloader/pkg/media"
|
|
||||||
"m3u8-downloader/pkg/transfer"
|
|
||||||
"os"
|
|
||||||
"os/signal"
|
|
||||||
"sync"
|
|
||||||
"syscall"
|
|
||||||
"time"
|
|
||||||
)
|
|
||||||
|
|
||||||
func main() {
|
|
||||||
ctx, cancel := context.WithCancel(context.Background())
|
|
||||||
defer cancel()
|
|
||||||
|
|
||||||
// Goroutine to listen for shutdown signals
|
|
||||||
sigChan := make(chan os.Signal, 1)
|
|
||||||
signal.Notify(sigChan, syscall.SIGINT, syscall.SIGTERM)
|
|
||||||
go func() {
|
|
||||||
<-sigChan
|
|
||||||
log.Println("Shutting down...")
|
|
||||||
cancel()
|
|
||||||
}()
|
|
||||||
|
|
||||||
var wg sync.WaitGroup
|
|
||||||
var transferService *transfer.TransferService
|
|
||||||
if constants.EnableNASTransfer {
|
|
||||||
ts, err := transfer.NewTrasferService(constants.NASPath)
|
|
||||||
if err != nil {
|
|
||||||
log.Printf("Failed to create transfer service: %v", err)
|
|
||||||
log.Println("Continuing without transfer service...")
|
|
||||||
} else {
|
|
||||||
transferService = ts
|
|
||||||
wg.Add(1)
|
|
||||||
go func() {
|
|
||||||
defer wg.Done()
|
|
||||||
if err := transferService.Start(ctx); err != nil && err != context.Canceled {
|
|
||||||
log.Printf("Transfer service error: %v", err)
|
|
||||||
}
|
|
||||||
}()
|
|
||||||
log.Println("Transfer service started.")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
variants, err := media.GetAllVariants(constants.MasterURL)
|
|
||||||
if err != nil {
|
|
||||||
log.Fatalf("Failed to get variants: %v", err)
|
|
||||||
}
|
|
||||||
log.Printf("Found %d variants", len(variants))
|
|
||||||
|
|
||||||
sem := make(chan struct{}, constants.WorkerCount*len(variants))
|
|
||||||
|
|
||||||
for _, variant := range variants {
|
|
||||||
wg.Add(1)
|
|
||||||
go func(v *media.StreamVariant) {
|
|
||||||
defer wg.Done()
|
|
||||||
media.VariantDownloader(ctx, v, sem)
|
|
||||||
}(variant)
|
|
||||||
}
|
|
||||||
|
|
||||||
wg.Wait()
|
|
||||||
log.Println("All variant downloaders finished.")
|
|
||||||
|
|
||||||
if transferService != nil {
|
|
||||||
shutdownCtx, shutdownCancel := context.WithTimeout(context.Background(), 30*time.Second)
|
|
||||||
defer shutdownCancel()
|
|
||||||
transferService.Shutdown(shutdownCtx)
|
|
||||||
}
|
|
||||||
|
|
||||||
log.Println("All Services shut down.")
|
|
||||||
}
|
|
||||||
53
cmd/main/main.go
Normal file
53
cmd/main/main.go
Normal file
@ -0,0 +1,53 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bufio"
|
||||||
|
"flag"
|
||||||
|
"fmt"
|
||||||
|
"m3u8-downloader/cmd/downloader"
|
||||||
|
"m3u8-downloader/cmd/processor"
|
||||||
|
"m3u8-downloader/cmd/transfer"
|
||||||
|
"m3u8-downloader/pkg/constants"
|
||||||
|
"m3u8-downloader/pkg/logging"
|
||||||
|
"os"
|
||||||
|
"strings"
|
||||||
|
)
|
||||||
|
|
||||||
|
func main() {
|
||||||
|
url := flag.String("url", "", "M3U8 playlist URL")
|
||||||
|
eventName := flag.String("event", "", "Event name")
|
||||||
|
debug := flag.Bool("debug", false, "Enable debug mode")
|
||||||
|
transferOnly := flag.Bool("transfer", false, "Transfer-only mode: transfer existing files without downloading")
|
||||||
|
processOnly := flag.Bool("process", false, "Process-only mode: process existing files without downloading")
|
||||||
|
|
||||||
|
flag.Parse()
|
||||||
|
|
||||||
|
// Must happen before the first logging.MustGetFactory() call anywhere in
|
||||||
|
// the program, since the logging factory reads its level from this
|
||||||
|
// config singleton once and caches it.
|
||||||
|
if *debug {
|
||||||
|
constants.MustGetConfig().Logging.Level = "debug"
|
||||||
|
}
|
||||||
|
logging.MustGetFactory().App.Info("m3u8-downloader starting", "debug", *debug)
|
||||||
|
|
||||||
|
if *transferOnly {
|
||||||
|
transfer.RunTransferOnly(*eventName)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
if *processOnly {
|
||||||
|
processor.Process(*eventName)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
if *url == "" {
|
||||||
|
reader := bufio.NewReader(os.Stdin)
|
||||||
|
fmt.Print("Enter M3U8 playlist URL: ")
|
||||||
|
inputUrl, _ := reader.ReadString('\n')
|
||||||
|
inputUrl = strings.TrimSpace(inputUrl)
|
||||||
|
downloader.Download(inputUrl, *eventName, *debug)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
downloader.Download(*url, *eventName, *debug)
|
||||||
|
}
|
||||||
@ -1 +0,0 @@
|
|||||||
package proc
|
|
||||||
42
cmd/processor/process.go
Normal file
42
cmd/processor/process.go
Normal file
@ -0,0 +1,42 @@
|
|||||||
|
package processor
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"log"
|
||||||
|
"m3u8-downloader/pkg/constants"
|
||||||
|
"m3u8-downloader/pkg/health"
|
||||||
|
"m3u8-downloader/pkg/logging"
|
||||||
|
"m3u8-downloader/pkg/processing"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
func Process(eventName string) {
|
||||||
|
appLog := logging.MustGetFactory().App
|
||||||
|
log.Printf("Starting processing for event: %s", eventName)
|
||||||
|
appLog.Info("starting processing", logging.KeyEvent, eventName)
|
||||||
|
cfg := constants.MustGetConfig()
|
||||||
|
|
||||||
|
if cfg.Health.Enabled {
|
||||||
|
healthSrv, err := health.Start(context.Background(), &cfg.Health, health.MustGetRegistry())
|
||||||
|
if err != nil {
|
||||||
|
appLog.Error("failed to start health server", logging.KeyError, err.Error())
|
||||||
|
} else {
|
||||||
|
appLog.Info("health server listening", "addr", cfg.Health.Addr)
|
||||||
|
defer func() {
|
||||||
|
shutdownCtx, shutdownCancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||||
|
defer shutdownCancel()
|
||||||
|
_ = healthSrv.Shutdown(shutdownCtx)
|
||||||
|
}()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
ps, err := processing.NewProcessingService(eventName, cfg)
|
||||||
|
if err != nil {
|
||||||
|
appLog.Error("failed to create processing service", logging.KeyError, err.Error())
|
||||||
|
log.Fatalf("Failed to create processing service: %v", err)
|
||||||
|
}
|
||||||
|
if err := ps.Start(context.Background()); err != nil {
|
||||||
|
appLog.Error("processing service failed", logging.KeyError, err.Error())
|
||||||
|
log.Fatalf("Failed to run processing service: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
139
cmd/transfer/transfer.go
Normal file
139
cmd/transfer/transfer.go
Normal file
@ -0,0 +1,139 @@
|
|||||||
|
package transfer
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bufio"
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"log"
|
||||||
|
"m3u8-downloader/pkg/config"
|
||||||
|
"m3u8-downloader/pkg/constants"
|
||||||
|
"m3u8-downloader/pkg/health"
|
||||||
|
"m3u8-downloader/pkg/logging"
|
||||||
|
"m3u8-downloader/pkg/transfer"
|
||||||
|
"m3u8-downloader/pkg/utils"
|
||||||
|
"os"
|
||||||
|
"os/signal"
|
||||||
|
"strconv"
|
||||||
|
"strings"
|
||||||
|
"syscall"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
func getEventDirs(cfg *config.Config) ([]string, error) {
|
||||||
|
dirs, err := os.ReadDir(cfg.Paths.LocalOutput)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("failed to read directory: %w", err)
|
||||||
|
}
|
||||||
|
var eventDirs []string
|
||||||
|
for _, dir := range dirs {
|
||||||
|
if dir.IsDir() {
|
||||||
|
eventDirs = append(eventDirs, dir.Name())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return eventDirs, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func RunTransferOnly(eventName string) {
|
||||||
|
appLog := logging.MustGetFactory().App
|
||||||
|
cfg := constants.MustGetConfig()
|
||||||
|
|
||||||
|
// Check if NAS transfer is enabled
|
||||||
|
if !cfg.NAS.EnableTransfer {
|
||||||
|
appLog.Error("NAS transfer disabled, cannot run transfer-only mode")
|
||||||
|
log.Fatal("NAS transfer is disabled in configuration. Please enable it to use transfer-only mode.")
|
||||||
|
}
|
||||||
|
|
||||||
|
if eventName == "" {
|
||||||
|
events, err := getEventDirs(cfg)
|
||||||
|
if err != nil {
|
||||||
|
log.Fatalf("Failed to get event directories: %v", err)
|
||||||
|
}
|
||||||
|
if len(events) == 0 {
|
||||||
|
log.Fatal("No events found")
|
||||||
|
}
|
||||||
|
if len(events) > 1 {
|
||||||
|
fmt.Println("Multiple events found, please select one:")
|
||||||
|
for i, event := range events {
|
||||||
|
fmt.Printf("%d. %s\n", i+1, event)
|
||||||
|
}
|
||||||
|
reader := bufio.NewReader(os.Stdin)
|
||||||
|
input, _ := reader.ReadString('\n')
|
||||||
|
input = strings.TrimSpace(input)
|
||||||
|
index, err := strconv.Atoi(input)
|
||||||
|
if err != nil {
|
||||||
|
log.Fatalf("Failed to parse input: %v", err)
|
||||||
|
}
|
||||||
|
if index < 1 || index > len(events) {
|
||||||
|
log.Fatal("Invalid input")
|
||||||
|
}
|
||||||
|
eventName = events[index-1]
|
||||||
|
} else {
|
||||||
|
eventName = events[0]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
log.Printf("Starting transfer-only mode for event: %s", eventName)
|
||||||
|
|
||||||
|
// Setup context and signal handling
|
||||||
|
ctx, cancel := context.WithCancel(context.Background())
|
||||||
|
defer cancel()
|
||||||
|
|
||||||
|
sigChan := make(chan os.Signal, 1)
|
||||||
|
signal.Notify(sigChan, syscall.SIGINT, syscall.SIGTERM)
|
||||||
|
go func() {
|
||||||
|
<-sigChan
|
||||||
|
log.Println("Shutting down transfer service...")
|
||||||
|
appLog.Info("shutdown signal received")
|
||||||
|
cancel()
|
||||||
|
}()
|
||||||
|
|
||||||
|
if cfg.Health.Enabled {
|
||||||
|
healthSrv, err := health.Start(ctx, &cfg.Health, health.MustGetRegistry())
|
||||||
|
if err != nil {
|
||||||
|
appLog.Error("failed to start health server", logging.KeyError, err.Error())
|
||||||
|
} else {
|
||||||
|
appLog.Info("health server listening", "addr", cfg.Health.Addr)
|
||||||
|
defer func() {
|
||||||
|
shutdownCtx, shutdownCancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||||
|
defer shutdownCancel()
|
||||||
|
_ = healthSrv.Shutdown(shutdownCtx)
|
||||||
|
}()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Verify local event directory exists
|
||||||
|
localEventPath := cfg.GetEventPath(eventName)
|
||||||
|
if !utils.PathExists(localEventPath) {
|
||||||
|
appLog.Error("local event directory does not exist", logging.KeyEvent, eventName)
|
||||||
|
log.Fatalf("Local event directory does not exist: %s", localEventPath)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Create transfer service
|
||||||
|
transferService, err := transfer.NewTrasferService(cfg.NAS.OutputPath, eventName)
|
||||||
|
if err != nil {
|
||||||
|
appLog.Error("failed to create transfer service", logging.KeyError, err.Error())
|
||||||
|
log.Fatalf("Failed to create transfer service: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Find and queue existing files
|
||||||
|
if err := transferService.QueueExistingFiles(localEventPath); err != nil {
|
||||||
|
appLog.Error("failed to queue existing files", logging.KeyError, err.Error())
|
||||||
|
log.Fatalf("Failed to queue existing files: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Start transfer service
|
||||||
|
log.Println("Starting transfer service...")
|
||||||
|
appLog.Info("starting transfer service", logging.KeyEvent, eventName)
|
||||||
|
if err := transferService.Start(ctx); err != nil && err != context.Canceled {
|
||||||
|
log.Printf("Transfer service error: %v", err)
|
||||||
|
appLog.Error("transfer service error", logging.KeyError, err.Error())
|
||||||
|
}
|
||||||
|
|
||||||
|
// Graceful shutdown
|
||||||
|
shutdownCtx, shutdownCancel := context.WithTimeout(context.Background(), 30*time.Second)
|
||||||
|
defer shutdownCancel()
|
||||||
|
transferService.Shutdown(shutdownCtx)
|
||||||
|
|
||||||
|
log.Println("Transfer-only mode completed.")
|
||||||
|
appLog.Info("transfer-only mode completed", logging.KeyEvent, eventName)
|
||||||
|
}
|
||||||
6
go.sum
Normal file
6
go.sum
Normal file
@ -0,0 +1,6 @@
|
|||||||
|
github.com/fsnotify/fsnotify v1.9.0 h1:2Ml+OJNzbYCTzsxtv8vKSFD9PbJjmhYF14k/jKC7S9k=
|
||||||
|
github.com/fsnotify/fsnotify v1.9.0/go.mod h1:8jBTzvmWwFyi3Pb8djgCCO5IBqzKJ/Jwo8TRcHyHii0=
|
||||||
|
github.com/grafov/m3u8 v0.12.1 h1:DuP1uA1kvRRmGNAZ0m+ObLv1dvrfNO0TPx0c/enNk0s=
|
||||||
|
github.com/grafov/m3u8 v0.12.1/go.mod h1:nqzOkfBiZJENr52zTVd/Dcl03yzphIMbJqkXGu+u080=
|
||||||
|
golang.org/x/sys v0.13.0 h1:Af8nKPmuFypiUBjVoU9V20FiaFXOcuZI21p0ycVYYGE=
|
||||||
|
golang.org/x/sys v0.13.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||||
278
pkg/config/config.go
Normal file
278
pkg/config/config.go
Normal file
@ -0,0 +1,278 @@
|
|||||||
|
package config
|
||||||
|
|
||||||
|
import (
|
||||||
|
"fmt"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strconv"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
type Config struct {
|
||||||
|
Core CoreConfig
|
||||||
|
HTTP HTTPConfig
|
||||||
|
NAS NASConfig
|
||||||
|
Processing ProcessingConfig
|
||||||
|
Transfer TransferConfig
|
||||||
|
Cleanup CleanupConfig
|
||||||
|
Paths PathsConfig
|
||||||
|
Logging LoggingConfig
|
||||||
|
Health HealthConfig
|
||||||
|
}
|
||||||
|
|
||||||
|
type CoreConfig struct {
|
||||||
|
WorkerCount int
|
||||||
|
RefreshDelay time.Duration
|
||||||
|
}
|
||||||
|
|
||||||
|
type HTTPConfig struct {
|
||||||
|
UserAgent string
|
||||||
|
Referer string
|
||||||
|
}
|
||||||
|
|
||||||
|
type NASConfig struct {
|
||||||
|
EnableTransfer bool
|
||||||
|
OutputPath string
|
||||||
|
Username string
|
||||||
|
Password string
|
||||||
|
Timeout time.Duration
|
||||||
|
RetryLimit int
|
||||||
|
}
|
||||||
|
|
||||||
|
type ProcessingConfig struct {
|
||||||
|
Enabled bool
|
||||||
|
AutoProcess bool
|
||||||
|
WorkerCount int
|
||||||
|
FFmpegPath string
|
||||||
|
}
|
||||||
|
|
||||||
|
type TransferConfig struct {
|
||||||
|
WorkerCount int
|
||||||
|
RetryLimit int
|
||||||
|
Timeout time.Duration
|
||||||
|
FileSettlingDelay time.Duration
|
||||||
|
QueueSize int
|
||||||
|
BatchSize int
|
||||||
|
}
|
||||||
|
|
||||||
|
type CleanupConfig struct {
|
||||||
|
AfterTransfer bool
|
||||||
|
BatchSize int
|
||||||
|
RetainHours int
|
||||||
|
}
|
||||||
|
|
||||||
|
type PathsConfig struct {
|
||||||
|
BaseDir string
|
||||||
|
LocalOutput string
|
||||||
|
ProcessOutput string
|
||||||
|
ManifestDir string
|
||||||
|
PersistenceFile string
|
||||||
|
}
|
||||||
|
|
||||||
|
type LoggingConfig struct {
|
||||||
|
Dir string
|
||||||
|
Level string
|
||||||
|
}
|
||||||
|
|
||||||
|
type HealthConfig struct {
|
||||||
|
Enabled bool
|
||||||
|
Addr string
|
||||||
|
}
|
||||||
|
|
||||||
|
var defaultConfig = Config{
|
||||||
|
Core: CoreConfig{
|
||||||
|
WorkerCount: 4,
|
||||||
|
RefreshDelay: 3 * time.Second,
|
||||||
|
},
|
||||||
|
HTTP: HTTPConfig{
|
||||||
|
UserAgent: "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36",
|
||||||
|
Referer: "https://www.flomarching.com",
|
||||||
|
},
|
||||||
|
NAS: NASConfig{
|
||||||
|
EnableTransfer: true,
|
||||||
|
OutputPath: "\\\\HomeLabNAS\\dci\\streams",
|
||||||
|
Username: "NASAdmin",
|
||||||
|
Password: "s3tkY6tzA&KN6M",
|
||||||
|
Timeout: 30 * time.Second,
|
||||||
|
RetryLimit: 3,
|
||||||
|
},
|
||||||
|
Processing: ProcessingConfig{
|
||||||
|
Enabled: true,
|
||||||
|
AutoProcess: true,
|
||||||
|
WorkerCount: 2,
|
||||||
|
FFmpegPath: "ffmpeg",
|
||||||
|
},
|
||||||
|
Transfer: TransferConfig{
|
||||||
|
WorkerCount: 2,
|
||||||
|
RetryLimit: 3,
|
||||||
|
Timeout: 30 * time.Second,
|
||||||
|
FileSettlingDelay: 5 * time.Second,
|
||||||
|
QueueSize: 100000,
|
||||||
|
BatchSize: 1000,
|
||||||
|
},
|
||||||
|
Cleanup: CleanupConfig{
|
||||||
|
AfterTransfer: true,
|
||||||
|
BatchSize: 1000,
|
||||||
|
RetainHours: 0,
|
||||||
|
},
|
||||||
|
Paths: PathsConfig{
|
||||||
|
BaseDir: "data",
|
||||||
|
LocalOutput: "data",
|
||||||
|
ProcessOutput: "out",
|
||||||
|
ManifestDir: "data",
|
||||||
|
PersistenceFile: "transfer_queue.json",
|
||||||
|
},
|
||||||
|
Logging: LoggingConfig{
|
||||||
|
Dir: "data/logs",
|
||||||
|
Level: "info",
|
||||||
|
},
|
||||||
|
Health: HealthConfig{
|
||||||
|
Enabled: true,
|
||||||
|
Addr: ":8090",
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
func Load() (*Config, error) {
|
||||||
|
cfg := defaultConfig
|
||||||
|
|
||||||
|
if err := cfg.loadFromEnvironment(); err != nil {
|
||||||
|
return nil, fmt.Errorf("failed to load environment config: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := cfg.resolveAndValidatePaths(); err != nil {
|
||||||
|
return nil, fmt.Errorf("path validation failed: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
return &cfg, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *Config) loadFromEnvironment() error {
|
||||||
|
if val := os.Getenv("WORKER_COUNT"); val != "" {
|
||||||
|
if parsed, err := strconv.Atoi(val); err == nil {
|
||||||
|
c.Core.WorkerCount = parsed
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if val := os.Getenv("REFRESH_DELAY_SECONDS"); val != "" {
|
||||||
|
if parsed, err := strconv.Atoi(val); err == nil {
|
||||||
|
c.Core.RefreshDelay = time.Duration(parsed) * time.Second
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if val := os.Getenv("NAS_OUTPUT_PATH"); val != "" {
|
||||||
|
c.NAS.OutputPath = val
|
||||||
|
}
|
||||||
|
|
||||||
|
if val := os.Getenv("NAS_USERNAME"); val != "" {
|
||||||
|
c.NAS.Username = val
|
||||||
|
}
|
||||||
|
|
||||||
|
if val := os.Getenv("NAS_PASSWORD"); val != "" {
|
||||||
|
c.NAS.Password = val
|
||||||
|
}
|
||||||
|
|
||||||
|
if val := os.Getenv("ENABLE_NAS_TRANSFER"); val != "" {
|
||||||
|
c.NAS.EnableTransfer = val == "true"
|
||||||
|
}
|
||||||
|
|
||||||
|
if val := os.Getenv("LOCAL_OUTPUT_DIR"); val != "" {
|
||||||
|
c.Paths.LocalOutput = val
|
||||||
|
}
|
||||||
|
|
||||||
|
if val := os.Getenv("PROCESS_OUTPUT_DIR"); val != "" {
|
||||||
|
c.Paths.ProcessOutput = val
|
||||||
|
}
|
||||||
|
|
||||||
|
if val := os.Getenv("FFMPEG_PATH"); val != "" {
|
||||||
|
c.Processing.FFmpegPath = val
|
||||||
|
}
|
||||||
|
|
||||||
|
if val := os.Getenv("LOG_DIR"); val != "" {
|
||||||
|
c.Logging.Dir = val
|
||||||
|
}
|
||||||
|
|
||||||
|
if val := os.Getenv("LOG_LEVEL"); val != "" {
|
||||||
|
c.Logging.Level = val
|
||||||
|
}
|
||||||
|
|
||||||
|
if val := os.Getenv("HEALTH_ENABLED"); val != "" {
|
||||||
|
c.Health.Enabled = val == "true"
|
||||||
|
}
|
||||||
|
|
||||||
|
if val := os.Getenv("HEALTH_ADDR"); val != "" {
|
||||||
|
c.Health.Addr = val
|
||||||
|
}
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *Config) resolveAndValidatePaths() error {
|
||||||
|
cwd, err := os.Getwd()
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("failed to get working directory: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Only join with cwd if path is not already absolute
|
||||||
|
if !filepath.IsAbs(c.Paths.BaseDir) {
|
||||||
|
c.Paths.BaseDir = filepath.Join(cwd, c.Paths.BaseDir)
|
||||||
|
}
|
||||||
|
if !filepath.IsAbs(c.Paths.LocalOutput) {
|
||||||
|
c.Paths.LocalOutput = filepath.Join(cwd, c.Paths.LocalOutput)
|
||||||
|
}
|
||||||
|
if !filepath.IsAbs(c.Paths.ProcessOutput) {
|
||||||
|
c.Paths.ProcessOutput = filepath.Join(cwd, c.Paths.ProcessOutput)
|
||||||
|
}
|
||||||
|
if !filepath.IsAbs(c.Paths.ManifestDir) {
|
||||||
|
c.Paths.ManifestDir = filepath.Join(cwd, c.Paths.ManifestDir)
|
||||||
|
}
|
||||||
|
if !filepath.IsAbs(c.Paths.PersistenceFile) {
|
||||||
|
c.Paths.PersistenceFile = filepath.Join(c.Paths.BaseDir, c.Paths.PersistenceFile)
|
||||||
|
}
|
||||||
|
if !filepath.IsAbs(c.Logging.Dir) {
|
||||||
|
c.Logging.Dir = filepath.Join(cwd, c.Logging.Dir)
|
||||||
|
}
|
||||||
|
|
||||||
|
requiredDirs := []string{
|
||||||
|
c.Paths.BaseDir,
|
||||||
|
c.Paths.LocalOutput,
|
||||||
|
c.Paths.ProcessOutput,
|
||||||
|
c.Paths.ManifestDir,
|
||||||
|
c.Logging.Dir,
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, dir := range requiredDirs {
|
||||||
|
if err := os.MkdirAll(dir, 0755); err != nil {
|
||||||
|
return fmt.Errorf("failed to create directory %s: %w", dir, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if c.NAS.EnableTransfer && c.NAS.OutputPath == "" {
|
||||||
|
return fmt.Errorf("NAS output path is required when transfer is enabled")
|
||||||
|
}
|
||||||
|
|
||||||
|
if c.Processing.Enabled && c.Processing.FFmpegPath == "" {
|
||||||
|
return fmt.Errorf("FFmpeg path is required when processing is enabled")
|
||||||
|
}
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *Config) GetEventPath(eventName string) string {
|
||||||
|
return filepath.Join(c.Paths.LocalOutput, eventName)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *Config) GetManifestPath(eventName string) string {
|
||||||
|
return filepath.Join(c.Paths.ManifestDir, eventName+".json")
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *Config) GetNASEventPath(eventName string) string {
|
||||||
|
return filepath.Join(c.NAS.OutputPath, eventName)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *Config) GetProcessOutputPath(eventName string) string {
|
||||||
|
return filepath.Join(c.Paths.ProcessOutput, eventName)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *Config) GetQualityPath(eventName, quality string) string {
|
||||||
|
return filepath.Join(c.GetEventPath(eventName), quality)
|
||||||
|
}
|
||||||
181
pkg/config/config_test.go
Normal file
181
pkg/config/config_test.go
Normal file
@ -0,0 +1,181 @@
|
|||||||
|
package config
|
||||||
|
|
||||||
|
import (
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestConfig_Load(t *testing.T) {
|
||||||
|
// Save original env vars
|
||||||
|
originalVars := map[string]string{
|
||||||
|
"WORKER_COUNT": os.Getenv("WORKER_COUNT"),
|
||||||
|
"NAS_USERNAME": os.Getenv("NAS_USERNAME"),
|
||||||
|
"LOCAL_OUTPUT_DIR": os.Getenv("LOCAL_OUTPUT_DIR"),
|
||||||
|
"ENABLE_NAS_TRANSFER": os.Getenv("ENABLE_NAS_TRANSFER"),
|
||||||
|
}
|
||||||
|
defer func() {
|
||||||
|
// Restore original env vars
|
||||||
|
for key, value := range originalVars {
|
||||||
|
if value == "" {
|
||||||
|
os.Unsetenv(key)
|
||||||
|
} else {
|
||||||
|
os.Setenv(key, value)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
|
||||||
|
// Test default config load
|
||||||
|
cfg, err := Load()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Load() failed: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Verify defaults
|
||||||
|
if cfg.Core.WorkerCount != 4 {
|
||||||
|
t.Errorf("Expected WorkerCount=4, got %d", cfg.Core.WorkerCount)
|
||||||
|
}
|
||||||
|
if cfg.Core.RefreshDelay != 3*time.Second {
|
||||||
|
t.Errorf("Expected RefreshDelay=3s, got %v", cfg.Core.RefreshDelay)
|
||||||
|
}
|
||||||
|
if !cfg.NAS.EnableTransfer {
|
||||||
|
t.Errorf("Expected NAS.EnableTransfer=true, got false")
|
||||||
|
}
|
||||||
|
|
||||||
|
// Test environment variable override
|
||||||
|
os.Setenv("WORKER_COUNT", "8")
|
||||||
|
os.Setenv("NAS_USERNAME", "testuser")
|
||||||
|
os.Setenv("ENABLE_NAS_TRANSFER", "false")
|
||||||
|
os.Setenv("LOCAL_OUTPUT_DIR", "custom_data")
|
||||||
|
|
||||||
|
cfg2, err := Load()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Load() with env vars failed: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if cfg2.Core.WorkerCount != 8 {
|
||||||
|
t.Errorf("Expected WorkerCount=8 from env, got %d", cfg2.Core.WorkerCount)
|
||||||
|
}
|
||||||
|
if cfg2.NAS.Username != "testuser" {
|
||||||
|
t.Errorf("Expected NAS.Username='testuser' from env, got %s", cfg2.NAS.Username)
|
||||||
|
}
|
||||||
|
if cfg2.NAS.EnableTransfer {
|
||||||
|
t.Errorf("Expected NAS.EnableTransfer=false from env, got true")
|
||||||
|
}
|
||||||
|
if !strings.Contains(cfg2.Paths.LocalOutput, "custom_data") {
|
||||||
|
t.Errorf("Expected LocalOutput to contain 'custom_data', got %s", cfg2.Paths.LocalOutput)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestConfig_PathMethods(t *testing.T) {
|
||||||
|
cfg, err := Load()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Load() failed: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
testEvent := "test-event"
|
||||||
|
testQuality := "1080p"
|
||||||
|
|
||||||
|
// Test GetEventPath
|
||||||
|
eventPath := cfg.GetEventPath(testEvent)
|
||||||
|
if !strings.Contains(eventPath, testEvent) {
|
||||||
|
t.Errorf("GetEventPath should contain event name, got %s", eventPath)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Test GetManifestPath
|
||||||
|
manifestPath := cfg.GetManifestPath(testEvent)
|
||||||
|
if !strings.Contains(manifestPath, testEvent) {
|
||||||
|
t.Errorf("GetManifestPath should contain event name, got %s", manifestPath)
|
||||||
|
}
|
||||||
|
if !strings.HasSuffix(manifestPath, ".json") {
|
||||||
|
t.Errorf("GetManifestPath should end with .json, got %s", manifestPath)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Test GetNASEventPath
|
||||||
|
nasPath := cfg.GetNASEventPath(testEvent)
|
||||||
|
if !strings.Contains(nasPath, testEvent) {
|
||||||
|
t.Errorf("GetNASEventPath should contain event name, got %s", nasPath)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Test GetProcessOutputPath
|
||||||
|
processPath := cfg.GetProcessOutputPath(testEvent)
|
||||||
|
if !strings.Contains(processPath, testEvent) {
|
||||||
|
t.Errorf("GetProcessOutputPath should contain event name, got %s", processPath)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Test GetQualityPath
|
||||||
|
qualityPath := cfg.GetQualityPath(testEvent, testQuality)
|
||||||
|
if !strings.Contains(qualityPath, testEvent) {
|
||||||
|
t.Errorf("GetQualityPath should contain event name, got %s", qualityPath)
|
||||||
|
}
|
||||||
|
if !strings.Contains(qualityPath, testQuality) {
|
||||||
|
t.Errorf("GetQualityPath should contain quality, got %s", qualityPath)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestConfig_PathValidation(t *testing.T) {
|
||||||
|
// Create a temporary directory for testing
|
||||||
|
tempDir, err := os.MkdirTemp("", "config_test_*")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Failed to create temp dir: %v", err)
|
||||||
|
}
|
||||||
|
defer os.RemoveAll(tempDir)
|
||||||
|
|
||||||
|
// Set environment variables to use temp directory
|
||||||
|
os.Setenv("LOCAL_OUTPUT_DIR", filepath.Join(tempDir, "data"))
|
||||||
|
defer os.Unsetenv("LOCAL_OUTPUT_DIR")
|
||||||
|
|
||||||
|
cfg, err := Load()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Load() failed: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Verify directories were created
|
||||||
|
if _, err := os.Stat(cfg.Paths.LocalOutput); os.IsNotExist(err) {
|
||||||
|
t.Errorf("LocalOutput directory should have been created: %s", cfg.Paths.LocalOutput)
|
||||||
|
}
|
||||||
|
if _, err := os.Stat(cfg.Paths.ProcessOutput); os.IsNotExist(err) {
|
||||||
|
t.Errorf("ProcessOutput directory should have been created: %s", cfg.Paths.ProcessOutput)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestConfig_ValidationErrors(t *testing.T) {
|
||||||
|
// Save original env vars
|
||||||
|
originalNASPath := os.Getenv("NAS_OUTPUT_PATH")
|
||||||
|
originalFFmpegPath := os.Getenv("FFMPEG_PATH")
|
||||||
|
defer func() {
|
||||||
|
if originalNASPath == "" {
|
||||||
|
os.Unsetenv("NAS_OUTPUT_PATH")
|
||||||
|
} else {
|
||||||
|
os.Setenv("NAS_OUTPUT_PATH", originalNASPath)
|
||||||
|
}
|
||||||
|
if originalFFmpegPath == "" {
|
||||||
|
os.Unsetenv("FFMPEG_PATH")
|
||||||
|
} else {
|
||||||
|
os.Setenv("FFMPEG_PATH", originalFFmpegPath)
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
|
||||||
|
// Note: Validation tests are limited because the default config
|
||||||
|
// has working defaults. We can test that Load() works with valid configs.
|
||||||
|
|
||||||
|
// Test that Load works with proper paths set
|
||||||
|
tempDir2, err := os.MkdirTemp("", "config_validation_test_*")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Failed to create temp dir: %v", err)
|
||||||
|
}
|
||||||
|
defer os.RemoveAll(tempDir2)
|
||||||
|
|
||||||
|
os.Setenv("NAS_OUTPUT_PATH", "\\\\test\\path")
|
||||||
|
os.Setenv("LOCAL_OUTPUT_DIR", tempDir2)
|
||||||
|
|
||||||
|
cfg, err := Load()
|
||||||
|
if err != nil {
|
||||||
|
t.Errorf("Load() should work with valid config: %v", err)
|
||||||
|
}
|
||||||
|
if cfg == nil {
|
||||||
|
t.Error("Config should not be nil")
|
||||||
|
}
|
||||||
|
}
|
||||||
@ -1,29 +1,51 @@
|
|||||||
package constants
|
package constants
|
||||||
|
|
||||||
import "time"
|
import (
|
||||||
|
"m3u8-downloader/pkg/config"
|
||||||
|
"sync"
|
||||||
|
)
|
||||||
|
|
||||||
|
var (
|
||||||
|
globalConfig *config.Config
|
||||||
|
configOnce sync.Once
|
||||||
|
configError error
|
||||||
|
)
|
||||||
|
|
||||||
|
func GetConfig() (*config.Config, error) {
|
||||||
|
configOnce.Do(func() {
|
||||||
|
globalConfig, configError = config.Load()
|
||||||
|
})
|
||||||
|
return globalConfig, configError
|
||||||
|
}
|
||||||
|
|
||||||
|
func MustGetConfig() *config.Config {
|
||||||
|
cfg, err := GetConfig()
|
||||||
|
if err != nil {
|
||||||
|
panic("Failed to load configuration: " + err.Error())
|
||||||
|
}
|
||||||
|
return cfg
|
||||||
|
}
|
||||||
|
|
||||||
const (
|
const (
|
||||||
MasterURL = "https://live-fastly.flosports.tv/streams/mr159021-260419/playlist.m3u8?token=st%3D1753571418%7Eexp%3D1753571448%7Eacl%3D%2Fstreams%2Fmr159021-260419%2Fplaylist.m3u8%7Edata%3Dssai%3A0%3BuserId%3A14025903%3BstreamId%3A260419%3BmediaPackageRegion%3Afalse%3BdvrMinutes%3A360%3BtokenId%3Abadd289a-ade5-48fe-852f-7dbd1d57aca8%3Bpv%3A86400%7Ehmac2%3D8de65c26b185084a6be77e788cb0ba41be5fcac3ab86159b06f7572ca925d77ba7bd182124af2a432953d4223548f198742d1a238e937d875976cd42fe549838&mid_origin=media_store&keyName=FLOSPORTS_TOKEN_KEY_2023-08-02&streamCode=mr159021-260419"
|
|
||||||
WorkerCount = 4
|
WorkerCount = 4
|
||||||
RefreshDelay = 3 * time.Second
|
RefreshDelay = 3
|
||||||
|
|
||||||
HTTPUserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36"
|
HTTPUserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36"
|
||||||
REFERRER = "https://www.flomarching.com"
|
REFERRER = "https://www.flomarching.com"
|
||||||
OutputDirPath = "./data/flo_radio"
|
|
||||||
|
|
||||||
EnableNASTransfer = true
|
DefaultNASOutputPath = "\\\\HomeLabNAS\\dci\\streams"
|
||||||
NASPath = "\\\\HomeLabNAS\\dci\\streams\\2025_Atlanta"
|
DefaultNASUsername = "NASAdmin"
|
||||||
NASUsername = ""
|
|
||||||
NASPassword = ""
|
|
||||||
TransferWorkerCount = 2
|
|
||||||
TransferRetryLimit = 3
|
|
||||||
TransferTimeout = 30 * time.Second
|
|
||||||
FileSettlingDelay = 5 * time.Second
|
|
||||||
PersistencePath = "./data/transfer_queue.json"
|
|
||||||
TransferQueueSize = 1000
|
|
||||||
BatchSize = 10
|
|
||||||
|
|
||||||
CleanupAfterTransfer = true
|
DefaultTransferWorkerCount = 2
|
||||||
CleanupBatchSize = 10
|
DefaultTransferRetryLimit = 3
|
||||||
RetainLocalHours = 0
|
DefaultTransferTimeout = 30
|
||||||
|
DefaultFileSettlingDelay = 5
|
||||||
|
DefaultTransferQueueSize = 100000
|
||||||
|
DefaultBatchSize = 1000
|
||||||
|
|
||||||
|
DefaultCleanupBatchSize = 1000
|
||||||
|
DefaultRetainLocalHours = 0
|
||||||
|
|
||||||
|
DefaultProcessWorkerCount = 2
|
||||||
|
DefaultFFmpegPath = "ffmpeg"
|
||||||
)
|
)
|
||||||
|
|||||||
240
pkg/constants/constants_test.go
Normal file
240
pkg/constants/constants_test.go
Normal file
@ -0,0 +1,240 @@
|
|||||||
|
package constants
|
||||||
|
|
||||||
|
import (
|
||||||
|
"os"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestGetConfig(t *testing.T) {
|
||||||
|
// Test successful config loading
|
||||||
|
cfg, err := GetConfig()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("GetConfig() failed: %v", err)
|
||||||
|
}
|
||||||
|
if cfg == nil {
|
||||||
|
t.Fatal("GetConfig() returned nil config")
|
||||||
|
}
|
||||||
|
|
||||||
|
// Test that subsequent calls return the same instance (singleton)
|
||||||
|
cfg2, err := GetConfig()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Second GetConfig() call failed: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Both should be the same instance due to sync.Once
|
||||||
|
if cfg != cfg2 {
|
||||||
|
t.Error("GetConfig() should return the same instance (singleton)")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestMustGetConfig(t *testing.T) {
|
||||||
|
// This should not panic with valid environment
|
||||||
|
cfg := MustGetConfig()
|
||||||
|
if cfg == nil {
|
||||||
|
t.Fatal("MustGetConfig() returned nil")
|
||||||
|
}
|
||||||
|
|
||||||
|
// Verify it returns a properly initialized config
|
||||||
|
if cfg.Core.WorkerCount <= 0 {
|
||||||
|
t.Errorf("Expected positive WorkerCount, got %d", cfg.Core.WorkerCount)
|
||||||
|
}
|
||||||
|
if cfg.Core.RefreshDelay <= 0 {
|
||||||
|
t.Errorf("Expected positive RefreshDelay, got %v", cfg.Core.RefreshDelay)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestMustGetConfig_Panic(t *testing.T) {
|
||||||
|
// We can't easily test the panic scenario without breaking the singleton,
|
||||||
|
// but we can test that MustGetConfig works normally
|
||||||
|
defer func() {
|
||||||
|
if r := recover(); r != nil {
|
||||||
|
t.Errorf("MustGetConfig() panicked unexpectedly: %v", r)
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
|
||||||
|
cfg := MustGetConfig()
|
||||||
|
if cfg == nil {
|
||||||
|
t.Fatal("MustGetConfig() returned nil without panicking")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestConfigSingleton(t *testing.T) {
|
||||||
|
// Reset the singleton for this test (this is a bit hacky but necessary for testing)
|
||||||
|
// We'll create multiple goroutines to test concurrent access
|
||||||
|
|
||||||
|
configs := make(chan interface{}, 10)
|
||||||
|
|
||||||
|
// Launch multiple goroutines to call GetConfig concurrently
|
||||||
|
for i := 0; i < 10; i++ {
|
||||||
|
go func() {
|
||||||
|
cfg, _ := GetConfig()
|
||||||
|
configs <- cfg
|
||||||
|
}()
|
||||||
|
}
|
||||||
|
|
||||||
|
// Collect all configs
|
||||||
|
var allConfigs []interface{}
|
||||||
|
for i := 0; i < 10; i++ {
|
||||||
|
allConfigs = append(allConfigs, <-configs)
|
||||||
|
}
|
||||||
|
|
||||||
|
// All should be the same instance
|
||||||
|
firstConfig := allConfigs[0]
|
||||||
|
for i, cfg := range allConfigs {
|
||||||
|
if cfg != firstConfig {
|
||||||
|
t.Errorf("Config %d is different from first config", i)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestConstants_Values(t *testing.T) {
|
||||||
|
// Test that constants have expected values
|
||||||
|
if WorkerCount != 4 {
|
||||||
|
t.Errorf("Expected WorkerCount=4, got %d", WorkerCount)
|
||||||
|
}
|
||||||
|
if RefreshDelay != 3 {
|
||||||
|
t.Errorf("Expected RefreshDelay=3, got %d", RefreshDelay)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Test HTTP constants
|
||||||
|
if HTTPUserAgent == "" {
|
||||||
|
t.Error("HTTPUserAgent should not be empty")
|
||||||
|
}
|
||||||
|
if !strings.Contains(HTTPUserAgent, "Mozilla") {
|
||||||
|
t.Error("HTTPUserAgent should contain 'Mozilla'")
|
||||||
|
}
|
||||||
|
if REFERRER != "https://www.flomarching.com" {
|
||||||
|
t.Errorf("Expected REFERRER='https://www.flomarching.com', got '%s'", REFERRER)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Test default NAS constants
|
||||||
|
if DefaultNASOutputPath != "\\\\HomeLabNAS\\dci\\streams" {
|
||||||
|
t.Errorf("Expected DefaultNASOutputPath='\\\\HomeLabNAS\\dci\\streams', got '%s'", DefaultNASOutputPath)
|
||||||
|
}
|
||||||
|
if DefaultNASUsername != "NASAdmin" {
|
||||||
|
t.Errorf("Expected DefaultNASUsername='NASAdmin', got '%s'", DefaultNASUsername)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Test transfer constants
|
||||||
|
if DefaultTransferWorkerCount != 2 {
|
||||||
|
t.Errorf("Expected DefaultTransferWorkerCount=2, got %d", DefaultTransferWorkerCount)
|
||||||
|
}
|
||||||
|
if DefaultTransferRetryLimit != 3 {
|
||||||
|
t.Errorf("Expected DefaultTransferRetryLimit=3, got %d", DefaultTransferRetryLimit)
|
||||||
|
}
|
||||||
|
if DefaultTransferTimeout != 30 {
|
||||||
|
t.Errorf("Expected DefaultTransferTimeout=30, got %d", DefaultTransferTimeout)
|
||||||
|
}
|
||||||
|
if DefaultFileSettlingDelay != 5 {
|
||||||
|
t.Errorf("Expected DefaultFileSettlingDelay=5, got %d", DefaultFileSettlingDelay)
|
||||||
|
}
|
||||||
|
if DefaultTransferQueueSize != 100000 {
|
||||||
|
t.Errorf("Expected DefaultTransferQueueSize=100000, got %d", DefaultTransferQueueSize)
|
||||||
|
}
|
||||||
|
if DefaultBatchSize != 1000 {
|
||||||
|
t.Errorf("Expected DefaultBatchSize=1000, got %d", DefaultBatchSize)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Test cleanup constants
|
||||||
|
if DefaultCleanupBatchSize != 1000 {
|
||||||
|
t.Errorf("Expected DefaultCleanupBatchSize=1000, got %d", DefaultCleanupBatchSize)
|
||||||
|
}
|
||||||
|
if DefaultRetainLocalHours != 0 {
|
||||||
|
t.Errorf("Expected DefaultRetainLocalHours=0, got %d", DefaultRetainLocalHours)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Test processing constants
|
||||||
|
if DefaultProcessWorkerCount != 2 {
|
||||||
|
t.Errorf("Expected DefaultProcessWorkerCount=2, got %d", DefaultProcessWorkerCount)
|
||||||
|
}
|
||||||
|
if DefaultFFmpegPath != "ffmpeg" {
|
||||||
|
t.Errorf("Expected DefaultFFmpegPath='ffmpeg', got '%s'", DefaultFFmpegPath)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestConfig_Integration(t *testing.T) {
|
||||||
|
cfg := MustGetConfig()
|
||||||
|
|
||||||
|
// Test that config values match or override constants appropriately
|
||||||
|
if cfg.Core.WorkerCount != WorkerCount && os.Getenv("WORKER_COUNT") == "" {
|
||||||
|
t.Errorf("Config WorkerCount (%d) should match constant (%d) when no env override", cfg.Core.WorkerCount, WorkerCount)
|
||||||
|
}
|
||||||
|
|
||||||
|
if cfg.Core.RefreshDelay != time.Duration(RefreshDelay)*time.Second && os.Getenv("REFRESH_DELAY_SECONDS") == "" {
|
||||||
|
t.Errorf("Config RefreshDelay (%v) should match constant (%v) when no env override", cfg.Core.RefreshDelay, time.Duration(RefreshDelay)*time.Second)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Test HTTP settings
|
||||||
|
if cfg.HTTP.UserAgent != HTTPUserAgent {
|
||||||
|
t.Errorf("Config UserAgent (%s) should match constant (%s)", cfg.HTTP.UserAgent, HTTPUserAgent)
|
||||||
|
}
|
||||||
|
if cfg.HTTP.Referer != REFERRER {
|
||||||
|
t.Errorf("Config Referer (%s) should match constant (%s)", cfg.HTTP.Referer, REFERRER)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestConfig_PathMethods(t *testing.T) {
|
||||||
|
cfg := MustGetConfig()
|
||||||
|
|
||||||
|
testEvent := "test-event-123"
|
||||||
|
testQuality := "1080p"
|
||||||
|
|
||||||
|
// Test GetEventPath
|
||||||
|
eventPath := cfg.GetEventPath(testEvent)
|
||||||
|
if !strings.Contains(eventPath, testEvent) {
|
||||||
|
t.Errorf("GetEventPath should contain event name '%s', got: %s", testEvent, eventPath)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Test GetManifestPath
|
||||||
|
manifestPath := cfg.GetManifestPath(testEvent)
|
||||||
|
if !strings.Contains(manifestPath, testEvent) {
|
||||||
|
t.Errorf("GetManifestPath should contain event name '%s', got: %s", testEvent, manifestPath)
|
||||||
|
}
|
||||||
|
if !strings.HasSuffix(manifestPath, ".json") {
|
||||||
|
t.Errorf("GetManifestPath should end with '.json', got: %s", manifestPath)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Test GetNASEventPath
|
||||||
|
nasPath := cfg.GetNASEventPath(testEvent)
|
||||||
|
if !strings.Contains(nasPath, testEvent) {
|
||||||
|
t.Errorf("GetNASEventPath should contain event name '%s', got: %s", testEvent, nasPath)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Test GetProcessOutputPath
|
||||||
|
processPath := cfg.GetProcessOutputPath(testEvent)
|
||||||
|
if !strings.Contains(processPath, testEvent) {
|
||||||
|
t.Errorf("GetProcessOutputPath should contain event name '%s', got: %s", testEvent, processPath)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Test GetQualityPath
|
||||||
|
qualityPath := cfg.GetQualityPath(testEvent, testQuality)
|
||||||
|
if !strings.Contains(qualityPath, testEvent) {
|
||||||
|
t.Errorf("GetQualityPath should contain event name '%s', got: %s", testEvent, qualityPath)
|
||||||
|
}
|
||||||
|
if !strings.Contains(qualityPath, testQuality) {
|
||||||
|
t.Errorf("GetQualityPath should contain quality '%s', got: %s", testQuality, qualityPath)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestConfig_DefaultValues(t *testing.T) {
|
||||||
|
cfg := MustGetConfig()
|
||||||
|
|
||||||
|
// Test that default values are reasonable
|
||||||
|
if cfg.Transfer.QueueSize != DefaultTransferQueueSize {
|
||||||
|
t.Errorf("Expected transfer queue size %d, got %d", DefaultTransferQueueSize, cfg.Transfer.QueueSize)
|
||||||
|
}
|
||||||
|
|
||||||
|
if cfg.Transfer.BatchSize != DefaultBatchSize {
|
||||||
|
t.Errorf("Expected transfer batch size %d, got %d", DefaultBatchSize, cfg.Transfer.BatchSize)
|
||||||
|
}
|
||||||
|
|
||||||
|
if cfg.Processing.WorkerCount != DefaultProcessWorkerCount {
|
||||||
|
t.Errorf("Expected processing worker count %d, got %d", DefaultProcessWorkerCount, cfg.Processing.WorkerCount)
|
||||||
|
}
|
||||||
|
|
||||||
|
if cfg.Cleanup.BatchSize != DefaultCleanupBatchSize {
|
||||||
|
t.Errorf("Expected cleanup batch size %d, got %d", DefaultCleanupBatchSize, cfg.Cleanup.BatchSize)
|
||||||
|
}
|
||||||
|
}
|
||||||
323
pkg/health/dashboard.html
Normal file
323
pkg/health/dashboard.html
Normal file
@ -0,0 +1,323 @@
|
|||||||
|
<!DOCTYPE html>
|
||||||
|
<html lang="en">
|
||||||
|
<head>
|
||||||
|
<meta charset="UTF-8">
|
||||||
|
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||||
|
<title>m3u8-downloader status</title>
|
||||||
|
<style>
|
||||||
|
:root {
|
||||||
|
--bg: #0f1115;
|
||||||
|
--panel: #161922;
|
||||||
|
--border: #262b38;
|
||||||
|
--text: #e6e8ee;
|
||||||
|
--muted: #8b909e;
|
||||||
|
--green: #3ecf8e;
|
||||||
|
--red: #f2545b;
|
||||||
|
--amber: #f2b705;
|
||||||
|
--blue: #4f8ef7;
|
||||||
|
}
|
||||||
|
* { box-sizing: border-box; }
|
||||||
|
body {
|
||||||
|
margin: 0;
|
||||||
|
font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif;
|
||||||
|
background: var(--bg);
|
||||||
|
color: var(--text);
|
||||||
|
}
|
||||||
|
header {
|
||||||
|
display: flex;
|
||||||
|
align-items: center;
|
||||||
|
justify-content: space-between;
|
||||||
|
flex-wrap: wrap;
|
||||||
|
gap: 12px;
|
||||||
|
padding: 16px 24px;
|
||||||
|
border-bottom: 1px solid var(--border);
|
||||||
|
}
|
||||||
|
header h1 {
|
||||||
|
font-size: 18px;
|
||||||
|
margin: 0;
|
||||||
|
font-weight: 600;
|
||||||
|
}
|
||||||
|
.header-meta {
|
||||||
|
display: flex;
|
||||||
|
align-items: center;
|
||||||
|
gap: 16px;
|
||||||
|
color: var(--muted);
|
||||||
|
font-size: 13px;
|
||||||
|
}
|
||||||
|
.badge {
|
||||||
|
display: inline-flex;
|
||||||
|
align-items: center;
|
||||||
|
gap: 6px;
|
||||||
|
padding: 4px 12px;
|
||||||
|
border-radius: 999px;
|
||||||
|
font-size: 13px;
|
||||||
|
font-weight: 600;
|
||||||
|
text-transform: uppercase;
|
||||||
|
letter-spacing: 0.03em;
|
||||||
|
}
|
||||||
|
.badge-dot { width: 8px; height: 8px; border-radius: 50%; }
|
||||||
|
.badge.healthy { background: rgba(62,207,142,0.15); color: var(--green); }
|
||||||
|
.badge.healthy .badge-dot { background: var(--green); }
|
||||||
|
.badge.degraded { background: rgba(242,84,91,0.15); color: var(--red); }
|
||||||
|
.badge.degraded .badge-dot { background: var(--red); }
|
||||||
|
.badge.unknown { background: rgba(139,144,158,0.15); color: var(--muted); }
|
||||||
|
.badge.unknown .badge-dot { background: var(--muted); }
|
||||||
|
|
||||||
|
#conn-banner {
|
||||||
|
display: none;
|
||||||
|
background: var(--red);
|
||||||
|
color: #fff;
|
||||||
|
text-align: center;
|
||||||
|
padding: 8px;
|
||||||
|
font-size: 13px;
|
||||||
|
font-weight: 600;
|
||||||
|
}
|
||||||
|
#conn-banner.show { display: block; }
|
||||||
|
|
||||||
|
main {
|
||||||
|
padding: 24px;
|
||||||
|
display: grid;
|
||||||
|
grid-template-columns: repeat(auto-fit, minmax(340px, 1fr));
|
||||||
|
gap: 20px;
|
||||||
|
max-width: 1400px;
|
||||||
|
margin: 0 auto;
|
||||||
|
}
|
||||||
|
.card {
|
||||||
|
background: var(--panel);
|
||||||
|
border: 1px solid var(--border);
|
||||||
|
border-radius: 10px;
|
||||||
|
padding: 18px 20px;
|
||||||
|
}
|
||||||
|
.card h2 {
|
||||||
|
font-size: 14px;
|
||||||
|
text-transform: uppercase;
|
||||||
|
letter-spacing: 0.05em;
|
||||||
|
color: var(--muted);
|
||||||
|
margin: 0 0 14px 0;
|
||||||
|
display: flex;
|
||||||
|
align-items: center;
|
||||||
|
justify-content: space-between;
|
||||||
|
}
|
||||||
|
table { width: 100%; border-collapse: collapse; font-size: 13px; }
|
||||||
|
th, td { text-align: left; padding: 6px 8px; border-bottom: 1px solid var(--border); }
|
||||||
|
th { color: var(--muted); font-weight: 500; font-size: 12px; }
|
||||||
|
tr.failed-row { background: rgba(242,84,91,0.08); }
|
||||||
|
tr.inactive-row { opacity: 0.5; }
|
||||||
|
.dot { display: inline-block; width: 8px; height: 8px; border-radius: 50%; margin-right: 6px; }
|
||||||
|
.dot.on { background: var(--green); }
|
||||||
|
.dot.off { background: var(--red); }
|
||||||
|
|
||||||
|
.stat-grid {
|
||||||
|
display: grid;
|
||||||
|
grid-template-columns: repeat(3, 1fr);
|
||||||
|
gap: 10px;
|
||||||
|
margin-bottom: 14px;
|
||||||
|
}
|
||||||
|
.stat {
|
||||||
|
background: rgba(255,255,255,0.03);
|
||||||
|
border-radius: 8px;
|
||||||
|
padding: 10px 12px;
|
||||||
|
}
|
||||||
|
.stat .value { font-size: 20px; font-weight: 600; }
|
||||||
|
.stat .label { font-size: 11px; color: var(--muted); text-transform: uppercase; letter-spacing: 0.03em; }
|
||||||
|
.stat.failed .value { color: var(--red); }
|
||||||
|
|
||||||
|
.state-pill {
|
||||||
|
display: inline-block;
|
||||||
|
padding: 3px 10px;
|
||||||
|
border-radius: 999px;
|
||||||
|
font-size: 12px;
|
||||||
|
font-weight: 600;
|
||||||
|
text-transform: uppercase;
|
||||||
|
}
|
||||||
|
.state-idle { background: rgba(139,144,158,0.15); color: var(--muted); }
|
||||||
|
.state-scanning, .state-running { background: rgba(79,142,247,0.15); color: var(--blue); }
|
||||||
|
.state-completed { background: rgba(62,207,142,0.15); color: var(--green); }
|
||||||
|
.state-failed { background: rgba(242,84,91,0.15); color: var(--red); }
|
||||||
|
|
||||||
|
.kv { display: flex; justify-content: space-between; padding: 4px 0; font-size: 13px; border-bottom: 1px solid var(--border); }
|
||||||
|
.kv:last-child { border-bottom: none; }
|
||||||
|
.kv .k { color: var(--muted); }
|
||||||
|
|
||||||
|
.error-list { margin: 10px 0 0 0; padding: 0; list-style: none; }
|
||||||
|
.error-list li {
|
||||||
|
font-size: 12px;
|
||||||
|
padding: 6px 8px;
|
||||||
|
margin-bottom: 4px;
|
||||||
|
background: rgba(242,84,91,0.08);
|
||||||
|
border-left: 3px solid var(--red);
|
||||||
|
border-radius: 4px;
|
||||||
|
color: var(--text);
|
||||||
|
}
|
||||||
|
.error-list .err-time { color: var(--muted); margin-right: 6px; }
|
||||||
|
.empty-note { color: var(--muted); font-size: 13px; font-style: italic; }
|
||||||
|
</style>
|
||||||
|
</head>
|
||||||
|
<body>
|
||||||
|
|
||||||
|
<div id="conn-banner">Lost connection to status endpoint - retrying...</div>
|
||||||
|
|
||||||
|
<header>
|
||||||
|
<h1>m3u8-downloader</h1>
|
||||||
|
<div class="header-meta">
|
||||||
|
<span id="overall-badge" class="badge unknown"><span class="badge-dot"></span>unknown</span>
|
||||||
|
<span id="uptime">uptime: -</span>
|
||||||
|
<span id="last-updated">last updated: -</span>
|
||||||
|
</div>
|
||||||
|
</header>
|
||||||
|
|
||||||
|
<main>
|
||||||
|
<section class="card">
|
||||||
|
<h2>Download</h2>
|
||||||
|
<table>
|
||||||
|
<thead>
|
||||||
|
<tr><th>Resolution</th><th>Active</th><th>OK</th><th>Failed</th><th>Last activity</th></tr>
|
||||||
|
</thead>
|
||||||
|
<tbody id="variant-rows"></tbody>
|
||||||
|
</table>
|
||||||
|
<div id="download-empty" class="empty-note" style="display:none;">No variants reporting yet.</div>
|
||||||
|
<ul id="download-errors" class="error-list"></ul>
|
||||||
|
</section>
|
||||||
|
|
||||||
|
<section class="card">
|
||||||
|
<h2>Transfer</h2>
|
||||||
|
<div class="kv"><span class="k">NAS connection</span><span><span id="nas-dot" class="dot off"></span><span id="nas-text">unknown</span></span></div>
|
||||||
|
<div class="stat-grid" style="margin-top:12px;">
|
||||||
|
<div class="stat"><div class="value" id="t-added">-</div><div class="label">Added</div></div>
|
||||||
|
<div class="stat"><div class="value" id="t-completed">-</div><div class="label">Completed</div></div>
|
||||||
|
<div class="stat failed"><div class="value" id="t-failed">-</div><div class="label">Failed</div></div>
|
||||||
|
<div class="stat"><div class="value" id="t-pending">-</div><div class="label">Pending</div></div>
|
||||||
|
<div class="stat"><div class="value" id="t-queue">-</div><div class="label">Queue size</div></div>
|
||||||
|
<div class="stat"><div class="value" id="t-cleanup">-</div><div class="label">Cleanup pending</div></div>
|
||||||
|
</div>
|
||||||
|
<div class="kv"><span class="k">Transferred</span><span id="t-bytes">-</span></div>
|
||||||
|
<ul id="transfer-errors" class="error-list"></ul>
|
||||||
|
</section>
|
||||||
|
|
||||||
|
<section class="card">
|
||||||
|
<h2>Processing</h2>
|
||||||
|
<div class="kv"><span class="k">State</span><span id="p-state" class="state-pill state-idle">idle</span></div>
|
||||||
|
<div class="kv"><span class="k">Event</span><span id="p-event">-</span></div>
|
||||||
|
<div class="kv"><span class="k">Segments found</span><span id="p-segments">-</span></div>
|
||||||
|
<div class="kv"><span class="k">Started</span><span id="p-started">-</span></div>
|
||||||
|
<div class="kv"><span class="k">Finished</span><span id="p-finished">-</span></div>
|
||||||
|
<ul id="processing-errors" class="error-list"></ul>
|
||||||
|
</section>
|
||||||
|
</main>
|
||||||
|
|
||||||
|
<script>
|
||||||
|
const POLL_INTERVAL_MS = 3000;
|
||||||
|
const connBanner = document.getElementById('conn-banner');
|
||||||
|
|
||||||
|
function fmtBytes(n) {
|
||||||
|
if (!n || n <= 0) return '0 B';
|
||||||
|
const units = ['B', 'KB', 'MB', 'GB', 'TB'];
|
||||||
|
let i = 0;
|
||||||
|
while (n >= 1024 && i < units.length - 1) { n /= 1024; i++; }
|
||||||
|
return n.toFixed(i === 0 ? 0 : 1) + ' ' + units[i];
|
||||||
|
}
|
||||||
|
|
||||||
|
function fmtDuration(sec) {
|
||||||
|
sec = Math.floor(sec || 0);
|
||||||
|
const h = Math.floor(sec / 3600);
|
||||||
|
const m = Math.floor((sec % 3600) / 60);
|
||||||
|
const s = sec % 60;
|
||||||
|
const parts = [];
|
||||||
|
if (h) parts.push(h + 'h');
|
||||||
|
if (m || h) parts.push(m + 'm');
|
||||||
|
parts.push(s + 's');
|
||||||
|
return parts.join(' ');
|
||||||
|
}
|
||||||
|
|
||||||
|
function fmtTime(iso) {
|
||||||
|
if (!iso) return '-';
|
||||||
|
const d = new Date(iso);
|
||||||
|
if (isNaN(d.getTime()) || d.getFullYear() < 1971) return '-';
|
||||||
|
return d.toLocaleTimeString();
|
||||||
|
}
|
||||||
|
|
||||||
|
function renderErrors(listEl, errors) {
|
||||||
|
listEl.innerHTML = '';
|
||||||
|
(errors || []).slice().reverse().forEach(e => {
|
||||||
|
const li = document.createElement('li');
|
||||||
|
const t = document.createElement('span');
|
||||||
|
t.className = 'err-time';
|
||||||
|
t.textContent = fmtTime(e.time);
|
||||||
|
li.appendChild(t);
|
||||||
|
li.appendChild(document.createTextNode(e.message));
|
||||||
|
listEl.appendChild(li);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
function render(data) {
|
||||||
|
const badge = document.getElementById('overall-badge');
|
||||||
|
badge.className = 'badge ' + (data.status === 'healthy' ? 'healthy' : 'degraded');
|
||||||
|
badge.innerHTML = '<span class="badge-dot"></span>' + data.status;
|
||||||
|
|
||||||
|
document.getElementById('uptime').textContent = 'uptime: ' + fmtDuration(data.uptime_seconds);
|
||||||
|
document.getElementById('last-updated').textContent = 'last updated: ' + new Date().toLocaleTimeString();
|
||||||
|
|
||||||
|
// Download
|
||||||
|
const variants = (data.download && data.download.variants) || [];
|
||||||
|
const rows = document.getElementById('variant-rows');
|
||||||
|
rows.innerHTML = '';
|
||||||
|
document.getElementById('download-empty').style.display = variants.length ? 'none' : 'block';
|
||||||
|
variants.forEach(v => {
|
||||||
|
const tr = document.createElement('tr');
|
||||||
|
if (v.segments_failed > 0) tr.classList.add('failed-row');
|
||||||
|
if (!v.active) tr.classList.add('inactive-row');
|
||||||
|
tr.innerHTML =
|
||||||
|
'<td>' + v.resolution + '</td>' +
|
||||||
|
'<td><span class="dot ' + (v.active ? 'on' : 'off') + '"></span>' + (v.active ? 'active' : 'stopped') + '</td>' +
|
||||||
|
'<td>' + v.segments_ok + '</td>' +
|
||||||
|
'<td>' + v.segments_failed + '</td>' +
|
||||||
|
'<td>' + fmtTime(v.last_activity) + '</td>';
|
||||||
|
rows.appendChild(tr);
|
||||||
|
});
|
||||||
|
renderErrors(document.getElementById('download-errors'), data.download && data.download.recent_errors);
|
||||||
|
|
||||||
|
// Transfer
|
||||||
|
const t = data.transfer || {};
|
||||||
|
const nasDot = document.getElementById('nas-dot');
|
||||||
|
nasDot.className = 'dot ' + (t.nas_connected ? 'on' : 'off');
|
||||||
|
document.getElementById('nas-text').textContent = t.nas_connected ? 'connected' : 'disconnected';
|
||||||
|
document.getElementById('t-added').textContent = t.added ?? '-';
|
||||||
|
document.getElementById('t-completed').textContent = t.completed ?? '-';
|
||||||
|
document.getElementById('t-failed').textContent = t.failed ?? '-';
|
||||||
|
document.getElementById('t-pending').textContent = t.pending ?? '-';
|
||||||
|
document.getElementById('t-queue').textContent = t.queue_size ?? '-';
|
||||||
|
document.getElementById('t-cleanup').textContent = t.cleanup_pending ?? '-';
|
||||||
|
document.getElementById('t-bytes').textContent = fmtBytes(t.bytes_transferred);
|
||||||
|
renderErrors(document.getElementById('transfer-errors'), t.recent_errors);
|
||||||
|
|
||||||
|
// Processing
|
||||||
|
const p = data.processing || {};
|
||||||
|
const stateEl = document.getElementById('p-state');
|
||||||
|
const state = p.state || 'idle';
|
||||||
|
stateEl.textContent = state;
|
||||||
|
stateEl.className = 'state-pill state-' + state;
|
||||||
|
document.getElementById('p-event').textContent = p.event_name || '-';
|
||||||
|
document.getElementById('p-segments').textContent = p.segments_found ?? '-';
|
||||||
|
document.getElementById('p-started').textContent = fmtTime(p.started_at);
|
||||||
|
document.getElementById('p-finished').textContent = fmtTime(p.finished_at);
|
||||||
|
renderErrors(document.getElementById('processing-errors'), p.recent_errors);
|
||||||
|
}
|
||||||
|
|
||||||
|
async function poll() {
|
||||||
|
try {
|
||||||
|
const resp = await fetch('/status', { cache: 'no-store' });
|
||||||
|
if (!resp.ok) throw new Error('status ' + resp.status);
|
||||||
|
const data = await resp.json();
|
||||||
|
connBanner.classList.remove('show');
|
||||||
|
render(data);
|
||||||
|
} catch (e) {
|
||||||
|
connBanner.classList.add('show');
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
poll();
|
||||||
|
setInterval(poll, POLL_INTERVAL_MS);
|
||||||
|
</script>
|
||||||
|
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
352
pkg/health/registry.go
Normal file
352
pkg/health/registry.go
Normal file
@ -0,0 +1,352 @@
|
|||||||
|
package health
|
||||||
|
|
||||||
|
import (
|
||||||
|
"sort"
|
||||||
|
"sync"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
const maxRecentErrors = 10
|
||||||
|
|
||||||
|
type ErrorEntry struct {
|
||||||
|
Time time.Time `json:"time"`
|
||||||
|
Message string `json:"message"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type VariantStatus struct {
|
||||||
|
Resolution string `json:"resolution"`
|
||||||
|
Bandwidth uint32 `json:"bandwidth"`
|
||||||
|
Active bool `json:"active"`
|
||||||
|
SegmentsOK int `json:"segments_ok"`
|
||||||
|
SegmentsFailed int `json:"segments_failed"`
|
||||||
|
LastActivity time.Time `json:"last_activity"`
|
||||||
|
LastError string `json:"last_error,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// StatsProvider is satisfied by *transfer.QueueStats without pkg/health
|
||||||
|
// needing to import pkg/transfer (which would create an import cycle, since
|
||||||
|
// pkg/transfer calls into pkg/health to report status).
|
||||||
|
type StatsProvider interface {
|
||||||
|
GetStats() (added, completed, failed, pending int, bytes int64)
|
||||||
|
}
|
||||||
|
|
||||||
|
type downloadState struct {
|
||||||
|
variants map[string]*VariantStatus
|
||||||
|
recentErrors []ErrorEntry
|
||||||
|
}
|
||||||
|
|
||||||
|
type transferState struct {
|
||||||
|
statsProvider StatsProvider
|
||||||
|
queueSizeFn func() int
|
||||||
|
cleanupPendingFn func() int
|
||||||
|
nasConnectedFn func() bool
|
||||||
|
nasConnected bool
|
||||||
|
recentErrors []ErrorEntry
|
||||||
|
}
|
||||||
|
|
||||||
|
type processingState struct {
|
||||||
|
state string // idle | scanning | running | completed | failed
|
||||||
|
eventName string
|
||||||
|
segmentsFound int
|
||||||
|
startedAt time.Time
|
||||||
|
finishedAt time.Time
|
||||||
|
lastError string
|
||||||
|
recentErrors []ErrorEntry
|
||||||
|
}
|
||||||
|
|
||||||
|
// Registry is the process-wide health/metrics store. All subsystems report
|
||||||
|
// into it via the Record* methods; the health HTTP server reads it via
|
||||||
|
// Snapshot(). Safe for concurrent use.
|
||||||
|
type Registry struct {
|
||||||
|
mu sync.RWMutex
|
||||||
|
startedAt time.Time
|
||||||
|
download downloadState
|
||||||
|
transfer transferState
|
||||||
|
processing processingState
|
||||||
|
}
|
||||||
|
|
||||||
|
var (
|
||||||
|
globalRegistry *Registry
|
||||||
|
registryOnce sync.Once
|
||||||
|
)
|
||||||
|
|
||||||
|
// MustGetRegistry returns the process-wide registry, mirroring the
|
||||||
|
// constants.MustGetConfig() / logging.MustGetFactory() singleton style.
|
||||||
|
func MustGetRegistry() *Registry {
|
||||||
|
registryOnce.Do(func() {
|
||||||
|
globalRegistry = New()
|
||||||
|
})
|
||||||
|
return globalRegistry
|
||||||
|
}
|
||||||
|
|
||||||
|
func New() *Registry {
|
||||||
|
return &Registry{
|
||||||
|
startedAt: time.Now(),
|
||||||
|
download: downloadState{
|
||||||
|
variants: make(map[string]*VariantStatus),
|
||||||
|
},
|
||||||
|
processing: processingState{
|
||||||
|
state: "idle",
|
||||||
|
},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func appendBounded(errs []ErrorEntry, e ErrorEntry) []ErrorEntry {
|
||||||
|
errs = append(errs, e)
|
||||||
|
if len(errs) > maxRecentErrors {
|
||||||
|
errs = errs[len(errs)-maxRecentErrors:]
|
||||||
|
}
|
||||||
|
return errs
|
||||||
|
}
|
||||||
|
|
||||||
|
func errString(err error) string {
|
||||||
|
if err == nil {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
return err.Error()
|
||||||
|
}
|
||||||
|
|
||||||
|
// --- Download ---
|
||||||
|
|
||||||
|
func (r *Registry) RecordVariantStarted(resolution string, bandwidth uint32) {
|
||||||
|
r.mu.Lock()
|
||||||
|
defer r.mu.Unlock()
|
||||||
|
r.download.variants[resolution] = &VariantStatus{
|
||||||
|
Resolution: resolution,
|
||||||
|
Bandwidth: bandwidth,
|
||||||
|
Active: true,
|
||||||
|
LastActivity: time.Now(),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *Registry) RecordVariantStopped(resolution string) {
|
||||||
|
r.mu.Lock()
|
||||||
|
defer r.mu.Unlock()
|
||||||
|
if v, ok := r.download.variants[resolution]; ok {
|
||||||
|
v.Active = false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *Registry) RecordSegmentDownloaded(resolution string) {
|
||||||
|
r.mu.Lock()
|
||||||
|
defer r.mu.Unlock()
|
||||||
|
v := r.ensureVariantLocked(resolution)
|
||||||
|
v.SegmentsOK++
|
||||||
|
v.LastActivity = time.Now()
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *Registry) RecordSegmentFailed(resolution string, err error) {
|
||||||
|
r.mu.Lock()
|
||||||
|
defer r.mu.Unlock()
|
||||||
|
v := r.ensureVariantLocked(resolution)
|
||||||
|
v.SegmentsFailed++
|
||||||
|
v.LastActivity = time.Now()
|
||||||
|
v.LastError = errString(err)
|
||||||
|
r.download.recentErrors = appendBounded(r.download.recentErrors, ErrorEntry{
|
||||||
|
Time: time.Now(),
|
||||||
|
Message: resolution + ": " + errString(err),
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *Registry) RecordPlaylistError(resolution string, err error) {
|
||||||
|
r.mu.Lock()
|
||||||
|
defer r.mu.Unlock()
|
||||||
|
v := r.ensureVariantLocked(resolution)
|
||||||
|
v.LastError = errString(err)
|
||||||
|
r.download.recentErrors = appendBounded(r.download.recentErrors, ErrorEntry{
|
||||||
|
Time: time.Now(),
|
||||||
|
Message: resolution + " playlist: " + errString(err),
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *Registry) ensureVariantLocked(resolution string) *VariantStatus {
|
||||||
|
v, ok := r.download.variants[resolution]
|
||||||
|
if !ok {
|
||||||
|
v = &VariantStatus{Resolution: resolution, Active: true}
|
||||||
|
r.download.variants[resolution] = v
|
||||||
|
}
|
||||||
|
return v
|
||||||
|
}
|
||||||
|
|
||||||
|
// --- Transfer ---
|
||||||
|
|
||||||
|
// AttachTransfer wires the registry to the live counters/state already
|
||||||
|
// maintained by pkg/transfer instead of duplicating them. Called once from
|
||||||
|
// TransferService construction.
|
||||||
|
func (r *Registry) AttachTransfer(stats StatsProvider, queueSizeFn, cleanupPendingFn func() int, nasConnectedFn func() bool) {
|
||||||
|
r.mu.Lock()
|
||||||
|
defer r.mu.Unlock()
|
||||||
|
r.transfer.statsProvider = stats
|
||||||
|
r.transfer.queueSizeFn = queueSizeFn
|
||||||
|
r.transfer.cleanupPendingFn = cleanupPendingFn
|
||||||
|
r.transfer.nasConnectedFn = nasConnectedFn
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *Registry) RecordNASConnectivity(connected bool) {
|
||||||
|
r.mu.Lock()
|
||||||
|
defer r.mu.Unlock()
|
||||||
|
r.transfer.nasConnected = connected
|
||||||
|
}
|
||||||
|
|
||||||
|
// RecordTransferError captures failures QueueStats' cumulative counters
|
||||||
|
// don't individually retain (pre-queue failures, cleanup failures) so they
|
||||||
|
// still surface in the bounded recent-errors list.
|
||||||
|
func (r *Registry) RecordTransferError(sourcePath string, err error) {
|
||||||
|
r.mu.Lock()
|
||||||
|
defer r.mu.Unlock()
|
||||||
|
r.transfer.recentErrors = appendBounded(r.transfer.recentErrors, ErrorEntry{
|
||||||
|
Time: time.Now(),
|
||||||
|
Message: sourcePath + ": " + errString(err),
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// --- Processing ---
|
||||||
|
|
||||||
|
func (r *Registry) RecordProcessingStageChange(state, eventName string) {
|
||||||
|
r.mu.Lock()
|
||||||
|
defer r.mu.Unlock()
|
||||||
|
r.processing.state = state
|
||||||
|
if eventName != "" {
|
||||||
|
r.processing.eventName = eventName
|
||||||
|
}
|
||||||
|
switch state {
|
||||||
|
case "scanning":
|
||||||
|
r.processing.startedAt = time.Now()
|
||||||
|
r.processing.finishedAt = time.Time{}
|
||||||
|
r.processing.lastError = ""
|
||||||
|
case "completed", "failed":
|
||||||
|
r.processing.finishedAt = time.Now()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *Registry) RecordProcessingSegmentsFound(count int) {
|
||||||
|
r.mu.Lock()
|
||||||
|
defer r.mu.Unlock()
|
||||||
|
r.processing.segmentsFound = count
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *Registry) RecordProcessingError(err error) {
|
||||||
|
r.mu.Lock()
|
||||||
|
defer r.mu.Unlock()
|
||||||
|
msg := errString(err)
|
||||||
|
r.processing.lastError = msg
|
||||||
|
r.processing.recentErrors = appendBounded(r.processing.recentErrors, ErrorEntry{
|
||||||
|
Time: time.Now(),
|
||||||
|
Message: msg,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// --- Snapshot ---
|
||||||
|
|
||||||
|
type DownloadSnapshot struct {
|
||||||
|
Variants []*VariantStatus `json:"variants"`
|
||||||
|
RecentErrors []ErrorEntry `json:"recent_errors"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type TransferSnapshot struct {
|
||||||
|
NASConnected bool `json:"nas_connected"`
|
||||||
|
QueueSize int `json:"queue_size"`
|
||||||
|
CleanupPending int `json:"cleanup_pending"`
|
||||||
|
Added int `json:"added"`
|
||||||
|
Completed int `json:"completed"`
|
||||||
|
Failed int `json:"failed"`
|
||||||
|
Pending int `json:"pending"`
|
||||||
|
BytesTransferred int64 `json:"bytes_transferred"`
|
||||||
|
RecentErrors []ErrorEntry `json:"recent_errors"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type ProcessingSnapshot struct {
|
||||||
|
State string `json:"state"`
|
||||||
|
EventName string `json:"event_name,omitempty"`
|
||||||
|
SegmentsFound int `json:"segments_found"`
|
||||||
|
StartedAt *time.Time `json:"started_at,omitempty"`
|
||||||
|
FinishedAt *time.Time `json:"finished_at,omitempty"`
|
||||||
|
LastError string `json:"last_error,omitempty"`
|
||||||
|
RecentErrors []ErrorEntry `json:"recent_errors"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type Snapshot struct {
|
||||||
|
Status string `json:"status"`
|
||||||
|
UptimeSec float64 `json:"uptime_seconds"`
|
||||||
|
Download DownloadSnapshot `json:"download"`
|
||||||
|
Transfer TransferSnapshot `json:"transfer"`
|
||||||
|
Processing ProcessingSnapshot `json:"processing"`
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *Registry) Snapshot() Snapshot {
|
||||||
|
r.mu.RLock()
|
||||||
|
defer r.mu.RUnlock()
|
||||||
|
|
||||||
|
variants := make([]*VariantStatus, 0, len(r.download.variants))
|
||||||
|
for _, v := range r.download.variants {
|
||||||
|
vCopy := *v
|
||||||
|
variants = append(variants, &vCopy)
|
||||||
|
}
|
||||||
|
sort.Slice(variants, func(i, j int) bool { return variants[i].Resolution < variants[j].Resolution })
|
||||||
|
|
||||||
|
dl := DownloadSnapshot{
|
||||||
|
Variants: variants,
|
||||||
|
RecentErrors: append([]ErrorEntry(nil), r.download.recentErrors...),
|
||||||
|
}
|
||||||
|
|
||||||
|
tr := TransferSnapshot{
|
||||||
|
NASConnected: r.transfer.nasConnected,
|
||||||
|
RecentErrors: append([]ErrorEntry(nil), r.transfer.recentErrors...),
|
||||||
|
}
|
||||||
|
if r.transfer.statsProvider != nil {
|
||||||
|
added, completed, failed, pending, bytes := r.transfer.statsProvider.GetStats()
|
||||||
|
tr.Added, tr.Completed, tr.Failed, tr.Pending, tr.BytesTransferred = added, completed, failed, pending, bytes
|
||||||
|
}
|
||||||
|
if r.transfer.queueSizeFn != nil {
|
||||||
|
tr.QueueSize = r.transfer.queueSizeFn()
|
||||||
|
}
|
||||||
|
if r.transfer.cleanupPendingFn != nil {
|
||||||
|
tr.CleanupPending = r.transfer.cleanupPendingFn()
|
||||||
|
}
|
||||||
|
if r.transfer.nasConnectedFn != nil {
|
||||||
|
tr.NASConnected = r.transfer.nasConnectedFn()
|
||||||
|
}
|
||||||
|
|
||||||
|
proc := ProcessingSnapshot{
|
||||||
|
State: r.processing.state,
|
||||||
|
EventName: r.processing.eventName,
|
||||||
|
SegmentsFound: r.processing.segmentsFound,
|
||||||
|
LastError: r.processing.lastError,
|
||||||
|
RecentErrors: append([]ErrorEntry(nil), r.processing.recentErrors...),
|
||||||
|
}
|
||||||
|
if !r.processing.startedAt.IsZero() {
|
||||||
|
t := r.processing.startedAt
|
||||||
|
proc.StartedAt = &t
|
||||||
|
}
|
||||||
|
if !r.processing.finishedAt.IsZero() {
|
||||||
|
t := r.processing.finishedAt
|
||||||
|
proc.FinishedAt = &t
|
||||||
|
}
|
||||||
|
|
||||||
|
snap := Snapshot{
|
||||||
|
UptimeSec: time.Since(r.startedAt).Seconds(),
|
||||||
|
Download: dl,
|
||||||
|
Transfer: tr,
|
||||||
|
Processing: proc,
|
||||||
|
}
|
||||||
|
snap.Status = computeStatus(dl, tr, proc)
|
||||||
|
return snap
|
||||||
|
}
|
||||||
|
|
||||||
|
// computeStatus gives one glanceable field: "degraded" the moment anything
|
||||||
|
// worth investigating shows up, "healthy" otherwise. Transfer is only
|
||||||
|
// considered for degradation once it's actually in use (Attached + has seen
|
||||||
|
// activity), so a download-only run with NAS transfer disabled doesn't
|
||||||
|
// falsely report degraded.
|
||||||
|
func computeStatus(dl DownloadSnapshot, tr TransferSnapshot, proc ProcessingSnapshot) string {
|
||||||
|
if len(dl.RecentErrors) > 0 || len(tr.RecentErrors) > 0 || len(proc.RecentErrors) > 0 {
|
||||||
|
return "degraded"
|
||||||
|
}
|
||||||
|
if !tr.NASConnected && (tr.Added > 0 || tr.QueueSize > 0) {
|
||||||
|
return "degraded"
|
||||||
|
}
|
||||||
|
if proc.State == "failed" {
|
||||||
|
return "degraded"
|
||||||
|
}
|
||||||
|
return "healthy"
|
||||||
|
}
|
||||||
60
pkg/health/server.go
Normal file
60
pkg/health/server.go
Normal file
@ -0,0 +1,60 @@
|
|||||||
|
package health
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
_ "embed"
|
||||||
|
"encoding/json"
|
||||||
|
"fmt"
|
||||||
|
"m3u8-downloader/pkg/config"
|
||||||
|
"m3u8-downloader/pkg/logging"
|
||||||
|
"net"
|
||||||
|
"net/http"
|
||||||
|
)
|
||||||
|
|
||||||
|
//go:embed dashboard.html
|
||||||
|
var dashboardHTML []byte
|
||||||
|
|
||||||
|
// Start binds the health/status HTTP server and returns immediately once the
|
||||||
|
// listener is bound, so callers see bind failures (e.g. port already in use)
|
||||||
|
// synchronously rather than losing them in a background goroutine. The
|
||||||
|
// returned *http.Server should be shut down by the caller via Shutdown(ctx).
|
||||||
|
//
|
||||||
|
// Security note: cfg.Addr is expected to be a bind-all address (e.g. ":8090")
|
||||||
|
// so status can be checked from other devices on the local network during a
|
||||||
|
// live event. There is no authentication or TLS on these endpoints - this is
|
||||||
|
// only safe on a trusted local network, never exposed to the open internet.
|
||||||
|
func Start(ctx context.Context, cfg *config.HealthConfig, reg *Registry) (*http.Server, error) {
|
||||||
|
mux := http.NewServeMux()
|
||||||
|
|
||||||
|
mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
w.Header().Set("Content-Type", "text/html; charset=utf-8")
|
||||||
|
w.Write(dashboardHTML)
|
||||||
|
})
|
||||||
|
|
||||||
|
mux.HandleFunc("/status", func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
w.Header().Set("Content-Type", "application/json")
|
||||||
|
if err := json.NewEncoder(w).Encode(reg.Snapshot()); err != nil {
|
||||||
|
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
mux.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
w.WriteHeader(http.StatusOK)
|
||||||
|
_, _ = w.Write([]byte("ok"))
|
||||||
|
})
|
||||||
|
|
||||||
|
ln, err := net.Listen("tcp", cfg.Addr)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("failed to bind health server to %s: %w", cfg.Addr, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
srv := &http.Server{Handler: mux}
|
||||||
|
|
||||||
|
go func() {
|
||||||
|
if err := srv.Serve(ln); err != nil && err != http.ErrServerClosed {
|
||||||
|
logging.MustGetFactory().App.Error("health server stopped unexpectedly", logging.KeyError, err.Error())
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
|
||||||
|
return srv, nil
|
||||||
|
}
|
||||||
@ -5,6 +5,50 @@ import (
|
|||||||
"fmt"
|
"fmt"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
// HTTPError represents an HTTP error with status code and message
|
||||||
|
type HTTPError struct {
|
||||||
|
StatusCode int
|
||||||
|
Message string
|
||||||
|
}
|
||||||
|
|
||||||
|
// Error returns the string representation of the HTTP error
|
||||||
|
func (e *HTTPError) Error() string {
|
||||||
|
return fmt.Sprintf("HTTP %d: %s", e.StatusCode, e.Message)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Is implements error comparison for errors.Is
|
||||||
|
func (e *HTTPError) Is(target error) bool {
|
||||||
|
var httpErr *HTTPError
|
||||||
|
if errors.As(target, &httpErr) {
|
||||||
|
return e.StatusCode == httpErr.StatusCode
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
// NewHTTPError creates a new HTTP error
|
||||||
|
func NewHTTPError(statusCode int, message string) error {
|
||||||
|
return &HTTPError{
|
||||||
|
StatusCode: statusCode,
|
||||||
|
Message: message,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// IsHTTPError checks if an error is an HTTP error
|
||||||
|
func IsHTTPError(err error) bool {
|
||||||
|
var httpErr *HTTPError
|
||||||
|
return errors.As(err, &httpErr)
|
||||||
|
}
|
||||||
|
|
||||||
|
// GetHTTPStatusCode extracts the status code from an HTTP error
|
||||||
|
func GetHTTPStatusCode(err error) int {
|
||||||
|
var httpErr *HTTPError
|
||||||
|
if errors.As(err, &httpErr) {
|
||||||
|
return httpErr.StatusCode
|
||||||
|
}
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
|
||||||
|
// Legacy support for existing code
|
||||||
type HttpError struct {
|
type HttpError struct {
|
||||||
Code int
|
Code int
|
||||||
}
|
}
|
||||||
|
|||||||
318
pkg/httpClient/error_test.go
Normal file
318
pkg/httpClient/error_test.go
Normal file
@ -0,0 +1,318 @@
|
|||||||
|
package httpClient
|
||||||
|
|
||||||
|
import (
|
||||||
|
"fmt"
|
||||||
|
"net/http"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestHTTPError_Error(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
statusCode int
|
||||||
|
message string
|
||||||
|
want string
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
name: "basic http error",
|
||||||
|
statusCode: 404,
|
||||||
|
message: "Not Found",
|
||||||
|
want: "HTTP 404: Not Found",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "server error",
|
||||||
|
statusCode: 500,
|
||||||
|
message: "Internal Server Error",
|
||||||
|
want: "HTTP 500: Internal Server Error",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "unauthorized error",
|
||||||
|
statusCode: 401,
|
||||||
|
message: "Unauthorized",
|
||||||
|
want: "HTTP 401: Unauthorized",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "empty message",
|
||||||
|
statusCode: 400,
|
||||||
|
message: "",
|
||||||
|
want: "HTTP 400: ",
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
err := &HTTPError{
|
||||||
|
StatusCode: tt.statusCode,
|
||||||
|
Message: tt.message,
|
||||||
|
}
|
||||||
|
|
||||||
|
got := err.Error()
|
||||||
|
if got != tt.want {
|
||||||
|
t.Errorf("HTTPError.Error() = %q, want %q", got, tt.want)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestHTTPError_Is(t *testing.T) {
|
||||||
|
err404 := &HTTPError{StatusCode: 404, Message: "Not Found"}
|
||||||
|
err500 := &HTTPError{StatusCode: 500, Message: "Server Error"}
|
||||||
|
otherErr404 := &HTTPError{StatusCode: 404, Message: "Different message"}
|
||||||
|
regularError := fmt.Errorf("regular error")
|
||||||
|
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
err error
|
||||||
|
target error
|
||||||
|
want bool
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
name: "same error instance",
|
||||||
|
err: err404,
|
||||||
|
target: err404,
|
||||||
|
want: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "different HTTP errors with same status",
|
||||||
|
err: err404,
|
||||||
|
target: otherErr404,
|
||||||
|
want: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "different HTTP errors with different status",
|
||||||
|
err: err404,
|
||||||
|
target: err500,
|
||||||
|
want: false,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "HTTP error vs regular error",
|
||||||
|
err: err404,
|
||||||
|
target: regularError,
|
||||||
|
want: false,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "regular error vs HTTP error",
|
||||||
|
err: regularError,
|
||||||
|
target: err404,
|
||||||
|
want: false,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "nil target",
|
||||||
|
err: err404,
|
||||||
|
target: nil,
|
||||||
|
want: false,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
var got bool
|
||||||
|
if httpErr, ok := tt.err.(*HTTPError); ok {
|
||||||
|
got = httpErr.Is(tt.target)
|
||||||
|
} else {
|
||||||
|
got = false // Non-HTTP errors return false
|
||||||
|
}
|
||||||
|
if got != tt.want {
|
||||||
|
t.Errorf("HTTPError.Is() = %v, want %v", got, tt.want)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestIsHTTPError(t *testing.T) {
|
||||||
|
httpErr := &HTTPError{StatusCode: 404, Message: "Not Found"}
|
||||||
|
regularErr := fmt.Errorf("regular error")
|
||||||
|
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
err error
|
||||||
|
want bool
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
name: "http error",
|
||||||
|
err: httpErr,
|
||||||
|
want: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "regular error",
|
||||||
|
err: regularErr,
|
||||||
|
want: false,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "nil error",
|
||||||
|
err: nil,
|
||||||
|
want: false,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "wrapped http error",
|
||||||
|
err: fmt.Errorf("wrapped: %w", httpErr),
|
||||||
|
want: true,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
got := IsHTTPError(tt.err)
|
||||||
|
if got != tt.want {
|
||||||
|
t.Errorf("IsHTTPError() = %v, want %v", got, tt.want)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestGetHTTPStatusCode(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
err error
|
||||||
|
want int
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
name: "http error 404",
|
||||||
|
err: &HTTPError{StatusCode: 404, Message: "Not Found"},
|
||||||
|
want: 404,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "http error 500",
|
||||||
|
err: &HTTPError{StatusCode: 500, Message: "Server Error"},
|
||||||
|
want: 500,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "wrapped http error",
|
||||||
|
err: fmt.Errorf("wrapped: %w", &HTTPError{StatusCode: 403, Message: "Forbidden"}),
|
||||||
|
want: 403,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "regular error",
|
||||||
|
err: fmt.Errorf("regular error"),
|
||||||
|
want: 0,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "nil error",
|
||||||
|
err: nil,
|
||||||
|
want: 0,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
got := GetHTTPStatusCode(tt.err)
|
||||||
|
if got != tt.want {
|
||||||
|
t.Errorf("GetHTTPStatusCode() = %v, want %v", got, tt.want)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestNewHTTPError(t *testing.T) {
|
||||||
|
statusCode := 404
|
||||||
|
message := "Page not found"
|
||||||
|
|
||||||
|
err := NewHTTPError(statusCode, message)
|
||||||
|
|
||||||
|
// Check type
|
||||||
|
httpErr, ok := err.(*HTTPError)
|
||||||
|
if !ok {
|
||||||
|
t.Fatalf("NewHTTPError should return *HTTPError, got %T", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Check fields
|
||||||
|
if httpErr.StatusCode != statusCode {
|
||||||
|
t.Errorf("Expected StatusCode=%d, got %d", statusCode, httpErr.StatusCode)
|
||||||
|
}
|
||||||
|
if httpErr.Message != message {
|
||||||
|
t.Errorf("Expected Message=%q, got %q", message, httpErr.Message)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Check error string
|
||||||
|
expectedErrorString := fmt.Sprintf("HTTP %d: %s", statusCode, message)
|
||||||
|
if httpErr.Error() != expectedErrorString {
|
||||||
|
t.Errorf("Expected error string=%q, got %q", expectedErrorString, httpErr.Error())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestHTTPError_StatusCodeChecks(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
statusCode int
|
||||||
|
isClient bool
|
||||||
|
isServer bool
|
||||||
|
}{
|
||||||
|
{"200 OK", 200, false, false},
|
||||||
|
{"400 Bad Request", 400, true, false},
|
||||||
|
{"401 Unauthorized", 401, true, false},
|
||||||
|
{"404 Not Found", 404, true, false},
|
||||||
|
{"499 Client Error", 499, true, false},
|
||||||
|
{"500 Server Error", 500, false, true},
|
||||||
|
{"502 Bad Gateway", 502, false, true},
|
||||||
|
{"599 Server Error", 599, false, true},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
err := &HTTPError{StatusCode: tt.statusCode, Message: "test"}
|
||||||
|
|
||||||
|
isClient := err.StatusCode >= 400 && err.StatusCode < 500
|
||||||
|
isServer := err.StatusCode >= 500 && err.StatusCode < 600
|
||||||
|
|
||||||
|
if isClient != tt.isClient {
|
||||||
|
t.Errorf("Status %d: expected isClient=%v, got %v", tt.statusCode, tt.isClient, isClient)
|
||||||
|
}
|
||||||
|
if isServer != tt.isServer {
|
||||||
|
t.Errorf("Status %d: expected isServer=%v, got %v", tt.statusCode, tt.isServer, isServer)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestHTTPError_Integration(t *testing.T) {
|
||||||
|
// Test that HTTPError integrates well with standard error handling
|
||||||
|
err := NewHTTPError(http.StatusNotFound, "Resource not found")
|
||||||
|
|
||||||
|
// Should be able to use with errors.Is
|
||||||
|
target := &HTTPError{StatusCode: http.StatusNotFound}
|
||||||
|
if !err.(*HTTPError).Is(target) {
|
||||||
|
t.Error("HTTPError should match target with same status code")
|
||||||
|
}
|
||||||
|
|
||||||
|
// Should be detectable as HTTPError
|
||||||
|
if !IsHTTPError(err) {
|
||||||
|
t.Error("Should be detectable as HTTPError")
|
||||||
|
}
|
||||||
|
|
||||||
|
// Should return correct status code
|
||||||
|
if GetHTTPStatusCode(err) != http.StatusNotFound {
|
||||||
|
t.Error("Should return correct status code")
|
||||||
|
}
|
||||||
|
|
||||||
|
// Should have meaningful string representation
|
||||||
|
errorString := err.Error()
|
||||||
|
if !strings.Contains(errorString, "404") {
|
||||||
|
t.Error("Error string should contain status code")
|
||||||
|
}
|
||||||
|
if !strings.Contains(errorString, "Resource not found") {
|
||||||
|
t.Error("Error string should contain message")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestHTTPError_EdgeCases(t *testing.T) {
|
||||||
|
// Test with zero status code
|
||||||
|
err := NewHTTPError(0, "Zero status")
|
||||||
|
if err.Error() != "HTTP 0: Zero status" {
|
||||||
|
t.Errorf("Unexpected error string for zero status: %s", err.Error())
|
||||||
|
}
|
||||||
|
|
||||||
|
// Test with very long message
|
||||||
|
longMessage := strings.Repeat("a", 1000)
|
||||||
|
err = NewHTTPError(500, longMessage)
|
||||||
|
if !strings.Contains(err.Error(), longMessage) {
|
||||||
|
t.Error("Long message should be preserved")
|
||||||
|
}
|
||||||
|
|
||||||
|
// Test status code boundaries
|
||||||
|
for _, code := range []int{399, 400, 499, 500, 599, 600} {
|
||||||
|
err := &HTTPError{StatusCode: code}
|
||||||
|
// Should not panic
|
||||||
|
_ = err.Error()
|
||||||
|
}
|
||||||
|
}
|
||||||
72
pkg/logging/factory.go
Normal file
72
pkg/logging/factory.go
Normal file
@ -0,0 +1,72 @@
|
|||||||
|
package logging
|
||||||
|
|
||||||
|
import (
|
||||||
|
"log/slog"
|
||||||
|
"m3u8-downloader/pkg/config"
|
||||||
|
"m3u8-downloader/pkg/constants"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"sync"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Factory holds one structured logger per pipeline stage, each writing JSON
|
||||||
|
// lines to its own file under LoggingConfig.Dir. These are additive to the
|
||||||
|
// existing stdlib log.* calls, not a replacement for them.
|
||||||
|
type Factory struct {
|
||||||
|
App *slog.Logger
|
||||||
|
Download *slog.Logger
|
||||||
|
Transfer *slog.Logger
|
||||||
|
Processing *slog.Logger
|
||||||
|
}
|
||||||
|
|
||||||
|
var (
|
||||||
|
globalFactory *Factory
|
||||||
|
factoryOnce sync.Once
|
||||||
|
)
|
||||||
|
|
||||||
|
// MustGetFactory returns the process-wide logging factory, building it from
|
||||||
|
// the config singleton on first use. Mirrors constants.MustGetConfig().
|
||||||
|
func MustGetFactory() *Factory {
|
||||||
|
factoryOnce.Do(func() {
|
||||||
|
globalFactory = New(&constants.MustGetConfig().Logging)
|
||||||
|
})
|
||||||
|
return globalFactory
|
||||||
|
}
|
||||||
|
|
||||||
|
func New(cfg *config.LoggingConfig) *Factory {
|
||||||
|
level := parseLevel(cfg.Level)
|
||||||
|
return &Factory{
|
||||||
|
App: newSubsystemLogger(cfg.Dir, "app.log", "app", level),
|
||||||
|
Download: newSubsystemLogger(cfg.Dir, "download.log", "download", level),
|
||||||
|
Transfer: newSubsystemLogger(cfg.Dir, "transfer.log", "transfer", level),
|
||||||
|
Processing: newSubsystemLogger(cfg.Dir, "processing.log", "processing", level),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// newSubsystemLogger opens (or creates) the subsystem's log file in append
|
||||||
|
// mode. A logging misconfiguration must never take down an active recording,
|
||||||
|
// so failure to open the file falls back to stderr instead of panicking.
|
||||||
|
func newSubsystemLogger(dir, filename, component string, level slog.Level) *slog.Logger {
|
||||||
|
path := filepath.Join(dir, filename)
|
||||||
|
f, err := os.OpenFile(path, os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0666)
|
||||||
|
if err != nil {
|
||||||
|
fallback := slog.New(slog.NewJSONHandler(os.Stderr, &slog.HandlerOptions{Level: level})).With("component", component)
|
||||||
|
fallback.Error("failed to open log file, falling back to stderr", KeyFile, path, KeyError, err.Error())
|
||||||
|
return fallback
|
||||||
|
}
|
||||||
|
return slog.New(slog.NewJSONHandler(f, &slog.HandlerOptions{Level: level})).With("component", component)
|
||||||
|
}
|
||||||
|
|
||||||
|
func parseLevel(level string) slog.Level {
|
||||||
|
switch strings.ToLower(strings.TrimSpace(level)) {
|
||||||
|
case "debug":
|
||||||
|
return slog.LevelDebug
|
||||||
|
case "warn", "warning":
|
||||||
|
return slog.LevelWarn
|
||||||
|
case "error":
|
||||||
|
return slog.LevelError
|
||||||
|
default:
|
||||||
|
return slog.LevelInfo
|
||||||
|
}
|
||||||
|
}
|
||||||
13
pkg/logging/keys.go
Normal file
13
pkg/logging/keys.go
Normal file
@ -0,0 +1,13 @@
|
|||||||
|
package logging
|
||||||
|
|
||||||
|
const (
|
||||||
|
KeyResolution = "resolution"
|
||||||
|
KeyBandwidth = "bandwidth"
|
||||||
|
KeySegment = "segment"
|
||||||
|
KeySeq = "seq"
|
||||||
|
KeyFile = "file"
|
||||||
|
KeyBytes = "bytes"
|
||||||
|
KeyEvent = "event"
|
||||||
|
KeyError = "error"
|
||||||
|
KeyAttempt = "attempt"
|
||||||
|
)
|
||||||
90
pkg/media/manifest.go
Normal file
90
pkg/media/manifest.go
Normal file
@ -0,0 +1,90 @@
|
|||||||
|
package media
|
||||||
|
|
||||||
|
import (
|
||||||
|
"encoding/json"
|
||||||
|
"log"
|
||||||
|
"m3u8-downloader/pkg/constants"
|
||||||
|
"m3u8-downloader/pkg/logging"
|
||||||
|
"m3u8-downloader/pkg/utils"
|
||||||
|
"os"
|
||||||
|
"sort"
|
||||||
|
)
|
||||||
|
|
||||||
|
type ManifestWriter struct {
|
||||||
|
ManifestPath string
|
||||||
|
Segments []ManifestItem
|
||||||
|
Index map[string]*ManifestItem
|
||||||
|
}
|
||||||
|
|
||||||
|
type ManifestItem struct {
|
||||||
|
SeqNo string `json:"seqNo"`
|
||||||
|
Resolution string `json:"resolution"`
|
||||||
|
}
|
||||||
|
|
||||||
|
func NewManifestWriter(eventName string) *ManifestWriter {
|
||||||
|
cfg := constants.MustGetConfig()
|
||||||
|
return &ManifestWriter{
|
||||||
|
ManifestPath: cfg.GetManifestPath(eventName),
|
||||||
|
Segments: make([]ManifestItem, 0),
|
||||||
|
Index: make(map[string]*ManifestItem),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (m *ManifestWriter) AddOrUpdateSegment(seqNo string, resolution string) {
|
||||||
|
if m.Index == nil {
|
||||||
|
m.Index = make(map[string]*ManifestItem)
|
||||||
|
}
|
||||||
|
|
||||||
|
if m.Segments == nil {
|
||||||
|
m.Segments = make([]ManifestItem, 0)
|
||||||
|
}
|
||||||
|
|
||||||
|
if existing, ok := m.Index[seqNo]; ok {
|
||||||
|
if resolution > existing.Resolution {
|
||||||
|
existing.Resolution = resolution
|
||||||
|
}
|
||||||
|
return
|
||||||
|
} else {
|
||||||
|
item := ManifestItem{
|
||||||
|
SeqNo: seqNo,
|
||||||
|
Resolution: resolution,
|
||||||
|
}
|
||||||
|
m.Segments = append(m.Segments, item)
|
||||||
|
m.Index[seqNo] = &item
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (m *ManifestWriter) WriteManifest() {
|
||||||
|
sort.Slice(m.Segments, func(i, j int) bool {
|
||||||
|
return m.Segments[i].SeqNo < m.Segments[j].SeqNo
|
||||||
|
})
|
||||||
|
|
||||||
|
data, err := json.MarshalIndent(m.Segments, "", " ")
|
||||||
|
if err != nil {
|
||||||
|
log.Printf("Failed to marshal manifest: %v", err)
|
||||||
|
logging.MustGetFactory().Download.Error("failed to marshal manifest", logging.KeyError, err.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := utils.ValidateWritablePath(m.ManifestPath); err != nil {
|
||||||
|
log.Printf("Manifest path validation failed: %v", err)
|
||||||
|
logging.MustGetFactory().Download.Error("manifest path validation failed", logging.KeyError, err.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
file, err := os.Create(m.ManifestPath)
|
||||||
|
if err != nil {
|
||||||
|
log.Printf("Failed to create manifest file: %v", err)
|
||||||
|
logging.MustGetFactory().Download.Error("failed to create manifest file", logging.KeyError, err.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
defer file.Close()
|
||||||
|
|
||||||
|
_, err = file.Write(data)
|
||||||
|
if err != nil {
|
||||||
|
log.Printf("Failed to write manifest file: %v", err)
|
||||||
|
logging.MustGetFactory().Download.Error("failed to write manifest file", logging.KeyError, err.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
}
|
||||||
241
pkg/media/manifest_test.go
Normal file
241
pkg/media/manifest_test.go
Normal file
@ -0,0 +1,241 @@
|
|||||||
|
package media
|
||||||
|
|
||||||
|
import (
|
||||||
|
"encoding/json"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestManifestWriter_NewManifestWriter(t *testing.T) {
|
||||||
|
// Set up temporary environment for testing
|
||||||
|
tempDir, err := os.MkdirTemp("", "manifest_test_*")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Failed to create temp dir: %v", err)
|
||||||
|
}
|
||||||
|
defer os.RemoveAll(tempDir)
|
||||||
|
|
||||||
|
// Set environment variable to use temp directory
|
||||||
|
os.Setenv("LOCAL_OUTPUT_DIR", tempDir)
|
||||||
|
defer os.Unsetenv("LOCAL_OUTPUT_DIR")
|
||||||
|
|
||||||
|
eventName := "test-event"
|
||||||
|
writer := NewManifestWriter(eventName)
|
||||||
|
|
||||||
|
if writer == nil {
|
||||||
|
t.Fatal("NewManifestWriter() returned nil")
|
||||||
|
}
|
||||||
|
|
||||||
|
if writer.Segments == nil {
|
||||||
|
t.Error("Segments should be initialized")
|
||||||
|
}
|
||||||
|
if writer.Index == nil {
|
||||||
|
t.Error("Index should be initialized")
|
||||||
|
}
|
||||||
|
if len(writer.Segments) != 0 {
|
||||||
|
t.Errorf("Segments should be empty, got %d items", len(writer.Segments))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestManifestWriter_AddOrUpdateSegment(t *testing.T) {
|
||||||
|
writer := &ManifestWriter{
|
||||||
|
ManifestPath: "test.json",
|
||||||
|
Segments: make([]ManifestItem, 0),
|
||||||
|
Index: make(map[string]*ManifestItem),
|
||||||
|
}
|
||||||
|
|
||||||
|
// Test adding new segment
|
||||||
|
writer.AddOrUpdateSegment("1001", "1080p")
|
||||||
|
|
||||||
|
if len(writer.Segments) != 1 {
|
||||||
|
t.Errorf("Expected 1 segment, got %d", len(writer.Segments))
|
||||||
|
}
|
||||||
|
if writer.Segments[0].SeqNo != "1001" {
|
||||||
|
t.Errorf("Expected SeqNo '1001', got '%s'", writer.Segments[0].SeqNo)
|
||||||
|
}
|
||||||
|
if writer.Segments[0].Resolution != "1080p" {
|
||||||
|
t.Errorf("Expected Resolution '1080p', got '%s'", writer.Segments[0].Resolution)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Test updating existing segment with higher resolution
|
||||||
|
writer.AddOrUpdateSegment("1001", "1440p")
|
||||||
|
|
||||||
|
if len(writer.Segments) != 1 {
|
||||||
|
t.Errorf("Segments count should remain 1 after update, got %d", len(writer.Segments))
|
||||||
|
}
|
||||||
|
if writer.Segments[0].Resolution != "1440p" {
|
||||||
|
t.Errorf("Expected updated resolution '1440p', got '%s'", writer.Segments[0].Resolution)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Test updating existing segment with lower resolution (should not change)
|
||||||
|
writer.AddOrUpdateSegment("1001", "720p")
|
||||||
|
|
||||||
|
if writer.Segments[0].Resolution != "1440p" {
|
||||||
|
t.Errorf("Resolution should remain '1440p', got '%s'", writer.Segments[0].Resolution)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Test adding different segment
|
||||||
|
writer.AddOrUpdateSegment("1002", "720p")
|
||||||
|
|
||||||
|
if len(writer.Segments) != 2 {
|
||||||
|
t.Errorf("Expected 2 segments, got %d", len(writer.Segments))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestManifestWriter_AddOrUpdateSegment_NilFields(t *testing.T) {
|
||||||
|
writer := &ManifestWriter{
|
||||||
|
ManifestPath: "test.json",
|
||||||
|
}
|
||||||
|
|
||||||
|
// Test with nil fields (should initialize them)
|
||||||
|
writer.AddOrUpdateSegment("1001", "1080p")
|
||||||
|
|
||||||
|
if writer.Segments == nil {
|
||||||
|
t.Error("Segments should be initialized")
|
||||||
|
}
|
||||||
|
if writer.Index == nil {
|
||||||
|
t.Error("Index should be initialized")
|
||||||
|
}
|
||||||
|
if len(writer.Segments) != 1 {
|
||||||
|
t.Errorf("Expected 1 segment, got %d", len(writer.Segments))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestManifestWriter_WriteManifest(t *testing.T) {
|
||||||
|
tempDir, err := os.MkdirTemp("", "manifest_test_*")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Failed to create temp dir: %v", err)
|
||||||
|
}
|
||||||
|
defer os.RemoveAll(tempDir)
|
||||||
|
|
||||||
|
manifestPath := filepath.Join(tempDir, "test-manifest.json")
|
||||||
|
writer := &ManifestWriter{
|
||||||
|
ManifestPath: manifestPath,
|
||||||
|
Segments: make([]ManifestItem, 0),
|
||||||
|
Index: make(map[string]*ManifestItem),
|
||||||
|
}
|
||||||
|
|
||||||
|
// Add some test segments out of order
|
||||||
|
writer.AddOrUpdateSegment("1003", "1080p")
|
||||||
|
writer.AddOrUpdateSegment("1001", "720p")
|
||||||
|
writer.AddOrUpdateSegment("1002", "1080p")
|
||||||
|
|
||||||
|
// Write manifest
|
||||||
|
writer.WriteManifest()
|
||||||
|
|
||||||
|
// Verify file was created
|
||||||
|
if _, err := os.Stat(manifestPath); os.IsNotExist(err) {
|
||||||
|
t.Fatalf("Manifest file was not created: %s", manifestPath)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Read and verify content
|
||||||
|
content, err := os.ReadFile(manifestPath)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Failed to read manifest file: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
var segments []ManifestItem
|
||||||
|
err = json.Unmarshal(content, &segments)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Failed to unmarshal manifest JSON: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Verify segments are sorted by sequence number
|
||||||
|
if len(segments) != 3 {
|
||||||
|
t.Errorf("Expected 3 segments in manifest, got %d", len(segments))
|
||||||
|
}
|
||||||
|
|
||||||
|
expectedOrder := []string{"1001", "1002", "1003"}
|
||||||
|
for i, segment := range segments {
|
||||||
|
if segment.SeqNo != expectedOrder[i] {
|
||||||
|
t.Errorf("Segment %d: expected SeqNo '%s', got '%s'", i, expectedOrder[i], segment.SeqNo)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Verify content structure
|
||||||
|
if segments[0].Resolution != "720p" {
|
||||||
|
t.Errorf("Expected first segment resolution '720p', got '%s'", segments[0].Resolution)
|
||||||
|
}
|
||||||
|
if segments[1].Resolution != "1080p" {
|
||||||
|
t.Errorf("Expected second segment resolution '1080p', got '%s'", segments[1].Resolution)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestManifestWriter_WriteManifest_EmptySegments(t *testing.T) {
|
||||||
|
tempDir, err := os.MkdirTemp("", "manifest_test_*")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Failed to create temp dir: %v", err)
|
||||||
|
}
|
||||||
|
defer os.RemoveAll(tempDir)
|
||||||
|
|
||||||
|
manifestPath := filepath.Join(tempDir, "empty-manifest.json")
|
||||||
|
writer := &ManifestWriter{
|
||||||
|
ManifestPath: manifestPath,
|
||||||
|
Segments: make([]ManifestItem, 0),
|
||||||
|
Index: make(map[string]*ManifestItem),
|
||||||
|
}
|
||||||
|
|
||||||
|
// Write empty manifest
|
||||||
|
writer.WriteManifest()
|
||||||
|
|
||||||
|
// Verify file was created
|
||||||
|
if _, err := os.Stat(manifestPath); os.IsNotExist(err) {
|
||||||
|
t.Fatalf("Manifest file was not created: %s", manifestPath)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Read and verify content is empty array
|
||||||
|
content, err := os.ReadFile(manifestPath)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Failed to read manifest file: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
var segments []ManifestItem
|
||||||
|
err = json.Unmarshal(content, &segments)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Failed to unmarshal manifest JSON: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if len(segments) != 0 {
|
||||||
|
t.Errorf("Expected empty segments array, got %d items", len(segments))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestManifestWriter_WriteManifest_InvalidPath(t *testing.T) {
|
||||||
|
writer := &ManifestWriter{
|
||||||
|
ManifestPath: "/invalid/path/that/does/not/exist/manifest.json",
|
||||||
|
Segments: []ManifestItem{{SeqNo: "1001", Resolution: "1080p"}},
|
||||||
|
Index: make(map[string]*ManifestItem),
|
||||||
|
}
|
||||||
|
|
||||||
|
// This should not panic, just fail gracefully
|
||||||
|
writer.WriteManifest()
|
||||||
|
|
||||||
|
// Test passes if no panic occurs
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestManifestItem_JSONSerialization(t *testing.T) {
|
||||||
|
item := ManifestItem{
|
||||||
|
SeqNo: "1001",
|
||||||
|
Resolution: "1080p",
|
||||||
|
}
|
||||||
|
|
||||||
|
// Test marshaling
|
||||||
|
data, err := json.Marshal(item)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Failed to marshal ManifestItem: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Test unmarshaling
|
||||||
|
var unmarshaled ManifestItem
|
||||||
|
err = json.Unmarshal(data, &unmarshaled)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Failed to unmarshal ManifestItem: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if unmarshaled.SeqNo != item.SeqNo {
|
||||||
|
t.Errorf("SeqNo mismatch: expected '%s', got '%s'", item.SeqNo, unmarshaled.SeqNo)
|
||||||
|
}
|
||||||
|
if unmarshaled.Resolution != item.Resolution {
|
||||||
|
t.Errorf("Resolution mismatch: expected '%s', got '%s'", item.Resolution, unmarshaled.Resolution)
|
||||||
|
}
|
||||||
|
}
|
||||||
@ -6,6 +6,7 @@ import (
|
|||||||
"io"
|
"io"
|
||||||
"m3u8-downloader/pkg/constants"
|
"m3u8-downloader/pkg/constants"
|
||||||
"m3u8-downloader/pkg/httpClient"
|
"m3u8-downloader/pkg/httpClient"
|
||||||
|
"m3u8-downloader/pkg/logging"
|
||||||
"net/http"
|
"net/http"
|
||||||
"net/url"
|
"net/url"
|
||||||
"os"
|
"os"
|
||||||
@ -33,6 +34,7 @@ func (j SegmentJob) Key() string {
|
|||||||
func DownloadSegment(ctx context.Context, client *http.Client, segmentURL string, outputDir string) error {
|
func DownloadSegment(ctx context.Context, client *http.Client, segmentURL string, outputDir string) error {
|
||||||
for attempt := 0; attempt < 2; attempt++ {
|
for attempt := 0; attempt < 2; attempt++ {
|
||||||
if attempt > 0 {
|
if attempt > 0 {
|
||||||
|
logging.MustGetFactory().Download.Debug("retrying segment download", logging.KeyAttempt, attempt, "url", segmentURL)
|
||||||
time.Sleep(300 * time.Millisecond)
|
time.Sleep(300 * time.Millisecond)
|
||||||
}
|
}
|
||||||
req, err := http.NewRequestWithContext(ctx, "GET", segmentURL, nil)
|
req, err := http.NewRequestWithContext(ctx, "GET", segmentURL, nil)
|
||||||
|
|||||||
@ -2,11 +2,14 @@ package media
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
|
"errors"
|
||||||
"fmt"
|
"fmt"
|
||||||
"github.com/grafov/m3u8"
|
"github.com/grafov/m3u8"
|
||||||
"log"
|
"log"
|
||||||
"m3u8-downloader/pkg/constants"
|
"m3u8-downloader/pkg/constants"
|
||||||
|
"m3u8-downloader/pkg/health"
|
||||||
"m3u8-downloader/pkg/httpClient"
|
"m3u8-downloader/pkg/httpClient"
|
||||||
|
"m3u8-downloader/pkg/logging"
|
||||||
"net/http"
|
"net/http"
|
||||||
"net/url"
|
"net/url"
|
||||||
"path"
|
"path"
|
||||||
@ -21,6 +24,7 @@ type StreamVariant struct {
|
|||||||
ID int
|
ID int
|
||||||
Resolution string
|
Resolution string
|
||||||
OutputDir string
|
OutputDir string
|
||||||
|
Writer *ManifestWriter
|
||||||
}
|
}
|
||||||
|
|
||||||
func extractResolution(variant *m3u8.Variant) string {
|
func extractResolution(variant *m3u8.Variant) string {
|
||||||
@ -44,19 +48,21 @@ func extractResolution(variant *m3u8.Variant) string {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func GetAllVariants(masterURL string) ([]*StreamVariant, error) {
|
func GetAllVariants(masterURL string, outputDir string, writer *ManifestWriter) ([]*StreamVariant, error) {
|
||||||
client := &http.Client{}
|
client := &http.Client{}
|
||||||
req, _ := http.NewRequest("GET", masterURL, nil)
|
req, _ := http.NewRequest("GET", masterURL, nil)
|
||||||
req.Header.Set("User-Agent", constants.HTTPUserAgent)
|
req.Header.Set("User-Agent", constants.HTTPUserAgent)
|
||||||
req.Header.Set("Referer", constants.REFERRER)
|
req.Header.Set("Referer", constants.REFERRER)
|
||||||
resp, err := client.Do(req)
|
resp, err := client.Do(req)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
logging.MustGetFactory().Download.Error("failed to fetch master playlist", logging.KeyError, err.Error())
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
defer resp.Body.Close()
|
defer resp.Body.Close()
|
||||||
|
|
||||||
playlist, listType, err := m3u8.DecodeFrom(resp.Body, true)
|
playlist, listType, err := m3u8.DecodeFrom(resp.Body, true)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
logging.MustGetFactory().Download.Error("failed to decode master playlist", logging.KeyError, err.Error())
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
|
|
||||||
@ -69,7 +75,8 @@ func GetAllVariants(masterURL string) ([]*StreamVariant, error) {
|
|||||||
BaseURL: base,
|
BaseURL: base,
|
||||||
ID: 0,
|
ID: 0,
|
||||||
Resolution: "unknown",
|
Resolution: "unknown",
|
||||||
OutputDir: path.Join(constants.NASPath, "unknown"),
|
OutputDir: path.Join(outputDir, "unknown"),
|
||||||
|
Writer: writer,
|
||||||
}}, nil
|
}}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
@ -83,7 +90,7 @@ func GetAllVariants(masterURL string) ([]*StreamVariant, error) {
|
|||||||
vURL, _ := url.Parse(v.URI)
|
vURL, _ := url.Parse(v.URI)
|
||||||
fullURL := base.ResolveReference(vURL).String()
|
fullURL := base.ResolveReference(vURL).String()
|
||||||
resolution := extractResolution(v)
|
resolution := extractResolution(v)
|
||||||
outputDir := path.Join(constants.NASPath, resolution)
|
outputDir := path.Join(outputDir, resolution)
|
||||||
variants = append(variants, &StreamVariant{
|
variants = append(variants, &StreamVariant{
|
||||||
URL: fullURL,
|
URL: fullURL,
|
||||||
Bandwidth: v.Bandwidth,
|
Bandwidth: v.Bandwidth,
|
||||||
@ -93,11 +100,17 @@ func GetAllVariants(masterURL string) ([]*StreamVariant, error) {
|
|||||||
OutputDir: outputDir,
|
OutputDir: outputDir,
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
logging.MustGetFactory().Download.Info("discovered variants", "count", len(variants))
|
||||||
return variants, nil
|
return variants, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func VariantDownloader(ctx context.Context, variant *StreamVariant, sem chan struct{}) {
|
func VariantDownloader(ctx context.Context, variant *StreamVariant, sem chan struct{}, manifest *ManifestWriter) {
|
||||||
log.Printf("Starting %s variant downloader (bandwidth: %d)", variant.Resolution, variant.Bandwidth)
|
log.Printf("Starting %s variant downloader (bandwidth: %d)", variant.Resolution, variant.Bandwidth)
|
||||||
|
dlLog := logging.MustGetFactory().Download.With(logging.KeyResolution, variant.Resolution)
|
||||||
|
registry := health.MustGetRegistry()
|
||||||
|
dlLog.Info("variant downloader started", logging.KeyBandwidth, variant.Bandwidth)
|
||||||
|
registry.RecordVariantStarted(variant.Resolution, variant.Bandwidth)
|
||||||
|
|
||||||
ticker := time.NewTicker(constants.RefreshDelay)
|
ticker := time.NewTicker(constants.RefreshDelay)
|
||||||
defer ticker.Stop()
|
defer ticker.Stop()
|
||||||
client := &http.Client{}
|
client := &http.Client{}
|
||||||
@ -106,16 +119,20 @@ func VariantDownloader(ctx context.Context, variant *StreamVariant, sem chan str
|
|||||||
for {
|
for {
|
||||||
select {
|
select {
|
||||||
case <-ctx.Done():
|
case <-ctx.Done():
|
||||||
|
registry.RecordVariantStopped(variant.Resolution)
|
||||||
return
|
return
|
||||||
default:
|
default:
|
||||||
}
|
}
|
||||||
|
|
||||||
|
var seq uint64
|
||||||
playlist, err := LoadMediaPlaylist(variant.URL)
|
playlist, err := LoadMediaPlaylist(variant.URL)
|
||||||
seq := playlist.SeqNo
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
log.Printf("%s: Error loading playlist playlist: %v", variant.Resolution, err)
|
log.Printf("%s: Error loading playlist playlist: %v", variant.Resolution, err)
|
||||||
|
dlLog.Warn("failed to load media playlist", logging.KeyError, err.Error())
|
||||||
|
registry.RecordPlaylistError(variant.Resolution, err)
|
||||||
goto waitTick
|
goto waitTick
|
||||||
}
|
}
|
||||||
|
seq = playlist.SeqNo
|
||||||
|
|
||||||
for _, seg := range playlist.Segments {
|
for _, seg := range playlist.Segments {
|
||||||
if seg == nil {
|
if seg == nil {
|
||||||
@ -142,25 +159,42 @@ func VariantDownloader(ctx context.Context, variant *StreamVariant, sem chan str
|
|||||||
|
|
||||||
err := DownloadSegment(ctx, client, j.AbsoluteURL(), j.Variant.OutputDir)
|
err := DownloadSegment(ctx, client, j.AbsoluteURL(), j.Variant.OutputDir)
|
||||||
name := strings.TrimSuffix(path.Base(j.Key()), path.Ext(path.Base(j.Key())))
|
name := strings.TrimSuffix(path.Base(j.Key()), path.Ext(path.Base(j.Key())))
|
||||||
|
|
||||||
if err == nil {
|
if err == nil {
|
||||||
log.Printf("✓ %s downloaded segment %s", j.Variant.Resolution, name)
|
log.Printf("✓ %s downloaded segment %s", j.Variant.Resolution, name)
|
||||||
} else if httpClient.IsHTTPStatus(err, 403) {
|
dlLog.Debug("segment downloaded", logging.KeySegment, name)
|
||||||
|
registry.RecordSegmentDownloaded(variant.Resolution)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
if errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) {
|
||||||
|
// Suppress log: shutdown in progress
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
if httpClient.IsHTTPStatus(err, 403) {
|
||||||
log.Printf("✗ %s failed to download segment %s (403)", j.Variant.Resolution, name)
|
log.Printf("✗ %s failed to download segment %s (403)", j.Variant.Resolution, name)
|
||||||
|
dlLog.Warn("segment download failed (403)", logging.KeySegment, name)
|
||||||
} else {
|
} else {
|
||||||
log.Printf("✗ %s failed to download segment %s: %v", j.Variant.Resolution, name, err)
|
log.Printf("✗ %s failed to download segment %s: %v", j.Variant.Resolution, name, err)
|
||||||
|
dlLog.Error("segment download failed", logging.KeySegment, name, logging.KeyError, err.Error())
|
||||||
}
|
}
|
||||||
|
registry.RecordSegmentFailed(variant.Resolution, err)
|
||||||
}(job)
|
}(job)
|
||||||
seq++
|
seq++
|
||||||
}
|
}
|
||||||
|
|
||||||
if playlist.Closed {
|
if playlist.Closed {
|
||||||
log.Printf("%s: Playlist closed (#EXT-X-ENDLIST)", variant.Resolution)
|
log.Printf("%s: Playlist closed (#EXT-X-ENDLIST)", variant.Resolution)
|
||||||
|
dlLog.Info("playlist closed")
|
||||||
|
registry.RecordVariantStopped(variant.Resolution)
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
waitTick:
|
waitTick:
|
||||||
select {
|
select {
|
||||||
case <-ctx.Done():
|
case <-ctx.Done():
|
||||||
|
registry.RecordVariantStopped(variant.Resolution)
|
||||||
return
|
return
|
||||||
case <-ticker.C:
|
case <-ticker.C:
|
||||||
}
|
}
|
||||||
|
|||||||
12
pkg/nas/config.go
Normal file
12
pkg/nas/config.go
Normal file
@ -0,0 +1,12 @@
|
|||||||
|
package nas
|
||||||
|
|
||||||
|
import "time"
|
||||||
|
|
||||||
|
type NASConfig struct {
|
||||||
|
Path string
|
||||||
|
Username string
|
||||||
|
Password string
|
||||||
|
Timeout time.Duration
|
||||||
|
RetryLimit int
|
||||||
|
VerifySize bool
|
||||||
|
}
|
||||||
202
pkg/nas/nas.go
Normal file
202
pkg/nas/nas.go
Normal file
@ -0,0 +1,202 @@
|
|||||||
|
package nas
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"log"
|
||||||
|
"os"
|
||||||
|
"os/exec"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
)
|
||||||
|
|
||||||
|
type NASService struct {
|
||||||
|
Config NASConfig
|
||||||
|
connected bool
|
||||||
|
}
|
||||||
|
|
||||||
|
func NewNASService(config NASConfig) *NASService {
|
||||||
|
nt := &NASService{
|
||||||
|
Config: config,
|
||||||
|
}
|
||||||
|
|
||||||
|
// Establish network connection with credentials before accessing the path
|
||||||
|
if err := nt.EstablishConnection(); err != nil {
|
||||||
|
log.Fatalf("Failed to establish network connection to %s: %v", nt.Config.Path, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
err := nt.EnsureDirectoryExists(nt.Config.Path)
|
||||||
|
if err != nil {
|
||||||
|
log.Fatalf("Failed to create directory %s: %v", nt.Config.Path, err)
|
||||||
|
}
|
||||||
|
return nt
|
||||||
|
}
|
||||||
|
|
||||||
|
func (nt *NASService) CopyFile(ctx context.Context, srcPath, destPath string) error {
|
||||||
|
src, err := os.Open(srcPath)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("Failed to open source file: %w", err)
|
||||||
|
}
|
||||||
|
defer src.Close()
|
||||||
|
|
||||||
|
dest, err := os.Create(destPath)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("Failed to create destination file: %w", err)
|
||||||
|
}
|
||||||
|
defer dest.Close()
|
||||||
|
|
||||||
|
done := make(chan error, 1)
|
||||||
|
go func() {
|
||||||
|
_, err := io.Copy(dest, src)
|
||||||
|
done <- err
|
||||||
|
}()
|
||||||
|
|
||||||
|
select {
|
||||||
|
case <-ctx.Done():
|
||||||
|
return ctx.Err()
|
||||||
|
case err := <-done:
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
return dest.Sync()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (nt *NASService) VerifyTransfer(srcPath, destPath string) error {
|
||||||
|
srcInfo, err := os.Stat(srcPath)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("Failed to stat source file: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
destInfo, err := os.Stat(destPath)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("Failed to stat destination file: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if srcInfo.Size() != destInfo.Size() {
|
||||||
|
return fmt.Errorf("size mismatch: source=%d, dest=%d", srcInfo.Size(), destInfo.Size())
|
||||||
|
}
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (nt *NASService) EnsureDirectoryExists(path string) error {
|
||||||
|
if err := os.MkdirAll(path, 0755); err != nil {
|
||||||
|
return fmt.Errorf("Failed to create directory: %w", err)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (nt *NASService) EstablishConnection() error {
|
||||||
|
networkPath := nt.ExtractNetworkPath(nt.Config.Path)
|
||||||
|
if networkPath == "" {
|
||||||
|
return nil // local path, no network mount needed
|
||||||
|
}
|
||||||
|
|
||||||
|
log.Printf("Establishing network connection to %s with user %s", networkPath, nt.Config.Username)
|
||||||
|
|
||||||
|
var cmd *exec.Cmd
|
||||||
|
if nt.Config.Username != "" && nt.Config.Password != "" {
|
||||||
|
cmd = exec.Command("net", "use", networkPath, "/user:"+nt.Config.Username, nt.Config.Password, "/persistent:no")
|
||||||
|
} else {
|
||||||
|
cmd = exec.Command("net", "use", networkPath, "/persistent:no")
|
||||||
|
}
|
||||||
|
|
||||||
|
output, err := cmd.CombinedOutput()
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("failed to establish network connection: %w\nOutput: %s", err, string(output))
|
||||||
|
}
|
||||||
|
|
||||||
|
log.Printf("Network connection established successfully")
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (nt *NASService) ExtractNetworkPath(fullPath string) string {
|
||||||
|
// Extract \\server\share from paths like \\server\share\folder\subfolder
|
||||||
|
if !strings.HasPrefix(fullPath, "\\\\") {
|
||||||
|
return "" // Not a UNC path
|
||||||
|
}
|
||||||
|
|
||||||
|
parts := strings.Split(fullPath[2:], "\\") // Remove leading \\
|
||||||
|
if len(parts) < 2 {
|
||||||
|
return "" // Invalid UNC path
|
||||||
|
}
|
||||||
|
|
||||||
|
return "\\\\" + parts[0] + "\\" + parts[1]
|
||||||
|
}
|
||||||
|
|
||||||
|
func (nt *NASService) TestConnection() error {
|
||||||
|
testFile := filepath.Join(nt.Config.Path, ".connection_test")
|
||||||
|
|
||||||
|
f, err := os.Create(testFile)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("Failed to create test file: %w", err)
|
||||||
|
}
|
||||||
|
f.Close()
|
||||||
|
|
||||||
|
os.Remove(testFile)
|
||||||
|
|
||||||
|
nt.connected = true
|
||||||
|
log.Printf("Connected to NAS at %s", nt.Config.Path)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (nt *NASService) IsConnected() bool {
|
||||||
|
return nt.connected
|
||||||
|
}
|
||||||
|
|
||||||
|
// Disconnect removes the network connection
|
||||||
|
func (nt *NASService) Disconnect() error {
|
||||||
|
networkPath := nt.ExtractNetworkPath(nt.Config.Path)
|
||||||
|
if networkPath == "" {
|
||||||
|
return nil // Local path, nothing to disconnect
|
||||||
|
}
|
||||||
|
|
||||||
|
cmd := exec.Command("net", "use", networkPath, "/delete")
|
||||||
|
output, err := cmd.CombinedOutput()
|
||||||
|
if err != nil {
|
||||||
|
log.Printf("Warning: failed to disconnect from %s: %v\nOutput: %s", networkPath, err, string(output))
|
||||||
|
// Don't return error since this is cleanup
|
||||||
|
} else {
|
||||||
|
log.Printf("Disconnected from network path: %s", networkPath)
|
||||||
|
}
|
||||||
|
|
||||||
|
nt.connected = false
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// FileExists checks if a file already exists on the NAS and optionally verifies size
|
||||||
|
func (nt *NASService) FileExists(destinationPath string, expectedSize int64) (bool, error) {
|
||||||
|
fullDestPath := filepath.Join(nt.Config.Path, destinationPath)
|
||||||
|
|
||||||
|
destInfo, err := os.Stat(fullDestPath)
|
||||||
|
if err != nil {
|
||||||
|
if os.IsNotExist(err) {
|
||||||
|
return false, nil // File doesn't exist, no error
|
||||||
|
}
|
||||||
|
return false, fmt.Errorf("failed to stat NAS file: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// File exists, check size if expected size is provided
|
||||||
|
if expectedSize > 0 && destInfo.Size() != expectedSize {
|
||||||
|
log.Printf("NAS file size mismatch for %s: expected=%d, actual=%d",
|
||||||
|
fullDestPath, expectedSize, destInfo.Size())
|
||||||
|
return false, nil // File exists but wrong size, treat as not existing
|
||||||
|
}
|
||||||
|
|
||||||
|
return true, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// GetFileSize returns the size of a file on the NAS
|
||||||
|
func (nt *NASService) GetFileSize(destinationPath string) (int64, error) {
|
||||||
|
fullDestPath := filepath.Join(nt.Config.Path, destinationPath)
|
||||||
|
|
||||||
|
destInfo, err := os.Stat(fullDestPath)
|
||||||
|
if err != nil {
|
||||||
|
return 0, fmt.Errorf("failed to stat NAS file: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
return destInfo.Size(), nil
|
||||||
|
}
|
||||||
7
pkg/processing/segment.go
Normal file
7
pkg/processing/segment.go
Normal file
@ -0,0 +1,7 @@
|
|||||||
|
package processing
|
||||||
|
|
||||||
|
type SegmentInfo struct {
|
||||||
|
Name string
|
||||||
|
SeqNo int
|
||||||
|
Resolution string
|
||||||
|
}
|
||||||
365
pkg/processing/service.go
Normal file
365
pkg/processing/service.go
Normal file
@ -0,0 +1,365 @@
|
|||||||
|
package processing
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bufio"
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"log"
|
||||||
|
"m3u8-downloader/pkg/config"
|
||||||
|
"m3u8-downloader/pkg/health"
|
||||||
|
"m3u8-downloader/pkg/logging"
|
||||||
|
"m3u8-downloader/pkg/nas"
|
||||||
|
"m3u8-downloader/pkg/utils"
|
||||||
|
"os"
|
||||||
|
"os/exec"
|
||||||
|
"path/filepath"
|
||||||
|
"regexp"
|
||||||
|
"runtime"
|
||||||
|
"sort"
|
||||||
|
"strconv"
|
||||||
|
"strings"
|
||||||
|
"sync"
|
||||||
|
)
|
||||||
|
|
||||||
|
type ProcessingService struct {
|
||||||
|
config *config.Config
|
||||||
|
eventName string
|
||||||
|
nas *nas.NASService
|
||||||
|
}
|
||||||
|
|
||||||
|
func NewProcessingService(eventName string, cfg *config.Config) (*ProcessingService, error) {
|
||||||
|
if cfg == nil {
|
||||||
|
return nil, fmt.Errorf("configuration is required")
|
||||||
|
}
|
||||||
|
|
||||||
|
nasConfig := nas.NASConfig{
|
||||||
|
Path: cfg.NAS.OutputPath,
|
||||||
|
Username: cfg.NAS.Username,
|
||||||
|
Password: cfg.NAS.Password,
|
||||||
|
Timeout: cfg.NAS.Timeout,
|
||||||
|
RetryLimit: cfg.NAS.RetryLimit,
|
||||||
|
VerifySize: true,
|
||||||
|
}
|
||||||
|
|
||||||
|
nasService := nas.NewNASService(nasConfig)
|
||||||
|
|
||||||
|
if err := nasService.TestConnection(); err != nil {
|
||||||
|
logging.MustGetFactory().Processing.Error("failed to connect to NAS", logging.KeyError, err.Error())
|
||||||
|
return nil, fmt.Errorf("failed to connect to NAS: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
return &ProcessingService{
|
||||||
|
config: cfg,
|
||||||
|
eventName: eventName,
|
||||||
|
nas: nasService,
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (ps *ProcessingService) GetSegmentInfo() (map[int]string, error) {
|
||||||
|
return nil, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (ps *ProcessingService) GetEventDirs() ([]string, error) {
|
||||||
|
if ps.eventName == "" {
|
||||||
|
sourcePath := ps.config.NAS.OutputPath
|
||||||
|
dirs, err := os.ReadDir(sourcePath)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("failed to read directory %s: %w", sourcePath, err)
|
||||||
|
}
|
||||||
|
var eventDirs []string
|
||||||
|
for _, dir := range dirs {
|
||||||
|
if dir.IsDir() {
|
||||||
|
eventDirs = append(eventDirs, dir.Name())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return eventDirs, nil
|
||||||
|
} else {
|
||||||
|
return []string{ps.eventName}, nil
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (ps *ProcessingService) Start(ctx context.Context) error {
|
||||||
|
procLog := logging.MustGetFactory().Processing
|
||||||
|
registry := health.MustGetRegistry()
|
||||||
|
|
||||||
|
if !ps.config.Processing.Enabled {
|
||||||
|
log.Println("Processing service disabled")
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
registry.RecordProcessingStageChange("scanning", ps.eventName)
|
||||||
|
|
||||||
|
// fail centralizes the log/registry bookkeeping for every error return
|
||||||
|
// site below instead of repeating it at each one.
|
||||||
|
fail := func(err error) error {
|
||||||
|
procLog.Error("processing failed", logging.KeyEvent, ps.eventName, logging.KeyError, err.Error())
|
||||||
|
registry.RecordProcessingStageChange("failed", ps.eventName)
|
||||||
|
registry.RecordProcessingError(err)
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
if ps.eventName == "" {
|
||||||
|
events, err := ps.GetEventDirs()
|
||||||
|
if err != nil {
|
||||||
|
return fail(fmt.Errorf("failed to get event directories: %w", err))
|
||||||
|
}
|
||||||
|
if len(events) == 0 {
|
||||||
|
return fail(fmt.Errorf("no events found"))
|
||||||
|
}
|
||||||
|
if len(events) > 1 {
|
||||||
|
fmt.Println("Multiple events found, please select one:")
|
||||||
|
for i, event := range events {
|
||||||
|
fmt.Printf("%d. %s\n", i+1, event)
|
||||||
|
}
|
||||||
|
reader := bufio.NewReader(os.Stdin)
|
||||||
|
input, _ := reader.ReadString('\n')
|
||||||
|
input = strings.TrimSpace(input)
|
||||||
|
index, err := strconv.Atoi(input)
|
||||||
|
if err != nil {
|
||||||
|
return fail(fmt.Errorf("failed to parse input: %w", err))
|
||||||
|
}
|
||||||
|
if index < 1 || index > len(events) {
|
||||||
|
return fail(fmt.Errorf("invalid input"))
|
||||||
|
}
|
||||||
|
ps.eventName = events[index-1]
|
||||||
|
} else {
|
||||||
|
ps.eventName = events[0]
|
||||||
|
}
|
||||||
|
registry.RecordProcessingStageChange("scanning", ps.eventName)
|
||||||
|
}
|
||||||
|
procLog.Info("processing started", logging.KeyEvent, ps.eventName)
|
||||||
|
|
||||||
|
//Get all present resolutions
|
||||||
|
dirs, err := ps.GetResolutions()
|
||||||
|
if err != nil {
|
||||||
|
return fail(fmt.Errorf("Failed to get resolutions: %w", err))
|
||||||
|
}
|
||||||
|
|
||||||
|
//Spawn a worker per resolution
|
||||||
|
ch := make(chan SegmentInfo, 100)
|
||||||
|
var wg sync.WaitGroup
|
||||||
|
|
||||||
|
for _, resolution := range dirs {
|
||||||
|
wg.Add(1)
|
||||||
|
go ps.ParseResolutionDirectory(resolution, ch, &wg)
|
||||||
|
}
|
||||||
|
go func() {
|
||||||
|
wg.Wait()
|
||||||
|
close(ch)
|
||||||
|
}()
|
||||||
|
|
||||||
|
segments, err := ps.AggregateSegmentInfo(ch)
|
||||||
|
if err != nil {
|
||||||
|
return fail(fmt.Errorf("Failed to aggregate segment info: %w", err))
|
||||||
|
}
|
||||||
|
registry.RecordProcessingSegmentsFound(len(segments))
|
||||||
|
|
||||||
|
aggFile, err := ps.WriteConcatFile(segments)
|
||||||
|
if err != nil {
|
||||||
|
return fail(fmt.Errorf("Failed to write concat file: %w", err))
|
||||||
|
}
|
||||||
|
|
||||||
|
// Feed info to ffmpeg to stitch files together
|
||||||
|
outPath := ps.config.GetProcessOutputPath(ps.eventName)
|
||||||
|
if err := utils.EnsureDir(outPath); err != nil {
|
||||||
|
return fail(fmt.Errorf("failed to create output directory: %w", err))
|
||||||
|
}
|
||||||
|
|
||||||
|
registry.RecordProcessingStageChange("running", ps.eventName)
|
||||||
|
concatErr := ps.RunFFmpeg(aggFile, outPath)
|
||||||
|
if concatErr != nil {
|
||||||
|
return fail(concatErr)
|
||||||
|
}
|
||||||
|
|
||||||
|
procLog.Info("processing completed", logging.KeyEvent, ps.eventName, "segments", len(segments))
|
||||||
|
registry.RecordProcessingStageChange("completed", ps.eventName)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (ps *ProcessingService) GetResolutions() ([]string, error) {
|
||||||
|
eventPath := ps.config.GetNASEventPath(ps.eventName)
|
||||||
|
dirs, err := os.ReadDir(eventPath)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("failed to read source directory %s: %w", eventPath, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
re := regexp.MustCompile(`^\d+p$`)
|
||||||
|
|
||||||
|
var resolutions []string
|
||||||
|
for _, dir := range dirs {
|
||||||
|
if dir.IsDir() && re.MatchString(dir.Name()) {
|
||||||
|
resolutions = append(resolutions, dir.Name())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return resolutions, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (ps *ProcessingService) ParseResolutionDirectory(resolution string, ch chan<- SegmentInfo, wg *sync.WaitGroup) {
|
||||||
|
defer wg.Done()
|
||||||
|
|
||||||
|
resolutionPath := utils.SafeJoin(ps.config.GetNASEventPath(ps.eventName), resolution)
|
||||||
|
files, err := os.ReadDir(resolutionPath)
|
||||||
|
if err != nil {
|
||||||
|
log.Printf("Failed to read resolution directory %s: %v", resolutionPath, err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, file := range files {
|
||||||
|
if !file.IsDir() {
|
||||||
|
if !strings.HasSuffix(strings.ToLower(file.Name()), ".ts") {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
no, err := strconv.Atoi(file.Name()[6:10])
|
||||||
|
if err != nil {
|
||||||
|
log.Printf("Failed to parse segment number: %v", err)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
ch <- SegmentInfo{
|
||||||
|
Name: file.Name(),
|
||||||
|
SeqNo: no,
|
||||||
|
Resolution: resolution,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (ps *ProcessingService) AggregateSegmentInfo(ch <-chan SegmentInfo) (map[int]SegmentInfo, error) {
|
||||||
|
segmentMap := make(map[int]SegmentInfo)
|
||||||
|
|
||||||
|
rank := map[string]int{
|
||||||
|
"1080p": 1,
|
||||||
|
"720p": 2,
|
||||||
|
"540p": 3,
|
||||||
|
"480p": 4,
|
||||||
|
"450p": 5,
|
||||||
|
"360p": 6,
|
||||||
|
"270p": 7,
|
||||||
|
"240p": 8,
|
||||||
|
}
|
||||||
|
|
||||||
|
for segment := range ch {
|
||||||
|
fmt.Printf("Received segment %s in resolution %s \n", segment.Name, segment.Resolution)
|
||||||
|
logging.MustGetFactory().Processing.Debug("segment received", logging.KeySegment, segment.Name, logging.KeyResolution, segment.Resolution)
|
||||||
|
current, exists := segmentMap[segment.SeqNo]
|
||||||
|
if !exists || rank[segment.Resolution] > rank[current.Resolution] {
|
||||||
|
segmentMap[segment.SeqNo] = segment
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return segmentMap, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (ps *ProcessingService) WriteConcatFile(segmentMap map[int]SegmentInfo) (string, error) {
|
||||||
|
concatPath := ps.config.GetProcessOutputPath(ps.eventName)
|
||||||
|
|
||||||
|
if err := utils.EnsureDir(concatPath); err != nil {
|
||||||
|
return "", fmt.Errorf("failed to create directories for concat path: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
concatFilePath := utils.SafeJoin(concatPath, ps.eventName+".txt")
|
||||||
|
f, err := os.Create(concatFilePath)
|
||||||
|
if err != nil {
|
||||||
|
return "", fmt.Errorf("failed to create concat file: %w", err)
|
||||||
|
}
|
||||||
|
defer f.Close()
|
||||||
|
|
||||||
|
// Sort keys to preserve order
|
||||||
|
keys := make([]int, 0, len(segmentMap))
|
||||||
|
for k := range segmentMap {
|
||||||
|
keys = append(keys, k)
|
||||||
|
}
|
||||||
|
sort.Ints(keys)
|
||||||
|
|
||||||
|
for _, seq := range keys {
|
||||||
|
segment := segmentMap[seq]
|
||||||
|
filePath := utils.SafeJoin(ps.config.GetNASEventPath(ps.eventName), segment.Resolution, segment.Name)
|
||||||
|
line := fmt.Sprintf("file '%s'\n", filePath)
|
||||||
|
if _, err := f.WriteString(line); err != nil {
|
||||||
|
return "", fmt.Errorf("failed to write to concat file: %w", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return concatFilePath, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (ps *ProcessingService) getFFmpegPath() (string, error) {
|
||||||
|
// First try the configured path
|
||||||
|
configuredPath := ps.config.Processing.FFmpegPath
|
||||||
|
if configuredPath != "" {
|
||||||
|
// Check if it's just the command name or a full path
|
||||||
|
if filepath.IsAbs(configuredPath) {
|
||||||
|
return configuredPath, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// Try to find it in PATH
|
||||||
|
if fullPath, err := exec.LookPath(configuredPath); err == nil {
|
||||||
|
return fullPath, nil
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Fallback: try local bin directory
|
||||||
|
var baseDir string
|
||||||
|
exePath, err := os.Executable()
|
||||||
|
if err == nil {
|
||||||
|
baseDir = filepath.Dir(exePath)
|
||||||
|
} else {
|
||||||
|
baseDir, err = os.Getwd()
|
||||||
|
if err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
ffmpeg := utils.SafeJoin(baseDir, "bin", "ffmpeg")
|
||||||
|
if runtime.GOOS == "windows" {
|
||||||
|
ffmpeg += ".exe"
|
||||||
|
}
|
||||||
|
|
||||||
|
if utils.PathExists(ffmpeg) {
|
||||||
|
return ffmpeg, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// Try current working directory
|
||||||
|
cwd, err := os.Getwd()
|
||||||
|
if err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
ffmpeg = utils.SafeJoin(cwd, "bin", "ffmpeg")
|
||||||
|
if runtime.GOOS == "windows" {
|
||||||
|
ffmpeg += ".exe"
|
||||||
|
}
|
||||||
|
|
||||||
|
if utils.PathExists(ffmpeg) {
|
||||||
|
return ffmpeg, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
return "", fmt.Errorf("FFmpeg not found. Please install FFmpeg or set FFMPEG_PATH environment variable")
|
||||||
|
}
|
||||||
|
|
||||||
|
func (ps *ProcessingService) RunFFmpeg(inputPath, outputPath string) error {
|
||||||
|
fmt.Println("Running ffmpeg...")
|
||||||
|
|
||||||
|
fileOutPath := utils.SafeJoin(outputPath, ps.eventName+".mp4")
|
||||||
|
fmt.Println("Input path:", inputPath)
|
||||||
|
fmt.Println("Output path:", fileOutPath)
|
||||||
|
logging.MustGetFactory().Processing.Info("running ffmpeg", "input", inputPath, "output", fileOutPath)
|
||||||
|
|
||||||
|
path, err := ps.getFFmpegPath()
|
||||||
|
if err != nil {
|
||||||
|
logging.MustGetFactory().Processing.Error("failed to find ffmpeg", logging.KeyError, err.Error())
|
||||||
|
return fmt.Errorf("failed to find FFmpeg: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
cmd := exec.Command(path, "-f", "concat", "-safe", "0", "-i", inputPath, "-c", "copy", fileOutPath)
|
||||||
|
cmd.Stdout = os.Stdout
|
||||||
|
cmd.Stderr = os.Stderr
|
||||||
|
|
||||||
|
if err := cmd.Run(); err != nil {
|
||||||
|
logging.MustGetFactory().Processing.Error("ffmpeg run failed", logging.KeyError, err.Error())
|
||||||
|
return fmt.Errorf("failed to run ffmpeg: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
fmt.Println("FFmpeg completed successfully")
|
||||||
|
logging.MustGetFactory().Processing.Info("ffmpeg completed successfully", "output", fileOutPath)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
384
pkg/processing/service_test.go
Normal file
384
pkg/processing/service_test.go
Normal file
@ -0,0 +1,384 @@
|
|||||||
|
package processing
|
||||||
|
|
||||||
|
import (
|
||||||
|
"m3u8-downloader/pkg/config"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"runtime"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
func createTestConfig(tempDir string) *config.Config {
|
||||||
|
return &config.Config{
|
||||||
|
Core: config.CoreConfig{
|
||||||
|
WorkerCount: 2,
|
||||||
|
RefreshDelay: 1 * time.Second,
|
||||||
|
},
|
||||||
|
NAS: config.NASConfig{
|
||||||
|
OutputPath: filepath.Join(tempDir, "nas"),
|
||||||
|
Username: "testuser",
|
||||||
|
Password: "testpass",
|
||||||
|
Timeout: 10 * time.Second,
|
||||||
|
RetryLimit: 2,
|
||||||
|
EnableTransfer: false, // Disable to avoid NAS connection
|
||||||
|
},
|
||||||
|
Processing: config.ProcessingConfig{
|
||||||
|
Enabled: true,
|
||||||
|
AutoProcess: true,
|
||||||
|
WorkerCount: 1,
|
||||||
|
FFmpegPath: "echo", // Use echo command for testing
|
||||||
|
},
|
||||||
|
Paths: config.PathsConfig{
|
||||||
|
LocalOutput: filepath.Join(tempDir, "data"),
|
||||||
|
ProcessOutput: filepath.Join(tempDir, "out"),
|
||||||
|
ManifestDir: filepath.Join(tempDir, "data"),
|
||||||
|
PersistenceFile: filepath.Join(tempDir, "queue.json"),
|
||||||
|
},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestNewProcessingService_Success(t *testing.T) {
|
||||||
|
tempDir, err := os.MkdirTemp("", "processing_test_*")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Failed to create temp dir: %v", err)
|
||||||
|
}
|
||||||
|
defer os.RemoveAll(tempDir)
|
||||||
|
|
||||||
|
cfg := createTestConfig(tempDir)
|
||||||
|
cfg.NAS.EnableTransfer = false // Disable NAS to avoid connection
|
||||||
|
|
||||||
|
// We can't test actual NAS connection, so we'll skip the constructor test
|
||||||
|
// that requires NAS connectivity. Instead, test the configuration handling.
|
||||||
|
|
||||||
|
if cfg.Processing.FFmpegPath != "echo" {
|
||||||
|
t.Errorf("Expected FFmpegPath='echo', got '%s'", cfg.Processing.FFmpegPath)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestNewProcessingService_NilConfig(t *testing.T) {
|
||||||
|
_, err := NewProcessingService("test-event", nil)
|
||||||
|
if err == nil {
|
||||||
|
t.Error("Expected error for nil config")
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), "configuration is required") {
|
||||||
|
t.Errorf("Expected 'configuration is required' error, got: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestProcessingService_GetEventDirs(t *testing.T) {
|
||||||
|
tempDir, err := os.MkdirTemp("", "processing_test_*")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Failed to create temp dir: %v", err)
|
||||||
|
}
|
||||||
|
defer os.RemoveAll(tempDir)
|
||||||
|
|
||||||
|
cfg := createTestConfig(tempDir)
|
||||||
|
|
||||||
|
// Create mock NAS directory structure
|
||||||
|
nasDir := cfg.NAS.OutputPath
|
||||||
|
os.MkdirAll(filepath.Join(nasDir, "event1"), 0755)
|
||||||
|
os.MkdirAll(filepath.Join(nasDir, "event2"), 0755)
|
||||||
|
os.MkdirAll(filepath.Join(nasDir, "event3"), 0755)
|
||||||
|
// Create a file (should be ignored)
|
||||||
|
os.WriteFile(filepath.Join(nasDir, "not_a_dir.txt"), []byte("test"), 0644)
|
||||||
|
|
||||||
|
ps := &ProcessingService{
|
||||||
|
config: cfg,
|
||||||
|
eventName: "", // Empty to test directory discovery
|
||||||
|
}
|
||||||
|
|
||||||
|
dirs, err := ps.GetEventDirs()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("GetEventDirs() failed: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if len(dirs) != 3 {
|
||||||
|
t.Errorf("Expected 3 event directories, got %d", len(dirs))
|
||||||
|
}
|
||||||
|
|
||||||
|
expectedDirs := []string{"event1", "event2", "event3"}
|
||||||
|
for _, expected := range expectedDirs {
|
||||||
|
found := false
|
||||||
|
for _, actual := range dirs {
|
||||||
|
if actual == expected {
|
||||||
|
found = true
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if !found {
|
||||||
|
t.Errorf("Expected to find directory '%s' in results: %v", expected, dirs)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestProcessingService_GetEventDirs_WithEventName(t *testing.T) {
|
||||||
|
cfg := createTestConfig("/tmp")
|
||||||
|
eventName := "specific-event"
|
||||||
|
|
||||||
|
ps := &ProcessingService{
|
||||||
|
config: cfg,
|
||||||
|
eventName: eventName,
|
||||||
|
}
|
||||||
|
|
||||||
|
dirs, err := ps.GetEventDirs()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("GetEventDirs() failed: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if len(dirs) != 1 {
|
||||||
|
t.Errorf("Expected 1 directory, got %d", len(dirs))
|
||||||
|
}
|
||||||
|
if dirs[0] != eventName {
|
||||||
|
t.Errorf("Expected directory '%s', got '%s'", eventName, dirs[0])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestProcessingService_GetResolutions(t *testing.T) {
|
||||||
|
tempDir, err := os.MkdirTemp("", "processing_test_*")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Failed to create temp dir: %v", err)
|
||||||
|
}
|
||||||
|
defer os.RemoveAll(tempDir)
|
||||||
|
|
||||||
|
cfg := createTestConfig(tempDir)
|
||||||
|
eventName := "test-event"
|
||||||
|
|
||||||
|
// Create mock event directory with quality subdirectories
|
||||||
|
eventPath := filepath.Join(cfg.NAS.OutputPath, eventName)
|
||||||
|
os.MkdirAll(filepath.Join(eventPath, "1080p"), 0755)
|
||||||
|
os.MkdirAll(filepath.Join(eventPath, "720p"), 0755)
|
||||||
|
os.MkdirAll(filepath.Join(eventPath, "480p"), 0755)
|
||||||
|
os.MkdirAll(filepath.Join(eventPath, "not_resolution"), 0755) // Should be ignored
|
||||||
|
os.WriteFile(filepath.Join(eventPath, "file.txt"), []byte("test"), 0644) // Should be ignored
|
||||||
|
|
||||||
|
ps := &ProcessingService{
|
||||||
|
config: cfg,
|
||||||
|
eventName: eventName,
|
||||||
|
}
|
||||||
|
|
||||||
|
resolutions, err := ps.GetResolutions()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("GetResolutions() failed: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
expectedResolutions := []string{"1080p", "720p", "480p"}
|
||||||
|
if len(resolutions) != len(expectedResolutions) {
|
||||||
|
t.Errorf("Expected %d resolutions, got %d: %v", len(expectedResolutions), len(resolutions), resolutions)
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, expected := range expectedResolutions {
|
||||||
|
found := false
|
||||||
|
for _, actual := range resolutions {
|
||||||
|
if actual == expected {
|
||||||
|
found = true
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if !found {
|
||||||
|
t.Errorf("Expected to find resolution '%s' in results: %v", expected, resolutions)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestProcessingService_AggregateSegmentInfo(t *testing.T) {
|
||||||
|
ps := &ProcessingService{}
|
||||||
|
|
||||||
|
// Create test channel with segments
|
||||||
|
ch := make(chan SegmentInfo, 5)
|
||||||
|
|
||||||
|
// Add segments with different qualities for same sequence
|
||||||
|
ch <- SegmentInfo{Name: "seg_1001.ts", SeqNo: 1001, Resolution: "720p"}
|
||||||
|
ch <- SegmentInfo{Name: "seg_1001.ts", SeqNo: 1001, Resolution: "1080p"} // Higher quality, should win
|
||||||
|
ch <- SegmentInfo{Name: "seg_1002.ts", SeqNo: 1002, Resolution: "480p"}
|
||||||
|
ch <- SegmentInfo{Name: "seg_1003.ts", SeqNo: 1003, Resolution: "1080p"}
|
||||||
|
ch <- SegmentInfo{Name: "seg_1001.ts", SeqNo: 1001, Resolution: "540p"} // Lower than 1080p, should not replace
|
||||||
|
|
||||||
|
close(ch)
|
||||||
|
|
||||||
|
segmentMap, err := ps.AggregateSegmentInfo(ch)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("AggregateSegmentInfo() failed: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Should have 3 unique sequence numbers
|
||||||
|
if len(segmentMap) != 3 {
|
||||||
|
t.Errorf("Expected 3 unique segments, got %d", len(segmentMap))
|
||||||
|
}
|
||||||
|
|
||||||
|
// Check sequence 1001 has the highest quality (1080p)
|
||||||
|
seg1001, exists := segmentMap[1001]
|
||||||
|
if !exists {
|
||||||
|
t.Fatal("Segment 1001 should exist")
|
||||||
|
}
|
||||||
|
if seg1001.Resolution != "1080p" {
|
||||||
|
t.Errorf("Expected segment 1001 to have resolution '1080p', got '%s'", seg1001.Resolution)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Check sequence 1002 has 480p
|
||||||
|
seg1002, exists := segmentMap[1002]
|
||||||
|
if !exists {
|
||||||
|
t.Fatal("Segment 1002 should exist")
|
||||||
|
}
|
||||||
|
if seg1002.Resolution != "480p" {
|
||||||
|
t.Errorf("Expected segment 1002 to have resolution '480p', got '%s'", seg1002.Resolution)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Check sequence 1003 has 1080p
|
||||||
|
seg1003, exists := segmentMap[1003]
|
||||||
|
if !exists {
|
||||||
|
t.Fatal("Segment 1003 should exist")
|
||||||
|
}
|
||||||
|
if seg1003.Resolution != "1080p" {
|
||||||
|
t.Errorf("Expected segment 1003 to have resolution '1080p', got '%s'", seg1003.Resolution)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestProcessingService_WriteConcatFile(t *testing.T) {
|
||||||
|
tempDir, err := os.MkdirTemp("", "processing_test_*")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Failed to create temp dir: %v", err)
|
||||||
|
}
|
||||||
|
defer os.RemoveAll(tempDir)
|
||||||
|
|
||||||
|
cfg := createTestConfig(tempDir)
|
||||||
|
eventName := "test-event"
|
||||||
|
|
||||||
|
ps := &ProcessingService{
|
||||||
|
config: cfg,
|
||||||
|
eventName: eventName,
|
||||||
|
}
|
||||||
|
|
||||||
|
// Create test segment map
|
||||||
|
segmentMap := map[int]SegmentInfo{
|
||||||
|
1003: {Name: "seg_1003.ts", SeqNo: 1003, Resolution: "1080p"},
|
||||||
|
1001: {Name: "seg_1001.ts", SeqNo: 1001, Resolution: "720p"},
|
||||||
|
1002: {Name: "seg_1002.ts", SeqNo: 1002, Resolution: "1080p"},
|
||||||
|
}
|
||||||
|
|
||||||
|
concatFilePath, err := ps.WriteConcatFile(segmentMap)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("WriteConcatFile() failed: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Verify file was created
|
||||||
|
if _, err := os.Stat(concatFilePath); os.IsNotExist(err) {
|
||||||
|
t.Fatalf("Concat file was not created: %s", concatFilePath)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Read and verify content
|
||||||
|
content, err := os.ReadFile(concatFilePath)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Failed to read concat file: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
contentStr := string(content)
|
||||||
|
lines := strings.Split(strings.TrimSpace(contentStr), "\n")
|
||||||
|
|
||||||
|
if len(lines) != 3 {
|
||||||
|
t.Errorf("Expected 3 lines in concat file, got %d", len(lines))
|
||||||
|
}
|
||||||
|
|
||||||
|
// Verify segments are sorted by sequence number
|
||||||
|
expectedOrder := []string{"seg_1001.ts", "seg_1002.ts", "seg_1003.ts"}
|
||||||
|
for i, line := range lines {
|
||||||
|
if !strings.Contains(line, expectedOrder[i]) {
|
||||||
|
t.Errorf("Line %d should contain '%s', got: %s", i, expectedOrder[i], line)
|
||||||
|
}
|
||||||
|
if !strings.HasPrefix(line, "file '") {
|
||||||
|
t.Errorf("Line %d should start with 'file ', got: %s", i, line)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestProcessingService_getFFmpegPath(t *testing.T) {
|
||||||
|
cfg := createTestConfig("/tmp")
|
||||||
|
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
ffmpegPath string
|
||||||
|
shouldFind bool
|
||||||
|
expectedError string
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
name: "echo command (should be found in PATH)",
|
||||||
|
ffmpegPath: "echo",
|
||||||
|
shouldFind: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "absolute path test",
|
||||||
|
ffmpegPath: func() string {
|
||||||
|
if runtime.GOOS == "windows" {
|
||||||
|
return "C:\\Windows\\System32\\cmd.exe"
|
||||||
|
}
|
||||||
|
return "/bin/echo"
|
||||||
|
}(),
|
||||||
|
shouldFind: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "nonexistent command",
|
||||||
|
ffmpegPath: "nonexistent_ffmpeg_command_12345",
|
||||||
|
shouldFind: false,
|
||||||
|
expectedError: "FFmpeg not found",
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
testCfg := *cfg
|
||||||
|
testCfg.Processing.FFmpegPath = tt.ffmpegPath
|
||||||
|
|
||||||
|
ps := &ProcessingService{
|
||||||
|
config: &testCfg,
|
||||||
|
eventName: "test",
|
||||||
|
}
|
||||||
|
|
||||||
|
path, err := ps.getFFmpegPath()
|
||||||
|
|
||||||
|
if tt.shouldFind {
|
||||||
|
if err != nil {
|
||||||
|
t.Errorf("Expected to find FFmpeg, but got error: %v", err)
|
||||||
|
}
|
||||||
|
if path == "" {
|
||||||
|
t.Error("Expected non-empty path")
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
if err == nil {
|
||||||
|
t.Error("Expected error for nonexistent FFmpeg")
|
||||||
|
}
|
||||||
|
if tt.expectedError != "" && !strings.Contains(err.Error(), tt.expectedError) {
|
||||||
|
t.Errorf("Expected error containing '%s', got: %v", tt.expectedError, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestSegmentInfo_Structure(t *testing.T) {
|
||||||
|
segment := SegmentInfo{
|
||||||
|
Name: "test_segment.ts",
|
||||||
|
SeqNo: 1001,
|
||||||
|
Resolution: "1080p",
|
||||||
|
}
|
||||||
|
|
||||||
|
if segment.Name != "test_segment.ts" {
|
||||||
|
t.Errorf("Expected Name='test_segment.ts', got '%s'", segment.Name)
|
||||||
|
}
|
||||||
|
if segment.SeqNo != 1001 {
|
||||||
|
t.Errorf("Expected SeqNo=1001, got %d", segment.SeqNo)
|
||||||
|
}
|
||||||
|
if segment.Resolution != "1080p" {
|
||||||
|
t.Errorf("Expected Resolution='1080p', got '%s'", segment.Resolution)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestProcessJob_Structure(t *testing.T) {
|
||||||
|
job := ProcessJob{
|
||||||
|
EventName: "test-event",
|
||||||
|
}
|
||||||
|
|
||||||
|
if job.EventName != "test-event" {
|
||||||
|
t.Errorf("Expected EventName='test-event', got '%s'", job.EventName)
|
||||||
|
}
|
||||||
|
}
|
||||||
5
pkg/processing/types.go
Normal file
5
pkg/processing/types.go
Normal file
@ -0,0 +1,5 @@
|
|||||||
|
package processing
|
||||||
|
|
||||||
|
type ProcessJob struct {
|
||||||
|
EventName string
|
||||||
|
}
|
||||||
@ -4,6 +4,7 @@ import (
|
|||||||
"context"
|
"context"
|
||||||
"fmt"
|
"fmt"
|
||||||
"log"
|
"log"
|
||||||
|
"m3u8-downloader/pkg/logging"
|
||||||
"os"
|
"os"
|
||||||
"sync"
|
"sync"
|
||||||
"time"
|
"time"
|
||||||
@ -41,6 +42,7 @@ func (cs *CleanupService) Start(ctx context.Context) error {
|
|||||||
}
|
}
|
||||||
|
|
||||||
log.Printf("Cleanup service started (retention: %v, batch: %d)", cs.config.RetentionPeriod, cs.config.BatchSize)
|
log.Printf("Cleanup service started (retention: %v, batch: %d)", cs.config.RetentionPeriod, cs.config.BatchSize)
|
||||||
|
logging.MustGetFactory().Transfer.Info("cleanup service started", "retention", cs.config.RetentionPeriod.String(), "batch_size", cs.config.BatchSize)
|
||||||
|
|
||||||
ticker := time.NewTicker(cs.config.CheckInterval)
|
ticker := time.NewTicker(cs.config.CheckInterval)
|
||||||
defer ticker.Stop()
|
defer ticker.Stop()
|
||||||
@ -69,7 +71,6 @@ func (cs *CleanupService) ExecuteCleanup(ctx context.Context) error {
|
|||||||
if batchSize > len(cs.pendingFiles) {
|
if batchSize > len(cs.pendingFiles) {
|
||||||
batchSize = len(cs.pendingFiles)
|
batchSize = len(cs.pendingFiles)
|
||||||
}
|
}
|
||||||
cs.mu.Unlock()
|
|
||||||
|
|
||||||
log.Printf("Executing cleanup batch (size: %d)", batchSize)
|
log.Printf("Executing cleanup batch (size: %d)", batchSize)
|
||||||
|
|
||||||
@ -98,6 +99,7 @@ func (cs *CleanupService) ExecuteCleanup(ctx context.Context) error {
|
|||||||
}
|
}
|
||||||
|
|
||||||
log.Printf("Cleanup batch completed (cleaned: %d, errors: %d)", cleanedCount, len(errors))
|
log.Printf("Cleanup batch completed (cleaned: %d, errors: %d)", cleanedCount, len(errors))
|
||||||
|
logging.MustGetFactory().Transfer.Info("cleanup batch completed", "cleaned", cleanedCount, "errors", len(errors))
|
||||||
|
|
||||||
if len(errors) > 0 {
|
if len(errors) > 0 {
|
||||||
for i, err := range errors {
|
for i, err := range errors {
|
||||||
@ -106,6 +108,7 @@ func (cs *CleanupService) ExecuteCleanup(ctx context.Context) error {
|
|||||||
break
|
break
|
||||||
}
|
}
|
||||||
log.Printf("Error: %v", err)
|
log.Printf("Error: %v", err)
|
||||||
|
logging.MustGetFactory().Transfer.Warn("cleanup error", logging.KeyError, err.Error())
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@ -3,39 +3,29 @@ package transfer
|
|||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
"fmt"
|
"fmt"
|
||||||
"io"
|
|
||||||
"log"
|
"log"
|
||||||
|
"m3u8-downloader/pkg/logging"
|
||||||
|
"m3u8-downloader/pkg/nas"
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
)
|
)
|
||||||
|
|
||||||
type NASTransfer struct {
|
func TransferFile(nt *nas.NASService, ctx context.Context, item *TransferItem) error {
|
||||||
config NASConfig
|
destPath := filepath.Join(nt.Config.Path, item.DestinationPath)
|
||||||
connected bool
|
|
||||||
}
|
|
||||||
|
|
||||||
func NewNASTransfer(config NASConfig) *NASTransfer {
|
|
||||||
return &NASTransfer{
|
|
||||||
config: config,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func (nt *NASTransfer) TransferFile(ctx context.Context, item *TransferItem) error {
|
|
||||||
destPath := filepath.Join(nt.config.Path, item.DestinationPath)
|
|
||||||
|
|
||||||
destDir := filepath.Dir(destPath)
|
destDir := filepath.Dir(destPath)
|
||||||
if err := nt.ensureDirectoryExists(destDir); err != nil {
|
if err := nt.EnsureDirectoryExists(destDir); err != nil {
|
||||||
return fmt.Errorf("Failed to create directory %s: %w", destDir, err)
|
return fmt.Errorf("Failed to create directory %s: %w", destDir, err)
|
||||||
}
|
}
|
||||||
|
|
||||||
transferCtx, cancel := context.WithTimeout(ctx, nt.config.Timeout)
|
transferCtx, cancel := context.WithTimeout(ctx, nt.Config.Timeout)
|
||||||
defer cancel()
|
defer cancel()
|
||||||
|
|
||||||
if err := nt.copyFile(transferCtx, item.SourcePath, destPath); err != nil {
|
if err := nt.CopyFile(transferCtx, item.SourcePath, destPath); err != nil {
|
||||||
return fmt.Errorf("Failed to copy file %s to %s: %w", item.SourcePath, destPath, err)
|
return fmt.Errorf("Failed to copy file %s to %s: %w", item.SourcePath, destPath, err)
|
||||||
}
|
}
|
||||||
|
|
||||||
if nt.config.VerifySize {
|
if nt.Config.VerifySize {
|
||||||
if err := nt.VerifyTransfer(item.SourcePath, destPath); err != nil {
|
if err := nt.VerifyTransfer(item.SourcePath, destPath); err != nil {
|
||||||
os.Remove(destPath)
|
os.Remove(destPath)
|
||||||
return fmt.Errorf("Failed to verify transfer: %w", err)
|
return fmt.Errorf("Failed to verify transfer: %w", err)
|
||||||
@ -43,82 +33,7 @@ func (nt *NASTransfer) TransferFile(ctx context.Context, item *TransferItem) err
|
|||||||
}
|
}
|
||||||
|
|
||||||
log.Printf("File transfer completed: %s -> %s", item.SourcePath, destPath)
|
log.Printf("File transfer completed: %s -> %s", item.SourcePath, destPath)
|
||||||
|
logging.MustGetFactory().Transfer.Debug("nas copy completed", logging.KeyFile, item.SourcePath, "dest", destPath)
|
||||||
|
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func (nt *NASTransfer) copyFile(ctx context.Context, srcPath, destPath string) error {
|
|
||||||
src, err := os.Open(srcPath)
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf("Failed to open source file: %w", err)
|
|
||||||
}
|
|
||||||
defer src.Close()
|
|
||||||
|
|
||||||
dest, err := os.Create(destPath)
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf("Failed to create destination file: %w", err)
|
|
||||||
}
|
|
||||||
defer dest.Close()
|
|
||||||
|
|
||||||
done := make(chan error, 1)
|
|
||||||
go func() {
|
|
||||||
_, err := io.Copy(dest, src)
|
|
||||||
done <- err
|
|
||||||
}()
|
|
||||||
|
|
||||||
select {
|
|
||||||
case <-ctx.Done():
|
|
||||||
return ctx.Err()
|
|
||||||
case err := <-done:
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
return dest.Sync()
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func (nt *NASTransfer) VerifyTransfer(srcPath, destPath string) error {
|
|
||||||
srcInfo, err := os.Stat(srcPath)
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf("Failed to stat source file: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
destInfo, err := os.Stat(destPath)
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf("Failed to stat destination file: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if srcInfo.Size() != destInfo.Size() {
|
|
||||||
return fmt.Errorf("size mismatch: source=%d, dest=%d", srcInfo.Size(), destInfo.Size())
|
|
||||||
}
|
|
||||||
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func (nt *NASTransfer) ensureDirectoryExists(path string) error {
|
|
||||||
if err := os.MkdirAll(path, 0755); err != nil {
|
|
||||||
return fmt.Errorf("Failed to create directory: %w", err)
|
|
||||||
}
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func (nt *NASTransfer) TestConnection() error {
|
|
||||||
testFile := filepath.Join(nt.config.Path, ".connection_test")
|
|
||||||
|
|
||||||
f, err := os.Create(testFile)
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf("Failed to create test file: %w", err)
|
|
||||||
}
|
|
||||||
f.Close()
|
|
||||||
|
|
||||||
os.Remove(testFile)
|
|
||||||
|
|
||||||
nt.connected = true
|
|
||||||
log.Printf("Connected to NAS at %s", nt.config.Path)
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func (nt *NASTransfer) IsConnected() bool {
|
|
||||||
return nt.connected
|
|
||||||
}
|
|
||||||
|
|||||||
@ -6,6 +6,9 @@ import (
|
|||||||
"encoding/json"
|
"encoding/json"
|
||||||
"fmt"
|
"fmt"
|
||||||
"log"
|
"log"
|
||||||
|
"m3u8-downloader/pkg/health"
|
||||||
|
"m3u8-downloader/pkg/logging"
|
||||||
|
"m3u8-downloader/pkg/nas"
|
||||||
"os"
|
"os"
|
||||||
"sync"
|
"sync"
|
||||||
"time"
|
"time"
|
||||||
@ -15,7 +18,7 @@ type TransferQueue struct {
|
|||||||
config QueueConfig
|
config QueueConfig
|
||||||
items *PriorityQueue
|
items *PriorityQueue
|
||||||
stats *QueueStats
|
stats *QueueStats
|
||||||
nasTransfer *NASTransfer
|
nasService *nas.NASService
|
||||||
cleanup *CleanupService
|
cleanup *CleanupService
|
||||||
workers []chan TransferItem
|
workers []chan TransferItem
|
||||||
mu sync.RWMutex
|
mu sync.RWMutex
|
||||||
@ -48,7 +51,7 @@ func (pq *PriorityQueue) Pop() interface{} {
|
|||||||
return item
|
return item
|
||||||
}
|
}
|
||||||
|
|
||||||
func NewTransferQueue(config QueueConfig, nasTransfer *NASTransfer, cleanup *CleanupService) *TransferQueue {
|
func NewTransferQueue(config QueueConfig, nasTransfer *nas.NASService, cleanup *CleanupService) *TransferQueue {
|
||||||
pq := &PriorityQueue{}
|
pq := &PriorityQueue{}
|
||||||
heap.Init(pq)
|
heap.Init(pq)
|
||||||
|
|
||||||
@ -56,7 +59,7 @@ func NewTransferQueue(config QueueConfig, nasTransfer *NASTransfer, cleanup *Cle
|
|||||||
config: config,
|
config: config,
|
||||||
items: pq,
|
items: pq,
|
||||||
stats: &QueueStats{},
|
stats: &QueueStats{},
|
||||||
nasTransfer: nasTransfer,
|
nasService: nasTransfer,
|
||||||
cleanup: cleanup,
|
cleanup: cleanup,
|
||||||
workers: make([]chan TransferItem, config.WorkerCount),
|
workers: make([]chan TransferItem, config.WorkerCount),
|
||||||
}
|
}
|
||||||
@ -73,7 +76,10 @@ func (tq *TransferQueue) Add(item TransferItem) error {
|
|||||||
defer tq.mu.Unlock()
|
defer tq.mu.Unlock()
|
||||||
|
|
||||||
if tq.items.Len() >= tq.config.MaxQueueSize {
|
if tq.items.Len() >= tq.config.MaxQueueSize {
|
||||||
return fmt.Errorf("Queue is full (max size: %d)", tq.config.MaxQueueSize)
|
err := fmt.Errorf("Queue is full (max size: %d)", tq.config.MaxQueueSize)
|
||||||
|
logging.MustGetFactory().Transfer.Error("queue full, dropping file", logging.KeyFile, item.SourcePath, logging.KeyError, err.Error())
|
||||||
|
health.MustGetRegistry().RecordTransferError(item.SourcePath, err)
|
||||||
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
heap.Push(tq.items, &item)
|
heap.Push(tq.items, &item)
|
||||||
@ -145,6 +151,24 @@ func (tq *TransferQueue) dispatchWork() {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func (tq *TransferQueue) processItem(ctx context.Context, item TransferItem) {
|
func (tq *TransferQueue) processItem(ctx context.Context, item TransferItem) {
|
||||||
|
// Check if file already exists on NAS before attempting transfer
|
||||||
|
if exists, err := tq.nasService.FileExists(item.DestinationPath, item.FileSize); err != nil {
|
||||||
|
log.Printf("Failed to check if file exists on NAS for %s: %v", item.SourcePath, err)
|
||||||
|
// Continue with transfer attempt on error
|
||||||
|
} else if exists {
|
||||||
|
log.Printf("File already exists on NAS, skipping transfer: %s", item.SourcePath)
|
||||||
|
item.Status = StatusCompleted
|
||||||
|
tq.stats.IncrementCompleted(item.FileSize)
|
||||||
|
|
||||||
|
// Schedule for cleanup
|
||||||
|
if tq.cleanup != nil {
|
||||||
|
if err := tq.cleanup.ScheduleCleanup(item.SourcePath); err != nil {
|
||||||
|
log.Printf("Failed to schedule cleanup for existing file %s: %v", item.SourcePath, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
maxRetries := 3
|
maxRetries := 3
|
||||||
|
|
||||||
for attempt := 1; attempt <= maxRetries; attempt++ {
|
for attempt := 1; attempt <= maxRetries; attempt++ {
|
||||||
@ -160,7 +184,7 @@ func (tq *TransferQueue) processItem(ctx context.Context, item TransferItem) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
err := tq.nasTransfer.TransferFile(ctx, &item)
|
err := TransferFile(tq.nasService, ctx, &item)
|
||||||
if err == nil {
|
if err == nil {
|
||||||
item.Status = StatusCompleted
|
item.Status = StatusCompleted
|
||||||
tq.stats.IncrementCompleted(item.FileSize)
|
tq.stats.IncrementCompleted(item.FileSize)
|
||||||
@ -168,9 +192,11 @@ func (tq *TransferQueue) processItem(ctx context.Context, item TransferItem) {
|
|||||||
if tq.cleanup != nil {
|
if tq.cleanup != nil {
|
||||||
if err := tq.cleanup.ScheduleCleanup(item.SourcePath); err != nil {
|
if err := tq.cleanup.ScheduleCleanup(item.SourcePath); err != nil {
|
||||||
log.Printf("Failed to add file to cleanup list: %v", err)
|
log.Printf("Failed to add file to cleanup list: %v", err)
|
||||||
|
logging.MustGetFactory().Transfer.Warn("failed to schedule cleanup", logging.KeyFile, item.SourcePath, logging.KeyError, err.Error())
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
log.Printf("File transfer completed: %s", item.SourcePath)
|
log.Printf("File transfer completed: %s", item.SourcePath)
|
||||||
|
logging.MustGetFactory().Transfer.Info("file transfer completed", logging.KeyFile, item.SourcePath, logging.KeyBytes, item.FileSize)
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
@ -178,11 +204,14 @@ func (tq *TransferQueue) processItem(ctx context.Context, item TransferItem) {
|
|||||||
item.RetryCount++
|
item.RetryCount++
|
||||||
|
|
||||||
log.Printf("File transfer failed: %s (attempt %d/%d): %v", item.SourcePath, item.RetryCount, maxRetries, err)
|
log.Printf("File transfer failed: %s (attempt %d/%d): %v", item.SourcePath, item.RetryCount, maxRetries, err)
|
||||||
|
logging.MustGetFactory().Transfer.Warn("file transfer attempt failed", logging.KeyFile, item.SourcePath, logging.KeyAttempt, item.RetryCount, logging.KeyError, err.Error())
|
||||||
|
|
||||||
if attempt == maxRetries {
|
if attempt == maxRetries {
|
||||||
item.Status = StatusFailed
|
item.Status = StatusFailed
|
||||||
tq.stats.IncrementFailed()
|
tq.stats.IncrementFailed()
|
||||||
log.Printf("Transfer permanently failed for file: %s", item.SourcePath)
|
log.Printf("Transfer permanently failed for file: %s", item.SourcePath)
|
||||||
|
logging.MustGetFactory().Transfer.Error("transfer permanently failed", logging.KeyFile, item.SourcePath, logging.KeyError, err.Error())
|
||||||
|
health.MustGetRegistry().RecordTransferError(item.SourcePath, err)
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@ -206,10 +235,12 @@ func (tq *TransferQueue) SaveState() error {
|
|||||||
"timestamp": time.Now(),
|
"timestamp": time.Now(),
|
||||||
}, "", " ")
|
}, "", " ")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
logging.MustGetFactory().Transfer.Error("failed to marshal queue state", logging.KeyError, err.Error())
|
||||||
return fmt.Errorf("Failed to marshal queue state: %w", err)
|
return fmt.Errorf("Failed to marshal queue state: %w", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
if err := os.WriteFile(tq.config.PersistencePath, data, 0644); err != nil {
|
if err := os.WriteFile(tq.config.PersistencePath, data, 0644); err != nil {
|
||||||
|
logging.MustGetFactory().Transfer.Error("failed to save queue state", logging.KeyError, err.Error())
|
||||||
return fmt.Errorf("Failed to save queue state: %w", err)
|
return fmt.Errorf("Failed to save queue state: %w", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
@ -222,6 +253,7 @@ func (tq *TransferQueue) LoadState() error {
|
|||||||
if os.IsNotExist(err) {
|
if os.IsNotExist(err) {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
logging.MustGetFactory().Transfer.Error("failed to load queue state", logging.KeyError, err.Error())
|
||||||
return fmt.Errorf("Failed to load queue state: %w", err)
|
return fmt.Errorf("Failed to load queue state: %w", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@ -5,6 +5,13 @@ import (
|
|||||||
"fmt"
|
"fmt"
|
||||||
"log"
|
"log"
|
||||||
"m3u8-downloader/pkg/constants"
|
"m3u8-downloader/pkg/constants"
|
||||||
|
"m3u8-downloader/pkg/health"
|
||||||
|
"m3u8-downloader/pkg/logging"
|
||||||
|
nas2 "m3u8-downloader/pkg/nas"
|
||||||
|
"m3u8-downloader/pkg/utils"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
"sync"
|
"sync"
|
||||||
"time"
|
"time"
|
||||||
)
|
)
|
||||||
@ -12,47 +19,60 @@ import (
|
|||||||
type TransferService struct {
|
type TransferService struct {
|
||||||
watcher *FileWatcher
|
watcher *FileWatcher
|
||||||
queue *TransferQueue
|
queue *TransferQueue
|
||||||
nas *NASTransfer
|
nas *nas2.NASService
|
||||||
cleanup *CleanupService
|
cleanup *CleanupService
|
||||||
stats *QueueStats
|
stats *QueueStats
|
||||||
}
|
}
|
||||||
|
|
||||||
func NewTrasferService(outputDir string) (*TransferService, error) {
|
func NewTrasferService(outputDir string, eventName string) (*TransferService, error) {
|
||||||
nasConfig := NASConfig{
|
cfg := constants.MustGetConfig()
|
||||||
Path: constants.NASPath,
|
|
||||||
Username: constants.NASUsername,
|
nasConfig := nas2.NASConfig{
|
||||||
Password: constants.NASPassword,
|
Path: outputDir,
|
||||||
Timeout: constants.TransferTimeout,
|
Username: cfg.NAS.Username,
|
||||||
RetryLimit: constants.TransferRetryLimit,
|
Password: cfg.NAS.Password,
|
||||||
|
Timeout: cfg.NAS.Timeout,
|
||||||
|
RetryLimit: cfg.NAS.RetryLimit,
|
||||||
VerifySize: true,
|
VerifySize: true,
|
||||||
}
|
}
|
||||||
nas := NewNASTransfer(nasConfig)
|
nas := nas2.NewNASService(nasConfig)
|
||||||
|
|
||||||
if err := nas.TestConnection(); err != nil {
|
if err := nas.TestConnection(); err != nil {
|
||||||
return nil, fmt.Errorf("Failed to connect to NAS: %w", err)
|
logging.MustGetFactory().Transfer.Error("failed to connect to NAS", logging.KeyError, err.Error())
|
||||||
|
return nil, fmt.Errorf("failed to connect to NAS: %w", err)
|
||||||
}
|
}
|
||||||
|
logging.MustGetFactory().Transfer.Info("connected to NAS", "path", outputDir)
|
||||||
|
health.MustGetRegistry().RecordNASConnectivity(true)
|
||||||
|
|
||||||
cleanupConfig := CleanupConfig{
|
cleanupConfig := CleanupConfig{
|
||||||
Enabled: constants.CleanupAfterTransfer,
|
Enabled: cfg.Cleanup.AfterTransfer,
|
||||||
RetentionPeriod: constants.RetainLocalHours,
|
RetentionPeriod: time.Duration(cfg.Cleanup.RetainHours) * time.Hour,
|
||||||
BatchSize: constants.CleanupBatchSize,
|
BatchSize: cfg.Cleanup.BatchSize,
|
||||||
CheckInterval: constants.FileSettlingDelay,
|
CheckInterval: cfg.Transfer.FileSettlingDelay,
|
||||||
}
|
}
|
||||||
cleanup := NewCleanupService(cleanupConfig)
|
cleanup := NewCleanupService(cleanupConfig)
|
||||||
|
|
||||||
queueConfig := QueueConfig{
|
queueConfig := QueueConfig{
|
||||||
WorkerCount: constants.TransferWorkerCount,
|
WorkerCount: cfg.Transfer.WorkerCount,
|
||||||
PersistencePath: constants.PersistencePath,
|
PersistencePath: cfg.Paths.PersistenceFile,
|
||||||
MaxQueueSize: constants.TransferQueueSize,
|
MaxQueueSize: cfg.Transfer.QueueSize,
|
||||||
BatchSize: constants.BatchSize,
|
BatchSize: cfg.Transfer.BatchSize,
|
||||||
}
|
}
|
||||||
queue := NewTransferQueue(queueConfig, nas, cleanup)
|
queue := NewTransferQueue(queueConfig, nas, cleanup)
|
||||||
|
|
||||||
watcher, err := NewFileWatcher(outputDir, queue)
|
// Create local output directory if it doesn't exist
|
||||||
if err != nil {
|
localOutputPath := cfg.GetEventPath(eventName)
|
||||||
return nil, fmt.Errorf("Failed to create file watcher: %w", err)
|
if err := utils.EnsureDir(localOutputPath); err != nil {
|
||||||
|
return nil, fmt.Errorf("failed to create local output directory: %w", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
watcher, err := NewFileWatcher(localOutputPath, queue, cfg.Transfer.FileSettlingDelay)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("failed to create file watcher: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
health.MustGetRegistry().AttachTransfer(queue.stats, queue.GetQueueSize, cleanup.GetPendingCount, nas.IsConnected)
|
||||||
|
|
||||||
return &TransferService{
|
return &TransferService{
|
||||||
watcher: watcher,
|
watcher: watcher,
|
||||||
queue: queue,
|
queue: queue,
|
||||||
@ -96,6 +116,7 @@ func (ts *TransferService) Start(ctx context.Context) error {
|
|||||||
}()
|
}()
|
||||||
|
|
||||||
log.Println("Transfer service started")
|
log.Println("Transfer service started")
|
||||||
|
logging.MustGetFactory().Transfer.Info("transfer service started")
|
||||||
wg.Wait()
|
wg.Wait()
|
||||||
|
|
||||||
return nil
|
return nil
|
||||||
@ -115,6 +136,9 @@ func (ts *TransferService) reportStats(ctx context.Context) {
|
|||||||
cleanupPending := ts.cleanup.GetPendingCount()
|
cleanupPending := ts.cleanup.GetPendingCount()
|
||||||
|
|
||||||
log.Printf("Transfer Stats: Added: %d, Completed: %d, Failed: %d, Pending: %d, Bytes: %d, Queue Size: %d, Cleanup Pending: %d", added, completed, failed, pending, bytes, queueSize, cleanupPending)
|
log.Printf("Transfer Stats: Added: %d, Completed: %d, Failed: %d, Pending: %d, Bytes: %d, Queue Size: %d, Cleanup Pending: %d", added, completed, failed, pending, bytes, queueSize, cleanupPending)
|
||||||
|
logging.MustGetFactory().Transfer.Info("transfer stats",
|
||||||
|
"added", added, "completed", completed, "failed", failed, "pending", pending,
|
||||||
|
logging.KeyBytes, bytes, "queue_size", queueSize, "cleanup_pending", cleanupPending)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@ -130,7 +154,119 @@ func (ts *TransferService) Shutdown(ctx context.Context) error {
|
|||||||
return fmt.Errorf("Failed to force cleanup: %w", err)
|
return fmt.Errorf("Failed to force cleanup: %w", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Disconnect from NAS
|
||||||
|
if err := ts.nas.Disconnect(); err != nil {
|
||||||
|
log.Printf("Warning: failed to disconnect from NAS: %v", err)
|
||||||
|
logging.MustGetFactory().Transfer.Warn("failed to disconnect from NAS", logging.KeyError, err.Error())
|
||||||
|
}
|
||||||
|
health.MustGetRegistry().RecordNASConnectivity(false)
|
||||||
|
|
||||||
log.Println("Transfer service shut down")
|
log.Println("Transfer service shut down")
|
||||||
|
logging.MustGetFactory().Transfer.Info("transfer service shut down")
|
||||||
|
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// QueueExistingFiles scans a directory for .ts files and queues them for transfer
|
||||||
|
func (ts *TransferService) QueueExistingFiles(localEventPath string) error {
|
||||||
|
cfg := constants.MustGetConfig()
|
||||||
|
log.Printf("Scanning for existing files in: %s", localEventPath)
|
||||||
|
|
||||||
|
var fileCount, alreadyTransferred, scheduledForCleanup int
|
||||||
|
|
||||||
|
// Extract event name from path for NAS destination
|
||||||
|
eventName := filepath.Base(localEventPath)
|
||||||
|
|
||||||
|
err := filepath.Walk(localEventPath, func(path string, info os.FileInfo, err error) error {
|
||||||
|
if err != nil {
|
||||||
|
log.Printf("Error accessing path %s: %v", path, err)
|
||||||
|
return nil // Continue walking
|
||||||
|
}
|
||||||
|
|
||||||
|
// Only process .ts files
|
||||||
|
if !info.IsDir() && strings.HasSuffix(strings.ToLower(info.Name()), ".ts") {
|
||||||
|
// Extract resolution from directory path
|
||||||
|
resolution := ts.extractResolutionFromPath(path)
|
||||||
|
|
||||||
|
// Get relative path from event directory
|
||||||
|
relPath, err := filepath.Rel(localEventPath, path)
|
||||||
|
if err != nil {
|
||||||
|
log.Printf("Failed to get relative path for %s: %v", path, err)
|
||||||
|
logging.MustGetFactory().Transfer.Warn("failed to get relative path", logging.KeyFile, path, logging.KeyError, err.Error())
|
||||||
|
health.MustGetRegistry().RecordTransferError(path, err)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// Build NAS destination path (eventName/relPath)
|
||||||
|
nasDestPath := filepath.Join(eventName, relPath)
|
||||||
|
|
||||||
|
// Check if file already exists on NAS with matching size
|
||||||
|
exists, err := ts.nas.FileExists(nasDestPath, info.Size())
|
||||||
|
if err != nil {
|
||||||
|
log.Printf("Failed to check NAS file existence for %s: %v", path, err)
|
||||||
|
// Continue with transfer attempt on error
|
||||||
|
} else if exists {
|
||||||
|
log.Printf("File already exists on NAS: %s (%s, %d bytes)", path, resolution, info.Size())
|
||||||
|
alreadyTransferred++
|
||||||
|
|
||||||
|
// Schedule for cleanup if cleanup is enabled
|
||||||
|
if cfg.Cleanup.AfterTransfer {
|
||||||
|
if err := ts.cleanup.ScheduleCleanup(path); err != nil {
|
||||||
|
log.Printf("Failed to schedule cleanup for already-transferred file %s: %v", path, err)
|
||||||
|
} else {
|
||||||
|
scheduledForCleanup++
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil // Skip queuing this file
|
||||||
|
}
|
||||||
|
|
||||||
|
// Create transfer item
|
||||||
|
item := TransferItem{
|
||||||
|
ID: ts.generateTransferID(),
|
||||||
|
SourcePath: path,
|
||||||
|
DestinationPath: nasDestPath,
|
||||||
|
Resolution: resolution,
|
||||||
|
Timestamp: info.ModTime(),
|
||||||
|
Status: StatusPending,
|
||||||
|
FileSize: info.Size(),
|
||||||
|
}
|
||||||
|
|
||||||
|
// Add to queue
|
||||||
|
if err := ts.queue.Add(item); err != nil {
|
||||||
|
log.Printf("Failed to queue file %s: %v", path, err)
|
||||||
|
logging.MustGetFactory().Transfer.Error("failed to queue file", logging.KeyFile, path, logging.KeyError, err.Error())
|
||||||
|
health.MustGetRegistry().RecordTransferError(path, err)
|
||||||
|
} else {
|
||||||
|
log.Printf("Queued file: %s (%s, %d bytes)", path, resolution, info.Size())
|
||||||
|
fileCount++
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return nil
|
||||||
|
})
|
||||||
|
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("failed to walk directory: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
log.Printf("File scan completed - Queued: %d, Already transferred: %d, Scheduled for cleanup: %d",
|
||||||
|
fileCount, alreadyTransferred, scheduledForCleanup)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (ts *TransferService) extractResolutionFromPath(filePath string) string {
|
||||||
|
dir := filepath.Dir(filePath)
|
||||||
|
parts := strings.Split(dir, string(filepath.Separator))
|
||||||
|
|
||||||
|
for _, part := range parts {
|
||||||
|
if strings.HasSuffix(part, "p") {
|
||||||
|
return part
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return "unknown"
|
||||||
|
}
|
||||||
|
|
||||||
|
func (ts *TransferService) generateTransferID() string {
|
||||||
|
return fmt.Sprintf("transfer_existing_%d", time.Now().UnixNano())
|
||||||
|
}
|
||||||
|
|||||||
@ -44,15 +44,6 @@ func (s TransferStatus) String() string {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
type NASConfig struct {
|
|
||||||
Path string
|
|
||||||
Username string
|
|
||||||
Password string
|
|
||||||
Timeout time.Duration
|
|
||||||
RetryLimit int
|
|
||||||
VerifySize bool
|
|
||||||
}
|
|
||||||
|
|
||||||
type QueueConfig struct {
|
type QueueConfig struct {
|
||||||
WorkerCount int
|
WorkerCount int
|
||||||
PersistencePath string
|
PersistencePath string
|
||||||
|
|||||||
@ -4,6 +4,8 @@ import (
|
|||||||
"context"
|
"context"
|
||||||
"fmt"
|
"fmt"
|
||||||
"log"
|
"log"
|
||||||
|
"m3u8-downloader/pkg/health"
|
||||||
|
"m3u8-downloader/pkg/logging"
|
||||||
"math/rand"
|
"math/rand"
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
@ -23,7 +25,7 @@ type FileWatcher struct {
|
|||||||
mu sync.Mutex
|
mu sync.Mutex
|
||||||
}
|
}
|
||||||
|
|
||||||
func NewFileWatcher(outputDir string, queue *TransferQueue) (*FileWatcher, error) {
|
func NewFileWatcher(outputDir string, queue *TransferQueue, settlingDelay time.Duration) (*FileWatcher, error) {
|
||||||
watcher, err := fsnotify.NewWatcher()
|
watcher, err := fsnotify.NewWatcher()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
@ -32,7 +34,7 @@ func NewFileWatcher(outputDir string, queue *TransferQueue) (*FileWatcher, error
|
|||||||
outputDir: outputDir,
|
outputDir: outputDir,
|
||||||
queue: queue,
|
queue: queue,
|
||||||
watcher: watcher,
|
watcher: watcher,
|
||||||
settingDelay: time.Second,
|
settingDelay: settlingDelay,
|
||||||
pendingFiles: make(map[string]*time.Timer),
|
pendingFiles: make(map[string]*time.Timer),
|
||||||
}, nil
|
}, nil
|
||||||
}
|
}
|
||||||
@ -45,11 +47,13 @@ func (fw *FileWatcher) Start(ctx context.Context) error {
|
|||||||
}
|
}
|
||||||
|
|
||||||
log.Printf("Starting file watcher on %s", fw.outputDir)
|
log.Printf("Starting file watcher on %s", fw.outputDir)
|
||||||
|
logging.MustGetFactory().Transfer.Info("file watcher started", "dir", fw.outputDir)
|
||||||
|
|
||||||
for {
|
for {
|
||||||
select {
|
select {
|
||||||
case <-ctx.Done():
|
case <-ctx.Done():
|
||||||
log.Println("File watcher shutting down...")
|
log.Println("File watcher shutting down...")
|
||||||
|
logging.MustGetFactory().Transfer.Info("file watcher shutting down")
|
||||||
return ctx.Err()
|
return ctx.Err()
|
||||||
|
|
||||||
case event, ok := <-fw.watcher.Events:
|
case event, ok := <-fw.watcher.Events:
|
||||||
@ -63,6 +67,7 @@ func (fw *FileWatcher) Start(ctx context.Context) error {
|
|||||||
return fmt.Errorf("Watcher errors channel closed")
|
return fmt.Errorf("Watcher errors channel closed")
|
||||||
}
|
}
|
||||||
log.Printf("Watcher error: %v", err)
|
log.Printf("Watcher error: %v", err)
|
||||||
|
logging.MustGetFactory().Transfer.Error("watcher error", logging.KeyError, err.Error())
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@ -144,6 +149,8 @@ func (fw *FileWatcher) processFile(filePath string) {
|
|||||||
info, err := os.Stat(filePath)
|
info, err := os.Stat(filePath)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
log.Printf("Failed to stat file %s: %v", filePath, err)
|
log.Printf("Failed to stat file %s: %v", filePath, err)
|
||||||
|
logging.MustGetFactory().Transfer.Warn("failed to stat file", logging.KeyFile, filePath, logging.KeyError, err.Error())
|
||||||
|
health.MustGetRegistry().RecordTransferError(filePath, err)
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
@ -152,6 +159,8 @@ func (fw *FileWatcher) processFile(filePath string) {
|
|||||||
relPath, err := filepath.Rel(fw.outputDir, filePath)
|
relPath, err := filepath.Rel(fw.outputDir, filePath)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
log.Printf("Failed to get relative path for file %s: %v", filePath, err)
|
log.Printf("Failed to get relative path for file %s: %v", filePath, err)
|
||||||
|
logging.MustGetFactory().Transfer.Warn("failed to get relative path", logging.KeyFile, filePath, logging.KeyError, err.Error())
|
||||||
|
health.MustGetRegistry().RecordTransferError(filePath, err)
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
@ -167,8 +176,11 @@ func (fw *FileWatcher) processFile(filePath string) {
|
|||||||
|
|
||||||
if err := fw.queue.Add(item); err != nil {
|
if err := fw.queue.Add(item); err != nil {
|
||||||
log.Printf("Failed to add file to queue: %v", err)
|
log.Printf("Failed to add file to queue: %v", err)
|
||||||
|
logging.MustGetFactory().Transfer.Error("failed to add file to queue", logging.KeyFile, filePath, logging.KeyError, err.Error())
|
||||||
|
health.MustGetRegistry().RecordTransferError(filePath, err)
|
||||||
} else {
|
} else {
|
||||||
log.Printf("Added file to queue: %s", filePath)
|
log.Printf("Added file to queue: %s", filePath)
|
||||||
|
logging.MustGetFactory().Transfer.Debug("file queued", logging.KeyFile, filePath)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
62
pkg/utils/paths.go
Normal file
62
pkg/utils/paths.go
Normal file
@ -0,0 +1,62 @@
|
|||||||
|
package utils
|
||||||
|
|
||||||
|
import (
|
||||||
|
"fmt"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
)
|
||||||
|
|
||||||
|
func SafeJoin(base string, elements ...string) string {
|
||||||
|
path := filepath.Join(append([]string{base}, elements...)...)
|
||||||
|
return filepath.Clean(path)
|
||||||
|
}
|
||||||
|
|
||||||
|
func EnsureDir(path string) error {
|
||||||
|
if err := os.MkdirAll(path, 0755); err != nil {
|
||||||
|
return fmt.Errorf("failed to create directory %s: %w", path, err)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func PathExists(path string) bool {
|
||||||
|
_, err := os.Stat(path)
|
||||||
|
return !os.IsNotExist(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
func IsValidPath(path string) bool {
|
||||||
|
if path == "" {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
return !strings.ContainsAny(path, "<>:\"|?*")
|
||||||
|
}
|
||||||
|
|
||||||
|
func NormalizePath(path string) string {
|
||||||
|
return filepath.Clean(strings.ReplaceAll(path, "\\", string(filepath.Separator)))
|
||||||
|
}
|
||||||
|
|
||||||
|
func GetRelativePath(basePath, targetPath string) (string, error) {
|
||||||
|
rel, err := filepath.Rel(basePath, targetPath)
|
||||||
|
if err != nil {
|
||||||
|
return "", fmt.Errorf("failed to get relative path: %w", err)
|
||||||
|
}
|
||||||
|
return rel, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func ValidateWritablePath(path string) error {
|
||||||
|
dir := filepath.Dir(path)
|
||||||
|
if err := EnsureDir(dir); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
testFile := filepath.Join(dir, ".write_test")
|
||||||
|
file, err := os.Create(testFile)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("path %s is not writable: %w", dir, err)
|
||||||
|
}
|
||||||
|
file.Close()
|
||||||
|
os.Remove(testFile)
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
235
pkg/utils/paths_test.go
Normal file
235
pkg/utils/paths_test.go
Normal file
@ -0,0 +1,235 @@
|
|||||||
|
package utils
|
||||||
|
|
||||||
|
import (
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"runtime"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestSafeJoin(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
base string
|
||||||
|
elements []string
|
||||||
|
want string
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
name: "basic join",
|
||||||
|
base: "data",
|
||||||
|
elements: []string{"events", "test-event"},
|
||||||
|
want: filepath.Join("data", "events", "test-event"),
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "empty elements",
|
||||||
|
base: "data",
|
||||||
|
elements: []string{},
|
||||||
|
want: "data",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "with path separators",
|
||||||
|
base: "data/events",
|
||||||
|
elements: []string{"test-event", "1080p"},
|
||||||
|
want: filepath.Join("data", "events", "test-event", "1080p"),
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
got := SafeJoin(tt.base, tt.elements...)
|
||||||
|
if got != tt.want {
|
||||||
|
t.Errorf("SafeJoin() = %v, want %v", got, tt.want)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestEnsureDir(t *testing.T) {
|
||||||
|
tempDir, err := os.MkdirTemp("", "utils_test_*")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Failed to create temp dir: %v", err)
|
||||||
|
}
|
||||||
|
defer os.RemoveAll(tempDir)
|
||||||
|
|
||||||
|
testPath := filepath.Join(tempDir, "test", "nested", "directory")
|
||||||
|
|
||||||
|
// Test creating nested directories
|
||||||
|
err = EnsureDir(testPath)
|
||||||
|
if err != nil {
|
||||||
|
t.Errorf("EnsureDir() failed: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Verify directory was created
|
||||||
|
if _, err := os.Stat(testPath); os.IsNotExist(err) {
|
||||||
|
t.Errorf("Directory was not created: %s", testPath)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Test with existing directory (should not fail)
|
||||||
|
err = EnsureDir(testPath)
|
||||||
|
if err != nil {
|
||||||
|
t.Errorf("EnsureDir() failed on existing directory: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestPathExists(t *testing.T) {
|
||||||
|
tempDir, err := os.MkdirTemp("", "utils_test_*")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Failed to create temp dir: %v", err)
|
||||||
|
}
|
||||||
|
defer os.RemoveAll(tempDir)
|
||||||
|
|
||||||
|
// Test existing path
|
||||||
|
if !PathExists(tempDir) {
|
||||||
|
t.Errorf("PathExists() should return true for existing path: %s", tempDir)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Test non-existing path
|
||||||
|
nonExistentPath := filepath.Join(tempDir, "does-not-exist")
|
||||||
|
if PathExists(nonExistentPath) {
|
||||||
|
t.Errorf("PathExists() should return false for non-existent path: %s", nonExistentPath)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Test with file
|
||||||
|
testFile := filepath.Join(tempDir, "test.txt")
|
||||||
|
f, err := os.Create(testFile)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Failed to create test file: %v", err)
|
||||||
|
}
|
||||||
|
f.Close()
|
||||||
|
|
||||||
|
if !PathExists(testFile) {
|
||||||
|
t.Errorf("PathExists() should return true for existing file: %s", testFile)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestIsValidPath(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
path string
|
||||||
|
want bool
|
||||||
|
}{
|
||||||
|
{"empty path", "", false},
|
||||||
|
{"valid path", "data/events/test", true},
|
||||||
|
{"path with colon", "data:events", false},
|
||||||
|
{"path with pipe", "data|events", false},
|
||||||
|
{"path with question mark", "data?events", false},
|
||||||
|
{"path with asterisk", "data*events", false},
|
||||||
|
{"path with quotes", "data\"events", false},
|
||||||
|
{"path with angle brackets", "data<events>", false},
|
||||||
|
{"normal windows path", "C:\\data\\events", true}, // Windows path separators are actually OK
|
||||||
|
{"unix path", "/home/user/data", true},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
got := IsValidPath(tt.path)
|
||||||
|
if got != tt.want {
|
||||||
|
t.Errorf("IsValidPath(%q) = %v, want %v", tt.path, got, tt.want)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestNormalizePath(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
path string
|
||||||
|
want string
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
name: "windows backslashes",
|
||||||
|
path: "data\\events\\test",
|
||||||
|
want: filepath.Join("data", "events", "test"),
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "unix forward slashes",
|
||||||
|
path: "data/events/test",
|
||||||
|
want: filepath.Join("data", "events", "test"),
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "mixed slashes",
|
||||||
|
path: "data\\events/test\\file",
|
||||||
|
want: filepath.Join("data", "events", "test", "file"),
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "redundant separators",
|
||||||
|
path: "data//events\\\\test",
|
||||||
|
want: filepath.Join("data", "events", "test"),
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
got := NormalizePath(tt.path)
|
||||||
|
if got != tt.want {
|
||||||
|
t.Errorf("NormalizePath(%q) = %q, want %q", tt.path, got, tt.want)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestGetRelativePath(t *testing.T) {
|
||||||
|
tempDir, err := os.MkdirTemp("", "utils_test_*")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Failed to create temp dir: %v", err)
|
||||||
|
}
|
||||||
|
defer os.RemoveAll(tempDir)
|
||||||
|
|
||||||
|
basePath := tempDir
|
||||||
|
targetPath := filepath.Join(tempDir, "subdir", "file.txt")
|
||||||
|
|
||||||
|
rel, err := GetRelativePath(basePath, targetPath)
|
||||||
|
if err != nil {
|
||||||
|
t.Errorf("GetRelativePath() failed: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
expected := filepath.Join("subdir", "file.txt")
|
||||||
|
if rel != expected {
|
||||||
|
t.Errorf("GetRelativePath() = %q, want %q", rel, expected)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Test with invalid paths
|
||||||
|
_, err = GetRelativePath("", "")
|
||||||
|
if err == nil {
|
||||||
|
t.Error("GetRelativePath() should fail with empty paths")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestValidateWritablePath(t *testing.T) {
|
||||||
|
tempDir, err := os.MkdirTemp("", "utils_test_*")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Failed to create temp dir: %v", err)
|
||||||
|
}
|
||||||
|
defer os.RemoveAll(tempDir)
|
||||||
|
|
||||||
|
// Test writable path
|
||||||
|
writablePath := filepath.Join(tempDir, "test", "file.txt")
|
||||||
|
err = ValidateWritablePath(writablePath)
|
||||||
|
if err != nil {
|
||||||
|
t.Errorf("ValidateWritablePath() failed for writable path: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Verify directory was created
|
||||||
|
dir := filepath.Dir(writablePath)
|
||||||
|
if _, err := os.Stat(dir); os.IsNotExist(err) {
|
||||||
|
t.Errorf("Directory should have been created: %s", dir)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Test with read-only directory (if supported by OS)
|
||||||
|
if runtime.GOOS != "windows" { // Skip on Windows as it's more complex
|
||||||
|
readOnlyDir := filepath.Join(tempDir, "readonly")
|
||||||
|
os.MkdirAll(readOnlyDir, 0755)
|
||||||
|
os.Chmod(readOnlyDir, 0444) // Read-only
|
||||||
|
defer os.Chmod(readOnlyDir, 0755) // Restore permissions for cleanup
|
||||||
|
|
||||||
|
readOnlyPath := filepath.Join(readOnlyDir, "file.txt")
|
||||||
|
err = ValidateWritablePath(readOnlyPath)
|
||||||
|
if err == nil {
|
||||||
|
t.Error("ValidateWritablePath() should fail for read-only directory")
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), "not writable") {
|
||||||
|
t.Errorf("Expected 'not writable' error, got: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
146
test_runner.go
Normal file
146
test_runner.go
Normal file
@ -0,0 +1,146 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"fmt"
|
||||||
|
"os"
|
||||||
|
"os/exec"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
func main() {
|
||||||
|
fmt.Println("🧪 StreamRecorder Test Suite")
|
||||||
|
fmt.Println("============================")
|
||||||
|
|
||||||
|
startTime := time.Now()
|
||||||
|
|
||||||
|
// Set test environment to avoid interfering with real data
|
||||||
|
originalEnv := setupTestEnvironment()
|
||||||
|
defer restoreEnvironment(originalEnv)
|
||||||
|
|
||||||
|
// Create temporary directory for test data
|
||||||
|
tempDir, err := os.MkdirTemp("", "streamrecorder_test_*")
|
||||||
|
if err != nil {
|
||||||
|
fmt.Printf("❌ Failed to create temp directory: %v\n", err)
|
||||||
|
os.Exit(1)
|
||||||
|
}
|
||||||
|
defer func() {
|
||||||
|
fmt.Printf("🧹 Cleaning up test directory: %s\n", tempDir)
|
||||||
|
os.RemoveAll(tempDir)
|
||||||
|
}()
|
||||||
|
|
||||||
|
// Set test-specific environment variables
|
||||||
|
os.Setenv("LOCAL_OUTPUT_DIR", filepath.Join(tempDir, "data"))
|
||||||
|
os.Setenv("PROCESS_OUTPUT_DIR", filepath.Join(tempDir, "out"))
|
||||||
|
os.Setenv("ENABLE_NAS_TRANSFER", "false") // Disable NAS for tests
|
||||||
|
os.Setenv("PROCESSING_ENABLED", "false") // Disable processing that needs FFmpeg
|
||||||
|
|
||||||
|
fmt.Printf("📁 Using temporary directory: %s\n", tempDir)
|
||||||
|
fmt.Println()
|
||||||
|
|
||||||
|
// Run tests for each package
|
||||||
|
packages := []string{
|
||||||
|
"./pkg/config",
|
||||||
|
"./pkg/utils",
|
||||||
|
"./pkg/constants",
|
||||||
|
"./pkg/httpClient",
|
||||||
|
"./pkg/media",
|
||||||
|
"./pkg/processing",
|
||||||
|
}
|
||||||
|
|
||||||
|
var failedPackages []string
|
||||||
|
totalTests := 0
|
||||||
|
passedTests := 0
|
||||||
|
|
||||||
|
for _, pkg := range packages {
|
||||||
|
fmt.Printf("🔍 Testing package: %s\n", pkg)
|
||||||
|
|
||||||
|
cmd := exec.Command("go", "test", "-v", pkg)
|
||||||
|
cmd.Dir = "."
|
||||||
|
|
||||||
|
output, err := cmd.CombinedOutput()
|
||||||
|
outputStr := string(output)
|
||||||
|
|
||||||
|
// Count tests
|
||||||
|
testCount := strings.Count(outputStr, "=== RUN")
|
||||||
|
passCount := strings.Count(outputStr, "--- PASS:")
|
||||||
|
|
||||||
|
totalTests += testCount
|
||||||
|
passedTests += passCount
|
||||||
|
|
||||||
|
if err != nil {
|
||||||
|
fmt.Printf("❌ FAILED: %s (%d/%d tests passed)\n", pkg, passCount, testCount)
|
||||||
|
failedPackages = append(failedPackages, pkg)
|
||||||
|
|
||||||
|
// Show failure details
|
||||||
|
lines := strings.Split(outputStr, "\n")
|
||||||
|
for _, line := range lines {
|
||||||
|
if strings.Contains(line, "FAIL:") ||
|
||||||
|
strings.Contains(line, "Error:") ||
|
||||||
|
strings.Contains(line, "panic:") {
|
||||||
|
fmt.Printf(" %s\n", line)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
fmt.Printf("✅ PASSED: %s (%d tests)\n", pkg, testCount)
|
||||||
|
}
|
||||||
|
fmt.Println()
|
||||||
|
}
|
||||||
|
|
||||||
|
// Print summary
|
||||||
|
duration := time.Since(startTime)
|
||||||
|
fmt.Println("📊 Test Summary")
|
||||||
|
fmt.Println("===============")
|
||||||
|
fmt.Printf("Total packages: %d\n", len(packages))
|
||||||
|
fmt.Printf("Passed packages: %d\n", len(packages)-len(failedPackages))
|
||||||
|
fmt.Printf("Failed packages: %d\n", len(failedPackages))
|
||||||
|
fmt.Printf("Total tests: %d\n", totalTests)
|
||||||
|
fmt.Printf("Passed tests: %d\n", passedTests)
|
||||||
|
fmt.Printf("Failed tests: %d\n", totalTests-passedTests)
|
||||||
|
fmt.Printf("Duration: %v\n", duration.Round(time.Millisecond))
|
||||||
|
|
||||||
|
if len(failedPackages) > 0 {
|
||||||
|
fmt.Println()
|
||||||
|
fmt.Println("❌ Failed packages:")
|
||||||
|
for _, pkg := range failedPackages {
|
||||||
|
fmt.Printf(" - %s\n", pkg)
|
||||||
|
}
|
||||||
|
os.Exit(1)
|
||||||
|
} else {
|
||||||
|
fmt.Println()
|
||||||
|
fmt.Println("🎉 All tests passed!")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func setupTestEnvironment() map[string]string {
|
||||||
|
// Save original environment variables that we'll modify
|
||||||
|
originalEnv := make(map[string]string)
|
||||||
|
|
||||||
|
envVars := []string{
|
||||||
|
"LOCAL_OUTPUT_DIR",
|
||||||
|
"PROCESS_OUTPUT_DIR",
|
||||||
|
"ENABLE_NAS_TRANSFER",
|
||||||
|
"PROCESSING_ENABLED",
|
||||||
|
"NAS_OUTPUT_PATH",
|
||||||
|
"FFMPEG_PATH",
|
||||||
|
"WORKER_COUNT",
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, envVar := range envVars {
|
||||||
|
originalEnv[envVar] = os.Getenv(envVar)
|
||||||
|
}
|
||||||
|
|
||||||
|
return originalEnv
|
||||||
|
}
|
||||||
|
|
||||||
|
func restoreEnvironment(originalEnv map[string]string) {
|
||||||
|
fmt.Println("🔄 Restoring original environment...")
|
||||||
|
for envVar, originalValue := range originalEnv {
|
||||||
|
if originalValue == "" {
|
||||||
|
os.Unsetenv(envVar)
|
||||||
|
} else {
|
||||||
|
os.Setenv(envVar, originalValue)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
Loading…
x
Reference in New Issue
Block a user