228 lines
11 KiB
Markdown
228 lines
11 KiB
Markdown
# CLAUDE.md
|
|
|
|
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
|
|
|
## Project Overview
|
|
|
|
This is a Go-based HLS (HTTP Live Streaming) recorder that monitors M3U8 playlists and downloads video segments in real-time with automatic NAS transfer capabilities. The program takes a master M3U8 playlist URL, parses all available stream variants (different qualities/bitrates), continuously monitors each variant's chunklist for new segments, downloads them locally, and optionally transfers them to network storage for long-term archival.
|
|
|
|
## Architecture
|
|
|
|
The project follows a modular architecture with clear separation of concerns:
|
|
|
|
- **cmd/**: Entry points for different execution modes
|
|
- **main/main.go**: Primary CLI entry point with URL input, event naming, and mode selection
|
|
- **downloader/download.go**: Core download orchestration logic with transfer service integration
|
|
- **processor/process.go**: Alternative processing entry point
|
|
- **transfer/transfer.go**: Transfer-only mode entry point
|
|
- **pkg/**: Core packages containing the application logic
|
|
- **media/**: HLS streaming and download logic
|
|
- **stream.go**: Stream variant parsing and downloading orchestration (`GetAllVariants`, `VariantDownloader`)
|
|
- **playlist.go**: M3U8 playlist loading and parsing (`LoadMediaPlaylist`)
|
|
- **segment.go**: Individual segment downloading logic (`DownloadSegment`, `SegmentJob`)
|
|
- **manifest.go**: Manifest generation and segment tracking (`ManifestWriter`, `ManifestItem`)
|
|
- **transfer/**: NAS transfer system (complete implementation available)
|
|
- **service.go**: Transfer service orchestration
|
|
- **watcher.go**: File system monitoring for new downloads
|
|
- **queue.go**: Priority queue with worker pool management
|
|
- **nas.go**: NAS file transfer with retry logic
|
|
- **cleanup.go**: Local file cleanup after successful transfer
|
|
- **types.go**: Transfer system data structures
|
|
- **processing/**: Video processing and concatenation system
|
|
- **service.go**: Processing service orchestration with FFmpeg integration
|
|
- **segment.go**: Individual segment processing logic
|
|
- **types.go**: Processing system data structures
|
|
- **nas/**: NAS connection and file operations
|
|
- **config.go**: NAS configuration structure
|
|
- **nas.go**: NAS service with connection management and file operations
|
|
- **config/**: Centralized configuration management with validation
|
|
- **config.go**: Configuration loading, validation, and path resolution
|
|
- **utils/**: Utility functions for cross-platform compatibility
|
|
- **paths.go**: Path manipulation and validation utilities
|
|
- **constants/constants.go**: Configuration constants and singleton access
|
|
- **httpClient/error.go**: HTTP error handling utilities
|
|
|
|
## Core Functionality
|
|
|
|
### Download Workflow
|
|
1. **Parse Master Playlist**: `GetAllVariants()` fetches and parses the master M3U8 to extract all stream variants with different qualities/bitrates
|
|
2. **Concurrent Monitoring**: Each variant gets its own goroutine running `VariantDownloader()` that continuously polls for playlist updates
|
|
3. **Segment Detection**: When new segments appear in a variant's playlist, they are queued for download
|
|
4. **Parallel Downloads**: Segments are downloaded concurrently with configurable worker pools and retry logic
|
|
5. **Quality Organization**: Downloaded segments are organized by resolution (1080p, 720p, etc.) in separate directories
|
|
6. **Manifest Generation**: `ManifestWriter` tracks all downloaded segments with sequence numbers and resolutions
|
|
|
|
### NAS Transfer Workflow (Optional)
|
|
1. **File Watching**: `FileWatcher` monitors download directories for new `.ts` files
|
|
2. **Transfer Queuing**: New files are added to a priority queue after a settling delay
|
|
3. **Background Transfer**: Worker pool transfers files to NAS with retry logic and verification
|
|
4. **Local Cleanup**: Successfully transferred files are automatically cleaned up locally
|
|
5. **State Persistence**: Queue state is persisted to survive crashes and restarts
|
|
|
|
### Video Processing Workflow (Optional)
|
|
1. **Segment Collection**: Processing service reads downloaded segments from NAS storage
|
|
2. **Quality Selection**: Automatically selects the highest quality variant available
|
|
3. **FFmpeg Processing**: Uses FFmpeg to concatenate segments into a single MP4 file
|
|
4. **Output Management**: Processed videos are saved to the configured output directory
|
|
5. **Concurrent Processing**: Multiple events can be processed simultaneously with worker pools
|
|
|
|
## Key Data Structures
|
|
|
|
- `StreamVariant`: Represents a stream quality variant with URL, bandwidth, resolution, output directory, and manifest writer
|
|
- `SegmentJob`: Represents a segment download task with URI, sequence number, and variant info
|
|
- `ManifestWriter`: Tracks downloaded segments and generates JSON manifests
|
|
- `ManifestItem`: Individual segment record with sequence number and resolution
|
|
- `TransferItem`: Transfer queue item with source, destination, retry count, and status
|
|
- `TransferService`: Orchestrates file watching, queuing, transfer, and cleanup
|
|
- `ProcessingService`: Manages video processing operations with FFmpeg integration
|
|
- `ProcessConfig`: Configuration for processing operations including worker count and paths
|
|
- `NASService`: Handles NAS connection, authentication, and file operations
|
|
- `NASConfig`: Configuration structure for NAS connection parameters
|
|
|
|
## Configuration
|
|
|
|
Configuration is managed through a centralized system in `pkg/config/config.go` with environment variable support for deployment flexibility. The system provides validation, cross-platform path resolution, and sensible defaults:
|
|
|
|
### Core Settings
|
|
- `Core.WorkerCount`: Number of concurrent segment downloaders per variant (4) - ENV: `WORKER_COUNT`
|
|
- `Core.RefreshDelay`: How often to check for playlist updates (3 seconds) - ENV: `REFRESH_DELAY_SECONDS`
|
|
|
|
### Path Configuration
|
|
- `Paths.LocalOutput`: Base directory for local downloads (`data/`) - ENV: `LOCAL_OUTPUT_DIR`
|
|
- `Paths.ProcessOutput`: Directory for processed videos (`out/`) - ENV: `PROCESS_OUTPUT_DIR`
|
|
- `Paths.ManifestDir`: Directory for manifest JSON files (`data/`)
|
|
- `Paths.PersistenceFile`: Transfer queue state file location
|
|
|
|
### HTTP Settings
|
|
- `HTTPUserAgent`: User agent string for HTTP requests
|
|
- `REFERRER`: Referer header for HTTP requests (`https://www.flomarching.com`)
|
|
|
|
### NAS Transfer Settings
|
|
- `NAS.EnableTransfer`: Enable/disable automatic NAS transfer (true) - ENV: `ENABLE_NAS_TRANSFER`
|
|
- `NAS.OutputPath`: UNC path to NAS storage (`\\HomeLabNAS\dci\streams`) - ENV: `NAS_OUTPUT_PATH`
|
|
- `NAS.Username`/`NAS.Password`: NAS credentials for authentication - ENV: `NAS_USERNAME`/`NAS_PASSWORD`
|
|
- `Transfer.WorkerCount`: Concurrent transfer workers (2)
|
|
- `Transfer.RetryLimit`: Max retry attempts per file (3)
|
|
- `Transfer.Timeout`: Timeout per file transfer (30 seconds)
|
|
- `Transfer.FileSettlingDelay`: Wait before queuing new files (5 seconds)
|
|
- `Transfer.QueueSize`: Maximum queue size (100000)
|
|
- `Transfer.BatchSize`: Batch processing size (1000)
|
|
|
|
### Processing Settings
|
|
- `Processing.AutoProcess`: Enable automatic processing after download (true)
|
|
- `Processing.Enabled`: Enable processing functionality (true)
|
|
- `Processing.WorkerCount`: Concurrent processing workers (2)
|
|
- `Processing.FFmpegPath`: Path to FFmpeg executable (`ffmpeg`) - ENV: `FFMPEG_PATH`
|
|
|
|
### Cleanup Settings
|
|
- `Cleanup.AfterTransfer`: Delete local files after NAS transfer (true)
|
|
- `Cleanup.BatchSize`: Files processed per cleanup batch (1000)
|
|
- `Cleanup.RetainHours`: Hours to keep local files (0 = immediate cleanup)
|
|
|
|
### Configuration Access
|
|
```go
|
|
cfg := constants.MustGetConfig() // Get validated config singleton
|
|
eventPath := cfg.GetEventPath("my-event") // Get cross-platform paths
|
|
```
|
|
|
|
See `DEPLOYMENT.md` for detailed environment variable configuration and deployment examples.
|
|
|
|
## Common Development Commands
|
|
|
|
```bash
|
|
# Build the main application
|
|
go build -o stream-recorder ./cmd/main
|
|
|
|
# Run with URL prompt
|
|
go run ./cmd/main/main.go
|
|
|
|
# Run with command line arguments
|
|
go run ./cmd/main/main.go -url="https://example.com/playlist.m3u8" -event="my-event" -debug=true
|
|
|
|
# Run with module support
|
|
go mod tidy
|
|
|
|
# Test the project (when tests are added)
|
|
go test ./...
|
|
|
|
# Format code
|
|
go fmt ./...
|
|
```
|
|
|
|
## Command Line Options
|
|
|
|
- `-url`: M3U8 playlist URL (if not provided, prompts for input)
|
|
- `-event`: Event name for organizing downloads (defaults to current date)
|
|
- `-debug`: Debug mode (only downloads 1080p variant for easier testing)
|
|
- `-transfer`: Transfer-only mode (transfer existing files without downloading)
|
|
- `-process`: Process-only mode (process existing files without downloading)
|
|
|
|
## Monitoring and Downloads
|
|
|
|
The application implements comprehensive real-time stream monitoring:
|
|
|
|
### Download Features
|
|
- **Continuous Polling**: Each variant playlist is checked every 3 seconds for new segments
|
|
- **Deduplication**: Uses segment URIs and sequence numbers to avoid re-downloading
|
|
- **Graceful Shutdown**: Responds to SIGINT/SIGTERM signals for clean exit
|
|
- **Error Resilience**: Retries failed downloads and handles HTTP 403 errors specially
|
|
- **Quality Detection**: Automatically determines resolution from bandwidth or explicit resolution data
|
|
- **Context Cancellation**: Proper timeout and cancellation handling for clean shutdowns
|
|
|
|
### Transfer Features (when enabled)
|
|
- **Real-time Transfer**: Files are transferred to NAS as soon as they're downloaded
|
|
- **Queue Persistence**: Transfer queue survives application restarts
|
|
- **Retry Logic**: Failed transfers are retried with exponential backoff
|
|
- **Verification**: File sizes are verified after transfer
|
|
- **Automatic Cleanup**: Local files are removed after successful NAS transfer
|
|
- **Statistics Reporting**: Transfer progress and statistics are logged regularly
|
|
|
|
### Manifest Generation
|
|
- **Segment Tracking**: All downloaded segments are tracked with sequence numbers
|
|
- **Resolution Mapping**: Segments are associated with their quality variants
|
|
- **JSON Output**: Manifest files are generated as sorted JSON arrays for easy processing
|
|
|
|
## Error Handling
|
|
|
|
The implementation uses proper Go error handling patterns:
|
|
- **Custom HTTP Errors**: Structured error types for HTTP failures
|
|
- **Context-Aware Cancellation**: Proper handling of shutdown scenarios
|
|
- **Retry Logic**: Exponential backoff for transient failures
|
|
- **Logging**: Clear status indicators (✓ for success, ✗ for failure)
|
|
- **Graceful Degradation**: Transfer service failures don't stop downloads
|
|
|
|
## Dependencies
|
|
|
|
- `github.com/grafov/m3u8`: M3U8 playlist parsing
|
|
- `github.com/fsnotify/fsnotify`: File system event monitoring for NAS transfers
|
|
|
|
## Data Organization
|
|
|
|
Downloaded files are organized as:
|
|
```
|
|
./data/
|
|
├── {event-name}.json # Manifest file
|
|
├── {event-name}/ # Event-specific directory
|
|
│ ├── 1080p/ # High quality segments
|
|
│ ├── 720p/ # Medium quality segments
|
|
│ └── 480p/ # Lower quality segments
|
|
├── transfer_queue.json # Transfer queue state
|
|
├── refresh_token.txt # Authentication tokens
|
|
└── tokens.txt # Session tokens
|
|
```
|
|
|
|
NAS files mirror the local structure:
|
|
```
|
|
\\HomeLabNAS\dci\streams\
|
|
└── {event-name}/
|
|
├── 1080p/
|
|
├── 720p/
|
|
└── 480p/
|
|
```
|
|
|
|
Processed files are output to:
|
|
```
|
|
./out/
|
|
└── {event-name}/
|
|
└── concatenated_segments.mp4 # Final processed video
|
|
``` |