StreamRecorder/CLAUDE.md

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Project Overview

This is a Go-based HLS (HTTP Live Streaming) recorder that monitors M3U8 playlists and downloads video segments in real-time with automatic NAS transfer capabilities. The program takes a master M3U8 playlist URL, parses all available stream variants (different qualities/bitrates), continuously monitors each variant's chunklist for new segments, downloads them locally, and optionally transfers them to network storage for long-term archival.

Architecture

The project follows a modular architecture with clear separation of concerns:

• cmd/: Entry points for different execution modes
  • main/main.go: Primary CLI entry point with URL input, event naming, and mode selection
  • downloader/download.go: Core download orchestration logic with transfer service integration
  • processor/process.go: Alternative processing entry point
  • transfer/transfer.go: Transfer-only mode entry point
• pkg/: Core packages containing the application logic
  • media/: HLS streaming and download logic
    • stream.go: Stream variant parsing and downloading orchestration (GetAllVariants, VariantDownloader)
    • playlist.go: M3U8 playlist loading and parsing (LoadMediaPlaylist)
    • segment.go: Individual segment downloading logic (DownloadSegment, SegmentJob)
    • manifest.go: Manifest generation and segment tracking (ManifestWriter, ManifestItem)
  • transfer/: NAS transfer system (complete implementation available)
    • service.go: Transfer service orchestration
    • watcher.go: File system monitoring for new downloads
    • queue.go: Priority queue with worker pool management
    • nas.go: NAS file transfer with retry logic
    • cleanup.go: Local file cleanup after successful transfer
    • types.go: Transfer system data structures
  • processing/: Video processing and concatenation system
    • service.go: Processing service orchestration with FFmpeg integration
    • segment.go: Individual segment processing logic
    • types.go: Processing system data structures
  • nas/: NAS connection and file operations
    • config.go: NAS configuration structure
    • nas.go: NAS service with connection management and file operations
  • config/: Centralized configuration management with validation
    • config.go: Configuration loading, validation, and path resolution
  • utils/: Utility functions for cross-platform compatibility
    • paths.go: Path manipulation and validation utilities
  • constants/constants.go: Configuration constants and singleton access
  • httpClient/error.go: HTTP error handling utilities

Core Functionality

Download Workflow

1. Parse Master Playlist: GetAllVariants() fetches and parses the master M3U8 to extract all stream variants with different qualities/bitrates
2. Concurrent Monitoring: Each variant gets its own goroutine running VariantDownloader() that continuously polls for playlist updates
3. Segment Detection: When new segments appear in a variant's playlist, they are queued for download
4. Parallel Downloads: Segments are downloaded concurrently with configurable worker pools and retry logic
5. Quality Organization: Downloaded segments are organized by resolution (1080p, 720p, etc.) in separate directories
6. Manifest Generation: ManifestWriter tracks all downloaded segments with sequence numbers and resolutions

NAS Transfer Workflow (Optional)

1. File Watching: FileWatcher monitors download directories for new .ts files
2. Transfer Queuing: New files are added to a priority queue after a settling delay
3. Background Transfer: Worker pool transfers files to NAS with retry logic and verification
4. Local Cleanup: Successfully transferred files are automatically cleaned up locally
5. State Persistence: Queue state is persisted to survive crashes and restarts

Video Processing Workflow (Optional)

1. Segment Collection: Processing service reads downloaded segments from NAS storage
2. Quality Selection: Automatically selects the highest quality variant available
3. FFmpeg Processing: Uses FFmpeg to concatenate segments into a single MP4 file
4. Output Management: Processed videos are saved to the configured output directory
5. Concurrent Processing: Multiple events can be processed simultaneously with worker pools

Key Data Structures

• StreamVariant: Represents a stream quality variant with URL, bandwidth, resolution, output directory, and manifest writer
• SegmentJob: Represents a segment download task with URI, sequence number, and variant info
• ManifestWriter: Tracks downloaded segments and generates JSON manifests
• ManifestItem: Individual segment record with sequence number and resolution
• TransferItem: Transfer queue item with source, destination, retry count, and status
• TransferService: Orchestrates file watching, queuing, transfer, and cleanup
• ProcessingService: Manages video processing operations with FFmpeg integration
• ProcessConfig: Configuration for processing operations including worker count and paths
• NASService: Handles NAS connection, authentication, and file operations
• NASConfig: Configuration structure for NAS connection parameters

Configuration

Configuration is managed through a centralized system in pkg/config/config.go with environment variable support for deployment flexibility. The system provides validation, cross-platform path resolution, and sensible defaults:

Core Settings

• Core.WorkerCount: Number of concurrent segment downloaders per variant (4) - ENV: WORKER_COUNT
• Core.RefreshDelay: How often to check for playlist updates (3 seconds) - ENV: REFRESH_DELAY_SECONDS

Path Configuration

• Paths.LocalOutput: Base directory for local downloads (data/) - ENV: LOCAL_OUTPUT_DIR
• Paths.ProcessOutput: Directory for processed videos (out/) - ENV: PROCESS_OUTPUT_DIR
• Paths.ManifestDir: Directory for manifest JSON files (data/)
• Paths.PersistenceFile: Transfer queue state file location

HTTP Settings

• HTTPUserAgent: User agent string for HTTP requests
• REFERRER: Referer header for HTTP requests (https://www.flomarching.com)

NAS Transfer Settings

• NAS.EnableTransfer: Enable/disable automatic NAS transfer (true) - ENV: ENABLE_NAS_TRANSFER
• NAS.OutputPath: UNC path to NAS storage (\\HomeLabNAS\dci\streams) - ENV: NAS_OUTPUT_PATH
• NAS.Username/NAS.Password: NAS credentials for authentication - ENV: NAS_USERNAME/NAS_PASSWORD
• Transfer.WorkerCount: Concurrent transfer workers (2)
• Transfer.RetryLimit: Max retry attempts per file (3)
• Transfer.Timeout: Timeout per file transfer (30 seconds)
• Transfer.FileSettlingDelay: Wait before queuing new files (5 seconds)
• Transfer.QueueSize: Maximum queue size (100000)
• Transfer.BatchSize: Batch processing size (1000)

Processing Settings

• Processing.AutoProcess: Enable automatic processing after download (true)
• Processing.Enabled: Enable processing functionality (true)
• Processing.WorkerCount: Concurrent processing workers (2)
• Processing.FFmpegPath: Path to FFmpeg executable (ffmpeg) - ENV: FFMPEG_PATH

Cleanup Settings

• Cleanup.AfterTransfer: Delete local files after NAS transfer (true)
• Cleanup.BatchSize: Files processed per cleanup batch (1000)
• Cleanup.RetainHours: Hours to keep local files (0 = immediate cleanup)

Configuration Access

cfg := constants.MustGetConfig()  // Get validated config singleton
eventPath := cfg.GetEventPath("my-event")  // Get cross-platform paths

See DEPLOYMENT.md for detailed environment variable configuration and deployment examples.

Common Development Commands

# Build the main application
go build -o stream-recorder ./cmd/main

# Run with URL prompt
go run ./cmd/main/main.go

# Run with command line arguments
go run ./cmd/main/main.go -url="https://example.com/playlist.m3u8" -event="my-event" -debug=true

# Run with module support
go mod tidy

# Test the project (when tests are added)
go test ./...

# Format code
go fmt ./...

Command Line Options

• -url: M3U8 playlist URL (if not provided, prompts for input)
• -event: Event name for organizing downloads (defaults to current date)
• -debug: Debug mode (only downloads 1080p variant for easier testing)
• -transfer: Transfer-only mode (transfer existing files without downloading)
• -process: Process-only mode (process existing files without downloading)

Monitoring and Downloads

The application implements comprehensive real-time stream monitoring:

Download Features

• Continuous Polling: Each variant playlist is checked every 3 seconds for new segments
• Deduplication: Uses segment URIs and sequence numbers to avoid re-downloading
• Graceful Shutdown: Responds to SIGINT/SIGTERM signals for clean exit
• Error Resilience: Retries failed downloads and handles HTTP 403 errors specially
• Quality Detection: Automatically determines resolution from bandwidth or explicit resolution data
• Context Cancellation: Proper timeout and cancellation handling for clean shutdowns

Transfer Features (when enabled)

• Real-time Transfer: Files are transferred to NAS as soon as they're downloaded
• Queue Persistence: Transfer queue survives application restarts
• Retry Logic: Failed transfers are retried with exponential backoff
• Verification: File sizes are verified after transfer
• Automatic Cleanup: Local files are removed after successful NAS transfer
• Statistics Reporting: Transfer progress and statistics are logged regularly

Manifest Generation

• Segment Tracking: All downloaded segments are tracked with sequence numbers
• Resolution Mapping: Segments are associated with their quality variants
• JSON Output: Manifest files are generated as sorted JSON arrays for easy processing

Error Handling

The implementation uses proper Go error handling patterns:

• Custom HTTP Errors: Structured error types for HTTP failures
• Context-Aware Cancellation: Proper handling of shutdown scenarios
• Retry Logic: Exponential backoff for transient failures
• Logging: Clear status indicators (✓ for success, ✗ for failure)
• Graceful Degradation: Transfer service failures don't stop downloads

Dependencies

• github.com/grafov/m3u8: M3U8 playlist parsing
• github.com/fsnotify/fsnotify: File system event monitoring for NAS transfers

Data Organization

Downloaded files are organized as:

./data/
├── {event-name}.json          # Manifest file
├── {event-name}/              # Event-specific directory
│   ├── 1080p/                 # High quality segments
│   ├── 720p/                  # Medium quality segments
│   └── 480p/                  # Lower quality segments
├── transfer_queue.json        # Transfer queue state
├── refresh_token.txt          # Authentication tokens
└── tokens.txt                 # Session tokens

NAS files mirror the local structure:

\\HomeLabNAS\dci\streams\
└── {event-name}/
    ├── 1080p/
    ├── 720p/
    └── 480p/

Processed files are output to:

./out/
└── {event-name}/
    └── concatenated_segments.mp4   # Final processed video