StreamRecorder/CLAUDE.md

# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## Project Overview

This is a Go-based HLS (HTTP Live Streaming) recorder that monitors M3U8 playlists and downloads video segments in real-time with automatic NAS transfer capabilities. The program takes a master M3U8 playlist URL, parses all available stream variants (different qualities/bitrates), continuously monitors each variant's chunklist for new segments, downloads them locally, and optionally transfers them to network storage for long-term archival.

## Architecture

The project follows a modular architecture with clear separation of concerns:

- **cmd/**: Entry points for different execution modes
  - **main/main.go**: Primary CLI entry point with URL input and event naming
  - **downloader/download.go**: Core download orchestration logic with transfer service integration
  - **processor/process.go**: Alternative processing entry point
- **pkg/**: Core packages containing the application logic
  - **media/**: HLS streaming and download logic
    - **stream.go**: Stream variant parsing and downloading orchestration (`GetAllVariants`, `VariantDownloader`)
    - **playlist.go**: M3U8 playlist loading and parsing (`LoadMediaPlaylist`)
    - **segment.go**: Individual segment downloading logic (`DownloadSegment`, `SegmentJob`)
    - **manifest.go**: Manifest generation and segment tracking (`ManifestWriter`, `ManifestItem`)
  - **transfer/**: NAS transfer system (complete implementation available)
    - **service.go**: Transfer service orchestration
    - **watcher.go**: File system monitoring for new downloads
    - **queue.go**: Priority queue with worker pool management
    - **nas.go**: NAS file transfer with retry logic
    - **cleanup.go**: Local file cleanup after successful transfer
    - **types.go**: Transfer system data structures
  - **constants/constants.go**: Configuration constants (paths, timeouts, NAS settings)
  - **httpClient/error.go**: HTTP error handling utilities

## Core Functionality

### Download Workflow
1. **Parse Master Playlist**: `GetAllVariants()` fetches and parses the master M3U8 to extract all stream variants with different qualities/bitrates
2. **Concurrent Monitoring**: Each variant gets its own goroutine running `VariantDownloader()` that continuously polls for playlist updates
3. **Segment Detection**: When new segments appear in a variant's playlist, they are queued for download
4. **Parallel Downloads**: Segments are downloaded concurrently with configurable worker pools and retry logic
5. **Quality Organization**: Downloaded segments are organized by resolution (1080p, 720p, etc.) in separate directories
6. **Manifest Generation**: `ManifestWriter` tracks all downloaded segments with sequence numbers and resolutions

### NAS Transfer Workflow (Optional)
1. **File Watching**: `FileWatcher` monitors download directories for new `.ts` files
2. **Transfer Queuing**: New files are added to a priority queue after a settling delay
3. **Background Transfer**: Worker pool transfers files to NAS with retry logic and verification
4. **Local Cleanup**: Successfully transferred files are automatically cleaned up locally
5. **State Persistence**: Queue state is persisted to survive crashes and restarts

## Key Data Structures

- `StreamVariant`: Represents a stream quality variant with URL, bandwidth, resolution, output directory, and manifest writer
- `SegmentJob`: Represents a segment download task with URI, sequence number, and variant info
- `ManifestWriter`: Tracks downloaded segments and generates JSON manifests
- `ManifestItem`: Individual segment record with sequence number and resolution
- `TransferItem`: Transfer queue item with source, destination, retry count, and status
- `TransferService`: Orchestrates file watching, queuing, transfer, and cleanup

## Configuration

Key configuration is managed in `pkg/constants/constants.go`:

### Core Settings
- `WorkerCount`: Number of concurrent segment downloaders per variant (4)
- `RefreshDelay`: How often to check for playlist updates (3 seconds)
- `LocalOutputDirPath`: Base directory for local downloads (`./data/`)
- `ManifestPath`: Directory for manifest JSON files (`./data`)

### HTTP Settings
- `HTTPUserAgent`: User agent string for HTTP requests
- `REFERRER`: Referer header for HTTP requests

### NAS Transfer Settings
- `EnableNASTransfer`: Enable/disable automatic NAS transfer (true)
- `NASOutputPath`: UNC path to NAS storage (`\\HomeLabNAS\dci\streams`)
- `NASUsername`/`NASPassword`: NAS credentials (empty for current user)
- `TransferWorkerCount`: Concurrent transfer workers (2)
- `TransferRetryLimit`: Max retry attempts per file (3)
- `TransferTimeout`: Timeout per file transfer (30 seconds)
- `FileSettlingDelay`: Wait before queuing new files (5 seconds)
- `PersistencePath`: Transfer queue state file (`./data/transfer_queue.json`)

### Cleanup Settings
- `CleanupAfterTransfer`: Delete local files after NAS transfer (true)
- `CleanupBatchSize`: Files processed per cleanup batch (10)
- `RetainLocalHours`: Hours to keep local files (0 = immediate cleanup)

## Common Development Commands

```bash
# Build the main application
go build -o stream-recorder ./cmd/main

# Run with URL prompt
go run ./cmd/main/main.go

# Run with command line arguments
go run ./cmd/main/main.go -url="https://example.com/playlist.m3u8" -event="my-event" -debug=true

# Run with module support
go mod tidy

# Test the project (when tests are added)
go test ./...

# Format code
go fmt ./...
```

## Command Line Options

- `-url`: M3U8 playlist URL (if not provided, prompts for input)
- `-event`: Event name for organizing downloads (defaults to current date)
- `-debug`: Debug mode (only downloads 1080p variant for easier testing)

## Monitoring and Downloads

The application implements comprehensive real-time stream monitoring:

### Download Features
- **Continuous Polling**: Each variant playlist is checked every 3 seconds for new segments
- **Deduplication**: Uses segment URIs and sequence numbers to avoid re-downloading
- **Graceful Shutdown**: Responds to SIGINT/SIGTERM signals for clean exit
- **Error Resilience**: Retries failed downloads and handles HTTP 403 errors specially
- **Quality Detection**: Automatically determines resolution from bandwidth or explicit resolution data
- **Context Cancellation**: Proper timeout and cancellation handling for clean shutdowns

### Transfer Features (when enabled)
- **Real-time Transfer**: Files are transferred to NAS as soon as they're downloaded
- **Queue Persistence**: Transfer queue survives application restarts
- **Retry Logic**: Failed transfers are retried with exponential backoff
- **Verification**: File sizes are verified after transfer
- **Automatic Cleanup**: Local files are removed after successful NAS transfer
- **Statistics Reporting**: Transfer progress and statistics are logged regularly

### Manifest Generation
- **Segment Tracking**: All downloaded segments are tracked with sequence numbers
- **Resolution Mapping**: Segments are associated with their quality variants
- **JSON Output**: Manifest files are generated as sorted JSON arrays for easy processing

## Error Handling

The implementation uses proper Go error handling patterns:
- **Custom HTTP Errors**: Structured error types for HTTP failures
- **Context-Aware Cancellation**: Proper handling of shutdown scenarios
- **Retry Logic**: Exponential backoff for transient failures  
- **Logging**: Clear status indicators (✓ for success, ✗ for failure)
- **Graceful Degradation**: Transfer service failures don't stop downloads

## Dependencies

- `github.com/grafov/m3u8`: M3U8 playlist parsing
- `github.com/fsnotify/fsnotify`: File system event monitoring for NAS transfers

## Data Organization

Downloaded files are organized as:
```
./data/
├── {event-name}.json          # Manifest file
├── {event-name}/              # Event-specific directory
│   ├── 1080p/                 # High quality segments
│   ├── 720p/                  # Medium quality segments
│   └── 480p/                  # Lower quality segments
└── transfer_queue.json        # Transfer queue state
```

NAS files mirror the local structure:
```
\\HomeLabNAS\dci\streams\
└── {event-name}/
    ├── 1080p/
    ├── 720p/
    └── 480p/
```