diff --git a/CLAUDE.md b/CLAUDE.md index 165cd0e..54ec9d1 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -33,9 +33,15 @@ The project follows a modular architecture with clear separation of concerns: - **service.go**: Processing service orchestration with FFmpeg integration - **segment.go**: Individual segment processing logic - **types.go**: Processing system data structures - - **nas/**: NAS connection and file operations + - **nas/**: NAS connection and file operations (platform-agnostic: Windows and Linux) - **config.go**: NAS configuration structure - - **nas.go**: NAS service with connection management and file operations + - **nas.go**: `NASService` — resolves the destination to a host-native path, then copies/verifies files against it + - **location.go**: Parses UNC / `//server/share` / `smb://` / local paths into a host-neutral `Location` + - **connector.go**: `Connector` interface + runtime selection by GOOS (no build tags, so both platforms' code is compiled and tested everywhere) + - **connector_netuse.go**: Windows strategy — `net use`, UNC path used directly + - **connector_mount.go**: Linux strategy — resolve the share against `/proc/mounts`, optional root-only `mount.cifs` + - **connector_local.go**: Already-mounted/plain directory on either platform + - **mounts.go**: `/proc/mounts` parsing and share-to-mount-point matching - **config/**: Centralized configuration management with validation - **config.go**: Configuration loading, validation, and path resolution - **utils/**: Utility functions for cross-platform compatibility @@ -100,8 +106,10 @@ Configuration is managed through a centralized system in `pkg/config/config.go` ### NAS Transfer Settings - `NAS.EnableTransfer`: Enable/disable automatic NAS transfer (true) - ENV: `ENABLE_NAS_TRANSFER` -- `NAS.OutputPath`: UNC path to NAS storage (`\\HomeLabNAS\dci\streams`) - ENV: `NAS_OUTPUT_PATH` +- `NAS.OutputPath`: NAS destination (`\\HomeLabNAS\dci\streams`) - ENV: `NAS_OUTPUT_PATH`. Accepts UNC, `//server/share`, `smb://`, or a local/mounted directory; Windows connects it with `net use`, Linux resolves it against `/proc/mounts` - `NAS.Username`/`NAS.Password`: NAS credentials for authentication - ENV: `NAS_USERNAME`/`NAS_PASSWORD` +- `NAS.AutoMount`: Linux-only, root-required auto-mount of the SMB share (false) - ENV: `NAS_AUTO_MOUNT` +- `NAS.MountPoint`: Override for the auto-mount location - ENV: `NAS_MOUNT_POINT` - `Transfer.WorkerCount`: Concurrent transfer workers (2) - `Transfer.RetryLimit`: Max retry attempts per file (3) - `Transfer.Timeout`: Timeout per file transfer (30 seconds) diff --git a/DEPLOYMENT.md b/DEPLOYMENT.md index 8eaab62..ed28f11 100644 --- a/DEPLOYMENT.md +++ b/DEPLOYMENT.md @@ -11,10 +11,23 @@ The application supports configuration through environment variables for flexibl - `REFRESH_DELAY_SECONDS`: How often to check for playlist updates in seconds (default: 3) ### NAS Transfer Settings -- `NAS_OUTPUT_PATH`: UNC path to NAS storage (default: "\\\\HomeLabNAS\\dci\\streams") +- `NAS_OUTPUT_PATH`: NAS destination (default: "\\\\HomeLabNAS\\dci\\streams"). Accepts a UNC path, `//server/share/dir`, `smb://server/share/dir`, or a local/mounted directory - `NAS_USERNAME`: NAS authentication username - `NAS_PASSWORD`: NAS authentication password - `ENABLE_NAS_TRANSFER`: Enable/disable automatic NAS transfer (default: true) +- `NAS_AUTO_MOUNT`: Linux only — mount the SMB share if the host hasn't already (default: false, requires root) +- `NAS_MOUNT_POINT`: Where `NAS_AUTO_MOUNT` attaches the share (default: `/mnt/streamrecorder//`) + +The same `NAS_OUTPUT_PATH` works on Windows and Linux. Windows authenticates the +UNC path with `net use`; Linux resolves the share against `/proc/mounts` and +rewrites the path onto the mount point, so `\\\\HomeLabNAS\\dci\\streams` with +`//homelabnas/dci` mounted at `/mnt/dci` becomes `/mnt/dci/streams`. Mounting the +share on the host (fstab/autofs/systemd, or `docker run -v`) is the recommended +Linux setup; `NAS_AUTO_MOUNT` is the root-privileged fallback. Verify either with: + +```bash +./stream-recorder -check +``` ### Path Configuration - `LOCAL_OUTPUT_DIR`: Base directory for local downloads (default: "data") diff --git a/README.md b/README.md index c3d2ae7..75b62f2 100644 --- a/README.md +++ b/README.md @@ -43,7 +43,7 @@ Segments land on disk under `Paths.LocalOutput///`. - `service.go`: `TransferService` wires together the watcher, queue, and cleanup service, and owns the connection to NAS for the run. - `watcher.go`: `FileWatcher` (via `fsnotify`) watches the local event directory for new `.ts` files and queues them after a short settling delay (`Transfer.FileSettlingDelay`, default 5s), so a file isn't queued while still being written. - `queue.go`: `TransferQueue` is a priority-queued worker pool (`Transfer.WorkerCount`, default 2) that transfers queued files to NAS with retry + exponential backoff, and persists its state to `transfer_queue.json` so it survives a restart. -- `nas.go` / `pkg/nas`: `NASService` establishes the NAS connection (on Windows, via `net use` for UNC paths) and does the actual file copy + size verification. +- `nas.go` / `pkg/nas`: `NASService` attaches the NAS destination for whatever host it's running on and does the actual file copy + size verification. See [NAS connectivity](#nas-connectivity-windows-and-linux). - `cleanup.go`: `CleanupService` deletes local copies after a successful, verified transfer (`Cleanup.AfterTransfer`, default true). During a normal download run, this whole stage starts automatically in the same process as the downloader (see the note above) — you don't invoke transfer separately for a live recording. The standalone `-transfer` mode exists only to catch up files that a prior download session ended without fully transferring. @@ -83,8 +83,10 @@ All settings have sane defaults; override via environment variables as needed. | `WORKER_COUNT` | `4` | Concurrent segment downloads per variant | | `REFRESH_DELAY_SECONDS` | `3` | How often each variant's chunklist is polled | | `ENABLE_NAS_TRANSFER` | `true` | Whether to run the transfer service alongside download | -| `NAS_OUTPUT_PATH` | `\\HomeLabNAS\dci\streams` | UNC/NAS root path segments are copied to | -| `NAS_USERNAME` / `NAS_PASSWORD` | *(see note below)* | NAS auth credentials | +| `NAS_OUTPUT_PATH` | `\\HomeLabNAS\dci\streams` | NAS root segments are copied to. UNC (`\\server\share\dir`), `//server/share/dir`, `smb://server/share/dir`, or a local/mounted directory — see [NAS connectivity](#nas-connectivity-windows-and-linux) | +| `NAS_USERNAME` / `NAS_PASSWORD` | *(see note below)* | NAS auth credentials. Used by `net use` on Windows and by `NAS_AUTO_MOUNT` on Linux; ignored for an already-mounted path | +| `NAS_AUTO_MOUNT` | `false` | Linux only: mount the SMB share if the host hasn't. Requires root | +| `NAS_MOUNT_POINT` | *(derived)* | Where `NAS_AUTO_MOUNT` attaches the share (default `/mnt/streamrecorder//`) | | `LOCAL_OUTPUT_DIR` | `data` | Local base directory for downloaded segments | | `PROCESS_OUTPUT_DIR` | `out` | Output directory for the final concatenated `.mp4` | | `FFMPEG_PATH` | `ffmpeg` | Path to the FFmpeg binary used by the processing stage | @@ -95,6 +97,25 @@ All settings have sane defaults; override via environment variables as needed. > **Security note:** the default NAS credentials currently ship as hardcoded defaults in `pkg/config/config.go` for this home-lab setup. If you fork or redeploy this, set `NAS_USERNAME`/`NAS_PASSWORD` explicitly and rotate the shipped default — see `TODO.md` for the full flagged list of housekeeping items like this one. +## NAS connectivity (Windows and Linux) + +The same `NAS_OUTPUT_PATH` works on both platforms; only the mechanism for reaching it differs, and the recorder picks it at runtime. + +| Host | Destination | What happens | +|---|---|---| +| Windows | UNC / `//server/share` / `smb://` | `net use \\server\share` with `NAS_USERNAME`/`NAS_PASSWORD`, then the UNC path is used directly | +| Linux | UNC / `//server/share` / `smb://` | The share is looked up in `/proc/mounts` and the UNC path is rewritten onto the mount point (`\\nas\dci\streams` + `//nas/dci` mounted at `/mnt/dci` → `/mnt/dci/streams`). Set `NAS_AUTO_MOUNT=true` to mount it instead | +| Either | A local or already-mounted directory | Used as-is, no network step | + +Notes: + +- **Server and share names match case-insensitively**, and both `cifs`/`smb3` and `nfs`/`nfs4` mounts are recognised. If the share is mounted at a deeper level than configured (`//nas/dci/streams` at `/mnt/streams`), the deepest matching mount wins. +- **Nothing the host already set up is torn down.** A share found already mounted (or already connected via `net use`) is left alone at shutdown; only a mount or connection this process created is undone. +- **`NAS_AUTO_MOUNT` requires root**, so the usual Linux setup is to mount the share yourself via fstab/autofs/systemd and either point `NAS_OUTPUT_PATH` at the mount point or leave it as the UNC path and let it resolve. Credentials are passed to `mount.cifs` through a `0600` credentials file, never on the command line where the process table would expose the password. +- **A Windows-shaped path on Linux is rejected, not silently created.** Backslashes and drive letters are legal filename characters on Linux, so `C:\data` or an unresolved `\\nas\dci` would otherwise be created as a local directory named exactly that — transfers would report success, cleanup would delete the local originals, and nothing would reach the NAS. + +Run `-check` to see which strategy applies and where the destination resolved to. + ## Building ```bash diff --git a/cmd/check/check.go b/cmd/check/check.go index ef045bd..80d8348 100644 --- a/cmd/check/check.go +++ b/cmd/check/check.go @@ -7,6 +7,7 @@ package check import ( "context" "fmt" + "log" "m3u8-downloader/pkg/config" "m3u8-downloader/pkg/constants" "m3u8-downloader/pkg/httpClient" @@ -155,46 +156,51 @@ func checkMasterPlaylist(masterURL string) ([]string, error) { return details, nil } -// checkNAS runs the same connect/mkdir/write sequence the transfer service -// performs at startup, but without the log.Fatalf that path takes on failure. +// checkNAS runs the same attach/mkdir/write sequence the transfer service +// performs at startup, so the check exercises the real code path for this host +// — net use on Windows, mount resolution on Linux — rather than a simulation of +// it. It disconnects afterwards so the check leaves no mount behind. func checkNAS(cfg *config.Config) ([]string, error) { - svc := &nas.NASService{Config: nas.NASConfig{ + svc, err := nas.New(nas.NASConfig{ Path: cfg.NAS.OutputPath, Username: cfg.NAS.Username, Password: cfg.NAS.Password, Timeout: cfg.NAS.Timeout, RetryLimit: cfg.NAS.RetryLimit, VerifySize: true, - }} - - details := []string{"path: " + cfg.NAS.OutputPath} - if share := svc.ExtractNetworkPath(cfg.NAS.OutputPath); share != "" { - details = append(details, "share: "+share, "user: "+usernameLabel(cfg.NAS.Username)) - } else { - details = append(details, "local or already-mounted path (no network authentication)") + AutoMount: cfg.NAS.AutoMount, + MountPoint: cfg.NAS.MountPoint, + }) + if err != nil { + return []string{"path: " + cfg.NAS.OutputPath}, err } start := time.Now() result := <-probeAsync(cfg.NAS.Timeout, func() ([]string, error) { - if err := svc.EstablishConnection(); err != nil { - return nil, fmt.Errorf("failed to connect to share: %w", err) + ctx, cancel := context.WithTimeout(context.Background(), cfg.NAS.Timeout) + defer cancel() + + if err := svc.Connect(ctx); err != nil { + return svc.ConnectionInfo(), err } - if err := svc.EnsureDirectoryExists(cfg.NAS.OutputPath); err != nil { - return nil, fmt.Errorf("share reachable but target directory unusable: %w", err) - } - if err := svc.TestConnection(); err != nil { - return nil, fmt.Errorf("target directory is not writable: %w", err) - } - return []string{"connect, mkdir and write probe succeeded"}, nil + defer func() { + if err := svc.Disconnect(context.Background()); err != nil { + log.Printf("Warning: failed to release NAS connection after check: %v", err) + } + }() + + return append(svc.ConnectionInfo(), "attach, mkdir and write probe succeeded"), nil }) + details := result.details + if len(details) == 0 { + details = svc.ConnectionInfo() + } if result.err != nil { return details, result.err } - details = append(details, result.details...) - details = append(details, "completed in "+formatDuration(time.Since(start))) - return details, nil + return append(details, "completed in "+formatDuration(time.Since(start))), nil } func formatDuration(d time.Duration) string { @@ -276,10 +282,3 @@ func enabledLabel(enabled bool) string { } return "disabled" } - -func usernameLabel(username string) string { - if username == "" { - return "(none, using ambient credentials)" - } - return username -} diff --git a/cmd/check/check_test.go b/cmd/check/check_test.go index f795a3a..b9f8bda 100644 --- a/cmd/check/check_test.go +++ b/cmd/check/check_test.go @@ -154,7 +154,7 @@ func TestCheckNASLocalPath(t *testing.T) { } joined := joinDetails(details) - for _, want := range []string{"path: " + target, "write probe succeeded"} { + for _, want := range []string{"configured: " + target, "strategy: local", "write probe succeeded"} { if !strings.Contains(joined, want) { t.Errorf("details missing %q:\n%s", want, joined) } @@ -176,7 +176,7 @@ func TestCheckNASUnusableTarget(t *testing.T) { if err == nil { t.Fatal("expected an unusable NAS target to fail the check") } - if joined := joinDetails(details); !strings.Contains(joined, "path: ") { + if joined := joinDetails(details); !strings.Contains(joined, "configured: ") { t.Errorf("expected path detail on failure:\n%s", joined) } } diff --git a/cmd/downloader/download.go b/cmd/downloader/download.go index ac6fbf2..8deb831 100644 --- a/cmd/downloader/download.go +++ b/cmd/downloader/download.go @@ -71,7 +71,9 @@ func Download(masterURL string, eventName string, debug bool) { } } - manifestWriter := media.NewManifestWriter(eventName) + // One writer for the whole run: every variant records into it, and it is + // the one written to disk at the end. + manifest := media.NewManifestWriter(eventName) eventPath := cfg.GetEventPath(eventName) if err := utils.EnsureDir(eventPath); err != nil { @@ -79,7 +81,7 @@ func Download(masterURL string, eventName string, debug bool) { log.Fatalf("Failed to create event directory: %v", err) } - variants, err := media.GetAllVariants(masterURL, eventPath, manifestWriter) + variants, err := media.GetAllVariants(masterURL, eventPath) if err != nil { appLog.Error("failed to get variants", logging.KeyError, err.Error()) log.Fatalf("Failed to get variants: %v", err) @@ -88,8 +90,6 @@ func Download(masterURL string, eventName string, debug bool) { sem := make(chan struct{}, constants.WorkerCount*len(variants)) - manifest := media.NewManifestWriter(eventName) - for _, variant := range variants { // Debug mode only tracks one variant for easier debugging if debug { @@ -117,7 +117,7 @@ func Download(masterURL string, eventName string, debug bool) { log.Println("All Services shut down.") appLog.Info("all services shut down") - manifestWriter.WriteManifest() + manifest.WriteManifest() log.Println("Manifest written.") - appLog.Info("manifest written", logging.KeyEvent, eventName) + appLog.Info("manifest written", logging.KeyEvent, eventName, "segments", manifest.Len()) } diff --git a/pkg/config/config.go b/pkg/config/config.go index 621b41d..df7898c 100644 --- a/pkg/config/config.go +++ b/pkg/config/config.go @@ -32,11 +32,19 @@ type HTTPConfig struct { type NASConfig struct { EnableTransfer bool - OutputPath string - Username string - Password string - Timeout time.Duration - RetryLimit int + // OutputPath is the destination as configured: a UNC path, an smb:// URL, + // or a directory on this host. How it is reached depends on the platform; + // see pkg/nas. + OutputPath string + Username string + Password string + Timeout time.Duration + RetryLimit int + // AutoMount lets the recorder mount an SMB share itself when the host has + // not already mounted it. Linux only, and requires root. + AutoMount bool + // MountPoint overrides where AutoMount attaches the share. + MountPoint string } type ProcessingConfig struct { @@ -95,6 +103,8 @@ var defaultConfig = Config{ Password: "s3tkY6tzA&KN6M", Timeout: 30 * time.Second, RetryLimit: 3, + AutoMount: false, + MountPoint: "", }, Processing: ProcessingConfig{ Enabled: true, @@ -175,8 +185,25 @@ func (c *Config) loadFromEnvironment() error { c.NAS.EnableTransfer = val == "true" } + if val := os.Getenv("NAS_AUTO_MOUNT"); val != "" { + c.NAS.AutoMount = val == "true" + } + + if val := os.Getenv("NAS_MOUNT_POINT"); val != "" { + c.NAS.MountPoint = val + } + if val := os.Getenv("LOCAL_OUTPUT_DIR"); val != "" { c.Paths.LocalOutput = val + // The manifest describes the event directory and is documented as + // sitting beside it, so it follows the output directory. Without this + // it stays in the default data/ while the segments move elsewhere. + c.Paths.ManifestDir = val + } + + // Applied after LOCAL_OUTPUT_DIR so an explicit setting wins. + if val := os.Getenv("MANIFEST_DIR"); val != "" { + c.Paths.ManifestDir = val } if val := os.Getenv("PROCESS_OUTPUT_DIR"); val != "" { diff --git a/pkg/media/manifest.go b/pkg/media/manifest.go index ee8a6ff..8f1719c 100644 --- a/pkg/media/manifest.go +++ b/pkg/media/manifest.go @@ -8,12 +8,22 @@ import ( "m3u8-downloader/pkg/utils" "os" "sort" + "strconv" + "sync" ) type ManifestWriter struct { ManifestPath string Segments []ManifestItem - Index map[string]*ManifestItem + // Index maps a sequence number to its position in Segments. It holds + // positions rather than pointers because append reallocates the backing + // array, which would leave previously handed-out element pointers aimed at + // a stale copy. + Index map[string]int + + // mu guards the fields above: every variant downloads concurrently, and + // each reports its segments as they land. + mu sync.Mutex } type ManifestItem struct { @@ -26,40 +36,55 @@ func NewManifestWriter(eventName string) *ManifestWriter { return &ManifestWriter{ ManifestPath: cfg.GetManifestPath(eventName), Segments: make([]ManifestItem, 0), - Index: make(map[string]*ManifestItem), + Index: make(map[string]int), } } +// AddOrUpdateSegment records that a segment was downloaded, keeping the +// highest-resolution variant seen for each sequence number. func (m *ManifestWriter) AddOrUpdateSegment(seqNo string, resolution string) { + m.mu.Lock() + defer m.mu.Unlock() + if m.Index == nil { - m.Index = make(map[string]*ManifestItem) + m.Index = make(map[string]int) } if m.Segments == nil { m.Segments = make([]ManifestItem, 0) } - if existing, ok := m.Index[seqNo]; ok { - if resolution > existing.Resolution { - existing.Resolution = resolution + if pos, ok := m.Index[seqNo]; ok { + if ResolutionHeight(resolution) > ResolutionHeight(m.Segments[pos].Resolution) { + m.Segments[pos].Resolution = resolution } return - } else { - item := ManifestItem{ - SeqNo: seqNo, - Resolution: resolution, - } - m.Segments = append(m.Segments, item) - m.Index[seqNo] = &item } + + m.Segments = append(m.Segments, ManifestItem{SeqNo: seqNo, Resolution: resolution}) + m.Index[seqNo] = len(m.Segments) - 1 +} + +// Len reports how many distinct sequence numbers have been recorded. +func (m *ManifestWriter) Len() int { + m.mu.Lock() + defer m.mu.Unlock() + return len(m.Segments) } func (m *ManifestWriter) WriteManifest() { - sort.Slice(m.Segments, func(i, j int) bool { - return m.Segments[i].SeqNo < m.Segments[j].SeqNo + // Sort a copy: reordering Segments in place would invalidate Index, and a + // recording may still be writing into it. + m.mu.Lock() + segments := make([]ManifestItem, len(m.Segments)) + copy(segments, m.Segments) + m.mu.Unlock() + + sort.Slice(segments, func(i, j int) bool { + return lessSeqNo(segments[i].SeqNo, segments[j].SeqNo) }) - data, err := json.MarshalIndent(m.Segments, "", " ") + data, err := json.MarshalIndent(segments, "", " ") if err != nil { log.Printf("Failed to marshal manifest: %v", err) logging.MustGetFactory().Download.Error("failed to marshal manifest", logging.KeyError, err.Error()) @@ -88,3 +113,15 @@ func (m *ManifestWriter) WriteManifest() { return } } + +// lessSeqNo orders sequence numbers numerically, so segment 9 precedes segment +// 10 instead of sorting after it as the strings would. Values that are not +// numeric fall back to lexicographic order. +func lessSeqNo(a, b string) bool { + ai, aErr := strconv.ParseUint(a, 10, 64) + bi, bErr := strconv.ParseUint(b, 10, 64) + if aErr == nil && bErr == nil { + return ai < bi + } + return a < b +} diff --git a/pkg/media/manifest_test.go b/pkg/media/manifest_test.go index a4520db..7a8a29b 100644 --- a/pkg/media/manifest_test.go +++ b/pkg/media/manifest_test.go @@ -4,6 +4,8 @@ import ( "encoding/json" "os" "path/filepath" + "strconv" + "sync" "testing" ) @@ -41,7 +43,7 @@ func TestManifestWriter_AddOrUpdateSegment(t *testing.T) { writer := &ManifestWriter{ ManifestPath: "test.json", Segments: make([]ManifestItem, 0), - Index: make(map[string]*ManifestItem), + Index: make(map[string]int), } // Test adding new segment @@ -112,7 +114,7 @@ func TestManifestWriter_WriteManifest(t *testing.T) { writer := &ManifestWriter{ ManifestPath: manifestPath, Segments: make([]ManifestItem, 0), - Index: make(map[string]*ManifestItem), + Index: make(map[string]int), } // Add some test segments out of order @@ -172,7 +174,7 @@ func TestManifestWriter_WriteManifest_EmptySegments(t *testing.T) { writer := &ManifestWriter{ ManifestPath: manifestPath, Segments: make([]ManifestItem, 0), - Index: make(map[string]*ManifestItem), + Index: make(map[string]int), } // Write empty manifest @@ -204,7 +206,7 @@ func TestManifestWriter_WriteManifest_InvalidPath(t *testing.T) { writer := &ManifestWriter{ ManifestPath: "/invalid/path/that/does/not/exist/manifest.json", Segments: []ManifestItem{{SeqNo: "1001", Resolution: "1080p"}}, - Index: make(map[string]*ManifestItem), + Index: make(map[string]int), } // This should not panic, just fail gracefully @@ -239,3 +241,169 @@ func TestManifestItem_JSONSerialization(t *testing.T) { t.Errorf("Resolution mismatch: expected '%s', got '%s'", item.Resolution, unmarshaled.Resolution) } } + +func TestResolutionHeight(t *testing.T) { + cases := map[string]int{ + "1080p": 1080, + "720p": 720, + "2160p": 2160, + "1440p": 1440, + "240p": 240, + // Not a p label: must rank below every real resolution rather + // than accidentally outranking them. + "unknown": 0, + "": 0, + "p": 0, + } + + for label, want := range cases { + if got := ResolutionHeight(label); got != want { + t.Errorf("ResolutionHeight(%q) = %d, want %d", label, got, want) + } + } +} + +// TestAddOrUpdateSegmentKeepsHighestResolution pins the comparison used to +// choose between variants. Comparing the labels as strings — as this once did — +// ranks "720p" above "1080p", quietly downgrading the manifest. +func TestAddOrUpdateSegmentKeepsHighestResolution(t *testing.T) { + cases := []struct { + name string + arriving []string + want string + }{ + {"higher replaces lower", []string{"720p", "1080p"}, "1080p"}, + {"lower does not replace higher", []string{"1080p", "720p"}, "1080p"}, + {"fewer digits must not win", []string{"1080p", "480p"}, "1080p"}, + {"4k beats 1080p", []string{"1080p", "2160p"}, "2160p"}, + {"unknown loses to a real resolution", []string{"unknown", "480p"}, "480p"}, + {"unknown does not displace a real resolution", []string{"1080p", "unknown"}, "1080p"}, + {"repeated identical resolution", []string{"720p", "720p"}, "720p"}, + } + + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + w := &ManifestWriter{} + for _, res := range tc.arriving { + w.AddOrUpdateSegment("1001", res) + } + + if len(w.Segments) != 1 { + t.Fatalf("expected a single entry, got %d", len(w.Segments)) + } + if got := w.Segments[0].Resolution; got != tc.want { + t.Errorf("kept %q, want %q (arrival order %v)", got, tc.want, tc.arriving) + } + }) + } +} + +// TestAddOrUpdateSegmentUpdatesTheStoredEntry guards the aliasing bug: the +// index once held a pointer to a local copy rather than to the stored element, +// so updates were applied to an orphan and never reached the written manifest. +func TestAddOrUpdateSegmentUpdatesTheStoredEntry(t *testing.T) { + w := &ManifestWriter{} + + // Enough entries to force at least one append reallocation, which would + // invalidate any pointer held into the backing array. + for i := 0; i < 64; i++ { + w.AddOrUpdateSegment(strconv.Itoa(1000+i), "480p") + } + for i := 0; i < 64; i++ { + w.AddOrUpdateSegment(strconv.Itoa(1000+i), "1080p") + } + + if len(w.Segments) != 64 { + t.Fatalf("expected 64 entries, got %d", len(w.Segments)) + } + for _, seg := range w.Segments { + if seg.Resolution != "1080p" { + t.Fatalf("segment %s kept %q; the upgrade never reached the stored entry", seg.SeqNo, seg.Resolution) + } + } +} + +// TestAddOrUpdateSegmentIsConcurrencySafe reflects real use: every variant +// downloads in its own goroutine and reports segments as they land. +func TestAddOrUpdateSegmentIsConcurrencySafe(t *testing.T) { + w := &ManifestWriter{} + resolutions := []string{"240p", "480p", "720p", "1080p"} + + var wg sync.WaitGroup + for _, res := range resolutions { + wg.Add(1) + go func(res string) { + defer wg.Done() + for seq := 0; seq < 200; seq++ { + w.AddOrUpdateSegment(strconv.Itoa(seq), res) + } + }(res) + } + wg.Wait() + + if got := w.Len(); got != 200 { + t.Errorf("expected 200 distinct sequence numbers, got %d", got) + } + for _, seg := range w.Segments { + if seg.Resolution != "1080p" { + t.Errorf("segment %s = %q, want the highest variant 1080p", seg.SeqNo, seg.Resolution) + } + } +} + +// TestWriteManifestSortsNumerically checks that segment 9 precedes segment 10, +// which string ordering would reverse. +func TestWriteManifestSortsNumerically(t *testing.T) { + dir := t.TempDir() + w := &ManifestWriter{ManifestPath: filepath.Join(dir, "event.json")} + + for _, seq := range []string{"100", "9", "1000", "10", "1"} { + w.AddOrUpdateSegment(seq, "1080p") + } + w.WriteManifest() + + data, err := os.ReadFile(w.ManifestPath) + if err != nil { + t.Fatalf("failed to read manifest: %v", err) + } + + var items []ManifestItem + if err := json.Unmarshal(data, &items); err != nil { + t.Fatalf("manifest is not valid JSON: %v", err) + } + + want := []string{"1", "9", "10", "100", "1000"} + if len(items) != len(want) { + t.Fatalf("expected %d entries, got %d", len(want), len(items)) + } + for i, seq := range want { + if items[i].SeqNo != seq { + t.Errorf("position %d = %q, want %q (full order: %v)", i, items[i].SeqNo, seq, items) + } + } +} + +// TestWriteManifestLeavesIndexUsable ensures writing mid-recording does not +// corrupt the writer: sorting in place would desynchronise Index from Segments. +func TestWriteManifestLeavesIndexUsable(t *testing.T) { + dir := t.TempDir() + w := &ManifestWriter{ManifestPath: filepath.Join(dir, "event.json")} + + w.AddOrUpdateSegment("20", "480p") + w.AddOrUpdateSegment("10", "480p") + w.WriteManifest() + + // After the write, an upgrade must still land on the right entry. + w.AddOrUpdateSegment("20", "1080p") + + pos, ok := w.Index["20"] + if !ok { + t.Fatal("sequence 20 missing from the index after WriteManifest") + } + if got := w.Segments[pos].Resolution; got != "1080p" { + t.Errorf("sequence 20 = %q, want 1080p; the index no longer matches the slice", got) + } + if w.Segments[w.Index["10"]].SeqNo != "10" { + t.Error("index and slice disagree after WriteManifest") + } +} diff --git a/pkg/media/stream.go b/pkg/media/stream.go index 3851674..e5bd509 100644 --- a/pkg/media/stream.go +++ b/pkg/media/stream.go @@ -13,6 +13,7 @@ import ( "net/http" "net/url" "path" + "strconv" "strings" "time" ) @@ -24,7 +25,27 @@ type StreamVariant struct { ID int Resolution string OutputDir string - Writer *ManifestWriter +} + +// ResolutionHeight converts a label produced by ExtractResolution back into +// the pixel height it stands for, so variants can be ranked by quality. +// +// The height is parsed rather than looked up in a table: labels come from the +// playlist's own RESOLUTION attribute, so a stream can legitimately offer +// 1440p or 2160p that no fixed table would anticipate. Anything not in +// p form — "unknown", from a playlist with no variants — returns 0 and +// therefore ranks below every recognised resolution. +// +// Comparing the labels as strings instead is a trap worth naming: "720p" sorts +// after "1080p", so lexicographic comparison silently prefers lower quality. +func ResolutionHeight(resolution string) int { + trimmed := strings.TrimSuffix(strings.ToLower(strings.TrimSpace(resolution)), "p") + + height, err := strconv.Atoi(trimmed) + if err != nil || height < 0 { + return 0 + } + return height } // ExtractResolution maps a variant to the label used for its output directory, @@ -50,7 +71,10 @@ func ExtractResolution(variant *m3u8.Variant) string { } } -func GetAllVariants(masterURL string, outputDir string, writer *ManifestWriter) ([]*StreamVariant, error) { +// GetAllVariants fetches the master playlist and returns one entry per stream +// variant. Segments are recorded into the manifest by VariantDownloader, which +// receives it directly, so no writer is threaded through here. +func GetAllVariants(masterURL string, outputDir string) ([]*StreamVariant, error) { client := &http.Client{} req, _ := http.NewRequest("GET", masterURL, nil) req.Header.Set("User-Agent", constants.HTTPUserAgent) @@ -78,7 +102,6 @@ func GetAllVariants(masterURL string, outputDir string, writer *ManifestWriter) ID: 0, Resolution: "unknown", OutputDir: path.Join(outputDir, "unknown"), - Writer: writer, }}, nil } @@ -166,6 +189,9 @@ func VariantDownloader(ctx context.Context, variant *StreamVariant, sem chan str log.Printf("✓ %s downloaded segment %s", j.Variant.Resolution, name) dlLog.Debug("segment downloaded", logging.KeySegment, name) registry.RecordSegmentDownloaded(variant.Resolution) + if manifest != nil { + manifest.AddOrUpdateSegment(strconv.FormatUint(j.Seq, 10), variant.Resolution) + } return } diff --git a/pkg/nas/config.go b/pkg/nas/config.go index cc40ac9..aa01e47 100644 --- a/pkg/nas/config.go +++ b/pkg/nas/config.go @@ -3,10 +3,18 @@ package nas import "time" type NASConfig struct { + // Path is the destination as configured. It may be a UNC path, an + // smb:// URL, or a directory on this host; see ParseLocation. Path string Username string Password string Timeout time.Duration RetryLimit int VerifySize bool + // AutoMount allows the recorder to mount an SMB share itself when the + // host has not already mounted it. Linux only, and requires root, so it + // is off by default in favour of a clear "mount it first" error. + AutoMount bool + // MountPoint overrides where AutoMount attaches the share. + MountPoint string } diff --git a/pkg/nas/connector.go b/pkg/nas/connector.go new file mode 100644 index 0000000..5a6012d --- /dev/null +++ b/pkg/nas/connector.go @@ -0,0 +1,98 @@ +package nas + +import ( + "context" + "fmt" + "os/exec" + "runtime" +) + +// Connector attaches a configured Location to the host filesystem so that +// ordinary file operations reach the NAS. +// +// Windows and Linux disagree about what "connecting to a share" means — +// Windows authenticates a UNC path in place, Linux needs the share mounted +// somewhere in its single filesystem tree — so each host gets its own +// implementation and NASService stays platform-neutral. Selection happens at +// runtime rather than through build tags so that every implementation is +// compiled, vetted and unit-testable on every platform; only the final exec +// call is genuinely host-specific. +type Connector interface { + // Name identifies the strategy in logs and check output. + Name() string + // Attach makes loc reachable and reports where to write. + Attach(ctx context.Context, loc Location, cfg NASConfig) (Attachment, error) + // Detach undoes whatever Attach set up. Implementations must undo only + // what they created: a share this process found already mounted belongs + // to the host, and unmounting it could break unrelated software. + Detach(ctx context.Context, att Attachment) error +} + +// Attachment is the result of a successful Attach. +type Attachment struct { + // BasePath is the host-native directory that maps to the configured + // destination. Every file operation is resolved against it. + BasePath string + // Notes explain how the host reached the destination, for -check output + // and startup logs. + Notes []string + // Owned records that this process created the connection or mount, and + // so may tear it down on shutdown. + Owned bool + // mountPoint is the directory this process mounted, when Owned is set by + // the Linux connector. + mountPoint string + // share is the network path this process connected, when Owned is set by + // the Windows connector. + share string +} + +// commandRunner executes an external command. Implementations keep argument +// construction in a pure function and call through this, so tests can assert +// exactly what would run without running it. +type commandRunner func(ctx context.Context, name string, args ...string) ([]byte, error) + +// execRunner is the production runner: it combines stdout and stderr because +// mount and net both report the useful diagnosis on one or the other. +func execRunner(ctx context.Context, name string, args ...string) ([]byte, error) { + return exec.CommandContext(ctx, name, args...).CombinedOutput() +} + +// connectorFor picks the strategy for a destination on a given OS. A +// destination that is already an ordinary directory needs no network step on +// any platform, so it short-circuits before the OS switch. +func connectorFor(loc Location, goos string) Connector { + if !loc.IsRemote() { + return &localConnector{goos: goos} + } + + switch goos { + case "windows": + return &windowsConnector{run: execRunner} + case "linux": + return &linuxConnector{run: execRunner, mountsPath: procMountsPath} + default: + return &unsupportedConnector{goos: goos} + } +} + +// defaultConnector selects the strategy for the running host. +func defaultConnector(loc Location) Connector { + return connectorFor(loc, runtime.GOOS) +} + +// unsupportedConnector fails clearly on platforms this project does not target, +// rather than silently writing a share-shaped directory to local disk. +type unsupportedConnector struct { + goos string +} + +func (c *unsupportedConnector) Name() string { return "unsupported" } + +func (c *unsupportedConnector) Attach(_ context.Context, loc Location, _ NASConfig) (Attachment, error) { + return Attachment{}, fmt.Errorf( + "connecting to network share %s is not supported on %s; mount the share and set NAS_OUTPUT_PATH to the mount point", + loc.SMBSource(), c.goos) +} + +func (c *unsupportedConnector) Detach(context.Context, Attachment) error { return nil } diff --git a/pkg/nas/connector_local.go b/pkg/nas/connector_local.go new file mode 100644 index 0000000..6c8adf8 --- /dev/null +++ b/pkg/nas/connector_local.go @@ -0,0 +1,57 @@ +package nas + +import ( + "context" + "fmt" + "path/filepath" + "strings" +) + +// localConnector handles destinations that are already part of the host +// filesystem: a plain output directory, or — the normal arrangement on Linux — +// an SMB/NFS share mounted by fstab, autofs or systemd before the recorder +// starts. There is nothing to connect, but there is something to check. +type localConnector struct { + goos string +} + +func (c *localConnector) Name() string { return "local" } + +func (c *localConnector) Attach(_ context.Context, loc Location, _ NASConfig) (Attachment, error) { + if err := c.validate(loc.Raw); err != nil { + return Attachment{}, err + } + + return Attachment{ + BasePath: filepath.Clean(loc.Raw), + Notes: []string{"local or already-mounted path (no network authentication)"}, + }, nil +} + +func (c *localConnector) Detach(context.Context, Attachment) error { return nil } + +// validate rejects a Windows-shaped path on a non-Windows host. This looks +// pedantic and is not: backslashes and colons are legal characters in Linux +// filenames, so a stray C:\data or \\server\share would be created verbatim as +// a local directory. Transfers would report success, cleanup would delete the +// local originals, and nothing would ever reach the NAS. +func (c *localConnector) validate(path string) error { + if c.goos == "windows" { + return nil + } + + if driveLetter.MatchString(path) { + return fmt.Errorf( + "%q is a Windows drive path but this host is %s; set NAS_OUTPUT_PATH to a mount point such as /mnt/nas/streams", + path, c.goos) + } + + if strings.Contains(path, `\`) { + return fmt.Errorf( + "%q contains backslashes but this host is %s, where they are ordinary filename characters rather than separators; "+ + "set NAS_OUTPUT_PATH to a mount point such as /mnt/nas/streams, or to \\\\server\\share\\folder to have the share resolved automatically", + path, c.goos) + } + + return nil +} diff --git a/pkg/nas/connector_mount.go b/pkg/nas/connector_mount.go new file mode 100644 index 0000000..c36dfa7 --- /dev/null +++ b/pkg/nas/connector_mount.go @@ -0,0 +1,178 @@ +package nas + +import ( + "context" + "fmt" + "os" + "path/filepath" + "strings" +) + +// linuxConnector reaches a share the way Linux does: through a mount point in +// the single filesystem tree. There is no per-path authentication, so the +// question is not "can we connect" but "where is this share mounted". +// +// The default and by far the safest answer is that the host already mounted it +// (fstab, autofs, systemd, or `docker run -v`), so the connector looks the +// share up in the mount table. Mounting it here is opt-in because it requires +// root and changes system state that outlives a failed run. +type linuxConnector struct { + run commandRunner + mountsPath string + // geteuid is indirected for tests; production uses os.Geteuid. + geteuid func() int +} + +func (c *linuxConnector) Name() string { return "mount" } + +func (c *linuxConnector) Attach(ctx context.Context, loc Location, cfg NASConfig) (Attachment, error) { + entries, err := readMountTable(c.mountsPath) + if err != nil { + return Attachment{}, err + } + + if point, rest, ok := findMount(entries, loc); ok { + return Attachment{ + BasePath: filepath.Join(append([]string{point}, rest...)...), + Notes: []string{ + "share: " + loc.SMBSource(), + fmt.Sprintf("resolved via existing mount at %s", point), + }, + }, nil + } + + if !cfg.AutoMount { + return Attachment{}, fmt.Errorf("%s", notMountedMessage(loc, cfg)) + } + + return c.mount(ctx, loc, cfg) +} + +func (c *linuxConnector) Detach(ctx context.Context, att Attachment) error { + // Never unmount a share the host had already mounted; other software may + // be using it, and we did not put it there. + if !att.Owned || att.mountPoint == "" { + return nil + } + + if out, err := c.run(ctx, "umount", att.mountPoint); err != nil { + return fmt.Errorf("failed to unmount %s: %v", att.mountPoint, summarize(err, out)) + } + return nil +} + +// mount performs an opt-in mount.cifs of the share. +func (c *linuxConnector) mount(ctx context.Context, loc Location, cfg NASConfig) (Attachment, error) { + geteuid := c.geteuid + if geteuid == nil { + geteuid = os.Geteuid + } + if geteuid() != 0 { + return Attachment{}, fmt.Errorf( + "NAS_AUTO_MOUNT is set but mounting %s requires root; run as root, or mount the share beforehand and %s", + loc.SMBSource(), suggestMountPointConfig(loc)) + } + + point := autoMountPoint(loc, cfg) + if err := os.MkdirAll(point, 0755); err != nil { + return Attachment{}, fmt.Errorf("failed to create mount point %s: %w", point, err) + } + + credentialsPath, cleanup, err := writeCredentialsFile(cfg) + if err != nil { + return Attachment{}, err + } + defer cleanup() + + if cfg.Timeout > 0 { + var cancel context.CancelFunc + ctx, cancel = context.WithTimeout(ctx, cfg.Timeout) + defer cancel() + } + + if out, err := c.run(ctx, "mount", mountArgs(loc, point, credentialsPath)...); err != nil { + return Attachment{}, fmt.Errorf("failed to mount %s at %s: %v", loc.SMBSource(), point, summarize(err, out)) + } + + return Attachment{ + BasePath: filepath.Join(append([]string{point}, loc.Sub...)...), + Owned: true, + mountPoint: point, + Notes: []string{ + "share: " + loc.SMBSource(), + "user: " + usernameNote(cfg.Username), + fmt.Sprintf("mounted at %s by this process (will unmount on shutdown)", point), + }, + }, nil +} + +// mountArgs builds the mount.cifs invocation. Kept pure so it can be asserted +// without mounting anything. +func mountArgs(loc Location, mountPoint, credentialsPath string) []string { + options := "rw" + if credentialsPath != "" { + options += ",credentials=" + credentialsPath + } else { + options += ",guest" + } + return []string{"-t", "cifs", loc.SMBSource(), mountPoint, "-o", options} +} + +// writeCredentialsFile stages the NAS credentials in a 0600 file. mount.cifs +// also accepts user=/pass= options, but those land in the process table where +// any local user can read the password; a credentials file does not. +func writeCredentialsFile(cfg NASConfig) (path string, cleanup func(), err error) { + if cfg.Username == "" || cfg.Password == "" { + return "", func() {}, nil + } + + dir, err := os.MkdirTemp("", "streamrecorder-cifs-") + if err != nil { + return "", func() {}, fmt.Errorf("failed to stage NAS credentials: %w", err) + } + cleanup = func() { os.RemoveAll(dir) } + + path = filepath.Join(dir, "credentials") + content := fmt.Sprintf("username=%s\npassword=%s\n", cfg.Username, cfg.Password) + if err := os.WriteFile(path, []byte(content), 0600); err != nil { + cleanup() + return "", func() {}, fmt.Errorf("failed to write NAS credentials file: %w", err) + } + + return path, cleanup, nil +} + +// autoMountPoint chooses where to mount the share when the operator has not +// named a location. Everything lands under one predictable directory so a +// leftover mount is easy to find and clean up. +func autoMountPoint(loc Location, cfg NASConfig) string { + if cfg.MountPoint != "" { + return cfg.MountPoint + } + return filepath.Join("/mnt", "streamrecorder", strings.ToLower(loc.Server), strings.ToLower(loc.Share)) +} + +// notMountedMessage explains the failure in terms of what the operator should +// actually type. A bare "no such mount" would leave them guessing. +func notMountedMessage(loc Location, cfg NASConfig) string { + source := loc.SMBSource() + example := filepath.Join("/mnt", strings.ToLower(loc.Share)) + + credentials := "" + if cfg.Username != "" { + credentials = fmt.Sprintf(",username=%s,password=***", cfg.Username) + } + + return fmt.Sprintf( + "%s is not mounted on this host. Either mount it and %s, for example:\n"+ + " sudo mkdir -p %s && sudo mount -t cifs %s %s -o rw%s\n"+ + "or set NAS_AUTO_MOUNT=true to have the recorder mount it (requires root)", + source, suggestMountPointConfig(loc), example, source, example, credentials) +} + +func suggestMountPointConfig(loc Location) string { + if len(loc.Sub) == 0 { + return "set NAS_OUTPUT_PATH to the mount point" + } + return fmt.Sprintf("set NAS_OUTPUT_PATH to the mount point plus %s", loc.SubPath()) +} diff --git a/pkg/nas/connector_netuse.go b/pkg/nas/connector_netuse.go new file mode 100644 index 0000000..1453c32 --- /dev/null +++ b/pkg/nas/connector_netuse.go @@ -0,0 +1,107 @@ +package nas + +import ( + "context" + "fmt" + "path/filepath" + "strings" +) + +// windowsConnector reaches a share the way Windows does: authenticate the UNC +// path with `net use`, then use the UNC path directly as a filesystem path. +// Nothing is mounted into a tree, so the resolved base path is the configured +// UNC path itself. +type windowsConnector struct { + run commandRunner +} + +func (c *windowsConnector) Name() string { return "net use" } + +func (c *windowsConnector) Attach(ctx context.Context, loc Location, cfg NASConfig) (Attachment, error) { + share := loc.ShareUNC() + att := Attachment{ + BasePath: loc.UNC(), + share: share, + Notes: []string{"share: " + share, "user: " + usernameNote(cfg.Username)}, + } + + if cfg.Timeout > 0 { + var cancel context.CancelFunc + ctx, cancel = context.WithTimeout(ctx, cfg.Timeout) + defer cancel() + } + + out, err := c.run(ctx, "net", netUseArgs(loc, cfg)...) + if err != nil { + // A failure here is not decisive. `net use` also fails when the share + // is already connected — commonly because Explorer or another process + // got there first — in which case the path is perfectly usable. Record + // the reason and let the caller's write probe deliver the verdict, + // rather than refusing to run against a working share. + att.Notes = append(att.Notes, + fmt.Sprintf("net use did not complete (%v); continuing, the share may already be connected", summarize(err, out))) + return att, nil + } + + att.Owned = true + att.Notes = append(att.Notes, "connected via net use") + return att, nil +} + +func (c *windowsConnector) Detach(ctx context.Context, att Attachment) error { + // Only drop connections this process made. Deleting a mapping the user or + // another application established would break them. + if !att.Owned || att.share == "" { + return nil + } + + if out, err := c.run(ctx, "net", "use", att.share, "/delete"); err != nil { + return fmt.Errorf("failed to disconnect %s: %v", att.share, summarize(err, out)) + } + return nil +} + +// netUseArgs builds the `net use` invocation. Kept pure so the Windows code +// path can be asserted from any host, including the Linux box this is +// developed on, without executing anything. +func netUseArgs(loc Location, cfg NASConfig) []string { + args := []string{"use", loc.ShareUNC()} + if cfg.Username != "" && cfg.Password != "" { + args = append(args, "/user:"+cfg.Username, cfg.Password) + } + return append(args, "/persistent:no") +} + +// summarize pairs an exec error with the first line of command output, which is +// where net and mount put the actual diagnosis. +func summarize(err error, out []byte) string { + text := strings.TrimSpace(string(out)) + if text == "" { + return err.Error() + } + if idx := strings.IndexByte(text, '\n'); idx >= 0 { + text = text[:idx] + } + return fmt.Sprintf("%v: %s", err, text) +} + +func usernameNote(username string) string { + if username == "" { + return "(none, using ambient credentials)" + } + return username +} + +// joinBase resolves a relative path against a base that may be a UNC path. +// filepath.Join is correct on Windows and would mangle the leading \\ anywhere +// else, so UNC bases are joined with backslashes explicitly. +func joinBase(base string, elem ...string) string { + if len(elem) == 0 { + return base + } + if strings.HasPrefix(base, `\\`) { + parts := append([]string{strings.TrimRight(base, `\`)}, elem...) + return strings.Join(parts, `\`) + } + return filepath.Join(append([]string{base}, elem...)...) +} diff --git a/pkg/nas/connector_test.go b/pkg/nas/connector_test.go new file mode 100644 index 0000000..334c18f --- /dev/null +++ b/pkg/nas/connector_test.go @@ -0,0 +1,353 @@ +package nas + +import ( + "context" + "fmt" + "os" + "path/filepath" + "strings" + "testing" + "time" +) + +// recordingRunner captures the commands a connector would run instead of +// running them. This is what lets the Windows path be tested from a Linux +// development machine: the decision logic and the exact argv are asserted +// without ever invoking net.exe. +type recordingRunner struct { + calls [][]string + output []byte + err error +} + +func (r *recordingRunner) run(_ context.Context, name string, args ...string) ([]byte, error) { + r.calls = append(r.calls, append([]string{name}, args...)) + return r.output, r.err +} + +func (r *recordingRunner) lastCall() string { + if len(r.calls) == 0 { + return "" + } + return strings.Join(r.calls[len(r.calls)-1], " ") +} + +func remoteLocation(t *testing.T, raw string) Location { + t.Helper() + loc, err := ParseLocation(raw) + if err != nil { + t.Fatalf("ParseLocation(%q) failed: %v", raw, err) + } + return loc +} + +func TestConnectorSelection(t *testing.T) { + remote := remoteLocation(t, `\\nas\dci\streams`) + local := remoteLocation(t, "/mnt/dci/streams") + + cases := []struct { + name string + loc Location + goos string + want string + }{ + {"windows share", remote, "windows", "net use"}, + {"linux share", remote, "linux", "mount"}, + {"darwin share", remote, "darwin", "unsupported"}, + {"local path on windows", local, "windows", "local"}, + {"local path on linux", local, "linux", "local"}, + // A already-mounted path needs no network step even where shares are + // unsupported, so it must not fall through to the error connector. + {"local path on darwin", local, "darwin", "local"}, + } + + for _, tc := range cases { + if got := connectorFor(tc.loc, tc.goos).Name(); got != tc.want { + t.Errorf("%s: connectorFor = %q, want %q", tc.name, got, tc.want) + } + } +} + +func TestWindowsConnectorAttach(t *testing.T) { + runner := &recordingRunner{} + c := &windowsConnector{run: runner.run} + loc := remoteLocation(t, `\\HomeLabNAS\dci\streams`) + + att, err := c.Attach(context.Background(), loc, NASConfig{Username: "NASAdmin", Password: "secret"}) + if err != nil { + t.Fatalf("Attach failed: %v", err) + } + + // The base path is the full UNC path; net use authenticates only the share. + if want := `\\HomeLabNAS\dci\streams`; att.BasePath != want { + t.Errorf("BasePath = %q, want %q", att.BasePath, want) + } + if want := `net use \\HomeLabNAS\dci /user:NASAdmin secret /persistent:no`; runner.lastCall() != want { + t.Errorf("command = %q, want %q", runner.lastCall(), want) + } + if !att.Owned { + t.Error("a successful net use should be owned so it can be released on shutdown") + } +} + +func TestWindowsConnectorOmitsCredentialsWhenUnset(t *testing.T) { + runner := &recordingRunner{} + c := &windowsConnector{run: runner.run} + + if _, err := c.Attach(context.Background(), remoteLocation(t, `\\nas\dci`), NASConfig{}); err != nil { + t.Fatalf("Attach failed: %v", err) + } + + if want := `net use \\nas\dci /persistent:no`; runner.lastCall() != want { + t.Errorf("command = %q, want %q", runner.lastCall(), want) + } +} + +func TestWindowsConnectorContinuesWhenAlreadyConnected(t *testing.T) { + // net use fails when a connection already exists, but the share is usable; + // refusing to run would be wrong. The write probe is the real verdict. + runner := &recordingRunner{ + err: fmt.Errorf("exit status 2"), + output: []byte("The network connection could not be found.\nMore help is available..."), + } + c := &windowsConnector{run: runner.run} + + att, err := c.Attach(context.Background(), remoteLocation(t, `\\nas\dci`), NASConfig{}) + if err != nil { + t.Fatalf("Attach should tolerate a net use failure, got: %v", err) + } + if att.Owned { + t.Error("a failed net use must not be marked owned, or Detach would drop someone else's connection") + } + + notes := strings.Join(att.Notes, "\n") + if !strings.Contains(notes, "net use did not complete") { + t.Errorf("expected the failure to be recorded in notes:\n%s", notes) + } + // Only the first line of output is worth surfacing. + if strings.Contains(notes, "More help is available") { + t.Errorf("notes should not carry the full usage dump:\n%s", notes) + } +} + +func TestWindowsConnectorDetachOnlyReleasesOwned(t *testing.T) { + runner := &recordingRunner{} + c := &windowsConnector{run: runner.run} + + if err := c.Detach(context.Background(), Attachment{share: `\\nas\dci`, Owned: false}); err != nil { + t.Fatalf("Detach failed: %v", err) + } + if len(runner.calls) != 0 { + t.Errorf("must not disconnect a share this process did not connect, ran: %v", runner.calls) + } + + if err := c.Detach(context.Background(), Attachment{share: `\\nas\dci`, Owned: true}); err != nil { + t.Fatalf("Detach failed: %v", err) + } + if want := `net use \\nas\dci /delete`; runner.lastCall() != want { + t.Errorf("command = %q, want %q", runner.lastCall(), want) + } +} + +// writeMountTable stages a /proc/mounts-format file. +func writeMountTable(t *testing.T, lines ...string) string { + t.Helper() + path := filepath.Join(t.TempDir(), "mounts") + body := strings.Join(lines, "\n") + "\n" + if err := os.WriteFile(path, []byte(body), 0644); err != nil { + t.Fatalf("failed to write mount table: %v", err) + } + return path +} + +func TestLinuxConnectorResolvesExistingMount(t *testing.T) { + mounts := writeMountTable(t, + "sysfs /sys sysfs rw,nosuid 0 0", + "//homelabnas/dci /mnt/dci cifs rw,relatime,username=nasadmin 0 0", + ) + c := &linuxConnector{run: (&recordingRunner{}).run, mountsPath: mounts} + + att, err := c.Attach(context.Background(), remoteLocation(t, `\\HomeLabNAS\dci\streams`), NASConfig{}) + if err != nil { + t.Fatalf("Attach failed: %v", err) + } + + // Server and share names are case-insensitive on SMB, so the configured + // \\HomeLabNAS\dci must match a //homelabnas/dci mount. + if want := "/mnt/dci/streams"; att.BasePath != want { + t.Errorf("BasePath = %q, want %q", att.BasePath, want) + } + if att.Owned { + t.Error("a pre-existing mount must not be owned, or shutdown would unmount it") + } +} + +func TestLinuxConnectorPrefersDeepestMount(t *testing.T) { + // Both mounts could serve the destination; the more specific one is the + // only one guaranteed to actually contain it. + mounts := writeMountTable(t, + "//nas/dci /mnt/dci cifs rw 0 0", + "//nas/dci/streams /mnt/streams cifs rw 0 0", + ) + c := &linuxConnector{run: (&recordingRunner{}).run, mountsPath: mounts} + + att, err := c.Attach(context.Background(), remoteLocation(t, `\\nas\dci\streams\2026`), NASConfig{}) + if err != nil { + t.Fatalf("Attach failed: %v", err) + } + if want := "/mnt/streams/2026"; att.BasePath != want { + t.Errorf("BasePath = %q, want %q", att.BasePath, want) + } +} + +func TestLinuxConnectorMatchesNFSAndEscapedPoints(t *testing.T) { + mounts := writeMountTable(t, + `nas:/dci /mnt/my\040share nfs4 rw 0 0`, + ) + c := &linuxConnector{run: (&recordingRunner{}).run, mountsPath: mounts} + + att, err := c.Attach(context.Background(), remoteLocation(t, `\\nas\dci\streams`), NASConfig{}) + if err != nil { + t.Fatalf("Attach failed: %v", err) + } + if want := "/mnt/my share/streams"; att.BasePath != want { + t.Errorf("BasePath = %q, want %q", att.BasePath, want) + } +} + +func TestLinuxConnectorIgnoresNonNetworkMounts(t *testing.T) { + // A local directory that merely looks like the share must not match. + mounts := writeMountTable(t, "//nas/dci /mnt/dci ext4 rw 0 0") + c := &linuxConnector{run: (&recordingRunner{}).run, mountsPath: mounts} + + if _, err := c.Attach(context.Background(), remoteLocation(t, `\\nas\dci`), NASConfig{}); err == nil { + t.Fatal("expected an ext4 mount not to satisfy a network share") + } +} + +func TestLinuxConnectorUnmountedGivesActionableError(t *testing.T) { + mounts := writeMountTable(t, "sysfs /sys sysfs rw 0 0") + c := &linuxConnector{run: (&recordingRunner{}).run, mountsPath: mounts} + + _, err := c.Attach(context.Background(), remoteLocation(t, `\\nas\dci\streams`), NASConfig{Username: "admin"}) + if err == nil { + t.Fatal("expected an unmounted share to fail") + } + + msg := err.Error() + for _, want := range []string{"//nas/dci", "mount -t cifs", "NAS_AUTO_MOUNT", "NAS_OUTPUT_PATH"} { + if !strings.Contains(msg, want) { + t.Errorf("error should mention %q:\n%s", want, msg) + } + } + // The remedy must not echo the real password. + if strings.Contains(msg, "hunter2") { + t.Error("error message leaked a credential") + } +} + +func TestLinuxConnectorAutoMountRequiresRoot(t *testing.T) { + mounts := writeMountTable(t, "sysfs /sys sysfs rw 0 0") + runner := &recordingRunner{} + c := &linuxConnector{run: runner.run, mountsPath: mounts, geteuid: func() int { return 1000 }} + + _, err := c.Attach(context.Background(), remoteLocation(t, `\\nas\dci`), NASConfig{AutoMount: true}) + if err == nil || !strings.Contains(err.Error(), "requires root") { + t.Fatalf("expected a root requirement error, got: %v", err) + } + if len(runner.calls) != 0 { + t.Errorf("must not attempt mount without privileges, ran: %v", runner.calls) + } +} + +func TestLinuxConnectorAutoMount(t *testing.T) { + mounts := writeMountTable(t, "sysfs /sys sysfs rw 0 0") + runner := &recordingRunner{} + point := filepath.Join(t.TempDir(), "mnt", "dci") + c := &linuxConnector{run: runner.run, mountsPath: mounts, geteuid: func() int { return 0 }} + + cfg := NASConfig{AutoMount: true, MountPoint: point, Username: "admin", Password: "hunter2", Timeout: 5 * time.Second} + att, err := c.Attach(context.Background(), remoteLocation(t, `\\nas\dci\streams`), cfg) + if err != nil { + t.Fatalf("Attach failed: %v", err) + } + + if want := filepath.Join(point, "streams"); att.BasePath != want { + t.Errorf("BasePath = %q, want %q", att.BasePath, want) + } + if !att.Owned || att.mountPoint != point { + t.Errorf("a self-created mount must be owned and recorded, got owned=%v point=%q", att.Owned, att.mountPoint) + } + + call := runner.lastCall() + if !strings.HasPrefix(call, "mount -t cifs //nas/dci "+point+" -o rw,credentials=") { + t.Errorf("unexpected mount command: %q", call) + } + // The password must reach mount.cifs through a credentials file, never + // through argv where any local user could read it from the process table. + if strings.Contains(call, "hunter2") { + t.Errorf("password leaked into the mount command line: %q", call) + } + // The staged credentials file must not outlive the mount call. + credPath := call[strings.Index(call, "credentials=")+len("credentials="):] + if _, err := os.Stat(credPath); !os.IsNotExist(err) { + t.Errorf("credentials file %s should have been removed, stat err = %v", credPath, err) + } + + if err := c.Detach(context.Background(), att); err != nil { + t.Fatalf("Detach failed: %v", err) + } + if want := "umount " + point; runner.lastCall() != want { + t.Errorf("detach command = %q, want %q", runner.lastCall(), want) + } +} + +func TestLinuxConnectorDetachLeavesForeignMounts(t *testing.T) { + runner := &recordingRunner{} + c := &linuxConnector{run: runner.run, mountsPath: writeMountTable(t)} + + if err := c.Detach(context.Background(), Attachment{BasePath: "/mnt/dci", Owned: false}); err != nil { + t.Fatalf("Detach failed: %v", err) + } + if len(runner.calls) != 0 { + t.Errorf("must not unmount a share the host mounted, ran: %v", runner.calls) + } +} + +func TestLocalConnectorRejectsWindowsPathsOffWindows(t *testing.T) { + c := &localConnector{goos: "linux"} + + for _, raw := range []string{`C:\data\streams`, `data\streams`} { + loc, err := ParseLocation(raw) + if err != nil { + t.Fatalf("ParseLocation(%q) failed: %v", raw, err) + } + if _, err := c.Attach(context.Background(), loc, NASConfig{}); err == nil { + t.Errorf("%q should be rejected on Linux: backslashes are legal filename characters there, "+ + "so it would silently create a local directory instead of writing to the NAS", raw) + } + } +} + +func TestLocalConnectorAcceptsWindowsPathsOnWindows(t *testing.T) { + c := &localConnector{goos: "windows"} + loc, err := ParseLocation(`C:\data\streams`) + if err != nil { + t.Fatalf("ParseLocation failed: %v", err) + } + + if _, err := c.Attach(context.Background(), loc, NASConfig{}); err != nil { + t.Errorf("a drive path is valid on Windows: %v", err) + } +} + +func TestUnsupportedConnectorExplainsItself(t *testing.T) { + c := &unsupportedConnector{goos: "plan9"} + _, err := c.Attach(context.Background(), remoteLocation(t, `\\nas\dci`), NASConfig{}) + if err == nil { + t.Fatal("expected an error on an unsupported platform") + } + if !strings.Contains(err.Error(), "plan9") || !strings.Contains(err.Error(), "NAS_OUTPUT_PATH") { + t.Errorf("error should name the platform and the remedy: %v", err) + } +} diff --git a/pkg/nas/location.go b/pkg/nas/location.go new file mode 100644 index 0000000..0dd9f74 --- /dev/null +++ b/pkg/nas/location.go @@ -0,0 +1,144 @@ +package nas + +import ( + "fmt" + "path/filepath" + "regexp" + "strings" +) + +// Location is a configured NAS destination parsed into a host-neutral form. +// +// The same destination is spelled differently depending on where you're +// standing: Windows wants \\HomeLabNAS\dci\streams, a Linux host reaches the +// same bytes through a mount point like /mnt/dci/streams, and either host may +// simply be handed a plain local directory. Parsing happens once, up front, +// and platform-specific rendering is left to the Connector implementations. +type Location struct { + // Raw is the destination exactly as configured, for error messages. + Raw string + // Server is the host serving the share; empty for a plain local path. + Server string + // Share is the top-level export name; empty for a plain local path. + Share string + // Sub holds the path components below the share, if any. + Sub []string +} + +// driveLetter matches a Windows drive-qualified path such as C:\data. +var driveLetter = regexp.MustCompile(`^[A-Za-z]:[\\/]`) + +// ParseLocation accepts every spelling of a destination this project supports: +// Windows UNC (\\server\share\dir), the SMB source form (//server/share/dir), +// a smb:// URL, or a plain local/mounted directory. It does no I/O and makes no +// assumptions about the host OS, so the result is safe to compute anywhere. +func ParseLocation(raw string) (Location, error) { + trimmed := strings.TrimSpace(raw) + if trimmed == "" { + return Location{}, fmt.Errorf("NAS path is empty") + } + + loc := Location{Raw: trimmed} + + remainder, isRemote := stripRemotePrefix(trimmed) + if !isRemote { + return loc, nil + } + + parts := splitPathComponents(remainder) + if len(parts) < 2 { + return Location{}, fmt.Errorf( + "%q names a server but no share; expected a form like \\\\server\\share\\folder", trimmed) + } + + loc.Server = parts[0] + loc.Share = parts[1] + loc.Sub = parts[2:] + return loc, nil +} + +// stripRemotePrefix reports whether raw names a network share and returns the +// server/share/subpath portion when it does. A leading // is treated as a share +// rather than a local path: POSIX leaves paths beginning with exactly two +// slashes implementation-defined, and nobody configures a NAS destination that +// way by accident. +func stripRemotePrefix(raw string) (string, bool) { + lower := strings.ToLower(raw) + switch { + case strings.HasPrefix(lower, "smb://"): + return raw[len("smb://"):], true + case strings.HasPrefix(lower, "cifs://"): + return raw[len("cifs://"):], true + case strings.HasPrefix(raw, `\\`): + return raw[2:], true + case strings.HasPrefix(raw, "//"): + return raw[2:], true + default: + return raw, false + } +} + +// splitPathComponents splits on either separator and drops empty components, so +// that \\server\share\\folder\ and //server/share/folder agree. +func splitPathComponents(path string) []string { + fields := strings.FieldsFunc(path, func(r rune) bool { return r == '/' || r == '\\' }) + out := make([]string, 0, len(fields)) + for _, f := range fields { + if f != "" { + out = append(out, f) + } + } + return out +} + +// IsRemote reports whether reaching this destination requires attaching a +// network share first, as opposed to writing to a directory that is already +// part of the local filesystem. +func (l Location) IsRemote() bool { + return l.Server != "" && l.Share != "" +} + +// UNC renders the destination in Windows form: \\server\share\sub. +func (l Location) UNC() string { + if !l.IsRemote() { + return l.Raw + } + return `\\` + strings.Join(append([]string{l.Server, l.Share}, l.Sub...), `\`) +} + +// ShareUNC renders just the share in Windows form: \\server\share. This is the +// unit `net use` authenticates against; the subpath below it is an ordinary +// directory on the connected share. +func (l Location) ShareUNC() string { + if !l.IsRemote() { + return "" + } + return `\\` + l.Server + `\` + l.Share +} + +// SMBSource renders the share in the //server/share form used by mount tables +// and by mount.cifs. It deliberately omits Sub: that part is a path *within* +// the mounted share, not part of the share's identity. +func (l Location) SMBSource() string { + if !l.IsRemote() { + return "" + } + return "//" + l.Server + "/" + l.Share +} + +// SubPath renders the components below the share using the host's separator, or +// "" when the destination is the share root. +func (l Location) SubPath() string { + if len(l.Sub) == 0 { + return "" + } + return filepath.Join(l.Sub...) +} + +// String describes the destination for logs and check output. +func (l Location) String() string { + if l.IsRemote() { + return l.UNC() + } + return l.Raw +} diff --git a/pkg/nas/location_test.go b/pkg/nas/location_test.go new file mode 100644 index 0000000..1b958d7 --- /dev/null +++ b/pkg/nas/location_test.go @@ -0,0 +1,123 @@ +package nas + +import ( + "strings" + "testing" +) + +func TestParseLocationRemoteForms(t *testing.T) { + // Every spelling of the same destination must parse identically, so that + // an operator can configure it the way their platform's documentation + // describes without changing behaviour. + equivalent := []string{ + `\\HomeLabNAS\dci\streams`, + `//HomeLabNAS/dci/streams`, + `smb://HomeLabNAS/dci/streams`, + `SMB://HomeLabNAS/dci/streams`, + `cifs://HomeLabNAS/dci/streams`, + `\\HomeLabNAS\dci\streams\`, + `\\HomeLabNAS\\dci\\streams`, + } + + for _, raw := range equivalent { + loc, err := ParseLocation(raw) + if err != nil { + t.Errorf("ParseLocation(%q) failed: %v", raw, err) + continue + } + if !loc.IsRemote() { + t.Errorf("ParseLocation(%q) should be remote", raw) + } + if loc.Server != "HomeLabNAS" || loc.Share != "dci" { + t.Errorf("ParseLocation(%q) = server %q share %q, want HomeLabNAS/dci", raw, loc.Server, loc.Share) + } + if len(loc.Sub) != 1 || loc.Sub[0] != "streams" { + t.Errorf("ParseLocation(%q) sub = %v, want [streams]", raw, loc.Sub) + } + } +} + +func TestParseLocationRendering(t *testing.T) { + loc, err := ParseLocation(`//HomeLabNAS/dci/streams/2026`) + if err != nil { + t.Fatalf("unexpected error: %v", err) + } + + if got, want := loc.UNC(), `\\HomeLabNAS\dci\streams\2026`; got != want { + t.Errorf("UNC() = %q, want %q", got, want) + } + if got, want := loc.ShareUNC(), `\\HomeLabNAS\dci`; got != want { + t.Errorf("ShareUNC() = %q, want %q", got, want) + } + if got, want := loc.SMBSource(), "//HomeLabNAS/dci"; got != want { + t.Errorf("SMBSource() = %q, want %q", got, want) + } +} + +func TestParseLocationShareRoot(t *testing.T) { + loc, err := ParseLocation(`\\nas\media`) + if err != nil { + t.Fatalf("unexpected error: %v", err) + } + if len(loc.Sub) != 0 { + t.Errorf("expected no subpath, got %v", loc.Sub) + } + if got, want := loc.UNC(), `\\nas\media`; got != want { + t.Errorf("UNC() = %q, want %q", got, want) + } +} + +func TestParseLocationLocalPaths(t *testing.T) { + for _, raw := range []string{"/mnt/nas/streams", "data/streams", `C:\data\streams`, "/mnt"} { + loc, err := ParseLocation(raw) + if err != nil { + t.Errorf("ParseLocation(%q) failed: %v", raw, err) + continue + } + if loc.IsRemote() { + t.Errorf("ParseLocation(%q) should not be treated as a network share", raw) + } + if loc.Raw != raw { + t.Errorf("ParseLocation(%q).Raw = %q", raw, loc.Raw) + } + } +} + +func TestParseLocationRejectsIncomplete(t *testing.T) { + for _, raw := range []string{"", " ", `\\`, `\\server`, "//server", "smb://server"} { + if _, err := ParseLocation(raw); err == nil { + t.Errorf("ParseLocation(%q) should have failed", raw) + } + } +} + +func TestParseLocationTrimsWhitespace(t *testing.T) { + loc, err := ParseLocation(" \\\\nas\\dci ") + if err != nil { + t.Fatalf("unexpected error: %v", err) + } + if loc.Server != "nas" || loc.Share != "dci" { + t.Errorf("whitespace not trimmed: server %q share %q", loc.Server, loc.Share) + } +} + +func TestJoinBaseKeepsUNCSeparators(t *testing.T) { + // filepath.Join would produce \\nas\dci/event on Linux, which is not a + // path Windows would accept if the value were ever written to state. + got := joinBase(`\\nas\dci`, "event", "1080p") + if want := `\\nas\dci\event\1080p`; got != want { + t.Errorf("joinBase UNC = %q, want %q", got, want) + } + + if got, want := joinBase(`\\nas\dci\`, "event"), `\\nas\dci\event`; got != want { + t.Errorf("joinBase trailing separator = %q, want %q", got, want) + } + + if got := joinBase("/mnt/dci"); got != "/mnt/dci" { + t.Errorf("joinBase with no elements = %q", got) + } + + if got, want := joinBase("/mnt/dci", "event"), "/mnt/dci/event"; !strings.HasSuffix(got, want) { + t.Errorf("joinBase local = %q, want suffix %q", got, want) + } +} diff --git a/pkg/nas/mounts.go b/pkg/nas/mounts.go new file mode 100644 index 0000000..327993a --- /dev/null +++ b/pkg/nas/mounts.go @@ -0,0 +1,135 @@ +package nas + +import ( + "bufio" + "fmt" + "os" + "strconv" + "strings" +) + +// procMountsPath is the kernel's view of what is mounted in this process's +// mount namespace, which is the right question to ask inside a container too. +const procMountsPath = "/proc/mounts" + +// mountEntry is one line of the mount table. +type mountEntry struct { + Source string + Point string + FSType string +} + +// networkFilesystems are the mount types that can plausibly be serving a NAS +// share. Restricting the search avoids matching, say, a local directory that +// merely happens to be named like a server. +var networkFilesystems = map[string]bool{ + "cifs": true, + "smb3": true, + "smbfs": true, + "nfs": true, + "nfs4": true, + "fuse.smbnetfs": true, +} + +// readMountTable parses /proc/mounts (or any file in its format). Fields are +// whitespace-separated: source, mount point, type, options, dump, pass. +func readMountTable(path string) ([]mountEntry, error) { + f, err := os.Open(path) + if err != nil { + return nil, fmt.Errorf("failed to read mount table %s: %w", path, err) + } + defer f.Close() + + var entries []mountEntry + scanner := bufio.NewScanner(f) + for scanner.Scan() { + fields := strings.Fields(scanner.Text()) + if len(fields) < 3 { + continue + } + entries = append(entries, mountEntry{ + Source: unescapeMountField(fields[0]), + Point: unescapeMountField(fields[1]), + FSType: fields[2], + }) + } + if err := scanner.Err(); err != nil { + return nil, fmt.Errorf("failed to scan mount table %s: %w", path, err) + } + + return entries, nil +} + +// unescapeMountField decodes the octal escapes the kernel uses for characters +// that would otherwise break the whitespace-separated format — most often \040 +// for a space in a mount point. +func unescapeMountField(field string) string { + if !strings.Contains(field, `\`) { + return field + } + + var b strings.Builder + for i := 0; i < len(field); i++ { + if field[i] == '\\' && i+3 < len(field) { + if v, err := strconv.ParseUint(field[i+1:i+4], 8, 8); err == nil { + b.WriteByte(byte(v)) + i += 3 + continue + } + } + b.WriteByte(field[i]) + } + return b.String() +} + +// findMount locates the mount serving loc and returns its mount point plus the +// path components still to be appended below it. +// +// The search runs deepest-first because a share may be mounted at any level: +// //nas/dci might be mounted at /mnt/dci, or //nas/dci/streams mounted directly +// at /mnt/streams. Preferring the longest matching source keeps both working +// and avoids pointing at a parent that the mount does not actually cover. +func findMount(entries []mountEntry, loc Location) (point string, rest []string, ok bool) { + if !loc.IsRemote() { + return "", nil, false + } + + for depth := len(loc.Sub); depth >= 0; depth-- { + for _, candidate := range mountSourceForms(loc, depth) { + for _, entry := range entries { + if !networkFilesystems[entry.FSType] { + continue + } + if normalizeMountSource(entry.Source) == candidate { + return entry.Point, loc.Sub[depth:], true + } + } + } + } + + return "", nil, false +} + +// mountSourceForms renders the share, plus its first `depth` subdirectories, in +// the spellings a mount table might use for it. +func mountSourceForms(loc Location, depth int) []string { + segments := append([]string{loc.Share}, loc.Sub[:depth]...) + tail := strings.Join(segments, "/") + + return []string{ + normalizeMountSource("//" + loc.Server + "/" + tail), // cifs/smb3 + normalizeMountSource(loc.Server + ":/" + tail), // nfs + } +} + +// normalizeMountSource makes sources comparable: SMB servers and shares are +// case-insensitive, separators may be either slash, and a trailing slash is not +// meaningful. +func normalizeMountSource(source string) string { + s := strings.ReplaceAll(source, `\`, "/") + s = strings.ToLower(strings.TrimSpace(s)) + if len(s) > 1 { + s = strings.TrimRight(s, "/") + } + return s +} diff --git a/pkg/nas/mounts_test.go b/pkg/nas/mounts_test.go new file mode 100644 index 0000000..39e70af --- /dev/null +++ b/pkg/nas/mounts_test.go @@ -0,0 +1,96 @@ +package nas + +import ( + "os" + "testing" +) + +func TestUnescapeMountField(t *testing.T) { + cases := map[string]string{ + `/mnt/plain`: "/mnt/plain", + `/mnt/my\040share`: "/mnt/my share", + `/mnt/tab\011here`: "/mnt/tab\there", + `/mnt/back\134slash`: `/mnt/back\slash`, + `//nas/dci`: "//nas/dci", + `/mnt/not\9octal`: `/mnt/not\9octal`, + `/mnt/trailing\`: `/mnt/trailing\`, + } + + for in, want := range cases { + if got := unescapeMountField(in); got != want { + t.Errorf("unescapeMountField(%q) = %q, want %q", in, got, want) + } + } +} + +func TestNormalizeMountSource(t *testing.T) { + cases := map[string]string{ + "//NAS/DCI": "//nas/dci", + "//nas/dci/": "//nas/dci", + `\\nas\dci`: "//nas/dci", + " //nas/dci ": "//nas/dci", + "/": "/", + } + + for in, want := range cases { + if got := normalizeMountSource(in); got != want { + t.Errorf("normalizeMountSource(%q) = %q, want %q", in, got, want) + } + } +} + +func TestReadMountTableSkipsMalformedLines(t *testing.T) { + path := writeMountTable(t, + "source-only", + "source /point-but-no-type", + "", + "//nas/dci /mnt/dci cifs rw 0 0", + ) + + entries, err := readMountTable(path) + if err != nil { + t.Fatalf("readMountTable failed: %v", err) + } + if len(entries) != 1 { + t.Fatalf("expected 1 usable entry, got %d: %v", len(entries), entries) + } + if entries[0].Source != "//nas/dci" || entries[0].Point != "/mnt/dci" || entries[0].FSType != "cifs" { + t.Errorf("unexpected entry: %+v", entries[0]) + } +} + +func TestReadMountTableMissingFile(t *testing.T) { + if _, err := readMountTable("/definitely/not/a/mount/table"); err == nil { + t.Error("expected an error for a missing mount table") + } +} + +// TestReadRealMountTable closes the loop between the fixtures above and the +// kernel's actual output: the parser must cope with whatever this host has +// mounted, not just hand-written examples. +func TestReadRealMountTable(t *testing.T) { + if _, err := os.Stat(procMountsPath); err != nil { + t.Skipf("%s is not available on this platform", procMountsPath) + } + + entries, err := readMountTable(procMountsPath) + if err != nil { + t.Fatalf("failed to parse the host's %s: %v", procMountsPath, err) + } + if len(entries) == 0 { + t.Fatalf("%s parsed to zero entries, which cannot be right", procMountsPath) + } + + foundRoot := false + for _, e := range entries { + if e.Point == "" || e.FSType == "" { + t.Errorf("entry parsed with empty fields: %+v", e) + } + if e.Point == "/" { + foundRoot = true + } + } + if !foundRoot { + t.Error("expected the root filesystem in the host's mount table") + } +} diff --git a/pkg/nas/nas.go b/pkg/nas/nas.go index 9a59663..02a5ca1 100644 --- a/pkg/nas/nas.go +++ b/pkg/nas/nas.go @@ -6,31 +6,168 @@ import ( "io" "log" "os" - "os/exec" - "path/filepath" - "strings" + "sync" ) +// NASService writes to a NAS destination without caring how this host reaches +// it. Parsing the destination and attaching it are delegated to a Location and +// a Connector; everything below operates on the resolved base path, which is +// always an ordinary path this host's filesystem understands. type NASService struct { - Config NASConfig + Config NASConfig + + location Location + connector Connector + + mu sync.RWMutex + attached Attachment connected bool } -func NewNASService(config NASConfig) *NASService { - nt := &NASService{ - Config: config, - } - - // Establish network connection with credentials before accessing the path - if err := nt.EstablishConnection(); err != nil { - log.Fatalf("Failed to establish network connection to %s: %v", nt.Config.Path, err) - } - - err := nt.EnsureDirectoryExists(nt.Config.Path) +// New parses the configured destination and selects a connection strategy for +// the running host. It performs no I/O and does not touch the network, so it is +// safe to call before deciding whether a transfer will actually happen; call +// Connect to attach the destination. +func New(config NASConfig) (*NASService, error) { + loc, err := ParseLocation(config.Path) if err != nil { - log.Fatalf("Failed to create directory %s: %v", nt.Config.Path, err) + return nil, err } - return nt + + return &NASService{ + Config: config, + location: loc, + connector: defaultConnector(loc), + }, nil +} + +// Connect attaches the destination, ensures the target directory exists, and +// proves it is writable. The write probe matters: on both platforms a share can +// be reachable and still reject writes, and finding that out at startup beats +// discovering it mid-recording. +func (nt *NASService) Connect(ctx context.Context) error { + att, err := nt.connector.Attach(ctx, nt.location, nt.Config) + if err != nil { + return err + } + + nt.mu.Lock() + nt.attached = att + nt.mu.Unlock() + + // From here on the attachment exists, so any failure has to undo it — + // otherwise a mount this process created would outlive the failed startup + // and be left behind on the host. + if err := nt.EnsureDirectoryExists(att.BasePath); err != nil { + nt.abandon(ctx, att) + return fmt.Errorf("%s is reachable but %s is unusable: %w", nt.location, att.BasePath, err) + } + + if err := nt.writeProbe(att.BasePath); err != nil { + nt.abandon(ctx, att) + return err + } + + nt.mu.Lock() + nt.connected = true + nt.mu.Unlock() + + log.Printf("Connected to NAS at %s", att.BasePath) + return nil +} + +// abandon releases a half-finished attachment after Connect fails. The +// original error is the one worth reporting, so a teardown failure is only +// logged. +func (nt *NASService) abandon(ctx context.Context, att Attachment) { + // Keep the notes: they explain how far the host got, which is exactly what + // the caller needs to report. Everything else is cleared so the service + // cannot be mistaken for usable and Detach cannot run twice. + nt.mu.Lock() + nt.attached = Attachment{Notes: att.Notes} + nt.connected = false + nt.mu.Unlock() + + if err := nt.connector.Detach(ctx, att); err != nil { + log.Printf("Warning: failed to release NAS attachment after a failed connect: %v", err) + } +} + +// writeProbe creates and removes a file to confirm the destination accepts +// writes. +func (nt *NASService) writeProbe(basePath string) error { + testFile := joinBase(basePath, ".connection_test") + + f, err := os.Create(testFile) + if err != nil { + return fmt.Errorf("%s is not writable: %w", basePath, err) + } + f.Close() + + if err := os.Remove(testFile); err != nil { + log.Printf("Warning: failed to remove connection test file %s: %v", testFile, err) + } + return nil +} + +// Disconnect releases only what Connect established. A share that was already +// mounted or already connected when the recorder started is left exactly as it +// was found. +func (nt *NASService) Disconnect(ctx context.Context) error { + nt.mu.Lock() + att := nt.attached + nt.attached = Attachment{} + nt.connected = false + nt.mu.Unlock() + + if err := nt.connector.Detach(ctx, att); err != nil { + return err + } + + if att.Owned { + log.Printf("Released NAS connection to %s", nt.location) + } + return nil +} + +// LocalPath is the resolved base directory on this host, valid after Connect. +func (nt *NASService) LocalPath() string { + nt.mu.RLock() + defer nt.mu.RUnlock() + return nt.attached.BasePath +} + +// Join resolves elements against the NAS base path using whatever separator +// this host's base path requires. +func (nt *NASService) Join(elem ...string) string { + return joinBase(nt.LocalPath(), elem...) +} + +// Location returns the parsed destination. +func (nt *NASService) Location() Location { return nt.location } + +// ConnectionInfo describes how this host reaches the destination, for check +// output and startup logs. Safe to call before Connect, when it reports only +// what is known from configuration. +func (nt *NASService) ConnectionInfo() []string { + nt.mu.RLock() + defer nt.mu.RUnlock() + + info := []string{ + "configured: " + nt.location.String(), + "strategy: " + nt.connector.Name(), + } + info = append(info, nt.attached.Notes...) + if nt.attached.BasePath != "" && nt.attached.BasePath != nt.location.Raw { + info = append(info, "resolved path: "+nt.attached.BasePath) + } + return info +} + +func (nt *NASService) IsConnected() bool { + nt.mu.RLock() + defer nt.mu.RUnlock() + return nt.connected } func (nt *NASService) CopyFile(ctx context.Context, srcPath, destPath string) error { @@ -89,87 +226,10 @@ func (nt *NASService) EnsureDirectoryExists(path string) error { return nil } -func (nt *NASService) EstablishConnection() error { - networkPath := nt.ExtractNetworkPath(nt.Config.Path) - if networkPath == "" { - return nil // local path, no network mount needed - } - - log.Printf("Establishing network connection to %s with user %s", networkPath, nt.Config.Username) - - var cmd *exec.Cmd - if nt.Config.Username != "" && nt.Config.Password != "" { - cmd = exec.Command("net", "use", networkPath, "/user:"+nt.Config.Username, nt.Config.Password, "/persistent:no") - } else { - cmd = exec.Command("net", "use", networkPath, "/persistent:no") - } - - output, err := cmd.CombinedOutput() - if err != nil { - return fmt.Errorf("failed to establish network connection: %w\nOutput: %s", err, string(output)) - } - - log.Printf("Network connection established successfully") - return nil -} - -func (nt *NASService) ExtractNetworkPath(fullPath string) string { - // Extract \\server\share from paths like \\server\share\folder\subfolder - if !strings.HasPrefix(fullPath, "\\\\") { - return "" // Not a UNC path - } - - parts := strings.Split(fullPath[2:], "\\") // Remove leading \\ - if len(parts) < 2 { - return "" // Invalid UNC path - } - - return "\\\\" + parts[0] + "\\" + parts[1] -} - -func (nt *NASService) TestConnection() error { - testFile := filepath.Join(nt.Config.Path, ".connection_test") - - f, err := os.Create(testFile) - if err != nil { - return fmt.Errorf("Failed to create test file: %w", err) - } - f.Close() - - os.Remove(testFile) - - nt.connected = true - log.Printf("Connected to NAS at %s", nt.Config.Path) - return nil -} - -func (nt *NASService) IsConnected() bool { - return nt.connected -} - -// Disconnect removes the network connection -func (nt *NASService) Disconnect() error { - networkPath := nt.ExtractNetworkPath(nt.Config.Path) - if networkPath == "" { - return nil // Local path, nothing to disconnect - } - - cmd := exec.Command("net", "use", networkPath, "/delete") - output, err := cmd.CombinedOutput() - if err != nil { - log.Printf("Warning: failed to disconnect from %s: %v\nOutput: %s", networkPath, err, string(output)) - // Don't return error since this is cleanup - } else { - log.Printf("Disconnected from network path: %s", networkPath) - } - - nt.connected = false - return nil -} - -// FileExists checks if a file already exists on the NAS and optionally verifies size +// FileExists checks whether a file already exists on the NAS, optionally +// verifying its size. destinationPath is relative to the NAS base path. func (nt *NASService) FileExists(destinationPath string, expectedSize int64) (bool, error) { - fullDestPath := filepath.Join(nt.Config.Path, destinationPath) + fullDestPath := nt.Join(destinationPath) destInfo, err := os.Stat(fullDestPath) if err != nil { @@ -189,9 +249,9 @@ func (nt *NASService) FileExists(destinationPath string, expectedSize int64) (bo return true, nil } -// GetFileSize returns the size of a file on the NAS +// GetFileSize returns the size of a file on the NAS, relative to the base path. func (nt *NASService) GetFileSize(destinationPath string) (int64, error) { - fullDestPath := filepath.Join(nt.Config.Path, destinationPath) + fullDestPath := nt.Join(destinationPath) destInfo, err := os.Stat(fullDestPath) if err != nil { diff --git a/pkg/nas/nas_test.go b/pkg/nas/nas_test.go new file mode 100644 index 0000000..b9c7de1 --- /dev/null +++ b/pkg/nas/nas_test.go @@ -0,0 +1,308 @@ +package nas + +import ( + "context" + "fmt" + "os" + "path/filepath" + "strings" + "testing" +) + +// fakeConnector stands in for a platform strategy so the service's own +// behaviour — probing, path resolution, teardown — can be tested without a NAS. +type fakeConnector struct { + basePath string + attachErr error + owned bool + attachCount int + detachCount int + detached Attachment +} + +func (f *fakeConnector) Name() string { return "fake" } + +func (f *fakeConnector) Attach(context.Context, Location, NASConfig) (Attachment, error) { + f.attachCount++ + if f.attachErr != nil { + return Attachment{}, f.attachErr + } + return Attachment{BasePath: f.basePath, Owned: f.owned, Notes: []string{"fake attachment"}}, nil +} + +func (f *fakeConnector) Detach(_ context.Context, att Attachment) error { + f.detachCount++ + f.detached = att + return nil +} + +func newTestService(t *testing.T, configuredPath string, conn Connector) *NASService { + t.Helper() + svc, err := New(NASConfig{Path: configuredPath}) + if err != nil { + t.Fatalf("New(%q) failed: %v", configuredPath, err) + } + svc.connector = conn + return svc +} + +func TestNewRejectsUnusableDestination(t *testing.T) { + if _, err := New(NASConfig{Path: ""}); err == nil { + t.Error("an empty NAS path should be rejected") + } + if _, err := New(NASConfig{Path: `\\server`}); err == nil { + t.Error("a server without a share should be rejected") + } +} + +func TestNewDoesNoIO(t *testing.T) { + // Constructing the service must not touch the network or filesystem, so + // that modes which never transfer can build one harmlessly. + svc, err := New(NASConfig{Path: `\\nas\dci\streams`}) + if err != nil { + t.Fatalf("New failed: %v", err) + } + if svc.IsConnected() { + t.Error("service should not report connected before Connect") + } + if svc.LocalPath() != "" { + t.Errorf("no path should be resolved before Connect, got %q", svc.LocalPath()) + } +} + +func TestConnectResolvesAndProbes(t *testing.T) { + base := filepath.Join(t.TempDir(), "streams") + conn := &fakeConnector{basePath: base} + svc := newTestService(t, `\\nas\dci\streams`, conn) + + if err := svc.Connect(context.Background()); err != nil { + t.Fatalf("Connect failed: %v", err) + } + + if !svc.IsConnected() { + t.Error("service should report connected") + } + if svc.LocalPath() != base { + t.Errorf("LocalPath = %q, want %q", svc.LocalPath(), base) + } + // Connect creates the destination directory if it is missing. + if info, err := os.Stat(base); err != nil || !info.IsDir() { + t.Errorf("Connect should have created %s: %v", base, err) + } + // The write probe must clean up after itself. + if _, err := os.Stat(filepath.Join(base, ".connection_test")); !os.IsNotExist(err) { + t.Errorf("probe file should have been removed, stat err = %v", err) + } +} + +func TestConnectSurfacesAttachFailure(t *testing.T) { + conn := &fakeConnector{attachErr: fmt.Errorf("share is not mounted")} + svc := newTestService(t, `\\nas\dci`, conn) + + err := svc.Connect(context.Background()) + if err == nil || !strings.Contains(err.Error(), "not mounted") { + t.Fatalf("expected the connector's error to surface, got: %v", err) + } + if svc.IsConnected() { + t.Error("a failed attach must not leave the service marked connected") + } +} + +func TestConnectFailsWhenDestinationUnwritable(t *testing.T) { + // A file where the directory should be: reachable, but unusable. + blocker := filepath.Join(t.TempDir(), "blocker") + if err := os.WriteFile(blocker, []byte("x"), 0644); err != nil { + t.Fatalf("setup failed: %v", err) + } + + svc := newTestService(t, "/tmp/whatever", &fakeConnector{basePath: filepath.Join(blocker, "streams")}) + if err := svc.Connect(context.Background()); err == nil { + t.Fatal("expected an unusable destination to fail the connect") + } + if svc.IsConnected() { + t.Error("service should not report connected after a failed probe") + } +} + +func TestFailedProbeReleasesTheAttachment(t *testing.T) { + // A mount created by Attach must not outlive a failed Connect, or a failed + // startup leaves a mount behind on the host. + blocker := filepath.Join(t.TempDir(), "blocker") + if err := os.WriteFile(blocker, []byte("x"), 0644); err != nil { + t.Fatalf("setup failed: %v", err) + } + + conn := &fakeConnector{basePath: filepath.Join(blocker, "streams"), owned: true} + svc := newTestService(t, `\\nas\dci`, conn) + + if err := svc.Connect(context.Background()); err == nil { + t.Fatal("expected Connect to fail") + } + + if conn.detachCount != 1 { + t.Errorf("Detach called %d times after a failed connect, want 1", conn.detachCount) + } + if !conn.detached.Owned { + t.Error("Detach should have been told the attachment was owned") + } + if svc.LocalPath() != "" { + t.Errorf("a released attachment should expose no path, got %q", svc.LocalPath()) + } + // The diagnosis of how far it got must survive for the caller to report. + if info := strings.Join(svc.ConnectionInfo(), "\n"); !strings.Contains(info, "fake attachment") { + t.Errorf("connection notes should survive a failed connect:\n%s", info) + } + + // A later Disconnect must not detach a second time. + if err := svc.Disconnect(context.Background()); err != nil { + t.Fatalf("Disconnect failed: %v", err) + } + if conn.detachCount != 2 || conn.detached.Owned { + t.Errorf("Disconnect after release should be a no-op teardown, detachCount=%d owned=%v", + conn.detachCount, conn.detached.Owned) + } +} + +func TestJoinUsesResolvedPath(t *testing.T) { + base := t.TempDir() + svc := newTestService(t, `\\nas\dci\streams`, &fakeConnector{basePath: base}) + if err := svc.Connect(context.Background()); err != nil { + t.Fatalf("Connect failed: %v", err) + } + + // Crucially the join is against the resolved path, not the configured UNC + // path — that is the whole point of the indirection. + if got, want := svc.Join("event", "1080p"), filepath.Join(base, "event", "1080p"); got != want { + t.Errorf("Join = %q, want %q", got, want) + } +} + +func TestFileExistsAndGetFileSizeUseResolvedPath(t *testing.T) { + base := t.TempDir() + svc := newTestService(t, `\\nas\dci`, &fakeConnector{basePath: base}) + if err := svc.Connect(context.Background()); err != nil { + t.Fatalf("Connect failed: %v", err) + } + + rel := filepath.Join("event", "seg.ts") + if err := os.MkdirAll(filepath.Join(base, "event"), 0755); err != nil { + t.Fatalf("setup failed: %v", err) + } + if err := os.WriteFile(filepath.Join(base, rel), []byte("0123456789"), 0644); err != nil { + t.Fatalf("setup failed: %v", err) + } + + exists, err := svc.FileExists(rel, 10) + if err != nil || !exists { + t.Errorf("FileExists = %v, %v; want true, nil", exists, err) + } + + // A size mismatch means a partial transfer, which should not count. + exists, err = svc.FileExists(rel, 99) + if err != nil || exists { + t.Errorf("FileExists with wrong size = %v, %v; want false, nil", exists, err) + } + + exists, err = svc.FileExists(filepath.Join("event", "missing.ts"), 0) + if err != nil || exists { + t.Errorf("FileExists for missing file = %v, %v; want false, nil", exists, err) + } + + size, err := svc.GetFileSize(rel) + if err != nil || size != 10 { + t.Errorf("GetFileSize = %d, %v; want 10, nil", size, err) + } +} + +func TestDisconnectDelegatesAndResets(t *testing.T) { + conn := &fakeConnector{basePath: t.TempDir(), owned: true} + svc := newTestService(t, `\\nas\dci`, conn) + if err := svc.Connect(context.Background()); err != nil { + t.Fatalf("Connect failed: %v", err) + } + + if err := svc.Disconnect(context.Background()); err != nil { + t.Fatalf("Disconnect failed: %v", err) + } + + if conn.detachCount != 1 { + t.Errorf("Detach called %d times, want 1", conn.detachCount) + } + if !conn.detached.Owned { + t.Error("Detach should receive the attachment so it knows what it owns") + } + if svc.IsConnected() { + t.Error("service should not report connected after Disconnect") + } +} + +func TestConnectionInfoDescribesResolution(t *testing.T) { + base := t.TempDir() + svc := newTestService(t, `\\nas\dci\streams`, &fakeConnector{basePath: base}) + + // Before connecting, only configuration is known. + before := strings.Join(svc.ConnectionInfo(), "\n") + if !strings.Contains(before, `\\nas\dci\streams`) || !strings.Contains(before, "fake") { + t.Errorf("pre-connect info should name the destination and strategy:\n%s", before) + } + + if err := svc.Connect(context.Background()); err != nil { + t.Fatalf("Connect failed: %v", err) + } + + after := strings.Join(svc.ConnectionInfo(), "\n") + if !strings.Contains(after, "resolved path: "+base) { + t.Errorf("post-connect info should report where the destination resolved to:\n%s", after) + } +} + +func TestVerifyTransferDetectsSizeMismatch(t *testing.T) { + dir := t.TempDir() + src := filepath.Join(dir, "src.ts") + dst := filepath.Join(dir, "dst.ts") + if err := os.WriteFile(src, []byte("0123456789"), 0644); err != nil { + t.Fatalf("setup failed: %v", err) + } + if err := os.WriteFile(dst, []byte("012"), 0644); err != nil { + t.Fatalf("setup failed: %v", err) + } + + svc := newTestService(t, dir, &fakeConnector{basePath: dir}) + if err := svc.VerifyTransfer(src, dst); err == nil { + t.Error("a truncated destination should fail verification") + } + + if err := os.WriteFile(dst, []byte("0123456789"), 0644); err != nil { + t.Fatalf("setup failed: %v", err) + } + if err := svc.VerifyTransfer(src, dst); err != nil { + t.Errorf("matching sizes should verify: %v", err) + } +} + +func TestCopyFile(t *testing.T) { + dir := t.TempDir() + src := filepath.Join(dir, "src.ts") + dst := filepath.Join(dir, "dst.ts") + payload := []byte("segment payload") + if err := os.WriteFile(src, payload, 0644); err != nil { + t.Fatalf("setup failed: %v", err) + } + + svc := newTestService(t, dir, &fakeConnector{basePath: dir}) + if err := svc.CopyFile(context.Background(), src, dst); err != nil { + t.Fatalf("CopyFile failed: %v", err) + } + + got, err := os.ReadFile(dst) + if err != nil { + t.Fatalf("failed to read copy: %v", err) + } + if string(got) != string(payload) { + t.Errorf("copied contents = %q, want %q", got, payload) + } + + if err := svc.CopyFile(context.Background(), filepath.Join(dir, "missing.ts"), dst); err == nil { + t.Error("copying a missing source should fail") + } +} diff --git a/pkg/processing/service.go b/pkg/processing/service.go index c55ae40..52d8107 100644 --- a/pkg/processing/service.go +++ b/pkg/processing/service.go @@ -39,11 +39,19 @@ func NewProcessingService(eventName string, cfg *config.Config) (*ProcessingServ Timeout: cfg.NAS.Timeout, RetryLimit: cfg.NAS.RetryLimit, VerifySize: true, + AutoMount: cfg.NAS.AutoMount, + MountPoint: cfg.NAS.MountPoint, } - nasService := nas.NewNASService(nasConfig) + nasService, err := nas.New(nasConfig) + if err != nil { + logging.MustGetFactory().Processing.Error("invalid NAS destination", logging.KeyError, err.Error()) + return nil, fmt.Errorf("invalid NAS destination: %w", err) + } - if err := nasService.TestConnection(); err != nil { + connectCtx, cancel := context.WithTimeout(context.Background(), cfg.NAS.Timeout) + defer cancel() + if err := nasService.Connect(connectCtx); err != nil { logging.MustGetFactory().Processing.Error("failed to connect to NAS", logging.KeyError, err.Error()) return nil, fmt.Errorf("failed to connect to NAS: %w", err) } @@ -59,9 +67,17 @@ func (ps *ProcessingService) GetSegmentInfo() (map[int]string, error) { return nil, nil } +// nasEventPath is the event's directory on the NAS as reached from this host. +// It goes through the NAS service rather than config.GetNASEventPath because +// the configured path may be a UNC path that this host reaches via a mount +// point instead. +func (ps *ProcessingService) nasEventPath() string { + return ps.nas.Join(ps.eventName) +} + func (ps *ProcessingService) GetEventDirs() ([]string, error) { if ps.eventName == "" { - sourcePath := ps.config.NAS.OutputPath + sourcePath := ps.nas.LocalPath() dirs, err := os.ReadDir(sourcePath) if err != nil { return nil, fmt.Errorf("failed to read directory %s: %w", sourcePath, err) @@ -177,7 +193,7 @@ func (ps *ProcessingService) Start(ctx context.Context) error { } func (ps *ProcessingService) GetResolutions() ([]string, error) { - eventPath := ps.config.GetNASEventPath(ps.eventName) + eventPath := ps.nasEventPath() dirs, err := os.ReadDir(eventPath) if err != nil { return nil, fmt.Errorf("failed to read source directory %s: %w", eventPath, err) @@ -198,7 +214,7 @@ func (ps *ProcessingService) GetResolutions() ([]string, error) { func (ps *ProcessingService) ParseResolutionDirectory(resolution string, ch chan<- SegmentInfo, wg *sync.WaitGroup) { defer wg.Done() - resolutionPath := utils.SafeJoin(ps.config.GetNASEventPath(ps.eventName), resolution) + resolutionPath := utils.SafeJoin(ps.nasEventPath(), resolution) files, err := os.ReadDir(resolutionPath) if err != nil { log.Printf("Failed to read resolution directory %s: %v", resolutionPath, err) @@ -227,22 +243,14 @@ func (ps *ProcessingService) ParseResolutionDirectory(resolution string, ch chan func (ps *ProcessingService) AggregateSegmentInfo(ch <-chan SegmentInfo) (map[int]SegmentInfo, error) { segmentMap := make(map[int]SegmentInfo) - rank := map[string]int{ - "1080p": 1, - "720p": 2, - "540p": 3, - "480p": 4, - "450p": 5, - "360p": 6, - "270p": 7, - "240p": 8, - } - + // Each sequence number arrives once per downloaded variant; keep the + // highest-resolution copy of each, since that is the one worth stitching + // into the final video. for segment := range ch { fmt.Printf("Received segment %s in resolution %s \n", segment.Name, segment.Resolution) logging.MustGetFactory().Processing.Debug("segment received", logging.KeySegment, segment.Name, logging.KeyResolution, segment.Resolution) current, exists := segmentMap[segment.SeqNo] - if !exists || rank[segment.Resolution] > rank[current.Resolution] { + if !exists || resolutionHeight(segment.Resolution) > resolutionHeight(current.Resolution) { segmentMap[segment.SeqNo] = segment } } @@ -250,6 +258,24 @@ func (ps *ProcessingService) AggregateSegmentInfo(ch <-chan SegmentInfo) (map[in return segmentMap, nil } +// resolutionHeight converts a resolution label such as "1080p" into the pixel +// height it stands for, so variants can be ranked by quality. +// +// The height is parsed rather than looked up in a table: labels come from the +// playlist's own RESOLUTION attribute, so a stream can legitimately offer +// 1440p or 2160p that no fixed table would anticipate. Anything not in +// p form — "unknown", from a playlist with no variants — returns 0 and +// therefore loses to every recognised resolution. +func resolutionHeight(resolution string) int { + trimmed := strings.TrimSuffix(strings.ToLower(strings.TrimSpace(resolution)), "p") + + height, err := strconv.Atoi(trimmed) + if err != nil || height < 0 { + return 0 + } + return height +} + func (ps *ProcessingService) WriteConcatFile(segmentMap map[int]SegmentInfo) (string, error) { concatPath := ps.config.GetProcessOutputPath(ps.eventName) @@ -273,7 +299,7 @@ func (ps *ProcessingService) WriteConcatFile(segmentMap map[int]SegmentInfo) (st for _, seq := range keys { segment := segmentMap[seq] - filePath := utils.SafeJoin(ps.config.GetNASEventPath(ps.eventName), segment.Resolution, segment.Name) + filePath := utils.SafeJoin(ps.nasEventPath(), segment.Resolution, segment.Name) line := fmt.Sprintf("file '%s'\n", filePath) if _, err := f.WriteString(line); err != nil { return "", fmt.Errorf("failed to write to concat file: %w", err) diff --git a/pkg/processing/service_test.go b/pkg/processing/service_test.go index 46c5cd9..97f9999 100644 --- a/pkg/processing/service_test.go +++ b/pkg/processing/service_test.go @@ -1,7 +1,9 @@ package processing import ( + "context" "m3u8-downloader/pkg/config" + "m3u8-downloader/pkg/nas" "os" "path/filepath" "runtime" @@ -10,6 +12,22 @@ import ( "time" ) +// connectedNAS attaches a NAS service to a local directory. Reaching the NAS +// is platform-specific, but an already-mounted or plain directory resolves the +// same way everywhere, which is what makes this usable as a test double. +func connectedNAS(t *testing.T, path string) *nas.NASService { + t.Helper() + + svc, err := nas.New(nas.NASConfig{Path: path, Timeout: 5 * time.Second}) + if err != nil { + t.Fatalf("Failed to create NAS service for %s: %v", path, err) + } + if err := svc.Connect(context.Background()); err != nil { + t.Fatalf("Failed to connect NAS service to %s: %v", path, err) + } + return svc +} + func createTestConfig(tempDir string) *config.Config { return &config.Config{ Core: config.CoreConfig{ @@ -87,6 +105,7 @@ func TestProcessingService_GetEventDirs(t *testing.T) { ps := &ProcessingService{ config: cfg, eventName: "", // Empty to test directory discovery + nas: connectedNAS(t, cfg.NAS.OutputPath), } dirs, err := ps.GetEventDirs() @@ -114,12 +133,15 @@ func TestProcessingService_GetEventDirs(t *testing.T) { } func TestProcessingService_GetEventDirs_WithEventName(t *testing.T) { - cfg := createTestConfig("/tmp") + // Scoped to a temp dir rather than /tmp directly: attaching the NAS + // service creates the destination, and tests should not leave one behind. + cfg := createTestConfig(t.TempDir()) eventName := "specific-event" ps := &ProcessingService{ config: cfg, eventName: eventName, + nas: connectedNAS(t, cfg.NAS.OutputPath), } dirs, err := ps.GetEventDirs() @@ -156,6 +178,7 @@ func TestProcessingService_GetResolutions(t *testing.T) { ps := &ProcessingService{ config: cfg, eventName: eventName, + nas: connectedNAS(t, cfg.NAS.OutputPath), } resolutions, err := ps.GetResolutions() @@ -248,6 +271,7 @@ func TestProcessingService_WriteConcatFile(t *testing.T) { ps := &ProcessingService{ config: cfg, eventName: eventName, + nas: connectedNAS(t, cfg.NAS.OutputPath), } // Create test segment map @@ -382,3 +406,67 @@ func TestProcessJob_Structure(t *testing.T) { t.Errorf("Expected EventName='test-event', got '%s'", job.EventName) } } + +func TestResolutionHeight(t *testing.T) { + cases := map[string]int{ + "1080p": 1080, + "720p": 720, + "2160p": 2160, // beyond anything a fixed quality table would list + "1440p": 1440, + "240p": 240, + // Not in p form: must rank below every real resolution rather + // than accidentally outranking them. + "unknown": 0, + "": 0, + "p": 0, + "-720p": 0, + } + + for label, want := range cases { + if got := resolutionHeight(label); got != want { + t.Errorf("resolutionHeight(%q) = %d, want %d", label, got, want) + } + } +} + +// TestAggregateSegmentInfoPrefersHighestResolution covers the variants a +// hardcoded rank table could not express: resolutions above 1080p, and the +// "unknown" label produced for a playlist with no variants. +func TestAggregateSegmentInfoPrefersHighestResolution(t *testing.T) { + cases := []struct { + name string + arriving []string + want string + }{ + {"1080p beats 720p regardless of order", []string{"720p", "1080p"}, "1080p"}, + {"1080p beats 720p when it arrives first", []string{"1080p", "720p"}, "1080p"}, + {"4k beats 1080p", []string{"1080p", "2160p"}, "2160p"}, + {"1440p beats 1080p", []string{"1080p", "1440p"}, "1440p"}, + {"a known resolution beats unknown", []string{"unknown", "480p"}, "480p"}, + {"unknown does not displace a known resolution", []string{"1080p", "unknown"}, "1080p"}, + {"unknown alone is still kept", []string{"unknown"}, "unknown"}, + {"lowest only", []string{"240p"}, "240p"}, + } + + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + ps := &ProcessingService{} + ch := make(chan SegmentInfo, len(tc.arriving)) + for _, res := range tc.arriving { + ch <- SegmentInfo{Name: "seg_1001.ts", SeqNo: 1001, Resolution: res} + } + close(ch) + + segments, err := ps.AggregateSegmentInfo(ch) + if err != nil { + t.Fatalf("AggregateSegmentInfo() failed: %v", err) + } + if len(segments) != 1 { + t.Fatalf("expected a single sequence number, got %d", len(segments)) + } + if got := segments[1001].Resolution; got != tc.want { + t.Errorf("kept %q, want %q (arrival order: %v)", got, tc.want, tc.arriving) + } + }) + } +} diff --git a/pkg/transfer/cleanup.go b/pkg/transfer/cleanup.go index 8e3ecaf..ab6ef8b 100644 --- a/pkg/transfer/cleanup.go +++ b/pkg/transfer/cleanup.go @@ -2,6 +2,7 @@ package transfer import ( "context" + "errors" "fmt" "log" "m3u8-downloader/pkg/logging" @@ -10,6 +11,11 @@ import ( "time" ) +// errRetained reports that a file is still inside its retention window. It is +// not a failure: the file simply has to be reconsidered on a later pass rather +// than being discarded from the pending list. +var errRetained = errors.New("file is within the retention period") + type CleanupService struct { config CleanupConfig pendingFiles []string @@ -82,29 +88,43 @@ func (cs *CleanupService) ExecuteCleanup(ctx context.Context) error { log.Printf("Processing %d files for cleanup", len(batch)) var cleanedCount int - var errors []error + var failures []error + // requeue collects files this pass must not drop: ones still inside their + // retention window, and — if the context is cancelled mid-batch — the + // remainder that was never looked at. They were already removed from + // pendingFiles above, so without this they would never be cleaned at all. + var requeue []string - for _, filePath := range batch { + for i, filePath := range batch { select { case <-ctx.Done(): + requeue = append(requeue, batch[i:]...) + cs.requeue(requeue) return ctx.Err() default: } - if err := cs.cleanupFile(filePath); err != nil { - errors = append(errors, fmt.Errorf("Failed to cleanup file %s: %w", filePath, err)) - } else { + err := cs.cleanupFile(filePath) + switch { + case errors.Is(err, errRetained): + requeue = append(requeue, filePath) + case err != nil: + failures = append(failures, fmt.Errorf("Failed to cleanup file %s: %w", filePath, err)) + default: cleanedCount++ } } - log.Printf("Cleanup batch completed (cleaned: %d, errors: %d)", cleanedCount, len(errors)) - logging.MustGetFactory().Transfer.Info("cleanup batch completed", "cleaned", cleanedCount, "errors", len(errors)) + cs.requeue(requeue) - if len(errors) > 0 { - for i, err := range errors { + log.Printf("Cleanup batch completed (cleaned: %d, retained: %d, errors: %d)", cleanedCount, len(requeue), len(failures)) + logging.MustGetFactory().Transfer.Info("cleanup batch completed", + "cleaned", cleanedCount, "retained", len(requeue), "errors", len(failures)) + + if len(failures) > 0 { + for i, err := range failures { if i >= 3 { - log.Printf("... and %d more errors", len(errors)-3) + log.Printf("... and %d more errors", len(failures)-3) break } log.Printf("Error: %v", err) @@ -113,7 +133,17 @@ func (cs *CleanupService) ExecuteCleanup(ctx context.Context) error { } return nil +} +// requeue returns files to the pending list for a later pass. +func (cs *CleanupService) requeue(files []string) { + if len(files) == 0 { + return + } + + cs.mu.Lock() + defer cs.mu.Unlock() + cs.pendingFiles = append(cs.pendingFiles, files...) } func (cs *CleanupService) cleanupFile(filePath string) error { @@ -128,7 +158,7 @@ func (cs *CleanupService) cleanupFile(filePath string) error { if cs.config.RetentionPeriod > 0 { if time.Since(info.ModTime()) < cs.config.RetentionPeriod { log.Printf("File too new to cleanup: %s", filePath) - return nil + return errRetained } } @@ -162,6 +192,18 @@ func (cs *CleanupService) ForceCleanupAll(ctx context.Context) error { return err } + // Stop when a pass makes no progress. Files still inside their + // retention window are requeued by design, so requiring the list to + // reach zero would spin here until the shutdown timeout expired. + cs.mu.Lock() + remaining := len(cs.pendingFiles) + cs.mu.Unlock() + + if remaining >= pendingCount { + log.Printf("Force cleanup stopping with %d file(s) still pending (within retention period)", remaining) + break + } + select { case <-ctx.Done(): return ctx.Err() diff --git a/pkg/transfer/cleanup_test.go b/pkg/transfer/cleanup_test.go new file mode 100644 index 0000000..937835b --- /dev/null +++ b/pkg/transfer/cleanup_test.go @@ -0,0 +1,190 @@ +package transfer + +import ( + "context" + "fmt" + "os" + "path/filepath" + "testing" + "time" +) + +func writeAgedFile(t *testing.T, dir, name string, age time.Duration) string { + t.Helper() + + path := filepath.Join(dir, name) + if err := os.WriteFile(path, []byte("segment"), 0644); err != nil { + t.Fatalf("failed to write %s: %v", path, err) + } + + modTime := time.Now().Add(-age) + if err := os.Chtimes(path, modTime, modTime); err != nil { + t.Fatalf("failed to age %s: %v", path, err) + } + return path +} + +func TestCleanupRemovesTransferredFiles(t *testing.T) { + dir := t.TempDir() + cs := NewCleanupService(CleanupConfig{Enabled: true, BatchSize: 10, CheckInterval: time.Second}) + + path := writeAgedFile(t, dir, "seg_1001.ts", 0) + if err := cs.ScheduleCleanup(path); err != nil { + t.Fatalf("ScheduleCleanup failed: %v", err) + } + + if err := cs.ExecuteCleanup(context.Background()); err != nil { + t.Fatalf("ExecuteCleanup failed: %v", err) + } + + if _, err := os.Stat(path); !os.IsNotExist(err) { + t.Errorf("file should have been removed, stat err = %v", err) + } + if got := cs.GetPendingCount(); got != 0 { + t.Errorf("pending count = %d, want 0", got) + } +} + +// TestCleanupRetainsThenRemoves is the regression test for a file inside its +// retention window being dropped from the pending list: it was reported as +// cleaned, never deleted, and never reconsidered, so local files accumulated +// forever whenever RetainHours was set. +func TestCleanupRetainsThenRemoves(t *testing.T) { + dir := t.TempDir() + cs := NewCleanupService(CleanupConfig{ + Enabled: true, + RetentionPeriod: time.Hour, + BatchSize: 10, + CheckInterval: time.Second, + }) + + fresh := writeAgedFile(t, dir, "fresh.ts", time.Minute) + old := writeAgedFile(t, dir, "old.ts", 2*time.Hour) + + for _, p := range []string{fresh, old} { + if err := cs.ScheduleCleanup(p); err != nil { + t.Fatalf("ScheduleCleanup failed: %v", err) + } + } + + if err := cs.ExecuteCleanup(context.Background()); err != nil { + t.Fatalf("ExecuteCleanup failed: %v", err) + } + + if _, err := os.Stat(old); !os.IsNotExist(err) { + t.Errorf("a file past its retention period should be removed, stat err = %v", err) + } + if _, err := os.Stat(fresh); err != nil { + t.Errorf("a file inside its retention period should survive: %v", err) + } + if got := cs.GetPendingCount(); got != 1 { + t.Fatalf("the retained file must stay queued for a later pass, pending = %d", got) + } + + // Once it ages past the window, the next pass must actually remove it. + past := time.Now().Add(-2 * time.Hour) + if err := os.Chtimes(fresh, past, past); err != nil { + t.Fatalf("failed to age file: %v", err) + } + if err := cs.ExecuteCleanup(context.Background()); err != nil { + t.Fatalf("ExecuteCleanup failed: %v", err) + } + if _, err := os.Stat(fresh); !os.IsNotExist(err) { + t.Errorf("file should have been removed once past retention, stat err = %v", err) + } + if got := cs.GetPendingCount(); got != 0 { + t.Errorf("pending count = %d, want 0", got) + } +} + +// TestForceCleanupAllTerminatesWithRetainedFiles guards shutdown: retained +// files are requeued by design, so a loop demanding an empty list would spin +// until the shutdown timeout expired. +func TestForceCleanupAllTerminatesWithRetainedFiles(t *testing.T) { + dir := t.TempDir() + cs := NewCleanupService(CleanupConfig{ + Enabled: true, + RetentionPeriod: time.Hour, + BatchSize: 10, + CheckInterval: time.Second, + }) + + if err := cs.ScheduleCleanup(writeAgedFile(t, dir, "fresh.ts", time.Minute)); err != nil { + t.Fatalf("ScheduleCleanup failed: %v", err) + } + + done := make(chan error, 1) + go func() { done <- cs.ForceCleanupAll(context.Background()) }() + + select { + case err := <-done: + if err != nil { + t.Fatalf("ForceCleanupAll failed: %v", err) + } + case <-time.After(3 * time.Second): + t.Fatal("ForceCleanupAll did not terminate with a retained file pending") + } + + if got := cs.GetPendingCount(); got != 1 { + t.Errorf("the retained file should still be pending, got %d", got) + } +} + +func TestCleanupMissingFileIsNotAnError(t *testing.T) { + cs := NewCleanupService(CleanupConfig{Enabled: true, BatchSize: 10, CheckInterval: time.Second}) + + // Already gone — a file transferred and removed by something else. + if err := cs.ScheduleCleanup(filepath.Join(t.TempDir(), "gone.ts")); err != nil { + t.Fatalf("ScheduleCleanup failed: %v", err) + } + if err := cs.ExecuteCleanup(context.Background()); err != nil { + t.Fatalf("ExecuteCleanup failed: %v", err) + } + if got := cs.GetPendingCount(); got != 0 { + t.Errorf("a missing file should not be requeued, pending = %d", got) + } +} + +func TestCleanupDisabledSchedulesNothing(t *testing.T) { + dir := t.TempDir() + cs := NewCleanupService(CleanupConfig{Enabled: false, BatchSize: 10, CheckInterval: time.Second}) + + path := writeAgedFile(t, dir, "seg.ts", time.Hour) + if err := cs.ScheduleCleanup(path); err != nil { + t.Fatalf("ScheduleCleanup failed: %v", err) + } + + if got := cs.GetPendingCount(); got != 0 { + t.Errorf("cleanup is disabled, nothing should be queued, got %d", got) + } + if err := cs.ExecuteCleanup(context.Background()); err != nil { + t.Fatalf("ExecuteCleanup failed: %v", err) + } + if _, err := os.Stat(path); err != nil { + t.Errorf("file must survive when cleanup is disabled: %v", err) + } +} + +// TestCleanupRequeuesOnCancellation covers shutdown arriving mid-batch: the +// files already pulled off the pending list must go back, not vanish. +func TestCleanupRequeuesOnCancellation(t *testing.T) { + dir := t.TempDir() + cs := NewCleanupService(CleanupConfig{Enabled: true, BatchSize: 10, CheckInterval: time.Second}) + + for i := 0; i < 5; i++ { + path := writeAgedFile(t, dir, fmt.Sprintf("seg_%d.ts", i), 0) + if err := cs.ScheduleCleanup(path); err != nil { + t.Fatalf("ScheduleCleanup failed: %v", err) + } + } + + ctx, cancel := context.WithCancel(context.Background()) + cancel() + + if err := cs.ExecuteCleanup(ctx); err == nil { + t.Fatal("expected the cancelled context to be reported") + } + if got := cs.GetPendingCount(); got != 5 { + t.Errorf("all files should be requeued after cancellation, pending = %d, want 5", got) + } +} diff --git a/pkg/transfer/nas.go b/pkg/transfer/nas.go index e655a89..f62fc6f 100644 --- a/pkg/transfer/nas.go +++ b/pkg/transfer/nas.go @@ -11,7 +11,7 @@ import ( ) func TransferFile(nt *nas.NASService, ctx context.Context, item *TransferItem) error { - destPath := filepath.Join(nt.Config.Path, item.DestinationPath) + destPath := nt.Join(item.DestinationPath) destDir := filepath.Dir(destPath) if err := nt.EnsureDirectoryExists(destDir); err != nil { diff --git a/pkg/transfer/queue.go b/pkg/transfer/queue.go index 56f7354..87dafd0 100644 --- a/pkg/transfer/queue.go +++ b/pkg/transfer/queue.go @@ -169,13 +169,20 @@ func (tq *TransferQueue) processItem(ctx context.Context, item TransferItem) { return } - maxRetries := 3 + maxRetries := tq.config.RetryLimit + if maxRetries < 1 { + maxRetries = 1 + } for attempt := 1; attempt <= maxRetries; attempt++ { - if attempt > 0 { + // Back off before a retry, never before the first attempt. The guard + // was `attempt > 0`, which is always true, so every single transfer — + // including healthy ones — paid a one-second delay before it began, + // throttling the whole pipeline to the worker count per second. + if attempt > 1 { item.Status = StatusRetrying backoff := time.Duration(attempt*attempt) * time.Second - log.Printf("Backing off for %d seconds before retrying (attempt %d/%d)", backoff, attempt, maxRetries) + log.Printf("Backing off for %s before retrying (attempt %d/%d)", backoff, attempt, maxRetries) select { case <-time.After(backoff): diff --git a/pkg/transfer/service.go b/pkg/transfer/service.go index 8b2d030..54e290f 100644 --- a/pkg/transfer/service.go +++ b/pkg/transfer/service.go @@ -34,14 +34,23 @@ func NewTrasferService(outputDir string, eventName string) (*TransferService, er Timeout: cfg.NAS.Timeout, RetryLimit: cfg.NAS.RetryLimit, VerifySize: true, + AutoMount: cfg.NAS.AutoMount, + MountPoint: cfg.NAS.MountPoint, + } + nas, err := nas2.New(nasConfig) + if err != nil { + logging.MustGetFactory().Transfer.Error("invalid NAS destination", logging.KeyError, err.Error()) + return nil, fmt.Errorf("invalid NAS destination: %w", err) } - nas := nas2.NewNASService(nasConfig) - if err := nas.TestConnection(); err != nil { + connectCtx, cancelConnect := context.WithTimeout(context.Background(), cfg.NAS.Timeout) + defer cancelConnect() + if err := nas.Connect(connectCtx); err != nil { logging.MustGetFactory().Transfer.Error("failed to connect to NAS", logging.KeyError, err.Error()) return nil, fmt.Errorf("failed to connect to NAS: %w", err) } - logging.MustGetFactory().Transfer.Info("connected to NAS", "path", outputDir) + logging.MustGetFactory().Transfer.Info("connected to NAS", + "path", nas.LocalPath(), "configured", outputDir, "how", strings.Join(nas.ConnectionInfo(), "; ")) health.MustGetRegistry().RecordNASConnectivity(true) cleanupConfig := CleanupConfig{ @@ -57,6 +66,7 @@ func NewTrasferService(outputDir string, eventName string) (*TransferService, er PersistencePath: cfg.Paths.PersistenceFile, MaxQueueSize: cfg.Transfer.QueueSize, BatchSize: cfg.Transfer.BatchSize, + RetryLimit: cfg.Transfer.RetryLimit, } queue := NewTransferQueue(queueConfig, nas, cleanup) @@ -66,7 +76,7 @@ func NewTrasferService(outputDir string, eventName string) (*TransferService, er return nil, fmt.Errorf("failed to create local output directory: %w", err) } - watcher, err := NewFileWatcher(localOutputPath, queue, cfg.Transfer.FileSettlingDelay) + watcher, err := NewFileWatcher(localOutputPath, eventName, queue, cfg.Transfer.FileSettlingDelay) if err != nil { return nil, fmt.Errorf("failed to create file watcher: %w", err) } @@ -155,7 +165,7 @@ func (ts *TransferService) Shutdown(ctx context.Context) error { } // Disconnect from NAS - if err := ts.nas.Disconnect(); err != nil { + if err := ts.nas.Disconnect(ctx); err != nil { log.Printf("Warning: failed to disconnect from NAS: %v", err) logging.MustGetFactory().Transfer.Warn("failed to disconnect from NAS", logging.KeyError, err.Error()) } diff --git a/pkg/transfer/types.go b/pkg/transfer/types.go index be35896..3cb5a0e 100644 --- a/pkg/transfer/types.go +++ b/pkg/transfer/types.go @@ -49,6 +49,9 @@ type QueueConfig struct { PersistencePath string MaxQueueSize int BatchSize int + // RetryLimit is the total number of attempts made per file before it is + // marked failed. + RetryLimit int } type CleanupConfig struct { diff --git a/pkg/transfer/watcher.go b/pkg/transfer/watcher.go index a37f6ac..5781674 100644 --- a/pkg/transfer/watcher.go +++ b/pkg/transfer/watcher.go @@ -17,7 +17,12 @@ import ( ) type FileWatcher struct { - outputDir string + outputDir string + // eventName namespaces the NAS destination. It is passed in rather than + // derived from outputDir so that an unnamed event resolves to the NAS root + // — matching where the downloader put the files — instead of picking up + // the local output directory's own name. + eventName string queue *TransferQueue watcher *fsnotify.Watcher settingDelay time.Duration @@ -25,13 +30,14 @@ type FileWatcher struct { mu sync.Mutex } -func NewFileWatcher(outputDir string, queue *TransferQueue, settlingDelay time.Duration) (*FileWatcher, error) { +func NewFileWatcher(outputDir, eventName string, queue *TransferQueue, settlingDelay time.Duration) (*FileWatcher, error) { watcher, err := fsnotify.NewWatcher() if err != nil { return nil, err } return &FileWatcher{ outputDir: outputDir, + eventName: eventName, queue: queue, watcher: watcher, settingDelay: settlingDelay, @@ -92,6 +98,18 @@ func (fw *FileWatcher) addWatchRecursive(root string) error { } func (fw *FileWatcher) handleFileEvent(event fsnotify.Event) { + // Directories must be handled before the .ts filter below. fsnotify is not + // recursive, so a resolution directory created after the watcher starts — + // which is the normal case, since the transfer service starts before the + // downloader creates 1080p/ and friends — has to be picked up here or + // nothing inside it is ever seen. + if event.Op&fsnotify.Create == fsnotify.Create { + if info, err := os.Stat(event.Name); err == nil && info.IsDir() { + fw.watchNewDirectory(event.Name) + return + } + } + if !strings.HasSuffix(strings.ToLower(event.Name), ".ts") { return } @@ -104,15 +122,33 @@ func (fw *FileWatcher) handleFileEvent(event fsnotify.Event) { case event.Op&fsnotify.Remove == fsnotify.Remove: fw.cancelPendingTransfer(event.Name) } +} - if event.Op&fsnotify.Create == fsnotify.Create { - if info, err := os.Stat(event.Name); err == nil && info.IsDir() { - if err := fw.watcher.Add(event.Name); err != nil { - log.Printf("Failed to watch directory %s: %v", event.Name, err) - } else { - log.Printf("Watching directory %s", event.Name) - } +// watchNewDirectory starts watching a directory that appeared after startup, +// and sweeps whatever is already inside it. +// +// The sweep is not redundant: the downloader creates a resolution directory and +// writes segments into it immediately, so files can land between the mkdir and +// the watch being registered. Those writes generate no event we would see, and +// without the sweep the first segments of every recording would be lost. +func (fw *FileWatcher) watchNewDirectory(dir string) { + if err := fw.addWatchRecursive(dir); err != nil { + log.Printf("Failed to watch new directory %s: %v", dir, err) + logging.MustGetFactory().Transfer.Warn("failed to watch new directory", "dir", dir, logging.KeyError, err.Error()) + return + } + + err := filepath.Walk(dir, func(path string, info os.FileInfo, err error) error { + if err != nil { + return nil // keep sweeping the rest } + if !info.IsDir() && strings.HasSuffix(strings.ToLower(info.Name()), ".ts") { + fw.scheduleTransfer(path) + } + return nil + }) + if err != nil { + log.Printf("Failed to sweep new directory %s: %v", dir, err) } } @@ -164,10 +200,13 @@ func (fw *FileWatcher) processFile(filePath string) { return } + // The destination is namespaced by event, mirroring the local layout and + // matching what QueueExistingFiles produces for the same file — otherwise + // live transfers and catch-up transfers would write to different places. item := TransferItem{ ID: generateID(), SourcePath: filePath, - DestinationPath: relPath, + DestinationPath: filepath.Join(fw.eventName, relPath), Resolution: resolution, Timestamp: time.Now(), Status: StatusPending, diff --git a/pkg/transfer/watcher_test.go b/pkg/transfer/watcher_test.go new file mode 100644 index 0000000..99346d4 --- /dev/null +++ b/pkg/transfer/watcher_test.go @@ -0,0 +1,223 @@ +package transfer + +import ( + "context" + "os" + "path/filepath" + "sort" + "testing" + "time" +) + +// settlingDelay is short so tests exercise the real timer path without +// waiting the production five seconds. +const testSettlingDelay = 40 * time.Millisecond + +// newTestQueue builds a queue that accepts items but never transfers them: +// ProcessQueue is not started, so Add is a pure recording of intent. +func newTestQueue(t *testing.T) *TransferQueue { + t.Helper() + return NewTransferQueue(QueueConfig{ + WorkerCount: 1, + PersistencePath: filepath.Join(t.TempDir(), "queue.json"), + MaxQueueSize: 100, + BatchSize: 10, + }, nil, nil) +} + +// queuedDestinations returns the destination paths currently queued, sorted. +func queuedDestinations(q *TransferQueue) []string { + q.mu.RLock() + defer q.mu.RUnlock() + + out := make([]string, 0, q.items.Len()) + for _, item := range *q.items { + out = append(out, filepath.ToSlash(item.DestinationPath)) + } + sort.Strings(out) + return out +} + +// waitForQueue polls until the queue holds want items or the deadline passes. +// Polling beats a fixed sleep: it keeps the test fast when the watcher is +// prompt and tolerant when a loaded CI machine is not. +func waitForQueue(t *testing.T, q *TransferQueue, want int) []string { + t.Helper() + + deadline := time.Now().Add(3 * time.Second) + var got []string + for time.Now().Before(deadline) { + got = queuedDestinations(q) + if len(got) >= want { + return got + } + time.Sleep(10 * time.Millisecond) + } + return got +} + +func startWatcher(t *testing.T, dir, eventName string, q *TransferQueue) { + t.Helper() + + fw, err := NewFileWatcher(dir, eventName, q, testSettlingDelay) + if err != nil { + t.Fatalf("NewFileWatcher failed: %v", err) + } + + ctx, cancel := context.WithCancel(context.Background()) + t.Cleanup(cancel) + + started := make(chan struct{}) + go func() { + close(started) + _ = fw.Start(ctx) + }() + <-started + + // Give Start a moment to register its initial watches before the test + // begins creating files under them. + time.Sleep(100 * time.Millisecond) +} + +// TestWatcherPicksUpDirectoriesCreatedAfterStart covers the normal recording +// sequence: the transfer service starts watching the event directory, and only +// then does the downloader create the per-resolution directories inside it. +// fsnotify is not recursive, so those directories have to be picked up from +// the create event or nothing inside them is ever transferred. +func TestWatcherPicksUpDirectoriesCreatedAfterStart(t *testing.T) { + eventDir := t.TempDir() + queue := newTestQueue(t) + startWatcher(t, eventDir, "2026-regionals", queue) + + resolutionDir := filepath.Join(eventDir, "1080p") + if err := os.MkdirAll(resolutionDir, 0755); err != nil { + t.Fatalf("failed to create resolution dir: %v", err) + } + if err := os.WriteFile(filepath.Join(resolutionDir, "seg_1001.ts"), []byte("payload"), 0644); err != nil { + t.Fatalf("failed to write segment: %v", err) + } + + got := waitForQueue(t, queue, 1) + if len(got) != 1 { + t.Fatalf("segment written to a new resolution directory was never queued, queue = %v", got) + } + if want := "2026-regionals/1080p/seg_1001.ts"; got[0] != want { + t.Errorf("destination = %q, want %q", got[0], want) + } +} + +// TestWatcherSweepsFilesRacingDirectoryCreation covers the tighter race where +// segments are written immediately after the mkdir, before the new watch can +// be registered. Those writes produce no event the watcher would receive, so +// the directory has to be swept when it is first seen. +func TestWatcherSweepsFilesRacingDirectoryCreation(t *testing.T) { + eventDir := t.TempDir() + queue := newTestQueue(t) + startWatcher(t, eventDir, "event", queue) + + // Stage the tree elsewhere and move it into place, so the files are + // guaranteed to already exist the instant the directory appears. + staging := filepath.Join(t.TempDir(), "1080p") + if err := os.MkdirAll(staging, 0755); err != nil { + t.Fatalf("failed to stage: %v", err) + } + for _, name := range []string{"seg_1001.ts", "seg_1002.ts"} { + if err := os.WriteFile(filepath.Join(staging, name), []byte("payload"), 0644); err != nil { + t.Fatalf("failed to stage segment: %v", err) + } + } + if err := os.Rename(staging, filepath.Join(eventDir, "1080p")); err != nil { + t.Fatalf("failed to move staged directory into place: %v", err) + } + + got := waitForQueue(t, queue, 2) + want := []string{"event/1080p/seg_1001.ts", "event/1080p/seg_1002.ts"} + if len(got) != 2 || got[0] != want[0] || got[1] != want[1] { + t.Errorf("queued %v, want %v", got, want) + } +} + +// TestWatcherDestinationIncludesEventName pins the layout agreement between +// live transfers and the catch-up path in QueueExistingFiles: both must write +// to ///. +func TestWatcherDestinationIncludesEventName(t *testing.T) { + eventDir := t.TempDir() + resolutionDir := filepath.Join(eventDir, "720p") + if err := os.MkdirAll(resolutionDir, 0755); err != nil { + t.Fatalf("setup failed: %v", err) + } + + queue := newTestQueue(t) + startWatcher(t, eventDir, "my-event", queue) + + if err := os.WriteFile(filepath.Join(resolutionDir, "seg_1001.ts"), []byte("payload"), 0644); err != nil { + t.Fatalf("failed to write segment: %v", err) + } + + got := waitForQueue(t, queue, 1) + if len(got) != 1 { + t.Fatalf("expected one queued item, got %v", got) + } + if want := "my-event/720p/seg_1001.ts"; got[0] != want { + t.Errorf("destination = %q, want %q", got[0], want) + } +} + +// TestWatcherWithoutEventNameUsesRoot mirrors a download run with no -event: +// segments land under the local output root, so the NAS destination must not +// invent a directory level for them. +func TestWatcherWithoutEventNameUsesRoot(t *testing.T) { + eventDir := t.TempDir() + resolutionDir := filepath.Join(eventDir, "1080p") + if err := os.MkdirAll(resolutionDir, 0755); err != nil { + t.Fatalf("setup failed: %v", err) + } + + queue := newTestQueue(t) + startWatcher(t, eventDir, "", queue) + + if err := os.WriteFile(filepath.Join(resolutionDir, "seg_1001.ts"), []byte("payload"), 0644); err != nil { + t.Fatalf("failed to write segment: %v", err) + } + + got := waitForQueue(t, queue, 1) + if len(got) != 1 { + t.Fatalf("expected one queued item, got %v", got) + } + if want := "1080p/seg_1001.ts"; got[0] != want { + t.Errorf("destination = %q, want %q", got[0], want) + } +} + +func TestWatcherIgnoresNonSegmentFiles(t *testing.T) { + eventDir := t.TempDir() + queue := newTestQueue(t) + startWatcher(t, eventDir, "event", queue) + + for _, name := range []string{"notes.txt", "manifest.json", "segment.ts.part"} { + if err := os.WriteFile(filepath.Join(eventDir, name), []byte("x"), 0644); err != nil { + t.Fatalf("failed to write %s: %v", name, err) + } + } + + time.Sleep(testSettlingDelay + 300*time.Millisecond) + if got := queuedDestinations(queue); len(got) != 0 { + t.Errorf("non-segment files should not be queued, got %v", got) + } +} + +func TestExtractResolution(t *testing.T) { + fw := &FileWatcher{} + + cases := map[string]string{ + filepath.Join("data", "event", "1080p", "seg.ts"): "1080p", + filepath.Join("data", "event", "720p", "seg.ts"): "720p", + filepath.Join("data", "event", "seg.ts"): "", + } + + for path, want := range cases { + if got := fw.extractResolution(path); got != want { + t.Errorf("extractResolution(%q) = %q, want %q", path, got, want) + } + } +} diff --git a/pkg/utils/paths.go b/pkg/utils/paths.go index 83a698a..331d4f2 100644 --- a/pkg/utils/paths.go +++ b/pkg/utils/paths.go @@ -24,12 +24,28 @@ func PathExists(path string) bool { return !os.IsNotExist(err) } +// IsValidPath reports whether path is usable on Windows, the stricter of the +// two platforms this project targets, which reserves <>:"|?* in names. +// +// A leading drive letter is checked separately: its colon is legal there and +// nowhere else in the path. Without that carve-out the function rejects every +// absolute Windows path, including the drive-qualified ones it exists to +// validate. func IsValidPath(path string) bool { if path == "" { return false } - return !strings.ContainsAny(path, "<>:\"|?*") + rest := path + if len(rest) >= 2 && rest[1] == ':' && isDriveLetter(rest[0]) { + rest = rest[2:] + } + + return !strings.ContainsAny(rest, `<>:"|?*`) +} + +func isDriveLetter(c byte) bool { + return (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z') } func NormalizePath(path string) string { @@ -37,6 +53,13 @@ func NormalizePath(path string) string { } func GetRelativePath(basePath, targetPath string) (string, error) { + // filepath.Rel("", "") succeeds and returns ".", which is a meaningless + // answer to a meaningless question; callers passing an empty path have a + // bug worth surfacing rather than papering over. + if basePath == "" || targetPath == "" { + return "", fmt.Errorf("both base and target paths are required (base %q, target %q)", basePath, targetPath) + } + rel, err := filepath.Rel(basePath, targetPath) if err != nil { return "", fmt.Errorf("failed to get relative path: %w", err)