Claude bugfixes
Some checks failed
Build Go Binaries / build (push) Has been cancelled

This commit is contained in:
Aaron 2026-08-07 03:49:37 +00:00
parent 7f4078961e
commit e2566fdefa
33 changed files with 2838 additions and 221 deletions

View File

@ -33,9 +33,15 @@ The project follows a modular architecture with clear separation of concerns:
- **service.go**: Processing service orchestration with FFmpeg integration
- **segment.go**: Individual segment processing logic
- **types.go**: Processing system data structures
- **nas/**: NAS connection and file operations
- **nas/**: NAS connection and file operations (platform-agnostic: Windows and Linux)
- **config.go**: NAS configuration structure
- **nas.go**: NAS service with connection management and file operations
- **nas.go**: `NASService` — resolves the destination to a host-native path, then copies/verifies files against it
- **location.go**: Parses UNC / `//server/share` / `smb://` / local paths into a host-neutral `Location`
- **connector.go**: `Connector` interface + runtime selection by GOOS (no build tags, so both platforms' code is compiled and tested everywhere)
- **connector_netuse.go**: Windows strategy — `net use`, UNC path used directly
- **connector_mount.go**: Linux strategy — resolve the share against `/proc/mounts`, optional root-only `mount.cifs`
- **connector_local.go**: Already-mounted/plain directory on either platform
- **mounts.go**: `/proc/mounts` parsing and share-to-mount-point matching
- **config/**: Centralized configuration management with validation
- **config.go**: Configuration loading, validation, and path resolution
- **utils/**: Utility functions for cross-platform compatibility
@ -100,8 +106,10 @@ Configuration is managed through a centralized system in `pkg/config/config.go`
### NAS Transfer Settings
- `NAS.EnableTransfer`: Enable/disable automatic NAS transfer (true) - ENV: `ENABLE_NAS_TRANSFER`
- `NAS.OutputPath`: UNC path to NAS storage (`\\HomeLabNAS\dci\streams`) - ENV: `NAS_OUTPUT_PATH`
- `NAS.OutputPath`: NAS destination (`\\HomeLabNAS\dci\streams`) - ENV: `NAS_OUTPUT_PATH`. Accepts UNC, `//server/share`, `smb://`, or a local/mounted directory; Windows connects it with `net use`, Linux resolves it against `/proc/mounts`
- `NAS.Username`/`NAS.Password`: NAS credentials for authentication - ENV: `NAS_USERNAME`/`NAS_PASSWORD`
- `NAS.AutoMount`: Linux-only, root-required auto-mount of the SMB share (false) - ENV: `NAS_AUTO_MOUNT`
- `NAS.MountPoint`: Override for the auto-mount location - ENV: `NAS_MOUNT_POINT`
- `Transfer.WorkerCount`: Concurrent transfer workers (2)
- `Transfer.RetryLimit`: Max retry attempts per file (3)
- `Transfer.Timeout`: Timeout per file transfer (30 seconds)

View File

@ -11,10 +11,23 @@ The application supports configuration through environment variables for flexibl
- `REFRESH_DELAY_SECONDS`: How often to check for playlist updates in seconds (default: 3)
### NAS Transfer Settings
- `NAS_OUTPUT_PATH`: UNC path to NAS storage (default: "\\\\HomeLabNAS\\dci\\streams")
- `NAS_OUTPUT_PATH`: NAS destination (default: "\\\\HomeLabNAS\\dci\\streams"). Accepts a UNC path, `//server/share/dir`, `smb://server/share/dir`, or a local/mounted directory
- `NAS_USERNAME`: NAS authentication username
- `NAS_PASSWORD`: NAS authentication password
- `ENABLE_NAS_TRANSFER`: Enable/disable automatic NAS transfer (default: true)
- `NAS_AUTO_MOUNT`: Linux only — mount the SMB share if the host hasn't already (default: false, requires root)
- `NAS_MOUNT_POINT`: Where `NAS_AUTO_MOUNT` attaches the share (default: `/mnt/streamrecorder/<server>/<share>`)
The same `NAS_OUTPUT_PATH` works on Windows and Linux. Windows authenticates the
UNC path with `net use`; Linux resolves the share against `/proc/mounts` and
rewrites the path onto the mount point, so `\\\\HomeLabNAS\\dci\\streams` with
`//homelabnas/dci` mounted at `/mnt/dci` becomes `/mnt/dci/streams`. Mounting the
share on the host (fstab/autofs/systemd, or `docker run -v`) is the recommended
Linux setup; `NAS_AUTO_MOUNT` is the root-privileged fallback. Verify either with:
```bash
./stream-recorder -check
```
### Path Configuration
- `LOCAL_OUTPUT_DIR`: Base directory for local downloads (default: "data")

View File

@ -43,7 +43,7 @@ Segments land on disk under `Paths.LocalOutput/<event>/<resolution>/`.
- `service.go`: `TransferService` wires together the watcher, queue, and cleanup service, and owns the connection to NAS for the run.
- `watcher.go`: `FileWatcher` (via `fsnotify`) watches the local event directory for new `.ts` files and queues them after a short settling delay (`Transfer.FileSettlingDelay`, default 5s), so a file isn't queued while still being written.
- `queue.go`: `TransferQueue` is a priority-queued worker pool (`Transfer.WorkerCount`, default 2) that transfers queued files to NAS with retry + exponential backoff, and persists its state to `transfer_queue.json` so it survives a restart.
- `nas.go` / `pkg/nas`: `NASService` establishes the NAS connection (on Windows, via `net use` for UNC paths) and does the actual file copy + size verification.
- `nas.go` / `pkg/nas`: `NASService` attaches the NAS destination for whatever host it's running on and does the actual file copy + size verification. See [NAS connectivity](#nas-connectivity-windows-and-linux).
- `cleanup.go`: `CleanupService` deletes local copies after a successful, verified transfer (`Cleanup.AfterTransfer`, default true).
During a normal download run, this whole stage starts automatically in the same process as the downloader (see the note above) — you don't invoke transfer separately for a live recording. The standalone `-transfer` mode exists only to catch up files that a prior download session ended without fully transferring.
@ -83,8 +83,10 @@ All settings have sane defaults; override via environment variables as needed.
| `WORKER_COUNT` | `4` | Concurrent segment downloads per variant |
| `REFRESH_DELAY_SECONDS` | `3` | How often each variant's chunklist is polled |
| `ENABLE_NAS_TRANSFER` | `true` | Whether to run the transfer service alongside download |
| `NAS_OUTPUT_PATH` | `\\HomeLabNAS\dci\streams` | UNC/NAS root path segments are copied to |
| `NAS_USERNAME` / `NAS_PASSWORD` | *(see note below)* | NAS auth credentials |
| `NAS_OUTPUT_PATH` | `\\HomeLabNAS\dci\streams` | NAS root segments are copied to. UNC (`\\server\share\dir`), `//server/share/dir`, `smb://server/share/dir`, or a local/mounted directory — see [NAS connectivity](#nas-connectivity-windows-and-linux) |
| `NAS_USERNAME` / `NAS_PASSWORD` | *(see note below)* | NAS auth credentials. Used by `net use` on Windows and by `NAS_AUTO_MOUNT` on Linux; ignored for an already-mounted path |
| `NAS_AUTO_MOUNT` | `false` | Linux only: mount the SMB share if the host hasn't. Requires root |
| `NAS_MOUNT_POINT` | *(derived)* | Where `NAS_AUTO_MOUNT` attaches the share (default `/mnt/streamrecorder/<server>/<share>`) |
| `LOCAL_OUTPUT_DIR` | `data` | Local base directory for downloaded segments |
| `PROCESS_OUTPUT_DIR` | `out` | Output directory for the final concatenated `.mp4` |
| `FFMPEG_PATH` | `ffmpeg` | Path to the FFmpeg binary used by the processing stage |
@ -95,6 +97,25 @@ All settings have sane defaults; override via environment variables as needed.
> **Security note:** the default NAS credentials currently ship as hardcoded defaults in `pkg/config/config.go` for this home-lab setup. If you fork or redeploy this, set `NAS_USERNAME`/`NAS_PASSWORD` explicitly and rotate the shipped default — see `TODO.md` for the full flagged list of housekeeping items like this one.
## NAS connectivity (Windows and Linux)
The same `NAS_OUTPUT_PATH` works on both platforms; only the mechanism for reaching it differs, and the recorder picks it at runtime.
| Host | Destination | What happens |
|---|---|---|
| Windows | UNC / `//server/share` / `smb://` | `net use \\server\share` with `NAS_USERNAME`/`NAS_PASSWORD`, then the UNC path is used directly |
| Linux | UNC / `//server/share` / `smb://` | The share is looked up in `/proc/mounts` and the UNC path is rewritten onto the mount point (`\\nas\dci\streams` + `//nas/dci` mounted at `/mnt/dci` → `/mnt/dci/streams`). Set `NAS_AUTO_MOUNT=true` to mount it instead |
| Either | A local or already-mounted directory | Used as-is, no network step |
Notes:
- **Server and share names match case-insensitively**, and both `cifs`/`smb3` and `nfs`/`nfs4` mounts are recognised. If the share is mounted at a deeper level than configured (`//nas/dci/streams` at `/mnt/streams`), the deepest matching mount wins.
- **Nothing the host already set up is torn down.** A share found already mounted (or already connected via `net use`) is left alone at shutdown; only a mount or connection this process created is undone.
- **`NAS_AUTO_MOUNT` requires root**, so the usual Linux setup is to mount the share yourself via fstab/autofs/systemd and either point `NAS_OUTPUT_PATH` at the mount point or leave it as the UNC path and let it resolve. Credentials are passed to `mount.cifs` through a `0600` credentials file, never on the command line where the process table would expose the password.
- **A Windows-shaped path on Linux is rejected, not silently created.** Backslashes and drive letters are legal filename characters on Linux, so `C:\data` or an unresolved `\\nas\dci` would otherwise be created as a local directory named exactly that — transfers would report success, cleanup would delete the local originals, and nothing would reach the NAS.
Run `-check` to see which strategy applies and where the destination resolved to.
## Building
```bash

View File

@ -7,6 +7,7 @@ package check
import (
"context"
"fmt"
"log"
"m3u8-downloader/pkg/config"
"m3u8-downloader/pkg/constants"
"m3u8-downloader/pkg/httpClient"
@ -155,46 +156,51 @@ func checkMasterPlaylist(masterURL string) ([]string, error) {
return details, nil
}
// checkNAS runs the same connect/mkdir/write sequence the transfer service
// performs at startup, but without the log.Fatalf that path takes on failure.
// checkNAS runs the same attach/mkdir/write sequence the transfer service
// performs at startup, so the check exercises the real code path for this host
// — net use on Windows, mount resolution on Linux — rather than a simulation of
// it. It disconnects afterwards so the check leaves no mount behind.
func checkNAS(cfg *config.Config) ([]string, error) {
svc := &nas.NASService{Config: nas.NASConfig{
svc, err := nas.New(nas.NASConfig{
Path: cfg.NAS.OutputPath,
Username: cfg.NAS.Username,
Password: cfg.NAS.Password,
Timeout: cfg.NAS.Timeout,
RetryLimit: cfg.NAS.RetryLimit,
VerifySize: true,
}}
details := []string{"path: " + cfg.NAS.OutputPath}
if share := svc.ExtractNetworkPath(cfg.NAS.OutputPath); share != "" {
details = append(details, "share: "+share, "user: "+usernameLabel(cfg.NAS.Username))
} else {
details = append(details, "local or already-mounted path (no network authentication)")
AutoMount: cfg.NAS.AutoMount,
MountPoint: cfg.NAS.MountPoint,
})
if err != nil {
return []string{"path: " + cfg.NAS.OutputPath}, err
}
start := time.Now()
result := <-probeAsync(cfg.NAS.Timeout, func() ([]string, error) {
if err := svc.EstablishConnection(); err != nil {
return nil, fmt.Errorf("failed to connect to share: %w", err)
ctx, cancel := context.WithTimeout(context.Background(), cfg.NAS.Timeout)
defer cancel()
if err := svc.Connect(ctx); err != nil {
return svc.ConnectionInfo(), err
}
if err := svc.EnsureDirectoryExists(cfg.NAS.OutputPath); err != nil {
return nil, fmt.Errorf("share reachable but target directory unusable: %w", err)
defer func() {
if err := svc.Disconnect(context.Background()); err != nil {
log.Printf("Warning: failed to release NAS connection after check: %v", err)
}
if err := svc.TestConnection(); err != nil {
return nil, fmt.Errorf("target directory is not writable: %w", err)
}
return []string{"connect, mkdir and write probe succeeded"}, nil
}()
return append(svc.ConnectionInfo(), "attach, mkdir and write probe succeeded"), nil
})
details := result.details
if len(details) == 0 {
details = svc.ConnectionInfo()
}
if result.err != nil {
return details, result.err
}
details = append(details, result.details...)
details = append(details, "completed in "+formatDuration(time.Since(start)))
return details, nil
return append(details, "completed in "+formatDuration(time.Since(start))), nil
}
func formatDuration(d time.Duration) string {
@ -276,10 +282,3 @@ func enabledLabel(enabled bool) string {
}
return "disabled"
}
func usernameLabel(username string) string {
if username == "" {
return "(none, using ambient credentials)"
}
return username
}

View File

@ -154,7 +154,7 @@ func TestCheckNASLocalPath(t *testing.T) {
}
joined := joinDetails(details)
for _, want := range []string{"path: " + target, "write probe succeeded"} {
for _, want := range []string{"configured: " + target, "strategy: local", "write probe succeeded"} {
if !strings.Contains(joined, want) {
t.Errorf("details missing %q:\n%s", want, joined)
}
@ -176,7 +176,7 @@ func TestCheckNASUnusableTarget(t *testing.T) {
if err == nil {
t.Fatal("expected an unusable NAS target to fail the check")
}
if joined := joinDetails(details); !strings.Contains(joined, "path: ") {
if joined := joinDetails(details); !strings.Contains(joined, "configured: ") {
t.Errorf("expected path detail on failure:\n%s", joined)
}
}

View File

@ -71,7 +71,9 @@ func Download(masterURL string, eventName string, debug bool) {
}
}
manifestWriter := media.NewManifestWriter(eventName)
// One writer for the whole run: every variant records into it, and it is
// the one written to disk at the end.
manifest := media.NewManifestWriter(eventName)
eventPath := cfg.GetEventPath(eventName)
if err := utils.EnsureDir(eventPath); err != nil {
@ -79,7 +81,7 @@ func Download(masterURL string, eventName string, debug bool) {
log.Fatalf("Failed to create event directory: %v", err)
}
variants, err := media.GetAllVariants(masterURL, eventPath, manifestWriter)
variants, err := media.GetAllVariants(masterURL, eventPath)
if err != nil {
appLog.Error("failed to get variants", logging.KeyError, err.Error())
log.Fatalf("Failed to get variants: %v", err)
@ -88,8 +90,6 @@ func Download(masterURL string, eventName string, debug bool) {
sem := make(chan struct{}, constants.WorkerCount*len(variants))
manifest := media.NewManifestWriter(eventName)
for _, variant := range variants {
// Debug mode only tracks one variant for easier debugging
if debug {
@ -117,7 +117,7 @@ func Download(masterURL string, eventName string, debug bool) {
log.Println("All Services shut down.")
appLog.Info("all services shut down")
manifestWriter.WriteManifest()
manifest.WriteManifest()
log.Println("Manifest written.")
appLog.Info("manifest written", logging.KeyEvent, eventName)
appLog.Info("manifest written", logging.KeyEvent, eventName, "segments", manifest.Len())
}

View File

@ -32,11 +32,19 @@ type HTTPConfig struct {
type NASConfig struct {
EnableTransfer bool
// OutputPath is the destination as configured: a UNC path, an smb:// URL,
// or a directory on this host. How it is reached depends on the platform;
// see pkg/nas.
OutputPath string
Username string
Password string
Timeout time.Duration
RetryLimit int
// AutoMount lets the recorder mount an SMB share itself when the host has
// not already mounted it. Linux only, and requires root.
AutoMount bool
// MountPoint overrides where AutoMount attaches the share.
MountPoint string
}
type ProcessingConfig struct {
@ -95,6 +103,8 @@ var defaultConfig = Config{
Password: "s3tkY6tzA&KN6M",
Timeout: 30 * time.Second,
RetryLimit: 3,
AutoMount: false,
MountPoint: "",
},
Processing: ProcessingConfig{
Enabled: true,
@ -175,8 +185,25 @@ func (c *Config) loadFromEnvironment() error {
c.NAS.EnableTransfer = val == "true"
}
if val := os.Getenv("NAS_AUTO_MOUNT"); val != "" {
c.NAS.AutoMount = val == "true"
}
if val := os.Getenv("NAS_MOUNT_POINT"); val != "" {
c.NAS.MountPoint = val
}
if val := os.Getenv("LOCAL_OUTPUT_DIR"); val != "" {
c.Paths.LocalOutput = val
// The manifest describes the event directory and is documented as
// sitting beside it, so it follows the output directory. Without this
// it stays in the default data/ while the segments move elsewhere.
c.Paths.ManifestDir = val
}
// Applied after LOCAL_OUTPUT_DIR so an explicit setting wins.
if val := os.Getenv("MANIFEST_DIR"); val != "" {
c.Paths.ManifestDir = val
}
if val := os.Getenv("PROCESS_OUTPUT_DIR"); val != "" {

View File

@ -8,12 +8,22 @@ import (
"m3u8-downloader/pkg/utils"
"os"
"sort"
"strconv"
"sync"
)
type ManifestWriter struct {
ManifestPath string
Segments []ManifestItem
Index map[string]*ManifestItem
// Index maps a sequence number to its position in Segments. It holds
// positions rather than pointers because append reallocates the backing
// array, which would leave previously handed-out element pointers aimed at
// a stale copy.
Index map[string]int
// mu guards the fields above: every variant downloads concurrently, and
// each reports its segments as they land.
mu sync.Mutex
}
type ManifestItem struct {
@ -26,40 +36,55 @@ func NewManifestWriter(eventName string) *ManifestWriter {
return &ManifestWriter{
ManifestPath: cfg.GetManifestPath(eventName),
Segments: make([]ManifestItem, 0),
Index: make(map[string]*ManifestItem),
Index: make(map[string]int),
}
}
// AddOrUpdateSegment records that a segment was downloaded, keeping the
// highest-resolution variant seen for each sequence number.
func (m *ManifestWriter) AddOrUpdateSegment(seqNo string, resolution string) {
m.mu.Lock()
defer m.mu.Unlock()
if m.Index == nil {
m.Index = make(map[string]*ManifestItem)
m.Index = make(map[string]int)
}
if m.Segments == nil {
m.Segments = make([]ManifestItem, 0)
}
if existing, ok := m.Index[seqNo]; ok {
if resolution > existing.Resolution {
existing.Resolution = resolution
if pos, ok := m.Index[seqNo]; ok {
if ResolutionHeight(resolution) > ResolutionHeight(m.Segments[pos].Resolution) {
m.Segments[pos].Resolution = resolution
}
return
} else {
item := ManifestItem{
SeqNo: seqNo,
Resolution: resolution,
}
m.Segments = append(m.Segments, item)
m.Index[seqNo] = &item
m.Segments = append(m.Segments, ManifestItem{SeqNo: seqNo, Resolution: resolution})
m.Index[seqNo] = len(m.Segments) - 1
}
// Len reports how many distinct sequence numbers have been recorded.
func (m *ManifestWriter) Len() int {
m.mu.Lock()
defer m.mu.Unlock()
return len(m.Segments)
}
func (m *ManifestWriter) WriteManifest() {
sort.Slice(m.Segments, func(i, j int) bool {
return m.Segments[i].SeqNo < m.Segments[j].SeqNo
// Sort a copy: reordering Segments in place would invalidate Index, and a
// recording may still be writing into it.
m.mu.Lock()
segments := make([]ManifestItem, len(m.Segments))
copy(segments, m.Segments)
m.mu.Unlock()
sort.Slice(segments, func(i, j int) bool {
return lessSeqNo(segments[i].SeqNo, segments[j].SeqNo)
})
data, err := json.MarshalIndent(m.Segments, "", " ")
data, err := json.MarshalIndent(segments, "", " ")
if err != nil {
log.Printf("Failed to marshal manifest: %v", err)
logging.MustGetFactory().Download.Error("failed to marshal manifest", logging.KeyError, err.Error())
@ -88,3 +113,15 @@ func (m *ManifestWriter) WriteManifest() {
return
}
}
// lessSeqNo orders sequence numbers numerically, so segment 9 precedes segment
// 10 instead of sorting after it as the strings would. Values that are not
// numeric fall back to lexicographic order.
func lessSeqNo(a, b string) bool {
ai, aErr := strconv.ParseUint(a, 10, 64)
bi, bErr := strconv.ParseUint(b, 10, 64)
if aErr == nil && bErr == nil {
return ai < bi
}
return a < b
}

View File

@ -4,6 +4,8 @@ import (
"encoding/json"
"os"
"path/filepath"
"strconv"
"sync"
"testing"
)
@ -41,7 +43,7 @@ func TestManifestWriter_AddOrUpdateSegment(t *testing.T) {
writer := &ManifestWriter{
ManifestPath: "test.json",
Segments: make([]ManifestItem, 0),
Index: make(map[string]*ManifestItem),
Index: make(map[string]int),
}
// Test adding new segment
@ -112,7 +114,7 @@ func TestManifestWriter_WriteManifest(t *testing.T) {
writer := &ManifestWriter{
ManifestPath: manifestPath,
Segments: make([]ManifestItem, 0),
Index: make(map[string]*ManifestItem),
Index: make(map[string]int),
}
// Add some test segments out of order
@ -172,7 +174,7 @@ func TestManifestWriter_WriteManifest_EmptySegments(t *testing.T) {
writer := &ManifestWriter{
ManifestPath: manifestPath,
Segments: make([]ManifestItem, 0),
Index: make(map[string]*ManifestItem),
Index: make(map[string]int),
}
// Write empty manifest
@ -204,7 +206,7 @@ func TestManifestWriter_WriteManifest_InvalidPath(t *testing.T) {
writer := &ManifestWriter{
ManifestPath: "/invalid/path/that/does/not/exist/manifest.json",
Segments: []ManifestItem{{SeqNo: "1001", Resolution: "1080p"}},
Index: make(map[string]*ManifestItem),
Index: make(map[string]int),
}
// This should not panic, just fail gracefully
@ -239,3 +241,169 @@ func TestManifestItem_JSONSerialization(t *testing.T) {
t.Errorf("Resolution mismatch: expected '%s', got '%s'", item.Resolution, unmarshaled.Resolution)
}
}
func TestResolutionHeight(t *testing.T) {
cases := map[string]int{
"1080p": 1080,
"720p": 720,
"2160p": 2160,
"1440p": 1440,
"240p": 240,
// Not a <height>p label: must rank below every real resolution rather
// than accidentally outranking them.
"unknown": 0,
"": 0,
"p": 0,
}
for label, want := range cases {
if got := ResolutionHeight(label); got != want {
t.Errorf("ResolutionHeight(%q) = %d, want %d", label, got, want)
}
}
}
// TestAddOrUpdateSegmentKeepsHighestResolution pins the comparison used to
// choose between variants. Comparing the labels as strings — as this once did —
// ranks "720p" above "1080p", quietly downgrading the manifest.
func TestAddOrUpdateSegmentKeepsHighestResolution(t *testing.T) {
cases := []struct {
name string
arriving []string
want string
}{
{"higher replaces lower", []string{"720p", "1080p"}, "1080p"},
{"lower does not replace higher", []string{"1080p", "720p"}, "1080p"},
{"fewer digits must not win", []string{"1080p", "480p"}, "1080p"},
{"4k beats 1080p", []string{"1080p", "2160p"}, "2160p"},
{"unknown loses to a real resolution", []string{"unknown", "480p"}, "480p"},
{"unknown does not displace a real resolution", []string{"1080p", "unknown"}, "1080p"},
{"repeated identical resolution", []string{"720p", "720p"}, "720p"},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
w := &ManifestWriter{}
for _, res := range tc.arriving {
w.AddOrUpdateSegment("1001", res)
}
if len(w.Segments) != 1 {
t.Fatalf("expected a single entry, got %d", len(w.Segments))
}
if got := w.Segments[0].Resolution; got != tc.want {
t.Errorf("kept %q, want %q (arrival order %v)", got, tc.want, tc.arriving)
}
})
}
}
// TestAddOrUpdateSegmentUpdatesTheStoredEntry guards the aliasing bug: the
// index once held a pointer to a local copy rather than to the stored element,
// so updates were applied to an orphan and never reached the written manifest.
func TestAddOrUpdateSegmentUpdatesTheStoredEntry(t *testing.T) {
w := &ManifestWriter{}
// Enough entries to force at least one append reallocation, which would
// invalidate any pointer held into the backing array.
for i := 0; i < 64; i++ {
w.AddOrUpdateSegment(strconv.Itoa(1000+i), "480p")
}
for i := 0; i < 64; i++ {
w.AddOrUpdateSegment(strconv.Itoa(1000+i), "1080p")
}
if len(w.Segments) != 64 {
t.Fatalf("expected 64 entries, got %d", len(w.Segments))
}
for _, seg := range w.Segments {
if seg.Resolution != "1080p" {
t.Fatalf("segment %s kept %q; the upgrade never reached the stored entry", seg.SeqNo, seg.Resolution)
}
}
}
// TestAddOrUpdateSegmentIsConcurrencySafe reflects real use: every variant
// downloads in its own goroutine and reports segments as they land.
func TestAddOrUpdateSegmentIsConcurrencySafe(t *testing.T) {
w := &ManifestWriter{}
resolutions := []string{"240p", "480p", "720p", "1080p"}
var wg sync.WaitGroup
for _, res := range resolutions {
wg.Add(1)
go func(res string) {
defer wg.Done()
for seq := 0; seq < 200; seq++ {
w.AddOrUpdateSegment(strconv.Itoa(seq), res)
}
}(res)
}
wg.Wait()
if got := w.Len(); got != 200 {
t.Errorf("expected 200 distinct sequence numbers, got %d", got)
}
for _, seg := range w.Segments {
if seg.Resolution != "1080p" {
t.Errorf("segment %s = %q, want the highest variant 1080p", seg.SeqNo, seg.Resolution)
}
}
}
// TestWriteManifestSortsNumerically checks that segment 9 precedes segment 10,
// which string ordering would reverse.
func TestWriteManifestSortsNumerically(t *testing.T) {
dir := t.TempDir()
w := &ManifestWriter{ManifestPath: filepath.Join(dir, "event.json")}
for _, seq := range []string{"100", "9", "1000", "10", "1"} {
w.AddOrUpdateSegment(seq, "1080p")
}
w.WriteManifest()
data, err := os.ReadFile(w.ManifestPath)
if err != nil {
t.Fatalf("failed to read manifest: %v", err)
}
var items []ManifestItem
if err := json.Unmarshal(data, &items); err != nil {
t.Fatalf("manifest is not valid JSON: %v", err)
}
want := []string{"1", "9", "10", "100", "1000"}
if len(items) != len(want) {
t.Fatalf("expected %d entries, got %d", len(want), len(items))
}
for i, seq := range want {
if items[i].SeqNo != seq {
t.Errorf("position %d = %q, want %q (full order: %v)", i, items[i].SeqNo, seq, items)
}
}
}
// TestWriteManifestLeavesIndexUsable ensures writing mid-recording does not
// corrupt the writer: sorting in place would desynchronise Index from Segments.
func TestWriteManifestLeavesIndexUsable(t *testing.T) {
dir := t.TempDir()
w := &ManifestWriter{ManifestPath: filepath.Join(dir, "event.json")}
w.AddOrUpdateSegment("20", "480p")
w.AddOrUpdateSegment("10", "480p")
w.WriteManifest()
// After the write, an upgrade must still land on the right entry.
w.AddOrUpdateSegment("20", "1080p")
pos, ok := w.Index["20"]
if !ok {
t.Fatal("sequence 20 missing from the index after WriteManifest")
}
if got := w.Segments[pos].Resolution; got != "1080p" {
t.Errorf("sequence 20 = %q, want 1080p; the index no longer matches the slice", got)
}
if w.Segments[w.Index["10"]].SeqNo != "10" {
t.Error("index and slice disagree after WriteManifest")
}
}

View File

@ -13,6 +13,7 @@ import (
"net/http"
"net/url"
"path"
"strconv"
"strings"
"time"
)
@ -24,7 +25,27 @@ type StreamVariant struct {
ID int
Resolution string
OutputDir string
Writer *ManifestWriter
}
// ResolutionHeight converts a label produced by ExtractResolution back into
// the pixel height it stands for, so variants can be ranked by quality.
//
// The height is parsed rather than looked up in a table: labels come from the
// playlist's own RESOLUTION attribute, so a stream can legitimately offer
// 1440p or 2160p that no fixed table would anticipate. Anything not in
// <height>p form — "unknown", from a playlist with no variants — returns 0 and
// therefore ranks below every recognised resolution.
//
// Comparing the labels as strings instead is a trap worth naming: "720p" sorts
// after "1080p", so lexicographic comparison silently prefers lower quality.
func ResolutionHeight(resolution string) int {
trimmed := strings.TrimSuffix(strings.ToLower(strings.TrimSpace(resolution)), "p")
height, err := strconv.Atoi(trimmed)
if err != nil || height < 0 {
return 0
}
return height
}
// ExtractResolution maps a variant to the label used for its output directory,
@ -50,7 +71,10 @@ func ExtractResolution(variant *m3u8.Variant) string {
}
}
func GetAllVariants(masterURL string, outputDir string, writer *ManifestWriter) ([]*StreamVariant, error) {
// GetAllVariants fetches the master playlist and returns one entry per stream
// variant. Segments are recorded into the manifest by VariantDownloader, which
// receives it directly, so no writer is threaded through here.
func GetAllVariants(masterURL string, outputDir string) ([]*StreamVariant, error) {
client := &http.Client{}
req, _ := http.NewRequest("GET", masterURL, nil)
req.Header.Set("User-Agent", constants.HTTPUserAgent)
@ -78,7 +102,6 @@ func GetAllVariants(masterURL string, outputDir string, writer *ManifestWriter)
ID: 0,
Resolution: "unknown",
OutputDir: path.Join(outputDir, "unknown"),
Writer: writer,
}}, nil
}
@ -166,6 +189,9 @@ func VariantDownloader(ctx context.Context, variant *StreamVariant, sem chan str
log.Printf("✓ %s downloaded segment %s", j.Variant.Resolution, name)
dlLog.Debug("segment downloaded", logging.KeySegment, name)
registry.RecordSegmentDownloaded(variant.Resolution)
if manifest != nil {
manifest.AddOrUpdateSegment(strconv.FormatUint(j.Seq, 10), variant.Resolution)
}
return
}

View File

@ -3,10 +3,18 @@ package nas
import "time"
type NASConfig struct {
// Path is the destination as configured. It may be a UNC path, an
// smb:// URL, or a directory on this host; see ParseLocation.
Path string
Username string
Password string
Timeout time.Duration
RetryLimit int
VerifySize bool
// AutoMount allows the recorder to mount an SMB share itself when the
// host has not already mounted it. Linux only, and requires root, so it
// is off by default in favour of a clear "mount it first" error.
AutoMount bool
// MountPoint overrides where AutoMount attaches the share.
MountPoint string
}

98
pkg/nas/connector.go Normal file
View File

@ -0,0 +1,98 @@
package nas
import (
"context"
"fmt"
"os/exec"
"runtime"
)
// Connector attaches a configured Location to the host filesystem so that
// ordinary file operations reach the NAS.
//
// Windows and Linux disagree about what "connecting to a share" means —
// Windows authenticates a UNC path in place, Linux needs the share mounted
// somewhere in its single filesystem tree — so each host gets its own
// implementation and NASService stays platform-neutral. Selection happens at
// runtime rather than through build tags so that every implementation is
// compiled, vetted and unit-testable on every platform; only the final exec
// call is genuinely host-specific.
type Connector interface {
// Name identifies the strategy in logs and check output.
Name() string
// Attach makes loc reachable and reports where to write.
Attach(ctx context.Context, loc Location, cfg NASConfig) (Attachment, error)
// Detach undoes whatever Attach set up. Implementations must undo only
// what they created: a share this process found already mounted belongs
// to the host, and unmounting it could break unrelated software.
Detach(ctx context.Context, att Attachment) error
}
// Attachment is the result of a successful Attach.
type Attachment struct {
// BasePath is the host-native directory that maps to the configured
// destination. Every file operation is resolved against it.
BasePath string
// Notes explain how the host reached the destination, for -check output
// and startup logs.
Notes []string
// Owned records that this process created the connection or mount, and
// so may tear it down on shutdown.
Owned bool
// mountPoint is the directory this process mounted, when Owned is set by
// the Linux connector.
mountPoint string
// share is the network path this process connected, when Owned is set by
// the Windows connector.
share string
}
// commandRunner executes an external command. Implementations keep argument
// construction in a pure function and call through this, so tests can assert
// exactly what would run without running it.
type commandRunner func(ctx context.Context, name string, args ...string) ([]byte, error)
// execRunner is the production runner: it combines stdout and stderr because
// mount and net both report the useful diagnosis on one or the other.
func execRunner(ctx context.Context, name string, args ...string) ([]byte, error) {
return exec.CommandContext(ctx, name, args...).CombinedOutput()
}
// connectorFor picks the strategy for a destination on a given OS. A
// destination that is already an ordinary directory needs no network step on
// any platform, so it short-circuits before the OS switch.
func connectorFor(loc Location, goos string) Connector {
if !loc.IsRemote() {
return &localConnector{goos: goos}
}
switch goos {
case "windows":
return &windowsConnector{run: execRunner}
case "linux":
return &linuxConnector{run: execRunner, mountsPath: procMountsPath}
default:
return &unsupportedConnector{goos: goos}
}
}
// defaultConnector selects the strategy for the running host.
func defaultConnector(loc Location) Connector {
return connectorFor(loc, runtime.GOOS)
}
// unsupportedConnector fails clearly on platforms this project does not target,
// rather than silently writing a share-shaped directory to local disk.
type unsupportedConnector struct {
goos string
}
func (c *unsupportedConnector) Name() string { return "unsupported" }
func (c *unsupportedConnector) Attach(_ context.Context, loc Location, _ NASConfig) (Attachment, error) {
return Attachment{}, fmt.Errorf(
"connecting to network share %s is not supported on %s; mount the share and set NAS_OUTPUT_PATH to the mount point",
loc.SMBSource(), c.goos)
}
func (c *unsupportedConnector) Detach(context.Context, Attachment) error { return nil }

View File

@ -0,0 +1,57 @@
package nas
import (
"context"
"fmt"
"path/filepath"
"strings"
)
// localConnector handles destinations that are already part of the host
// filesystem: a plain output directory, or — the normal arrangement on Linux —
// an SMB/NFS share mounted by fstab, autofs or systemd before the recorder
// starts. There is nothing to connect, but there is something to check.
type localConnector struct {
goos string
}
func (c *localConnector) Name() string { return "local" }
func (c *localConnector) Attach(_ context.Context, loc Location, _ NASConfig) (Attachment, error) {
if err := c.validate(loc.Raw); err != nil {
return Attachment{}, err
}
return Attachment{
BasePath: filepath.Clean(loc.Raw),
Notes: []string{"local or already-mounted path (no network authentication)"},
}, nil
}
func (c *localConnector) Detach(context.Context, Attachment) error { return nil }
// validate rejects a Windows-shaped path on a non-Windows host. This looks
// pedantic and is not: backslashes and colons are legal characters in Linux
// filenames, so a stray C:\data or \\server\share would be created verbatim as
// a local directory. Transfers would report success, cleanup would delete the
// local originals, and nothing would ever reach the NAS.
func (c *localConnector) validate(path string) error {
if c.goos == "windows" {
return nil
}
if driveLetter.MatchString(path) {
return fmt.Errorf(
"%q is a Windows drive path but this host is %s; set NAS_OUTPUT_PATH to a mount point such as /mnt/nas/streams",
path, c.goos)
}
if strings.Contains(path, `\`) {
return fmt.Errorf(
"%q contains backslashes but this host is %s, where they are ordinary filename characters rather than separators; "+
"set NAS_OUTPUT_PATH to a mount point such as /mnt/nas/streams, or to \\\\server\\share\\folder to have the share resolved automatically",
path, c.goos)
}
return nil
}

178
pkg/nas/connector_mount.go Normal file
View File

@ -0,0 +1,178 @@
package nas
import (
"context"
"fmt"
"os"
"path/filepath"
"strings"
)
// linuxConnector reaches a share the way Linux does: through a mount point in
// the single filesystem tree. There is no per-path authentication, so the
// question is not "can we connect" but "where is this share mounted".
//
// The default and by far the safest answer is that the host already mounted it
// (fstab, autofs, systemd, or `docker run -v`), so the connector looks the
// share up in the mount table. Mounting it here is opt-in because it requires
// root and changes system state that outlives a failed run.
type linuxConnector struct {
run commandRunner
mountsPath string
// geteuid is indirected for tests; production uses os.Geteuid.
geteuid func() int
}
func (c *linuxConnector) Name() string { return "mount" }
func (c *linuxConnector) Attach(ctx context.Context, loc Location, cfg NASConfig) (Attachment, error) {
entries, err := readMountTable(c.mountsPath)
if err != nil {
return Attachment{}, err
}
if point, rest, ok := findMount(entries, loc); ok {
return Attachment{
BasePath: filepath.Join(append([]string{point}, rest...)...),
Notes: []string{
"share: " + loc.SMBSource(),
fmt.Sprintf("resolved via existing mount at %s", point),
},
}, nil
}
if !cfg.AutoMount {
return Attachment{}, fmt.Errorf("%s", notMountedMessage(loc, cfg))
}
return c.mount(ctx, loc, cfg)
}
func (c *linuxConnector) Detach(ctx context.Context, att Attachment) error {
// Never unmount a share the host had already mounted; other software may
// be using it, and we did not put it there.
if !att.Owned || att.mountPoint == "" {
return nil
}
if out, err := c.run(ctx, "umount", att.mountPoint); err != nil {
return fmt.Errorf("failed to unmount %s: %v", att.mountPoint, summarize(err, out))
}
return nil
}
// mount performs an opt-in mount.cifs of the share.
func (c *linuxConnector) mount(ctx context.Context, loc Location, cfg NASConfig) (Attachment, error) {
geteuid := c.geteuid
if geteuid == nil {
geteuid = os.Geteuid
}
if geteuid() != 0 {
return Attachment{}, fmt.Errorf(
"NAS_AUTO_MOUNT is set but mounting %s requires root; run as root, or mount the share beforehand and %s",
loc.SMBSource(), suggestMountPointConfig(loc))
}
point := autoMountPoint(loc, cfg)
if err := os.MkdirAll(point, 0755); err != nil {
return Attachment{}, fmt.Errorf("failed to create mount point %s: %w", point, err)
}
credentialsPath, cleanup, err := writeCredentialsFile(cfg)
if err != nil {
return Attachment{}, err
}
defer cleanup()
if cfg.Timeout > 0 {
var cancel context.CancelFunc
ctx, cancel = context.WithTimeout(ctx, cfg.Timeout)
defer cancel()
}
if out, err := c.run(ctx, "mount", mountArgs(loc, point, credentialsPath)...); err != nil {
return Attachment{}, fmt.Errorf("failed to mount %s at %s: %v", loc.SMBSource(), point, summarize(err, out))
}
return Attachment{
BasePath: filepath.Join(append([]string{point}, loc.Sub...)...),
Owned: true,
mountPoint: point,
Notes: []string{
"share: " + loc.SMBSource(),
"user: " + usernameNote(cfg.Username),
fmt.Sprintf("mounted at %s by this process (will unmount on shutdown)", point),
},
}, nil
}
// mountArgs builds the mount.cifs invocation. Kept pure so it can be asserted
// without mounting anything.
func mountArgs(loc Location, mountPoint, credentialsPath string) []string {
options := "rw"
if credentialsPath != "" {
options += ",credentials=" + credentialsPath
} else {
options += ",guest"
}
return []string{"-t", "cifs", loc.SMBSource(), mountPoint, "-o", options}
}
// writeCredentialsFile stages the NAS credentials in a 0600 file. mount.cifs
// also accepts user=/pass= options, but those land in the process table where
// any local user can read the password; a credentials file does not.
func writeCredentialsFile(cfg NASConfig) (path string, cleanup func(), err error) {
if cfg.Username == "" || cfg.Password == "" {
return "", func() {}, nil
}
dir, err := os.MkdirTemp("", "streamrecorder-cifs-")
if err != nil {
return "", func() {}, fmt.Errorf("failed to stage NAS credentials: %w", err)
}
cleanup = func() { os.RemoveAll(dir) }
path = filepath.Join(dir, "credentials")
content := fmt.Sprintf("username=%s\npassword=%s\n", cfg.Username, cfg.Password)
if err := os.WriteFile(path, []byte(content), 0600); err != nil {
cleanup()
return "", func() {}, fmt.Errorf("failed to write NAS credentials file: %w", err)
}
return path, cleanup, nil
}
// autoMountPoint chooses where to mount the share when the operator has not
// named a location. Everything lands under one predictable directory so a
// leftover mount is easy to find and clean up.
func autoMountPoint(loc Location, cfg NASConfig) string {
if cfg.MountPoint != "" {
return cfg.MountPoint
}
return filepath.Join("/mnt", "streamrecorder", strings.ToLower(loc.Server), strings.ToLower(loc.Share))
}
// notMountedMessage explains the failure in terms of what the operator should
// actually type. A bare "no such mount" would leave them guessing.
func notMountedMessage(loc Location, cfg NASConfig) string {
source := loc.SMBSource()
example := filepath.Join("/mnt", strings.ToLower(loc.Share))
credentials := ""
if cfg.Username != "" {
credentials = fmt.Sprintf(",username=%s,password=***", cfg.Username)
}
return fmt.Sprintf(
"%s is not mounted on this host. Either mount it and %s, for example:\n"+
" sudo mkdir -p %s && sudo mount -t cifs %s %s -o rw%s\n"+
"or set NAS_AUTO_MOUNT=true to have the recorder mount it (requires root)",
source, suggestMountPointConfig(loc), example, source, example, credentials)
}
func suggestMountPointConfig(loc Location) string {
if len(loc.Sub) == 0 {
return "set NAS_OUTPUT_PATH to the mount point"
}
return fmt.Sprintf("set NAS_OUTPUT_PATH to the mount point plus %s", loc.SubPath())
}

107
pkg/nas/connector_netuse.go Normal file
View File

@ -0,0 +1,107 @@
package nas
import (
"context"
"fmt"
"path/filepath"
"strings"
)
// windowsConnector reaches a share the way Windows does: authenticate the UNC
// path with `net use`, then use the UNC path directly as a filesystem path.
// Nothing is mounted into a tree, so the resolved base path is the configured
// UNC path itself.
type windowsConnector struct {
run commandRunner
}
func (c *windowsConnector) Name() string { return "net use" }
func (c *windowsConnector) Attach(ctx context.Context, loc Location, cfg NASConfig) (Attachment, error) {
share := loc.ShareUNC()
att := Attachment{
BasePath: loc.UNC(),
share: share,
Notes: []string{"share: " + share, "user: " + usernameNote(cfg.Username)},
}
if cfg.Timeout > 0 {
var cancel context.CancelFunc
ctx, cancel = context.WithTimeout(ctx, cfg.Timeout)
defer cancel()
}
out, err := c.run(ctx, "net", netUseArgs(loc, cfg)...)
if err != nil {
// A failure here is not decisive. `net use` also fails when the share
// is already connected — commonly because Explorer or another process
// got there first — in which case the path is perfectly usable. Record
// the reason and let the caller's write probe deliver the verdict,
// rather than refusing to run against a working share.
att.Notes = append(att.Notes,
fmt.Sprintf("net use did not complete (%v); continuing, the share may already be connected", summarize(err, out)))
return att, nil
}
att.Owned = true
att.Notes = append(att.Notes, "connected via net use")
return att, nil
}
func (c *windowsConnector) Detach(ctx context.Context, att Attachment) error {
// Only drop connections this process made. Deleting a mapping the user or
// another application established would break them.
if !att.Owned || att.share == "" {
return nil
}
if out, err := c.run(ctx, "net", "use", att.share, "/delete"); err != nil {
return fmt.Errorf("failed to disconnect %s: %v", att.share, summarize(err, out))
}
return nil
}
// netUseArgs builds the `net use` invocation. Kept pure so the Windows code
// path can be asserted from any host, including the Linux box this is
// developed on, without executing anything.
func netUseArgs(loc Location, cfg NASConfig) []string {
args := []string{"use", loc.ShareUNC()}
if cfg.Username != "" && cfg.Password != "" {
args = append(args, "/user:"+cfg.Username, cfg.Password)
}
return append(args, "/persistent:no")
}
// summarize pairs an exec error with the first line of command output, which is
// where net and mount put the actual diagnosis.
func summarize(err error, out []byte) string {
text := strings.TrimSpace(string(out))
if text == "" {
return err.Error()
}
if idx := strings.IndexByte(text, '\n'); idx >= 0 {
text = text[:idx]
}
return fmt.Sprintf("%v: %s", err, text)
}
func usernameNote(username string) string {
if username == "" {
return "(none, using ambient credentials)"
}
return username
}
// joinBase resolves a relative path against a base that may be a UNC path.
// filepath.Join is correct on Windows and would mangle the leading \\ anywhere
// else, so UNC bases are joined with backslashes explicitly.
func joinBase(base string, elem ...string) string {
if len(elem) == 0 {
return base
}
if strings.HasPrefix(base, `\\`) {
parts := append([]string{strings.TrimRight(base, `\`)}, elem...)
return strings.Join(parts, `\`)
}
return filepath.Join(append([]string{base}, elem...)...)
}

353
pkg/nas/connector_test.go Normal file
View File

@ -0,0 +1,353 @@
package nas
import (
"context"
"fmt"
"os"
"path/filepath"
"strings"
"testing"
"time"
)
// recordingRunner captures the commands a connector would run instead of
// running them. This is what lets the Windows path be tested from a Linux
// development machine: the decision logic and the exact argv are asserted
// without ever invoking net.exe.
type recordingRunner struct {
calls [][]string
output []byte
err error
}
func (r *recordingRunner) run(_ context.Context, name string, args ...string) ([]byte, error) {
r.calls = append(r.calls, append([]string{name}, args...))
return r.output, r.err
}
func (r *recordingRunner) lastCall() string {
if len(r.calls) == 0 {
return ""
}
return strings.Join(r.calls[len(r.calls)-1], " ")
}
func remoteLocation(t *testing.T, raw string) Location {
t.Helper()
loc, err := ParseLocation(raw)
if err != nil {
t.Fatalf("ParseLocation(%q) failed: %v", raw, err)
}
return loc
}
func TestConnectorSelection(t *testing.T) {
remote := remoteLocation(t, `\\nas\dci\streams`)
local := remoteLocation(t, "/mnt/dci/streams")
cases := []struct {
name string
loc Location
goos string
want string
}{
{"windows share", remote, "windows", "net use"},
{"linux share", remote, "linux", "mount"},
{"darwin share", remote, "darwin", "unsupported"},
{"local path on windows", local, "windows", "local"},
{"local path on linux", local, "linux", "local"},
// A already-mounted path needs no network step even where shares are
// unsupported, so it must not fall through to the error connector.
{"local path on darwin", local, "darwin", "local"},
}
for _, tc := range cases {
if got := connectorFor(tc.loc, tc.goos).Name(); got != tc.want {
t.Errorf("%s: connectorFor = %q, want %q", tc.name, got, tc.want)
}
}
}
func TestWindowsConnectorAttach(t *testing.T) {
runner := &recordingRunner{}
c := &windowsConnector{run: runner.run}
loc := remoteLocation(t, `\\HomeLabNAS\dci\streams`)
att, err := c.Attach(context.Background(), loc, NASConfig{Username: "NASAdmin", Password: "secret"})
if err != nil {
t.Fatalf("Attach failed: %v", err)
}
// The base path is the full UNC path; net use authenticates only the share.
if want := `\\HomeLabNAS\dci\streams`; att.BasePath != want {
t.Errorf("BasePath = %q, want %q", att.BasePath, want)
}
if want := `net use \\HomeLabNAS\dci /user:NASAdmin secret /persistent:no`; runner.lastCall() != want {
t.Errorf("command = %q, want %q", runner.lastCall(), want)
}
if !att.Owned {
t.Error("a successful net use should be owned so it can be released on shutdown")
}
}
func TestWindowsConnectorOmitsCredentialsWhenUnset(t *testing.T) {
runner := &recordingRunner{}
c := &windowsConnector{run: runner.run}
if _, err := c.Attach(context.Background(), remoteLocation(t, `\\nas\dci`), NASConfig{}); err != nil {
t.Fatalf("Attach failed: %v", err)
}
if want := `net use \\nas\dci /persistent:no`; runner.lastCall() != want {
t.Errorf("command = %q, want %q", runner.lastCall(), want)
}
}
func TestWindowsConnectorContinuesWhenAlreadyConnected(t *testing.T) {
// net use fails when a connection already exists, but the share is usable;
// refusing to run would be wrong. The write probe is the real verdict.
runner := &recordingRunner{
err: fmt.Errorf("exit status 2"),
output: []byte("The network connection could not be found.\nMore help is available..."),
}
c := &windowsConnector{run: runner.run}
att, err := c.Attach(context.Background(), remoteLocation(t, `\\nas\dci`), NASConfig{})
if err != nil {
t.Fatalf("Attach should tolerate a net use failure, got: %v", err)
}
if att.Owned {
t.Error("a failed net use must not be marked owned, or Detach would drop someone else's connection")
}
notes := strings.Join(att.Notes, "\n")
if !strings.Contains(notes, "net use did not complete") {
t.Errorf("expected the failure to be recorded in notes:\n%s", notes)
}
// Only the first line of output is worth surfacing.
if strings.Contains(notes, "More help is available") {
t.Errorf("notes should not carry the full usage dump:\n%s", notes)
}
}
func TestWindowsConnectorDetachOnlyReleasesOwned(t *testing.T) {
runner := &recordingRunner{}
c := &windowsConnector{run: runner.run}
if err := c.Detach(context.Background(), Attachment{share: `\\nas\dci`, Owned: false}); err != nil {
t.Fatalf("Detach failed: %v", err)
}
if len(runner.calls) != 0 {
t.Errorf("must not disconnect a share this process did not connect, ran: %v", runner.calls)
}
if err := c.Detach(context.Background(), Attachment{share: `\\nas\dci`, Owned: true}); err != nil {
t.Fatalf("Detach failed: %v", err)
}
if want := `net use \\nas\dci /delete`; runner.lastCall() != want {
t.Errorf("command = %q, want %q", runner.lastCall(), want)
}
}
// writeMountTable stages a /proc/mounts-format file.
func writeMountTable(t *testing.T, lines ...string) string {
t.Helper()
path := filepath.Join(t.TempDir(), "mounts")
body := strings.Join(lines, "\n") + "\n"
if err := os.WriteFile(path, []byte(body), 0644); err != nil {
t.Fatalf("failed to write mount table: %v", err)
}
return path
}
func TestLinuxConnectorResolvesExistingMount(t *testing.T) {
mounts := writeMountTable(t,
"sysfs /sys sysfs rw,nosuid 0 0",
"//homelabnas/dci /mnt/dci cifs rw,relatime,username=nasadmin 0 0",
)
c := &linuxConnector{run: (&recordingRunner{}).run, mountsPath: mounts}
att, err := c.Attach(context.Background(), remoteLocation(t, `\\HomeLabNAS\dci\streams`), NASConfig{})
if err != nil {
t.Fatalf("Attach failed: %v", err)
}
// Server and share names are case-insensitive on SMB, so the configured
// \\HomeLabNAS\dci must match a //homelabnas/dci mount.
if want := "/mnt/dci/streams"; att.BasePath != want {
t.Errorf("BasePath = %q, want %q", att.BasePath, want)
}
if att.Owned {
t.Error("a pre-existing mount must not be owned, or shutdown would unmount it")
}
}
func TestLinuxConnectorPrefersDeepestMount(t *testing.T) {
// Both mounts could serve the destination; the more specific one is the
// only one guaranteed to actually contain it.
mounts := writeMountTable(t,
"//nas/dci /mnt/dci cifs rw 0 0",
"//nas/dci/streams /mnt/streams cifs rw 0 0",
)
c := &linuxConnector{run: (&recordingRunner{}).run, mountsPath: mounts}
att, err := c.Attach(context.Background(), remoteLocation(t, `\\nas\dci\streams\2026`), NASConfig{})
if err != nil {
t.Fatalf("Attach failed: %v", err)
}
if want := "/mnt/streams/2026"; att.BasePath != want {
t.Errorf("BasePath = %q, want %q", att.BasePath, want)
}
}
func TestLinuxConnectorMatchesNFSAndEscapedPoints(t *testing.T) {
mounts := writeMountTable(t,
`nas:/dci /mnt/my\040share nfs4 rw 0 0`,
)
c := &linuxConnector{run: (&recordingRunner{}).run, mountsPath: mounts}
att, err := c.Attach(context.Background(), remoteLocation(t, `\\nas\dci\streams`), NASConfig{})
if err != nil {
t.Fatalf("Attach failed: %v", err)
}
if want := "/mnt/my share/streams"; att.BasePath != want {
t.Errorf("BasePath = %q, want %q", att.BasePath, want)
}
}
func TestLinuxConnectorIgnoresNonNetworkMounts(t *testing.T) {
// A local directory that merely looks like the share must not match.
mounts := writeMountTable(t, "//nas/dci /mnt/dci ext4 rw 0 0")
c := &linuxConnector{run: (&recordingRunner{}).run, mountsPath: mounts}
if _, err := c.Attach(context.Background(), remoteLocation(t, `\\nas\dci`), NASConfig{}); err == nil {
t.Fatal("expected an ext4 mount not to satisfy a network share")
}
}
func TestLinuxConnectorUnmountedGivesActionableError(t *testing.T) {
mounts := writeMountTable(t, "sysfs /sys sysfs rw 0 0")
c := &linuxConnector{run: (&recordingRunner{}).run, mountsPath: mounts}
_, err := c.Attach(context.Background(), remoteLocation(t, `\\nas\dci\streams`), NASConfig{Username: "admin"})
if err == nil {
t.Fatal("expected an unmounted share to fail")
}
msg := err.Error()
for _, want := range []string{"//nas/dci", "mount -t cifs", "NAS_AUTO_MOUNT", "NAS_OUTPUT_PATH"} {
if !strings.Contains(msg, want) {
t.Errorf("error should mention %q:\n%s", want, msg)
}
}
// The remedy must not echo the real password.
if strings.Contains(msg, "hunter2") {
t.Error("error message leaked a credential")
}
}
func TestLinuxConnectorAutoMountRequiresRoot(t *testing.T) {
mounts := writeMountTable(t, "sysfs /sys sysfs rw 0 0")
runner := &recordingRunner{}
c := &linuxConnector{run: runner.run, mountsPath: mounts, geteuid: func() int { return 1000 }}
_, err := c.Attach(context.Background(), remoteLocation(t, `\\nas\dci`), NASConfig{AutoMount: true})
if err == nil || !strings.Contains(err.Error(), "requires root") {
t.Fatalf("expected a root requirement error, got: %v", err)
}
if len(runner.calls) != 0 {
t.Errorf("must not attempt mount without privileges, ran: %v", runner.calls)
}
}
func TestLinuxConnectorAutoMount(t *testing.T) {
mounts := writeMountTable(t, "sysfs /sys sysfs rw 0 0")
runner := &recordingRunner{}
point := filepath.Join(t.TempDir(), "mnt", "dci")
c := &linuxConnector{run: runner.run, mountsPath: mounts, geteuid: func() int { return 0 }}
cfg := NASConfig{AutoMount: true, MountPoint: point, Username: "admin", Password: "hunter2", Timeout: 5 * time.Second}
att, err := c.Attach(context.Background(), remoteLocation(t, `\\nas\dci\streams`), cfg)
if err != nil {
t.Fatalf("Attach failed: %v", err)
}
if want := filepath.Join(point, "streams"); att.BasePath != want {
t.Errorf("BasePath = %q, want %q", att.BasePath, want)
}
if !att.Owned || att.mountPoint != point {
t.Errorf("a self-created mount must be owned and recorded, got owned=%v point=%q", att.Owned, att.mountPoint)
}
call := runner.lastCall()
if !strings.HasPrefix(call, "mount -t cifs //nas/dci "+point+" -o rw,credentials=") {
t.Errorf("unexpected mount command: %q", call)
}
// The password must reach mount.cifs through a credentials file, never
// through argv where any local user could read it from the process table.
if strings.Contains(call, "hunter2") {
t.Errorf("password leaked into the mount command line: %q", call)
}
// The staged credentials file must not outlive the mount call.
credPath := call[strings.Index(call, "credentials=")+len("credentials="):]
if _, err := os.Stat(credPath); !os.IsNotExist(err) {
t.Errorf("credentials file %s should have been removed, stat err = %v", credPath, err)
}
if err := c.Detach(context.Background(), att); err != nil {
t.Fatalf("Detach failed: %v", err)
}
if want := "umount " + point; runner.lastCall() != want {
t.Errorf("detach command = %q, want %q", runner.lastCall(), want)
}
}
func TestLinuxConnectorDetachLeavesForeignMounts(t *testing.T) {
runner := &recordingRunner{}
c := &linuxConnector{run: runner.run, mountsPath: writeMountTable(t)}
if err := c.Detach(context.Background(), Attachment{BasePath: "/mnt/dci", Owned: false}); err != nil {
t.Fatalf("Detach failed: %v", err)
}
if len(runner.calls) != 0 {
t.Errorf("must not unmount a share the host mounted, ran: %v", runner.calls)
}
}
func TestLocalConnectorRejectsWindowsPathsOffWindows(t *testing.T) {
c := &localConnector{goos: "linux"}
for _, raw := range []string{`C:\data\streams`, `data\streams`} {
loc, err := ParseLocation(raw)
if err != nil {
t.Fatalf("ParseLocation(%q) failed: %v", raw, err)
}
if _, err := c.Attach(context.Background(), loc, NASConfig{}); err == nil {
t.Errorf("%q should be rejected on Linux: backslashes are legal filename characters there, "+
"so it would silently create a local directory instead of writing to the NAS", raw)
}
}
}
func TestLocalConnectorAcceptsWindowsPathsOnWindows(t *testing.T) {
c := &localConnector{goos: "windows"}
loc, err := ParseLocation(`C:\data\streams`)
if err != nil {
t.Fatalf("ParseLocation failed: %v", err)
}
if _, err := c.Attach(context.Background(), loc, NASConfig{}); err != nil {
t.Errorf("a drive path is valid on Windows: %v", err)
}
}
func TestUnsupportedConnectorExplainsItself(t *testing.T) {
c := &unsupportedConnector{goos: "plan9"}
_, err := c.Attach(context.Background(), remoteLocation(t, `\\nas\dci`), NASConfig{})
if err == nil {
t.Fatal("expected an error on an unsupported platform")
}
if !strings.Contains(err.Error(), "plan9") || !strings.Contains(err.Error(), "NAS_OUTPUT_PATH") {
t.Errorf("error should name the platform and the remedy: %v", err)
}
}

144
pkg/nas/location.go Normal file
View File

@ -0,0 +1,144 @@
package nas
import (
"fmt"
"path/filepath"
"regexp"
"strings"
)
// Location is a configured NAS destination parsed into a host-neutral form.
//
// The same destination is spelled differently depending on where you're
// standing: Windows wants \\HomeLabNAS\dci\streams, a Linux host reaches the
// same bytes through a mount point like /mnt/dci/streams, and either host may
// simply be handed a plain local directory. Parsing happens once, up front,
// and platform-specific rendering is left to the Connector implementations.
type Location struct {
// Raw is the destination exactly as configured, for error messages.
Raw string
// Server is the host serving the share; empty for a plain local path.
Server string
// Share is the top-level export name; empty for a plain local path.
Share string
// Sub holds the path components below the share, if any.
Sub []string
}
// driveLetter matches a Windows drive-qualified path such as C:\data.
var driveLetter = regexp.MustCompile(`^[A-Za-z]:[\\/]`)
// ParseLocation accepts every spelling of a destination this project supports:
// Windows UNC (\\server\share\dir), the SMB source form (//server/share/dir),
// a smb:// URL, or a plain local/mounted directory. It does no I/O and makes no
// assumptions about the host OS, so the result is safe to compute anywhere.
func ParseLocation(raw string) (Location, error) {
trimmed := strings.TrimSpace(raw)
if trimmed == "" {
return Location{}, fmt.Errorf("NAS path is empty")
}
loc := Location{Raw: trimmed}
remainder, isRemote := stripRemotePrefix(trimmed)
if !isRemote {
return loc, nil
}
parts := splitPathComponents(remainder)
if len(parts) < 2 {
return Location{}, fmt.Errorf(
"%q names a server but no share; expected a form like \\\\server\\share\\folder", trimmed)
}
loc.Server = parts[0]
loc.Share = parts[1]
loc.Sub = parts[2:]
return loc, nil
}
// stripRemotePrefix reports whether raw names a network share and returns the
// server/share/subpath portion when it does. A leading // is treated as a share
// rather than a local path: POSIX leaves paths beginning with exactly two
// slashes implementation-defined, and nobody configures a NAS destination that
// way by accident.
func stripRemotePrefix(raw string) (string, bool) {
lower := strings.ToLower(raw)
switch {
case strings.HasPrefix(lower, "smb://"):
return raw[len("smb://"):], true
case strings.HasPrefix(lower, "cifs://"):
return raw[len("cifs://"):], true
case strings.HasPrefix(raw, `\\`):
return raw[2:], true
case strings.HasPrefix(raw, "//"):
return raw[2:], true
default:
return raw, false
}
}
// splitPathComponents splits on either separator and drops empty components, so
// that \\server\share\\folder\ and //server/share/folder agree.
func splitPathComponents(path string) []string {
fields := strings.FieldsFunc(path, func(r rune) bool { return r == '/' || r == '\\' })
out := make([]string, 0, len(fields))
for _, f := range fields {
if f != "" {
out = append(out, f)
}
}
return out
}
// IsRemote reports whether reaching this destination requires attaching a
// network share first, as opposed to writing to a directory that is already
// part of the local filesystem.
func (l Location) IsRemote() bool {
return l.Server != "" && l.Share != ""
}
// UNC renders the destination in Windows form: \\server\share\sub.
func (l Location) UNC() string {
if !l.IsRemote() {
return l.Raw
}
return `\\` + strings.Join(append([]string{l.Server, l.Share}, l.Sub...), `\`)
}
// ShareUNC renders just the share in Windows form: \\server\share. This is the
// unit `net use` authenticates against; the subpath below it is an ordinary
// directory on the connected share.
func (l Location) ShareUNC() string {
if !l.IsRemote() {
return ""
}
return `\\` + l.Server + `\` + l.Share
}
// SMBSource renders the share in the //server/share form used by mount tables
// and by mount.cifs. It deliberately omits Sub: that part is a path *within*
// the mounted share, not part of the share's identity.
func (l Location) SMBSource() string {
if !l.IsRemote() {
return ""
}
return "//" + l.Server + "/" + l.Share
}
// SubPath renders the components below the share using the host's separator, or
// "" when the destination is the share root.
func (l Location) SubPath() string {
if len(l.Sub) == 0 {
return ""
}
return filepath.Join(l.Sub...)
}
// String describes the destination for logs and check output.
func (l Location) String() string {
if l.IsRemote() {
return l.UNC()
}
return l.Raw
}

123
pkg/nas/location_test.go Normal file
View File

@ -0,0 +1,123 @@
package nas
import (
"strings"
"testing"
)
func TestParseLocationRemoteForms(t *testing.T) {
// Every spelling of the same destination must parse identically, so that
// an operator can configure it the way their platform's documentation
// describes without changing behaviour.
equivalent := []string{
`\\HomeLabNAS\dci\streams`,
`//HomeLabNAS/dci/streams`,
`smb://HomeLabNAS/dci/streams`,
`SMB://HomeLabNAS/dci/streams`,
`cifs://HomeLabNAS/dci/streams`,
`\\HomeLabNAS\dci\streams\`,
`\\HomeLabNAS\\dci\\streams`,
}
for _, raw := range equivalent {
loc, err := ParseLocation(raw)
if err != nil {
t.Errorf("ParseLocation(%q) failed: %v", raw, err)
continue
}
if !loc.IsRemote() {
t.Errorf("ParseLocation(%q) should be remote", raw)
}
if loc.Server != "HomeLabNAS" || loc.Share != "dci" {
t.Errorf("ParseLocation(%q) = server %q share %q, want HomeLabNAS/dci", raw, loc.Server, loc.Share)
}
if len(loc.Sub) != 1 || loc.Sub[0] != "streams" {
t.Errorf("ParseLocation(%q) sub = %v, want [streams]", raw, loc.Sub)
}
}
}
func TestParseLocationRendering(t *testing.T) {
loc, err := ParseLocation(`//HomeLabNAS/dci/streams/2026`)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if got, want := loc.UNC(), `\\HomeLabNAS\dci\streams\2026`; got != want {
t.Errorf("UNC() = %q, want %q", got, want)
}
if got, want := loc.ShareUNC(), `\\HomeLabNAS\dci`; got != want {
t.Errorf("ShareUNC() = %q, want %q", got, want)
}
if got, want := loc.SMBSource(), "//HomeLabNAS/dci"; got != want {
t.Errorf("SMBSource() = %q, want %q", got, want)
}
}
func TestParseLocationShareRoot(t *testing.T) {
loc, err := ParseLocation(`\\nas\media`)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if len(loc.Sub) != 0 {
t.Errorf("expected no subpath, got %v", loc.Sub)
}
if got, want := loc.UNC(), `\\nas\media`; got != want {
t.Errorf("UNC() = %q, want %q", got, want)
}
}
func TestParseLocationLocalPaths(t *testing.T) {
for _, raw := range []string{"/mnt/nas/streams", "data/streams", `C:\data\streams`, "/mnt"} {
loc, err := ParseLocation(raw)
if err != nil {
t.Errorf("ParseLocation(%q) failed: %v", raw, err)
continue
}
if loc.IsRemote() {
t.Errorf("ParseLocation(%q) should not be treated as a network share", raw)
}
if loc.Raw != raw {
t.Errorf("ParseLocation(%q).Raw = %q", raw, loc.Raw)
}
}
}
func TestParseLocationRejectsIncomplete(t *testing.T) {
for _, raw := range []string{"", " ", `\\`, `\\server`, "//server", "smb://server"} {
if _, err := ParseLocation(raw); err == nil {
t.Errorf("ParseLocation(%q) should have failed", raw)
}
}
}
func TestParseLocationTrimsWhitespace(t *testing.T) {
loc, err := ParseLocation(" \\\\nas\\dci ")
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if loc.Server != "nas" || loc.Share != "dci" {
t.Errorf("whitespace not trimmed: server %q share %q", loc.Server, loc.Share)
}
}
func TestJoinBaseKeepsUNCSeparators(t *testing.T) {
// filepath.Join would produce \\nas\dci/event on Linux, which is not a
// path Windows would accept if the value were ever written to state.
got := joinBase(`\\nas\dci`, "event", "1080p")
if want := `\\nas\dci\event\1080p`; got != want {
t.Errorf("joinBase UNC = %q, want %q", got, want)
}
if got, want := joinBase(`\\nas\dci\`, "event"), `\\nas\dci\event`; got != want {
t.Errorf("joinBase trailing separator = %q, want %q", got, want)
}
if got := joinBase("/mnt/dci"); got != "/mnt/dci" {
t.Errorf("joinBase with no elements = %q", got)
}
if got, want := joinBase("/mnt/dci", "event"), "/mnt/dci/event"; !strings.HasSuffix(got, want) {
t.Errorf("joinBase local = %q, want suffix %q", got, want)
}
}

135
pkg/nas/mounts.go Normal file
View File

@ -0,0 +1,135 @@
package nas
import (
"bufio"
"fmt"
"os"
"strconv"
"strings"
)
// procMountsPath is the kernel's view of what is mounted in this process's
// mount namespace, which is the right question to ask inside a container too.
const procMountsPath = "/proc/mounts"
// mountEntry is one line of the mount table.
type mountEntry struct {
Source string
Point string
FSType string
}
// networkFilesystems are the mount types that can plausibly be serving a NAS
// share. Restricting the search avoids matching, say, a local directory that
// merely happens to be named like a server.
var networkFilesystems = map[string]bool{
"cifs": true,
"smb3": true,
"smbfs": true,
"nfs": true,
"nfs4": true,
"fuse.smbnetfs": true,
}
// readMountTable parses /proc/mounts (or any file in its format). Fields are
// whitespace-separated: source, mount point, type, options, dump, pass.
func readMountTable(path string) ([]mountEntry, error) {
f, err := os.Open(path)
if err != nil {
return nil, fmt.Errorf("failed to read mount table %s: %w", path, err)
}
defer f.Close()
var entries []mountEntry
scanner := bufio.NewScanner(f)
for scanner.Scan() {
fields := strings.Fields(scanner.Text())
if len(fields) < 3 {
continue
}
entries = append(entries, mountEntry{
Source: unescapeMountField(fields[0]),
Point: unescapeMountField(fields[1]),
FSType: fields[2],
})
}
if err := scanner.Err(); err != nil {
return nil, fmt.Errorf("failed to scan mount table %s: %w", path, err)
}
return entries, nil
}
// unescapeMountField decodes the octal escapes the kernel uses for characters
// that would otherwise break the whitespace-separated format — most often \040
// for a space in a mount point.
func unescapeMountField(field string) string {
if !strings.Contains(field, `\`) {
return field
}
var b strings.Builder
for i := 0; i < len(field); i++ {
if field[i] == '\\' && i+3 < len(field) {
if v, err := strconv.ParseUint(field[i+1:i+4], 8, 8); err == nil {
b.WriteByte(byte(v))
i += 3
continue
}
}
b.WriteByte(field[i])
}
return b.String()
}
// findMount locates the mount serving loc and returns its mount point plus the
// path components still to be appended below it.
//
// The search runs deepest-first because a share may be mounted at any level:
// //nas/dci might be mounted at /mnt/dci, or //nas/dci/streams mounted directly
// at /mnt/streams. Preferring the longest matching source keeps both working
// and avoids pointing at a parent that the mount does not actually cover.
func findMount(entries []mountEntry, loc Location) (point string, rest []string, ok bool) {
if !loc.IsRemote() {
return "", nil, false
}
for depth := len(loc.Sub); depth >= 0; depth-- {
for _, candidate := range mountSourceForms(loc, depth) {
for _, entry := range entries {
if !networkFilesystems[entry.FSType] {
continue
}
if normalizeMountSource(entry.Source) == candidate {
return entry.Point, loc.Sub[depth:], true
}
}
}
}
return "", nil, false
}
// mountSourceForms renders the share, plus its first `depth` subdirectories, in
// the spellings a mount table might use for it.
func mountSourceForms(loc Location, depth int) []string {
segments := append([]string{loc.Share}, loc.Sub[:depth]...)
tail := strings.Join(segments, "/")
return []string{
normalizeMountSource("//" + loc.Server + "/" + tail), // cifs/smb3
normalizeMountSource(loc.Server + ":/" + tail), // nfs
}
}
// normalizeMountSource makes sources comparable: SMB servers and shares are
// case-insensitive, separators may be either slash, and a trailing slash is not
// meaningful.
func normalizeMountSource(source string) string {
s := strings.ReplaceAll(source, `\`, "/")
s = strings.ToLower(strings.TrimSpace(s))
if len(s) > 1 {
s = strings.TrimRight(s, "/")
}
return s
}

96
pkg/nas/mounts_test.go Normal file
View File

@ -0,0 +1,96 @@
package nas
import (
"os"
"testing"
)
func TestUnescapeMountField(t *testing.T) {
cases := map[string]string{
`/mnt/plain`: "/mnt/plain",
`/mnt/my\040share`: "/mnt/my share",
`/mnt/tab\011here`: "/mnt/tab\there",
`/mnt/back\134slash`: `/mnt/back\slash`,
`//nas/dci`: "//nas/dci",
`/mnt/not\9octal`: `/mnt/not\9octal`,
`/mnt/trailing\`: `/mnt/trailing\`,
}
for in, want := range cases {
if got := unescapeMountField(in); got != want {
t.Errorf("unescapeMountField(%q) = %q, want %q", in, got, want)
}
}
}
func TestNormalizeMountSource(t *testing.T) {
cases := map[string]string{
"//NAS/DCI": "//nas/dci",
"//nas/dci/": "//nas/dci",
`\\nas\dci`: "//nas/dci",
" //nas/dci ": "//nas/dci",
"/": "/",
}
for in, want := range cases {
if got := normalizeMountSource(in); got != want {
t.Errorf("normalizeMountSource(%q) = %q, want %q", in, got, want)
}
}
}
func TestReadMountTableSkipsMalformedLines(t *testing.T) {
path := writeMountTable(t,
"source-only",
"source /point-but-no-type",
"",
"//nas/dci /mnt/dci cifs rw 0 0",
)
entries, err := readMountTable(path)
if err != nil {
t.Fatalf("readMountTable failed: %v", err)
}
if len(entries) != 1 {
t.Fatalf("expected 1 usable entry, got %d: %v", len(entries), entries)
}
if entries[0].Source != "//nas/dci" || entries[0].Point != "/mnt/dci" || entries[0].FSType != "cifs" {
t.Errorf("unexpected entry: %+v", entries[0])
}
}
func TestReadMountTableMissingFile(t *testing.T) {
if _, err := readMountTable("/definitely/not/a/mount/table"); err == nil {
t.Error("expected an error for a missing mount table")
}
}
// TestReadRealMountTable closes the loop between the fixtures above and the
// kernel's actual output: the parser must cope with whatever this host has
// mounted, not just hand-written examples.
func TestReadRealMountTable(t *testing.T) {
if _, err := os.Stat(procMountsPath); err != nil {
t.Skipf("%s is not available on this platform", procMountsPath)
}
entries, err := readMountTable(procMountsPath)
if err != nil {
t.Fatalf("failed to parse the host's %s: %v", procMountsPath, err)
}
if len(entries) == 0 {
t.Fatalf("%s parsed to zero entries, which cannot be right", procMountsPath)
}
foundRoot := false
for _, e := range entries {
if e.Point == "" || e.FSType == "" {
t.Errorf("entry parsed with empty fields: %+v", e)
}
if e.Point == "/" {
foundRoot = true
}
}
if !foundRoot {
t.Error("expected the root filesystem in the host's mount table")
}
}

View File

@ -6,31 +6,168 @@ import (
"io"
"log"
"os"
"os/exec"
"path/filepath"
"strings"
"sync"
)
// NASService writes to a NAS destination without caring how this host reaches
// it. Parsing the destination and attaching it are delegated to a Location and
// a Connector; everything below operates on the resolved base path, which is
// always an ordinary path this host's filesystem understands.
type NASService struct {
Config NASConfig
location Location
connector Connector
mu sync.RWMutex
attached Attachment
connected bool
}
func NewNASService(config NASConfig) *NASService {
nt := &NASService{
Config: config,
}
// Establish network connection with credentials before accessing the path
if err := nt.EstablishConnection(); err != nil {
log.Fatalf("Failed to establish network connection to %s: %v", nt.Config.Path, err)
}
err := nt.EnsureDirectoryExists(nt.Config.Path)
// New parses the configured destination and selects a connection strategy for
// the running host. It performs no I/O and does not touch the network, so it is
// safe to call before deciding whether a transfer will actually happen; call
// Connect to attach the destination.
func New(config NASConfig) (*NASService, error) {
loc, err := ParseLocation(config.Path)
if err != nil {
log.Fatalf("Failed to create directory %s: %v", nt.Config.Path, err)
return nil, err
}
return nt
return &NASService{
Config: config,
location: loc,
connector: defaultConnector(loc),
}, nil
}
// Connect attaches the destination, ensures the target directory exists, and
// proves it is writable. The write probe matters: on both platforms a share can
// be reachable and still reject writes, and finding that out at startup beats
// discovering it mid-recording.
func (nt *NASService) Connect(ctx context.Context) error {
att, err := nt.connector.Attach(ctx, nt.location, nt.Config)
if err != nil {
return err
}
nt.mu.Lock()
nt.attached = att
nt.mu.Unlock()
// From here on the attachment exists, so any failure has to undo it —
// otherwise a mount this process created would outlive the failed startup
// and be left behind on the host.
if err := nt.EnsureDirectoryExists(att.BasePath); err != nil {
nt.abandon(ctx, att)
return fmt.Errorf("%s is reachable but %s is unusable: %w", nt.location, att.BasePath, err)
}
if err := nt.writeProbe(att.BasePath); err != nil {
nt.abandon(ctx, att)
return err
}
nt.mu.Lock()
nt.connected = true
nt.mu.Unlock()
log.Printf("Connected to NAS at %s", att.BasePath)
return nil
}
// abandon releases a half-finished attachment after Connect fails. The
// original error is the one worth reporting, so a teardown failure is only
// logged.
func (nt *NASService) abandon(ctx context.Context, att Attachment) {
// Keep the notes: they explain how far the host got, which is exactly what
// the caller needs to report. Everything else is cleared so the service
// cannot be mistaken for usable and Detach cannot run twice.
nt.mu.Lock()
nt.attached = Attachment{Notes: att.Notes}
nt.connected = false
nt.mu.Unlock()
if err := nt.connector.Detach(ctx, att); err != nil {
log.Printf("Warning: failed to release NAS attachment after a failed connect: %v", err)
}
}
// writeProbe creates and removes a file to confirm the destination accepts
// writes.
func (nt *NASService) writeProbe(basePath string) error {
testFile := joinBase(basePath, ".connection_test")
f, err := os.Create(testFile)
if err != nil {
return fmt.Errorf("%s is not writable: %w", basePath, err)
}
f.Close()
if err := os.Remove(testFile); err != nil {
log.Printf("Warning: failed to remove connection test file %s: %v", testFile, err)
}
return nil
}
// Disconnect releases only what Connect established. A share that was already
// mounted or already connected when the recorder started is left exactly as it
// was found.
func (nt *NASService) Disconnect(ctx context.Context) error {
nt.mu.Lock()
att := nt.attached
nt.attached = Attachment{}
nt.connected = false
nt.mu.Unlock()
if err := nt.connector.Detach(ctx, att); err != nil {
return err
}
if att.Owned {
log.Printf("Released NAS connection to %s", nt.location)
}
return nil
}
// LocalPath is the resolved base directory on this host, valid after Connect.
func (nt *NASService) LocalPath() string {
nt.mu.RLock()
defer nt.mu.RUnlock()
return nt.attached.BasePath
}
// Join resolves elements against the NAS base path using whatever separator
// this host's base path requires.
func (nt *NASService) Join(elem ...string) string {
return joinBase(nt.LocalPath(), elem...)
}
// Location returns the parsed destination.
func (nt *NASService) Location() Location { return nt.location }
// ConnectionInfo describes how this host reaches the destination, for check
// output and startup logs. Safe to call before Connect, when it reports only
// what is known from configuration.
func (nt *NASService) ConnectionInfo() []string {
nt.mu.RLock()
defer nt.mu.RUnlock()
info := []string{
"configured: " + nt.location.String(),
"strategy: " + nt.connector.Name(),
}
info = append(info, nt.attached.Notes...)
if nt.attached.BasePath != "" && nt.attached.BasePath != nt.location.Raw {
info = append(info, "resolved path: "+nt.attached.BasePath)
}
return info
}
func (nt *NASService) IsConnected() bool {
nt.mu.RLock()
defer nt.mu.RUnlock()
return nt.connected
}
func (nt *NASService) CopyFile(ctx context.Context, srcPath, destPath string) error {
@ -89,87 +226,10 @@ func (nt *NASService) EnsureDirectoryExists(path string) error {
return nil
}
func (nt *NASService) EstablishConnection() error {
networkPath := nt.ExtractNetworkPath(nt.Config.Path)
if networkPath == "" {
return nil // local path, no network mount needed
}
log.Printf("Establishing network connection to %s with user %s", networkPath, nt.Config.Username)
var cmd *exec.Cmd
if nt.Config.Username != "" && nt.Config.Password != "" {
cmd = exec.Command("net", "use", networkPath, "/user:"+nt.Config.Username, nt.Config.Password, "/persistent:no")
} else {
cmd = exec.Command("net", "use", networkPath, "/persistent:no")
}
output, err := cmd.CombinedOutput()
if err != nil {
return fmt.Errorf("failed to establish network connection: %w\nOutput: %s", err, string(output))
}
log.Printf("Network connection established successfully")
return nil
}
func (nt *NASService) ExtractNetworkPath(fullPath string) string {
// Extract \\server\share from paths like \\server\share\folder\subfolder
if !strings.HasPrefix(fullPath, "\\\\") {
return "" // Not a UNC path
}
parts := strings.Split(fullPath[2:], "\\") // Remove leading \\
if len(parts) < 2 {
return "" // Invalid UNC path
}
return "\\\\" + parts[0] + "\\" + parts[1]
}
func (nt *NASService) TestConnection() error {
testFile := filepath.Join(nt.Config.Path, ".connection_test")
f, err := os.Create(testFile)
if err != nil {
return fmt.Errorf("Failed to create test file: %w", err)
}
f.Close()
os.Remove(testFile)
nt.connected = true
log.Printf("Connected to NAS at %s", nt.Config.Path)
return nil
}
func (nt *NASService) IsConnected() bool {
return nt.connected
}
// Disconnect removes the network connection
func (nt *NASService) Disconnect() error {
networkPath := nt.ExtractNetworkPath(nt.Config.Path)
if networkPath == "" {
return nil // Local path, nothing to disconnect
}
cmd := exec.Command("net", "use", networkPath, "/delete")
output, err := cmd.CombinedOutput()
if err != nil {
log.Printf("Warning: failed to disconnect from %s: %v\nOutput: %s", networkPath, err, string(output))
// Don't return error since this is cleanup
} else {
log.Printf("Disconnected from network path: %s", networkPath)
}
nt.connected = false
return nil
}
// FileExists checks if a file already exists on the NAS and optionally verifies size
// FileExists checks whether a file already exists on the NAS, optionally
// verifying its size. destinationPath is relative to the NAS base path.
func (nt *NASService) FileExists(destinationPath string, expectedSize int64) (bool, error) {
fullDestPath := filepath.Join(nt.Config.Path, destinationPath)
fullDestPath := nt.Join(destinationPath)
destInfo, err := os.Stat(fullDestPath)
if err != nil {
@ -189,9 +249,9 @@ func (nt *NASService) FileExists(destinationPath string, expectedSize int64) (bo
return true, nil
}
// GetFileSize returns the size of a file on the NAS
// GetFileSize returns the size of a file on the NAS, relative to the base path.
func (nt *NASService) GetFileSize(destinationPath string) (int64, error) {
fullDestPath := filepath.Join(nt.Config.Path, destinationPath)
fullDestPath := nt.Join(destinationPath)
destInfo, err := os.Stat(fullDestPath)
if err != nil {

308
pkg/nas/nas_test.go Normal file
View File

@ -0,0 +1,308 @@
package nas
import (
"context"
"fmt"
"os"
"path/filepath"
"strings"
"testing"
)
// fakeConnector stands in for a platform strategy so the service's own
// behaviour — probing, path resolution, teardown — can be tested without a NAS.
type fakeConnector struct {
basePath string
attachErr error
owned bool
attachCount int
detachCount int
detached Attachment
}
func (f *fakeConnector) Name() string { return "fake" }
func (f *fakeConnector) Attach(context.Context, Location, NASConfig) (Attachment, error) {
f.attachCount++
if f.attachErr != nil {
return Attachment{}, f.attachErr
}
return Attachment{BasePath: f.basePath, Owned: f.owned, Notes: []string{"fake attachment"}}, nil
}
func (f *fakeConnector) Detach(_ context.Context, att Attachment) error {
f.detachCount++
f.detached = att
return nil
}
func newTestService(t *testing.T, configuredPath string, conn Connector) *NASService {
t.Helper()
svc, err := New(NASConfig{Path: configuredPath})
if err != nil {
t.Fatalf("New(%q) failed: %v", configuredPath, err)
}
svc.connector = conn
return svc
}
func TestNewRejectsUnusableDestination(t *testing.T) {
if _, err := New(NASConfig{Path: ""}); err == nil {
t.Error("an empty NAS path should be rejected")
}
if _, err := New(NASConfig{Path: `\\server`}); err == nil {
t.Error("a server without a share should be rejected")
}
}
func TestNewDoesNoIO(t *testing.T) {
// Constructing the service must not touch the network or filesystem, so
// that modes which never transfer can build one harmlessly.
svc, err := New(NASConfig{Path: `\\nas\dci\streams`})
if err != nil {
t.Fatalf("New failed: %v", err)
}
if svc.IsConnected() {
t.Error("service should not report connected before Connect")
}
if svc.LocalPath() != "" {
t.Errorf("no path should be resolved before Connect, got %q", svc.LocalPath())
}
}
func TestConnectResolvesAndProbes(t *testing.T) {
base := filepath.Join(t.TempDir(), "streams")
conn := &fakeConnector{basePath: base}
svc := newTestService(t, `\\nas\dci\streams`, conn)
if err := svc.Connect(context.Background()); err != nil {
t.Fatalf("Connect failed: %v", err)
}
if !svc.IsConnected() {
t.Error("service should report connected")
}
if svc.LocalPath() != base {
t.Errorf("LocalPath = %q, want %q", svc.LocalPath(), base)
}
// Connect creates the destination directory if it is missing.
if info, err := os.Stat(base); err != nil || !info.IsDir() {
t.Errorf("Connect should have created %s: %v", base, err)
}
// The write probe must clean up after itself.
if _, err := os.Stat(filepath.Join(base, ".connection_test")); !os.IsNotExist(err) {
t.Errorf("probe file should have been removed, stat err = %v", err)
}
}
func TestConnectSurfacesAttachFailure(t *testing.T) {
conn := &fakeConnector{attachErr: fmt.Errorf("share is not mounted")}
svc := newTestService(t, `\\nas\dci`, conn)
err := svc.Connect(context.Background())
if err == nil || !strings.Contains(err.Error(), "not mounted") {
t.Fatalf("expected the connector's error to surface, got: %v", err)
}
if svc.IsConnected() {
t.Error("a failed attach must not leave the service marked connected")
}
}
func TestConnectFailsWhenDestinationUnwritable(t *testing.T) {
// A file where the directory should be: reachable, but unusable.
blocker := filepath.Join(t.TempDir(), "blocker")
if err := os.WriteFile(blocker, []byte("x"), 0644); err != nil {
t.Fatalf("setup failed: %v", err)
}
svc := newTestService(t, "/tmp/whatever", &fakeConnector{basePath: filepath.Join(blocker, "streams")})
if err := svc.Connect(context.Background()); err == nil {
t.Fatal("expected an unusable destination to fail the connect")
}
if svc.IsConnected() {
t.Error("service should not report connected after a failed probe")
}
}
func TestFailedProbeReleasesTheAttachment(t *testing.T) {
// A mount created by Attach must not outlive a failed Connect, or a failed
// startup leaves a mount behind on the host.
blocker := filepath.Join(t.TempDir(), "blocker")
if err := os.WriteFile(blocker, []byte("x"), 0644); err != nil {
t.Fatalf("setup failed: %v", err)
}
conn := &fakeConnector{basePath: filepath.Join(blocker, "streams"), owned: true}
svc := newTestService(t, `\\nas\dci`, conn)
if err := svc.Connect(context.Background()); err == nil {
t.Fatal("expected Connect to fail")
}
if conn.detachCount != 1 {
t.Errorf("Detach called %d times after a failed connect, want 1", conn.detachCount)
}
if !conn.detached.Owned {
t.Error("Detach should have been told the attachment was owned")
}
if svc.LocalPath() != "" {
t.Errorf("a released attachment should expose no path, got %q", svc.LocalPath())
}
// The diagnosis of how far it got must survive for the caller to report.
if info := strings.Join(svc.ConnectionInfo(), "\n"); !strings.Contains(info, "fake attachment") {
t.Errorf("connection notes should survive a failed connect:\n%s", info)
}
// A later Disconnect must not detach a second time.
if err := svc.Disconnect(context.Background()); err != nil {
t.Fatalf("Disconnect failed: %v", err)
}
if conn.detachCount != 2 || conn.detached.Owned {
t.Errorf("Disconnect after release should be a no-op teardown, detachCount=%d owned=%v",
conn.detachCount, conn.detached.Owned)
}
}
func TestJoinUsesResolvedPath(t *testing.T) {
base := t.TempDir()
svc := newTestService(t, `\\nas\dci\streams`, &fakeConnector{basePath: base})
if err := svc.Connect(context.Background()); err != nil {
t.Fatalf("Connect failed: %v", err)
}
// Crucially the join is against the resolved path, not the configured UNC
// path — that is the whole point of the indirection.
if got, want := svc.Join("event", "1080p"), filepath.Join(base, "event", "1080p"); got != want {
t.Errorf("Join = %q, want %q", got, want)
}
}
func TestFileExistsAndGetFileSizeUseResolvedPath(t *testing.T) {
base := t.TempDir()
svc := newTestService(t, `\\nas\dci`, &fakeConnector{basePath: base})
if err := svc.Connect(context.Background()); err != nil {
t.Fatalf("Connect failed: %v", err)
}
rel := filepath.Join("event", "seg.ts")
if err := os.MkdirAll(filepath.Join(base, "event"), 0755); err != nil {
t.Fatalf("setup failed: %v", err)
}
if err := os.WriteFile(filepath.Join(base, rel), []byte("0123456789"), 0644); err != nil {
t.Fatalf("setup failed: %v", err)
}
exists, err := svc.FileExists(rel, 10)
if err != nil || !exists {
t.Errorf("FileExists = %v, %v; want true, nil", exists, err)
}
// A size mismatch means a partial transfer, which should not count.
exists, err = svc.FileExists(rel, 99)
if err != nil || exists {
t.Errorf("FileExists with wrong size = %v, %v; want false, nil", exists, err)
}
exists, err = svc.FileExists(filepath.Join("event", "missing.ts"), 0)
if err != nil || exists {
t.Errorf("FileExists for missing file = %v, %v; want false, nil", exists, err)
}
size, err := svc.GetFileSize(rel)
if err != nil || size != 10 {
t.Errorf("GetFileSize = %d, %v; want 10, nil", size, err)
}
}
func TestDisconnectDelegatesAndResets(t *testing.T) {
conn := &fakeConnector{basePath: t.TempDir(), owned: true}
svc := newTestService(t, `\\nas\dci`, conn)
if err := svc.Connect(context.Background()); err != nil {
t.Fatalf("Connect failed: %v", err)
}
if err := svc.Disconnect(context.Background()); err != nil {
t.Fatalf("Disconnect failed: %v", err)
}
if conn.detachCount != 1 {
t.Errorf("Detach called %d times, want 1", conn.detachCount)
}
if !conn.detached.Owned {
t.Error("Detach should receive the attachment so it knows what it owns")
}
if svc.IsConnected() {
t.Error("service should not report connected after Disconnect")
}
}
func TestConnectionInfoDescribesResolution(t *testing.T) {
base := t.TempDir()
svc := newTestService(t, `\\nas\dci\streams`, &fakeConnector{basePath: base})
// Before connecting, only configuration is known.
before := strings.Join(svc.ConnectionInfo(), "\n")
if !strings.Contains(before, `\\nas\dci\streams`) || !strings.Contains(before, "fake") {
t.Errorf("pre-connect info should name the destination and strategy:\n%s", before)
}
if err := svc.Connect(context.Background()); err != nil {
t.Fatalf("Connect failed: %v", err)
}
after := strings.Join(svc.ConnectionInfo(), "\n")
if !strings.Contains(after, "resolved path: "+base) {
t.Errorf("post-connect info should report where the destination resolved to:\n%s", after)
}
}
func TestVerifyTransferDetectsSizeMismatch(t *testing.T) {
dir := t.TempDir()
src := filepath.Join(dir, "src.ts")
dst := filepath.Join(dir, "dst.ts")
if err := os.WriteFile(src, []byte("0123456789"), 0644); err != nil {
t.Fatalf("setup failed: %v", err)
}
if err := os.WriteFile(dst, []byte("012"), 0644); err != nil {
t.Fatalf("setup failed: %v", err)
}
svc := newTestService(t, dir, &fakeConnector{basePath: dir})
if err := svc.VerifyTransfer(src, dst); err == nil {
t.Error("a truncated destination should fail verification")
}
if err := os.WriteFile(dst, []byte("0123456789"), 0644); err != nil {
t.Fatalf("setup failed: %v", err)
}
if err := svc.VerifyTransfer(src, dst); err != nil {
t.Errorf("matching sizes should verify: %v", err)
}
}
func TestCopyFile(t *testing.T) {
dir := t.TempDir()
src := filepath.Join(dir, "src.ts")
dst := filepath.Join(dir, "dst.ts")
payload := []byte("segment payload")
if err := os.WriteFile(src, payload, 0644); err != nil {
t.Fatalf("setup failed: %v", err)
}
svc := newTestService(t, dir, &fakeConnector{basePath: dir})
if err := svc.CopyFile(context.Background(), src, dst); err != nil {
t.Fatalf("CopyFile failed: %v", err)
}
got, err := os.ReadFile(dst)
if err != nil {
t.Fatalf("failed to read copy: %v", err)
}
if string(got) != string(payload) {
t.Errorf("copied contents = %q, want %q", got, payload)
}
if err := svc.CopyFile(context.Background(), filepath.Join(dir, "missing.ts"), dst); err == nil {
t.Error("copying a missing source should fail")
}
}

View File

@ -39,11 +39,19 @@ func NewProcessingService(eventName string, cfg *config.Config) (*ProcessingServ
Timeout: cfg.NAS.Timeout,
RetryLimit: cfg.NAS.RetryLimit,
VerifySize: true,
AutoMount: cfg.NAS.AutoMount,
MountPoint: cfg.NAS.MountPoint,
}
nasService := nas.NewNASService(nasConfig)
nasService, err := nas.New(nasConfig)
if err != nil {
logging.MustGetFactory().Processing.Error("invalid NAS destination", logging.KeyError, err.Error())
return nil, fmt.Errorf("invalid NAS destination: %w", err)
}
if err := nasService.TestConnection(); err != nil {
connectCtx, cancel := context.WithTimeout(context.Background(), cfg.NAS.Timeout)
defer cancel()
if err := nasService.Connect(connectCtx); err != nil {
logging.MustGetFactory().Processing.Error("failed to connect to NAS", logging.KeyError, err.Error())
return nil, fmt.Errorf("failed to connect to NAS: %w", err)
}
@ -59,9 +67,17 @@ func (ps *ProcessingService) GetSegmentInfo() (map[int]string, error) {
return nil, nil
}
// nasEventPath is the event's directory on the NAS as reached from this host.
// It goes through the NAS service rather than config.GetNASEventPath because
// the configured path may be a UNC path that this host reaches via a mount
// point instead.
func (ps *ProcessingService) nasEventPath() string {
return ps.nas.Join(ps.eventName)
}
func (ps *ProcessingService) GetEventDirs() ([]string, error) {
if ps.eventName == "" {
sourcePath := ps.config.NAS.OutputPath
sourcePath := ps.nas.LocalPath()
dirs, err := os.ReadDir(sourcePath)
if err != nil {
return nil, fmt.Errorf("failed to read directory %s: %w", sourcePath, err)
@ -177,7 +193,7 @@ func (ps *ProcessingService) Start(ctx context.Context) error {
}
func (ps *ProcessingService) GetResolutions() ([]string, error) {
eventPath := ps.config.GetNASEventPath(ps.eventName)
eventPath := ps.nasEventPath()
dirs, err := os.ReadDir(eventPath)
if err != nil {
return nil, fmt.Errorf("failed to read source directory %s: %w", eventPath, err)
@ -198,7 +214,7 @@ func (ps *ProcessingService) GetResolutions() ([]string, error) {
func (ps *ProcessingService) ParseResolutionDirectory(resolution string, ch chan<- SegmentInfo, wg *sync.WaitGroup) {
defer wg.Done()
resolutionPath := utils.SafeJoin(ps.config.GetNASEventPath(ps.eventName), resolution)
resolutionPath := utils.SafeJoin(ps.nasEventPath(), resolution)
files, err := os.ReadDir(resolutionPath)
if err != nil {
log.Printf("Failed to read resolution directory %s: %v", resolutionPath, err)
@ -227,22 +243,14 @@ func (ps *ProcessingService) ParseResolutionDirectory(resolution string, ch chan
func (ps *ProcessingService) AggregateSegmentInfo(ch <-chan SegmentInfo) (map[int]SegmentInfo, error) {
segmentMap := make(map[int]SegmentInfo)
rank := map[string]int{
"1080p": 1,
"720p": 2,
"540p": 3,
"480p": 4,
"450p": 5,
"360p": 6,
"270p": 7,
"240p": 8,
}
// Each sequence number arrives once per downloaded variant; keep the
// highest-resolution copy of each, since that is the one worth stitching
// into the final video.
for segment := range ch {
fmt.Printf("Received segment %s in resolution %s \n", segment.Name, segment.Resolution)
logging.MustGetFactory().Processing.Debug("segment received", logging.KeySegment, segment.Name, logging.KeyResolution, segment.Resolution)
current, exists := segmentMap[segment.SeqNo]
if !exists || rank[segment.Resolution] > rank[current.Resolution] {
if !exists || resolutionHeight(segment.Resolution) > resolutionHeight(current.Resolution) {
segmentMap[segment.SeqNo] = segment
}
}
@ -250,6 +258,24 @@ func (ps *ProcessingService) AggregateSegmentInfo(ch <-chan SegmentInfo) (map[in
return segmentMap, nil
}
// resolutionHeight converts a resolution label such as "1080p" into the pixel
// height it stands for, so variants can be ranked by quality.
//
// The height is parsed rather than looked up in a table: labels come from the
// playlist's own RESOLUTION attribute, so a stream can legitimately offer
// 1440p or 2160p that no fixed table would anticipate. Anything not in
// <height>p form — "unknown", from a playlist with no variants — returns 0 and
// therefore loses to every recognised resolution.
func resolutionHeight(resolution string) int {
trimmed := strings.TrimSuffix(strings.ToLower(strings.TrimSpace(resolution)), "p")
height, err := strconv.Atoi(trimmed)
if err != nil || height < 0 {
return 0
}
return height
}
func (ps *ProcessingService) WriteConcatFile(segmentMap map[int]SegmentInfo) (string, error) {
concatPath := ps.config.GetProcessOutputPath(ps.eventName)
@ -273,7 +299,7 @@ func (ps *ProcessingService) WriteConcatFile(segmentMap map[int]SegmentInfo) (st
for _, seq := range keys {
segment := segmentMap[seq]
filePath := utils.SafeJoin(ps.config.GetNASEventPath(ps.eventName), segment.Resolution, segment.Name)
filePath := utils.SafeJoin(ps.nasEventPath(), segment.Resolution, segment.Name)
line := fmt.Sprintf("file '%s'\n", filePath)
if _, err := f.WriteString(line); err != nil {
return "", fmt.Errorf("failed to write to concat file: %w", err)

View File

@ -1,7 +1,9 @@
package processing
import (
"context"
"m3u8-downloader/pkg/config"
"m3u8-downloader/pkg/nas"
"os"
"path/filepath"
"runtime"
@ -10,6 +12,22 @@ import (
"time"
)
// connectedNAS attaches a NAS service to a local directory. Reaching the NAS
// is platform-specific, but an already-mounted or plain directory resolves the
// same way everywhere, which is what makes this usable as a test double.
func connectedNAS(t *testing.T, path string) *nas.NASService {
t.Helper()
svc, err := nas.New(nas.NASConfig{Path: path, Timeout: 5 * time.Second})
if err != nil {
t.Fatalf("Failed to create NAS service for %s: %v", path, err)
}
if err := svc.Connect(context.Background()); err != nil {
t.Fatalf("Failed to connect NAS service to %s: %v", path, err)
}
return svc
}
func createTestConfig(tempDir string) *config.Config {
return &config.Config{
Core: config.CoreConfig{
@ -87,6 +105,7 @@ func TestProcessingService_GetEventDirs(t *testing.T) {
ps := &ProcessingService{
config: cfg,
eventName: "", // Empty to test directory discovery
nas: connectedNAS(t, cfg.NAS.OutputPath),
}
dirs, err := ps.GetEventDirs()
@ -114,12 +133,15 @@ func TestProcessingService_GetEventDirs(t *testing.T) {
}
func TestProcessingService_GetEventDirs_WithEventName(t *testing.T) {
cfg := createTestConfig("/tmp")
// Scoped to a temp dir rather than /tmp directly: attaching the NAS
// service creates the destination, and tests should not leave one behind.
cfg := createTestConfig(t.TempDir())
eventName := "specific-event"
ps := &ProcessingService{
config: cfg,
eventName: eventName,
nas: connectedNAS(t, cfg.NAS.OutputPath),
}
dirs, err := ps.GetEventDirs()
@ -156,6 +178,7 @@ func TestProcessingService_GetResolutions(t *testing.T) {
ps := &ProcessingService{
config: cfg,
eventName: eventName,
nas: connectedNAS(t, cfg.NAS.OutputPath),
}
resolutions, err := ps.GetResolutions()
@ -248,6 +271,7 @@ func TestProcessingService_WriteConcatFile(t *testing.T) {
ps := &ProcessingService{
config: cfg,
eventName: eventName,
nas: connectedNAS(t, cfg.NAS.OutputPath),
}
// Create test segment map
@ -382,3 +406,67 @@ func TestProcessJob_Structure(t *testing.T) {
t.Errorf("Expected EventName='test-event', got '%s'", job.EventName)
}
}
func TestResolutionHeight(t *testing.T) {
cases := map[string]int{
"1080p": 1080,
"720p": 720,
"2160p": 2160, // beyond anything a fixed quality table would list
"1440p": 1440,
"240p": 240,
// Not in <height>p form: must rank below every real resolution rather
// than accidentally outranking them.
"unknown": 0,
"": 0,
"p": 0,
"-720p": 0,
}
for label, want := range cases {
if got := resolutionHeight(label); got != want {
t.Errorf("resolutionHeight(%q) = %d, want %d", label, got, want)
}
}
}
// TestAggregateSegmentInfoPrefersHighestResolution covers the variants a
// hardcoded rank table could not express: resolutions above 1080p, and the
// "unknown" label produced for a playlist with no variants.
func TestAggregateSegmentInfoPrefersHighestResolution(t *testing.T) {
cases := []struct {
name string
arriving []string
want string
}{
{"1080p beats 720p regardless of order", []string{"720p", "1080p"}, "1080p"},
{"1080p beats 720p when it arrives first", []string{"1080p", "720p"}, "1080p"},
{"4k beats 1080p", []string{"1080p", "2160p"}, "2160p"},
{"1440p beats 1080p", []string{"1080p", "1440p"}, "1440p"},
{"a known resolution beats unknown", []string{"unknown", "480p"}, "480p"},
{"unknown does not displace a known resolution", []string{"1080p", "unknown"}, "1080p"},
{"unknown alone is still kept", []string{"unknown"}, "unknown"},
{"lowest only", []string{"240p"}, "240p"},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
ps := &ProcessingService{}
ch := make(chan SegmentInfo, len(tc.arriving))
for _, res := range tc.arriving {
ch <- SegmentInfo{Name: "seg_1001.ts", SeqNo: 1001, Resolution: res}
}
close(ch)
segments, err := ps.AggregateSegmentInfo(ch)
if err != nil {
t.Fatalf("AggregateSegmentInfo() failed: %v", err)
}
if len(segments) != 1 {
t.Fatalf("expected a single sequence number, got %d", len(segments))
}
if got := segments[1001].Resolution; got != tc.want {
t.Errorf("kept %q, want %q (arrival order: %v)", got, tc.want, tc.arriving)
}
})
}
}

View File

@ -2,6 +2,7 @@ package transfer
import (
"context"
"errors"
"fmt"
"log"
"m3u8-downloader/pkg/logging"
@ -10,6 +11,11 @@ import (
"time"
)
// errRetained reports that a file is still inside its retention window. It is
// not a failure: the file simply has to be reconsidered on a later pass rather
// than being discarded from the pending list.
var errRetained = errors.New("file is within the retention period")
type CleanupService struct {
config CleanupConfig
pendingFiles []string
@ -82,29 +88,43 @@ func (cs *CleanupService) ExecuteCleanup(ctx context.Context) error {
log.Printf("Processing %d files for cleanup", len(batch))
var cleanedCount int
var errors []error
var failures []error
// requeue collects files this pass must not drop: ones still inside their
// retention window, and — if the context is cancelled mid-batch — the
// remainder that was never looked at. They were already removed from
// pendingFiles above, so without this they would never be cleaned at all.
var requeue []string
for _, filePath := range batch {
for i, filePath := range batch {
select {
case <-ctx.Done():
requeue = append(requeue, batch[i:]...)
cs.requeue(requeue)
return ctx.Err()
default:
}
if err := cs.cleanupFile(filePath); err != nil {
errors = append(errors, fmt.Errorf("Failed to cleanup file %s: %w", filePath, err))
} else {
err := cs.cleanupFile(filePath)
switch {
case errors.Is(err, errRetained):
requeue = append(requeue, filePath)
case err != nil:
failures = append(failures, fmt.Errorf("Failed to cleanup file %s: %w", filePath, err))
default:
cleanedCount++
}
}
log.Printf("Cleanup batch completed (cleaned: %d, errors: %d)", cleanedCount, len(errors))
logging.MustGetFactory().Transfer.Info("cleanup batch completed", "cleaned", cleanedCount, "errors", len(errors))
cs.requeue(requeue)
if len(errors) > 0 {
for i, err := range errors {
log.Printf("Cleanup batch completed (cleaned: %d, retained: %d, errors: %d)", cleanedCount, len(requeue), len(failures))
logging.MustGetFactory().Transfer.Info("cleanup batch completed",
"cleaned", cleanedCount, "retained", len(requeue), "errors", len(failures))
if len(failures) > 0 {
for i, err := range failures {
if i >= 3 {
log.Printf("... and %d more errors", len(errors)-3)
log.Printf("... and %d more errors", len(failures)-3)
break
}
log.Printf("Error: %v", err)
@ -113,7 +133,17 @@ func (cs *CleanupService) ExecuteCleanup(ctx context.Context) error {
}
return nil
}
// requeue returns files to the pending list for a later pass.
func (cs *CleanupService) requeue(files []string) {
if len(files) == 0 {
return
}
cs.mu.Lock()
defer cs.mu.Unlock()
cs.pendingFiles = append(cs.pendingFiles, files...)
}
func (cs *CleanupService) cleanupFile(filePath string) error {
@ -128,7 +158,7 @@ func (cs *CleanupService) cleanupFile(filePath string) error {
if cs.config.RetentionPeriod > 0 {
if time.Since(info.ModTime()) < cs.config.RetentionPeriod {
log.Printf("File too new to cleanup: %s", filePath)
return nil
return errRetained
}
}
@ -162,6 +192,18 @@ func (cs *CleanupService) ForceCleanupAll(ctx context.Context) error {
return err
}
// Stop when a pass makes no progress. Files still inside their
// retention window are requeued by design, so requiring the list to
// reach zero would spin here until the shutdown timeout expired.
cs.mu.Lock()
remaining := len(cs.pendingFiles)
cs.mu.Unlock()
if remaining >= pendingCount {
log.Printf("Force cleanup stopping with %d file(s) still pending (within retention period)", remaining)
break
}
select {
case <-ctx.Done():
return ctx.Err()

View File

@ -0,0 +1,190 @@
package transfer
import (
"context"
"fmt"
"os"
"path/filepath"
"testing"
"time"
)
func writeAgedFile(t *testing.T, dir, name string, age time.Duration) string {
t.Helper()
path := filepath.Join(dir, name)
if err := os.WriteFile(path, []byte("segment"), 0644); err != nil {
t.Fatalf("failed to write %s: %v", path, err)
}
modTime := time.Now().Add(-age)
if err := os.Chtimes(path, modTime, modTime); err != nil {
t.Fatalf("failed to age %s: %v", path, err)
}
return path
}
func TestCleanupRemovesTransferredFiles(t *testing.T) {
dir := t.TempDir()
cs := NewCleanupService(CleanupConfig{Enabled: true, BatchSize: 10, CheckInterval: time.Second})
path := writeAgedFile(t, dir, "seg_1001.ts", 0)
if err := cs.ScheduleCleanup(path); err != nil {
t.Fatalf("ScheduleCleanup failed: %v", err)
}
if err := cs.ExecuteCleanup(context.Background()); err != nil {
t.Fatalf("ExecuteCleanup failed: %v", err)
}
if _, err := os.Stat(path); !os.IsNotExist(err) {
t.Errorf("file should have been removed, stat err = %v", err)
}
if got := cs.GetPendingCount(); got != 0 {
t.Errorf("pending count = %d, want 0", got)
}
}
// TestCleanupRetainsThenRemoves is the regression test for a file inside its
// retention window being dropped from the pending list: it was reported as
// cleaned, never deleted, and never reconsidered, so local files accumulated
// forever whenever RetainHours was set.
func TestCleanupRetainsThenRemoves(t *testing.T) {
dir := t.TempDir()
cs := NewCleanupService(CleanupConfig{
Enabled: true,
RetentionPeriod: time.Hour,
BatchSize: 10,
CheckInterval: time.Second,
})
fresh := writeAgedFile(t, dir, "fresh.ts", time.Minute)
old := writeAgedFile(t, dir, "old.ts", 2*time.Hour)
for _, p := range []string{fresh, old} {
if err := cs.ScheduleCleanup(p); err != nil {
t.Fatalf("ScheduleCleanup failed: %v", err)
}
}
if err := cs.ExecuteCleanup(context.Background()); err != nil {
t.Fatalf("ExecuteCleanup failed: %v", err)
}
if _, err := os.Stat(old); !os.IsNotExist(err) {
t.Errorf("a file past its retention period should be removed, stat err = %v", err)
}
if _, err := os.Stat(fresh); err != nil {
t.Errorf("a file inside its retention period should survive: %v", err)
}
if got := cs.GetPendingCount(); got != 1 {
t.Fatalf("the retained file must stay queued for a later pass, pending = %d", got)
}
// Once it ages past the window, the next pass must actually remove it.
past := time.Now().Add(-2 * time.Hour)
if err := os.Chtimes(fresh, past, past); err != nil {
t.Fatalf("failed to age file: %v", err)
}
if err := cs.ExecuteCleanup(context.Background()); err != nil {
t.Fatalf("ExecuteCleanup failed: %v", err)
}
if _, err := os.Stat(fresh); !os.IsNotExist(err) {
t.Errorf("file should have been removed once past retention, stat err = %v", err)
}
if got := cs.GetPendingCount(); got != 0 {
t.Errorf("pending count = %d, want 0", got)
}
}
// TestForceCleanupAllTerminatesWithRetainedFiles guards shutdown: retained
// files are requeued by design, so a loop demanding an empty list would spin
// until the shutdown timeout expired.
func TestForceCleanupAllTerminatesWithRetainedFiles(t *testing.T) {
dir := t.TempDir()
cs := NewCleanupService(CleanupConfig{
Enabled: true,
RetentionPeriod: time.Hour,
BatchSize: 10,
CheckInterval: time.Second,
})
if err := cs.ScheduleCleanup(writeAgedFile(t, dir, "fresh.ts", time.Minute)); err != nil {
t.Fatalf("ScheduleCleanup failed: %v", err)
}
done := make(chan error, 1)
go func() { done <- cs.ForceCleanupAll(context.Background()) }()
select {
case err := <-done:
if err != nil {
t.Fatalf("ForceCleanupAll failed: %v", err)
}
case <-time.After(3 * time.Second):
t.Fatal("ForceCleanupAll did not terminate with a retained file pending")
}
if got := cs.GetPendingCount(); got != 1 {
t.Errorf("the retained file should still be pending, got %d", got)
}
}
func TestCleanupMissingFileIsNotAnError(t *testing.T) {
cs := NewCleanupService(CleanupConfig{Enabled: true, BatchSize: 10, CheckInterval: time.Second})
// Already gone — a file transferred and removed by something else.
if err := cs.ScheduleCleanup(filepath.Join(t.TempDir(), "gone.ts")); err != nil {
t.Fatalf("ScheduleCleanup failed: %v", err)
}
if err := cs.ExecuteCleanup(context.Background()); err != nil {
t.Fatalf("ExecuteCleanup failed: %v", err)
}
if got := cs.GetPendingCount(); got != 0 {
t.Errorf("a missing file should not be requeued, pending = %d", got)
}
}
func TestCleanupDisabledSchedulesNothing(t *testing.T) {
dir := t.TempDir()
cs := NewCleanupService(CleanupConfig{Enabled: false, BatchSize: 10, CheckInterval: time.Second})
path := writeAgedFile(t, dir, "seg.ts", time.Hour)
if err := cs.ScheduleCleanup(path); err != nil {
t.Fatalf("ScheduleCleanup failed: %v", err)
}
if got := cs.GetPendingCount(); got != 0 {
t.Errorf("cleanup is disabled, nothing should be queued, got %d", got)
}
if err := cs.ExecuteCleanup(context.Background()); err != nil {
t.Fatalf("ExecuteCleanup failed: %v", err)
}
if _, err := os.Stat(path); err != nil {
t.Errorf("file must survive when cleanup is disabled: %v", err)
}
}
// TestCleanupRequeuesOnCancellation covers shutdown arriving mid-batch: the
// files already pulled off the pending list must go back, not vanish.
func TestCleanupRequeuesOnCancellation(t *testing.T) {
dir := t.TempDir()
cs := NewCleanupService(CleanupConfig{Enabled: true, BatchSize: 10, CheckInterval: time.Second})
for i := 0; i < 5; i++ {
path := writeAgedFile(t, dir, fmt.Sprintf("seg_%d.ts", i), 0)
if err := cs.ScheduleCleanup(path); err != nil {
t.Fatalf("ScheduleCleanup failed: %v", err)
}
}
ctx, cancel := context.WithCancel(context.Background())
cancel()
if err := cs.ExecuteCleanup(ctx); err == nil {
t.Fatal("expected the cancelled context to be reported")
}
if got := cs.GetPendingCount(); got != 5 {
t.Errorf("all files should be requeued after cancellation, pending = %d, want 5", got)
}
}

View File

@ -11,7 +11,7 @@ import (
)
func TransferFile(nt *nas.NASService, ctx context.Context, item *TransferItem) error {
destPath := filepath.Join(nt.Config.Path, item.DestinationPath)
destPath := nt.Join(item.DestinationPath)
destDir := filepath.Dir(destPath)
if err := nt.EnsureDirectoryExists(destDir); err != nil {

View File

@ -169,13 +169,20 @@ func (tq *TransferQueue) processItem(ctx context.Context, item TransferItem) {
return
}
maxRetries := 3
maxRetries := tq.config.RetryLimit
if maxRetries < 1 {
maxRetries = 1
}
for attempt := 1; attempt <= maxRetries; attempt++ {
if attempt > 0 {
// Back off before a retry, never before the first attempt. The guard
// was `attempt > 0`, which is always true, so every single transfer —
// including healthy ones — paid a one-second delay before it began,
// throttling the whole pipeline to the worker count per second.
if attempt > 1 {
item.Status = StatusRetrying
backoff := time.Duration(attempt*attempt) * time.Second
log.Printf("Backing off for %d seconds before retrying (attempt %d/%d)", backoff, attempt, maxRetries)
log.Printf("Backing off for %s before retrying (attempt %d/%d)", backoff, attempt, maxRetries)
select {
case <-time.After(backoff):

View File

@ -34,14 +34,23 @@ func NewTrasferService(outputDir string, eventName string) (*TransferService, er
Timeout: cfg.NAS.Timeout,
RetryLimit: cfg.NAS.RetryLimit,
VerifySize: true,
AutoMount: cfg.NAS.AutoMount,
MountPoint: cfg.NAS.MountPoint,
}
nas, err := nas2.New(nasConfig)
if err != nil {
logging.MustGetFactory().Transfer.Error("invalid NAS destination", logging.KeyError, err.Error())
return nil, fmt.Errorf("invalid NAS destination: %w", err)
}
nas := nas2.NewNASService(nasConfig)
if err := nas.TestConnection(); err != nil {
connectCtx, cancelConnect := context.WithTimeout(context.Background(), cfg.NAS.Timeout)
defer cancelConnect()
if err := nas.Connect(connectCtx); err != nil {
logging.MustGetFactory().Transfer.Error("failed to connect to NAS", logging.KeyError, err.Error())
return nil, fmt.Errorf("failed to connect to NAS: %w", err)
}
logging.MustGetFactory().Transfer.Info("connected to NAS", "path", outputDir)
logging.MustGetFactory().Transfer.Info("connected to NAS",
"path", nas.LocalPath(), "configured", outputDir, "how", strings.Join(nas.ConnectionInfo(), "; "))
health.MustGetRegistry().RecordNASConnectivity(true)
cleanupConfig := CleanupConfig{
@ -57,6 +66,7 @@ func NewTrasferService(outputDir string, eventName string) (*TransferService, er
PersistencePath: cfg.Paths.PersistenceFile,
MaxQueueSize: cfg.Transfer.QueueSize,
BatchSize: cfg.Transfer.BatchSize,
RetryLimit: cfg.Transfer.RetryLimit,
}
queue := NewTransferQueue(queueConfig, nas, cleanup)
@ -66,7 +76,7 @@ func NewTrasferService(outputDir string, eventName string) (*TransferService, er
return nil, fmt.Errorf("failed to create local output directory: %w", err)
}
watcher, err := NewFileWatcher(localOutputPath, queue, cfg.Transfer.FileSettlingDelay)
watcher, err := NewFileWatcher(localOutputPath, eventName, queue, cfg.Transfer.FileSettlingDelay)
if err != nil {
return nil, fmt.Errorf("failed to create file watcher: %w", err)
}
@ -155,7 +165,7 @@ func (ts *TransferService) Shutdown(ctx context.Context) error {
}
// Disconnect from NAS
if err := ts.nas.Disconnect(); err != nil {
if err := ts.nas.Disconnect(ctx); err != nil {
log.Printf("Warning: failed to disconnect from NAS: %v", err)
logging.MustGetFactory().Transfer.Warn("failed to disconnect from NAS", logging.KeyError, err.Error())
}

View File

@ -49,6 +49,9 @@ type QueueConfig struct {
PersistencePath string
MaxQueueSize int
BatchSize int
// RetryLimit is the total number of attempts made per file before it is
// marked failed.
RetryLimit int
}
type CleanupConfig struct {

View File

@ -18,6 +18,11 @@ import (
type FileWatcher struct {
outputDir string
// eventName namespaces the NAS destination. It is passed in rather than
// derived from outputDir so that an unnamed event resolves to the NAS root
// — matching where the downloader put the files — instead of picking up
// the local output directory's own name.
eventName string
queue *TransferQueue
watcher *fsnotify.Watcher
settingDelay time.Duration
@ -25,13 +30,14 @@ type FileWatcher struct {
mu sync.Mutex
}
func NewFileWatcher(outputDir string, queue *TransferQueue, settlingDelay time.Duration) (*FileWatcher, error) {
func NewFileWatcher(outputDir, eventName string, queue *TransferQueue, settlingDelay time.Duration) (*FileWatcher, error) {
watcher, err := fsnotify.NewWatcher()
if err != nil {
return nil, err
}
return &FileWatcher{
outputDir: outputDir,
eventName: eventName,
queue: queue,
watcher: watcher,
settingDelay: settlingDelay,
@ -92,6 +98,18 @@ func (fw *FileWatcher) addWatchRecursive(root string) error {
}
func (fw *FileWatcher) handleFileEvent(event fsnotify.Event) {
// Directories must be handled before the .ts filter below. fsnotify is not
// recursive, so a resolution directory created after the watcher starts —
// which is the normal case, since the transfer service starts before the
// downloader creates 1080p/ and friends — has to be picked up here or
// nothing inside it is ever seen.
if event.Op&fsnotify.Create == fsnotify.Create {
if info, err := os.Stat(event.Name); err == nil && info.IsDir() {
fw.watchNewDirectory(event.Name)
return
}
}
if !strings.HasSuffix(strings.ToLower(event.Name), ".ts") {
return
}
@ -104,15 +122,33 @@ func (fw *FileWatcher) handleFileEvent(event fsnotify.Event) {
case event.Op&fsnotify.Remove == fsnotify.Remove:
fw.cancelPendingTransfer(event.Name)
}
}
if event.Op&fsnotify.Create == fsnotify.Create {
if info, err := os.Stat(event.Name); err == nil && info.IsDir() {
if err := fw.watcher.Add(event.Name); err != nil {
log.Printf("Failed to watch directory %s: %v", event.Name, err)
} else {
log.Printf("Watching directory %s", event.Name)
// watchNewDirectory starts watching a directory that appeared after startup,
// and sweeps whatever is already inside it.
//
// The sweep is not redundant: the downloader creates a resolution directory and
// writes segments into it immediately, so files can land between the mkdir and
// the watch being registered. Those writes generate no event we would see, and
// without the sweep the first segments of every recording would be lost.
func (fw *FileWatcher) watchNewDirectory(dir string) {
if err := fw.addWatchRecursive(dir); err != nil {
log.Printf("Failed to watch new directory %s: %v", dir, err)
logging.MustGetFactory().Transfer.Warn("failed to watch new directory", "dir", dir, logging.KeyError, err.Error())
return
}
err := filepath.Walk(dir, func(path string, info os.FileInfo, err error) error {
if err != nil {
return nil // keep sweeping the rest
}
if !info.IsDir() && strings.HasSuffix(strings.ToLower(info.Name()), ".ts") {
fw.scheduleTransfer(path)
}
return nil
})
if err != nil {
log.Printf("Failed to sweep new directory %s: %v", dir, err)
}
}
@ -164,10 +200,13 @@ func (fw *FileWatcher) processFile(filePath string) {
return
}
// The destination is namespaced by event, mirroring the local layout and
// matching what QueueExistingFiles produces for the same file — otherwise
// live transfers and catch-up transfers would write to different places.
item := TransferItem{
ID: generateID(),
SourcePath: filePath,
DestinationPath: relPath,
DestinationPath: filepath.Join(fw.eventName, relPath),
Resolution: resolution,
Timestamp: time.Now(),
Status: StatusPending,

View File

@ -0,0 +1,223 @@
package transfer
import (
"context"
"os"
"path/filepath"
"sort"
"testing"
"time"
)
// settlingDelay is short so tests exercise the real timer path without
// waiting the production five seconds.
const testSettlingDelay = 40 * time.Millisecond
// newTestQueue builds a queue that accepts items but never transfers them:
// ProcessQueue is not started, so Add is a pure recording of intent.
func newTestQueue(t *testing.T) *TransferQueue {
t.Helper()
return NewTransferQueue(QueueConfig{
WorkerCount: 1,
PersistencePath: filepath.Join(t.TempDir(), "queue.json"),
MaxQueueSize: 100,
BatchSize: 10,
}, nil, nil)
}
// queuedDestinations returns the destination paths currently queued, sorted.
func queuedDestinations(q *TransferQueue) []string {
q.mu.RLock()
defer q.mu.RUnlock()
out := make([]string, 0, q.items.Len())
for _, item := range *q.items {
out = append(out, filepath.ToSlash(item.DestinationPath))
}
sort.Strings(out)
return out
}
// waitForQueue polls until the queue holds want items or the deadline passes.
// Polling beats a fixed sleep: it keeps the test fast when the watcher is
// prompt and tolerant when a loaded CI machine is not.
func waitForQueue(t *testing.T, q *TransferQueue, want int) []string {
t.Helper()
deadline := time.Now().Add(3 * time.Second)
var got []string
for time.Now().Before(deadline) {
got = queuedDestinations(q)
if len(got) >= want {
return got
}
time.Sleep(10 * time.Millisecond)
}
return got
}
func startWatcher(t *testing.T, dir, eventName string, q *TransferQueue) {
t.Helper()
fw, err := NewFileWatcher(dir, eventName, q, testSettlingDelay)
if err != nil {
t.Fatalf("NewFileWatcher failed: %v", err)
}
ctx, cancel := context.WithCancel(context.Background())
t.Cleanup(cancel)
started := make(chan struct{})
go func() {
close(started)
_ = fw.Start(ctx)
}()
<-started
// Give Start a moment to register its initial watches before the test
// begins creating files under them.
time.Sleep(100 * time.Millisecond)
}
// TestWatcherPicksUpDirectoriesCreatedAfterStart covers the normal recording
// sequence: the transfer service starts watching the event directory, and only
// then does the downloader create the per-resolution directories inside it.
// fsnotify is not recursive, so those directories have to be picked up from
// the create event or nothing inside them is ever transferred.
func TestWatcherPicksUpDirectoriesCreatedAfterStart(t *testing.T) {
eventDir := t.TempDir()
queue := newTestQueue(t)
startWatcher(t, eventDir, "2026-regionals", queue)
resolutionDir := filepath.Join(eventDir, "1080p")
if err := os.MkdirAll(resolutionDir, 0755); err != nil {
t.Fatalf("failed to create resolution dir: %v", err)
}
if err := os.WriteFile(filepath.Join(resolutionDir, "seg_1001.ts"), []byte("payload"), 0644); err != nil {
t.Fatalf("failed to write segment: %v", err)
}
got := waitForQueue(t, queue, 1)
if len(got) != 1 {
t.Fatalf("segment written to a new resolution directory was never queued, queue = %v", got)
}
if want := "2026-regionals/1080p/seg_1001.ts"; got[0] != want {
t.Errorf("destination = %q, want %q", got[0], want)
}
}
// TestWatcherSweepsFilesRacingDirectoryCreation covers the tighter race where
// segments are written immediately after the mkdir, before the new watch can
// be registered. Those writes produce no event the watcher would receive, so
// the directory has to be swept when it is first seen.
func TestWatcherSweepsFilesRacingDirectoryCreation(t *testing.T) {
eventDir := t.TempDir()
queue := newTestQueue(t)
startWatcher(t, eventDir, "event", queue)
// Stage the tree elsewhere and move it into place, so the files are
// guaranteed to already exist the instant the directory appears.
staging := filepath.Join(t.TempDir(), "1080p")
if err := os.MkdirAll(staging, 0755); err != nil {
t.Fatalf("failed to stage: %v", err)
}
for _, name := range []string{"seg_1001.ts", "seg_1002.ts"} {
if err := os.WriteFile(filepath.Join(staging, name), []byte("payload"), 0644); err != nil {
t.Fatalf("failed to stage segment: %v", err)
}
}
if err := os.Rename(staging, filepath.Join(eventDir, "1080p")); err != nil {
t.Fatalf("failed to move staged directory into place: %v", err)
}
got := waitForQueue(t, queue, 2)
want := []string{"event/1080p/seg_1001.ts", "event/1080p/seg_1002.ts"}
if len(got) != 2 || got[0] != want[0] || got[1] != want[1] {
t.Errorf("queued %v, want %v", got, want)
}
}
// TestWatcherDestinationIncludesEventName pins the layout agreement between
// live transfers and the catch-up path in QueueExistingFiles: both must write
// to <nas>/<event>/<resolution>/<file>.
func TestWatcherDestinationIncludesEventName(t *testing.T) {
eventDir := t.TempDir()
resolutionDir := filepath.Join(eventDir, "720p")
if err := os.MkdirAll(resolutionDir, 0755); err != nil {
t.Fatalf("setup failed: %v", err)
}
queue := newTestQueue(t)
startWatcher(t, eventDir, "my-event", queue)
if err := os.WriteFile(filepath.Join(resolutionDir, "seg_1001.ts"), []byte("payload"), 0644); err != nil {
t.Fatalf("failed to write segment: %v", err)
}
got := waitForQueue(t, queue, 1)
if len(got) != 1 {
t.Fatalf("expected one queued item, got %v", got)
}
if want := "my-event/720p/seg_1001.ts"; got[0] != want {
t.Errorf("destination = %q, want %q", got[0], want)
}
}
// TestWatcherWithoutEventNameUsesRoot mirrors a download run with no -event:
// segments land under the local output root, so the NAS destination must not
// invent a directory level for them.
func TestWatcherWithoutEventNameUsesRoot(t *testing.T) {
eventDir := t.TempDir()
resolutionDir := filepath.Join(eventDir, "1080p")
if err := os.MkdirAll(resolutionDir, 0755); err != nil {
t.Fatalf("setup failed: %v", err)
}
queue := newTestQueue(t)
startWatcher(t, eventDir, "", queue)
if err := os.WriteFile(filepath.Join(resolutionDir, "seg_1001.ts"), []byte("payload"), 0644); err != nil {
t.Fatalf("failed to write segment: %v", err)
}
got := waitForQueue(t, queue, 1)
if len(got) != 1 {
t.Fatalf("expected one queued item, got %v", got)
}
if want := "1080p/seg_1001.ts"; got[0] != want {
t.Errorf("destination = %q, want %q", got[0], want)
}
}
func TestWatcherIgnoresNonSegmentFiles(t *testing.T) {
eventDir := t.TempDir()
queue := newTestQueue(t)
startWatcher(t, eventDir, "event", queue)
for _, name := range []string{"notes.txt", "manifest.json", "segment.ts.part"} {
if err := os.WriteFile(filepath.Join(eventDir, name), []byte("x"), 0644); err != nil {
t.Fatalf("failed to write %s: %v", name, err)
}
}
time.Sleep(testSettlingDelay + 300*time.Millisecond)
if got := queuedDestinations(queue); len(got) != 0 {
t.Errorf("non-segment files should not be queued, got %v", got)
}
}
func TestExtractResolution(t *testing.T) {
fw := &FileWatcher{}
cases := map[string]string{
filepath.Join("data", "event", "1080p", "seg.ts"): "1080p",
filepath.Join("data", "event", "720p", "seg.ts"): "720p",
filepath.Join("data", "event", "seg.ts"): "",
}
for path, want := range cases {
if got := fw.extractResolution(path); got != want {
t.Errorf("extractResolution(%q) = %q, want %q", path, got, want)
}
}
}

View File

@ -24,12 +24,28 @@ func PathExists(path string) bool {
return !os.IsNotExist(err)
}
// IsValidPath reports whether path is usable on Windows, the stricter of the
// two platforms this project targets, which reserves <>:"|?* in names.
//
// A leading drive letter is checked separately: its colon is legal there and
// nowhere else in the path. Without that carve-out the function rejects every
// absolute Windows path, including the drive-qualified ones it exists to
// validate.
func IsValidPath(path string) bool {
if path == "" {
return false
}
return !strings.ContainsAny(path, "<>:\"|?*")
rest := path
if len(rest) >= 2 && rest[1] == ':' && isDriveLetter(rest[0]) {
rest = rest[2:]
}
return !strings.ContainsAny(rest, `<>:"|?*`)
}
func isDriveLetter(c byte) bool {
return (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z')
}
func NormalizePath(path string) string {
@ -37,6 +53,13 @@ func NormalizePath(path string) string {
}
func GetRelativePath(basePath, targetPath string) (string, error) {
// filepath.Rel("", "") succeeds and returns ".", which is a meaningless
// answer to a meaningless question; callers passing an empty path have a
// bug worth surfacing rather than papering over.
if basePath == "" || targetPath == "" {
return "", fmt.Errorf("both base and target paths are required (base %q, target %q)", basePath, targetPath)
}
rel, err := filepath.Rel(basePath, targetPath)
if err != nil {
return "", fmt.Errorf("failed to get relative path: %w", err)