Claude bugfixes
Some checks failed
Build Go Binaries / build (push) Has been cancelled

This commit is contained in:
Aaron 2026-08-07 03:49:37 +00:00
parent 7f4078961e
commit e2566fdefa
33 changed files with 2838 additions and 221 deletions

View File

@ -33,9 +33,15 @@ The project follows a modular architecture with clear separation of concerns:
- **service.go**: Processing service orchestration with FFmpeg integration - **service.go**: Processing service orchestration with FFmpeg integration
- **segment.go**: Individual segment processing logic - **segment.go**: Individual segment processing logic
- **types.go**: Processing system data structures - **types.go**: Processing system data structures
- **nas/**: NAS connection and file operations - **nas/**: NAS connection and file operations (platform-agnostic: Windows and Linux)
- **config.go**: NAS configuration structure - **config.go**: NAS configuration structure
- **nas.go**: NAS service with connection management and file operations - **nas.go**: `NASService` — resolves the destination to a host-native path, then copies/verifies files against it
- **location.go**: Parses UNC / `//server/share` / `smb://` / local paths into a host-neutral `Location`
- **connector.go**: `Connector` interface + runtime selection by GOOS (no build tags, so both platforms' code is compiled and tested everywhere)
- **connector_netuse.go**: Windows strategy — `net use`, UNC path used directly
- **connector_mount.go**: Linux strategy — resolve the share against `/proc/mounts`, optional root-only `mount.cifs`
- **connector_local.go**: Already-mounted/plain directory on either platform
- **mounts.go**: `/proc/mounts` parsing and share-to-mount-point matching
- **config/**: Centralized configuration management with validation - **config/**: Centralized configuration management with validation
- **config.go**: Configuration loading, validation, and path resolution - **config.go**: Configuration loading, validation, and path resolution
- **utils/**: Utility functions for cross-platform compatibility - **utils/**: Utility functions for cross-platform compatibility
@ -100,8 +106,10 @@ Configuration is managed through a centralized system in `pkg/config/config.go`
### NAS Transfer Settings ### NAS Transfer Settings
- `NAS.EnableTransfer`: Enable/disable automatic NAS transfer (true) - ENV: `ENABLE_NAS_TRANSFER` - `NAS.EnableTransfer`: Enable/disable automatic NAS transfer (true) - ENV: `ENABLE_NAS_TRANSFER`
- `NAS.OutputPath`: UNC path to NAS storage (`\\HomeLabNAS\dci\streams`) - ENV: `NAS_OUTPUT_PATH` - `NAS.OutputPath`: NAS destination (`\\HomeLabNAS\dci\streams`) - ENV: `NAS_OUTPUT_PATH`. Accepts UNC, `//server/share`, `smb://`, or a local/mounted directory; Windows connects it with `net use`, Linux resolves it against `/proc/mounts`
- `NAS.Username`/`NAS.Password`: NAS credentials for authentication - ENV: `NAS_USERNAME`/`NAS_PASSWORD` - `NAS.Username`/`NAS.Password`: NAS credentials for authentication - ENV: `NAS_USERNAME`/`NAS_PASSWORD`
- `NAS.AutoMount`: Linux-only, root-required auto-mount of the SMB share (false) - ENV: `NAS_AUTO_MOUNT`
- `NAS.MountPoint`: Override for the auto-mount location - ENV: `NAS_MOUNT_POINT`
- `Transfer.WorkerCount`: Concurrent transfer workers (2) - `Transfer.WorkerCount`: Concurrent transfer workers (2)
- `Transfer.RetryLimit`: Max retry attempts per file (3) - `Transfer.RetryLimit`: Max retry attempts per file (3)
- `Transfer.Timeout`: Timeout per file transfer (30 seconds) - `Transfer.Timeout`: Timeout per file transfer (30 seconds)

View File

@ -11,10 +11,23 @@ The application supports configuration through environment variables for flexibl
- `REFRESH_DELAY_SECONDS`: How often to check for playlist updates in seconds (default: 3) - `REFRESH_DELAY_SECONDS`: How often to check for playlist updates in seconds (default: 3)
### NAS Transfer Settings ### NAS Transfer Settings
- `NAS_OUTPUT_PATH`: UNC path to NAS storage (default: "\\\\HomeLabNAS\\dci\\streams") - `NAS_OUTPUT_PATH`: NAS destination (default: "\\\\HomeLabNAS\\dci\\streams"). Accepts a UNC path, `//server/share/dir`, `smb://server/share/dir`, or a local/mounted directory
- `NAS_USERNAME`: NAS authentication username - `NAS_USERNAME`: NAS authentication username
- `NAS_PASSWORD`: NAS authentication password - `NAS_PASSWORD`: NAS authentication password
- `ENABLE_NAS_TRANSFER`: Enable/disable automatic NAS transfer (default: true) - `ENABLE_NAS_TRANSFER`: Enable/disable automatic NAS transfer (default: true)
- `NAS_AUTO_MOUNT`: Linux only — mount the SMB share if the host hasn't already (default: false, requires root)
- `NAS_MOUNT_POINT`: Where `NAS_AUTO_MOUNT` attaches the share (default: `/mnt/streamrecorder/<server>/<share>`)
The same `NAS_OUTPUT_PATH` works on Windows and Linux. Windows authenticates the
UNC path with `net use`; Linux resolves the share against `/proc/mounts` and
rewrites the path onto the mount point, so `\\\\HomeLabNAS\\dci\\streams` with
`//homelabnas/dci` mounted at `/mnt/dci` becomes `/mnt/dci/streams`. Mounting the
share on the host (fstab/autofs/systemd, or `docker run -v`) is the recommended
Linux setup; `NAS_AUTO_MOUNT` is the root-privileged fallback. Verify either with:
```bash
./stream-recorder -check
```
### Path Configuration ### Path Configuration
- `LOCAL_OUTPUT_DIR`: Base directory for local downloads (default: "data") - `LOCAL_OUTPUT_DIR`: Base directory for local downloads (default: "data")

View File

@ -43,7 +43,7 @@ Segments land on disk under `Paths.LocalOutput/<event>/<resolution>/`.
- `service.go`: `TransferService` wires together the watcher, queue, and cleanup service, and owns the connection to NAS for the run. - `service.go`: `TransferService` wires together the watcher, queue, and cleanup service, and owns the connection to NAS for the run.
- `watcher.go`: `FileWatcher` (via `fsnotify`) watches the local event directory for new `.ts` files and queues them after a short settling delay (`Transfer.FileSettlingDelay`, default 5s), so a file isn't queued while still being written. - `watcher.go`: `FileWatcher` (via `fsnotify`) watches the local event directory for new `.ts` files and queues them after a short settling delay (`Transfer.FileSettlingDelay`, default 5s), so a file isn't queued while still being written.
- `queue.go`: `TransferQueue` is a priority-queued worker pool (`Transfer.WorkerCount`, default 2) that transfers queued files to NAS with retry + exponential backoff, and persists its state to `transfer_queue.json` so it survives a restart. - `queue.go`: `TransferQueue` is a priority-queued worker pool (`Transfer.WorkerCount`, default 2) that transfers queued files to NAS with retry + exponential backoff, and persists its state to `transfer_queue.json` so it survives a restart.
- `nas.go` / `pkg/nas`: `NASService` establishes the NAS connection (on Windows, via `net use` for UNC paths) and does the actual file copy + size verification. - `nas.go` / `pkg/nas`: `NASService` attaches the NAS destination for whatever host it's running on and does the actual file copy + size verification. See [NAS connectivity](#nas-connectivity-windows-and-linux).
- `cleanup.go`: `CleanupService` deletes local copies after a successful, verified transfer (`Cleanup.AfterTransfer`, default true). - `cleanup.go`: `CleanupService` deletes local copies after a successful, verified transfer (`Cleanup.AfterTransfer`, default true).
During a normal download run, this whole stage starts automatically in the same process as the downloader (see the note above) — you don't invoke transfer separately for a live recording. The standalone `-transfer` mode exists only to catch up files that a prior download session ended without fully transferring. During a normal download run, this whole stage starts automatically in the same process as the downloader (see the note above) — you don't invoke transfer separately for a live recording. The standalone `-transfer` mode exists only to catch up files that a prior download session ended without fully transferring.
@ -83,8 +83,10 @@ All settings have sane defaults; override via environment variables as needed.
| `WORKER_COUNT` | `4` | Concurrent segment downloads per variant | | `WORKER_COUNT` | `4` | Concurrent segment downloads per variant |
| `REFRESH_DELAY_SECONDS` | `3` | How often each variant's chunklist is polled | | `REFRESH_DELAY_SECONDS` | `3` | How often each variant's chunklist is polled |
| `ENABLE_NAS_TRANSFER` | `true` | Whether to run the transfer service alongside download | | `ENABLE_NAS_TRANSFER` | `true` | Whether to run the transfer service alongside download |
| `NAS_OUTPUT_PATH` | `\\HomeLabNAS\dci\streams` | UNC/NAS root path segments are copied to | | `NAS_OUTPUT_PATH` | `\\HomeLabNAS\dci\streams` | NAS root segments are copied to. UNC (`\\server\share\dir`), `//server/share/dir`, `smb://server/share/dir`, or a local/mounted directory — see [NAS connectivity](#nas-connectivity-windows-and-linux) |
| `NAS_USERNAME` / `NAS_PASSWORD` | *(see note below)* | NAS auth credentials | | `NAS_USERNAME` / `NAS_PASSWORD` | *(see note below)* | NAS auth credentials. Used by `net use` on Windows and by `NAS_AUTO_MOUNT` on Linux; ignored for an already-mounted path |
| `NAS_AUTO_MOUNT` | `false` | Linux only: mount the SMB share if the host hasn't. Requires root |
| `NAS_MOUNT_POINT` | *(derived)* | Where `NAS_AUTO_MOUNT` attaches the share (default `/mnt/streamrecorder/<server>/<share>`) |
| `LOCAL_OUTPUT_DIR` | `data` | Local base directory for downloaded segments | | `LOCAL_OUTPUT_DIR` | `data` | Local base directory for downloaded segments |
| `PROCESS_OUTPUT_DIR` | `out` | Output directory for the final concatenated `.mp4` | | `PROCESS_OUTPUT_DIR` | `out` | Output directory for the final concatenated `.mp4` |
| `FFMPEG_PATH` | `ffmpeg` | Path to the FFmpeg binary used by the processing stage | | `FFMPEG_PATH` | `ffmpeg` | Path to the FFmpeg binary used by the processing stage |
@ -95,6 +97,25 @@ All settings have sane defaults; override via environment variables as needed.
> **Security note:** the default NAS credentials currently ship as hardcoded defaults in `pkg/config/config.go` for this home-lab setup. If you fork or redeploy this, set `NAS_USERNAME`/`NAS_PASSWORD` explicitly and rotate the shipped default — see `TODO.md` for the full flagged list of housekeeping items like this one. > **Security note:** the default NAS credentials currently ship as hardcoded defaults in `pkg/config/config.go` for this home-lab setup. If you fork or redeploy this, set `NAS_USERNAME`/`NAS_PASSWORD` explicitly and rotate the shipped default — see `TODO.md` for the full flagged list of housekeeping items like this one.
## NAS connectivity (Windows and Linux)
The same `NAS_OUTPUT_PATH` works on both platforms; only the mechanism for reaching it differs, and the recorder picks it at runtime.
| Host | Destination | What happens |
|---|---|---|
| Windows | UNC / `//server/share` / `smb://` | `net use \\server\share` with `NAS_USERNAME`/`NAS_PASSWORD`, then the UNC path is used directly |
| Linux | UNC / `//server/share` / `smb://` | The share is looked up in `/proc/mounts` and the UNC path is rewritten onto the mount point (`\\nas\dci\streams` + `//nas/dci` mounted at `/mnt/dci` → `/mnt/dci/streams`). Set `NAS_AUTO_MOUNT=true` to mount it instead |
| Either | A local or already-mounted directory | Used as-is, no network step |
Notes:
- **Server and share names match case-insensitively**, and both `cifs`/`smb3` and `nfs`/`nfs4` mounts are recognised. If the share is mounted at a deeper level than configured (`//nas/dci/streams` at `/mnt/streams`), the deepest matching mount wins.
- **Nothing the host already set up is torn down.** A share found already mounted (or already connected via `net use`) is left alone at shutdown; only a mount or connection this process created is undone.
- **`NAS_AUTO_MOUNT` requires root**, so the usual Linux setup is to mount the share yourself via fstab/autofs/systemd and either point `NAS_OUTPUT_PATH` at the mount point or leave it as the UNC path and let it resolve. Credentials are passed to `mount.cifs` through a `0600` credentials file, never on the command line where the process table would expose the password.
- **A Windows-shaped path on Linux is rejected, not silently created.** Backslashes and drive letters are legal filename characters on Linux, so `C:\data` or an unresolved `\\nas\dci` would otherwise be created as a local directory named exactly that — transfers would report success, cleanup would delete the local originals, and nothing would reach the NAS.
Run `-check` to see which strategy applies and where the destination resolved to.
## Building ## Building
```bash ```bash

View File

@ -7,6 +7,7 @@ package check
import ( import (
"context" "context"
"fmt" "fmt"
"log"
"m3u8-downloader/pkg/config" "m3u8-downloader/pkg/config"
"m3u8-downloader/pkg/constants" "m3u8-downloader/pkg/constants"
"m3u8-downloader/pkg/httpClient" "m3u8-downloader/pkg/httpClient"
@ -155,46 +156,51 @@ func checkMasterPlaylist(masterURL string) ([]string, error) {
return details, nil return details, nil
} }
// checkNAS runs the same connect/mkdir/write sequence the transfer service // checkNAS runs the same attach/mkdir/write sequence the transfer service
// performs at startup, but without the log.Fatalf that path takes on failure. // performs at startup, so the check exercises the real code path for this host
// — net use on Windows, mount resolution on Linux — rather than a simulation of
// it. It disconnects afterwards so the check leaves no mount behind.
func checkNAS(cfg *config.Config) ([]string, error) { func checkNAS(cfg *config.Config) ([]string, error) {
svc := &nas.NASService{Config: nas.NASConfig{ svc, err := nas.New(nas.NASConfig{
Path: cfg.NAS.OutputPath, Path: cfg.NAS.OutputPath,
Username: cfg.NAS.Username, Username: cfg.NAS.Username,
Password: cfg.NAS.Password, Password: cfg.NAS.Password,
Timeout: cfg.NAS.Timeout, Timeout: cfg.NAS.Timeout,
RetryLimit: cfg.NAS.RetryLimit, RetryLimit: cfg.NAS.RetryLimit,
VerifySize: true, VerifySize: true,
}} AutoMount: cfg.NAS.AutoMount,
MountPoint: cfg.NAS.MountPoint,
details := []string{"path: " + cfg.NAS.OutputPath} })
if share := svc.ExtractNetworkPath(cfg.NAS.OutputPath); share != "" { if err != nil {
details = append(details, "share: "+share, "user: "+usernameLabel(cfg.NAS.Username)) return []string{"path: " + cfg.NAS.OutputPath}, err
} else {
details = append(details, "local or already-mounted path (no network authentication)")
} }
start := time.Now() start := time.Now()
result := <-probeAsync(cfg.NAS.Timeout, func() ([]string, error) { result := <-probeAsync(cfg.NAS.Timeout, func() ([]string, error) {
if err := svc.EstablishConnection(); err != nil { ctx, cancel := context.WithTimeout(context.Background(), cfg.NAS.Timeout)
return nil, fmt.Errorf("failed to connect to share: %w", err) defer cancel()
if err := svc.Connect(ctx); err != nil {
return svc.ConnectionInfo(), err
} }
if err := svc.EnsureDirectoryExists(cfg.NAS.OutputPath); err != nil { defer func() {
return nil, fmt.Errorf("share reachable but target directory unusable: %w", err) if err := svc.Disconnect(context.Background()); err != nil {
log.Printf("Warning: failed to release NAS connection after check: %v", err)
} }
if err := svc.TestConnection(); err != nil { }()
return nil, fmt.Errorf("target directory is not writable: %w", err)
} return append(svc.ConnectionInfo(), "attach, mkdir and write probe succeeded"), nil
return []string{"connect, mkdir and write probe succeeded"}, nil
}) })
details := result.details
if len(details) == 0 {
details = svc.ConnectionInfo()
}
if result.err != nil { if result.err != nil {
return details, result.err return details, result.err
} }
details = append(details, result.details...) return append(details, "completed in "+formatDuration(time.Since(start))), nil
details = append(details, "completed in "+formatDuration(time.Since(start)))
return details, nil
} }
func formatDuration(d time.Duration) string { func formatDuration(d time.Duration) string {
@ -276,10 +282,3 @@ func enabledLabel(enabled bool) string {
} }
return "disabled" return "disabled"
} }
func usernameLabel(username string) string {
if username == "" {
return "(none, using ambient credentials)"
}
return username
}

View File

@ -154,7 +154,7 @@ func TestCheckNASLocalPath(t *testing.T) {
} }
joined := joinDetails(details) joined := joinDetails(details)
for _, want := range []string{"path: " + target, "write probe succeeded"} { for _, want := range []string{"configured: " + target, "strategy: local", "write probe succeeded"} {
if !strings.Contains(joined, want) { if !strings.Contains(joined, want) {
t.Errorf("details missing %q:\n%s", want, joined) t.Errorf("details missing %q:\n%s", want, joined)
} }
@ -176,7 +176,7 @@ func TestCheckNASUnusableTarget(t *testing.T) {
if err == nil { if err == nil {
t.Fatal("expected an unusable NAS target to fail the check") t.Fatal("expected an unusable NAS target to fail the check")
} }
if joined := joinDetails(details); !strings.Contains(joined, "path: ") { if joined := joinDetails(details); !strings.Contains(joined, "configured: ") {
t.Errorf("expected path detail on failure:\n%s", joined) t.Errorf("expected path detail on failure:\n%s", joined)
} }
} }

View File

@ -71,7 +71,9 @@ func Download(masterURL string, eventName string, debug bool) {
} }
} }
manifestWriter := media.NewManifestWriter(eventName) // One writer for the whole run: every variant records into it, and it is
// the one written to disk at the end.
manifest := media.NewManifestWriter(eventName)
eventPath := cfg.GetEventPath(eventName) eventPath := cfg.GetEventPath(eventName)
if err := utils.EnsureDir(eventPath); err != nil { if err := utils.EnsureDir(eventPath); err != nil {
@ -79,7 +81,7 @@ func Download(masterURL string, eventName string, debug bool) {
log.Fatalf("Failed to create event directory: %v", err) log.Fatalf("Failed to create event directory: %v", err)
} }
variants, err := media.GetAllVariants(masterURL, eventPath, manifestWriter) variants, err := media.GetAllVariants(masterURL, eventPath)
if err != nil { if err != nil {
appLog.Error("failed to get variants", logging.KeyError, err.Error()) appLog.Error("failed to get variants", logging.KeyError, err.Error())
log.Fatalf("Failed to get variants: %v", err) log.Fatalf("Failed to get variants: %v", err)
@ -88,8 +90,6 @@ func Download(masterURL string, eventName string, debug bool) {
sem := make(chan struct{}, constants.WorkerCount*len(variants)) sem := make(chan struct{}, constants.WorkerCount*len(variants))
manifest := media.NewManifestWriter(eventName)
for _, variant := range variants { for _, variant := range variants {
// Debug mode only tracks one variant for easier debugging // Debug mode only tracks one variant for easier debugging
if debug { if debug {
@ -117,7 +117,7 @@ func Download(masterURL string, eventName string, debug bool) {
log.Println("All Services shut down.") log.Println("All Services shut down.")
appLog.Info("all services shut down") appLog.Info("all services shut down")
manifestWriter.WriteManifest() manifest.WriteManifest()
log.Println("Manifest written.") log.Println("Manifest written.")
appLog.Info("manifest written", logging.KeyEvent, eventName) appLog.Info("manifest written", logging.KeyEvent, eventName, "segments", manifest.Len())
} }

View File

@ -32,11 +32,19 @@ type HTTPConfig struct {
type NASConfig struct { type NASConfig struct {
EnableTransfer bool EnableTransfer bool
// OutputPath is the destination as configured: a UNC path, an smb:// URL,
// or a directory on this host. How it is reached depends on the platform;
// see pkg/nas.
OutputPath string OutputPath string
Username string Username string
Password string Password string
Timeout time.Duration Timeout time.Duration
RetryLimit int RetryLimit int
// AutoMount lets the recorder mount an SMB share itself when the host has
// not already mounted it. Linux only, and requires root.
AutoMount bool
// MountPoint overrides where AutoMount attaches the share.
MountPoint string
} }
type ProcessingConfig struct { type ProcessingConfig struct {
@ -95,6 +103,8 @@ var defaultConfig = Config{
Password: "s3tkY6tzA&KN6M", Password: "s3tkY6tzA&KN6M",
Timeout: 30 * time.Second, Timeout: 30 * time.Second,
RetryLimit: 3, RetryLimit: 3,
AutoMount: false,
MountPoint: "",
}, },
Processing: ProcessingConfig{ Processing: ProcessingConfig{
Enabled: true, Enabled: true,
@ -175,8 +185,25 @@ func (c *Config) loadFromEnvironment() error {
c.NAS.EnableTransfer = val == "true" c.NAS.EnableTransfer = val == "true"
} }
if val := os.Getenv("NAS_AUTO_MOUNT"); val != "" {
c.NAS.AutoMount = val == "true"
}
if val := os.Getenv("NAS_MOUNT_POINT"); val != "" {
c.NAS.MountPoint = val
}
if val := os.Getenv("LOCAL_OUTPUT_DIR"); val != "" { if val := os.Getenv("LOCAL_OUTPUT_DIR"); val != "" {
c.Paths.LocalOutput = val c.Paths.LocalOutput = val
// The manifest describes the event directory and is documented as
// sitting beside it, so it follows the output directory. Without this
// it stays in the default data/ while the segments move elsewhere.
c.Paths.ManifestDir = val
}
// Applied after LOCAL_OUTPUT_DIR so an explicit setting wins.
if val := os.Getenv("MANIFEST_DIR"); val != "" {
c.Paths.ManifestDir = val
} }
if val := os.Getenv("PROCESS_OUTPUT_DIR"); val != "" { if val := os.Getenv("PROCESS_OUTPUT_DIR"); val != "" {

View File

@ -8,12 +8,22 @@ import (
"m3u8-downloader/pkg/utils" "m3u8-downloader/pkg/utils"
"os" "os"
"sort" "sort"
"strconv"
"sync"
) )
type ManifestWriter struct { type ManifestWriter struct {
ManifestPath string ManifestPath string
Segments []ManifestItem Segments []ManifestItem
Index map[string]*ManifestItem // Index maps a sequence number to its position in Segments. It holds
// positions rather than pointers because append reallocates the backing
// array, which would leave previously handed-out element pointers aimed at
// a stale copy.
Index map[string]int
// mu guards the fields above: every variant downloads concurrently, and
// each reports its segments as they land.
mu sync.Mutex
} }
type ManifestItem struct { type ManifestItem struct {
@ -26,40 +36,55 @@ func NewManifestWriter(eventName string) *ManifestWriter {
return &ManifestWriter{ return &ManifestWriter{
ManifestPath: cfg.GetManifestPath(eventName), ManifestPath: cfg.GetManifestPath(eventName),
Segments: make([]ManifestItem, 0), Segments: make([]ManifestItem, 0),
Index: make(map[string]*ManifestItem), Index: make(map[string]int),
} }
} }
// AddOrUpdateSegment records that a segment was downloaded, keeping the
// highest-resolution variant seen for each sequence number.
func (m *ManifestWriter) AddOrUpdateSegment(seqNo string, resolution string) { func (m *ManifestWriter) AddOrUpdateSegment(seqNo string, resolution string) {
m.mu.Lock()
defer m.mu.Unlock()
if m.Index == nil { if m.Index == nil {
m.Index = make(map[string]*ManifestItem) m.Index = make(map[string]int)
} }
if m.Segments == nil { if m.Segments == nil {
m.Segments = make([]ManifestItem, 0) m.Segments = make([]ManifestItem, 0)
} }
if existing, ok := m.Index[seqNo]; ok { if pos, ok := m.Index[seqNo]; ok {
if resolution > existing.Resolution { if ResolutionHeight(resolution) > ResolutionHeight(m.Segments[pos].Resolution) {
existing.Resolution = resolution m.Segments[pos].Resolution = resolution
} }
return return
} else {
item := ManifestItem{
SeqNo: seqNo,
Resolution: resolution,
} }
m.Segments = append(m.Segments, item)
m.Index[seqNo] = &item m.Segments = append(m.Segments, ManifestItem{SeqNo: seqNo, Resolution: resolution})
m.Index[seqNo] = len(m.Segments) - 1
} }
// Len reports how many distinct sequence numbers have been recorded.
func (m *ManifestWriter) Len() int {
m.mu.Lock()
defer m.mu.Unlock()
return len(m.Segments)
} }
func (m *ManifestWriter) WriteManifest() { func (m *ManifestWriter) WriteManifest() {
sort.Slice(m.Segments, func(i, j int) bool { // Sort a copy: reordering Segments in place would invalidate Index, and a
return m.Segments[i].SeqNo < m.Segments[j].SeqNo // recording may still be writing into it.
m.mu.Lock()
segments := make([]ManifestItem, len(m.Segments))
copy(segments, m.Segments)
m.mu.Unlock()
sort.Slice(segments, func(i, j int) bool {
return lessSeqNo(segments[i].SeqNo, segments[j].SeqNo)
}) })
data, err := json.MarshalIndent(m.Segments, "", " ") data, err := json.MarshalIndent(segments, "", " ")
if err != nil { if err != nil {
log.Printf("Failed to marshal manifest: %v", err) log.Printf("Failed to marshal manifest: %v", err)
logging.MustGetFactory().Download.Error("failed to marshal manifest", logging.KeyError, err.Error()) logging.MustGetFactory().Download.Error("failed to marshal manifest", logging.KeyError, err.Error())
@ -88,3 +113,15 @@ func (m *ManifestWriter) WriteManifest() {
return return
} }
} }
// lessSeqNo orders sequence numbers numerically, so segment 9 precedes segment
// 10 instead of sorting after it as the strings would. Values that are not
// numeric fall back to lexicographic order.
func lessSeqNo(a, b string) bool {
ai, aErr := strconv.ParseUint(a, 10, 64)
bi, bErr := strconv.ParseUint(b, 10, 64)
if aErr == nil && bErr == nil {
return ai < bi
}
return a < b
}

View File

@ -4,6 +4,8 @@ import (
"encoding/json" "encoding/json"
"os" "os"
"path/filepath" "path/filepath"
"strconv"
"sync"
"testing" "testing"
) )
@ -41,7 +43,7 @@ func TestManifestWriter_AddOrUpdateSegment(t *testing.T) {
writer := &ManifestWriter{ writer := &ManifestWriter{
ManifestPath: "test.json", ManifestPath: "test.json",
Segments: make([]ManifestItem, 0), Segments: make([]ManifestItem, 0),
Index: make(map[string]*ManifestItem), Index: make(map[string]int),
} }
// Test adding new segment // Test adding new segment
@ -112,7 +114,7 @@ func TestManifestWriter_WriteManifest(t *testing.T) {
writer := &ManifestWriter{ writer := &ManifestWriter{
ManifestPath: manifestPath, ManifestPath: manifestPath,
Segments: make([]ManifestItem, 0), Segments: make([]ManifestItem, 0),
Index: make(map[string]*ManifestItem), Index: make(map[string]int),
} }
// Add some test segments out of order // Add some test segments out of order
@ -172,7 +174,7 @@ func TestManifestWriter_WriteManifest_EmptySegments(t *testing.T) {
writer := &ManifestWriter{ writer := &ManifestWriter{
ManifestPath: manifestPath, ManifestPath: manifestPath,
Segments: make([]ManifestItem, 0), Segments: make([]ManifestItem, 0),
Index: make(map[string]*ManifestItem), Index: make(map[string]int),
} }
// Write empty manifest // Write empty manifest
@ -204,7 +206,7 @@ func TestManifestWriter_WriteManifest_InvalidPath(t *testing.T) {
writer := &ManifestWriter{ writer := &ManifestWriter{
ManifestPath: "/invalid/path/that/does/not/exist/manifest.json", ManifestPath: "/invalid/path/that/does/not/exist/manifest.json",
Segments: []ManifestItem{{SeqNo: "1001", Resolution: "1080p"}}, Segments: []ManifestItem{{SeqNo: "1001", Resolution: "1080p"}},
Index: make(map[string]*ManifestItem), Index: make(map[string]int),
} }
// This should not panic, just fail gracefully // This should not panic, just fail gracefully
@ -239,3 +241,169 @@ func TestManifestItem_JSONSerialization(t *testing.T) {
t.Errorf("Resolution mismatch: expected '%s', got '%s'", item.Resolution, unmarshaled.Resolution) t.Errorf("Resolution mismatch: expected '%s', got '%s'", item.Resolution, unmarshaled.Resolution)
} }
} }
func TestResolutionHeight(t *testing.T) {
cases := map[string]int{
"1080p": 1080,
"720p": 720,
"2160p": 2160,
"1440p": 1440,
"240p": 240,
// Not a <height>p label: must rank below every real resolution rather
// than accidentally outranking them.
"unknown": 0,
"": 0,
"p": 0,
}
for label, want := range cases {
if got := ResolutionHeight(label); got != want {
t.Errorf("ResolutionHeight(%q) = %d, want %d", label, got, want)
}
}
}
// TestAddOrUpdateSegmentKeepsHighestResolution pins the comparison used to
// choose between variants. Comparing the labels as strings — as this once did —
// ranks "720p" above "1080p", quietly downgrading the manifest.
func TestAddOrUpdateSegmentKeepsHighestResolution(t *testing.T) {
cases := []struct {
name string
arriving []string
want string
}{
{"higher replaces lower", []string{"720p", "1080p"}, "1080p"},
{"lower does not replace higher", []string{"1080p", "720p"}, "1080p"},
{"fewer digits must not win", []string{"1080p", "480p"}, "1080p"},
{"4k beats 1080p", []string{"1080p", "2160p"}, "2160p"},
{"unknown loses to a real resolution", []string{"unknown", "480p"}, "480p"},
{"unknown does not displace a real resolution", []string{"1080p", "unknown"}, "1080p"},
{"repeated identical resolution", []string{"720p", "720p"}, "720p"},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
w := &ManifestWriter{}
for _, res := range tc.arriving {
w.AddOrUpdateSegment("1001", res)
}
if len(w.Segments) != 1 {
t.Fatalf("expected a single entry, got %d", len(w.Segments))
}
if got := w.Segments[0].Resolution; got != tc.want {
t.Errorf("kept %q, want %q (arrival order %v)", got, tc.want, tc.arriving)
}
})
}
}
// TestAddOrUpdateSegmentUpdatesTheStoredEntry guards the aliasing bug: the
// index once held a pointer to a local copy rather than to the stored element,
// so updates were applied to an orphan and never reached the written manifest.
func TestAddOrUpdateSegmentUpdatesTheStoredEntry(t *testing.T) {
w := &ManifestWriter{}
// Enough entries to force at least one append reallocation, which would
// invalidate any pointer held into the backing array.
for i := 0; i < 64; i++ {
w.AddOrUpdateSegment(strconv.Itoa(1000+i), "480p")
}
for i := 0; i < 64; i++ {
w.AddOrUpdateSegment(strconv.Itoa(1000+i), "1080p")
}
if len(w.Segments) != 64 {
t.Fatalf("expected 64 entries, got %d", len(w.Segments))
}
for _, seg := range w.Segments {
if seg.Resolution != "1080p" {
t.Fatalf("segment %s kept %q; the upgrade never reached the stored entry", seg.SeqNo, seg.Resolution)
}
}
}
// TestAddOrUpdateSegmentIsConcurrencySafe reflects real use: every variant
// downloads in its own goroutine and reports segments as they land.
func TestAddOrUpdateSegmentIsConcurrencySafe(t *testing.T) {
w := &ManifestWriter{}
resolutions := []string{"240p", "480p", "720p", "1080p"}
var wg sync.WaitGroup
for _, res := range resolutions {
wg.Add(1)
go func(res string) {
defer wg.Done()
for seq := 0; seq < 200; seq++ {
w.AddOrUpdateSegment(strconv.Itoa(seq), res)
}
}(res)
}
wg.Wait()
if got := w.Len(); got != 200 {
t.Errorf("expected 200 distinct sequence numbers, got %d", got)
}
for _, seg := range w.Segments {
if seg.Resolution != "1080p" {
t.Errorf("segment %s = %q, want the highest variant 1080p", seg.SeqNo, seg.Resolution)
}
}
}
// TestWriteManifestSortsNumerically checks that segment 9 precedes segment 10,
// which string ordering would reverse.
func TestWriteManifestSortsNumerically(t *testing.T) {
dir := t.TempDir()
w := &ManifestWriter{ManifestPath: filepath.Join(dir, "event.json")}
for _, seq := range []string{"100", "9", "1000", "10", "1"} {
w.AddOrUpdateSegment(seq, "1080p")
}
w.WriteManifest()
data, err := os.ReadFile(w.ManifestPath)
if err != nil {
t.Fatalf("failed to read manifest: %v", err)
}
var items []ManifestItem
if err := json.Unmarshal(data, &items); err != nil {
t.Fatalf("manifest is not valid JSON: %v", err)
}
want := []string{"1", "9", "10", "100", "1000"}
if len(items) != len(want) {
t.Fatalf("expected %d entries, got %d", len(want), len(items))
}
for i, seq := range want {
if items[i].SeqNo != seq {
t.Errorf("position %d = %q, want %q (full order: %v)", i, items[i].SeqNo, seq, items)
}
}
}
// TestWriteManifestLeavesIndexUsable ensures writing mid-recording does not
// corrupt the writer: sorting in place would desynchronise Index from Segments.
func TestWriteManifestLeavesIndexUsable(t *testing.T) {
dir := t.TempDir()
w := &ManifestWriter{ManifestPath: filepath.Join(dir, "event.json")}
w.AddOrUpdateSegment("20", "480p")
w.AddOrUpdateSegment("10", "480p")
w.WriteManifest()
// After the write, an upgrade must still land on the right entry.
w.AddOrUpdateSegment("20", "1080p")
pos, ok := w.Index["20"]
if !ok {
t.Fatal("sequence 20 missing from the index after WriteManifest")
}
if got := w.Segments[pos].Resolution; got != "1080p" {
t.Errorf("sequence 20 = %q, want 1080p; the index no longer matches the slice", got)
}
if w.Segments[w.Index["10"]].SeqNo != "10" {
t.Error("index and slice disagree after WriteManifest")
}
}

View File

@ -13,6 +13,7 @@ import (
"net/http" "net/http"
"net/url" "net/url"
"path" "path"
"strconv"
"strings" "strings"
"time" "time"
) )
@ -24,7 +25,27 @@ type StreamVariant struct {
ID int ID int
Resolution string Resolution string
OutputDir string OutputDir string
Writer *ManifestWriter }
// ResolutionHeight converts a label produced by ExtractResolution back into
// the pixel height it stands for, so variants can be ranked by quality.
//
// The height is parsed rather than looked up in a table: labels come from the
// playlist's own RESOLUTION attribute, so a stream can legitimately offer
// 1440p or 2160p that no fixed table would anticipate. Anything not in
// <height>p form — "unknown", from a playlist with no variants — returns 0 and
// therefore ranks below every recognised resolution.
//
// Comparing the labels as strings instead is a trap worth naming: "720p" sorts
// after "1080p", so lexicographic comparison silently prefers lower quality.
func ResolutionHeight(resolution string) int {
trimmed := strings.TrimSuffix(strings.ToLower(strings.TrimSpace(resolution)), "p")
height, err := strconv.Atoi(trimmed)
if err != nil || height < 0 {
return 0
}
return height
} }
// ExtractResolution maps a variant to the label used for its output directory, // ExtractResolution maps a variant to the label used for its output directory,
@ -50,7 +71,10 @@ func ExtractResolution(variant *m3u8.Variant) string {
} }
} }
func GetAllVariants(masterURL string, outputDir string, writer *ManifestWriter) ([]*StreamVariant, error) { // GetAllVariants fetches the master playlist and returns one entry per stream
// variant. Segments are recorded into the manifest by VariantDownloader, which
// receives it directly, so no writer is threaded through here.
func GetAllVariants(masterURL string, outputDir string) ([]*StreamVariant, error) {
client := &http.Client{} client := &http.Client{}
req, _ := http.NewRequest("GET", masterURL, nil) req, _ := http.NewRequest("GET", masterURL, nil)
req.Header.Set("User-Agent", constants.HTTPUserAgent) req.Header.Set("User-Agent", constants.HTTPUserAgent)
@ -78,7 +102,6 @@ func GetAllVariants(masterURL string, outputDir string, writer *ManifestWriter)
ID: 0, ID: 0,
Resolution: "unknown", Resolution: "unknown",
OutputDir: path.Join(outputDir, "unknown"), OutputDir: path.Join(outputDir, "unknown"),
Writer: writer,
}}, nil }}, nil
} }
@ -166,6 +189,9 @@ func VariantDownloader(ctx context.Context, variant *StreamVariant, sem chan str
log.Printf("✓ %s downloaded segment %s", j.Variant.Resolution, name) log.Printf("✓ %s downloaded segment %s", j.Variant.Resolution, name)
dlLog.Debug("segment downloaded", logging.KeySegment, name) dlLog.Debug("segment downloaded", logging.KeySegment, name)
registry.RecordSegmentDownloaded(variant.Resolution) registry.RecordSegmentDownloaded(variant.Resolution)
if manifest != nil {
manifest.AddOrUpdateSegment(strconv.FormatUint(j.Seq, 10), variant.Resolution)
}
return return
} }

View File

@ -3,10 +3,18 @@ package nas
import "time" import "time"
type NASConfig struct { type NASConfig struct {
// Path is the destination as configured. It may be a UNC path, an
// smb:// URL, or a directory on this host; see ParseLocation.
Path string Path string
Username string Username string
Password string Password string
Timeout time.Duration Timeout time.Duration
RetryLimit int RetryLimit int
VerifySize bool VerifySize bool
// AutoMount allows the recorder to mount an SMB share itself when the
// host has not already mounted it. Linux only, and requires root, so it
// is off by default in favour of a clear "mount it first" error.
AutoMount bool
// MountPoint overrides where AutoMount attaches the share.
MountPoint string
} }

98
pkg/nas/connector.go Normal file
View File

@ -0,0 +1,98 @@
package nas
import (
"context"
"fmt"
"os/exec"
"runtime"
)
// Connector attaches a configured Location to the host filesystem so that
// ordinary file operations reach the NAS.
//
// Windows and Linux disagree about what "connecting to a share" means —
// Windows authenticates a UNC path in place, Linux needs the share mounted
// somewhere in its single filesystem tree — so each host gets its own
// implementation and NASService stays platform-neutral. Selection happens at
// runtime rather than through build tags so that every implementation is
// compiled, vetted and unit-testable on every platform; only the final exec
// call is genuinely host-specific.
type Connector interface {
// Name identifies the strategy in logs and check output.
Name() string
// Attach makes loc reachable and reports where to write.
Attach(ctx context.Context, loc Location, cfg NASConfig) (Attachment, error)
// Detach undoes whatever Attach set up. Implementations must undo only
// what they created: a share this process found already mounted belongs
// to the host, and unmounting it could break unrelated software.
Detach(ctx context.Context, att Attachment) error
}
// Attachment is the result of a successful Attach.
type Attachment struct {
// BasePath is the host-native directory that maps to the configured
// destination. Every file operation is resolved against it.
BasePath string
// Notes explain how the host reached the destination, for -check output
// and startup logs.
Notes []string
// Owned records that this process created the connection or mount, and
// so may tear it down on shutdown.
Owned bool
// mountPoint is the directory this process mounted, when Owned is set by
// the Linux connector.
mountPoint string
// share is the network path this process connected, when Owned is set by
// the Windows connector.
share string
}
// commandRunner executes an external command. Implementations keep argument
// construction in a pure function and call through this, so tests can assert
// exactly what would run without running it.
type commandRunner func(ctx context.Context, name string, args ...string) ([]byte, error)
// execRunner is the production runner: it combines stdout and stderr because
// mount and net both report the useful diagnosis on one or the other.
func execRunner(ctx context.Context, name string, args ...string) ([]byte, error) {
return exec.CommandContext(ctx, name, args...).CombinedOutput()
}
// connectorFor picks the strategy for a destination on a given OS. A
// destination that is already an ordinary directory needs no network step on
// any platform, so it short-circuits before the OS switch.
func connectorFor(loc Location, goos string) Connector {
if !loc.IsRemote() {
return &localConnector{goos: goos}
}
switch goos {
case "windows":
return &windowsConnector{run: execRunner}
case "linux":
return &linuxConnector{run: execRunner, mountsPath: procMountsPath}
default:
return &unsupportedConnector{goos: goos}
}
}
// defaultConnector selects the strategy for the running host.
func defaultConnector(loc Location) Connector {
return connectorFor(loc, runtime.GOOS)
}
// unsupportedConnector fails clearly on platforms this project does not target,
// rather than silently writing a share-shaped directory to local disk.
type unsupportedConnector struct {
goos string
}
func (c *unsupportedConnector) Name() string { return "unsupported" }
func (c *unsupportedConnector) Attach(_ context.Context, loc Location, _ NASConfig) (Attachment, error) {
return Attachment{}, fmt.Errorf(
"connecting to network share %s is not supported on %s; mount the share and set NAS_OUTPUT_PATH to the mount point",
loc.SMBSource(), c.goos)
}
func (c *unsupportedConnector) Detach(context.Context, Attachment) error { return nil }

View File

@ -0,0 +1,57 @@
package nas
import (
"context"
"fmt"
"path/filepath"
"strings"
)
// localConnector handles destinations that are already part of the host
// filesystem: a plain output directory, or — the normal arrangement on Linux —
// an SMB/NFS share mounted by fstab, autofs or systemd before the recorder
// starts. There is nothing to connect, but there is something to check.
type localConnector struct {
goos string
}
func (c *localConnector) Name() string { return "local" }
func (c *localConnector) Attach(_ context.Context, loc Location, _ NASConfig) (Attachment, error) {
if err := c.validate(loc.Raw); err != nil {
return Attachment{}, err
}
return Attachment{
BasePath: filepath.Clean(loc.Raw),
Notes: []string{"local or already-mounted path (no network authentication)"},
}, nil
}
func (c *localConnector) Detach(context.Context, Attachment) error { return nil }
// validate rejects a Windows-shaped path on a non-Windows host. This looks
// pedantic and is not: backslashes and colons are legal characters in Linux
// filenames, so a stray C:\data or \\server\share would be created verbatim as
// a local directory. Transfers would report success, cleanup would delete the
// local originals, and nothing would ever reach the NAS.
func (c *localConnector) validate(path string) error {
if c.goos == "windows" {
return nil
}
if driveLetter.MatchString(path) {
return fmt.Errorf(
"%q is a Windows drive path but this host is %s; set NAS_OUTPUT_PATH to a mount point such as /mnt/nas/streams",
path, c.goos)
}
if strings.Contains(path, `\`) {
return fmt.Errorf(
"%q contains backslashes but this host is %s, where they are ordinary filename characters rather than separators; "+
"set NAS_OUTPUT_PATH to a mount point such as /mnt/nas/streams, or to \\\\server\\share\\folder to have the share resolved automatically",
path, c.goos)
}
return nil
}

178
pkg/nas/connector_mount.go Normal file
View File

@ -0,0 +1,178 @@
package nas
import (
"context"
"fmt"
"os"
"path/filepath"
"strings"
)
// linuxConnector reaches a share the way Linux does: through a mount point in
// the single filesystem tree. There is no per-path authentication, so the
// question is not "can we connect" but "where is this share mounted".
//
// The default and by far the safest answer is that the host already mounted it
// (fstab, autofs, systemd, or `docker run -v`), so the connector looks the
// share up in the mount table. Mounting it here is opt-in because it requires
// root and changes system state that outlives a failed run.
type linuxConnector struct {
run commandRunner
mountsPath string
// geteuid is indirected for tests; production uses os.Geteuid.
geteuid func() int
}
func (c *linuxConnector) Name() string { return "mount" }
func (c *linuxConnector) Attach(ctx context.Context, loc Location, cfg NASConfig) (Attachment, error) {
entries, err := readMountTable(c.mountsPath)
if err != nil {
return Attachment{}, err
}
if point, rest, ok := findMount(entries, loc); ok {
return Attachment{
BasePath: filepath.Join(append([]string{point}, rest...)...),
Notes: []string{
"share: " + loc.SMBSource(),
fmt.Sprintf("resolved via existing mount at %s", point),
},
}, nil
}
if !cfg.AutoMount {
return Attachment{}, fmt.Errorf("%s", notMountedMessage(loc, cfg))
}
return c.mount(ctx, loc, cfg)
}
func (c *linuxConnector) Detach(ctx context.Context, att Attachment) error {
// Never unmount a share the host had already mounted; other software may
// be using it, and we did not put it there.
if !att.Owned || att.mountPoint == "" {
return nil
}
if out, err := c.run(ctx, "umount", att.mountPoint); err != nil {
return fmt.Errorf("failed to unmount %s: %v", att.mountPoint, summarize(err, out))
}
return nil
}
// mount performs an opt-in mount.cifs of the share.
func (c *linuxConnector) mount(ctx context.Context, loc Location, cfg NASConfig) (Attachment, error) {
geteuid := c.geteuid
if geteuid == nil {
geteuid = os.Geteuid
}
if geteuid() != 0 {
return Attachment{}, fmt.Errorf(
"NAS_AUTO_MOUNT is set but mounting %s requires root; run as root, or mount the share beforehand and %s",
loc.SMBSource(), suggestMountPointConfig(loc))
}
point := autoMountPoint(loc, cfg)
if err := os.MkdirAll(point, 0755); err != nil {
return Attachment{}, fmt.Errorf("failed to create mount point %s: %w", point, err)
}
credentialsPath, cleanup, err := writeCredentialsFile(cfg)
if err != nil {
return Attachment{}, err
}
defer cleanup()
if cfg.Timeout > 0 {
var cancel context.CancelFunc
ctx, cancel = context.WithTimeout(ctx, cfg.Timeout)
defer cancel()
}
if out, err := c.run(ctx, "mount", mountArgs(loc, point, credentialsPath)...); err != nil {
return Attachment{}, fmt.Errorf("failed to mount %s at %s: %v", loc.SMBSource(), point, summarize(err, out))
}
return Attachment{
BasePath: filepath.Join(append([]string{point}, loc.Sub...)...),
Owned: true,
mountPoint: point,
Notes: []string{
"share: " + loc.SMBSource(),
"user: " + usernameNote(cfg.Username),
fmt.Sprintf("mounted at %s by this process (will unmount on shutdown)", point),
},
}, nil
}
// mountArgs builds the mount.cifs invocation. Kept pure so it can be asserted
// without mounting anything.
func mountArgs(loc Location, mountPoint, credentialsPath string) []string {
options := "rw"
if credentialsPath != "" {
options += ",credentials=" + credentialsPath
} else {
options += ",guest"
}
return []string{"-t", "cifs", loc.SMBSource(), mountPoint, "-o", options}
}
// writeCredentialsFile stages the NAS credentials in a 0600 file. mount.cifs
// also accepts user=/pass= options, but those land in the process table where
// any local user can read the password; a credentials file does not.
func writeCredentialsFile(cfg NASConfig) (path string, cleanup func(), err error) {
if cfg.Username == "" || cfg.Password == "" {
return "", func() {}, nil
}
dir, err := os.MkdirTemp("", "streamrecorder-cifs-")
if err != nil {
return "", func() {}, fmt.Errorf("failed to stage NAS credentials: %w", err)
}
cleanup = func() { os.RemoveAll(dir) }
path = filepath.Join(dir, "credentials")
content := fmt.Sprintf("username=%s\npassword=%s\n", cfg.Username, cfg.Password)
if err := os.WriteFile(path, []byte(content), 0600); err != nil {
cleanup()
return "", func() {}, fmt.Errorf("failed to write NAS credentials file: %w", err)
}
return path, cleanup, nil
}
// autoMountPoint chooses where to mount the share when the operator has not
// named a location. Everything lands under one predictable directory so a
// leftover mount is easy to find and clean up.
func autoMountPoint(loc Location, cfg NASConfig) string {
if cfg.MountPoint != "" {
return cfg.MountPoint
}
return filepath.Join("/mnt", "streamrecorder", strings.ToLower(loc.Server), strings.ToLower(loc.Share))
}
// notMountedMessage explains the failure in terms of what the operator should
// actually type. A bare "no such mount" would leave them guessing.
func notMountedMessage(loc Location, cfg NASConfig) string {
source := loc.SMBSource()
example := filepath.Join("/mnt", strings.ToLower(loc.Share))
credentials := ""
if cfg.Username != "" {
credentials = fmt.Sprintf(",username=%s,password=***", cfg.Username)
}
return fmt.Sprintf(
"%s is not mounted on this host. Either mount it and %s, for example:\n"+
" sudo mkdir -p %s && sudo mount -t cifs %s %s -o rw%s\n"+
"or set NAS_AUTO_MOUNT=true to have the recorder mount it (requires root)",
source, suggestMountPointConfig(loc), example, source, example, credentials)
}
func suggestMountPointConfig(loc Location) string {
if len(loc.Sub) == 0 {
return "set NAS_OUTPUT_PATH to the mount point"
}
return fmt.Sprintf("set NAS_OUTPUT_PATH to the mount point plus %s", loc.SubPath())
}

107
pkg/nas/connector_netuse.go Normal file
View File

@ -0,0 +1,107 @@
package nas
import (
"context"
"fmt"
"path/filepath"
"strings"
)
// windowsConnector reaches a share the way Windows does: authenticate the UNC
// path with `net use`, then use the UNC path directly as a filesystem path.
// Nothing is mounted into a tree, so the resolved base path is the configured
// UNC path itself.
type windowsConnector struct {
run commandRunner
}
func (c *windowsConnector) Name() string { return "net use" }
func (c *windowsConnector) Attach(ctx context.Context, loc Location, cfg NASConfig) (Attachment, error) {
share := loc.ShareUNC()
att := Attachment{
BasePath: loc.UNC(),
share: share,
Notes: []string{"share: " + share, "user: " + usernameNote(cfg.Username)},
}
if cfg.Timeout > 0 {
var cancel context.CancelFunc
ctx, cancel = context.WithTimeout(ctx, cfg.Timeout)
defer cancel()
}
out, err := c.run(ctx, "net", netUseArgs(loc, cfg)...)
if err != nil {
// A failure here is not decisive. `net use` also fails when the share
// is already connected — commonly because Explorer or another process
// got there first — in which case the path is perfectly usable. Record
// the reason and let the caller's write probe deliver the verdict,
// rather than refusing to run against a working share.
att.Notes = append(att.Notes,
fmt.Sprintf("net use did not complete (%v); continuing, the share may already be connected", summarize(err, out)))
return att, nil
}
att.Owned = true
att.Notes = append(att.Notes, "connected via net use")
return att, nil
}
func (c *windowsConnector) Detach(ctx context.Context, att Attachment) error {
// Only drop connections this process made. Deleting a mapping the user or
// another application established would break them.
if !att.Owned || att.share == "" {
return nil
}
if out, err := c.run(ctx, "net", "use", att.share, "/delete"); err != nil {
return fmt.Errorf("failed to disconnect %s: %v", att.share, summarize(err, out))
}
return nil
}
// netUseArgs builds the `net use` invocation. Kept pure so the Windows code
// path can be asserted from any host, including the Linux box this is
// developed on, without executing anything.
func netUseArgs(loc Location, cfg NASConfig) []string {
args := []string{"use", loc.ShareUNC()}
if cfg.Username != "" && cfg.Password != "" {
args = append(args, "/user:"+cfg.Username, cfg.Password)
}
return append(args, "/persistent:no")
}
// summarize pairs an exec error with the first line of command output, which is
// where net and mount put the actual diagnosis.
func summarize(err error, out []byte) string {
text := strings.TrimSpace(string(out))
if text == "" {
return err.Error()
}
if idx := strings.IndexByte(text, '\n'); idx >= 0 {
text = text[:idx]
}
return fmt.Sprintf("%v: %s", err, text)
}
func usernameNote(username string) string {
if username == "" {
return "(none, using ambient credentials)"
}
return username
}
// joinBase resolves a relative path against a base that may be a UNC path.
// filepath.Join is correct on Windows and would mangle the leading \\ anywhere
// else, so UNC bases are joined with backslashes explicitly.
func joinBase(base string, elem ...string) string {
if len(elem) == 0 {
return base
}
if strings.HasPrefix(base, `\\`) {
parts := append([]string{strings.TrimRight(base, `\`)}, elem...)
return strings.Join(parts, `\`)
}
return filepath.Join(append([]string{base}, elem...)...)
}

353
pkg/nas/connector_test.go Normal file
View File

@ -0,0 +1,353 @@
package nas
import (
"context"
"fmt"
"os"
"path/filepath"
"strings"
"testing"
"time"
)
// recordingRunner captures the commands a connector would run instead of
// running them. This is what lets the Windows path be tested from a Linux
// development machine: the decision logic and the exact argv are asserted
// without ever invoking net.exe.
type recordingRunner struct {
calls [][]string
output []byte
err error
}
func (r *recordingRunner) run(_ context.Context, name string, args ...string) ([]byte, error) {
r.calls = append(r.calls, append([]string{name}, args...))
return r.output, r.err
}
func (r *recordingRunner) lastCall() string {
if len(r.calls) == 0 {
return ""
}
return strings.Join(r.calls[len(r.calls)-1], " ")
}
func remoteLocation(t *testing.T, raw string) Location {
t.Helper()
loc, err := ParseLocation(raw)
if err != nil {
t.Fatalf("ParseLocation(%q) failed: %v", raw, err)
}
return loc
}
func TestConnectorSelection(t *testing.T) {
remote := remoteLocation(t, `\\nas\dci\streams`)
local := remoteLocation(t, "/mnt/dci/streams")
cases := []struct {
name string
loc Location
goos string
want string
}{
{"windows share", remote, "windows", "net use"},
{"linux share", remote, "linux", "mount"},
{"darwin share", remote, "darwin", "unsupported"},
{"local path on windows", local, "windows", "local"},
{"local path on linux", local, "linux", "local"},
// A already-mounted path needs no network step even where shares are
// unsupported, so it must not fall through to the error connector.
{"local path on darwin", local, "darwin", "local"},
}
for _, tc := range cases {
if got := connectorFor(tc.loc, tc.goos).Name(); got != tc.want {
t.Errorf("%s: connectorFor = %q, want %q", tc.name, got, tc.want)
}
}
}
func TestWindowsConnectorAttach(t *testing.T) {
runner := &recordingRunner{}
c := &windowsConnector{run: runner.run}
loc := remoteLocation(t, `\\HomeLabNAS\dci\streams`)
att, err := c.Attach(context.Background(), loc, NASConfig{Username: "NASAdmin", Password: "secret"})
if err != nil {
t.Fatalf("Attach failed: %v", err)
}
// The base path is the full UNC path; net use authenticates only the share.
if want := `\\HomeLabNAS\dci\streams`; att.BasePath != want {
t.Errorf("BasePath = %q, want %q", att.BasePath, want)
}
if want := `net use \\HomeLabNAS\dci /user:NASAdmin secret /persistent:no`; runner.lastCall() != want {
t.Errorf("command = %q, want %q", runner.lastCall(), want)
}
if !att.Owned {
t.Error("a successful net use should be owned so it can be released on shutdown")
}
}
func TestWindowsConnectorOmitsCredentialsWhenUnset(t *testing.T) {
runner := &recordingRunner{}
c := &windowsConnector{run: runner.run}
if _, err := c.Attach(context.Background(), remoteLocation(t, `\\nas\dci`), NASConfig{}); err != nil {
t.Fatalf("Attach failed: %v", err)
}
if want := `net use \\nas\dci /persistent:no`; runner.lastCall() != want {
t.Errorf("command = %q, want %q", runner.lastCall(), want)
}
}
func TestWindowsConnectorContinuesWhenAlreadyConnected(t *testing.T) {
// net use fails when a connection already exists, but the share is usable;
// refusing to run would be wrong. The write probe is the real verdict.
runner := &recordingRunner{
err: fmt.Errorf("exit status 2"),
output: []byte("The network connection could not be found.\nMore help is available..."),
}
c := &windowsConnector{run: runner.run}
att, err := c.Attach(context.Background(), remoteLocation(t, `\\nas\dci`), NASConfig{})
if err != nil {
t.Fatalf("Attach should tolerate a net use failure, got: %v", err)
}
if att.Owned {
t.Error("a failed net use must not be marked owned, or Detach would drop someone else's connection")
}
notes := strings.Join(att.Notes, "\n")
if !strings.Contains(notes, "net use did not complete") {
t.Errorf("expected the failure to be recorded in notes:\n%s", notes)
}
// Only the first line of output is worth surfacing.
if strings.Contains(notes, "More help is available") {
t.Errorf("notes should not carry the full usage dump:\n%s", notes)
}
}
func TestWindowsConnectorDetachOnlyReleasesOwned(t *testing.T) {
runner := &recordingRunner{}
c := &windowsConnector{run: runner.run}
if err := c.Detach(context.Background(), Attachment{share: `\\nas\dci`, Owned: false}); err != nil {
t.Fatalf("Detach failed: %v", err)
}
if len(runner.calls) != 0 {
t.Errorf("must not disconnect a share this process did not connect, ran: %v", runner.calls)
}
if err := c.Detach(context.Background(), Attachment{share: `\\nas\dci`, Owned: true}); err != nil {
t.Fatalf("Detach failed: %v", err)
}
if want := `net use \\nas\dci /delete`; runner.lastCall() != want {
t.Errorf("command = %q, want %q", runner.lastCall(), want)
}
}
// writeMountTable stages a /proc/mounts-format file.
func writeMountTable(t *testing.T, lines ...string) string {
t.Helper()
path := filepath.Join(t.TempDir(), "mounts")
body := strings.Join(lines, "\n") + "\n"
if err := os.WriteFile(path, []byte(body), 0644); err != nil {
t.Fatalf("failed to write mount table: %v", err)
}
return path
}
func TestLinuxConnectorResolvesExistingMount(t *testing.T) {
mounts := writeMountTable(t,
"sysfs /sys sysfs rw,nosuid 0 0",
"//homelabnas/dci /mnt/dci cifs rw,relatime,username=nasadmin 0 0",
)
c := &linuxConnector{run: (&recordingRunner{}).run, mountsPath: mounts}
att, err := c.Attach(context.Background(), remoteLocation(t, `\\HomeLabNAS\dci\streams`), NASConfig{})
if err != nil {
t.Fatalf("Attach failed: %v", err)
}
// Server and share names are case-insensitive on SMB, so the configured
// \\HomeLabNAS\dci must match a //homelabnas/dci mount.
if want := "/mnt/dci/streams"; att.BasePath != want {
t.Errorf("BasePath = %q, want %q", att.BasePath, want)
}
if att.Owned {
t.Error("a pre-existing mount must not be owned, or shutdown would unmount it")
}
}
func TestLinuxConnectorPrefersDeepestMount(t *testing.T) {
// Both mounts could serve the destination; the more specific one is the
// only one guaranteed to actually contain it.
mounts := writeMountTable(t,
"//nas/dci /mnt/dci cifs rw 0 0",
"//nas/dci/streams /mnt/streams cifs rw 0 0",
)
c := &linuxConnector{run: (&recordingRunner{}).run, mountsPath: mounts}
att, err := c.Attach(context.Background(), remoteLocation(t, `\\nas\dci\streams\2026`), NASConfig{})
if err != nil {
t.Fatalf("Attach failed: %v", err)
}
if want := "/mnt/streams/2026"; att.BasePath != want {
t.Errorf("BasePath = %q, want %q", att.BasePath, want)
}
}
func TestLinuxConnectorMatchesNFSAndEscapedPoints(t *testing.T) {
mounts := writeMountTable(t,
`nas:/dci /mnt/my\040share nfs4 rw 0 0`,
)
c := &linuxConnector{run: (&recordingRunner{}).run, mountsPath: mounts}
att, err := c.Attach(context.Background(), remoteLocation(t, `\\nas\dci\streams`), NASConfig{})
if err != nil {
t.Fatalf("Attach failed: %v", err)
}
if want := "/mnt/my share/streams"; att.BasePath != want {
t.Errorf("BasePath = %q, want %q", att.BasePath, want)
}
}
func TestLinuxConnectorIgnoresNonNetworkMounts(t *testing.T) {
// A local directory that merely looks like the share must not match.
mounts := writeMountTable(t, "//nas/dci /mnt/dci ext4 rw 0 0")
c := &linuxConnector{run: (&recordingRunner{}).run, mountsPath: mounts}
if _, err := c.Attach(context.Background(), remoteLocation(t, `\\nas\dci`), NASConfig{}); err == nil {
t.Fatal("expected an ext4 mount not to satisfy a network share")
}
}
func TestLinuxConnectorUnmountedGivesActionableError(t *testing.T) {
mounts := writeMountTable(t, "sysfs /sys sysfs rw 0 0")
c := &linuxConnector{run: (&recordingRunner{}).run, mountsPath: mounts}
_, err := c.Attach(context.Background(), remoteLocation(t, `\\nas\dci\streams`), NASConfig{Username: "admin"})
if err == nil {
t.Fatal("expected an unmounted share to fail")
}
msg := err.Error()
for _, want := range []string{"//nas/dci", "mount -t cifs", "NAS_AUTO_MOUNT", "NAS_OUTPUT_PATH"} {
if !strings.Contains(msg, want) {
t.Errorf("error should mention %q:\n%s", want, msg)
}
}
// The remedy must not echo the real password.
if strings.Contains(msg, "hunter2") {
t.Error("error message leaked a credential")
}
}
func TestLinuxConnectorAutoMountRequiresRoot(t *testing.T) {
mounts := writeMountTable(t, "sysfs /sys sysfs rw 0 0")
runner := &recordingRunner{}
c := &linuxConnector{run: runner.run, mountsPath: mounts, geteuid: func() int { return 1000 }}
_, err := c.Attach(context.Background(), remoteLocation(t, `\\nas\dci`), NASConfig{AutoMount: true})
if err == nil || !strings.Contains(err.Error(), "requires root") {
t.Fatalf("expected a root requirement error, got: %v", err)
}
if len(runner.calls) != 0 {
t.Errorf("must not attempt mount without privileges, ran: %v", runner.calls)
}
}
func TestLinuxConnectorAutoMount(t *testing.T) {
mounts := writeMountTable(t, "sysfs /sys sysfs rw 0 0")
runner := &recordingRunner{}
point := filepath.Join(t.TempDir(), "mnt", "dci")
c := &linuxConnector{run: runner.run, mountsPath: mounts, geteuid: func() int { return 0 }}
cfg := NASConfig{AutoMount: true, MountPoint: point, Username: "admin", Password: "hunter2", Timeout: 5 * time.Second}
att, err := c.Attach(context.Background(), remoteLocation(t, `\\nas\dci\streams`), cfg)
if err != nil {
t.Fatalf("Attach failed: %v", err)
}
if want := filepath.Join(point, "streams"); att.BasePath != want {
t.Errorf("BasePath = %q, want %q", att.BasePath, want)
}
if !att.Owned || att.mountPoint != point {
t.Errorf("a self-created mount must be owned and recorded, got owned=%v point=%q", att.Owned, att.mountPoint)
}
call := runner.lastCall()
if !strings.HasPrefix(call, "mount -t cifs //nas/dci "+point+" -o rw,credentials=") {
t.Errorf("unexpected mount command: %q", call)
}
// The password must reach mount.cifs through a credentials file, never
// through argv where any local user could read it from the process table.
if strings.Contains(call, "hunter2") {
t.Errorf("password leaked into the mount command line: %q", call)
}
// The staged credentials file must not outlive the mount call.
credPath := call[strings.Index(call, "credentials=")+len("credentials="):]
if _, err := os.Stat(credPath); !os.IsNotExist(err) {
t.Errorf("credentials file %s should have been removed, stat err = %v", credPath, err)
}
if err := c.Detach(context.Background(), att); err != nil {
t.Fatalf("Detach failed: %v", err)
}
if want := "umount " + point; runner.lastCall() != want {
t.Errorf("detach command = %q, want %q", runner.lastCall(), want)
}
}
func TestLinuxConnectorDetachLeavesForeignMounts(t *testing.T) {
runner := &recordingRunner{}
c := &linuxConnector{run: runner.run, mountsPath: writeMountTable(t)}
if err := c.Detach(context.Background(), Attachment{BasePath: "/mnt/dci", Owned: false}); err != nil {
t.Fatalf("Detach failed: %v", err)
}
if len(runner.calls) != 0 {
t.Errorf("must not unmount a share the host mounted, ran: %v", runner.calls)
}
}
func TestLocalConnectorRejectsWindowsPathsOffWindows(t *testing.T) {
c := &localConnector{goos: "linux"}
for _, raw := range []string{`C:\data\streams`, `data\streams`} {
loc, err := ParseLocation(raw)
if err != nil {
t.Fatalf("ParseLocation(%q) failed: %v", raw, err)
}
if _, err := c.Attach(context.Background(), loc, NASConfig{}); err == nil {
t.Errorf("%q should be rejected on Linux: backslashes are legal filename characters there, "+
"so it would silently create a local directory instead of writing to the NAS", raw)
}
}
}
func TestLocalConnectorAcceptsWindowsPathsOnWindows(t *testing.T) {
c := &localConnector{goos: "windows"}
loc, err := ParseLocation(`C:\data\streams`)
if err != nil {
t.Fatalf("ParseLocation failed: %v", err)
}
if _, err := c.Attach(context.Background(), loc, NASConfig{}); err != nil {
t.Errorf("a drive path is valid on Windows: %v", err)
}
}
func TestUnsupportedConnectorExplainsItself(t *testing.T) {
c := &unsupportedConnector{goos: "plan9"}
_, err := c.Attach(context.Background(), remoteLocation(t, `\\nas\dci`), NASConfig{})
if err == nil {
t.Fatal("expected an error on an unsupported platform")
}
if !strings.Contains(err.Error(), "plan9") || !strings.Contains(err.Error(), "NAS_OUTPUT_PATH") {
t.Errorf("error should name the platform and the remedy: %v", err)
}
}

144
pkg/nas/location.go Normal file
View File

@ -0,0 +1,144 @@
package nas
import (
"fmt"
"path/filepath"
"regexp"
"strings"
)
// Location is a configured NAS destination parsed into a host-neutral form.
//
// The same destination is spelled differently depending on where you're
// standing: Windows wants \\HomeLabNAS\dci\streams, a Linux host reaches the
// same bytes through a mount point like /mnt/dci/streams, and either host may
// simply be handed a plain local directory. Parsing happens once, up front,
// and platform-specific rendering is left to the Connector implementations.
type Location struct {
// Raw is the destination exactly as configured, for error messages.
Raw string
// Server is the host serving the share; empty for a plain local path.
Server string
// Share is the top-level export name; empty for a plain local path.
Share string
// Sub holds the path components below the share, if any.
Sub []string
}
// driveLetter matches a Windows drive-qualified path such as C:\data.
var driveLetter = regexp.MustCompile(`^[A-Za-z]:[\\/]`)
// ParseLocation accepts every spelling of a destination this project supports:
// Windows UNC (\\server\share\dir), the SMB source form (//server/share/dir),
// a smb:// URL, or a plain local/mounted directory. It does no I/O and makes no
// assumptions about the host OS, so the result is safe to compute anywhere.
func ParseLocation(raw string) (Location, error) {
trimmed := strings.TrimSpace(raw)
if trimmed == "" {
return Location{}, fmt.Errorf("NAS path is empty")
}
loc := Location{Raw: trimmed}
remainder, isRemote := stripRemotePrefix(trimmed)
if !isRemote {
return loc, nil
}
parts := splitPathComponents(remainder)
if len(parts) < 2 {
return Location{}, fmt.Errorf(
"%q names a server but no share; expected a form like \\\\server\\share\\folder", trimmed)
}
loc.Server = parts[0]
loc.Share = parts[1]
loc.Sub = parts[2:]
return loc, nil
}
// stripRemotePrefix reports whether raw names a network share and returns the
// server/share/subpath portion when it does. A leading // is treated as a share
// rather than a local path: POSIX leaves paths beginning with exactly two
// slashes implementation-defined, and nobody configures a NAS destination that
// way by accident.
func stripRemotePrefix(raw string) (string, bool) {
lower := strings.ToLower(raw)
switch {
case strings.HasPrefix(lower, "smb://"):
return raw[len("smb://"):], true
case strings.HasPrefix(lower, "cifs://"):
return raw[len("cifs://"):], true
case strings.HasPrefix(raw, `\\`):
return raw[2:], true
case strings.HasPrefix(raw, "//"):
return raw[2:], true
default:
return raw, false
}
}
// splitPathComponents splits on either separator and drops empty components, so
// that \\server\share\\folder\ and //server/share/folder agree.
func splitPathComponents(path string) []string {
fields := strings.FieldsFunc(path, func(r rune) bool { return r == '/' || r == '\\' })
out := make([]string, 0, len(fields))
for _, f := range fields {
if f != "" {
out = append(out, f)
}
}
return out
}
// IsRemote reports whether reaching this destination requires attaching a
// network share first, as opposed to writing to a directory that is already
// part of the local filesystem.
func (l Location) IsRemote() bool {
return l.Server != "" && l.Share != ""
}
// UNC renders the destination in Windows form: \\server\share\sub.
func (l Location) UNC() string {
if !l.IsRemote() {
return l.Raw
}
return `\\` + strings.Join(append([]string{l.Server, l.Share}, l.Sub...), `\`)
}
// ShareUNC renders just the share in Windows form: \\server\share. This is the
// unit `net use` authenticates against; the subpath below it is an ordinary
// directory on the connected share.
func (l Location) ShareUNC() string {
if !l.IsRemote() {
return ""
}
return `\\` + l.Server + `\` + l.Share
}
// SMBSource renders the share in the //server/share form used by mount tables
// and by mount.cifs. It deliberately omits Sub: that part is a path *within*
// the mounted share, not part of the share's identity.
func (l Location) SMBSource() string {
if !l.IsRemote() {
return ""
}
return "//" + l.Server + "/" + l.Share
}
// SubPath renders the components below the share using the host's separator, or
// "" when the destination is the share root.
func (l Location) SubPath() string {
if len(l.Sub) == 0 {
return ""
}
return filepath.Join(l.Sub...)
}
// String describes the destination for logs and check output.
func (l Location) String() string {
if l.IsRemote() {
return l.UNC()
}
return l.Raw
}

123
pkg/nas/location_test.go Normal file
View File

@ -0,0 +1,123 @@
package nas
import (
"strings"
"testing"
)
func TestParseLocationRemoteForms(t *testing.T) {
// Every spelling of the same destination must parse identically, so that
// an operator can configure it the way their platform's documentation
// describes without changing behaviour.
equivalent := []string{
`\\HomeLabNAS\dci\streams`,
`//HomeLabNAS/dci/streams`,
`smb://HomeLabNAS/dci/streams`,
`SMB://HomeLabNAS/dci/streams`,
`cifs://HomeLabNAS/dci/streams`,
`\\HomeLabNAS\dci\streams\`,
`\\HomeLabNAS\\dci\\streams`,
}
for _, raw := range equivalent {
loc, err := ParseLocation(raw)
if err != nil {
t.Errorf("ParseLocation(%q) failed: %v", raw, err)
continue
}
if !loc.IsRemote() {
t.Errorf("ParseLocation(%q) should be remote", raw)
}
if loc.Server != "HomeLabNAS" || loc.Share != "dci" {
t.Errorf("ParseLocation(%q) = server %q share %q, want HomeLabNAS/dci", raw, loc.Server, loc.Share)
}
if len(loc.Sub) != 1 || loc.Sub[0] != "streams" {
t.Errorf("ParseLocation(%q) sub = %v, want [streams]", raw, loc.Sub)
}
}
}
func TestParseLocationRendering(t *testing.T) {
loc, err := ParseLocation(`//HomeLabNAS/dci/streams/2026`)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if got, want := loc.UNC(), `\\HomeLabNAS\dci\streams\2026`; got != want {
t.Errorf("UNC() = %q, want %q", got, want)
}
if got, want := loc.ShareUNC(), `\\HomeLabNAS\dci`; got != want {
t.Errorf("ShareUNC() = %q, want %q", got, want)
}
if got, want := loc.SMBSource(), "//HomeLabNAS/dci"; got != want {
t.Errorf("SMBSource() = %q, want %q", got, want)
}
}
func TestParseLocationShareRoot(t *testing.T) {
loc, err := ParseLocation(`\\nas\media`)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if len(loc.Sub) != 0 {
t.Errorf("expected no subpath, got %v", loc.Sub)
}
if got, want := loc.UNC(), `\\nas\media`; got != want {
t.Errorf("UNC() = %q, want %q", got, want)
}
}
func TestParseLocationLocalPaths(t *testing.T) {
for _, raw := range []string{"/mnt/nas/streams", "data/streams", `C:\data\streams`, "/mnt"} {
loc, err := ParseLocation(raw)
if err != nil {
t.Errorf("ParseLocation(%q) failed: %v", raw, err)
continue
}
if loc.IsRemote() {
t.Errorf("ParseLocation(%q) should not be treated as a network share", raw)
}
if loc.Raw != raw {
t.Errorf("ParseLocation(%q).Raw = %q", raw, loc.Raw)
}
}
}
func TestParseLocationRejectsIncomplete(t *testing.T) {
for _, raw := range []string{"", " ", `\\`, `\\server`, "//server", "smb://server"} {
if _, err := ParseLocation(raw); err == nil {
t.Errorf("ParseLocation(%q) should have failed", raw)
}
}
}
func TestParseLocationTrimsWhitespace(t *testing.T) {
loc, err := ParseLocation(" \\\\nas\\dci ")
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if loc.Server != "nas" || loc.Share != "dci" {
t.Errorf("whitespace not trimmed: server %q share %q", loc.Server, loc.Share)
}
}
func TestJoinBaseKeepsUNCSeparators(t *testing.T) {
// filepath.Join would produce \\nas\dci/event on Linux, which is not a
// path Windows would accept if the value were ever written to state.
got := joinBase(`\\nas\dci`, "event", "1080p")
if want := `\\nas\dci\event\1080p`; got != want {
t.Errorf("joinBase UNC = %q, want %q", got, want)
}
if got, want := joinBase(`\\nas\dci\`, "event"), `\\nas\dci\event`; got != want {
t.Errorf("joinBase trailing separator = %q, want %q", got, want)
}
if got := joinBase("/mnt/dci"); got != "/mnt/dci" {
t.Errorf("joinBase with no elements = %q", got)
}
if got, want := joinBase("/mnt/dci", "event"), "/mnt/dci/event"; !strings.HasSuffix(got, want) {
t.Errorf("joinBase local = %q, want suffix %q", got, want)
}
}

135
pkg/nas/mounts.go Normal file
View File

@ -0,0 +1,135 @@
package nas
import (
"bufio"
"fmt"
"os"
"strconv"
"strings"
)
// procMountsPath is the kernel's view of what is mounted in this process's
// mount namespace, which is the right question to ask inside a container too.
const procMountsPath = "/proc/mounts"
// mountEntry is one line of the mount table.
type mountEntry struct {
Source string
Point string
FSType string
}
// networkFilesystems are the mount types that can plausibly be serving a NAS
// share. Restricting the search avoids matching, say, a local directory that
// merely happens to be named like a server.
var networkFilesystems = map[string]bool{
"cifs": true,
"smb3": true,
"smbfs": true,
"nfs": true,
"nfs4": true,
"fuse.smbnetfs": true,
}
// readMountTable parses /proc/mounts (or any file in its format). Fields are
// whitespace-separated: source, mount point, type, options, dump, pass.
func readMountTable(path string) ([]mountEntry, error) {
f, err := os.Open(path)
if err != nil {
return nil, fmt.Errorf("failed to read mount table %s: %w", path, err)
}
defer f.Close()
var entries []mountEntry
scanner := bufio.NewScanner(f)
for scanner.Scan() {
fields := strings.Fields(scanner.Text())
if len(fields) < 3 {
continue
}
entries = append(entries, mountEntry{
Source: unescapeMountField(fields[0]),
Point: unescapeMountField(fields[1]),
FSType: fields[2],
})
}
if err := scanner.Err(); err != nil {
return nil, fmt.Errorf("failed to scan mount table %s: %w", path, err)
}
return entries, nil
}
// unescapeMountField decodes the octal escapes the kernel uses for characters
// that would otherwise break the whitespace-separated format — most often \040
// for a space in a mount point.
func unescapeMountField(field string) string {
if !strings.Contains(field, `\`) {
return field
}
var b strings.Builder
for i := 0; i < len(field); i++ {
if field[i] == '\\' && i+3 < len(field) {
if v, err := strconv.ParseUint(field[i+1:i+4], 8, 8); err == nil {
b.WriteByte(byte(v))
i += 3
continue
}
}
b.WriteByte(field[i])
}
return b.String()
}
// findMount locates the mount serving loc and returns its mount point plus the
// path components still to be appended below it.
//
// The search runs deepest-first because a share may be mounted at any level:
// //nas/dci might be mounted at /mnt/dci, or //nas/dci/streams mounted directly
// at /mnt/streams. Preferring the longest matching source keeps both working
// and avoids pointing at a parent that the mount does not actually cover.
func findMount(entries []mountEntry, loc Location) (point string, rest []string, ok bool) {
if !loc.IsRemote() {
return "", nil, false
}
for depth := len(loc.Sub); depth >= 0; depth-- {
for _, candidate := range mountSourceForms(loc, depth) {
for _, entry := range entries {
if !networkFilesystems[entry.FSType] {
continue
}
if normalizeMountSource(entry.Source) == candidate {
return entry.Point, loc.Sub[depth:], true
}
}
}
}
return "", nil, false
}
// mountSourceForms renders the share, plus its first `depth` subdirectories, in
// the spellings a mount table might use for it.
func mountSourceForms(loc Location, depth int) []string {
segments := append([]string{loc.Share}, loc.Sub[:depth]...)
tail := strings.Join(segments, "/")
return []string{
normalizeMountSource("//" + loc.Server + "/" + tail), // cifs/smb3
normalizeMountSource(loc.Server + ":/" + tail), // nfs
}
}
// normalizeMountSource makes sources comparable: SMB servers and shares are
// case-insensitive, separators may be either slash, and a trailing slash is not
// meaningful.
func normalizeMountSource(source string) string {
s := strings.ReplaceAll(source, `\`, "/")
s = strings.ToLower(strings.TrimSpace(s))
if len(s) > 1 {
s = strings.TrimRight(s, "/")
}
return s
}

96
pkg/nas/mounts_test.go Normal file
View File

@ -0,0 +1,96 @@
package nas
import (
"os"
"testing"
)
func TestUnescapeMountField(t *testing.T) {
cases := map[string]string{
`/mnt/plain`: "/mnt/plain",
`/mnt/my\040share`: "/mnt/my share",
`/mnt/tab\011here`: "/mnt/tab\there",
`/mnt/back\134slash`: `/mnt/back\slash`,
`//nas/dci`: "//nas/dci",
`/mnt/not\9octal`: `/mnt/not\9octal`,
`/mnt/trailing\`: `/mnt/trailing\`,
}
for in, want := range cases {
if got := unescapeMountField(in); got != want {
t.Errorf("unescapeMountField(%q) = %q, want %q", in, got, want)
}
}
}
func TestNormalizeMountSource(t *testing.T) {
cases := map[string]string{
"//NAS/DCI": "//nas/dci",
"//nas/dci/": "//nas/dci",
`\\nas\dci`: "//nas/dci",
" //nas/dci ": "//nas/dci",
"/": "/",
}
for in, want := range cases {
if got := normalizeMountSource(in); got != want {
t.Errorf("normalizeMountSource(%q) = %q, want %q", in, got, want)
}
}
}
func TestReadMountTableSkipsMalformedLines(t *testing.T) {
path := writeMountTable(t,
"source-only",
"source /point-but-no-type",
"",
"//nas/dci /mnt/dci cifs rw 0 0",
)
entries, err := readMountTable(path)
if err != nil {
t.Fatalf("readMountTable failed: %v", err)
}
if len(entries) != 1 {
t.Fatalf("expected 1 usable entry, got %d: %v", len(entries), entries)
}
if entries[0].Source != "//nas/dci" || entries[0].Point != "/mnt/dci" || entries[0].FSType != "cifs" {
t.Errorf("unexpected entry: %+v", entries[0])
}
}
func TestReadMountTableMissingFile(t *testing.T) {
if _, err := readMountTable("/definitely/not/a/mount/table"); err == nil {
t.Error("expected an error for a missing mount table")
}
}
// TestReadRealMountTable closes the loop between the fixtures above and the
// kernel's actual output: the parser must cope with whatever this host has
// mounted, not just hand-written examples.
func TestReadRealMountTable(t *testing.T) {
if _, err := os.Stat(procMountsPath); err != nil {
t.Skipf("%s is not available on this platform", procMountsPath)
}
entries, err := readMountTable(procMountsPath)
if err != nil {
t.Fatalf("failed to parse the host's %s: %v", procMountsPath, err)
}
if len(entries) == 0 {
t.Fatalf("%s parsed to zero entries, which cannot be right", procMountsPath)
}
foundRoot := false
for _, e := range entries {
if e.Point == "" || e.FSType == "" {
t.Errorf("entry parsed with empty fields: %+v", e)
}
if e.Point == "/" {
foundRoot = true
}
}
if !foundRoot {
t.Error("expected the root filesystem in the host's mount table")
}
}

View File

@ -6,31 +6,168 @@ import (
"io" "io"
"log" "log"
"os" "os"
"os/exec" "sync"
"path/filepath"
"strings"
) )
// NASService writes to a NAS destination without caring how this host reaches
// it. Parsing the destination and attaching it are delegated to a Location and
// a Connector; everything below operates on the resolved base path, which is
// always an ordinary path this host's filesystem understands.
type NASService struct { type NASService struct {
Config NASConfig Config NASConfig
location Location
connector Connector
mu sync.RWMutex
attached Attachment
connected bool connected bool
} }
func NewNASService(config NASConfig) *NASService { // New parses the configured destination and selects a connection strategy for
nt := &NASService{ // the running host. It performs no I/O and does not touch the network, so it is
Config: config, // safe to call before deciding whether a transfer will actually happen; call
} // Connect to attach the destination.
func New(config NASConfig) (*NASService, error) {
// Establish network connection with credentials before accessing the path loc, err := ParseLocation(config.Path)
if err := nt.EstablishConnection(); err != nil {
log.Fatalf("Failed to establish network connection to %s: %v", nt.Config.Path, err)
}
err := nt.EnsureDirectoryExists(nt.Config.Path)
if err != nil { if err != nil {
log.Fatalf("Failed to create directory %s: %v", nt.Config.Path, err) return nil, err
} }
return nt
return &NASService{
Config: config,
location: loc,
connector: defaultConnector(loc),
}, nil
}
// Connect attaches the destination, ensures the target directory exists, and
// proves it is writable. The write probe matters: on both platforms a share can
// be reachable and still reject writes, and finding that out at startup beats
// discovering it mid-recording.
func (nt *NASService) Connect(ctx context.Context) error {
att, err := nt.connector.Attach(ctx, nt.location, nt.Config)
if err != nil {
return err
}
nt.mu.Lock()
nt.attached = att
nt.mu.Unlock()
// From here on the attachment exists, so any failure has to undo it —
// otherwise a mount this process created would outlive the failed startup
// and be left behind on the host.
if err := nt.EnsureDirectoryExists(att.BasePath); err != nil {
nt.abandon(ctx, att)
return fmt.Errorf("%s is reachable but %s is unusable: %w", nt.location, att.BasePath, err)
}
if err := nt.writeProbe(att.BasePath); err != nil {
nt.abandon(ctx, att)
return err
}
nt.mu.Lock()
nt.connected = true
nt.mu.Unlock()
log.Printf("Connected to NAS at %s", att.BasePath)
return nil
}
// abandon releases a half-finished attachment after Connect fails. The
// original error is the one worth reporting, so a teardown failure is only
// logged.
func (nt *NASService) abandon(ctx context.Context, att Attachment) {
// Keep the notes: they explain how far the host got, which is exactly what
// the caller needs to report. Everything else is cleared so the service
// cannot be mistaken for usable and Detach cannot run twice.
nt.mu.Lock()
nt.attached = Attachment{Notes: att.Notes}
nt.connected = false
nt.mu.Unlock()
if err := nt.connector.Detach(ctx, att); err != nil {
log.Printf("Warning: failed to release NAS attachment after a failed connect: %v", err)
}
}
// writeProbe creates and removes a file to confirm the destination accepts
// writes.
func (nt *NASService) writeProbe(basePath string) error {
testFile := joinBase(basePath, ".connection_test")
f, err := os.Create(testFile)
if err != nil {
return fmt.Errorf("%s is not writable: %w", basePath, err)
}
f.Close()
if err := os.Remove(testFile); err != nil {
log.Printf("Warning: failed to remove connection test file %s: %v", testFile, err)
}
return nil
}
// Disconnect releases only what Connect established. A share that was already
// mounted or already connected when the recorder started is left exactly as it
// was found.
func (nt *NASService) Disconnect(ctx context.Context) error {
nt.mu.Lock()
att := nt.attached
nt.attached = Attachment{}
nt.connected = false
nt.mu.Unlock()
if err := nt.connector.Detach(ctx, att); err != nil {
return err
}
if att.Owned {
log.Printf("Released NAS connection to %s", nt.location)
}
return nil
}
// LocalPath is the resolved base directory on this host, valid after Connect.
func (nt *NASService) LocalPath() string {
nt.mu.RLock()
defer nt.mu.RUnlock()
return nt.attached.BasePath
}
// Join resolves elements against the NAS base path using whatever separator
// this host's base path requires.
func (nt *NASService) Join(elem ...string) string {
return joinBase(nt.LocalPath(), elem...)
}
// Location returns the parsed destination.
func (nt *NASService) Location() Location { return nt.location }
// ConnectionInfo describes how this host reaches the destination, for check
// output and startup logs. Safe to call before Connect, when it reports only
// what is known from configuration.
func (nt *NASService) ConnectionInfo() []string {
nt.mu.RLock()
defer nt.mu.RUnlock()
info := []string{
"configured: " + nt.location.String(),
"strategy: " + nt.connector.Name(),
}
info = append(info, nt.attached.Notes...)
if nt.attached.BasePath != "" && nt.attached.BasePath != nt.location.Raw {
info = append(info, "resolved path: "+nt.attached.BasePath)
}
return info
}
func (nt *NASService) IsConnected() bool {
nt.mu.RLock()
defer nt.mu.RUnlock()
return nt.connected
} }
func (nt *NASService) CopyFile(ctx context.Context, srcPath, destPath string) error { func (nt *NASService) CopyFile(ctx context.Context, srcPath, destPath string) error {
@ -89,87 +226,10 @@ func (nt *NASService) EnsureDirectoryExists(path string) error {
return nil return nil
} }
func (nt *NASService) EstablishConnection() error { // FileExists checks whether a file already exists on the NAS, optionally
networkPath := nt.ExtractNetworkPath(nt.Config.Path) // verifying its size. destinationPath is relative to the NAS base path.
if networkPath == "" {
return nil // local path, no network mount needed
}
log.Printf("Establishing network connection to %s with user %s", networkPath, nt.Config.Username)
var cmd *exec.Cmd
if nt.Config.Username != "" && nt.Config.Password != "" {
cmd = exec.Command("net", "use", networkPath, "/user:"+nt.Config.Username, nt.Config.Password, "/persistent:no")
} else {
cmd = exec.Command("net", "use", networkPath, "/persistent:no")
}
output, err := cmd.CombinedOutput()
if err != nil {
return fmt.Errorf("failed to establish network connection: %w\nOutput: %s", err, string(output))
}
log.Printf("Network connection established successfully")
return nil
}
func (nt *NASService) ExtractNetworkPath(fullPath string) string {
// Extract \\server\share from paths like \\server\share\folder\subfolder
if !strings.HasPrefix(fullPath, "\\\\") {
return "" // Not a UNC path
}
parts := strings.Split(fullPath[2:], "\\") // Remove leading \\
if len(parts) < 2 {
return "" // Invalid UNC path
}
return "\\\\" + parts[0] + "\\" + parts[1]
}
func (nt *NASService) TestConnection() error {
testFile := filepath.Join(nt.Config.Path, ".connection_test")
f, err := os.Create(testFile)
if err != nil {
return fmt.Errorf("Failed to create test file: %w", err)
}
f.Close()
os.Remove(testFile)
nt.connected = true
log.Printf("Connected to NAS at %s", nt.Config.Path)
return nil
}
func (nt *NASService) IsConnected() bool {
return nt.connected
}
// Disconnect removes the network connection
func (nt *NASService) Disconnect() error {
networkPath := nt.ExtractNetworkPath(nt.Config.Path)
if networkPath == "" {
return nil // Local path, nothing to disconnect
}
cmd := exec.Command("net", "use", networkPath, "/delete")
output, err := cmd.CombinedOutput()
if err != nil {
log.Printf("Warning: failed to disconnect from %s: %v\nOutput: %s", networkPath, err, string(output))
// Don't return error since this is cleanup
} else {
log.Printf("Disconnected from network path: %s", networkPath)
}
nt.connected = false
return nil
}
// FileExists checks if a file already exists on the NAS and optionally verifies size
func (nt *NASService) FileExists(destinationPath string, expectedSize int64) (bool, error) { func (nt *NASService) FileExists(destinationPath string, expectedSize int64) (bool, error) {
fullDestPath := filepath.Join(nt.Config.Path, destinationPath) fullDestPath := nt.Join(destinationPath)
destInfo, err := os.Stat(fullDestPath) destInfo, err := os.Stat(fullDestPath)
if err != nil { if err != nil {
@ -189,9 +249,9 @@ func (nt *NASService) FileExists(destinationPath string, expectedSize int64) (bo
return true, nil return true, nil
} }
// GetFileSize returns the size of a file on the NAS // GetFileSize returns the size of a file on the NAS, relative to the base path.
func (nt *NASService) GetFileSize(destinationPath string) (int64, error) { func (nt *NASService) GetFileSize(destinationPath string) (int64, error) {
fullDestPath := filepath.Join(nt.Config.Path, destinationPath) fullDestPath := nt.Join(destinationPath)
destInfo, err := os.Stat(fullDestPath) destInfo, err := os.Stat(fullDestPath)
if err != nil { if err != nil {

308
pkg/nas/nas_test.go Normal file
View File

@ -0,0 +1,308 @@
package nas
import (
"context"
"fmt"
"os"
"path/filepath"
"strings"
"testing"
)
// fakeConnector stands in for a platform strategy so the service's own
// behaviour — probing, path resolution, teardown — can be tested without a NAS.
type fakeConnector struct {
basePath string
attachErr error
owned bool
attachCount int
detachCount int
detached Attachment
}
func (f *fakeConnector) Name() string { return "fake" }
func (f *fakeConnector) Attach(context.Context, Location, NASConfig) (Attachment, error) {
f.attachCount++
if f.attachErr != nil {
return Attachment{}, f.attachErr
}
return Attachment{BasePath: f.basePath, Owned: f.owned, Notes: []string{"fake attachment"}}, nil
}
func (f *fakeConnector) Detach(_ context.Context, att Attachment) error {
f.detachCount++
f.detached = att
return nil
}
func newTestService(t *testing.T, configuredPath string, conn Connector) *NASService {
t.Helper()
svc, err := New(NASConfig{Path: configuredPath})
if err != nil {
t.Fatalf("New(%q) failed: %v", configuredPath, err)
}
svc.connector = conn
return svc
}
func TestNewRejectsUnusableDestination(t *testing.T) {
if _, err := New(NASConfig{Path: ""}); err == nil {
t.Error("an empty NAS path should be rejected")
}
if _, err := New(NASConfig{Path: `\\server`}); err == nil {
t.Error("a server without a share should be rejected")
}
}
func TestNewDoesNoIO(t *testing.T) {
// Constructing the service must not touch the network or filesystem, so
// that modes which never transfer can build one harmlessly.
svc, err := New(NASConfig{Path: `\\nas\dci\streams`})
if err != nil {
t.Fatalf("New failed: %v", err)
}
if svc.IsConnected() {
t.Error("service should not report connected before Connect")
}
if svc.LocalPath() != "" {
t.Errorf("no path should be resolved before Connect, got %q", svc.LocalPath())
}
}
func TestConnectResolvesAndProbes(t *testing.T) {
base := filepath.Join(t.TempDir(), "streams")
conn := &fakeConnector{basePath: base}
svc := newTestService(t, `\\nas\dci\streams`, conn)
if err := svc.Connect(context.Background()); err != nil {
t.Fatalf("Connect failed: %v", err)
}
if !svc.IsConnected() {
t.Error("service should report connected")
}
if svc.LocalPath() != base {
t.Errorf("LocalPath = %q, want %q", svc.LocalPath(), base)
}
// Connect creates the destination directory if it is missing.
if info, err := os.Stat(base); err != nil || !info.IsDir() {
t.Errorf("Connect should have created %s: %v", base, err)
}
// The write probe must clean up after itself.
if _, err := os.Stat(filepath.Join(base, ".connection_test")); !os.IsNotExist(err) {
t.Errorf("probe file should have been removed, stat err = %v", err)
}
}
func TestConnectSurfacesAttachFailure(t *testing.T) {
conn := &fakeConnector{attachErr: fmt.Errorf("share is not mounted")}
svc := newTestService(t, `\\nas\dci`, conn)
err := svc.Connect(context.Background())
if err == nil || !strings.Contains(err.Error(), "not mounted") {
t.Fatalf("expected the connector's error to surface, got: %v", err)
}
if svc.IsConnected() {
t.Error("a failed attach must not leave the service marked connected")
}
}
func TestConnectFailsWhenDestinationUnwritable(t *testing.T) {
// A file where the directory should be: reachable, but unusable.
blocker := filepath.Join(t.TempDir(), "blocker")
if err := os.WriteFile(blocker, []byte("x"), 0644); err != nil {
t.Fatalf("setup failed: %v", err)
}
svc := newTestService(t, "/tmp/whatever", &fakeConnector{basePath: filepath.Join(blocker, "streams")})
if err := svc.Connect(context.Background()); err == nil {
t.Fatal("expected an unusable destination to fail the connect")
}
if svc.IsConnected() {
t.Error("service should not report connected after a failed probe")
}
}
func TestFailedProbeReleasesTheAttachment(t *testing.T) {
// A mount created by Attach must not outlive a failed Connect, or a failed
// startup leaves a mount behind on the host.
blocker := filepath.Join(t.TempDir(), "blocker")
if err := os.WriteFile(blocker, []byte("x"), 0644); err != nil {
t.Fatalf("setup failed: %v", err)
}
conn := &fakeConnector{basePath: filepath.Join(blocker, "streams"), owned: true}
svc := newTestService(t, `\\nas\dci`, conn)
if err := svc.Connect(context.Background()); err == nil {
t.Fatal("expected Connect to fail")
}
if conn.detachCount != 1 {
t.Errorf("Detach called %d times after a failed connect, want 1", conn.detachCount)
}
if !conn.detached.Owned {
t.Error("Detach should have been told the attachment was owned")
}
if svc.LocalPath() != "" {
t.Errorf("a released attachment should expose no path, got %q", svc.LocalPath())
}
// The diagnosis of how far it got must survive for the caller to report.
if info := strings.Join(svc.ConnectionInfo(), "\n"); !strings.Contains(info, "fake attachment") {
t.Errorf("connection notes should survive a failed connect:\n%s", info)
}
// A later Disconnect must not detach a second time.
if err := svc.Disconnect(context.Background()); err != nil {
t.Fatalf("Disconnect failed: %v", err)
}
if conn.detachCount != 2 || conn.detached.Owned {
t.Errorf("Disconnect after release should be a no-op teardown, detachCount=%d owned=%v",
conn.detachCount, conn.detached.Owned)
}
}
func TestJoinUsesResolvedPath(t *testing.T) {
base := t.TempDir()
svc := newTestService(t, `\\nas\dci\streams`, &fakeConnector{basePath: base})
if err := svc.Connect(context.Background()); err != nil {
t.Fatalf("Connect failed: %v", err)
}
// Crucially the join is against the resolved path, not the configured UNC
// path — that is the whole point of the indirection.
if got, want := svc.Join("event", "1080p"), filepath.Join(base, "event", "1080p"); got != want {
t.Errorf("Join = %q, want %q", got, want)
}
}
func TestFileExistsAndGetFileSizeUseResolvedPath(t *testing.T) {
base := t.TempDir()
svc := newTestService(t, `\\nas\dci`, &fakeConnector{basePath: base})
if err := svc.Connect(context.Background()); err != nil {
t.Fatalf("Connect failed: %v", err)
}
rel := filepath.Join("event", "seg.ts")
if err := os.MkdirAll(filepath.Join(base, "event"), 0755); err != nil {
t.Fatalf("setup failed: %v", err)
}
if err := os.WriteFile(filepath.Join(base, rel), []byte("0123456789"), 0644); err != nil {
t.Fatalf("setup failed: %v", err)
}
exists, err := svc.FileExists(rel, 10)
if err != nil || !exists {
t.Errorf("FileExists = %v, %v; want true, nil", exists, err)
}
// A size mismatch means a partial transfer, which should not count.
exists, err = svc.FileExists(rel, 99)
if err != nil || exists {
t.Errorf("FileExists with wrong size = %v, %v; want false, nil", exists, err)
}
exists, err = svc.FileExists(filepath.Join("event", "missing.ts"), 0)
if err != nil || exists {
t.Errorf("FileExists for missing file = %v, %v; want false, nil", exists, err)
}
size, err := svc.GetFileSize(rel)
if err != nil || size != 10 {
t.Errorf("GetFileSize = %d, %v; want 10, nil", size, err)
}
}
func TestDisconnectDelegatesAndResets(t *testing.T) {
conn := &fakeConnector{basePath: t.TempDir(), owned: true}
svc := newTestService(t, `\\nas\dci`, conn)
if err := svc.Connect(context.Background()); err != nil {
t.Fatalf("Connect failed: %v", err)
}
if err := svc.Disconnect(context.Background()); err != nil {
t.Fatalf("Disconnect failed: %v", err)
}
if conn.detachCount != 1 {
t.Errorf("Detach called %d times, want 1", conn.detachCount)
}
if !conn.detached.Owned {
t.Error("Detach should receive the attachment so it knows what it owns")
}
if svc.IsConnected() {
t.Error("service should not report connected after Disconnect")
}
}
func TestConnectionInfoDescribesResolution(t *testing.T) {
base := t.TempDir()
svc := newTestService(t, `\\nas\dci\streams`, &fakeConnector{basePath: base})
// Before connecting, only configuration is known.
before := strings.Join(svc.ConnectionInfo(), "\n")
if !strings.Contains(before, `\\nas\dci\streams`) || !strings.Contains(before, "fake") {
t.Errorf("pre-connect info should name the destination and strategy:\n%s", before)
}
if err := svc.Connect(context.Background()); err != nil {
t.Fatalf("Connect failed: %v", err)
}
after := strings.Join(svc.ConnectionInfo(), "\n")
if !strings.Contains(after, "resolved path: "+base) {
t.Errorf("post-connect info should report where the destination resolved to:\n%s", after)
}
}
func TestVerifyTransferDetectsSizeMismatch(t *testing.T) {
dir := t.TempDir()
src := filepath.Join(dir, "src.ts")
dst := filepath.Join(dir, "dst.ts")
if err := os.WriteFile(src, []byte("0123456789"), 0644); err != nil {
t.Fatalf("setup failed: %v", err)
}
if err := os.WriteFile(dst, []byte("012"), 0644); err != nil {
t.Fatalf("setup failed: %v", err)
}
svc := newTestService(t, dir, &fakeConnector{basePath: dir})
if err := svc.VerifyTransfer(src, dst); err == nil {
t.Error("a truncated destination should fail verification")
}
if err := os.WriteFile(dst, []byte("0123456789"), 0644); err != nil {
t.Fatalf("setup failed: %v", err)
}
if err := svc.VerifyTransfer(src, dst); err != nil {
t.Errorf("matching sizes should verify: %v", err)
}
}
func TestCopyFile(t *testing.T) {
dir := t.TempDir()
src := filepath.Join(dir, "src.ts")
dst := filepath.Join(dir, "dst.ts")
payload := []byte("segment payload")
if err := os.WriteFile(src, payload, 0644); err != nil {
t.Fatalf("setup failed: %v", err)
}
svc := newTestService(t, dir, &fakeConnector{basePath: dir})
if err := svc.CopyFile(context.Background(), src, dst); err != nil {
t.Fatalf("CopyFile failed: %v", err)
}
got, err := os.ReadFile(dst)
if err != nil {
t.Fatalf("failed to read copy: %v", err)
}
if string(got) != string(payload) {
t.Errorf("copied contents = %q, want %q", got, payload)
}
if err := svc.CopyFile(context.Background(), filepath.Join(dir, "missing.ts"), dst); err == nil {
t.Error("copying a missing source should fail")
}
}

View File

@ -39,11 +39,19 @@ func NewProcessingService(eventName string, cfg *config.Config) (*ProcessingServ
Timeout: cfg.NAS.Timeout, Timeout: cfg.NAS.Timeout,
RetryLimit: cfg.NAS.RetryLimit, RetryLimit: cfg.NAS.RetryLimit,
VerifySize: true, VerifySize: true,
AutoMount: cfg.NAS.AutoMount,
MountPoint: cfg.NAS.MountPoint,
} }
nasService := nas.NewNASService(nasConfig) nasService, err := nas.New(nasConfig)
if err != nil {
logging.MustGetFactory().Processing.Error("invalid NAS destination", logging.KeyError, err.Error())
return nil, fmt.Errorf("invalid NAS destination: %w", err)
}
if err := nasService.TestConnection(); err != nil { connectCtx, cancel := context.WithTimeout(context.Background(), cfg.NAS.Timeout)
defer cancel()
if err := nasService.Connect(connectCtx); err != nil {
logging.MustGetFactory().Processing.Error("failed to connect to NAS", logging.KeyError, err.Error()) logging.MustGetFactory().Processing.Error("failed to connect to NAS", logging.KeyError, err.Error())
return nil, fmt.Errorf("failed to connect to NAS: %w", err) return nil, fmt.Errorf("failed to connect to NAS: %w", err)
} }
@ -59,9 +67,17 @@ func (ps *ProcessingService) GetSegmentInfo() (map[int]string, error) {
return nil, nil return nil, nil
} }
// nasEventPath is the event's directory on the NAS as reached from this host.
// It goes through the NAS service rather than config.GetNASEventPath because
// the configured path may be a UNC path that this host reaches via a mount
// point instead.
func (ps *ProcessingService) nasEventPath() string {
return ps.nas.Join(ps.eventName)
}
func (ps *ProcessingService) GetEventDirs() ([]string, error) { func (ps *ProcessingService) GetEventDirs() ([]string, error) {
if ps.eventName == "" { if ps.eventName == "" {
sourcePath := ps.config.NAS.OutputPath sourcePath := ps.nas.LocalPath()
dirs, err := os.ReadDir(sourcePath) dirs, err := os.ReadDir(sourcePath)
if err != nil { if err != nil {
return nil, fmt.Errorf("failed to read directory %s: %w", sourcePath, err) return nil, fmt.Errorf("failed to read directory %s: %w", sourcePath, err)
@ -177,7 +193,7 @@ func (ps *ProcessingService) Start(ctx context.Context) error {
} }
func (ps *ProcessingService) GetResolutions() ([]string, error) { func (ps *ProcessingService) GetResolutions() ([]string, error) {
eventPath := ps.config.GetNASEventPath(ps.eventName) eventPath := ps.nasEventPath()
dirs, err := os.ReadDir(eventPath) dirs, err := os.ReadDir(eventPath)
if err != nil { if err != nil {
return nil, fmt.Errorf("failed to read source directory %s: %w", eventPath, err) return nil, fmt.Errorf("failed to read source directory %s: %w", eventPath, err)
@ -198,7 +214,7 @@ func (ps *ProcessingService) GetResolutions() ([]string, error) {
func (ps *ProcessingService) ParseResolutionDirectory(resolution string, ch chan<- SegmentInfo, wg *sync.WaitGroup) { func (ps *ProcessingService) ParseResolutionDirectory(resolution string, ch chan<- SegmentInfo, wg *sync.WaitGroup) {
defer wg.Done() defer wg.Done()
resolutionPath := utils.SafeJoin(ps.config.GetNASEventPath(ps.eventName), resolution) resolutionPath := utils.SafeJoin(ps.nasEventPath(), resolution)
files, err := os.ReadDir(resolutionPath) files, err := os.ReadDir(resolutionPath)
if err != nil { if err != nil {
log.Printf("Failed to read resolution directory %s: %v", resolutionPath, err) log.Printf("Failed to read resolution directory %s: %v", resolutionPath, err)
@ -227,22 +243,14 @@ func (ps *ProcessingService) ParseResolutionDirectory(resolution string, ch chan
func (ps *ProcessingService) AggregateSegmentInfo(ch <-chan SegmentInfo) (map[int]SegmentInfo, error) { func (ps *ProcessingService) AggregateSegmentInfo(ch <-chan SegmentInfo) (map[int]SegmentInfo, error) {
segmentMap := make(map[int]SegmentInfo) segmentMap := make(map[int]SegmentInfo)
rank := map[string]int{ // Each sequence number arrives once per downloaded variant; keep the
"1080p": 1, // highest-resolution copy of each, since that is the one worth stitching
"720p": 2, // into the final video.
"540p": 3,
"480p": 4,
"450p": 5,
"360p": 6,
"270p": 7,
"240p": 8,
}
for segment := range ch { for segment := range ch {
fmt.Printf("Received segment %s in resolution %s \n", segment.Name, segment.Resolution) fmt.Printf("Received segment %s in resolution %s \n", segment.Name, segment.Resolution)
logging.MustGetFactory().Processing.Debug("segment received", logging.KeySegment, segment.Name, logging.KeyResolution, segment.Resolution) logging.MustGetFactory().Processing.Debug("segment received", logging.KeySegment, segment.Name, logging.KeyResolution, segment.Resolution)
current, exists := segmentMap[segment.SeqNo] current, exists := segmentMap[segment.SeqNo]
if !exists || rank[segment.Resolution] > rank[current.Resolution] { if !exists || resolutionHeight(segment.Resolution) > resolutionHeight(current.Resolution) {
segmentMap[segment.SeqNo] = segment segmentMap[segment.SeqNo] = segment
} }
} }
@ -250,6 +258,24 @@ func (ps *ProcessingService) AggregateSegmentInfo(ch <-chan SegmentInfo) (map[in
return segmentMap, nil return segmentMap, nil
} }
// resolutionHeight converts a resolution label such as "1080p" into the pixel
// height it stands for, so variants can be ranked by quality.
//
// The height is parsed rather than looked up in a table: labels come from the
// playlist's own RESOLUTION attribute, so a stream can legitimately offer
// 1440p or 2160p that no fixed table would anticipate. Anything not in
// <height>p form — "unknown", from a playlist with no variants — returns 0 and
// therefore loses to every recognised resolution.
func resolutionHeight(resolution string) int {
trimmed := strings.TrimSuffix(strings.ToLower(strings.TrimSpace(resolution)), "p")
height, err := strconv.Atoi(trimmed)
if err != nil || height < 0 {
return 0
}
return height
}
func (ps *ProcessingService) WriteConcatFile(segmentMap map[int]SegmentInfo) (string, error) { func (ps *ProcessingService) WriteConcatFile(segmentMap map[int]SegmentInfo) (string, error) {
concatPath := ps.config.GetProcessOutputPath(ps.eventName) concatPath := ps.config.GetProcessOutputPath(ps.eventName)
@ -273,7 +299,7 @@ func (ps *ProcessingService) WriteConcatFile(segmentMap map[int]SegmentInfo) (st
for _, seq := range keys { for _, seq := range keys {
segment := segmentMap[seq] segment := segmentMap[seq]
filePath := utils.SafeJoin(ps.config.GetNASEventPath(ps.eventName), segment.Resolution, segment.Name) filePath := utils.SafeJoin(ps.nasEventPath(), segment.Resolution, segment.Name)
line := fmt.Sprintf("file '%s'\n", filePath) line := fmt.Sprintf("file '%s'\n", filePath)
if _, err := f.WriteString(line); err != nil { if _, err := f.WriteString(line); err != nil {
return "", fmt.Errorf("failed to write to concat file: %w", err) return "", fmt.Errorf("failed to write to concat file: %w", err)

View File

@ -1,7 +1,9 @@
package processing package processing
import ( import (
"context"
"m3u8-downloader/pkg/config" "m3u8-downloader/pkg/config"
"m3u8-downloader/pkg/nas"
"os" "os"
"path/filepath" "path/filepath"
"runtime" "runtime"
@ -10,6 +12,22 @@ import (
"time" "time"
) )
// connectedNAS attaches a NAS service to a local directory. Reaching the NAS
// is platform-specific, but an already-mounted or plain directory resolves the
// same way everywhere, which is what makes this usable as a test double.
func connectedNAS(t *testing.T, path string) *nas.NASService {
t.Helper()
svc, err := nas.New(nas.NASConfig{Path: path, Timeout: 5 * time.Second})
if err != nil {
t.Fatalf("Failed to create NAS service for %s: %v", path, err)
}
if err := svc.Connect(context.Background()); err != nil {
t.Fatalf("Failed to connect NAS service to %s: %v", path, err)
}
return svc
}
func createTestConfig(tempDir string) *config.Config { func createTestConfig(tempDir string) *config.Config {
return &config.Config{ return &config.Config{
Core: config.CoreConfig{ Core: config.CoreConfig{
@ -87,6 +105,7 @@ func TestProcessingService_GetEventDirs(t *testing.T) {
ps := &ProcessingService{ ps := &ProcessingService{
config: cfg, config: cfg,
eventName: "", // Empty to test directory discovery eventName: "", // Empty to test directory discovery
nas: connectedNAS(t, cfg.NAS.OutputPath),
} }
dirs, err := ps.GetEventDirs() dirs, err := ps.GetEventDirs()
@ -114,12 +133,15 @@ func TestProcessingService_GetEventDirs(t *testing.T) {
} }
func TestProcessingService_GetEventDirs_WithEventName(t *testing.T) { func TestProcessingService_GetEventDirs_WithEventName(t *testing.T) {
cfg := createTestConfig("/tmp") // Scoped to a temp dir rather than /tmp directly: attaching the NAS
// service creates the destination, and tests should not leave one behind.
cfg := createTestConfig(t.TempDir())
eventName := "specific-event" eventName := "specific-event"
ps := &ProcessingService{ ps := &ProcessingService{
config: cfg, config: cfg,
eventName: eventName, eventName: eventName,
nas: connectedNAS(t, cfg.NAS.OutputPath),
} }
dirs, err := ps.GetEventDirs() dirs, err := ps.GetEventDirs()
@ -156,6 +178,7 @@ func TestProcessingService_GetResolutions(t *testing.T) {
ps := &ProcessingService{ ps := &ProcessingService{
config: cfg, config: cfg,
eventName: eventName, eventName: eventName,
nas: connectedNAS(t, cfg.NAS.OutputPath),
} }
resolutions, err := ps.GetResolutions() resolutions, err := ps.GetResolutions()
@ -248,6 +271,7 @@ func TestProcessingService_WriteConcatFile(t *testing.T) {
ps := &ProcessingService{ ps := &ProcessingService{
config: cfg, config: cfg,
eventName: eventName, eventName: eventName,
nas: connectedNAS(t, cfg.NAS.OutputPath),
} }
// Create test segment map // Create test segment map
@ -382,3 +406,67 @@ func TestProcessJob_Structure(t *testing.T) {
t.Errorf("Expected EventName='test-event', got '%s'", job.EventName) t.Errorf("Expected EventName='test-event', got '%s'", job.EventName)
} }
} }
func TestResolutionHeight(t *testing.T) {
cases := map[string]int{
"1080p": 1080,
"720p": 720,
"2160p": 2160, // beyond anything a fixed quality table would list
"1440p": 1440,
"240p": 240,
// Not in <height>p form: must rank below every real resolution rather
// than accidentally outranking them.
"unknown": 0,
"": 0,
"p": 0,
"-720p": 0,
}
for label, want := range cases {
if got := resolutionHeight(label); got != want {
t.Errorf("resolutionHeight(%q) = %d, want %d", label, got, want)
}
}
}
// TestAggregateSegmentInfoPrefersHighestResolution covers the variants a
// hardcoded rank table could not express: resolutions above 1080p, and the
// "unknown" label produced for a playlist with no variants.
func TestAggregateSegmentInfoPrefersHighestResolution(t *testing.T) {
cases := []struct {
name string
arriving []string
want string
}{
{"1080p beats 720p regardless of order", []string{"720p", "1080p"}, "1080p"},
{"1080p beats 720p when it arrives first", []string{"1080p", "720p"}, "1080p"},
{"4k beats 1080p", []string{"1080p", "2160p"}, "2160p"},
{"1440p beats 1080p", []string{"1080p", "1440p"}, "1440p"},
{"a known resolution beats unknown", []string{"unknown", "480p"}, "480p"},
{"unknown does not displace a known resolution", []string{"1080p", "unknown"}, "1080p"},
{"unknown alone is still kept", []string{"unknown"}, "unknown"},
{"lowest only", []string{"240p"}, "240p"},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
ps := &ProcessingService{}
ch := make(chan SegmentInfo, len(tc.arriving))
for _, res := range tc.arriving {
ch <- SegmentInfo{Name: "seg_1001.ts", SeqNo: 1001, Resolution: res}
}
close(ch)
segments, err := ps.AggregateSegmentInfo(ch)
if err != nil {
t.Fatalf("AggregateSegmentInfo() failed: %v", err)
}
if len(segments) != 1 {
t.Fatalf("expected a single sequence number, got %d", len(segments))
}
if got := segments[1001].Resolution; got != tc.want {
t.Errorf("kept %q, want %q (arrival order: %v)", got, tc.want, tc.arriving)
}
})
}
}

View File

@ -2,6 +2,7 @@ package transfer
import ( import (
"context" "context"
"errors"
"fmt" "fmt"
"log" "log"
"m3u8-downloader/pkg/logging" "m3u8-downloader/pkg/logging"
@ -10,6 +11,11 @@ import (
"time" "time"
) )
// errRetained reports that a file is still inside its retention window. It is
// not a failure: the file simply has to be reconsidered on a later pass rather
// than being discarded from the pending list.
var errRetained = errors.New("file is within the retention period")
type CleanupService struct { type CleanupService struct {
config CleanupConfig config CleanupConfig
pendingFiles []string pendingFiles []string
@ -82,29 +88,43 @@ func (cs *CleanupService) ExecuteCleanup(ctx context.Context) error {
log.Printf("Processing %d files for cleanup", len(batch)) log.Printf("Processing %d files for cleanup", len(batch))
var cleanedCount int var cleanedCount int
var errors []error var failures []error
// requeue collects files this pass must not drop: ones still inside their
// retention window, and — if the context is cancelled mid-batch — the
// remainder that was never looked at. They were already removed from
// pendingFiles above, so without this they would never be cleaned at all.
var requeue []string
for _, filePath := range batch { for i, filePath := range batch {
select { select {
case <-ctx.Done(): case <-ctx.Done():
requeue = append(requeue, batch[i:]...)
cs.requeue(requeue)
return ctx.Err() return ctx.Err()
default: default:
} }
if err := cs.cleanupFile(filePath); err != nil { err := cs.cleanupFile(filePath)
errors = append(errors, fmt.Errorf("Failed to cleanup file %s: %w", filePath, err)) switch {
} else { case errors.Is(err, errRetained):
requeue = append(requeue, filePath)
case err != nil:
failures = append(failures, fmt.Errorf("Failed to cleanup file %s: %w", filePath, err))
default:
cleanedCount++ cleanedCount++
} }
} }
log.Printf("Cleanup batch completed (cleaned: %d, errors: %d)", cleanedCount, len(errors)) cs.requeue(requeue)
logging.MustGetFactory().Transfer.Info("cleanup batch completed", "cleaned", cleanedCount, "errors", len(errors))
if len(errors) > 0 { log.Printf("Cleanup batch completed (cleaned: %d, retained: %d, errors: %d)", cleanedCount, len(requeue), len(failures))
for i, err := range errors { logging.MustGetFactory().Transfer.Info("cleanup batch completed",
"cleaned", cleanedCount, "retained", len(requeue), "errors", len(failures))
if len(failures) > 0 {
for i, err := range failures {
if i >= 3 { if i >= 3 {
log.Printf("... and %d more errors", len(errors)-3) log.Printf("... and %d more errors", len(failures)-3)
break break
} }
log.Printf("Error: %v", err) log.Printf("Error: %v", err)
@ -113,7 +133,17 @@ func (cs *CleanupService) ExecuteCleanup(ctx context.Context) error {
} }
return nil return nil
}
// requeue returns files to the pending list for a later pass.
func (cs *CleanupService) requeue(files []string) {
if len(files) == 0 {
return
}
cs.mu.Lock()
defer cs.mu.Unlock()
cs.pendingFiles = append(cs.pendingFiles, files...)
} }
func (cs *CleanupService) cleanupFile(filePath string) error { func (cs *CleanupService) cleanupFile(filePath string) error {
@ -128,7 +158,7 @@ func (cs *CleanupService) cleanupFile(filePath string) error {
if cs.config.RetentionPeriod > 0 { if cs.config.RetentionPeriod > 0 {
if time.Since(info.ModTime()) < cs.config.RetentionPeriod { if time.Since(info.ModTime()) < cs.config.RetentionPeriod {
log.Printf("File too new to cleanup: %s", filePath) log.Printf("File too new to cleanup: %s", filePath)
return nil return errRetained
} }
} }
@ -162,6 +192,18 @@ func (cs *CleanupService) ForceCleanupAll(ctx context.Context) error {
return err return err
} }
// Stop when a pass makes no progress. Files still inside their
// retention window are requeued by design, so requiring the list to
// reach zero would spin here until the shutdown timeout expired.
cs.mu.Lock()
remaining := len(cs.pendingFiles)
cs.mu.Unlock()
if remaining >= pendingCount {
log.Printf("Force cleanup stopping with %d file(s) still pending (within retention period)", remaining)
break
}
select { select {
case <-ctx.Done(): case <-ctx.Done():
return ctx.Err() return ctx.Err()

View File

@ -0,0 +1,190 @@
package transfer
import (
"context"
"fmt"
"os"
"path/filepath"
"testing"
"time"
)
func writeAgedFile(t *testing.T, dir, name string, age time.Duration) string {
t.Helper()
path := filepath.Join(dir, name)
if err := os.WriteFile(path, []byte("segment"), 0644); err != nil {
t.Fatalf("failed to write %s: %v", path, err)
}
modTime := time.Now().Add(-age)
if err := os.Chtimes(path, modTime, modTime); err != nil {
t.Fatalf("failed to age %s: %v", path, err)
}
return path
}
func TestCleanupRemovesTransferredFiles(t *testing.T) {
dir := t.TempDir()
cs := NewCleanupService(CleanupConfig{Enabled: true, BatchSize: 10, CheckInterval: time.Second})
path := writeAgedFile(t, dir, "seg_1001.ts", 0)
if err := cs.ScheduleCleanup(path); err != nil {
t.Fatalf("ScheduleCleanup failed: %v", err)
}
if err := cs.ExecuteCleanup(context.Background()); err != nil {
t.Fatalf("ExecuteCleanup failed: %v", err)
}
if _, err := os.Stat(path); !os.IsNotExist(err) {
t.Errorf("file should have been removed, stat err = %v", err)
}
if got := cs.GetPendingCount(); got != 0 {
t.Errorf("pending count = %d, want 0", got)
}
}
// TestCleanupRetainsThenRemoves is the regression test for a file inside its
// retention window being dropped from the pending list: it was reported as
// cleaned, never deleted, and never reconsidered, so local files accumulated
// forever whenever RetainHours was set.
func TestCleanupRetainsThenRemoves(t *testing.T) {
dir := t.TempDir()
cs := NewCleanupService(CleanupConfig{
Enabled: true,
RetentionPeriod: time.Hour,
BatchSize: 10,
CheckInterval: time.Second,
})
fresh := writeAgedFile(t, dir, "fresh.ts", time.Minute)
old := writeAgedFile(t, dir, "old.ts", 2*time.Hour)
for _, p := range []string{fresh, old} {
if err := cs.ScheduleCleanup(p); err != nil {
t.Fatalf("ScheduleCleanup failed: %v", err)
}
}
if err := cs.ExecuteCleanup(context.Background()); err != nil {
t.Fatalf("ExecuteCleanup failed: %v", err)
}
if _, err := os.Stat(old); !os.IsNotExist(err) {
t.Errorf("a file past its retention period should be removed, stat err = %v", err)
}
if _, err := os.Stat(fresh); err != nil {
t.Errorf("a file inside its retention period should survive: %v", err)
}
if got := cs.GetPendingCount(); got != 1 {
t.Fatalf("the retained file must stay queued for a later pass, pending = %d", got)
}
// Once it ages past the window, the next pass must actually remove it.
past := time.Now().Add(-2 * time.Hour)
if err := os.Chtimes(fresh, past, past); err != nil {
t.Fatalf("failed to age file: %v", err)
}
if err := cs.ExecuteCleanup(context.Background()); err != nil {
t.Fatalf("ExecuteCleanup failed: %v", err)
}
if _, err := os.Stat(fresh); !os.IsNotExist(err) {
t.Errorf("file should have been removed once past retention, stat err = %v", err)
}
if got := cs.GetPendingCount(); got != 0 {
t.Errorf("pending count = %d, want 0", got)
}
}
// TestForceCleanupAllTerminatesWithRetainedFiles guards shutdown: retained
// files are requeued by design, so a loop demanding an empty list would spin
// until the shutdown timeout expired.
func TestForceCleanupAllTerminatesWithRetainedFiles(t *testing.T) {
dir := t.TempDir()
cs := NewCleanupService(CleanupConfig{
Enabled: true,
RetentionPeriod: time.Hour,
BatchSize: 10,
CheckInterval: time.Second,
})
if err := cs.ScheduleCleanup(writeAgedFile(t, dir, "fresh.ts", time.Minute)); err != nil {
t.Fatalf("ScheduleCleanup failed: %v", err)
}
done := make(chan error, 1)
go func() { done <- cs.ForceCleanupAll(context.Background()) }()
select {
case err := <-done:
if err != nil {
t.Fatalf("ForceCleanupAll failed: %v", err)
}
case <-time.After(3 * time.Second):
t.Fatal("ForceCleanupAll did not terminate with a retained file pending")
}
if got := cs.GetPendingCount(); got != 1 {
t.Errorf("the retained file should still be pending, got %d", got)
}
}
func TestCleanupMissingFileIsNotAnError(t *testing.T) {
cs := NewCleanupService(CleanupConfig{Enabled: true, BatchSize: 10, CheckInterval: time.Second})
// Already gone — a file transferred and removed by something else.
if err := cs.ScheduleCleanup(filepath.Join(t.TempDir(), "gone.ts")); err != nil {
t.Fatalf("ScheduleCleanup failed: %v", err)
}
if err := cs.ExecuteCleanup(context.Background()); err != nil {
t.Fatalf("ExecuteCleanup failed: %v", err)
}
if got := cs.GetPendingCount(); got != 0 {
t.Errorf("a missing file should not be requeued, pending = %d", got)
}
}
func TestCleanupDisabledSchedulesNothing(t *testing.T) {
dir := t.TempDir()
cs := NewCleanupService(CleanupConfig{Enabled: false, BatchSize: 10, CheckInterval: time.Second})
path := writeAgedFile(t, dir, "seg.ts", time.Hour)
if err := cs.ScheduleCleanup(path); err != nil {
t.Fatalf("ScheduleCleanup failed: %v", err)
}
if got := cs.GetPendingCount(); got != 0 {
t.Errorf("cleanup is disabled, nothing should be queued, got %d", got)
}
if err := cs.ExecuteCleanup(context.Background()); err != nil {
t.Fatalf("ExecuteCleanup failed: %v", err)
}
if _, err := os.Stat(path); err != nil {
t.Errorf("file must survive when cleanup is disabled: %v", err)
}
}
// TestCleanupRequeuesOnCancellation covers shutdown arriving mid-batch: the
// files already pulled off the pending list must go back, not vanish.
func TestCleanupRequeuesOnCancellation(t *testing.T) {
dir := t.TempDir()
cs := NewCleanupService(CleanupConfig{Enabled: true, BatchSize: 10, CheckInterval: time.Second})
for i := 0; i < 5; i++ {
path := writeAgedFile(t, dir, fmt.Sprintf("seg_%d.ts", i), 0)
if err := cs.ScheduleCleanup(path); err != nil {
t.Fatalf("ScheduleCleanup failed: %v", err)
}
}
ctx, cancel := context.WithCancel(context.Background())
cancel()
if err := cs.ExecuteCleanup(ctx); err == nil {
t.Fatal("expected the cancelled context to be reported")
}
if got := cs.GetPendingCount(); got != 5 {
t.Errorf("all files should be requeued after cancellation, pending = %d, want 5", got)
}
}

View File

@ -11,7 +11,7 @@ import (
) )
func TransferFile(nt *nas.NASService, ctx context.Context, item *TransferItem) error { func TransferFile(nt *nas.NASService, ctx context.Context, item *TransferItem) error {
destPath := filepath.Join(nt.Config.Path, item.DestinationPath) destPath := nt.Join(item.DestinationPath)
destDir := filepath.Dir(destPath) destDir := filepath.Dir(destPath)
if err := nt.EnsureDirectoryExists(destDir); err != nil { if err := nt.EnsureDirectoryExists(destDir); err != nil {

View File

@ -169,13 +169,20 @@ func (tq *TransferQueue) processItem(ctx context.Context, item TransferItem) {
return return
} }
maxRetries := 3 maxRetries := tq.config.RetryLimit
if maxRetries < 1 {
maxRetries = 1
}
for attempt := 1; attempt <= maxRetries; attempt++ { for attempt := 1; attempt <= maxRetries; attempt++ {
if attempt > 0 { // Back off before a retry, never before the first attempt. The guard
// was `attempt > 0`, which is always true, so every single transfer —
// including healthy ones — paid a one-second delay before it began,
// throttling the whole pipeline to the worker count per second.
if attempt > 1 {
item.Status = StatusRetrying item.Status = StatusRetrying
backoff := time.Duration(attempt*attempt) * time.Second backoff := time.Duration(attempt*attempt) * time.Second
log.Printf("Backing off for %d seconds before retrying (attempt %d/%d)", backoff, attempt, maxRetries) log.Printf("Backing off for %s before retrying (attempt %d/%d)", backoff, attempt, maxRetries)
select { select {
case <-time.After(backoff): case <-time.After(backoff):

View File

@ -34,14 +34,23 @@ func NewTrasferService(outputDir string, eventName string) (*TransferService, er
Timeout: cfg.NAS.Timeout, Timeout: cfg.NAS.Timeout,
RetryLimit: cfg.NAS.RetryLimit, RetryLimit: cfg.NAS.RetryLimit,
VerifySize: true, VerifySize: true,
AutoMount: cfg.NAS.AutoMount,
MountPoint: cfg.NAS.MountPoint,
}
nas, err := nas2.New(nasConfig)
if err != nil {
logging.MustGetFactory().Transfer.Error("invalid NAS destination", logging.KeyError, err.Error())
return nil, fmt.Errorf("invalid NAS destination: %w", err)
} }
nas := nas2.NewNASService(nasConfig)
if err := nas.TestConnection(); err != nil { connectCtx, cancelConnect := context.WithTimeout(context.Background(), cfg.NAS.Timeout)
defer cancelConnect()
if err := nas.Connect(connectCtx); err != nil {
logging.MustGetFactory().Transfer.Error("failed to connect to NAS", logging.KeyError, err.Error()) logging.MustGetFactory().Transfer.Error("failed to connect to NAS", logging.KeyError, err.Error())
return nil, fmt.Errorf("failed to connect to NAS: %w", err) return nil, fmt.Errorf("failed to connect to NAS: %w", err)
} }
logging.MustGetFactory().Transfer.Info("connected to NAS", "path", outputDir) logging.MustGetFactory().Transfer.Info("connected to NAS",
"path", nas.LocalPath(), "configured", outputDir, "how", strings.Join(nas.ConnectionInfo(), "; "))
health.MustGetRegistry().RecordNASConnectivity(true) health.MustGetRegistry().RecordNASConnectivity(true)
cleanupConfig := CleanupConfig{ cleanupConfig := CleanupConfig{
@ -57,6 +66,7 @@ func NewTrasferService(outputDir string, eventName string) (*TransferService, er
PersistencePath: cfg.Paths.PersistenceFile, PersistencePath: cfg.Paths.PersistenceFile,
MaxQueueSize: cfg.Transfer.QueueSize, MaxQueueSize: cfg.Transfer.QueueSize,
BatchSize: cfg.Transfer.BatchSize, BatchSize: cfg.Transfer.BatchSize,
RetryLimit: cfg.Transfer.RetryLimit,
} }
queue := NewTransferQueue(queueConfig, nas, cleanup) queue := NewTransferQueue(queueConfig, nas, cleanup)
@ -66,7 +76,7 @@ func NewTrasferService(outputDir string, eventName string) (*TransferService, er
return nil, fmt.Errorf("failed to create local output directory: %w", err) return nil, fmt.Errorf("failed to create local output directory: %w", err)
} }
watcher, err := NewFileWatcher(localOutputPath, queue, cfg.Transfer.FileSettlingDelay) watcher, err := NewFileWatcher(localOutputPath, eventName, queue, cfg.Transfer.FileSettlingDelay)
if err != nil { if err != nil {
return nil, fmt.Errorf("failed to create file watcher: %w", err) return nil, fmt.Errorf("failed to create file watcher: %w", err)
} }
@ -155,7 +165,7 @@ func (ts *TransferService) Shutdown(ctx context.Context) error {
} }
// Disconnect from NAS // Disconnect from NAS
if err := ts.nas.Disconnect(); err != nil { if err := ts.nas.Disconnect(ctx); err != nil {
log.Printf("Warning: failed to disconnect from NAS: %v", err) log.Printf("Warning: failed to disconnect from NAS: %v", err)
logging.MustGetFactory().Transfer.Warn("failed to disconnect from NAS", logging.KeyError, err.Error()) logging.MustGetFactory().Transfer.Warn("failed to disconnect from NAS", logging.KeyError, err.Error())
} }

View File

@ -49,6 +49,9 @@ type QueueConfig struct {
PersistencePath string PersistencePath string
MaxQueueSize int MaxQueueSize int
BatchSize int BatchSize int
// RetryLimit is the total number of attempts made per file before it is
// marked failed.
RetryLimit int
} }
type CleanupConfig struct { type CleanupConfig struct {

View File

@ -18,6 +18,11 @@ import (
type FileWatcher struct { type FileWatcher struct {
outputDir string outputDir string
// eventName namespaces the NAS destination. It is passed in rather than
// derived from outputDir so that an unnamed event resolves to the NAS root
// — matching where the downloader put the files — instead of picking up
// the local output directory's own name.
eventName string
queue *TransferQueue queue *TransferQueue
watcher *fsnotify.Watcher watcher *fsnotify.Watcher
settingDelay time.Duration settingDelay time.Duration
@ -25,13 +30,14 @@ type FileWatcher struct {
mu sync.Mutex mu sync.Mutex
} }
func NewFileWatcher(outputDir string, queue *TransferQueue, settlingDelay time.Duration) (*FileWatcher, error) { func NewFileWatcher(outputDir, eventName string, queue *TransferQueue, settlingDelay time.Duration) (*FileWatcher, error) {
watcher, err := fsnotify.NewWatcher() watcher, err := fsnotify.NewWatcher()
if err != nil { if err != nil {
return nil, err return nil, err
} }
return &FileWatcher{ return &FileWatcher{
outputDir: outputDir, outputDir: outputDir,
eventName: eventName,
queue: queue, queue: queue,
watcher: watcher, watcher: watcher,
settingDelay: settlingDelay, settingDelay: settlingDelay,
@ -92,6 +98,18 @@ func (fw *FileWatcher) addWatchRecursive(root string) error {
} }
func (fw *FileWatcher) handleFileEvent(event fsnotify.Event) { func (fw *FileWatcher) handleFileEvent(event fsnotify.Event) {
// Directories must be handled before the .ts filter below. fsnotify is not
// recursive, so a resolution directory created after the watcher starts —
// which is the normal case, since the transfer service starts before the
// downloader creates 1080p/ and friends — has to be picked up here or
// nothing inside it is ever seen.
if event.Op&fsnotify.Create == fsnotify.Create {
if info, err := os.Stat(event.Name); err == nil && info.IsDir() {
fw.watchNewDirectory(event.Name)
return
}
}
if !strings.HasSuffix(strings.ToLower(event.Name), ".ts") { if !strings.HasSuffix(strings.ToLower(event.Name), ".ts") {
return return
} }
@ -104,15 +122,33 @@ func (fw *FileWatcher) handleFileEvent(event fsnotify.Event) {
case event.Op&fsnotify.Remove == fsnotify.Remove: case event.Op&fsnotify.Remove == fsnotify.Remove:
fw.cancelPendingTransfer(event.Name) fw.cancelPendingTransfer(event.Name)
} }
}
if event.Op&fsnotify.Create == fsnotify.Create { // watchNewDirectory starts watching a directory that appeared after startup,
if info, err := os.Stat(event.Name); err == nil && info.IsDir() { // and sweeps whatever is already inside it.
if err := fw.watcher.Add(event.Name); err != nil { //
log.Printf("Failed to watch directory %s: %v", event.Name, err) // The sweep is not redundant: the downloader creates a resolution directory and
} else { // writes segments into it immediately, so files can land between the mkdir and
log.Printf("Watching directory %s", event.Name) // the watch being registered. Those writes generate no event we would see, and
// without the sweep the first segments of every recording would be lost.
func (fw *FileWatcher) watchNewDirectory(dir string) {
if err := fw.addWatchRecursive(dir); err != nil {
log.Printf("Failed to watch new directory %s: %v", dir, err)
logging.MustGetFactory().Transfer.Warn("failed to watch new directory", "dir", dir, logging.KeyError, err.Error())
return
} }
err := filepath.Walk(dir, func(path string, info os.FileInfo, err error) error {
if err != nil {
return nil // keep sweeping the rest
} }
if !info.IsDir() && strings.HasSuffix(strings.ToLower(info.Name()), ".ts") {
fw.scheduleTransfer(path)
}
return nil
})
if err != nil {
log.Printf("Failed to sweep new directory %s: %v", dir, err)
} }
} }
@ -164,10 +200,13 @@ func (fw *FileWatcher) processFile(filePath string) {
return return
} }
// The destination is namespaced by event, mirroring the local layout and
// matching what QueueExistingFiles produces for the same file — otherwise
// live transfers and catch-up transfers would write to different places.
item := TransferItem{ item := TransferItem{
ID: generateID(), ID: generateID(),
SourcePath: filePath, SourcePath: filePath,
DestinationPath: relPath, DestinationPath: filepath.Join(fw.eventName, relPath),
Resolution: resolution, Resolution: resolution,
Timestamp: time.Now(), Timestamp: time.Now(),
Status: StatusPending, Status: StatusPending,

View File

@ -0,0 +1,223 @@
package transfer
import (
"context"
"os"
"path/filepath"
"sort"
"testing"
"time"
)
// settlingDelay is short so tests exercise the real timer path without
// waiting the production five seconds.
const testSettlingDelay = 40 * time.Millisecond
// newTestQueue builds a queue that accepts items but never transfers them:
// ProcessQueue is not started, so Add is a pure recording of intent.
func newTestQueue(t *testing.T) *TransferQueue {
t.Helper()
return NewTransferQueue(QueueConfig{
WorkerCount: 1,
PersistencePath: filepath.Join(t.TempDir(), "queue.json"),
MaxQueueSize: 100,
BatchSize: 10,
}, nil, nil)
}
// queuedDestinations returns the destination paths currently queued, sorted.
func queuedDestinations(q *TransferQueue) []string {
q.mu.RLock()
defer q.mu.RUnlock()
out := make([]string, 0, q.items.Len())
for _, item := range *q.items {
out = append(out, filepath.ToSlash(item.DestinationPath))
}
sort.Strings(out)
return out
}
// waitForQueue polls until the queue holds want items or the deadline passes.
// Polling beats a fixed sleep: it keeps the test fast when the watcher is
// prompt and tolerant when a loaded CI machine is not.
func waitForQueue(t *testing.T, q *TransferQueue, want int) []string {
t.Helper()
deadline := time.Now().Add(3 * time.Second)
var got []string
for time.Now().Before(deadline) {
got = queuedDestinations(q)
if len(got) >= want {
return got
}
time.Sleep(10 * time.Millisecond)
}
return got
}
func startWatcher(t *testing.T, dir, eventName string, q *TransferQueue) {
t.Helper()
fw, err := NewFileWatcher(dir, eventName, q, testSettlingDelay)
if err != nil {
t.Fatalf("NewFileWatcher failed: %v", err)
}
ctx, cancel := context.WithCancel(context.Background())
t.Cleanup(cancel)
started := make(chan struct{})
go func() {
close(started)
_ = fw.Start(ctx)
}()
<-started
// Give Start a moment to register its initial watches before the test
// begins creating files under them.
time.Sleep(100 * time.Millisecond)
}
// TestWatcherPicksUpDirectoriesCreatedAfterStart covers the normal recording
// sequence: the transfer service starts watching the event directory, and only
// then does the downloader create the per-resolution directories inside it.
// fsnotify is not recursive, so those directories have to be picked up from
// the create event or nothing inside them is ever transferred.
func TestWatcherPicksUpDirectoriesCreatedAfterStart(t *testing.T) {
eventDir := t.TempDir()
queue := newTestQueue(t)
startWatcher(t, eventDir, "2026-regionals", queue)
resolutionDir := filepath.Join(eventDir, "1080p")
if err := os.MkdirAll(resolutionDir, 0755); err != nil {
t.Fatalf("failed to create resolution dir: %v", err)
}
if err := os.WriteFile(filepath.Join(resolutionDir, "seg_1001.ts"), []byte("payload"), 0644); err != nil {
t.Fatalf("failed to write segment: %v", err)
}
got := waitForQueue(t, queue, 1)
if len(got) != 1 {
t.Fatalf("segment written to a new resolution directory was never queued, queue = %v", got)
}
if want := "2026-regionals/1080p/seg_1001.ts"; got[0] != want {
t.Errorf("destination = %q, want %q", got[0], want)
}
}
// TestWatcherSweepsFilesRacingDirectoryCreation covers the tighter race where
// segments are written immediately after the mkdir, before the new watch can
// be registered. Those writes produce no event the watcher would receive, so
// the directory has to be swept when it is first seen.
func TestWatcherSweepsFilesRacingDirectoryCreation(t *testing.T) {
eventDir := t.TempDir()
queue := newTestQueue(t)
startWatcher(t, eventDir, "event", queue)
// Stage the tree elsewhere and move it into place, so the files are
// guaranteed to already exist the instant the directory appears.
staging := filepath.Join(t.TempDir(), "1080p")
if err := os.MkdirAll(staging, 0755); err != nil {
t.Fatalf("failed to stage: %v", err)
}
for _, name := range []string{"seg_1001.ts", "seg_1002.ts"} {
if err := os.WriteFile(filepath.Join(staging, name), []byte("payload"), 0644); err != nil {
t.Fatalf("failed to stage segment: %v", err)
}
}
if err := os.Rename(staging, filepath.Join(eventDir, "1080p")); err != nil {
t.Fatalf("failed to move staged directory into place: %v", err)
}
got := waitForQueue(t, queue, 2)
want := []string{"event/1080p/seg_1001.ts", "event/1080p/seg_1002.ts"}
if len(got) != 2 || got[0] != want[0] || got[1] != want[1] {
t.Errorf("queued %v, want %v", got, want)
}
}
// TestWatcherDestinationIncludesEventName pins the layout agreement between
// live transfers and the catch-up path in QueueExistingFiles: both must write
// to <nas>/<event>/<resolution>/<file>.
func TestWatcherDestinationIncludesEventName(t *testing.T) {
eventDir := t.TempDir()
resolutionDir := filepath.Join(eventDir, "720p")
if err := os.MkdirAll(resolutionDir, 0755); err != nil {
t.Fatalf("setup failed: %v", err)
}
queue := newTestQueue(t)
startWatcher(t, eventDir, "my-event", queue)
if err := os.WriteFile(filepath.Join(resolutionDir, "seg_1001.ts"), []byte("payload"), 0644); err != nil {
t.Fatalf("failed to write segment: %v", err)
}
got := waitForQueue(t, queue, 1)
if len(got) != 1 {
t.Fatalf("expected one queued item, got %v", got)
}
if want := "my-event/720p/seg_1001.ts"; got[0] != want {
t.Errorf("destination = %q, want %q", got[0], want)
}
}
// TestWatcherWithoutEventNameUsesRoot mirrors a download run with no -event:
// segments land under the local output root, so the NAS destination must not
// invent a directory level for them.
func TestWatcherWithoutEventNameUsesRoot(t *testing.T) {
eventDir := t.TempDir()
resolutionDir := filepath.Join(eventDir, "1080p")
if err := os.MkdirAll(resolutionDir, 0755); err != nil {
t.Fatalf("setup failed: %v", err)
}
queue := newTestQueue(t)
startWatcher(t, eventDir, "", queue)
if err := os.WriteFile(filepath.Join(resolutionDir, "seg_1001.ts"), []byte("payload"), 0644); err != nil {
t.Fatalf("failed to write segment: %v", err)
}
got := waitForQueue(t, queue, 1)
if len(got) != 1 {
t.Fatalf("expected one queued item, got %v", got)
}
if want := "1080p/seg_1001.ts"; got[0] != want {
t.Errorf("destination = %q, want %q", got[0], want)
}
}
func TestWatcherIgnoresNonSegmentFiles(t *testing.T) {
eventDir := t.TempDir()
queue := newTestQueue(t)
startWatcher(t, eventDir, "event", queue)
for _, name := range []string{"notes.txt", "manifest.json", "segment.ts.part"} {
if err := os.WriteFile(filepath.Join(eventDir, name), []byte("x"), 0644); err != nil {
t.Fatalf("failed to write %s: %v", name, err)
}
}
time.Sleep(testSettlingDelay + 300*time.Millisecond)
if got := queuedDestinations(queue); len(got) != 0 {
t.Errorf("non-segment files should not be queued, got %v", got)
}
}
func TestExtractResolution(t *testing.T) {
fw := &FileWatcher{}
cases := map[string]string{
filepath.Join("data", "event", "1080p", "seg.ts"): "1080p",
filepath.Join("data", "event", "720p", "seg.ts"): "720p",
filepath.Join("data", "event", "seg.ts"): "",
}
for path, want := range cases {
if got := fw.extractResolution(path); got != want {
t.Errorf("extractResolution(%q) = %q, want %q", path, got, want)
}
}
}

View File

@ -24,12 +24,28 @@ func PathExists(path string) bool {
return !os.IsNotExist(err) return !os.IsNotExist(err)
} }
// IsValidPath reports whether path is usable on Windows, the stricter of the
// two platforms this project targets, which reserves <>:"|?* in names.
//
// A leading drive letter is checked separately: its colon is legal there and
// nowhere else in the path. Without that carve-out the function rejects every
// absolute Windows path, including the drive-qualified ones it exists to
// validate.
func IsValidPath(path string) bool { func IsValidPath(path string) bool {
if path == "" { if path == "" {
return false return false
} }
return !strings.ContainsAny(path, "<>:\"|?*") rest := path
if len(rest) >= 2 && rest[1] == ':' && isDriveLetter(rest[0]) {
rest = rest[2:]
}
return !strings.ContainsAny(rest, `<>:"|?*`)
}
func isDriveLetter(c byte) bool {
return (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z')
} }
func NormalizePath(path string) string { func NormalizePath(path string) string {
@ -37,6 +53,13 @@ func NormalizePath(path string) string {
} }
func GetRelativePath(basePath, targetPath string) (string, error) { func GetRelativePath(basePath, targetPath string) (string, error) {
// filepath.Rel("", "") succeeds and returns ".", which is a meaningless
// answer to a meaningless question; callers passing an empty path have a
// bug worth surfacing rather than papering over.
if basePath == "" || targetPath == "" {
return "", fmt.Errorf("both base and target paths are required (base %q, target %q)", basePath, targetPath)
}
rel, err := filepath.Rel(basePath, targetPath) rel, err := filepath.Rel(basePath, targetPath)
if err != nil { if err != nil {
return "", fmt.Errorf("failed to get relative path: %w", err) return "", fmt.Errorf("failed to get relative path: %w", err)