This commit is contained in:
parent
7f4078961e
commit
e2566fdefa
14
CLAUDE.md
14
CLAUDE.md
@ -33,9 +33,15 @@ The project follows a modular architecture with clear separation of concerns:
|
||||
- **service.go**: Processing service orchestration with FFmpeg integration
|
||||
- **segment.go**: Individual segment processing logic
|
||||
- **types.go**: Processing system data structures
|
||||
- **nas/**: NAS connection and file operations
|
||||
- **nas/**: NAS connection and file operations (platform-agnostic: Windows and Linux)
|
||||
- **config.go**: NAS configuration structure
|
||||
- **nas.go**: NAS service with connection management and file operations
|
||||
- **nas.go**: `NASService` — resolves the destination to a host-native path, then copies/verifies files against it
|
||||
- **location.go**: Parses UNC / `//server/share` / `smb://` / local paths into a host-neutral `Location`
|
||||
- **connector.go**: `Connector` interface + runtime selection by GOOS (no build tags, so both platforms' code is compiled and tested everywhere)
|
||||
- **connector_netuse.go**: Windows strategy — `net use`, UNC path used directly
|
||||
- **connector_mount.go**: Linux strategy — resolve the share against `/proc/mounts`, optional root-only `mount.cifs`
|
||||
- **connector_local.go**: Already-mounted/plain directory on either platform
|
||||
- **mounts.go**: `/proc/mounts` parsing and share-to-mount-point matching
|
||||
- **config/**: Centralized configuration management with validation
|
||||
- **config.go**: Configuration loading, validation, and path resolution
|
||||
- **utils/**: Utility functions for cross-platform compatibility
|
||||
@ -100,8 +106,10 @@ Configuration is managed through a centralized system in `pkg/config/config.go`
|
||||
|
||||
### NAS Transfer Settings
|
||||
- `NAS.EnableTransfer`: Enable/disable automatic NAS transfer (true) - ENV: `ENABLE_NAS_TRANSFER`
|
||||
- `NAS.OutputPath`: UNC path to NAS storage (`\\HomeLabNAS\dci\streams`) - ENV: `NAS_OUTPUT_PATH`
|
||||
- `NAS.OutputPath`: NAS destination (`\\HomeLabNAS\dci\streams`) - ENV: `NAS_OUTPUT_PATH`. Accepts UNC, `//server/share`, `smb://`, or a local/mounted directory; Windows connects it with `net use`, Linux resolves it against `/proc/mounts`
|
||||
- `NAS.Username`/`NAS.Password`: NAS credentials for authentication - ENV: `NAS_USERNAME`/`NAS_PASSWORD`
|
||||
- `NAS.AutoMount`: Linux-only, root-required auto-mount of the SMB share (false) - ENV: `NAS_AUTO_MOUNT`
|
||||
- `NAS.MountPoint`: Override for the auto-mount location - ENV: `NAS_MOUNT_POINT`
|
||||
- `Transfer.WorkerCount`: Concurrent transfer workers (2)
|
||||
- `Transfer.RetryLimit`: Max retry attempts per file (3)
|
||||
- `Transfer.Timeout`: Timeout per file transfer (30 seconds)
|
||||
|
||||
@ -11,10 +11,23 @@ The application supports configuration through environment variables for flexibl
|
||||
- `REFRESH_DELAY_SECONDS`: How often to check for playlist updates in seconds (default: 3)
|
||||
|
||||
### NAS Transfer Settings
|
||||
- `NAS_OUTPUT_PATH`: UNC path to NAS storage (default: "\\\\HomeLabNAS\\dci\\streams")
|
||||
- `NAS_OUTPUT_PATH`: NAS destination (default: "\\\\HomeLabNAS\\dci\\streams"). Accepts a UNC path, `//server/share/dir`, `smb://server/share/dir`, or a local/mounted directory
|
||||
- `NAS_USERNAME`: NAS authentication username
|
||||
- `NAS_PASSWORD`: NAS authentication password
|
||||
- `ENABLE_NAS_TRANSFER`: Enable/disable automatic NAS transfer (default: true)
|
||||
- `NAS_AUTO_MOUNT`: Linux only — mount the SMB share if the host hasn't already (default: false, requires root)
|
||||
- `NAS_MOUNT_POINT`: Where `NAS_AUTO_MOUNT` attaches the share (default: `/mnt/streamrecorder/<server>/<share>`)
|
||||
|
||||
The same `NAS_OUTPUT_PATH` works on Windows and Linux. Windows authenticates the
|
||||
UNC path with `net use`; Linux resolves the share against `/proc/mounts` and
|
||||
rewrites the path onto the mount point, so `\\\\HomeLabNAS\\dci\\streams` with
|
||||
`//homelabnas/dci` mounted at `/mnt/dci` becomes `/mnt/dci/streams`. Mounting the
|
||||
share on the host (fstab/autofs/systemd, or `docker run -v`) is the recommended
|
||||
Linux setup; `NAS_AUTO_MOUNT` is the root-privileged fallback. Verify either with:
|
||||
|
||||
```bash
|
||||
./stream-recorder -check
|
||||
```
|
||||
|
||||
### Path Configuration
|
||||
- `LOCAL_OUTPUT_DIR`: Base directory for local downloads (default: "data")
|
||||
|
||||
27
README.md
27
README.md
@ -43,7 +43,7 @@ Segments land on disk under `Paths.LocalOutput/<event>/<resolution>/`.
|
||||
- `service.go`: `TransferService` wires together the watcher, queue, and cleanup service, and owns the connection to NAS for the run.
|
||||
- `watcher.go`: `FileWatcher` (via `fsnotify`) watches the local event directory for new `.ts` files and queues them after a short settling delay (`Transfer.FileSettlingDelay`, default 5s), so a file isn't queued while still being written.
|
||||
- `queue.go`: `TransferQueue` is a priority-queued worker pool (`Transfer.WorkerCount`, default 2) that transfers queued files to NAS with retry + exponential backoff, and persists its state to `transfer_queue.json` so it survives a restart.
|
||||
- `nas.go` / `pkg/nas`: `NASService` establishes the NAS connection (on Windows, via `net use` for UNC paths) and does the actual file copy + size verification.
|
||||
- `nas.go` / `pkg/nas`: `NASService` attaches the NAS destination for whatever host it's running on and does the actual file copy + size verification. See [NAS connectivity](#nas-connectivity-windows-and-linux).
|
||||
- `cleanup.go`: `CleanupService` deletes local copies after a successful, verified transfer (`Cleanup.AfterTransfer`, default true).
|
||||
|
||||
During a normal download run, this whole stage starts automatically in the same process as the downloader (see the note above) — you don't invoke transfer separately for a live recording. The standalone `-transfer` mode exists only to catch up files that a prior download session ended without fully transferring.
|
||||
@ -83,8 +83,10 @@ All settings have sane defaults; override via environment variables as needed.
|
||||
| `WORKER_COUNT` | `4` | Concurrent segment downloads per variant |
|
||||
| `REFRESH_DELAY_SECONDS` | `3` | How often each variant's chunklist is polled |
|
||||
| `ENABLE_NAS_TRANSFER` | `true` | Whether to run the transfer service alongside download |
|
||||
| `NAS_OUTPUT_PATH` | `\\HomeLabNAS\dci\streams` | UNC/NAS root path segments are copied to |
|
||||
| `NAS_USERNAME` / `NAS_PASSWORD` | *(see note below)* | NAS auth credentials |
|
||||
| `NAS_OUTPUT_PATH` | `\\HomeLabNAS\dci\streams` | NAS root segments are copied to. UNC (`\\server\share\dir`), `//server/share/dir`, `smb://server/share/dir`, or a local/mounted directory — see [NAS connectivity](#nas-connectivity-windows-and-linux) |
|
||||
| `NAS_USERNAME` / `NAS_PASSWORD` | *(see note below)* | NAS auth credentials. Used by `net use` on Windows and by `NAS_AUTO_MOUNT` on Linux; ignored for an already-mounted path |
|
||||
| `NAS_AUTO_MOUNT` | `false` | Linux only: mount the SMB share if the host hasn't. Requires root |
|
||||
| `NAS_MOUNT_POINT` | *(derived)* | Where `NAS_AUTO_MOUNT` attaches the share (default `/mnt/streamrecorder/<server>/<share>`) |
|
||||
| `LOCAL_OUTPUT_DIR` | `data` | Local base directory for downloaded segments |
|
||||
| `PROCESS_OUTPUT_DIR` | `out` | Output directory for the final concatenated `.mp4` |
|
||||
| `FFMPEG_PATH` | `ffmpeg` | Path to the FFmpeg binary used by the processing stage |
|
||||
@ -95,6 +97,25 @@ All settings have sane defaults; override via environment variables as needed.
|
||||
|
||||
> **Security note:** the default NAS credentials currently ship as hardcoded defaults in `pkg/config/config.go` for this home-lab setup. If you fork or redeploy this, set `NAS_USERNAME`/`NAS_PASSWORD` explicitly and rotate the shipped default — see `TODO.md` for the full flagged list of housekeeping items like this one.
|
||||
|
||||
## NAS connectivity (Windows and Linux)
|
||||
|
||||
The same `NAS_OUTPUT_PATH` works on both platforms; only the mechanism for reaching it differs, and the recorder picks it at runtime.
|
||||
|
||||
| Host | Destination | What happens |
|
||||
|---|---|---|
|
||||
| Windows | UNC / `//server/share` / `smb://` | `net use \\server\share` with `NAS_USERNAME`/`NAS_PASSWORD`, then the UNC path is used directly |
|
||||
| Linux | UNC / `//server/share` / `smb://` | The share is looked up in `/proc/mounts` and the UNC path is rewritten onto the mount point (`\\nas\dci\streams` + `//nas/dci` mounted at `/mnt/dci` → `/mnt/dci/streams`). Set `NAS_AUTO_MOUNT=true` to mount it instead |
|
||||
| Either | A local or already-mounted directory | Used as-is, no network step |
|
||||
|
||||
Notes:
|
||||
|
||||
- **Server and share names match case-insensitively**, and both `cifs`/`smb3` and `nfs`/`nfs4` mounts are recognised. If the share is mounted at a deeper level than configured (`//nas/dci/streams` at `/mnt/streams`), the deepest matching mount wins.
|
||||
- **Nothing the host already set up is torn down.** A share found already mounted (or already connected via `net use`) is left alone at shutdown; only a mount or connection this process created is undone.
|
||||
- **`NAS_AUTO_MOUNT` requires root**, so the usual Linux setup is to mount the share yourself via fstab/autofs/systemd and either point `NAS_OUTPUT_PATH` at the mount point or leave it as the UNC path and let it resolve. Credentials are passed to `mount.cifs` through a `0600` credentials file, never on the command line where the process table would expose the password.
|
||||
- **A Windows-shaped path on Linux is rejected, not silently created.** Backslashes and drive letters are legal filename characters on Linux, so `C:\data` or an unresolved `\\nas\dci` would otherwise be created as a local directory named exactly that — transfers would report success, cleanup would delete the local originals, and nothing would reach the NAS.
|
||||
|
||||
Run `-check` to see which strategy applies and where the destination resolved to.
|
||||
|
||||
## Building
|
||||
|
||||
```bash
|
||||
|
||||
@ -7,6 +7,7 @@ package check
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"log"
|
||||
"m3u8-downloader/pkg/config"
|
||||
"m3u8-downloader/pkg/constants"
|
||||
"m3u8-downloader/pkg/httpClient"
|
||||
@ -155,46 +156,51 @@ func checkMasterPlaylist(masterURL string) ([]string, error) {
|
||||
return details, nil
|
||||
}
|
||||
|
||||
// checkNAS runs the same connect/mkdir/write sequence the transfer service
|
||||
// performs at startup, but without the log.Fatalf that path takes on failure.
|
||||
// checkNAS runs the same attach/mkdir/write sequence the transfer service
|
||||
// performs at startup, so the check exercises the real code path for this host
|
||||
// — net use on Windows, mount resolution on Linux — rather than a simulation of
|
||||
// it. It disconnects afterwards so the check leaves no mount behind.
|
||||
func checkNAS(cfg *config.Config) ([]string, error) {
|
||||
svc := &nas.NASService{Config: nas.NASConfig{
|
||||
svc, err := nas.New(nas.NASConfig{
|
||||
Path: cfg.NAS.OutputPath,
|
||||
Username: cfg.NAS.Username,
|
||||
Password: cfg.NAS.Password,
|
||||
Timeout: cfg.NAS.Timeout,
|
||||
RetryLimit: cfg.NAS.RetryLimit,
|
||||
VerifySize: true,
|
||||
}}
|
||||
|
||||
details := []string{"path: " + cfg.NAS.OutputPath}
|
||||
if share := svc.ExtractNetworkPath(cfg.NAS.OutputPath); share != "" {
|
||||
details = append(details, "share: "+share, "user: "+usernameLabel(cfg.NAS.Username))
|
||||
} else {
|
||||
details = append(details, "local or already-mounted path (no network authentication)")
|
||||
AutoMount: cfg.NAS.AutoMount,
|
||||
MountPoint: cfg.NAS.MountPoint,
|
||||
})
|
||||
if err != nil {
|
||||
return []string{"path: " + cfg.NAS.OutputPath}, err
|
||||
}
|
||||
|
||||
start := time.Now()
|
||||
result := <-probeAsync(cfg.NAS.Timeout, func() ([]string, error) {
|
||||
if err := svc.EstablishConnection(); err != nil {
|
||||
return nil, fmt.Errorf("failed to connect to share: %w", err)
|
||||
ctx, cancel := context.WithTimeout(context.Background(), cfg.NAS.Timeout)
|
||||
defer cancel()
|
||||
|
||||
if err := svc.Connect(ctx); err != nil {
|
||||
return svc.ConnectionInfo(), err
|
||||
}
|
||||
if err := svc.EnsureDirectoryExists(cfg.NAS.OutputPath); err != nil {
|
||||
return nil, fmt.Errorf("share reachable but target directory unusable: %w", err)
|
||||
defer func() {
|
||||
if err := svc.Disconnect(context.Background()); err != nil {
|
||||
log.Printf("Warning: failed to release NAS connection after check: %v", err)
|
||||
}
|
||||
if err := svc.TestConnection(); err != nil {
|
||||
return nil, fmt.Errorf("target directory is not writable: %w", err)
|
||||
}
|
||||
return []string{"connect, mkdir and write probe succeeded"}, nil
|
||||
}()
|
||||
|
||||
return append(svc.ConnectionInfo(), "attach, mkdir and write probe succeeded"), nil
|
||||
})
|
||||
|
||||
details := result.details
|
||||
if len(details) == 0 {
|
||||
details = svc.ConnectionInfo()
|
||||
}
|
||||
if result.err != nil {
|
||||
return details, result.err
|
||||
}
|
||||
|
||||
details = append(details, result.details...)
|
||||
details = append(details, "completed in "+formatDuration(time.Since(start)))
|
||||
return details, nil
|
||||
return append(details, "completed in "+formatDuration(time.Since(start))), nil
|
||||
}
|
||||
|
||||
func formatDuration(d time.Duration) string {
|
||||
@ -276,10 +282,3 @@ func enabledLabel(enabled bool) string {
|
||||
}
|
||||
return "disabled"
|
||||
}
|
||||
|
||||
func usernameLabel(username string) string {
|
||||
if username == "" {
|
||||
return "(none, using ambient credentials)"
|
||||
}
|
||||
return username
|
||||
}
|
||||
|
||||
@ -154,7 +154,7 @@ func TestCheckNASLocalPath(t *testing.T) {
|
||||
}
|
||||
|
||||
joined := joinDetails(details)
|
||||
for _, want := range []string{"path: " + target, "write probe succeeded"} {
|
||||
for _, want := range []string{"configured: " + target, "strategy: local", "write probe succeeded"} {
|
||||
if !strings.Contains(joined, want) {
|
||||
t.Errorf("details missing %q:\n%s", want, joined)
|
||||
}
|
||||
@ -176,7 +176,7 @@ func TestCheckNASUnusableTarget(t *testing.T) {
|
||||
if err == nil {
|
||||
t.Fatal("expected an unusable NAS target to fail the check")
|
||||
}
|
||||
if joined := joinDetails(details); !strings.Contains(joined, "path: ") {
|
||||
if joined := joinDetails(details); !strings.Contains(joined, "configured: ") {
|
||||
t.Errorf("expected path detail on failure:\n%s", joined)
|
||||
}
|
||||
}
|
||||
|
||||
@ -71,7 +71,9 @@ func Download(masterURL string, eventName string, debug bool) {
|
||||
}
|
||||
}
|
||||
|
||||
manifestWriter := media.NewManifestWriter(eventName)
|
||||
// One writer for the whole run: every variant records into it, and it is
|
||||
// the one written to disk at the end.
|
||||
manifest := media.NewManifestWriter(eventName)
|
||||
|
||||
eventPath := cfg.GetEventPath(eventName)
|
||||
if err := utils.EnsureDir(eventPath); err != nil {
|
||||
@ -79,7 +81,7 @@ func Download(masterURL string, eventName string, debug bool) {
|
||||
log.Fatalf("Failed to create event directory: %v", err)
|
||||
}
|
||||
|
||||
variants, err := media.GetAllVariants(masterURL, eventPath, manifestWriter)
|
||||
variants, err := media.GetAllVariants(masterURL, eventPath)
|
||||
if err != nil {
|
||||
appLog.Error("failed to get variants", logging.KeyError, err.Error())
|
||||
log.Fatalf("Failed to get variants: %v", err)
|
||||
@ -88,8 +90,6 @@ func Download(masterURL string, eventName string, debug bool) {
|
||||
|
||||
sem := make(chan struct{}, constants.WorkerCount*len(variants))
|
||||
|
||||
manifest := media.NewManifestWriter(eventName)
|
||||
|
||||
for _, variant := range variants {
|
||||
// Debug mode only tracks one variant for easier debugging
|
||||
if debug {
|
||||
@ -117,7 +117,7 @@ func Download(masterURL string, eventName string, debug bool) {
|
||||
log.Println("All Services shut down.")
|
||||
appLog.Info("all services shut down")
|
||||
|
||||
manifestWriter.WriteManifest()
|
||||
manifest.WriteManifest()
|
||||
log.Println("Manifest written.")
|
||||
appLog.Info("manifest written", logging.KeyEvent, eventName)
|
||||
appLog.Info("manifest written", logging.KeyEvent, eventName, "segments", manifest.Len())
|
||||
}
|
||||
|
||||
@ -32,11 +32,19 @@ type HTTPConfig struct {
|
||||
|
||||
type NASConfig struct {
|
||||
EnableTransfer bool
|
||||
// OutputPath is the destination as configured: a UNC path, an smb:// URL,
|
||||
// or a directory on this host. How it is reached depends on the platform;
|
||||
// see pkg/nas.
|
||||
OutputPath string
|
||||
Username string
|
||||
Password string
|
||||
Timeout time.Duration
|
||||
RetryLimit int
|
||||
// AutoMount lets the recorder mount an SMB share itself when the host has
|
||||
// not already mounted it. Linux only, and requires root.
|
||||
AutoMount bool
|
||||
// MountPoint overrides where AutoMount attaches the share.
|
||||
MountPoint string
|
||||
}
|
||||
|
||||
type ProcessingConfig struct {
|
||||
@ -95,6 +103,8 @@ var defaultConfig = Config{
|
||||
Password: "s3tkY6tzA&KN6M",
|
||||
Timeout: 30 * time.Second,
|
||||
RetryLimit: 3,
|
||||
AutoMount: false,
|
||||
MountPoint: "",
|
||||
},
|
||||
Processing: ProcessingConfig{
|
||||
Enabled: true,
|
||||
@ -175,8 +185,25 @@ func (c *Config) loadFromEnvironment() error {
|
||||
c.NAS.EnableTransfer = val == "true"
|
||||
}
|
||||
|
||||
if val := os.Getenv("NAS_AUTO_MOUNT"); val != "" {
|
||||
c.NAS.AutoMount = val == "true"
|
||||
}
|
||||
|
||||
if val := os.Getenv("NAS_MOUNT_POINT"); val != "" {
|
||||
c.NAS.MountPoint = val
|
||||
}
|
||||
|
||||
if val := os.Getenv("LOCAL_OUTPUT_DIR"); val != "" {
|
||||
c.Paths.LocalOutput = val
|
||||
// The manifest describes the event directory and is documented as
|
||||
// sitting beside it, so it follows the output directory. Without this
|
||||
// it stays in the default data/ while the segments move elsewhere.
|
||||
c.Paths.ManifestDir = val
|
||||
}
|
||||
|
||||
// Applied after LOCAL_OUTPUT_DIR so an explicit setting wins.
|
||||
if val := os.Getenv("MANIFEST_DIR"); val != "" {
|
||||
c.Paths.ManifestDir = val
|
||||
}
|
||||
|
||||
if val := os.Getenv("PROCESS_OUTPUT_DIR"); val != "" {
|
||||
|
||||
@ -8,12 +8,22 @@ import (
|
||||
"m3u8-downloader/pkg/utils"
|
||||
"os"
|
||||
"sort"
|
||||
"strconv"
|
||||
"sync"
|
||||
)
|
||||
|
||||
type ManifestWriter struct {
|
||||
ManifestPath string
|
||||
Segments []ManifestItem
|
||||
Index map[string]*ManifestItem
|
||||
// Index maps a sequence number to its position in Segments. It holds
|
||||
// positions rather than pointers because append reallocates the backing
|
||||
// array, which would leave previously handed-out element pointers aimed at
|
||||
// a stale copy.
|
||||
Index map[string]int
|
||||
|
||||
// mu guards the fields above: every variant downloads concurrently, and
|
||||
// each reports its segments as they land.
|
||||
mu sync.Mutex
|
||||
}
|
||||
|
||||
type ManifestItem struct {
|
||||
@ -26,40 +36,55 @@ func NewManifestWriter(eventName string) *ManifestWriter {
|
||||
return &ManifestWriter{
|
||||
ManifestPath: cfg.GetManifestPath(eventName),
|
||||
Segments: make([]ManifestItem, 0),
|
||||
Index: make(map[string]*ManifestItem),
|
||||
Index: make(map[string]int),
|
||||
}
|
||||
}
|
||||
|
||||
// AddOrUpdateSegment records that a segment was downloaded, keeping the
|
||||
// highest-resolution variant seen for each sequence number.
|
||||
func (m *ManifestWriter) AddOrUpdateSegment(seqNo string, resolution string) {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
|
||||
if m.Index == nil {
|
||||
m.Index = make(map[string]*ManifestItem)
|
||||
m.Index = make(map[string]int)
|
||||
}
|
||||
|
||||
if m.Segments == nil {
|
||||
m.Segments = make([]ManifestItem, 0)
|
||||
}
|
||||
|
||||
if existing, ok := m.Index[seqNo]; ok {
|
||||
if resolution > existing.Resolution {
|
||||
existing.Resolution = resolution
|
||||
if pos, ok := m.Index[seqNo]; ok {
|
||||
if ResolutionHeight(resolution) > ResolutionHeight(m.Segments[pos].Resolution) {
|
||||
m.Segments[pos].Resolution = resolution
|
||||
}
|
||||
return
|
||||
} else {
|
||||
item := ManifestItem{
|
||||
SeqNo: seqNo,
|
||||
Resolution: resolution,
|
||||
}
|
||||
m.Segments = append(m.Segments, item)
|
||||
m.Index[seqNo] = &item
|
||||
}
|
||||
|
||||
m.Segments = append(m.Segments, ManifestItem{SeqNo: seqNo, Resolution: resolution})
|
||||
m.Index[seqNo] = len(m.Segments) - 1
|
||||
}
|
||||
|
||||
// Len reports how many distinct sequence numbers have been recorded.
|
||||
func (m *ManifestWriter) Len() int {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
return len(m.Segments)
|
||||
}
|
||||
|
||||
func (m *ManifestWriter) WriteManifest() {
|
||||
sort.Slice(m.Segments, func(i, j int) bool {
|
||||
return m.Segments[i].SeqNo < m.Segments[j].SeqNo
|
||||
// Sort a copy: reordering Segments in place would invalidate Index, and a
|
||||
// recording may still be writing into it.
|
||||
m.mu.Lock()
|
||||
segments := make([]ManifestItem, len(m.Segments))
|
||||
copy(segments, m.Segments)
|
||||
m.mu.Unlock()
|
||||
|
||||
sort.Slice(segments, func(i, j int) bool {
|
||||
return lessSeqNo(segments[i].SeqNo, segments[j].SeqNo)
|
||||
})
|
||||
|
||||
data, err := json.MarshalIndent(m.Segments, "", " ")
|
||||
data, err := json.MarshalIndent(segments, "", " ")
|
||||
if err != nil {
|
||||
log.Printf("Failed to marshal manifest: %v", err)
|
||||
logging.MustGetFactory().Download.Error("failed to marshal manifest", logging.KeyError, err.Error())
|
||||
@ -88,3 +113,15 @@ func (m *ManifestWriter) WriteManifest() {
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
// lessSeqNo orders sequence numbers numerically, so segment 9 precedes segment
|
||||
// 10 instead of sorting after it as the strings would. Values that are not
|
||||
// numeric fall back to lexicographic order.
|
||||
func lessSeqNo(a, b string) bool {
|
||||
ai, aErr := strconv.ParseUint(a, 10, 64)
|
||||
bi, bErr := strconv.ParseUint(b, 10, 64)
|
||||
if aErr == nil && bErr == nil {
|
||||
return ai < bi
|
||||
}
|
||||
return a < b
|
||||
}
|
||||
|
||||
@ -4,6 +4,8 @@ import (
|
||||
"encoding/json"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"sync"
|
||||
"testing"
|
||||
)
|
||||
|
||||
@ -41,7 +43,7 @@ func TestManifestWriter_AddOrUpdateSegment(t *testing.T) {
|
||||
writer := &ManifestWriter{
|
||||
ManifestPath: "test.json",
|
||||
Segments: make([]ManifestItem, 0),
|
||||
Index: make(map[string]*ManifestItem),
|
||||
Index: make(map[string]int),
|
||||
}
|
||||
|
||||
// Test adding new segment
|
||||
@ -112,7 +114,7 @@ func TestManifestWriter_WriteManifest(t *testing.T) {
|
||||
writer := &ManifestWriter{
|
||||
ManifestPath: manifestPath,
|
||||
Segments: make([]ManifestItem, 0),
|
||||
Index: make(map[string]*ManifestItem),
|
||||
Index: make(map[string]int),
|
||||
}
|
||||
|
||||
// Add some test segments out of order
|
||||
@ -172,7 +174,7 @@ func TestManifestWriter_WriteManifest_EmptySegments(t *testing.T) {
|
||||
writer := &ManifestWriter{
|
||||
ManifestPath: manifestPath,
|
||||
Segments: make([]ManifestItem, 0),
|
||||
Index: make(map[string]*ManifestItem),
|
||||
Index: make(map[string]int),
|
||||
}
|
||||
|
||||
// Write empty manifest
|
||||
@ -204,7 +206,7 @@ func TestManifestWriter_WriteManifest_InvalidPath(t *testing.T) {
|
||||
writer := &ManifestWriter{
|
||||
ManifestPath: "/invalid/path/that/does/not/exist/manifest.json",
|
||||
Segments: []ManifestItem{{SeqNo: "1001", Resolution: "1080p"}},
|
||||
Index: make(map[string]*ManifestItem),
|
||||
Index: make(map[string]int),
|
||||
}
|
||||
|
||||
// This should not panic, just fail gracefully
|
||||
@ -239,3 +241,169 @@ func TestManifestItem_JSONSerialization(t *testing.T) {
|
||||
t.Errorf("Resolution mismatch: expected '%s', got '%s'", item.Resolution, unmarshaled.Resolution)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolutionHeight(t *testing.T) {
|
||||
cases := map[string]int{
|
||||
"1080p": 1080,
|
||||
"720p": 720,
|
||||
"2160p": 2160,
|
||||
"1440p": 1440,
|
||||
"240p": 240,
|
||||
// Not a <height>p label: must rank below every real resolution rather
|
||||
// than accidentally outranking them.
|
||||
"unknown": 0,
|
||||
"": 0,
|
||||
"p": 0,
|
||||
}
|
||||
|
||||
for label, want := range cases {
|
||||
if got := ResolutionHeight(label); got != want {
|
||||
t.Errorf("ResolutionHeight(%q) = %d, want %d", label, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestAddOrUpdateSegmentKeepsHighestResolution pins the comparison used to
|
||||
// choose between variants. Comparing the labels as strings — as this once did —
|
||||
// ranks "720p" above "1080p", quietly downgrading the manifest.
|
||||
func TestAddOrUpdateSegmentKeepsHighestResolution(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
arriving []string
|
||||
want string
|
||||
}{
|
||||
{"higher replaces lower", []string{"720p", "1080p"}, "1080p"},
|
||||
{"lower does not replace higher", []string{"1080p", "720p"}, "1080p"},
|
||||
{"fewer digits must not win", []string{"1080p", "480p"}, "1080p"},
|
||||
{"4k beats 1080p", []string{"1080p", "2160p"}, "2160p"},
|
||||
{"unknown loses to a real resolution", []string{"unknown", "480p"}, "480p"},
|
||||
{"unknown does not displace a real resolution", []string{"1080p", "unknown"}, "1080p"},
|
||||
{"repeated identical resolution", []string{"720p", "720p"}, "720p"},
|
||||
}
|
||||
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
w := &ManifestWriter{}
|
||||
for _, res := range tc.arriving {
|
||||
w.AddOrUpdateSegment("1001", res)
|
||||
}
|
||||
|
||||
if len(w.Segments) != 1 {
|
||||
t.Fatalf("expected a single entry, got %d", len(w.Segments))
|
||||
}
|
||||
if got := w.Segments[0].Resolution; got != tc.want {
|
||||
t.Errorf("kept %q, want %q (arrival order %v)", got, tc.want, tc.arriving)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestAddOrUpdateSegmentUpdatesTheStoredEntry guards the aliasing bug: the
|
||||
// index once held a pointer to a local copy rather than to the stored element,
|
||||
// so updates were applied to an orphan and never reached the written manifest.
|
||||
func TestAddOrUpdateSegmentUpdatesTheStoredEntry(t *testing.T) {
|
||||
w := &ManifestWriter{}
|
||||
|
||||
// Enough entries to force at least one append reallocation, which would
|
||||
// invalidate any pointer held into the backing array.
|
||||
for i := 0; i < 64; i++ {
|
||||
w.AddOrUpdateSegment(strconv.Itoa(1000+i), "480p")
|
||||
}
|
||||
for i := 0; i < 64; i++ {
|
||||
w.AddOrUpdateSegment(strconv.Itoa(1000+i), "1080p")
|
||||
}
|
||||
|
||||
if len(w.Segments) != 64 {
|
||||
t.Fatalf("expected 64 entries, got %d", len(w.Segments))
|
||||
}
|
||||
for _, seg := range w.Segments {
|
||||
if seg.Resolution != "1080p" {
|
||||
t.Fatalf("segment %s kept %q; the upgrade never reached the stored entry", seg.SeqNo, seg.Resolution)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestAddOrUpdateSegmentIsConcurrencySafe reflects real use: every variant
|
||||
// downloads in its own goroutine and reports segments as they land.
|
||||
func TestAddOrUpdateSegmentIsConcurrencySafe(t *testing.T) {
|
||||
w := &ManifestWriter{}
|
||||
resolutions := []string{"240p", "480p", "720p", "1080p"}
|
||||
|
||||
var wg sync.WaitGroup
|
||||
for _, res := range resolutions {
|
||||
wg.Add(1)
|
||||
go func(res string) {
|
||||
defer wg.Done()
|
||||
for seq := 0; seq < 200; seq++ {
|
||||
w.AddOrUpdateSegment(strconv.Itoa(seq), res)
|
||||
}
|
||||
}(res)
|
||||
}
|
||||
wg.Wait()
|
||||
|
||||
if got := w.Len(); got != 200 {
|
||||
t.Errorf("expected 200 distinct sequence numbers, got %d", got)
|
||||
}
|
||||
for _, seg := range w.Segments {
|
||||
if seg.Resolution != "1080p" {
|
||||
t.Errorf("segment %s = %q, want the highest variant 1080p", seg.SeqNo, seg.Resolution)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestWriteManifestSortsNumerically checks that segment 9 precedes segment 10,
|
||||
// which string ordering would reverse.
|
||||
func TestWriteManifestSortsNumerically(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
w := &ManifestWriter{ManifestPath: filepath.Join(dir, "event.json")}
|
||||
|
||||
for _, seq := range []string{"100", "9", "1000", "10", "1"} {
|
||||
w.AddOrUpdateSegment(seq, "1080p")
|
||||
}
|
||||
w.WriteManifest()
|
||||
|
||||
data, err := os.ReadFile(w.ManifestPath)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to read manifest: %v", err)
|
||||
}
|
||||
|
||||
var items []ManifestItem
|
||||
if err := json.Unmarshal(data, &items); err != nil {
|
||||
t.Fatalf("manifest is not valid JSON: %v", err)
|
||||
}
|
||||
|
||||
want := []string{"1", "9", "10", "100", "1000"}
|
||||
if len(items) != len(want) {
|
||||
t.Fatalf("expected %d entries, got %d", len(want), len(items))
|
||||
}
|
||||
for i, seq := range want {
|
||||
if items[i].SeqNo != seq {
|
||||
t.Errorf("position %d = %q, want %q (full order: %v)", i, items[i].SeqNo, seq, items)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestWriteManifestLeavesIndexUsable ensures writing mid-recording does not
|
||||
// corrupt the writer: sorting in place would desynchronise Index from Segments.
|
||||
func TestWriteManifestLeavesIndexUsable(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
w := &ManifestWriter{ManifestPath: filepath.Join(dir, "event.json")}
|
||||
|
||||
w.AddOrUpdateSegment("20", "480p")
|
||||
w.AddOrUpdateSegment("10", "480p")
|
||||
w.WriteManifest()
|
||||
|
||||
// After the write, an upgrade must still land on the right entry.
|
||||
w.AddOrUpdateSegment("20", "1080p")
|
||||
|
||||
pos, ok := w.Index["20"]
|
||||
if !ok {
|
||||
t.Fatal("sequence 20 missing from the index after WriteManifest")
|
||||
}
|
||||
if got := w.Segments[pos].Resolution; got != "1080p" {
|
||||
t.Errorf("sequence 20 = %q, want 1080p; the index no longer matches the slice", got)
|
||||
}
|
||||
if w.Segments[w.Index["10"]].SeqNo != "10" {
|
||||
t.Error("index and slice disagree after WriteManifest")
|
||||
}
|
||||
}
|
||||
|
||||
@ -13,6 +13,7 @@ import (
|
||||
"net/http"
|
||||
"net/url"
|
||||
"path"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
@ -24,7 +25,27 @@ type StreamVariant struct {
|
||||
ID int
|
||||
Resolution string
|
||||
OutputDir string
|
||||
Writer *ManifestWriter
|
||||
}
|
||||
|
||||
// ResolutionHeight converts a label produced by ExtractResolution back into
|
||||
// the pixel height it stands for, so variants can be ranked by quality.
|
||||
//
|
||||
// The height is parsed rather than looked up in a table: labels come from the
|
||||
// playlist's own RESOLUTION attribute, so a stream can legitimately offer
|
||||
// 1440p or 2160p that no fixed table would anticipate. Anything not in
|
||||
// <height>p form — "unknown", from a playlist with no variants — returns 0 and
|
||||
// therefore ranks below every recognised resolution.
|
||||
//
|
||||
// Comparing the labels as strings instead is a trap worth naming: "720p" sorts
|
||||
// after "1080p", so lexicographic comparison silently prefers lower quality.
|
||||
func ResolutionHeight(resolution string) int {
|
||||
trimmed := strings.TrimSuffix(strings.ToLower(strings.TrimSpace(resolution)), "p")
|
||||
|
||||
height, err := strconv.Atoi(trimmed)
|
||||
if err != nil || height < 0 {
|
||||
return 0
|
||||
}
|
||||
return height
|
||||
}
|
||||
|
||||
// ExtractResolution maps a variant to the label used for its output directory,
|
||||
@ -50,7 +71,10 @@ func ExtractResolution(variant *m3u8.Variant) string {
|
||||
}
|
||||
}
|
||||
|
||||
func GetAllVariants(masterURL string, outputDir string, writer *ManifestWriter) ([]*StreamVariant, error) {
|
||||
// GetAllVariants fetches the master playlist and returns one entry per stream
|
||||
// variant. Segments are recorded into the manifest by VariantDownloader, which
|
||||
// receives it directly, so no writer is threaded through here.
|
||||
func GetAllVariants(masterURL string, outputDir string) ([]*StreamVariant, error) {
|
||||
client := &http.Client{}
|
||||
req, _ := http.NewRequest("GET", masterURL, nil)
|
||||
req.Header.Set("User-Agent", constants.HTTPUserAgent)
|
||||
@ -78,7 +102,6 @@ func GetAllVariants(masterURL string, outputDir string, writer *ManifestWriter)
|
||||
ID: 0,
|
||||
Resolution: "unknown",
|
||||
OutputDir: path.Join(outputDir, "unknown"),
|
||||
Writer: writer,
|
||||
}}, nil
|
||||
}
|
||||
|
||||
@ -166,6 +189,9 @@ func VariantDownloader(ctx context.Context, variant *StreamVariant, sem chan str
|
||||
log.Printf("✓ %s downloaded segment %s", j.Variant.Resolution, name)
|
||||
dlLog.Debug("segment downloaded", logging.KeySegment, name)
|
||||
registry.RecordSegmentDownloaded(variant.Resolution)
|
||||
if manifest != nil {
|
||||
manifest.AddOrUpdateSegment(strconv.FormatUint(j.Seq, 10), variant.Resolution)
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
|
||||
@ -3,10 +3,18 @@ package nas
|
||||
import "time"
|
||||
|
||||
type NASConfig struct {
|
||||
// Path is the destination as configured. It may be a UNC path, an
|
||||
// smb:// URL, or a directory on this host; see ParseLocation.
|
||||
Path string
|
||||
Username string
|
||||
Password string
|
||||
Timeout time.Duration
|
||||
RetryLimit int
|
||||
VerifySize bool
|
||||
// AutoMount allows the recorder to mount an SMB share itself when the
|
||||
// host has not already mounted it. Linux only, and requires root, so it
|
||||
// is off by default in favour of a clear "mount it first" error.
|
||||
AutoMount bool
|
||||
// MountPoint overrides where AutoMount attaches the share.
|
||||
MountPoint string
|
||||
}
|
||||
|
||||
98
pkg/nas/connector.go
Normal file
98
pkg/nas/connector.go
Normal file
@ -0,0 +1,98 @@
|
||||
package nas
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os/exec"
|
||||
"runtime"
|
||||
)
|
||||
|
||||
// Connector attaches a configured Location to the host filesystem so that
|
||||
// ordinary file operations reach the NAS.
|
||||
//
|
||||
// Windows and Linux disagree about what "connecting to a share" means —
|
||||
// Windows authenticates a UNC path in place, Linux needs the share mounted
|
||||
// somewhere in its single filesystem tree — so each host gets its own
|
||||
// implementation and NASService stays platform-neutral. Selection happens at
|
||||
// runtime rather than through build tags so that every implementation is
|
||||
// compiled, vetted and unit-testable on every platform; only the final exec
|
||||
// call is genuinely host-specific.
|
||||
type Connector interface {
|
||||
// Name identifies the strategy in logs and check output.
|
||||
Name() string
|
||||
// Attach makes loc reachable and reports where to write.
|
||||
Attach(ctx context.Context, loc Location, cfg NASConfig) (Attachment, error)
|
||||
// Detach undoes whatever Attach set up. Implementations must undo only
|
||||
// what they created: a share this process found already mounted belongs
|
||||
// to the host, and unmounting it could break unrelated software.
|
||||
Detach(ctx context.Context, att Attachment) error
|
||||
}
|
||||
|
||||
// Attachment is the result of a successful Attach.
|
||||
type Attachment struct {
|
||||
// BasePath is the host-native directory that maps to the configured
|
||||
// destination. Every file operation is resolved against it.
|
||||
BasePath string
|
||||
// Notes explain how the host reached the destination, for -check output
|
||||
// and startup logs.
|
||||
Notes []string
|
||||
// Owned records that this process created the connection or mount, and
|
||||
// so may tear it down on shutdown.
|
||||
Owned bool
|
||||
// mountPoint is the directory this process mounted, when Owned is set by
|
||||
// the Linux connector.
|
||||
mountPoint string
|
||||
// share is the network path this process connected, when Owned is set by
|
||||
// the Windows connector.
|
||||
share string
|
||||
}
|
||||
|
||||
// commandRunner executes an external command. Implementations keep argument
|
||||
// construction in a pure function and call through this, so tests can assert
|
||||
// exactly what would run without running it.
|
||||
type commandRunner func(ctx context.Context, name string, args ...string) ([]byte, error)
|
||||
|
||||
// execRunner is the production runner: it combines stdout and stderr because
|
||||
// mount and net both report the useful diagnosis on one or the other.
|
||||
func execRunner(ctx context.Context, name string, args ...string) ([]byte, error) {
|
||||
return exec.CommandContext(ctx, name, args...).CombinedOutput()
|
||||
}
|
||||
|
||||
// connectorFor picks the strategy for a destination on a given OS. A
|
||||
// destination that is already an ordinary directory needs no network step on
|
||||
// any platform, so it short-circuits before the OS switch.
|
||||
func connectorFor(loc Location, goos string) Connector {
|
||||
if !loc.IsRemote() {
|
||||
return &localConnector{goos: goos}
|
||||
}
|
||||
|
||||
switch goos {
|
||||
case "windows":
|
||||
return &windowsConnector{run: execRunner}
|
||||
case "linux":
|
||||
return &linuxConnector{run: execRunner, mountsPath: procMountsPath}
|
||||
default:
|
||||
return &unsupportedConnector{goos: goos}
|
||||
}
|
||||
}
|
||||
|
||||
// defaultConnector selects the strategy for the running host.
|
||||
func defaultConnector(loc Location) Connector {
|
||||
return connectorFor(loc, runtime.GOOS)
|
||||
}
|
||||
|
||||
// unsupportedConnector fails clearly on platforms this project does not target,
|
||||
// rather than silently writing a share-shaped directory to local disk.
|
||||
type unsupportedConnector struct {
|
||||
goos string
|
||||
}
|
||||
|
||||
func (c *unsupportedConnector) Name() string { return "unsupported" }
|
||||
|
||||
func (c *unsupportedConnector) Attach(_ context.Context, loc Location, _ NASConfig) (Attachment, error) {
|
||||
return Attachment{}, fmt.Errorf(
|
||||
"connecting to network share %s is not supported on %s; mount the share and set NAS_OUTPUT_PATH to the mount point",
|
||||
loc.SMBSource(), c.goos)
|
||||
}
|
||||
|
||||
func (c *unsupportedConnector) Detach(context.Context, Attachment) error { return nil }
|
||||
57
pkg/nas/connector_local.go
Normal file
57
pkg/nas/connector_local.go
Normal file
@ -0,0 +1,57 @@
|
||||
package nas
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// localConnector handles destinations that are already part of the host
|
||||
// filesystem: a plain output directory, or — the normal arrangement on Linux —
|
||||
// an SMB/NFS share mounted by fstab, autofs or systemd before the recorder
|
||||
// starts. There is nothing to connect, but there is something to check.
|
||||
type localConnector struct {
|
||||
goos string
|
||||
}
|
||||
|
||||
func (c *localConnector) Name() string { return "local" }
|
||||
|
||||
func (c *localConnector) Attach(_ context.Context, loc Location, _ NASConfig) (Attachment, error) {
|
||||
if err := c.validate(loc.Raw); err != nil {
|
||||
return Attachment{}, err
|
||||
}
|
||||
|
||||
return Attachment{
|
||||
BasePath: filepath.Clean(loc.Raw),
|
||||
Notes: []string{"local or already-mounted path (no network authentication)"},
|
||||
}, nil
|
||||
}
|
||||
|
||||
func (c *localConnector) Detach(context.Context, Attachment) error { return nil }
|
||||
|
||||
// validate rejects a Windows-shaped path on a non-Windows host. This looks
|
||||
// pedantic and is not: backslashes and colons are legal characters in Linux
|
||||
// filenames, so a stray C:\data or \\server\share would be created verbatim as
|
||||
// a local directory. Transfers would report success, cleanup would delete the
|
||||
// local originals, and nothing would ever reach the NAS.
|
||||
func (c *localConnector) validate(path string) error {
|
||||
if c.goos == "windows" {
|
||||
return nil
|
||||
}
|
||||
|
||||
if driveLetter.MatchString(path) {
|
||||
return fmt.Errorf(
|
||||
"%q is a Windows drive path but this host is %s; set NAS_OUTPUT_PATH to a mount point such as /mnt/nas/streams",
|
||||
path, c.goos)
|
||||
}
|
||||
|
||||
if strings.Contains(path, `\`) {
|
||||
return fmt.Errorf(
|
||||
"%q contains backslashes but this host is %s, where they are ordinary filename characters rather than separators; "+
|
||||
"set NAS_OUTPUT_PATH to a mount point such as /mnt/nas/streams, or to \\\\server\\share\\folder to have the share resolved automatically",
|
||||
path, c.goos)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
178
pkg/nas/connector_mount.go
Normal file
178
pkg/nas/connector_mount.go
Normal file
@ -0,0 +1,178 @@
|
||||
package nas
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// linuxConnector reaches a share the way Linux does: through a mount point in
|
||||
// the single filesystem tree. There is no per-path authentication, so the
|
||||
// question is not "can we connect" but "where is this share mounted".
|
||||
//
|
||||
// The default and by far the safest answer is that the host already mounted it
|
||||
// (fstab, autofs, systemd, or `docker run -v`), so the connector looks the
|
||||
// share up in the mount table. Mounting it here is opt-in because it requires
|
||||
// root and changes system state that outlives a failed run.
|
||||
type linuxConnector struct {
|
||||
run commandRunner
|
||||
mountsPath string
|
||||
// geteuid is indirected for tests; production uses os.Geteuid.
|
||||
geteuid func() int
|
||||
}
|
||||
|
||||
func (c *linuxConnector) Name() string { return "mount" }
|
||||
|
||||
func (c *linuxConnector) Attach(ctx context.Context, loc Location, cfg NASConfig) (Attachment, error) {
|
||||
entries, err := readMountTable(c.mountsPath)
|
||||
if err != nil {
|
||||
return Attachment{}, err
|
||||
}
|
||||
|
||||
if point, rest, ok := findMount(entries, loc); ok {
|
||||
return Attachment{
|
||||
BasePath: filepath.Join(append([]string{point}, rest...)...),
|
||||
Notes: []string{
|
||||
"share: " + loc.SMBSource(),
|
||||
fmt.Sprintf("resolved via existing mount at %s", point),
|
||||
},
|
||||
}, nil
|
||||
}
|
||||
|
||||
if !cfg.AutoMount {
|
||||
return Attachment{}, fmt.Errorf("%s", notMountedMessage(loc, cfg))
|
||||
}
|
||||
|
||||
return c.mount(ctx, loc, cfg)
|
||||
}
|
||||
|
||||
func (c *linuxConnector) Detach(ctx context.Context, att Attachment) error {
|
||||
// Never unmount a share the host had already mounted; other software may
|
||||
// be using it, and we did not put it there.
|
||||
if !att.Owned || att.mountPoint == "" {
|
||||
return nil
|
||||
}
|
||||
|
||||
if out, err := c.run(ctx, "umount", att.mountPoint); err != nil {
|
||||
return fmt.Errorf("failed to unmount %s: %v", att.mountPoint, summarize(err, out))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// mount performs an opt-in mount.cifs of the share.
|
||||
func (c *linuxConnector) mount(ctx context.Context, loc Location, cfg NASConfig) (Attachment, error) {
|
||||
geteuid := c.geteuid
|
||||
if geteuid == nil {
|
||||
geteuid = os.Geteuid
|
||||
}
|
||||
if geteuid() != 0 {
|
||||
return Attachment{}, fmt.Errorf(
|
||||
"NAS_AUTO_MOUNT is set but mounting %s requires root; run as root, or mount the share beforehand and %s",
|
||||
loc.SMBSource(), suggestMountPointConfig(loc))
|
||||
}
|
||||
|
||||
point := autoMountPoint(loc, cfg)
|
||||
if err := os.MkdirAll(point, 0755); err != nil {
|
||||
return Attachment{}, fmt.Errorf("failed to create mount point %s: %w", point, err)
|
||||
}
|
||||
|
||||
credentialsPath, cleanup, err := writeCredentialsFile(cfg)
|
||||
if err != nil {
|
||||
return Attachment{}, err
|
||||
}
|
||||
defer cleanup()
|
||||
|
||||
if cfg.Timeout > 0 {
|
||||
var cancel context.CancelFunc
|
||||
ctx, cancel = context.WithTimeout(ctx, cfg.Timeout)
|
||||
defer cancel()
|
||||
}
|
||||
|
||||
if out, err := c.run(ctx, "mount", mountArgs(loc, point, credentialsPath)...); err != nil {
|
||||
return Attachment{}, fmt.Errorf("failed to mount %s at %s: %v", loc.SMBSource(), point, summarize(err, out))
|
||||
}
|
||||
|
||||
return Attachment{
|
||||
BasePath: filepath.Join(append([]string{point}, loc.Sub...)...),
|
||||
Owned: true,
|
||||
mountPoint: point,
|
||||
Notes: []string{
|
||||
"share: " + loc.SMBSource(),
|
||||
"user: " + usernameNote(cfg.Username),
|
||||
fmt.Sprintf("mounted at %s by this process (will unmount on shutdown)", point),
|
||||
},
|
||||
}, nil
|
||||
}
|
||||
|
||||
// mountArgs builds the mount.cifs invocation. Kept pure so it can be asserted
|
||||
// without mounting anything.
|
||||
func mountArgs(loc Location, mountPoint, credentialsPath string) []string {
|
||||
options := "rw"
|
||||
if credentialsPath != "" {
|
||||
options += ",credentials=" + credentialsPath
|
||||
} else {
|
||||
options += ",guest"
|
||||
}
|
||||
return []string{"-t", "cifs", loc.SMBSource(), mountPoint, "-o", options}
|
||||
}
|
||||
|
||||
// writeCredentialsFile stages the NAS credentials in a 0600 file. mount.cifs
|
||||
// also accepts user=/pass= options, but those land in the process table where
|
||||
// any local user can read the password; a credentials file does not.
|
||||
func writeCredentialsFile(cfg NASConfig) (path string, cleanup func(), err error) {
|
||||
if cfg.Username == "" || cfg.Password == "" {
|
||||
return "", func() {}, nil
|
||||
}
|
||||
|
||||
dir, err := os.MkdirTemp("", "streamrecorder-cifs-")
|
||||
if err != nil {
|
||||
return "", func() {}, fmt.Errorf("failed to stage NAS credentials: %w", err)
|
||||
}
|
||||
cleanup = func() { os.RemoveAll(dir) }
|
||||
|
||||
path = filepath.Join(dir, "credentials")
|
||||
content := fmt.Sprintf("username=%s\npassword=%s\n", cfg.Username, cfg.Password)
|
||||
if err := os.WriteFile(path, []byte(content), 0600); err != nil {
|
||||
cleanup()
|
||||
return "", func() {}, fmt.Errorf("failed to write NAS credentials file: %w", err)
|
||||
}
|
||||
|
||||
return path, cleanup, nil
|
||||
}
|
||||
|
||||
// autoMountPoint chooses where to mount the share when the operator has not
|
||||
// named a location. Everything lands under one predictable directory so a
|
||||
// leftover mount is easy to find and clean up.
|
||||
func autoMountPoint(loc Location, cfg NASConfig) string {
|
||||
if cfg.MountPoint != "" {
|
||||
return cfg.MountPoint
|
||||
}
|
||||
return filepath.Join("/mnt", "streamrecorder", strings.ToLower(loc.Server), strings.ToLower(loc.Share))
|
||||
}
|
||||
|
||||
// notMountedMessage explains the failure in terms of what the operator should
|
||||
// actually type. A bare "no such mount" would leave them guessing.
|
||||
func notMountedMessage(loc Location, cfg NASConfig) string {
|
||||
source := loc.SMBSource()
|
||||
example := filepath.Join("/mnt", strings.ToLower(loc.Share))
|
||||
|
||||
credentials := ""
|
||||
if cfg.Username != "" {
|
||||
credentials = fmt.Sprintf(",username=%s,password=***", cfg.Username)
|
||||
}
|
||||
|
||||
return fmt.Sprintf(
|
||||
"%s is not mounted on this host. Either mount it and %s, for example:\n"+
|
||||
" sudo mkdir -p %s && sudo mount -t cifs %s %s -o rw%s\n"+
|
||||
"or set NAS_AUTO_MOUNT=true to have the recorder mount it (requires root)",
|
||||
source, suggestMountPointConfig(loc), example, source, example, credentials)
|
||||
}
|
||||
|
||||
func suggestMountPointConfig(loc Location) string {
|
||||
if len(loc.Sub) == 0 {
|
||||
return "set NAS_OUTPUT_PATH to the mount point"
|
||||
}
|
||||
return fmt.Sprintf("set NAS_OUTPUT_PATH to the mount point plus %s", loc.SubPath())
|
||||
}
|
||||
107
pkg/nas/connector_netuse.go
Normal file
107
pkg/nas/connector_netuse.go
Normal file
@ -0,0 +1,107 @@
|
||||
package nas
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// windowsConnector reaches a share the way Windows does: authenticate the UNC
|
||||
// path with `net use`, then use the UNC path directly as a filesystem path.
|
||||
// Nothing is mounted into a tree, so the resolved base path is the configured
|
||||
// UNC path itself.
|
||||
type windowsConnector struct {
|
||||
run commandRunner
|
||||
}
|
||||
|
||||
func (c *windowsConnector) Name() string { return "net use" }
|
||||
|
||||
func (c *windowsConnector) Attach(ctx context.Context, loc Location, cfg NASConfig) (Attachment, error) {
|
||||
share := loc.ShareUNC()
|
||||
att := Attachment{
|
||||
BasePath: loc.UNC(),
|
||||
share: share,
|
||||
Notes: []string{"share: " + share, "user: " + usernameNote(cfg.Username)},
|
||||
}
|
||||
|
||||
if cfg.Timeout > 0 {
|
||||
var cancel context.CancelFunc
|
||||
ctx, cancel = context.WithTimeout(ctx, cfg.Timeout)
|
||||
defer cancel()
|
||||
}
|
||||
|
||||
out, err := c.run(ctx, "net", netUseArgs(loc, cfg)...)
|
||||
if err != nil {
|
||||
// A failure here is not decisive. `net use` also fails when the share
|
||||
// is already connected — commonly because Explorer or another process
|
||||
// got there first — in which case the path is perfectly usable. Record
|
||||
// the reason and let the caller's write probe deliver the verdict,
|
||||
// rather than refusing to run against a working share.
|
||||
att.Notes = append(att.Notes,
|
||||
fmt.Sprintf("net use did not complete (%v); continuing, the share may already be connected", summarize(err, out)))
|
||||
return att, nil
|
||||
}
|
||||
|
||||
att.Owned = true
|
||||
att.Notes = append(att.Notes, "connected via net use")
|
||||
return att, nil
|
||||
}
|
||||
|
||||
func (c *windowsConnector) Detach(ctx context.Context, att Attachment) error {
|
||||
// Only drop connections this process made. Deleting a mapping the user or
|
||||
// another application established would break them.
|
||||
if !att.Owned || att.share == "" {
|
||||
return nil
|
||||
}
|
||||
|
||||
if out, err := c.run(ctx, "net", "use", att.share, "/delete"); err != nil {
|
||||
return fmt.Errorf("failed to disconnect %s: %v", att.share, summarize(err, out))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// netUseArgs builds the `net use` invocation. Kept pure so the Windows code
|
||||
// path can be asserted from any host, including the Linux box this is
|
||||
// developed on, without executing anything.
|
||||
func netUseArgs(loc Location, cfg NASConfig) []string {
|
||||
args := []string{"use", loc.ShareUNC()}
|
||||
if cfg.Username != "" && cfg.Password != "" {
|
||||
args = append(args, "/user:"+cfg.Username, cfg.Password)
|
||||
}
|
||||
return append(args, "/persistent:no")
|
||||
}
|
||||
|
||||
// summarize pairs an exec error with the first line of command output, which is
|
||||
// where net and mount put the actual diagnosis.
|
||||
func summarize(err error, out []byte) string {
|
||||
text := strings.TrimSpace(string(out))
|
||||
if text == "" {
|
||||
return err.Error()
|
||||
}
|
||||
if idx := strings.IndexByte(text, '\n'); idx >= 0 {
|
||||
text = text[:idx]
|
||||
}
|
||||
return fmt.Sprintf("%v: %s", err, text)
|
||||
}
|
||||
|
||||
func usernameNote(username string) string {
|
||||
if username == "" {
|
||||
return "(none, using ambient credentials)"
|
||||
}
|
||||
return username
|
||||
}
|
||||
|
||||
// joinBase resolves a relative path against a base that may be a UNC path.
|
||||
// filepath.Join is correct on Windows and would mangle the leading \\ anywhere
|
||||
// else, so UNC bases are joined with backslashes explicitly.
|
||||
func joinBase(base string, elem ...string) string {
|
||||
if len(elem) == 0 {
|
||||
return base
|
||||
}
|
||||
if strings.HasPrefix(base, `\\`) {
|
||||
parts := append([]string{strings.TrimRight(base, `\`)}, elem...)
|
||||
return strings.Join(parts, `\`)
|
||||
}
|
||||
return filepath.Join(append([]string{base}, elem...)...)
|
||||
}
|
||||
353
pkg/nas/connector_test.go
Normal file
353
pkg/nas/connector_test.go
Normal file
@ -0,0 +1,353 @@
|
||||
package nas
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// recordingRunner captures the commands a connector would run instead of
|
||||
// running them. This is what lets the Windows path be tested from a Linux
|
||||
// development machine: the decision logic and the exact argv are asserted
|
||||
// without ever invoking net.exe.
|
||||
type recordingRunner struct {
|
||||
calls [][]string
|
||||
output []byte
|
||||
err error
|
||||
}
|
||||
|
||||
func (r *recordingRunner) run(_ context.Context, name string, args ...string) ([]byte, error) {
|
||||
r.calls = append(r.calls, append([]string{name}, args...))
|
||||
return r.output, r.err
|
||||
}
|
||||
|
||||
func (r *recordingRunner) lastCall() string {
|
||||
if len(r.calls) == 0 {
|
||||
return ""
|
||||
}
|
||||
return strings.Join(r.calls[len(r.calls)-1], " ")
|
||||
}
|
||||
|
||||
func remoteLocation(t *testing.T, raw string) Location {
|
||||
t.Helper()
|
||||
loc, err := ParseLocation(raw)
|
||||
if err != nil {
|
||||
t.Fatalf("ParseLocation(%q) failed: %v", raw, err)
|
||||
}
|
||||
return loc
|
||||
}
|
||||
|
||||
func TestConnectorSelection(t *testing.T) {
|
||||
remote := remoteLocation(t, `\\nas\dci\streams`)
|
||||
local := remoteLocation(t, "/mnt/dci/streams")
|
||||
|
||||
cases := []struct {
|
||||
name string
|
||||
loc Location
|
||||
goos string
|
||||
want string
|
||||
}{
|
||||
{"windows share", remote, "windows", "net use"},
|
||||
{"linux share", remote, "linux", "mount"},
|
||||
{"darwin share", remote, "darwin", "unsupported"},
|
||||
{"local path on windows", local, "windows", "local"},
|
||||
{"local path on linux", local, "linux", "local"},
|
||||
// A already-mounted path needs no network step even where shares are
|
||||
// unsupported, so it must not fall through to the error connector.
|
||||
{"local path on darwin", local, "darwin", "local"},
|
||||
}
|
||||
|
||||
for _, tc := range cases {
|
||||
if got := connectorFor(tc.loc, tc.goos).Name(); got != tc.want {
|
||||
t.Errorf("%s: connectorFor = %q, want %q", tc.name, got, tc.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestWindowsConnectorAttach(t *testing.T) {
|
||||
runner := &recordingRunner{}
|
||||
c := &windowsConnector{run: runner.run}
|
||||
loc := remoteLocation(t, `\\HomeLabNAS\dci\streams`)
|
||||
|
||||
att, err := c.Attach(context.Background(), loc, NASConfig{Username: "NASAdmin", Password: "secret"})
|
||||
if err != nil {
|
||||
t.Fatalf("Attach failed: %v", err)
|
||||
}
|
||||
|
||||
// The base path is the full UNC path; net use authenticates only the share.
|
||||
if want := `\\HomeLabNAS\dci\streams`; att.BasePath != want {
|
||||
t.Errorf("BasePath = %q, want %q", att.BasePath, want)
|
||||
}
|
||||
if want := `net use \\HomeLabNAS\dci /user:NASAdmin secret /persistent:no`; runner.lastCall() != want {
|
||||
t.Errorf("command = %q, want %q", runner.lastCall(), want)
|
||||
}
|
||||
if !att.Owned {
|
||||
t.Error("a successful net use should be owned so it can be released on shutdown")
|
||||
}
|
||||
}
|
||||
|
||||
func TestWindowsConnectorOmitsCredentialsWhenUnset(t *testing.T) {
|
||||
runner := &recordingRunner{}
|
||||
c := &windowsConnector{run: runner.run}
|
||||
|
||||
if _, err := c.Attach(context.Background(), remoteLocation(t, `\\nas\dci`), NASConfig{}); err != nil {
|
||||
t.Fatalf("Attach failed: %v", err)
|
||||
}
|
||||
|
||||
if want := `net use \\nas\dci /persistent:no`; runner.lastCall() != want {
|
||||
t.Errorf("command = %q, want %q", runner.lastCall(), want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestWindowsConnectorContinuesWhenAlreadyConnected(t *testing.T) {
|
||||
// net use fails when a connection already exists, but the share is usable;
|
||||
// refusing to run would be wrong. The write probe is the real verdict.
|
||||
runner := &recordingRunner{
|
||||
err: fmt.Errorf("exit status 2"),
|
||||
output: []byte("The network connection could not be found.\nMore help is available..."),
|
||||
}
|
||||
c := &windowsConnector{run: runner.run}
|
||||
|
||||
att, err := c.Attach(context.Background(), remoteLocation(t, `\\nas\dci`), NASConfig{})
|
||||
if err != nil {
|
||||
t.Fatalf("Attach should tolerate a net use failure, got: %v", err)
|
||||
}
|
||||
if att.Owned {
|
||||
t.Error("a failed net use must not be marked owned, or Detach would drop someone else's connection")
|
||||
}
|
||||
|
||||
notes := strings.Join(att.Notes, "\n")
|
||||
if !strings.Contains(notes, "net use did not complete") {
|
||||
t.Errorf("expected the failure to be recorded in notes:\n%s", notes)
|
||||
}
|
||||
// Only the first line of output is worth surfacing.
|
||||
if strings.Contains(notes, "More help is available") {
|
||||
t.Errorf("notes should not carry the full usage dump:\n%s", notes)
|
||||
}
|
||||
}
|
||||
|
||||
func TestWindowsConnectorDetachOnlyReleasesOwned(t *testing.T) {
|
||||
runner := &recordingRunner{}
|
||||
c := &windowsConnector{run: runner.run}
|
||||
|
||||
if err := c.Detach(context.Background(), Attachment{share: `\\nas\dci`, Owned: false}); err != nil {
|
||||
t.Fatalf("Detach failed: %v", err)
|
||||
}
|
||||
if len(runner.calls) != 0 {
|
||||
t.Errorf("must not disconnect a share this process did not connect, ran: %v", runner.calls)
|
||||
}
|
||||
|
||||
if err := c.Detach(context.Background(), Attachment{share: `\\nas\dci`, Owned: true}); err != nil {
|
||||
t.Fatalf("Detach failed: %v", err)
|
||||
}
|
||||
if want := `net use \\nas\dci /delete`; runner.lastCall() != want {
|
||||
t.Errorf("command = %q, want %q", runner.lastCall(), want)
|
||||
}
|
||||
}
|
||||
|
||||
// writeMountTable stages a /proc/mounts-format file.
|
||||
func writeMountTable(t *testing.T, lines ...string) string {
|
||||
t.Helper()
|
||||
path := filepath.Join(t.TempDir(), "mounts")
|
||||
body := strings.Join(lines, "\n") + "\n"
|
||||
if err := os.WriteFile(path, []byte(body), 0644); err != nil {
|
||||
t.Fatalf("failed to write mount table: %v", err)
|
||||
}
|
||||
return path
|
||||
}
|
||||
|
||||
func TestLinuxConnectorResolvesExistingMount(t *testing.T) {
|
||||
mounts := writeMountTable(t,
|
||||
"sysfs /sys sysfs rw,nosuid 0 0",
|
||||
"//homelabnas/dci /mnt/dci cifs rw,relatime,username=nasadmin 0 0",
|
||||
)
|
||||
c := &linuxConnector{run: (&recordingRunner{}).run, mountsPath: mounts}
|
||||
|
||||
att, err := c.Attach(context.Background(), remoteLocation(t, `\\HomeLabNAS\dci\streams`), NASConfig{})
|
||||
if err != nil {
|
||||
t.Fatalf("Attach failed: %v", err)
|
||||
}
|
||||
|
||||
// Server and share names are case-insensitive on SMB, so the configured
|
||||
// \\HomeLabNAS\dci must match a //homelabnas/dci mount.
|
||||
if want := "/mnt/dci/streams"; att.BasePath != want {
|
||||
t.Errorf("BasePath = %q, want %q", att.BasePath, want)
|
||||
}
|
||||
if att.Owned {
|
||||
t.Error("a pre-existing mount must not be owned, or shutdown would unmount it")
|
||||
}
|
||||
}
|
||||
|
||||
func TestLinuxConnectorPrefersDeepestMount(t *testing.T) {
|
||||
// Both mounts could serve the destination; the more specific one is the
|
||||
// only one guaranteed to actually contain it.
|
||||
mounts := writeMountTable(t,
|
||||
"//nas/dci /mnt/dci cifs rw 0 0",
|
||||
"//nas/dci/streams /mnt/streams cifs rw 0 0",
|
||||
)
|
||||
c := &linuxConnector{run: (&recordingRunner{}).run, mountsPath: mounts}
|
||||
|
||||
att, err := c.Attach(context.Background(), remoteLocation(t, `\\nas\dci\streams\2026`), NASConfig{})
|
||||
if err != nil {
|
||||
t.Fatalf("Attach failed: %v", err)
|
||||
}
|
||||
if want := "/mnt/streams/2026"; att.BasePath != want {
|
||||
t.Errorf("BasePath = %q, want %q", att.BasePath, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLinuxConnectorMatchesNFSAndEscapedPoints(t *testing.T) {
|
||||
mounts := writeMountTable(t,
|
||||
`nas:/dci /mnt/my\040share nfs4 rw 0 0`,
|
||||
)
|
||||
c := &linuxConnector{run: (&recordingRunner{}).run, mountsPath: mounts}
|
||||
|
||||
att, err := c.Attach(context.Background(), remoteLocation(t, `\\nas\dci\streams`), NASConfig{})
|
||||
if err != nil {
|
||||
t.Fatalf("Attach failed: %v", err)
|
||||
}
|
||||
if want := "/mnt/my share/streams"; att.BasePath != want {
|
||||
t.Errorf("BasePath = %q, want %q", att.BasePath, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLinuxConnectorIgnoresNonNetworkMounts(t *testing.T) {
|
||||
// A local directory that merely looks like the share must not match.
|
||||
mounts := writeMountTable(t, "//nas/dci /mnt/dci ext4 rw 0 0")
|
||||
c := &linuxConnector{run: (&recordingRunner{}).run, mountsPath: mounts}
|
||||
|
||||
if _, err := c.Attach(context.Background(), remoteLocation(t, `\\nas\dci`), NASConfig{}); err == nil {
|
||||
t.Fatal("expected an ext4 mount not to satisfy a network share")
|
||||
}
|
||||
}
|
||||
|
||||
func TestLinuxConnectorUnmountedGivesActionableError(t *testing.T) {
|
||||
mounts := writeMountTable(t, "sysfs /sys sysfs rw 0 0")
|
||||
c := &linuxConnector{run: (&recordingRunner{}).run, mountsPath: mounts}
|
||||
|
||||
_, err := c.Attach(context.Background(), remoteLocation(t, `\\nas\dci\streams`), NASConfig{Username: "admin"})
|
||||
if err == nil {
|
||||
t.Fatal("expected an unmounted share to fail")
|
||||
}
|
||||
|
||||
msg := err.Error()
|
||||
for _, want := range []string{"//nas/dci", "mount -t cifs", "NAS_AUTO_MOUNT", "NAS_OUTPUT_PATH"} {
|
||||
if !strings.Contains(msg, want) {
|
||||
t.Errorf("error should mention %q:\n%s", want, msg)
|
||||
}
|
||||
}
|
||||
// The remedy must not echo the real password.
|
||||
if strings.Contains(msg, "hunter2") {
|
||||
t.Error("error message leaked a credential")
|
||||
}
|
||||
}
|
||||
|
||||
func TestLinuxConnectorAutoMountRequiresRoot(t *testing.T) {
|
||||
mounts := writeMountTable(t, "sysfs /sys sysfs rw 0 0")
|
||||
runner := &recordingRunner{}
|
||||
c := &linuxConnector{run: runner.run, mountsPath: mounts, geteuid: func() int { return 1000 }}
|
||||
|
||||
_, err := c.Attach(context.Background(), remoteLocation(t, `\\nas\dci`), NASConfig{AutoMount: true})
|
||||
if err == nil || !strings.Contains(err.Error(), "requires root") {
|
||||
t.Fatalf("expected a root requirement error, got: %v", err)
|
||||
}
|
||||
if len(runner.calls) != 0 {
|
||||
t.Errorf("must not attempt mount without privileges, ran: %v", runner.calls)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLinuxConnectorAutoMount(t *testing.T) {
|
||||
mounts := writeMountTable(t, "sysfs /sys sysfs rw 0 0")
|
||||
runner := &recordingRunner{}
|
||||
point := filepath.Join(t.TempDir(), "mnt", "dci")
|
||||
c := &linuxConnector{run: runner.run, mountsPath: mounts, geteuid: func() int { return 0 }}
|
||||
|
||||
cfg := NASConfig{AutoMount: true, MountPoint: point, Username: "admin", Password: "hunter2", Timeout: 5 * time.Second}
|
||||
att, err := c.Attach(context.Background(), remoteLocation(t, `\\nas\dci\streams`), cfg)
|
||||
if err != nil {
|
||||
t.Fatalf("Attach failed: %v", err)
|
||||
}
|
||||
|
||||
if want := filepath.Join(point, "streams"); att.BasePath != want {
|
||||
t.Errorf("BasePath = %q, want %q", att.BasePath, want)
|
||||
}
|
||||
if !att.Owned || att.mountPoint != point {
|
||||
t.Errorf("a self-created mount must be owned and recorded, got owned=%v point=%q", att.Owned, att.mountPoint)
|
||||
}
|
||||
|
||||
call := runner.lastCall()
|
||||
if !strings.HasPrefix(call, "mount -t cifs //nas/dci "+point+" -o rw,credentials=") {
|
||||
t.Errorf("unexpected mount command: %q", call)
|
||||
}
|
||||
// The password must reach mount.cifs through a credentials file, never
|
||||
// through argv where any local user could read it from the process table.
|
||||
if strings.Contains(call, "hunter2") {
|
||||
t.Errorf("password leaked into the mount command line: %q", call)
|
||||
}
|
||||
// The staged credentials file must not outlive the mount call.
|
||||
credPath := call[strings.Index(call, "credentials=")+len("credentials="):]
|
||||
if _, err := os.Stat(credPath); !os.IsNotExist(err) {
|
||||
t.Errorf("credentials file %s should have been removed, stat err = %v", credPath, err)
|
||||
}
|
||||
|
||||
if err := c.Detach(context.Background(), att); err != nil {
|
||||
t.Fatalf("Detach failed: %v", err)
|
||||
}
|
||||
if want := "umount " + point; runner.lastCall() != want {
|
||||
t.Errorf("detach command = %q, want %q", runner.lastCall(), want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLinuxConnectorDetachLeavesForeignMounts(t *testing.T) {
|
||||
runner := &recordingRunner{}
|
||||
c := &linuxConnector{run: runner.run, mountsPath: writeMountTable(t)}
|
||||
|
||||
if err := c.Detach(context.Background(), Attachment{BasePath: "/mnt/dci", Owned: false}); err != nil {
|
||||
t.Fatalf("Detach failed: %v", err)
|
||||
}
|
||||
if len(runner.calls) != 0 {
|
||||
t.Errorf("must not unmount a share the host mounted, ran: %v", runner.calls)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLocalConnectorRejectsWindowsPathsOffWindows(t *testing.T) {
|
||||
c := &localConnector{goos: "linux"}
|
||||
|
||||
for _, raw := range []string{`C:\data\streams`, `data\streams`} {
|
||||
loc, err := ParseLocation(raw)
|
||||
if err != nil {
|
||||
t.Fatalf("ParseLocation(%q) failed: %v", raw, err)
|
||||
}
|
||||
if _, err := c.Attach(context.Background(), loc, NASConfig{}); err == nil {
|
||||
t.Errorf("%q should be rejected on Linux: backslashes are legal filename characters there, "+
|
||||
"so it would silently create a local directory instead of writing to the NAS", raw)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestLocalConnectorAcceptsWindowsPathsOnWindows(t *testing.T) {
|
||||
c := &localConnector{goos: "windows"}
|
||||
loc, err := ParseLocation(`C:\data\streams`)
|
||||
if err != nil {
|
||||
t.Fatalf("ParseLocation failed: %v", err)
|
||||
}
|
||||
|
||||
if _, err := c.Attach(context.Background(), loc, NASConfig{}); err != nil {
|
||||
t.Errorf("a drive path is valid on Windows: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestUnsupportedConnectorExplainsItself(t *testing.T) {
|
||||
c := &unsupportedConnector{goos: "plan9"}
|
||||
_, err := c.Attach(context.Background(), remoteLocation(t, `\\nas\dci`), NASConfig{})
|
||||
if err == nil {
|
||||
t.Fatal("expected an error on an unsupported platform")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "plan9") || !strings.Contains(err.Error(), "NAS_OUTPUT_PATH") {
|
||||
t.Errorf("error should name the platform and the remedy: %v", err)
|
||||
}
|
||||
}
|
||||
144
pkg/nas/location.go
Normal file
144
pkg/nas/location.go
Normal file
@ -0,0 +1,144 @@
|
||||
package nas
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Location is a configured NAS destination parsed into a host-neutral form.
|
||||
//
|
||||
// The same destination is spelled differently depending on where you're
|
||||
// standing: Windows wants \\HomeLabNAS\dci\streams, a Linux host reaches the
|
||||
// same bytes through a mount point like /mnt/dci/streams, and either host may
|
||||
// simply be handed a plain local directory. Parsing happens once, up front,
|
||||
// and platform-specific rendering is left to the Connector implementations.
|
||||
type Location struct {
|
||||
// Raw is the destination exactly as configured, for error messages.
|
||||
Raw string
|
||||
// Server is the host serving the share; empty for a plain local path.
|
||||
Server string
|
||||
// Share is the top-level export name; empty for a plain local path.
|
||||
Share string
|
||||
// Sub holds the path components below the share, if any.
|
||||
Sub []string
|
||||
}
|
||||
|
||||
// driveLetter matches a Windows drive-qualified path such as C:\data.
|
||||
var driveLetter = regexp.MustCompile(`^[A-Za-z]:[\\/]`)
|
||||
|
||||
// ParseLocation accepts every spelling of a destination this project supports:
|
||||
// Windows UNC (\\server\share\dir), the SMB source form (//server/share/dir),
|
||||
// a smb:// URL, or a plain local/mounted directory. It does no I/O and makes no
|
||||
// assumptions about the host OS, so the result is safe to compute anywhere.
|
||||
func ParseLocation(raw string) (Location, error) {
|
||||
trimmed := strings.TrimSpace(raw)
|
||||
if trimmed == "" {
|
||||
return Location{}, fmt.Errorf("NAS path is empty")
|
||||
}
|
||||
|
||||
loc := Location{Raw: trimmed}
|
||||
|
||||
remainder, isRemote := stripRemotePrefix(trimmed)
|
||||
if !isRemote {
|
||||
return loc, nil
|
||||
}
|
||||
|
||||
parts := splitPathComponents(remainder)
|
||||
if len(parts) < 2 {
|
||||
return Location{}, fmt.Errorf(
|
||||
"%q names a server but no share; expected a form like \\\\server\\share\\folder", trimmed)
|
||||
}
|
||||
|
||||
loc.Server = parts[0]
|
||||
loc.Share = parts[1]
|
||||
loc.Sub = parts[2:]
|
||||
return loc, nil
|
||||
}
|
||||
|
||||
// stripRemotePrefix reports whether raw names a network share and returns the
|
||||
// server/share/subpath portion when it does. A leading // is treated as a share
|
||||
// rather than a local path: POSIX leaves paths beginning with exactly two
|
||||
// slashes implementation-defined, and nobody configures a NAS destination that
|
||||
// way by accident.
|
||||
func stripRemotePrefix(raw string) (string, bool) {
|
||||
lower := strings.ToLower(raw)
|
||||
switch {
|
||||
case strings.HasPrefix(lower, "smb://"):
|
||||
return raw[len("smb://"):], true
|
||||
case strings.HasPrefix(lower, "cifs://"):
|
||||
return raw[len("cifs://"):], true
|
||||
case strings.HasPrefix(raw, `\\`):
|
||||
return raw[2:], true
|
||||
case strings.HasPrefix(raw, "//"):
|
||||
return raw[2:], true
|
||||
default:
|
||||
return raw, false
|
||||
}
|
||||
}
|
||||
|
||||
// splitPathComponents splits on either separator and drops empty components, so
|
||||
// that \\server\share\\folder\ and //server/share/folder agree.
|
||||
func splitPathComponents(path string) []string {
|
||||
fields := strings.FieldsFunc(path, func(r rune) bool { return r == '/' || r == '\\' })
|
||||
out := make([]string, 0, len(fields))
|
||||
for _, f := range fields {
|
||||
if f != "" {
|
||||
out = append(out, f)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// IsRemote reports whether reaching this destination requires attaching a
|
||||
// network share first, as opposed to writing to a directory that is already
|
||||
// part of the local filesystem.
|
||||
func (l Location) IsRemote() bool {
|
||||
return l.Server != "" && l.Share != ""
|
||||
}
|
||||
|
||||
// UNC renders the destination in Windows form: \\server\share\sub.
|
||||
func (l Location) UNC() string {
|
||||
if !l.IsRemote() {
|
||||
return l.Raw
|
||||
}
|
||||
return `\\` + strings.Join(append([]string{l.Server, l.Share}, l.Sub...), `\`)
|
||||
}
|
||||
|
||||
// ShareUNC renders just the share in Windows form: \\server\share. This is the
|
||||
// unit `net use` authenticates against; the subpath below it is an ordinary
|
||||
// directory on the connected share.
|
||||
func (l Location) ShareUNC() string {
|
||||
if !l.IsRemote() {
|
||||
return ""
|
||||
}
|
||||
return `\\` + l.Server + `\` + l.Share
|
||||
}
|
||||
|
||||
// SMBSource renders the share in the //server/share form used by mount tables
|
||||
// and by mount.cifs. It deliberately omits Sub: that part is a path *within*
|
||||
// the mounted share, not part of the share's identity.
|
||||
func (l Location) SMBSource() string {
|
||||
if !l.IsRemote() {
|
||||
return ""
|
||||
}
|
||||
return "//" + l.Server + "/" + l.Share
|
||||
}
|
||||
|
||||
// SubPath renders the components below the share using the host's separator, or
|
||||
// "" when the destination is the share root.
|
||||
func (l Location) SubPath() string {
|
||||
if len(l.Sub) == 0 {
|
||||
return ""
|
||||
}
|
||||
return filepath.Join(l.Sub...)
|
||||
}
|
||||
|
||||
// String describes the destination for logs and check output.
|
||||
func (l Location) String() string {
|
||||
if l.IsRemote() {
|
||||
return l.UNC()
|
||||
}
|
||||
return l.Raw
|
||||
}
|
||||
123
pkg/nas/location_test.go
Normal file
123
pkg/nas/location_test.go
Normal file
@ -0,0 +1,123 @@
|
||||
package nas
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestParseLocationRemoteForms(t *testing.T) {
|
||||
// Every spelling of the same destination must parse identically, so that
|
||||
// an operator can configure it the way their platform's documentation
|
||||
// describes without changing behaviour.
|
||||
equivalent := []string{
|
||||
`\\HomeLabNAS\dci\streams`,
|
||||
`//HomeLabNAS/dci/streams`,
|
||||
`smb://HomeLabNAS/dci/streams`,
|
||||
`SMB://HomeLabNAS/dci/streams`,
|
||||
`cifs://HomeLabNAS/dci/streams`,
|
||||
`\\HomeLabNAS\dci\streams\`,
|
||||
`\\HomeLabNAS\\dci\\streams`,
|
||||
}
|
||||
|
||||
for _, raw := range equivalent {
|
||||
loc, err := ParseLocation(raw)
|
||||
if err != nil {
|
||||
t.Errorf("ParseLocation(%q) failed: %v", raw, err)
|
||||
continue
|
||||
}
|
||||
if !loc.IsRemote() {
|
||||
t.Errorf("ParseLocation(%q) should be remote", raw)
|
||||
}
|
||||
if loc.Server != "HomeLabNAS" || loc.Share != "dci" {
|
||||
t.Errorf("ParseLocation(%q) = server %q share %q, want HomeLabNAS/dci", raw, loc.Server, loc.Share)
|
||||
}
|
||||
if len(loc.Sub) != 1 || loc.Sub[0] != "streams" {
|
||||
t.Errorf("ParseLocation(%q) sub = %v, want [streams]", raw, loc.Sub)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseLocationRendering(t *testing.T) {
|
||||
loc, err := ParseLocation(`//HomeLabNAS/dci/streams/2026`)
|
||||
if err != nil {
|
||||
t.Fatalf("unexpected error: %v", err)
|
||||
}
|
||||
|
||||
if got, want := loc.UNC(), `\\HomeLabNAS\dci\streams\2026`; got != want {
|
||||
t.Errorf("UNC() = %q, want %q", got, want)
|
||||
}
|
||||
if got, want := loc.ShareUNC(), `\\HomeLabNAS\dci`; got != want {
|
||||
t.Errorf("ShareUNC() = %q, want %q", got, want)
|
||||
}
|
||||
if got, want := loc.SMBSource(), "//HomeLabNAS/dci"; got != want {
|
||||
t.Errorf("SMBSource() = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseLocationShareRoot(t *testing.T) {
|
||||
loc, err := ParseLocation(`\\nas\media`)
|
||||
if err != nil {
|
||||
t.Fatalf("unexpected error: %v", err)
|
||||
}
|
||||
if len(loc.Sub) != 0 {
|
||||
t.Errorf("expected no subpath, got %v", loc.Sub)
|
||||
}
|
||||
if got, want := loc.UNC(), `\\nas\media`; got != want {
|
||||
t.Errorf("UNC() = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseLocationLocalPaths(t *testing.T) {
|
||||
for _, raw := range []string{"/mnt/nas/streams", "data/streams", `C:\data\streams`, "/mnt"} {
|
||||
loc, err := ParseLocation(raw)
|
||||
if err != nil {
|
||||
t.Errorf("ParseLocation(%q) failed: %v", raw, err)
|
||||
continue
|
||||
}
|
||||
if loc.IsRemote() {
|
||||
t.Errorf("ParseLocation(%q) should not be treated as a network share", raw)
|
||||
}
|
||||
if loc.Raw != raw {
|
||||
t.Errorf("ParseLocation(%q).Raw = %q", raw, loc.Raw)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseLocationRejectsIncomplete(t *testing.T) {
|
||||
for _, raw := range []string{"", " ", `\\`, `\\server`, "//server", "smb://server"} {
|
||||
if _, err := ParseLocation(raw); err == nil {
|
||||
t.Errorf("ParseLocation(%q) should have failed", raw)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseLocationTrimsWhitespace(t *testing.T) {
|
||||
loc, err := ParseLocation(" \\\\nas\\dci ")
|
||||
if err != nil {
|
||||
t.Fatalf("unexpected error: %v", err)
|
||||
}
|
||||
if loc.Server != "nas" || loc.Share != "dci" {
|
||||
t.Errorf("whitespace not trimmed: server %q share %q", loc.Server, loc.Share)
|
||||
}
|
||||
}
|
||||
|
||||
func TestJoinBaseKeepsUNCSeparators(t *testing.T) {
|
||||
// filepath.Join would produce \\nas\dci/event on Linux, which is not a
|
||||
// path Windows would accept if the value were ever written to state.
|
||||
got := joinBase(`\\nas\dci`, "event", "1080p")
|
||||
if want := `\\nas\dci\event\1080p`; got != want {
|
||||
t.Errorf("joinBase UNC = %q, want %q", got, want)
|
||||
}
|
||||
|
||||
if got, want := joinBase(`\\nas\dci\`, "event"), `\\nas\dci\event`; got != want {
|
||||
t.Errorf("joinBase trailing separator = %q, want %q", got, want)
|
||||
}
|
||||
|
||||
if got := joinBase("/mnt/dci"); got != "/mnt/dci" {
|
||||
t.Errorf("joinBase with no elements = %q", got)
|
||||
}
|
||||
|
||||
if got, want := joinBase("/mnt/dci", "event"), "/mnt/dci/event"; !strings.HasSuffix(got, want) {
|
||||
t.Errorf("joinBase local = %q, want suffix %q", got, want)
|
||||
}
|
||||
}
|
||||
135
pkg/nas/mounts.go
Normal file
135
pkg/nas/mounts.go
Normal file
@ -0,0 +1,135 @@
|
||||
package nas
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"fmt"
|
||||
"os"
|
||||
"strconv"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// procMountsPath is the kernel's view of what is mounted in this process's
|
||||
// mount namespace, which is the right question to ask inside a container too.
|
||||
const procMountsPath = "/proc/mounts"
|
||||
|
||||
// mountEntry is one line of the mount table.
|
||||
type mountEntry struct {
|
||||
Source string
|
||||
Point string
|
||||
FSType string
|
||||
}
|
||||
|
||||
// networkFilesystems are the mount types that can plausibly be serving a NAS
|
||||
// share. Restricting the search avoids matching, say, a local directory that
|
||||
// merely happens to be named like a server.
|
||||
var networkFilesystems = map[string]bool{
|
||||
"cifs": true,
|
||||
"smb3": true,
|
||||
"smbfs": true,
|
||||
"nfs": true,
|
||||
"nfs4": true,
|
||||
"fuse.smbnetfs": true,
|
||||
}
|
||||
|
||||
// readMountTable parses /proc/mounts (or any file in its format). Fields are
|
||||
// whitespace-separated: source, mount point, type, options, dump, pass.
|
||||
func readMountTable(path string) ([]mountEntry, error) {
|
||||
f, err := os.Open(path)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to read mount table %s: %w", path, err)
|
||||
}
|
||||
defer f.Close()
|
||||
|
||||
var entries []mountEntry
|
||||
scanner := bufio.NewScanner(f)
|
||||
for scanner.Scan() {
|
||||
fields := strings.Fields(scanner.Text())
|
||||
if len(fields) < 3 {
|
||||
continue
|
||||
}
|
||||
entries = append(entries, mountEntry{
|
||||
Source: unescapeMountField(fields[0]),
|
||||
Point: unescapeMountField(fields[1]),
|
||||
FSType: fields[2],
|
||||
})
|
||||
}
|
||||
if err := scanner.Err(); err != nil {
|
||||
return nil, fmt.Errorf("failed to scan mount table %s: %w", path, err)
|
||||
}
|
||||
|
||||
return entries, nil
|
||||
}
|
||||
|
||||
// unescapeMountField decodes the octal escapes the kernel uses for characters
|
||||
// that would otherwise break the whitespace-separated format — most often \040
|
||||
// for a space in a mount point.
|
||||
func unescapeMountField(field string) string {
|
||||
if !strings.Contains(field, `\`) {
|
||||
return field
|
||||
}
|
||||
|
||||
var b strings.Builder
|
||||
for i := 0; i < len(field); i++ {
|
||||
if field[i] == '\\' && i+3 < len(field) {
|
||||
if v, err := strconv.ParseUint(field[i+1:i+4], 8, 8); err == nil {
|
||||
b.WriteByte(byte(v))
|
||||
i += 3
|
||||
continue
|
||||
}
|
||||
}
|
||||
b.WriteByte(field[i])
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
|
||||
// findMount locates the mount serving loc and returns its mount point plus the
|
||||
// path components still to be appended below it.
|
||||
//
|
||||
// The search runs deepest-first because a share may be mounted at any level:
|
||||
// //nas/dci might be mounted at /mnt/dci, or //nas/dci/streams mounted directly
|
||||
// at /mnt/streams. Preferring the longest matching source keeps both working
|
||||
// and avoids pointing at a parent that the mount does not actually cover.
|
||||
func findMount(entries []mountEntry, loc Location) (point string, rest []string, ok bool) {
|
||||
if !loc.IsRemote() {
|
||||
return "", nil, false
|
||||
}
|
||||
|
||||
for depth := len(loc.Sub); depth >= 0; depth-- {
|
||||
for _, candidate := range mountSourceForms(loc, depth) {
|
||||
for _, entry := range entries {
|
||||
if !networkFilesystems[entry.FSType] {
|
||||
continue
|
||||
}
|
||||
if normalizeMountSource(entry.Source) == candidate {
|
||||
return entry.Point, loc.Sub[depth:], true
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return "", nil, false
|
||||
}
|
||||
|
||||
// mountSourceForms renders the share, plus its first `depth` subdirectories, in
|
||||
// the spellings a mount table might use for it.
|
||||
func mountSourceForms(loc Location, depth int) []string {
|
||||
segments := append([]string{loc.Share}, loc.Sub[:depth]...)
|
||||
tail := strings.Join(segments, "/")
|
||||
|
||||
return []string{
|
||||
normalizeMountSource("//" + loc.Server + "/" + tail), // cifs/smb3
|
||||
normalizeMountSource(loc.Server + ":/" + tail), // nfs
|
||||
}
|
||||
}
|
||||
|
||||
// normalizeMountSource makes sources comparable: SMB servers and shares are
|
||||
// case-insensitive, separators may be either slash, and a trailing slash is not
|
||||
// meaningful.
|
||||
func normalizeMountSource(source string) string {
|
||||
s := strings.ReplaceAll(source, `\`, "/")
|
||||
s = strings.ToLower(strings.TrimSpace(s))
|
||||
if len(s) > 1 {
|
||||
s = strings.TrimRight(s, "/")
|
||||
}
|
||||
return s
|
||||
}
|
||||
96
pkg/nas/mounts_test.go
Normal file
96
pkg/nas/mounts_test.go
Normal file
@ -0,0 +1,96 @@
|
||||
package nas
|
||||
|
||||
import (
|
||||
"os"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestUnescapeMountField(t *testing.T) {
|
||||
cases := map[string]string{
|
||||
`/mnt/plain`: "/mnt/plain",
|
||||
`/mnt/my\040share`: "/mnt/my share",
|
||||
`/mnt/tab\011here`: "/mnt/tab\there",
|
||||
`/mnt/back\134slash`: `/mnt/back\slash`,
|
||||
`//nas/dci`: "//nas/dci",
|
||||
`/mnt/not\9octal`: `/mnt/not\9octal`,
|
||||
`/mnt/trailing\`: `/mnt/trailing\`,
|
||||
}
|
||||
|
||||
for in, want := range cases {
|
||||
if got := unescapeMountField(in); got != want {
|
||||
t.Errorf("unescapeMountField(%q) = %q, want %q", in, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestNormalizeMountSource(t *testing.T) {
|
||||
cases := map[string]string{
|
||||
"//NAS/DCI": "//nas/dci",
|
||||
"//nas/dci/": "//nas/dci",
|
||||
`\\nas\dci`: "//nas/dci",
|
||||
" //nas/dci ": "//nas/dci",
|
||||
"/": "/",
|
||||
}
|
||||
|
||||
for in, want := range cases {
|
||||
if got := normalizeMountSource(in); got != want {
|
||||
t.Errorf("normalizeMountSource(%q) = %q, want %q", in, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestReadMountTableSkipsMalformedLines(t *testing.T) {
|
||||
path := writeMountTable(t,
|
||||
"source-only",
|
||||
"source /point-but-no-type",
|
||||
"",
|
||||
"//nas/dci /mnt/dci cifs rw 0 0",
|
||||
)
|
||||
|
||||
entries, err := readMountTable(path)
|
||||
if err != nil {
|
||||
t.Fatalf("readMountTable failed: %v", err)
|
||||
}
|
||||
if len(entries) != 1 {
|
||||
t.Fatalf("expected 1 usable entry, got %d: %v", len(entries), entries)
|
||||
}
|
||||
if entries[0].Source != "//nas/dci" || entries[0].Point != "/mnt/dci" || entries[0].FSType != "cifs" {
|
||||
t.Errorf("unexpected entry: %+v", entries[0])
|
||||
}
|
||||
}
|
||||
|
||||
func TestReadMountTableMissingFile(t *testing.T) {
|
||||
if _, err := readMountTable("/definitely/not/a/mount/table"); err == nil {
|
||||
t.Error("expected an error for a missing mount table")
|
||||
}
|
||||
}
|
||||
|
||||
// TestReadRealMountTable closes the loop between the fixtures above and the
|
||||
// kernel's actual output: the parser must cope with whatever this host has
|
||||
// mounted, not just hand-written examples.
|
||||
func TestReadRealMountTable(t *testing.T) {
|
||||
if _, err := os.Stat(procMountsPath); err != nil {
|
||||
t.Skipf("%s is not available on this platform", procMountsPath)
|
||||
}
|
||||
|
||||
entries, err := readMountTable(procMountsPath)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to parse the host's %s: %v", procMountsPath, err)
|
||||
}
|
||||
if len(entries) == 0 {
|
||||
t.Fatalf("%s parsed to zero entries, which cannot be right", procMountsPath)
|
||||
}
|
||||
|
||||
foundRoot := false
|
||||
for _, e := range entries {
|
||||
if e.Point == "" || e.FSType == "" {
|
||||
t.Errorf("entry parsed with empty fields: %+v", e)
|
||||
}
|
||||
if e.Point == "/" {
|
||||
foundRoot = true
|
||||
}
|
||||
}
|
||||
if !foundRoot {
|
||||
t.Error("expected the root filesystem in the host's mount table")
|
||||
}
|
||||
}
|
||||
256
pkg/nas/nas.go
256
pkg/nas/nas.go
@ -6,31 +6,168 @@ import (
|
||||
"io"
|
||||
"log"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync"
|
||||
)
|
||||
|
||||
// NASService writes to a NAS destination without caring how this host reaches
|
||||
// it. Parsing the destination and attaching it are delegated to a Location and
|
||||
// a Connector; everything below operates on the resolved base path, which is
|
||||
// always an ordinary path this host's filesystem understands.
|
||||
type NASService struct {
|
||||
Config NASConfig
|
||||
|
||||
location Location
|
||||
connector Connector
|
||||
|
||||
mu sync.RWMutex
|
||||
attached Attachment
|
||||
connected bool
|
||||
}
|
||||
|
||||
func NewNASService(config NASConfig) *NASService {
|
||||
nt := &NASService{
|
||||
Config: config,
|
||||
}
|
||||
|
||||
// Establish network connection with credentials before accessing the path
|
||||
if err := nt.EstablishConnection(); err != nil {
|
||||
log.Fatalf("Failed to establish network connection to %s: %v", nt.Config.Path, err)
|
||||
}
|
||||
|
||||
err := nt.EnsureDirectoryExists(nt.Config.Path)
|
||||
// New parses the configured destination and selects a connection strategy for
|
||||
// the running host. It performs no I/O and does not touch the network, so it is
|
||||
// safe to call before deciding whether a transfer will actually happen; call
|
||||
// Connect to attach the destination.
|
||||
func New(config NASConfig) (*NASService, error) {
|
||||
loc, err := ParseLocation(config.Path)
|
||||
if err != nil {
|
||||
log.Fatalf("Failed to create directory %s: %v", nt.Config.Path, err)
|
||||
return nil, err
|
||||
}
|
||||
return nt
|
||||
|
||||
return &NASService{
|
||||
Config: config,
|
||||
location: loc,
|
||||
connector: defaultConnector(loc),
|
||||
}, nil
|
||||
}
|
||||
|
||||
// Connect attaches the destination, ensures the target directory exists, and
|
||||
// proves it is writable. The write probe matters: on both platforms a share can
|
||||
// be reachable and still reject writes, and finding that out at startup beats
|
||||
// discovering it mid-recording.
|
||||
func (nt *NASService) Connect(ctx context.Context) error {
|
||||
att, err := nt.connector.Attach(ctx, nt.location, nt.Config)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
nt.mu.Lock()
|
||||
nt.attached = att
|
||||
nt.mu.Unlock()
|
||||
|
||||
// From here on the attachment exists, so any failure has to undo it —
|
||||
// otherwise a mount this process created would outlive the failed startup
|
||||
// and be left behind on the host.
|
||||
if err := nt.EnsureDirectoryExists(att.BasePath); err != nil {
|
||||
nt.abandon(ctx, att)
|
||||
return fmt.Errorf("%s is reachable but %s is unusable: %w", nt.location, att.BasePath, err)
|
||||
}
|
||||
|
||||
if err := nt.writeProbe(att.BasePath); err != nil {
|
||||
nt.abandon(ctx, att)
|
||||
return err
|
||||
}
|
||||
|
||||
nt.mu.Lock()
|
||||
nt.connected = true
|
||||
nt.mu.Unlock()
|
||||
|
||||
log.Printf("Connected to NAS at %s", att.BasePath)
|
||||
return nil
|
||||
}
|
||||
|
||||
// abandon releases a half-finished attachment after Connect fails. The
|
||||
// original error is the one worth reporting, so a teardown failure is only
|
||||
// logged.
|
||||
func (nt *NASService) abandon(ctx context.Context, att Attachment) {
|
||||
// Keep the notes: they explain how far the host got, which is exactly what
|
||||
// the caller needs to report. Everything else is cleared so the service
|
||||
// cannot be mistaken for usable and Detach cannot run twice.
|
||||
nt.mu.Lock()
|
||||
nt.attached = Attachment{Notes: att.Notes}
|
||||
nt.connected = false
|
||||
nt.mu.Unlock()
|
||||
|
||||
if err := nt.connector.Detach(ctx, att); err != nil {
|
||||
log.Printf("Warning: failed to release NAS attachment after a failed connect: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// writeProbe creates and removes a file to confirm the destination accepts
|
||||
// writes.
|
||||
func (nt *NASService) writeProbe(basePath string) error {
|
||||
testFile := joinBase(basePath, ".connection_test")
|
||||
|
||||
f, err := os.Create(testFile)
|
||||
if err != nil {
|
||||
return fmt.Errorf("%s is not writable: %w", basePath, err)
|
||||
}
|
||||
f.Close()
|
||||
|
||||
if err := os.Remove(testFile); err != nil {
|
||||
log.Printf("Warning: failed to remove connection test file %s: %v", testFile, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Disconnect releases only what Connect established. A share that was already
|
||||
// mounted or already connected when the recorder started is left exactly as it
|
||||
// was found.
|
||||
func (nt *NASService) Disconnect(ctx context.Context) error {
|
||||
nt.mu.Lock()
|
||||
att := nt.attached
|
||||
nt.attached = Attachment{}
|
||||
nt.connected = false
|
||||
nt.mu.Unlock()
|
||||
|
||||
if err := nt.connector.Detach(ctx, att); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if att.Owned {
|
||||
log.Printf("Released NAS connection to %s", nt.location)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// LocalPath is the resolved base directory on this host, valid after Connect.
|
||||
func (nt *NASService) LocalPath() string {
|
||||
nt.mu.RLock()
|
||||
defer nt.mu.RUnlock()
|
||||
return nt.attached.BasePath
|
||||
}
|
||||
|
||||
// Join resolves elements against the NAS base path using whatever separator
|
||||
// this host's base path requires.
|
||||
func (nt *NASService) Join(elem ...string) string {
|
||||
return joinBase(nt.LocalPath(), elem...)
|
||||
}
|
||||
|
||||
// Location returns the parsed destination.
|
||||
func (nt *NASService) Location() Location { return nt.location }
|
||||
|
||||
// ConnectionInfo describes how this host reaches the destination, for check
|
||||
// output and startup logs. Safe to call before Connect, when it reports only
|
||||
// what is known from configuration.
|
||||
func (nt *NASService) ConnectionInfo() []string {
|
||||
nt.mu.RLock()
|
||||
defer nt.mu.RUnlock()
|
||||
|
||||
info := []string{
|
||||
"configured: " + nt.location.String(),
|
||||
"strategy: " + nt.connector.Name(),
|
||||
}
|
||||
info = append(info, nt.attached.Notes...)
|
||||
if nt.attached.BasePath != "" && nt.attached.BasePath != nt.location.Raw {
|
||||
info = append(info, "resolved path: "+nt.attached.BasePath)
|
||||
}
|
||||
return info
|
||||
}
|
||||
|
||||
func (nt *NASService) IsConnected() bool {
|
||||
nt.mu.RLock()
|
||||
defer nt.mu.RUnlock()
|
||||
return nt.connected
|
||||
}
|
||||
|
||||
func (nt *NASService) CopyFile(ctx context.Context, srcPath, destPath string) error {
|
||||
@ -89,87 +226,10 @@ func (nt *NASService) EnsureDirectoryExists(path string) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
func (nt *NASService) EstablishConnection() error {
|
||||
networkPath := nt.ExtractNetworkPath(nt.Config.Path)
|
||||
if networkPath == "" {
|
||||
return nil // local path, no network mount needed
|
||||
}
|
||||
|
||||
log.Printf("Establishing network connection to %s with user %s", networkPath, nt.Config.Username)
|
||||
|
||||
var cmd *exec.Cmd
|
||||
if nt.Config.Username != "" && nt.Config.Password != "" {
|
||||
cmd = exec.Command("net", "use", networkPath, "/user:"+nt.Config.Username, nt.Config.Password, "/persistent:no")
|
||||
} else {
|
||||
cmd = exec.Command("net", "use", networkPath, "/persistent:no")
|
||||
}
|
||||
|
||||
output, err := cmd.CombinedOutput()
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to establish network connection: %w\nOutput: %s", err, string(output))
|
||||
}
|
||||
|
||||
log.Printf("Network connection established successfully")
|
||||
return nil
|
||||
}
|
||||
|
||||
func (nt *NASService) ExtractNetworkPath(fullPath string) string {
|
||||
// Extract \\server\share from paths like \\server\share\folder\subfolder
|
||||
if !strings.HasPrefix(fullPath, "\\\\") {
|
||||
return "" // Not a UNC path
|
||||
}
|
||||
|
||||
parts := strings.Split(fullPath[2:], "\\") // Remove leading \\
|
||||
if len(parts) < 2 {
|
||||
return "" // Invalid UNC path
|
||||
}
|
||||
|
||||
return "\\\\" + parts[0] + "\\" + parts[1]
|
||||
}
|
||||
|
||||
func (nt *NASService) TestConnection() error {
|
||||
testFile := filepath.Join(nt.Config.Path, ".connection_test")
|
||||
|
||||
f, err := os.Create(testFile)
|
||||
if err != nil {
|
||||
return fmt.Errorf("Failed to create test file: %w", err)
|
||||
}
|
||||
f.Close()
|
||||
|
||||
os.Remove(testFile)
|
||||
|
||||
nt.connected = true
|
||||
log.Printf("Connected to NAS at %s", nt.Config.Path)
|
||||
return nil
|
||||
}
|
||||
|
||||
func (nt *NASService) IsConnected() bool {
|
||||
return nt.connected
|
||||
}
|
||||
|
||||
// Disconnect removes the network connection
|
||||
func (nt *NASService) Disconnect() error {
|
||||
networkPath := nt.ExtractNetworkPath(nt.Config.Path)
|
||||
if networkPath == "" {
|
||||
return nil // Local path, nothing to disconnect
|
||||
}
|
||||
|
||||
cmd := exec.Command("net", "use", networkPath, "/delete")
|
||||
output, err := cmd.CombinedOutput()
|
||||
if err != nil {
|
||||
log.Printf("Warning: failed to disconnect from %s: %v\nOutput: %s", networkPath, err, string(output))
|
||||
// Don't return error since this is cleanup
|
||||
} else {
|
||||
log.Printf("Disconnected from network path: %s", networkPath)
|
||||
}
|
||||
|
||||
nt.connected = false
|
||||
return nil
|
||||
}
|
||||
|
||||
// FileExists checks if a file already exists on the NAS and optionally verifies size
|
||||
// FileExists checks whether a file already exists on the NAS, optionally
|
||||
// verifying its size. destinationPath is relative to the NAS base path.
|
||||
func (nt *NASService) FileExists(destinationPath string, expectedSize int64) (bool, error) {
|
||||
fullDestPath := filepath.Join(nt.Config.Path, destinationPath)
|
||||
fullDestPath := nt.Join(destinationPath)
|
||||
|
||||
destInfo, err := os.Stat(fullDestPath)
|
||||
if err != nil {
|
||||
@ -189,9 +249,9 @@ func (nt *NASService) FileExists(destinationPath string, expectedSize int64) (bo
|
||||
return true, nil
|
||||
}
|
||||
|
||||
// GetFileSize returns the size of a file on the NAS
|
||||
// GetFileSize returns the size of a file on the NAS, relative to the base path.
|
||||
func (nt *NASService) GetFileSize(destinationPath string) (int64, error) {
|
||||
fullDestPath := filepath.Join(nt.Config.Path, destinationPath)
|
||||
fullDestPath := nt.Join(destinationPath)
|
||||
|
||||
destInfo, err := os.Stat(fullDestPath)
|
||||
if err != nil {
|
||||
|
||||
308
pkg/nas/nas_test.go
Normal file
308
pkg/nas/nas_test.go
Normal file
@ -0,0 +1,308 @@
|
||||
package nas
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// fakeConnector stands in for a platform strategy so the service's own
|
||||
// behaviour — probing, path resolution, teardown — can be tested without a NAS.
|
||||
type fakeConnector struct {
|
||||
basePath string
|
||||
attachErr error
|
||||
owned bool
|
||||
attachCount int
|
||||
detachCount int
|
||||
detached Attachment
|
||||
}
|
||||
|
||||
func (f *fakeConnector) Name() string { return "fake" }
|
||||
|
||||
func (f *fakeConnector) Attach(context.Context, Location, NASConfig) (Attachment, error) {
|
||||
f.attachCount++
|
||||
if f.attachErr != nil {
|
||||
return Attachment{}, f.attachErr
|
||||
}
|
||||
return Attachment{BasePath: f.basePath, Owned: f.owned, Notes: []string{"fake attachment"}}, nil
|
||||
}
|
||||
|
||||
func (f *fakeConnector) Detach(_ context.Context, att Attachment) error {
|
||||
f.detachCount++
|
||||
f.detached = att
|
||||
return nil
|
||||
}
|
||||
|
||||
func newTestService(t *testing.T, configuredPath string, conn Connector) *NASService {
|
||||
t.Helper()
|
||||
svc, err := New(NASConfig{Path: configuredPath})
|
||||
if err != nil {
|
||||
t.Fatalf("New(%q) failed: %v", configuredPath, err)
|
||||
}
|
||||
svc.connector = conn
|
||||
return svc
|
||||
}
|
||||
|
||||
func TestNewRejectsUnusableDestination(t *testing.T) {
|
||||
if _, err := New(NASConfig{Path: ""}); err == nil {
|
||||
t.Error("an empty NAS path should be rejected")
|
||||
}
|
||||
if _, err := New(NASConfig{Path: `\\server`}); err == nil {
|
||||
t.Error("a server without a share should be rejected")
|
||||
}
|
||||
}
|
||||
|
||||
func TestNewDoesNoIO(t *testing.T) {
|
||||
// Constructing the service must not touch the network or filesystem, so
|
||||
// that modes which never transfer can build one harmlessly.
|
||||
svc, err := New(NASConfig{Path: `\\nas\dci\streams`})
|
||||
if err != nil {
|
||||
t.Fatalf("New failed: %v", err)
|
||||
}
|
||||
if svc.IsConnected() {
|
||||
t.Error("service should not report connected before Connect")
|
||||
}
|
||||
if svc.LocalPath() != "" {
|
||||
t.Errorf("no path should be resolved before Connect, got %q", svc.LocalPath())
|
||||
}
|
||||
}
|
||||
|
||||
func TestConnectResolvesAndProbes(t *testing.T) {
|
||||
base := filepath.Join(t.TempDir(), "streams")
|
||||
conn := &fakeConnector{basePath: base}
|
||||
svc := newTestService(t, `\\nas\dci\streams`, conn)
|
||||
|
||||
if err := svc.Connect(context.Background()); err != nil {
|
||||
t.Fatalf("Connect failed: %v", err)
|
||||
}
|
||||
|
||||
if !svc.IsConnected() {
|
||||
t.Error("service should report connected")
|
||||
}
|
||||
if svc.LocalPath() != base {
|
||||
t.Errorf("LocalPath = %q, want %q", svc.LocalPath(), base)
|
||||
}
|
||||
// Connect creates the destination directory if it is missing.
|
||||
if info, err := os.Stat(base); err != nil || !info.IsDir() {
|
||||
t.Errorf("Connect should have created %s: %v", base, err)
|
||||
}
|
||||
// The write probe must clean up after itself.
|
||||
if _, err := os.Stat(filepath.Join(base, ".connection_test")); !os.IsNotExist(err) {
|
||||
t.Errorf("probe file should have been removed, stat err = %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestConnectSurfacesAttachFailure(t *testing.T) {
|
||||
conn := &fakeConnector{attachErr: fmt.Errorf("share is not mounted")}
|
||||
svc := newTestService(t, `\\nas\dci`, conn)
|
||||
|
||||
err := svc.Connect(context.Background())
|
||||
if err == nil || !strings.Contains(err.Error(), "not mounted") {
|
||||
t.Fatalf("expected the connector's error to surface, got: %v", err)
|
||||
}
|
||||
if svc.IsConnected() {
|
||||
t.Error("a failed attach must not leave the service marked connected")
|
||||
}
|
||||
}
|
||||
|
||||
func TestConnectFailsWhenDestinationUnwritable(t *testing.T) {
|
||||
// A file where the directory should be: reachable, but unusable.
|
||||
blocker := filepath.Join(t.TempDir(), "blocker")
|
||||
if err := os.WriteFile(blocker, []byte("x"), 0644); err != nil {
|
||||
t.Fatalf("setup failed: %v", err)
|
||||
}
|
||||
|
||||
svc := newTestService(t, "/tmp/whatever", &fakeConnector{basePath: filepath.Join(blocker, "streams")})
|
||||
if err := svc.Connect(context.Background()); err == nil {
|
||||
t.Fatal("expected an unusable destination to fail the connect")
|
||||
}
|
||||
if svc.IsConnected() {
|
||||
t.Error("service should not report connected after a failed probe")
|
||||
}
|
||||
}
|
||||
|
||||
func TestFailedProbeReleasesTheAttachment(t *testing.T) {
|
||||
// A mount created by Attach must not outlive a failed Connect, or a failed
|
||||
// startup leaves a mount behind on the host.
|
||||
blocker := filepath.Join(t.TempDir(), "blocker")
|
||||
if err := os.WriteFile(blocker, []byte("x"), 0644); err != nil {
|
||||
t.Fatalf("setup failed: %v", err)
|
||||
}
|
||||
|
||||
conn := &fakeConnector{basePath: filepath.Join(blocker, "streams"), owned: true}
|
||||
svc := newTestService(t, `\\nas\dci`, conn)
|
||||
|
||||
if err := svc.Connect(context.Background()); err == nil {
|
||||
t.Fatal("expected Connect to fail")
|
||||
}
|
||||
|
||||
if conn.detachCount != 1 {
|
||||
t.Errorf("Detach called %d times after a failed connect, want 1", conn.detachCount)
|
||||
}
|
||||
if !conn.detached.Owned {
|
||||
t.Error("Detach should have been told the attachment was owned")
|
||||
}
|
||||
if svc.LocalPath() != "" {
|
||||
t.Errorf("a released attachment should expose no path, got %q", svc.LocalPath())
|
||||
}
|
||||
// The diagnosis of how far it got must survive for the caller to report.
|
||||
if info := strings.Join(svc.ConnectionInfo(), "\n"); !strings.Contains(info, "fake attachment") {
|
||||
t.Errorf("connection notes should survive a failed connect:\n%s", info)
|
||||
}
|
||||
|
||||
// A later Disconnect must not detach a second time.
|
||||
if err := svc.Disconnect(context.Background()); err != nil {
|
||||
t.Fatalf("Disconnect failed: %v", err)
|
||||
}
|
||||
if conn.detachCount != 2 || conn.detached.Owned {
|
||||
t.Errorf("Disconnect after release should be a no-op teardown, detachCount=%d owned=%v",
|
||||
conn.detachCount, conn.detached.Owned)
|
||||
}
|
||||
}
|
||||
|
||||
func TestJoinUsesResolvedPath(t *testing.T) {
|
||||
base := t.TempDir()
|
||||
svc := newTestService(t, `\\nas\dci\streams`, &fakeConnector{basePath: base})
|
||||
if err := svc.Connect(context.Background()); err != nil {
|
||||
t.Fatalf("Connect failed: %v", err)
|
||||
}
|
||||
|
||||
// Crucially the join is against the resolved path, not the configured UNC
|
||||
// path — that is the whole point of the indirection.
|
||||
if got, want := svc.Join("event", "1080p"), filepath.Join(base, "event", "1080p"); got != want {
|
||||
t.Errorf("Join = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestFileExistsAndGetFileSizeUseResolvedPath(t *testing.T) {
|
||||
base := t.TempDir()
|
||||
svc := newTestService(t, `\\nas\dci`, &fakeConnector{basePath: base})
|
||||
if err := svc.Connect(context.Background()); err != nil {
|
||||
t.Fatalf("Connect failed: %v", err)
|
||||
}
|
||||
|
||||
rel := filepath.Join("event", "seg.ts")
|
||||
if err := os.MkdirAll(filepath.Join(base, "event"), 0755); err != nil {
|
||||
t.Fatalf("setup failed: %v", err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(base, rel), []byte("0123456789"), 0644); err != nil {
|
||||
t.Fatalf("setup failed: %v", err)
|
||||
}
|
||||
|
||||
exists, err := svc.FileExists(rel, 10)
|
||||
if err != nil || !exists {
|
||||
t.Errorf("FileExists = %v, %v; want true, nil", exists, err)
|
||||
}
|
||||
|
||||
// A size mismatch means a partial transfer, which should not count.
|
||||
exists, err = svc.FileExists(rel, 99)
|
||||
if err != nil || exists {
|
||||
t.Errorf("FileExists with wrong size = %v, %v; want false, nil", exists, err)
|
||||
}
|
||||
|
||||
exists, err = svc.FileExists(filepath.Join("event", "missing.ts"), 0)
|
||||
if err != nil || exists {
|
||||
t.Errorf("FileExists for missing file = %v, %v; want false, nil", exists, err)
|
||||
}
|
||||
|
||||
size, err := svc.GetFileSize(rel)
|
||||
if err != nil || size != 10 {
|
||||
t.Errorf("GetFileSize = %d, %v; want 10, nil", size, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDisconnectDelegatesAndResets(t *testing.T) {
|
||||
conn := &fakeConnector{basePath: t.TempDir(), owned: true}
|
||||
svc := newTestService(t, `\\nas\dci`, conn)
|
||||
if err := svc.Connect(context.Background()); err != nil {
|
||||
t.Fatalf("Connect failed: %v", err)
|
||||
}
|
||||
|
||||
if err := svc.Disconnect(context.Background()); err != nil {
|
||||
t.Fatalf("Disconnect failed: %v", err)
|
||||
}
|
||||
|
||||
if conn.detachCount != 1 {
|
||||
t.Errorf("Detach called %d times, want 1", conn.detachCount)
|
||||
}
|
||||
if !conn.detached.Owned {
|
||||
t.Error("Detach should receive the attachment so it knows what it owns")
|
||||
}
|
||||
if svc.IsConnected() {
|
||||
t.Error("service should not report connected after Disconnect")
|
||||
}
|
||||
}
|
||||
|
||||
func TestConnectionInfoDescribesResolution(t *testing.T) {
|
||||
base := t.TempDir()
|
||||
svc := newTestService(t, `\\nas\dci\streams`, &fakeConnector{basePath: base})
|
||||
|
||||
// Before connecting, only configuration is known.
|
||||
before := strings.Join(svc.ConnectionInfo(), "\n")
|
||||
if !strings.Contains(before, `\\nas\dci\streams`) || !strings.Contains(before, "fake") {
|
||||
t.Errorf("pre-connect info should name the destination and strategy:\n%s", before)
|
||||
}
|
||||
|
||||
if err := svc.Connect(context.Background()); err != nil {
|
||||
t.Fatalf("Connect failed: %v", err)
|
||||
}
|
||||
|
||||
after := strings.Join(svc.ConnectionInfo(), "\n")
|
||||
if !strings.Contains(after, "resolved path: "+base) {
|
||||
t.Errorf("post-connect info should report where the destination resolved to:\n%s", after)
|
||||
}
|
||||
}
|
||||
|
||||
func TestVerifyTransferDetectsSizeMismatch(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
src := filepath.Join(dir, "src.ts")
|
||||
dst := filepath.Join(dir, "dst.ts")
|
||||
if err := os.WriteFile(src, []byte("0123456789"), 0644); err != nil {
|
||||
t.Fatalf("setup failed: %v", err)
|
||||
}
|
||||
if err := os.WriteFile(dst, []byte("012"), 0644); err != nil {
|
||||
t.Fatalf("setup failed: %v", err)
|
||||
}
|
||||
|
||||
svc := newTestService(t, dir, &fakeConnector{basePath: dir})
|
||||
if err := svc.VerifyTransfer(src, dst); err == nil {
|
||||
t.Error("a truncated destination should fail verification")
|
||||
}
|
||||
|
||||
if err := os.WriteFile(dst, []byte("0123456789"), 0644); err != nil {
|
||||
t.Fatalf("setup failed: %v", err)
|
||||
}
|
||||
if err := svc.VerifyTransfer(src, dst); err != nil {
|
||||
t.Errorf("matching sizes should verify: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCopyFile(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
src := filepath.Join(dir, "src.ts")
|
||||
dst := filepath.Join(dir, "dst.ts")
|
||||
payload := []byte("segment payload")
|
||||
if err := os.WriteFile(src, payload, 0644); err != nil {
|
||||
t.Fatalf("setup failed: %v", err)
|
||||
}
|
||||
|
||||
svc := newTestService(t, dir, &fakeConnector{basePath: dir})
|
||||
if err := svc.CopyFile(context.Background(), src, dst); err != nil {
|
||||
t.Fatalf("CopyFile failed: %v", err)
|
||||
}
|
||||
|
||||
got, err := os.ReadFile(dst)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to read copy: %v", err)
|
||||
}
|
||||
if string(got) != string(payload) {
|
||||
t.Errorf("copied contents = %q, want %q", got, payload)
|
||||
}
|
||||
|
||||
if err := svc.CopyFile(context.Background(), filepath.Join(dir, "missing.ts"), dst); err == nil {
|
||||
t.Error("copying a missing source should fail")
|
||||
}
|
||||
}
|
||||
@ -39,11 +39,19 @@ func NewProcessingService(eventName string, cfg *config.Config) (*ProcessingServ
|
||||
Timeout: cfg.NAS.Timeout,
|
||||
RetryLimit: cfg.NAS.RetryLimit,
|
||||
VerifySize: true,
|
||||
AutoMount: cfg.NAS.AutoMount,
|
||||
MountPoint: cfg.NAS.MountPoint,
|
||||
}
|
||||
|
||||
nasService := nas.NewNASService(nasConfig)
|
||||
nasService, err := nas.New(nasConfig)
|
||||
if err != nil {
|
||||
logging.MustGetFactory().Processing.Error("invalid NAS destination", logging.KeyError, err.Error())
|
||||
return nil, fmt.Errorf("invalid NAS destination: %w", err)
|
||||
}
|
||||
|
||||
if err := nasService.TestConnection(); err != nil {
|
||||
connectCtx, cancel := context.WithTimeout(context.Background(), cfg.NAS.Timeout)
|
||||
defer cancel()
|
||||
if err := nasService.Connect(connectCtx); err != nil {
|
||||
logging.MustGetFactory().Processing.Error("failed to connect to NAS", logging.KeyError, err.Error())
|
||||
return nil, fmt.Errorf("failed to connect to NAS: %w", err)
|
||||
}
|
||||
@ -59,9 +67,17 @@ func (ps *ProcessingService) GetSegmentInfo() (map[int]string, error) {
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
// nasEventPath is the event's directory on the NAS as reached from this host.
|
||||
// It goes through the NAS service rather than config.GetNASEventPath because
|
||||
// the configured path may be a UNC path that this host reaches via a mount
|
||||
// point instead.
|
||||
func (ps *ProcessingService) nasEventPath() string {
|
||||
return ps.nas.Join(ps.eventName)
|
||||
}
|
||||
|
||||
func (ps *ProcessingService) GetEventDirs() ([]string, error) {
|
||||
if ps.eventName == "" {
|
||||
sourcePath := ps.config.NAS.OutputPath
|
||||
sourcePath := ps.nas.LocalPath()
|
||||
dirs, err := os.ReadDir(sourcePath)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to read directory %s: %w", sourcePath, err)
|
||||
@ -177,7 +193,7 @@ func (ps *ProcessingService) Start(ctx context.Context) error {
|
||||
}
|
||||
|
||||
func (ps *ProcessingService) GetResolutions() ([]string, error) {
|
||||
eventPath := ps.config.GetNASEventPath(ps.eventName)
|
||||
eventPath := ps.nasEventPath()
|
||||
dirs, err := os.ReadDir(eventPath)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to read source directory %s: %w", eventPath, err)
|
||||
@ -198,7 +214,7 @@ func (ps *ProcessingService) GetResolutions() ([]string, error) {
|
||||
func (ps *ProcessingService) ParseResolutionDirectory(resolution string, ch chan<- SegmentInfo, wg *sync.WaitGroup) {
|
||||
defer wg.Done()
|
||||
|
||||
resolutionPath := utils.SafeJoin(ps.config.GetNASEventPath(ps.eventName), resolution)
|
||||
resolutionPath := utils.SafeJoin(ps.nasEventPath(), resolution)
|
||||
files, err := os.ReadDir(resolutionPath)
|
||||
if err != nil {
|
||||
log.Printf("Failed to read resolution directory %s: %v", resolutionPath, err)
|
||||
@ -227,22 +243,14 @@ func (ps *ProcessingService) ParseResolutionDirectory(resolution string, ch chan
|
||||
func (ps *ProcessingService) AggregateSegmentInfo(ch <-chan SegmentInfo) (map[int]SegmentInfo, error) {
|
||||
segmentMap := make(map[int]SegmentInfo)
|
||||
|
||||
rank := map[string]int{
|
||||
"1080p": 1,
|
||||
"720p": 2,
|
||||
"540p": 3,
|
||||
"480p": 4,
|
||||
"450p": 5,
|
||||
"360p": 6,
|
||||
"270p": 7,
|
||||
"240p": 8,
|
||||
}
|
||||
|
||||
// Each sequence number arrives once per downloaded variant; keep the
|
||||
// highest-resolution copy of each, since that is the one worth stitching
|
||||
// into the final video.
|
||||
for segment := range ch {
|
||||
fmt.Printf("Received segment %s in resolution %s \n", segment.Name, segment.Resolution)
|
||||
logging.MustGetFactory().Processing.Debug("segment received", logging.KeySegment, segment.Name, logging.KeyResolution, segment.Resolution)
|
||||
current, exists := segmentMap[segment.SeqNo]
|
||||
if !exists || rank[segment.Resolution] > rank[current.Resolution] {
|
||||
if !exists || resolutionHeight(segment.Resolution) > resolutionHeight(current.Resolution) {
|
||||
segmentMap[segment.SeqNo] = segment
|
||||
}
|
||||
}
|
||||
@ -250,6 +258,24 @@ func (ps *ProcessingService) AggregateSegmentInfo(ch <-chan SegmentInfo) (map[in
|
||||
return segmentMap, nil
|
||||
}
|
||||
|
||||
// resolutionHeight converts a resolution label such as "1080p" into the pixel
|
||||
// height it stands for, so variants can be ranked by quality.
|
||||
//
|
||||
// The height is parsed rather than looked up in a table: labels come from the
|
||||
// playlist's own RESOLUTION attribute, so a stream can legitimately offer
|
||||
// 1440p or 2160p that no fixed table would anticipate. Anything not in
|
||||
// <height>p form — "unknown", from a playlist with no variants — returns 0 and
|
||||
// therefore loses to every recognised resolution.
|
||||
func resolutionHeight(resolution string) int {
|
||||
trimmed := strings.TrimSuffix(strings.ToLower(strings.TrimSpace(resolution)), "p")
|
||||
|
||||
height, err := strconv.Atoi(trimmed)
|
||||
if err != nil || height < 0 {
|
||||
return 0
|
||||
}
|
||||
return height
|
||||
}
|
||||
|
||||
func (ps *ProcessingService) WriteConcatFile(segmentMap map[int]SegmentInfo) (string, error) {
|
||||
concatPath := ps.config.GetProcessOutputPath(ps.eventName)
|
||||
|
||||
@ -273,7 +299,7 @@ func (ps *ProcessingService) WriteConcatFile(segmentMap map[int]SegmentInfo) (st
|
||||
|
||||
for _, seq := range keys {
|
||||
segment := segmentMap[seq]
|
||||
filePath := utils.SafeJoin(ps.config.GetNASEventPath(ps.eventName), segment.Resolution, segment.Name)
|
||||
filePath := utils.SafeJoin(ps.nasEventPath(), segment.Resolution, segment.Name)
|
||||
line := fmt.Sprintf("file '%s'\n", filePath)
|
||||
if _, err := f.WriteString(line); err != nil {
|
||||
return "", fmt.Errorf("failed to write to concat file: %w", err)
|
||||
|
||||
@ -1,7 +1,9 @@
|
||||
package processing
|
||||
|
||||
import (
|
||||
"context"
|
||||
"m3u8-downloader/pkg/config"
|
||||
"m3u8-downloader/pkg/nas"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"runtime"
|
||||
@ -10,6 +12,22 @@ import (
|
||||
"time"
|
||||
)
|
||||
|
||||
// connectedNAS attaches a NAS service to a local directory. Reaching the NAS
|
||||
// is platform-specific, but an already-mounted or plain directory resolves the
|
||||
// same way everywhere, which is what makes this usable as a test double.
|
||||
func connectedNAS(t *testing.T, path string) *nas.NASService {
|
||||
t.Helper()
|
||||
|
||||
svc, err := nas.New(nas.NASConfig{Path: path, Timeout: 5 * time.Second})
|
||||
if err != nil {
|
||||
t.Fatalf("Failed to create NAS service for %s: %v", path, err)
|
||||
}
|
||||
if err := svc.Connect(context.Background()); err != nil {
|
||||
t.Fatalf("Failed to connect NAS service to %s: %v", path, err)
|
||||
}
|
||||
return svc
|
||||
}
|
||||
|
||||
func createTestConfig(tempDir string) *config.Config {
|
||||
return &config.Config{
|
||||
Core: config.CoreConfig{
|
||||
@ -87,6 +105,7 @@ func TestProcessingService_GetEventDirs(t *testing.T) {
|
||||
ps := &ProcessingService{
|
||||
config: cfg,
|
||||
eventName: "", // Empty to test directory discovery
|
||||
nas: connectedNAS(t, cfg.NAS.OutputPath),
|
||||
}
|
||||
|
||||
dirs, err := ps.GetEventDirs()
|
||||
@ -114,12 +133,15 @@ func TestProcessingService_GetEventDirs(t *testing.T) {
|
||||
}
|
||||
|
||||
func TestProcessingService_GetEventDirs_WithEventName(t *testing.T) {
|
||||
cfg := createTestConfig("/tmp")
|
||||
// Scoped to a temp dir rather than /tmp directly: attaching the NAS
|
||||
// service creates the destination, and tests should not leave one behind.
|
||||
cfg := createTestConfig(t.TempDir())
|
||||
eventName := "specific-event"
|
||||
|
||||
ps := &ProcessingService{
|
||||
config: cfg,
|
||||
eventName: eventName,
|
||||
nas: connectedNAS(t, cfg.NAS.OutputPath),
|
||||
}
|
||||
|
||||
dirs, err := ps.GetEventDirs()
|
||||
@ -156,6 +178,7 @@ func TestProcessingService_GetResolutions(t *testing.T) {
|
||||
ps := &ProcessingService{
|
||||
config: cfg,
|
||||
eventName: eventName,
|
||||
nas: connectedNAS(t, cfg.NAS.OutputPath),
|
||||
}
|
||||
|
||||
resolutions, err := ps.GetResolutions()
|
||||
@ -248,6 +271,7 @@ func TestProcessingService_WriteConcatFile(t *testing.T) {
|
||||
ps := &ProcessingService{
|
||||
config: cfg,
|
||||
eventName: eventName,
|
||||
nas: connectedNAS(t, cfg.NAS.OutputPath),
|
||||
}
|
||||
|
||||
// Create test segment map
|
||||
@ -382,3 +406,67 @@ func TestProcessJob_Structure(t *testing.T) {
|
||||
t.Errorf("Expected EventName='test-event', got '%s'", job.EventName)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolutionHeight(t *testing.T) {
|
||||
cases := map[string]int{
|
||||
"1080p": 1080,
|
||||
"720p": 720,
|
||||
"2160p": 2160, // beyond anything a fixed quality table would list
|
||||
"1440p": 1440,
|
||||
"240p": 240,
|
||||
// Not in <height>p form: must rank below every real resolution rather
|
||||
// than accidentally outranking them.
|
||||
"unknown": 0,
|
||||
"": 0,
|
||||
"p": 0,
|
||||
"-720p": 0,
|
||||
}
|
||||
|
||||
for label, want := range cases {
|
||||
if got := resolutionHeight(label); got != want {
|
||||
t.Errorf("resolutionHeight(%q) = %d, want %d", label, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestAggregateSegmentInfoPrefersHighestResolution covers the variants a
|
||||
// hardcoded rank table could not express: resolutions above 1080p, and the
|
||||
// "unknown" label produced for a playlist with no variants.
|
||||
func TestAggregateSegmentInfoPrefersHighestResolution(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
arriving []string
|
||||
want string
|
||||
}{
|
||||
{"1080p beats 720p regardless of order", []string{"720p", "1080p"}, "1080p"},
|
||||
{"1080p beats 720p when it arrives first", []string{"1080p", "720p"}, "1080p"},
|
||||
{"4k beats 1080p", []string{"1080p", "2160p"}, "2160p"},
|
||||
{"1440p beats 1080p", []string{"1080p", "1440p"}, "1440p"},
|
||||
{"a known resolution beats unknown", []string{"unknown", "480p"}, "480p"},
|
||||
{"unknown does not displace a known resolution", []string{"1080p", "unknown"}, "1080p"},
|
||||
{"unknown alone is still kept", []string{"unknown"}, "unknown"},
|
||||
{"lowest only", []string{"240p"}, "240p"},
|
||||
}
|
||||
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
ps := &ProcessingService{}
|
||||
ch := make(chan SegmentInfo, len(tc.arriving))
|
||||
for _, res := range tc.arriving {
|
||||
ch <- SegmentInfo{Name: "seg_1001.ts", SeqNo: 1001, Resolution: res}
|
||||
}
|
||||
close(ch)
|
||||
|
||||
segments, err := ps.AggregateSegmentInfo(ch)
|
||||
if err != nil {
|
||||
t.Fatalf("AggregateSegmentInfo() failed: %v", err)
|
||||
}
|
||||
if len(segments) != 1 {
|
||||
t.Fatalf("expected a single sequence number, got %d", len(segments))
|
||||
}
|
||||
if got := segments[1001].Resolution; got != tc.want {
|
||||
t.Errorf("kept %q, want %q (arrival order: %v)", got, tc.want, tc.arriving)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
@ -2,6 +2,7 @@ package transfer
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"log"
|
||||
"m3u8-downloader/pkg/logging"
|
||||
@ -10,6 +11,11 @@ import (
|
||||
"time"
|
||||
)
|
||||
|
||||
// errRetained reports that a file is still inside its retention window. It is
|
||||
// not a failure: the file simply has to be reconsidered on a later pass rather
|
||||
// than being discarded from the pending list.
|
||||
var errRetained = errors.New("file is within the retention period")
|
||||
|
||||
type CleanupService struct {
|
||||
config CleanupConfig
|
||||
pendingFiles []string
|
||||
@ -82,29 +88,43 @@ func (cs *CleanupService) ExecuteCleanup(ctx context.Context) error {
|
||||
log.Printf("Processing %d files for cleanup", len(batch))
|
||||
|
||||
var cleanedCount int
|
||||
var errors []error
|
||||
var failures []error
|
||||
// requeue collects files this pass must not drop: ones still inside their
|
||||
// retention window, and — if the context is cancelled mid-batch — the
|
||||
// remainder that was never looked at. They were already removed from
|
||||
// pendingFiles above, so without this they would never be cleaned at all.
|
||||
var requeue []string
|
||||
|
||||
for _, filePath := range batch {
|
||||
for i, filePath := range batch {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
requeue = append(requeue, batch[i:]...)
|
||||
cs.requeue(requeue)
|
||||
return ctx.Err()
|
||||
default:
|
||||
}
|
||||
|
||||
if err := cs.cleanupFile(filePath); err != nil {
|
||||
errors = append(errors, fmt.Errorf("Failed to cleanup file %s: %w", filePath, err))
|
||||
} else {
|
||||
err := cs.cleanupFile(filePath)
|
||||
switch {
|
||||
case errors.Is(err, errRetained):
|
||||
requeue = append(requeue, filePath)
|
||||
case err != nil:
|
||||
failures = append(failures, fmt.Errorf("Failed to cleanup file %s: %w", filePath, err))
|
||||
default:
|
||||
cleanedCount++
|
||||
}
|
||||
}
|
||||
|
||||
log.Printf("Cleanup batch completed (cleaned: %d, errors: %d)", cleanedCount, len(errors))
|
||||
logging.MustGetFactory().Transfer.Info("cleanup batch completed", "cleaned", cleanedCount, "errors", len(errors))
|
||||
cs.requeue(requeue)
|
||||
|
||||
if len(errors) > 0 {
|
||||
for i, err := range errors {
|
||||
log.Printf("Cleanup batch completed (cleaned: %d, retained: %d, errors: %d)", cleanedCount, len(requeue), len(failures))
|
||||
logging.MustGetFactory().Transfer.Info("cleanup batch completed",
|
||||
"cleaned", cleanedCount, "retained", len(requeue), "errors", len(failures))
|
||||
|
||||
if len(failures) > 0 {
|
||||
for i, err := range failures {
|
||||
if i >= 3 {
|
||||
log.Printf("... and %d more errors", len(errors)-3)
|
||||
log.Printf("... and %d more errors", len(failures)-3)
|
||||
break
|
||||
}
|
||||
log.Printf("Error: %v", err)
|
||||
@ -113,7 +133,17 @@ func (cs *CleanupService) ExecuteCleanup(ctx context.Context) error {
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// requeue returns files to the pending list for a later pass.
|
||||
func (cs *CleanupService) requeue(files []string) {
|
||||
if len(files) == 0 {
|
||||
return
|
||||
}
|
||||
|
||||
cs.mu.Lock()
|
||||
defer cs.mu.Unlock()
|
||||
cs.pendingFiles = append(cs.pendingFiles, files...)
|
||||
}
|
||||
|
||||
func (cs *CleanupService) cleanupFile(filePath string) error {
|
||||
@ -128,7 +158,7 @@ func (cs *CleanupService) cleanupFile(filePath string) error {
|
||||
if cs.config.RetentionPeriod > 0 {
|
||||
if time.Since(info.ModTime()) < cs.config.RetentionPeriod {
|
||||
log.Printf("File too new to cleanup: %s", filePath)
|
||||
return nil
|
||||
return errRetained
|
||||
}
|
||||
}
|
||||
|
||||
@ -162,6 +192,18 @@ func (cs *CleanupService) ForceCleanupAll(ctx context.Context) error {
|
||||
return err
|
||||
}
|
||||
|
||||
// Stop when a pass makes no progress. Files still inside their
|
||||
// retention window are requeued by design, so requiring the list to
|
||||
// reach zero would spin here until the shutdown timeout expired.
|
||||
cs.mu.Lock()
|
||||
remaining := len(cs.pendingFiles)
|
||||
cs.mu.Unlock()
|
||||
|
||||
if remaining >= pendingCount {
|
||||
log.Printf("Force cleanup stopping with %d file(s) still pending (within retention period)", remaining)
|
||||
break
|
||||
}
|
||||
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return ctx.Err()
|
||||
|
||||
190
pkg/transfer/cleanup_test.go
Normal file
190
pkg/transfer/cleanup_test.go
Normal file
@ -0,0 +1,190 @@
|
||||
package transfer
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func writeAgedFile(t *testing.T, dir, name string, age time.Duration) string {
|
||||
t.Helper()
|
||||
|
||||
path := filepath.Join(dir, name)
|
||||
if err := os.WriteFile(path, []byte("segment"), 0644); err != nil {
|
||||
t.Fatalf("failed to write %s: %v", path, err)
|
||||
}
|
||||
|
||||
modTime := time.Now().Add(-age)
|
||||
if err := os.Chtimes(path, modTime, modTime); err != nil {
|
||||
t.Fatalf("failed to age %s: %v", path, err)
|
||||
}
|
||||
return path
|
||||
}
|
||||
|
||||
func TestCleanupRemovesTransferredFiles(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
cs := NewCleanupService(CleanupConfig{Enabled: true, BatchSize: 10, CheckInterval: time.Second})
|
||||
|
||||
path := writeAgedFile(t, dir, "seg_1001.ts", 0)
|
||||
if err := cs.ScheduleCleanup(path); err != nil {
|
||||
t.Fatalf("ScheduleCleanup failed: %v", err)
|
||||
}
|
||||
|
||||
if err := cs.ExecuteCleanup(context.Background()); err != nil {
|
||||
t.Fatalf("ExecuteCleanup failed: %v", err)
|
||||
}
|
||||
|
||||
if _, err := os.Stat(path); !os.IsNotExist(err) {
|
||||
t.Errorf("file should have been removed, stat err = %v", err)
|
||||
}
|
||||
if got := cs.GetPendingCount(); got != 0 {
|
||||
t.Errorf("pending count = %d, want 0", got)
|
||||
}
|
||||
}
|
||||
|
||||
// TestCleanupRetainsThenRemoves is the regression test for a file inside its
|
||||
// retention window being dropped from the pending list: it was reported as
|
||||
// cleaned, never deleted, and never reconsidered, so local files accumulated
|
||||
// forever whenever RetainHours was set.
|
||||
func TestCleanupRetainsThenRemoves(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
cs := NewCleanupService(CleanupConfig{
|
||||
Enabled: true,
|
||||
RetentionPeriod: time.Hour,
|
||||
BatchSize: 10,
|
||||
CheckInterval: time.Second,
|
||||
})
|
||||
|
||||
fresh := writeAgedFile(t, dir, "fresh.ts", time.Minute)
|
||||
old := writeAgedFile(t, dir, "old.ts", 2*time.Hour)
|
||||
|
||||
for _, p := range []string{fresh, old} {
|
||||
if err := cs.ScheduleCleanup(p); err != nil {
|
||||
t.Fatalf("ScheduleCleanup failed: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
if err := cs.ExecuteCleanup(context.Background()); err != nil {
|
||||
t.Fatalf("ExecuteCleanup failed: %v", err)
|
||||
}
|
||||
|
||||
if _, err := os.Stat(old); !os.IsNotExist(err) {
|
||||
t.Errorf("a file past its retention period should be removed, stat err = %v", err)
|
||||
}
|
||||
if _, err := os.Stat(fresh); err != nil {
|
||||
t.Errorf("a file inside its retention period should survive: %v", err)
|
||||
}
|
||||
if got := cs.GetPendingCount(); got != 1 {
|
||||
t.Fatalf("the retained file must stay queued for a later pass, pending = %d", got)
|
||||
}
|
||||
|
||||
// Once it ages past the window, the next pass must actually remove it.
|
||||
past := time.Now().Add(-2 * time.Hour)
|
||||
if err := os.Chtimes(fresh, past, past); err != nil {
|
||||
t.Fatalf("failed to age file: %v", err)
|
||||
}
|
||||
if err := cs.ExecuteCleanup(context.Background()); err != nil {
|
||||
t.Fatalf("ExecuteCleanup failed: %v", err)
|
||||
}
|
||||
if _, err := os.Stat(fresh); !os.IsNotExist(err) {
|
||||
t.Errorf("file should have been removed once past retention, stat err = %v", err)
|
||||
}
|
||||
if got := cs.GetPendingCount(); got != 0 {
|
||||
t.Errorf("pending count = %d, want 0", got)
|
||||
}
|
||||
}
|
||||
|
||||
// TestForceCleanupAllTerminatesWithRetainedFiles guards shutdown: retained
|
||||
// files are requeued by design, so a loop demanding an empty list would spin
|
||||
// until the shutdown timeout expired.
|
||||
func TestForceCleanupAllTerminatesWithRetainedFiles(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
cs := NewCleanupService(CleanupConfig{
|
||||
Enabled: true,
|
||||
RetentionPeriod: time.Hour,
|
||||
BatchSize: 10,
|
||||
CheckInterval: time.Second,
|
||||
})
|
||||
|
||||
if err := cs.ScheduleCleanup(writeAgedFile(t, dir, "fresh.ts", time.Minute)); err != nil {
|
||||
t.Fatalf("ScheduleCleanup failed: %v", err)
|
||||
}
|
||||
|
||||
done := make(chan error, 1)
|
||||
go func() { done <- cs.ForceCleanupAll(context.Background()) }()
|
||||
|
||||
select {
|
||||
case err := <-done:
|
||||
if err != nil {
|
||||
t.Fatalf("ForceCleanupAll failed: %v", err)
|
||||
}
|
||||
case <-time.After(3 * time.Second):
|
||||
t.Fatal("ForceCleanupAll did not terminate with a retained file pending")
|
||||
}
|
||||
|
||||
if got := cs.GetPendingCount(); got != 1 {
|
||||
t.Errorf("the retained file should still be pending, got %d", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCleanupMissingFileIsNotAnError(t *testing.T) {
|
||||
cs := NewCleanupService(CleanupConfig{Enabled: true, BatchSize: 10, CheckInterval: time.Second})
|
||||
|
||||
// Already gone — a file transferred and removed by something else.
|
||||
if err := cs.ScheduleCleanup(filepath.Join(t.TempDir(), "gone.ts")); err != nil {
|
||||
t.Fatalf("ScheduleCleanup failed: %v", err)
|
||||
}
|
||||
if err := cs.ExecuteCleanup(context.Background()); err != nil {
|
||||
t.Fatalf("ExecuteCleanup failed: %v", err)
|
||||
}
|
||||
if got := cs.GetPendingCount(); got != 0 {
|
||||
t.Errorf("a missing file should not be requeued, pending = %d", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCleanupDisabledSchedulesNothing(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
cs := NewCleanupService(CleanupConfig{Enabled: false, BatchSize: 10, CheckInterval: time.Second})
|
||||
|
||||
path := writeAgedFile(t, dir, "seg.ts", time.Hour)
|
||||
if err := cs.ScheduleCleanup(path); err != nil {
|
||||
t.Fatalf("ScheduleCleanup failed: %v", err)
|
||||
}
|
||||
|
||||
if got := cs.GetPendingCount(); got != 0 {
|
||||
t.Errorf("cleanup is disabled, nothing should be queued, got %d", got)
|
||||
}
|
||||
if err := cs.ExecuteCleanup(context.Background()); err != nil {
|
||||
t.Fatalf("ExecuteCleanup failed: %v", err)
|
||||
}
|
||||
if _, err := os.Stat(path); err != nil {
|
||||
t.Errorf("file must survive when cleanup is disabled: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestCleanupRequeuesOnCancellation covers shutdown arriving mid-batch: the
|
||||
// files already pulled off the pending list must go back, not vanish.
|
||||
func TestCleanupRequeuesOnCancellation(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
cs := NewCleanupService(CleanupConfig{Enabled: true, BatchSize: 10, CheckInterval: time.Second})
|
||||
|
||||
for i := 0; i < 5; i++ {
|
||||
path := writeAgedFile(t, dir, fmt.Sprintf("seg_%d.ts", i), 0)
|
||||
if err := cs.ScheduleCleanup(path); err != nil {
|
||||
t.Fatalf("ScheduleCleanup failed: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
cancel()
|
||||
|
||||
if err := cs.ExecuteCleanup(ctx); err == nil {
|
||||
t.Fatal("expected the cancelled context to be reported")
|
||||
}
|
||||
if got := cs.GetPendingCount(); got != 5 {
|
||||
t.Errorf("all files should be requeued after cancellation, pending = %d, want 5", got)
|
||||
}
|
||||
}
|
||||
@ -11,7 +11,7 @@ import (
|
||||
)
|
||||
|
||||
func TransferFile(nt *nas.NASService, ctx context.Context, item *TransferItem) error {
|
||||
destPath := filepath.Join(nt.Config.Path, item.DestinationPath)
|
||||
destPath := nt.Join(item.DestinationPath)
|
||||
|
||||
destDir := filepath.Dir(destPath)
|
||||
if err := nt.EnsureDirectoryExists(destDir); err != nil {
|
||||
|
||||
@ -169,13 +169,20 @@ func (tq *TransferQueue) processItem(ctx context.Context, item TransferItem) {
|
||||
return
|
||||
}
|
||||
|
||||
maxRetries := 3
|
||||
maxRetries := tq.config.RetryLimit
|
||||
if maxRetries < 1 {
|
||||
maxRetries = 1
|
||||
}
|
||||
|
||||
for attempt := 1; attempt <= maxRetries; attempt++ {
|
||||
if attempt > 0 {
|
||||
// Back off before a retry, never before the first attempt. The guard
|
||||
// was `attempt > 0`, which is always true, so every single transfer —
|
||||
// including healthy ones — paid a one-second delay before it began,
|
||||
// throttling the whole pipeline to the worker count per second.
|
||||
if attempt > 1 {
|
||||
item.Status = StatusRetrying
|
||||
backoff := time.Duration(attempt*attempt) * time.Second
|
||||
log.Printf("Backing off for %d seconds before retrying (attempt %d/%d)", backoff, attempt, maxRetries)
|
||||
log.Printf("Backing off for %s before retrying (attempt %d/%d)", backoff, attempt, maxRetries)
|
||||
|
||||
select {
|
||||
case <-time.After(backoff):
|
||||
|
||||
@ -34,14 +34,23 @@ func NewTrasferService(outputDir string, eventName string) (*TransferService, er
|
||||
Timeout: cfg.NAS.Timeout,
|
||||
RetryLimit: cfg.NAS.RetryLimit,
|
||||
VerifySize: true,
|
||||
AutoMount: cfg.NAS.AutoMount,
|
||||
MountPoint: cfg.NAS.MountPoint,
|
||||
}
|
||||
nas, err := nas2.New(nasConfig)
|
||||
if err != nil {
|
||||
logging.MustGetFactory().Transfer.Error("invalid NAS destination", logging.KeyError, err.Error())
|
||||
return nil, fmt.Errorf("invalid NAS destination: %w", err)
|
||||
}
|
||||
nas := nas2.NewNASService(nasConfig)
|
||||
|
||||
if err := nas.TestConnection(); err != nil {
|
||||
connectCtx, cancelConnect := context.WithTimeout(context.Background(), cfg.NAS.Timeout)
|
||||
defer cancelConnect()
|
||||
if err := nas.Connect(connectCtx); err != nil {
|
||||
logging.MustGetFactory().Transfer.Error("failed to connect to NAS", logging.KeyError, err.Error())
|
||||
return nil, fmt.Errorf("failed to connect to NAS: %w", err)
|
||||
}
|
||||
logging.MustGetFactory().Transfer.Info("connected to NAS", "path", outputDir)
|
||||
logging.MustGetFactory().Transfer.Info("connected to NAS",
|
||||
"path", nas.LocalPath(), "configured", outputDir, "how", strings.Join(nas.ConnectionInfo(), "; "))
|
||||
health.MustGetRegistry().RecordNASConnectivity(true)
|
||||
|
||||
cleanupConfig := CleanupConfig{
|
||||
@ -57,6 +66,7 @@ func NewTrasferService(outputDir string, eventName string) (*TransferService, er
|
||||
PersistencePath: cfg.Paths.PersistenceFile,
|
||||
MaxQueueSize: cfg.Transfer.QueueSize,
|
||||
BatchSize: cfg.Transfer.BatchSize,
|
||||
RetryLimit: cfg.Transfer.RetryLimit,
|
||||
}
|
||||
queue := NewTransferQueue(queueConfig, nas, cleanup)
|
||||
|
||||
@ -66,7 +76,7 @@ func NewTrasferService(outputDir string, eventName string) (*TransferService, er
|
||||
return nil, fmt.Errorf("failed to create local output directory: %w", err)
|
||||
}
|
||||
|
||||
watcher, err := NewFileWatcher(localOutputPath, queue, cfg.Transfer.FileSettlingDelay)
|
||||
watcher, err := NewFileWatcher(localOutputPath, eventName, queue, cfg.Transfer.FileSettlingDelay)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to create file watcher: %w", err)
|
||||
}
|
||||
@ -155,7 +165,7 @@ func (ts *TransferService) Shutdown(ctx context.Context) error {
|
||||
}
|
||||
|
||||
// Disconnect from NAS
|
||||
if err := ts.nas.Disconnect(); err != nil {
|
||||
if err := ts.nas.Disconnect(ctx); err != nil {
|
||||
log.Printf("Warning: failed to disconnect from NAS: %v", err)
|
||||
logging.MustGetFactory().Transfer.Warn("failed to disconnect from NAS", logging.KeyError, err.Error())
|
||||
}
|
||||
|
||||
@ -49,6 +49,9 @@ type QueueConfig struct {
|
||||
PersistencePath string
|
||||
MaxQueueSize int
|
||||
BatchSize int
|
||||
// RetryLimit is the total number of attempts made per file before it is
|
||||
// marked failed.
|
||||
RetryLimit int
|
||||
}
|
||||
|
||||
type CleanupConfig struct {
|
||||
|
||||
@ -18,6 +18,11 @@ import (
|
||||
|
||||
type FileWatcher struct {
|
||||
outputDir string
|
||||
// eventName namespaces the NAS destination. It is passed in rather than
|
||||
// derived from outputDir so that an unnamed event resolves to the NAS root
|
||||
// — matching where the downloader put the files — instead of picking up
|
||||
// the local output directory's own name.
|
||||
eventName string
|
||||
queue *TransferQueue
|
||||
watcher *fsnotify.Watcher
|
||||
settingDelay time.Duration
|
||||
@ -25,13 +30,14 @@ type FileWatcher struct {
|
||||
mu sync.Mutex
|
||||
}
|
||||
|
||||
func NewFileWatcher(outputDir string, queue *TransferQueue, settlingDelay time.Duration) (*FileWatcher, error) {
|
||||
func NewFileWatcher(outputDir, eventName string, queue *TransferQueue, settlingDelay time.Duration) (*FileWatcher, error) {
|
||||
watcher, err := fsnotify.NewWatcher()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &FileWatcher{
|
||||
outputDir: outputDir,
|
||||
eventName: eventName,
|
||||
queue: queue,
|
||||
watcher: watcher,
|
||||
settingDelay: settlingDelay,
|
||||
@ -92,6 +98,18 @@ func (fw *FileWatcher) addWatchRecursive(root string) error {
|
||||
}
|
||||
|
||||
func (fw *FileWatcher) handleFileEvent(event fsnotify.Event) {
|
||||
// Directories must be handled before the .ts filter below. fsnotify is not
|
||||
// recursive, so a resolution directory created after the watcher starts —
|
||||
// which is the normal case, since the transfer service starts before the
|
||||
// downloader creates 1080p/ and friends — has to be picked up here or
|
||||
// nothing inside it is ever seen.
|
||||
if event.Op&fsnotify.Create == fsnotify.Create {
|
||||
if info, err := os.Stat(event.Name); err == nil && info.IsDir() {
|
||||
fw.watchNewDirectory(event.Name)
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
if !strings.HasSuffix(strings.ToLower(event.Name), ".ts") {
|
||||
return
|
||||
}
|
||||
@ -104,15 +122,33 @@ func (fw *FileWatcher) handleFileEvent(event fsnotify.Event) {
|
||||
case event.Op&fsnotify.Remove == fsnotify.Remove:
|
||||
fw.cancelPendingTransfer(event.Name)
|
||||
}
|
||||
}
|
||||
|
||||
if event.Op&fsnotify.Create == fsnotify.Create {
|
||||
if info, err := os.Stat(event.Name); err == nil && info.IsDir() {
|
||||
if err := fw.watcher.Add(event.Name); err != nil {
|
||||
log.Printf("Failed to watch directory %s: %v", event.Name, err)
|
||||
} else {
|
||||
log.Printf("Watching directory %s", event.Name)
|
||||
// watchNewDirectory starts watching a directory that appeared after startup,
|
||||
// and sweeps whatever is already inside it.
|
||||
//
|
||||
// The sweep is not redundant: the downloader creates a resolution directory and
|
||||
// writes segments into it immediately, so files can land between the mkdir and
|
||||
// the watch being registered. Those writes generate no event we would see, and
|
||||
// without the sweep the first segments of every recording would be lost.
|
||||
func (fw *FileWatcher) watchNewDirectory(dir string) {
|
||||
if err := fw.addWatchRecursive(dir); err != nil {
|
||||
log.Printf("Failed to watch new directory %s: %v", dir, err)
|
||||
logging.MustGetFactory().Transfer.Warn("failed to watch new directory", "dir", dir, logging.KeyError, err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
err := filepath.Walk(dir, func(path string, info os.FileInfo, err error) error {
|
||||
if err != nil {
|
||||
return nil // keep sweeping the rest
|
||||
}
|
||||
if !info.IsDir() && strings.HasSuffix(strings.ToLower(info.Name()), ".ts") {
|
||||
fw.scheduleTransfer(path)
|
||||
}
|
||||
return nil
|
||||
})
|
||||
if err != nil {
|
||||
log.Printf("Failed to sweep new directory %s: %v", dir, err)
|
||||
}
|
||||
}
|
||||
|
||||
@ -164,10 +200,13 @@ func (fw *FileWatcher) processFile(filePath string) {
|
||||
return
|
||||
}
|
||||
|
||||
// The destination is namespaced by event, mirroring the local layout and
|
||||
// matching what QueueExistingFiles produces for the same file — otherwise
|
||||
// live transfers and catch-up transfers would write to different places.
|
||||
item := TransferItem{
|
||||
ID: generateID(),
|
||||
SourcePath: filePath,
|
||||
DestinationPath: relPath,
|
||||
DestinationPath: filepath.Join(fw.eventName, relPath),
|
||||
Resolution: resolution,
|
||||
Timestamp: time.Now(),
|
||||
Status: StatusPending,
|
||||
|
||||
223
pkg/transfer/watcher_test.go
Normal file
223
pkg/transfer/watcher_test.go
Normal file
@ -0,0 +1,223 @@
|
||||
package transfer
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// settlingDelay is short so tests exercise the real timer path without
|
||||
// waiting the production five seconds.
|
||||
const testSettlingDelay = 40 * time.Millisecond
|
||||
|
||||
// newTestQueue builds a queue that accepts items but never transfers them:
|
||||
// ProcessQueue is not started, so Add is a pure recording of intent.
|
||||
func newTestQueue(t *testing.T) *TransferQueue {
|
||||
t.Helper()
|
||||
return NewTransferQueue(QueueConfig{
|
||||
WorkerCount: 1,
|
||||
PersistencePath: filepath.Join(t.TempDir(), "queue.json"),
|
||||
MaxQueueSize: 100,
|
||||
BatchSize: 10,
|
||||
}, nil, nil)
|
||||
}
|
||||
|
||||
// queuedDestinations returns the destination paths currently queued, sorted.
|
||||
func queuedDestinations(q *TransferQueue) []string {
|
||||
q.mu.RLock()
|
||||
defer q.mu.RUnlock()
|
||||
|
||||
out := make([]string, 0, q.items.Len())
|
||||
for _, item := range *q.items {
|
||||
out = append(out, filepath.ToSlash(item.DestinationPath))
|
||||
}
|
||||
sort.Strings(out)
|
||||
return out
|
||||
}
|
||||
|
||||
// waitForQueue polls until the queue holds want items or the deadline passes.
|
||||
// Polling beats a fixed sleep: it keeps the test fast when the watcher is
|
||||
// prompt and tolerant when a loaded CI machine is not.
|
||||
func waitForQueue(t *testing.T, q *TransferQueue, want int) []string {
|
||||
t.Helper()
|
||||
|
||||
deadline := time.Now().Add(3 * time.Second)
|
||||
var got []string
|
||||
for time.Now().Before(deadline) {
|
||||
got = queuedDestinations(q)
|
||||
if len(got) >= want {
|
||||
return got
|
||||
}
|
||||
time.Sleep(10 * time.Millisecond)
|
||||
}
|
||||
return got
|
||||
}
|
||||
|
||||
func startWatcher(t *testing.T, dir, eventName string, q *TransferQueue) {
|
||||
t.Helper()
|
||||
|
||||
fw, err := NewFileWatcher(dir, eventName, q, testSettlingDelay)
|
||||
if err != nil {
|
||||
t.Fatalf("NewFileWatcher failed: %v", err)
|
||||
}
|
||||
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
t.Cleanup(cancel)
|
||||
|
||||
started := make(chan struct{})
|
||||
go func() {
|
||||
close(started)
|
||||
_ = fw.Start(ctx)
|
||||
}()
|
||||
<-started
|
||||
|
||||
// Give Start a moment to register its initial watches before the test
|
||||
// begins creating files under them.
|
||||
time.Sleep(100 * time.Millisecond)
|
||||
}
|
||||
|
||||
// TestWatcherPicksUpDirectoriesCreatedAfterStart covers the normal recording
|
||||
// sequence: the transfer service starts watching the event directory, and only
|
||||
// then does the downloader create the per-resolution directories inside it.
|
||||
// fsnotify is not recursive, so those directories have to be picked up from
|
||||
// the create event or nothing inside them is ever transferred.
|
||||
func TestWatcherPicksUpDirectoriesCreatedAfterStart(t *testing.T) {
|
||||
eventDir := t.TempDir()
|
||||
queue := newTestQueue(t)
|
||||
startWatcher(t, eventDir, "2026-regionals", queue)
|
||||
|
||||
resolutionDir := filepath.Join(eventDir, "1080p")
|
||||
if err := os.MkdirAll(resolutionDir, 0755); err != nil {
|
||||
t.Fatalf("failed to create resolution dir: %v", err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(resolutionDir, "seg_1001.ts"), []byte("payload"), 0644); err != nil {
|
||||
t.Fatalf("failed to write segment: %v", err)
|
||||
}
|
||||
|
||||
got := waitForQueue(t, queue, 1)
|
||||
if len(got) != 1 {
|
||||
t.Fatalf("segment written to a new resolution directory was never queued, queue = %v", got)
|
||||
}
|
||||
if want := "2026-regionals/1080p/seg_1001.ts"; got[0] != want {
|
||||
t.Errorf("destination = %q, want %q", got[0], want)
|
||||
}
|
||||
}
|
||||
|
||||
// TestWatcherSweepsFilesRacingDirectoryCreation covers the tighter race where
|
||||
// segments are written immediately after the mkdir, before the new watch can
|
||||
// be registered. Those writes produce no event the watcher would receive, so
|
||||
// the directory has to be swept when it is first seen.
|
||||
func TestWatcherSweepsFilesRacingDirectoryCreation(t *testing.T) {
|
||||
eventDir := t.TempDir()
|
||||
queue := newTestQueue(t)
|
||||
startWatcher(t, eventDir, "event", queue)
|
||||
|
||||
// Stage the tree elsewhere and move it into place, so the files are
|
||||
// guaranteed to already exist the instant the directory appears.
|
||||
staging := filepath.Join(t.TempDir(), "1080p")
|
||||
if err := os.MkdirAll(staging, 0755); err != nil {
|
||||
t.Fatalf("failed to stage: %v", err)
|
||||
}
|
||||
for _, name := range []string{"seg_1001.ts", "seg_1002.ts"} {
|
||||
if err := os.WriteFile(filepath.Join(staging, name), []byte("payload"), 0644); err != nil {
|
||||
t.Fatalf("failed to stage segment: %v", err)
|
||||
}
|
||||
}
|
||||
if err := os.Rename(staging, filepath.Join(eventDir, "1080p")); err != nil {
|
||||
t.Fatalf("failed to move staged directory into place: %v", err)
|
||||
}
|
||||
|
||||
got := waitForQueue(t, queue, 2)
|
||||
want := []string{"event/1080p/seg_1001.ts", "event/1080p/seg_1002.ts"}
|
||||
if len(got) != 2 || got[0] != want[0] || got[1] != want[1] {
|
||||
t.Errorf("queued %v, want %v", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
// TestWatcherDestinationIncludesEventName pins the layout agreement between
|
||||
// live transfers and the catch-up path in QueueExistingFiles: both must write
|
||||
// to <nas>/<event>/<resolution>/<file>.
|
||||
func TestWatcherDestinationIncludesEventName(t *testing.T) {
|
||||
eventDir := t.TempDir()
|
||||
resolutionDir := filepath.Join(eventDir, "720p")
|
||||
if err := os.MkdirAll(resolutionDir, 0755); err != nil {
|
||||
t.Fatalf("setup failed: %v", err)
|
||||
}
|
||||
|
||||
queue := newTestQueue(t)
|
||||
startWatcher(t, eventDir, "my-event", queue)
|
||||
|
||||
if err := os.WriteFile(filepath.Join(resolutionDir, "seg_1001.ts"), []byte("payload"), 0644); err != nil {
|
||||
t.Fatalf("failed to write segment: %v", err)
|
||||
}
|
||||
|
||||
got := waitForQueue(t, queue, 1)
|
||||
if len(got) != 1 {
|
||||
t.Fatalf("expected one queued item, got %v", got)
|
||||
}
|
||||
if want := "my-event/720p/seg_1001.ts"; got[0] != want {
|
||||
t.Errorf("destination = %q, want %q", got[0], want)
|
||||
}
|
||||
}
|
||||
|
||||
// TestWatcherWithoutEventNameUsesRoot mirrors a download run with no -event:
|
||||
// segments land under the local output root, so the NAS destination must not
|
||||
// invent a directory level for them.
|
||||
func TestWatcherWithoutEventNameUsesRoot(t *testing.T) {
|
||||
eventDir := t.TempDir()
|
||||
resolutionDir := filepath.Join(eventDir, "1080p")
|
||||
if err := os.MkdirAll(resolutionDir, 0755); err != nil {
|
||||
t.Fatalf("setup failed: %v", err)
|
||||
}
|
||||
|
||||
queue := newTestQueue(t)
|
||||
startWatcher(t, eventDir, "", queue)
|
||||
|
||||
if err := os.WriteFile(filepath.Join(resolutionDir, "seg_1001.ts"), []byte("payload"), 0644); err != nil {
|
||||
t.Fatalf("failed to write segment: %v", err)
|
||||
}
|
||||
|
||||
got := waitForQueue(t, queue, 1)
|
||||
if len(got) != 1 {
|
||||
t.Fatalf("expected one queued item, got %v", got)
|
||||
}
|
||||
if want := "1080p/seg_1001.ts"; got[0] != want {
|
||||
t.Errorf("destination = %q, want %q", got[0], want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestWatcherIgnoresNonSegmentFiles(t *testing.T) {
|
||||
eventDir := t.TempDir()
|
||||
queue := newTestQueue(t)
|
||||
startWatcher(t, eventDir, "event", queue)
|
||||
|
||||
for _, name := range []string{"notes.txt", "manifest.json", "segment.ts.part"} {
|
||||
if err := os.WriteFile(filepath.Join(eventDir, name), []byte("x"), 0644); err != nil {
|
||||
t.Fatalf("failed to write %s: %v", name, err)
|
||||
}
|
||||
}
|
||||
|
||||
time.Sleep(testSettlingDelay + 300*time.Millisecond)
|
||||
if got := queuedDestinations(queue); len(got) != 0 {
|
||||
t.Errorf("non-segment files should not be queued, got %v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExtractResolution(t *testing.T) {
|
||||
fw := &FileWatcher{}
|
||||
|
||||
cases := map[string]string{
|
||||
filepath.Join("data", "event", "1080p", "seg.ts"): "1080p",
|
||||
filepath.Join("data", "event", "720p", "seg.ts"): "720p",
|
||||
filepath.Join("data", "event", "seg.ts"): "",
|
||||
}
|
||||
|
||||
for path, want := range cases {
|
||||
if got := fw.extractResolution(path); got != want {
|
||||
t.Errorf("extractResolution(%q) = %q, want %q", path, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
@ -24,12 +24,28 @@ func PathExists(path string) bool {
|
||||
return !os.IsNotExist(err)
|
||||
}
|
||||
|
||||
// IsValidPath reports whether path is usable on Windows, the stricter of the
|
||||
// two platforms this project targets, which reserves <>:"|?* in names.
|
||||
//
|
||||
// A leading drive letter is checked separately: its colon is legal there and
|
||||
// nowhere else in the path. Without that carve-out the function rejects every
|
||||
// absolute Windows path, including the drive-qualified ones it exists to
|
||||
// validate.
|
||||
func IsValidPath(path string) bool {
|
||||
if path == "" {
|
||||
return false
|
||||
}
|
||||
|
||||
return !strings.ContainsAny(path, "<>:\"|?*")
|
||||
rest := path
|
||||
if len(rest) >= 2 && rest[1] == ':' && isDriveLetter(rest[0]) {
|
||||
rest = rest[2:]
|
||||
}
|
||||
|
||||
return !strings.ContainsAny(rest, `<>:"|?*`)
|
||||
}
|
||||
|
||||
func isDriveLetter(c byte) bool {
|
||||
return (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z')
|
||||
}
|
||||
|
||||
func NormalizePath(path string) string {
|
||||
@ -37,6 +53,13 @@ func NormalizePath(path string) string {
|
||||
}
|
||||
|
||||
func GetRelativePath(basePath, targetPath string) (string, error) {
|
||||
// filepath.Rel("", "") succeeds and returns ".", which is a meaningless
|
||||
// answer to a meaningless question; callers passing an empty path have a
|
||||
// bug worth surfacing rather than papering over.
|
||||
if basePath == "" || targetPath == "" {
|
||||
return "", fmt.Errorf("both base and target paths are required (base %q, target %q)", basePath, targetPath)
|
||||
}
|
||||
|
||||
rel, err := filepath.Rel(basePath, targetPath)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("failed to get relative path: %w", err)
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user