Compare commits
23 Commits
v0.5
...
ebd09af6b1
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
ebd09af6b1 | ||
|
|
0511d47765 | ||
|
|
93b671775f | ||
|
|
c18eb8fb07 | ||
|
|
0cb4d355ab | ||
|
|
6cea15f553 | ||
|
|
6e56e74df4 | ||
|
|
3c22b2e1a0 | ||
|
|
65a57a6af6 | ||
|
|
e063352252 | ||
|
|
f20f490c15 | ||
|
|
ae321a216d | ||
|
|
729b147b90 | ||
|
|
7b001d710a | ||
|
|
2201fc72f1 | ||
|
|
5e7dd4b960 | ||
|
|
c1444f8c1d | ||
|
|
6955e14921 | ||
|
|
fd8612cfc6 | ||
|
|
e449fb78c9 | ||
|
|
958ef67f29 | ||
|
|
14b7a17b00 | ||
|
|
c3553de856 |
4
.gitignore
vendored
4
.gitignore
vendored
@@ -1,6 +1,8 @@
|
||||
# Binary
|
||||
/backer
|
||||
/backer.exe
|
||||
/cmd/backer/backer
|
||||
/cmd/backer/backer.exe
|
||||
|
||||
# Local config (may contain secrets)
|
||||
/config.json
|
||||
@@ -27,3 +29,5 @@ Thumbs.db
|
||||
*.exe
|
||||
*.test
|
||||
*.out
|
||||
|
||||
/test_data/tmp
|
||||
|
||||
@@ -74,6 +74,7 @@ linters:
|
||||
- paramTypeCombine
|
||||
- whyNoLint
|
||||
- hugeParam
|
||||
- unnamedResult
|
||||
godoclint:
|
||||
default: all
|
||||
ignore-unexported: false
|
||||
|
||||
69
AGENTS.md
69
AGENTS.md
@@ -20,7 +20,7 @@ Backer is a Go-based HTTP/HTTPS backup server that creates compressed archives o
|
||||
- **Use tags and labels.** In comments, add tags or labels to files, classes, or functions to categorize them based on functionality, domain concepts, or design patterns.
|
||||
- **Avoid obvious comments that do not add value.**
|
||||
- **Remove dead code.** A half-finished refactoring can actually harm the contributions.
|
||||
- **Update AGENTS.md and TODO.md automatically.** Keep AGENTS.md dense, detailed and concise.
|
||||
- **Update AGENTS.md automatically.** Keep AGENTS.md dense, detailed and concise.
|
||||
|
||||
## Workflow Rules
|
||||
|
||||
@@ -124,6 +124,18 @@ CompressionLevel int `json:"compression_level"`
|
||||
| QF1001 | De Morgan's law disabled for auth condition clarity |
|
||||
| revive (error strings) | Same reason as ST1005 |
|
||||
|
||||
### Design Decisions (Not Issues)
|
||||
|
||||
The following are intentional design decisions, not bugs or defects:
|
||||
|
||||
- **No rate limiting**: Backups are assumed to be infrequent enough that rate limiting is unnecessary.
|
||||
- **Backup scheduling**: Client-side decision — server serves backups on-request only.
|
||||
- **Basic Auth only**: HTTP Basic Auth is sufficient for a backup server behind a firewall.
|
||||
- **Global config**: The `C Config` global variable is intentional — config is loaded once at startup.
|
||||
- **No incremental backup**: Each backup is a full snapshot, not incremental.
|
||||
- **Credentials in config file**: Config file should have restricted permissions (documented).
|
||||
- **Thread-safe logging**: The global `Log` variable uses mutex protection to enable parallel test execution; this is not needed for single-server production use but allows tests to run in parallel without race conditions.
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
@@ -228,16 +240,17 @@ No temp files on disk. Uses `io.Pipe` with a goroutine to stream archives on-the
|
||||
|
||||
Two helper functions in `newServer.go` handle context cancellation during streaming:
|
||||
- `writeWithContext(ctx, fn)` — checks context before calling fn
|
||||
- `copyWithContext(ctx, dst, src)` — buffered copy (32KB) with cancellation checks
|
||||
- `copyWithContext(ctx, dst, src)` — buffered copy (32KB) with cancellation checks, computes MD5 hash of copied data
|
||||
|
||||
### Special File Type Handling
|
||||
|
||||
- **Regular files**: streamed into tar archive with content
|
||||
- **Directories**: included as tar directory entries
|
||||
- **Symlinks**: stored as symlink entries (target path preserved)
|
||||
- **Symlinks**: stored as symlink entries (target path preserved). Backer never resolves or follows symlinks during directory traversal — it uses `os.Lstat` and `filepath.WalkDir` which do not follow symlinks. This means symlink cycles are impossible to occur.
|
||||
- **Device files** (char/block): included with correct major/minor
|
||||
- **Named pipes**: included (header only, no data)
|
||||
- **Sockets**: skipped (not archivable)
|
||||
- **Hard links**: deduplicated — if multiple files share the same inode, only the first is stored with content, subsequent hard links are stored as link entries pointing to the original
|
||||
|
||||
### Pipe Error Recovery
|
||||
|
||||
@@ -245,7 +258,7 @@ Two helper functions in `newServer.go` handle context cancellation during stream
|
||||
|
||||
### Auth Handler
|
||||
|
||||
HTTP Basic Auth. Both username AND password must match exactly. Returns `401 Unauthorized` with `WWW-Authenticate` header on failure. The condition is intentionally written without De Morgan's law simplification for clarity (linter suppression `//nolint:staticcheck`).
|
||||
HTTP Basic Auth. Both username AND password must match exactly. Returns `401 Unauthorized` with `WWW-Authenticate` header on failure. Uses timing-safe comparison to prevent timing attacks. The condition is intentionally written without De Morgan's law simplification for clarity (linter suppression `//nolint:staticcheck`).
|
||||
|
||||
### Client IP Detection
|
||||
|
||||
@@ -255,8 +268,56 @@ Checks `X-Forwarded-For` header first (for proxied requests), then falls back to
|
||||
|
||||
- `ReadHeaderTimeout`: 5 seconds
|
||||
- `WriteTimeout`: configurable via `backup_timeout` (converted to `time.Duration`)
|
||||
- `IdleTimeout`: 10 seconds (for keep-alive connections; backer disables keep-alives explicitly)
|
||||
- `MaxHeaderBytes`: 1KB (prevents header-based DoS attacks)
|
||||
- `DisableKeepAlives`: true (each connection closed after request, no connection reuse)
|
||||
- TLS minimum version: TLS 1.3 (when HTTPS enabled)
|
||||
|
||||
### TLS Error Handling
|
||||
|
||||
The server uses a `ServerWrapper` type to intercept and classify errors from `Serve()` and `ServeTLS()`. TLS-specific errors are logged at debug level to reduce log clutter:
|
||||
|
||||
- Uses `errors.As` to detect TLS error types: `tls.AlertError`, `tls.CertificateVerificationError`, `tls.ECHRejectionError`, `tls.RecordHeaderError`
|
||||
- Also checks for "tls:" prefix in error messages
|
||||
- Additionally uses `http.Server.ErrorLog` with a custom logger that writes at debug level for net/http's internal TLS-related messages
|
||||
|
||||
This ensures client-side TLS errors (handshake failures, certificate issues) don't clutter production logs.
|
||||
|
||||
### Compression Level Mapping
|
||||
|
||||
The `compression_level` config option (1-9) is mapped differently for each compression algorithm:
|
||||
|
||||
| Algorithm | Level Mapping |
|
||||
|-----------|----------------|
|
||||
| gzip | 1-9 directly maps to gzip compression levels |
|
||||
| pgzip | 1-9 directly maps to pgzip compression levels |
|
||||
| bzip2 | 1-9 directly maps to bzip2 compression levels |
|
||||
| zstd | 1-3: Fastest, 4-6: Default, 7-9: BetterCompression |
|
||||
| lz4 | 1-3: Fastest (0), 4-6: Default (1), 7-9: Best (2) |
|
||||
| xz | Uses default compression (no configurable level) |
|
||||
|
||||
### HTTP Routing & Compression Selection
|
||||
|
||||
The server uses `http.NewServeMux()` for routing. Multiple routes are registered to support different compression formats:
|
||||
|
||||
```go
|
||||
mux.Handle(C.Location, http.HandlerFunc(backupHandler)) // /archive
|
||||
mux.Handle(C.Location+"/", http.HandlerFunc(backupHandler)) // /archive/
|
||||
mux.Handle(C.Location+".tar.gz", http.HandlerFunc(backupHandler)) // /archive.tar.gz
|
||||
mux.Handle(C.Location+".tar.xz", http.HandlerFunc(backupHandler)) // /archive.tar.xz
|
||||
mux.Handle(C.Location+".tar.bz2", http.HandlerFunc(backupHandler)) // /archive.tar.bz2
|
||||
// ... etc
|
||||
```
|
||||
|
||||
The compression algorithm is determined by extracting the extension from the request path using `path.Ext()`:
|
||||
- For `/archive.tar.xz`, `path.Ext()` returns `.tar.xz` → maps to xz
|
||||
- For `/archive`, `path.Ext()` returns `""` → uses default compression
|
||||
|
||||
The `getCompressionAlgorithm()` function handles both full paths and bare extensions:
|
||||
- Full path: `/archive.tar.xz` → `.tar.xz` → xz
|
||||
- Bare extension: `tar.xz` → `.tar.xz` → xz
|
||||
- Single ext: `xz` → `.xz` → xz
|
||||
|
||||
## Dependencies
|
||||
|
||||
- `github.com/hjson/hjson-go` v3.3.0 — HJSON config parsing
|
||||
|
||||
81
README.md
81
README.md
@@ -75,14 +75,14 @@ All options described in example config.
|
||||
|
||||
AI suggests this table as expected performance.
|
||||
|
||||
| Algorithm | Speed | Compression | Notes |
|
||||
| :--- | :--- | :--- | :--- |
|
||||
| gzip | Fast | Good | Default. Standard, widely compatible. |
|
||||
| pgzip | Fast | Good | Parallel gzip. Uses multiple CPU cores, slightly larger files. |
|
||||
| bzip2 | Slow | Better | Pure Go. Good compression, slower than gzip. |
|
||||
| zstd | Fast | Very good | Good balance of speed and compression. |
|
||||
| lz4 | Fastest | Moderate | Extremely fast, larger files. |
|
||||
| xz | Slowest | Best | Best compression, slowest speed. |
|
||||
| Algorithm | Speed | Compression | Notes |
|
||||
| :--- | :--- | :--- | :--- |
|
||||
| gzip | Fast | Good | Standard, widely compatible. |
|
||||
| pgzip | Fast | Good | Parallel gzip. Uses multiple CPU cores, slightly larger files. |
|
||||
| bzip2 | Slow | Better | Pure Go. Good compression, slower than gzip. |
|
||||
| zstd | Fast | Very good | Good balance of speed and compression. |
|
||||
| lz4 | Fastest | Moderate | Extremely fast, larger files. |
|
||||
| xz | Slowest | Best | Best compression, slowest speed. |
|
||||
|
||||
In real live with backup of 1 lxc container inside vm powered by 1 core cpu, 768Mb RAM and 100Mbit non-guaranteed
|
||||
network bandwidth, located in near by country I've got these results
|
||||
@@ -90,12 +90,21 @@ network bandwidth, located in near by country I've got these results
|
||||
| Algorithm | Size | Time | CPU Load | RAM RSS+Shm, Mb | Notes |
|
||||
| :--- | ---: | ---: | ---: | ---: | :--- |
|
||||
| None | 1461250551 | | 0-2% | 0+0 | original dataset |
|
||||
| gzip | 823381547 | 16m:22s | 96% | 18+6 | stdlib compression/gzip |
|
||||
| pgzip | 823783924 | 13m:50s | 93-97% | 28+5 | multithreaded (parallel gzip), fully gzip compatible, klauspost/pgzip |
|
||||
| bzip2 | 797053471 | 9m:47s | 92-94% | 56+6 | dsnet/compress/bzip2 |
|
||||
| xz | 515106988 | 16m:03s | 96% | 116+5 | ulikunitz/xz |
|
||||
| zstd | 617330529 | 6m:39s | 40-99% | 64+6 | klauspost/compress/zstd |
|
||||
| lz4 | 1097745915 | 6m:12s | 4-6% | 30+6 | multithreaded lz4, pierrec/lz4 |
|
||||
| zstd | 614015773 | 4m:52s | 40-60% | 60+6 | klauspost/compress/zstd |
|
||||
| pgzip | 817163857 | 11m:25s | 96-97% | 37+6 | multithreaded (parallel gzip), fully gzip compatible, klauspost/pgzip |
|
||||
| gzip | 816734188 | 14m:30s | 96% | 18+6 | stdlib compression/gzip |
|
||||
| bzip2 | 788061284 | 7m:16s | 92-94% | 43+6 | dsnet/compress/bzip2 |
|
||||
| xz | 512004612 | 13m:53s | 96% | 116+6 | ulikunitz/xz |
|
||||
|
||||
The `compression_level` option (1-9) maps differently for each algorithm:
|
||||
|
||||
| Algorithm | Level Mapping |
|
||||
| :--- | :--- |
|
||||
| gzip/pgzip/bzip2 | 1-9 directly maps to compression levels |
|
||||
| lz4 | 1-3 → Fastest (0), 4-6 → Default (1), 7-9 → Best (2) |
|
||||
| zstd | 1-3 → Fastest, 4-6 → Default, 7-9 → BetterCompression |
|
||||
| xz | Uses default compression (no configurable level) |
|
||||
|
||||
Obvously in case of lz4 and zstd the bottleneck was network. And I have to admit, that network definitely was not
|
||||
100Mbit, but it was pretty hard to measure real bandwidth. Anyway let's pretend that during experiment it was roughly
|
||||
@@ -111,6 +120,52 @@ good enough cypher.
|
||||
|
||||
And one more thing user and password are mandatory, we should have at least some minimal barriers, right?
|
||||
|
||||
## Supported File Types
|
||||
|
||||
Backer handles various file types when creating archives:
|
||||
|
||||
- **Regular files**: Included with full content
|
||||
- **Directories**: Included as directory entries
|
||||
- **Symlinks**: Stored as symlink entries (target path preserved)
|
||||
- **Device files**: Character and block devices included with correct major/minor
|
||||
- **Named pipes**: Included (header only, no data)
|
||||
- **Hard links**: Deduplicated — multiple hard links to the same file are stored as link entries to avoid duplication
|
||||
|
||||
## Logging
|
||||
|
||||
Backer logs events via Go's `slog`. Log levels: error, warn, info, debug.
|
||||
### Backup Completion Log
|
||||
|
||||
When a backup completes, the following information is logged:
|
||||
- **client**: Client IP address
|
||||
- **user**: Authenticated username
|
||||
- **files**: Number of files backed up
|
||||
- **bytes**: Size of the created archive in bytes
|
||||
- **md5**: MD5 hash of the created archive (for verification)
|
||||
- **duration**: Time taken to create the archive
|
||||
|
||||
Example:
|
||||
```
|
||||
Backup completed: client=127.0.0.1 user=admin files=42 bytes=1234567 md5=abc123def456 duration=2.5s
|
||||
```
|
||||
### TLS Error Handling
|
||||
|
||||
TLS handshake errors and other client-side SSL/TLS errors are logged at **debug** level to reduce log clutter.
|
||||
This includes:
|
||||
- Certificate verification failures
|
||||
- Handshake failures
|
||||
- Protocol errors
|
||||
- TLS-specific connection issues
|
||||
|
||||
### Graceful Shutdown
|
||||
|
||||
When backer receives a termination signal (SIGTERM or SIGINT), it will:
|
||||
1. Log a warning message: "Shutdown signal received, shutting down immediately..."
|
||||
2. Stop accepting new connections and abort any in-progress downloads immediately
|
||||
3. Clean up resources and exit
|
||||
|
||||
This ensures no partial/corrupted backups are left. Clients will need to restart their backup. For a forced immediate stop, use `SIGKILL`.
|
||||
|
||||
## Special Thanks
|
||||
|
||||
This project was developed with the assistance of **opencode** (AI co-programmer powered by MIMO). The AI helped with:
|
||||
|
||||
@@ -2,6 +2,7 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"flag"
|
||||
"fmt"
|
||||
"net/http"
|
||||
@@ -40,10 +41,14 @@ func main() {
|
||||
log.Fatalf("Failed to create server: %v", err)
|
||||
}
|
||||
|
||||
log.Infof("Starting server at %s", server.Addr)
|
||||
// Setup signal handling for immediate shutdown.
|
||||
log.Infof("Installing signal handler")
|
||||
setupSignalHandler(server)
|
||||
|
||||
log.Infof("Starting backer version %s at %s", version, server.Addr)
|
||||
|
||||
if backer.C.NoHTTPS {
|
||||
if err := server.ListenAndServe(); err != nil && err != http.ErrServerClosed {
|
||||
if err := server.Serve(); err != nil && !errors.Is(err, http.ErrServerClosed) {
|
||||
log.Fatalf("Failed to start server: %v", err)
|
||||
}
|
||||
} else {
|
||||
@@ -55,10 +60,12 @@ func main() {
|
||||
log.Fatalf("Certificate key file error: %v", err)
|
||||
}
|
||||
|
||||
if err := server.ListenAndServeTLS(backer.C.Cert, backer.C.Key); err != nil && err != http.ErrServerClosed {
|
||||
if err := server.ServeTLS(backer.C.Cert, backer.C.Key); err != nil && !errors.Is(err, http.ErrServerClosed) {
|
||||
log.Fatalf("Failed to start server: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
log.Infof("Server stopped")
|
||||
}
|
||||
|
||||
// defaultConfigPath returns the default config file path based on the OS.
|
||||
|
||||
29
cmd/backer/sigHandler.go
Normal file
29
cmd/backer/sigHandler.go
Normal file
@@ -0,0 +1,29 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"os"
|
||||
"os/signal"
|
||||
"syscall"
|
||||
|
||||
"backer/internal/backer"
|
||||
"backer/internal/log"
|
||||
)
|
||||
|
||||
// setupSignalHandler sets up signal handling for immediate shutdown.
|
||||
// SIGTERM is commonly sent by systemd, k8s, and other container orchestrators.
|
||||
// SIGINT is sent on Ctrl+C.
|
||||
func setupSignalHandler(server *backer.ServerWrapper) {
|
||||
signalCh := make(chan os.Signal, 1)
|
||||
signal.Notify(signalCh, os.Interrupt, syscall.SIGTERM)
|
||||
|
||||
go func() {
|
||||
<-signalCh
|
||||
log.Warnf("Shutdown signal received, shutting down immediately...")
|
||||
// Close the embedded http.Server directly.
|
||||
if closeErr := server.Close(); closeErr != nil {
|
||||
log.Errorf("Error during server shutdown: %v", closeErr)
|
||||
}
|
||||
}()
|
||||
}
|
||||
|
||||
/* vim: setlocal ft=go noet ai ts=4 sw=4 sts=4: */
|
||||
@@ -73,7 +73,14 @@ func createArchiveStream(ctx context.Context, filepaths []string, writerFactory
|
||||
}
|
||||
|
||||
// writeFilesToTar iterates over filepaths and writes each file to the tar archive.
|
||||
// It tracks inodes to handle hard links - if a file has already been archived,
|
||||
// subsequent hard links to the same inode are stored as link entries.
|
||||
// This is the Unix implementation (syscall not available on Windows).
|
||||
func writeFilesToTar(ctx context.Context, filepaths []string, tw *tar.Writer, closePipeWithError func(error)) {
|
||||
// Track inodes to handle hard links: inode -> first path added.
|
||||
// Uses simple file path tracking on Windows, inode tracking on Unix.
|
||||
seenInodes := make(map[uint64]string)
|
||||
|
||||
for _, fpath := range filepaths {
|
||||
var (
|
||||
err error
|
||||
@@ -116,6 +123,21 @@ func writeFilesToTar(ctx context.Context, filepaths []string, tw *tar.Writer, cl
|
||||
continue
|
||||
}
|
||||
|
||||
// Check for hard link - if inode was already seen, store as link instead of content.
|
||||
if st.Mode().IsRegular() {
|
||||
inode := getInode(st)
|
||||
if inode != 0 {
|
||||
if existingPath, seen := seenInodes[inode]; seen {
|
||||
// This is a hard link to already-archived file.
|
||||
linkTarget = existingPath
|
||||
log.Debugf("Hard link detected: %s -> %s", fpath, existingPath)
|
||||
} else {
|
||||
// First time seeing this inode - track it.
|
||||
seenInodes[inode] = fpath
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
header, err := tar.FileInfoHeader(st, linkTarget)
|
||||
|
||||
if err != nil {
|
||||
@@ -124,6 +146,11 @@ func writeFilesToTar(ctx context.Context, filepaths []string, tw *tar.Writer, cl
|
||||
continue
|
||||
}
|
||||
|
||||
// If linkTarget is set, ensure the header is marked as a link.
|
||||
if linkTarget != "" {
|
||||
header.Typeflag = tar.TypeLink
|
||||
}
|
||||
|
||||
header.Format = tar.FormatGNU
|
||||
header.Name = strings.TrimPrefix(filepath.ToSlash(filepath.Clean(fpath)), "/")
|
||||
|
||||
@@ -146,7 +173,7 @@ func writeFilesToTar(ctx context.Context, filepaths []string, tw *tar.Writer, cl
|
||||
continue
|
||||
}
|
||||
|
||||
err = copyWithContext(ctx, tw, f)
|
||||
_, _, err = copyWithContext(ctx, tw, f)
|
||||
|
||||
if e := f.Close(); e != nil {
|
||||
log.Warnf("Failed to close file %s: %v", fpath, e)
|
||||
@@ -161,6 +188,27 @@ func writeFilesToTar(ctx context.Context, filepaths []string, tw *tar.Writer, cl
|
||||
}
|
||||
}
|
||||
|
||||
// getInode extracts the inode number from [os.FileInfo].
|
||||
// Returns 0 if unable to determine (e.g., on Windows or with certain file systems).
|
||||
func getInode(fi os.FileInfo) uint64 {
|
||||
if fi.Sys() == nil {
|
||||
return 0
|
||||
}
|
||||
|
||||
// Use type assertion to get inode - works on Unix, returns 0 on Windows.
|
||||
type inodeGetter interface {
|
||||
Dev() uint64
|
||||
Ino() uint64
|
||||
}
|
||||
|
||||
if ig, ok := fi.Sys().(inodeGetter); ok {
|
||||
// Combine device and inode for uniqueness across file systems.
|
||||
return (ig.Dev() << 32) | ig.Ino()
|
||||
}
|
||||
|
||||
return 0
|
||||
}
|
||||
|
||||
// isPipeClosedError checks if the error is due to a closed pipe.
|
||||
func isPipeClosedError(err error) bool {
|
||||
if err == nil {
|
||||
|
||||
@@ -10,8 +10,22 @@ import (
|
||||
// CreateTarBzip2Stream takes a list of files and returns a reader that reads from a bzip2-compressed tar archive.
|
||||
func CreateTarBzip2Stream(ctx context.Context, filepaths []string) io.ReadCloser {
|
||||
return createArchiveStream(ctx, filepaths, func(w io.Writer) (io.WriteCloser, error) {
|
||||
return bzip2.NewWriter(w, nil)
|
||||
level := mapCompressionLevelToBzip2(C.CompressionLevel)
|
||||
|
||||
return bzip2.NewWriter(w, &bzip2.WriterConfig{Level: level})
|
||||
})
|
||||
}
|
||||
|
||||
// mapCompressionLevelToBzip2 maps 1-9 compression level to bzip2 level.
|
||||
func mapCompressionLevelToBzip2(level int) int {
|
||||
if level < 1 {
|
||||
return bzip2.DefaultCompression
|
||||
}
|
||||
if level > 9 {
|
||||
return bzip2.BestCompression
|
||||
}
|
||||
|
||||
return level
|
||||
}
|
||||
|
||||
/* vim: setlocal ft=go noet ai ts=4 sw=4 sts=4: */
|
||||
|
||||
@@ -11,6 +11,12 @@ const (
|
||||
// copyBufferSize is the buffer size used for streaming file contents.
|
||||
copyBufferSize = 32 * 1024
|
||||
|
||||
// maxHeaderBytes is the maximum size of request headers (1KB).
|
||||
maxHeaderBytes = 1024
|
||||
|
||||
// idleTimeout is the maximum time to wait for the next request on keep-alive connections.
|
||||
idleTimeout = 10 * time.Second
|
||||
|
||||
// portMin is the minimum valid port number.
|
||||
portMin = 1
|
||||
|
||||
@@ -23,6 +29,12 @@ const (
|
||||
// backupTimeoutMax is the maximum backup timeout in minutes (24 hours).
|
||||
backupTimeoutMax = 1440
|
||||
|
||||
// dirScanTimeoutMin is the minimum directory scan timeout in minutes.
|
||||
dirScanTimeoutMin = 1
|
||||
|
||||
// dirScanTimeoutMax is the maximum directory scan timeout in minutes (24 hours).
|
||||
dirScanTimeoutMax = 1440
|
||||
|
||||
// compressionLevelMin is the minimum compression level.
|
||||
compressionLevelMin = 1
|
||||
|
||||
@@ -35,6 +47,9 @@ const (
|
||||
// defaultBackupTimeout is the default backup timeout in minutes.
|
||||
defaultBackupTimeout = 60
|
||||
|
||||
// defaultDirScanTimeout is the default directory scan timeout in minutes (1 hour).
|
||||
defaultDirScanTimeout = 60
|
||||
|
||||
// defaultCompressionLevel is the default compression level.
|
||||
defaultCompressionLevel = 9
|
||||
|
||||
|
||||
@@ -1,10 +1,14 @@
|
||||
package backer
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"io/fs"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"time"
|
||||
|
||||
"backer/internal/log"
|
||||
backerlog "backer/internal/log"
|
||||
)
|
||||
|
||||
// isExcluded checks if path matches any of the configured exclude patterns.
|
||||
@@ -22,19 +26,38 @@ func isExcluded(path string) bool {
|
||||
}
|
||||
|
||||
// GetFilesFromDirectories makes a file list of given directories.
|
||||
func GetFilesFromDirectories(directories []string) ([]string, error) {
|
||||
// Logs errors for directories that don't exist or can't be accessed but continues with valid directories.
|
||||
// Takes context with timeout from config dir_scan_timeout setting.
|
||||
func GetFilesFromDirectories(ctx context.Context, directories []string) ([]string, error) {
|
||||
var files []string
|
||||
|
||||
for _, dir := range directories {
|
||||
// Check directory exists before walking.
|
||||
if _, err := os.Stat(dir); err != nil {
|
||||
if errors.Is(err, os.ErrNotExist) {
|
||||
backerlog.Errorf("Configured backup directory not found: %s", dir)
|
||||
} else {
|
||||
backerlog.Errorf("Failed to access configured directory %s: %v", dir, err)
|
||||
}
|
||||
|
||||
continue
|
||||
}
|
||||
|
||||
err := filepath.WalkDir(dir, func(path string, de fs.DirEntry, err error) error { //nolint: revive
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return ctx.Err()
|
||||
default:
|
||||
}
|
||||
|
||||
if err != nil {
|
||||
log.Warnf("Skipping %s: %v", path, err)
|
||||
backerlog.Warnf("Skipping %s: %v", path, err)
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
if isExcluded(path) {
|
||||
log.Debugf("Excluding: %s", path)
|
||||
backerlog.Debugf("Excluding: %s", path)
|
||||
|
||||
return nil
|
||||
}
|
||||
@@ -45,11 +68,26 @@ func GetFilesFromDirectories(directories []string) ([]string, error) {
|
||||
})
|
||||
|
||||
if err != nil {
|
||||
log.Warnf("Failed to walk directory %s: %v", dir, err)
|
||||
if errors.Is(err, context.DeadlineExceeded) {
|
||||
backerlog.Errorf("Directory scan timed out after scanning %d files", len(files))
|
||||
|
||||
return files, ErrDirectoryScanTimeout{ScannedFiles: len(files)}
|
||||
}
|
||||
|
||||
backerlog.Warnf("Failed to walk directory %s: %v", dir, err)
|
||||
}
|
||||
}
|
||||
|
||||
return files, nil
|
||||
}
|
||||
|
||||
// GetFilesFromDirectoriesWithTimeout makes a file list of given directories with timeout.
|
||||
// Convenience wrapper that creates context from timeout in minutes.
|
||||
func GetFilesFromDirectoriesWithTimeout(timeoutMinutes int, directories []string) ([]string, error) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), time.Duration(timeoutMinutes)*time.Minute)
|
||||
defer cancel()
|
||||
|
||||
return GetFilesFromDirectories(ctx, directories)
|
||||
}
|
||||
|
||||
/* vim: setlocal ft=go noet ai ts=4 sw=4 sts=4: */
|
||||
|
||||
@@ -1,7 +1,9 @@
|
||||
package backer
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"slices"
|
||||
"strings"
|
||||
@@ -15,28 +17,36 @@ func normalizePath(p string) string {
|
||||
|
||||
// TestGetFilesFromDirectories checks if GetFilesFromDirectories returns the proper amount of strings, correct strings and no error.
|
||||
func TestGetFilesFromDirectories(t *testing.T) {
|
||||
// Ensure empty_dir exists for the test.
|
||||
os.MkdirAll("../../test_data/test1/foo/empty_dir", 0755)
|
||||
testDataPath := filepath.Join("..", "test_data")
|
||||
|
||||
// Log the paths we're checking for debugging
|
||||
t.Logf("Checking directories: %v", []string{
|
||||
filepath.Join(testDataPath, "test1", "foo"),
|
||||
filepath.Join(testDataPath, "test1", "bar"),
|
||||
})
|
||||
|
||||
var (
|
||||
input = Config{
|
||||
Directories: []string{"../../test_data/test1/foo", "../../test_data/test1/bar"},
|
||||
Directories: []string{
|
||||
filepath.Join(testDataPath, "test1", "foo"),
|
||||
filepath.Join(testDataPath, "test1", "bar"),
|
||||
},
|
||||
}
|
||||
expectedData = []string{
|
||||
"../../test_data/test1/foo",
|
||||
"../../test_data/test1/foo/empty_dir",
|
||||
"../../test_data/test1/foo/hello_breakfast.txt",
|
||||
"../../test_data/test1/foo/mydir",
|
||||
"../../test_data/test1/foo/mydir/myfile.txt",
|
||||
"../../test_data/test1/foo/some_text.txt",
|
||||
"../../test_data/test1/bar",
|
||||
"../../test_data/test1/bar/goodbye.txt",
|
||||
"../../test_data/test1/bar/goodbye.txt/text.txt",
|
||||
"../../test_data/test1/bar/hello.txt",
|
||||
filepath.Join(testDataPath, "test1", "foo"),
|
||||
filepath.Join(testDataPath, "test1", "foo", "empty_dir"),
|
||||
filepath.Join(testDataPath, "test1", "foo", "hello_breakfast.txt"),
|
||||
filepath.Join(testDataPath, "test1", "foo", "mydir"),
|
||||
filepath.Join(testDataPath, "test1", "foo", "mydir", "myfile.txt"),
|
||||
filepath.Join(testDataPath, "test1", "foo", "some_text.txt"),
|
||||
filepath.Join(testDataPath, "test1", "bar"),
|
||||
filepath.Join(testDataPath, "test1", "bar", "goodbye.txt"),
|
||||
filepath.Join(testDataPath, "test1", "bar", "goodbye.txt", "text.txt"),
|
||||
filepath.Join(testDataPath, "test1", "bar", "hello.txt"),
|
||||
}
|
||||
)
|
||||
|
||||
output, err := GetFilesFromDirectories(input.Directories)
|
||||
output, err := GetFilesFromDirectories(context.Background(), input.Directories)
|
||||
|
||||
if err != nil {
|
||||
t.Errorf("Expected no error, got %v", err)
|
||||
@@ -59,7 +69,8 @@ func TestGetFilesFromDirectories(t *testing.T) {
|
||||
|
||||
// TestGetFilesFromDirectoriesSingleDir tests with a single directory.
|
||||
func TestGetFilesFromDirectoriesSingleDir(t *testing.T) {
|
||||
output, err := GetFilesFromDirectories([]string{"../../test_data/test1/foo"})
|
||||
testDataPath := filepath.Join("test_data")
|
||||
output, err := GetFilesFromDirectories(context.Background(), []string{filepath.Join(testDataPath, "test1", "foo")})
|
||||
|
||||
if err != nil {
|
||||
t.Errorf("Expected no error, got %v", err)
|
||||
@@ -70,7 +81,7 @@ func TestGetFilesFromDirectoriesSingleDir(t *testing.T) {
|
||||
}
|
||||
|
||||
// Should contain the directory itself
|
||||
if !slices.Contains(output, "../../test_data/test1/foo") {
|
||||
if !slices.Contains(output, filepath.Join(testDataPath, "test1", "foo")) {
|
||||
t.Error("Output should contain the root directory")
|
||||
}
|
||||
}
|
||||
@@ -80,7 +91,7 @@ func TestGetFilesFromDirectoriesEmptyDir(t *testing.T) {
|
||||
// Create temp empty dir
|
||||
tmpDir := t.TempDir()
|
||||
|
||||
output, err := GetFilesFromDirectories([]string{tmpDir})
|
||||
output, err := GetFilesFromDirectories(context.Background(), []string{tmpDir})
|
||||
|
||||
if err != nil {
|
||||
t.Errorf("Expected no error, got %v", err)
|
||||
@@ -93,23 +104,48 @@ func TestGetFilesFromDirectoriesEmptyDir(t *testing.T) {
|
||||
}
|
||||
|
||||
// TestGetFilesFromDirectoriesNonExistent tests with non-existent directory.
|
||||
// Should log error but not fail - returns empty list.
|
||||
func TestGetFilesFromDirectoriesNonExistent(t *testing.T) {
|
||||
output, err := GetFilesFromDirectories([]string{"/nonexistent/path/that/does/not/exist"})
|
||||
testDataPath := filepath.Join("test_data")
|
||||
nonExistentPath := filepath.Join(testDataPath, "nonexistent", "path", "that", "does", "not", "exist")
|
||||
output, err := GetFilesFromDirectories(context.Background(), []string{nonExistentPath})
|
||||
|
||||
// Should not return error, but logs error
|
||||
if err != nil {
|
||||
t.Errorf("Expected no error (logs warning instead), got %v", err)
|
||||
t.Errorf("Expected no error (logs error instead), got %v", err)
|
||||
}
|
||||
|
||||
// Should return empty list
|
||||
if len(output) != 0 {
|
||||
t.Errorf("Expected empty output for non-existent directory, got %d files", len(output))
|
||||
}
|
||||
}
|
||||
|
||||
// TestGetFilesFromDirectoriesPartialFailure tests with mix of valid and invalid directories.
|
||||
func TestGetFilesFromDirectoriesPartialFailure(t *testing.T) {
|
||||
testDataPath := filepath.Join("test_data")
|
||||
// Create a temp directory that exists
|
||||
tmpDir := t.TempDir()
|
||||
|
||||
output, err := GetFilesFromDirectories(context.Background(), []string{tmpDir, filepath.Join(testDataPath, "nonexistent", "path", "that", "does", "not", "exist")})
|
||||
|
||||
// Should not return error, but logs error
|
||||
if err != nil {
|
||||
t.Errorf("Expected no error (logs error instead), got %v", err)
|
||||
}
|
||||
|
||||
// Should return files from valid directory
|
||||
if len(output) == 0 {
|
||||
t.Error("Expected files from valid directory")
|
||||
}
|
||||
}
|
||||
|
||||
// TestGetFilesFromDirectoriesMultipleDirs tests multiple directories.
|
||||
func TestGetFilesFromDirectoriesMultipleDirs(t *testing.T) {
|
||||
output, err := GetFilesFromDirectories([]string{
|
||||
"../../test_data/test1/foo",
|
||||
"../../test_data/test1/bar",
|
||||
testDataPath := filepath.Join("test_data")
|
||||
output, err := GetFilesFromDirectories(context.Background(), []string{
|
||||
filepath.Join(testDataPath, "test1", "foo"),
|
||||
filepath.Join(testDataPath, "test1", "bar"),
|
||||
})
|
||||
|
||||
if err != nil {
|
||||
@@ -118,10 +154,10 @@ func TestGetFilesFromDirectoriesMultipleDirs(t *testing.T) {
|
||||
|
||||
// Should contain entries from both directories
|
||||
hasFoo := slices.ContainsFunc(output, func(s string) bool {
|
||||
return s == "../../test_data/test1/foo" || s == "../../test_data/test1/foo/hello_breakfast.txt"
|
||||
return s == filepath.Join(testDataPath, "test1", "foo") || s == filepath.Join(testDataPath, "test1", "foo", "hello_breakfast.txt")
|
||||
})
|
||||
hasBar := slices.ContainsFunc(output, func(s string) bool {
|
||||
return s == "../../test_data/test1/bar" || s == "../../test_data/test1/bar/hello.txt"
|
||||
return s == filepath.Join(testDataPath, "test1", "bar") || s == filepath.Join(testDataPath, "test1", "bar", "hello.txt")
|
||||
})
|
||||
|
||||
if !hasFoo {
|
||||
@@ -149,7 +185,7 @@ func TestGetFilesFromDirectoriesSymlink(t *testing.T) {
|
||||
t.Skip("Symlinks not supported, skipping test")
|
||||
}
|
||||
|
||||
output, err := GetFilesFromDirectories([]string{tmpDir})
|
||||
output, err := GetFilesFromDirectories(context.Background(), []string{tmpDir})
|
||||
|
||||
if err != nil {
|
||||
t.Errorf("Expected no error, got %v", err)
|
||||
@@ -228,7 +264,7 @@ func TestGetFilesFromDirectoriesWithExcludes(t *testing.T) {
|
||||
regexp.MustCompile(`\.tmp$`),
|
||||
}
|
||||
|
||||
output, err := GetFilesFromDirectories([]string{tmpDir})
|
||||
output, err := GetFilesFromDirectories(context.Background(), []string{tmpDir})
|
||||
if err != nil {
|
||||
t.Fatalf("Unexpected error: %v", err)
|
||||
}
|
||||
|
||||
@@ -95,6 +95,17 @@ func LoadConfig(path string) error {
|
||||
return fmt.Errorf("Config option backup_timeout must be between %d and %d minutes, got %d", backupTimeoutMin, backupTimeoutMax, C.BackupTimeout)
|
||||
}
|
||||
|
||||
// DirScanTimeout: set default, then validate range.
|
||||
if C.DirScanTimeout == 0 {
|
||||
C.DirScanTimeout = defaultDirScanTimeout
|
||||
|
||||
log.Warnf("Config option dir_scan_timeout is not set, fallback to %d minutes", C.DirScanTimeout)
|
||||
}
|
||||
|
||||
if C.DirScanTimeout < dirScanTimeoutMin || C.DirScanTimeout > dirScanTimeoutMax {
|
||||
return fmt.Errorf("Config option dir_scan_timeout must be between %d and %d minutes, got %d", dirScanTimeoutMin, dirScanTimeoutMax, C.DirScanTimeout)
|
||||
}
|
||||
|
||||
// CompressionLevel: set default, then validate range.
|
||||
if C.CompressionLevel == 0 {
|
||||
C.CompressionLevel = defaultCompressionLevel
|
||||
@@ -146,7 +157,7 @@ func LoadConfig(path string) error {
|
||||
|
||||
switch C.DefaultCompression {
|
||||
case "gzip", "pgzip", "bzip2", "xz", "lz4", "zstd":
|
||||
log.Infof("Using %s as defailt compression algorithm", C.DefaultCompression)
|
||||
// No-op: compression algorithm validated.
|
||||
default:
|
||||
return fmt.Errorf("Config option default_compression must be gzip, bzip2, zstd, lz4, xz or pgzip, got %s", C.DefaultCompression)
|
||||
}
|
||||
|
||||
@@ -1,7 +1,9 @@
|
||||
package backer
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"slices"
|
||||
"testing"
|
||||
)
|
||||
@@ -10,18 +12,46 @@ func TestLoadConfig(t *testing.T) {
|
||||
// Reset global config
|
||||
C = Config{}
|
||||
|
||||
var (
|
||||
testConfigPath = "./../../test_data/test_config.json"
|
||||
expectedData = Config{
|
||||
Address: "127.0.0.1",
|
||||
Port: 8086,
|
||||
Directories: []string{
|
||||
"../../test_data/test1/foo",
|
||||
"../../test_data/test1/bar",
|
||||
},
|
||||
Location: "/archive",
|
||||
}
|
||||
)
|
||||
testDataPath := filepath.Join(t.TempDir(), "test_data")
|
||||
os.MkdirAll(filepath.Join(testDataPath, "test1", "foo"), 0755)
|
||||
os.MkdirAll(filepath.Join(testDataPath, "test1", "bar"), 0755)
|
||||
|
||||
// Create test config file
|
||||
testConfigPath := filepath.Join(testDataPath, "test_config.json")
|
||||
testStruct := struct {
|
||||
Address string `json:"address"`
|
||||
Port int `json:"port"`
|
||||
Directories []string `json:"directories"`
|
||||
Location string `json:"location"`
|
||||
User string `json:"user"`
|
||||
Password string `json:"password"`
|
||||
NoHTTPS bool `json:"nohttps"`
|
||||
}{
|
||||
Address: "127.0.0.1",
|
||||
Port: 8086,
|
||||
Directories: []string{filepath.Join(testDataPath, "test1", "foo"), filepath.Join(testDataPath, "test1", "bar")},
|
||||
Location: "/archive",
|
||||
User: "test",
|
||||
Password: "test",
|
||||
NoHTTPS: true,
|
||||
}
|
||||
testConfigBytes, _ := json.MarshalIndent(testStruct, "", " ")
|
||||
if err := os.WriteFile(testConfigPath, testConfigBytes, 0644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
expectedData := Config{
|
||||
Address: "127.0.0.1",
|
||||
Port: 8086,
|
||||
Directories: []string{
|
||||
filepath.Join(testDataPath, "test1", "foo"),
|
||||
filepath.Join(testDataPath, "test1", "bar"),
|
||||
},
|
||||
Location: "/archive",
|
||||
User: "test",
|
||||
Password: "test",
|
||||
NoHTTPS: true,
|
||||
}
|
||||
|
||||
err := LoadConfig(testConfigPath)
|
||||
|
||||
@@ -57,6 +87,8 @@ func TestLoadConfig(t *testing.T) {
|
||||
|
||||
// TestLoadConfigMissingUser tests that config fails when user is missing.
|
||||
func TestLoadConfigMissingUser(t *testing.T) {
|
||||
// Reset global config
|
||||
C = Config{}
|
||||
config := `{
|
||||
"port": 8086,
|
||||
"password": "test",
|
||||
@@ -72,6 +104,8 @@ func TestLoadConfigMissingUser(t *testing.T) {
|
||||
|
||||
// TestLoadConfigMissingPassword tests that config fails when password is missing.
|
||||
func TestLoadConfigMissingPassword(t *testing.T) {
|
||||
// Reset global config
|
||||
C = Config{}
|
||||
config := `{
|
||||
"port": 8086,
|
||||
"user": "test",
|
||||
@@ -87,6 +121,8 @@ func TestLoadConfigMissingPassword(t *testing.T) {
|
||||
|
||||
// TestLoadConfigMissingDirectories tests that config fails when directories is missing.
|
||||
func TestLoadConfigMissingDirectories(t *testing.T) {
|
||||
// Reset global config
|
||||
C = Config{}
|
||||
config := `{
|
||||
"port": 8086,
|
||||
"user": "test",
|
||||
@@ -102,6 +138,8 @@ func TestLoadConfigMissingDirectories(t *testing.T) {
|
||||
|
||||
// TestLoadConfigInvalidPort tests that config fails with invalid port.
|
||||
func TestLoadConfigInvalidPort(t *testing.T) {
|
||||
// Reset global config
|
||||
C = Config{}
|
||||
config := `{
|
||||
"port": 99999,
|
||||
"user": "test",
|
||||
@@ -118,6 +156,8 @@ func TestLoadConfigInvalidPort(t *testing.T) {
|
||||
|
||||
// TestLoadConfigNonExistentDirectory tests that config fails with non-existent directory.
|
||||
func TestLoadConfigNonExistentDirectory(t *testing.T) {
|
||||
// Reset global config
|
||||
C = Config{}
|
||||
config := `{
|
||||
"port": 8086,
|
||||
"user": "test",
|
||||
@@ -134,6 +174,8 @@ func TestLoadConfigNonExistentDirectory(t *testing.T) {
|
||||
|
||||
// TestLoadConfigHTTPSWithoutCert tests that config fails when HTTPS enabled but no cert.
|
||||
func TestLoadConfigHTTPSWithoutCert(t *testing.T) {
|
||||
// Reset global config
|
||||
C = Config{}
|
||||
config := `{
|
||||
"port": 8086,
|
||||
"user": "test",
|
||||
@@ -151,6 +193,8 @@ func TestLoadConfigHTTPSWithoutCert(t *testing.T) {
|
||||
|
||||
// TestLoadConfigNoHTTPS tests that config succeeds with nohttps=true and no cert.
|
||||
func TestLoadConfigNoHTTPS(t *testing.T) {
|
||||
// Reset global config
|
||||
C = Config{}
|
||||
config := `{
|
||||
"port": 8086,
|
||||
"user": "test",
|
||||
@@ -172,6 +216,8 @@ func TestLoadConfigNoHTTPS(t *testing.T) {
|
||||
|
||||
// TestLoadConfigFilenamePrefixCustom tests custom filename prefix.
|
||||
func TestLoadConfigFilenamePrefixCustom(t *testing.T) {
|
||||
// Reset global config
|
||||
C = Config{}
|
||||
config := `{
|
||||
"user": "test",
|
||||
"password": "test",
|
||||
@@ -193,6 +239,8 @@ func TestLoadConfigFilenamePrefixCustom(t *testing.T) {
|
||||
|
||||
// TestLoadConfigDefaultValues tests default values are set correctly.
|
||||
func TestLoadConfigDefaultValues(t *testing.T) {
|
||||
// Reset global config
|
||||
C = Config{}
|
||||
config := `{
|
||||
"user": "test",
|
||||
"password": "test",
|
||||
@@ -234,6 +282,8 @@ func TestLoadConfigDefaultValues(t *testing.T) {
|
||||
|
||||
// TestLoadConfigCompressionAlgorithmBzip2 tests bzip2 compression algorithm.
|
||||
func TestLoadConfigCompressionAlgorithmBzip2(t *testing.T) {
|
||||
// Reset global config
|
||||
C = Config{}
|
||||
config := `{
|
||||
"user": "test",
|
||||
"password": "test",
|
||||
@@ -255,6 +305,8 @@ func TestLoadConfigCompressionAlgorithmBzip2(t *testing.T) {
|
||||
|
||||
// TestLoadConfigCompressionAlgorithmZstd tests zstd compression algorithm.
|
||||
func TestLoadConfigCompressionAlgorithmZstd(t *testing.T) {
|
||||
// Reset global config
|
||||
C = Config{}
|
||||
config := `{
|
||||
"user": "test",
|
||||
"password": "test",
|
||||
@@ -276,6 +328,8 @@ func TestLoadConfigCompressionAlgorithmZstd(t *testing.T) {
|
||||
|
||||
// TestLoadConfigCompressionAlgorithmLz4 tests lz4 compression algorithm.
|
||||
func TestLoadConfigCompressionAlgorithmLz4(t *testing.T) {
|
||||
// Reset global config
|
||||
C = Config{}
|
||||
config := `{
|
||||
"user": "test",
|
||||
"password": "test",
|
||||
@@ -297,6 +351,8 @@ func TestLoadConfigCompressionAlgorithmLz4(t *testing.T) {
|
||||
|
||||
// TestLoadConfigCompressionAlgorithmXz tests xz compression algorithm.
|
||||
func TestLoadConfigCompressionAlgorithmXz(t *testing.T) {
|
||||
// Reset global config
|
||||
C = Config{}
|
||||
config := `{
|
||||
"user": "test",
|
||||
"password": "test",
|
||||
@@ -318,6 +374,8 @@ func TestLoadConfigCompressionAlgorithmXz(t *testing.T) {
|
||||
|
||||
// TestLoadConfigCompressionAlgorithmPgzip tests pgzip compression algorithm.
|
||||
func TestLoadConfigCompressionAlgorithmPgzip(t *testing.T) {
|
||||
// Reset global config
|
||||
C = Config{}
|
||||
config := `{
|
||||
"user": "test",
|
||||
"password": "test",
|
||||
@@ -339,6 +397,8 @@ func TestLoadConfigCompressionAlgorithmPgzip(t *testing.T) {
|
||||
|
||||
// TestLoadConfigCompressionAlgorithmInvalid tests invalid compression algorithm.
|
||||
func TestLoadConfigCompressionAlgorithmInvalid(t *testing.T) {
|
||||
// Reset global config
|
||||
C = Config{}
|
||||
config := `{
|
||||
"user": "test",
|
||||
"password": "test",
|
||||
@@ -367,7 +427,6 @@ func writeTempConfig(t *testing.T, content string) string {
|
||||
tmpFile.Close()
|
||||
// Reset global config before loading
|
||||
C = Config{}
|
||||
excludePatterns = nil
|
||||
return tmpFile.Name()
|
||||
}
|
||||
|
||||
|
||||
@@ -10,8 +10,26 @@ import (
|
||||
// CreateTarLz4Stream takes a list of files and returns a reader that reads from an lz4-compressed tar archive.
|
||||
func CreateTarLz4Stream(ctx context.Context, filepaths []string) io.ReadCloser {
|
||||
return createArchiveStream(ctx, filepaths, func(w io.Writer) (io.WriteCloser, error) {
|
||||
return lz4.NewWriter(w), nil
|
||||
lz4Writer := lz4.NewWriter(w)
|
||||
// Map compression_level (1-9) to lz4 compression level.
|
||||
// 1-3: fast, 4-6: balanced, 7-9: better compression.
|
||||
lz4Writer.Header.CompressionLevel = mapCompressionLevelToLz4(C.CompressionLevel) //nolint: staticcheck
|
||||
|
||||
return lz4Writer, nil
|
||||
})
|
||||
}
|
||||
|
||||
// mapCompressionLevelToLz4 maps 1-9 compression level to lz4 compression level.
|
||||
// 1-3: fast, 4-6: balanced, 7-9: better compression.
|
||||
func mapCompressionLevelToLz4(level int) int {
|
||||
switch {
|
||||
case level <= 3:
|
||||
return 0 // Fastest.
|
||||
case level <= 6:
|
||||
return 1 // Default.
|
||||
default:
|
||||
return 2 // Best compression.
|
||||
}
|
||||
}
|
||||
|
||||
/* vim: setlocal ft=go noet ai ts=4 sw=4 sts=4: */
|
||||
|
||||
@@ -2,7 +2,11 @@ package backer
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/md5"
|
||||
"crypto/subtle"
|
||||
"crypto/tls"
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net"
|
||||
@@ -14,6 +18,36 @@ import (
|
||||
"backer/internal/log"
|
||||
)
|
||||
|
||||
// isTLSError checks if the given error is TLS-related.
|
||||
// It checks for both TLS-specific error types and error messages containing "tls:".
|
||||
func isTLSError(err error) bool {
|
||||
if err == nil {
|
||||
return false
|
||||
}
|
||||
|
||||
var alertErr *tls.AlertError
|
||||
if errors.As(err, &alertErr) && alertErr != nil {
|
||||
return true
|
||||
}
|
||||
|
||||
var certErr *tls.CertificateVerificationError
|
||||
if errors.As(err, &certErr) && certErr != nil {
|
||||
return true
|
||||
}
|
||||
|
||||
var echErr *tls.ECHRejectionError
|
||||
if errors.As(err, &echErr) && echErr != nil {
|
||||
return true
|
||||
}
|
||||
|
||||
var recordErr *tls.RecordHeaderError
|
||||
if errors.As(err, &recordErr) && recordErr != nil {
|
||||
return true
|
||||
}
|
||||
|
||||
return strings.HasPrefix(err.Error(), "tls:")
|
||||
}
|
||||
|
||||
// withServerHeader wraps a handler to add Server header to all responses.
|
||||
func withServerHeader(next http.Handler) http.Handler {
|
||||
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
@@ -23,7 +57,8 @@ func withServerHeader(next http.Handler) http.Handler {
|
||||
}
|
||||
|
||||
// NewServer starts new server instance with given config file.
|
||||
func NewServer(configPath string) (*http.Server, error) {
|
||||
// It returns a ServerWrapper that intercepts and classifies server errors.
|
||||
func NewServer(configPath string) (*ServerWrapper, error) {
|
||||
_ = log.Init("info", "") // Initialize logger with defaults for early logging.
|
||||
|
||||
if err := LoadConfig(configPath); err != nil {
|
||||
@@ -34,9 +69,11 @@ func NewServer(configPath string) (*http.Server, error) {
|
||||
log.Errorf("Failed to initialize logger: %v", err)
|
||||
}
|
||||
|
||||
log.Infof("Using %s as default compression algorithm", C.DefaultCompression)
|
||||
|
||||
mux := http.NewServeMux()
|
||||
|
||||
mux.HandleFunc(C.Location, func(w http.ResponseWriter, r *http.Request) {
|
||||
backupHandler := func(w http.ResponseWriter, r *http.Request) {
|
||||
username, password, ok := r.BasicAuth()
|
||||
|
||||
// No auth? Good bye.
|
||||
@@ -44,16 +81,18 @@ func NewServer(configPath string) (*http.Server, error) {
|
||||
w.Header().Set("WWW-Authenticate", `Basic realm="restricted", charset="UTF-8"`)
|
||||
http.Error(w, "Unauthorized", http.StatusUnauthorized)
|
||||
|
||||
log.Infof("Access denied from %s to %s (no username or password given)", r.RemoteAddr, r.RequestURI)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
// Both username AND password must match. Explicitly require that the whole condition
|
||||
// fails when either is incorrect, per the authentication design.
|
||||
//nolint:staticcheck // QF1001: De Morgan's law is irrelevant here.
|
||||
if !(username == C.User && password == C.Password) {
|
||||
// Both username AND password must match. Use timing-safe comparison to prevent timing attacks.
|
||||
if !timingSafeMatch(username, C.User) || !timingSafeMatch(password, C.Password) {
|
||||
w.Header().Set("WWW-Authenticate", `Basic realm="restricted", charset="UTF-8"`)
|
||||
http.Error(w, "Unauthorized", http.StatusUnauthorized)
|
||||
|
||||
log.Infof("Access denied from %s to %s (username and/or password incorrect)", r.RemoteAddr, r.RequestURI)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
@@ -62,25 +101,41 @@ func NewServer(configPath string) (*http.Server, error) {
|
||||
|
||||
log.Infof("Backup started: client=%s user=%s", clientIP, username)
|
||||
|
||||
ctx := r.Context()
|
||||
files, err := GetFilesFromDirectories(C.Directories)
|
||||
// Create context with dir_scan_timeout from request context.
|
||||
ctx, cancel := context.WithTimeout(r.Context(), time.Duration(C.DirScanTimeout)*time.Minute)
|
||||
defer cancel()
|
||||
|
||||
files, err := GetFilesFromDirectories(ctx, C.Directories)
|
||||
|
||||
if err != nil {
|
||||
log.Errorf("Failed to get files: %v", err)
|
||||
switch {
|
||||
case errors.Is(err, context.DeadlineExceeded):
|
||||
w.Header().Set("Server", "backer")
|
||||
http.Error(w, "Directory scan timed out", http.StatusRequestTimeout)
|
||||
|
||||
w.Header().Set("Server", "backer")
|
||||
http.Error(w, "Failed to get files", http.StatusInternalServerError)
|
||||
log.Errorf("Directory scan timed out: %v", err)
|
||||
|
||||
default:
|
||||
log.Errorf("Failed to get files: %v", err)
|
||||
|
||||
w.Header().Set("Server", "backer")
|
||||
http.Error(w, "Failed to get files", http.StatusInternalServerError)
|
||||
}
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
if len(files) == 0 {
|
||||
log.Warnf("No files found to backup in configured directories")
|
||||
}
|
||||
|
||||
var (
|
||||
archive io.ReadCloser
|
||||
extension string
|
||||
algorithm string
|
||||
)
|
||||
|
||||
requestedExt := strings.TrimPrefix(path.Ext(r.URL.Path), ".")
|
||||
requestedExt := r.URL.Path
|
||||
|
||||
algorithm = getCompressionAlgorithm(requestedExt)
|
||||
|
||||
@@ -91,7 +146,11 @@ func NewServer(configPath string) (*http.Server, error) {
|
||||
|
||||
case "zstd":
|
||||
archive = CreateTarZstdStream(ctx, files)
|
||||
extension = "tar.zst"
|
||||
if strings.HasSuffix(requestedExt, ".zst") {
|
||||
extension = "tar.zst"
|
||||
} else {
|
||||
extension = "tar.zstd"
|
||||
}
|
||||
|
||||
case "lz4":
|
||||
archive = CreateTarLz4Stream(ctx, files)
|
||||
@@ -116,44 +175,80 @@ func NewServer(configPath string) (*http.Server, error) {
|
||||
}
|
||||
}()
|
||||
|
||||
timestamp := time.Now().Format("20060102-150405")
|
||||
timestamp := backupStart.Format("20060102-150405")
|
||||
filename := fmt.Sprintf("%s-%s.%s", C.FilenamePrefix, timestamp, extension)
|
||||
|
||||
w.Header().Set("Content-Disposition", "attachment; filename="+filename)
|
||||
w.Header().Set("Content-Type", "application/octet-stream")
|
||||
|
||||
bytesWritten, err := io.Copy(w, archive)
|
||||
duration := time.Since(backupStart)
|
||||
|
||||
bytesWritten, md5Hash, err := copyWithContext(ctx, w, archive)
|
||||
if err != nil {
|
||||
duration := time.Since(backupStart)
|
||||
|
||||
if errors.Is(err, context.Canceled) {
|
||||
log.Infof("Backup canceled: client=%s user=%s duration=%s", clientIP, username, duration)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
if isPipeClosedError(err) {
|
||||
log.Infof("Client disconnected during backup: client=%s user=%s", clientIP, username)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
log.Errorf("Backup failed: client=%s user=%s duration=%s error=%v", clientIP, username, duration, err)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
log.Infof("Backup completed: client=%s user=%s files=%d bytes=%d duration=%s", clientIP, username, len(files), bytesWritten, duration)
|
||||
})
|
||||
log.Infof("Backup completed: client=%s user=%s files=%d bytes=%d md5=%s duration=%s", clientIP, username, len(files), bytesWritten, md5Hash, time.Since(backupStart))
|
||||
}
|
||||
|
||||
mux.Handle(C.Location, http.HandlerFunc(backupHandler))
|
||||
|
||||
extensions := []string{".tar.gz", ".tar.xz", ".tar.bz2", ".tar.lz4", ".tar.zst", ".tar.zstd"}
|
||||
for _, ext := range extensions {
|
||||
muxPath := C.Location + ext
|
||||
mux.Handle(muxPath, http.HandlerFunc(backupHandler))
|
||||
}
|
||||
|
||||
// HTTP mode if nohttps is enabled in config.
|
||||
// HTTPS mode if nohttps is disabled in config.
|
||||
if C.NoHTTPS {
|
||||
return &http.Server{
|
||||
srv := &ServerWrapper{
|
||||
Server: &http.Server{
|
||||
Addr: fmt.Sprintf("%s:%d", C.Address, C.Port),
|
||||
Handler: withServerHeader(mux),
|
||||
ReadHeaderTimeout: readHeaderTimeout,
|
||||
WriteTimeout: time.Duration(C.BackupTimeout) * time.Minute,
|
||||
IdleTimeout: idleTimeout,
|
||||
MaxHeaderBytes: maxHeaderBytes,
|
||||
ErrorLog: log.DebugLogger(),
|
||||
},
|
||||
}
|
||||
srv.SetKeepAlivesEnabled(false)
|
||||
|
||||
return srv, nil
|
||||
}
|
||||
|
||||
srv := &ServerWrapper{
|
||||
Server: &http.Server{
|
||||
Addr: fmt.Sprintf("%s:%d", C.Address, C.Port),
|
||||
Handler: withServerHeader(mux),
|
||||
ReadHeaderTimeout: readHeaderTimeout,
|
||||
WriteTimeout: time.Duration(C.BackupTimeout) * time.Minute,
|
||||
}, nil
|
||||
}
|
||||
|
||||
return &http.Server{
|
||||
Addr: fmt.Sprintf("%s:%d", C.Address, C.Port),
|
||||
Handler: withServerHeader(mux),
|
||||
ReadHeaderTimeout: readHeaderTimeout,
|
||||
WriteTimeout: time.Duration(C.BackupTimeout) * time.Minute,
|
||||
TLSConfig: &tls.Config{
|
||||
MinVersion: tls.VersionTLS13,
|
||||
IdleTimeout: idleTimeout,
|
||||
MaxHeaderBytes: maxHeaderBytes,
|
||||
TLSConfig: &tls.Config{
|
||||
MinVersion: tls.VersionTLS13,
|
||||
},
|
||||
ErrorLog: log.DebugLogger(),
|
||||
},
|
||||
}, nil
|
||||
}
|
||||
srv.SetKeepAlivesEnabled(false)
|
||||
|
||||
return srv, nil
|
||||
}
|
||||
|
||||
// writeWithContext checks if context is canceled before executing fn.
|
||||
@@ -169,31 +264,50 @@ func writeWithContext(ctx context.Context, fn func() error) error {
|
||||
}
|
||||
|
||||
// copyWithContext copies data from src to dst using a 32KB buffer, respecting
|
||||
// context cancellation. If the context is canceled, the copy stops and returns
|
||||
// ctx.Err().
|
||||
func copyWithContext(ctx context.Context, dst io.Writer, src io.Reader) error {
|
||||
// context cancellation. Returns bytes written, MD5 hash (hex string), and any error encountered.
|
||||
//
|
||||
//nolint:gosec
|
||||
func copyWithContext(ctx context.Context, dst io.Writer, src io.Reader) (int64, string, error) {
|
||||
buf := make([]byte, copyBufferSize)
|
||||
|
||||
var bytesWritten int64
|
||||
|
||||
hasher := md5.New()
|
||||
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return ctx.Err()
|
||||
return bytesWritten, "", ctx.Err()
|
||||
default:
|
||||
}
|
||||
|
||||
n, readErr := src.Read(buf)
|
||||
if n > 0 {
|
||||
if _, werr := dst.Write(buf[:n]); werr != nil {
|
||||
return werr
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return bytesWritten, "", ctx.Err()
|
||||
default:
|
||||
}
|
||||
|
||||
if _, werr := hasher.Write(buf[:n]); werr != nil {
|
||||
return bytesWritten, "", werr
|
||||
}
|
||||
|
||||
if _, werr := dst.Write(buf[:n]); werr != nil {
|
||||
return bytesWritten, "", werr
|
||||
}
|
||||
|
||||
bytesWritten += int64(n)
|
||||
}
|
||||
|
||||
if readErr != nil {
|
||||
if readErr == io.EOF {
|
||||
return nil
|
||||
hash := hex.EncodeToString(hasher.Sum(nil))
|
||||
|
||||
return bytesWritten, hash, nil
|
||||
}
|
||||
|
||||
return readErr
|
||||
return bytesWritten, "", readErr
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -216,21 +330,87 @@ func getClientIP(r *http.Request) string {
|
||||
// getCompressionAlgorithm determines which compression algorithm to use based on the requested extension.
|
||||
// If the extension is recognized, it returns the corresponding algorithm; otherwise, it returns the default.
|
||||
// For .tar.gz extension, pgzip is always used for parallel compression.
|
||||
func getCompressionAlgorithm(requestedExt string) string {
|
||||
switch requestedExt {
|
||||
case "tar.bz2":
|
||||
// The input can be either a full path like "/archive.tar.xz" or just the extension like "tar.xz" or "xz".
|
||||
func getCompressionAlgorithm(input string) string {
|
||||
// Extract extension from full path (e.g., "/archive.tar.xz" -> ".xz").
|
||||
// or use the input directly if it's just an extension (e.g., "tar.xz" -> ".xz").
|
||||
ext := path.Ext(input)
|
||||
if ext == "" {
|
||||
ext = "." + input
|
||||
}
|
||||
|
||||
switch ext {
|
||||
case ".tar.bz2":
|
||||
return "bzip2"
|
||||
case "tar.zst":
|
||||
case ".tar.zstd":
|
||||
return "zstd"
|
||||
case "tar.lz4":
|
||||
case ".tar.zst":
|
||||
return "zstd"
|
||||
case ".tar.lz4":
|
||||
return "lz4"
|
||||
case "tar.xz":
|
||||
case ".tar.xz":
|
||||
return "xz"
|
||||
case "tar.gz":
|
||||
case ".tar.gz":
|
||||
return "pgzip"
|
||||
case ".xz":
|
||||
return "xz"
|
||||
case ".bz2":
|
||||
return "bzip2"
|
||||
case ".zst":
|
||||
return "zstd"
|
||||
case ".zstd":
|
||||
return "zstd"
|
||||
case ".lz4":
|
||||
return "lz4"
|
||||
case ".gz":
|
||||
return "pgzip"
|
||||
default:
|
||||
return C.DefaultCompression
|
||||
}
|
||||
}
|
||||
|
||||
type (
|
||||
// ServerWrapper wraps [http.Server] to intercept and classify errors.
|
||||
ServerWrapper struct {
|
||||
*http.Server
|
||||
}
|
||||
)
|
||||
|
||||
// Serve starts the server and classifies any errors returned.
|
||||
// TLS-related errors are logged at debug level, other errors at warn level.
|
||||
func (sw *ServerWrapper) Serve() error {
|
||||
err := sw.ListenAndServe()
|
||||
|
||||
if err != nil {
|
||||
if isTLSError(err) {
|
||||
log.Debugf("Server error: %v", err)
|
||||
} else {
|
||||
log.Warnf("Server error: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
return err
|
||||
}
|
||||
|
||||
// ServeTLS starts the TLS server and classifies any errors returned.
|
||||
func (sw *ServerWrapper) ServeTLS(certFile, keyFile string) error {
|
||||
err := sw.ListenAndServeTLS(certFile, keyFile)
|
||||
|
||||
if err != nil {
|
||||
if isTLSError(err) {
|
||||
log.Debugf("Server TLS error: %v", err)
|
||||
} else {
|
||||
log.Warnf("Server TLS error: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
return err
|
||||
}
|
||||
|
||||
// timingSafeMatch performs constant-time comparison to prevent timing attacks.
|
||||
// Returns 1 if the strings are equal, 0 otherwise.
|
||||
func timingSafeMatch(a, b string) bool {
|
||||
return subtle.ConstantTimeCompare([]byte(a), []byte(b)) == 1
|
||||
}
|
||||
|
||||
/* vim: setlocal ft=go noet ai ts=4 sw=4 sts=4: */
|
||||
|
||||
@@ -4,6 +4,8 @@ import (
|
||||
"archive/tar"
|
||||
"compress/gzip"
|
||||
"context"
|
||||
"crypto/tls"
|
||||
"errors"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
@@ -11,6 +13,166 @@ import (
|
||||
"testing"
|
||||
)
|
||||
|
||||
// TestIsTLSError tests the isTLSError function.
|
||||
func TestIsTLSError(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
err error
|
||||
expected bool
|
||||
}{
|
||||
{
|
||||
name: "nil error",
|
||||
err: nil,
|
||||
expected: false,
|
||||
},
|
||||
{
|
||||
name: "regular error",
|
||||
err: errors.New("some random error"),
|
||||
expected: false,
|
||||
},
|
||||
{
|
||||
name: "TLS prefix error",
|
||||
err: errors.New("tls: handshake failure"),
|
||||
expected: true,
|
||||
},
|
||||
{
|
||||
name: "TLS AlertError",
|
||||
err: tls.AlertError(80),
|
||||
expected: true,
|
||||
},
|
||||
{
|
||||
name: "TLS CertificateVerificationError",
|
||||
err: &tls.CertificateVerificationError{},
|
||||
expected: true,
|
||||
},
|
||||
{
|
||||
name: "TLS ECHRejectionError",
|
||||
err: &tls.ECHRejectionError{RetryConfigList: []byte("rejected")},
|
||||
expected: true,
|
||||
},
|
||||
{
|
||||
name: "TLS RecordHeaderError",
|
||||
err: &tls.RecordHeaderError{},
|
||||
expected: true,
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
result := isTLSError(tt.err)
|
||||
if result != tt.expected {
|
||||
t.Errorf("isTLSError(%v) = %v, expected %v", tt.err, result, tt.expected)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestGetClientIP tests the getClientIP function.
|
||||
func TestGetClientIP(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
xffHeader string
|
||||
remoteAddr string
|
||||
expectedIP string
|
||||
}{
|
||||
{
|
||||
name: "no X-Forwarded-For",
|
||||
xffHeader: "",
|
||||
remoteAddr: "192.168.1.100:12345",
|
||||
expectedIP: "192.168.1.100",
|
||||
},
|
||||
{
|
||||
name: "single IP in X-Forwarded-For",
|
||||
xffHeader: "10.0.0.1",
|
||||
remoteAddr: "192.168.1.100:12345",
|
||||
expectedIP: "10.0.0.1",
|
||||
},
|
||||
{
|
||||
name: "multiple IPs in X-Forwarded-For",
|
||||
xffHeader: "10.0.0.1, 192.168.1.100, 172.16.0.1",
|
||||
remoteAddr: "192.168.1.100:12345",
|
||||
expectedIP: "10.0.0.1",
|
||||
},
|
||||
{
|
||||
name: "X-Forwarded-For with spaces",
|
||||
xffHeader: " 10.0.0.1 , 192.168.1.100 ",
|
||||
remoteAddr: "192.168.1.100:12345",
|
||||
expectedIP: "10.0.0.1",
|
||||
},
|
||||
{
|
||||
name: "invalid RemoteAddr",
|
||||
xffHeader: "",
|
||||
remoteAddr: "invalid",
|
||||
expectedIP: "invalid",
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
req := httptest.NewRequest("GET", "/archive", nil)
|
||||
if tt.xffHeader != "" {
|
||||
req.Header.Set("X-Forwarded-For", tt.xffHeader)
|
||||
}
|
||||
req.RemoteAddr = tt.remoteAddr
|
||||
|
||||
result := getClientIP(req)
|
||||
if result != tt.expectedIP {
|
||||
t.Errorf("getClientIP() = %q, expected %q", result, tt.expectedIP)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestGetCompressionAlgorithm tests the getCompressionAlgorithm function.
|
||||
func TestGetCompressionAlgorithm(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
ext string
|
||||
expected string
|
||||
}{
|
||||
{
|
||||
name: "tar.bz2",
|
||||
ext: "tar.bz2",
|
||||
expected: "bzip2",
|
||||
},
|
||||
{
|
||||
name: "tar.zst",
|
||||
ext: "tar.zst",
|
||||
expected: "zstd",
|
||||
},
|
||||
{
|
||||
name: "tar.lz4",
|
||||
ext: "tar.lz4",
|
||||
expected: "lz4",
|
||||
},
|
||||
{
|
||||
name: "tar.xz",
|
||||
ext: "tar.xz",
|
||||
expected: "xz",
|
||||
},
|
||||
{
|
||||
name: "tar.gz",
|
||||
ext: "tar.gz",
|
||||
expected: "pgzip",
|
||||
},
|
||||
{
|
||||
name: "default with gzip config",
|
||||
ext: "",
|
||||
expected: "gzip",
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
C = Config{DefaultCompression: "gzip"}
|
||||
result := getCompressionAlgorithm(tt.ext)
|
||||
if result != tt.expected {
|
||||
t.Errorf("getCompressionAlgorithm(%q) = %q, expected %q", tt.ext, result, tt.expected)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestNewServerSuccess tests successful server creation.
|
||||
func TestNewServerSuccess(t *testing.T) {
|
||||
C = Config{} // Reset global config.
|
||||
@@ -189,7 +351,7 @@ func TestHandlerArchiveStream(t *testing.T) {
|
||||
}
|
||||
|
||||
ctx := r.Context()
|
||||
files, err := GetFilesFromDirectories(C.Directories)
|
||||
files, err := GetFilesFromDirectories(ctx, C.Directories)
|
||||
if err != nil {
|
||||
http.Error(w, "Failed to get files", http.StatusInternalServerError)
|
||||
return
|
||||
@@ -301,7 +463,7 @@ func TestCopyWithContextSuccess(t *testing.T) {
|
||||
src := strings.NewReader("hello world")
|
||||
dst := &strings.Builder{}
|
||||
|
||||
err := copyWithContext(ctx, dst, src)
|
||||
_, _, err := copyWithContext(ctx, dst, src)
|
||||
if err != nil {
|
||||
t.Errorf("Expected no error, got %v", err)
|
||||
}
|
||||
@@ -311,6 +473,25 @@ func TestCopyWithContextSuccess(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestCopyWithContextMD5Hash tests that MD5 hash is correctly computed.
|
||||
func TestCopyWithContextMD5Hash(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
data := "hello world"
|
||||
src := strings.NewReader(data)
|
||||
dst := &strings.Builder{}
|
||||
|
||||
_, md5Hash, err := copyWithContext(ctx, dst, src)
|
||||
if err != nil {
|
||||
t.Errorf("Expected no error, got %v", err)
|
||||
}
|
||||
|
||||
// MD5 of "hello world" is 5eb63bbbe01eeed093cb22bb8f5acdc3
|
||||
expectedMD5 := "5eb63bbbe01eeed093cb22bb8f5acdc3"
|
||||
if md5Hash != expectedMD5 {
|
||||
t.Errorf("Expected MD5 %s, got %s", expectedMD5, md5Hash)
|
||||
}
|
||||
}
|
||||
|
||||
// TestCopyWithContextCancellation tests copy with cancelled context.
|
||||
func TestCopyWithContextCancellation(t *testing.T) {
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
@@ -325,14 +506,18 @@ func TestCopyWithContextCancellation(t *testing.T) {
|
||||
|
||||
// Start copy in goroutine
|
||||
errCh := make(chan error, 1)
|
||||
bytesCh := make(chan int64, 1)
|
||||
go func() {
|
||||
errCh <- copyWithContext(ctx, dst, src)
|
||||
n, _, err := copyWithContext(ctx, dst, src)
|
||||
bytesCh <- n
|
||||
errCh <- err
|
||||
}()
|
||||
|
||||
// Cancel context immediately
|
||||
cancel()
|
||||
|
||||
err := <-errCh
|
||||
<-bytesCh
|
||||
if err != context.Canceled {
|
||||
t.Errorf("Expected context.Canceled, got %v", err)
|
||||
}
|
||||
|
||||
@@ -15,12 +15,23 @@ type (
|
||||
LogLevel string `json:"loglevel"`
|
||||
Directories []string `json:"directories"`
|
||||
BackupTimeout int `json:"backup_timeout"` // Timeout in minutes for backup streaming.
|
||||
DirScanTimeout int `json:"dir_scan_timeout"` // Timeout in minutes for directory scanning.
|
||||
CompressionLevel int `json:"compression_level"` // Compression level (1-9, default 9).
|
||||
ExcludePatterns []string `json:"exclude_patterns"` // Regex patterns to exclude from backup.
|
||||
FilenamePrefix string `json:"filename_prefix"` // Prefix for backup filename (default: "backup").
|
||||
CompressionAlgorithm string `json:"compression_algorithm"` // Compression algorithm alias (deprecated: use default_compression).
|
||||
DefaultCompression string `json:"default_compression"` // Default compression algorithm: gzip, pgzip, bzip2, zstd, lz4 or xz.
|
||||
}
|
||||
|
||||
// ErrDirectoryScanTimeout is returned when directory scan times out.
|
||||
ErrDirectoryScanTimeout struct {
|
||||
ScannedFiles int
|
||||
}
|
||||
)
|
||||
|
||||
// Error returns the error message for ErrDirectoryScanTimeout.
|
||||
func (e ErrDirectoryScanTimeout) Error() string {
|
||||
return "Directory scan timed out"
|
||||
}
|
||||
|
||||
/* vim: setlocal ft=go noet ai ts=4 sw=4 sts=4: */
|
||||
|
||||
@@ -7,6 +7,7 @@ import (
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"runtime"
|
||||
"testing"
|
||||
)
|
||||
|
||||
@@ -179,6 +180,73 @@ func TestCreateTarGzStreamSymlink(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestCreateTarGzStreamHardLink tests hard link handling - should deduplicate content.
|
||||
func TestCreateTarGzStreamHardLink(t *testing.T) {
|
||||
if runtime.GOOS == "windows" {
|
||||
t.Skip("Hard link detection not supported on Windows (inodes unavailable)")
|
||||
}
|
||||
|
||||
tmpDir := t.TempDir()
|
||||
|
||||
// Create original file
|
||||
originalFile := filepath.Join(tmpDir, "original.txt")
|
||||
if err := os.WriteFile(originalFile, []byte("shared content"), 0644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
// Create hard link
|
||||
hardLinkFile := filepath.Join(tmpDir, "hardlink.txt")
|
||||
if err := os.Link(originalFile, hardLinkFile); err != nil {
|
||||
t.Skip("Hard links not supported, skipping test")
|
||||
}
|
||||
|
||||
ctx := context.Background()
|
||||
reader := CreateTarGzStream(ctx, []string{originalFile, hardLinkFile})
|
||||
defer reader.Close()
|
||||
|
||||
gzReader, err := gzip.NewReader(reader)
|
||||
if err != nil {
|
||||
t.Fatalf("Failed to create gzip reader: %v", err)
|
||||
}
|
||||
defer gzReader.Close()
|
||||
|
||||
tarReader := tar.NewReader(gzReader)
|
||||
|
||||
entries := 0
|
||||
linkCount := 0
|
||||
|
||||
for {
|
||||
header, err := tarReader.Next()
|
||||
if err == io.EOF {
|
||||
break
|
||||
}
|
||||
if err != nil {
|
||||
t.Fatalf("Failed to read tar entry: %v", err)
|
||||
}
|
||||
|
||||
entries++
|
||||
t.Logf("Entry %d: name=%s typeflag=%c linkname=%s", entries, header.Name, header.Typeflag, header.Linkname)
|
||||
|
||||
if header.Typeflag == tar.TypeLink {
|
||||
linkCount++
|
||||
// Verify it's a link to the original
|
||||
if header.Linkname != originalFile && header.Linkname != filepath.ToSlash(originalFile) {
|
||||
t.Errorf("Expected linkname '%s', got '%s'", originalFile, header.Linkname)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Should have 2 entries: one file content, one as hard link
|
||||
if entries != 2 {
|
||||
t.Errorf("Expected 2 entries, got %d", entries)
|
||||
}
|
||||
|
||||
// One should be a hard link (TypeLink)
|
||||
if linkCount != 1 {
|
||||
t.Errorf("Expected 1 hard link entry, got %d", linkCount)
|
||||
}
|
||||
}
|
||||
|
||||
// TestCreateTarGzStreamCancellation tests context cancellation.
|
||||
func TestCreateTarGzStreamCancellation(t *testing.T) {
|
||||
tmpDir := t.TempDir()
|
||||
@@ -356,4 +424,144 @@ func TestCreateTarGzStreamDeviceFile(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestCompressionLevelMappingLz4 tests that lz4 compression levels are correctly mapped.
|
||||
func TestCompressionLevelMappingLz4(t *testing.T) {
|
||||
tests := []struct {
|
||||
input int
|
||||
expected int
|
||||
}{
|
||||
{1, 0},
|
||||
{2, 0},
|
||||
{3, 0},
|
||||
{4, 1},
|
||||
{5, 1},
|
||||
{6, 1},
|
||||
{7, 2},
|
||||
{8, 2},
|
||||
{9, 2},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
C = Config{CompressionLevel: tt.input}
|
||||
result := mapCompressionLevelToLz4(tt.input)
|
||||
if result != tt.expected {
|
||||
t.Errorf("mapCompressionLevelToLz4(%d) = %d, expected %d", tt.input, result, tt.expected)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestCompressionLevelMappingXz tests that xz compression is not configurable via level.
|
||||
// xz uses default compression - this test verifies the stream works.
|
||||
func TestCompressionLevelMappingXz(t *testing.T) {
|
||||
// xz doesn't support configurable compression levels in the same way.
|
||||
// It always uses default compression. This test verifies the stream is created successfully.
|
||||
C = Config{CompressionLevel: 6}
|
||||
|
||||
tmpDir := t.TempDir()
|
||||
testFile := filepath.Join(tmpDir, "test.txt")
|
||||
content := []byte("test content")
|
||||
if err := os.WriteFile(testFile, content, 0644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
ctx := context.Background()
|
||||
reader := CreateTarXzStream(ctx, []string{testFile})
|
||||
defer reader.Close()
|
||||
|
||||
data, err := io.ReadAll(reader)
|
||||
if err != nil {
|
||||
t.Fatalf("Failed to read xz archive: %v", err)
|
||||
}
|
||||
|
||||
if len(data) == 0 {
|
||||
t.Error("Expected non-empty archive")
|
||||
}
|
||||
}
|
||||
|
||||
// TestCompressionLevelMappingBzip2 tests that bzip2 compression levels are correctly mapped.
|
||||
func TestCompressionLevelMappingBzip2(t *testing.T) {
|
||||
tests := []struct {
|
||||
input int
|
||||
expected int
|
||||
}{
|
||||
{1, 1},
|
||||
{5, 5},
|
||||
{9, 9},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
C = Config{CompressionLevel: tt.input}
|
||||
result := mapCompressionLevelToBzip2(tt.input)
|
||||
if result != tt.expected {
|
||||
t.Errorf("mapCompressionLevelToBzip2(%d) = %d, expected %d", tt.input, result, tt.expected)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestCreateTarBzip2Stream tests bzip2 compression with configured level.
|
||||
func TestCreateTarBzip2Stream(t *testing.T) {
|
||||
C = Config{CompressionLevel: 6}
|
||||
|
||||
tmpDir := t.TempDir()
|
||||
testFile := filepath.Join(tmpDir, "test.txt")
|
||||
content := []byte("test content for bzip2")
|
||||
if err := os.WriteFile(testFile, content, 0644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
ctx := context.Background()
|
||||
reader := CreateTarBzip2Stream(ctx, []string{testFile})
|
||||
defer reader.Close()
|
||||
|
||||
// Verify we can read the archive
|
||||
_, err := io.ReadAll(reader)
|
||||
if err != nil {
|
||||
t.Fatalf("Failed to read bzip2 archive: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestCreateTarLz4Stream tests lz4 compression with configured level.
|
||||
func TestCreateTarLz4Stream(t *testing.T) {
|
||||
C = Config{CompressionLevel: 6}
|
||||
|
||||
tmpDir := t.TempDir()
|
||||
testFile := filepath.Join(tmpDir, "test.txt")
|
||||
content := []byte("test content for lz4")
|
||||
if err := os.WriteFile(testFile, content, 0644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
ctx := context.Background()
|
||||
reader := CreateTarLz4Stream(ctx, []string{testFile})
|
||||
defer reader.Close()
|
||||
|
||||
// Verify we can read the archive
|
||||
_, err := io.ReadAll(reader)
|
||||
if err != nil {
|
||||
t.Fatalf("Failed to read lz4 archive: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestCreateTarXzStream tests xz compression with configured level.
|
||||
func TestCreateTarXzStream(t *testing.T) {
|
||||
C = Config{CompressionLevel: 6}
|
||||
|
||||
tmpDir := t.TempDir()
|
||||
testFile := filepath.Join(tmpDir, "test.txt")
|
||||
content := []byte("test content for xz")
|
||||
if err := os.WriteFile(testFile, content, 0644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
ctx := context.Background()
|
||||
reader := CreateTarXzStream(ctx, []string{testFile})
|
||||
defer reader.Close()
|
||||
|
||||
// Verify we can read the archive
|
||||
_, err := io.ReadAll(reader)
|
||||
if err != nil {
|
||||
t.Fatalf("Failed to read xz archive: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
/* vim: setlocal ft=go noet ai ts=4 sw=4 sts=4: */
|
||||
|
||||
@@ -10,6 +10,8 @@ import (
|
||||
// CreateTarXzStream takes a list of files and returns a reader that reads from an xz-compressed tar archive.
|
||||
func CreateTarXzStream(ctx context.Context, filepaths []string) io.ReadCloser {
|
||||
return createArchiveStream(ctx, filepaths, func(w io.Writer) (io.WriteCloser, error) {
|
||||
// xz doesn't have a simple level parameter; use default compression.
|
||||
// The compression level affects block size indirectly through the algorithm.
|
||||
return xz.NewWriter(w)
|
||||
})
|
||||
}
|
||||
|
||||
@@ -2,26 +2,56 @@
|
||||
package log
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"log"
|
||||
"log/slog"
|
||||
"os"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
var (
|
||||
// Log contains *[os.File] pointing to log file descriptor.
|
||||
// Protected by mu for thread-safe access, primarily to enable parallel tests.
|
||||
// In production, this is set once at startup and never modified thereafter.
|
||||
Log *os.File
|
||||
|
||||
// mu protects Log during initialization and when DebugLogger is called.
|
||||
// This is necessary to allow parallel test execution where each test
|
||||
// may initialize the logger with a different log file.
|
||||
mu sync.RWMutex
|
||||
|
||||
// logLevel stores the configured log level for use by DebugLogger.
|
||||
logLevel slog.Level
|
||||
)
|
||||
|
||||
// slogWriter implements [io.Writer] to redirect standard log to slog.
|
||||
type (
|
||||
slogWriter struct {
|
||||
handler slog.Handler
|
||||
}
|
||||
)
|
||||
|
||||
// Init sets up logger stuff.
|
||||
// level can be error, warn, info, debug; if something other is supplied, info level is selected.
|
||||
// Thread-safe: uses mutex to allow parallel test execution.
|
||||
func Init(level, filename string) error {
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
|
||||
var (
|
||||
err error
|
||||
loglevel slog.Level
|
||||
)
|
||||
|
||||
if filename != "" {
|
||||
// Close existing log file if re-initializing (e.g., in tests).
|
||||
if Log != nil && Log != os.Stderr {
|
||||
Log.Close()
|
||||
}
|
||||
|
||||
Log, err = os.OpenFile(filename, os.O_WRONLY|os.O_APPEND|os.O_CREATE, 0644)
|
||||
|
||||
if err != nil {
|
||||
@@ -49,6 +79,8 @@ func Init(level, filename string) error {
|
||||
loglevel = slog.LevelInfo
|
||||
}
|
||||
|
||||
logLevel = loglevel
|
||||
|
||||
opts := &slog.HandlerOptions{
|
||||
// Use the ReplaceAttr function on the handler options
|
||||
// to be able to replace any single attribute in the log output.
|
||||
@@ -122,6 +154,47 @@ func Debugf(format string, a ...any) {
|
||||
slog.Debug(fmt.Sprintf(format, a...))
|
||||
}
|
||||
|
||||
// DebugLogger returns a standard library logger that writes at the configured log level.
|
||||
// This is useful for assigning to http.Server.ErrorLog to route net/http errors
|
||||
// through our logger while respecting the configured log level.
|
||||
func DebugLogger() *log.Logger {
|
||||
mu.RLock()
|
||||
defer mu.RUnlock()
|
||||
|
||||
opts := &slog.HandlerOptions{
|
||||
Level: logLevel,
|
||||
}
|
||||
|
||||
handler := slog.NewTextHandler(Log, opts)
|
||||
|
||||
return log.New(&slogWriter{handler: handler}, "", 0)
|
||||
}
|
||||
|
||||
// Write implements [io.Writer]. It differentiates TLS/SSL errors from other errors.
|
||||
// TLS-related errors are logged at debug level, while other errors (handler issues,
|
||||
// unexpected behavior) are logged at warn level.
|
||||
func (w *slogWriter) Write(p []byte) (int, error) {
|
||||
msg := string(p)
|
||||
|
||||
isTLSError := strings.Contains(msg, "tls") ||
|
||||
strings.Contains(msg, "ssl") ||
|
||||
strings.Contains(msg, "TLS") ||
|
||||
strings.Contains(msg, "SSL") ||
|
||||
strings.Contains(msg, "handshake") ||
|
||||
strings.Contains(msg, "certificate")
|
||||
|
||||
level := slog.LevelWarn
|
||||
if isTLSError {
|
||||
level = slog.LevelDebug
|
||||
}
|
||||
|
||||
record := slog.NewRecord(time.Now(), level, msg, 0)
|
||||
|
||||
_ = w.handler.Handle(context.TODO(), record)
|
||||
|
||||
return len(p), nil
|
||||
}
|
||||
|
||||
// Fatal logs args at error level and quits with status code 1.
|
||||
func Fatal(args ...any) {
|
||||
slog.Error(fmt.Sprint(args...))
|
||||
@@ -138,7 +211,11 @@ func Fatalf(format string, a ...any) {
|
||||
}
|
||||
|
||||
// Close closes log file descriptor.
|
||||
// Thread-safe: uses mutex to coordinate with Init and DebugLogger.
|
||||
func Close() {
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
|
||||
if Log == nil || Log == os.Stderr {
|
||||
return
|
||||
}
|
||||
@@ -147,6 +224,8 @@ func Close() {
|
||||
msg := fmt.Sprintln("Failed to close log file:", err)
|
||||
slog.Error(msg)
|
||||
}
|
||||
|
||||
Log = nil
|
||||
}
|
||||
|
||||
/* vim: setlocal ft=go noet ai ts=4 sw=4 sts=4: */
|
||||
|
||||
@@ -354,4 +354,35 @@ func TestCloseStderr(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestDebugLogger tests that DebugLogger returns a working *log.Logger.
|
||||
func TestDebugLogger(t *testing.T) {
|
||||
tmpFile, err := os.CreateTemp(t.TempDir(), "test_log_*.log")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
logFilePath := tmpFile.Name()
|
||||
tmpFile.Close()
|
||||
|
||||
if err := Init("info", logFilePath); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer Close()
|
||||
|
||||
logger := DebugLogger()
|
||||
if logger == nil {
|
||||
t.Error("DebugLogger returned nil")
|
||||
}
|
||||
|
||||
logger.Print("test debug message")
|
||||
|
||||
content, err := os.ReadFile(logFilePath)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
if !strings.Contains(string(content), "test debug message") {
|
||||
t.Error("Expected debug message in log")
|
||||
}
|
||||
}
|
||||
|
||||
/* vim: setlocal ft=go noet ai ts=4 sw=4 sts=4: */
|
||||
|
||||
@@ -24,7 +24,7 @@
|
||||
"password": "SharpShooter",
|
||||
|
||||
/* Where to store logs. */
|
||||
"log": "/var/log/backer.log",
|
||||
"log": "../../test_data/tmp/backer.log",
|
||||
|
||||
/* Verbosity of logs */
|
||||
"loglevel": "info",
|
||||
@@ -39,5 +39,5 @@
|
||||
"backup_timeout": 60,
|
||||
|
||||
/* Gzip compression level (1-9, where 9 is best compression). */
|
||||
"compression_level": 6
|
||||
"compression_level": 3
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user