24 Commits

Author SHA1 Message Date
eleksir
ebd09af6b1 Update tests results, add compression level mapping table to README.md 2026-04-05 01:08:11 +03:00
eleksir
0511d47765 Update rules for linter 2026-04-05 01:06:52 +03:00
eleksir
93b671775f Update AGENTS.md 2026-04-05 01:06:28 +03:00
eleksir
c18eb8fb07 Update .gitignore 2026-04-05 01:06:04 +03:00
eleksir
0cb4d355ab Fix tests path fragility 2026-04-05 01:05:25 +03:00
eleksir
6cea15f553 Implement dir_scan_timeout config option 2026-04-04 23:32:37 +03:00
eleksir
6e56e74df4 If configured directory is missing, archive will be created anyway, but this error will be logged. 2026-04-04 22:45:00 +03:00
eleksir
3c22b2e1a0 Now TLS handshake errors will respect your loglevel setting 2026-04-04 22:25:23 +03:00
eleksir
65a57a6af6 Fixed bug with file extentions, enhance logging
- Fix 404 on downloading archive with extension
- Add downloaded archive length to log
- Add md5 hash to log
2026-04-04 21:45:33 +03:00
eleksir
e063352252 Compression levels does not apply for bzip2, xz, zstd, lz4, now we map then as best as we can. 2026-04-04 16:45:07 +03:00
eleksir
f20f490c15 Disable keepalive connections and set idle timeout to 10 seconds, explicitly 2026-04-04 16:17:21 +03:00
eleksir
ae321a216d Explicitly hardcode MaxHeaderBytes as 1k, to prevent slowloris attack 2026-04-04 16:08:57 +03:00
eleksir
729b147b90 If no files found in configured directories warn in log about it 2026-04-04 16:03:08 +03:00
eleksir
7b001d710a Implement timing-safe string comparison for authentication
Prevents timing attacks where an attacker could deduce correct
credentials by measuring response times. Not sure if it is real threat,
but anyway.
2026-04-04 15:59:40 +03:00
eleksir
2201fc72f1 Implement proper signal handler 2026-04-04 15:50:09 +03:00
eleksir
5e7dd4b960 Make logging thread safe, useful to run tests in parallel 2026-04-04 15:32:42 +03:00
eleksir
c1444f8c1d Add cancellation after read and before write, to be able to cancel operation more quickly 2026-04-04 15:22:50 +03:00
eleksir
6955e14921 Add hardlink detection for *nix platforms, they stored as links in tar, so content is effectively deduplites 2026-04-04 15:08:54 +03:00
eleksir
fd8612cfc6 Log all requests, not only ones that hit backup target, fix comments to be docstring friendly. 2026-04-04 14:44:48 +03:00
eleksir
e449fb78c9 Add more tests 2026-04-04 13:59:09 +03:00
eleksir
958ef67f29 Move ssl/tls related error to debug loglevel, to avoid log cluttering. 2026-04-04 13:16:22 +03:00
eleksir
14b7a17b00 Move log message about selected default compression algo after logging subsystem fully initialized 2026-04-04 12:44:47 +03:00
eleksir
c3553de856 Add version and software name in first message during startup, fix typo in log message during config loading. 2026-04-04 12:33:40 +03:00
eleksir
06ee9e081f Do not create log dir in openrc init file as per default config 2026-04-04 12:07:32 +03:00
23 changed files with 1211 additions and 120 deletions

4
.gitignore vendored
View File

@@ -1,6 +1,8 @@
# Binary
/backer
/backer.exe
/cmd/backer/backer
/cmd/backer/backer.exe
# Local config (may contain secrets)
/config.json
@@ -27,3 +29,5 @@ Thumbs.db
*.exe
*.test
*.out
/test_data/tmp

View File

@@ -74,6 +74,7 @@ linters:
- paramTypeCombine
- whyNoLint
- hugeParam
- unnamedResult
godoclint:
default: all
ignore-unexported: false

View File

@@ -20,7 +20,7 @@ Backer is a Go-based HTTP/HTTPS backup server that creates compressed archives o
- **Use tags and labels.** In comments, add tags or labels to files, classes, or functions to categorize them based on functionality, domain concepts, or design patterns.
- **Avoid obvious comments that do not add value.**
- **Remove dead code.** A half-finished refactoring can actually harm the contributions.
- **Update AGENTS.md and TODO.md automatically.** Keep AGENTS.md dense, detailed and concise.
- **Update AGENTS.md automatically.** Keep AGENTS.md dense, detailed and concise.
## Workflow Rules
@@ -124,6 +124,18 @@ CompressionLevel int `json:"compression_level"`
| QF1001 | De Morgan's law disabled for auth condition clarity |
| revive (error strings) | Same reason as ST1005 |
### Design Decisions (Not Issues)
The following are intentional design decisions, not bugs or defects:
- **No rate limiting**: Backups are assumed to be infrequent enough that rate limiting is unnecessary.
- **Backup scheduling**: Client-side decision — server serves backups on-request only.
- **Basic Auth only**: HTTP Basic Auth is sufficient for a backup server behind a firewall.
- **Global config**: The `C Config` global variable is intentional — config is loaded once at startup.
- **No incremental backup**: Each backup is a full snapshot, not incremental.
- **Credentials in config file**: Config file should have restricted permissions (documented).
- **Thread-safe logging**: The global `Log` variable uses mutex protection to enable parallel test execution; this is not needed for single-server production use but allows tests to run in parallel without race conditions.
## Project Structure
```
@@ -228,16 +240,17 @@ No temp files on disk. Uses `io.Pipe` with a goroutine to stream archives on-the
Two helper functions in `newServer.go` handle context cancellation during streaming:
- `writeWithContext(ctx, fn)` — checks context before calling fn
- `copyWithContext(ctx, dst, src)` — buffered copy (32KB) with cancellation checks
- `copyWithContext(ctx, dst, src)` — buffered copy (32KB) with cancellation checks, computes MD5 hash of copied data
### Special File Type Handling
- **Regular files**: streamed into tar archive with content
- **Directories**: included as tar directory entries
- **Symlinks**: stored as symlink entries (target path preserved)
- **Symlinks**: stored as symlink entries (target path preserved). Backer never resolves or follows symlinks during directory traversal — it uses `os.Lstat` and `filepath.WalkDir` which do not follow symlinks. This means symlink cycles are impossible to occur.
- **Device files** (char/block): included with correct major/minor
- **Named pipes**: included (header only, no data)
- **Sockets**: skipped (not archivable)
- **Hard links**: deduplicated — if multiple files share the same inode, only the first is stored with content, subsequent hard links are stored as link entries pointing to the original
### Pipe Error Recovery
@@ -245,7 +258,7 @@ Two helper functions in `newServer.go` handle context cancellation during stream
### Auth Handler
HTTP Basic Auth. Both username AND password must match exactly. Returns `401 Unauthorized` with `WWW-Authenticate` header on failure. The condition is intentionally written without De Morgan's law simplification for clarity (linter suppression `//nolint:staticcheck`).
HTTP Basic Auth. Both username AND password must match exactly. Returns `401 Unauthorized` with `WWW-Authenticate` header on failure. Uses timing-safe comparison to prevent timing attacks. The condition is intentionally written without De Morgan's law simplification for clarity (linter suppression `//nolint:staticcheck`).
### Client IP Detection
@@ -255,8 +268,56 @@ Checks `X-Forwarded-For` header first (for proxied requests), then falls back to
- `ReadHeaderTimeout`: 5 seconds
- `WriteTimeout`: configurable via `backup_timeout` (converted to `time.Duration`)
- `IdleTimeout`: 10 seconds (for keep-alive connections; backer disables keep-alives explicitly)
- `MaxHeaderBytes`: 1KB (prevents header-based DoS attacks)
- `DisableKeepAlives`: true (each connection closed after request, no connection reuse)
- TLS minimum version: TLS 1.3 (when HTTPS enabled)
### TLS Error Handling
The server uses a `ServerWrapper` type to intercept and classify errors from `Serve()` and `ServeTLS()`. TLS-specific errors are logged at debug level to reduce log clutter:
- Uses `errors.As` to detect TLS error types: `tls.AlertError`, `tls.CertificateVerificationError`, `tls.ECHRejectionError`, `tls.RecordHeaderError`
- Also checks for "tls:" prefix in error messages
- Additionally uses `http.Server.ErrorLog` with a custom logger that writes at debug level for net/http's internal TLS-related messages
This ensures client-side TLS errors (handshake failures, certificate issues) don't clutter production logs.
### Compression Level Mapping
The `compression_level` config option (1-9) is mapped differently for each compression algorithm:
| Algorithm | Level Mapping |
|-----------|----------------|
| gzip | 1-9 directly maps to gzip compression levels |
| pgzip | 1-9 directly maps to pgzip compression levels |
| bzip2 | 1-9 directly maps to bzip2 compression levels |
| zstd | 1-3: Fastest, 4-6: Default, 7-9: BetterCompression |
| lz4 | 1-3: Fastest (0), 4-6: Default (1), 7-9: Best (2) |
| xz | Uses default compression (no configurable level) |
### HTTP Routing & Compression Selection
The server uses `http.NewServeMux()` for routing. Multiple routes are registered to support different compression formats:
```go
mux.Handle(C.Location, http.HandlerFunc(backupHandler)) // /archive
mux.Handle(C.Location+"/", http.HandlerFunc(backupHandler)) // /archive/
mux.Handle(C.Location+".tar.gz", http.HandlerFunc(backupHandler)) // /archive.tar.gz
mux.Handle(C.Location+".tar.xz", http.HandlerFunc(backupHandler)) // /archive.tar.xz
mux.Handle(C.Location+".tar.bz2", http.HandlerFunc(backupHandler)) // /archive.tar.bz2
// ... etc
```
The compression algorithm is determined by extracting the extension from the request path using `path.Ext()`:
- For `/archive.tar.xz`, `path.Ext()` returns `.tar.xz` → maps to xz
- For `/archive`, `path.Ext()` returns `""` → uses default compression
The `getCompressionAlgorithm()` function handles both full paths and bare extensions:
- Full path: `/archive.tar.xz` → `.tar.xz` → xz
- Bare extension: `tar.xz` → `.tar.xz` → xz
- Single ext: `xz` → `.xz` → xz
## Dependencies
- `github.com/hjson/hjson-go` v3.3.0 — HJSON config parsing

View File

@@ -77,7 +77,7 @@ AI suggests this table as expected performance.
| Algorithm | Speed | Compression | Notes |
| :--- | :--- | :--- | :--- |
| gzip | Fast | Good | Default. Standard, widely compatible. |
| gzip | Fast | Good | Standard, widely compatible. |
| pgzip | Fast | Good | Parallel gzip. Uses multiple CPU cores, slightly larger files. |
| bzip2 | Slow | Better | Pure Go. Good compression, slower than gzip. |
| zstd | Fast | Very good | Good balance of speed and compression. |
@@ -90,12 +90,21 @@ network bandwidth, located in near by country I've got these results
| Algorithm | Size | Time | CPU Load | RAM RSS+Shm, Mb | Notes |
| :--- | ---: | ---: | ---: | ---: | :--- |
| None | 1461250551 | | 0-2% | 0+0 | original dataset |
| gzip | 823381547 | 16m:22s | 96% | 18+6 | stdlib compression/gzip |
| pgzip | 823783924 | 13m:50s | 93-97% | 28+5 | multithreaded (parallel gzip), fully gzip compatible, klauspost/pgzip |
| bzip2 | 797053471 | 9m:47s | 92-94% | 56+6 | dsnet/compress/bzip2 |
| xz | 515106988 | 16m:03s | 96% | 116+5 | ulikunitz/xz |
| zstd | 617330529 | 6m:39s | 40-99% | 64+6 | klauspost/compress/zstd |
| lz4 | 1097745915 | 6m:12s | 4-6% | 30+6 | multithreaded lz4, pierrec/lz4 |
| zstd | 614015773 | 4m:52s | 40-60% | 60+6 | klauspost/compress/zstd |
| pgzip | 817163857 | 11m:25s | 96-97% | 37+6 | multithreaded (parallel gzip), fully gzip compatible, klauspost/pgzip |
| gzip | 816734188 | 14m:30s | 96% | 18+6 | stdlib compression/gzip |
| bzip2 | 788061284 | 7m:16s | 92-94% | 43+6 | dsnet/compress/bzip2 |
| xz | 512004612 | 13m:53s | 96% | 116+6 | ulikunitz/xz |
The `compression_level` option (1-9) maps differently for each algorithm:
| Algorithm | Level Mapping |
| :--- | :--- |
| gzip/pgzip/bzip2 | 1-9 directly maps to compression levels |
| lz4 | 1-3 → Fastest (0), 4-6 → Default (1), 7-9 → Best (2) |
| zstd | 1-3 → Fastest, 4-6 → Default, 7-9 → BetterCompression |
| xz | Uses default compression (no configurable level) |
Obvously in case of lz4 and zstd the bottleneck was network. And I have to admit, that network definitely was not
100Mbit, but it was pretty hard to measure real bandwidth. Anyway let's pretend that during experiment it was roughly
@@ -111,6 +120,52 @@ good enough cypher.
And one more thing user and password are mandatory, we should have at least some minimal barriers, right?
## Supported File Types
Backer handles various file types when creating archives:
- **Regular files**: Included with full content
- **Directories**: Included as directory entries
- **Symlinks**: Stored as symlink entries (target path preserved)
- **Device files**: Character and block devices included with correct major/minor
- **Named pipes**: Included (header only, no data)
- **Hard links**: Deduplicated — multiple hard links to the same file are stored as link entries to avoid duplication
## Logging
Backer logs events via Go's `slog`. Log levels: error, warn, info, debug.
### Backup Completion Log
When a backup completes, the following information is logged:
- **client**: Client IP address
- **user**: Authenticated username
- **files**: Number of files backed up
- **bytes**: Size of the created archive in bytes
- **md5**: MD5 hash of the created archive (for verification)
- **duration**: Time taken to create the archive
Example:
```
Backup completed: client=127.0.0.1 user=admin files=42 bytes=1234567 md5=abc123def456 duration=2.5s
```
### TLS Error Handling
TLS handshake errors and other client-side SSL/TLS errors are logged at **debug** level to reduce log clutter.
This includes:
- Certificate verification failures
- Handshake failures
- Protocol errors
- TLS-specific connection issues
### Graceful Shutdown
When backer receives a termination signal (SIGTERM or SIGINT), it will:
1. Log a warning message: "Shutdown signal received, shutting down immediately..."
2. Stop accepting new connections and abort any in-progress downloads immediately
3. Clean up resources and exit
This ensures no partial/corrupted backups are left. Clients will need to restart their backup. For a forced immediate stop, use `SIGKILL`.
## Special Thanks
This project was developed with the assistance of **opencode** (AI co-programmer powered by MIMO). The AI helped with:

View File

@@ -2,6 +2,7 @@
package main
import (
"errors"
"flag"
"fmt"
"net/http"
@@ -40,10 +41,14 @@ func main() {
log.Fatalf("Failed to create server: %v", err)
}
log.Infof("Starting server at %s", server.Addr)
// Setup signal handling for immediate shutdown.
log.Infof("Installing signal handler")
setupSignalHandler(server)
log.Infof("Starting backer version %s at %s", version, server.Addr)
if backer.C.NoHTTPS {
if err := server.ListenAndServe(); err != nil && err != http.ErrServerClosed {
if err := server.Serve(); err != nil && !errors.Is(err, http.ErrServerClosed) {
log.Fatalf("Failed to start server: %v", err)
}
} else {
@@ -55,10 +60,12 @@ func main() {
log.Fatalf("Certificate key file error: %v", err)
}
if err := server.ListenAndServeTLS(backer.C.Cert, backer.C.Key); err != nil && err != http.ErrServerClosed {
if err := server.ServeTLS(backer.C.Cert, backer.C.Key); err != nil && !errors.Is(err, http.ErrServerClosed) {
log.Fatalf("Failed to start server: %v", err)
}
}
log.Infof("Server stopped")
}
// defaultConfigPath returns the default config file path based on the OS.

29
cmd/backer/sigHandler.go Normal file
View File

@@ -0,0 +1,29 @@
package main
import (
"os"
"os/signal"
"syscall"
"backer/internal/backer"
"backer/internal/log"
)
// setupSignalHandler sets up signal handling for immediate shutdown.
// SIGTERM is commonly sent by systemd, k8s, and other container orchestrators.
// SIGINT is sent on Ctrl+C.
func setupSignalHandler(server *backer.ServerWrapper) {
signalCh := make(chan os.Signal, 1)
signal.Notify(signalCh, os.Interrupt, syscall.SIGTERM)
go func() {
<-signalCh
log.Warnf("Shutdown signal received, shutting down immediately...")
// Close the embedded http.Server directly.
if closeErr := server.Close(); closeErr != nil {
log.Errorf("Error during server shutdown: %v", closeErr)
}
}()
}
/* vim: setlocal ft=go noet ai ts=4 sw=4 sts=4: */

View File

@@ -11,7 +11,6 @@ depend() {
}
start_pre() {
mkdir -p /var/log/$SVCNAME
mkdir -p /run/$SVCNAME
}

View File

@@ -73,7 +73,14 @@ func createArchiveStream(ctx context.Context, filepaths []string, writerFactory
}
// writeFilesToTar iterates over filepaths and writes each file to the tar archive.
// It tracks inodes to handle hard links - if a file has already been archived,
// subsequent hard links to the same inode are stored as link entries.
// This is the Unix implementation (syscall not available on Windows).
func writeFilesToTar(ctx context.Context, filepaths []string, tw *tar.Writer, closePipeWithError func(error)) {
// Track inodes to handle hard links: inode -> first path added.
// Uses simple file path tracking on Windows, inode tracking on Unix.
seenInodes := make(map[uint64]string)
for _, fpath := range filepaths {
var (
err error
@@ -116,6 +123,21 @@ func writeFilesToTar(ctx context.Context, filepaths []string, tw *tar.Writer, cl
continue
}
// Check for hard link - if inode was already seen, store as link instead of content.
if st.Mode().IsRegular() {
inode := getInode(st)
if inode != 0 {
if existingPath, seen := seenInodes[inode]; seen {
// This is a hard link to already-archived file.
linkTarget = existingPath
log.Debugf("Hard link detected: %s -> %s", fpath, existingPath)
} else {
// First time seeing this inode - track it.
seenInodes[inode] = fpath
}
}
}
header, err := tar.FileInfoHeader(st, linkTarget)
if err != nil {
@@ -124,6 +146,11 @@ func writeFilesToTar(ctx context.Context, filepaths []string, tw *tar.Writer, cl
continue
}
// If linkTarget is set, ensure the header is marked as a link.
if linkTarget != "" {
header.Typeflag = tar.TypeLink
}
header.Format = tar.FormatGNU
header.Name = strings.TrimPrefix(filepath.ToSlash(filepath.Clean(fpath)), "/")
@@ -146,7 +173,7 @@ func writeFilesToTar(ctx context.Context, filepaths []string, tw *tar.Writer, cl
continue
}
err = copyWithContext(ctx, tw, f)
_, _, err = copyWithContext(ctx, tw, f)
if e := f.Close(); e != nil {
log.Warnf("Failed to close file %s: %v", fpath, e)
@@ -161,6 +188,27 @@ func writeFilesToTar(ctx context.Context, filepaths []string, tw *tar.Writer, cl
}
}
// getInode extracts the inode number from [os.FileInfo].
// Returns 0 if unable to determine (e.g., on Windows or with certain file systems).
func getInode(fi os.FileInfo) uint64 {
if fi.Sys() == nil {
return 0
}
// Use type assertion to get inode - works on Unix, returns 0 on Windows.
type inodeGetter interface {
Dev() uint64
Ino() uint64
}
if ig, ok := fi.Sys().(inodeGetter); ok {
// Combine device and inode for uniqueness across file systems.
return (ig.Dev() << 32) | ig.Ino()
}
return 0
}
// isPipeClosedError checks if the error is due to a closed pipe.
func isPipeClosedError(err error) bool {
if err == nil {

View File

@@ -10,8 +10,22 @@ import (
// CreateTarBzip2Stream takes a list of files and returns a reader that reads from a bzip2-compressed tar archive.
func CreateTarBzip2Stream(ctx context.Context, filepaths []string) io.ReadCloser {
return createArchiveStream(ctx, filepaths, func(w io.Writer) (io.WriteCloser, error) {
return bzip2.NewWriter(w, nil)
level := mapCompressionLevelToBzip2(C.CompressionLevel)
return bzip2.NewWriter(w, &bzip2.WriterConfig{Level: level})
})
}
// mapCompressionLevelToBzip2 maps 1-9 compression level to bzip2 level.
func mapCompressionLevelToBzip2(level int) int {
if level < 1 {
return bzip2.DefaultCompression
}
if level > 9 {
return bzip2.BestCompression
}
return level
}
/* vim: setlocal ft=go noet ai ts=4 sw=4 sts=4: */

View File

@@ -11,6 +11,12 @@ const (
// copyBufferSize is the buffer size used for streaming file contents.
copyBufferSize = 32 * 1024
// maxHeaderBytes is the maximum size of request headers (1KB).
maxHeaderBytes = 1024
// idleTimeout is the maximum time to wait for the next request on keep-alive connections.
idleTimeout = 10 * time.Second
// portMin is the minimum valid port number.
portMin = 1
@@ -23,6 +29,12 @@ const (
// backupTimeoutMax is the maximum backup timeout in minutes (24 hours).
backupTimeoutMax = 1440
// dirScanTimeoutMin is the minimum directory scan timeout in minutes.
dirScanTimeoutMin = 1
// dirScanTimeoutMax is the maximum directory scan timeout in minutes (24 hours).
dirScanTimeoutMax = 1440
// compressionLevelMin is the minimum compression level.
compressionLevelMin = 1
@@ -35,6 +47,9 @@ const (
// defaultBackupTimeout is the default backup timeout in minutes.
defaultBackupTimeout = 60
// defaultDirScanTimeout is the default directory scan timeout in minutes (1 hour).
defaultDirScanTimeout = 60
// defaultCompressionLevel is the default compression level.
defaultCompressionLevel = 9

View File

@@ -1,10 +1,14 @@
package backer
import (
"context"
"errors"
"io/fs"
"os"
"path/filepath"
"time"
"backer/internal/log"
backerlog "backer/internal/log"
)
// isExcluded checks if path matches any of the configured exclude patterns.
@@ -22,19 +26,38 @@ func isExcluded(path string) bool {
}
// GetFilesFromDirectories makes a file list of given directories.
func GetFilesFromDirectories(directories []string) ([]string, error) {
// Logs errors for directories that don't exist or can't be accessed but continues with valid directories.
// Takes context with timeout from config dir_scan_timeout setting.
func GetFilesFromDirectories(ctx context.Context, directories []string) ([]string, error) {
var files []string
for _, dir := range directories {
// Check directory exists before walking.
if _, err := os.Stat(dir); err != nil {
if errors.Is(err, os.ErrNotExist) {
backerlog.Errorf("Configured backup directory not found: %s", dir)
} else {
backerlog.Errorf("Failed to access configured directory %s: %v", dir, err)
}
continue
}
err := filepath.WalkDir(dir, func(path string, de fs.DirEntry, err error) error { //nolint: revive
select {
case <-ctx.Done():
return ctx.Err()
default:
}
if err != nil {
log.Warnf("Skipping %s: %v", path, err)
backerlog.Warnf("Skipping %s: %v", path, err)
return nil
}
if isExcluded(path) {
log.Debugf("Excluding: %s", path)
backerlog.Debugf("Excluding: %s", path)
return nil
}
@@ -45,11 +68,26 @@ func GetFilesFromDirectories(directories []string) ([]string, error) {
})
if err != nil {
log.Warnf("Failed to walk directory %s: %v", dir, err)
if errors.Is(err, context.DeadlineExceeded) {
backerlog.Errorf("Directory scan timed out after scanning %d files", len(files))
return files, ErrDirectoryScanTimeout{ScannedFiles: len(files)}
}
backerlog.Warnf("Failed to walk directory %s: %v", dir, err)
}
}
return files, nil
}
// GetFilesFromDirectoriesWithTimeout makes a file list of given directories with timeout.
// Convenience wrapper that creates context from timeout in minutes.
func GetFilesFromDirectoriesWithTimeout(timeoutMinutes int, directories []string) ([]string, error) {
ctx, cancel := context.WithTimeout(context.Background(), time.Duration(timeoutMinutes)*time.Minute)
defer cancel()
return GetFilesFromDirectories(ctx, directories)
}
/* vim: setlocal ft=go noet ai ts=4 sw=4 sts=4: */

View File

@@ -1,7 +1,9 @@
package backer
import (
"context"
"os"
"path/filepath"
"regexp"
"slices"
"strings"
@@ -15,28 +17,36 @@ func normalizePath(p string) string {
// TestGetFilesFromDirectories checks if GetFilesFromDirectories returns the proper amount of strings, correct strings and no error.
func TestGetFilesFromDirectories(t *testing.T) {
// Ensure empty_dir exists for the test.
os.MkdirAll("../../test_data/test1/foo/empty_dir", 0755)
testDataPath := filepath.Join("..", "test_data")
// Log the paths we're checking for debugging
t.Logf("Checking directories: %v", []string{
filepath.Join(testDataPath, "test1", "foo"),
filepath.Join(testDataPath, "test1", "bar"),
})
var (
input = Config{
Directories: []string{"../../test_data/test1/foo", "../../test_data/test1/bar"},
Directories: []string{
filepath.Join(testDataPath, "test1", "foo"),
filepath.Join(testDataPath, "test1", "bar"),
},
}
expectedData = []string{
"../../test_data/test1/foo",
"../../test_data/test1/foo/empty_dir",
"../../test_data/test1/foo/hello_breakfast.txt",
"../../test_data/test1/foo/mydir",
"../../test_data/test1/foo/mydir/myfile.txt",
"../../test_data/test1/foo/some_text.txt",
"../../test_data/test1/bar",
"../../test_data/test1/bar/goodbye.txt",
"../../test_data/test1/bar/goodbye.txt/text.txt",
"../../test_data/test1/bar/hello.txt",
filepath.Join(testDataPath, "test1", "foo"),
filepath.Join(testDataPath, "test1", "foo", "empty_dir"),
filepath.Join(testDataPath, "test1", "foo", "hello_breakfast.txt"),
filepath.Join(testDataPath, "test1", "foo", "mydir"),
filepath.Join(testDataPath, "test1", "foo", "mydir", "myfile.txt"),
filepath.Join(testDataPath, "test1", "foo", "some_text.txt"),
filepath.Join(testDataPath, "test1", "bar"),
filepath.Join(testDataPath, "test1", "bar", "goodbye.txt"),
filepath.Join(testDataPath, "test1", "bar", "goodbye.txt", "text.txt"),
filepath.Join(testDataPath, "test1", "bar", "hello.txt"),
}
)
output, err := GetFilesFromDirectories(input.Directories)
output, err := GetFilesFromDirectories(context.Background(), input.Directories)
if err != nil {
t.Errorf("Expected no error, got %v", err)
@@ -59,7 +69,8 @@ func TestGetFilesFromDirectories(t *testing.T) {
// TestGetFilesFromDirectoriesSingleDir tests with a single directory.
func TestGetFilesFromDirectoriesSingleDir(t *testing.T) {
output, err := GetFilesFromDirectories([]string{"../../test_data/test1/foo"})
testDataPath := filepath.Join("test_data")
output, err := GetFilesFromDirectories(context.Background(), []string{filepath.Join(testDataPath, "test1", "foo")})
if err != nil {
t.Errorf("Expected no error, got %v", err)
@@ -70,7 +81,7 @@ func TestGetFilesFromDirectoriesSingleDir(t *testing.T) {
}
// Should contain the directory itself
if !slices.Contains(output, "../../test_data/test1/foo") {
if !slices.Contains(output, filepath.Join(testDataPath, "test1", "foo")) {
t.Error("Output should contain the root directory")
}
}
@@ -80,7 +91,7 @@ func TestGetFilesFromDirectoriesEmptyDir(t *testing.T) {
// Create temp empty dir
tmpDir := t.TempDir()
output, err := GetFilesFromDirectories([]string{tmpDir})
output, err := GetFilesFromDirectories(context.Background(), []string{tmpDir})
if err != nil {
t.Errorf("Expected no error, got %v", err)
@@ -93,23 +104,48 @@ func TestGetFilesFromDirectoriesEmptyDir(t *testing.T) {
}
// TestGetFilesFromDirectoriesNonExistent tests with non-existent directory.
// Should log error but not fail - returns empty list.
func TestGetFilesFromDirectoriesNonExistent(t *testing.T) {
output, err := GetFilesFromDirectories([]string{"/nonexistent/path/that/does/not/exist"})
testDataPath := filepath.Join("test_data")
nonExistentPath := filepath.Join(testDataPath, "nonexistent", "path", "that", "does", "not", "exist")
output, err := GetFilesFromDirectories(context.Background(), []string{nonExistentPath})
// Should not return error, but logs error
if err != nil {
t.Errorf("Expected no error (logs warning instead), got %v", err)
t.Errorf("Expected no error (logs error instead), got %v", err)
}
// Should return empty list
if len(output) != 0 {
t.Errorf("Expected empty output for non-existent directory, got %d files", len(output))
}
}
// TestGetFilesFromDirectoriesPartialFailure tests with mix of valid and invalid directories.
func TestGetFilesFromDirectoriesPartialFailure(t *testing.T) {
testDataPath := filepath.Join("test_data")
// Create a temp directory that exists
tmpDir := t.TempDir()
output, err := GetFilesFromDirectories(context.Background(), []string{tmpDir, filepath.Join(testDataPath, "nonexistent", "path", "that", "does", "not", "exist")})
// Should not return error, but logs error
if err != nil {
t.Errorf("Expected no error (logs error instead), got %v", err)
}
// Should return files from valid directory
if len(output) == 0 {
t.Error("Expected files from valid directory")
}
}
// TestGetFilesFromDirectoriesMultipleDirs tests multiple directories.
func TestGetFilesFromDirectoriesMultipleDirs(t *testing.T) {
output, err := GetFilesFromDirectories([]string{
"../../test_data/test1/foo",
"../../test_data/test1/bar",
testDataPath := filepath.Join("test_data")
output, err := GetFilesFromDirectories(context.Background(), []string{
filepath.Join(testDataPath, "test1", "foo"),
filepath.Join(testDataPath, "test1", "bar"),
})
if err != nil {
@@ -118,10 +154,10 @@ func TestGetFilesFromDirectoriesMultipleDirs(t *testing.T) {
// Should contain entries from both directories
hasFoo := slices.ContainsFunc(output, func(s string) bool {
return s == "../../test_data/test1/foo" || s == "../../test_data/test1/foo/hello_breakfast.txt"
return s == filepath.Join(testDataPath, "test1", "foo") || s == filepath.Join(testDataPath, "test1", "foo", "hello_breakfast.txt")
})
hasBar := slices.ContainsFunc(output, func(s string) bool {
return s == "../../test_data/test1/bar" || s == "../../test_data/test1/bar/hello.txt"
return s == filepath.Join(testDataPath, "test1", "bar") || s == filepath.Join(testDataPath, "test1", "bar", "hello.txt")
})
if !hasFoo {
@@ -149,7 +185,7 @@ func TestGetFilesFromDirectoriesSymlink(t *testing.T) {
t.Skip("Symlinks not supported, skipping test")
}
output, err := GetFilesFromDirectories([]string{tmpDir})
output, err := GetFilesFromDirectories(context.Background(), []string{tmpDir})
if err != nil {
t.Errorf("Expected no error, got %v", err)
@@ -228,7 +264,7 @@ func TestGetFilesFromDirectoriesWithExcludes(t *testing.T) {
regexp.MustCompile(`\.tmp$`),
}
output, err := GetFilesFromDirectories([]string{tmpDir})
output, err := GetFilesFromDirectories(context.Background(), []string{tmpDir})
if err != nil {
t.Fatalf("Unexpected error: %v", err)
}

View File

@@ -95,6 +95,17 @@ func LoadConfig(path string) error {
return fmt.Errorf("Config option backup_timeout must be between %d and %d minutes, got %d", backupTimeoutMin, backupTimeoutMax, C.BackupTimeout)
}
// DirScanTimeout: set default, then validate range.
if C.DirScanTimeout == 0 {
C.DirScanTimeout = defaultDirScanTimeout
log.Warnf("Config option dir_scan_timeout is not set, fallback to %d minutes", C.DirScanTimeout)
}
if C.DirScanTimeout < dirScanTimeoutMin || C.DirScanTimeout > dirScanTimeoutMax {
return fmt.Errorf("Config option dir_scan_timeout must be between %d and %d minutes, got %d", dirScanTimeoutMin, dirScanTimeoutMax, C.DirScanTimeout)
}
// CompressionLevel: set default, then validate range.
if C.CompressionLevel == 0 {
C.CompressionLevel = defaultCompressionLevel
@@ -146,7 +157,7 @@ func LoadConfig(path string) error {
switch C.DefaultCompression {
case "gzip", "pgzip", "bzip2", "xz", "lz4", "zstd":
log.Infof("Using %s as defailt compression algorithm", C.DefaultCompression)
// No-op: compression algorithm validated.
default:
return fmt.Errorf("Config option default_compression must be gzip, bzip2, zstd, lz4, xz or pgzip, got %s", C.DefaultCompression)
}

View File

@@ -1,7 +1,9 @@
package backer
import (
"encoding/json"
"os"
"path/filepath"
"slices"
"testing"
)
@@ -10,18 +12,46 @@ func TestLoadConfig(t *testing.T) {
// Reset global config
C = Config{}
var (
testConfigPath = "./../../test_data/test_config.json"
expectedData = Config{
testDataPath := filepath.Join(t.TempDir(), "test_data")
os.MkdirAll(filepath.Join(testDataPath, "test1", "foo"), 0755)
os.MkdirAll(filepath.Join(testDataPath, "test1", "bar"), 0755)
// Create test config file
testConfigPath := filepath.Join(testDataPath, "test_config.json")
testStruct := struct {
Address string `json:"address"`
Port int `json:"port"`
Directories []string `json:"directories"`
Location string `json:"location"`
User string `json:"user"`
Password string `json:"password"`
NoHTTPS bool `json:"nohttps"`
}{
Address: "127.0.0.1",
Port: 8086,
Directories: []string{filepath.Join(testDataPath, "test1", "foo"), filepath.Join(testDataPath, "test1", "bar")},
Location: "/archive",
User: "test",
Password: "test",
NoHTTPS: true,
}
testConfigBytes, _ := json.MarshalIndent(testStruct, "", " ")
if err := os.WriteFile(testConfigPath, testConfigBytes, 0644); err != nil {
t.Fatal(err)
}
expectedData := Config{
Address: "127.0.0.1",
Port: 8086,
Directories: []string{
"../../test_data/test1/foo",
"../../test_data/test1/bar",
filepath.Join(testDataPath, "test1", "foo"),
filepath.Join(testDataPath, "test1", "bar"),
},
Location: "/archive",
User: "test",
Password: "test",
NoHTTPS: true,
}
)
err := LoadConfig(testConfigPath)
@@ -57,6 +87,8 @@ func TestLoadConfig(t *testing.T) {
// TestLoadConfigMissingUser tests that config fails when user is missing.
func TestLoadConfigMissingUser(t *testing.T) {
// Reset global config
C = Config{}
config := `{
"port": 8086,
"password": "test",
@@ -72,6 +104,8 @@ func TestLoadConfigMissingUser(t *testing.T) {
// TestLoadConfigMissingPassword tests that config fails when password is missing.
func TestLoadConfigMissingPassword(t *testing.T) {
// Reset global config
C = Config{}
config := `{
"port": 8086,
"user": "test",
@@ -87,6 +121,8 @@ func TestLoadConfigMissingPassword(t *testing.T) {
// TestLoadConfigMissingDirectories tests that config fails when directories is missing.
func TestLoadConfigMissingDirectories(t *testing.T) {
// Reset global config
C = Config{}
config := `{
"port": 8086,
"user": "test",
@@ -102,6 +138,8 @@ func TestLoadConfigMissingDirectories(t *testing.T) {
// TestLoadConfigInvalidPort tests that config fails with invalid port.
func TestLoadConfigInvalidPort(t *testing.T) {
// Reset global config
C = Config{}
config := `{
"port": 99999,
"user": "test",
@@ -118,6 +156,8 @@ func TestLoadConfigInvalidPort(t *testing.T) {
// TestLoadConfigNonExistentDirectory tests that config fails with non-existent directory.
func TestLoadConfigNonExistentDirectory(t *testing.T) {
// Reset global config
C = Config{}
config := `{
"port": 8086,
"user": "test",
@@ -134,6 +174,8 @@ func TestLoadConfigNonExistentDirectory(t *testing.T) {
// TestLoadConfigHTTPSWithoutCert tests that config fails when HTTPS enabled but no cert.
func TestLoadConfigHTTPSWithoutCert(t *testing.T) {
// Reset global config
C = Config{}
config := `{
"port": 8086,
"user": "test",
@@ -151,6 +193,8 @@ func TestLoadConfigHTTPSWithoutCert(t *testing.T) {
// TestLoadConfigNoHTTPS tests that config succeeds with nohttps=true and no cert.
func TestLoadConfigNoHTTPS(t *testing.T) {
// Reset global config
C = Config{}
config := `{
"port": 8086,
"user": "test",
@@ -172,6 +216,8 @@ func TestLoadConfigNoHTTPS(t *testing.T) {
// TestLoadConfigFilenamePrefixCustom tests custom filename prefix.
func TestLoadConfigFilenamePrefixCustom(t *testing.T) {
// Reset global config
C = Config{}
config := `{
"user": "test",
"password": "test",
@@ -193,6 +239,8 @@ func TestLoadConfigFilenamePrefixCustom(t *testing.T) {
// TestLoadConfigDefaultValues tests default values are set correctly.
func TestLoadConfigDefaultValues(t *testing.T) {
// Reset global config
C = Config{}
config := `{
"user": "test",
"password": "test",
@@ -234,6 +282,8 @@ func TestLoadConfigDefaultValues(t *testing.T) {
// TestLoadConfigCompressionAlgorithmBzip2 tests bzip2 compression algorithm.
func TestLoadConfigCompressionAlgorithmBzip2(t *testing.T) {
// Reset global config
C = Config{}
config := `{
"user": "test",
"password": "test",
@@ -255,6 +305,8 @@ func TestLoadConfigCompressionAlgorithmBzip2(t *testing.T) {
// TestLoadConfigCompressionAlgorithmZstd tests zstd compression algorithm.
func TestLoadConfigCompressionAlgorithmZstd(t *testing.T) {
// Reset global config
C = Config{}
config := `{
"user": "test",
"password": "test",
@@ -276,6 +328,8 @@ func TestLoadConfigCompressionAlgorithmZstd(t *testing.T) {
// TestLoadConfigCompressionAlgorithmLz4 tests lz4 compression algorithm.
func TestLoadConfigCompressionAlgorithmLz4(t *testing.T) {
// Reset global config
C = Config{}
config := `{
"user": "test",
"password": "test",
@@ -297,6 +351,8 @@ func TestLoadConfigCompressionAlgorithmLz4(t *testing.T) {
// TestLoadConfigCompressionAlgorithmXz tests xz compression algorithm.
func TestLoadConfigCompressionAlgorithmXz(t *testing.T) {
// Reset global config
C = Config{}
config := `{
"user": "test",
"password": "test",
@@ -318,6 +374,8 @@ func TestLoadConfigCompressionAlgorithmXz(t *testing.T) {
// TestLoadConfigCompressionAlgorithmPgzip tests pgzip compression algorithm.
func TestLoadConfigCompressionAlgorithmPgzip(t *testing.T) {
// Reset global config
C = Config{}
config := `{
"user": "test",
"password": "test",
@@ -339,6 +397,8 @@ func TestLoadConfigCompressionAlgorithmPgzip(t *testing.T) {
// TestLoadConfigCompressionAlgorithmInvalid tests invalid compression algorithm.
func TestLoadConfigCompressionAlgorithmInvalid(t *testing.T) {
// Reset global config
C = Config{}
config := `{
"user": "test",
"password": "test",
@@ -367,7 +427,6 @@ func writeTempConfig(t *testing.T, content string) string {
tmpFile.Close()
// Reset global config before loading
C = Config{}
excludePatterns = nil
return tmpFile.Name()
}

View File

@@ -10,8 +10,26 @@ import (
// CreateTarLz4Stream takes a list of files and returns a reader that reads from an lz4-compressed tar archive.
func CreateTarLz4Stream(ctx context.Context, filepaths []string) io.ReadCloser {
return createArchiveStream(ctx, filepaths, func(w io.Writer) (io.WriteCloser, error) {
return lz4.NewWriter(w), nil
lz4Writer := lz4.NewWriter(w)
// Map compression_level (1-9) to lz4 compression level.
// 1-3: fast, 4-6: balanced, 7-9: better compression.
lz4Writer.Header.CompressionLevel = mapCompressionLevelToLz4(C.CompressionLevel) //nolint: staticcheck
return lz4Writer, nil
})
}
// mapCompressionLevelToLz4 maps 1-9 compression level to lz4 compression level.
// 1-3: fast, 4-6: balanced, 7-9: better compression.
func mapCompressionLevelToLz4(level int) int {
switch {
case level <= 3:
return 0 // Fastest.
case level <= 6:
return 1 // Default.
default:
return 2 // Best compression.
}
}
/* vim: setlocal ft=go noet ai ts=4 sw=4 sts=4: */

View File

@@ -2,7 +2,11 @@ package backer
import (
"context"
"crypto/md5"
"crypto/subtle"
"crypto/tls"
"encoding/hex"
"errors"
"fmt"
"io"
"net"
@@ -14,6 +18,36 @@ import (
"backer/internal/log"
)
// isTLSError checks if the given error is TLS-related.
// It checks for both TLS-specific error types and error messages containing "tls:".
func isTLSError(err error) bool {
if err == nil {
return false
}
var alertErr *tls.AlertError
if errors.As(err, &alertErr) && alertErr != nil {
return true
}
var certErr *tls.CertificateVerificationError
if errors.As(err, &certErr) && certErr != nil {
return true
}
var echErr *tls.ECHRejectionError
if errors.As(err, &echErr) && echErr != nil {
return true
}
var recordErr *tls.RecordHeaderError
if errors.As(err, &recordErr) && recordErr != nil {
return true
}
return strings.HasPrefix(err.Error(), "tls:")
}
// withServerHeader wraps a handler to add Server header to all responses.
func withServerHeader(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
@@ -23,7 +57,8 @@ func withServerHeader(next http.Handler) http.Handler {
}
// NewServer starts new server instance with given config file.
func NewServer(configPath string) (*http.Server, error) {
// It returns a ServerWrapper that intercepts and classifies server errors.
func NewServer(configPath string) (*ServerWrapper, error) {
_ = log.Init("info", "") // Initialize logger with defaults for early logging.
if err := LoadConfig(configPath); err != nil {
@@ -34,9 +69,11 @@ func NewServer(configPath string) (*http.Server, error) {
log.Errorf("Failed to initialize logger: %v", err)
}
log.Infof("Using %s as default compression algorithm", C.DefaultCompression)
mux := http.NewServeMux()
mux.HandleFunc(C.Location, func(w http.ResponseWriter, r *http.Request) {
backupHandler := func(w http.ResponseWriter, r *http.Request) {
username, password, ok := r.BasicAuth()
// No auth? Good bye.
@@ -44,16 +81,18 @@ func NewServer(configPath string) (*http.Server, error) {
w.Header().Set("WWW-Authenticate", `Basic realm="restricted", charset="UTF-8"`)
http.Error(w, "Unauthorized", http.StatusUnauthorized)
log.Infof("Access denied from %s to %s (no username or password given)", r.RemoteAddr, r.RequestURI)
return
}
// Both username AND password must match. Explicitly require that the whole condition
// fails when either is incorrect, per the authentication design.
//nolint:staticcheck // QF1001: De Morgan's law is irrelevant here.
if !(username == C.User && password == C.Password) {
// Both username AND password must match. Use timing-safe comparison to prevent timing attacks.
if !timingSafeMatch(username, C.User) || !timingSafeMatch(password, C.Password) {
w.Header().Set("WWW-Authenticate", `Basic realm="restricted", charset="UTF-8"`)
http.Error(w, "Unauthorized", http.StatusUnauthorized)
log.Infof("Access denied from %s to %s (username and/or password incorrect)", r.RemoteAddr, r.RequestURI)
return
}
@@ -62,25 +101,41 @@ func NewServer(configPath string) (*http.Server, error) {
log.Infof("Backup started: client=%s user=%s", clientIP, username)
ctx := r.Context()
files, err := GetFilesFromDirectories(C.Directories)
// Create context with dir_scan_timeout from request context.
ctx, cancel := context.WithTimeout(r.Context(), time.Duration(C.DirScanTimeout)*time.Minute)
defer cancel()
files, err := GetFilesFromDirectories(ctx, C.Directories)
if err != nil {
switch {
case errors.Is(err, context.DeadlineExceeded):
w.Header().Set("Server", "backer")
http.Error(w, "Directory scan timed out", http.StatusRequestTimeout)
log.Errorf("Directory scan timed out: %v", err)
default:
log.Errorf("Failed to get files: %v", err)
w.Header().Set("Server", "backer")
http.Error(w, "Failed to get files", http.StatusInternalServerError)
}
return
}
if len(files) == 0 {
log.Warnf("No files found to backup in configured directories")
}
var (
archive io.ReadCloser
extension string
algorithm string
)
requestedExt := strings.TrimPrefix(path.Ext(r.URL.Path), ".")
requestedExt := r.URL.Path
algorithm = getCompressionAlgorithm(requestedExt)
@@ -91,7 +146,11 @@ func NewServer(configPath string) (*http.Server, error) {
case "zstd":
archive = CreateTarZstdStream(ctx, files)
if strings.HasSuffix(requestedExt, ".zst") {
extension = "tar.zst"
} else {
extension = "tar.zstd"
}
case "lz4":
archive = CreateTarLz4Stream(ctx, files)
@@ -116,44 +175,80 @@ func NewServer(configPath string) (*http.Server, error) {
}
}()
timestamp := time.Now().Format("20060102-150405")
timestamp := backupStart.Format("20060102-150405")
filename := fmt.Sprintf("%s-%s.%s", C.FilenamePrefix, timestamp, extension)
w.Header().Set("Content-Disposition", "attachment; filename="+filename)
w.Header().Set("Content-Type", "application/octet-stream")
bytesWritten, err := io.Copy(w, archive)
bytesWritten, md5Hash, err := copyWithContext(ctx, w, archive)
if err != nil {
duration := time.Since(backupStart)
if err != nil {
if errors.Is(err, context.Canceled) {
log.Infof("Backup canceled: client=%s user=%s duration=%s", clientIP, username, duration)
return
}
if isPipeClosedError(err) {
log.Infof("Client disconnected during backup: client=%s user=%s", clientIP, username)
return
}
log.Errorf("Backup failed: client=%s user=%s duration=%s error=%v", clientIP, username, duration, err)
return
}
log.Infof("Backup completed: client=%s user=%s files=%d bytes=%d duration=%s", clientIP, username, len(files), bytesWritten, duration)
})
log.Infof("Backup completed: client=%s user=%s files=%d bytes=%d md5=%s duration=%s", clientIP, username, len(files), bytesWritten, md5Hash, time.Since(backupStart))
}
mux.Handle(C.Location, http.HandlerFunc(backupHandler))
extensions := []string{".tar.gz", ".tar.xz", ".tar.bz2", ".tar.lz4", ".tar.zst", ".tar.zstd"}
for _, ext := range extensions {
muxPath := C.Location + ext
mux.Handle(muxPath, http.HandlerFunc(backupHandler))
}
// HTTP mode if nohttps is enabled in config.
// HTTPS mode if nohttps is disabled in config.
if C.NoHTTPS {
return &http.Server{
srv := &ServerWrapper{
Server: &http.Server{
Addr: fmt.Sprintf("%s:%d", C.Address, C.Port),
Handler: withServerHeader(mux),
ReadHeaderTimeout: readHeaderTimeout,
WriteTimeout: time.Duration(C.BackupTimeout) * time.Minute,
}, nil
IdleTimeout: idleTimeout,
MaxHeaderBytes: maxHeaderBytes,
ErrorLog: log.DebugLogger(),
},
}
srv.SetKeepAlivesEnabled(false)
return srv, nil
}
return &http.Server{
srv := &ServerWrapper{
Server: &http.Server{
Addr: fmt.Sprintf("%s:%d", C.Address, C.Port),
Handler: withServerHeader(mux),
ReadHeaderTimeout: readHeaderTimeout,
WriteTimeout: time.Duration(C.BackupTimeout) * time.Minute,
IdleTimeout: idleTimeout,
MaxHeaderBytes: maxHeaderBytes,
TLSConfig: &tls.Config{
MinVersion: tls.VersionTLS13,
},
}, nil
ErrorLog: log.DebugLogger(),
},
}
srv.SetKeepAlivesEnabled(false)
return srv, nil
}
// writeWithContext checks if context is canceled before executing fn.
@@ -169,31 +264,50 @@ func writeWithContext(ctx context.Context, fn func() error) error {
}
// copyWithContext copies data from src to dst using a 32KB buffer, respecting
// context cancellation. If the context is canceled, the copy stops and returns
// ctx.Err().
func copyWithContext(ctx context.Context, dst io.Writer, src io.Reader) error {
// context cancellation. Returns bytes written, MD5 hash (hex string), and any error encountered.
//
//nolint:gosec
func copyWithContext(ctx context.Context, dst io.Writer, src io.Reader) (int64, string, error) {
buf := make([]byte, copyBufferSize)
var bytesWritten int64
hasher := md5.New()
for {
select {
case <-ctx.Done():
return ctx.Err()
return bytesWritten, "", ctx.Err()
default:
}
n, readErr := src.Read(buf)
if n > 0 {
if _, werr := dst.Write(buf[:n]); werr != nil {
return werr
select {
case <-ctx.Done():
return bytesWritten, "", ctx.Err()
default:
}
if _, werr := hasher.Write(buf[:n]); werr != nil {
return bytesWritten, "", werr
}
if _, werr := dst.Write(buf[:n]); werr != nil {
return bytesWritten, "", werr
}
bytesWritten += int64(n)
}
if readErr != nil {
if readErr == io.EOF {
return nil
hash := hex.EncodeToString(hasher.Sum(nil))
return bytesWritten, hash, nil
}
return readErr
return bytesWritten, "", readErr
}
}
}
@@ -216,21 +330,87 @@ func getClientIP(r *http.Request) string {
// getCompressionAlgorithm determines which compression algorithm to use based on the requested extension.
// If the extension is recognized, it returns the corresponding algorithm; otherwise, it returns the default.
// For .tar.gz extension, pgzip is always used for parallel compression.
func getCompressionAlgorithm(requestedExt string) string {
switch requestedExt {
case "tar.bz2":
// The input can be either a full path like "/archive.tar.xz" or just the extension like "tar.xz" or "xz".
func getCompressionAlgorithm(input string) string {
// Extract extension from full path (e.g., "/archive.tar.xz" -> ".xz").
// or use the input directly if it's just an extension (e.g., "tar.xz" -> ".xz").
ext := path.Ext(input)
if ext == "" {
ext = "." + input
}
switch ext {
case ".tar.bz2":
return "bzip2"
case "tar.zst":
case ".tar.zstd":
return "zstd"
case "tar.lz4":
case ".tar.zst":
return "zstd"
case ".tar.lz4":
return "lz4"
case "tar.xz":
case ".tar.xz":
return "xz"
case "tar.gz":
case ".tar.gz":
return "pgzip"
case ".xz":
return "xz"
case ".bz2":
return "bzip2"
case ".zst":
return "zstd"
case ".zstd":
return "zstd"
case ".lz4":
return "lz4"
case ".gz":
return "pgzip"
default:
return C.DefaultCompression
}
}
type (
// ServerWrapper wraps [http.Server] to intercept and classify errors.
ServerWrapper struct {
*http.Server
}
)
// Serve starts the server and classifies any errors returned.
// TLS-related errors are logged at debug level, other errors at warn level.
func (sw *ServerWrapper) Serve() error {
err := sw.ListenAndServe()
if err != nil {
if isTLSError(err) {
log.Debugf("Server error: %v", err)
} else {
log.Warnf("Server error: %v", err)
}
}
return err
}
// ServeTLS starts the TLS server and classifies any errors returned.
func (sw *ServerWrapper) ServeTLS(certFile, keyFile string) error {
err := sw.ListenAndServeTLS(certFile, keyFile)
if err != nil {
if isTLSError(err) {
log.Debugf("Server TLS error: %v", err)
} else {
log.Warnf("Server TLS error: %v", err)
}
}
return err
}
// timingSafeMatch performs constant-time comparison to prevent timing attacks.
// Returns 1 if the strings are equal, 0 otherwise.
func timingSafeMatch(a, b string) bool {
return subtle.ConstantTimeCompare([]byte(a), []byte(b)) == 1
}
/* vim: setlocal ft=go noet ai ts=4 sw=4 sts=4: */

View File

@@ -4,6 +4,8 @@ import (
"archive/tar"
"compress/gzip"
"context"
"crypto/tls"
"errors"
"io"
"net/http"
"net/http/httptest"
@@ -11,6 +13,166 @@ import (
"testing"
)
// TestIsTLSError tests the isTLSError function.
func TestIsTLSError(t *testing.T) {
tests := []struct {
name string
err error
expected bool
}{
{
name: "nil error",
err: nil,
expected: false,
},
{
name: "regular error",
err: errors.New("some random error"),
expected: false,
},
{
name: "TLS prefix error",
err: errors.New("tls: handshake failure"),
expected: true,
},
{
name: "TLS AlertError",
err: tls.AlertError(80),
expected: true,
},
{
name: "TLS CertificateVerificationError",
err: &tls.CertificateVerificationError{},
expected: true,
},
{
name: "TLS ECHRejectionError",
err: &tls.ECHRejectionError{RetryConfigList: []byte("rejected")},
expected: true,
},
{
name: "TLS RecordHeaderError",
err: &tls.RecordHeaderError{},
expected: true,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
result := isTLSError(tt.err)
if result != tt.expected {
t.Errorf("isTLSError(%v) = %v, expected %v", tt.err, result, tt.expected)
}
})
}
}
// TestGetClientIP tests the getClientIP function.
func TestGetClientIP(t *testing.T) {
tests := []struct {
name string
xffHeader string
remoteAddr string
expectedIP string
}{
{
name: "no X-Forwarded-For",
xffHeader: "",
remoteAddr: "192.168.1.100:12345",
expectedIP: "192.168.1.100",
},
{
name: "single IP in X-Forwarded-For",
xffHeader: "10.0.0.1",
remoteAddr: "192.168.1.100:12345",
expectedIP: "10.0.0.1",
},
{
name: "multiple IPs in X-Forwarded-For",
xffHeader: "10.0.0.1, 192.168.1.100, 172.16.0.1",
remoteAddr: "192.168.1.100:12345",
expectedIP: "10.0.0.1",
},
{
name: "X-Forwarded-For with spaces",
xffHeader: " 10.0.0.1 , 192.168.1.100 ",
remoteAddr: "192.168.1.100:12345",
expectedIP: "10.0.0.1",
},
{
name: "invalid RemoteAddr",
xffHeader: "",
remoteAddr: "invalid",
expectedIP: "invalid",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
req := httptest.NewRequest("GET", "/archive", nil)
if tt.xffHeader != "" {
req.Header.Set("X-Forwarded-For", tt.xffHeader)
}
req.RemoteAddr = tt.remoteAddr
result := getClientIP(req)
if result != tt.expectedIP {
t.Errorf("getClientIP() = %q, expected %q", result, tt.expectedIP)
}
})
}
}
// TestGetCompressionAlgorithm tests the getCompressionAlgorithm function.
func TestGetCompressionAlgorithm(t *testing.T) {
tests := []struct {
name string
ext string
expected string
}{
{
name: "tar.bz2",
ext: "tar.bz2",
expected: "bzip2",
},
{
name: "tar.zst",
ext: "tar.zst",
expected: "zstd",
},
{
name: "tar.lz4",
ext: "tar.lz4",
expected: "lz4",
},
{
name: "tar.xz",
ext: "tar.xz",
expected: "xz",
},
{
name: "tar.gz",
ext: "tar.gz",
expected: "pgzip",
},
{
name: "default with gzip config",
ext: "",
expected: "gzip",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
C = Config{DefaultCompression: "gzip"}
result := getCompressionAlgorithm(tt.ext)
if result != tt.expected {
t.Errorf("getCompressionAlgorithm(%q) = %q, expected %q", tt.ext, result, tt.expected)
}
})
}
}
// TestNewServerSuccess tests successful server creation.
func TestNewServerSuccess(t *testing.T) {
C = Config{} // Reset global config.
@@ -189,7 +351,7 @@ func TestHandlerArchiveStream(t *testing.T) {
}
ctx := r.Context()
files, err := GetFilesFromDirectories(C.Directories)
files, err := GetFilesFromDirectories(ctx, C.Directories)
if err != nil {
http.Error(w, "Failed to get files", http.StatusInternalServerError)
return
@@ -301,7 +463,7 @@ func TestCopyWithContextSuccess(t *testing.T) {
src := strings.NewReader("hello world")
dst := &strings.Builder{}
err := copyWithContext(ctx, dst, src)
_, _, err := copyWithContext(ctx, dst, src)
if err != nil {
t.Errorf("Expected no error, got %v", err)
}
@@ -311,6 +473,25 @@ func TestCopyWithContextSuccess(t *testing.T) {
}
}
// TestCopyWithContextMD5Hash tests that MD5 hash is correctly computed.
func TestCopyWithContextMD5Hash(t *testing.T) {
ctx := context.Background()
data := "hello world"
src := strings.NewReader(data)
dst := &strings.Builder{}
_, md5Hash, err := copyWithContext(ctx, dst, src)
if err != nil {
t.Errorf("Expected no error, got %v", err)
}
// MD5 of "hello world" is 5eb63bbbe01eeed093cb22bb8f5acdc3
expectedMD5 := "5eb63bbbe01eeed093cb22bb8f5acdc3"
if md5Hash != expectedMD5 {
t.Errorf("Expected MD5 %s, got %s", expectedMD5, md5Hash)
}
}
// TestCopyWithContextCancellation tests copy with cancelled context.
func TestCopyWithContextCancellation(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
@@ -325,14 +506,18 @@ func TestCopyWithContextCancellation(t *testing.T) {
// Start copy in goroutine
errCh := make(chan error, 1)
bytesCh := make(chan int64, 1)
go func() {
errCh <- copyWithContext(ctx, dst, src)
n, _, err := copyWithContext(ctx, dst, src)
bytesCh <- n
errCh <- err
}()
// Cancel context immediately
cancel()
err := <-errCh
<-bytesCh
if err != context.Canceled {
t.Errorf("Expected context.Canceled, got %v", err)
}

View File

@@ -15,12 +15,23 @@ type (
LogLevel string `json:"loglevel"`
Directories []string `json:"directories"`
BackupTimeout int `json:"backup_timeout"` // Timeout in minutes for backup streaming.
DirScanTimeout int `json:"dir_scan_timeout"` // Timeout in minutes for directory scanning.
CompressionLevel int `json:"compression_level"` // Compression level (1-9, default 9).
ExcludePatterns []string `json:"exclude_patterns"` // Regex patterns to exclude from backup.
FilenamePrefix string `json:"filename_prefix"` // Prefix for backup filename (default: "backup").
CompressionAlgorithm string `json:"compression_algorithm"` // Compression algorithm alias (deprecated: use default_compression).
DefaultCompression string `json:"default_compression"` // Default compression algorithm: gzip, pgzip, bzip2, zstd, lz4 or xz.
}
// ErrDirectoryScanTimeout is returned when directory scan times out.
ErrDirectoryScanTimeout struct {
ScannedFiles int
}
)
// Error returns the error message for ErrDirectoryScanTimeout.
func (e ErrDirectoryScanTimeout) Error() string {
return "Directory scan timed out"
}
/* vim: setlocal ft=go noet ai ts=4 sw=4 sts=4: */

View File

@@ -7,6 +7,7 @@ import (
"io"
"os"
"path/filepath"
"runtime"
"testing"
)
@@ -179,6 +180,73 @@ func TestCreateTarGzStreamSymlink(t *testing.T) {
}
}
// TestCreateTarGzStreamHardLink tests hard link handling - should deduplicate content.
func TestCreateTarGzStreamHardLink(t *testing.T) {
if runtime.GOOS == "windows" {
t.Skip("Hard link detection not supported on Windows (inodes unavailable)")
}
tmpDir := t.TempDir()
// Create original file
originalFile := filepath.Join(tmpDir, "original.txt")
if err := os.WriteFile(originalFile, []byte("shared content"), 0644); err != nil {
t.Fatal(err)
}
// Create hard link
hardLinkFile := filepath.Join(tmpDir, "hardlink.txt")
if err := os.Link(originalFile, hardLinkFile); err != nil {
t.Skip("Hard links not supported, skipping test")
}
ctx := context.Background()
reader := CreateTarGzStream(ctx, []string{originalFile, hardLinkFile})
defer reader.Close()
gzReader, err := gzip.NewReader(reader)
if err != nil {
t.Fatalf("Failed to create gzip reader: %v", err)
}
defer gzReader.Close()
tarReader := tar.NewReader(gzReader)
entries := 0
linkCount := 0
for {
header, err := tarReader.Next()
if err == io.EOF {
break
}
if err != nil {
t.Fatalf("Failed to read tar entry: %v", err)
}
entries++
t.Logf("Entry %d: name=%s typeflag=%c linkname=%s", entries, header.Name, header.Typeflag, header.Linkname)
if header.Typeflag == tar.TypeLink {
linkCount++
// Verify it's a link to the original
if header.Linkname != originalFile && header.Linkname != filepath.ToSlash(originalFile) {
t.Errorf("Expected linkname '%s', got '%s'", originalFile, header.Linkname)
}
}
}
// Should have 2 entries: one file content, one as hard link
if entries != 2 {
t.Errorf("Expected 2 entries, got %d", entries)
}
// One should be a hard link (TypeLink)
if linkCount != 1 {
t.Errorf("Expected 1 hard link entry, got %d", linkCount)
}
}
// TestCreateTarGzStreamCancellation tests context cancellation.
func TestCreateTarGzStreamCancellation(t *testing.T) {
tmpDir := t.TempDir()
@@ -356,4 +424,144 @@ func TestCreateTarGzStreamDeviceFile(t *testing.T) {
}
}
// TestCompressionLevelMappingLz4 tests that lz4 compression levels are correctly mapped.
func TestCompressionLevelMappingLz4(t *testing.T) {
tests := []struct {
input int
expected int
}{
{1, 0},
{2, 0},
{3, 0},
{4, 1},
{5, 1},
{6, 1},
{7, 2},
{8, 2},
{9, 2},
}
for _, tt := range tests {
C = Config{CompressionLevel: tt.input}
result := mapCompressionLevelToLz4(tt.input)
if result != tt.expected {
t.Errorf("mapCompressionLevelToLz4(%d) = %d, expected %d", tt.input, result, tt.expected)
}
}
}
// TestCompressionLevelMappingXz tests that xz compression is not configurable via level.
// xz uses default compression - this test verifies the stream works.
func TestCompressionLevelMappingXz(t *testing.T) {
// xz doesn't support configurable compression levels in the same way.
// It always uses default compression. This test verifies the stream is created successfully.
C = Config{CompressionLevel: 6}
tmpDir := t.TempDir()
testFile := filepath.Join(tmpDir, "test.txt")
content := []byte("test content")
if err := os.WriteFile(testFile, content, 0644); err != nil {
t.Fatal(err)
}
ctx := context.Background()
reader := CreateTarXzStream(ctx, []string{testFile})
defer reader.Close()
data, err := io.ReadAll(reader)
if err != nil {
t.Fatalf("Failed to read xz archive: %v", err)
}
if len(data) == 0 {
t.Error("Expected non-empty archive")
}
}
// TestCompressionLevelMappingBzip2 tests that bzip2 compression levels are correctly mapped.
func TestCompressionLevelMappingBzip2(t *testing.T) {
tests := []struct {
input int
expected int
}{
{1, 1},
{5, 5},
{9, 9},
}
for _, tt := range tests {
C = Config{CompressionLevel: tt.input}
result := mapCompressionLevelToBzip2(tt.input)
if result != tt.expected {
t.Errorf("mapCompressionLevelToBzip2(%d) = %d, expected %d", tt.input, result, tt.expected)
}
}
}
// TestCreateTarBzip2Stream tests bzip2 compression with configured level.
func TestCreateTarBzip2Stream(t *testing.T) {
C = Config{CompressionLevel: 6}
tmpDir := t.TempDir()
testFile := filepath.Join(tmpDir, "test.txt")
content := []byte("test content for bzip2")
if err := os.WriteFile(testFile, content, 0644); err != nil {
t.Fatal(err)
}
ctx := context.Background()
reader := CreateTarBzip2Stream(ctx, []string{testFile})
defer reader.Close()
// Verify we can read the archive
_, err := io.ReadAll(reader)
if err != nil {
t.Fatalf("Failed to read bzip2 archive: %v", err)
}
}
// TestCreateTarLz4Stream tests lz4 compression with configured level.
func TestCreateTarLz4Stream(t *testing.T) {
C = Config{CompressionLevel: 6}
tmpDir := t.TempDir()
testFile := filepath.Join(tmpDir, "test.txt")
content := []byte("test content for lz4")
if err := os.WriteFile(testFile, content, 0644); err != nil {
t.Fatal(err)
}
ctx := context.Background()
reader := CreateTarLz4Stream(ctx, []string{testFile})
defer reader.Close()
// Verify we can read the archive
_, err := io.ReadAll(reader)
if err != nil {
t.Fatalf("Failed to read lz4 archive: %v", err)
}
}
// TestCreateTarXzStream tests xz compression with configured level.
func TestCreateTarXzStream(t *testing.T) {
C = Config{CompressionLevel: 6}
tmpDir := t.TempDir()
testFile := filepath.Join(tmpDir, "test.txt")
content := []byte("test content for xz")
if err := os.WriteFile(testFile, content, 0644); err != nil {
t.Fatal(err)
}
ctx := context.Background()
reader := CreateTarXzStream(ctx, []string{testFile})
defer reader.Close()
// Verify we can read the archive
_, err := io.ReadAll(reader)
if err != nil {
t.Fatalf("Failed to read xz archive: %v", err)
}
}
/* vim: setlocal ft=go noet ai ts=4 sw=4 sts=4: */

View File

@@ -10,6 +10,8 @@ import (
// CreateTarXzStream takes a list of files and returns a reader that reads from an xz-compressed tar archive.
func CreateTarXzStream(ctx context.Context, filepaths []string) io.ReadCloser {
return createArchiveStream(ctx, filepaths, func(w io.Writer) (io.WriteCloser, error) {
// xz doesn't have a simple level parameter; use default compression.
// The compression level affects block size indirectly through the algorithm.
return xz.NewWriter(w)
})
}

View File

@@ -2,26 +2,56 @@
package log
import (
"context"
"fmt"
"log"
"log/slog"
"os"
"strings"
"sync"
"time"
)
var (
// Log contains *[os.File] pointing to log file descriptor.
// Protected by mu for thread-safe access, primarily to enable parallel tests.
// In production, this is set once at startup and never modified thereafter.
Log *os.File
// mu protects Log during initialization and when DebugLogger is called.
// This is necessary to allow parallel test execution where each test
// may initialize the logger with a different log file.
mu sync.RWMutex
// logLevel stores the configured log level for use by DebugLogger.
logLevel slog.Level
)
// slogWriter implements [io.Writer] to redirect standard log to slog.
type (
slogWriter struct {
handler slog.Handler
}
)
// Init sets up logger stuff.
// level can be error, warn, info, debug; if something other is supplied, info level is selected.
// Thread-safe: uses mutex to allow parallel test execution.
func Init(level, filename string) error {
mu.Lock()
defer mu.Unlock()
var (
err error
loglevel slog.Level
)
if filename != "" {
// Close existing log file if re-initializing (e.g., in tests).
if Log != nil && Log != os.Stderr {
Log.Close()
}
Log, err = os.OpenFile(filename, os.O_WRONLY|os.O_APPEND|os.O_CREATE, 0644)
if err != nil {
@@ -49,6 +79,8 @@ func Init(level, filename string) error {
loglevel = slog.LevelInfo
}
logLevel = loglevel
opts := &slog.HandlerOptions{
// Use the ReplaceAttr function on the handler options
// to be able to replace any single attribute in the log output.
@@ -122,6 +154,47 @@ func Debugf(format string, a ...any) {
slog.Debug(fmt.Sprintf(format, a...))
}
// DebugLogger returns a standard library logger that writes at the configured log level.
// This is useful for assigning to http.Server.ErrorLog to route net/http errors
// through our logger while respecting the configured log level.
func DebugLogger() *log.Logger {
mu.RLock()
defer mu.RUnlock()
opts := &slog.HandlerOptions{
Level: logLevel,
}
handler := slog.NewTextHandler(Log, opts)
return log.New(&slogWriter{handler: handler}, "", 0)
}
// Write implements [io.Writer]. It differentiates TLS/SSL errors from other errors.
// TLS-related errors are logged at debug level, while other errors (handler issues,
// unexpected behavior) are logged at warn level.
func (w *slogWriter) Write(p []byte) (int, error) {
msg := string(p)
isTLSError := strings.Contains(msg, "tls") ||
strings.Contains(msg, "ssl") ||
strings.Contains(msg, "TLS") ||
strings.Contains(msg, "SSL") ||
strings.Contains(msg, "handshake") ||
strings.Contains(msg, "certificate")
level := slog.LevelWarn
if isTLSError {
level = slog.LevelDebug
}
record := slog.NewRecord(time.Now(), level, msg, 0)
_ = w.handler.Handle(context.TODO(), record)
return len(p), nil
}
// Fatal logs args at error level and quits with status code 1.
func Fatal(args ...any) {
slog.Error(fmt.Sprint(args...))
@@ -138,7 +211,11 @@ func Fatalf(format string, a ...any) {
}
// Close closes log file descriptor.
// Thread-safe: uses mutex to coordinate with Init and DebugLogger.
func Close() {
mu.Lock()
defer mu.Unlock()
if Log == nil || Log == os.Stderr {
return
}
@@ -147,6 +224,8 @@ func Close() {
msg := fmt.Sprintln("Failed to close log file:", err)
slog.Error(msg)
}
Log = nil
}
/* vim: setlocal ft=go noet ai ts=4 sw=4 sts=4: */

View File

@@ -354,4 +354,35 @@ func TestCloseStderr(t *testing.T) {
}
}
// TestDebugLogger tests that DebugLogger returns a working *log.Logger.
func TestDebugLogger(t *testing.T) {
tmpFile, err := os.CreateTemp(t.TempDir(), "test_log_*.log")
if err != nil {
t.Fatal(err)
}
logFilePath := tmpFile.Name()
tmpFile.Close()
if err := Init("info", logFilePath); err != nil {
t.Fatal(err)
}
defer Close()
logger := DebugLogger()
if logger == nil {
t.Error("DebugLogger returned nil")
}
logger.Print("test debug message")
content, err := os.ReadFile(logFilePath)
if err != nil {
t.Fatal(err)
}
if !strings.Contains(string(content), "test debug message") {
t.Error("Expected debug message in log")
}
}
/* vim: setlocal ft=go noet ai ts=4 sw=4 sts=4: */

View File

@@ -24,7 +24,7 @@
"password": "SharpShooter",
/* Where to store logs. */
"log": "/var/log/backer.log",
"log": "../../test_data/tmp/backer.log",
/* Verbosity of logs */
"loglevel": "info",
@@ -39,5 +39,5 @@
"backup_timeout": 60,
/* Gzip compression level (1-9, where 9 is best compression). */
"compression_level": 6
"compression_level": 3
}