6 Commits

15 changed files with 474 additions and 64 deletions

4
.gitignore vendored
View File

@@ -30,4 +30,6 @@ Thumbs.db
*.test
*.out
/test_data/tmp
# Test data (generated by tests)
test_data/tmp
test_data/test1

View File

@@ -27,6 +27,36 @@ Backer is a Go-based HTTP/HTTPS backup server that creates compressed archives o
- **Don't use `context` package unless absolutely necessary.** Try to find another solution first.
- **Create functional and unit tests if possible/necessary** to verify changes that software code works at is should.
### Test Rules
- **Test data must be self-contained and created at test setup.** All test data should be created in `TestMain` in `internal/backer/main_test.go` before any tests run, not rely on pre-existing files in the repository. Use `test_data/test1/` as the test data directory.
- **Use `test_data/tmp/` for temporary files.** When tests need temporary storage, use this directory (not system `/tmp`) to keep test data isolated and predictable.
- **Always check errors with meaningful diagnostics.** Never discard errors with `_ =`. Use patterns like:
```go
if err := os.MkdirAll(dir, 0755); err != nil {
t.Fatalf("Failed to create directory %s: %v", dir, err)
}
```
- **Tests must not depend on external state.** Test data should be created fresh for each test run and cleaned up afterward. Use `TestMain` with setup/cleanup functions.
- **Use `t.Fatal` or `t.Fatalf` for setup errors.** When test setup fails, fail fast with clear diagnostic messages showing what failed and why.
- **Reset global state between tests.** For tests that modify global variables (like `C` config), save the original value and restore it in a `defer`:
```go
original := C
defer func() { C = original }()
```
- **Skip tests gracefully when features are unavailable.** For platform-specific features (hardlinks, symlinks), use `t.Skip()` when the feature is not supported:
```go
if err := os.Link(orig, link); err != nil {
t.Skip("Hard links not supported, skipping test")
}
```
## Build & Test Commands
```bash
@@ -359,10 +389,14 @@ Tests live alongside source files as `*_test.go` (package name, not `_test` suff
### Test Data
- `test_data/test_config.json` — valid test config (user: JohnnyGoode, password: SharpShooter)
- `test_data/test1/foo/` — test directory with files, subdirs, empty dirs
All test data is created by `TestMain` in `internal/backer/main_test.go` before tests run and cleaned up afterward. This ensures tests are independent of pre-existing files.
- `test_data/test1/foo/` — test directory with files, subdirs, empty_dir
- `test_data/test1/bar/` — test directory with files and a directory named `goodbye.txt/` (edge case: directory with file extension)
- `test_data/example.crt` / `example.key` — TLS test certificates
- `test_data/test1/hardlinks/` — test directory with hard-linked files (original.txt, hardlink1.txt, hardlink2.txt)
- `test_data/test1/symlinks/` — test directory with symbolic links
- `test_data/test1/empty_dir/` — empty directory for testing
- `test_data/tmp/` — reserved for test temporary files (use instead of system `/tmp`)
## Service Deployment

View File

@@ -23,13 +23,15 @@ build:
clean:
rm -f $(BINARY)
rm -rf test_data/tmp
rm -rf test_data/test1
upgrade:
go get -u ./...
go mod tidy
go mod vendor
test:
test: clean
go test ./...
help:

View File

@@ -9,6 +9,7 @@ import (
"os"
"path/filepath"
"strings"
"syscall"
"backer/internal/log"
)
@@ -85,6 +86,8 @@ func writeFilesToTar(ctx context.Context, filepaths []string, tw *tar.Writer, cl
var (
err error
linkTarget string
isSymlink bool
isHardLink bool
)
select {
@@ -106,6 +109,7 @@ func writeFilesToTar(ctx context.Context, filepaths []string, tw *tar.Writer, cl
continue
}
// Handle symlinks - read target path and mark as symbolic link.
if st.Mode()&os.ModeSymlink != 0 {
linkTarget, err = os.Readlink(fpath)
@@ -114,6 +118,8 @@ func writeFilesToTar(ctx context.Context, filepaths []string, tw *tar.Writer, cl
continue
}
isSymlink = true
}
// Skip sockets explicitly (they cannot be backed up).
@@ -124,12 +130,15 @@ func writeFilesToTar(ctx context.Context, filepaths []string, tw *tar.Writer, cl
}
// Check for hard link - if inode was already seen, store as link instead of content.
if st.Mode().IsRegular() {
// Only check for regular files (not symlinks).
if st.Mode().IsRegular() && !isSymlink {
inode := getInode(st)
if inode != 0 {
if existingPath, seen := seenInodes[inode]; seen {
// This is a hard link to already-archived file.
linkTarget = existingPath
isHardLink = true
log.Debugf("Hard link detected: %s -> %s", fpath, existingPath)
} else {
// First time seeing this inode - track it.
@@ -146,9 +155,13 @@ func writeFilesToTar(ctx context.Context, filepaths []string, tw *tar.Writer, cl
continue
}
// If linkTarget is set, ensure the header is marked as a link.
if linkTarget != "" {
// Set correct type flag for symbolic links or hard links.
if isSymlink {
header.Typeflag = tar.TypeSymlink
header.Linkname = linkTarget
} else if isHardLink {
header.Typeflag = tar.TypeLink
header.Linkname = linkTarget
}
header.Format = tar.FormatGNU
@@ -164,8 +177,8 @@ func writeFilesToTar(ctx context.Context, filepaths []string, tw *tar.Writer, cl
return
}
// Only stream file contents for regular files.
if mode.IsRegular() {
// Only stream file contents for regular files that are not hard links.
if mode.IsRegular() && linkTarget == "" {
f, err := os.Open(fpath)
if err != nil {
log.Warnf("Skipping %s: open failed: %v", fpath, err)
@@ -195,15 +208,12 @@ func getInode(fi os.FileInfo) uint64 {
return 0
}
// Use type assertion to get inode - works on Unix, returns 0 on Windows.
type inodeGetter interface {
Dev() uint64
Ino() uint64
}
if ig, ok := fi.Sys().(inodeGetter); ok {
// Combine device and inode for uniqueness across file systems.
return (ig.Dev() << 32) | ig.Ino()
// Use reflection to get Dev and Ino fields from the syscall.Stat_t struct.
st := fi.Sys()
// Try type assertion for the struct with fields.
switch v := st.(type) {
case *syscall.Stat_t:
return (v.Dev << 32) | v.Ino
}
return 0

View File

@@ -13,8 +13,12 @@ func TestLoadConfig(t *testing.T) {
C = Config{}
testDataPath := filepath.Join(t.TempDir(), "test_data")
os.MkdirAll(filepath.Join(testDataPath, "test1", "foo"), 0755)
os.MkdirAll(filepath.Join(testDataPath, "test1", "bar"), 0755)
if err := os.MkdirAll(filepath.Join(testDataPath, "test1", "foo"), 0755); err != nil {
t.Fatalf("Failed to create test directory foo: %v", err)
}
if err := os.MkdirAll(filepath.Join(testDataPath, "test1", "bar"), 0755); err != nil {
t.Fatalf("Failed to create test directory bar: %v", err)
}
// Create test config file
testConfigPath := filepath.Join(testDataPath, "test_config.json")

View File

@@ -0,0 +1,103 @@
package backer
import (
"context"
"fmt"
"os"
"path/filepath"
"testing"
)
const testDataPath = "../../test_data"
func TestMain(m *testing.M) {
if err := setupTestData(); err != nil {
fmt.Fprintf(os.Stderr, "FATAL: Failed to setup test data: %v\n", err)
os.Exit(1)
}
code := m.Run()
if err := cleanupTestData(); err != nil {
fmt.Fprintf(os.Stderr, "WARNING: Failed to cleanup test data: %v\n", err)
}
os.Exit(code)
}
func setupTestData() error {
testDir := filepath.Join(testDataPath, "test1")
dirs := []string{
filepath.Join(testDir, "foo", "mydir"),
filepath.Join(testDir, "bar", "goodbye.txt"),
filepath.Join(testDir, "hardlinks"),
filepath.Join(testDir, "symlinks"),
filepath.Join(testDir, "foo", "empty_dir"),
filepath.Join(testDataPath, "tmp"),
}
for _, dir := range dirs {
if err := os.MkdirAll(dir, 0755); err != nil {
return fmt.Errorf("Failed to create directory %s: %w", dir, err)
}
}
files := map[string]string{
filepath.Join(testDir, "foo", "hello_breakfast.txt"): "hello breakfast content",
filepath.Join(testDir, "foo", "some_text.txt"): "some text content",
filepath.Join(testDir, "foo", "mydir", "myfile.txt"): "my file content",
filepath.Join(testDir, "bar", "hello.txt"): "hello content",
filepath.Join(testDir, "bar", "goodbye.txt", "text.txt"): "text content",
filepath.Join(testDir, "hardlinks", "original.txt"): "original content",
filepath.Join(testDir, "symlinks", "target.txt"): "target content",
}
for path, content := range files {
if err := os.WriteFile(path, []byte(content), 0644); err != nil {
return fmt.Errorf("Failed to write file %s: %w", path, err)
}
}
orig := filepath.Join(testDir, "hardlinks", "original.txt")
link1 := filepath.Join(testDir, "hardlinks", "hardlink1.txt")
if err := os.Link(orig, link1); err != nil {
return fmt.Errorf("Failed to create hardlink %s -> %s: %w", link1, orig, err)
}
link2 := filepath.Join(testDir, "hardlinks", "hardlink2.txt")
if err := os.Link(orig, link2); err != nil {
return fmt.Errorf("Failed to create hardlink %s -> %s: %w", link2, orig, err)
}
target := filepath.Join(testDir, "symlinks", "target.txt")
link := filepath.Join(testDir, "symlinks", "link.txt")
if err := os.Symlink(target, link); err != nil {
return fmt.Errorf("Failed to create symlink %s -> %s: %w", link, target, err)
}
return nil
}
func cleanupTestData() error {
testDir := filepath.Join(testDataPath, "test1")
if err := os.RemoveAll(testDir); err != nil {
return fmt.Errorf("Failed to remove test directory %s: %w", testDir, err)
}
tmpDir := filepath.Join(testDataPath, "tmp")
if err := os.RemoveAll(tmpDir); err != nil {
return fmt.Errorf("Failed to remove tmp directory %s: %w", tmpDir, err)
}
return nil
}
func GetTestDataPath() string {
return testDataPath
}
func GetTestFilesFromDirectories(ctx context.Context, directories []string) ([]string, error) {
return GetFilesFromDirectories(ctx, directories)
}

View File

@@ -125,46 +125,43 @@ func TestGetClientIP(t *testing.T) {
// TestGetCompressionAlgorithm tests the getCompressionAlgorithm function.
func TestGetCompressionAlgorithm(t *testing.T) {
// Save original config
original := C
defer func() { C = original }()
tests := []struct {
name string
ext string
expected string
name string
ext string
expected string
defaultCompression string
}{
{
name: "tar.bz2",
ext: "tar.bz2",
expected: "bzip2",
},
{
name: "tar.zst",
ext: "tar.zst",
expected: "zstd",
},
{
name: "tar.lz4",
ext: "tar.lz4",
expected: "lz4",
},
{
name: "tar.xz",
ext: "tar.xz",
expected: "xz",
},
{
name: "tar.gz",
ext: "tar.gz",
expected: "pgzip",
},
{
name: "default with gzip config",
ext: "",
expected: "gzip",
},
{name: "tar.bz2", ext: "tar.bz2", expected: "bzip2"},
{name: "tar.zstd", ext: "tar.zstd", expected: "zstd"},
{name: "tar.zst", ext: "tar.zst", expected: "zstd"},
{name: "tar.lz4", ext: "tar.lz4", expected: "lz4"},
{name: "tar.xz", ext: "tar.xz", expected: "xz"},
{name: "tar.gz", ext: "tar.gz", expected: "pgzip"},
{name: "bz2", ext: "bz2", expected: "bzip2"},
{name: "zst", ext: "zst", expected: "zstd"},
{name: "lz4", ext: "lz4", expected: "lz4"},
{name: "xz", ext: "xz", expected: "xz"},
{name: "gz", ext: "gz", expected: "pgzip"},
{name: "zstd", ext: "zstd", expected: "zstd"},
{name: "full path /archive.tar.gz", ext: "/archive.tar.gz", expected: "pgzip"},
{name: "full path /archive.tar.xz", ext: "/archive.tar.xz", expected: "xz"},
{name: "full path /archive", ext: "/archive", expected: "gzip"},
{name: "empty uses default", ext: "", expected: "gzip"},
{name: "unknown uses default", ext: "unknown", expected: "gzip"},
{name: "unknown with default zstd", ext: "unknown", expected: "zstd", defaultCompression: "zstd"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
C = Config{DefaultCompression: "gzip"}
if tt.defaultCompression != "" {
C = Config{DefaultCompression: tt.defaultCompression}
} else {
C = Config{DefaultCompression: "gzip"}
}
result := getCompressionAlgorithm(tt.ext)
if result != tt.expected {
t.Errorf("getCompressionAlgorithm(%q) = %q, expected %q", tt.ext, result, tt.expected)
@@ -191,6 +188,46 @@ func TestNewServerSuccess(t *testing.T) {
}
}
// TestWithServerHeader tests that the Server header is added to responses.
func TestWithServerHeader(t *testing.T) {
handler := withServerHeader(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusOK)
}))
rec := &responseWriterWrapper{
ResponseWriter: &testResponseWriter{},
statusCode: 0,
}
req, _ := http.NewRequest("GET", "/test", nil)
handler.ServeHTTP(rec, req)
if rec.Header().Get("Server") != "backer" {
t.Error("Expected Server header to be set to 'backer'")
}
}
// testResponseWriter implements http.ResponseWriter for testing.
type testResponseWriter struct {
header http.Header
status int
}
func (tw *testResponseWriter) Header() http.Header {
if tw.header == nil {
tw.header = make(http.Header)
}
return tw.header
}
func (tw *testResponseWriter) Write(data []byte) (int, error) {
return len(data), nil
}
func (tw *testResponseWriter) WriteHeader(status int) {
tw.status = status
}
// TestNewServerInvalidConfig tests server creation with invalid config.
func TestNewServerInvalidConfig(t *testing.T) {
C = Config{} // Reset global config.
@@ -618,4 +655,30 @@ func TestMethodNotAllowedHandler(t *testing.T) {
}
}
// TestTimingSafeMatch tests the timingSafeMatch function for constant-time comparison.
func TestTimingSafeMatch(t *testing.T) {
tests := []struct {
name string
a string
b string
expected bool
}{
{"empty strings", "", "", true},
{"identical strings", "test", "test", true},
{"different strings", "test", "Test", false},
{"different length", "test", "testing", false},
{"first empty", "", "test", false},
{"second empty", "test", "", false},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
result := timingSafeMatch(tt.a, tt.b)
if result != tt.expected {
t.Errorf("timingSafeMatch(%q, %q) = %v, expected %v", tt.a, tt.b, result, tt.expected)
}
})
}
}
/* vim: setlocal ft=go noet ai ts=4 sw=4 sts=4: */

View File

@@ -0,0 +1,17 @@
package backer
import (
"testing"
)
func TestErrDirectoryScanTimeout(t *testing.T) {
err := ErrDirectoryScanTimeout{ScannedFiles: 42}
if err.Error() != "Directory scan timed out" {
t.Errorf("Expected error message 'Directory scan timed out', got %q", err.Error())
}
if err.ScannedFiles != 42 {
t.Errorf("Expected ScannedFiles=42, got %d", err.ScannedFiles)
}
}

View File

@@ -9,6 +9,8 @@ import (
"path/filepath"
"runtime"
"testing"
"github.com/klauspost/compress/zstd"
)
// TestCreateTarGzStreamBasic tests basic tar.gz creation with a single file.
@@ -247,6 +249,59 @@ func TestCreateTarGzStreamHardLink(t *testing.T) {
}
}
// TestCreateTarGzStreamHardLinkFromTestData tests hard link handling with test data.
func TestCreateTarGzStreamHardLinkFromTestData(t *testing.T) {
if runtime.GOOS == "windows" {
t.Skip("Hard link detection not supported on Windows (inodes unavailable)")
}
testDataPath := filepath.Join("../../test_data", "test1", "hardlinks")
filepaths, err := GetFilesFromDirectories(context.Background(), []string{testDataPath})
if err != nil {
t.Fatalf("Failed to get files from directory: %v", err)
}
ctx := context.Background()
reader := CreateTarGzStream(ctx, filepaths)
defer reader.Close()
gzReader, err := gzip.NewReader(reader)
if err != nil {
t.Fatalf("Failed to create gzip reader: %v", err)
}
defer gzReader.Close()
tarReader := tar.NewReader(gzReader)
entries := 0
linkCount := 0
for {
header, err := tarReader.Next()
if err == io.EOF {
break
}
if err != nil {
t.Fatalf("Failed to read tar entry: %v", err)
}
entries++
t.Logf("Entry %d: name=%s typeflag=%c linkname=%s", entries, header.Name, header.Typeflag, header.Linkname)
if header.Typeflag == tar.TypeLink {
linkCount++
}
}
if entries != 4 {
t.Errorf("Expected 4 entries (dir + 1 original + 2 hardlinks), got %d", entries)
}
if linkCount != 2 {
t.Errorf("Expected 2 hard link entries, got %d", linkCount)
}
}
// TestCreateTarGzStreamCancellation tests context cancellation.
func TestCreateTarGzStreamCancellation(t *testing.T) {
tmpDir := t.TempDir()
@@ -564,4 +619,115 @@ func TestCreateTarXzStream(t *testing.T) {
}
}
// TestCreateTarPgzipStream tests pgzip (parallel gzip) compression.
func TestCreateTarPgzipStream(t *testing.T) {
C = Config{CompressionLevel: 6}
tmpDir := t.TempDir()
testFile := filepath.Join(tmpDir, "test.txt")
content := []byte("test content for pgzip")
if err := os.WriteFile(testFile, content, 0644); err != nil {
t.Fatal(err)
}
ctx := context.Background()
reader := CreateTarPgzipStream(ctx, []string{testFile})
defer reader.Close()
gzReader, err := gzip.NewReader(reader)
if err != nil {
t.Fatalf("Failed to create gzip reader: %v", err)
}
defer gzReader.Close()
tarReader := tar.NewReader(gzReader)
header, err := tarReader.Next()
if err != nil {
t.Fatalf("Failed to read tar header: %v", err)
}
if header.Name != testFile && filepath.Base(header.Name) != "test.txt" {
t.Errorf("Expected file name 'test.txt', got '%s'", header.Name)
}
data, err := io.ReadAll(tarReader)
if err != nil {
t.Fatalf("Failed to read file content: %v", err)
}
if string(data) != string(content) {
t.Errorf("Expected content '%s', got '%s'", content, data)
}
}
// TestCreateTarZstdStream tests zstd compression.
func TestCreateTarZstdStream(t *testing.T) {
C = Config{CompressionLevel: 6}
tmpDir := t.TempDir()
testFile := filepath.Join(tmpDir, "test.txt")
content := []byte("test content for zstd")
if err := os.WriteFile(testFile, content, 0644); err != nil {
t.Fatal(err)
}
ctx := context.Background()
reader := CreateTarZstdStream(ctx, []string{testFile})
defer reader.Close()
// zstd.Reader is in github.com/klauspost/compress/zstd
zstdReader, err := zstd.NewReader(reader)
if err != nil {
t.Fatalf("Failed to create zstd reader: %v", err)
}
defer zstdReader.Close()
tarReader := tar.NewReader(zstdReader)
header, err := tarReader.Next()
if err != nil {
t.Fatalf("Failed to read tar header: %v", err)
}
if header.Name != testFile && filepath.Base(header.Name) != "test.txt" {
t.Errorf("Expected file name 'test.txt', got '%s'", header.Name)
}
data, err := io.ReadAll(tarReader)
if err != nil {
t.Fatalf("Failed to read file content: %v", err)
}
if string(data) != string(content) {
t.Errorf("Expected content '%s', got '%s'", content, data)
}
}
// TestZstdLevel tests zstd level mapping.
func TestZstdLevel(t *testing.T) {
tests := []struct {
input int
expected zstd.EncoderLevel
}{
{1, zstd.SpeedFastest},
{2, zstd.SpeedFastest},
{3, zstd.SpeedFastest},
{4, zstd.SpeedDefault},
{5, zstd.SpeedDefault},
{6, zstd.SpeedDefault},
{7, zstd.SpeedBetterCompression},
{8, zstd.SpeedBetterCompression},
{9, zstd.SpeedBetterCompression},
}
for _, tt := range tests {
C = Config{CompressionLevel: tt.input}
result := zstdLevel(tt.input)
if result != tt.expected {
t.Errorf("zstdLevel(%d) = %v, expected %v", tt.input, result, tt.expected)
}
}
}
/* vim: setlocal ft=go noet ai ts=4 sw=4 sts=4: */

View File

@@ -242,25 +242,38 @@ if C.Port < 1 || C.Port > 65535 {
## Testing
```bash
go test ./... # Run tests
go test ./... # Run tests (runs make clean first)
go test -v -run TestName ./... # Run specific test
golangci-lint run ./... # Lint
make test # Clean + test (deterministic)
```
Coverage: ~69% on internal/backer (~100% on internal/log)
Coverage: ~64% on internal/backer (~100% on internal/log)
### Test Data
All test data is created by `TestMain` in `internal/backer/main_test.go` before tests run and cleaned up afterward. This ensures tests are independent of pre-existing files.
- `test_data/test1/foo/` — test directory with files, subdirs, empty_dir
- `test_data/test1/bar/` — test directory with files and a directory named `goodbye.txt/` (edge case: directory with file extension)
- `test_data/test1/hardlinks/` — test directory with hard-linked files (original.txt, hardlink1.txt, hardlink2.txt)
- `test_data/test1/symlinks/` — test directory with symbolic links
- `test_data/test1/empty_dir/` — empty directory for testing
- `test_data/tmp/` — reserved for test temporary files (use instead of system `/tmp`)
### Test Conventions
- Reset global config: `C = Config{}`
- Reset exclude patterns: `excludePatterns = nil`
- Use `t.TempDir()` for temporary files
- Helper function `writeTempConfig()` for config tests
- Test data created by TestMain setup, cleaned after tests
- Always check errors with meaningful diagnostics (use `t.Fatalf` for setup errors)
## Build
```bash
make # Build binary (version from git or "dev")
make # Clean + build binary (version from git or "dev")
make build VERSION=0.1.0 # Build with specific version
make clean # Remove binary
make test # Run tests
make clean # Remove binary, test_data/tmp, test_data/test1
make test # Clean + test (deterministic)
make upgrade # Update dependencies
make help # Show targets
```

View File

@@ -1 +0,0 @@
Fucked up beyond recognotion.

View File

@@ -1,2 +0,0 @@
My dear Tweety,
I'd like to greet you as my breakfast.

View File

@@ -1 +0,0 @@
This is MY file!