Add explicit statement that our tar is gnu-compatible to README.md

- Fix 404 on downloading archive with extension
- Add downloaded archive length to log
- Add md5 hash to log

.gitignore

@@ -1,6 +1,8 @@
 # Binary
 /backer
 /backer.exe
+/cmd/backer/backer
+/cmd/backer/backer.exe
 
 # Local config (may contain secrets)
 /config.json
@@ -27,3 +29,7 @@ Thumbs.db
 *.exe
 *.test
 *.out
+
+# Test data (generated by tests)
+test_data/tmp
+test_data/test1

.golangci.yml

@@ -74,6 +74,7 @@ linters:
         - paramTypeCombine
         - whyNoLint
         - hugeParam
+        - unnamedResult
     godoclint:
       default: all
       ignore-unexported: false

AGENTS.md

@@ -2,19 +2,60 @@
 
 ## Project Overview
 
-Backer is a Go-based HTTP/HTTPS backup server that creates `tar.gz` archives on-the-fly from configured directories. It serves a single endpoint that clients (e.g., `curl`) can query to download a timestamped backup archive. Licensed under MIT.
+Backer is a Go-based HTTP/HTTPS backup server that creates compressed archives on-the-fly from configured directories. Supports multiple compression algorithms: gzip, pgzip, bzip2, zstd, lz4, xz. It serves a single endpoint that clients (e.g., `curl`) can query to download a timestamped backup archive. Licensed under MIT.
+
+## Fundamental Rules
+
+- **Do not apply changes without explicit user consent.** When asked to analyze, review, or suggest any changes, present your proposal first and wait for confirmation before editing any files. Always ask "Should I implement this?" or "Should I apply these changes?" before modifying anything.
+- **Keep codebase simple, small, maintainable, safe and human-readable.** Favour minimal, clear solutions over clever or abstract ones. Prefer clarity over cleverness. Prioritize explicitness. Minimize magic. Do not introduce patterns, layers, or indirection unless they directly solve a concrete problem. Every line of code must earn its place.
+- **Create small focused modules and separate functionality over files.** Do not make universal implementation; try to move functions to separate files and name files after function names.
+- **Use a consistent structure.** Subsequent searches will be faster and consume less context.
+- **Avoid duplicate names.** This is not strict rule. If codebase contain multiple files with the same name (e.g., index.ts), we have to read them all to discover the one exactly we need. This consumes time and context.
+- **When stuck, ask clarification questions, or present a short plan, or both.** Do not try to guess intent or make assumptions - ask.
+- **Always make TODO plan before any job and follow it closely.**
+- **Report any found defects, issues, bugs.** And keep track of them.
+- **Comment the code.** Add code comments to explain corner cases, complex logic, business rules, and design decisions. Focus on **what** problem the code is solving rather than **how** it is solving it. Write comments in easy to read and understand language, for humans to comprehend. Every public API and structure field must have a docstring.
+- **Use domain names.** Use naming conventions that reflect their purpose and domain concepts, not their technical implementation (e.g., OrderProcessor instead of OrderServiceFactory).
+- **Test corner cases.** Write comprehensive test suites that cover various scenarios and edge cases.
+- **Use tags and labels.** In comments, add tags or labels to files, classes, or functions to categorize them based on functionality, domain concepts, or design patterns.
+- **Avoid obvious comments that do not add value.**
+- **Remove dead code.** A half-finished refactoring can actually harm the contributions.
+- **Update AGENTS.md automatically.** Keep AGENTS.md dense, detailed and concise.
 
 ## Workflow Rules
 
-- **Do not apply changes without explicit user consent.** When asked to analyze, review, or suggest improvements, present your proposal first and wait for confirmation before editing any files. Always ask "Should I implement this?" or "Should I apply these changes?" before writing code.
-- **Keep the codebase simple, small, maintainable, safe, and human-readable.** Favor minimal, clear solutions over clever or abstract ones. Do not introduce patterns, layers, or indirection unless they directly solve a concrete problem. Every line of code must earn its place.
-- **When stuck, ask a clarification question, or propose a short plan, or both.** Do not guess intent or make assumptions — ask.
-- **Always make a TODO plan before acting. And try to follow this plan.**
-- **If you notice defects, bugs, or issues, always report it.**
-- **Track new features, known issues, bugs, and other defects in `TODO.org`.** Update it as you implement features, fix issues, fix bugs, or discover new ones — check off completed items, add new findings.
-- **Update `AGENTS.md` as the project evolves.** Keep it comprehensive but dense.
 - **Don't use `context` package unless absolutely necessary.** Try to find another solution first.
-- **Create unit tests if possible/necessary** to verify changes that are not covered by unit tests.
+- **Create functional and unit tests if possible/necessary** to verify changes that software code works at is should.
+
+### Test Rules
+
+- **Test data must be self-contained and created at test setup.** All test data should be created in `TestMain` in `internal/backer/main_test.go` before any tests run, not rely on pre-existing files in the repository. Use `test_data/test1/` as the test data directory.
+
+- **Use `test_data/tmp/` for temporary files.** When tests need temporary storage, use this directory (not system `/tmp`) to keep test data isolated and predictable.
+
+- **Always check errors with meaningful diagnostics.** Never discard errors with `_ =`. Use patterns like:
+  ```go
+  if err := os.MkdirAll(dir, 0755); err != nil {
+      t.Fatalf("Failed to create directory %s: %v", dir, err)
+  }
+  ```
+
+- **Tests must not depend on external state.** Test data should be created fresh for each test run and cleaned up afterward. Use `TestMain` with setup/cleanup functions.
+
+- **Use `t.Fatal` or `t.Fatalf` for setup errors.** When test setup fails, fail fast with clear diagnostic messages showing what failed and why.
+
+- **Reset global state between tests.** For tests that modify global variables (like `C` config), save the original value and restore it in a `defer`:
+  ```go
+  original := C
+  defer func() { C = original }()
+  ```
+
+- **Skip tests gracefully when features are unavailable.** For platform-specific features (hardlinks, symlinks), use `t.Skip()` when the feature is not supported:
+  ```go
+  if err := os.Link(orig, link); err != nil {
+      t.Skip("Hard links not supported, skipping test")
+  }
+  ```
 
 ## Build & Test Commands
 
@@ -76,8 +117,8 @@ if C.Port == 0 {
     log.Warnf("Config option port is not set, fallback to %d", C.Port)
 }
 
-if C.Port < 1 || C.Port > 65535 {
+if C.Port < portMin || C.Port > portMax {
-    return fmt.Errorf("Port must be between 1 and 65535, got %d", C.Port)
+    return fmt.Errorf("Config option port must be between %d and %d, got %d", portMin, portMax, C.Port)
 }
 ```
 
@@ -113,6 +154,18 @@ CompressionLevel  int `json:"compression_level"`
 | QF1001 | De Morgan's law disabled for auth condition clarity |
 | revive (error strings) | Same reason as ST1005 |
 
+### Design Decisions (Not Issues)
+
+The following are intentional design decisions, not bugs or defects:
+
+- **No rate limiting**: Backups are assumed to be infrequent enough that rate limiting is unnecessary.
+- **Backup scheduling**: Client-side decision — server serves backups on-request only.
+- **Basic Auth only**: HTTP Basic Auth is sufficient for a backup server behind a firewall.
+- **Global config**: The `C Config` global variable is intentional — config is loaded once at startup.
+- **No incremental backup**: Each backup is a full snapshot, not incremental.
+- **Credentials in config file**: Config file should have restricted permissions (documented).
+- **Thread-safe logging**: The global `Log` variable uses mutex protection to enable parallel test execution; this is not needed for single-server production use but allows tests to run in parallel without race conditions.
+
 ## Project Structure
 
 ```
@@ -123,15 +176,21 @@ cmd/
 internal/
   backer/
     types.go                        # Config struct definition
+    consts.go                       # Named constants for magic numbers
     globals.go                      # Global C Config variable, compiled excludePatterns
+    archive.go                      # Common archive logic (pipe, tar, file iteration)
+    gzip.go                         # CreateTarGzStream, CreateTarPgzipStream
+    bzip2.go                        # CreateTarBzip2Stream
+    zstd.go                         # CreateTarZstdStream, zstdLevel helper
+    lz4.go                          # CreateTarLz4Stream
+    xz.go                           # CreateTarXzStream
     loadConfig.go                   # HJSON/JSON config parsing with validation
     loadConfig_test.go              # Config validation tests
     newServer.go                    # HTTP server creation, auth handler, context helpers
     newServer_test.go               # Server, auth, and context helper tests
-    utils.go                        # CreateTarGzStream (tar.gz streaming via io.Pipe)
-    utils_test.go                   # Tar.gz stream tests
     getFilesFromDirectories.go      # Directory walking, exclusion filtering (isExcluded)
     getFilesFromDirectories_test.go # File traversal and exclude pattern tests
+    utils_test.go                   # Archive stream tests
   log/
     main.go                         # Logging wrapper around stdlib slog
     main_test.go                    # Log package tests
@@ -150,7 +209,6 @@ test_data/
   test1/                            # File/directory fixtures for tests
 
 Makefile                            # Build targets
-TODO.org                            # Project backlog (Org mode)
 README.md                           # User documentation
 LICENSE                             # MIT License
 llms.txt                            # LLM-friendly project reference
@@ -179,8 +237,12 @@ Config is JSON (HJSON with comments supported). Default path depends on OS:
 | loglevel | `info` | Log verbosity: error/warn/info/debug |
 | directories | — | Directories to backup (required, validated for existence) |
 | backup_timeout | `60` | Stream timeout in minutes (1-1440) |
+| dir_scan_timeout | `60` | Directory scan timeout in minutes (1-1440) |
 | compression_level | `9` | Gzip level (1-9) |
 | exclude_patterns | `[]` | Regex patterns to exclude from backup |
+| filename_prefix | `backup` | Prefix for backup filename in Content-Disposition header |
+| default_compression | `gzip` | Default compression algorithm: gzip, pgzip, bzip2, zstd, lz4 or xz |
+| compression_algorithm | `gzip` | Alias for default_compression (deprecated) |
 
 ## Architecture & Key Patterns
 
@@ -199,25 +261,47 @@ json.Unmarshal(buf, &C)     // decode into Config struct
 
 ### Streaming Archive Creation
 
-No temp files on disk. Uses `io.Pipe` with a goroutine to stream tar.gz on-the-fly:
+No temp files on disk. Uses `io.Pipe` with a goroutine to stream archives on-the-fly:
-- `CreateTarGzStream()` returns an `io.ReadCloser`
+- Common logic in `archive.go`: `createArchiveStream()` handles pipe, goroutine, tar writing, file iteration
-- Goroutine writes tar entries → gzip writer → pipe writer
+- Each compression algorithm has its own file: `gzip.go`, `bzip2.go`, `zstd.go`, `lz4.go`, `xz.go`
+- All return an `io.ReadCloser`
 - HTTP handler copies from pipe reader directly to HTTP response
 
 ### Context-Aware I/O
 
 Two helper functions in `newServer.go` handle context cancellation during streaming:
 - `writeWithContext(ctx, fn)` — checks context before calling fn
-- `copyWithContext(ctx, dst, src)` — buffered copy (32KB) with cancellation checks
+- `copyWithContext(ctx, dst, src)` — buffered copy (32KB) with cancellation checks, computes MD5 hash of copied data
 
 ### Special File Type Handling
 
 - **Regular files**: streamed into tar archive with content
 - **Directories**: included as tar directory entries
-- **Symlinks**: stored as symlink entries (target path preserved)
+- **Symlinks**: stored as symlink entries (target path preserved). Backer never resolves or follows symlinks during directory traversal — it uses `os.Lstat` and `filepath.WalkDir` which do not follow symlinks. This means symlink cycles are impossible to occur.
+- **Broken symlinks**: archived with their target path (even if target doesn't exist)
 - **Device files** (char/block): included with correct major/minor
-- **Named pipes**: included (header only, no data)
+- **Named pipes (FIFOs)**: included (header only, no data)
 - **Sockets**: skipped (not archivable)
+- **Hard links**: deduplicated — if multiple files share the same inode, only the first is stored with content, subsequent hard links are stored as link entries pointing to the original
+- **GNU tar format**: archives use GNU tar format (`header.Format = tar.FormatGNU`) for compatibility with GNU tar utility
+
+### Archive Test Coverage
+
+All entity types are tested in `internal/backer/utils_test.go`:
+
+| Entity Type | Test Function |
+|------------|---------------|
+| Regular file | `TestCreateTarGzStreamBasic` |
+| Empty file | `TestCreateTarGzStreamEmptyFile` |
+| Directory | `TestCreateTarGzStreamDirectory` |
+| Symlink | `TestCreateTarGzStreamSymlink` |
+| Broken symlink | `TestCreateTarGzStreamBrokenSymlink` |
+| Symlink to directory | `TestCreateTarGzStreamSymlinkToDirectory` |
+| Hard link | `TestCreateTarGzStreamHardLink` |
+| Character device | `TestCreateTarGzStreamDeviceFile` (falls back to char device) |
+| Named pipe (FIFO) | `TestCreateTarGzStreamNamedPipe` |
+| Special characters in names | `TestCreateTarGzStreamSpecialCharacters` |
+| Nested files | `TestCreateTarGzStreamNestedFiles` |
 
 ### Pipe Error Recovery
 
@@ -225,7 +309,7 @@ Two helper functions in `newServer.go` handle context cancellation during stream
 
 ### Auth Handler
 
-HTTP Basic Auth. Both username AND password must match exactly. Returns `401 Unauthorized` with `WWW-Authenticate` header on failure. The condition is intentionally written without De Morgan's law simplification for clarity (linter suppression `//nolint:staticcheck`).
+HTTP Basic Auth. Both username AND password must match exactly. Returns `401 Unauthorized` with `WWW-Authenticate` header on failure. Uses timing-safe comparison to prevent timing attacks. The condition is intentionally written without De Morgan's law simplification for clarity (linter suppression `//nolint:staticcheck`).
 
 ### Client IP Detection
 
@@ -235,11 +319,74 @@ Checks `X-Forwarded-For` header first (for proxied requests), then falls back to
 
 - `ReadHeaderTimeout`: 5 seconds
 - `WriteTimeout`: configurable via `backup_timeout` (converted to `time.Duration`)
+- `IdleTimeout`: 10 seconds (for keep-alive connections; backer disables keep-alives explicitly)
+- `MaxHeaderBytes`: 1KB (prevents header-based DoS attacks)
+- `DisableKeepAlives`: true (each connection closed after request, no connection reuse)
 - TLS minimum version: TLS 1.3 (when HTTPS enabled)
 
+### TLS Error Handling
+
+The server uses a `ServerWrapper` type to intercept and classify errors from `Serve()` and `ServeTLS()`. Two mechanisms:
+
+1. **Server errors** (`Serve()` / `ServeTLS()`): Uses `errors.As` to detect TLS error types (`tls.AlertError`, `tls.CertificateVerificationError`, `tls.ECHRejectionError`, `tls.RecordHeaderError`) and checks for "tls:" prefix. TLS-specific errors logged at debug level, others at warn level.
+
+2. **net/http internal errors** (`http.Server.ErrorLog`): Uses `DebugLogger()` which routes messages through `slogWriter.Write()`:
+   - TLS-related errors (keywords: "tls", "ssl", "handshake", "certificate") → logged at debug level
+   - Other errors → logged at warn level
+   - Both are filtered by configured `loglevel`. Set `loglevel: debug` to see all messages.
+
+This ensures client-side TLS errors (handshake failures, certificate issues) don't clutter production logs by default.
+
+### Compression Level Mapping
+
+The `compression_level` config option (1-9) is mapped differently for each compression algorithm:
+
+| Algorithm | Level Mapping |
+|-----------|----------------|
+| gzip | 1-9 directly maps to gzip compression levels |
+| pgzip | 1-9 directly maps to pgzip compression levels |
+| bzip2 | 1-9 directly maps to bzip2 compression levels |
+| zstd | 1-3: Fastest, 4-6: Default, 7-9: BetterCompression |
+| lz4 | 1-3: Fastest (0), 4-6: Default (1), 7-9: Best (2) |
+| xz | Uses default compression (no configurable level) |
+
+### HTTP Routing & Compression Selection
+
+The server uses `http.NewServeMux()` for routing. Multiple routes are registered to support different compression formats:
+
+```go
+mux.Handle(C.Location, http.HandlerFunc(backupHandler))           // /archive
+mux.Handle(C.Location+".tar.gz", http.HandlerFunc(backupHandler))  // /archive.tar.gz
+mux.Handle(C.Location+".tar.xz", http.HandlerFunc(backupHandler))   // /archive.tar.xz
+mux.Handle(C.Location+".tar.bz2", http.HandlerFunc(backupHandler)) // /archive.tar.bz2
+// ... etc
+```
+
+The compression algorithm is determined by extracting the extension from the request path using `path.Ext()`:
+- For `/archive.tar.xz`, `path.Ext()` returns `.tar.xz` → maps to xz
+- For `/archive`, `path.Ext()` returns `""` → uses default compression
+
+The `getCompressionAlgorithm()` function handles both full paths and bare extensions:
+- Full path: `/archive.tar.xz` → `.tar.xz` → xz
+- Bare extension: `tar.xz` → `.tar.xz` → xz
+- Single ext: `xz` → `.xz` → xz
+
+### Unmatched Route Logging
+
+The server wraps the `http.ResponseWriter` to capture status codes and log unmatched routes:
+- **404 Not Found**: Logged as `"Not Found from {clientIP} to {path}"`
+- **405 Method Not Allowed**: Logged as `"Method Not Allowed from {clientIP} for {method} {path}"`
+
+This uses a `responseWriterWrapper` that tracks the status code via the `WriteHeader()` method.
+
 ## Dependencies
 
 - `github.com/hjson/hjson-go` v3.3.0 — HJSON config parsing
+- `github.com/dsnet/compress` v0.0.1 — bzip2 compression
+- `github.com/klauspost/compress` v1.18.5 — zstd compression
+- `github.com/klauspost/pgzip` v1.2.6 — parallel gzip compression
+- `github.com/pierrec/lz4` v2.6.1 — lz4 compression
+- `github.com/ulikunitz/xz` v0.5.15 — xz compression
 
 ## Testing
 
@@ -262,10 +409,14 @@ Tests live alongside source files as `*_test.go` (package name, not `_test` suff
 
 ### Test Data
 
-- `test_data/test_config.json` — valid test config (user: JohnnyGoode, password: SharpShooter)
+All test data is created by `TestMain` in `internal/backer/main_test.go` before tests run and cleaned up afterward. This ensures tests are independent of pre-existing files.
-- `test_data/test1/foo/` — test directory with files, subdirs, empty dirs
+
+- `test_data/test1/foo/` — test directory with files, subdirs, empty_dir
 - `test_data/test1/bar/` — test directory with files and a directory named `goodbye.txt/` (edge case: directory with file extension)
-- `test_data/example.crt` / `example.key` — TLS test certificates
+- `test_data/test1/hardlinks/` — test directory with hard-linked files (original.txt, hardlink1.txt, hardlink2.txt)
+- `test_data/test1/symlinks/` — test directory with symbolic links
+- `test_data/test1/empty_dir/` — empty directory for testing
+- `test_data/tmp/` — reserved for test temporary files (use instead of system `/tmp`)
 
 ## Service Deployment
 
@@ -273,9 +424,3 @@ Init scripts provided in `contrib/`:
 - **systemd** (`backer.service`): `ExecStart=/usr/local/sbin/backer`, restart on failure
 - **OpenRC** (`backer`): uses `supervise-daemon`, creates log/pid dirs in `start_pre`
 - **FreeBSD rc.d** (`backer.freebsd`): daemonizes via `/usr/sbin/daemon`
-
-## TODO / Roadmap
-
-See `TODO.org` for current backlog:
-- Consider alternative compression algorithms (bz2, xz, lzo, lz4)
-- Configurable filename prefix for `Content-Disposition` header

Makefile

@@ -23,13 +23,15 @@ build:
 
 clean:
 	rm -f $(BINARY)
+	rm -rf test_data/tmp
+	rm -rf test_data/test1
 
 upgrade:
 	go get -u ./...
 	go mod tidy
 	go mod vendor
 
-test:
+test: clean
 	go test ./...
 
 help:

README.md

@@ -1,6 +1,6 @@
 # Backer
 
-Kind of backup utility. It makes tar.gz on-the-fly out of given in config.json files.
+HTTP/HTTPS backup server that creates compressed archives on-the-fly from configured directories. Supports multiple compression algorithms: gzip, pgzip (which is parallel gzip), bzip2, zstd, lz4, xz.
 
 ## How to build it
 
@@ -8,9 +8,8 @@ Kind of backup utility. It makes tar.gz on-the-fly out of given in config.json f
 make                                    # Build binary
 make build VERSION=0.1.0                # Build with version
 make test                               # Run tests
-backer                                  # Run
+make                                    # Clean + build
-backer --version                        # Show version
+make upgrade                            # Update dependencies
-backer -c /path/to/config.json          # Custom config
 ```
 
 That will produce binary named `backer`. Copy it somewhere in `/usr/local/sbin/backer`, for example. In `data`
@@ -29,33 +28,88 @@ If you running FreeBSD, use the rc.d script `contrib/backer.freebsd` and place i
 Build, install and configure backer. On your backup storage server you can do
 
 ```bash
-curl -u username:password https://your_server:8086/archive -o "backup-$(date +%Y%m%d-%H%M%S).tar.gz"
+wget --quiet --no-check-certificate --content-disposition --http-user=username --http-password=password https://your_server:8086/archive
 ```
 
-and you'll get your backup with timestamp in filename, like `backup-20260325-105100.tar.gz`
+The filename and extension depend on your `filename_prefix` and `default_compression` settings.
+
+You can also override compression by specifying the archive extension in the URL:
+
+```bash
+wget --quiet --no-check-certificate --content-disposition --http-user=username --http-password=password https://your_server:8086/archive.xz
+wget --quiet --no-check-certificate --content-disposition --http-user=username --http-password=password https://your_server:8086/archive.gz    # Uses pgzip
+wget --quiet --no-check-certificate --content-disposition --http-user=username --http-password=password https://your_server:8086/archive.bz2
+wget --quiet --no-check-certificate --content-disposition --http-user=username --http-password=password https://your_server:8086/archive.zst
+wget --quiet --no-check-certificate --content-disposition --http-user=username --http-password=password https://your_server:8086/archive.lz4
+```
+
+Note: `/archive.tar.gz` always uses pgzip (parallel gzip) for better performance.
 
 Of course, something can go wrong, so you must check http status code after downloading backup.
 
 ## About config options
 
-AI suggests to fill this section with useful information, despite that all options described in example config.
+All options described in example config.
 
 | Option | Default value | Wtf | Notes |
 | :--- | :---- | :--- | :--- |
 | address | "0.0.0.0" | Address at which server listens. | In many cases 0.0.0.0 is okay. |
-| port | 8086 | Port at which server binds | Number picked after intel first comercially successful cpu. |
+| port | 8086 | Port at which server binds. | Number picked after intel first commercially successful cpu. |
-| cert | "/path/to/ssl.crt" | Path to ssl certificate file | It is not enforced in obvious form, but you should limit access to certificate at least with file permissions. |
+| cert | — | Path to ssl certificate file. | Required when nohttps is false. |
-| key | "/path/to/ssl.key" | Path to ssl certificates key file | It is not enforced in obvious form, but you should limit access to certificates key at least with file permissions. |
+| key | — | Path to ssl certificates key file. | Required when nohttps is false. |
-| nohttps | false | ability to disable https and use plain http. | This option is for lab experiments or for development needs. In case of this utility you should consider using long enough certificate with well ciphers. |
+| nohttps | false | Disable https and use plain http. | For lab experiments or development. |
-| location | "/archive" | This is static location where curl (or more sophisticated client) should aim to download backup | /archive is okay, but you free to choose something more wild like "/backup", just for giggles. |
+| location | "/archive" | API endpoint path for backup download. | |
-| user | | You should pick up some flashy username for basic auth to prevent unauthorized access to your backup. | Enough said. |
+| user | — | Username for basic auth. | Required. |
-| password | | You should pick up something super-duper secretous to prevent unauthorized access to your backup. | Enough said. |
+| password | — | Password for basic auth. | Required. |
-| log | stderr | Where log messages are directed. If no value set, logs drops to stderr | |
+| log | stderr | Log output file path. | |
-| loglevel | | Verbosity of logs. | Can be error, warn, info, debug. Pick whatever you like. |
+| loglevel | "info" | Verbosity of logs. | Options: error, warn, info, debug. |
-| directories | | An array with directories. Backer can work only with directories, not single files. | Put here all the folders you'd like to backup! |
+| directories | — | Directories to backup. | Required. Array of paths. |
-| backup_timeout | 60 | Timeout in minutes for backup streaming operations. | If a backup takes longer than this, the connection will be terminated. Useful for large backups. Range: 1-1440 minutes (1 day max). |
+| backup_timeout | 60 | Timeout in minutes for backup streaming. | Range: 1-1440 minutes. |
-| compression_level | 9 | Gzip compression level for tar.gz archives. | 1 (fastest) to 9 (best compression). Higher values produce smaller files but take longer to compress. |
+| dir_scan_timeout | 60 | Timeout in minutes for getting file list from all backupped directories. | Range: 1-1440 minutes. |
-| exclude_patterns | [] | Regex patterns to exclude from backup. | E.g., `[".*\\.tmp$", "/node_modules/"]`. Files matching any pattern are skipped. |
+| compression_level | 9 | Compression level. | 1 (fastest) to 9 (best compression). |
+| exclude_patterns | [] | Regex patterns to exclude from backup. | E.g., `[".*\\.tmp$", "/node_modules/"]`. |
+| filename_prefix | "backup" | Prefix for backup filename. | E.g., "mybackup" produces `mybackup-20260325-092341.tar.gz`. |
+| default_compression | "gzip" | Default compression algorithm. | Options: gzip, pgzip, bzip2, zstd, lz4, xz. |
+| compression_algorithm | "gzip" | Alias for default_compression. | You should use default_compression instead. |
+
+## Compression algorithms
+
+AI suggests this table as expected performance.
+
+| Algorithm | Speed   | Compression | Notes |
+| :---      | :---    | :---        | :--- |
+| gzip      | Fast    | Good        | Standard, widely compatible. |
+| pgzip     | Fast    | Good        | Parallel gzip. Uses multiple CPU cores, slightly larger files. |
+| bzip2     | Slow    | Better      | Pure Go. Good compression, slower than gzip. |
+| zstd      | Fast    | Very good   | Good balance of speed and compression. |
+| lz4       | Fastest | Moderate    | Extremely fast, larger files. |
+| xz        | Slowest | Best        | Best compression, slowest speed. |
+
+In real live with backup of 1 lxc container inside vm powered by 1 core cpu, 768Mb RAM and 100Mbit non-guaranteed
+network bandwidth, located in near by country I've got these results
+
+| Algorithm | Size       | Time    | CPU Load | RAM RSS+Shm, Mb | Notes |
+| :---      | ---:       | ---:    | ---:     | ---:            | :---  |
+| None      | 1461250551 |         |     0-2% |             0+0 | original dataset |
+| gzip      |  823381547 | 16m:22s |      96% |            18+6 | stdlib compression/gzip |
+| pgzip     |  823783924 | 13m:50s |   93-97% |            28+5 | multithreaded (parallel gzip), fully gzip compatible, klauspost/pgzip |
+| bzip2     |  797053471 |  9m:47s |   92-94% |            56+6 | dsnet/compress/bzip2 |
+| xz        |  515106988 | 16m:03s |      96% |           116+5 | ulikunitz/xz |
+| zstd      |  617330529 |  6m:39s |   40-99% |            64+6 | klauspost/compress/zstd |
+| lz4       | 1097745915 |  6m:12s |     4-6% |            30+6 | multithreaded lz4, pierrec/lz4 |
+
+The `compression_level` option (1-9) maps differently for each algorithm:
+
+| Algorithm        | Level Mapping |
+| :---             | :---          |
+| gzip/pgzip/bzip2 | 1-9 directly maps to compression levels |
+| lz4              | 1-3 → Fastest (0), 4-6 → Default (1), 7-9 → Best (2) |
+| zstd             | 1-3 → Fastest, 4-6 → Default, 7-9 → BetterCompression |
+| xz               | Uses default compression (no configurable level) |
+
+Obvously in case of lz4 and zstd the bottleneck was network. And I have to admit, that network definitely was not
+100Mbit, but it was pretty hard to measure real bandwidth. Anyway let's pretend that during experiment it was roughly
+the same.
 
 ## Best practices
 
@@ -67,6 +121,55 @@ good enough cypher.
 
 And one more thing user and password are mandatory, we should have at least some minimal barriers, right?
 
+## Supported File Types
+
+Backer handles various file types when creating archives:
+
+- **Regular files**: Included with full content
+- **Directories**: Included as directory entries
+- **Symlinks**: Stored as symlink entries (target path preserved)
+- **Broken symlinks**: Archived with their target path (even if target doesn't exist)
+- **Device files**: Character and block devices included with correct major/minor
+- **Named pipes (FIFOs)**: Included (header only, no data)
+- **Sockets**: Skipped (not archivable)
+- **Hard links**: Deduplicated — multiple hard links to the same file are stored as link entries to avoid duplication
+- **GNU tar format**: Compatible with GNU tar utility
+
+## Logging
+
+Backer logs events via Go's `slog`. Log levels: error, warn, info, debug.
+### Backup Completion Log
+
+When a backup completes, the following information is logged:
+- **client**: Client IP address
+- **user**: Authenticated username
+- **files**: Number of files backed up
+- **bytes**: Size of the created archive in bytes
+- **md5**: MD5 hash of the created archive (for verification)
+- **duration**: Time taken to create the archive
+
+Example:
+```
+Backup completed: client=127.0.0.1 user=admin files=42 bytes=1234567 md5=abc123def456 duration=2.5s
+```
+### TLS Error Handling
+
+TLS handshake errors and other client-side SSL/TLS errors are logged at **debug** level to reduce log clutter.
+This includes:
+- Certificate verification failures
+- Handshake failures
+- Protocol errors
+- TLS-specific connection issues
+
+### Graceful Shutdown
+
+When backer receives a termination signal (SIGTERM or SIGINT), it will:
+1. Log a warning message: "Shutdown signal received, shutting down immediately..."
+2. Stop accepting new connections and abort any in-progress downloads immediately
+3. Clean up resources and exit
+
+This ensures no partial/corrupted backups are left. Clients will need to restart their backup. For a forced immediate stop, use `SIGKILL`.
+
 ## Special Thanks
 
 This project was developed with the assistance of **opencode** (AI co-programmer powered by MIMO). The AI helped with:

TODO.org

@@ -1,14 +0,0 @@
-#+TITLE: Backer TODO List
-#+AUTHOR: Backer Project
-#+DATE: <2026-03-25 Wed>
-
-* TODO Backer Tasks
-
-** TODO Consider different compression algorithms
-   - [ ] Evaluate alternatives to gzip: bz2, xz, lzo, lz4
-   - [ ] lz4 for speed, xz for better compression
-   - [ ] Add compression_algorithm config option
-
-** TODO Configurable filename prefix
-   - [ ] Move "backup" prefix to config (currently hardcoded in newServer.go)
-   - [ ] Default: "backup", e.g., "backup-20260325-092341.tar.gz"

cmd/backer/main.go

@@ -2,6 +2,7 @@
 package main
 
 import (
+	"errors"
 	"flag"
 	"fmt"
 	"net/http"
@@ -40,10 +41,14 @@ func main() {
 		log.Fatalf("Failed to create server: %v", err)
 	}
 
-	log.Infof("Starting server at %s", server.Addr)
+	// Setup signal handling for immediate shutdown.
+	log.Infof("Installing signal handler")
+	setupSignalHandler(server)
+
+	log.Infof("Starting backer version %s at %s", version, server.Addr)
 
 	if backer.C.NoHTTPS {
-		if err := server.ListenAndServe(); err != nil && err != http.ErrServerClosed {
+		if err := server.Serve(); err != nil && !errors.Is(err, http.ErrServerClosed) {
 			log.Fatalf("Failed to start server: %v", err)
 		}
 	} else {
@@ -55,10 +60,12 @@ func main() {
 			log.Fatalf("Certificate key file error: %v", err)
 		}
 
-		if err := server.ListenAndServeTLS(backer.C.Cert, backer.C.Key); err != nil && err != http.ErrServerClosed {
+		if err := server.ServeTLS(backer.C.Cert, backer.C.Key); err != nil && !errors.Is(err, http.ErrServerClosed) {
 			log.Fatalf("Failed to start server: %v", err)
 		}
 	}
+
+	log.Infof("Server stopped")
 }
 
 // defaultConfigPath returns the default config file path based on the OS.

cmd/backer/sigHandler.go

@@ -0,0 +1,29 @@
+package main
+
+import (
+	"os"
+	"os/signal"
+	"syscall"
+
+	"backer/internal/backer"
+	"backer/internal/log"
+)
+
+// setupSignalHandler sets up signal handling for immediate shutdown.
+// SIGTERM is commonly sent by systemd, k8s, and other container orchestrators.
+// SIGINT is sent on Ctrl+C.
+func setupSignalHandler(server *backer.ServerWrapper) {
+	signalCh := make(chan os.Signal, 1)
+	signal.Notify(signalCh, os.Interrupt, syscall.SIGTERM)
+
+	go func() {
+		<-signalCh
+		log.Warnf("Shutdown signal received, shutting down immediately...")
+		// Close the embedded http.Server directly.
+		if closeErr := server.Close(); closeErr != nil {
+			log.Errorf("Error during server shutdown: %v", closeErr)
+		}
+	}()
+}
+
+/* vim: setlocal ft=go noet ai ts=4 sw=4 sts=4: */

contrib/backer.openrc

@@ -1,7 +1,7 @@
 #!/sbin/openrc-run
 
 name="$SVCNAME"
-command="/var/lib/$SVCNAME/$SVCNAME"
+command="/usr/local/sbin/$SVCNAME"
 supervisor="supervise-daemon"
 pidfile="/run/$SVCNAME/$SVCNAME.pid"
 
@@ -11,7 +11,6 @@ depend() {
 }
 
 start_pre() {
-        mkdir -p /var/log/$SVCNAME
         mkdir -p /run/$SVCNAME
 }
 

data/config_example.json

@@ -26,10 +26,31 @@
     /* Where to store logs.                                                                                           */
     "log": "/var/log/backer.log",
 
-    /* Verbosity of logs                                                                                              */
+    /* Verbosity of logs.                                                                                             */
     "loglevel": "info",
 
-    /* Diretories to backup                                                                                           */
+    /* Backup streaming timeout in minutes.                                                                           */
+    "backup_timeout": 60,
+
+    /* Gzip compression level (1-9, where 9 is best compression).                                                     */
+    "compression_level": 9,
+
+    /* Prefix for backup filename in Content-Disposition header.                                                      */
+    "filename_prefix": "backup",
+
+    /* Compression algorithm alias (deprecated: use default_compression).                                             */
+    "compression_algorithm": "gzip",
+
+    /* Default compression algorithm: gzip, pgzip, bzip2, zstd, lz4 or xz.                                            */
+    "default_compression": "gzip",
+
+    /* Regex patterns to exclude from backup.                                                                         */
+    "exclude_patterns": [],
+
+    /* Maximum allowed time, spent for getting file list from backupped directories.                                  */
+    "dir_scan_timeout": 60,
+
+    /* Directories to backup.                                                                                         */
     "directories": [
         "/path/to/directory1",
         "/path/to/directory2"

go.mod

@@ -2,4 +2,13 @@ module backer
 
 go 1.25
 
-require github.com/hjson/hjson-go v3.3.0+incompatible
+require (
+	github.com/dsnet/compress v0.0.1
+	github.com/hjson/hjson-go v3.3.0+incompatible
+	github.com/klauspost/compress v1.18.5
+	github.com/klauspost/pgzip v1.2.6
+	github.com/pierrec/lz4 v2.6.1+incompatible
+	github.com/ulikunitz/xz v0.5.15
+)
+
+require github.com/frankban/quicktest v1.14.6 // indirect

go.sum

@@ -1,2 +1,28 @@
+github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E=
+github.com/dsnet/compress v0.0.1 h1:PlZu0n3Tuv04TzpfPbrnI0HW/YwodEXDS+oPKahKF0Q=
+github.com/dsnet/compress v0.0.1/go.mod h1:Aw8dCMJ7RioblQeTqt88akK31OvO8Dhf5JflhBbQEHo=
+github.com/dsnet/golib v0.0.0-20171103203638-1ea166775780/go.mod h1:Lj+Z9rebOhdfkVLjJ8T6VcRQv3SXugXy999NBtR9aFY=
+github.com/frankban/quicktest v1.14.6 h1:7Xjx+VpznH+oBnejlPUj8oUpdxnVs4f8XU8WnHkI4W8=
+github.com/frankban/quicktest v1.14.6/go.mod h1:4ptaffx2x8+WTWXmUCuVU6aPUX1/Mz7zb5vbUoiM6w0=
+github.com/google/go-cmp v0.5.9 h1:O2Tfq5qg4qc4AmwVlvv0oLiVAGB7enBSJ2x2DqQFi38=
+github.com/google/go-cmp v0.5.9/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
 github.com/hjson/hjson-go v3.3.0+incompatible h1:Rqr+Ya+0aCJMjaE4s8E9YKvuJLuLVpEvz4ONum52vnI=
 github.com/hjson/hjson-go v3.3.0+incompatible/go.mod h1:qsetwF8NlsTsOTwZTApNlTCerV+b2GjYRRcIk4JMFio=
+github.com/klauspost/compress v1.4.1/go.mod h1:RyIbtBH6LamlWaDj8nUwkbUhJ87Yi3uG0guNDohfE1A=
+github.com/klauspost/compress v1.18.5 h1:/h1gH5Ce+VWNLSWqPzOVn6XBO+vJbCNGvjoaGBFW2IE=
+github.com/klauspost/compress v1.18.5/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ=
+github.com/klauspost/cpuid v1.2.0/go.mod h1:Pj4uuM528wm8OyEC2QMXAi2YiTZ96dNQPGgoMS4s3ek=
+github.com/klauspost/pgzip v1.2.6 h1:8RXeL5crjEUFnR2/Sn6GJNWtSQ3Dk8pq4CL3jvdDyjU=
+github.com/klauspost/pgzip v1.2.6/go.mod h1:Ch1tH69qFZu15pkjo5kYi6mth2Zzwzt50oCQKQE9RUs=
+github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=
+github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk=
+github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
+github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
+github.com/pierrec/lz4 v2.6.1+incompatible h1:9UY3+iC23yxF0UfGaYrGplQ+79Rg+h/q9FV9ix19jjM=
+github.com/pierrec/lz4 v2.6.1+incompatible/go.mod h1:pdkljMzZIN41W+lC3N2tnIh5sFi+IEE17M5jbnwPHcY=
+github.com/pkg/diff v0.0.0-20210226163009-20ebb0f2a09e/go.mod h1:pJLUxLENpZxwdsKMEsNbx1VGcRFpLqf3715MtcvvzbA=
+github.com/rogpeppe/go-internal v1.9.0 h1:73kH8U+JUqXU8lRuOHeVHaa/SZPifC7BkcraZVejAe8=
+github.com/rogpeppe/go-internal v1.9.0/go.mod h1:WtVeX8xhTBvf0smdhujwtBcq4Qrzq/fJaraNFVN+nFs=
+github.com/ulikunitz/xz v0.5.6/go.mod h1:2bypXElzHzzJZwzH67Y6wb67pO62Rzfn7BSiF4ABRW8=
+github.com/ulikunitz/xz v0.5.15 h1:9DNdB5s+SgV3bQ2ApL10xRc35ck0DuIX/isZvIk+ubY=
+github.com/ulikunitz/xz v0.5.15/go.mod h1:nbz6k7qbPmH4IRqmfOplQw/tblSgqTqBwxkY0oWt/14=

internal/backer/archive.go

@@ -0,0 +1,234 @@
+// Package backer implements HTTP/HTTPS backup server.
+package backer
+
+import (
+	"archive/tar"
+	"context"
+	"fmt"
+	"io"
+	"os"
+	"path/filepath"
+	"strings"
+	"syscall"
+
+	"backer/internal/log"
+)
+
+// createArchiveStream creates a pipe and streams a tar archive compressed by the provided writer.
+// The writerFactory creates the compression writer wrapping the pipe writer.
+// This is the core of all CreateTar*Stream functions.
+func createArchiveStream(ctx context.Context, filepaths []string, writerFactory func(io.Writer) (io.WriteCloser, error)) io.ReadCloser {
+	pr, pw := io.Pipe()
+
+	go func() {
+		var pipeErr error
+
+		closePipeWithError := func(err error) {
+			if pipeErr != nil {
+				return
+			}
+
+			pipeErr = err
+			pw.CloseWithError(err)
+		}
+
+		defer func() {
+			if pipeErr == nil {
+				pw.Close()
+			}
+		}()
+
+		cw, err := writerFactory(pw)
+		if err != nil {
+			closePipeWithError(fmt.Errorf("compression writer error: %w", err))
+
+			return
+		}
+
+		defer func() {
+			if pipeErr == nil {
+				if err := cw.Close(); err != nil {
+					if !isPipeClosedError(err) {
+						log.Errorf("Failed to close compression writer: %v", err)
+					}
+				}
+			}
+		}()
+
+		tw := tar.NewWriter(cw)
+
+		defer func() {
+			if pipeErr == nil {
+				if err := tw.Close(); err != nil {
+					if !isPipeClosedError(err) {
+						log.Errorf("Failed to close tar writer: %v", err)
+					}
+				}
+			}
+		}()
+
+		writeFilesToTar(ctx, filepaths, tw, closePipeWithError)
+	}()
+
+	return pr
+}
+
+// writeFilesToTar iterates over filepaths and writes each file to the tar archive.
+// It tracks inodes to handle hard links - if a file has already been archived,
+// subsequent hard links to the same inode are stored as link entries.
+// This is the Unix implementation (syscall not available on Windows).
+func writeFilesToTar(ctx context.Context, filepaths []string, tw *tar.Writer, closePipeWithError func(error)) {
+	// Track inodes to handle hard links: inode -> first path added.
+	// Uses simple file path tracking on Windows, inode tracking on Unix.
+	seenInodes := make(map[uint64]string)
+
+	for _, fpath := range filepaths {
+		var (
+			err        error
+			linkTarget string
+			isSymlink  bool
+			isHardLink bool
+		)
+
+		select {
+		case <-ctx.Done():
+			closePipeWithError(ctx.Err())
+
+			return
+
+		default:
+		}
+
+		log.Debugf("Adding file: %s", fpath)
+
+		st, err := os.Lstat(fpath)
+
+		if err != nil {
+			log.Warnf("Skipping %s: %v", fpath, err)
+
+			continue
+		}
+
+		// Handle symlinks - read target path and mark as symbolic link.
+		if st.Mode()&os.ModeSymlink != 0 {
+			linkTarget, err = os.Readlink(fpath)
+
+			if err != nil {
+				log.Warnf("Skipping %s: readlink failed: %v", fpath, err)
+
+				continue
+			}
+
+			isSymlink = true
+		}
+
+		// Skip sockets explicitly (they cannot be backed up).
+		if st.Mode()&os.ModeSocket != 0 {
+			log.Debugf("Skipping socket: %s", fpath)
+
+			continue
+		}
+
+		// Check for hard link - if inode was already seen, store as link instead of content.
+		// Only check for regular files (not symlinks).
+		if st.Mode().IsRegular() && !isSymlink {
+			inode := getInode(st)
+			if inode != 0 {
+				if existingPath, seen := seenInodes[inode]; seen {
+					// This is a hard link to already-archived file.
+					// To avoid "tar: Removing leading `/' from hard link targets" message, remove leading slash.
+					linkTarget = strings.TrimPrefix(filepath.ToSlash(filepath.Clean(existingPath)), "/")
+					isHardLink = true
+
+					log.Debugf("Hard link detected: %s -> %s", fpath, existingPath)
+				} else {
+					// First time seeing this inode - track it.
+					seenInodes[inode] = fpath
+				}
+			}
+		}
+
+		header, err := tar.FileInfoHeader(st, linkTarget)
+
+		if err != nil {
+			log.Warnf("Skipping %s: tar header failed: %v", fpath, err)
+
+			continue
+		}
+
+		// Set correct type flag for symbolic links or hard links.
+		if isSymlink {
+			header.Typeflag = tar.TypeSymlink
+			header.Linkname = linkTarget
+		} else if isHardLink {
+			header.Typeflag = tar.TypeLink
+			header.Linkname = linkTarget
+		}
+
+		header.Format = tar.FormatGNU
+		header.Name = strings.TrimPrefix(filepath.ToSlash(filepath.Clean(fpath)), "/")
+
+		mode := st.Mode()
+
+		if err := writeWithContext(ctx, func() error {
+			return tw.WriteHeader(header)
+		}); err != nil {
+			closePipeWithError(err)
+
+			return
+		}
+
+		// Only stream file contents for regular files that are not hard links.
+		if mode.IsRegular() && linkTarget == "" {
+			f, err := os.Open(fpath)
+			if err != nil {
+				log.Warnf("Skipping %s: open failed: %v", fpath, err)
+
+				continue
+			}
+
+			_, _, err = copyWithContext(ctx, tw, f)
+
+			if e := f.Close(); e != nil {
+				log.Warnf("Failed to close file %s: %v", fpath, e)
+			}
+
+			if err != nil {
+				log.Warnf("Skipping %s: copy failed: %v", fpath, err)
+
+				continue
+			}
+		}
+	}
+}
+
+// getInode extracts the inode number from [os.FileInfo].
+// Returns 0 if unable to determine (e.g., on Windows or with certain file systems).
+func getInode(fi os.FileInfo) uint64 {
+	if fi.Sys() == nil {
+		return 0
+	}
+
+	// Use reflection to get Dev and Ino fields from the syscall.Stat_t struct.
+	st := fi.Sys()
+	// Try type assertion for the struct with fields.
+	switch v := st.(type) {
+	case *syscall.Stat_t:
+		return (v.Dev << 32) | v.Ino
+	}
+
+	return 0
+}
+
+// isPipeClosedError checks if the error is due to a closed pipe.
+func isPipeClosedError(err error) bool {
+	if err == nil {
+		return false
+	}
+
+	// Check for the specific pipe closed error message.
+	return err.Error() == "io: read/write on closed pipe" ||
+		err.Error() == "io: read on closed pipe"
+}
+
+/* vim: setlocal ft=go noet ai ts=4 sw=4 sts=4: */

internal/backer/bzip2.go

@@ -0,0 +1,31 @@
+package backer
+
+import (
+	"context"
+	"io"
+
+	"github.com/dsnet/compress/bzip2"
+)
+
+// CreateTarBzip2Stream takes a list of files and returns a reader that reads from a bzip2-compressed tar archive.
+func CreateTarBzip2Stream(ctx context.Context, filepaths []string) io.ReadCloser {
+	return createArchiveStream(ctx, filepaths, func(w io.Writer) (io.WriteCloser, error) {
+		level := mapCompressionLevelToBzip2(C.CompressionLevel)
+
+		return bzip2.NewWriter(w, &bzip2.WriterConfig{Level: level})
+	})
+}
+
+// mapCompressionLevelToBzip2 maps 1-9 compression level to bzip2 level.
+func mapCompressionLevelToBzip2(level int) int {
+	if level < 1 {
+		return bzip2.DefaultCompression
+	}
+	if level > 9 {
+		return bzip2.BestCompression
+	}
+
+	return level
+}
+
+/* vim: setlocal ft=go noet ai ts=4 sw=4 sts=4: */

internal/backer/consts.go

@@ -0,0 +1,72 @@
+package backer
+
+import (
+	"time"
+)
+
+const (
+	// readHeaderTimeout is the maximum duration for reading request headers.
+	readHeaderTimeout = 5 * time.Second
+
+	// copyBufferSize is the buffer size used for streaming file contents.
+	copyBufferSize = 32 * 1024
+
+	// maxHeaderBytes is the maximum size of request headers (1KB).
+	maxHeaderBytes = 1024
+
+	// idleTimeout is the maximum time to wait for the next request on keep-alive connections.
+	idleTimeout = 10 * time.Second
+
+	// portMin is the minimum valid port number.
+	portMin = 1
+
+	// portMax is the maximum valid port number.
+	portMax = 65535
+
+	// backupTimeoutMin is the minimum backup timeout in minutes.
+	backupTimeoutMin = 1
+
+	// backupTimeoutMax is the maximum backup timeout in minutes (24 hours).
+	backupTimeoutMax = 1440
+
+	// dirScanTimeoutMin is the minimum directory scan timeout in minutes.
+	dirScanTimeoutMin = 1
+
+	// dirScanTimeoutMax is the maximum directory scan timeout in minutes (24 hours).
+	dirScanTimeoutMax = 1440
+
+	// compressionLevelMin is the minimum compression level.
+	compressionLevelMin = 1
+
+	// compressionLevelMax is the maximum compression level.
+	compressionLevelMax = 9
+
+	// defaultPort is the default listen port.
+	defaultPort = 8086
+
+	// defaultBackupTimeout is the default backup timeout in minutes.
+	defaultBackupTimeout = 60
+
+	// defaultDirScanTimeout is the default directory scan timeout in minutes (1 hour).
+	defaultDirScanTimeout = 60
+
+	// defaultCompressionLevel is the default compression level.
+	defaultCompressionLevel = 9
+
+	// defaultLocation is the default API endpoint path.
+	defaultLocation = "/archive"
+
+	// defaultAddress is the default listen address.
+	defaultAddress = "0.0.0.0"
+
+	// defaultLogLevel is the default log verbosity.
+	defaultLogLevel = "info"
+
+	// defaultFilenamePrefix is the default prefix for backup filenames.
+	defaultFilenamePrefix = "backup"
+
+	// defaultDefaultCompression is the default compression algorithm.
+	defaultDefaultCompression = "gzip"
+)
+
+/* vim: setlocal ft=go noet ai ts=4 sw=4 sts=4: */

internal/backer/getFilesFromDirectories.go

@@ -1,11 +1,13 @@
-// Package backer implements simple backup server.
 package backer
 
 import (
+	"context"
+	"errors"
 	"io/fs"
+	"os"
 	"path/filepath"
 
-	"backer/internal/log"
+	backerlog "backer/internal/log"
 )
 
 // isExcluded checks if path matches any of the configured exclude patterns.
@@ -23,19 +25,38 @@ func isExcluded(path string) bool {
 }
 
 // GetFilesFromDirectories makes a file list of given directories.
-func GetFilesFromDirectories(directories []string) ([]string, error) {
+// Logs errors for directories that don't exist or can't be accessed but continues with valid directories.
+// Takes context with timeout from config dir_scan_timeout setting.
+func GetFilesFromDirectories(ctx context.Context, directories []string) ([]string, error) {
 	var files []string
 
 	for _, dir := range directories {
+		// Check directory exists before walking.
+		if _, err := os.Stat(dir); err != nil {
+			if errors.Is(err, os.ErrNotExist) {
+				backerlog.Errorf("Configured backup directory not found: %s", dir)
+			} else {
+				backerlog.Errorf("Failed to access configured directory %s: %v", dir, err)
+			}
+
+			continue
+		}
+
 		err := filepath.WalkDir(dir, func(path string, de fs.DirEntry, err error) error { //nolint: revive
+			select {
+			case <-ctx.Done():
+				return ctx.Err()
+			default:
+			}
+
 			if err != nil {
-				log.Warnf("Skipping %s: %v", path, err)
+				backerlog.Warnf("Skipping %s: %v", path, err)
 
 				return nil
 			}
 
 			if isExcluded(path) {
-				log.Debugf("Excluding: %s", path)
+				backerlog.Debugf("Excluding: %s", path)
 
 				return nil
 			}
@@ -46,7 +67,13 @@ func GetFilesFromDirectories(directories []string) ([]string, error) {
 		})
 
 		if err != nil {
-			log.Warnf("Failed to walk directory %s: %v", dir, err)
+			if errors.Is(err, context.DeadlineExceeded) {
+				backerlog.Errorf("Directory scan timed out after scanning %d files", len(files))
+
+				return files, ErrDirectoryScanTimeout{ScannedFiles: len(files)}
+			}
+
+			backerlog.Warnf("Failed to walk directory %s: %v", dir, err)
 		}
 	}
 

internal/backer/getFilesFromDirectories_test.go

@@ -1,7 +1,9 @@
 package backer
 
 import (
+	"context"
 	"os"
+	"path/filepath"
 	"regexp"
 	"slices"
 	"strings"
@@ -13,30 +15,47 @@ func normalizePath(p string) string {
 	return strings.ReplaceAll(p, "\\", "/")
 }
 
+// normalizePathSlice normalizes all paths in a slice.
+func normalizePathSlice(paths []string) []string {
+	result := make([]string, len(paths))
+	for i, p := range paths {
+		result[i] = normalizePath(p)
+	}
+	return result
+}
+
 // TestGetFilesFromDirectories checks if GetFilesFromDirectories returns the proper amount of strings, correct strings and no error.
 func TestGetFilesFromDirectories(t *testing.T) {
-	// Ensure empty_dir exists for the test.
+	testDataPath := "../../test_data"
-	os.MkdirAll("../../test_data/test1/foo/empty_dir", 0755)
+
+	// Log the paths we're checking for debugging
+	t.Logf("Checking directories: %v", []string{
+		filepath.Join(testDataPath, "test1", "foo"),
+		filepath.Join(testDataPath, "test1", "bar"),
+	})
 
 	var (
 		input = Config{
-			Directories: []string{"../../test_data/test1/foo", "../../test_data/test1/bar"},
+			Directories: []string{
+				filepath.Join(testDataPath, "test1", "foo"),
+				filepath.Join(testDataPath, "test1", "bar"),
+			},
 		}
 		expectedData = []string{
-			"../../test_data/test1/foo",
+			filepath.Join(testDataPath, "test1", "foo"),
-			"../../test_data/test1/foo/empty_dir",
+			filepath.Join(testDataPath, "test1", "foo", "empty_dir"),
-			"../../test_data/test1/foo/hello_breakfast.txt",
+			filepath.Join(testDataPath, "test1", "foo", "hello_breakfast.txt"),
-			"../../test_data/test1/foo/mydir",
+			filepath.Join(testDataPath, "test1", "foo", "mydir"),
-			"../../test_data/test1/foo/mydir/myfile.txt",
+			filepath.Join(testDataPath, "test1", "foo", "mydir", "myfile.txt"),
-			"../../test_data/test1/foo/some_text.txt",
+			filepath.Join(testDataPath, "test1", "foo", "some_text.txt"),
-			"../../test_data/test1/bar",
+			filepath.Join(testDataPath, "test1", "bar"),
-			"../../test_data/test1/bar/goodbye.txt",
+			filepath.Join(testDataPath, "test1", "bar", "goodbye.txt"),
-			"../../test_data/test1/bar/goodbye.txt/text.txt",
+			filepath.Join(testDataPath, "test1", "bar", "goodbye.txt", "text.txt"),
-			"../../test_data/test1/bar/hello.txt",
+			filepath.Join(testDataPath, "test1", "bar", "hello.txt"),
 		}
 	)
 
-	output, err := GetFilesFromDirectories(input.Directories)
+	output, err := GetFilesFromDirectories(context.Background(), input.Directories)
 
 	if err != nil {
 		t.Errorf("Expected no error, got %v", err)
@@ -51,7 +70,8 @@ func TestGetFilesFromDirectories(t *testing.T) {
 
 	for _, filepath := range output {
 		normalized := normalizePath(filepath)
-		if !slices.Contains(expectedData, normalized) {
+		normalizedExpected := normalizePathSlice(expectedData)
+		if !slices.Contains(normalizedExpected, normalized) {
 			t.Errorf("expectedData data does not contain string from output: %s", filepath)
 		}
 	}
@@ -59,7 +79,8 @@ func TestGetFilesFromDirectories(t *testing.T) {
 
 // TestGetFilesFromDirectoriesSingleDir tests with a single directory.
 func TestGetFilesFromDirectoriesSingleDir(t *testing.T) {
-	output, err := GetFilesFromDirectories([]string{"../../test_data/test1/foo"})
+	testDataPath := filepath.Join("../../test_data")
+	output, err := GetFilesFromDirectories(context.Background(), []string{filepath.Join(testDataPath, "test1", "foo")})
 
 	if err != nil {
 		t.Errorf("Expected no error, got %v", err)
@@ -70,7 +91,7 @@ func TestGetFilesFromDirectoriesSingleDir(t *testing.T) {
 	}
 
 	// Should contain the directory itself
-	if !slices.Contains(output, "../../test_data/test1/foo") {
+	if !slices.Contains(output, filepath.Join(testDataPath, "test1", "foo")) {
 		t.Error("Output should contain the root directory")
 	}
 }
@@ -80,7 +101,7 @@ func TestGetFilesFromDirectoriesEmptyDir(t *testing.T) {
 	// Create temp empty dir
 	tmpDir := t.TempDir()
 
-	output, err := GetFilesFromDirectories([]string{tmpDir})
+	output, err := GetFilesFromDirectories(context.Background(), []string{tmpDir})
 
 	if err != nil {
 		t.Errorf("Expected no error, got %v", err)
@@ -93,23 +114,48 @@ func TestGetFilesFromDirectoriesEmptyDir(t *testing.T) {
 }
 
 // TestGetFilesFromDirectoriesNonExistent tests with non-existent directory.
+// Should log error but not fail - returns empty list.
 func TestGetFilesFromDirectoriesNonExistent(t *testing.T) {
-	output, err := GetFilesFromDirectories([]string{"/nonexistent/path/that/does/not/exist"})
+	testDataPath := filepath.Join("../../test_data")
+	nonExistentPath := filepath.Join(testDataPath, "nonexistent", "path", "that", "does", "not", "exist")
+	output, err := GetFilesFromDirectories(context.Background(), []string{nonExistentPath})
 
+	// Should not return error, but logs error
 	if err != nil {
-		t.Errorf("Expected no error (logs warning instead), got %v", err)
+		t.Errorf("Expected no error (logs error instead), got %v", err)
 	}
 
+	// Should return empty list
 	if len(output) != 0 {
 		t.Errorf("Expected empty output for non-existent directory, got %d files", len(output))
 	}
 }
 
+// TestGetFilesFromDirectoriesPartialFailure tests with mix of valid and invalid directories.
+func TestGetFilesFromDirectoriesPartialFailure(t *testing.T) {
+	testDataPath := filepath.Join("../../test_data")
+	// Create a temp directory that exists
+	tmpDir := t.TempDir()
+
+	output, err := GetFilesFromDirectories(context.Background(), []string{tmpDir, filepath.Join(testDataPath, "nonexistent", "path", "that", "does", "not", "exist")})
+
+	// Should not return error, but logs error
+	if err != nil {
+		t.Errorf("Expected no error (logs error instead), got %v", err)
+	}
+
+	// Should return files from valid directory
+	if len(output) == 0 {
+		t.Error("Expected files from valid directory")
+	}
+}
+
 // TestGetFilesFromDirectoriesMultipleDirs tests multiple directories.
 func TestGetFilesFromDirectoriesMultipleDirs(t *testing.T) {
-	output, err := GetFilesFromDirectories([]string{
+	testDataPath := filepath.Join("../../test_data")
-		"../../test_data/test1/foo",
+	output, err := GetFilesFromDirectories(context.Background(), []string{
-		"../../test_data/test1/bar",
+		filepath.Join(testDataPath, "test1", "foo"),
+		filepath.Join(testDataPath, "test1", "bar"),
 	})
 
 	if err != nil {
@@ -118,10 +164,10 @@ func TestGetFilesFromDirectoriesMultipleDirs(t *testing.T) {
 
 	// Should contain entries from both directories
 	hasFoo := slices.ContainsFunc(output, func(s string) bool {
-		return s == "../../test_data/test1/foo" || s == "../../test_data/test1/foo/hello_breakfast.txt"
+		return s == filepath.Join(testDataPath, "test1", "foo") || s == filepath.Join(testDataPath, "test1", "foo", "hello_breakfast.txt")
 	})
 	hasBar := slices.ContainsFunc(output, func(s string) bool {
-		return s == "../../test_data/test1/bar" || s == "../../test_data/test1/bar/hello.txt"
+		return s == filepath.Join(testDataPath, "test1", "bar") || s == filepath.Join(testDataPath, "test1", "bar", "hello.txt")
 	})
 
 	if !hasFoo {
@@ -149,7 +195,7 @@ func TestGetFilesFromDirectoriesSymlink(t *testing.T) {
 		t.Skip("Symlinks not supported, skipping test")
 	}
 
-	output, err := GetFilesFromDirectories([]string{tmpDir})
+	output, err := GetFilesFromDirectories(context.Background(), []string{tmpDir})
 
 	if err != nil {
 		t.Errorf("Expected no error, got %v", err)
@@ -228,7 +274,7 @@ func TestGetFilesFromDirectoriesWithExcludes(t *testing.T) {
 		regexp.MustCompile(`\.tmp$`),
 	}
 
-	output, err := GetFilesFromDirectories([]string{tmpDir})
+	output, err := GetFilesFromDirectories(context.Background(), []string{tmpDir})
 	if err != nil {
 		t.Fatalf("Unexpected error: %v", err)
 	}

internal/backer/globals.go

@@ -6,8 +6,11 @@ import (
 
 var (
 	// C contains the global configuration loaded from config file.
+	// This is intentional — config is loaded once at startup and never modified concurrently.
 	C Config
+
 	// excludePatterns contains compiled regex patterns from C.ExcludePatterns.
+	// Populated by LoadConfig, used by isExcluded during directory walking.
 	excludePatterns []*regexp.Regexp
 )
 

internal/backer/gzip.go

@@ -0,0 +1,25 @@
+package backer
+
+import (
+	"compress/gzip"
+	"context"
+	"io"
+
+	"github.com/klauspost/pgzip"
+)
+
+// CreateTarGzStream takes a list of files and returns a reader that reads from a gzip-compressed tar archive.
+func CreateTarGzStream(ctx context.Context, filepaths []string) io.ReadCloser {
+	return createArchiveStream(ctx, filepaths, func(w io.Writer) (io.WriteCloser, error) {
+		return gzip.NewWriterLevel(w, C.CompressionLevel)
+	})
+}
+
+// CreateTarPgzipStream takes a list of files and returns a reader that reads from a parallel gzip-compressed tar archive.
+func CreateTarPgzipStream(ctx context.Context, filepaths []string) io.ReadCloser {
+	return createArchiveStream(ctx, filepaths, func(w io.Writer) (io.WriteCloser, error) {
+		return pgzip.NewWriterLevel(w, C.CompressionLevel)
+	})
+}
+
+/* vim: setlocal ft=go noet ai ts=4 sw=4 sts=4: */

internal/backer/loadConfig.go

@@ -38,25 +38,25 @@ func LoadConfig(path string) error {
 
 	// Address: default and no range validation needed.
 	if C.Address == "" {
-		C.Address = "0.0.0.0"
+		C.Address = defaultAddress
 
 		log.Warnf("Config option address is not set, fallback to %s", C.Address)
 	}
 
 	// Port: set default, then validate range.
 	if C.Port == 0 {
-		C.Port = 8086
+		C.Port = defaultPort
 
 		log.Warnf("Config option port is not set, fallback to %d", C.Port)
 	}
 
-	if C.Port < 1 || C.Port > 65535 {
+	if C.Port < portMin || C.Port > portMax {
-		return fmt.Errorf("Port must be between 1 and 65535, got %d", C.Port)
+		return fmt.Errorf("Config option port must be between %d and %d, got %d", portMin, portMax, C.Port)
 	}
 
 	// Location: default and no range validation needed.
 	if C.Location == "" {
-		C.Location = "/archive"
+		C.Location = defaultLocation
 
 		log.Warnf("Config option location is not set, fallback to %s", C.Location)
 	}
@@ -68,7 +68,7 @@ func LoadConfig(path string) error {
 
 	// LogLevel: default and no range validation needed.
 	if C.LogLevel == "" {
-		C.LogLevel = "info"
+		C.LogLevel = defaultLogLevel
 
 		log.Warnf("Config option loglevel is not set, fallback to %s", C.LogLevel)
 	}
@@ -86,24 +86,35 @@ func LoadConfig(path string) error {
 
 	// BackupTimeout: set default, then validate range.
 	if C.BackupTimeout == 0 {
-		C.BackupTimeout = 60
+		C.BackupTimeout = defaultBackupTimeout
 
 		log.Warnf("Config option backup_timeout is not set, fallback to %d minutes", C.BackupTimeout)
 	}
 
-	if C.BackupTimeout < 1 || C.BackupTimeout > 1440 {
+	if C.BackupTimeout < backupTimeoutMin || C.BackupTimeout > backupTimeoutMax {
-		return fmt.Errorf("Backup_timeout must be between 1 and 1440 minutes, got %d", C.BackupTimeout)
+		return fmt.Errorf("Config option backup_timeout must be between %d and %d minutes, got %d", backupTimeoutMin, backupTimeoutMax, C.BackupTimeout)
+	}
+
+	// DirScanTimeout: set default, then validate range.
+	if C.DirScanTimeout == 0 {
+		C.DirScanTimeout = defaultDirScanTimeout
+
+		log.Warnf("Config option dir_scan_timeout is not set, fallback to %d minutes", C.DirScanTimeout)
+	}
+
+	if C.DirScanTimeout < dirScanTimeoutMin || C.DirScanTimeout > dirScanTimeoutMax {
+		return fmt.Errorf("Config option dir_scan_timeout must be between %d and %d minutes, got %d", dirScanTimeoutMin, dirScanTimeoutMax, C.DirScanTimeout)
 	}
 
 	// CompressionLevel: set default, then validate range.
 	if C.CompressionLevel == 0 {
-		C.CompressionLevel = 9
+		C.CompressionLevel = defaultCompressionLevel
 
 		log.Warnf("Config option compression_level is not set, fallback to %d", C.CompressionLevel)
 	}
 
-	if C.CompressionLevel < 1 || C.CompressionLevel > 9 {
+	if C.CompressionLevel < compressionLevelMin || C.CompressionLevel > compressionLevelMax {
-		return fmt.Errorf("Compression_level must be between 1 and 9, got %d", C.CompressionLevel)
+		return fmt.Errorf("Config option compression_level must be between %d and %d, got %d", compressionLevelMin, compressionLevelMax, C.CompressionLevel)
 	}
 
 	// HTTPS: validate cert/key when HTTPS enabled.
@@ -120,12 +131,35 @@ func LoadConfig(path string) error {
 
 	// User: required.
 	if C.User == "" {
-		return errors.New("Option user is not set")
+		return errors.New("Config option user is not set")
 	}
 
 	// Password: required.
 	if C.Password == "" {
-		return errors.New("Option password is not set")
+		return errors.New("Config option password is not set")
+	}
+
+	// FilenamePrefix: default.
+	if C.FilenamePrefix == "" {
+		C.FilenamePrefix = defaultFilenamePrefix
+
+		log.Warnf("Config option filename_prefix is not set, fallback to %s", C.FilenamePrefix)
+	}
+
+	// DefaultCompression: default and validate. If compression_algorithm is set (deprecated), use it as fallback.
+	if C.DefaultCompression == "" {
+		if C.CompressionAlgorithm != "" {
+			C.DefaultCompression = C.CompressionAlgorithm
+		} else {
+			C.DefaultCompression = defaultDefaultCompression
+		}
+	}
+
+	switch C.DefaultCompression {
+	case "gzip", "pgzip", "bzip2", "xz", "lz4", "zstd":
+		// No-op: compression algorithm validated.
+	default:
+		return fmt.Errorf("Config option default_compression must be gzip, bzip2, zstd, lz4, xz or pgzip, got %s", C.DefaultCompression)
 	}
 
 	// Compile exclude patterns.

internal/backer/loadConfig_test.go

@@ -1,7 +1,9 @@
 package backer
 
 import (
+	"encoding/json"
 	"os"
+	"path/filepath"
 	"slices"
 	"testing"
 )
@@ -10,18 +12,50 @@ func TestLoadConfig(t *testing.T) {
 	// Reset global config
 	C = Config{}
 
-	var (
+	testDataPath := filepath.Join(t.TempDir(), "test_data")
-		testConfigPath = "./../../test_data/test_config.json"
+	if err := os.MkdirAll(filepath.Join(testDataPath, "test1", "foo"), 0755); err != nil {
-		expectedData   = Config{
+		t.Fatalf("Failed to create test directory foo: %v", err)
+	}
+	if err := os.MkdirAll(filepath.Join(testDataPath, "test1", "bar"), 0755); err != nil {
+		t.Fatalf("Failed to create test directory bar: %v", err)
+	}
+
+	// Create test config file
+	testConfigPath := filepath.Join(testDataPath, "test_config.json")
+	testStruct := struct {
+		Address     string   `json:"address"`
+		Port        int      `json:"port"`
+		Directories []string `json:"directories"`
+		Location    string   `json:"location"`
+		User        string   `json:"user"`
+		Password    string   `json:"password"`
+		NoHTTPS     bool     `json:"nohttps"`
+	}{
+		Address:     "127.0.0.1",
+		Port:        8086,
+		Directories: []string{filepath.Join(testDataPath, "test1", "foo"), filepath.Join(testDataPath, "test1", "bar")},
+		Location:    "/archive",
+		User:        "test",
+		Password:    "test",
+		NoHTTPS:     true,
+	}
+	testConfigBytes, _ := json.MarshalIndent(testStruct, "", "  ")
+	if err := os.WriteFile(testConfigPath, testConfigBytes, 0644); err != nil {
+		t.Fatal(err)
+	}
+
+	expectedData := Config{
 		Address: "127.0.0.1",
 		Port:    8086,
 		Directories: []string{
-				"../../test_data/test1/foo",
+			filepath.Join(testDataPath, "test1", "foo"),
-				"../../test_data/test1/bar",
+			filepath.Join(testDataPath, "test1", "bar"),
 		},
 		Location: "/archive",
+		User:     "test",
+		Password: "test",
+		NoHTTPS:  true,
 	}
-	)
 
 	err := LoadConfig(testConfigPath)
 
@@ -57,6 +91,8 @@ func TestLoadConfig(t *testing.T) {
 
 // TestLoadConfigMissingUser tests that config fails when user is missing.
 func TestLoadConfigMissingUser(t *testing.T) {
+	// Reset global config
+	C = Config{}
 	config := `{
 		"port": 8086,
 		"password": "test",
@@ -72,6 +108,8 @@ func TestLoadConfigMissingUser(t *testing.T) {
 
 // TestLoadConfigMissingPassword tests that config fails when password is missing.
 func TestLoadConfigMissingPassword(t *testing.T) {
+	// Reset global config
+	C = Config{}
 	config := `{
 		"port": 8086,
 		"user": "test",
@@ -87,6 +125,8 @@ func TestLoadConfigMissingPassword(t *testing.T) {
 
 // TestLoadConfigMissingDirectories tests that config fails when directories is missing.
 func TestLoadConfigMissingDirectories(t *testing.T) {
+	// Reset global config
+	C = Config{}
 	config := `{
 		"port": 8086,
 		"user": "test",
@@ -102,6 +142,8 @@ func TestLoadConfigMissingDirectories(t *testing.T) {
 
 // TestLoadConfigInvalidPort tests that config fails with invalid port.
 func TestLoadConfigInvalidPort(t *testing.T) {
+	// Reset global config
+	C = Config{}
 	config := `{
 		"port": 99999,
 		"user": "test",
@@ -118,6 +160,8 @@ func TestLoadConfigInvalidPort(t *testing.T) {
 
 // TestLoadConfigNonExistentDirectory tests that config fails with non-existent directory.
 func TestLoadConfigNonExistentDirectory(t *testing.T) {
+	// Reset global config
+	C = Config{}
 	config := `{
 		"port": 8086,
 		"user": "test",
@@ -134,6 +178,8 @@ func TestLoadConfigNonExistentDirectory(t *testing.T) {
 
 // TestLoadConfigHTTPSWithoutCert tests that config fails when HTTPS enabled but no cert.
 func TestLoadConfigHTTPSWithoutCert(t *testing.T) {
+	// Reset global config
+	C = Config{}
 	config := `{
 		"port": 8086,
 		"user": "test",
@@ -151,6 +197,8 @@ func TestLoadConfigHTTPSWithoutCert(t *testing.T) {
 
 // TestLoadConfigNoHTTPS tests that config succeeds with nohttps=true and no cert.
 func TestLoadConfigNoHTTPS(t *testing.T) {
+	// Reset global config
+	C = Config{}
 	config := `{
 		"port": 8086,
 		"user": "test",
@@ -170,8 +218,33 @@ func TestLoadConfigNoHTTPS(t *testing.T) {
 	}
 }
 
+// TestLoadConfigFilenamePrefixCustom tests custom filename prefix.
+func TestLoadConfigFilenamePrefixCustom(t *testing.T) {
+	// Reset global config
+	C = Config{}
+	config := `{
+		"user": "test",
+		"password": "test",
+		"directories": ["../../test_data/test1/foo"],
+		"nohttps": true,
+		"filename_prefix": "mybackup"
+	}`
+	tmpFile := writeTempConfig(t, config)
+
+	err := LoadConfig(tmpFile)
+	if err != nil {
+		t.Errorf("Expected no error, got %v", err)
+	}
+
+	if C.FilenamePrefix != "mybackup" {
+		t.Errorf("C.FilenamePrefix should be mybackup, got %s", C.FilenamePrefix)
+	}
+}
+
 // TestLoadConfigDefaultValues tests default values are set correctly.
 func TestLoadConfigDefaultValues(t *testing.T) {
+	// Reset global config
+	C = Config{}
 	config := `{
 		"user": "test",
 		"password": "test",
@@ -201,6 +274,148 @@ func TestLoadConfigDefaultValues(t *testing.T) {
 	if C.LogLevel != "info" {
 		t.Errorf("C.LogLevel should default to info, got %s", C.LogLevel)
 	}
+
+	if C.FilenamePrefix != "backup" {
+		t.Errorf("C.FilenamePrefix should default to backup, got %s", C.FilenamePrefix)
+	}
+
+	if C.DefaultCompression != "gzip" {
+		t.Errorf("C.DefaultCompression should default to gzip, got %s", C.DefaultCompression)
+	}
+}
+
+// TestLoadConfigCompressionAlgorithmBzip2 tests bzip2 compression algorithm.
+func TestLoadConfigCompressionAlgorithmBzip2(t *testing.T) {
+	// Reset global config
+	C = Config{}
+	config := `{
+		"user": "test",
+		"password": "test",
+		"directories": ["../../test_data/test1/foo"],
+		"nohttps": true,
+		"compression_algorithm": "bzip2"
+	}`
+	tmpFile := writeTempConfig(t, config)
+
+	err := LoadConfig(tmpFile)
+	if err != nil {
+		t.Errorf("Expected no error, got %v", err)
+	}
+
+	if C.DefaultCompression != "bzip2" {
+		t.Errorf("C.DefaultCompression should be bzip2, got %s", C.DefaultCompression)
+	}
+}
+
+// TestLoadConfigCompressionAlgorithmZstd tests zstd compression algorithm.
+func TestLoadConfigCompressionAlgorithmZstd(t *testing.T) {
+	// Reset global config
+	C = Config{}
+	config := `{
+		"user": "test",
+		"password": "test",
+		"directories": ["../../test_data/test1/foo"],
+		"nohttps": true,
+		"compression_algorithm": "zstd"
+	}`
+	tmpFile := writeTempConfig(t, config)
+
+	err := LoadConfig(tmpFile)
+	if err != nil {
+		t.Errorf("Expected no error, got %v", err)
+	}
+
+	if C.DefaultCompression != "zstd" {
+		t.Errorf("C.DefaultCompression should be zstd, got %s", C.DefaultCompression)
+	}
+}
+
+// TestLoadConfigCompressionAlgorithmLz4 tests lz4 compression algorithm.
+func TestLoadConfigCompressionAlgorithmLz4(t *testing.T) {
+	// Reset global config
+	C = Config{}
+	config := `{
+		"user": "test",
+		"password": "test",
+		"directories": ["../../test_data/test1/foo"],
+		"nohttps": true,
+		"compression_algorithm": "lz4"
+	}`
+	tmpFile := writeTempConfig(t, config)
+
+	err := LoadConfig(tmpFile)
+	if err != nil {
+		t.Errorf("Expected no error, got %v", err)
+	}
+
+	if C.DefaultCompression != "lz4" {
+		t.Errorf("C.DefaultCompression should be lz4, got %s", C.DefaultCompression)
+	}
+}
+
+// TestLoadConfigCompressionAlgorithmXz tests xz compression algorithm.
+func TestLoadConfigCompressionAlgorithmXz(t *testing.T) {
+	// Reset global config
+	C = Config{}
+	config := `{
+		"user": "test",
+		"password": "test",
+		"directories": ["../../test_data/test1/foo"],
+		"nohttps": true,
+		"compression_algorithm": "xz"
+	}`
+	tmpFile := writeTempConfig(t, config)
+
+	err := LoadConfig(tmpFile)
+	if err != nil {
+		t.Errorf("Expected no error, got %v", err)
+	}
+
+	if C.DefaultCompression != "xz" {
+		t.Errorf("C.DefaultCompression should be xz, got %s", C.DefaultCompression)
+	}
+}
+
+// TestLoadConfigCompressionAlgorithmPgzip tests pgzip compression algorithm.
+func TestLoadConfigCompressionAlgorithmPgzip(t *testing.T) {
+	// Reset global config
+	C = Config{}
+	config := `{
+		"user": "test",
+		"password": "test",
+		"directories": ["../../test_data/test1/foo"],
+		"nohttps": true,
+		"compression_algorithm": "pgzip"
+	}`
+	tmpFile := writeTempConfig(t, config)
+
+	err := LoadConfig(tmpFile)
+	if err != nil {
+		t.Errorf("Expected no error, got %v", err)
+	}
+
+	if C.DefaultCompression != "pgzip" {
+		t.Errorf("C.DefaultCompression should be pgzip, got %s", C.DefaultCompression)
+	}
+}
+
+// TestLoadConfigCompressionAlgorithmInvalid tests invalid compression algorithm.
+func TestLoadConfigCompressionAlgorithmInvalid(t *testing.T) {
+	// Reset global config
+	C = Config{}
+	config := `{
+		"user": "test",
+		"password": "test",
+		"directories": ["../../test_data/test1/foo"],
+		"nohttps": true,
+		"compression_algorithm": "invalid"
+	}`
+	tmpFile := writeTempConfig(t, config)
+
+	err := LoadConfig(tmpFile)
+	if err == nil {
+		t.Error("Expected error for invalid compression algorithm, got nil")
+	}
 }
 
 // writeTempConfig is a helper to write a temp config file for testing.
@@ -216,7 +431,6 @@ func writeTempConfig(t *testing.T, content string) string {
 	tmpFile.Close()
 	// Reset global config before loading
 	C = Config{}
-	excludePatterns = nil
 	return tmpFile.Name()
 }
 

internal/backer/lz4.go

@@ -0,0 +1,35 @@
+package backer
+
+import (
+	"context"
+	"io"
+
+	"github.com/pierrec/lz4"
+)
+
+// CreateTarLz4Stream takes a list of files and returns a reader that reads from an lz4-compressed tar archive.
+func CreateTarLz4Stream(ctx context.Context, filepaths []string) io.ReadCloser {
+	return createArchiveStream(ctx, filepaths, func(w io.Writer) (io.WriteCloser, error) {
+		lz4Writer := lz4.NewWriter(w)
+		// Map compression_level (1-9) to lz4 compression level.
+		// 1-3: fast, 4-6: balanced, 7-9: better compression.
+		lz4Writer.Header.CompressionLevel = mapCompressionLevelToLz4(C.CompressionLevel) //nolint: staticcheck
+
+		return lz4Writer, nil
+	})
+}
+
+// mapCompressionLevelToLz4 maps 1-9 compression level to lz4 compression level.
+// 1-3: fast, 4-6: balanced, 7-9: better compression.
+func mapCompressionLevelToLz4(level int) int {
+	switch {
+	case level <= 3:
+		return 0 // Fastest.
+	case level <= 6:
+		return 1 // Default.
+	default:
+		return 2 // Best compression.
+	}
+}
+
+/* vim: setlocal ft=go noet ai ts=4 sw=4 sts=4: */

internal/backer/main_test.go

@@ -0,0 +1,103 @@
+package backer
+
+import (
+	"context"
+	"fmt"
+	"os"
+	"path/filepath"
+	"testing"
+)
+
+const testDataPath = "../../test_data"
+
+func TestMain(m *testing.M) {
+	if err := setupTestData(); err != nil {
+		fmt.Fprintf(os.Stderr, "FATAL: Failed to setup test data: %v\n", err)
+		os.Exit(1)
+	}
+
+	code := m.Run()
+
+	if err := cleanupTestData(); err != nil {
+		fmt.Fprintf(os.Stderr, "WARNING: Failed to cleanup test data: %v\n", err)
+	}
+
+	os.Exit(code)
+}
+
+func setupTestData() error {
+	testDir := filepath.Join(testDataPath, "test1")
+
+	dirs := []string{
+		filepath.Join(testDir, "foo", "mydir"),
+		filepath.Join(testDir, "bar", "goodbye.txt"),
+		filepath.Join(testDir, "hardlinks"),
+		filepath.Join(testDir, "symlinks"),
+		filepath.Join(testDir, "foo", "empty_dir"),
+		filepath.Join(testDataPath, "tmp"),
+	}
+
+	for _, dir := range dirs {
+		if err := os.MkdirAll(dir, 0755); err != nil {
+			return fmt.Errorf("Failed to create directory %s: %w", dir, err)
+		}
+	}
+
+	files := map[string]string{
+		filepath.Join(testDir, "foo", "hello_breakfast.txt"):   "hello breakfast content",
+		filepath.Join(testDir, "foo", "some_text.txt"):         "some text content",
+		filepath.Join(testDir, "foo", "mydir", "myfile.txt"):   "my file content",
+		filepath.Join(testDir, "bar", "hello.txt"):             "hello content",
+		filepath.Join(testDir, "bar", "goodbye.txt", "text.txt"): "text content",
+		filepath.Join(testDir, "hardlinks", "original.txt"):    "original content",
+		filepath.Join(testDir, "symlinks", "target.txt"):       "target content",
+	}
+
+	for path, content := range files {
+		if err := os.WriteFile(path, []byte(content), 0644); err != nil {
+			return fmt.Errorf("Failed to write file %s: %w", path, err)
+		}
+	}
+
+	orig := filepath.Join(testDir, "hardlinks", "original.txt")
+	link1 := filepath.Join(testDir, "hardlinks", "hardlink1.txt")
+	if err := os.Link(orig, link1); err != nil {
+		return fmt.Errorf("Failed to create hardlink %s -> %s: %w", link1, orig, err)
+	}
+
+	link2 := filepath.Join(testDir, "hardlinks", "hardlink2.txt")
+	if err := os.Link(orig, link2); err != nil {
+		return fmt.Errorf("Failed to create hardlink %s -> %s: %w", link2, orig, err)
+	}
+
+	target := filepath.Join(testDir, "symlinks", "target.txt")
+	link := filepath.Join(testDir, "symlinks", "link.txt")
+	if err := os.Symlink(target, link); err != nil {
+		return fmt.Errorf("Failed to create symlink %s -> %s: %w", link, target, err)
+	}
+
+	return nil
+}
+
+func cleanupTestData() error {
+	testDir := filepath.Join(testDataPath, "test1")
+
+	if err := os.RemoveAll(testDir); err != nil {
+		return fmt.Errorf("Failed to remove test directory %s: %w", testDir, err)
+	}
+
+	tmpDir := filepath.Join(testDataPath, "tmp")
+	if err := os.RemoveAll(tmpDir); err != nil {
+		return fmt.Errorf("Failed to remove tmp directory %s: %w", tmpDir, err)
+	}
+
+	return nil
+}
+
+func GetTestDataPath() string {
+	return testDataPath
+}
+
+func GetTestFilesFromDirectories(ctx context.Context, directories []string) ([]string, error) {
+	return GetFilesFromDirectories(ctx, directories)
+}

internal/backer/newServer.go

@@ -2,17 +2,69 @@ package backer
 
 import (
 	"context"
+	"crypto/md5"
+	"crypto/subtle"
 	"crypto/tls"
+	"encoding/hex"
+	"errors"
 	"fmt"
 	"io"
 	"net"
 	"net/http"
+	"path"
 	"strings"
 	"time"
 
 	"backer/internal/log"
 )
 
+// isTLSError checks if the given error is TLS-related.
+// It checks for both TLS-specific error types and error messages containing "tls:".
+func isTLSError(err error) bool {
+	if err == nil {
+		return false
+	}
+
+	var alertErr *tls.AlertError
+	if errors.As(err, &alertErr) && alertErr != nil {
+		return true
+	}
+
+	var certErr *tls.CertificateVerificationError
+	if errors.As(err, &certErr) && certErr != nil {
+		return true
+	}
+
+	var echErr *tls.ECHRejectionError
+	if errors.As(err, &echErr) && echErr != nil {
+		return true
+	}
+
+	var recordErr *tls.RecordHeaderError
+	if errors.As(err, &recordErr) && recordErr != nil {
+		return true
+	}
+
+	return strings.HasPrefix(err.Error(), "tls:")
+}
+
+// withStatusLogging wraps a handler to log 404 and 405 responses.
+func withStatusLogging(next http.Handler) http.Handler {
+	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		rw := &responseWriterWrapper{ResponseWriter: w, statusCode: http.StatusOK}
+		next.ServeHTTP(rw, r)
+
+		clientIP := getClientIP(r)
+
+		switch rw.statusCode {
+		case http.StatusNotFound:
+			log.Infof("Not Found from %s to %s", clientIP, r.RequestURI)
+		case http.StatusMethodNotAllowed:
+			log.Infof("Method Not Allowed from %s for %s %s", clientIP, r.Method, r.RequestURI)
+		}
+	})
+}
+
 // withServerHeader wraps a handler to add Server header to all responses.
 func withServerHeader(next http.Handler) http.Handler {
 	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
@@ -22,7 +74,8 @@ func withServerHeader(next http.Handler) http.Handler {
 }
 
 // NewServer starts new server instance with given config file.
-func NewServer(configPath string) (*http.Server, error) {
+// It returns a ServerWrapper that intercepts and classifies server errors.
+func NewServer(configPath string) (*ServerWrapper, error) {
 	_ = log.Init("info", "") // Initialize logger with defaults for early logging.
 
 	if err := LoadConfig(configPath); err != nil {
@@ -33,9 +86,18 @@ func NewServer(configPath string) (*http.Server, error) {
 		log.Errorf("Failed to initialize logger: %v", err)
 	}
 
+	log.Infof("Using %s as default compression algorithm", C.DefaultCompression)
+
 	mux := http.NewServeMux()
 
-	mux.HandleFunc(C.Location, func(w http.ResponseWriter, r *http.Request) {
+	backupHandler := func(w http.ResponseWriter, r *http.Request) {
+		// Only GET method is allowed.
+		if r.Method != http.MethodGet {
+			http.Error(w, "Method Not Allowed", http.StatusMethodNotAllowed)
+
+			return
+		}
+
 		username, password, ok := r.BasicAuth()
 
 		// No auth? Good bye.
@@ -43,16 +105,18 @@ func NewServer(configPath string) (*http.Server, error) {
 			w.Header().Set("WWW-Authenticate", `Basic realm="restricted", charset="UTF-8"`)
 			http.Error(w, "Unauthorized", http.StatusUnauthorized)
 
+			log.Infof("Access denied from %s to %s (no username or password given)", r.RemoteAddr, r.RequestURI)
+
 			return
 		}
 
-		// Both username AND password must match. Fuck De Morgan and his bullshit laws - we explicitly require
+		// Both username AND password must match. Use timing-safe comparison to prevent timing attacks.
-		// that the whole condition fails. That *is* the point.
+		if !timingSafeMatch(username, C.User) || !timingSafeMatch(password, C.Password) {
-		//nolint:staticcheck // QF1001: De Morgan's law is irrelevant here.
-		if !(username == C.User && password == C.Password) {
 			w.Header().Set("WWW-Authenticate", `Basic realm="restricted", charset="UTF-8"`)
 			http.Error(w, "Unauthorized", http.StatusUnauthorized)
 
+			log.Infof("Access denied from %s to %s (username and/or password incorrect)", r.RemoteAddr, r.RequestURI)
+
 			return
 		}
 
@@ -61,53 +125,154 @@ func NewServer(configPath string) (*http.Server, error) {
 
 		log.Infof("Backup started: client=%s user=%s", clientIP, username)
 
-		ctx := r.Context()
+		// Create context with dir_scan_timeout from request context.
-		files, err := GetFilesFromDirectories(C.Directories)
+		ctx, cancel := context.WithTimeout(r.Context(), time.Duration(C.DirScanTimeout)*time.Minute)
+		defer cancel()
+
+		files, err := GetFilesFromDirectories(ctx, C.Directories)
 
 		if err != nil {
+			switch {
+			case errors.Is(err, context.DeadlineExceeded):
+				w.Header().Set("Server", "backer")
+				http.Error(w, "Directory scan timed out", http.StatusRequestTimeout)
+
+				log.Errorf("Directory scan timed out: %v", err)
+
+			default:
 				log.Errorf("Failed to get files: %v", err)
 
 				w.Header().Set("Server", "backer")
 				http.Error(w, "Failed to get files", http.StatusInternalServerError)
+			}
 
 			return
 		}
 
-		tarGz := CreateTarGzStream(ctx, files)
+		if len(files) == 0 {
+			log.Warnf("No files found to backup in configured directories")
+		}
+
+		var (
+			archive   io.ReadCloser
+			extension string
+			algorithm string
+		)
+
+		requestedExt := r.URL.Path
+
+		algorithm = getCompressionAlgorithm(requestedExt)
+
+		switch algorithm {
+		case "bzip2":
+			archive = CreateTarBzip2Stream(ctx, files)
+			extension = "tar.bz2"
+
+		case "zstd":
+			archive = CreateTarZstdStream(ctx, files)
+			if strings.HasSuffix(requestedExt, ".zst") {
+				extension = "tar.zst"
+			} else {
+				extension = "tar.zstd"
+			}
+
+		case "lz4":
+			archive = CreateTarLz4Stream(ctx, files)
+			extension = "tar.lz4"
+
+		case "xz":
+			archive = CreateTarXzStream(ctx, files)
+			extension = "tar.xz"
+
+		case "pgzip":
+			archive = CreateTarPgzipStream(ctx, files)
+			extension = "tar.gz"
+
+		default:
+			archive = CreateTarGzStream(ctx, files)
+			extension = "tar.gz"
+		}
 
 		defer func() {
-			if err := tarGz.Close(); err != nil {
+			if err := archive.Close(); err != nil {
-				log.Errorf("Failed to close tar.gz reader: %v", err)
+				log.Errorf("Failed to close archive reader: %v", err)
 			}
 		}()
 
-		timestamp := time.Now().Format("20060102-150405")
+		timestamp := backupStart.Format("20060102-150405")
-		filename := fmt.Sprintf("backup-%s.tar.gz", timestamp)
+		filename := fmt.Sprintf("%s-%s.%s", C.FilenamePrefix, timestamp, extension)
 
 		w.Header().Set("Content-Disposition", "attachment; filename="+filename)
 		w.Header().Set("Content-Type", "application/octet-stream")
 
-		bytesWritten, err := io.Copy(w, tarGz)
+		bytesWritten, md5Hash, err := copyWithContext(ctx, w, archive)
+		if err != nil {
 			duration := time.Since(backupStart)
 
-		if err != nil {
+			if errors.Is(err, context.Canceled) {
+				log.Infof("Backup canceled: client=%s user=%s duration=%s", clientIP, username, duration)
+
+				return
+			}
+
+			if isPipeClosedError(err) {
+				log.Infof("Client disconnected during backup: client=%s user=%s", clientIP, username)
+
+				return
+			}
+
 			log.Errorf("Backup failed: client=%s user=%s duration=%s error=%v", clientIP, username, duration, err)
 
 			return
 		}
 
-		log.Infof("Backup completed: client=%s user=%s files=%d bytes=%d duration=%s", clientIP, username, len(files), bytesWritten, duration)
+		log.Infof("Backup completed: client=%s user=%s files=%d bytes=%d md5=%s duration=%s", clientIP, username, len(files), bytesWritten, md5Hash, time.Since(backupStart))
-	})
+	}
 
-	return &http.Server{
+	mux.Handle(C.Location, http.HandlerFunc(backupHandler))
+
+	extensions := []string{".tar.gz", ".tar.xz", ".tar.bz2", ".tar.lz4", ".tar.zst", ".tar.zstd"}
+	for _, ext := range extensions {
+		muxPath := C.Location + ext
+		mux.Handle(muxPath, http.HandlerFunc(backupHandler))
+	}
+
+	// HTTP mode if nohttps is enabled in config.
+	// HTTPS mode if nohttps is disabled in config.
+	if C.NoHTTPS {
+		srv := &ServerWrapper{
+			Server: &http.Server{
 				Addr:              fmt.Sprintf("%s:%d", C.Address, C.Port),
-		Handler:           withServerHeader(mux),
+				Handler:           withServerHeader(withStatusLogging(mux)),
-		ReadHeaderTimeout: 5 * time.Second,
+				ReadHeaderTimeout: readHeaderTimeout,
 				WriteTimeout:      time.Duration(C.BackupTimeout) * time.Minute,
+				IdleTimeout:       idleTimeout,
+				MaxHeaderBytes:    maxHeaderBytes,
+				ErrorLog:          log.DebugLogger(),
+			},
+		}
+		srv.SetKeepAlivesEnabled(false)
+
+		return srv, nil
+	}
+
+	srv := &ServerWrapper{
+		Server: &http.Server{
+			Addr:              fmt.Sprintf("%s:%d", C.Address, C.Port),
+			Handler:           withServerHeader(withStatusLogging(mux)),
+			ReadHeaderTimeout: readHeaderTimeout,
+			WriteTimeout:      time.Duration(C.BackupTimeout) * time.Minute,
+			IdleTimeout:       idleTimeout,
+			MaxHeaderBytes:    maxHeaderBytes,
 			TLSConfig: &tls.Config{
 				MinVersion: tls.VersionTLS13,
 			},
-	}, nil
+			ErrorLog: log.DebugLogger(),
+		},
+	}
+	srv.SetKeepAlivesEnabled(false)
+
+	return srv, nil
 }
 
 // writeWithContext checks if context is canceled before executing fn.
@@ -123,31 +288,50 @@ func writeWithContext(ctx context.Context, fn func() error) error {
 }
 
 // copyWithContext copies data from src to dst using a 32KB buffer, respecting
-// context cancellation. If the context is canceled, the copy stops and returns
+// context cancellation. Returns bytes written, MD5 hash (hex string), and any error encountered.
-// ctx.Err().
+//
-func copyWithContext(ctx context.Context, dst io.Writer, src io.Reader) error {
+//nolint:gosec
-	buf := make([]byte, 32*1024)
+func copyWithContext(ctx context.Context, dst io.Writer, src io.Reader) (int64, string, error) {
+	buf := make([]byte, copyBufferSize)
+
+	var bytesWritten int64
+
+	hasher := md5.New()
 
 	for {
 		select {
 		case <-ctx.Done():
-			return ctx.Err()
+			return bytesWritten, "", ctx.Err()
 		default:
 		}
 
 		n, readErr := src.Read(buf)
 		if n > 0 {
-			if _, werr := dst.Write(buf[:n]); werr != nil {
+			select {
-				return werr
+			case <-ctx.Done():
+				return bytesWritten, "", ctx.Err()
+			default:
 			}
+
+			if _, werr := hasher.Write(buf[:n]); werr != nil {
+				return bytesWritten, "", werr
+			}
+
+			if _, werr := dst.Write(buf[:n]); werr != nil {
+				return bytesWritten, "", werr
+			}
+
+			bytesWritten += int64(n)
 		}
 
 		if readErr != nil {
 			if readErr == io.EOF {
-				return nil
+				hash := hex.EncodeToString(hasher.Sum(nil))
+
+				return bytesWritten, hash, nil
 			}
 
-			return readErr
+			return bytesWritten, "", readErr
 		}
 	}
 }
@@ -167,4 +351,102 @@ func getClientIP(r *http.Request) string {
 	return host
 }
 
+// getCompressionAlgorithm determines which compression algorithm to use based on the requested extension.
+// If the extension is recognized, it returns the corresponding algorithm; otherwise, it returns the default.
+// For .tar.gz extension, pgzip is always used for parallel compression.
+// The input can be either a full path like "/archive.tar.xz" or just the extension like "tar.xz" or "xz".
+func getCompressionAlgorithm(input string) string {
+	// Extract extension from full path (e.g., "/archive.tar.xz" -> ".xz").
+	// or use the input directly if it's just an extension (e.g., "tar.xz" -> ".xz").
+	ext := path.Ext(input)
+	if ext == "" {
+		ext = "." + input
+	}
+
+	switch ext {
+	case ".tar.bz2":
+		return "bzip2"
+	case ".tar.zstd":
+		return "zstd"
+	case ".tar.zst":
+		return "zstd"
+	case ".tar.lz4":
+		return "lz4"
+	case ".tar.xz":
+		return "xz"
+	case ".tar.gz":
+		return "pgzip"
+	case ".xz":
+		return "xz"
+	case ".bz2":
+		return "bzip2"
+	case ".zst":
+		return "zstd"
+	case ".zstd":
+		return "zstd"
+	case ".lz4":
+		return "lz4"
+	case ".gz":
+		return "pgzip"
+	default:
+		return C.DefaultCompression
+	}
+}
+
+type (
+	// ServerWrapper wraps [http.Server] to intercept and classify errors.
+	ServerWrapper struct {
+		*http.Server
+	}
+
+	// responseWriterWrapper wraps [http.ResponseWriter] to capture status code.
+	responseWriterWrapper struct {
+		http.ResponseWriter
+		statusCode int
+	}
+)
+
+// WriteHeader captures the status code for logging.
+func (rw *responseWriterWrapper) WriteHeader(code int) {
+	rw.statusCode = code
+	rw.ResponseWriter.WriteHeader(code)
+}
+
+// Serve starts the server and classifies any errors returned.
+// TLS-related errors are logged at debug level, other errors at warn level.
+func (sw *ServerWrapper) Serve() error {
+	err := sw.ListenAndServe()
+
+	if err != nil {
+		if isTLSError(err) {
+			log.Debugf("Server error: %v", err)
+		} else {
+			log.Warnf("Server error: %v", err)
+		}
+	}
+
+	return err
+}
+
+// ServeTLS starts the TLS server and classifies any errors returned.
+func (sw *ServerWrapper) ServeTLS(certFile, keyFile string) error {
+	err := sw.ListenAndServeTLS(certFile, keyFile)
+
+	if err != nil {
+		if isTLSError(err) {
+			log.Debugf("Server TLS error: %v", err)
+		} else {
+			log.Warnf("Server TLS error: %v", err)
+		}
+	}
+
+	return err
+}
+
+// timingSafeMatch performs constant-time comparison to prevent timing attacks.
+// Returns 1 if the strings are equal, 0 otherwise.
+func timingSafeMatch(a, b string) bool {
+	return subtle.ConstantTimeCompare([]byte(a), []byte(b)) == 1
+}
+
 /* vim: setlocal ft=go noet ai ts=4 sw=4 sts=4: */

internal/backer/newServer_test.go

@@ -4,6 +4,8 @@ import (
 	"archive/tar"
 	"compress/gzip"
 	"context"
+	"crypto/tls"
+	"errors"
 	"io"
 	"net/http"
 	"net/http/httptest"
@@ -11,6 +13,163 @@ import (
 	"testing"
 )
 
+// TestIsTLSError tests the isTLSError function.
+func TestIsTLSError(t *testing.T) {
+	tests := []struct {
+		name     string
+		err      error
+		expected bool
+	}{
+		{
+			name:     "nil error",
+			err:      nil,
+			expected: false,
+		},
+		{
+			name:     "regular error",
+			err:      errors.New("some random error"),
+			expected: false,
+		},
+		{
+			name:     "TLS prefix error",
+			err:      errors.New("tls: handshake failure"),
+			expected: true,
+		},
+		{
+			name:     "TLS AlertError",
+			err:      tls.AlertError(80),
+			expected: true,
+		},
+		{
+			name:     "TLS CertificateVerificationError",
+			err:      &tls.CertificateVerificationError{},
+			expected: true,
+		},
+		{
+			name:     "TLS ECHRejectionError",
+			err:      &tls.ECHRejectionError{RetryConfigList: []byte("rejected")},
+			expected: true,
+		},
+		{
+			name:     "TLS RecordHeaderError",
+			err:      &tls.RecordHeaderError{},
+			expected: true,
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			result := isTLSError(tt.err)
+			if result != tt.expected {
+				t.Errorf("isTLSError(%v) = %v, expected %v", tt.err, result, tt.expected)
+			}
+		})
+	}
+}
+
+// TestGetClientIP tests the getClientIP function.
+func TestGetClientIP(t *testing.T) {
+	tests := []struct {
+		name       string
+		xffHeader  string
+		remoteAddr string
+		expectedIP string
+	}{
+		{
+			name:       "no X-Forwarded-For",
+			xffHeader:  "",
+			remoteAddr: "192.168.1.100:12345",
+			expectedIP: "192.168.1.100",
+		},
+		{
+			name:       "single IP in X-Forwarded-For",
+			xffHeader:  "10.0.0.1",
+			remoteAddr: "192.168.1.100:12345",
+			expectedIP: "10.0.0.1",
+		},
+		{
+			name:       "multiple IPs in X-Forwarded-For",
+			xffHeader:  "10.0.0.1, 192.168.1.100, 172.16.0.1",
+			remoteAddr: "192.168.1.100:12345",
+			expectedIP: "10.0.0.1",
+		},
+		{
+			name:       "X-Forwarded-For with spaces",
+			xffHeader:  "  10.0.0.1  ,  192.168.1.100  ",
+			remoteAddr: "192.168.1.100:12345",
+			expectedIP: "10.0.0.1",
+		},
+		{
+			name:       "invalid RemoteAddr",
+			xffHeader:  "",
+			remoteAddr: "invalid",
+			expectedIP: "invalid",
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			req := httptest.NewRequest("GET", "/archive", nil)
+			if tt.xffHeader != "" {
+				req.Header.Set("X-Forwarded-For", tt.xffHeader)
+			}
+			req.RemoteAddr = tt.remoteAddr
+
+			result := getClientIP(req)
+			if result != tt.expectedIP {
+				t.Errorf("getClientIP() = %q, expected %q", result, tt.expectedIP)
+			}
+		})
+	}
+}
+
+// TestGetCompressionAlgorithm tests the getCompressionAlgorithm function.
+func TestGetCompressionAlgorithm(t *testing.T) {
+	// Save original config
+	original := C
+	defer func() { C = original }()
+
+	tests := []struct {
+		name              string
+		ext               string
+		expected          string
+		defaultCompression string
+	}{
+		{name: "tar.bz2", ext: "tar.bz2", expected: "bzip2"},
+		{name: "tar.zstd", ext: "tar.zstd", expected: "zstd"},
+		{name: "tar.zst", ext: "tar.zst", expected: "zstd"},
+		{name: "tar.lz4", ext: "tar.lz4", expected: "lz4"},
+		{name: "tar.xz", ext: "tar.xz", expected: "xz"},
+		{name: "tar.gz", ext: "tar.gz", expected: "pgzip"},
+		{name: "bz2", ext: "bz2", expected: "bzip2"},
+		{name: "zst", ext: "zst", expected: "zstd"},
+		{name: "lz4", ext: "lz4", expected: "lz4"},
+		{name: "xz", ext: "xz", expected: "xz"},
+		{name: "gz", ext: "gz", expected: "pgzip"},
+		{name: "zstd", ext: "zstd", expected: "zstd"},
+		{name: "full path /archive.tar.gz", ext: "/archive.tar.gz", expected: "pgzip"},
+		{name: "full path /archive.tar.xz", ext: "/archive.tar.xz", expected: "xz"},
+		{name: "full path /archive", ext: "/archive", expected: "gzip"},
+		{name: "empty uses default", ext: "", expected: "gzip"},
+		{name: "unknown uses default", ext: "unknown", expected: "gzip"},
+		{name: "unknown with default zstd", ext: "unknown", expected: "zstd", defaultCompression: "zstd"},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			if tt.defaultCompression != "" {
+				C = Config{DefaultCompression: tt.defaultCompression}
+			} else {
+				C = Config{DefaultCompression: "gzip"}
+			}
+			result := getCompressionAlgorithm(tt.ext)
+			if result != tt.expected {
+				t.Errorf("getCompressionAlgorithm(%q) = %q, expected %q", tt.ext, result, tt.expected)
+			}
+		})
+	}
+}
+
 // TestNewServerSuccess tests successful server creation.
 func TestNewServerSuccess(t *testing.T) {
 	C = Config{} // Reset global config.
@@ -29,6 +188,46 @@ func TestNewServerSuccess(t *testing.T) {
 	}
 }
 
+// TestWithServerHeader tests that the Server header is added to responses.
+func TestWithServerHeader(t *testing.T) {
+	handler := withServerHeader(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		w.WriteHeader(http.StatusOK)
+	}))
+
+	rec := &responseWriterWrapper{
+		ResponseWriter: &testResponseWriter{},
+		statusCode:     0,
+	}
+
+	req, _ := http.NewRequest("GET", "/test", nil)
+	handler.ServeHTTP(rec, req)
+
+	if rec.Header().Get("Server") != "backer" {
+		t.Error("Expected Server header to be set to 'backer'")
+	}
+}
+
+// testResponseWriter implements http.ResponseWriter for testing.
+type testResponseWriter struct {
+	header http.Header
+	status int
+}
+
+func (tw *testResponseWriter) Header() http.Header {
+	if tw.header == nil {
+		tw.header = make(http.Header)
+	}
+	return tw.header
+}
+
+func (tw *testResponseWriter) Write(data []byte) (int, error) {
+	return len(data), nil
+}
+
+func (tw *testResponseWriter) WriteHeader(status int) {
+	tw.status = status
+}
+
 // TestNewServerInvalidConfig tests server creation with invalid config.
 func TestNewServerInvalidConfig(t *testing.T) {
 	C = Config{} // Reset global config.
@@ -189,7 +388,7 @@ func TestHandlerArchiveStream(t *testing.T) {
 		}
 
 		ctx := r.Context()
-		files, err := GetFilesFromDirectories(C.Directories)
+		files, err := GetFilesFromDirectories(ctx, C.Directories)
 		if err != nil {
 			http.Error(w, "Failed to get files", http.StatusInternalServerError)
 			return
@@ -301,7 +500,7 @@ func TestCopyWithContextSuccess(t *testing.T) {
 	src := strings.NewReader("hello world")
 	dst := &strings.Builder{}
 
-	err := copyWithContext(ctx, dst, src)
+	_, _, err := copyWithContext(ctx, dst, src)
 	if err != nil {
 		t.Errorf("Expected no error, got %v", err)
 	}
@@ -311,6 +510,25 @@ func TestCopyWithContextSuccess(t *testing.T) {
 	}
 }
 
+// TestCopyWithContextMD5Hash tests that MD5 hash is correctly computed.
+func TestCopyWithContextMD5Hash(t *testing.T) {
+	ctx := context.Background()
+	data := "hello world"
+	src := strings.NewReader(data)
+	dst := &strings.Builder{}
+
+	_, md5Hash, err := copyWithContext(ctx, dst, src)
+	if err != nil {
+		t.Errorf("Expected no error, got %v", err)
+	}
+
+	// MD5 of "hello world" is 5eb63bbbe01eeed093cb22bb8f5acdc3
+	expectedMD5 := "5eb63bbbe01eeed093cb22bb8f5acdc3"
+	if md5Hash != expectedMD5 {
+		t.Errorf("Expected MD5 %s, got %s", expectedMD5, md5Hash)
+	}
+}
+
 // TestCopyWithContextCancellation tests copy with cancelled context.
 func TestCopyWithContextCancellation(t *testing.T) {
 	ctx, cancel := context.WithCancel(context.Background())
@@ -325,17 +543,142 @@ func TestCopyWithContextCancellation(t *testing.T) {
 
 	// Start copy in goroutine
 	errCh := make(chan error, 1)
+	bytesCh := make(chan int64, 1)
 	go func() {
-		errCh <- copyWithContext(ctx, dst, src)
+		n, _, err := copyWithContext(ctx, dst, src)
+		bytesCh <- n
+		errCh <- err
 	}()
 
 	// Cancel context immediately
 	cancel()
 
 	err := <-errCh
+	<-bytesCh
 	if err != context.Canceled {
 		t.Errorf("Expected context.Canceled, got %v", err)
 	}
 }
 
+// TestStatusLogging404 tests that 404 responses are logged.
+func TestStatusLogging404(t *testing.T) {
+	C = Config{} // Reset global config.
+	excludePatterns = nil
+
+	// Create a minimal mux that returns 404 for all paths
+	testMux := http.NewServeMux()
+	testMux.HandleFunc("/archive", func(w http.ResponseWriter, r *http.Request) {
+		w.WriteHeader(http.StatusOK)
+	})
+
+	wrappedMux := withStatusLogging(testMux)
+
+	// Use a ResponseRecorder to capture the response
+	rr := httptest.NewRecorder()
+	req, _ := http.NewRequest("GET", "/nonexistent", nil)
+	req.RemoteAddr = "127.0.0.1:12345"
+
+	wrappedMux.ServeHTTP(rr, req)
+
+	if rr.Code != http.StatusNotFound {
+		t.Errorf("Expected status 404, got %d", rr.Code)
+	}
+}
+
+// TestStatusLogging405 tests that 405 responses are logged.
+func TestStatusLogging405(t *testing.T) {
+	C = Config{} // Reset global config.
+	excludePatterns = nil
+
+	// Create a minimal mux that returns 405 for wrong method
+	testMux := http.NewServeMux()
+	testMux.HandleFunc("/archive", func(w http.ResponseWriter, r *http.Request) {
+		w.WriteHeader(http.StatusMethodNotAllowed)
+	})
+
+	wrappedMux := withStatusLogging(testMux)
+
+	rr := httptest.NewRecorder()
+	req, _ := http.NewRequest("GET", "/archive", nil)
+	req.RemoteAddr = "127.0.0.1:12345"
+
+	wrappedMux.ServeHTTP(rr, req)
+
+	if rr.Code != http.StatusMethodNotAllowed {
+		t.Errorf("Expected status 405, got %d", rr.Code)
+	}
+}
+
+// TestMethodNotAllowedHandler tests that non-GET methods return 405.
+func TestMethodNotAllowedHandler(t *testing.T) {
+	C = Config{
+		Address:     "0.0.0.0",
+		Port:        8086,
+		Location:    "/archive",
+		User:        "testuser",
+		Password:    "testpass",
+		Directories: []string{"../../test_data/test1/foo"},
+	}
+	excludePatterns = nil
+
+	// Create test server
+	server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		username, password, ok := r.BasicAuth()
+		if !ok || username != C.User || password != C.Password {
+			http.Error(w, "Unauthorized", http.StatusUnauthorized)
+
+			return
+		}
+
+		if r.Method != http.MethodGet {
+			w.Header().Set("Server", "backer")
+			http.Error(w, "Method Not Allowed", http.StatusMethodNotAllowed)
+
+			return
+		}
+
+		w.WriteHeader(http.StatusOK)
+	}))
+	defer server.Close()
+
+	// Test POST method
+	req, _ := http.NewRequest("POST", server.URL+"/archive", nil)
+	req.SetBasicAuth("testuser", "testpass")
+	resp, err := http.DefaultClient.Do(req)
+	if err != nil {
+		t.Fatalf("Request failed: %v", err)
+	}
+	defer resp.Body.Close()
+
+	if resp.StatusCode != http.StatusMethodNotAllowed {
+		t.Errorf("Expected status 405, got %d", resp.StatusCode)
+	}
+}
+
+// TestTimingSafeMatch tests the timingSafeMatch function for constant-time comparison.
+func TestTimingSafeMatch(t *testing.T) {
+	tests := []struct {
+		name     string
+		a        string
+		b        string
+		expected bool
+	}{
+		{"empty strings", "", "", true},
+		{"identical strings", "test", "test", true},
+		{"different strings", "test", "Test", false},
+		{"different length", "test", "testing", false},
+		{"first empty", "", "test", false},
+		{"second empty", "test", "", false},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			result := timingSafeMatch(tt.a, tt.b)
+			if result != tt.expected {
+				t.Errorf("timingSafeMatch(%q, %q) = %v, expected %v", tt.a, tt.b, result, tt.expected)
+			}
+		})
+	}
+}
+
 /* vim: setlocal ft=go noet ai ts=4 sw=4 sts=4: */

internal/backer/types.go

@@ -1,7 +1,7 @@
 package backer
 
-type (
 // Config main config struct.
+type (
 	Config struct {
 		Address              string   `json:"address"`
 		Port                 int      `json:"port"`
@@ -15,9 +15,23 @@ type (
 		LogLevel             string   `json:"loglevel"`
 		Directories          []string `json:"directories"`
 		BackupTimeout        int      `json:"backup_timeout"`        // Timeout in minutes for backup streaming.
-		CompressionLevel int      `json:"compression_level"` // Gzip compression level (1-9, default 9).
+		DirScanTimeout       int      `json:"dir_scan_timeout"`      // Timeout in minutes for directory scanning.
+		CompressionLevel     int      `json:"compression_level"`     // Compression level (1-9, default 9).
 		ExcludePatterns      []string `json:"exclude_patterns"`      // Regex patterns to exclude from backup.
+		FilenamePrefix       string   `json:"filename_prefix"`       // Prefix for backup filename (default: "backup").
+		CompressionAlgorithm string   `json:"compression_algorithm"` // Compression algorithm alias (deprecated: use default_compression).
+		DefaultCompression   string   `json:"default_compression"`   // Default compression algorithm: gzip, pgzip, bzip2, zstd, lz4 or xz.
+	}
+
+	// ErrDirectoryScanTimeout is returned when directory scan times out.
+	ErrDirectoryScanTimeout struct {
+		ScannedFiles int
 	}
 )
 
+// Error returns the error message for ErrDirectoryScanTimeout.
+func (e ErrDirectoryScanTimeout) Error() string {
+	return "Directory scan timed out"
+}
+
 /* vim: setlocal ft=go noet ai ts=4 sw=4 sts=4: */

internal/backer/types_test.go

@@ -0,0 +1,17 @@
+package backer
+
+import (
+	"testing"
+)
+
+func TestErrDirectoryScanTimeout(t *testing.T) {
+	err := ErrDirectoryScanTimeout{ScannedFiles: 42}
+
+	if err.Error() != "Directory scan timed out" {
+		t.Errorf("Expected error message 'Directory scan timed out', got %q", err.Error())
+	}
+
+	if err.ScannedFiles != 42 {
+		t.Errorf("Expected ScannedFiles=42, got %d", err.ScannedFiles)
+	}
+}

internal/backer/utils.go

@@ -1,176 +0,0 @@
-package backer
-
-import (
-	"archive/tar"
-	"compress/gzip"
-	"context"
-	"fmt"
-	"io"
-	"os"
-	"path/filepath"
-	"strings"
-
-	"backer/internal/log"
-)
-
-// CreateTarGzStream takes a list of files and returns a reader that reads from a tar archive containing these files.
-func CreateTarGzStream(ctx context.Context, filepaths []string) io.ReadCloser {
-	pr, pw := io.Pipe()
-
-	go func() {
-		var pipeErr error // Track if we close pipe with error.
-
-		// Helper to close pipe with error, only once.
-		closePipeWithError := func(err error) {
-			if pipeErr != nil {
-				return // Already closed with error.
-			}
-
-			pipeErr = err
-			pw.CloseWithError(err)
-		}
-
-		defer func() {
-			// Only do normal close if we haven't closed with error.
-			if pipeErr == nil {
-				pw.Close()
-			}
-		}()
-
-		gw, err := gzip.NewWriterLevel(pw, C.CompressionLevel)
-		if err != nil {
-			closePipeWithError(fmt.Errorf("gzip writer error: %w", err))
-
-			return
-		}
-
-		defer func() {
-			// Only close gzip writer if pipe is still open.
-			if pipeErr == nil {
-				if err := gw.Close(); err != nil {
-					// Ignore pipe closed errors - reader may have closed early.
-					if !isPipeClosedError(err) {
-						log.Errorf("Failed to close gzip writer: %v", err)
-					}
-				}
-			}
-		}()
-
-		tw := tar.NewWriter(gw)
-
-		defer func() {
-			// Only close tar writer if pipe is still open.
-			if pipeErr == nil {
-				if err := tw.Close(); err != nil {
-					// Ignore pipe closed errors - reader may have closed early.
-					if !isPipeClosedError(err) {
-						log.Errorf("Failed to close tar writer: %v", err)
-					}
-				}
-			}
-		}()
-
-		for _, fpath := range filepaths {
-			var (
-				err        error
-				linkTarget string
-			)
-
-			// Check for cancellation early.
-			select {
-			case <-ctx.Done():
-				closePipeWithError(ctx.Err())
-
-				return
-
-			default:
-			}
-
-			log.Debugf("Adding file: %s", fpath)
-
-			st, err := os.Lstat(fpath)
-
-			if err != nil {
-				log.Warnf("Skipping %s: %v", fpath, err)
-
-				continue
-			}
-
-			if st.Mode()&os.ModeSymlink != 0 {
-				linkTarget, err = os.Readlink(fpath)
-
-				if err != nil {
-					log.Warnf("Skipping %s: readlink failed: %v", fpath, err)
-
-					continue
-				}
-			}
-
-			// Skip sockets explicitly (they cannot be backed up).
-			if st.Mode()&os.ModeSocket != 0 {
-				log.Debugf("Skipping socket: %s", fpath)
-
-				continue
-			}
-
-			header, err := tar.FileInfoHeader(st, linkTarget)
-
-			if err != nil {
-				log.Warnf("Skipping %s: tar header failed: %v", fpath, err)
-
-				continue
-			}
-
-			header.Format = tar.FormatGNU
-			header.Name = strings.TrimPrefix(filepath.Clean(fpath), string(filepath.Separator))
-
-			mode := st.Mode()
-
-			// Write header with cancellation awareness.
-			if err := writeWithContext(ctx, func() error {
-				return tw.WriteHeader(header)
-			}); err != nil {
-				closePipeWithError(err)
-
-				return
-			}
-
-			// Only stream file contents for regular files.
-			if mode.IsRegular() {
-				f, err := os.Open(fpath)
-				if err != nil {
-					log.Warnf("Skipping %s: open failed: %v", fpath, err)
-
-					continue
-				}
-
-				err = copyWithContext(ctx, tw, f)
-
-				if e := f.Close(); e != nil {
-					log.Warnf("Failed to close file %s: %v", fpath, e)
-				}
-
-				if err != nil {
-					log.Warnf("Skipping %s: copy failed: %v", fpath, err)
-
-					continue
-				}
-			}
-		}
-	}()
-
-	return pr
-}
-
-// isPipeClosedError checks if the error is due to a closed pipe.
-func isPipeClosedError(err error) bool {
-	if err == nil {
-		return false
-	}
-
-	// Check for the specific pipe closed error message.
-	return err.Error() == "io: read/write on closed pipe" ||
-		err.Error() == "io: read on closed pipe"
-}
-
-/* vim: setlocal ft=go noet ai ts=4 sw=4 sts=4: */

internal/backer/utils_test.go

@@ -7,9 +7,20 @@ import (
 	"io"
 	"os"
 	"path/filepath"
+	"runtime"
+	"strings"
+	"syscall"
 	"testing"
+
+	"github.com/klauspost/compress/zstd"
 )
 
+// mkfifo creates a named pipe (FIFO).
+// Uses syscall.Mkfifo which is available on Unix.
+func mkfifo(path string, mode os.FileMode) error {
+	return syscall.Mkfifo(path, uint32(mode))
+}
+
 // TestCreateTarGzStreamBasic tests basic tar.gz creation with a single file.
 func TestCreateTarGzStreamBasic(t *testing.T) {
 	// Create temp file
@@ -63,6 +74,49 @@ func TestCreateTarGzStreamBasic(t *testing.T) {
 	}
 }
 
+// TestCreateTarGzStreamEmptyFile tests creation of archive with empty file.
+func TestCreateTarGzStreamEmptyFile(t *testing.T) {
+	tmpDir := t.TempDir()
+	emptyFile := filepath.Join(tmpDir, "empty.txt")
+	if err := os.WriteFile(emptyFile, []byte{}, 0644); err != nil {
+		t.Fatal(err)
+	}
+
+	ctx := context.Background()
+	reader := CreateTarGzStream(ctx, []string{emptyFile})
+	defer reader.Close()
+
+	gzReader, err := gzip.NewReader(reader)
+	if err != nil {
+		t.Fatalf("Failed to create gzip reader: %v", err)
+	}
+	defer gzReader.Close()
+
+	tarReader := tar.NewReader(gzReader)
+
+	header, err := tarReader.Next()
+	if err != nil {
+		t.Fatalf("Failed to read tar header: %v", err)
+	}
+
+	if header.Size != 0 {
+		t.Errorf("Expected empty file (size 0), got %d", header.Size)
+	}
+
+	if header.Typeflag != tar.TypeReg {
+		t.Errorf("Expected regular file type, got %c", header.Typeflag)
+	}
+
+	data, err := io.ReadAll(tarReader)
+	if err != nil {
+		t.Fatalf("Failed to read content: %v", err)
+	}
+
+	if len(data) != 0 {
+		t.Errorf("Expected empty content, got %d bytes", len(data))
+	}
+}
+
 // TestCreateTarGzStreamMultipleFiles tests with multiple files.
 func TestCreateTarGzStreamMultipleFiles(t *testing.T) {
 	tmpDir := t.TempDir()
@@ -179,6 +233,226 @@ func TestCreateTarGzStreamSymlink(t *testing.T) {
 	}
 }
 
+// TestCreateTarGzStreamBrokenSymlink tests handling of broken symlinks (target doesn't exist).
+func TestCreateTarGzStreamBrokenSymlink(t *testing.T) {
+	tmpDir := t.TempDir()
+	brokenLink := filepath.Join(tmpDir, "broken.txt")
+
+	if err := os.Symlink("/nonexistent/target", brokenLink); err != nil {
+		t.Skip("Symlinks not supported, skipping test")
+	}
+
+	ctx := context.Background()
+	reader := CreateTarGzStream(ctx, []string{brokenLink})
+	defer reader.Close()
+
+	gzReader, err := gzip.NewReader(reader)
+	if err != nil {
+		t.Fatalf("Failed to create gzip reader: %v", err)
+	}
+	defer gzReader.Close()
+
+	tarReader := tar.NewReader(gzReader)
+
+	header, err := tarReader.Next()
+	if err != nil {
+		t.Fatalf("Failed to read tar header: %v", err)
+	}
+
+	if header.Typeflag != tar.TypeSymlink {
+		t.Errorf("Expected symlink type, got %c", header.Typeflag)
+	}
+
+	// Broken symlinks are still stored - just the link target path
+	if header.Linkname != "/nonexistent/target" {
+		t.Errorf("Expected linkname '/nonexistent/target', got '%s'", header.Linkname)
+	}
+}
+
+// TestCreateTarGzStreamSymlinkToDirectory tests symlink pointing to a directory.
+func TestCreateTarGzStreamSymlinkToDirectory(t *testing.T) {
+	tmpDir := t.TempDir()
+	targetDir := filepath.Join(tmpDir, "targetdir")
+	linkToDir := filepath.Join(tmpDir, "linkdir")
+
+	if err := os.Mkdir(targetDir, 0755); err != nil {
+		t.Fatal(err)
+	}
+
+	if err := os.Symlink(targetDir, linkToDir); err != nil {
+		t.Skip("Symlinks not supported, skipping test")
+	}
+
+	ctx := context.Background()
+	reader := CreateTarGzStream(ctx, []string{linkToDir})
+	defer reader.Close()
+
+	gzReader, err := gzip.NewReader(reader)
+	if err != nil {
+		t.Fatalf("Failed to create gzip reader: %v", err)
+	}
+	defer gzReader.Close()
+
+	tarReader := tar.NewReader(gzReader)
+
+	header, err := tarReader.Next()
+	if err != nil {
+		t.Fatalf("Failed to read tar header: %v", err)
+	}
+
+	if header.Typeflag != tar.TypeSymlink {
+		t.Errorf("Expected symlink type, got %c", header.Typeflag)
+	}
+
+	if header.Linkname != targetDir {
+		t.Errorf("Expected linkname '%s', got '%s'", targetDir, header.Linkname)
+	}
+}
+
+// TestCreateTarGzStreamHardLink tests hard link handling - should deduplicate content.
+func TestCreateTarGzStreamHardLink(t *testing.T) {
+	if runtime.GOOS == "windows" {
+		t.Skip("Hard link detection not supported on Windows (inodes unavailable)")
+	}
+
+	tmpDir := t.TempDir()
+
+	// Create original file
+	originalFile := filepath.Join(tmpDir, "original.txt")
+	if err := os.WriteFile(originalFile, []byte("shared content"), 0644); err != nil {
+		t.Fatal(err)
+	}
+
+	// Create hard link
+	hardLinkFile := filepath.Join(tmpDir, "hardlink.txt")
+	if err := os.Link(originalFile, hardLinkFile); err != nil {
+		t.Skip("Hard links not supported, skipping test")
+	}
+
+	ctx := context.Background()
+	reader := CreateTarGzStream(ctx, []string{originalFile, hardLinkFile})
+	defer reader.Close()
+
+	gzReader, err := gzip.NewReader(reader)
+	if err != nil {
+		t.Fatalf("Failed to create gzip reader: %v", err)
+	}
+	defer gzReader.Close()
+
+	tarReader := tar.NewReader(gzReader)
+
+	entries := 0
+	linkCount := 0
+
+	for {
+		header, err := tarReader.Next()
+		if err == io.EOF {
+			break
+		}
+		if err != nil {
+			t.Fatalf("Failed to read tar entry: %v", err)
+		}
+
+		entries++
+		t.Logf("Entry %d: name=%s typeflag=%c linkname=%s", entries, header.Name, header.Typeflag, header.Linkname)
+
+		if header.Typeflag == tar.TypeLink {
+			linkCount++
+			// Verify it's a link to the original - code strips leading '/' to avoid tar warnings
+			expectedLinkname := strings.TrimPrefix(filepath.ToSlash(filepath.Clean(originalFile)), "/")
+			if header.Linkname != expectedLinkname {
+				t.Errorf("Expected linkname '%s', got '%s'", expectedLinkname, header.Linkname)
+			}
+		}
+	}
+
+	// Should have 2 entries: one file content, one as hard link
+	if entries != 2 {
+		t.Errorf("Expected 2 entries, got %d", entries)
+	}
+
+	// One should be a hard link (TypeLink)
+	if linkCount != 1 {
+		t.Errorf("Expected 1 hard link entry, got %d", linkCount)
+	}
+}
+
+// TestCreateTarGzStreamHardLinkFromTestData tests hard link handling with test data.
+func TestCreateTarGzStreamHardLinkFromTestData(t *testing.T) {
+	if runtime.GOOS == "windows" {
+		t.Skip("Hard link detection not supported on Windows (inodes unavailable)")
+	}
+
+	testDataPath := filepath.Join("../../test_data", "test1", "hardlinks")
+	filepaths, err := GetFilesFromDirectories(context.Background(), []string{testDataPath})
+	if err != nil {
+		t.Fatalf("Failed to get files from directory: %v", err)
+	}
+
+	ctx := context.Background()
+	reader := CreateTarGzStream(ctx, filepaths)
+	defer reader.Close()
+
+	gzReader, err := gzip.NewReader(reader)
+	if err != nil {
+		t.Fatalf("Failed to create gzip reader: %v", err)
+	}
+	defer gzReader.Close()
+
+	tarReader := tar.NewReader(gzReader)
+
+	type entryInfo struct {
+		name      string
+		typeflag byte
+		linkname string
+	}
+	var entries []entryInfo
+
+	for {
+		header, err := tarReader.Next()
+		if err == io.EOF {
+			break
+		}
+		if err != nil {
+			t.Fatalf("Failed to read tar entry: %v", err)
+		}
+
+		entries = append(entries, entryInfo{
+			name:      header.Name,
+			typeflag:  header.Typeflag,
+			linkname:  header.Linkname,
+		})
+		t.Logf("Entry: name=%s typeflag=%c linkname=%s", header.Name, header.Typeflag, header.Linkname)
+	}
+
+	// Check we have the expected number of entries
+	if len(entries) != 4 {
+		t.Errorf("Expected 4 entries (dir + 1 original + 2 hardlinks), got %d", len(entries))
+	}
+
+	// Find the original file (typeflag = TypeReg = '0') and verify hardlinks point to it
+	var originalName string
+	var hardlinkCount int
+
+	for _, e := range entries {
+		if e.typeflag == tar.TypeReg && e.linkname == "" {
+			originalName = filepath.Base(e.name)
+		}
+		if e.typeflag == tar.TypeLink {
+			hardlinkCount++
+			// Verify link points to the original file's base name
+			linkBase := filepath.Base(e.linkname)
+			if linkBase != originalName && linkBase != "original.txt" {
+				t.Errorf("Hard link %s points to %s, expected %s", filepath.Base(e.name), linkBase, originalName)
+			}
+		}
+	}
+
+	if hardlinkCount != 2 {
+		t.Errorf("Expected 2 hard link entries, got %d", hardlinkCount)
+	}
+}
+
 // TestCreateTarGzStreamCancellation tests context cancellation.
 func TestCreateTarGzStreamCancellation(t *testing.T) {
 	tmpDir := t.TempDir()
@@ -250,6 +524,64 @@ func TestCreateTarGzStreamEmptyList(t *testing.T) {
 	}
 }
 
+// TestCreateTarGzStreamSpecialCharacters tests files with special characters in names.
+func TestCreateTarGzStreamSpecialCharacters(t *testing.T) {
+	tmpDir := t.TempDir()
+
+	testCases := []struct {
+		name    string
+		expect string
+	}{
+		{"file with spaces.txt", "file with spaces.txt"},
+		{"file-with-dashes.txt", "file-with-dashes.txt"},
+		{"file_with_underscores.txt", "file_with_underscores.txt"},
+		{"file.multiple.dots.txt", "file.multiple.dots.txt"},
+	}
+
+	for _, tc := range testCases {
+		path := filepath.Join(tmpDir, tc.name)
+		if err := os.WriteFile(path, []byte(tc.name), 0644); err != nil {
+			t.Fatal(err)
+		}
+	}
+
+	ctx := context.Background()
+	filepaths := make([]string, len(testCases))
+	for i, tc := range testCases {
+		filepaths[i] = filepath.Join(tmpDir, tc.name)
+	}
+
+	reader := CreateTarGzStream(ctx, filepaths)
+	defer reader.Close()
+
+	gzReader, err := gzip.NewReader(reader)
+	if err != nil {
+		t.Fatalf("Failed to create gzip reader: %v", err)
+	}
+	defer gzReader.Close()
+
+	tarReader := tar.NewReader(gzReader)
+
+	found := make(map[string]bool)
+	for {
+		header, err := tarReader.Next()
+		if err == io.EOF {
+			break
+		}
+		if err != nil {
+			t.Fatalf("Failed to read tar entry: %v", err)
+		}
+
+		found[filepath.Base(header.Name)] = true
+	}
+
+	for _, tc := range testCases {
+		if !found[tc.expect] {
+			t.Errorf("Expected to find '%s' in archive", tc.expect)
+		}
+	}
+}
+
 // TestCreateTarGzStreamNestedFiles tests with nested directory structure.
 func TestCreateTarGzStreamNestedFiles(t *testing.T) {
 	tmpDir := t.TempDir()
@@ -307,7 +639,43 @@ func TestCreateTarGzStreamNestedFiles(t *testing.T) {
 	}
 }
 
-// TestCreateTarGzStreamDeviceFile tests that device files are included in the archive.
+// TestCreateTarGzStreamNamedPipe tests that named pipes (FIFOs) are included in the archive.
+func TestCreateTarGzStreamNamedPipe(t *testing.T) {
+	tmpDir := t.TempDir()
+	namedPipe := filepath.Join(tmpDir, "testpipe")
+
+	if err := mkfifo(namedPipe, 0644); err != nil {
+		t.Skip("Named pipes not supported, skipping test")
+	}
+
+	ctx := context.Background()
+	reader := CreateTarGzStream(ctx, []string{namedPipe})
+	defer reader.Close()
+
+	gzReader, err := gzip.NewReader(reader)
+	if err != nil {
+		t.Fatalf("Failed to create gzip reader: %v", err)
+	}
+	defer gzReader.Close()
+
+	tarReader := tar.NewReader(gzReader)
+
+	header, err := tarReader.Next()
+	if err != nil {
+		t.Fatalf("Failed to read tar header: %v", err)
+	}
+
+	if header.Typeflag != tar.TypeFifo {
+		t.Errorf("Expected FIFO type, got %c", header.Typeflag)
+	}
+
+	// Ensure there are no more entries
+	if _, err := tarReader.Next(); err != io.EOF {
+		t.Error("Expected EOF after FIFO entry")
+	}
+}
+
+// TestCreateTarGzStreamDeviceFile tests that device files (char and block) are included in the archive.
 func TestCreateTarGzStreamDeviceFile(t *testing.T) {
 	// Use /dev/null as a test device file (character device).
 	devNull := "/dev/null"
@@ -315,8 +683,15 @@ func TestCreateTarGzStreamDeviceFile(t *testing.T) {
 	if err != nil {
 		t.Skipf("Skipping device test: %v", err)
 	}
-	if st.Mode()&os.ModeCharDevice == 0 {
+
-		t.Skipf("Skipping: %s is not a character device", devNull)
+	var expectedTypeflag byte
+	switch {
+	case st.Mode()&os.ModeCharDevice != 0:
+		expectedTypeflag = tar.TypeChar
+	case st.Mode()&os.ModeDevice != 0:
+		expectedTypeflag = tar.TypeBlock
+	default:
+		t.Skipf("Skipping: %s is not a device file", devNull)
 	}
 
 	ctx := context.Background()
@@ -331,23 +706,13 @@ func TestCreateTarGzStreamDeviceFile(t *testing.T) {
 
 	tarReader := tar.NewReader(gzReader)
 
-	// Read the device entry
 	header, err := tarReader.Next()
 	if err != nil {
 		t.Fatalf("Failed to read tar header: %v", err)
 	}
 
-	if header.Typeflag != tar.TypeChar {
+	if header.Typeflag != expectedTypeflag {
-		t.Errorf("Expected character device type, got %c", header.Typeflag)
+		t.Errorf("Expected device type %c, got %c", expectedTypeflag, header.Typeflag)
-	}
-
-	// Check that the name matches the file path (without leading slash)
-	expectedName := devNull
-	if expectedName[0] == '/' {
-		expectedName = expectedName[1:]
-	}
-	if header.Name != expectedName {
-		t.Errorf("Expected header name %q, got %q", expectedName, header.Name)
 	}
 
 	// Ensure there are no more entries
@@ -356,4 +721,255 @@ func TestCreateTarGzStreamDeviceFile(t *testing.T) {
 	}
 }
 
+// TestCompressionLevelMappingLz4 tests that lz4 compression levels are correctly mapped.
+func TestCompressionLevelMappingLz4(t *testing.T) {
+	tests := []struct {
+		input    int
+		expected int
+	}{
+		{1, 0},
+		{2, 0},
+		{3, 0},
+		{4, 1},
+		{5, 1},
+		{6, 1},
+		{7, 2},
+		{8, 2},
+		{9, 2},
+	}
+
+	for _, tt := range tests {
+		C = Config{CompressionLevel: tt.input}
+		result := mapCompressionLevelToLz4(tt.input)
+		if result != tt.expected {
+			t.Errorf("mapCompressionLevelToLz4(%d) = %d, expected %d", tt.input, result, tt.expected)
+		}
+	}
+}
+
+// TestCompressionLevelMappingXz tests that xz compression is not configurable via level.
+// xz uses default compression - this test verifies the stream works.
+func TestCompressionLevelMappingXz(t *testing.T) {
+	// xz doesn't support configurable compression levels in the same way.
+	// It always uses default compression. This test verifies the stream is created successfully.
+	C = Config{CompressionLevel: 6}
+
+	tmpDir := t.TempDir()
+	testFile := filepath.Join(tmpDir, "test.txt")
+	content := []byte("test content")
+	if err := os.WriteFile(testFile, content, 0644); err != nil {
+		t.Fatal(err)
+	}
+
+	ctx := context.Background()
+	reader := CreateTarXzStream(ctx, []string{testFile})
+	defer reader.Close()
+
+	data, err := io.ReadAll(reader)
+	if err != nil {
+		t.Fatalf("Failed to read xz archive: %v", err)
+	}
+
+	if len(data) == 0 {
+		t.Error("Expected non-empty archive")
+	}
+}
+
+// TestCompressionLevelMappingBzip2 tests that bzip2 compression levels are correctly mapped.
+func TestCompressionLevelMappingBzip2(t *testing.T) {
+	tests := []struct {
+		input    int
+		expected int
+	}{
+		{1, 1},
+		{5, 5},
+		{9, 9},
+	}
+
+	for _, tt := range tests {
+		C = Config{CompressionLevel: tt.input}
+		result := mapCompressionLevelToBzip2(tt.input)
+		if result != tt.expected {
+			t.Errorf("mapCompressionLevelToBzip2(%d) = %d, expected %d", tt.input, result, tt.expected)
+		}
+	}
+}
+
+// TestCreateTarBzip2Stream tests bzip2 compression with configured level.
+func TestCreateTarBzip2Stream(t *testing.T) {
+	C = Config{CompressionLevel: 6}
+
+	tmpDir := t.TempDir()
+	testFile := filepath.Join(tmpDir, "test.txt")
+	content := []byte("test content for bzip2")
+	if err := os.WriteFile(testFile, content, 0644); err != nil {
+		t.Fatal(err)
+	}
+
+	ctx := context.Background()
+	reader := CreateTarBzip2Stream(ctx, []string{testFile})
+	defer reader.Close()
+
+	// Verify we can read the archive
+	_, err := io.ReadAll(reader)
+	if err != nil {
+		t.Fatalf("Failed to read bzip2 archive: %v", err)
+	}
+}
+
+// TestCreateTarLz4Stream tests lz4 compression with configured level.
+func TestCreateTarLz4Stream(t *testing.T) {
+	C = Config{CompressionLevel: 6}
+
+	tmpDir := t.TempDir()
+	testFile := filepath.Join(tmpDir, "test.txt")
+	content := []byte("test content for lz4")
+	if err := os.WriteFile(testFile, content, 0644); err != nil {
+		t.Fatal(err)
+	}
+
+	ctx := context.Background()
+	reader := CreateTarLz4Stream(ctx, []string{testFile})
+	defer reader.Close()
+
+	// Verify we can read the archive
+	_, err := io.ReadAll(reader)
+	if err != nil {
+		t.Fatalf("Failed to read lz4 archive: %v", err)
+	}
+}
+
+// TestCreateTarXzStream tests xz compression with configured level.
+func TestCreateTarXzStream(t *testing.T) {
+	C = Config{CompressionLevel: 6}
+
+	tmpDir := t.TempDir()
+	testFile := filepath.Join(tmpDir, "test.txt")
+	content := []byte("test content for xz")
+	if err := os.WriteFile(testFile, content, 0644); err != nil {
+		t.Fatal(err)
+	}
+
+	ctx := context.Background()
+	reader := CreateTarXzStream(ctx, []string{testFile})
+	defer reader.Close()
+
+	// Verify we can read the archive
+	_, err := io.ReadAll(reader)
+	if err != nil {
+		t.Fatalf("Failed to read xz archive: %v", err)
+	}
+}
+
+// TestCreateTarPgzipStream tests pgzip (parallel gzip) compression.
+func TestCreateTarPgzipStream(t *testing.T) {
+	C = Config{CompressionLevel: 6}
+
+	tmpDir := t.TempDir()
+	testFile := filepath.Join(tmpDir, "test.txt")
+	content := []byte("test content for pgzip")
+	if err := os.WriteFile(testFile, content, 0644); err != nil {
+		t.Fatal(err)
+	}
+
+	ctx := context.Background()
+	reader := CreateTarPgzipStream(ctx, []string{testFile})
+	defer reader.Close()
+
+	gzReader, err := gzip.NewReader(reader)
+	if err != nil {
+		t.Fatalf("Failed to create gzip reader: %v", err)
+	}
+	defer gzReader.Close()
+
+	tarReader := tar.NewReader(gzReader)
+
+	header, err := tarReader.Next()
+	if err != nil {
+		t.Fatalf("Failed to read tar header: %v", err)
+	}
+
+	if header.Name != testFile && filepath.Base(header.Name) != "test.txt" {
+		t.Errorf("Expected file name 'test.txt', got '%s'", header.Name)
+	}
+
+	data, err := io.ReadAll(tarReader)
+	if err != nil {
+		t.Fatalf("Failed to read file content: %v", err)
+	}
+
+	if string(data) != string(content) {
+		t.Errorf("Expected content '%s', got '%s'", content, data)
+	}
+}
+
+// TestCreateTarZstdStream tests zstd compression.
+func TestCreateTarZstdStream(t *testing.T) {
+	C = Config{CompressionLevel: 6}
+
+	tmpDir := t.TempDir()
+	testFile := filepath.Join(tmpDir, "test.txt")
+	content := []byte("test content for zstd")
+	if err := os.WriteFile(testFile, content, 0644); err != nil {
+		t.Fatal(err)
+	}
+
+	ctx := context.Background()
+	reader := CreateTarZstdStream(ctx, []string{testFile})
+	defer reader.Close()
+
+	// zstd.Reader is in github.com/klauspost/compress/zstd
+	zstdReader, err := zstd.NewReader(reader)
+	if err != nil {
+		t.Fatalf("Failed to create zstd reader: %v", err)
+	}
+	defer zstdReader.Close()
+
+	tarReader := tar.NewReader(zstdReader)
+
+	header, err := tarReader.Next()
+	if err != nil {
+		t.Fatalf("Failed to read tar header: %v", err)
+	}
+
+	if header.Name != testFile && filepath.Base(header.Name) != "test.txt" {
+		t.Errorf("Expected file name 'test.txt', got '%s'", header.Name)
+	}
+
+	data, err := io.ReadAll(tarReader)
+	if err != nil {
+		t.Fatalf("Failed to read file content: %v", err)
+	}
+
+	if string(data) != string(content) {
+		t.Errorf("Expected content '%s', got '%s'", content, data)
+	}
+}
+
+// TestZstdLevel tests zstd level mapping.
+func TestZstdLevel(t *testing.T) {
+	tests := []struct {
+		input    int
+		expected zstd.EncoderLevel
+	}{
+		{1, zstd.SpeedFastest},
+		{2, zstd.SpeedFastest},
+		{3, zstd.SpeedFastest},
+		{4, zstd.SpeedDefault},
+		{5, zstd.SpeedDefault},
+		{6, zstd.SpeedDefault},
+		{7, zstd.SpeedBetterCompression},
+		{8, zstd.SpeedBetterCompression},
+		{9, zstd.SpeedBetterCompression},
+	}
+
+	for _, tt := range tests {
+		C = Config{CompressionLevel: tt.input}
+		result := zstdLevel(tt.input)
+		if result != tt.expected {
+			t.Errorf("zstdLevel(%d) = %v, expected %v", tt.input, result, tt.expected)
+		}
+	}
+}
+
 /* vim: setlocal ft=go noet ai ts=4 sw=4 sts=4: */

internal/backer/xz.go

@@ -0,0 +1,19 @@
+package backer
+
+import (
+	"context"
+	"io"
+
+	"github.com/ulikunitz/xz"
+)
+
+// CreateTarXzStream takes a list of files and returns a reader that reads from an xz-compressed tar archive.
+func CreateTarXzStream(ctx context.Context, filepaths []string) io.ReadCloser {
+	return createArchiveStream(ctx, filepaths, func(w io.Writer) (io.WriteCloser, error) {
+		// xz doesn't have a simple level parameter; use default compression.
+		// The compression level affects block size indirectly through the algorithm.
+		return xz.NewWriter(w)
+	})
+}
+
+/* vim: setlocal ft=go noet ai ts=4 sw=4 sts=4: */

internal/backer/zstd.go

@@ -0,0 +1,29 @@
+package backer
+
+import (
+	"context"
+	"io"
+
+	"github.com/klauspost/compress/zstd"
+)
+
+// CreateTarZstdStream takes a list of files and returns a reader that reads from a zstd-compressed tar archive.
+func CreateTarZstdStream(ctx context.Context, filepaths []string) io.ReadCloser {
+	return createArchiveStream(ctx, filepaths, func(w io.Writer) (io.WriteCloser, error) {
+		return zstd.NewWriter(w, zstd.WithEncoderLevel(zstdLevel(C.CompressionLevel)))
+	})
+}
+
+// zstdLevel returns the zstd encoder level based on the 1-9 compression level.
+func zstdLevel(level int) zstd.EncoderLevel {
+	switch {
+	case level <= 3:
+		return zstd.SpeedFastest
+	case level <= 6:
+		return zstd.SpeedDefault
+	default:
+		return zstd.SpeedBetterCompression
+	}
+}
+
+/* vim: setlocal ft=go noet ai ts=4 sw=4 sts=4: */

internal/log/main.go

@@ -3,25 +3,52 @@ package log
 
 import (
 	"fmt"
+	"log"
 	"log/slog"
 	"os"
+	"strings"
+	"sync"
 	"time"
 )
 
 var (
 	// Log contains *[os.File] pointing to log file descriptor.
+	// Protected by mu for thread-safe access, primarily to enable parallel tests.
+	// In production, this is set once at startup and never modified thereafter.
 	Log *os.File
+
+	// mu protects Log during initialization and when DebugLogger is called.
+	// This is necessary to allow parallel test execution where each test
+	// may initialize the logger with a different log file.
+	mu sync.RWMutex
+
+	// logLevel stores the configured log level for use by DebugLogger.
+	logLevel slog.Level
+)
+
+type (
+	// slogWriter implements [io.Writer] to redirect standard log to slog.
+	slogWriter struct{}
 )
 
 // Init sets up logger stuff.
 // level can be error, warn, info, debug; if something other is supplied, info level is selected.
+// Thread-safe: uses mutex to allow parallel test execution.
 func Init(level, filename string) error {
+	mu.Lock()
+	defer mu.Unlock()
+
 	var (
 		err      error
 		loglevel slog.Level
 	)
 
 	if filename != "" {
+		// Close existing log file if re-initializing (e.g., in tests).
+		if Log != nil && Log != os.Stderr {
+			Log.Close()
+		}
+
 		Log, err = os.OpenFile(filename, os.O_WRONLY|os.O_APPEND|os.O_CREATE, 0644)
 
 		if err != nil {
@@ -49,6 +76,8 @@ func Init(level, filename string) error {
 		loglevel = slog.LevelInfo
 	}
 
+	logLevel = loglevel
+
 	opts := &slog.HandlerOptions{
 		// Use the ReplaceAttr function on the handler options
 		// to be able to replace any single attribute in the log output.
@@ -122,6 +151,46 @@ func Debugf(format string, a ...any) {
 	slog.Debug(fmt.Sprintf(format, a...))
 }
 
+// DebugLogger returns a standard library logger that writes at debug/warn level.
+// TLS-related errors are logged at debug level, others at warn level.
+// Both are filtered by the configured log level: set loglevel to debug to see all messages.
+func DebugLogger() *log.Logger {
+	mu.RLock()
+	defer mu.RUnlock()
+
+	opts := &slog.HandlerOptions{
+		Level: logLevel,
+	}
+
+	handler := slog.NewTextHandler(Log, opts)
+
+	_ = handler // satisfy linter (handler configures level for potential future use).
+
+	return log.New(&slogWriter{}, "", 0)
+}
+
+// Write implements [io.Writer]. It differentiates TLS/SSL errors from other errors.
+// TLS-related errors are logged at debug level, while other errors (handler issues,
+// unexpected behavior) are logged at warn level.
+func (w *slogWriter) Write(p []byte) (int, error) {
+	msg := string(p)
+
+	isTLSError := strings.Contains(msg, "tls") ||
+		strings.Contains(msg, "ssl") ||
+		strings.Contains(msg, "TLS") ||
+		strings.Contains(msg, "SSL") ||
+		strings.Contains(msg, "handshake") ||
+		strings.Contains(msg, "certificate")
+
+	if isTLSError {
+		Debug(msg)
+	} else {
+		Warn(msg)
+	}
+
+	return len(p), nil
+}
+
 // Fatal logs args at error level and quits with status code 1.
 func Fatal(args ...any) {
 	slog.Error(fmt.Sprint(args...))
@@ -138,7 +207,11 @@ func Fatalf(format string, a ...any) {
 }
 
 // Close closes log file descriptor.
+// Thread-safe: uses mutex to coordinate with Init and DebugLogger.
 func Close() {
+	mu.Lock()
+	defer mu.Unlock()
+
 	if Log == nil || Log == os.Stderr {
 		return
 	}
@@ -147,6 +220,8 @@ func Close() {
 		msg := fmt.Sprintln("Failed to close log file:", err)
 		slog.Error(msg)
 	}
+
+	Log = nil
 }
 
 /* vim: setlocal ft=go noet ai ts=4 sw=4 sts=4: */

internal/log/main_test.go

@@ -354,4 +354,116 @@ func TestCloseStderr(t *testing.T) {
 	}
 }
 
+// TestDebugLogger tests that DebugLogger returns a working *log.Logger.
+func TestDebugLogger(t *testing.T) {
+	tmpFile, err := os.CreateTemp(t.TempDir(), "test_log_*.log")
+	if err != nil {
+		t.Fatal(err)
+	}
+	logFilePath := tmpFile.Name()
+	tmpFile.Close()
+
+	if err := Init("debug", logFilePath); err != nil {
+		t.Fatal(err)
+	}
+	defer Close()
+
+	logger := DebugLogger()
+	if logger == nil {
+		t.Error("DebugLogger returned nil")
+	}
+
+	logger.Print("test debug message")
+
+	content, err := os.ReadFile(logFilePath)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	if !strings.Contains(string(content), "test debug message") {
+		t.Error("Expected debug message in log")
+	}
+}
+
+// TestDebugLoggerTLS tests that TLS errors are logged at debug level.
+func TestDebugLoggerTLS(t *testing.T) {
+	tmpFile, err := os.CreateTemp(t.TempDir(), "test_log_*.log")
+	if err != nil {
+		t.Fatal(err)
+	}
+	logFilePath := tmpFile.Name()
+	tmpFile.Close()
+
+	if err := Init("debug", logFilePath); err != nil {
+		t.Fatal(err)
+	}
+	defer Close()
+
+	logger := DebugLogger()
+	logger.Print("tls handshake timeout")
+
+	content, err := os.ReadFile(logFilePath)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	if !strings.Contains(string(content), "tls handshake timeout") {
+		t.Error("Expected TLS message in log")
+	}
+}
+
+// TestDebugLoggerNonTLS tests that non-TLS errors are logged at warn level.
+func TestDebugLoggerNonTLS(t *testing.T) {
+	tmpFile, err := os.CreateTemp(t.TempDir(), "test_log_*.log")
+	if err != nil {
+		t.Fatal(err)
+	}
+	logFilePath := tmpFile.Name()
+	tmpFile.Close()
+
+	if err := Init("warn", logFilePath); err != nil {
+		t.Fatal(err)
+	}
+	defer Close()
+
+	logger := DebugLogger()
+	logger.Print("http: connection reset by peer")
+
+	content, err := os.ReadFile(logFilePath)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	if !strings.Contains(string(content), "http: connection reset by peer") {
+		t.Error("Expected non-TLS message in log")
+	}
+}
+
+// TestDebugLoggerLevelFiltering tests that log level filtering works.
+func TestDebugLoggerLevelFiltering(t *testing.T) {
+	tmpFile, err := os.CreateTemp(t.TempDir(), "test_log_*.log")
+	if err != nil {
+		t.Fatal(err)
+	}
+	logFilePath := tmpFile.Name()
+	tmpFile.Close()
+
+	if err := Init("info", logFilePath); err != nil {
+		t.Fatal(err)
+	}
+	defer Close()
+
+	logger := DebugLogger()
+	logger.Print("tls handshake timeout") // TLS error at debug level - should be filtered
+
+	content, err := os.ReadFile(logFilePath)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	if strings.Contains(string(content), "tls handshake timeout") {
+		t.Error("Debug-level TLS message should be filtered at info log level")
+	}
+}
+
 /* vim: setlocal ft=go noet ai ts=4 sw=4 sts=4: */

llms.txt

@@ -1,7 +1,7 @@
 # llms.txt for Backer
 
 ## Overview
-Backer is a Go-based backup utility that creates tar.gz archives on-the-fly and serves them via HTTP/HTTPS with basic authentication.
+Backer is a Go-based backup utility that creates compressed archives on-the-fly and serves them via HTTP/HTTPS with basic authentication. Supports gzip, pgzip, bzip2, zstd, lz4, xz compression.
 
 ## Quick Start
 ```bash
@@ -26,11 +26,17 @@ curl -u user:pass https://server:8086/archive -o backup.tar.gz
 ├── internal/
 │   ├── backer/              # Core logic
 │   │   ├── types.go         # Config struct definition
+│   │   ├── consts.go        # Named constants for magic numbers
 │   │   ├── globals.go       # Global C Config variable
+│   │   ├── archive.go       # Common archive logic (createArchiveStream, writeFilesToTar)
+│   │   ├── gzip.go          # CreateTarGzStream, CreateTarPgzipStream
+│   │   ├── bzip2.go         # CreateTarBzip2Stream, mapCompressionLevelToBzip2
+│   │   ├── zstd.go          # CreateTarZstdStream, zstdLevel
+│   │   ├── lz4.go           # CreateTarLz4Stream, mapCompressionLevelToLz4
+│   │   ├── xz.go            # CreateTarXzStream
 │   │   ├── loadConfig.go    # HJSON/JSON config parsing with validation
 │   │   ├── newServer.go     # HTTP server, auth handler, context helpers
-│   │   ├── utils.go         # CreateTarGzStream, isPipeClosedError
+│   │   ├── getFilesFromDirectories.go  # Directory walking, exclusion filtering, isExcluded
-│   │   ├── getFilesFromDirectories.go  # Directory walking, exclusion filtering
 │   │   └── *_test.go        # Test files
 │   └── log/main.go          # Logging wrapper around stdlib slog
 ├── contrib/                 # Init scripts
@@ -40,7 +46,6 @@ curl -u user:pass https://server:8086/archive -o backup.tar.gz
 ├── test_data/               # Test fixtures
 ├── data/config_example.json  # Example config
 ├── Makefile                 # Build targets
-├── TODO.org                 # Project backlog
 ├── README.md                # User documentation
 └── LICENSE                  # MIT License
 ```
@@ -50,7 +55,7 @@ curl -u user:pass https://server:8086/archive -o backup.tar.gz
 |--------|---------|-------------|
 | address | 0.0.0.0 | Listen address |
 | port | 8086 | Listen port |
-| cert/key | - | TLS cert/key paths |
+| cert/key | - | TLS cert/key paths (required when nohttps=false) |
 | nohttps | false | Disable HTTPS |
 | location | /archive | API endpoint path |
 | user | - | Basic auth username (required) |
@@ -59,8 +64,23 @@ curl -u user:pass https://server:8086/archive -o backup.tar.gz
 | loglevel | info | Log verbosity (debug/info/warn/error) |
 | directories | - | Directories to backup (required) |
 | backup_timeout | 60 | Stream timeout in minutes (1-1440) |
-| compression_level | 9 | Gzip level (1-9) |
+| dir_scan_timeout | 60 | Directory scan timeout in minutes (1-1440) |
+| compression_level | 9 | Compression level (1-9) |
 | exclude_patterns | [] | Regex patterns to exclude from backup |
+| filename_prefix | backup | Prefix for backup filename in Content-Disposition header |
+| default_compression | gzip | Default compression algorithm (gzip, pgzip, bzip2, zstd, lz4, xz) |
+| compression_algorithm | gzip | Alias for default_compression (deprecated) |
+
+## Compression Level Mapping
+
+The `compression_level` config option (1-9) is mapped differently for each algorithm:
+
+| Algorithm | Level Mapping |
+|-----------|---------------|
+| gzip/pgzip/bzip2 | 1-9 directly maps to compression levels |
+| lz4 | 1-3 → Fastest (0), 4-6 → Default (1), 7-9 → Best (2) |
+| zstd | 1-3 → Fastest, 4-6 → Default, 7-9 → BetterCompression |
+| xz | Uses default compression (no configurable level) |
 
 ## Key Patterns
 
@@ -78,34 +98,86 @@ json.Marshal(tmp)
 json.Unmarshal(buf, &C)
 ```
 
-### Stream Cancellation
+### Context-Aware I/O
-Context-aware write and copy functions:
+Two helper functions in newServer.go handle context cancellation during streaming:
+- `writeWithContext(ctx, fn)` — checks context before calling fn
+- `copyWithContext(ctx, dst, src)` — buffered copy (32KB) with cancellation checks, computes MD5 hash
+
 ```go
 func writeWithContext(ctx context.Context, fn func() error) error
-func copyWithContext(ctx context.Context, dst io.Writer, src io.Reader) error
+func copyWithContext(ctx context.Context, dst io.Writer, src io.Reader) (int64, string, error)
 ```
 
+### HTTP Server Middleware
+Two middleware functions wrap the handler:
+- `withServerHeader` — adds "Server: backer" header to all responses
+- `withStatusLogging` — logs 404 and 405 responses
+
+```go
+Handler: withServerHeader(withStatusLogging(mux))
+```
+
+### Unmatched Route Logging
+The server wraps http.ResponseWriter to capture status codes:
+- **404 Not Found**: `"Not Found from {clientIP} to {path}"`
+- **405 Method Not Allowed**: `"Method Not Allowed from {clientIP} for {method} {path}"`
+
+Uses `responseWriterWrapper` that tracks status code via WriteHeader() method.
+
 ### Pipe Error Handling
 Uses `pipeErr` tracking to avoid noisy errors when reader closes early.
 Skips files that can't be read (logs warning, continues backup).
+- `isPipeClosedError()` checks for "io: read/write on closed pipe" errors
 
 ### Content-Disposition Filename
-Backup filenames include timestamp: `backup-20060102-150405.tar.gz`
+Backup filenames include timestamp and configured prefix: `backup-20060102-150405.tar.gz` (extension depends on compression_algorithm).
 
 ### Special File Types
-- **Device files** (character/block) are included with correct major/minor numbers.
+- **Regular files**: streamed into tar archive with content
-- **Sockets** are skipped (no entry created).
+- **Directories**: included as tar directory entries
-- **Symlinks** are stored as symlinks (target path stored).
+- **Symlinks**: stored as symlink entries (target path preserved) - uses os.Lstat, no cycles possible
-- **Named pipes** and other special files are included (tar header created).
+- **Broken symlinks**: archived with their target path (even if target doesn't exist)
-- **Regular files** are compressed; other file types have no data section.
+- **Device files** (char/block): included with correct major/minor numbers
+- **Named pipes (FIFOs)**: included (header only, no data)
+- **Sockets**: skipped (not archivable)
+- **Hard links**: deduplicated - first occurrence stored with content, subsequent as link entries
+- **GNU tar format**: uses `header.Format = tar.FormatGNU` for GNU tar compatibility
+
+### Test Coverage
+All entity types are tested: regular files, empty files, directories, symlinks, broken symlinks, symlinks to directories, hard links, device files, named pipes (FIFOs), special characters in filenames.
 
 ### Error Recovery
-- `writeWithContext` and `copyWithContext` recover from panics in callbacks and return an error.
+- `writeWithContext` checks context before calling fn.
+- `copyWithContext` does buffered copy with cancellation checks.
+
+### Client IP Detection
+Checks `X-Forwarded-For` header first (for proxied requests), then falls back to `RemoteAddr`.
+
+### TLS Error Handling
+Uses `ServerWrapper` to intercept and classify errors from Serve() and ServeTLS():
+- TLS-specific errors logged at debug level to reduce log clutter
+- Uses errors.As to detect: tls.AlertError, tls.CertificateVerificationError, tls.ECHRejectionError, tls.RecordHeaderError
+
+### HTTP Routing & Compression Selection
+The server uses http.NewServeMux() for routing. Multiple routes support different compression formats:
+```go
+mux.Handle(C.Location, http.HandlerFunc(backupHandler))           // /archive
+mux.Handle(C.Location+".tar.gz", http.HandlerFunc(backupHandler))  // archive.tar.gz
+mux.Handle(C.Location+".tar.xz", http.HandlerFunc(backupHandler))   // archive.tar.xz
+mux.Handle(C.Location+".tar.bz2", http.HandlerFunc(backupHandler)) // archive.tar.bz2
+// ... etc (.tar.lz4, .tar.zst, .tar.zstd)
+```
+
+Compression algorithm determined by path extension using `path.Ext()`:
+- Full path: `/archive.tar.xz` → `.xz` → xz
+- Bare extension: `tar.xz` → `.xz` → xz
+- Default when no extension matches
 
 ### Performance Notes
 - Exclude patterns are precompiled after config load for faster matching.
 - Compression level is configurable (1-9); default 9 uses more CPU.
 - Large directory walks use `filepath.WalkDir` (efficient).
+- Context cancellation checked on each file during directory scan.
 
 ## Coding Style & Conventions
 
@@ -120,7 +192,8 @@ GetFilesFromDirectories()
 // Unexported (private): verbNoun or isNoun pattern
 writeWithContext()
 copyWithContext()
-isPipeClosedError()  // predicate helper
+isPipeClosedError()
+isExcluded()
 defaultConfigPath()
 ```
 
@@ -129,6 +202,7 @@ Struct fields use PascalCase, json tags use snake_case:
 ```go
 BackupTimeout     int `json:"backup_timeout"`
 CompressionLevel  int `json:"compression_level"`
+DirScanTimeout    int `json:"dir_scan_timeout"`
 ```
 
 ### Error Messages
@@ -141,6 +215,7 @@ CompressionLevel  int `json:"compression_level"`
 - Past tense: `"Failed to close gzip writer: %v", err`
 - Context included: `"Skipping %s: %v", filename, err`
 - Appropriate level: Warn for recoverable, Error for issues, Fatal for startup failures
+- Log levels: error, warn, info, debug
 
 ### Config Validation Pattern
 For each config option: set default first, then immediately validate:
@@ -152,13 +227,13 @@ if C.Port == 0 {
 }
 
 if C.Port < 1 || C.Port > 65535 {
-    return fmt.Errorf("Port must be between 1 and 65535, got %d", C.Port)
+    return fmt.Errorf("Config option port must be between 1 and 65535, got %d", C.Port)
 }
 ```
 
 ### Go Formatting
 - Tabs for indentation
-- Vim modeline at EOF: `/* vim: setlocal ft=go noet ai ts=4 sw=4 sts=4: */`
+- Vim modeline at EOF: `/* vim: setlocal ft=go noet ai ts=4 sw=4 sts=4: */
 
 ### Linter Suppressions (intentional)
 | Rule | Reason |
@@ -173,24 +248,38 @@ if C.Port < 1 || C.Port > 65535 {
 
 ## Testing
 ```bash
-go test ./...                    # Run tests
+go test ./...                    # Run tests (runs make clean first)
 go test -v -run TestName ./...   # Run specific test
 golangci-lint run ./...          # Lint
+make test                        # Clean + test (deterministic)
 ```
 
-Coverage: ~69% on internal/backer (main and log packages not covered by design)
+Coverage: ~64% on internal/backer (~100% on internal/log)
+
+### Test Data
+All test data is created by `TestMain` in `internal/backer/main_test.go` before tests run and cleaned up afterward. This ensures tests are independent of pre-existing files.
+
+- `test_data/test1/foo/` — test directory with files, subdirs, empty_dir
+- `test_data/test1/bar/` — test directory with files and a directory named `goodbye.txt/` (edge case: directory with file extension)
+- `test_data/test1/hardlinks/` — test directory with hard-linked files (original.txt, hardlink1.txt, hardlink2.txt)
+- `test_data/test1/symlinks/` — test directory with symbolic links
+- `test_data/test1/empty_dir/` — empty directory for testing
+- `test_data/tmp/` — reserved for test temporary files (use instead of system `/tmp`)
 
 ### Test Conventions
 - Reset global config: `C = Config{}`
+- Reset exclude patterns: `excludePatterns = nil`
 - Use `t.TempDir()` for temporary files
 - Helper function `writeTempConfig()` for config tests
+- Test data created by TestMain setup, cleaned after tests
+- Always check errors with meaningful diagnostics (use `t.Fatalf` for setup errors)
 
 ## Build
 ```bash
-make           # Build binary (version from git or "dev")
+make           # Clean + build binary (version from git or "dev")
 make build VERSION=0.1.0  # Build with specific version
-make clean     # Remove binary
+make clean     # Remove binary, test_data/tmp, test_data/test1
-make test      # Run tests
+make test      # Clean + test (deterministic)
 make upgrade   # Update dependencies
 make help      # Show targets
 ```
@@ -201,7 +290,19 @@ Version is injected at build time via ldflags:
 var version = "dev"  // overridden by: -X main.version=$(VERSION)
 ```
 
-## Known Issues / TODO
+## Request Flow (Good Path)
-See TODO.org for current backlog:
+```
-- Consider different compression algorithms (bz2, xz, lzo, lz4)
+Request → Method Check (GET) → Auth Check → Get Files (with timeout) → Create Archive → Stream → Complete
-- Consider configurable filename prefix for content-disposition
+```
+
+## Request Flow (Error Paths)
+| Request | Response | Logged |
+|---------|----------|--------|
+| Non-GET method | 405 Method Not Allowed | Yes (info) |
+| No auth | 401 Unauthorized | Yes (info) |
+| Wrong credentials | 401 Unauthorized | Yes (info) |
+| Unknown path | 404 Not Found | Yes (info) |
+| Dir scan timeout | 504 Request Timeout | Yes (error) |
+| Failed to get files | 500 Internal Server Error | Yes (error) |
+| Client disconnect | - | Yes (info) |
+| Backup completed | 200 OK | Yes (info) |

test_data/test1/bar/goodbye.txt/text.txt

@@ -1 +0,0 @@
-Fucked up beyond recognotion.

test_data/test1/foo/hello_breakfast.txt

@@ -1,2 +0,0 @@
-My dear Tweety,
-I'd like to greet you as my breakfast.

test_data/test1/foo/mydir/myfile.txt

@@ -1 +0,0 @@
-This is MY file!

test_data/test_config.json

@@ -5,10 +5,10 @@
     /* By default listen on port 8086, like first intel commercially successful cpu model.                            */
     "port": 8086,
 
-    /* Path location of certificate file                                                                              */
+    /* Path location of test certificate file                                                                       */
     "cert": "../../test_data/example.crt",
 
-    /* Path location of certificate key                                                                               */
+    /* Path location of test certificate key                                                                          */
     "key": "../../test_data/example.key",
 
     /* We really want to use https for backup archive tranferring.                                                    */
@@ -24,7 +24,7 @@
     "password": "SharpShooter",
 
     /* Where to store logs.                                                                                           */
-    "log": "/var/log/backer.log",
+    "log": "../../test_data/tmp/backer.log",
 
     /* Verbosity of logs                                                                                              */
     "loglevel": "info",
@@ -39,5 +39,5 @@
     "backup_timeout": 60,
 
     /* Gzip compression level (1-9, where 9 is best compression).                                                    */
-    "compression_level": 6
+    "compression_level": 3
 }

vendor/github.com/dsnet/compress/.travis.yml

@@ -0,0 +1,36 @@
+sudo: false
+language: go
+before_install:
+  - curl -L https://github.com/google/brotli/archive/v1.0.2.tar.gz | tar -zxv
+  - (cd brotli-1.0.2 && mkdir out && cd out && ../configure-cmake && make && sudo make install)
+  - rm -rf brotli-1.0.2
+  - curl -L https://github.com/facebook/zstd/archive/v1.3.2.tar.gz | tar -zxv
+  - (cd zstd-1.3.2 && sudo make install)
+  - rm -rf zstd-1.3.2
+  - sudo ldconfig
+  - mkdir /tmp/go1.12
+  - curl -L -s https://dl.google.com/go/go1.12.linux-amd64.tar.gz | tar -zxf - -C /tmp/go1.12 --strip-components 1
+  - unset GOROOT
+  - (GO111MODULE=on /tmp/go1.12/bin/go mod vendor)
+  - (cd /tmp && GO111MODULE=on /tmp/go1.12/bin/go get golang.org/x/lint/golint@8f45f776aaf18cebc8d65861cc70c33c60471952)
+  - (cd /tmp && GO111MODULE=on /tmp/go1.12/bin/go get honnef.co/go/tools/cmd/staticcheck@2019.1)
+matrix:
+  include:
+    - go: 1.9.x
+      script:
+        - go test -v -race ./...
+    - go: 1.10.x
+      script:
+        - go test -v -race ./...
+    - go: 1.11.x
+      script:
+        - go test -v -race ./...
+    - go: 1.12.x
+      script:
+        - ./ztest.sh
+    - go: master
+      script:
+        - go test -v -race ./...
+  allow_failures:
+    - go: master
+  fast_finish: true

vendor/github.com/dsnet/compress/LICENSE.md

@@ -0,0 +1,24 @@
+Copyright © 2015, Joe Tsai and The Go Authors. All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+* Redistributions of source code must retain the above copyright notice, this
+list of conditions and the following disclaimer.
+* Redistributions in binary form must reproduce the above copyright notice,
+this list of conditions and the following disclaimer in the documentation and/or
+other materials provided with the distribution.
+* Neither the copyright holder nor the names of its contributors may be used to
+endorse or promote products derived from this software without specific prior
+written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
+ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER BE LIABLE FOR ANY
+DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
+(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
+SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

vendor/github.com/dsnet/compress/README.md

@@ -0,0 +1,75 @@
+# Collection of compression libraries for Go #
+
+[![GoDoc](https://godoc.org/github.com/dsnet/compress/cmp?status.svg)](https://godoc.org/github.com/dsnet/compress)
+[![Build Status](https://travis-ci.org/dsnet/compress.svg?branch=master)](https://travis-ci.org/dsnet/compress)
+[![Report Card](https://goreportcard.com/badge/github.com/dsnet/compress)](https://goreportcard.com/report/github.com/dsnet/compress)
+
+## Introduction ##
+
+**NOTE: This library is in active development. As such, there are no guarantees about the stability of the API. The author reserves the right to arbitrarily break the API for any reason.**
+
+This repository hosts a collection of compression related libraries. The goal of this project is to provide pure Go implementations for popular compression algorithms beyond what the Go standard library provides. The goals for these packages are as follows:
+* Maintainable: That the code remains well documented, well tested, readable, easy to maintain, and easy to verify that it conforms to the specification for the format being implemented.
+* Performant: To be able to compress and decompress within at least 80% of the rates that the C implementations are able to achieve.
+* Flexible: That the code provides low-level and fine granularity control over the compression streams similar to what the C APIs would provide.
+
+Of these three, the first objective is often at odds with the other two objectives and provides interesting challenges. Higher performance can often be achieved by muddling abstraction layers or using non-intuitive low-level primitives. Also, more features and functionality, while useful in some situations, often complicates the API. Thus, this package will attempt to satisfy all the goals, but will defer to favoring maintainability when the performance or flexibility benefits are not significant enough.
+
+
+## Library Status ##
+
+For the packages available, only some features are currently implemented:
+
+| Package | Reader | Writer |
+| ------- | :----: | :----: |
+| brotli | :white_check_mark: | |
+| bzip2 | :white_check_mark: | :white_check_mark: |
+| flate | :white_check_mark: | |
+| xflate | :white_check_mark: | :white_check_mark: |
+
+This library is in active development. As such, there are no guarantees about the stability of the API. The author reserves the right to arbitrarily break the API for any reason. When the library becomes more mature, it is planned to eventually conform to some strict versioning scheme like [Semantic Versioning](http://semver.org/).
+
+However, in the meanwhile, this library does provide some basic API guarantees. For the types defined below, the method signatures are guaranteed to not change. Note that the author still reserves the right to change the fields within each ```Reader``` and ```Writer``` structs.
+```go
+type ReaderConfig struct { ... }
+type Reader struct { ... }
+  func NewReader(io.Reader, *ReaderConfig) (*Reader, error) { ... }
+  func (*Reader) Read([]byte) (int, error)                  { ... }
+  func (*Reader) Close() error                              { ... }
+
+type WriterConfig struct { ... }
+type Writer struct { ... }
+  func NewWriter(io.Writer, *WriterConfig) (*Writer, error) { ... }
+  func (*Writer) Write([]byte) (int, error)                 { ... }
+  func (*Writer) Close() error                              { ... }
+```
+
+To see what work still remains, see the [Task List](https://github.com/dsnet/compress/wiki/Task-List).
+
+## Performance  ##
+
+See [Performance Metrics](https://github.com/dsnet/compress/wiki/Performance-Metrics).
+
+
+## Frequently Asked Questions ##
+
+See [Frequently Asked Questions](https://github.com/dsnet/compress/wiki/Frequently-Asked-Questions).
+
+
+## Installation ##
+
+Run the command:
+
+```go get -u github.com/dsnet/compress```
+
+This library requires `Go1.9` or higher in order to build.
+
+
+## Packages ##
+
+| Package | Description |
+| :------ | :---------- |
+| [brotli](http://godoc.org/github.com/dsnet/compress/brotli) | Package brotli implements the Brotli format, described in RFC 7932. |
+| [bzip2](http://godoc.org/github.com/dsnet/compress/bzip2) | Package bzip2 implements the BZip2 compressed data format. |
+| [flate](http://godoc.org/github.com/dsnet/compress/flate) | Package flate implements the DEFLATE format, described in RFC 1951. |
+| [xflate](http://godoc.org/github.com/dsnet/compress/xflate) | Package xflate implements the XFLATE format, an random-access extension to DEFLATE. |

vendor/github.com/dsnet/compress/api.go

@@ -0,0 +1,74 @@
+// Copyright 2015, Joe Tsai. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE.md file.
+
+// Package compress is a collection of compression libraries.
+package compress
+
+import (
+	"bufio"
+	"io"
+
+	"github.com/dsnet/compress/internal/errors"
+)
+
+// The Error interface identifies all compression related errors.
+type Error interface {
+	error
+	CompressError()
+
+	// IsDeprecated reports the use of a deprecated and unsupported feature.
+	IsDeprecated() bool
+
+	// IsCorrupted reports whether the input stream was corrupted.
+	IsCorrupted() bool
+}
+
+var _ Error = errors.Error{}
+
+// ByteReader is an interface accepted by all decompression Readers.
+// It guarantees that the decompressor never reads more data than is necessary
+// from the underlying io.Reader.
+type ByteReader interface {
+	io.Reader
+	io.ByteReader
+}
+
+var _ ByteReader = (*bufio.Reader)(nil)
+
+// BufferedReader is an interface accepted by all decompression Readers.
+// It guarantees that the decompressor never reads more data than is necessary
+// from the underlying io.Reader. Since BufferedReader allows a decompressor
+// to peek at bytes further along in the stream without advancing the read
+// pointer, decompression can experience a significant performance gain when
+// provided a reader that satisfies this interface. Thus, a decompressor will
+// prefer this interface over ByteReader for performance reasons.
+//
+// The bufio.Reader satisfies this interface.
+type BufferedReader interface {
+	io.Reader
+
+	// Buffered returns the number of bytes currently buffered.
+	//
+	// This value becomes invalid following the next Read/Discard operation.
+	Buffered() int
+
+	// Peek returns the next n bytes without advancing the reader.
+	//
+	// If Peek returns fewer than n bytes, it also returns an error explaining
+	// why the peek is short. Peek must support peeking of at least 8 bytes.
+	// If 0 <= n <= Buffered(), Peek is guaranteed to succeed without reading
+	// from the underlying io.Reader.
+	//
+	// This result becomes invalid following the next Read/Discard operation.
+	Peek(n int) ([]byte, error)
+
+	// Discard skips the next n bytes, returning the number of bytes discarded.
+	//
+	// If Discard skips fewer than n bytes, it also returns an error.
+	// If 0 <= n <= Buffered(), Discard is guaranteed to succeed without reading
+	// from the underlying io.Reader.
+	Discard(n int) (int, error)
+}
+
+var _ BufferedReader = (*bufio.Reader)(nil)

vendor/github.com/dsnet/compress/bzip2/bwt.go

@@ -0,0 +1,110 @@
+// Copyright 2015, Joe Tsai. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE.md file.
+
+package bzip2
+
+import "github.com/dsnet/compress/bzip2/internal/sais"
+
+// The Burrows-Wheeler Transform implementation used here is based on the
+// Suffix Array by Induced Sorting (SA-IS) methodology by Nong, Zhang, and Chan.
+// This implementation uses the sais algorithm originally written by Yuta Mori.
+//
+// The SA-IS algorithm runs in O(n) and outputs a Suffix Array. There is a
+// mathematical relationship between Suffix Arrays and the Burrows-Wheeler
+// Transform, such that a SA can be converted to a BWT in O(n) time.
+//
+// References:
+//	http://www.hpl.hp.com/techreports/Compaq-DEC/SRC-RR-124.pdf
+//	https://github.com/cscott/compressjs/blob/master/lib/BWT.js
+//	https://www.quora.com/How-can-I-optimize-burrows-wheeler-transform-and-inverse-transform-to-work-in-O-n-time-O-n-space
+type burrowsWheelerTransform struct {
+	buf  []byte
+	sa   []int
+	perm []uint32
+}
+
+func (bwt *burrowsWheelerTransform) Encode(buf []byte) (ptr int) {
+	if len(buf) == 0 {
+		return -1
+	}
+
+	// TODO(dsnet): Find a way to avoid the duplicate input string method.
+	// We only need to do this because suffix arrays (by definition) only
+	// operate non-wrapped suffixes of a string. On the other hand,
+	// the BWT specifically used in bzip2 operate on a strings that wrap-around
+	// when being sorted.
+
+	// Step 1: Concatenate the input string to itself so that we can use the
+	// suffix array algorithm for bzip2's variant of BWT.
+	n := len(buf)
+	bwt.buf = append(append(bwt.buf[:0], buf...), buf...)
+	if cap(bwt.sa) < 2*n {
+		bwt.sa = make([]int, 2*n)
+	}
+	t := bwt.buf[:2*n]
+	sa := bwt.sa[:2*n]
+
+	// Step 2: Compute the suffix array (SA). The input string, t, will not be
+	// modified, while the results will be written to the output, sa.
+	sais.ComputeSA(t, sa)
+
+	// Step 3: Convert the SA to a BWT. Since ComputeSA does not mutate the
+	// input, we have two copies of the input; in buf and buf2. Thus, we write
+	// the transformation to buf, while using buf2.
+	var j int
+	buf2 := t[n:]
+	for _, i := range sa {
+		if i < n {
+			if i == 0 {
+				ptr = j
+				i = n
+			}
+			buf[j] = buf2[i-1]
+			j++
+		}
+	}
+	return ptr
+}
+
+func (bwt *burrowsWheelerTransform) Decode(buf []byte, ptr int) {
+	if len(buf) == 0 {
+		return
+	}
+
+	// Step 1: Compute cumm, where cumm[ch] reports the total number of
+	// characters that precede the character ch in the alphabet.
+	var cumm [256]int
+	for _, v := range buf {
+		cumm[v]++
+	}
+	var sum int
+	for i, v := range cumm {
+		cumm[i] = sum
+		sum += v
+	}
+
+	// Step 2: Compute perm, where perm[ptr] contains a pointer to the next
+	// byte in buf and the next pointer in perm itself.
+	if cap(bwt.perm) < len(buf) {
+		bwt.perm = make([]uint32, len(buf))
+	}
+	perm := bwt.perm[:len(buf)]
+	for i, b := range buf {
+		perm[cumm[b]] = uint32(i)
+		cumm[b]++
+	}
+
+	// Step 3: Follow each pointer in perm to the next byte, starting with the
+	// origin pointer.
+	if cap(bwt.buf) < len(buf) {
+		bwt.buf = make([]byte, len(buf))
+	}
+	buf2 := bwt.buf[:len(buf)]
+	i := perm[ptr]
+	for j := range buf2 {
+		buf2[j] = buf[i]
+		i = perm[i]
+	}
+	copy(buf, buf2)
+}

vendor/github.com/dsnet/compress/bzip2/common.go

@@ -0,0 +1,110 @@
+// Copyright 2015, Joe Tsai. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE.md file.
+
+// Package bzip2 implements the BZip2 compressed data format.
+//
+// Canonical C implementation:
+//	http://bzip.org
+//
+// Unofficial format specification:
+//	https://github.com/dsnet/compress/blob/master/doc/bzip2-format.pdf
+package bzip2
+
+import (
+	"fmt"
+	"hash/crc32"
+
+	"github.com/dsnet/compress/internal"
+	"github.com/dsnet/compress/internal/errors"
+)
+
+// There does not exist a formal specification of the BZip2 format. As such,
+// much of this work is derived by either reverse engineering the original C
+// source code or using secondary sources.
+//
+// Significant amounts of fuzz testing is done to ensure that outputs from
+// this package is properly decoded by the C library. Furthermore, we test that
+// both this package and the C library agree about what inputs are invalid.
+//
+// Compression stack:
+//	Run-length encoding 1     (RLE1)
+//	Burrows-Wheeler transform (BWT)
+//	Move-to-front transform   (MTF)
+//	Run-length encoding 2     (RLE2)
+//	Prefix encoding           (PE)
+//
+// References:
+//	http://bzip.org/
+//	https://en.wikipedia.org/wiki/Bzip2
+//	https://code.google.com/p/jbzip2/
+
+const (
+	BestSpeed          = 1
+	BestCompression    = 9
+	DefaultCompression = 6
+)
+
+const (
+	hdrMagic = 0x425a         // Hex of "BZ"
+	blkMagic = 0x314159265359 // BCD of PI
+	endMagic = 0x177245385090 // BCD of sqrt(PI)
+
+	blockSize = 100000
+)
+
+func errorf(c int, f string, a ...interface{}) error {
+	return errors.Error{Code: c, Pkg: "bzip2", Msg: fmt.Sprintf(f, a...)}
+}
+
+func panicf(c int, f string, a ...interface{}) {
+	errors.Panic(errorf(c, f, a...))
+}
+
+// errWrap converts a lower-level errors.Error to be one from this package.
+// The replaceCode passed in will be used to replace the code for any errors
+// with the errors.Invalid code.
+//
+// For the Reader, set this to errors.Corrupted.
+// For the Writer, set this to errors.Internal.
+func errWrap(err error, replaceCode int) error {
+	if cerr, ok := err.(errors.Error); ok {
+		if errors.IsInvalid(cerr) {
+			cerr.Code = replaceCode
+		}
+		err = errorf(cerr.Code, "%s", cerr.Msg)
+	}
+	return err
+}
+
+var errClosed = errorf(errors.Closed, "")
+
+// crc computes the CRC-32 used by BZip2.
+//
+// The CRC-32 computation in bzip2 treats bytes as having bits in big-endian
+// order. That is, the MSB is read before the LSB. Thus, we can use the
+// standard library version of CRC-32 IEEE with some minor adjustments.
+//
+// The byte array is used as an intermediate buffer to swap the bits of every
+// byte of the input.
+type crc struct {
+	val uint32
+	buf [256]byte
+}
+
+// update computes the CRC-32 of appending buf to c.
+func (c *crc) update(buf []byte) {
+	cval := internal.ReverseUint32(c.val)
+	for len(buf) > 0 {
+		n := len(buf)
+		if n > len(c.buf) {
+			n = len(c.buf)
+		}
+		for i, b := range buf[:n] {
+			c.buf[i] = internal.ReverseLUT[b]
+		}
+		cval = crc32.Update(cval, crc32.IEEETable, c.buf[:n])
+		buf = buf[n:]
+	}
+	c.val = internal.ReverseUint32(cval)
+}

vendor/github.com/dsnet/compress/bzip2/fuzz_off.go

@@ -0,0 +1,13 @@
+// Copyright 2016, Joe Tsai. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE.md file.
+
+// +build !gofuzz
+
+// This file exists to suppress fuzzing details from release builds.
+
+package bzip2
+
+type fuzzReader struct{}
+
+func (*fuzzReader) updateChecksum(int64, uint32) {}

vendor/github.com/dsnet/compress/bzip2/fuzz_on.go

@@ -0,0 +1,77 @@
+// Copyright 2016, Joe Tsai. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE.md file.
+
+// +build gofuzz
+
+// This file exists to export internal implementation details for fuzz testing.
+
+package bzip2
+
+func ForwardBWT(buf []byte) (ptr int) {
+	var bwt burrowsWheelerTransform
+	return bwt.Encode(buf)
+}
+
+func ReverseBWT(buf []byte, ptr int) {
+	var bwt burrowsWheelerTransform
+	bwt.Decode(buf, ptr)
+}
+
+type fuzzReader struct {
+	Checksums Checksums
+}
+
+// updateChecksum updates Checksums.
+//
+// If a valid pos is provided, it appends the (pos, val) pair to the slice.
+// Otherwise, it will update the last record with the new value.
+func (fr *fuzzReader) updateChecksum(pos int64, val uint32) {
+	if pos >= 0 {
+		fr.Checksums = append(fr.Checksums, Checksum{pos, val})
+	} else {
+		fr.Checksums[len(fr.Checksums)-1].Value = val
+	}
+}
+
+type Checksum struct {
+	Offset int64  // Bit offset of the checksum
+	Value  uint32 // Checksum value
+}
+
+type Checksums []Checksum
+
+// Apply overwrites all checksum fields in d with the ones in cs.
+func (cs Checksums) Apply(d []byte) []byte {
+	d = append([]byte(nil), d...)
+	for _, c := range cs {
+		setU32(d, c.Offset, c.Value)
+	}
+	return d
+}
+
+func setU32(d []byte, pos int64, val uint32) {
+	for i := uint(0); i < 32; i++ {
+		bpos := uint64(pos) + uint64(i)
+		d[bpos/8] &= ^byte(1 << (7 - bpos%8))
+		d[bpos/8] |= byte(val>>(31-i)) << (7 - bpos%8)
+	}
+}
+
+// Verify checks that all checksum fields in d matches those in cs.
+func (cs Checksums) Verify(d []byte) bool {
+	for _, c := range cs {
+		if getU32(d, c.Offset) != c.Value {
+			return false
+		}
+	}
+	return true
+}
+
+func getU32(d []byte, pos int64) (val uint32) {
+	for i := uint(0); i < 32; i++ {
+		bpos := uint64(pos) + uint64(i)
+		val |= (uint32(d[bpos/8] >> (7 - bpos%8))) << (31 - i)
+	}
+	return val
+}

vendor/github.com/dsnet/compress/bzip2/internal/sais/common.go

@@ -0,0 +1,28 @@
+// Copyright 2015, Joe Tsai. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE.md file.
+
+// Package sais implements a linear time suffix array algorithm.
+package sais
+
+//go:generate go run sais_gen.go byte sais_byte.go
+//go:generate go run sais_gen.go int sais_int.go
+
+// This package ports the C sais implementation by Yuta Mori. The ports are
+// located in sais_byte.go and sais_int.go, which are identical to each other
+// except for the types. Since Go does not support generics, we use generators to
+// create the two files.
+//
+// References:
+//	https://sites.google.com/site/yuta256/sais
+//	https://www.researchgate.net/publication/221313676_Linear_Time_Suffix_Array_Construction_Using_D-Critical_Substrings
+//	https://www.researchgate.net/publication/224176324_Two_Efficient_Algorithms_for_Linear_Time_Suffix_Array_Construction
+
+// ComputeSA computes the suffix array of t and places the result in sa.
+// Both t and sa must be the same length.
+func ComputeSA(t []byte, sa []int) {
+	if len(sa) != len(t) {
+		panic("mismatching sizes")
+	}
+	computeSA_byte(t, sa, 0, len(t), 256)
+}

vendor/github.com/dsnet/compress/bzip2/internal/sais/sais_byte.go

@@ -0,0 +1,661 @@
+// Copyright 2015, Joe Tsai. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE.md file.
+
+// Code generated by sais_gen.go. DO NOT EDIT.
+
+// ====================================================
+// Copyright (c) 2008-2010 Yuta Mori All Rights Reserved.
+//
+// Permission is hereby granted, free of charge, to any person
+// obtaining a copy of this software and associated documentation
+// files (the "Software"), to deal in the Software without
+// restriction, including without limitation the rights to use,
+// copy, modify, merge, publish, distribute, sublicense, and/or sell
+// copies of the Software, and to permit persons to whom the
+// Software is furnished to do so, subject to the following
+// conditions:
+//
+// The above copyright notice and this permission notice shall be
+// included in all copies or substantial portions of the Software.
+//
+// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
+// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
+// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+// OTHER DEALINGS IN THE SOFTWARE.
+// ====================================================
+
+package sais
+
+func getCounts_byte(T []byte, C []int, n, k int) {
+	var i int
+	for i = 0; i < k; i++ {
+		C[i] = 0
+	}
+	for i = 0; i < n; i++ {
+		C[T[i]]++
+	}
+}
+
+func getBuckets_byte(C, B []int, k int, end bool) {
+	var i, sum int
+	if end {
+		for i = 0; i < k; i++ {
+			sum += C[i]
+			B[i] = sum
+		}
+	} else {
+		for i = 0; i < k; i++ {
+			sum += C[i]
+			B[i] = sum - C[i]
+		}
+	}
+}
+
+func sortLMS1_byte(T []byte, SA, C, B []int, n, k int) {
+	var b, i, j int
+	var c0, c1 int
+
+	// Compute SAl.
+	if &C[0] == &B[0] {
+		getCounts_byte(T, C, n, k)
+	}
+	getBuckets_byte(C, B, k, false) // Find starts of buckets
+	j = n - 1
+	c1 = int(T[j])
+	b = B[c1]
+	j--
+	if int(T[j]) < c1 {
+		SA[b] = ^j
+	} else {
+		SA[b] = j
+	}
+	b++
+	for i = 0; i < n; i++ {
+		if j = SA[i]; j > 0 {
+			if c0 = int(T[j]); c0 != c1 {
+				B[c1] = b
+				c1 = c0
+				b = B[c1]
+			}
+			j--
+			if int(T[j]) < c1 {
+				SA[b] = ^j
+			} else {
+				SA[b] = j
+			}
+			b++
+			SA[i] = 0
+		} else if j < 0 {
+			SA[i] = ^j
+		}
+	}
+
+	// Compute SAs.
+	if &C[0] == &B[0] {
+		getCounts_byte(T, C, n, k)
+	}
+	getBuckets_byte(C, B, k, true) // Find ends of buckets
+	c1 = 0
+	b = B[c1]
+	for i = n - 1; i >= 0; i-- {
+		if j = SA[i]; j > 0 {
+			if c0 = int(T[j]); c0 != c1 {
+				B[c1] = b
+				c1 = c0
+				b = B[c1]
+			}
+			j--
+			b--
+			if int(T[j]) > c1 {
+				SA[b] = ^(j + 1)
+			} else {
+				SA[b] = j
+			}
+			SA[i] = 0
+		}
+	}
+}
+
+func postProcLMS1_byte(T []byte, SA []int, n, m int) int {
+	var i, j, p, q, plen, qlen, name int
+	var c0, c1 int
+	var diff bool
+
+	// Compact all the sorted substrings into the first m items of SA.
+	// 2*m must be not larger than n (provable).
+	for i = 0; SA[i] < 0; i++ {
+		SA[i] = ^SA[i]
+	}
+	if i < m {
+		for j, i = i, i+1; ; i++ {
+			if p = SA[i]; p < 0 {
+				SA[j] = ^p
+				j++
+				SA[i] = 0
+				if j == m {
+					break
+				}
+			}
+		}
+	}
+
+	// Store the length of all substrings.
+	i = n - 1
+	j = n - 1
+	c0 = int(T[n-1])
+	for {
+		c1 = c0
+		if i--; i < 0 {
+			break
+		}
+		if c0 = int(T[i]); c0 < c1 {
+			break
+		}
+	}
+	for i >= 0 {
+		for {
+			c1 = c0
+			if i--; i < 0 {
+				break
+			}
+			if c0 = int(T[i]); c0 > c1 {
+				break
+			}
+		}
+		if i >= 0 {
+			SA[m+((i+1)>>1)] = j - i
+			j = i + 1
+			for {
+				c1 = c0
+				if i--; i < 0 {
+					break
+				}
+				if c0 = int(T[i]); c0 < c1 {
+					break
+				}
+			}
+		}
+	}
+
+	// Find the lexicographic names of all substrings.
+	name = 0
+	qlen = 0
+	for i, q = 0, n; i < m; i++ {
+		p = SA[i]
+		plen = SA[m+(p>>1)]
+		diff = true
+		if (plen == qlen) && ((q + plen) < n) {
+			for j = 0; (j < plen) && (T[p+j] == T[q+j]); j++ {
+			}
+			if j == plen {
+				diff = false
+			}
+		}
+		if diff {
+			name++
+			q = p
+			qlen = plen
+		}
+		SA[m+(p>>1)] = name
+	}
+	return name
+}
+
+func sortLMS2_byte(T []byte, SA, C, B, D []int, n, k int) {
+	var b, i, j, t, d int
+	var c0, c1 int
+
+	// Compute SAl.
+	getBuckets_byte(C, B, k, false) // Find starts of buckets
+	j = n - 1
+	c1 = int(T[j])
+	b = B[c1]
+	j--
+	if int(T[j]) < c1 {
+		t = 1
+	} else {
+		t = 0
+	}
+	j += n
+	if t&1 > 0 {
+		SA[b] = ^j
+	} else {
+		SA[b] = j
+	}
+	b++
+	for i, d = 0, 0; i < n; i++ {
+		if j = SA[i]; j > 0 {
+			if n <= j {
+				d += 1
+				j -= n
+			}
+			if c0 = int(T[j]); c0 != c1 {
+				B[c1] = b
+				c1 = c0
+				b = B[c1]
+			}
+			j--
+			t = int(c0) << 1
+			if int(T[j]) < c1 {
+				t |= 1
+			}
+			if D[t] != d {
+				j += n
+				D[t] = d
+			}
+			if t&1 > 0 {
+				SA[b] = ^j
+			} else {
+				SA[b] = j
+			}
+			b++
+			SA[i] = 0
+		} else if j < 0 {
+			SA[i] = ^j
+		}
+	}
+	for i = n - 1; 0 <= i; i-- {
+		if SA[i] > 0 {
+			if SA[i] < n {
+				SA[i] += n
+				for j = i - 1; SA[j] < n; j-- {
+				}
+				SA[j] -= n
+				i = j
+			}
+		}
+	}
+
+	// Compute SAs.
+	getBuckets_byte(C, B, k, true) // Find ends of buckets
+	c1 = 0
+	b = B[c1]
+	for i, d = n-1, d+1; i >= 0; i-- {
+		if j = SA[i]; j > 0 {
+			if n <= j {
+				d += 1
+				j -= n
+			}
+			if c0 = int(T[j]); c0 != c1 {
+				B[c1] = b
+				c1 = c0
+				b = B[c1]
+			}
+			j--
+			t = int(c0) << 1
+			if int(T[j]) > c1 {
+				t |= 1
+			}
+			if D[t] != d {
+				j += n
+				D[t] = d
+			}
+			b--
+			if t&1 > 0 {
+				SA[b] = ^(j + 1)
+			} else {
+				SA[b] = j
+			}
+			SA[i] = 0
+		}
+	}
+}
+
+func postProcLMS2_byte(SA []int, n, m int) int {
+	var i, j, d, name int
+
+	// Compact all the sorted LMS substrings into the first m items of SA.
+	name = 0
+	for i = 0; SA[i] < 0; i++ {
+		j = ^SA[i]
+		if n <= j {
+			name += 1
+		}
+		SA[i] = j
+	}
+	if i < m {
+		for d, i = i, i+1; ; i++ {
+			if j = SA[i]; j < 0 {
+				j = ^j
+				if n <= j {
+					name += 1
+				}
+				SA[d] = j
+				d++
+				SA[i] = 0
+				if d == m {
+					break
+				}
+			}
+		}
+	}
+	if name < m {
+		// Store the lexicographic names.
+		for i, d = m-1, name+1; 0 <= i; i-- {
+			if j = SA[i]; n <= j {
+				j -= n
+				d--
+			}
+			SA[m+(j>>1)] = d
+		}
+	} else {
+		// Unset flags.
+		for i = 0; i < m; i++ {
+			if j = SA[i]; n <= j {
+				j -= n
+				SA[i] = j
+			}
+		}
+	}
+	return name
+}
+
+func induceSA_byte(T []byte, SA, C, B []int, n, k int) {
+	var b, i, j int
+	var c0, c1 int
+
+	// Compute SAl.
+	if &C[0] == &B[0] {
+		getCounts_byte(T, C, n, k)
+	}
+	getBuckets_byte(C, B, k, false) // Find starts of buckets
+	j = n - 1
+	c1 = int(T[j])
+	b = B[c1]
+	if j > 0 && int(T[j-1]) < c1 {
+		SA[b] = ^j
+	} else {
+		SA[b] = j
+	}
+	b++
+	for i = 0; i < n; i++ {
+		j = SA[i]
+		SA[i] = ^j
+		if j > 0 {
+			j--
+			if c0 = int(T[j]); c0 != c1 {
+				B[c1] = b
+				c1 = c0
+				b = B[c1]
+			}
+			if j > 0 && int(T[j-1]) < c1 {
+				SA[b] = ^j
+			} else {
+				SA[b] = j
+			}
+			b++
+		}
+	}
+
+	// Compute SAs.
+	if &C[0] == &B[0] {
+		getCounts_byte(T, C, n, k)
+	}
+	getBuckets_byte(C, B, k, true) // Find ends of buckets
+	c1 = 0
+	b = B[c1]
+	for i = n - 1; i >= 0; i-- {
+		if j = SA[i]; j > 0 {
+			j--
+			if c0 = int(T[j]); c0 != c1 {
+				B[c1] = b
+				c1 = c0
+				b = B[c1]
+			}
+			b--
+			if (j == 0) || (int(T[j-1]) > c1) {
+				SA[b] = ^j
+			} else {
+				SA[b] = j
+			}
+		} else {
+			SA[i] = ^j
+		}
+	}
+}
+
+func computeSA_byte(T []byte, SA []int, fs, n, k int) {
+	const (
+		minBucketSize = 512
+		sortLMS2Limit = 0x3fffffff
+	)
+
+	var C, B, D, RA []int
+	var bo int // Offset of B relative to SA
+	var b, i, j, m, p, q, name, newfs int
+	var c0, c1 int
+	var flags uint
+
+	if k <= minBucketSize {
+		C = make([]int, k)
+		if k <= fs {
+			bo = n + fs - k
+			B = SA[bo:]
+			flags = 1
+		} else {
+			B = make([]int, k)
+			flags = 3
+		}
+	} else if k <= fs {
+		C = SA[n+fs-k:]
+		if k <= fs-k {
+			bo = n + fs - 2*k
+			B = SA[bo:]
+			flags = 0
+		} else if k <= 4*minBucketSize {
+			B = make([]int, k)
+			flags = 2
+		} else {
+			B = C
+			flags = 8
+		}
+	} else {
+		C = make([]int, k)
+		B = C
+		flags = 4 | 8
+	}
+	if n <= sortLMS2Limit && 2 <= (n/k) {
+		if flags&1 > 0 {
+			if 2*k <= fs-k {
+				flags |= 32
+			} else {
+				flags |= 16
+			}
+		} else if flags == 0 && 2*k <= (fs-2*k) {
+			flags |= 32
+		}
+	}
+
+	// Stage 1: Reduce the problem by at least 1/2.
+	// Sort all the LMS-substrings.
+	getCounts_byte(T, C, n, k)
+	getBuckets_byte(C, B, k, true) // Find ends of buckets
+	for i = 0; i < n; i++ {
+		SA[i] = 0
+	}
+	b = -1
+	i = n - 1
+	j = n
+	m = 0
+	c0 = int(T[n-1])
+	for {
+		c1 = c0
+		if i--; i < 0 {
+			break
+		}
+		if c0 = int(T[i]); c0 < c1 {
+			break
+		}
+	}
+	for i >= 0 {
+		for {
+			c1 = c0
+			if i--; i < 0 {
+				break
+			}
+			if c0 = int(T[i]); c0 > c1 {
+				break
+			}
+		}
+		if i >= 0 {
+			if b >= 0 {
+				SA[b] = j
+			}
+			B[c1]--
+			b = B[c1]
+			j = i
+			m++
+			for {
+				c1 = c0
+				if i--; i < 0 {
+					break
+				}
+				if c0 = int(T[i]); c0 < c1 {
+					break
+				}
+			}
+		}
+	}
+
+	if m > 1 {
+		if flags&(16|32) > 0 {
+			if flags&16 > 0 {
+				D = make([]int, 2*k)
+			} else {
+				D = SA[bo-2*k:]
+			}
+			B[T[j+1]]++
+			for i, j = 0, 0; i < k; i++ {
+				j += C[i]
+				if B[i] != j {
+					SA[B[i]] += n
+				}
+				D[i] = 0
+				D[i+k] = 0
+			}
+			sortLMS2_byte(T, SA, C, B, D, n, k)
+			name = postProcLMS2_byte(SA, n, m)
+		} else {
+			sortLMS1_byte(T, SA, C, B, n, k)
+			name = postProcLMS1_byte(T, SA, n, m)
+		}
+	} else if m == 1 {
+		SA[b] = j + 1
+		name = 1
+	} else {
+		name = 0
+	}
+
+	// Stage 2: Solve the reduced problem.
+	// Recurse if names are not yet unique.
+	if name < m {
+		newfs = n + fs - 2*m
+		if flags&(1|4|8) == 0 {
+			if k+name <= newfs {
+				newfs -= k
+			} else {
+				flags |= 8
+			}
+		}
+		RA = SA[m+newfs:]
+		for i, j = m+(n>>1)-1, m-1; m <= i; i-- {
+			if SA[i] != 0 {
+				RA[j] = SA[i] - 1
+				j--
+			}
+		}
+		computeSA_int(RA, SA, newfs, m, name)
+
+		i = n - 1
+		j = m - 1
+		c0 = int(T[n-1])
+		for {
+			c1 = c0
+			if i--; i < 0 {
+				break
+			}
+			if c0 = int(T[i]); c0 < c1 {
+				break
+			}
+		}
+		for i >= 0 {
+			for {
+				c1 = c0
+				if i--; i < 0 {
+					break
+				}
+				if c0 = int(T[i]); c0 > c1 {
+					break
+				}
+			}
+			if i >= 0 {
+				RA[j] = i + 1
+				j--
+				for {
+					c1 = c0
+					if i--; i < 0 {
+						break
+					}
+					if c0 = int(T[i]); c0 < c1 {
+						break
+					}
+				}
+			}
+		}
+		for i = 0; i < m; i++ {
+			SA[i] = RA[SA[i]]
+		}
+		if flags&4 > 0 {
+			B = make([]int, k)
+			C = B
+		}
+		if flags&2 > 0 {
+			B = make([]int, k)
+		}
+	}
+
+	// Stage 3: Induce the result for the original problem.
+	if flags&8 > 0 {
+		getCounts_byte(T, C, n, k)
+	}
+	// Put all left-most S characters into their buckets.
+	if m > 1 {
+		getBuckets_byte(C, B, k, true) // Find ends of buckets
+		i = m - 1
+		j = n
+		p = SA[m-1]
+		c1 = int(T[p])
+		for {
+			c0 = c1
+			q = B[c0]
+			for q < j {
+				j--
+				SA[j] = 0
+			}
+			for {
+				j--
+				SA[j] = p
+				if i--; i < 0 {
+					break
+				}
+				p = SA[i]
+				if c1 = int(T[p]); c1 != c0 {
+					break
+				}
+			}
+			if i < 0 {
+				break
+			}
+		}
+		for j > 0 {
+			j--
+			SA[j] = 0
+		}
+	}
+	induceSA_byte(T, SA, C, B, n, k)
+}

vendor/github.com/dsnet/compress/bzip2/internal/sais/sais_int.go

@@ -0,0 +1,661 @@
+// Copyright 2015, Joe Tsai. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE.md file.
+
+// Code generated by sais_gen.go. DO NOT EDIT.
+
+// ====================================================
+// Copyright (c) 2008-2010 Yuta Mori All Rights Reserved.
+//
+// Permission is hereby granted, free of charge, to any person
+// obtaining a copy of this software and associated documentation
+// files (the "Software"), to deal in the Software without
+// restriction, including without limitation the rights to use,
+// copy, modify, merge, publish, distribute, sublicense, and/or sell
+// copies of the Software, and to permit persons to whom the
+// Software is furnished to do so, subject to the following
+// conditions:
+//
+// The above copyright notice and this permission notice shall be
+// included in all copies or substantial portions of the Software.
+//
+// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
+// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
+// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+// OTHER DEALINGS IN THE SOFTWARE.
+// ====================================================
+
+package sais
+
+func getCounts_int(T []int, C []int, n, k int) {
+	var i int
+	for i = 0; i < k; i++ {
+		C[i] = 0
+	}
+	for i = 0; i < n; i++ {
+		C[T[i]]++
+	}
+}
+
+func getBuckets_int(C, B []int, k int, end bool) {
+	var i, sum int
+	if end {
+		for i = 0; i < k; i++ {
+			sum += C[i]
+			B[i] = sum
+		}
+	} else {
+		for i = 0; i < k; i++ {
+			sum += C[i]
+			B[i] = sum - C[i]
+		}
+	}
+}
+
+func sortLMS1_int(T []int, SA, C, B []int, n, k int) {
+	var b, i, j int
+	var c0, c1 int
+
+	// Compute SAl.
+	if &C[0] == &B[0] {
+		getCounts_int(T, C, n, k)
+	}
+	getBuckets_int(C, B, k, false) // Find starts of buckets
+	j = n - 1
+	c1 = int(T[j])
+	b = B[c1]
+	j--
+	if int(T[j]) < c1 {
+		SA[b] = ^j
+	} else {
+		SA[b] = j
+	}
+	b++
+	for i = 0; i < n; i++ {
+		if j = SA[i]; j > 0 {
+			if c0 = int(T[j]); c0 != c1 {
+				B[c1] = b
+				c1 = c0
+				b = B[c1]
+			}
+			j--
+			if int(T[j]) < c1 {
+				SA[b] = ^j
+			} else {
+				SA[b] = j
+			}
+			b++
+			SA[i] = 0
+		} else if j < 0 {
+			SA[i] = ^j
+		}
+	}
+
+	// Compute SAs.
+	if &C[0] == &B[0] {
+		getCounts_int(T, C, n, k)
+	}
+	getBuckets_int(C, B, k, true) // Find ends of buckets
+	c1 = 0
+	b = B[c1]
+	for i = n - 1; i >= 0; i-- {
+		if j = SA[i]; j > 0 {
+			if c0 = int(T[j]); c0 != c1 {
+				B[c1] = b
+				c1 = c0
+				b = B[c1]
+			}
+			j--
+			b--
+			if int(T[j]) > c1 {
+				SA[b] = ^(j + 1)
+			} else {
+				SA[b] = j
+			}
+			SA[i] = 0
+		}
+	}
+}
+
+func postProcLMS1_int(T []int, SA []int, n, m int) int {
+	var i, j, p, q, plen, qlen, name int
+	var c0, c1 int
+	var diff bool
+
+	// Compact all the sorted substrings into the first m items of SA.
+	// 2*m must be not larger than n (provable).
+	for i = 0; SA[i] < 0; i++ {
+		SA[i] = ^SA[i]
+	}
+	if i < m {
+		for j, i = i, i+1; ; i++ {
+			if p = SA[i]; p < 0 {
+				SA[j] = ^p
+				j++
+				SA[i] = 0
+				if j == m {
+					break
+				}
+			}
+		}
+	}
+
+	// Store the length of all substrings.
+	i = n - 1
+	j = n - 1
+	c0 = int(T[n-1])
+	for {
+		c1 = c0
+		if i--; i < 0 {
+			break
+		}
+		if c0 = int(T[i]); c0 < c1 {
+			break
+		}
+	}
+	for i >= 0 {
+		for {
+			c1 = c0
+			if i--; i < 0 {
+				break
+			}
+			if c0 = int(T[i]); c0 > c1 {
+				break
+			}
+		}
+		if i >= 0 {
+			SA[m+((i+1)>>1)] = j - i
+			j = i + 1
+			for {
+				c1 = c0
+				if i--; i < 0 {
+					break
+				}
+				if c0 = int(T[i]); c0 < c1 {
+					break
+				}
+			}
+		}
+	}
+
+	// Find the lexicographic names of all substrings.
+	name = 0
+	qlen = 0
+	for i, q = 0, n; i < m; i++ {
+		p = SA[i]
+		plen = SA[m+(p>>1)]
+		diff = true
+		if (plen == qlen) && ((q + plen) < n) {
+			for j = 0; (j < plen) && (T[p+j] == T[q+j]); j++ {
+			}
+			if j == plen {
+				diff = false
+			}
+		}
+		if diff {
+			name++
+			q = p
+			qlen = plen
+		}
+		SA[m+(p>>1)] = name
+	}
+	return name
+}
+
+func sortLMS2_int(T []int, SA, C, B, D []int, n, k int) {
+	var b, i, j, t, d int
+	var c0, c1 int
+
+	// Compute SAl.
+	getBuckets_int(C, B, k, false) // Find starts of buckets
+	j = n - 1
+	c1 = int(T[j])
+	b = B[c1]
+	j--
+	if int(T[j]) < c1 {
+		t = 1
+	} else {
+		t = 0
+	}
+	j += n
+	if t&1 > 0 {
+		SA[b] = ^j
+	} else {
+		SA[b] = j
+	}
+	b++
+	for i, d = 0, 0; i < n; i++ {
+		if j = SA[i]; j > 0 {
+			if n <= j {
+				d += 1
+				j -= n
+			}
+			if c0 = int(T[j]); c0 != c1 {
+				B[c1] = b
+				c1 = c0
+				b = B[c1]
+			}
+			j--
+			t = int(c0) << 1
+			if int(T[j]) < c1 {
+				t |= 1
+			}
+			if D[t] != d {
+				j += n
+				D[t] = d
+			}
+			if t&1 > 0 {
+				SA[b] = ^j
+			} else {
+				SA[b] = j
+			}
+			b++
+			SA[i] = 0
+		} else if j < 0 {
+			SA[i] = ^j
+		}
+	}
+	for i = n - 1; 0 <= i; i-- {
+		if SA[i] > 0 {
+			if SA[i] < n {
+				SA[i] += n
+				for j = i - 1; SA[j] < n; j-- {
+				}
+				SA[j] -= n
+				i = j
+			}
+		}
+	}
+
+	// Compute SAs.
+	getBuckets_int(C, B, k, true) // Find ends of buckets
+	c1 = 0
+	b = B[c1]
+	for i, d = n-1, d+1; i >= 0; i-- {
+		if j = SA[i]; j > 0 {
+			if n <= j {
+				d += 1
+				j -= n
+			}
+			if c0 = int(T[j]); c0 != c1 {
+				B[c1] = b
+				c1 = c0
+				b = B[c1]
+			}
+			j--
+			t = int(c0) << 1
+			if int(T[j]) > c1 {
+				t |= 1
+			}
+			if D[t] != d {
+				j += n
+				D[t] = d
+			}
+			b--
+			if t&1 > 0 {
+				SA[b] = ^(j + 1)
+			} else {
+				SA[b] = j
+			}
+			SA[i] = 0
+		}
+	}
+}
+
+func postProcLMS2_int(SA []int, n, m int) int {
+	var i, j, d, name int
+
+	// Compact all the sorted LMS substrings into the first m items of SA.
+	name = 0
+	for i = 0; SA[i] < 0; i++ {
+		j = ^SA[i]
+		if n <= j {
+			name += 1
+		}
+		SA[i] = j
+	}
+	if i < m {
+		for d, i = i, i+1; ; i++ {
+			if j = SA[i]; j < 0 {
+				j = ^j
+				if n <= j {
+					name += 1
+				}
+				SA[d] = j
+				d++
+				SA[i] = 0
+				if d == m {
+					break
+				}
+			}
+		}
+	}
+	if name < m {
+		// Store the lexicographic names.
+		for i, d = m-1, name+1; 0 <= i; i-- {
+			if j = SA[i]; n <= j {
+				j -= n
+				d--
+			}
+			SA[m+(j>>1)] = d
+		}
+	} else {
+		// Unset flags.
+		for i = 0; i < m; i++ {
+			if j = SA[i]; n <= j {
+				j -= n
+				SA[i] = j
+			}
+		}
+	}
+	return name
+}
+
+func induceSA_int(T []int, SA, C, B []int, n, k int) {
+	var b, i, j int
+	var c0, c1 int
+
+	// Compute SAl.
+	if &C[0] == &B[0] {
+		getCounts_int(T, C, n, k)
+	}
+	getBuckets_int(C, B, k, false) // Find starts of buckets
+	j = n - 1
+	c1 = int(T[j])
+	b = B[c1]
+	if j > 0 && int(T[j-1]) < c1 {
+		SA[b] = ^j
+	} else {
+		SA[b] = j
+	}
+	b++
+	for i = 0; i < n; i++ {
+		j = SA[i]
+		SA[i] = ^j
+		if j > 0 {
+			j--
+			if c0 = int(T[j]); c0 != c1 {
+				B[c1] = b
+				c1 = c0
+				b = B[c1]
+			}
+			if j > 0 && int(T[j-1]) < c1 {
+				SA[b] = ^j
+			} else {
+				SA[b] = j
+			}
+			b++
+		}
+	}
+
+	// Compute SAs.
+	if &C[0] == &B[0] {
+		getCounts_int(T, C, n, k)
+	}
+	getBuckets_int(C, B, k, true) // Find ends of buckets
+	c1 = 0
+	b = B[c1]
+	for i = n - 1; i >= 0; i-- {
+		if j = SA[i]; j > 0 {
+			j--
+			if c0 = int(T[j]); c0 != c1 {
+				B[c1] = b
+				c1 = c0
+				b = B[c1]
+			}
+			b--
+			if (j == 0) || (int(T[j-1]) > c1) {
+				SA[b] = ^j
+			} else {
+				SA[b] = j
+			}
+		} else {
+			SA[i] = ^j
+		}
+	}
+}
+
+func computeSA_int(T []int, SA []int, fs, n, k int) {
+	const (
+		minBucketSize = 512
+		sortLMS2Limit = 0x3fffffff
+	)
+
+	var C, B, D, RA []int
+	var bo int // Offset of B relative to SA
+	var b, i, j, m, p, q, name, newfs int
+	var c0, c1 int
+	var flags uint
+
+	if k <= minBucketSize {
+		C = make([]int, k)
+		if k <= fs {
+			bo = n + fs - k
+			B = SA[bo:]
+			flags = 1
+		} else {
+			B = make([]int, k)
+			flags = 3
+		}
+	} else if k <= fs {
+		C = SA[n+fs-k:]
+		if k <= fs-k {
+			bo = n + fs - 2*k
+			B = SA[bo:]
+			flags = 0
+		} else if k <= 4*minBucketSize {
+			B = make([]int, k)
+			flags = 2
+		} else {
+			B = C
+			flags = 8
+		}
+	} else {
+		C = make([]int, k)
+		B = C
+		flags = 4 | 8
+	}
+	if n <= sortLMS2Limit && 2 <= (n/k) {
+		if flags&1 > 0 {
+			if 2*k <= fs-k {
+				flags |= 32
+			} else {
+				flags |= 16
+			}
+		} else if flags == 0 && 2*k <= (fs-2*k) {
+			flags |= 32
+		}
+	}
+
+	// Stage 1: Reduce the problem by at least 1/2.
+	// Sort all the LMS-substrings.
+	getCounts_int(T, C, n, k)
+	getBuckets_int(C, B, k, true) // Find ends of buckets
+	for i = 0; i < n; i++ {
+		SA[i] = 0
+	}
+	b = -1
+	i = n - 1
+	j = n
+	m = 0
+	c0 = int(T[n-1])
+	for {
+		c1 = c0
+		if i--; i < 0 {
+			break
+		}
+		if c0 = int(T[i]); c0 < c1 {
+			break
+		}
+	}
+	for i >= 0 {
+		for {
+			c1 = c0
+			if i--; i < 0 {
+				break
+			}
+			if c0 = int(T[i]); c0 > c1 {
+				break
+			}
+		}
+		if i >= 0 {
+			if b >= 0 {
+				SA[b] = j
+			}
+			B[c1]--
+			b = B[c1]
+			j = i
+			m++
+			for {
+				c1 = c0
+				if i--; i < 0 {
+					break
+				}
+				if c0 = int(T[i]); c0 < c1 {
+					break
+				}
+			}
+		}
+	}
+
+	if m > 1 {
+		if flags&(16|32) > 0 {
+			if flags&16 > 0 {
+				D = make([]int, 2*k)
+			} else {
+				D = SA[bo-2*k:]
+			}
+			B[T[j+1]]++
+			for i, j = 0, 0; i < k; i++ {
+				j += C[i]
+				if B[i] != j {
+					SA[B[i]] += n
+				}
+				D[i] = 0
+				D[i+k] = 0
+			}
+			sortLMS2_int(T, SA, C, B, D, n, k)
+			name = postProcLMS2_int(SA, n, m)
+		} else {
+			sortLMS1_int(T, SA, C, B, n, k)
+			name = postProcLMS1_int(T, SA, n, m)
+		}
+	} else if m == 1 {
+		SA[b] = j + 1
+		name = 1
+	} else {
+		name = 0
+	}
+
+	// Stage 2: Solve the reduced problem.
+	// Recurse if names are not yet unique.
+	if name < m {
+		newfs = n + fs - 2*m
+		if flags&(1|4|8) == 0 {
+			if k+name <= newfs {
+				newfs -= k
+			} else {
+				flags |= 8
+			}
+		}
+		RA = SA[m+newfs:]
+		for i, j = m+(n>>1)-1, m-1; m <= i; i-- {
+			if SA[i] != 0 {
+				RA[j] = SA[i] - 1
+				j--
+			}
+		}
+		computeSA_int(RA, SA, newfs, m, name)
+
+		i = n - 1
+		j = m - 1
+		c0 = int(T[n-1])
+		for {
+			c1 = c0
+			if i--; i < 0 {
+				break
+			}
+			if c0 = int(T[i]); c0 < c1 {
+				break
+			}
+		}
+		for i >= 0 {
+			for {
+				c1 = c0
+				if i--; i < 0 {
+					break
+				}
+				if c0 = int(T[i]); c0 > c1 {
+					break
+				}
+			}
+			if i >= 0 {
+				RA[j] = i + 1
+				j--
+				for {
+					c1 = c0
+					if i--; i < 0 {
+						break
+					}
+					if c0 = int(T[i]); c0 < c1 {
+						break
+					}
+				}
+			}
+		}
+		for i = 0; i < m; i++ {
+			SA[i] = RA[SA[i]]
+		}
+		if flags&4 > 0 {
+			B = make([]int, k)
+			C = B
+		}
+		if flags&2 > 0 {
+			B = make([]int, k)
+		}
+	}
+
+	// Stage 3: Induce the result for the original problem.
+	if flags&8 > 0 {
+		getCounts_int(T, C, n, k)
+	}
+	// Put all left-most S characters into their buckets.
+	if m > 1 {
+		getBuckets_int(C, B, k, true) // Find ends of buckets
+		i = m - 1
+		j = n
+		p = SA[m-1]
+		c1 = int(T[p])
+		for {
+			c0 = c1
+			q = B[c0]
+			for q < j {
+				j--
+				SA[j] = 0
+			}
+			for {
+				j--
+				SA[j] = p
+				if i--; i < 0 {
+					break
+				}
+				p = SA[i]
+				if c1 = int(T[p]); c1 != c0 {
+					break
+				}
+			}
+			if i < 0 {
+				break
+			}
+		}
+		for j > 0 {
+			j--
+			SA[j] = 0
+		}
+	}
+	induceSA_int(T, SA, C, B, n, k)
+}

vendor/github.com/dsnet/compress/bzip2/mtf_rle2.go

@@ -0,0 +1,131 @@
+// Copyright 2015, Joe Tsai. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE.md file.
+
+package bzip2
+
+import "github.com/dsnet/compress/internal/errors"
+
+// moveToFront implements both the MTF and RLE stages of bzip2 at the same time.
+// Any runs of zeros in the encoded output will be replaced by a sequence of
+// RUNA and RUNB symbols are encode the length of the run.
+//
+// The RLE encoding used can actually be encoded to and decoded from using
+// normal two's complement arithmetic. The methodology for doing so is below.
+//
+// Assuming the following:
+//	num: The value being encoded by RLE encoding.
+//	run: A sequence of RUNA and RUNB symbols represented as a binary integer,
+//	where RUNA is the 0 bit, RUNB is the 1 bit, and least-significant RUN
+//	symbols are at the least-significant bit positions.
+//	cnt: The number of RUNA and RUNB symbols.
+//
+// Then the RLE encoding used by bzip2 has this mathematical property:
+//	num+1 == (1<<cnt) | run
+type moveToFront struct {
+	dictBuf [256]uint8
+	dictLen int
+
+	vals    []byte
+	syms    []uint16
+	blkSize int
+}
+
+func (mtf *moveToFront) Init(dict []uint8, blkSize int) {
+	if len(dict) > len(mtf.dictBuf) {
+		panicf(errors.Internal, "alphabet too large")
+	}
+	copy(mtf.dictBuf[:], dict)
+	mtf.dictLen = len(dict)
+	mtf.blkSize = blkSize
+}
+
+func (mtf *moveToFront) Encode(vals []byte) (syms []uint16) {
+	dict := mtf.dictBuf[:mtf.dictLen]
+	syms = mtf.syms[:0]
+
+	if len(vals) > mtf.blkSize {
+		panicf(errors.Internal, "exceeded block size")
+	}
+
+	var lastNum uint32
+	for _, val := range vals {
+		// Normal move-to-front transform.
+		var idx uint8 // Reverse lookup idx in dict
+		for di, dv := range dict {
+			if dv == val {
+				idx = uint8(di)
+				break
+			}
+		}
+		copy(dict[1:], dict[:idx])
+		dict[0] = val
+
+		// Run-length encoding augmentation.
+		if idx == 0 {
+			lastNum++
+			continue
+		}
+		if lastNum > 0 {
+			for rc := lastNum + 1; rc != 1; rc >>= 1 {
+				syms = append(syms, uint16(rc&1))
+			}
+			lastNum = 0
+		}
+		syms = append(syms, uint16(idx)+1)
+	}
+	if lastNum > 0 {
+		for rc := lastNum + 1; rc != 1; rc >>= 1 {
+			syms = append(syms, uint16(rc&1))
+		}
+	}
+	mtf.syms = syms
+	return syms
+}
+
+func (mtf *moveToFront) Decode(syms []uint16) (vals []byte) {
+	dict := mtf.dictBuf[:mtf.dictLen]
+	vals = mtf.vals[:0]
+
+	var lastCnt uint
+	var lastRun uint32
+	for _, sym := range syms {
+		// Run-length encoding augmentation.
+		if sym < 2 {
+			lastRun |= uint32(sym) << lastCnt
+			lastCnt++
+			continue
+		}
+		if lastCnt > 0 {
+			cnt := int((1<<lastCnt)|lastRun) - 1
+			if len(vals)+cnt > mtf.blkSize || lastCnt > 24 {
+				panicf(errors.Corrupted, "run-length decoding exceeded block size")
+			}
+			for i := cnt; i > 0; i-- {
+				vals = append(vals, dict[0])
+			}
+			lastCnt, lastRun = 0, 0
+		}
+
+		// Normal move-to-front transform.
+		val := dict[sym-1] // Forward lookup val in dict
+		copy(dict[1:], dict[:sym-1])
+		dict[0] = val
+
+		if len(vals) >= mtf.blkSize {
+			panicf(errors.Corrupted, "run-length decoding exceeded block size")
+		}
+		vals = append(vals, val)
+	}
+	if lastCnt > 0 {
+		cnt := int((1<<lastCnt)|lastRun) - 1
+		if len(vals)+cnt > mtf.blkSize || lastCnt > 24 {
+			panicf(errors.Corrupted, "run-length decoding exceeded block size")
+		}
+		for i := cnt; i > 0; i-- {
+			vals = append(vals, dict[0])
+		}
+	}
+	mtf.vals = vals
+	return vals
+}

vendor/github.com/dsnet/compress/bzip2/prefix.go

@@ -0,0 +1,374 @@
+// Copyright 2015, Joe Tsai. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE.md file.
+
+package bzip2
+
+import (
+	"io"
+
+	"github.com/dsnet/compress/internal"
+	"github.com/dsnet/compress/internal/errors"
+	"github.com/dsnet/compress/internal/prefix"
+)
+
+const (
+	minNumTrees = 2
+	maxNumTrees = 6
+
+	maxPrefixBits = 20      // Maximum bit-width of a prefix code
+	maxNumSyms    = 256 + 2 // Maximum number of symbols in the alphabet
+	numBlockSyms  = 50      // Number of bytes in a block
+)
+
+// encSel and decSel are used to handle the prefix encoding for tree selectors.
+// The prefix encoding is as follows:
+//
+//	Code         TreeIdx
+//	0        <=> 0
+//	10       <=> 1
+//	110      <=> 2
+//	1110     <=> 3
+//	11110    <=> 4
+//	111110   <=> 5
+//	111111   <=> 6	Invalid tree index, so should fail
+//
+var encSel, decSel = func() (e prefix.Encoder, d prefix.Decoder) {
+	var selCodes [maxNumTrees + 1]prefix.PrefixCode
+	for i := range selCodes {
+		selCodes[i] = prefix.PrefixCode{Sym: uint32(i), Len: uint32(i + 1)}
+	}
+	selCodes[maxNumTrees] = prefix.PrefixCode{Sym: maxNumTrees, Len: maxNumTrees}
+	prefix.GeneratePrefixes(selCodes[:])
+	e.Init(selCodes[:])
+	d.Init(selCodes[:])
+	return
+}()
+
+type prefixReader struct{ prefix.Reader }
+
+func (pr *prefixReader) Init(r io.Reader) {
+	pr.Reader.Init(r, true)
+}
+
+func (pr *prefixReader) ReadBitsBE64(nb uint) uint64 {
+	if nb <= 32 {
+		v := uint32(pr.ReadBits(nb))
+		return uint64(internal.ReverseUint32N(v, nb))
+	}
+	v0 := internal.ReverseUint32(uint32(pr.ReadBits(32)))
+	v1 := internal.ReverseUint32(uint32(pr.ReadBits(nb - 32)))
+	v := uint64(v0)<<32 | uint64(v1)
+	return v >> (64 - nb)
+}
+
+func (pr *prefixReader) ReadPrefixCodes(codes []prefix.PrefixCodes, trees []prefix.Decoder) {
+	for i, pc := range codes {
+		clen := int(pr.ReadBitsBE64(5))
+		sum := 1 << maxPrefixBits
+		for sym := range pc {
+			for {
+				if clen < 1 || clen > maxPrefixBits {
+					panicf(errors.Corrupted, "invalid prefix bit-length: %d", clen)
+				}
+
+				b, ok := pr.TryReadBits(1)
+				if !ok {
+					b = pr.ReadBits(1)
+				}
+				if b == 0 {
+					break
+				}
+
+				b, ok = pr.TryReadBits(1)
+				if !ok {
+					b = pr.ReadBits(1)
+				}
+				clen -= int(b*2) - 1 // +1 or -1
+			}
+			pc[sym] = prefix.PrefixCode{Sym: uint32(sym), Len: uint32(clen)}
+			sum -= (1 << maxPrefixBits) >> uint(clen)
+		}
+
+		if sum == 0 {
+			// Fast path, but only handles complete trees.
+			if err := prefix.GeneratePrefixes(pc); err != nil {
+				errors.Panic(err) // Using complete trees; should never fail
+			}
+		} else {
+			// Slow path, but handles anything.
+			pc = handleDegenerateCodes(pc) // Never fails, but may fail later
+			codes[i] = pc
+		}
+		trees[i].Init(pc)
+	}
+}
+
+type prefixWriter struct{ prefix.Writer }
+
+func (pw *prefixWriter) Init(w io.Writer) {
+	pw.Writer.Init(w, true)
+}
+
+func (pw *prefixWriter) WriteBitsBE64(v uint64, nb uint) {
+	if nb <= 32 {
+		v := internal.ReverseUint32N(uint32(v), nb)
+		pw.WriteBits(uint(v), nb)
+		return
+	}
+	v <<= (64 - nb)
+	v0 := internal.ReverseUint32(uint32(v >> 32))
+	v1 := internal.ReverseUint32(uint32(v))
+	pw.WriteBits(uint(v0), 32)
+	pw.WriteBits(uint(v1), nb-32)
+	return
+}
+
+func (pw *prefixWriter) WritePrefixCodes(codes []prefix.PrefixCodes, trees []prefix.Encoder) {
+	for i, pc := range codes {
+		if err := prefix.GeneratePrefixes(pc); err != nil {
+			errors.Panic(err) // Using complete trees; should never fail
+		}
+		trees[i].Init(pc)
+
+		clen := int(pc[0].Len)
+		pw.WriteBitsBE64(uint64(clen), 5)
+		for _, c := range pc {
+			for int(c.Len) < clen {
+				pw.WriteBits(3, 2) // 11
+				clen--
+			}
+			for int(c.Len) > clen {
+				pw.WriteBits(1, 2) // 10
+				clen++
+			}
+			pw.WriteBits(0, 1)
+		}
+	}
+}
+
+// handleDegenerateCodes converts a degenerate tree into a canonical tree.
+//
+// For example, when the input is an under-subscribed tree:
+//	input:  []PrefixCode{
+//		{Sym: 0, Len: 3},
+//		{Sym: 1, Len: 4},
+//		{Sym: 2, Len: 3},
+//	}
+//	output: []PrefixCode{
+//		{Sym:   0, Len: 3, Val:  0}, //  000
+//		{Sym:   1, Len: 4, Val:  2}, // 0010
+//		{Sym:   2, Len: 3, Val:  4}, //  100
+//		{Sym: 258, Len: 4, Val: 10}, // 1010
+//		{Sym: 259, Len: 3, Val:  6}, //  110
+//		{Sym: 260, Len: 1, Val:  1}, //    1
+//	}
+//
+// For example, when the input is an over-subscribed tree:
+//	input:  []PrefixCode{
+//		{Sym: 0, Len: 1},
+//		{Sym: 1, Len: 3},
+//		{Sym: 2, Len: 4},
+//		{Sym: 3, Len: 3},
+//		{Sym: 4, Len: 2},
+//	}
+//	output: []PrefixCode{
+//		{Sym: 0, Len: 1, Val: 0}, //   0
+//		{Sym: 1, Len: 3, Val: 3}, // 011
+//		{Sym: 3, Len: 3, Val: 7}, // 111
+//		{Sym: 4, Len: 2, Val: 1}, //  01
+//	}
+func handleDegenerateCodes(codes prefix.PrefixCodes) prefix.PrefixCodes {
+	// Since there is no formal definition for the BZip2 format, there is no
+	// specification that says that the code lengths must form a complete
+	// prefix tree (IE: it is neither over-subscribed nor under-subscribed).
+	// Thus, the original C implementation becomes the reference for how prefix
+	// decoding is done in these edge cases. Unfortunately, the C version does
+	// not error when an invalid tree is used, but rather allows decoding to
+	// continue and only errors if some bit pattern happens to cause an error.
+	// Thus, it is possible for an invalid tree to end up decoding an input
+	// "properly" so long as invalid bit patterns are not present. In order to
+	// replicate this non-specified behavior, we use a ported version of the
+	// C code to generate the codes as a valid canonical tree by substituting
+	// invalid nodes with invalid symbols.
+	//
+	// ====================================================
+	// This program, "bzip2", the associated library "libbzip2", and all
+	// documentation, are copyright (C) 1996-2010 Julian R Seward.  All
+	// rights reserved.
+	//
+	// Redistribution and use in source and binary forms, with or without
+	// modification, are permitted provided that the following conditions
+	// are met:
+	//
+	// 1. Redistributions of source code must retain the above copyright
+	//    notice, this list of conditions and the following disclaimer.
+	//
+	// 2. The origin of this software must not be misrepresented; you must
+	//    not claim that you wrote the original software.  If you use this
+	//    software in a product, an acknowledgment in the product
+	//    documentation would be appreciated but is not required.
+	//
+	// 3. Altered source versions must be plainly marked as such, and must
+	//    not be misrepresented as being the original software.
+	//
+	// 4. The name of the author may not be used to endorse or promote
+	//    products derived from this software without specific prior written
+	//    permission.
+	//
+	// THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS
+	// OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+	// WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+	// ARE DISCLAIMED.  IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY
+	// DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+	// DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE
+	// GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+	// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
+	// WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
+	// NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
+	// SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+	//
+	// Julian Seward, jseward@bzip.org
+	// bzip2/libbzip2 version 1.0.6 of 6 September 2010
+	// ====================================================
+	var (
+		limits [maxPrefixBits + 2]int32
+		bases  [maxPrefixBits + 2]int32
+		perms  [maxNumSyms]int32
+
+		minLen = uint32(maxPrefixBits)
+		maxLen = uint32(0)
+	)
+
+	const (
+		statusOkay = iota
+		statusInvalid
+		statusNeedBits
+		statusMaxBits
+	)
+
+	// createTables is the BZ2_hbCreateDecodeTables function from the C code.
+	createTables := func(codes []prefix.PrefixCode) {
+		for _, c := range codes {
+			if c.Len > maxLen {
+				maxLen = c.Len
+			}
+			if c.Len < minLen {
+				minLen = c.Len
+			}
+		}
+
+		var pp int
+		for i := minLen; i <= maxLen; i++ {
+			for j, c := range codes {
+				if c.Len == i {
+					perms[pp] = int32(j)
+					pp++
+				}
+			}
+		}
+
+		var vec int32
+		for _, c := range codes {
+			bases[c.Len+1]++
+		}
+		for i := 1; i < len(bases); i++ {
+			bases[i] += bases[i-1]
+		}
+		for i := minLen; i <= maxLen; i++ {
+			vec += bases[i+1] - bases[i]
+			limits[i] = vec - 1
+			vec <<= 1
+		}
+		for i := minLen + 1; i <= maxLen; i++ {
+			bases[i] = ((limits[i-1] + 1) << 1) - bases[i]
+		}
+	}
+
+	// getSymbol is the GET_MTF_VAL macro from the C code.
+	getSymbol := func(c prefix.PrefixCode) (uint32, int) {
+		v := internal.ReverseUint32(c.Val)
+		n := c.Len
+
+		zn := minLen
+		if zn > n {
+			return 0, statusNeedBits
+		}
+		zvec := int32(v >> (32 - zn))
+		v <<= zn
+		for {
+			if zn > maxLen {
+				return 0, statusMaxBits
+			}
+			if zvec <= limits[zn] {
+				break
+			}
+			zn++
+			if zn > n {
+				return 0, statusNeedBits
+			}
+			zvec = (zvec << 1) | int32(v>>31)
+			v <<= 1
+		}
+		if zvec-bases[zn] < 0 || zvec-bases[zn] >= maxNumSyms {
+			return 0, statusInvalid
+		}
+		return uint32(perms[zvec-bases[zn]]), statusOkay
+	}
+
+	// Step 1: Create the prefix trees using the C algorithm.
+	createTables(codes)
+
+	// Step 2: Starting with the shortest bit pattern, explore the whole tree.
+	// If tree is under-subscribed, the worst-case runtime is O(1<<maxLen).
+	// If tree is over-subscribed, the worst-case runtime is O(maxNumSyms).
+	var pcodesArr [2 * maxNumSyms]prefix.PrefixCode
+	pcodes := pcodesArr[:maxNumSyms]
+	var exploreCode func(prefix.PrefixCode) bool
+	exploreCode = func(c prefix.PrefixCode) (term bool) {
+		sym, status := getSymbol(c)
+		switch status {
+		case statusOkay:
+			// This code is valid, so insert it.
+			c.Sym = sym
+			pcodes[sym] = c
+			term = true
+		case statusInvalid:
+			// This code is invalid, so insert an invalid symbol.
+			c.Sym = uint32(len(pcodes))
+			pcodes = append(pcodes, c)
+			term = true
+		case statusNeedBits:
+			// This code is too short, so explore both children.
+			c.Len++
+			c0, c1 := c, c
+			c1.Val |= 1 << (c.Len - 1)
+
+			b0 := exploreCode(c0)
+			b1 := exploreCode(c1)
+			switch {
+			case !b0 && b1:
+				c0.Sym = uint32(len(pcodes))
+				pcodes = append(pcodes, c0)
+			case !b1 && b0:
+				c1.Sym = uint32(len(pcodes))
+				pcodes = append(pcodes, c1)
+			}
+			term = b0 || b1
+		case statusMaxBits:
+			// This code is too long, so report it upstream.
+			term = false
+		}
+		return term // Did this code terminate?
+	}
+	exploreCode(prefix.PrefixCode{})
+
+	// Step 3: Copy new sparse codes to old output codes.
+	codes = codes[:0]
+	for _, c := range pcodes {
+		if c.Len > 0 {
+			codes = append(codes, c)
+		}
+	}
+	return codes
+}

vendor/github.com/dsnet/compress/bzip2/reader.go

@@ -0,0 +1,274 @@
+// Copyright 2015, Joe Tsai. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE.md file.
+
+package bzip2
+
+import (
+	"io"
+
+	"github.com/dsnet/compress/internal"
+	"github.com/dsnet/compress/internal/errors"
+	"github.com/dsnet/compress/internal/prefix"
+)
+
+type Reader struct {
+	InputOffset  int64 // Total number of bytes read from underlying io.Reader
+	OutputOffset int64 // Total number of bytes emitted from Read
+
+	rd       prefixReader
+	err      error
+	level    int    // The current compression level
+	rdHdrFtr int    // Number of times we read the stream header and footer
+	blkCRC   uint32 // CRC-32 IEEE of each block (as stored)
+	endCRC   uint32 // Checksum of all blocks using bzip2's custom method
+
+	crc crc
+	mtf moveToFront
+	bwt burrowsWheelerTransform
+	rle runLengthEncoding
+
+	// These fields are allocated with Reader and re-used later.
+	treeSels []uint8
+	codes2D  [maxNumTrees][maxNumSyms]prefix.PrefixCode
+	codes1D  [maxNumTrees]prefix.PrefixCodes
+	trees1D  [maxNumTrees]prefix.Decoder
+	syms     []uint16
+
+	fuzzReader // Exported functionality when fuzz testing
+}
+
+type ReaderConfig struct {
+	_ struct{} // Blank field to prevent unkeyed struct literals
+}
+
+func NewReader(r io.Reader, conf *ReaderConfig) (*Reader, error) {
+	zr := new(Reader)
+	zr.Reset(r)
+	return zr, nil
+}
+
+func (zr *Reader) Reset(r io.Reader) error {
+	*zr = Reader{
+		rd: zr.rd,
+
+		mtf: zr.mtf,
+		bwt: zr.bwt,
+		rle: zr.rle,
+
+		treeSels: zr.treeSels,
+		trees1D:  zr.trees1D,
+		syms:     zr.syms,
+	}
+	zr.rd.Init(r)
+	return nil
+}
+
+func (zr *Reader) Read(buf []byte) (int, error) {
+	for {
+		cnt, err := zr.rle.Read(buf)
+		if err != rleDone && zr.err == nil {
+			zr.err = err
+		}
+		if cnt > 0 {
+			zr.crc.update(buf[:cnt])
+			zr.OutputOffset += int64(cnt)
+			return cnt, nil
+		}
+		if zr.err != nil || len(buf) == 0 {
+			return 0, zr.err
+		}
+
+		// Read the next chunk.
+		zr.rd.Offset = zr.InputOffset
+		func() {
+			defer errors.Recover(&zr.err)
+			if zr.rdHdrFtr%2 == 0 {
+				// Check if we are already at EOF.
+				if err := zr.rd.PullBits(1); err != nil {
+					if err == io.ErrUnexpectedEOF && zr.rdHdrFtr > 0 {
+						err = io.EOF // EOF is okay if we read at least one stream
+					}
+					errors.Panic(err)
+				}
+
+				// Read stream header.
+				if zr.rd.ReadBitsBE64(16) != hdrMagic {
+					panicf(errors.Corrupted, "invalid stream magic")
+				}
+				if ver := zr.rd.ReadBitsBE64(8); ver != 'h' {
+					if ver == '0' {
+						panicf(errors.Deprecated, "bzip1 format is not supported")
+					}
+					panicf(errors.Corrupted, "invalid version: %q", ver)
+				}
+				lvl := int(zr.rd.ReadBitsBE64(8)) - '0'
+				if lvl < BestSpeed || lvl > BestCompression {
+					panicf(errors.Corrupted, "invalid block size: %d", lvl*blockSize)
+				}
+				zr.level = lvl
+				zr.rdHdrFtr++
+			} else {
+				// Check and update the CRC.
+				if internal.GoFuzz {
+					zr.updateChecksum(-1, zr.crc.val) // Update with value
+					zr.blkCRC = zr.crc.val            // Suppress CRC failures
+				}
+				if zr.blkCRC != zr.crc.val {
+					panicf(errors.Corrupted, "mismatching block checksum")
+				}
+				zr.endCRC = (zr.endCRC<<1 | zr.endCRC>>31) ^ zr.blkCRC
+			}
+			buf := zr.decodeBlock()
+			zr.rle.Init(buf)
+		}()
+		if zr.InputOffset, err = zr.rd.Flush(); zr.err == nil {
+			zr.err = err
+		}
+		if zr.err != nil {
+			zr.err = errWrap(zr.err, errors.Corrupted)
+			return 0, zr.err
+		}
+	}
+}
+
+func (zr *Reader) Close() error {
+	if zr.err == io.EOF || zr.err == errClosed {
+		zr.rle.Init(nil) // Make sure future reads fail
+		zr.err = errClosed
+		return nil
+	}
+	return zr.err // Return the persistent error
+}
+
+func (zr *Reader) decodeBlock() []byte {
+	if magic := zr.rd.ReadBitsBE64(48); magic != blkMagic {
+		if magic == endMagic {
+			endCRC := uint32(zr.rd.ReadBitsBE64(32))
+			if internal.GoFuzz {
+				zr.updateChecksum(zr.rd.BitsRead()-32, zr.endCRC)
+				endCRC = zr.endCRC // Suppress CRC failures
+			}
+			if zr.endCRC != endCRC {
+				panicf(errors.Corrupted, "mismatching stream checksum")
+			}
+			zr.endCRC = 0
+			zr.rd.ReadPads()
+			zr.rdHdrFtr++
+			return nil
+		}
+		panicf(errors.Corrupted, "invalid block or footer magic")
+	}
+
+	zr.crc.val = 0
+	zr.blkCRC = uint32(zr.rd.ReadBitsBE64(32))
+	if internal.GoFuzz {
+		zr.updateChecksum(zr.rd.BitsRead()-32, 0) // Record offset only
+	}
+	if zr.rd.ReadBitsBE64(1) != 0 {
+		panicf(errors.Deprecated, "block randomization is not supported")
+	}
+
+	// Read BWT related fields.
+	ptr := int(zr.rd.ReadBitsBE64(24)) // BWT origin pointer
+
+	// Read MTF related fields.
+	var dictArr [256]uint8
+	dict := dictArr[:0]
+	bmapHi := uint16(zr.rd.ReadBits(16))
+	for i := 0; i < 256; i, bmapHi = i+16, bmapHi>>1 {
+		if bmapHi&1 > 0 {
+			bmapLo := uint16(zr.rd.ReadBits(16))
+			for j := 0; j < 16; j, bmapLo = j+1, bmapLo>>1 {
+				if bmapLo&1 > 0 {
+					dict = append(dict, uint8(i+j))
+				}
+			}
+		}
+	}
+
+	// Step 1: Prefix encoding.
+	syms := zr.decodePrefix(len(dict))
+
+	// Step 2: Move-to-front transform and run-length encoding.
+	zr.mtf.Init(dict, zr.level*blockSize)
+	buf := zr.mtf.Decode(syms)
+
+	// Step 3: Burrows-Wheeler transformation.
+	if ptr >= len(buf) {
+		panicf(errors.Corrupted, "origin pointer (0x%06x) exceeds block size: %d", ptr, len(buf))
+	}
+	zr.bwt.Decode(buf, ptr)
+
+	return buf
+}
+
+func (zr *Reader) decodePrefix(numSyms int) (syms []uint16) {
+	numSyms += 2 // Remove 0 symbol, add RUNA, RUNB, and EOF symbols
+	if numSyms < 3 {
+		panicf(errors.Corrupted, "not enough prefix symbols: %d", numSyms)
+	}
+
+	// Read information about the trees and tree selectors.
+	var mtf internal.MoveToFront
+	numTrees := int(zr.rd.ReadBitsBE64(3))
+	if numTrees < minNumTrees || numTrees > maxNumTrees {
+		panicf(errors.Corrupted, "invalid number of prefix trees: %d", numTrees)
+	}
+	numSels := int(zr.rd.ReadBitsBE64(15))
+	if cap(zr.treeSels) < numSels {
+		zr.treeSels = make([]uint8, numSels)
+	}
+	treeSels := zr.treeSels[:numSels]
+	for i := range treeSels {
+		sym, ok := zr.rd.TryReadSymbol(&decSel)
+		if !ok {
+			sym = zr.rd.ReadSymbol(&decSel)
+		}
+		if int(sym) >= numTrees {
+			panicf(errors.Corrupted, "invalid prefix tree selector: %d", sym)
+		}
+		treeSels[i] = uint8(sym)
+	}
+	mtf.Decode(treeSels)
+	zr.treeSels = treeSels
+
+	// Initialize prefix codes.
+	for i := range zr.codes2D[:numTrees] {
+		zr.codes1D[i] = zr.codes2D[i][:numSyms]
+	}
+	zr.rd.ReadPrefixCodes(zr.codes1D[:numTrees], zr.trees1D[:numTrees])
+
+	// Read prefix encoded symbols of compressed data.
+	var tree *prefix.Decoder
+	var blkLen, selIdx int
+	syms = zr.syms[:0]
+	for {
+		if blkLen == 0 {
+			blkLen = numBlockSyms
+			if selIdx >= len(treeSels) {
+				panicf(errors.Corrupted, "not enough prefix tree selectors")
+			}
+			tree = &zr.trees1D[treeSels[selIdx]]
+			selIdx++
+		}
+		blkLen--
+		sym, ok := zr.rd.TryReadSymbol(tree)
+		if !ok {
+			sym = zr.rd.ReadSymbol(tree)
+		}
+
+		if int(sym) == numSyms-1 {
+			break // EOF marker
+		}
+		if int(sym) >= numSyms {
+			panicf(errors.Corrupted, "invalid prefix symbol: %d", sym)
+		}
+		if len(syms) >= zr.level*blockSize {
+			panicf(errors.Corrupted, "number of prefix symbols exceeds block size")
+		}
+		syms = append(syms, uint16(sym))
+	}
+	zr.syms = syms
+	return syms
+}

vendor/github.com/dsnet/compress/bzip2/rle1.go

@@ -0,0 +1,101 @@
+// Copyright 2015, Joe Tsai. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE.md file.
+
+package bzip2
+
+import "github.com/dsnet/compress/internal/errors"
+
+// rleDone is a special "error" to indicate that the RLE stage is done.
+var rleDone = errorf(errors.Unknown, "RLE1 stage is completed")
+
+// runLengthEncoding implements the first RLE stage of bzip2. Every sequence
+// of 4..255 duplicated bytes is replaced by only the first 4 bytes, and a
+// single byte representing the repeat length. Similar to the C bzip2
+// implementation, the encoder will always terminate repeat sequences with a
+// count (even if it is the end of the buffer), and it will also never produce
+// run lengths of 256..259. The decoder can handle the latter case.
+//
+// For example, if the input was:
+//	input:  "AAAAAAABBBBCCCD"
+//
+// Then the output will be:
+//	output: "AAAA\x03BBBB\x00CCCD"
+type runLengthEncoding struct {
+	buf     []byte
+	idx     int
+	lastVal byte
+	lastCnt int
+}
+
+func (rle *runLengthEncoding) Init(buf []byte) {
+	*rle = runLengthEncoding{buf: buf}
+}
+
+func (rle *runLengthEncoding) Write(buf []byte) (int, error) {
+	for i, b := range buf {
+		if rle.lastVal != b {
+			rle.lastCnt = 0
+		}
+		rle.lastCnt++
+		switch {
+		case rle.lastCnt < 4:
+			if rle.idx >= len(rle.buf) {
+				return i, rleDone
+			}
+			rle.buf[rle.idx] = b
+			rle.idx++
+		case rle.lastCnt == 4:
+			if rle.idx+1 >= len(rle.buf) {
+				return i, rleDone
+			}
+			rle.buf[rle.idx] = b
+			rle.idx++
+			rle.buf[rle.idx] = 0
+			rle.idx++
+		case rle.lastCnt < 256:
+			rle.buf[rle.idx-1]++
+		default:
+			if rle.idx >= len(rle.buf) {
+				return i, rleDone
+			}
+			rle.lastCnt = 1
+			rle.buf[rle.idx] = b
+			rle.idx++
+		}
+		rle.lastVal = b
+	}
+	return len(buf), nil
+}
+
+func (rle *runLengthEncoding) Read(buf []byte) (int, error) {
+	for i := range buf {
+		switch {
+		case rle.lastCnt == -4:
+			if rle.idx >= len(rle.buf) {
+				return i, errorf(errors.Corrupted, "missing terminating run-length repeater")
+			}
+			rle.lastCnt = int(rle.buf[rle.idx])
+			rle.idx++
+			if rle.lastCnt > 0 {
+				break // Break the switch
+			}
+			fallthrough // Count was zero, continue the work
+		case rle.lastCnt <= 0:
+			if rle.idx >= len(rle.buf) {
+				return i, rleDone
+			}
+			b := rle.buf[rle.idx]
+			rle.idx++
+			if b != rle.lastVal {
+				rle.lastCnt = 0
+				rle.lastVal = b
+			}
+		}
+		buf[i] = rle.lastVal
+		rle.lastCnt--
+	}
+	return len(buf), nil
+}
+
+func (rle *runLengthEncoding) Bytes() []byte { return rle.buf[:rle.idx] }

vendor/github.com/dsnet/compress/bzip2/writer.go

@@ -0,0 +1,307 @@
+// Copyright 2015, Joe Tsai. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE.md file.
+
+package bzip2
+
+import (
+	"io"
+
+	"github.com/dsnet/compress/internal"
+	"github.com/dsnet/compress/internal/errors"
+	"github.com/dsnet/compress/internal/prefix"
+)
+
+type Writer struct {
+	InputOffset  int64 // Total number of bytes issued to Write
+	OutputOffset int64 // Total number of bytes written to underlying io.Writer
+
+	wr     prefixWriter
+	err    error
+	level  int    // The current compression level
+	wrHdr  bool   // Have we written the stream header?
+	blkCRC uint32 // CRC-32 IEEE of each block
+	endCRC uint32 // Checksum of all blocks using bzip2's custom method
+
+	crc crc
+	rle runLengthEncoding
+	bwt burrowsWheelerTransform
+	mtf moveToFront
+
+	// These fields are allocated with Writer and re-used later.
+	buf         []byte
+	treeSels    []uint8
+	treeSelsMTF []uint8
+	codes2D     [maxNumTrees][maxNumSyms]prefix.PrefixCode
+	codes1D     [maxNumTrees]prefix.PrefixCodes
+	trees1D     [maxNumTrees]prefix.Encoder
+}
+
+type WriterConfig struct {
+	Level int
+
+	_ struct{} // Blank field to prevent unkeyed struct literals
+}
+
+func NewWriter(w io.Writer, conf *WriterConfig) (*Writer, error) {
+	var lvl int
+	if conf != nil {
+		lvl = conf.Level
+	}
+	if lvl == 0 {
+		lvl = DefaultCompression
+	}
+	if lvl < BestSpeed || lvl > BestCompression {
+		return nil, errorf(errors.Invalid, "compression level: %d", lvl)
+	}
+	zw := new(Writer)
+	zw.level = lvl
+	zw.Reset(w)
+	return zw, nil
+}
+
+func (zw *Writer) Reset(w io.Writer) error {
+	*zw = Writer{
+		wr:    zw.wr,
+		level: zw.level,
+
+		rle: zw.rle,
+		bwt: zw.bwt,
+		mtf: zw.mtf,
+
+		buf:         zw.buf,
+		treeSels:    zw.treeSels,
+		treeSelsMTF: zw.treeSelsMTF,
+		trees1D:     zw.trees1D,
+	}
+	zw.wr.Init(w)
+	if len(zw.buf) != zw.level*blockSize {
+		zw.buf = make([]byte, zw.level*blockSize)
+	}
+	zw.rle.Init(zw.buf)
+	return nil
+}
+
+func (zw *Writer) Write(buf []byte) (int, error) {
+	if zw.err != nil {
+		return 0, zw.err
+	}
+
+	cnt := len(buf)
+	for {
+		wrCnt, err := zw.rle.Write(buf)
+		if err != rleDone && zw.err == nil {
+			zw.err = err
+		}
+		zw.crc.update(buf[:wrCnt])
+		buf = buf[wrCnt:]
+		if len(buf) == 0 {
+			zw.InputOffset += int64(cnt)
+			return cnt, nil
+		}
+		if zw.err = zw.flush(); zw.err != nil {
+			return 0, zw.err
+		}
+	}
+}
+
+func (zw *Writer) flush() error {
+	vals := zw.rle.Bytes()
+	if len(vals) == 0 {
+		return nil
+	}
+	zw.wr.Offset = zw.OutputOffset
+	func() {
+		defer errors.Recover(&zw.err)
+		if !zw.wrHdr {
+			// Write stream header.
+			zw.wr.WriteBitsBE64(hdrMagic, 16)
+			zw.wr.WriteBitsBE64('h', 8)
+			zw.wr.WriteBitsBE64(uint64('0'+zw.level), 8)
+			zw.wrHdr = true
+		}
+		zw.encodeBlock(vals)
+	}()
+	var err error
+	if zw.OutputOffset, err = zw.wr.Flush(); zw.err == nil {
+		zw.err = err
+	}
+	if zw.err != nil {
+		zw.err = errWrap(zw.err, errors.Internal)
+		return zw.err
+	}
+	zw.endCRC = (zw.endCRC<<1 | zw.endCRC>>31) ^ zw.blkCRC
+	zw.blkCRC = 0
+	zw.rle.Init(zw.buf)
+	return nil
+}
+
+func (zw *Writer) Close() error {
+	if zw.err == errClosed {
+		return nil
+	}
+
+	// Flush RLE buffer if there is left-over data.
+	if zw.err = zw.flush(); zw.err != nil {
+		return zw.err
+	}
+
+	// Write stream footer.
+	zw.wr.Offset = zw.OutputOffset
+	func() {
+		defer errors.Recover(&zw.err)
+		if !zw.wrHdr {
+			// Write stream header.
+			zw.wr.WriteBitsBE64(hdrMagic, 16)
+			zw.wr.WriteBitsBE64('h', 8)
+			zw.wr.WriteBitsBE64(uint64('0'+zw.level), 8)
+			zw.wrHdr = true
+		}
+		zw.wr.WriteBitsBE64(endMagic, 48)
+		zw.wr.WriteBitsBE64(uint64(zw.endCRC), 32)
+		zw.wr.WritePads(0)
+	}()
+	var err error
+	if zw.OutputOffset, err = zw.wr.Flush(); zw.err == nil {
+		zw.err = err
+	}
+	if zw.err != nil {
+		zw.err = errWrap(zw.err, errors.Internal)
+		return zw.err
+	}
+
+	zw.err = errClosed
+	return nil
+}
+
+func (zw *Writer) encodeBlock(buf []byte) {
+	zw.blkCRC = zw.crc.val
+	zw.wr.WriteBitsBE64(blkMagic, 48)
+	zw.wr.WriteBitsBE64(uint64(zw.blkCRC), 32)
+	zw.wr.WriteBitsBE64(0, 1)
+	zw.crc.val = 0
+
+	// Step 1: Burrows-Wheeler transformation.
+	ptr := zw.bwt.Encode(buf)
+	zw.wr.WriteBitsBE64(uint64(ptr), 24)
+
+	// Step 2: Move-to-front transform and run-length encoding.
+	var dictMap [256]bool
+	for _, c := range buf {
+		dictMap[c] = true
+	}
+
+	var dictArr [256]uint8
+	var bmapLo [16]uint16
+	dict := dictArr[:0]
+	bmapHi := uint16(0)
+	for i, b := range dictMap {
+		if b {
+			c := uint8(i)
+			dict = append(dict, c)
+			bmapHi |= 1 << (c >> 4)
+			bmapLo[c>>4] |= 1 << (c & 0xf)
+		}
+	}
+
+	zw.wr.WriteBits(uint(bmapHi), 16)
+	for _, m := range bmapLo {
+		if m > 0 {
+			zw.wr.WriteBits(uint(m), 16)
+		}
+	}
+
+	zw.mtf.Init(dict, len(buf))
+	syms := zw.mtf.Encode(buf)
+
+	// Step 3: Prefix encoding.
+	zw.encodePrefix(syms, len(dict))
+}
+
+func (zw *Writer) encodePrefix(syms []uint16, numSyms int) {
+	numSyms += 2 // Remove 0 symbol, add RUNA, RUNB, and EOB symbols
+	if numSyms < 3 {
+		panicf(errors.Internal, "unable to encode EOB marker")
+	}
+	syms = append(syms, uint16(numSyms-1)) // EOB marker
+
+	// Compute number of prefix trees needed.
+	numTrees := maxNumTrees
+	for i, lim := range []int{200, 600, 1200, 2400} {
+		if len(syms) < lim {
+			numTrees = minNumTrees + i
+			break
+		}
+	}
+
+	// Compute number of block selectors.
+	numSels := (len(syms) + numBlockSyms - 1) / numBlockSyms
+	if cap(zw.treeSels) < numSels {
+		zw.treeSels = make([]uint8, numSels)
+	}
+	treeSels := zw.treeSels[:numSels]
+	for i := range treeSels {
+		treeSels[i] = uint8(i % numTrees)
+	}
+
+	// Initialize prefix codes.
+	for i := range zw.codes2D[:numTrees] {
+		pc := zw.codes2D[i][:numSyms]
+		for j := range pc {
+			pc[j] = prefix.PrefixCode{Sym: uint32(j)}
+		}
+		zw.codes1D[i] = pc
+	}
+
+	// First cut at assigning prefix trees to each group.
+	var codes prefix.PrefixCodes
+	var blkLen, selIdx int
+	for _, sym := range syms {
+		if blkLen == 0 {
+			blkLen = numBlockSyms
+			codes = zw.codes2D[treeSels[selIdx]][:numSyms]
+			selIdx++
+		}
+		blkLen--
+		codes[sym].Cnt++
+	}
+
+	// TODO(dsnet): Use K-means to cluster groups to each prefix tree.
+
+	// Generate lengths and prefixes based on symbol frequencies.
+	for i := range zw.trees1D[:numTrees] {
+		pc := prefix.PrefixCodes(zw.codes2D[i][:numSyms])
+		pc.SortByCount()
+		if err := prefix.GenerateLengths(pc, maxPrefixBits); err != nil {
+			errors.Panic(err)
+		}
+		pc.SortBySymbol()
+	}
+
+	// Write out information about the trees and tree selectors.
+	var mtf internal.MoveToFront
+	zw.wr.WriteBitsBE64(uint64(numTrees), 3)
+	zw.wr.WriteBitsBE64(uint64(numSels), 15)
+	zw.treeSelsMTF = append(zw.treeSelsMTF[:0], treeSels...)
+	mtf.Encode(zw.treeSelsMTF)
+	for _, sym := range zw.treeSelsMTF {
+		zw.wr.WriteSymbol(uint(sym), &encSel)
+	}
+	zw.wr.WritePrefixCodes(zw.codes1D[:numTrees], zw.trees1D[:numTrees])
+
+	// Write out prefix encoded symbols of compressed data.
+	var tree *prefix.Encoder
+	blkLen, selIdx = 0, 0
+	for _, sym := range syms {
+		if blkLen == 0 {
+			blkLen = numBlockSyms
+			tree = &zw.trees1D[treeSels[selIdx]]
+			selIdx++
+		}
+		blkLen--
+		ok := zw.wr.TryWriteSymbol(uint(sym), tree)
+		if !ok {
+			zw.wr.WriteSymbol(uint(sym), tree)
+		}
+	}
+}

vendor/github.com/dsnet/compress/internal/common.go

@@ -0,0 +1,107 @@
+// Copyright 2015, Joe Tsai. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE.md file.
+
+// Package internal is a collection of common compression algorithms.
+//
+// For performance reasons, these packages lack strong error checking and
+// require that the caller to ensure that strict invariants are kept.
+package internal
+
+var (
+	// IdentityLUT returns the input key itself.
+	IdentityLUT = func() (lut [256]byte) {
+		for i := range lut {
+			lut[i] = uint8(i)
+		}
+		return lut
+	}()
+
+	// ReverseLUT returns the input key with its bits reversed.
+	ReverseLUT = func() (lut [256]byte) {
+		for i := range lut {
+			b := uint8(i)
+			b = (b&0xaa)>>1 | (b&0x55)<<1
+			b = (b&0xcc)>>2 | (b&0x33)<<2
+			b = (b&0xf0)>>4 | (b&0x0f)<<4
+			lut[i] = b
+		}
+		return lut
+	}()
+)
+
+// ReverseUint32 reverses all bits of v.
+func ReverseUint32(v uint32) (x uint32) {
+	x |= uint32(ReverseLUT[byte(v>>0)]) << 24
+	x |= uint32(ReverseLUT[byte(v>>8)]) << 16
+	x |= uint32(ReverseLUT[byte(v>>16)]) << 8
+	x |= uint32(ReverseLUT[byte(v>>24)]) << 0
+	return x
+}
+
+// ReverseUint32N reverses the lower n bits of v.
+func ReverseUint32N(v uint32, n uint) (x uint32) {
+	return ReverseUint32(v << (32 - n))
+}
+
+// ReverseUint64 reverses all bits of v.
+func ReverseUint64(v uint64) (x uint64) {
+	x |= uint64(ReverseLUT[byte(v>>0)]) << 56
+	x |= uint64(ReverseLUT[byte(v>>8)]) << 48
+	x |= uint64(ReverseLUT[byte(v>>16)]) << 40
+	x |= uint64(ReverseLUT[byte(v>>24)]) << 32
+	x |= uint64(ReverseLUT[byte(v>>32)]) << 24
+	x |= uint64(ReverseLUT[byte(v>>40)]) << 16
+	x |= uint64(ReverseLUT[byte(v>>48)]) << 8
+	x |= uint64(ReverseLUT[byte(v>>56)]) << 0
+	return x
+}
+
+// ReverseUint64N reverses the lower n bits of v.
+func ReverseUint64N(v uint64, n uint) (x uint64) {
+	return ReverseUint64(v << (64 - n))
+}
+
+// MoveToFront is a data structure that allows for more efficient move-to-front
+// transformations. This specific implementation assumes that the alphabet is
+// densely packed within 0..255.
+type MoveToFront struct {
+	dict [256]uint8 // Mapping from indexes to values
+	tail int        // Number of tail bytes that are already ordered
+}
+
+func (m *MoveToFront) Encode(vals []uint8) {
+	copy(m.dict[:], IdentityLUT[:256-m.tail]) // Reset dict to be identity
+
+	var max int
+	for i, val := range vals {
+		var idx uint8 // Reverse lookup idx in dict
+		for di, dv := range m.dict {
+			if dv == val {
+				idx = uint8(di)
+				break
+			}
+		}
+		vals[i] = idx
+
+		max |= int(idx)
+		copy(m.dict[1:], m.dict[:idx])
+		m.dict[0] = val
+	}
+	m.tail = 256 - max - 1
+}
+
+func (m *MoveToFront) Decode(idxs []uint8) {
+	copy(m.dict[:], IdentityLUT[:256-m.tail]) // Reset dict to be identity
+
+	var max int
+	for i, idx := range idxs {
+		val := m.dict[idx] // Forward lookup val in dict
+		idxs[i] = val
+
+		max |= int(idx)
+		copy(m.dict[1:], m.dict[:idx])
+		m.dict[0] = val
+	}
+	m.tail = 256 - max - 1
+}

vendor/github.com/dsnet/compress/internal/debug.go

@@ -0,0 +1,12 @@
+// Copyright 2015, Joe Tsai. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE.md file.
+
+// +build debug,!gofuzz
+
+package internal
+
+const (
+	Debug  = true
+	GoFuzz = false
+)

vendor/github.com/dsnet/compress/internal/errors/errors.go

@@ -0,0 +1,120 @@
+// Copyright 2016, Joe Tsai. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE.md file.
+
+// Package errors implements functions to manipulate compression errors.
+//
+// In idiomatic Go, it is an anti-pattern to use panics as a form of error
+// reporting in the API. Instead, the expected way to transmit errors is by
+// returning an error value. Unfortunately, the checking of "err != nil" in
+// tight loops commonly found in compression causes non-negligible performance
+// degradation. While this may not be idiomatic, the internal packages of this
+// repository rely on panics as a normal means to convey errors. In order to
+// ensure that these panics do not leak across the public API, the public
+// packages must recover from these panics and present an error value.
+//
+// The Panic and Recover functions in this package provide a safe way to
+// recover from errors only generated from within this repository.
+//
+// Example usage:
+//	func Foo() (err error) {
+//		defer errors.Recover(&err)
+//
+//		if rand.Intn(2) == 0 {
+//			// Unexpected panics will not be caught by Recover.
+//			io.Closer(nil).Close()
+//		} else {
+//			// Errors thrown by Panic will be caught by Recover.
+//			errors.Panic(errors.New("whoopsie"))
+//		}
+//	}
+//
+package errors
+
+import "strings"
+
+const (
+	// Unknown indicates that there is no classification for this error.
+	Unknown = iota
+
+	// Internal indicates that this error is due to an internal bug.
+	// Users should file a issue report if this type of error is encountered.
+	Internal
+
+	// Invalid indicates that this error is due to the user misusing the API
+	// and is indicative of a bug on the user's part.
+	Invalid
+
+	// Deprecated indicates the use of a deprecated and unsupported feature.
+	Deprecated
+
+	// Corrupted indicates that the input stream is corrupted.
+	Corrupted
+
+	// Closed indicates that the handlers are closed.
+	Closed
+)
+
+var codeMap = map[int]string{
+	Unknown:    "unknown error",
+	Internal:   "internal error",
+	Invalid:    "invalid argument",
+	Deprecated: "deprecated format",
+	Corrupted:  "corrupted input",
+	Closed:     "closed handler",
+}
+
+type Error struct {
+	Code int    // The error type
+	Pkg  string // Name of the package where the error originated
+	Msg  string // Descriptive message about the error (optional)
+}
+
+func (e Error) Error() string {
+	var ss []string
+	for _, s := range []string{e.Pkg, codeMap[e.Code], e.Msg} {
+		if s != "" {
+			ss = append(ss, s)
+		}
+	}
+	return strings.Join(ss, ": ")
+}
+
+func (e Error) CompressError()     {}
+func (e Error) IsInternal() bool   { return e.Code == Internal }
+func (e Error) IsInvalid() bool    { return e.Code == Invalid }
+func (e Error) IsDeprecated() bool { return e.Code == Deprecated }
+func (e Error) IsCorrupted() bool  { return e.Code == Corrupted }
+func (e Error) IsClosed() bool     { return e.Code == Closed }
+
+func IsInternal(err error) bool   { return isCode(err, Internal) }
+func IsInvalid(err error) bool    { return isCode(err, Invalid) }
+func IsDeprecated(err error) bool { return isCode(err, Deprecated) }
+func IsCorrupted(err error) bool  { return isCode(err, Corrupted) }
+func IsClosed(err error) bool     { return isCode(err, Closed) }
+
+func isCode(err error, code int) bool {
+	if cerr, ok := err.(Error); ok && cerr.Code == code {
+		return true
+	}
+	return false
+}
+
+// errWrap is used by Panic and Recover to ensure that only errors raised by
+// Panic are recovered by Recover.
+type errWrap struct{ e *error }
+
+func Recover(err *error) {
+	switch ex := recover().(type) {
+	case nil:
+		// Do nothing.
+	case errWrap:
+		*err = *ex.e
+	default:
+		panic(ex)
+	}
+}
+
+func Panic(err error) {
+	panic(errWrap{&err})
+}

vendor/github.com/dsnet/compress/internal/gofuzz.go

@@ -0,0 +1,12 @@
+// Copyright 2016, Joe Tsai. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE.md file.
+
+// +build gofuzz
+
+package internal
+
+const (
+	Debug  = true
+	GoFuzz = true
+)

vendor/github.com/dsnet/compress/internal/prefix/debug.go

@@ -0,0 +1,159 @@
+// Copyright 2015, Joe Tsai. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE.md file.
+
+// +build debug
+
+package prefix
+
+import (
+	"fmt"
+	"math"
+	"strings"
+)
+
+func max(a, b int) int {
+	if a > b {
+		return a
+	}
+	return b
+}
+
+func lenBase2(n uint) int {
+	return int(math.Ceil(math.Log2(float64(n + 1))))
+}
+func padBase2(v, n uint, m int) string {
+	s := fmt.Sprintf("%b", 1<<n|v)[1:]
+	if pad := m - len(s); pad > 0 {
+		return strings.Repeat(" ", pad) + s
+	}
+	return s
+}
+
+func lenBase10(n int) int {
+	return int(math.Ceil(math.Log10(float64(n + 1))))
+}
+func padBase10(n, m int) string {
+	s := fmt.Sprintf("%d", n)
+	if pad := m - len(s); pad > 0 {
+		return strings.Repeat(" ", pad) + s
+	}
+	return s
+}
+
+func (rc RangeCodes) String() string {
+	var maxLen, maxBase int
+	for _, c := range rc {
+		maxLen = max(maxLen, int(c.Len))
+		maxBase = max(maxBase, int(c.Base))
+	}
+
+	var ss []string
+	ss = append(ss, "{")
+	for i, c := range rc {
+		base := padBase10(int(c.Base), lenBase10(maxBase))
+		if c.Len > 0 {
+			base += fmt.Sprintf("-%d", c.End()-1)
+		}
+		ss = append(ss, fmt.Sprintf("\t%s:  {len: %s, range: %s},",
+			padBase10(int(i), lenBase10(len(rc)-1)),
+			padBase10(int(c.Len), lenBase10(maxLen)),
+			base,
+		))
+	}
+	ss = append(ss, "}")
+	return strings.Join(ss, "\n")
+}
+
+func (pc PrefixCodes) String() string {
+	var maxSym, maxLen, maxCnt int
+	for _, c := range pc {
+		maxSym = max(maxSym, int(c.Sym))
+		maxLen = max(maxLen, int(c.Len))
+		maxCnt = max(maxCnt, int(c.Cnt))
+	}
+
+	var ss []string
+	ss = append(ss, "{")
+	for _, c := range pc {
+		var cntStr string
+		if maxCnt > 0 {
+			cnt := int(32*float32(c.Cnt)/float32(maxCnt) + 0.5)
+			cntStr = fmt.Sprintf("%s |%s",
+				padBase10(int(c.Cnt), lenBase10(maxCnt)),
+				strings.Repeat("#", cnt),
+			)
+		}
+		ss = append(ss, fmt.Sprintf("\t%s:  %s,  %s",
+			padBase10(int(c.Sym), lenBase10(maxSym)),
+			padBase2(uint(c.Val), uint(c.Len), maxLen),
+			cntStr,
+		))
+	}
+	ss = append(ss, "}")
+	return strings.Join(ss, "\n")
+}
+
+func (pd Decoder) String() string {
+	var ss []string
+	ss = append(ss, "{")
+	if len(pd.chunks) > 0 {
+		ss = append(ss, "\tchunks: {")
+		for i, c := range pd.chunks {
+			label := "sym"
+			if uint(c&countMask) > uint(pd.chunkBits) {
+				label = "idx"
+			}
+			ss = append(ss, fmt.Sprintf("\t\t%s:  {%s: %s, len: %s}",
+				padBase2(uint(i), uint(pd.chunkBits), int(pd.chunkBits)),
+				label, padBase10(int(c>>countBits), 3),
+				padBase10(int(c&countMask), 2),
+			))
+		}
+		ss = append(ss, "\t},")
+
+		for j, links := range pd.links {
+			ss = append(ss, fmt.Sprintf("\tlinks[%d]: {", j))
+			linkBits := lenBase2(uint(pd.linkMask))
+			for i, c := range links {
+				ss = append(ss, fmt.Sprintf("\t\t%s:  {sym: %s, len: %s},",
+					padBase2(uint(i), uint(linkBits), int(linkBits)),
+					padBase10(int(c>>countBits), 3),
+					padBase10(int(c&countMask), 2),
+				))
+			}
+			ss = append(ss, "\t},")
+		}
+	}
+	ss = append(ss, fmt.Sprintf("\tchunkMask: %b,", pd.chunkMask))
+	ss = append(ss, fmt.Sprintf("\tlinkMask:  %b,", pd.linkMask))
+	ss = append(ss, fmt.Sprintf("\tchunkBits: %d,", pd.chunkBits))
+	ss = append(ss, fmt.Sprintf("\tMinBits:   %d,", pd.MinBits))
+	ss = append(ss, fmt.Sprintf("\tNumSyms:   %d,", pd.NumSyms))
+	ss = append(ss, "}")
+	return strings.Join(ss, "\n")
+}
+
+func (pe Encoder) String() string {
+	var maxLen int
+	for _, c := range pe.chunks {
+		maxLen = max(maxLen, int(c&countMask))
+	}
+
+	var ss []string
+	ss = append(ss, "{")
+	if len(pe.chunks) > 0 {
+		ss = append(ss, "\tchunks: {")
+		for i, c := range pe.chunks {
+			ss = append(ss, fmt.Sprintf("\t\t%s:  %s,",
+				padBase10(i, 3),
+				padBase2(uint(c>>countBits), uint(c&countMask), maxLen),
+			))
+		}
+		ss = append(ss, "\t},")
+	}
+	ss = append(ss, fmt.Sprintf("\tchunkMask: %b,", pe.chunkMask))
+	ss = append(ss, fmt.Sprintf("\tNumSyms:   %d,", pe.NumSyms))
+	ss = append(ss, "}")
+	return strings.Join(ss, "\n")
+}

vendor/github.com/dsnet/compress/internal/prefix/decoder.go

@@ -0,0 +1,136 @@
+// Copyright 2015, Joe Tsai. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE.md file.
+
+package prefix
+
+import (
+	"sort"
+
+	"github.com/dsnet/compress/internal"
+)
+
+// The algorithm used to decode variable length codes is based on the lookup
+// method in zlib. If the code is less-than-or-equal to maxChunkBits,
+// then the symbol can be decoded using a single lookup into the chunks table.
+// Otherwise, the links table will be used for a second level lookup.
+//
+// The chunks slice is keyed by the contents of the bit buffer ANDed with
+// the chunkMask to avoid a out-of-bounds lookup. The value of chunks is a tuple
+// that is decoded as follow:
+//
+//	var length = chunks[bitBuffer&chunkMask] & countMask
+//	var symbol = chunks[bitBuffer&chunkMask] >> countBits
+//
+// If the decoded length is larger than chunkBits, then an overflow link table
+// must be used for further decoding. In this case, the symbol is actually the
+// index into the links tables. The second-level links table returned is
+// processed in the same way as the chunks table.
+//
+//	if length > chunkBits {
+//		var index = symbol // Previous symbol is index into links tables
+//		length = links[index][bitBuffer>>chunkBits & linkMask] & countMask
+//		symbol = links[index][bitBuffer>>chunkBits & linkMask] >> countBits
+//	}
+//
+// See the following:
+//	http://www.gzip.org/algorithm.txt
+
+type Decoder struct {
+	chunks    []uint32   // First-level lookup map
+	links     [][]uint32 // Second-level lookup map
+	chunkMask uint32     // Mask the length of the chunks table
+	linkMask  uint32     // Mask the length of the link table
+	chunkBits uint32     // Bit-length of the chunks table
+
+	MinBits uint32 // The minimum number of bits to safely make progress
+	NumSyms uint32 // Number of symbols
+}
+
+// Init initializes Decoder according to the codes provided.
+func (pd *Decoder) Init(codes PrefixCodes) {
+	// Handle special case trees.
+	if len(codes) <= 1 {
+		switch {
+		case len(codes) == 0: // Empty tree (should error if used later)
+			*pd = Decoder{chunks: pd.chunks[:0], links: pd.links[:0], NumSyms: 0}
+		case len(codes) == 1 && codes[0].Len == 0: // Single code tree (bit-length of zero)
+			pd.chunks = append(pd.chunks[:0], codes[0].Sym<<countBits|0)
+			*pd = Decoder{chunks: pd.chunks[:1], links: pd.links[:0], NumSyms: 1}
+		default:
+			panic("invalid codes")
+		}
+		return
+	}
+	if internal.Debug && !sort.IsSorted(prefixCodesBySymbol(codes)) {
+		panic("input codes is not sorted")
+	}
+	if internal.Debug && !(codes.checkLengths() && codes.checkPrefixes()) {
+		panic("detected incomplete or overlapping codes")
+	}
+
+	var minBits, maxBits uint32 = valueBits, 0
+	for _, c := range codes {
+		if minBits > c.Len {
+			minBits = c.Len
+		}
+		if maxBits < c.Len {
+			maxBits = c.Len
+		}
+	}
+
+	// Allocate chunks table as needed.
+	const maxChunkBits = 9 // This can be tuned for better performance
+	pd.NumSyms = uint32(len(codes))
+	pd.MinBits = minBits
+	pd.chunkBits = maxBits
+	if pd.chunkBits > maxChunkBits {
+		pd.chunkBits = maxChunkBits
+	}
+	numChunks := 1 << pd.chunkBits
+	pd.chunks = allocUint32s(pd.chunks, numChunks)
+	pd.chunkMask = uint32(numChunks - 1)
+
+	// Allocate links tables as needed.
+	pd.links = pd.links[:0]
+	pd.linkMask = 0
+	if pd.chunkBits < maxBits {
+		numLinks := 1 << (maxBits - pd.chunkBits)
+		pd.linkMask = uint32(numLinks - 1)
+
+		var linkIdx uint32
+		for i := range pd.chunks {
+			pd.chunks[i] = 0 // Logic below relies on zero value as uninitialized
+		}
+		for _, c := range codes {
+			if c.Len > pd.chunkBits && pd.chunks[c.Val&pd.chunkMask] == 0 {
+				pd.chunks[c.Val&pd.chunkMask] = (linkIdx << countBits) | (pd.chunkBits + 1)
+				linkIdx++
+			}
+		}
+
+		pd.links = extendSliceUint32s(pd.links, int(linkIdx))
+		linksFlat := allocUint32s(pd.links[0], numLinks*int(linkIdx))
+		for i, j := 0, 0; i < len(pd.links); i, j = i+1, j+numLinks {
+			pd.links[i] = linksFlat[j : j+numLinks]
+		}
+	}
+
+	// Fill out chunks and links tables with values.
+	for _, c := range codes {
+		chunk := c.Sym<<countBits | c.Len
+		if c.Len <= pd.chunkBits {
+			skip := 1 << uint(c.Len)
+			for j := int(c.Val); j < len(pd.chunks); j += skip {
+				pd.chunks[j] = chunk
+			}
+		} else {
+			linkIdx := pd.chunks[c.Val&pd.chunkMask] >> countBits
+			links := pd.links[linkIdx]
+			skip := 1 << uint(c.Len-pd.chunkBits)
+			for j := int(c.Val >> pd.chunkBits); j < len(links); j += skip {
+				links[j] = chunk
+			}
+		}
+	}
+}

vendor/github.com/dsnet/compress/internal/prefix/encoder.go

@@ -0,0 +1,66 @@
+// Copyright 2015, Joe Tsai. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE.md file.
+
+package prefix
+
+import (
+	"sort"
+
+	"github.com/dsnet/compress/internal"
+)
+
+type Encoder struct {
+	chunks    []uint32 // First-level lookup map
+	chunkMask uint32   // Mask the length of the chunks table
+
+	NumSyms uint32 // Number of symbols
+}
+
+// Init initializes Encoder according to the codes provided.
+func (pe *Encoder) Init(codes PrefixCodes) {
+	// Handle special case trees.
+	if len(codes) <= 1 {
+		switch {
+		case len(codes) == 0: // Empty tree (should error if used later)
+			*pe = Encoder{chunks: pe.chunks[:0], NumSyms: 0}
+		case len(codes) == 1 && codes[0].Len == 0: // Single code tree (bit-length of zero)
+			pe.chunks = append(pe.chunks[:0], codes[0].Val<<countBits|0)
+			*pe = Encoder{chunks: pe.chunks[:1], NumSyms: 1}
+		default:
+			panic("invalid codes")
+		}
+		return
+	}
+	if internal.Debug && !sort.IsSorted(prefixCodesBySymbol(codes)) {
+		panic("input codes is not sorted")
+	}
+	if internal.Debug && !(codes.checkLengths() && codes.checkPrefixes()) {
+		panic("detected incomplete or overlapping codes")
+	}
+
+	// Enough chunks to contain all the symbols.
+	numChunks := 1
+	for n := len(codes) - 1; n > 0; n >>= 1 {
+		numChunks <<= 1
+	}
+	pe.NumSyms = uint32(len(codes))
+
+retry:
+	// Allocate and reset chunks.
+	pe.chunks = allocUint32s(pe.chunks, numChunks)
+	pe.chunkMask = uint32(numChunks - 1)
+	for i := range pe.chunks {
+		pe.chunks[i] = 0 // Logic below relies on zero value as uninitialized
+	}
+
+	// Insert each symbol, checking that there are no conflicts.
+	for _, c := range codes {
+		if pe.chunks[c.Sym&pe.chunkMask] > 0 {
+			// Collision found our "hash" table, so grow and try again.
+			numChunks <<= 1
+			goto retry
+		}
+		pe.chunks[c.Sym&pe.chunkMask] = c.Val<<countBits | c.Len
+	}
+}

vendor/github.com/dsnet/compress/internal/prefix/prefix.go

@@ -0,0 +1,400 @@
+// Copyright 2015, Joe Tsai. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE.md file.
+
+// Package prefix implements bit readers and writers that use prefix encoding.
+package prefix
+
+import (
+	"fmt"
+	"sort"
+
+	"github.com/dsnet/compress/internal"
+	"github.com/dsnet/compress/internal/errors"
+)
+
+func errorf(c int, f string, a ...interface{}) error {
+	return errors.Error{Code: c, Pkg: "prefix", Msg: fmt.Sprintf(f, a...)}
+}
+
+func panicf(c int, f string, a ...interface{}) {
+	errors.Panic(errorf(c, f, a...))
+}
+
+const (
+	countBits = 5  // Number of bits to store the bit-length of the code
+	valueBits = 27 // Number of bits to store the code value
+
+	countMask = (1 << countBits) - 1
+)
+
+// PrefixCode is a representation of a prefix code, which is conceptually a
+// mapping from some arbitrary symbol to some bit-string.
+//
+// The Sym and Cnt fields are typically provided by the user,
+// while the Len and Val fields are generated by this package.
+type PrefixCode struct {
+	Sym uint32 // The symbol being mapped
+	Cnt uint32 // The number times this symbol is used
+	Len uint32 // Bit-length of the prefix code
+	Val uint32 // Value of the prefix code (must be in 0..(1<<Len)-1)
+}
+type PrefixCodes []PrefixCode
+
+type prefixCodesBySymbol []PrefixCode
+
+func (c prefixCodesBySymbol) Len() int           { return len(c) }
+func (c prefixCodesBySymbol) Less(i, j int) bool { return c[i].Sym < c[j].Sym }
+func (c prefixCodesBySymbol) Swap(i, j int)      { c[i], c[j] = c[j], c[i] }
+
+type prefixCodesByCount []PrefixCode
+
+func (c prefixCodesByCount) Len() int { return len(c) }
+func (c prefixCodesByCount) Less(i, j int) bool {
+	return c[i].Cnt < c[j].Cnt || (c[i].Cnt == c[j].Cnt && c[i].Sym < c[j].Sym)
+}
+func (c prefixCodesByCount) Swap(i, j int) { c[i], c[j] = c[j], c[i] }
+
+func (pc PrefixCodes) SortBySymbol() { sort.Sort(prefixCodesBySymbol(pc)) }
+func (pc PrefixCodes) SortByCount()  { sort.Sort(prefixCodesByCount(pc)) }
+
+// Length computes the total bit-length using the Len and Cnt fields.
+func (pc PrefixCodes) Length() (nb uint) {
+	for _, c := range pc {
+		nb += uint(c.Len * c.Cnt)
+	}
+	return nb
+}
+
+// checkLengths reports whether the codes form a complete prefix tree.
+func (pc PrefixCodes) checkLengths() bool {
+	sum := 1 << valueBits
+	for _, c := range pc {
+		sum -= (1 << valueBits) >> uint(c.Len)
+	}
+	return sum == 0 || len(pc) == 0
+}
+
+// checkPrefixes reports whether all codes have non-overlapping prefixes.
+func (pc PrefixCodes) checkPrefixes() bool {
+	for i, c1 := range pc {
+		for j, c2 := range pc {
+			mask := uint32(1)<<c1.Len - 1
+			if i != j && c1.Len <= c2.Len && c1.Val&mask == c2.Val&mask {
+				return false
+			}
+		}
+	}
+	return true
+}
+
+// checkCanonical reports whether all codes are canonical.
+// That is, they have the following properties:
+//
+//	1. All codes of a given bit-length are consecutive values.
+//	2. Shorter codes lexicographically precede longer codes.
+//
+// The codes must have unique symbols and be sorted by the symbol
+// The Len and Val fields in each code must be populated.
+func (pc PrefixCodes) checkCanonical() bool {
+	// Rule 1.
+	var vals [valueBits + 1]PrefixCode
+	for _, c := range pc {
+		if c.Len > 0 {
+			c.Val = internal.ReverseUint32N(c.Val, uint(c.Len))
+			if vals[c.Len].Cnt > 0 && vals[c.Len].Val+1 != c.Val {
+				return false
+			}
+			vals[c.Len].Val = c.Val
+			vals[c.Len].Cnt++
+		}
+	}
+
+	// Rule 2.
+	var last PrefixCode
+	for _, v := range vals {
+		if v.Cnt > 0 {
+			curVal := v.Val - v.Cnt + 1
+			if last.Cnt != 0 && last.Val >= curVal {
+				return false
+			}
+			last = v
+		}
+	}
+	return true
+}
+
+// GenerateLengths assigns non-zero bit-lengths to all codes. Codes with high
+// frequency counts will be assigned shorter codes to reduce bit entropy.
+// This function is used primarily by compressors.
+//
+// The input codes must have the Cnt field populated, be sorted by count.
+// Even if a code has a count of 0, a non-zero bit-length will be assigned.
+//
+// The result will have the Len field populated. The algorithm used guarantees
+// that Len <= maxBits and that it is a complete prefix tree. The resulting
+// codes will remain sorted by count.
+func GenerateLengths(codes PrefixCodes, maxBits uint) error {
+	if len(codes) <= 1 {
+		if len(codes) == 1 {
+			codes[0].Len = 0
+		}
+		return nil
+	}
+
+	// Verify that the codes are in ascending order by count.
+	cntLast := codes[0].Cnt
+	for _, c := range codes[1:] {
+		if c.Cnt < cntLast {
+			return errorf(errors.Invalid, "non-monotonically increasing symbol counts")
+		}
+		cntLast = c.Cnt
+	}
+
+	// Construct a Huffman tree used to generate the bit-lengths.
+	//
+	// The Huffman tree is a binary tree where each symbol lies as a leaf node
+	// on this tree. The length of the prefix code to assign is the depth of
+	// that leaf from the root. The Huffman algorithm, which runs in O(n),
+	// is used to generate the tree. It assumes that codes are sorted in
+	// increasing order of frequency.
+	//
+	// The algorithm is as follows:
+	//	1. Start with two queues, F and Q, where F contains all of the starting
+	//	symbols sorted such that symbols with lowest counts come first.
+	//	2. While len(F)+len(Q) > 1:
+	//		2a. Dequeue the node from F or Q that has the lowest weight as N0.
+	//		2b. Dequeue the node from F or Q that has the lowest weight as N1.
+	//		2c. Create a new node N that has N0 and N1 as its children.
+	//		2d. Enqueue N into the back of Q.
+	//	3. The tree's root node is Q[0].
+	type node struct {
+		cnt uint32
+
+		// n0 or c0 represent the left child of this node.
+		// Since Go does not have unions, only one of these will be set.
+		// Similarly, n1 or c1 represent the right child of this node.
+		//
+		// If n0 or n1 is set, then it represents a "pointer" to another
+		// node in the Huffman tree. Since Go's pointer analysis cannot reason
+		// that these node pointers do not escape (golang.org/issue/13493),
+		// we use an index to a node in the nodes slice as a pseudo-pointer.
+		//
+		// If c0 or c1 is set, then it represents a leaf "node" in the
+		// Huffman tree. The leaves are the PrefixCode values themselves.
+		n0, n1 int // Index to child nodes
+		c0, c1 *PrefixCode
+	}
+	var nodeIdx int
+	var nodeArr [1024]node // Large enough to handle most cases on the stack
+	nodes := nodeArr[:]
+	if len(nodes) < len(codes) {
+		nodes = make([]node, len(codes)) // Number of internal nodes < number of leaves
+	}
+	freqs, queue := codes, nodes[:0]
+	for len(freqs)+len(queue) > 1 {
+		// These are the two smallest nodes at the front of freqs and queue.
+		var n node
+		if len(queue) == 0 || (len(freqs) > 0 && freqs[0].Cnt <= queue[0].cnt) {
+			n.c0, freqs = &freqs[0], freqs[1:]
+			n.cnt += n.c0.Cnt
+		} else {
+			n.cnt += queue[0].cnt
+			n.n0 = nodeIdx // nodeIdx is same as &queue[0] - &nodes[0]
+			nodeIdx++
+			queue = queue[1:]
+		}
+		if len(queue) == 0 || (len(freqs) > 0 && freqs[0].Cnt <= queue[0].cnt) {
+			n.c1, freqs = &freqs[0], freqs[1:]
+			n.cnt += n.c1.Cnt
+		} else {
+			n.cnt += queue[0].cnt
+			n.n1 = nodeIdx // nodeIdx is same as &queue[0] - &nodes[0]
+			nodeIdx++
+			queue = queue[1:]
+		}
+		queue = append(queue, n)
+	}
+	rootIdx := nodeIdx
+
+	// Search the whole binary tree, noting when we hit each leaf node.
+	// We do not care about the exact Huffman tree structure, but rather we only
+	// care about depth of each of the leaf nodes. That is, the depth determines
+	// how long each symbol is in bits.
+	//
+	// Since the number of leaves is n, there is at most n internal nodes.
+	// Thus, this algorithm runs in O(n).
+	var fixBits bool
+	var explore func(int, uint)
+	explore = func(rootIdx int, level uint) {
+		root := &nodes[rootIdx]
+
+		// Explore left branch.
+		if root.c0 == nil {
+			explore(root.n0, level+1)
+		} else {
+			fixBits = fixBits || (level > maxBits)
+			root.c0.Len = uint32(level)
+		}
+
+		// Explore right branch.
+		if root.c1 == nil {
+			explore(root.n1, level+1)
+		} else {
+			fixBits = fixBits || (level > maxBits)
+			root.c1.Len = uint32(level)
+		}
+	}
+	explore(rootIdx, 1)
+
+	// Fix the bit-lengths if we violate the maxBits requirement.
+	if fixBits {
+		// Create histogram for number of symbols with each bit-length.
+		var symBitsArr [valueBits + 1]uint32
+		symBits := symBitsArr[:] // symBits[nb] indicates number of symbols using nb bits
+		for _, c := range codes {
+			for int(c.Len) >= len(symBits) {
+				symBits = append(symBits, 0)
+			}
+			symBits[c.Len]++
+		}
+
+		// Fudge the tree such that the largest bit-length is <= maxBits.
+		// This is accomplish by effectively doing a tree rotation. That is, we
+		// increase the bit-length of some higher frequency code, so that the
+		// bit-lengths of lower frequency codes can be decreased.
+		//
+		// Visually, this looks like the following transform:
+		//
+		//	Level   Before       After
+		//	          __          ___
+		//	         /  \        /   \
+		//	 n-1    X  / \      /\   /\
+		//	 n        X  /\    X  X X  X
+		//	 n+1        X  X
+		//
+		var treeRotate func(uint)
+		treeRotate = func(nb uint) {
+			if symBits[nb-1] == 0 {
+				treeRotate(nb - 1)
+			}
+			symBits[nb-1] -= 1 // Push this node to the level below
+			symBits[nb] += 3   // This level gets one node from above, two from below
+			symBits[nb+1] -= 2 // Push two nodes to the level above
+		}
+		for i := uint(len(symBits)) - 1; i > maxBits; i-- {
+			for symBits[i] > 0 {
+				treeRotate(i - 1)
+			}
+		}
+
+		// Assign bit-lengths to each code. Since codes is sorted in increasing
+		// order of frequency, that means that the most frequently used symbols
+		// should have the shortest bit-lengths. Thus, we copy symbols to codes
+		// from the back of codes first.
+		cs := codes
+		for nb, cnt := range symBits {
+			if cnt > 0 {
+				pos := len(cs) - int(cnt)
+				cs2 := cs[pos:]
+				for i := range cs2 {
+					cs2[i].Len = uint32(nb)
+				}
+				cs = cs[:pos]
+			}
+		}
+		if len(cs) != 0 {
+			panic("not all codes were used up")
+		}
+	}
+
+	if internal.Debug && !codes.checkLengths() {
+		panic("incomplete prefix tree detected")
+	}
+	return nil
+}
+
+// GeneratePrefixes assigns a prefix value to all codes according to the
+// bit-lengths. This function is used by both compressors and decompressors.
+//
+// The input codes must have the Sym and Len fields populated and be
+// sorted by symbol. The bit-lengths of each code must be properly allocated,
+// such that it forms a complete tree.
+//
+// The result will have the Val field populated and will produce a canonical
+// prefix tree. The resulting codes will remain sorted by symbol.
+func GeneratePrefixes(codes PrefixCodes) error {
+	if len(codes) <= 1 {
+		if len(codes) == 1 {
+			if codes[0].Len != 0 {
+				return errorf(errors.Invalid, "degenerate prefix tree with one node")
+			}
+			codes[0].Val = 0
+		}
+		return nil
+	}
+
+	// Compute basic statistics on the symbols.
+	var bitCnts [valueBits + 1]uint
+	c0 := codes[0]
+	bitCnts[c0.Len]++
+	minBits, maxBits, symLast := c0.Len, c0.Len, c0.Sym
+	for _, c := range codes[1:] {
+		if c.Sym <= symLast {
+			return errorf(errors.Invalid, "non-unique or non-monotonically increasing symbols")
+		}
+		if minBits > c.Len {
+			minBits = c.Len
+		}
+		if maxBits < c.Len {
+			maxBits = c.Len
+		}
+		bitCnts[c.Len]++ // Histogram of bit counts
+		symLast = c.Sym  // Keep track of last symbol
+	}
+	if minBits == 0 {
+		return errorf(errors.Invalid, "invalid prefix bit-length")
+	}
+
+	// Compute the next code for a symbol of a given bit length.
+	var nextCodes [valueBits + 1]uint
+	var code uint
+	for i := minBits; i <= maxBits; i++ {
+		code <<= 1
+		nextCodes[i] = code
+		code += bitCnts[i]
+	}
+	if code != 1<<maxBits {
+		return errorf(errors.Invalid, "degenerate prefix tree")
+	}
+
+	// Assign the code to each symbol.
+	for i, c := range codes {
+		codes[i].Val = internal.ReverseUint32N(uint32(nextCodes[c.Len]), uint(c.Len))
+		nextCodes[c.Len]++
+	}
+
+	if internal.Debug && !codes.checkPrefixes() {
+		panic("overlapping prefixes detected")
+	}
+	if internal.Debug && !codes.checkCanonical() {
+		panic("non-canonical prefixes detected")
+	}
+	return nil
+}
+
+func allocUint32s(s []uint32, n int) []uint32 {
+	if cap(s) >= n {
+		return s[:n]
+	}
+	return make([]uint32, n, n*3/2)
+}
+
+func extendSliceUint32s(s [][]uint32, n int) [][]uint32 {
+	if cap(s) >= n {
+		return s[:n]
+	}
+	ss := make([][]uint32, n, n*3/2)
+	copy(ss, s[:cap(s)])
+	return ss
+}

vendor/github.com/dsnet/compress/internal/prefix/range.go

@@ -0,0 +1,93 @@
+// Copyright 2015, Joe Tsai. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE.md file.
+
+package prefix
+
+type RangeCode struct {
+	Base uint32 // Starting base offset of the range
+	Len  uint32 // Bit-length of a subsequent integer to add to base offset
+}
+type RangeCodes []RangeCode
+
+type RangeEncoder struct {
+	rcs     RangeCodes
+	lut     [1024]uint32
+	minBase uint
+}
+
+// End reports the non-inclusive ending range.
+func (rc RangeCode) End() uint32 { return rc.Base + (1 << rc.Len) }
+
+// MakeRangeCodes creates a RangeCodes, where each region is assumed to be
+// contiguously stacked, without any gaps, with bit-lengths taken from bits.
+func MakeRangeCodes(minBase uint, bits []uint) (rc RangeCodes) {
+	for _, nb := range bits {
+		rc = append(rc, RangeCode{Base: uint32(minBase), Len: uint32(nb)})
+		minBase += 1 << nb
+	}
+	return rc
+}
+
+// Base reports the inclusive starting range for all ranges.
+func (rcs RangeCodes) Base() uint32 { return rcs[0].Base }
+
+// End reports the non-inclusive ending range for all ranges.
+func (rcs RangeCodes) End() uint32 { return rcs[len(rcs)-1].End() }
+
+// checkValid reports whether the RangeCodes is valid. In order to be valid,
+// the following must hold true:
+//	rcs[i-1].Base <= rcs[i].Base
+//	rcs[i-1].End  <= rcs[i].End
+//	rcs[i-1].End  >= rcs[i].Base
+//
+// Practically speaking, each range must be increasing and must not have any
+// gaps in between. It is okay for ranges to overlap.
+func (rcs RangeCodes) checkValid() bool {
+	if len(rcs) == 0 {
+		return false
+	}
+	pre := rcs[0]
+	for _, cur := range rcs[1:] {
+		preBase, preEnd := pre.Base, pre.End()
+		curBase, curEnd := cur.Base, cur.End()
+		if preBase > curBase || preEnd > curEnd || preEnd < curBase {
+			return false
+		}
+		pre = cur
+	}
+	return true
+}
+
+func (re *RangeEncoder) Init(rcs RangeCodes) {
+	if !rcs.checkValid() {
+		panic("invalid range codes")
+	}
+	*re = RangeEncoder{rcs: rcs, minBase: uint(rcs.Base())}
+	for sym, rc := range rcs {
+		base := int(rc.Base) - int(re.minBase)
+		end := int(rc.End()) - int(re.minBase)
+		if base >= len(re.lut) {
+			break
+		}
+		if end > len(re.lut) {
+			end = len(re.lut)
+		}
+		for i := base; i < end; i++ {
+			re.lut[i] = uint32(sym)
+		}
+	}
+}
+
+func (re *RangeEncoder) Encode(offset uint) (sym uint) {
+	if idx := int(offset - re.minBase); idx < len(re.lut) {
+		return uint(re.lut[idx])
+	}
+	sym = uint(re.lut[len(re.lut)-1])
+retry:
+	if int(sym) >= len(re.rcs) || re.rcs[sym].Base > uint32(offset) {
+		return sym - 1
+	}
+	sym++
+	goto retry // Avoid for-loop so that this function can be inlined
+}

vendor/github.com/dsnet/compress/internal/prefix/reader.go

@@ -0,0 +1,335 @@
+// Copyright 2015, Joe Tsai. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE.md file.
+
+package prefix
+
+import (
+	"bufio"
+	"bytes"
+	"encoding/binary"
+	"io"
+	"strings"
+
+	"github.com/dsnet/compress"
+	"github.com/dsnet/compress/internal"
+	"github.com/dsnet/compress/internal/errors"
+)
+
+// Reader implements a prefix decoder. If the input io.Reader satisfies the
+// compress.ByteReader or compress.BufferedReader interface, then it also
+// guarantees that it will never read more bytes than is necessary.
+//
+// For high performance, provide an io.Reader that satisfies the
+// compress.BufferedReader interface. If the input does not satisfy either
+// compress.ByteReader or compress.BufferedReader, then it will be internally
+// wrapped with a bufio.Reader.
+type Reader struct {
+	Offset int64 // Number of bytes read from the underlying io.Reader
+
+	rd     io.Reader
+	byteRd compress.ByteReader     // Set if rd is a ByteReader
+	bufRd  compress.BufferedReader // Set if rd is a BufferedReader
+
+	bufBits   uint64 // Buffer to hold some bits
+	numBits   uint   // Number of valid bits in bufBits
+	bigEndian bool   // Do we treat input bytes as big endian?
+
+	// These fields are only used if rd is a compress.BufferedReader.
+	bufPeek     []byte // Buffer for the Peek data
+	discardBits int    // Number of bits to discard from reader
+	fedBits     uint   // Number of bits fed in last call to PullBits
+
+	// These fields are used to reduce allocations.
+	bb *buffer
+	br *bytesReader
+	sr *stringReader
+	bu *bufio.Reader
+}
+
+// Init initializes the bit Reader to read from r. If bigEndian is true, then
+// bits will be read starting from the most-significant bits of a byte
+// (as done in bzip2), otherwise it will read starting from the
+// least-significant bits of a byte (such as for deflate and brotli).
+func (pr *Reader) Init(r io.Reader, bigEndian bool) {
+	*pr = Reader{
+		rd:        r,
+		bigEndian: bigEndian,
+
+		bb: pr.bb,
+		br: pr.br,
+		sr: pr.sr,
+		bu: pr.bu,
+	}
+	switch rr := r.(type) {
+	case *bytes.Buffer:
+		if pr.bb == nil {
+			pr.bb = new(buffer)
+		}
+		*pr.bb = buffer{Buffer: rr}
+		pr.bufRd = pr.bb
+	case *bytes.Reader:
+		if pr.br == nil {
+			pr.br = new(bytesReader)
+		}
+		*pr.br = bytesReader{Reader: rr}
+		pr.bufRd = pr.br
+	case *strings.Reader:
+		if pr.sr == nil {
+			pr.sr = new(stringReader)
+		}
+		*pr.sr = stringReader{Reader: rr}
+		pr.bufRd = pr.sr
+	case compress.BufferedReader:
+		pr.bufRd = rr
+	case compress.ByteReader:
+		pr.byteRd = rr
+	default:
+		if pr.bu == nil {
+			pr.bu = bufio.NewReader(nil)
+		}
+		pr.bu.Reset(r)
+		pr.rd, pr.bufRd = pr.bu, pr.bu
+	}
+}
+
+// BitsRead reports the total number of bits emitted from any Read method.
+func (pr *Reader) BitsRead() int64 {
+	offset := 8*pr.Offset - int64(pr.numBits)
+	if pr.bufRd != nil {
+		discardBits := pr.discardBits + int(pr.fedBits-pr.numBits)
+		offset = 8*pr.Offset + int64(discardBits)
+	}
+	return offset
+}
+
+// IsBufferedReader reports whether the underlying io.Reader is also a
+// compress.BufferedReader.
+func (pr *Reader) IsBufferedReader() bool {
+	return pr.bufRd != nil
+}
+
+// ReadPads reads 0-7 bits from the bit buffer to achieve byte-alignment.
+func (pr *Reader) ReadPads() uint {
+	nb := pr.numBits % 8
+	val := uint(pr.bufBits & uint64(1<<nb-1))
+	pr.bufBits >>= nb
+	pr.numBits -= nb
+	return val
+}
+
+// Read reads bytes into buf.
+// The bit-ordering mode does not affect this method.
+func (pr *Reader) Read(buf []byte) (cnt int, err error) {
+	if pr.numBits > 0 {
+		if pr.numBits%8 != 0 {
+			return 0, errorf(errors.Invalid, "non-aligned bit buffer")
+		}
+		for cnt = 0; len(buf) > cnt && pr.numBits > 0; cnt++ {
+			if pr.bigEndian {
+				buf[cnt] = internal.ReverseLUT[byte(pr.bufBits)]
+			} else {
+				buf[cnt] = byte(pr.bufBits)
+			}
+			pr.bufBits >>= 8
+			pr.numBits -= 8
+		}
+		return cnt, nil
+	}
+	if _, err := pr.Flush(); err != nil {
+		return 0, err
+	}
+	cnt, err = pr.rd.Read(buf)
+	pr.Offset += int64(cnt)
+	return cnt, err
+}
+
+// ReadOffset reads an offset value using the provided RangeCodes indexed by
+// the symbol read.
+func (pr *Reader) ReadOffset(pd *Decoder, rcs RangeCodes) uint {
+	rc := rcs[pr.ReadSymbol(pd)]
+	return uint(rc.Base) + pr.ReadBits(uint(rc.Len))
+}
+
+// TryReadBits attempts to read nb bits using the contents of the bit buffer
+// alone. It returns the value and whether it succeeded.
+//
+// This method is designed to be inlined for performance reasons.
+func (pr *Reader) TryReadBits(nb uint) (uint, bool) {
+	if pr.numBits < nb {
+		return 0, false
+	}
+	val := uint(pr.bufBits & uint64(1<<nb-1))
+	pr.bufBits >>= nb
+	pr.numBits -= nb
+	return val, true
+}
+
+// ReadBits reads nb bits in from the underlying reader.
+func (pr *Reader) ReadBits(nb uint) uint {
+	if err := pr.PullBits(nb); err != nil {
+		errors.Panic(err)
+	}
+	val := uint(pr.bufBits & uint64(1<<nb-1))
+	pr.bufBits >>= nb
+	pr.numBits -= nb
+	return val
+}
+
+// TryReadSymbol attempts to decode the next symbol using the contents of the
+// bit buffer alone. It returns the decoded symbol and whether it succeeded.
+//
+// This method is designed to be inlined for performance reasons.
+func (pr *Reader) TryReadSymbol(pd *Decoder) (uint, bool) {
+	if pr.numBits < uint(pd.MinBits) || len(pd.chunks) == 0 {
+		return 0, false
+	}
+	chunk := pd.chunks[uint32(pr.bufBits)&pd.chunkMask]
+	nb := uint(chunk & countMask)
+	if nb > pr.numBits || nb > uint(pd.chunkBits) {
+		return 0, false
+	}
+	pr.bufBits >>= nb
+	pr.numBits -= nb
+	return uint(chunk >> countBits), true
+}
+
+// ReadSymbol reads the next symbol using the provided prefix Decoder.
+func (pr *Reader) ReadSymbol(pd *Decoder) uint {
+	if len(pd.chunks) == 0 {
+		panicf(errors.Invalid, "decode with empty prefix tree")
+	}
+
+	nb := uint(pd.MinBits)
+	for {
+		if err := pr.PullBits(nb); err != nil {
+			errors.Panic(err)
+		}
+		chunk := pd.chunks[uint32(pr.bufBits)&pd.chunkMask]
+		nb = uint(chunk & countMask)
+		if nb > uint(pd.chunkBits) {
+			linkIdx := chunk >> countBits
+			chunk = pd.links[linkIdx][uint32(pr.bufBits>>pd.chunkBits)&pd.linkMask]
+			nb = uint(chunk & countMask)
+		}
+		if nb <= pr.numBits {
+			pr.bufBits >>= nb
+			pr.numBits -= nb
+			return uint(chunk >> countBits)
+		}
+	}
+}
+
+// Flush updates the read offset of the underlying ByteReader.
+// If reader is a compress.BufferedReader, then this calls Discard to update
+// the read offset.
+func (pr *Reader) Flush() (int64, error) {
+	if pr.bufRd == nil {
+		return pr.Offset, nil
+	}
+
+	// Update the number of total bits to discard.
+	pr.discardBits += int(pr.fedBits - pr.numBits)
+	pr.fedBits = pr.numBits
+
+	// Discard some bytes to update read offset.
+	var err error
+	nd := (pr.discardBits + 7) / 8 // Round up to nearest byte
+	nd, err = pr.bufRd.Discard(nd)
+	pr.discardBits -= nd * 8 // -7..0
+	pr.Offset += int64(nd)
+
+	// These are invalid after Discard.
+	pr.bufPeek = nil
+	return pr.Offset, err
+}
+
+// PullBits ensures that at least nb bits exist in the bit buffer.
+// If the underlying reader is a compress.BufferedReader, then this will fill
+// the bit buffer with as many bits as possible, relying on Peek and Discard to
+// properly advance the read offset. Otherwise, it will use ReadByte to fill the
+// buffer with just the right number of bits.
+func (pr *Reader) PullBits(nb uint) error {
+	if pr.bufRd != nil {
+		pr.discardBits += int(pr.fedBits - pr.numBits)
+		for {
+			if len(pr.bufPeek) == 0 {
+				pr.fedBits = pr.numBits // Don't discard bits just added
+				if _, err := pr.Flush(); err != nil {
+					return err
+				}
+
+				// Peek no more bytes than necessary.
+				// The computation for cntPeek computes the minimum number of
+				// bytes to Peek to fill nb bits.
+				var err error
+				cntPeek := int(nb+(-nb&7)) / 8
+				if cntPeek < pr.bufRd.Buffered() {
+					cntPeek = pr.bufRd.Buffered()
+				}
+				pr.bufPeek, err = pr.bufRd.Peek(cntPeek)
+				pr.bufPeek = pr.bufPeek[int(pr.numBits/8):] // Skip buffered bits
+				if len(pr.bufPeek) == 0 {
+					if pr.numBits >= nb {
+						break
+					}
+					if err == io.EOF {
+						err = io.ErrUnexpectedEOF
+					}
+					return err
+				}
+			}
+
+			n := int(64-pr.numBits) / 8 // Number of bytes to copy to bit buffer
+			if len(pr.bufPeek) >= 8 {
+				// Starting with Go 1.7, the compiler should use a wide integer
+				// load here if the architecture supports it.
+				u := binary.LittleEndian.Uint64(pr.bufPeek)
+				if pr.bigEndian {
+					// Swap all the bits within each byte.
+					u = (u&0xaaaaaaaaaaaaaaaa)>>1 | (u&0x5555555555555555)<<1
+					u = (u&0xcccccccccccccccc)>>2 | (u&0x3333333333333333)<<2
+					u = (u&0xf0f0f0f0f0f0f0f0)>>4 | (u&0x0f0f0f0f0f0f0f0f)<<4
+				}
+
+				pr.bufBits |= u << pr.numBits
+				pr.numBits += uint(n * 8)
+				pr.bufPeek = pr.bufPeek[n:]
+				break
+			} else {
+				if n > len(pr.bufPeek) {
+					n = len(pr.bufPeek)
+				}
+				for _, c := range pr.bufPeek[:n] {
+					if pr.bigEndian {
+						c = internal.ReverseLUT[c]
+					}
+					pr.bufBits |= uint64(c) << pr.numBits
+					pr.numBits += 8
+				}
+				pr.bufPeek = pr.bufPeek[n:]
+				if pr.numBits > 56 {
+					break
+				}
+			}
+		}
+		pr.fedBits = pr.numBits
+	} else {
+		for pr.numBits < nb {
+			c, err := pr.byteRd.ReadByte()
+			if err != nil {
+				if err == io.EOF {
+					err = io.ErrUnexpectedEOF
+				}
+				return err
+			}
+			if pr.bigEndian {
+				c = internal.ReverseLUT[c]
+			}
+			pr.bufBits |= uint64(c) << pr.numBits
+			pr.numBits += 8
+			pr.Offset++
+		}
+	}
+	return nil
+}

vendor/github.com/dsnet/compress/internal/prefix/wrap.go

@@ -0,0 +1,146 @@
+// Copyright 2015, Joe Tsai. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE.md file.
+
+package prefix
+
+import (
+	"bytes"
+	"io"
+	"strings"
+)
+
+// For some of the common Readers, we wrap and extend them to satisfy the
+// compress.BufferedReader interface to improve performance.
+
+type buffer struct {
+	*bytes.Buffer
+}
+
+type bytesReader struct {
+	*bytes.Reader
+	pos int64
+	buf []byte
+	arr [512]byte
+}
+
+type stringReader struct {
+	*strings.Reader
+	pos int64
+	buf []byte
+	arr [512]byte
+}
+
+func (r *buffer) Buffered() int {
+	return r.Len()
+}
+
+func (r *buffer) Peek(n int) ([]byte, error) {
+	b := r.Bytes()
+	if len(b) < n {
+		return b, io.EOF
+	}
+	return b[:n], nil
+}
+
+func (r *buffer) Discard(n int) (int, error) {
+	b := r.Next(n)
+	if len(b) < n {
+		return len(b), io.EOF
+	}
+	return n, nil
+}
+
+func (r *bytesReader) Buffered() int {
+	r.update()
+	if r.Len() > len(r.buf) {
+		return len(r.buf)
+	}
+	return r.Len()
+}
+
+func (r *bytesReader) Peek(n int) ([]byte, error) {
+	if n > len(r.arr) {
+		return nil, io.ErrShortBuffer
+	}
+
+	// Return sub-slice of local buffer if possible.
+	r.update()
+	if len(r.buf) >= n {
+		return r.buf[:n], nil
+	}
+
+	// Fill entire local buffer, and return appropriate sub-slice.
+	cnt, err := r.ReadAt(r.arr[:], r.pos)
+	r.buf = r.arr[:cnt]
+	if cnt < n {
+		return r.arr[:cnt], err
+	}
+	return r.arr[:n], nil
+}
+
+func (r *bytesReader) Discard(n int) (int, error) {
+	var err error
+	if n > r.Len() {
+		n, err = r.Len(), io.EOF
+	}
+	r.Seek(int64(n), io.SeekCurrent)
+	return n, err
+}
+
+// update reslices the internal buffer to be consistent with the read offset.
+func (r *bytesReader) update() {
+	pos, _ := r.Seek(0, io.SeekCurrent)
+	if off := pos - r.pos; off >= 0 && off < int64(len(r.buf)) {
+		r.buf, r.pos = r.buf[off:], pos
+	} else {
+		r.buf, r.pos = nil, pos
+	}
+}
+
+func (r *stringReader) Buffered() int {
+	r.update()
+	if r.Len() > len(r.buf) {
+		return len(r.buf)
+	}
+	return r.Len()
+}
+
+func (r *stringReader) Peek(n int) ([]byte, error) {
+	if n > len(r.arr) {
+		return nil, io.ErrShortBuffer
+	}
+
+	// Return sub-slice of local buffer if possible.
+	r.update()
+	if len(r.buf) >= n {
+		return r.buf[:n], nil
+	}
+
+	// Fill entire local buffer, and return appropriate sub-slice.
+	cnt, err := r.ReadAt(r.arr[:], r.pos)
+	r.buf = r.arr[:cnt]
+	if cnt < n {
+		return r.arr[:cnt], err
+	}
+	return r.arr[:n], nil
+}
+
+func (r *stringReader) Discard(n int) (int, error) {
+	var err error
+	if n > r.Len() {
+		n, err = r.Len(), io.EOF
+	}
+	r.Seek(int64(n), io.SeekCurrent)
+	return n, err
+}
+
+// update reslices the internal buffer to be consistent with the read offset.
+func (r *stringReader) update() {
+	pos, _ := r.Seek(0, io.SeekCurrent)
+	if off := pos - r.pos; off >= 0 && off < int64(len(r.buf)) {
+		r.buf, r.pos = r.buf[off:], pos
+	} else {
+		r.buf, r.pos = nil, pos
+	}
+}

vendor/github.com/dsnet/compress/internal/prefix/writer.go

@@ -0,0 +1,166 @@
+// Copyright 2015, Joe Tsai. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE.md file.
+
+package prefix
+
+import (
+	"encoding/binary"
+	"io"
+
+	"github.com/dsnet/compress/internal/errors"
+)
+
+// Writer implements a prefix encoder. For performance reasons, Writer will not
+// write bytes immediately to the underlying stream.
+type Writer struct {
+	Offset int64 // Number of bytes written to the underlying io.Writer
+
+	wr        io.Writer
+	bufBits   uint64 // Buffer to hold some bits
+	numBits   uint   // Number of valid bits in bufBits
+	bigEndian bool   // Are bits written in big-endian order?
+
+	buf    [512]byte
+	cntBuf int
+}
+
+// Init initializes the bit Writer to write to w. If bigEndian is true, then
+// bits will be written starting from the most-significant bits of a byte
+// (as done in bzip2), otherwise it will write starting from the
+// least-significant bits of a byte (such as for deflate and brotli).
+func (pw *Writer) Init(w io.Writer, bigEndian bool) {
+	*pw = Writer{wr: w, bigEndian: bigEndian}
+	return
+}
+
+// BitsWritten reports the total number of bits issued to any Write method.
+func (pw *Writer) BitsWritten() int64 {
+	return 8*pw.Offset + 8*int64(pw.cntBuf) + int64(pw.numBits)
+}
+
+// WritePads writes 0-7 bits to the bit buffer to achieve byte-alignment.
+func (pw *Writer) WritePads(v uint) {
+	nb := -pw.numBits & 7
+	pw.bufBits |= uint64(v) << pw.numBits
+	pw.numBits += nb
+}
+
+// Write writes bytes from buf.
+// The bit-ordering mode does not affect this method.
+func (pw *Writer) Write(buf []byte) (cnt int, err error) {
+	if pw.numBits > 0 || pw.cntBuf > 0 {
+		if pw.numBits%8 != 0 {
+			return 0, errorf(errors.Invalid, "non-aligned bit buffer")
+		}
+		if _, err := pw.Flush(); err != nil {
+			return 0, err
+		}
+	}
+	cnt, err = pw.wr.Write(buf)
+	pw.Offset += int64(cnt)
+	return cnt, err
+}
+
+// WriteOffset writes ofs in a (sym, extra) fashion using the provided prefix
+// Encoder and RangeEncoder.
+func (pw *Writer) WriteOffset(ofs uint, pe *Encoder, re *RangeEncoder) {
+	sym := re.Encode(ofs)
+	pw.WriteSymbol(sym, pe)
+	rc := re.rcs[sym]
+	pw.WriteBits(ofs-uint(rc.Base), uint(rc.Len))
+}
+
+// TryWriteBits attempts to write nb bits using the contents of the bit buffer
+// alone. It reports whether it succeeded.
+//
+// This method is designed to be inlined for performance reasons.
+func (pw *Writer) TryWriteBits(v, nb uint) bool {
+	if 64-pw.numBits < nb {
+		return false
+	}
+	pw.bufBits |= uint64(v) << pw.numBits
+	pw.numBits += nb
+	return true
+}
+
+// WriteBits writes nb bits of v to the underlying writer.
+func (pw *Writer) WriteBits(v, nb uint) {
+	if _, err := pw.PushBits(); err != nil {
+		errors.Panic(err)
+	}
+	pw.bufBits |= uint64(v) << pw.numBits
+	pw.numBits += nb
+}
+
+// TryWriteSymbol attempts to encode the next symbol using the contents of the
+// bit buffer alone. It reports whether it succeeded.
+//
+// This method is designed to be inlined for performance reasons.
+func (pw *Writer) TryWriteSymbol(sym uint, pe *Encoder) bool {
+	chunk := pe.chunks[uint32(sym)&pe.chunkMask]
+	nb := uint(chunk & countMask)
+	if 64-pw.numBits < nb {
+		return false
+	}
+	pw.bufBits |= uint64(chunk>>countBits) << pw.numBits
+	pw.numBits += nb
+	return true
+}
+
+// WriteSymbol writes the symbol using the provided prefix Encoder.
+func (pw *Writer) WriteSymbol(sym uint, pe *Encoder) {
+	if _, err := pw.PushBits(); err != nil {
+		errors.Panic(err)
+	}
+	chunk := pe.chunks[uint32(sym)&pe.chunkMask]
+	nb := uint(chunk & countMask)
+	pw.bufBits |= uint64(chunk>>countBits) << pw.numBits
+	pw.numBits += nb
+}
+
+// Flush flushes all complete bytes from the bit buffer to the byte buffer, and
+// then flushes all bytes in the byte buffer to the underlying writer.
+// After this call, the bit Writer is will only withhold 7 bits at most.
+func (pw *Writer) Flush() (int64, error) {
+	if pw.numBits < 8 && pw.cntBuf == 0 {
+		return pw.Offset, nil
+	}
+	if _, err := pw.PushBits(); err != nil {
+		return pw.Offset, err
+	}
+	cnt, err := pw.wr.Write(pw.buf[:pw.cntBuf])
+	pw.cntBuf -= cnt
+	pw.Offset += int64(cnt)
+	return pw.Offset, err
+}
+
+// PushBits pushes as many bytes as possible from the bit buffer to the byte
+// buffer, reporting the number of bits pushed.
+func (pw *Writer) PushBits() (uint, error) {
+	if pw.cntBuf >= len(pw.buf)-8 {
+		cnt, err := pw.wr.Write(pw.buf[:pw.cntBuf])
+		pw.cntBuf -= cnt
+		pw.Offset += int64(cnt)
+		if err != nil {
+			return 0, err
+		}
+	}
+
+	u := pw.bufBits
+	if pw.bigEndian {
+		// Swap all the bits within each byte.
+		u = (u&0xaaaaaaaaaaaaaaaa)>>1 | (u&0x5555555555555555)<<1
+		u = (u&0xcccccccccccccccc)>>2 | (u&0x3333333333333333)<<2
+		u = (u&0xf0f0f0f0f0f0f0f0)>>4 | (u&0x0f0f0f0f0f0f0f0f)<<4
+	}
+	// Starting with Go 1.7, the compiler should use a wide integer
+	// store here if the architecture supports it.
+	binary.LittleEndian.PutUint64(pw.buf[pw.cntBuf:], u)
+
+	nb := pw.numBits / 8 // Number of bytes to copy from bit buffer
+	pw.cntBuf += int(nb)
+	pw.bufBits >>= 8 * nb
+	pw.numBits -= 8 * nb
+	return 8 * nb, nil
+}

vendor/github.com/dsnet/compress/internal/release.go

@@ -0,0 +1,21 @@
+// Copyright 2015, Joe Tsai. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE.md file.
+
+// +build !debug,!gofuzz
+
+package internal
+
+// Debug indicates whether the debug build tag was set.
+//
+// If set, programs may choose to print with more human-readable
+// debug information and also perform sanity checks that would otherwise be too
+// expensive to run in a release build.
+const Debug = false
+
+// GoFuzz indicates whether the gofuzz build tag was set.
+//
+// If set, programs may choose to disable certain checks (like checksums) that
+// would be nearly impossible for gofuzz to properly get right.
+// If GoFuzz is set, it implies that Debug is set as well.
+const GoFuzz = false

vendor/github.com/dsnet/compress/zbench.sh

@@ -0,0 +1,12 @@
+#!/bin/bash
+#
+# Copyright 2017, Joe Tsai. All rights reserved.
+# Use of this source code is governed by a BSD-style
+# license that can be found in the LICENSE.md file.
+
+# zbench wraps internal/tool/bench and is useful for comparing benchmarks from
+# the implementations in this repository relative to other implementations.
+#
+# See internal/tool/bench/main.go for more details.
+cd $(dirname "${BASH_SOURCE[0]}")/internal/tool/bench
+go run $(go list -f '{{ join .GoFiles "\n" }}') "$@"

vendor/github.com/dsnet/compress/zfuzz.sh

@@ -0,0 +1,10 @@
+#!/bin/bash
+#
+# Copyright 2017, Joe Tsai. All rights reserved.
+# Use of this source code is governed by a BSD-style
+# license that can be found in the LICENSE.md file.
+
+# zfuzz wraps internal/tool/fuzz and is useful for fuzz testing each of
+# the implementations in this repository.
+cd $(dirname "${BASH_SOURCE[0]}")/internal/tool/fuzz
+./fuzz.sh "$@"

vendor/github.com/dsnet/compress/zprof.sh

@@ -0,0 +1,54 @@
+#!/bin/bash
+#
+# Copyright 2017, Joe Tsai. All rights reserved.
+# Use of this source code is governed by a BSD-style
+# license that can be found in the LICENSE.md file.
+
+if [ $# == 0 ]; then
+	echo "Usage: $0 PKG_PATH TEST_ARGS..."
+	echo ""
+	echo "Runs coverage and performance benchmarks for a given package."
+	echo "The results are stored in the _zprof_ directory."
+	echo ""
+	echo "Example:"
+	echo "	$0 flate -test.bench=Decode/Twain/Default"
+	exit 1
+fi
+
+DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+PKG_PATH=$1
+PKG_NAME=$(basename $PKG_PATH)
+shift
+
+TMPDIR=$(mktemp -d)
+trap "rm -rf $TMPDIR $PKG_PATH/$PKG_NAME.test" SIGINT SIGTERM EXIT
+
+(
+	cd $DIR/$PKG_PATH
+
+	# Print the go version.
+	go version
+
+	# Perform coverage profiling.
+	go test github.com/dsnet/compress/$PKG_PATH -coverprofile $TMPDIR/cover.profile
+	if [ $? != 0 ]; then exit 1; fi
+	go tool cover -html $TMPDIR/cover.profile -o cover.html
+
+	# Perform performance profiling.
+	if [ $# != 0 ]; then
+		go test -c github.com/dsnet/compress/$PKG_PATH
+		if [ $? != 0 ]; then exit 1; fi
+		./$PKG_NAME.test -test.cpuprofile $TMPDIR/cpu.profile -test.memprofile $TMPDIR/mem.profile -test.run - "$@"
+		PPROF="go tool pprof"
+		$PPROF -output=cpu.svg          -web                      $PKG_NAME.test $TMPDIR/cpu.profile 2> /dev/null
+		$PPROF -output=cpu.html         -weblist=.                $PKG_NAME.test $TMPDIR/cpu.profile 2> /dev/null
+		$PPROF -output=mem_objects.svg  -alloc_objects -web       $PKG_NAME.test $TMPDIR/mem.profile 2> /dev/null
+		$PPROF -output=mem_objects.html -alloc_objects -weblist=. $PKG_NAME.test $TMPDIR/mem.profile 2> /dev/null
+		$PPROF -output=mem_space.svg    -alloc_space   -web       $PKG_NAME.test $TMPDIR/mem.profile 2> /dev/null
+		$PPROF -output=mem_space.html   -alloc_space   -weblist=. $PKG_NAME.test $TMPDIR/mem.profile 2> /dev/null
+	fi
+
+	rm -rf $DIR/_zprof_/$PKG_NAME
+	mkdir -p $DIR/_zprof_/$PKG_NAME
+	mv *.html *.svg $DIR/_zprof_/$PKG_NAME 2> /dev/null
+)

vendor/github.com/dsnet/compress/ztest.sh

@@ -0,0 +1,54 @@
+#!/bin/bash
+#
+# Copyright 2017, Joe Tsai. All rights reserved.
+# Use of this source code is governed by a BSD-style
+# license that can be found in the LICENSE.md file.
+
+cd $(go list -f '{{ .Dir }}' github.com/dsnet/compress)
+
+BOLD="\x1b[1mRunning: "
+PASS="\x1b[32mPASS"
+FAIL="\x1b[31mFAIL"
+RESET="\x1b[0m"
+
+echo -e "${BOLD}fmt${RESET}"
+RET_FMT=$(find . -name "*.go" | egrep -v "/(_.*_|\..*|testdata)/" | xargs gofmt -d)
+if [[ ! -z "$RET_FMT" ]]; then echo "$RET_FMT"; echo; fi
+
+echo -e "${BOLD}test${RESET}"
+RET_TEST=$(go test -race ./... | egrep -v "^(ok|[?])\s+")
+if [[ ! -z "$RET_TEST" ]]; then echo "$RET_TEST"; echo; fi
+
+echo -e "${BOLD}staticcheck${RESET}"
+RET_SCHK=$(staticcheck \
+	-ignore "
+		github.com/dsnet/compress/brotli/*.go:SA4016
+		github.com/dsnet/compress/brotli/*.go:S1023
+		github.com/dsnet/compress/brotli/*.go:U1000
+		github.com/dsnet/compress/bzip2/*.go:S1023
+		github.com/dsnet/compress/flate/*.go:U1000
+		github.com/dsnet/compress/internal/cgo/lzma/*.go:SA4000
+		github.com/dsnet/compress/internal/prefix/*.go:S1004
+		github.com/dsnet/compress/internal/prefix/*.go:S1023
+		github.com/dsnet/compress/internal/prefix/*.go:SA4016
+		github.com/dsnet/compress/internal/tool/bench/*.go:S1007
+		github.com/dsnet/compress/xflate/internal/meta/*.go:S1023
+	" ./... 2>&1)
+if [[ ! -z "$RET_SCHK" ]]; then echo "$RET_SCHK"; echo; fi
+
+echo -e "${BOLD}lint${RESET}"
+RET_LINT=$(golint ./... 2>&1 |
+	egrep -v "^vendor/" |
+	egrep -v "should have comment(.*)or be unexported" |
+	egrep -v "^(.*)type name will be used as(.*)by other packages" |
+	egrep -v "^brotli/transform.go:(.*)replace i [+]= 1 with i[+]{2}" |
+	egrep -v "^internal/prefix/prefix.go:(.*)replace symBits(.*) [-]= 1 with symBits(.*)[-]{2}" |
+	egrep -v "^xflate/common.go:(.*)NoCompression should be of the form" |
+	egrep -v "^exit status")
+if [[ ! -z "$RET_LINT" ]]; then echo "$RET_LINT"; echo; fi
+
+if [[ ! -z "$RET_FMT" ]] || [ ! -z "$RET_TEST" ] || [[ ! -z "$RET_SCHK" ]] || [[ ! -z "$RET_LINT" ]]; then
+	echo -e "${FAIL}${RESET}"; exit 1
+else
+	echo -e "${PASS}${RESET}"; exit 0
+fi

vendor/github.com/hjson/hjson-go/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2016, 2017 Christian Zangl
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

vendor/github.com/hjson/hjson-go/README.md

@@ -0,0 +1,181 @@
+# hjson-go
+
+[![Build Status](https://github.com/hjson/hjson-go/workflows/test/badge.svg)](https://github.com/hjson/hjson-go/actions)
+[![Go Pkg](https://img.shields.io/github/release/hjson/hjson-go.svg?style=flat-square&label=go-pkg)](https://github.com/hjson/hjson-go/releases)
+[![Go Report Card](https://goreportcard.com/badge/github.com/hjson/hjson-go?style=flat-square)](https://goreportcard.com/report/github.com/hjson/hjson-go)
+[![coverage](https://img.shields.io/badge/coverage-ok-brightgreen.svg?style=flat-square)](http://gocover.io/github.com/hjson/hjson-go/)
+[![godoc](https://img.shields.io/badge/godoc-reference-blue.svg?style=flat-square)](https://godoc.org/github.com/hjson/hjson-go)
+
+![Hjson Intro](https://hjson.github.io/hjson1.gif)
+
+```
+{
+  # specify rate in requests/second (because comments are helpful!)
+  rate: 1000
+
+  // prefer c-style comments?
+  /* feeling old fashioned? */
+
+  # did you notice that rate doesn't need quotes?
+  hey: look ma, no quotes for strings either!
+
+  # best of all
+  notice: []
+  anything: ?
+
+  # yes, commas are optional!
+}
+```
+
+The Go implementation of Hjson is based on [hjson-js](https://github.com/hjson/hjson-js). For other platforms see [hjson.github.io](https://hjson.github.io).
+
+# Install
+
+Make sure you have a working Go environment. See the [install instructions](http://golang.org/doc/install.html).
+
+1. Get the sources
+```bash
+$ go get -u github.com/hjson/hjson-go
+```
+2. Build the **hjson-cli** commandline tool (optional)
+```bash
+$ cd $(go env GOPATH)/src/github.com/hjson/hjson-go/hjson-cli && go install
+$ hjson-cli --version
+```
+# Usage as command line tool
+```
+usage: hjson-cli [OPTIONS] [INPUT]
+hjson can be used to convert JSON from/to Hjson.
+
+hjson will read the given JSON/Hjson input file or read from stdin.
+
+Options:
+  -allowMinusZero
+      Allow -0.
+  -bracesSameLine
+      Print braces on the same line.
+  -c  Output as JSON.
+  -h  Show this screen.
+  -indentBy string
+      The indent string. (default "  ")
+  -j  Output as formatted JSON.
+  -omitRootBraces
+      Omit braces at the root.
+  -quoteAlways
+      Always quote string values.
+```
+
+Sample:
+- run `hjson-cli test.json > test.hjson` to convert to Hjson
+- run `hjson-cli -j test.hjson > test.json` to convert to JSON
+
+# Usage as a GO library
+
+```go
+
+package main
+
+import (
+  "github.com/hjson/hjson-go"
+  "fmt"
+)
+
+func main() {
+
+    // Now let's look at decoding Hjson data into Go
+    // values.
+    sampleText := []byte(`
+    {
+        # specify rate in requests/second
+        rate: 1000
+        array:
+        [
+            foo
+            bar
+        ]
+    }`)
+
+    // We need to provide a variable where Hjson
+    // can put the decoded data.
+    var dat map[string]interface{}
+
+    // Decode and a check for errors.
+    if err := hjson.Unmarshal(sampleText, &dat); err != nil {
+        panic(err)
+    }
+    fmt.Println(dat)
+
+    // In order to use the values in the decoded map,
+    // we'll need to cast them to their appropriate type.
+
+    rate := dat["rate"].(float64)
+    fmt.Println(rate)
+
+    array := dat["array"].([]interface{})
+    str1 := array[0].(string)
+    fmt.Println(str1)
+
+
+    // To encode to Hjson with default options:
+    sampleMap := map[string]int{"apple": 5, "lettuce": 7}
+    hjson, _ := hjson.Marshal(sampleMap)
+    // this is short for:
+    // options := hjson.DefaultOptions()
+    // hjson, _ := hjson.MarshalWithOptions(sampleMap, options)
+    fmt.Println(string(hjson))
+}
+```
+
+If you prefer, you can also unmarshal to Go objects by converting to JSON:
+
+```go
+
+package main
+
+import (
+  "github.com/hjson/hjson-go"
+  "encoding/json"
+  "fmt"
+)
+
+type Sample struct {
+    Rate  int
+    Array []string
+}
+
+func main() {
+
+    sampleText := []byte(`
+    {
+        # specify rate in requests/second
+        rate: 1000
+        array:
+        [
+            foo
+            bar
+        ]
+    }`)
+
+    // read Hjson
+    var dat map[string]interface{}
+    hjson.Unmarshal(sampleText, &dat)
+
+    // convert to JSON
+    b, _ := json.Marshal(dat)
+
+    // unmarshal
+    var sample Sample
+    json.Unmarshal(b, &sample)
+
+    fmt.Println(sample.Rate)
+    fmt.Println(sample.Array)
+}
+```
+
+# API
+
+[![godoc](https://godoc.org/github.com/hjson/hjson-go?status.svg)](http://godoc.org/github.com/hjson/hjson-go)
+
+# History
+
+[see releases](https://github.com/hjson/hjson-go/releases)

vendor/github.com/hjson/hjson-go/build_release.sh

@@ -0,0 +1,58 @@
+#!/bin/bash
+
+cd `dirname $0`
+ROOT=$PWD/_dist
+
+if [ -d "$ROOT" ]; then rm -rf $ROOT; fi
+
+mkdir $ROOT
+
+function build() {
+  export GOOS=$1
+  export GOARCH=$2
+
+  echo build $GOOS $GOARCH
+  OUT=$ROOT/${GOOS}_${GOARCH}
+  mkdir $OUT
+  cd $OUT
+  go build github.com/hjson/hjson-go/hjson-cli
+  if [[ $3 == "zip" ]]; then
+    mv $OUT/hjson-cli.exe $OUT/hjson.exe
+    zip -j ${OUT}.zip $OUT/*
+  else
+    mv $OUT/hjson-cli $OUT/hjson
+    tar -czf ${OUT}.tar.gz -C $OUT .
+  fi
+
+}
+
+# not all targets can be built on travis
+
+# build android arm
+build darwin 386
+build darwin amd64
+# build darwin arm
+# build darwin arm64
+build dragonfly amd64
+build freebsd 386
+build freebsd amd64
+build freebsd arm
+build linux 386
+build linux amd64
+build linux arm
+build linux arm64
+build linux mips64
+build linux mips64le
+build linux ppc64
+build linux ppc64le
+build netbsd 386
+build netbsd amd64
+build netbsd arm
+build openbsd 386
+build openbsd amd64
+build openbsd arm
+build plan9 386
+build plan9 amd64
+build solaris amd64
+build windows 386 zip
+build windows amd64 zip

vendor/github.com/hjson/hjson-go/decode.go

@@ -0,0 +1,480 @@
+package hjson
+
+import (
+	"bytes"
+	"fmt"
+	"reflect"
+	"strings"
+)
+
+type hjsonParser struct {
+	data []byte
+	at   int  // The index of the current character
+	ch   byte // The current character
+}
+
+func (p *hjsonParser) resetAt() {
+	p.at = 0
+	p.ch = ' '
+}
+
+func isPunctuatorChar(c byte) bool {
+	return c == '{' || c == '}' || c == '[' || c == ']' || c == ',' || c == ':'
+}
+
+func (p *hjsonParser) errAt(message string) error {
+	var i int
+	col := 0
+	line := 1
+	for i = p.at - 1; i > 0 && p.data[i] != '\n'; i-- {
+		col++
+	}
+	for ; i > 0; i-- {
+		if p.data[i] == '\n' {
+			line++
+		}
+	}
+	samEnd := p.at - col + 20
+	if samEnd > len(p.data) {
+		samEnd = len(p.data)
+	}
+	return fmt.Errorf("%s at line %d,%d >>> %s", message, line, col, string(p.data[p.at-col:samEnd]))
+}
+
+func (p *hjsonParser) next() bool {
+	// get the next character.
+	if p.at < len(p.data) {
+		p.ch = p.data[p.at]
+		p.at++
+		return true
+	}
+	p.ch = 0
+	return false
+}
+
+func (p *hjsonParser) peek(offs int) byte {
+	pos := p.at + offs
+	if pos >= 0 && pos < len(p.data) {
+		return p.data[p.at+offs]
+	}
+	return 0
+}
+
+var escapee = map[byte]byte{
+	'"':  '"',
+	'\'': '\'',
+	'\\': '\\',
+	'/':  '/',
+	'b':  '\b',
+	'f':  '\f',
+	'n':  '\n',
+	'r':  '\r',
+	't':  '\t',
+}
+
+func (p *hjsonParser) readString(allowML bool) (string, error) {
+
+	// Parse a string value.
+	res := new(bytes.Buffer)
+
+	// callers make sure that (ch === '"' || ch === "'")
+	// When parsing for string values, we must look for " and \ characters.
+	exitCh := p.ch
+	for p.next() {
+		if p.ch == exitCh {
+			p.next()
+			if allowML && exitCh == '\'' && p.ch == '\'' && res.Len() == 0 {
+				// ''' indicates a multiline string
+				p.next()
+				return p.readMLString()
+			} else {
+				return res.String(), nil
+			}
+		}
+		if p.ch == '\\' {
+			p.next()
+			if p.ch == 'u' {
+				uffff := 0
+				for i := 0; i < 4; i++ {
+					p.next()
+					var hex int
+					if p.ch >= '0' && p.ch <= '9' {
+						hex = int(p.ch - '0')
+					} else if p.ch >= 'a' && p.ch <= 'f' {
+						hex = int(p.ch - 'a' + 0xa)
+					} else if p.ch >= 'A' && p.ch <= 'F' {
+						hex = int(p.ch - 'A' + 0xa)
+					} else {
+						return "", p.errAt("Bad \\u char " + string(p.ch))
+					}
+					uffff = uffff*16 + hex
+				}
+				res.WriteRune(rune(uffff))
+			} else if ech, ok := escapee[p.ch]; ok {
+				res.WriteByte(ech)
+			} else {
+				return "", p.errAt("Bad escape \\" + string(p.ch))
+			}
+		} else if p.ch == '\n' || p.ch == '\r' {
+			return "", p.errAt("Bad string containing newline")
+		} else {
+			res.WriteByte(p.ch)
+		}
+	}
+	return "", p.errAt("Bad string")
+}
+
+func (p *hjsonParser) readMLString() (value string, err error) {
+
+	// Parse a multiline string value.
+	res := new(bytes.Buffer)
+	triple := 0
+
+	// we are at ''' +1 - get indent
+	indent := 0
+	for {
+		c := p.peek(-indent - 5)
+		if c == 0 || c == '\n' {
+			break
+		}
+		indent++
+	}
+
+	skipIndent := func() {
+		skip := indent
+		for p.ch > 0 && p.ch <= ' ' && p.ch != '\n' && skip > 0 {
+			skip--
+			p.next()
+		}
+	}
+
+	// skip white/to (newline)
+	for p.ch > 0 && p.ch <= ' ' && p.ch != '\n' {
+		p.next()
+	}
+	if p.ch == '\n' {
+		p.next()
+		skipIndent()
+	}
+
+	// When parsing multiline string values, we must look for ' characters.
+	lastLf := false
+	for {
+		if p.ch == 0 {
+			return "", p.errAt("Bad multiline string")
+		} else if p.ch == '\'' {
+			triple++
+			p.next()
+			if triple == 3 {
+				sres := res.Bytes()
+				if lastLf {
+					return string(sres[0 : len(sres)-1]), nil // remove last EOL
+				}
+				return string(sres), nil
+			}
+			continue
+		} else {
+			for triple > 0 {
+				res.WriteByte('\'')
+				triple--
+				lastLf = false
+			}
+		}
+		if p.ch == '\n' {
+			res.WriteByte('\n')
+			lastLf = true
+			p.next()
+			skipIndent()
+		} else {
+			if p.ch != '\r' {
+				res.WriteByte(p.ch)
+				lastLf = false
+			}
+			p.next()
+		}
+	}
+}
+
+func (p *hjsonParser) readKeyname() (string, error) {
+
+	// quotes for keys are optional in Hjson
+	// unless they include {}[],: or whitespace.
+
+	if p.ch == '"' || p.ch == '\'' {
+		return p.readString(false)
+	}
+
+	name := new(bytes.Buffer)
+	start := p.at
+	space := -1
+	for {
+		if p.ch == ':' {
+			if name.Len() == 0 {
+				return "", p.errAt("Found ':' but no key name (for an empty key name use quotes)")
+			} else if space >= 0 && space != name.Len() {
+				p.at = start + space
+				return "", p.errAt("Found whitespace in your key name (use quotes to include)")
+			}
+			return name.String(), nil
+		} else if p.ch <= ' ' {
+			if p.ch == 0 {
+				return "", p.errAt("Found EOF while looking for a key name (check your syntax)")
+			}
+			if space < 0 {
+				space = name.Len()
+			}
+		} else {
+			if isPunctuatorChar(p.ch) {
+				return "", p.errAt("Found '" + string(p.ch) + "' where a key name was expected (check your syntax or use quotes if the key name includes {}[],: or whitespace)")
+			}
+			name.WriteByte(p.ch)
+		}
+		p.next()
+	}
+}
+
+func (p *hjsonParser) white() {
+	for p.ch > 0 {
+		// Skip whitespace.
+		for p.ch > 0 && p.ch <= ' ' {
+			p.next()
+		}
+		// Hjson allows comments
+		if p.ch == '#' || p.ch == '/' && p.peek(0) == '/' {
+			for p.ch > 0 && p.ch != '\n' {
+				p.next()
+			}
+		} else if p.ch == '/' && p.peek(0) == '*' {
+			p.next()
+			p.next()
+			for p.ch > 0 && !(p.ch == '*' && p.peek(0) == '/') {
+				p.next()
+			}
+			if p.ch > 0 {
+				p.next()
+				p.next()
+			}
+		} else {
+			break
+		}
+	}
+}
+
+func (p *hjsonParser) readTfnns() (interface{}, error) {
+
+	// Hjson strings can be quoteless
+	// returns string, true, false, or null.
+
+	if isPunctuatorChar(p.ch) {
+		return nil, p.errAt("Found a punctuator character '" + string(p.ch) + "' when expecting a quoteless string (check your syntax)")
+	}
+	chf := p.ch
+	value := new(bytes.Buffer)
+	value.WriteByte(p.ch)
+
+	for {
+		p.next()
+		isEol := p.ch == '\r' || p.ch == '\n' || p.ch == 0
+		if isEol ||
+			p.ch == ',' || p.ch == '}' || p.ch == ']' ||
+			p.ch == '#' ||
+			p.ch == '/' && (p.peek(0) == '/' || p.peek(0) == '*') {
+			switch chf {
+			case 'f':
+				if strings.TrimSpace(value.String()) == "false" {
+					return false, nil
+				}
+			case 'n':
+				if strings.TrimSpace(value.String()) == "null" {
+					return nil, nil
+				}
+			case 't':
+				if strings.TrimSpace(value.String()) == "true" {
+					return true, nil
+				}
+			default:
+				if chf == '-' || chf >= '0' && chf <= '9' {
+					if n, err := tryParseNumber(value.Bytes(), false); err == nil {
+						return n, nil
+					}
+				}
+			}
+			if isEol {
+				// remove any whitespace at the end (ignored in quoteless strings)
+				return strings.TrimSpace(value.String()), nil
+			}
+		}
+		value.WriteByte(p.ch)
+	}
+}
+
+func (p *hjsonParser) readArray() (value interface{}, err error) {
+
+	// Parse an array value.
+	// assuming ch == '['
+
+	array := make([]interface{}, 0, 1)
+
+	p.next()
+	p.white()
+
+	if p.ch == ']' {
+		p.next()
+		return array, nil // empty array
+	}
+
+	for p.ch > 0 {
+		var val interface{}
+		if val, err = p.readValue(); err != nil {
+			return nil, err
+		}
+		array = append(array, val)
+		p.white()
+		// in Hjson the comma is optional and trailing commas are allowed
+		if p.ch == ',' {
+			p.next()
+			p.white()
+		}
+		if p.ch == ']' {
+			p.next()
+			return array, nil
+		}
+		p.white()
+	}
+
+	return nil, p.errAt("End of input while parsing an array (did you forget a closing ']'?)")
+}
+
+func (p *hjsonParser) readObject(withoutBraces bool) (value interface{}, err error) {
+	// Parse an object value.
+
+	object := make(map[string]interface{})
+
+	if !withoutBraces {
+		// assuming ch == '{'
+		p.next()
+	}
+
+	p.white()
+	if p.ch == '}' && !withoutBraces {
+		p.next()
+		return object, nil // empty object
+	}
+	for p.ch > 0 {
+		var key string
+		if key, err = p.readKeyname(); err != nil {
+			return nil, err
+		}
+		p.white()
+		if p.ch != ':' {
+			return nil, p.errAt("Expected ':' instead of '" + string(p.ch) + "'")
+		}
+		p.next()
+		// duplicate keys overwrite the previous value
+		var val interface{}
+		if val, err = p.readValue(); err != nil {
+			return nil, err
+		}
+		object[key] = val
+		p.white()
+		// in Hjson the comma is optional and trailing commas are allowed
+		if p.ch == ',' {
+			p.next()
+			p.white()
+		}
+		if p.ch == '}' && !withoutBraces {
+			p.next()
+			return object, nil
+		}
+		p.white()
+	}
+
+	if withoutBraces {
+		return object, nil
+	}
+	return nil, p.errAt("End of input while parsing an object (did you forget a closing '}'?)")
+}
+
+func (p *hjsonParser) readValue() (interface{}, error) {
+
+	// Parse a Hjson value. It could be an object, an array, a string, a number or a word.
+
+	p.white()
+	switch p.ch {
+	case '{':
+		return p.readObject(false)
+	case '[':
+		return p.readArray()
+	case '"', '\'':
+		return p.readString(true)
+	default:
+		return p.readTfnns()
+	}
+}
+
+func (p *hjsonParser) rootValue() (interface{}, error) {
+	// Braces for the root object are optional
+
+	p.white()
+	switch p.ch {
+	case '{':
+		return p.checkTrailing(p.readObject(false))
+	case '[':
+		return p.checkTrailing(p.readArray())
+	}
+
+	// assume we have a root object without braces
+	res, err := p.checkTrailing(p.readObject(true))
+	if err == nil {
+		return res, nil
+	}
+
+	// test if we are dealing with a single JSON value instead (true/false/null/num/"")
+	p.resetAt()
+	if res2, err2 := p.checkTrailing(p.readValue()); err2 == nil {
+		return res2, nil
+	}
+	return res, err
+}
+
+func (p *hjsonParser) checkTrailing(v interface{}, err error) (interface{}, error) {
+	if err != nil {
+		return nil, err
+	}
+	p.white()
+	if p.ch > 0 {
+		return nil, p.errAt("Syntax error, found trailing characters")
+	}
+	return v, nil
+}
+
+// Unmarshal parses the Hjson-encoded data and stores the result
+// in the value pointed to by v.
+//
+// Unmarshal uses the inverse of the encodings that
+// Marshal uses, allocating maps, slices, and pointers as necessary.
+//
+func Unmarshal(data []byte, v interface{}) (err error) {
+	var value interface{}
+	parser := &hjsonParser{data, 0, ' '}
+	parser.resetAt()
+	value, err = parser.rootValue()
+	if err != nil {
+		return err
+	}
+
+	rv := reflect.ValueOf(v)
+	if rv.Kind() != reflect.Ptr || rv.IsNil() {
+		return fmt.Errorf("non-pointer %v", reflect.TypeOf(v))
+	}
+	for rv.Kind() == reflect.Ptr {
+		rv = rv.Elem()
+	}
+	defer func() {
+		if e := recover(); e != nil {
+			err = fmt.Errorf("%v", e)
+		}
+	}()
+	rv.Set(reflect.ValueOf(value))
+	return err
+}

vendor/github.com/hjson/hjson-go/encode.go

@@ -0,0 +1,492 @@
+package hjson
+
+import (
+	"bytes"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"math"
+	"reflect"
+	"regexp"
+	"sort"
+	"strconv"
+	"strings"
+	"unicode/utf8"
+)
+
+// EncoderOptions defines options for encoding to Hjson.
+type EncoderOptions struct {
+	// End of line, should be either \n or \r\n
+	Eol string
+	// Place braces on the same line
+	BracesSameLine bool
+	// Emit braces for the root object
+	EmitRootBraces bool
+	// Always place string in quotes
+	QuoteAlways bool
+	// Place string in quotes if it could otherwise be a number, boolean or null
+	QuoteAmbiguousStrings bool
+	// Indent string
+	IndentBy string
+	// Base indentation string
+	BaseIndentation string
+	// Allow the -0 value (unlike ES6)
+	AllowMinusZero bool
+	// Encode unknown values as 'null'
+	UnknownAsNull bool
+}
+
+// DefaultOptions returns the default encoding options.
+func DefaultOptions() EncoderOptions {
+	opt := EncoderOptions{}
+	opt.Eol = "\n"
+	opt.BracesSameLine = false
+	opt.EmitRootBraces = true
+	opt.QuoteAlways = false
+	opt.QuoteAmbiguousStrings = true
+	opt.IndentBy = "  "
+	opt.BaseIndentation = ""
+	opt.AllowMinusZero = false
+	opt.UnknownAsNull = false
+	return opt
+}
+
+type hjsonEncoder struct {
+	bytes.Buffer // output
+	EncoderOptions
+	indent int
+}
+
+var needsEscape, needsQuotes, needsEscapeML, startsWithKeyword, needsEscapeName *regexp.Regexp
+
+func init() {
+	var commonRange = `\x7f-\x9f\x{00ad}\x{0600}-\x{0604}\x{070f}\x{17b4}\x{17b5}\x{200c}-\x{200f}\x{2028}-\x{202f}\x{2060}-\x{206f}\x{feff}\x{fff0}-\x{ffff}`
+	// needsEscape tests if the string can be written without escapes
+	needsEscape = regexp.MustCompile(`[\\\"\x00-\x1f` + commonRange + `]`)
+	// needsQuotes tests if the string can be written as a quoteless string (includes needsEscape but without \\ and \")
+	needsQuotes = regexp.MustCompile(`^\s|^"|^'|^#|^/\*|^//|^\{|^\}|^\[|^\]|^:|^,|\s$|[\x00-\x1f\x7f-\x9f\x{00ad}\x{0600}-\x{0604}\x{070f}\x{17b4}\x{17b5}\x{200c}-\x{200f}\x{2028}-\x{202f}\x{2060}-\x{206f}\x{feff}\x{fff0}-\x{ffff}]`)
+	// needsEscapeML tests if the string can be written as a multiline string (like needsEscape but without \n, \r, \\, \", \t)
+	var x08Or9 = `\x08` // `\x09` for the old behavior
+	needsEscapeML = regexp.MustCompile(`'''|^[\s]+$|[\x00-` + x08Or9 + `\x0b\x0c\x0e-\x1f` + commonRange + `]`)
+	// starts with a keyword and optionally is followed by a comment
+	startsWithKeyword = regexp.MustCompile(`^(true|false|null)\s*((,|\]|\}|#|//|/\*).*)?$`)
+	needsEscapeName = regexp.MustCompile(`[,\{\[\}\]\s:#"']|//|/\*`)
+}
+
+var meta = map[byte][]byte{
+	// table of character substitutions
+	'\b': []byte("\\b"),
+	'\t': []byte("\\t"),
+	'\n': []byte("\\n"),
+	'\f': []byte("\\f"),
+	'\r': []byte("\\r"),
+	'"':  []byte("\\\""),
+	'\\': []byte("\\\\"),
+}
+
+func (e *hjsonEncoder) quoteReplace(text string) string {
+	return string(needsEscape.ReplaceAllFunc([]byte(text), func(a []byte) []byte {
+		c := meta[a[0]]
+		if c != nil {
+			return c
+		}
+		r, _ := utf8.DecodeRune(a)
+		return []byte(fmt.Sprintf("\\u%04x", r))
+	}))
+}
+
+func (e *hjsonEncoder) quote(value string, separator string, isRootObject bool) {
+
+	// Check if we can insert this string without quotes
+	// see hjson syntax (must not parse as true, false, null or number)
+
+	if len(value) == 0 {
+		e.WriteString(separator + `""`)
+	} else if e.QuoteAlways ||
+		needsQuotes.MatchString(value) || (e.QuoteAmbiguousStrings && (startsWithNumber([]byte(value)) ||
+		startsWithKeyword.MatchString(value))) {
+
+		// If the string contains no control characters, no quote characters, and no
+		// backslash characters, then we can safely slap some quotes around it.
+		// Otherwise we first check if the string can be expressed in multiline
+		// format or we must replace the offending characters with safe escape
+		// sequences.
+
+		if !needsEscape.MatchString(value) {
+			e.WriteString(separator + `"` + value + `"`)
+		} else if !needsEscapeML.MatchString(value) && !isRootObject {
+			e.mlString(value, separator)
+		} else {
+			e.WriteString(separator + `"` + e.quoteReplace(value) + `"`)
+		}
+	} else {
+		// return without quotes
+		e.WriteString(separator + value)
+	}
+}
+
+func (e *hjsonEncoder) mlString(value string, separator string) {
+	// wrap the string into the ''' (multiline) format
+
+	a := strings.Split(strings.Replace(value, "\r", "", -1), "\n")
+
+	if len(a) == 1 {
+		// The string contains only a single line. We still use the multiline
+		// format as it avoids escaping the \ character (e.g. when used in a
+		// regex).
+		e.WriteString(separator + "'''")
+		e.WriteString(a[0])
+	} else {
+		e.writeIndent(e.indent + 1)
+		e.WriteString("'''")
+		for _, v := range a {
+			indent := e.indent + 1
+			if len(v) == 0 {
+				indent = 0
+			}
+			e.writeIndent(indent)
+			e.WriteString(v)
+		}
+		e.writeIndent(e.indent + 1)
+	}
+	e.WriteString("'''")
+}
+
+func (e *hjsonEncoder) quoteName(name string) string {
+	if len(name) == 0 {
+		return `""`
+	}
+
+	// Check if we can insert this name without quotes
+
+	if needsEscapeName.MatchString(name) {
+		if needsEscape.MatchString(name) {
+			name = e.quoteReplace(name)
+		}
+		return `"` + name + `"`
+	}
+	// without quotes
+	return name
+}
+
+type sortAlpha []reflect.Value
+
+func (s sortAlpha) Len() int {
+	return len(s)
+}
+func (s sortAlpha) Swap(i, j int) {
+	s[i], s[j] = s[j], s[i]
+}
+func (s sortAlpha) Less(i, j int) bool {
+	return s[i].String() < s[j].String()
+}
+
+func (e *hjsonEncoder) writeIndent(indent int) {
+	e.WriteString(e.Eol)
+	e.WriteString(e.BaseIndentation)
+	for i := 0; i < indent; i++ {
+		e.WriteString(e.IndentBy)
+	}
+}
+
+func (e *hjsonEncoder) useMarshaler(value reflect.Value, separator string) error {
+	b, err := value.Interface().(json.Marshaler).MarshalJSON()
+	if err != nil {
+		return err
+	}
+	e.WriteString(separator)
+	e.WriteString(string(b))
+	return nil
+}
+
+var marshaler = reflect.TypeOf((*json.Marshaler)(nil)).Elem()
+
+func (e *hjsonEncoder) str(value reflect.Value, noIndent bool, separator string, isRootObject bool) error {
+
+	// Produce a string from value.
+
+	kind := value.Kind()
+
+	if kind == reflect.Interface || kind == reflect.Ptr {
+		if value.IsNil() {
+			e.WriteString(separator)
+			e.WriteString("null")
+			return nil
+		}
+		return e.str(value.Elem(),noIndent,separator,isRootObject)
+	}
+
+	if value.Type().Implements(marshaler) {
+		return e.useMarshaler(value, separator)
+	}
+
+	switch kind {
+	case reflect.String:
+		e.quote(value.String(), separator, isRootObject)
+
+	case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64:
+		e.WriteString(separator)
+		e.WriteString(strconv.FormatInt(value.Int(), 10))
+
+	case reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64,
+		reflect.Uintptr:
+		e.WriteString(separator)
+		e.WriteString(strconv.FormatUint(value.Uint(), 10))
+
+	case reflect.Float32, reflect.Float64:
+		// JSON numbers must be finite. Encode non-finite numbers as null.
+		e.WriteString(separator)
+		number := value.Float()
+		if math.IsInf(number, 0) || math.IsNaN(number) {
+			e.WriteString("null")
+		} else if !e.AllowMinusZero && number == -0 {
+			e.WriteString("0")
+		} else {
+			// find shortest representation ('G' does not work)
+			val := strconv.FormatFloat(number, 'f', -1, 64)
+			exp := strconv.FormatFloat(number, 'E', -1, 64)
+			if len(exp) < len(val) {
+				val = strings.ToLower(exp)
+			}
+			e.WriteString(val)
+		}
+
+	case reflect.Bool:
+		e.WriteString(separator)
+		if value.Bool() {
+			e.WriteString("true")
+		} else {
+			e.WriteString("false")
+		}
+
+	case reflect.Slice, reflect.Array:
+
+		len := value.Len()
+		if len == 0 {
+			e.WriteString(separator)
+			e.WriteString("[]")
+			break
+		}
+
+		indent1 := e.indent
+		e.indent++
+
+		if !noIndent && !e.BracesSameLine {
+			e.writeIndent(indent1)
+		} else {
+			e.WriteString(separator)
+		}
+		e.WriteString("[")
+
+		// Join all of the element texts together, separated with newlines
+		for i := 0; i < len; i++ {
+			e.writeIndent(e.indent)
+			if err := e.str(value.Index(i), true, "", false); err != nil {
+				return err
+			}
+		}
+
+		e.writeIndent(indent1)
+		e.WriteString("]")
+
+		e.indent = indent1
+
+	case reflect.Map:
+
+		len := value.Len()
+		if len == 0 {
+			e.WriteString(separator)
+			e.WriteString("{}")
+			break
+		}
+
+		indent1 := e.indent
+		if !isRootObject || e.EmitRootBraces {
+			if !noIndent && !e.BracesSameLine {
+				e.writeIndent(e.indent)
+			} else {
+				e.WriteString(separator)
+			}
+
+			e.indent++
+			e.WriteString("{")
+		}
+
+		keys := value.MapKeys()
+		sort.Sort(sortAlpha(keys))
+
+		// Join all of the member texts together, separated with newlines
+		for i := 0; i < len; i++ {
+			if i > 0 || !isRootObject || e.EmitRootBraces {
+				e.writeIndent(e.indent)
+			}
+			e.WriteString(e.quoteName(keys[i].String()))
+			e.WriteString(":")
+			if err := e.str(value.MapIndex(keys[i]), false, " ", false); err != nil {
+				return err
+			}
+		}
+
+		if !isRootObject || e.EmitRootBraces {
+			e.writeIndent(indent1)
+			e.WriteString("}")
+		}
+		e.indent = indent1
+
+	case reflect.Struct:
+
+		l := value.NumField()
+		if l == 0 {
+			e.WriteString(separator)
+			e.WriteString("{}")
+			break
+		}
+
+		indent1 := e.indent
+		if !isRootObject || e.EmitRootBraces {
+			if !noIndent && !e.BracesSameLine {
+				e.writeIndent(e.indent)
+			} else {
+				e.WriteString(separator)
+			}
+
+			e.indent++
+			e.WriteString("{")
+		}
+
+		// Join all of the member texts together, separated with newlines
+		for i := 0; i < l; i++ {
+			curStructField := value.Type().Field(i)
+			curField := value.Field(i)
+
+			name := curStructField.Name
+			jsonTag := curStructField.Tag.Get("json")
+			jsonComment := curStructField.Tag.Get("comment")
+			omitEmpty := false
+			if jsonTag == "-" {
+				continue
+			}
+			splits := strings.Split(jsonTag, ",")
+			if splits[0] != "" {
+				name = splits[0]
+			}
+			if len(splits) > 1 {
+				for _, opt := range splits[1:] {
+					if opt == "omitempty" {
+						omitEmpty = true
+					}
+				}
+			}
+			if omitEmpty && isEmptyValue(curField) {
+				continue
+			}
+			if len(jsonComment) > 0 {
+				for _, line := range strings.Split(jsonComment, e.Eol) {
+					if i > 0 || !isRootObject || e.EmitRootBraces {
+						e.writeIndent(e.indent)
+					}
+					e.WriteString(fmt.Sprintf("# %s", line))
+				}
+			}
+			if i > 0 || !isRootObject || e.EmitRootBraces {
+				e.writeIndent(e.indent)
+			}
+			e.WriteString(e.quoteName(name))
+			e.WriteString(":")
+			if err := e.str(curField, false, " ", false); err != nil {
+				return err
+			}
+			if len(jsonComment) > 0 && i < l-1 {
+				e.WriteString(e.Eol)
+			}
+		}
+
+		if !isRootObject || e.EmitRootBraces {
+			e.writeIndent(indent1)
+			e.WriteString("}")
+		}
+
+		e.indent = indent1
+
+	default:
+		if e.UnknownAsNull {
+			// Use null as a placeholder for non-JSON values.
+			e.WriteString("null")
+		} else {
+			return errors.New("Unsupported type " + value.Type().String())
+		}
+	}
+	return nil
+}
+
+func isEmptyValue(v reflect.Value) bool {
+	switch v.Kind() {
+	case reflect.Array, reflect.Map, reflect.Slice, reflect.String:
+		return v.Len() == 0
+	case reflect.Bool:
+		return !v.Bool()
+	case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64:
+		return v.Int() == 0
+	case reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64, reflect.Uintptr:
+		return v.Uint() == 0
+	case reflect.Float32, reflect.Float64:
+		return v.Float() == 0
+	case reflect.Interface, reflect.Ptr:
+		return v.IsNil()
+	default:
+		return false
+	}
+}
+
+// Marshal returns the Hjson encoding of v using
+// default options.
+//
+// See MarshalWithOptions.
+//
+func Marshal(v interface{}) ([]byte, error) {
+	return MarshalWithOptions(v, DefaultOptions())
+}
+
+// MarshalWithOptions returns the Hjson encoding of v.
+//
+// Marshal traverses the value v recursively.
+//
+// Boolean values encode as JSON booleans.
+//
+// Floating point, integer, and Number values encode as JSON numbers.
+//
+// String values encode as Hjson strings (quoteless, multiline or
+// JSON).
+//
+// Array and slice values encode as JSON arrays.
+//
+// Map values encode as JSON objects. The map's key type must be a
+// string. The map keys are sorted and used as JSON object keys.
+//
+// Pointer values encode as the value pointed to.
+// A nil pointer encodes as the null JSON value.
+//
+// Interface values encode as the value contained in the interface.
+// A nil interface value encodes as the null JSON value.
+//
+// JSON cannot represent cyclic data structures and Marshal does not
+// handle them. Passing cyclic structures to Marshal will result in
+// an infinite recursion.
+//
+func MarshalWithOptions(v interface{}, options EncoderOptions) ([]byte, error) {
+	e := &hjsonEncoder{}
+	e.indent = 0
+	e.Eol = options.Eol
+	e.BracesSameLine = options.BracesSameLine
+	e.EmitRootBraces = options.EmitRootBraces
+	e.QuoteAlways = options.QuoteAlways
+	e.QuoteAmbiguousStrings = options.QuoteAmbiguousStrings
+	e.IndentBy = options.IndentBy
+	e.BaseIndentation = options.BaseIndentation
+
+	err := e.str(reflect.ValueOf(v), true, e.BaseIndentation, true)
+	if err != nil {
+		return nil, err
+	}
+	return e.Bytes(), nil
+}

vendor/github.com/hjson/hjson-go/parseNumber.go

@@ -0,0 +1,108 @@
+package hjson
+
+import (
+	"errors"
+	"math"
+	"strconv"
+)
+
+type parseNumber struct {
+	data []byte
+	at   int  // The index of the current character
+	ch   byte // The current character
+}
+
+func (p *parseNumber) next() bool {
+	// get the next character.
+	len := len(p.data)
+	if p.at < len {
+		p.ch = p.data[p.at]
+		p.at++
+		return true
+	}
+	if p.at == len {
+		p.at++
+		p.ch = 0
+	}
+	return false
+}
+
+func (p *parseNumber) peek(offs int) byte {
+	pos := p.at + offs
+	if pos >= 0 && pos < len(p.data) {
+		return p.data[pos]
+	}
+	return 0
+}
+
+func startsWithNumber(text []byte) bool {
+	if _, err := tryParseNumber(text, true); err == nil {
+		return true
+	}
+	return false
+}
+
+func tryParseNumber(text []byte, stopAtNext bool) (float64, error) {
+	// Parse a number value.
+
+	p := parseNumber{text, 0, ' '}
+	leadingZeros := 0
+	testLeading := true
+	p.next()
+	if p.ch == '-' {
+		p.next()
+	}
+	for p.ch >= '0' && p.ch <= '9' {
+		if testLeading {
+			if p.ch == '0' {
+				leadingZeros++
+			} else {
+				testLeading = false
+			}
+		}
+		p.next()
+	}
+	if testLeading {
+		leadingZeros--
+	} // single 0 is allowed
+	if p.ch == '.' {
+		for p.next() && p.ch >= '0' && p.ch <= '9' {
+		}
+	}
+	if p.ch == 'e' || p.ch == 'E' {
+		p.next()
+		if p.ch == '-' || p.ch == '+' {
+			p.next()
+		}
+		for p.ch >= '0' && p.ch <= '9' {
+			p.next()
+		}
+	}
+
+	end := p.at
+
+	// skip white/to (newline)
+	for p.ch > 0 && p.ch <= ' ' {
+		p.next()
+	}
+
+	if stopAtNext {
+		// end scan if we find a punctuator character like ,}] or a comment
+		if p.ch == ',' || p.ch == '}' || p.ch == ']' ||
+			p.ch == '#' || p.ch == '/' && (p.peek(0) == '/' || p.peek(0) == '*') {
+			p.ch = 0
+		}
+	}
+
+	if p.ch > 0 || leadingZeros != 0 {
+		return 0, errors.New("Invalid number")
+	}
+	number, err := strconv.ParseFloat(string(p.data[0:end-1]), 64)
+	if err != nil {
+		return 0, err
+	}
+	if math.IsInf(number, 0) || math.IsNaN(number) {
+		return 0, errors.New("Invalid number")
+	}
+	return number, nil
+}

vendor/github.com/klauspost/compress/.gitattributes

@@ -0,0 +1,2 @@
+* -text
+*.bin -text -diff

vendor/github.com/klauspost/compress/.gitignore

@@ -0,0 +1,32 @@
+# Compiled Object files, Static and Dynamic libs (Shared Objects)
+*.o
+*.a
+*.so
+
+# Folders
+_obj
+_test
+
+# Architecture specific extensions/prefixes
+*.[568vq]
+[568vq].out
+
+*.cgo1.go
+*.cgo2.c
+_cgo_defun.c
+_cgo_gotypes.go
+_cgo_export.*
+
+_testmain.go
+
+*.exe
+*.test
+*.prof
+/s2/cmd/_s2sx/sfx-exe
+
+# Linux perf files
+perf.data
+perf.data.old
+
+# gdb history
+.gdb_history

vendor/github.com/klauspost/compress/.goreleaser.yml

@@ -0,0 +1,132 @@
+version: 2
+
+before:
+  hooks:
+    - ./gen.sh
+
+builds:
+  -
+    id: "s2c"
+    binary: s2c
+    main: ./s2/cmd/s2c/main.go
+    flags:
+      - -trimpath
+    env:
+      - CGO_ENABLED=0
+    goos:
+      - aix
+      - linux
+      - freebsd
+      - netbsd
+      - windows
+      - darwin
+    goarch:
+      - 386
+      - amd64
+      - arm
+      - arm64
+      - ppc64
+      - ppc64le
+      - mips64
+      - mips64le
+    goarm:
+      - 7
+    ignore:
+      - goos: windows
+        goarch: arm
+  -
+    id: "s2d"
+    binary: s2d
+    main: ./s2/cmd/s2d/main.go
+    flags:
+      - -trimpath
+    env:
+      - CGO_ENABLED=0
+    goos:
+      - aix
+      - linux
+      - freebsd
+      - netbsd
+      - windows
+      - darwin
+    goarch:
+      - 386
+      - amd64
+      - arm
+      - arm64
+      - ppc64
+      - ppc64le
+      - mips64
+      - mips64le
+    goarm:
+      - 7
+    ignore:
+      - goos: windows
+        goarch: arm
+  -
+    id: "s2sx"
+    binary: s2sx
+    main: ./s2/cmd/_s2sx/main.go
+    flags:
+      - -modfile=s2sx.mod
+      - -trimpath
+    env:
+      - CGO_ENABLED=0
+    goos:
+      - aix
+      - linux
+      - freebsd
+      - netbsd
+      - windows
+      - darwin
+    goarch:
+      - 386
+      - amd64
+      - arm
+      - arm64
+      - ppc64
+      - ppc64le
+      - mips64
+      - mips64le
+    goarm:
+      - 7
+    ignore:
+      - goos: windows
+        goarch: arm
+
+archives:
+  -
+    id: s2-binaries
+    name_template: "s2-{{ .Os }}_{{ .Arch }}{{ if .Arm }}v{{ .Arm }}{{ end }}"
+    format_overrides:
+      - goos: windows
+        formats: ['zip']
+    files:
+      - unpack/*
+      - s2/LICENSE
+      - s2/README.md
+checksum:
+  name_template: 'checksums.txt'
+snapshot:
+  version_template: "{{ .Tag }}-next"
+changelog:
+  sort: asc
+  filters:
+    exclude:
+    - '^doc:'
+    - '^docs:'
+    - '^test:'
+    - '^tests:'
+    - '^Update\sREADME.md'
+
+nfpms:
+  -
+    file_name_template: "s2_package__{{ .Os }}_{{ .Arch }}{{ if .Arm }}v{{ .Arm }}{{ end }}"
+    vendor: Klaus Post
+    homepage: https://github.com/klauspost/compress
+    maintainer: Klaus Post <klauspost@gmail.com>
+    description: S2 Compression Tool
+    license: BSD 3-Clause
+    formats:
+      - deb
+      - rpm

vendor/github.com/klauspost/compress/LICENSE

@@ -0,0 +1,304 @@
+Copyright (c) 2012 The Go Authors. All rights reserved.
+Copyright (c) 2019 Klaus Post. All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are
+met:
+
+   * Redistributions of source code must retain the above copyright
+notice, this list of conditions and the following disclaimer.
+   * Redistributions in binary form must reproduce the above
+copyright notice, this list of conditions and the following disclaimer
+in the documentation and/or other materials provided with the
+distribution.
+   * Neither the name of Google Inc. nor the names of its
+contributors may be used to endorse or promote products derived from
+this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+------------------
+
+Files: gzhttp/*
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright 2016-2017 The New York Times Company
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+
+------------------
+
+Files: s2/cmd/internal/readahead/*
+
+The MIT License (MIT)
+
+Copyright (c) 2015 Klaus Post
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+
+---------------------
+Files: snappy/*
+Files: internal/snapref/*
+
+Copyright (c) 2011 The Snappy-Go Authors. All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are
+met:
+
+   * Redistributions of source code must retain the above copyright
+notice, this list of conditions and the following disclaimer.
+   * Redistributions in binary form must reproduce the above
+copyright notice, this list of conditions and the following disclaimer
+in the documentation and/or other materials provided with the
+distribution.
+   * Neither the name of Google Inc. nor the names of its
+contributors may be used to endorse or promote products derived from
+this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+-----------------
+
+Files: s2/cmd/internal/filepathx/*
+
+Copyright 2016 The filepathx Authors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

vendor/github.com/klauspost/compress/README.md

@@ -0,0 +1,700 @@
+# compress
+
+This package provides various compression algorithms.
+
+* [zstandard](https://github.com/klauspost/compress/tree/master/zstd#zstd) compression and decompression in pure Go.
+* [S2](https://github.com/klauspost/compress/tree/master/s2#s2-compression) is a high performance replacement for Snappy.
+* Optimized [deflate](https://godoc.org/github.com/klauspost/compress/flate) packages which can be used as a dropin replacement for [gzip](https://godoc.org/github.com/klauspost/compress/gzip), [zip](https://godoc.org/github.com/klauspost/compress/zip) and [zlib](https://godoc.org/github.com/klauspost/compress/zlib).
+* [snappy](https://github.com/klauspost/compress/tree/master/snappy) is a drop-in replacement for `github.com/golang/snappy` offering better compression and concurrent streams.
+* [huff0](https://github.com/klauspost/compress/tree/master/huff0) and [FSE](https://github.com/klauspost/compress/tree/master/fse) implementations for raw entropy encoding.
+* [gzhttp](https://github.com/klauspost/compress/tree/master/gzhttp) Provides client and server wrappers for handling gzipped/zstd HTTP requests efficiently.
+* [pgzip](https://github.com/klauspost/pgzip) is a separate package that provides a very fast parallel gzip implementation.
+
+[![Go Reference](https://pkg.go.dev/badge/klauspost/compress.svg)](https://pkg.go.dev/github.com/klauspost/compress?tab=subdirectories)
+[![Go](https://github.com/klauspost/compress/actions/workflows/go.yml/badge.svg)](https://github.com/klauspost/compress/actions/workflows/go.yml)
+[![Sourcegraph Badge](https://sourcegraph.com/github.com/klauspost/compress/-/badge.svg)](https://sourcegraph.com/github.com/klauspost/compress?badge)
+
+# package usage
+
+Use `go get github.com/klauspost/compress@latest` to add it to your project.
+
+This package will support the current Go version and 2 versions back.
+
+* Use the `nounsafe` tag to disable all use of the "unsafe" package.
+* Use the `noasm` tag to disable all assembly across packages.
+
+Use the links above for more information on each.
+
+# changelog
+
+* Feb 9th, 2026 [1.18.4](https://github.com/klauspost/compress/releases/tag/v1.18.4)
+	* gzhttp: Add zstandard to server handler wrapper https://github.com/klauspost/compress/pull/1121
+	* zstd: Add ResetWithOptions to encoder/decoder https://github.com/klauspost/compress/pull/1122
+	* gzhttp: preserve qvalue when extra parameters follow in Accept-Encoding by @analytically in https://github.com/klauspost/compress/pull/1116
+
+* Jan 16th, 2026 [1.18.3](https://github.com/klauspost/compress/releases/tag/v1.18.3)
+	* Downstream CVE-2025-61728. See [golang/go#77102](https://github.com/golang/go/issues/77102).
+
+* Dec 1st, 2025 - [1.18.2](https://github.com/klauspost/compress/releases/tag/v1.18.2)
+	* flate: Fix invalid encoding on level 9 with single value input in https://github.com/klauspost/compress/pull/1115
+	* flate: reduce stateless allocations by @RXamzin in https://github.com/klauspost/compress/pull/1106
+
+* Oct 20, 2025 - [1.18.1](https://github.com/klauspost/compress/releases/tag/v1.18.1) - RETRACTED
+  * zstd: Add simple zstd EncodeTo/DecodeTo functions  https://github.com/klauspost/compress/pull/1079
+  * zstd: Fix incorrect buffer size in dictionary encodes https://github.com/klauspost/compress/pull/1059
+  * s2: check for cap, not len of buffer in EncodeBetter/Best by @vdarulis in https://github.com/klauspost/compress/pull/1080
+  * zlib: Avoiding extra allocation in zlib.reader.Reset by @travelpolicy in https://github.com/klauspost/compress/pull/1086
+  * gzhttp: remove redundant err check in zstdReader by @ryanfowler in https://github.com/klauspost/compress/pull/1090
+  * flate: Faster load+store https://github.com/klauspost/compress/pull/1104
+  * flate: Simplify matchlen https://github.com/klauspost/compress/pull/1101
+  * flate: Use exact sizes for huffman tables https://github.com/klauspost/compress/pull/1103
+
+* Feb 19th, 2025 - [1.18.0](https://github.com/klauspost/compress/releases/tag/v1.18.0)
+  * Add unsafe little endian loaders https://github.com/klauspost/compress/pull/1036
+  * fix: check `r.err != nil` but return a nil value error `err` by @alingse in https://github.com/klauspost/compress/pull/1028
+  * flate: Simplify L4-6 loading https://github.com/klauspost/compress/pull/1043
+  * flate: Simplify matchlen (remove asm) https://github.com/klauspost/compress/pull/1045
+  * s2: Improve small block compression speed w/o asm https://github.com/klauspost/compress/pull/1048
+  * flate: Fix matchlen L5+L6 https://github.com/klauspost/compress/pull/1049
+  * flate: Cleanup & reduce casts https://github.com/klauspost/compress/pull/1050
+
+<details>
+	<summary>See changes to v1.17.x</summary>
+
+* Oct 11th, 2024 - [1.17.11](https://github.com/klauspost/compress/releases/tag/v1.17.11)
+  * zstd: Fix extra CRC written with multiple Close calls https://github.com/klauspost/compress/pull/1017
+  * s2: Don't use stack for index tables https://github.com/klauspost/compress/pull/1014
+  * gzhttp: No content-type on no body response code by @juliens in https://github.com/klauspost/compress/pull/1011
+  * gzhttp: Do not set the content-type when response has no body by @kevinpollet in https://github.com/klauspost/compress/pull/1013
+
+* Sep 23rd, 2024 - [1.17.10](https://github.com/klauspost/compress/releases/tag/v1.17.10)
+	* gzhttp: Add TransportAlwaysDecompress option. https://github.com/klauspost/compress/pull/978
+	* gzhttp: Add supported decompress request body by @mirecl in https://github.com/klauspost/compress/pull/1002
+	* s2: Add EncodeBuffer buffer recycling callback https://github.com/klauspost/compress/pull/982
+	* zstd: Improve memory usage on small streaming encodes https://github.com/klauspost/compress/pull/1007
+	* flate: read data written with partial flush by @vajexal in https://github.com/klauspost/compress/pull/996
+
+* Jun 12th, 2024 - [1.17.9](https://github.com/klauspost/compress/releases/tag/v1.17.9)
+	* s2: Reduce ReadFrom temporary allocations https://github.com/klauspost/compress/pull/949
+	* flate, zstd: Shave some bytes off amd64 matchLen by @greatroar in https://github.com/klauspost/compress/pull/963
+	* Upgrade zip/zlib to 1.22.4 upstream https://github.com/klauspost/compress/pull/970 https://github.com/klauspost/compress/pull/971
+	* zstd: BuildDict fails with RLE table https://github.com/klauspost/compress/pull/951
+
+* Apr 9th, 2024 - [1.17.8](https://github.com/klauspost/compress/releases/tag/v1.17.8)
+	* zstd: Reject blocks where reserved values are not 0 https://github.com/klauspost/compress/pull/885
+	* zstd: Add RLE detection+encoding https://github.com/klauspost/compress/pull/938
+
+* Feb 21st, 2024 - [1.17.7](https://github.com/klauspost/compress/releases/tag/v1.17.7)
+	* s2: Add AsyncFlush method: Complete the block without flushing by @Jille in https://github.com/klauspost/compress/pull/927
+	* s2: Fix literal+repeat exceeds dst crash https://github.com/klauspost/compress/pull/930
+  
+* Feb 5th, 2024 - [1.17.6](https://github.com/klauspost/compress/releases/tag/v1.17.6)
+	* zstd: Fix incorrect repeat coding in best mode https://github.com/klauspost/compress/pull/923
+	* s2: Fix DecodeConcurrent deadlock on errors https://github.com/klauspost/compress/pull/925
+  
+* Jan 26th, 2024 - [v1.17.5](https://github.com/klauspost/compress/releases/tag/v1.17.5)
+	* flate: Fix reset with dictionary on custom window encodes https://github.com/klauspost/compress/pull/912
+	* zstd: Add Frame header encoding and stripping https://github.com/klauspost/compress/pull/908
+	* zstd: Limit better/best default window to 8MB https://github.com/klauspost/compress/pull/913
+	* zstd: Speed improvements by @greatroar in https://github.com/klauspost/compress/pull/896 https://github.com/klauspost/compress/pull/910
+	* s2: Fix callbacks for skippable blocks and disallow 0xfe (Padding) by @Jille in https://github.com/klauspost/compress/pull/916 https://github.com/klauspost/compress/pull/917
+https://github.com/klauspost/compress/pull/919 https://github.com/klauspost/compress/pull/918
+
+* Dec 1st, 2023 - [v1.17.4](https://github.com/klauspost/compress/releases/tag/v1.17.4)
+	* huff0: Speed up symbol counting by @greatroar in https://github.com/klauspost/compress/pull/887
+	* huff0: Remove byteReader by @greatroar in https://github.com/klauspost/compress/pull/886
+	* gzhttp: Allow overriding decompression on transport https://github.com/klauspost/compress/pull/892
+	* gzhttp: Clamp compression level https://github.com/klauspost/compress/pull/890
+	* gzip: Error out if reserved bits are set https://github.com/klauspost/compress/pull/891
+
+* Nov 15th, 2023 - [v1.17.3](https://github.com/klauspost/compress/releases/tag/v1.17.3)
+	* fse: Fix max header size https://github.com/klauspost/compress/pull/881
+	* zstd: Improve better/best compression https://github.com/klauspost/compress/pull/877
+	* gzhttp: Fix missing content type on Close https://github.com/klauspost/compress/pull/883
+
+* Oct 22nd, 2023 - [v1.17.2](https://github.com/klauspost/compress/releases/tag/v1.17.2)
+	* zstd: Fix rare *CORRUPTION* output in "best" mode. See https://github.com/klauspost/compress/pull/876
+
+* Oct 14th, 2023 - [v1.17.1](https://github.com/klauspost/compress/releases/tag/v1.17.1)
+	* s2: Fix S2 "best" dictionary wrong encoding https://github.com/klauspost/compress/pull/871
+	* flate: Reduce allocations in decompressor and minor code improvements by @fakefloordiv in https://github.com/klauspost/compress/pull/869
+	* s2: Fix EstimateBlockSize on 6&7 length input https://github.com/klauspost/compress/pull/867
+
+* Sept 19th, 2023 - [v1.17.0](https://github.com/klauspost/compress/releases/tag/v1.17.0)
+	* Add experimental dictionary builder  https://github.com/klauspost/compress/pull/853
+	* Add xerial snappy read/writer https://github.com/klauspost/compress/pull/838
+	* flate: Add limited window compression https://github.com/klauspost/compress/pull/843
+	* s2: Do 2 overlapping match checks https://github.com/klauspost/compress/pull/839
+	* flate: Add amd64 assembly matchlen https://github.com/klauspost/compress/pull/837
+	* gzip: Copy bufio.Reader on Reset by @thatguystone in https://github.com/klauspost/compress/pull/860
+   
+</details>
+<details>
+	<summary>See changes to v1.16.x</summary>
+
+   
+* July 1st, 2023 - [v1.16.7](https://github.com/klauspost/compress/releases/tag/v1.16.7)
+	* zstd: Fix default level first dictionary encode https://github.com/klauspost/compress/pull/829
+	* s2: add GetBufferCapacity() method by @GiedriusS in https://github.com/klauspost/compress/pull/832
+
+* June 13, 2023 - [v1.16.6](https://github.com/klauspost/compress/releases/tag/v1.16.6)
+	* zstd: correctly ignore WithEncoderPadding(1) by @ianlancetaylor in https://github.com/klauspost/compress/pull/806
+	* zstd: Add amd64 match length assembly https://github.com/klauspost/compress/pull/824
+	* gzhttp: Handle informational headers by @rtribotte in https://github.com/klauspost/compress/pull/815
+	* s2: Improve Better compression slightly https://github.com/klauspost/compress/pull/663
+
+* Apr 16, 2023 - [v1.16.5](https://github.com/klauspost/compress/releases/tag/v1.16.5)
+	* zstd: readByte needs to use io.ReadFull by @jnoxon in https://github.com/klauspost/compress/pull/802
+	* gzip: Fix WriterTo after initial read https://github.com/klauspost/compress/pull/804
+
+* Apr 5, 2023 - [v1.16.4](https://github.com/klauspost/compress/releases/tag/v1.16.4)
+	* zstd: Improve zstd best efficiency by @greatroar and @klauspost in https://github.com/klauspost/compress/pull/784
+	* zstd: Respect WithAllLitEntropyCompression https://github.com/klauspost/compress/pull/792
+	* zstd: Fix amd64 not always detecting corrupt data https://github.com/klauspost/compress/pull/785
+	* zstd: Various minor improvements by @greatroar in https://github.com/klauspost/compress/pull/788 https://github.com/klauspost/compress/pull/794 https://github.com/klauspost/compress/pull/795
+	* s2: Fix huge block overflow https://github.com/klauspost/compress/pull/779
+	* s2: Allow CustomEncoder fallback https://github.com/klauspost/compress/pull/780
+	* gzhttp: Support ResponseWriter Unwrap() in gzhttp handler by @jgimenez in https://github.com/klauspost/compress/pull/799
+
+* Mar 13, 2023 - [v1.16.1](https://github.com/klauspost/compress/releases/tag/v1.16.1)
+	* zstd: Speed up + improve best encoder by @greatroar in https://github.com/klauspost/compress/pull/776
+	* gzhttp: Add optional [BREACH mitigation](https://github.com/klauspost/compress/tree/master/gzhttp#breach-mitigation). https://github.com/klauspost/compress/pull/762 https://github.com/klauspost/compress/pull/768 https://github.com/klauspost/compress/pull/769 https://github.com/klauspost/compress/pull/770 https://github.com/klauspost/compress/pull/767
+	* s2: Add Intel LZ4s converter https://github.com/klauspost/compress/pull/766
+	* zstd: Minor bug fixes https://github.com/klauspost/compress/pull/771 https://github.com/klauspost/compress/pull/772 https://github.com/klauspost/compress/pull/773
+	* huff0: Speed up compress1xDo by @greatroar in https://github.com/klauspost/compress/pull/774
+
+* Feb 26, 2023 - [v1.16.0](https://github.com/klauspost/compress/releases/tag/v1.16.0)
+	* s2: Add [Dictionary](https://github.com/klauspost/compress/tree/master/s2#dictionaries) support.  https://github.com/klauspost/compress/pull/685
+	* s2: Add Compression Size Estimate.  https://github.com/klauspost/compress/pull/752
+	* s2: Add support for custom stream encoder. https://github.com/klauspost/compress/pull/755
+	* s2: Add LZ4 block converter. https://github.com/klauspost/compress/pull/748
+	* s2: Support io.ReaderAt in ReadSeeker. https://github.com/klauspost/compress/pull/747
+	* s2c/s2sx: Use concurrent decoding. https://github.com/klauspost/compress/pull/746
+</details>
+
+<details>
+	<summary>See changes to v1.15.x</summary>
+	
+* Jan 21st, 2023 (v1.15.15)
+	* deflate: Improve level 7-9 https://github.com/klauspost/compress/pull/739
+	* zstd: Add delta encoding support by @greatroar in https://github.com/klauspost/compress/pull/728
+	* zstd: Various speed improvements by @greatroar https://github.com/klauspost/compress/pull/741 https://github.com/klauspost/compress/pull/734 https://github.com/klauspost/compress/pull/736 https://github.com/klauspost/compress/pull/744 https://github.com/klauspost/compress/pull/743 https://github.com/klauspost/compress/pull/745
+	* gzhttp: Add SuffixETag() and DropETag() options to prevent ETag collisions on compressed responses by @willbicks in https://github.com/klauspost/compress/pull/740
+
+* Jan 3rd, 2023 (v1.15.14)
+
+	* flate: Improve speed in big stateless blocks https://github.com/klauspost/compress/pull/718
+	* zstd: Minor speed tweaks by @greatroar in https://github.com/klauspost/compress/pull/716 https://github.com/klauspost/compress/pull/720
+	* export NoGzipResponseWriter for custom ResponseWriter wrappers by @harshavardhana in https://github.com/klauspost/compress/pull/722
+	* s2: Add example for indexing and existing stream https://github.com/klauspost/compress/pull/723
+
+* Dec 11, 2022 (v1.15.13)
+	* zstd: Add [MaxEncodedSize](https://pkg.go.dev/github.com/klauspost/compress@v1.15.13/zstd#Encoder.MaxEncodedSize) to encoder  https://github.com/klauspost/compress/pull/691
+	* zstd: Various tweaks and improvements https://github.com/klauspost/compress/pull/693 https://github.com/klauspost/compress/pull/695 https://github.com/klauspost/compress/pull/696 https://github.com/klauspost/compress/pull/701 https://github.com/klauspost/compress/pull/702 https://github.com/klauspost/compress/pull/703 https://github.com/klauspost/compress/pull/704 https://github.com/klauspost/compress/pull/705 https://github.com/klauspost/compress/pull/706 https://github.com/klauspost/compress/pull/707 https://github.com/klauspost/compress/pull/708
+
+* Oct 26, 2022 (v1.15.12)
+
+	* zstd: Tweak decoder allocs. https://github.com/klauspost/compress/pull/680
+	* gzhttp: Always delete `HeaderNoCompression` https://github.com/klauspost/compress/pull/683
+
+* Sept 26, 2022 (v1.15.11)
+
+	* flate: Improve level 1-3 compression  https://github.com/klauspost/compress/pull/678
+	* zstd: Improve "best" compression by @nightwolfz in https://github.com/klauspost/compress/pull/677
+	* zstd: Fix+reduce decompression allocations https://github.com/klauspost/compress/pull/668
+	* zstd: Fix non-effective noescape tag https://github.com/klauspost/compress/pull/667
+
+* Sept 16, 2022 (v1.15.10)
+
+	* zstd: Add [WithDecodeAllCapLimit](https://pkg.go.dev/github.com/klauspost/compress@v1.15.10/zstd#WithDecodeAllCapLimit) https://github.com/klauspost/compress/pull/649
+	* Add Go 1.19 - deprecate Go 1.16  https://github.com/klauspost/compress/pull/651
+	* flate: Improve level 5+6 compression https://github.com/klauspost/compress/pull/656
+	* zstd: Improve "better" compression  https://github.com/klauspost/compress/pull/657
+	* s2: Improve "best" compression https://github.com/klauspost/compress/pull/658
+	* s2: Improve "better" compression. https://github.com/klauspost/compress/pull/635
+	* s2: Slightly faster non-assembly decompression https://github.com/klauspost/compress/pull/646
+	* Use arrays for constant size copies https://github.com/klauspost/compress/pull/659
+
+* July 21, 2022 (v1.15.9)
+
+	* zstd: Fix decoder crash on amd64 (no BMI) on invalid input https://github.com/klauspost/compress/pull/645
+	* zstd: Disable decoder extended memory copies (amd64) due to possible crashes https://github.com/klauspost/compress/pull/644
+	* zstd: Allow single segments up to "max decoded size" https://github.com/klauspost/compress/pull/643
+
+* July 13, 2022 (v1.15.8)
+
+	* gzip: fix stack exhaustion bug in Reader.Read https://github.com/klauspost/compress/pull/641
+	* s2: Add Index header trim/restore https://github.com/klauspost/compress/pull/638
+	* zstd: Optimize seqdeq amd64 asm by @greatroar in https://github.com/klauspost/compress/pull/636
+	* zstd: Improve decoder memcopy https://github.com/klauspost/compress/pull/637
+	* huff0: Pass a single bitReader pointer to asm by @greatroar in https://github.com/klauspost/compress/pull/634
+	* zstd: Branchless getBits for amd64 w/o BMI2 by @greatroar in https://github.com/klauspost/compress/pull/640
+	* gzhttp: Remove header before writing https://github.com/klauspost/compress/pull/639
+
+* June 29, 2022 (v1.15.7)
+
+	* s2: Fix absolute forward seeks  https://github.com/klauspost/compress/pull/633
+	* zip: Merge upstream  https://github.com/klauspost/compress/pull/631
+	* zip: Re-add zip64 fix https://github.com/klauspost/compress/pull/624
+	* zstd: translate fseDecoder.buildDtable into asm by @WojciechMula in https://github.com/klauspost/compress/pull/598
+	* flate: Faster histograms  https://github.com/klauspost/compress/pull/620
+	* deflate: Use compound hcode  https://github.com/klauspost/compress/pull/622
+
+* June 3, 2022 (v1.15.6)
+	* s2: Improve coding for long, close matches https://github.com/klauspost/compress/pull/613
+	* s2c: Add Snappy/S2 stream recompression https://github.com/klauspost/compress/pull/611
+	* zstd: Always use configured block size https://github.com/klauspost/compress/pull/605
+	* zstd: Fix incorrect hash table placement for dict encoding in default https://github.com/klauspost/compress/pull/606
+	* zstd: Apply default config to ZipDecompressor without options https://github.com/klauspost/compress/pull/608
+	* gzhttp: Exclude more common archive formats https://github.com/klauspost/compress/pull/612
+	* s2: Add ReaderIgnoreCRC https://github.com/klauspost/compress/pull/609
+	* s2: Remove sanity load on index creation https://github.com/klauspost/compress/pull/607
+	* snappy: Use dedicated function for scoring https://github.com/klauspost/compress/pull/614
+	* s2c+s2d: Use official snappy framed extension https://github.com/klauspost/compress/pull/610
+
+* May 25, 2022 (v1.15.5)
+	* s2: Add concurrent stream decompression https://github.com/klauspost/compress/pull/602
+	* s2: Fix final emit oob read crash on amd64 https://github.com/klauspost/compress/pull/601
+	* huff0: asm implementation of Decompress1X by @WojciechMula https://github.com/klauspost/compress/pull/596
+	* zstd: Use 1 less goroutine for stream decoding https://github.com/klauspost/compress/pull/588
+	* zstd: Copy literal in 16 byte blocks when possible https://github.com/klauspost/compress/pull/592
+	* zstd: Speed up when WithDecoderLowmem(false) https://github.com/klauspost/compress/pull/599
+	* zstd: faster next state update in BMI2 version of decode by @WojciechMula in https://github.com/klauspost/compress/pull/593
+	* huff0: Do not check max size when reading table. https://github.com/klauspost/compress/pull/586
+	* flate: Inplace hashing for level 7-9 https://github.com/klauspost/compress/pull/590
+
+
+* May 11, 2022 (v1.15.4)
+	* huff0: decompress directly into output by @WojciechMula in [#577](https://github.com/klauspost/compress/pull/577)
+	* inflate: Keep dict on stack [#581](https://github.com/klauspost/compress/pull/581)
+	* zstd: Faster decoding memcopy in asm [#583](https://github.com/klauspost/compress/pull/583)
+	* zstd: Fix ignored crc [#580](https://github.com/klauspost/compress/pull/580)
+
+* May 5, 2022 (v1.15.3)
+	* zstd: Allow to ignore checksum checking by @WojciechMula [#572](https://github.com/klauspost/compress/pull/572)
+	* s2: Fix incorrect seek for io.SeekEnd in [#575](https://github.com/klauspost/compress/pull/575)
+
+* Apr 26, 2022 (v1.15.2)
+	* zstd: Add x86-64 assembly for decompression on streams and blocks. Contributed by [@WojciechMula](https://github.com/WojciechMula). Typically 2x faster.  [#528](https://github.com/klauspost/compress/pull/528) [#531](https://github.com/klauspost/compress/pull/531) [#545](https://github.com/klauspost/compress/pull/545) [#537](https://github.com/klauspost/compress/pull/537)
+	* zstd: Add options to ZipDecompressor and fixes [#539](https://github.com/klauspost/compress/pull/539)
+	* s2: Use sorted search for index [#555](https://github.com/klauspost/compress/pull/555)
+	* Minimum version is Go 1.16, added CI test on 1.18.
+
+* Mar 11, 2022 (v1.15.1)
+	* huff0: Add x86 assembly of Decode4X by @WojciechMula in [#512](https://github.com/klauspost/compress/pull/512)
+	* zstd: Reuse zip decoders in [#514](https://github.com/klauspost/compress/pull/514)
+	* zstd: Detect extra block data and report as corrupted in [#520](https://github.com/klauspost/compress/pull/520)
+	* zstd: Handle zero sized frame content size stricter in [#521](https://github.com/klauspost/compress/pull/521)
+	* zstd: Add stricter block size checks in [#523](https://github.com/klauspost/compress/pull/523)
+
+* Mar 3, 2022 (v1.15.0)
+	* zstd: Refactor decoder [#498](https://github.com/klauspost/compress/pull/498)
+	* zstd: Add stream encoding without goroutines [#505](https://github.com/klauspost/compress/pull/505)
+	* huff0: Prevent single blocks exceeding 16 bits by @klauspost in[#507](https://github.com/klauspost/compress/pull/507)
+	* flate: Inline literal emission [#509](https://github.com/klauspost/compress/pull/509)
+	* gzhttp: Add zstd to transport [#400](https://github.com/klauspost/compress/pull/400)
+	* gzhttp: Make content-type optional [#510](https://github.com/klauspost/compress/pull/510)
+
+Both compression and decompression now supports "synchronous" stream operations. This means that whenever "concurrency" is set to 1, they will operate without spawning goroutines.
+
+Stream decompression is now faster on asynchronous, since the goroutine allocation much more effectively splits the workload. On typical streams this will typically use 2 cores fully for decompression. When a stream has finished decoding no goroutines will be left over, so decoders can now safely be pooled and still be garbage collected.
+
+While the release has been extensively tested, it is recommended to testing when upgrading.
+
+</details>
+
+<details>
+	<summary>See changes to v1.14.x</summary>
+	
+* Feb 22, 2022 (v1.14.4)
+	* flate: Fix rare huffman only (-2) corruption. [#503](https://github.com/klauspost/compress/pull/503)
+	* zip: Update deprecated CreateHeaderRaw to correctly call CreateRaw by @saracen in [#502](https://github.com/klauspost/compress/pull/502)
+	* zip: don't read data descriptor early by @saracen in [#501](https://github.com/klauspost/compress/pull/501)  #501
+	* huff0: Use static decompression buffer up to 30% faster [#499](https://github.com/klauspost/compress/pull/499) [#500](https://github.com/klauspost/compress/pull/500)
+
+* Feb 17, 2022 (v1.14.3)
+	* flate: Improve fastest levels compression speed ~10% more throughput. [#482](https://github.com/klauspost/compress/pull/482) [#489](https://github.com/klauspost/compress/pull/489) [#490](https://github.com/klauspost/compress/pull/490) [#491](https://github.com/klauspost/compress/pull/491) [#494](https://github.com/klauspost/compress/pull/494)  [#478](https://github.com/klauspost/compress/pull/478)
+	* flate: Faster decompression speed, ~5-10%. [#483](https://github.com/klauspost/compress/pull/483)
+	* s2: Faster compression with Go v1.18 and amd64 microarch level 3+. [#484](https://github.com/klauspost/compress/pull/484) [#486](https://github.com/klauspost/compress/pull/486)
+
+* Jan 25, 2022 (v1.14.2)
+	* zstd: improve header decoder by @dsnet  [#476](https://github.com/klauspost/compress/pull/476)
+	* zstd: Add bigger default blocks  [#469](https://github.com/klauspost/compress/pull/469)
+	* zstd: Remove unused decompression buffer [#470](https://github.com/klauspost/compress/pull/470)
+	* zstd: Fix logically dead code by @ningmingxiao [#472](https://github.com/klauspost/compress/pull/472)
+	* flate: Improve level 7-9 [#471](https://github.com/klauspost/compress/pull/471) [#473](https://github.com/klauspost/compress/pull/473)
+	* zstd: Add noasm tag for xxhash [#475](https://github.com/klauspost/compress/pull/475)
+
+* Jan 11, 2022 (v1.14.1)
+	* s2: Add stream index in [#462](https://github.com/klauspost/compress/pull/462)
+	* flate: Speed and efficiency improvements in [#439](https://github.com/klauspost/compress/pull/439) [#461](https://github.com/klauspost/compress/pull/461) [#455](https://github.com/klauspost/compress/pull/455) [#452](https://github.com/klauspost/compress/pull/452) [#458](https://github.com/klauspost/compress/pull/458)
+	* zstd: Performance improvement in [#420]( https://github.com/klauspost/compress/pull/420) [#456](https://github.com/klauspost/compress/pull/456) [#437](https://github.com/klauspost/compress/pull/437) [#467](https://github.com/klauspost/compress/pull/467) [#468](https://github.com/klauspost/compress/pull/468)
+	* zstd: add arm64 xxhash assembly in [#464](https://github.com/klauspost/compress/pull/464)
+	* Add garbled for binaries for s2 in [#445](https://github.com/klauspost/compress/pull/445)
+</details>
+
+<details>
+	<summary>See changes to v1.13.x</summary>
+	
+* Aug 30, 2021 (v1.13.5)
+	* gz/zlib/flate: Alias stdlib errors [#425](https://github.com/klauspost/compress/pull/425)
+	* s2: Add block support to commandline tools [#413](https://github.com/klauspost/compress/pull/413)
+	* zstd: pooledZipWriter should return Writers to the same pool [#426](https://github.com/klauspost/compress/pull/426)
+	* Removed golang/snappy as external dependency for tests [#421](https://github.com/klauspost/compress/pull/421)
+
+* Aug 12, 2021 (v1.13.4)
+	* Add [snappy replacement package](https://github.com/klauspost/compress/tree/master/snappy).
+	* zstd: Fix incorrect encoding in "best" mode [#415](https://github.com/klauspost/compress/pull/415)
+
+* Aug 3, 2021 (v1.13.3) 
+	* zstd: Improve Best compression [#404](https://github.com/klauspost/compress/pull/404)
+	* zstd: Fix WriteTo error forwarding [#411](https://github.com/klauspost/compress/pull/411)
+	* gzhttp: Return http.HandlerFunc instead of http.Handler. Unlikely breaking change. [#406](https://github.com/klauspost/compress/pull/406)
+	* s2sx: Fix max size error [#399](https://github.com/klauspost/compress/pull/399)
+	* zstd: Add optional stream content size on reset [#401](https://github.com/klauspost/compress/pull/401)
+	* zstd: use SpeedBestCompression for level >= 10 [#410](https://github.com/klauspost/compress/pull/410)
+
+* Jun 14, 2021 (v1.13.1)
+	* s2: Add full Snappy output support  [#396](https://github.com/klauspost/compress/pull/396)
+	* zstd: Add configurable [Decoder window](https://pkg.go.dev/github.com/klauspost/compress/zstd#WithDecoderMaxWindow) size [#394](https://github.com/klauspost/compress/pull/394)
+	* gzhttp: Add header to skip compression  [#389](https://github.com/klauspost/compress/pull/389)
+	* s2: Improve speed with bigger output margin  [#395](https://github.com/klauspost/compress/pull/395)
+
+* Jun 3, 2021 (v1.13.0)
+	* Added [gzhttp](https://github.com/klauspost/compress/tree/master/gzhttp#gzip-handler) which allows wrapping HTTP servers and clients with GZIP compressors.
+	* zstd: Detect short invalid signatures [#382](https://github.com/klauspost/compress/pull/382)
+	* zstd: Spawn decoder goroutine only if needed. [#380](https://github.com/klauspost/compress/pull/380)
+</details>
+
+
+<details>
+	<summary>See changes to v1.12.x</summary>
+	
+* May 25, 2021 (v1.12.3)
+	* deflate: Better/faster Huffman encoding [#374](https://github.com/klauspost/compress/pull/374)
+	* deflate: Allocate less for history. [#375](https://github.com/klauspost/compress/pull/375)
+	* zstd: Forward read errors [#373](https://github.com/klauspost/compress/pull/373) 
+
+* Apr 27, 2021 (v1.12.2)
+	* zstd: Improve better/best compression [#360](https://github.com/klauspost/compress/pull/360) [#364](https://github.com/klauspost/compress/pull/364) [#365](https://github.com/klauspost/compress/pull/365)
+	* zstd: Add helpers to compress/decompress zstd inside zip files [#363](https://github.com/klauspost/compress/pull/363)
+	* deflate: Improve level 5+6 compression [#367](https://github.com/klauspost/compress/pull/367)
+	* s2: Improve better/best compression [#358](https://github.com/klauspost/compress/pull/358) [#359](https://github.com/klauspost/compress/pull/358)
+	* s2: Load after checking src limit on amd64. [#362](https://github.com/klauspost/compress/pull/362)
+	* s2sx: Limit max executable size [#368](https://github.com/klauspost/compress/pull/368) 
+
+* Apr 14, 2021 (v1.12.1)
+	* snappy package removed. Upstream added as dependency.
+	* s2: Better compression in "best" mode [#353](https://github.com/klauspost/compress/pull/353)
+	* s2sx: Add stdin input and detect pre-compressed from signature [#352](https://github.com/klauspost/compress/pull/352)
+	* s2c/s2d: Add http as possible input [#348](https://github.com/klauspost/compress/pull/348)
+	* s2c/s2d/s2sx: Always truncate when writing files [#352](https://github.com/klauspost/compress/pull/352)
+	* zstd: Reduce memory usage further when using [WithLowerEncoderMem](https://pkg.go.dev/github.com/klauspost/compress/zstd#WithLowerEncoderMem) [#346](https://github.com/klauspost/compress/pull/346)
+	* s2: Fix potential problem with amd64 assembly and profilers [#349](https://github.com/klauspost/compress/pull/349)
+</details>
+
+<details>
+	<summary>See changes to v1.11.x</summary>
+	
+* Mar 26, 2021 (v1.11.13)
+	* zstd: Big speedup on small dictionary encodes [#344](https://github.com/klauspost/compress/pull/344) [#345](https://github.com/klauspost/compress/pull/345)
+	* zstd: Add [WithLowerEncoderMem](https://pkg.go.dev/github.com/klauspost/compress/zstd#WithLowerEncoderMem) encoder option [#336](https://github.com/klauspost/compress/pull/336)
+	* deflate: Improve entropy compression [#338](https://github.com/klauspost/compress/pull/338)
+	* s2: Clean up and minor performance improvement in best [#341](https://github.com/klauspost/compress/pull/341)
+
+* Mar 5, 2021 (v1.11.12)
+	* s2: Add `s2sx` binary that creates [self extracting archives](https://github.com/klauspost/compress/tree/master/s2#s2sx-self-extracting-archives).
+	* s2: Speed up decompression on non-assembly platforms [#328](https://github.com/klauspost/compress/pull/328)
+
+* Mar 1, 2021 (v1.11.9)
+	* s2: Add ARM64 decompression assembly. Around 2x output speed. [#324](https://github.com/klauspost/compress/pull/324)
+	* s2: Improve "better" speed and efficiency. [#325](https://github.com/klauspost/compress/pull/325)
+	* s2: Fix binaries.
+
+* Feb 25, 2021 (v1.11.8)
+	* s2: Fixed occasional out-of-bounds write on amd64. Upgrade recommended.
+	* s2: Add AMD64 assembly for better mode. 25-50% faster. [#315](https://github.com/klauspost/compress/pull/315)
+	* s2: Less upfront decoder allocation. [#322](https://github.com/klauspost/compress/pull/322)
+	* zstd: Faster "compression" of incompressible data. [#314](https://github.com/klauspost/compress/pull/314)
+	* zip: Fix zip64 headers. [#313](https://github.com/klauspost/compress/pull/313)
+  
+* Jan 14, 2021 (v1.11.7)
+	* Use Bytes() interface to get bytes across packages. [#309](https://github.com/klauspost/compress/pull/309)
+	* s2: Add 'best' compression option.  [#310](https://github.com/klauspost/compress/pull/310)
+	* s2: Add ReaderMaxBlockSize, changes `s2.NewReader` signature to include varargs. [#311](https://github.com/klauspost/compress/pull/311)
+	* s2: Fix crash on small better buffers. [#308](https://github.com/klauspost/compress/pull/308)
+	* s2: Clean up decoder. [#312](https://github.com/klauspost/compress/pull/312)
+
+* Jan 7, 2021 (v1.11.6)
+	* zstd: Make decoder allocations smaller [#306](https://github.com/klauspost/compress/pull/306)
+	* zstd: Free Decoder resources when Reset is called with a nil io.Reader  [#305](https://github.com/klauspost/compress/pull/305)
+
+* Dec 20, 2020 (v1.11.4)
+	* zstd: Add Best compression mode [#304](https://github.com/klauspost/compress/pull/304)
+	* Add header decoder [#299](https://github.com/klauspost/compress/pull/299)
+	* s2: Add uncompressed stream option [#297](https://github.com/klauspost/compress/pull/297)
+	* Simplify/speed up small blocks with known max size. [#300](https://github.com/klauspost/compress/pull/300)
+	* zstd: Always reset literal dict encoder [#303](https://github.com/klauspost/compress/pull/303)
+
+* Nov 15, 2020 (v1.11.3)
+	* inflate: 10-15% faster decompression  [#293](https://github.com/klauspost/compress/pull/293)
+	* zstd: Tweak DecodeAll default allocation [#295](https://github.com/klauspost/compress/pull/295)
+
+* Oct 11, 2020 (v1.11.2)
+	* s2: Fix out of bounds read in "better" block compression [#291](https://github.com/klauspost/compress/pull/291)
+
+* Oct 1, 2020 (v1.11.1)
+	* zstd: Set allLitEntropy true in default configuration [#286](https://github.com/klauspost/compress/pull/286)
+
+* Sept 8, 2020 (v1.11.0)
+	* zstd: Add experimental compression [dictionaries](https://github.com/klauspost/compress/tree/master/zstd#dictionaries) [#281](https://github.com/klauspost/compress/pull/281)
+	* zstd: Fix mixed Write and ReadFrom calls [#282](https://github.com/klauspost/compress/pull/282)
+	* inflate/gz: Limit variable shifts, ~5% faster decompression [#274](https://github.com/klauspost/compress/pull/274)
+</details>
+
+<details>
+	<summary>See changes to v1.10.x</summary>
+ 
+* July 8, 2020 (v1.10.11) 
+	* zstd: Fix extra block when compressing with ReadFrom. [#278](https://github.com/klauspost/compress/pull/278)
+	* huff0: Also populate compression table when reading decoding table. [#275](https://github.com/klauspost/compress/pull/275)
+	
+* June 23, 2020 (v1.10.10) 
+	* zstd: Skip entropy compression in fastest mode when no matches. [#270](https://github.com/klauspost/compress/pull/270)
+	
+* June 16, 2020 (v1.10.9): 
+	* zstd: API change for specifying dictionaries. See [#268](https://github.com/klauspost/compress/pull/268)
+	* zip: update CreateHeaderRaw to handle zip64 fields. [#266](https://github.com/klauspost/compress/pull/266)
+	* Fuzzit tests removed. The service has been purchased and is no longer available.
+	
+* June 5, 2020 (v1.10.8): 
+	* 1.15x faster zstd block decompression. [#265](https://github.com/klauspost/compress/pull/265)
+	
+* June 1, 2020 (v1.10.7): 
+	* Added zstd decompression [dictionary support](https://github.com/klauspost/compress/tree/master/zstd#dictionaries)
+	* Increase zstd decompression speed up to 1.19x.  [#259](https://github.com/klauspost/compress/pull/259)
+	* Remove internal reset call in zstd compression and reduce allocations. [#263](https://github.com/klauspost/compress/pull/263)
+	
+* May 21, 2020: (v1.10.6) 
+	* zstd: Reduce allocations while decoding. [#258](https://github.com/klauspost/compress/pull/258), [#252](https://github.com/klauspost/compress/pull/252)
+	* zstd: Stricter decompression checks.
+	
+* April 12, 2020: (v1.10.5)
+	* s2-commands: Flush output when receiving SIGINT. [#239](https://github.com/klauspost/compress/pull/239)
+	
+* Apr 8, 2020: (v1.10.4) 
+	* zstd: Minor/special case optimizations. [#251](https://github.com/klauspost/compress/pull/251),  [#250](https://github.com/klauspost/compress/pull/250),  [#249](https://github.com/klauspost/compress/pull/249),  [#247](https://github.com/klauspost/compress/pull/247)
+* Mar 11, 2020: (v1.10.3) 
+	* s2: Use S2 encoder in pure Go mode for Snappy output as well. [#245](https://github.com/klauspost/compress/pull/245)
+	* s2: Fix pure Go block encoder. [#244](https://github.com/klauspost/compress/pull/244)
+	* zstd: Added "better compression" mode. [#240](https://github.com/klauspost/compress/pull/240)
+	* zstd: Improve speed of fastest compression mode by 5-10% [#241](https://github.com/klauspost/compress/pull/241)
+	* zstd: Skip creating encoders when not needed. [#238](https://github.com/klauspost/compress/pull/238)
+	
+* Feb 27, 2020: (v1.10.2) 
+	* Close to 50% speedup in inflate (gzip/zip decompression). [#236](https://github.com/klauspost/compress/pull/236) [#234](https://github.com/klauspost/compress/pull/234) [#232](https://github.com/klauspost/compress/pull/232)
+	* Reduce deflate level 1-6 memory usage up to 59%. [#227](https://github.com/klauspost/compress/pull/227)
+	
+* Feb 18, 2020: (v1.10.1)
+	* Fix zstd crash when resetting multiple times without sending data. [#226](https://github.com/klauspost/compress/pull/226)
+	* deflate: Fix dictionary use on level 1-6. [#224](https://github.com/klauspost/compress/pull/224)
+	* Remove deflate writer reference when closing. [#224](https://github.com/klauspost/compress/pull/224)
+	
+* Feb 4, 2020: (v1.10.0) 
+	* Add optional dictionary to [stateless deflate](https://pkg.go.dev/github.com/klauspost/compress/flate?tab=doc#StatelessDeflate). Breaking change, send `nil` for previous behaviour. [#216](https://github.com/klauspost/compress/pull/216)
+	* Fix buffer overflow on repeated small block deflate.  [#218](https://github.com/klauspost/compress/pull/218)
+	* Allow copying content from an existing ZIP file without decompressing+compressing. [#214](https://github.com/klauspost/compress/pull/214)
+	* Added [S2](https://github.com/klauspost/compress/tree/master/s2#s2-compression) AMD64 assembler and various optimizations. Stream speed >10GB/s.  [#186](https://github.com/klauspost/compress/pull/186)
+
+</details>
+
+<details>
+	<summary>See changes prior to v1.10.0</summary>
+
+* Jan 20,2020 (v1.9.8) Optimize gzip/deflate with better size estimates and faster table generation. [#207](https://github.com/klauspost/compress/pull/207) by [luyu6056](https://github.com/luyu6056),  [#206](https://github.com/klauspost/compress/pull/206).
+* Jan 11, 2020: S2 Encode/Decode will use provided buffer if capacity is big enough. [#204](https://github.com/klauspost/compress/pull/204) 
+* Jan 5, 2020: (v1.9.7) Fix another zstd regression in v1.9.5 - v1.9.6 removed.
+* Jan 4, 2020: (v1.9.6) Regression in v1.9.5 fixed causing corrupt zstd encodes in rare cases.
+* Jan 4, 2020: Faster IO in [s2c + s2d commandline tools](https://github.com/klauspost/compress/tree/master/s2#commandline-tools) compression/decompression. [#192](https://github.com/klauspost/compress/pull/192)
+* Dec 29, 2019: Removed v1.9.5 since fuzz tests showed a compatibility problem with the reference zstandard decoder.
+* Dec 29, 2019: (v1.9.5) zstd: 10-20% faster block compression. [#199](https://github.com/klauspost/compress/pull/199)
+* Dec 29, 2019: [zip](https://godoc.org/github.com/klauspost/compress/zip) package updated with latest Go features
+* Dec 29, 2019: zstd: Single segment flag condintions tweaked. [#197](https://github.com/klauspost/compress/pull/197)
+* Dec 18, 2019: s2: Faster compression when ReadFrom is used. [#198](https://github.com/klauspost/compress/pull/198)
+* Dec 10, 2019: s2: Fix repeat length output when just above at 16MB limit.
+* Dec 10, 2019: zstd: Add function to get decoder as io.ReadCloser. [#191](https://github.com/klauspost/compress/pull/191)
+* Dec 3, 2019: (v1.9.4) S2: limit max repeat length. [#188](https://github.com/klauspost/compress/pull/188)
+* Dec 3, 2019: Add [WithNoEntropyCompression](https://godoc.org/github.com/klauspost/compress/zstd#WithNoEntropyCompression) to zstd [#187](https://github.com/klauspost/compress/pull/187)
+* Dec 3, 2019: Reduce memory use for tests. Check for leaked goroutines.
+* Nov 28, 2019 (v1.9.3) Less allocations in stateless deflate.
+* Nov 28, 2019: 5-20% Faster huff0 decode. Impacts zstd as well. [#184](https://github.com/klauspost/compress/pull/184)
+* Nov 12, 2019 (v1.9.2) Added [Stateless Compression](#stateless-compression) for gzip/deflate.
+* Nov 12, 2019: Fixed zstd decompression of large single blocks. [#180](https://github.com/klauspost/compress/pull/180)
+* Nov 11, 2019: Set default  [s2c](https://github.com/klauspost/compress/tree/master/s2#commandline-tools) block size to 4MB.
+* Nov 11, 2019: Reduce inflate memory use by 1KB.
+* Nov 10, 2019: Less allocations in deflate bit writer.
+* Nov 10, 2019: Fix inconsistent error returned by zstd decoder.
+* Oct 28, 2019 (v1.9.1) ztsd: Fix crash when compressing blocks. [#174](https://github.com/klauspost/compress/pull/174)
+* Oct 24, 2019 (v1.9.0) zstd: Fix rare data corruption [#173](https://github.com/klauspost/compress/pull/173)
+* Oct 24, 2019 zstd: Fix huff0 out of buffer write [#171](https://github.com/klauspost/compress/pull/171) and always return errors [#172](https://github.com/klauspost/compress/pull/172) 
+* Oct 10, 2019: Big deflate rewrite, 30-40% faster with better compression [#105](https://github.com/klauspost/compress/pull/105)
+
+</details>
+
+<details>
+	<summary>See changes prior to v1.9.0</summary>
+
+* Oct 10, 2019: (v1.8.6) zstd: Allow partial reads to get flushed data. [#169](https://github.com/klauspost/compress/pull/169)
+* Oct 3, 2019: Fix inconsistent results on broken zstd streams.
+* Sep 25, 2019: Added `-rm` (remove source files) and `-q` (no output except errors) to `s2c` and `s2d` [commands](https://github.com/klauspost/compress/tree/master/s2#commandline-tools)
+* Sep 16, 2019: (v1.8.4) Add `s2c` and `s2d` [commandline tools](https://github.com/klauspost/compress/tree/master/s2#commandline-tools).
+* Sep 10, 2019: (v1.8.3) Fix s2 decoder [Skip](https://godoc.org/github.com/klauspost/compress/s2#Reader.Skip).
+* Sep 7, 2019: zstd: Added [WithWindowSize](https://godoc.org/github.com/klauspost/compress/zstd#WithWindowSize), contributed by [ianwilkes](https://github.com/ianwilkes).
+* Sep 5, 2019: (v1.8.2) Add [WithZeroFrames](https://godoc.org/github.com/klauspost/compress/zstd#WithZeroFrames) which adds full zero payload block encoding option.
+* Sep 5, 2019: Lazy initialization of zstandard predefined en/decoder tables.
+* Aug 26, 2019: (v1.8.1) S2: 1-2% compression increase in "better" compression mode.
+* Aug 26, 2019: zstd: Check maximum size of Huffman 1X compressed literals while decoding.
+* Aug 24, 2019: (v1.8.0) Added [S2 compression](https://github.com/klauspost/compress/tree/master/s2#s2-compression), a high performance replacement for Snappy. 
+* Aug 21, 2019: (v1.7.6) Fixed minor issues found by fuzzer. One could lead to zstd not decompressing.
+* Aug 18, 2019: Add [fuzzit](https://fuzzit.dev/) continuous fuzzing.
+* Aug 14, 2019: zstd: Skip incompressible data 2x faster.  [#147](https://github.com/klauspost/compress/pull/147)
+* Aug 4, 2019 (v1.7.5): Better literal compression. [#146](https://github.com/klauspost/compress/pull/146)
+* Aug 4, 2019: Faster zstd compression. [#143](https://github.com/klauspost/compress/pull/143) [#144](https://github.com/klauspost/compress/pull/144)
+* Aug 4, 2019: Faster zstd decompression. [#145](https://github.com/klauspost/compress/pull/145) [#143](https://github.com/klauspost/compress/pull/143) [#142](https://github.com/klauspost/compress/pull/142)
+* July 15, 2019 (v1.7.4): Fix double EOF block in rare cases on zstd encoder.
+* July 15, 2019 (v1.7.3): Minor speedup/compression increase in default zstd encoder.
+* July 14, 2019: zstd decoder: Fix decompression error on multiple uses with mixed content.
+* July 7, 2019 (v1.7.2): Snappy update, zstd decoder potential race fix.
+* June 17, 2019: zstd decompression bugfix.
+* June 17, 2019: fix 32 bit builds.
+* June 17, 2019: Easier use in modules (less dependencies).
+* June 9, 2019: New stronger "default" [zstd](https://github.com/klauspost/compress/tree/master/zstd#zstd) compression mode. Matches zstd default compression ratio.
+* June 5, 2019: 20-40% throughput in [zstandard](https://github.com/klauspost/compress/tree/master/zstd#zstd) compression and better compression.
+* June 5, 2019: deflate/gzip compression: Reduce memory usage of lower compression levels.
+* June 2, 2019: Added [zstandard](https://github.com/klauspost/compress/tree/master/zstd#zstd) compression!
+* May 25, 2019: deflate/gzip: 10% faster bit writer, mostly visible in lower levels.
+* Apr 22, 2019: [zstd](https://github.com/klauspost/compress/tree/master/zstd#zstd) decompression added.
+* Aug 1, 2018: Added [huff0 README](https://github.com/klauspost/compress/tree/master/huff0#huff0-entropy-compression).
+* Jul 8, 2018: Added [Performance Update 2018](#performance-update-2018) below.
+* Jun 23, 2018: Merged [Go 1.11 inflate optimizations](https://go-review.googlesource.com/c/go/+/102235). Go 1.9 is now required. Backwards compatible version tagged with [v1.3.0](https://github.com/klauspost/compress/releases/tag/v1.3.0).
+* Apr 2, 2018: Added [huff0](https://godoc.org/github.com/klauspost/compress/huff0) en/decoder. Experimental for now, API may change.
+* Mar 4, 2018: Added [FSE Entropy](https://godoc.org/github.com/klauspost/compress/fse) en/decoder. Experimental for now, API may change.
+* Nov 3, 2017: Add compression [Estimate](https://godoc.org/github.com/klauspost/compress#Estimate) function.
+* May 28, 2017: Reduce allocations when resetting decoder.
+* Apr 02, 2017: Change back to official crc32, since changes were merged in Go 1.7.
+* Jan 14, 2017: Reduce stack pressure due to array copies. See [Issue #18625](https://github.com/golang/go/issues/18625).
+* Oct 25, 2016: Level 2-4 have been rewritten and now offers significantly better performance than before.
+* Oct 20, 2016: Port zlib changes from Go 1.7 to fix zlib writer issue. Please update.
+* Oct 16, 2016: Go 1.7 changes merged. Apples to apples this package is a few percent faster, but has a significantly better balance between speed and compression per level. 
+* Mar 24, 2016: Always attempt Huffman encoding on level 4-7. This improves base 64 encoded data compression.
+* Mar 24, 2016: Small speedup for level 1-3.
+* Feb 19, 2016: Faster bit writer, level -2 is 15% faster, level 1 is 4% faster.
+* Feb 19, 2016: Handle small payloads faster in level 1-3.
+* Feb 19, 2016: Added faster level 2 + 3 compression modes.
+* Feb 19, 2016: [Rebalanced compression levels](https://blog.klauspost.com/rebalancing-deflate-compression-levels/), so there is a more even progression in terms of compression. New default level is 5.
+* Feb 14, 2016: Snappy: Merge upstream changes. 
+* Feb 14, 2016: Snappy: Fix aggressive skipping.
+* Feb 14, 2016: Snappy: Update benchmark.
+* Feb 13, 2016: Deflate: Fixed assembler problem that could lead to sub-optimal compression.
+* Feb 12, 2016: Snappy: Added AMD64 SSE 4.2 optimizations to matching, which makes easy to compress material run faster. Typical speedup is around 25%.
+* Feb 9, 2016: Added Snappy package fork. This version is 5-7% faster, much more on hard to compress content.
+* Jan 30, 2016: Optimize level 1 to 3 by not considering static dictionary or storing uncompressed. ~4-5% speedup.
+* Jan 16, 2016: Optimization on deflate level 1,2,3 compression.
+* Jan 8 2016: Merge [CL 18317](https://go-review.googlesource.com/#/c/18317): fix reading, writing of zip64 archives.
+* Dec 8 2015: Make level 1 and -2 deterministic even if write size differs.
+* Dec 8 2015: Split encoding functions, so hashing and matching can potentially be inlined. 1-3% faster on AMD64. 5% faster on other platforms.
+* Dec 8 2015: Fixed rare [one byte out-of bounds read](https://github.com/klauspost/compress/issues/20). Please update!
+* Nov 23 2015: Optimization on token writer. ~2-4% faster. Contributed by [@dsnet](https://github.com/dsnet).
+* Nov 20 2015: Small optimization to bit writer on 64 bit systems.
+* Nov 17 2015: Fixed out-of-bound errors if the underlying Writer returned an error. See [#15](https://github.com/klauspost/compress/issues/15).
+* Nov 12 2015: Added [io.WriterTo](https://golang.org/pkg/io/#WriterTo) support to gzip/inflate.
+* Nov 11 2015: Merged [CL 16669](https://go-review.googlesource.com/#/c/16669/4): archive/zip: enable overriding (de)compressors per file
+* Oct 15 2015: Added skipping on uncompressible data. Random data speed up >5x.
+
+</details>
+
+# deflate usage
+
+The packages are drop-in replacements for standard library [deflate](https://godoc.org/github.com/klauspost/compress/flate), [gzip](https://godoc.org/github.com/klauspost/compress/gzip), [zip](https://godoc.org/github.com/klauspost/compress/zip), and [zlib](https://godoc.org/github.com/klauspost/compress/zlib). Simply replace the import path to use them:
+
+Typical speed is about 2x of the standard library packages.
+
+| old import       | new import                            | Documentation                                                           |
+|------------------|---------------------------------------|-------------------------------------------------------------------------|
+| `compress/gzip`  | `github.com/klauspost/compress/gzip`  | [gzip](https://pkg.go.dev/github.com/klauspost/compress/gzip?tab=doc)   |
+| `compress/zlib`  | `github.com/klauspost/compress/zlib`  | [zlib](https://pkg.go.dev/github.com/klauspost/compress/zlib?tab=doc)   |
+| `archive/zip`    | `github.com/klauspost/compress/zip`   | [zip](https://pkg.go.dev/github.com/klauspost/compress/zip?tab=doc)     |
+| `compress/flate` | `github.com/klauspost/compress/flate` | [flate](https://pkg.go.dev/github.com/klauspost/compress/flate?tab=doc) |
+
+You may also be interested in [pgzip](https://github.com/klauspost/pgzip), which is a drop-in replacement for gzip, which support multithreaded compression on big files and the optimized [crc32](https://github.com/klauspost/crc32) package used by these packages.
+
+The packages implement the same API as the standard library, so you can use the original godoc documentation: [gzip](http://golang.org/pkg/compress/gzip/), [zip](http://golang.org/pkg/archive/zip/), [zlib](http://golang.org/pkg/compress/zlib/), [flate](http://golang.org/pkg/compress/flate/).
+
+Currently there is only minor speedup on decompression (mostly CRC32 calculation).
+
+Memory usage is typically 1MB for a Writer. stdlib is in the same range. 
+If you expect to have a lot of concurrently allocated Writers consider using 
+the stateless compression described below.
+
+For compression performance, see: [this spreadsheet](https://docs.google.com/spreadsheets/d/1nuNE2nPfuINCZJRMt6wFWhKpToF95I47XjSsc-1rbPQ/edit?usp=sharing).
+
+To disable all assembly add `-tags=noasm`. This works across all packages.
+
+# Stateless compression
+
+This package offers stateless compression as a special option for gzip/deflate. 
+It will do compression but without maintaining any state between Write calls.
+
+This means there will be no memory kept between Write calls, but compression and speed will be suboptimal.
+
+This is only relevant in cases where you expect to run many thousands of compressors concurrently, 
+but with very little activity. This is *not* intended for regular web servers serving individual requests.  
+
+Because of this, the size of actual Write calls will affect output size.
+
+In gzip, specify level `-3` / `gzip.StatelessCompression` to enable.
+
+For direct deflate use, NewStatelessWriter and StatelessDeflate are available. See [documentation](https://godoc.org/github.com/klauspost/compress/flate#NewStatelessWriter)
+
+A `bufio.Writer` can of course be used to control write sizes. For example, to use a 4KB buffer:
+
+```go
+	// replace 'ioutil.Discard' with your output.
+	gzw, err := gzip.NewWriterLevel(ioutil.Discard, gzip.StatelessCompression)
+	if err != nil {
+		return err
+	}
+	defer gzw.Close()
+
+	w := bufio.NewWriterSize(gzw, 4096)
+	defer w.Flush()
+	
+	// Write to 'w' 
+```
+
+This will only use up to 4KB in memory when the writer is idle. 
+
+Compression is almost always worse than the fastest compression level 
+and each write will allocate (a little) memory. 
+
+
+# Other packages
+
+Here are other packages of good quality and pure Go (no cgo wrappers or autoconverted code):
+
+* [github.com/pierrec/lz4](https://github.com/pierrec/lz4) - strong multithreaded LZ4 compression.
+* [github.com/cosnicolaou/pbzip2](https://github.com/cosnicolaou/pbzip2) - multithreaded bzip2 decompression.
+* [github.com/dsnet/compress](https://github.com/dsnet/compress) - brotli decompression, bzip2 writer.
+* [github.com/ronanh/intcomp](https://github.com/ronanh/intcomp) - Integer compression.
+* [github.com/spenczar/fpc](https://github.com/spenczar/fpc) - Float compression.
+* [github.com/minio/zipindex](https://github.com/minio/zipindex) - External ZIP directory index.
+* [github.com/ybirader/pzip](https://github.com/ybirader/pzip) - Fast concurrent zip archiver and extractor.
+
+# license
+
+This code is licensed under the same conditions as the original Go code. See LICENSE file.
+
+
+
+
+

vendor/github.com/klauspost/compress/SECURITY.md

@@ -0,0 +1,25 @@
+# Security Policy
+
+## Supported Versions
+
+Security updates are applied only to the latest release.
+
+## Vulnerability Definition
+
+A security vulnerability is a bug that with certain input triggers a crash or an infinite loop. Most calls will have varying execution time and only in rare cases will slow operation be considered a security vulnerability.
+
+Corrupted output generally is not considered a security vulnerability, unless independent operations are able to affect each other. Note that not all functionality is re-entrant and safe to use concurrently.
+
+Out-of-memory crashes only applies if the en/decoder uses an abnormal amount of memory, with appropriate options applied, to limit maximum window size, concurrency, etc. However, if you are in doubt you are welcome to file a security issue.
+
+It is assumed that all callers are trusted, meaning internal data exposed through reflection or inspection of returned data structures is not considered a vulnerability.
+
+Vulnerabilities resulting from compiler/assembler errors should be reported upstream. Depending on the severity this package may or may not implement a workaround.
+
+## Reporting a Vulnerability
+
+If you have discovered a security vulnerability in this project, please report it privately. **Do not disclose it as a public issue.** This gives us time to work with you to fix the issue before public exposure, reducing the chance that the exploit will be used before a patch is released.
+
+Please disclose it at [security advisory](https://github.com/klauspost/compress/security/advisories/new). If possible please provide a minimal reproducer. If the issue only applies to a single platform, it would be helpful to provide access to that.
+
+This project is maintained by a team of volunteers on a reasonable-effort basis. As such, vulnerabilities will be disclosed in a best effort base.

vendor/github.com/klauspost/compress/compressible.go

@@ -0,0 +1,85 @@
+package compress
+
+import "math"
+
+// Estimate returns a normalized compressibility estimate of block b.
+// Values close to zero are likely uncompressible.
+// Values above 0.1 are likely to be compressible.
+// Values above 0.5 are very compressible.
+// Very small lengths will return 0.
+func Estimate(b []byte) float64 {
+	if len(b) < 16 {
+		return 0
+	}
+
+	// Correctly predicted order 1
+	hits := 0
+	lastMatch := false
+	var o1 [256]byte
+	var hist [256]int
+	c1 := byte(0)
+	for _, c := range b {
+		if c == o1[c1] {
+			// We only count a hit if there was two correct predictions in a row.
+			if lastMatch {
+				hits++
+			}
+			lastMatch = true
+		} else {
+			lastMatch = false
+		}
+		o1[c1] = c
+		c1 = c
+		hist[c]++
+	}
+
+	// Use x^0.6 to give better spread
+	prediction := math.Pow(float64(hits)/float64(len(b)), 0.6)
+
+	// Calculate histogram distribution
+	variance := float64(0)
+	avg := float64(len(b)) / 256
+
+	for _, v := range hist {
+		Δ := float64(v) - avg
+		variance += Δ * Δ
+	}
+
+	stddev := math.Sqrt(float64(variance)) / float64(len(b))
+	exp := math.Sqrt(1 / float64(len(b)))
+
+	// Subtract expected stddev
+	stddev -= exp
+	if stddev < 0 {
+		stddev = 0
+	}
+	stddev *= 1 + exp
+
+	// Use x^0.4 to give better spread
+	entropy := math.Pow(stddev, 0.4)
+
+	// 50/50 weight between prediction and histogram distribution
+	return math.Pow((prediction+entropy)/2, 0.9)
+}
+
+// ShannonEntropyBits returns the number of bits minimum required to represent
+// an entropy encoding of the input bytes.
+// https://en.wiktionary.org/wiki/Shannon_entropy
+func ShannonEntropyBits(b []byte) int {
+	if len(b) == 0 {
+		return 0
+	}
+	var hist [256]int
+	for _, c := range b {
+		hist[c]++
+	}
+	shannon := float64(0)
+	invTotal := 1.0 / float64(len(b))
+	for _, v := range hist[:] {
+		if v > 0 {
+			n := float64(v)
+			shannon += math.Ceil(-math.Log2(n*invTotal) * n)
+		}
+	}
+	return int(math.Ceil(shannon))
+}

vendor/github.com/klauspost/compress/flate/deflate.go

@@ -0,0 +1,996 @@
+// Copyright 2009 The Go Authors. All rights reserved.
+// Copyright (c) 2015 Klaus Post
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE file.
+
+package flate
+
+import (
+	"errors"
+	"fmt"
+	"io"
+	"math"
+
+	"github.com/klauspost/compress/internal/le"
+)
+
+const (
+	NoCompression      = 0
+	BestSpeed          = 1
+	BestCompression    = 9
+	DefaultCompression = -1
+
+	// HuffmanOnly disables Lempel-Ziv match searching and only performs Huffman
+	// entropy encoding. This mode is useful in compressing data that has
+	// already been compressed with an LZ style algorithm (e.g. Snappy or LZ4)
+	// that lacks an entropy encoder. Compression gains are achieved when
+	// certain bytes in the input stream occur more frequently than others.
+	//
+	// Note that HuffmanOnly produces a compressed output that is
+	// RFC 1951 compliant. That is, any valid DEFLATE decompressor will
+	// continue to be able to decompress this output.
+	HuffmanOnly         = -2
+	ConstantCompression = HuffmanOnly // compatibility alias.
+
+	logWindowSize    = 15
+	windowSize       = 1 << logWindowSize
+	windowMask       = windowSize - 1
+	logMaxOffsetSize = 15  // Standard DEFLATE
+	minMatchLength   = 4   // The smallest match that the compressor looks for
+	maxMatchLength   = 258 // The longest match for the compressor
+	minOffsetSize    = 1   // The shortest offset that makes any sense
+
+	// The maximum number of tokens we will encode at the time.
+	// Smaller sizes usually creates less optimal blocks.
+	// Bigger can make context switching slow.
+	// We use this for levels 7-9, so we make it big.
+	maxFlateBlockTokens = 1 << 15
+	maxStoreBlockSize   = 65535
+	hashBits            = 17 // After 17 performance degrades
+	hashSize            = 1 << hashBits
+	hashMask            = (1 << hashBits) - 1
+	hashShift           = (hashBits + minMatchLength - 1) / minMatchLength
+	maxHashOffset       = 1 << 28
+
+	skipNever = math.MaxInt32
+
+	debugDeflate = false
+)
+
+type compressionLevel struct {
+	good, lazy, nice, chain, fastSkipHashing, level int
+}
+
+// Compression levels have been rebalanced from zlib deflate defaults
+// to give a bigger spread in speed and compression.
+// See https://blog.klauspost.com/rebalancing-deflate-compression-levels/
+var levels = []compressionLevel{
+	{}, // 0
+	// Level 1-6 uses specialized algorithm - values not used
+	{0, 0, 0, 0, 0, 1},
+	{0, 0, 0, 0, 0, 2},
+	{0, 0, 0, 0, 0, 3},
+	{0, 0, 0, 0, 0, 4},
+	{0, 0, 0, 0, 0, 5},
+	{0, 0, 0, 0, 0, 6},
+	// Levels 7-9 use increasingly more lazy matching
+	// and increasingly stringent conditions for "good enough".
+	{8, 12, 16, 24, skipNever, 7},
+	{16, 30, 40, 64, skipNever, 8},
+	{32, 258, 258, 1024, skipNever, 9},
+}
+
+// advancedState contains state for the advanced levels, with bigger hash tables, etc.
+type advancedState struct {
+	// deflate state
+	length         int
+	offset         int
+	maxInsertIndex int
+	chainHead      int
+	hashOffset     int
+
+	ii uint16 // position of last match, intended to overflow to reset.
+
+	// input window: unprocessed data is window[index:windowEnd]
+	index     int
+	hashMatch [maxMatchLength + minMatchLength]uint32
+
+	// Input hash chains
+	// hashHead[hashValue] contains the largest inputIndex with the specified hash value
+	// If hashHead[hashValue] is within the current window, then
+	// hashPrev[hashHead[hashValue] & windowMask] contains the previous index
+	// with the same hash value.
+	hashHead [hashSize]uint32
+	hashPrev [windowSize]uint32
+}
+
+type compressor struct {
+	compressionLevel
+
+	h *huffmanEncoder
+	w *huffmanBitWriter
+
+	// compression algorithm
+	fill func(*compressor, []byte) int // copy data to window
+	step func(*compressor)             // process window
+
+	window     []byte
+	windowEnd  int
+	blockStart int // window index where current tokens start
+	err        error
+
+	// queued output tokens
+	tokens tokens
+	fast   fastEnc
+	state  *advancedState
+
+	sync          bool // requesting flush
+	byteAvailable bool // if true, still need to process window[index-1].
+}
+
+func (d *compressor) fillDeflate(b []byte) int {
+	s := d.state
+	if s.index >= 2*windowSize-(minMatchLength+maxMatchLength) {
+		// shift the window by windowSize
+		//copy(d.window[:], d.window[windowSize:2*windowSize])
+		*(*[windowSize]byte)(d.window) = *(*[windowSize]byte)(d.window[windowSize:])
+		s.index -= windowSize
+		d.windowEnd -= windowSize
+		if d.blockStart >= windowSize {
+			d.blockStart -= windowSize
+		} else {
+			d.blockStart = math.MaxInt32
+		}
+		s.hashOffset += windowSize
+		if s.hashOffset > maxHashOffset {
+			delta := s.hashOffset - 1
+			s.hashOffset -= delta
+			s.chainHead -= delta
+			// Iterate over slices instead of arrays to avoid copying
+			// the entire table onto the stack (Issue #18625).
+			for i, v := range s.hashPrev[:] {
+				if int(v) > delta {
+					s.hashPrev[i] = uint32(int(v) - delta)
+				} else {
+					s.hashPrev[i] = 0
+				}
+			}
+			for i, v := range s.hashHead[:] {
+				if int(v) > delta {
+					s.hashHead[i] = uint32(int(v) - delta)
+				} else {
+					s.hashHead[i] = 0
+				}
+			}
+		}
+	}
+	n := copy(d.window[d.windowEnd:], b)
+	d.windowEnd += n
+	return n
+}
+
+func (d *compressor) writeBlock(tok *tokens, index int, eof bool) error {
+	if index > 0 || eof {
+		var window []byte
+		if d.blockStart <= index {
+			window = d.window[d.blockStart:index]
+		}
+		d.blockStart = index
+		//d.w.writeBlock(tok, eof, window)
+		d.w.writeBlockDynamic(tok, eof, window, d.sync)
+		return d.w.err
+	}
+	return nil
+}
+
+// writeBlockSkip writes the current block and uses the number of tokens
+// to determine if the block should be stored on no matches, or
+// only huffman encoded.
+func (d *compressor) writeBlockSkip(tok *tokens, index int, eof bool) error {
+	if index > 0 || eof {
+		if d.blockStart <= index {
+			window := d.window[d.blockStart:index]
+			// If we removed less than a 64th of all literals
+			// we huffman compress the block.
+			if int(tok.n) > len(window)-int(tok.n>>6) {
+				d.w.writeBlockHuff(eof, window, d.sync)
+			} else {
+				// Write a dynamic huffman block.
+				d.w.writeBlockDynamic(tok, eof, window, d.sync)
+			}
+		} else {
+			d.w.writeBlock(tok, eof, nil)
+		}
+		d.blockStart = index
+		return d.w.err
+	}
+	return nil
+}
+
+// fillWindow will fill the current window with the supplied
+// dictionary and calculate all hashes.
+// This is much faster than doing a full encode.
+// Should only be used after a start/reset.
+func (d *compressor) fillWindow(b []byte) {
+	// Do not fill window if we are in store-only or huffman mode.
+	if d.level <= 0 && d.level > -MinCustomWindowSize {
+		return
+	}
+	if d.fast != nil {
+		// encode the last data, but discard the result
+		if len(b) > maxMatchOffset {
+			b = b[len(b)-maxMatchOffset:]
+		}
+		d.fast.Encode(&d.tokens, b)
+		d.tokens.Reset()
+		return
+	}
+	s := d.state
+	// If we are given too much, cut it.
+	if len(b) > windowSize {
+		b = b[len(b)-windowSize:]
+	}
+	// Add all to window.
+	n := copy(d.window[d.windowEnd:], b)
+
+	// Calculate 256 hashes at the time (more L1 cache hits)
+	loops := (n + 256 - minMatchLength) / 256
+	for j := range loops {
+		startindex := j * 256
+		end := min(startindex+256+minMatchLength-1, n)
+		tocheck := d.window[startindex:end]
+		dstSize := len(tocheck) - minMatchLength + 1
+
+		if dstSize <= 0 {
+			continue
+		}
+
+		dst := s.hashMatch[:dstSize]
+		bulkHash4(tocheck, dst)
+		var newH uint32
+		for i, val := range dst {
+			di := i + startindex
+			newH = val & hashMask
+			// Get previous value with the same hash.
+			// Our chain should point to the previous value.
+			s.hashPrev[di&windowMask] = s.hashHead[newH]
+			// Set the head of the hash chain to us.
+			s.hashHead[newH] = uint32(di + s.hashOffset)
+		}
+	}
+	// Update window information.
+	d.windowEnd += n
+	s.index = n
+}
+
+// Try to find a match starting at index whose length is greater than prevSize.
+// We only look at chainCount possibilities before giving up.
+// pos = s.index, prevHead = s.chainHead-s.hashOffset, prevLength=minMatchLength-1, lookahead
+func (d *compressor) findMatch(pos int, prevHead int, lookahead int) (length, offset int, ok bool) {
+	minMatchLook := min(lookahead, maxMatchLength)
+
+	win := d.window[0 : pos+minMatchLook]
+
+	// We quit when we get a match that's at least nice long
+	nice := min(d.nice, len(win)-pos)
+
+	// If we've got a match that's good enough, only look in 1/4 the chain.
+	tries := d.chain
+	length = minMatchLength - 1
+
+	wEnd := win[pos+length]
+	wPos := win[pos:]
+	minIndex := max(pos-windowSize, 0)
+	offset = 0
+
+	if d.chain < 100 {
+		for i := prevHead; tries > 0; tries-- {
+			if wEnd == win[i+length] {
+				n := matchLen(win[i:i+minMatchLook], wPos)
+				if n > length {
+					length = n
+					offset = pos - i
+					ok = true
+					if n >= nice {
+						// The match is good enough that we don't try to find a better one.
+						break
+					}
+					wEnd = win[pos+n]
+				}
+			}
+			if i <= minIndex {
+				// hashPrev[i & windowMask] has already been overwritten, so stop now.
+				break
+			}
+			i = int(d.state.hashPrev[i&windowMask]) - d.state.hashOffset
+			if i < minIndex {
+				break
+			}
+		}
+		return
+	}
+
+	// Minimum gain to accept a match.
+	cGain := 4
+
+	// Some like it higher (CSV), some like it lower (JSON)
+	const baseCost = 3
+	// Base is 4 bytes at with an additional cost.
+	// Matches must be better than this.
+
+	for i := prevHead; tries > 0; tries-- {
+		if wEnd == win[i+length] {
+			n := matchLen(win[i:i+minMatchLook], wPos)
+			if n > length {
+				// Calculate gain. Estimate
+				newGain := d.h.bitLengthRaw(wPos[:n]) - int(offsetExtraBits[offsetCode(uint32(pos-i))]) - baseCost - int(lengthExtraBits[lengthCodes[(n-3)&255]])
+
+				//fmt.Println("gain:", newGain, "prev:", cGain, "raw:", d.h.bitLengthRaw(wPos[:n]), "this-len:", n, "prev-len:", length)
+				if newGain > cGain {
+					length = n
+					offset = pos - i
+					cGain = newGain
+					ok = true
+					if n >= nice {
+						// The match is good enough that we don't try to find a better one.
+						break
+					}
+					wEnd = win[pos+n]
+				}
+			}
+		}
+		if i <= minIndex {
+			// hashPrev[i & windowMask] has already been overwritten, so stop now.
+			break
+		}
+		i = int(d.state.hashPrev[i&windowMask]) - d.state.hashOffset
+		if i < minIndex {
+			break
+		}
+	}
+	return
+}
+
+func (d *compressor) writeStoredBlock(buf []byte) error {
+	if d.w.writeStoredHeader(len(buf), false); d.w.err != nil {
+		return d.w.err
+	}
+	d.w.writeBytes(buf)
+	return d.w.err
+}
+
+// hash4 returns a hash representation of the first 4 bytes
+// of the supplied slice.
+// The caller must ensure that len(b) >= 4.
+func hash4(b []byte) uint32 {
+	return hash4u(le.Load32(b, 0), hashBits)
+}
+
+// hash4 returns the hash of u to fit in a hash table with h bits.
+// Preferably h should be a constant and should always be <32.
+func hash4u(u uint32, h uint8) uint32 {
+	return (u * prime4bytes) >> (32 - h)
+}
+
+// bulkHash4 will compute hashes using the same
+// algorithm as hash4
+func bulkHash4(b []byte, dst []uint32) {
+	if len(b) < 4 {
+		return
+	}
+	hb := le.Load32(b, 0)
+
+	dst[0] = hash4u(hb, hashBits)
+	end := len(b) - 4 + 1
+	for i := 1; i < end; i++ {
+		hb = (hb >> 8) | uint32(b[i+3])<<24
+		dst[i] = hash4u(hb, hashBits)
+	}
+}
+
+func (d *compressor) initDeflate() {
+	d.window = make([]byte, 2*windowSize)
+	d.byteAvailable = false
+	d.err = nil
+	if d.state == nil {
+		return
+	}
+	s := d.state
+	s.index = 0
+	s.hashOffset = 1
+	s.length = minMatchLength - 1
+	s.offset = 0
+	s.chainHead = -1
+}
+
+// deflateLazy is the same as deflate, but with d.fastSkipHashing == skipNever,
+// meaning it always has lazy matching on.
+func (d *compressor) deflateLazy() {
+	s := d.state
+	// Sanity enables additional runtime tests.
+	// It's intended to be used during development
+	// to supplement the currently ad-hoc unit tests.
+	const sanity = debugDeflate
+
+	if d.windowEnd-s.index < minMatchLength+maxMatchLength && !d.sync {
+		return
+	}
+	if d.windowEnd != s.index && d.chain > 100 {
+		// Get literal huffman coder.
+		if d.h == nil {
+			d.h = newHuffmanEncoder(maxFlateBlockTokens)
+		}
+		var tmp [256]uint16
+		toIndex := d.window[s.index:d.windowEnd]
+		toIndex = toIndex[:min(len(toIndex), maxFlateBlockTokens)]
+		for _, v := range toIndex {
+			tmp[v]++
+		}
+		d.h.generate(tmp[:], 15)
+	}
+
+	s.maxInsertIndex = d.windowEnd - (minMatchLength - 1)
+
+	for {
+		if sanity && s.index > d.windowEnd {
+			panic("index > windowEnd")
+		}
+		lookahead := d.windowEnd - s.index
+		if lookahead < minMatchLength+maxMatchLength {
+			if !d.sync {
+				return
+			}
+			if sanity && s.index > d.windowEnd {
+				panic("index > windowEnd")
+			}
+			if lookahead == 0 {
+				// Flush current output block if any.
+				if d.byteAvailable {
+					// There is still one pending token that needs to be flushed
+					d.tokens.AddLiteral(d.window[s.index-1])
+					d.byteAvailable = false
+				}
+				if d.tokens.n > 0 {
+					if d.err = d.writeBlock(&d.tokens, s.index, false); d.err != nil {
+						return
+					}
+					d.tokens.Reset()
+				}
+				return
+			}
+		}
+		if s.index < s.maxInsertIndex {
+			// Update the hash
+			hash := hash4(d.window[s.index:])
+			ch := s.hashHead[hash]
+			s.chainHead = int(ch)
+			s.hashPrev[s.index&windowMask] = ch
+			s.hashHead[hash] = uint32(s.index + s.hashOffset)
+		}
+		prevLength := s.length
+		prevOffset := s.offset
+		s.length = minMatchLength - 1
+		s.offset = 0
+		minIndex := max(s.index-windowSize, 0)
+
+		if s.chainHead-s.hashOffset >= minIndex && lookahead > prevLength && prevLength < d.lazy {
+			if newLength, newOffset, ok := d.findMatch(s.index, s.chainHead-s.hashOffset, lookahead); ok {
+				s.length = newLength
+				s.offset = newOffset
+			}
+		}
+
+		if prevLength >= minMatchLength && s.length <= prevLength {
+			// No better match, but check for better match at end...
+			//
+			// Skip forward a number of bytes.
+			// Offset of 2 seems to yield best results. 3 is sometimes better.
+			const checkOff = 2
+
+			// Check all, except full length
+			if prevLength < maxMatchLength-checkOff {
+				prevIndex := s.index - 1
+				if prevIndex+prevLength < s.maxInsertIndex {
+					end := min(lookahead, maxMatchLength+checkOff)
+					end += prevIndex
+
+					// Hash at match end.
+					h := hash4(d.window[prevIndex+prevLength:])
+					ch2 := int(s.hashHead[h]) - s.hashOffset - prevLength
+					if prevIndex-ch2 != prevOffset && ch2 > minIndex+checkOff {
+						length := matchLen(d.window[prevIndex+checkOff:end], d.window[ch2+checkOff:])
+						// It seems like a pure length metric is best.
+						if length > prevLength {
+							prevLength = length
+							prevOffset = prevIndex - ch2
+
+							// Extend back...
+							for i := checkOff - 1; i >= 0; i-- {
+								if prevLength >= maxMatchLength || d.window[prevIndex+i] != d.window[ch2+i] {
+									// Emit tokens we "owe"
+									for j := 0; j <= i; j++ {
+										d.tokens.AddLiteral(d.window[prevIndex+j])
+										if d.tokens.n == maxFlateBlockTokens {
+											// The block includes the current character
+											if d.err = d.writeBlock(&d.tokens, s.index, false); d.err != nil {
+												return
+											}
+											d.tokens.Reset()
+										}
+										s.index++
+										if s.index < s.maxInsertIndex {
+											h := hash4(d.window[s.index:])
+											ch := s.hashHead[h]
+											s.chainHead = int(ch)
+											s.hashPrev[s.index&windowMask] = ch
+											s.hashHead[h] = uint32(s.index + s.hashOffset)
+										}
+									}
+									break
+								} else {
+									prevLength++
+								}
+							}
+						} else if false {
+							// Check one further ahead.
+							// Only rarely better, disabled for now.
+							prevIndex++
+							h := hash4(d.window[prevIndex+prevLength:])
+							ch2 := int(s.hashHead[h]) - s.hashOffset - prevLength
+							if prevIndex-ch2 != prevOffset && ch2 > minIndex+checkOff {
+								length := matchLen(d.window[prevIndex+checkOff:end], d.window[ch2+checkOff:])
+								// It seems like a pure length metric is best.
+								if length > prevLength+checkOff {
+									prevLength = length
+									prevOffset = prevIndex - ch2
+									prevIndex--
+
+									// Extend back...
+									for i := checkOff; i >= 0; i-- {
+										if prevLength >= maxMatchLength || d.window[prevIndex+i] != d.window[ch2+i-1] {
+											// Emit tokens we "owe"
+											for j := 0; j <= i; j++ {
+												d.tokens.AddLiteral(d.window[prevIndex+j])
+												if d.tokens.n == maxFlateBlockTokens {
+													// The block includes the current character
+													if d.err = d.writeBlock(&d.tokens, s.index, false); d.err != nil {
+														return
+													}
+													d.tokens.Reset()
+												}
+												s.index++
+												if s.index < s.maxInsertIndex {
+													h := hash4(d.window[s.index:])
+													ch := s.hashHead[h]
+													s.chainHead = int(ch)
+													s.hashPrev[s.index&windowMask] = ch
+													s.hashHead[h] = uint32(s.index + s.hashOffset)
+												}
+											}
+											break
+										} else {
+											prevLength++
+										}
+									}
+								}
+							}
+						}
+					}
+				}
+			}
+			// There was a match at the previous step, and the current match is
+			// not better. Output the previous match.
+			d.tokens.AddMatch(uint32(prevLength-3), uint32(prevOffset-minOffsetSize))
+
+			// Insert in the hash table all strings up to the end of the match.
+			// index and index-1 are already inserted. If there is not enough
+			// lookahead, the last two strings are not inserted into the hash
+			// table.
+			newIndex := s.index + prevLength - 1
+			// Calculate missing hashes
+			end := min(newIndex, s.maxInsertIndex)
+			end += minMatchLength - 1
+			startindex := min(s.index+1, s.maxInsertIndex)
+			tocheck := d.window[startindex:end]
+			dstSize := len(tocheck) - minMatchLength + 1
+			if dstSize > 0 {
+				dst := s.hashMatch[:dstSize]
+				bulkHash4(tocheck, dst)
+				var newH uint32
+				for i, val := range dst {
+					di := i + startindex
+					newH = val & hashMask
+					// Get previous value with the same hash.
+					// Our chain should point to the previous value.
+					s.hashPrev[di&windowMask] = s.hashHead[newH]
+					// Set the head of the hash chain to us.
+					s.hashHead[newH] = uint32(di + s.hashOffset)
+				}
+			}
+
+			s.index = newIndex
+			d.byteAvailable = false
+			s.length = minMatchLength - 1
+			if d.tokens.n == maxFlateBlockTokens {
+				// The block includes the current character
+				if d.err = d.writeBlock(&d.tokens, s.index, false); d.err != nil {
+					return
+				}
+				d.tokens.Reset()
+			}
+			s.ii = 0
+		} else {
+			// Reset, if we got a match this run.
+			if s.length >= minMatchLength {
+				s.ii = 0
+			}
+			// We have a byte waiting. Emit it.
+			if d.byteAvailable {
+				s.ii++
+				d.tokens.AddLiteral(d.window[s.index-1])
+				if d.tokens.n == maxFlateBlockTokens {
+					if d.err = d.writeBlock(&d.tokens, s.index, false); d.err != nil {
+						return
+					}
+					d.tokens.Reset()
+				}
+				s.index++
+
+				// If we have a long run of no matches, skip additional bytes
+				// Resets when s.ii overflows after 64KB.
+				if n := int(s.ii) - d.chain; n > 0 {
+					n = 1 + int(n>>6)
+					for j := 0; j < n; j++ {
+						if s.index >= d.windowEnd-1 {
+							break
+						}
+						d.tokens.AddLiteral(d.window[s.index-1])
+						if d.tokens.n == maxFlateBlockTokens {
+							if d.err = d.writeBlock(&d.tokens, s.index, false); d.err != nil {
+								return
+							}
+							d.tokens.Reset()
+						}
+						// Index...
+						if s.index < s.maxInsertIndex {
+							h := hash4(d.window[s.index:])
+							ch := s.hashHead[h]
+							s.chainHead = int(ch)
+							s.hashPrev[s.index&windowMask] = ch
+							s.hashHead[h] = uint32(s.index + s.hashOffset)
+						}
+						s.index++
+					}
+					// Flush last byte
+					d.tokens.AddLiteral(d.window[s.index-1])
+					d.byteAvailable = false
+					// s.length = minMatchLength - 1 // not needed, since s.ii is reset above, so it should never be > minMatchLength
+					if d.tokens.n == maxFlateBlockTokens {
+						if d.err = d.writeBlock(&d.tokens, s.index, false); d.err != nil {
+							return
+						}
+						d.tokens.Reset()
+					}
+				}
+			} else {
+				s.index++
+				d.byteAvailable = true
+			}
+		}
+	}
+}
+
+func (d *compressor) store() {
+	if d.windowEnd > 0 && (d.windowEnd == maxStoreBlockSize || d.sync) {
+		d.err = d.writeStoredBlock(d.window[:d.windowEnd])
+		d.windowEnd = 0
+	}
+}
+
+// fillWindow will fill the buffer with data for huffman-only compression.
+// The number of bytes copied is returned.
+func (d *compressor) fillBlock(b []byte) int {
+	n := copy(d.window[d.windowEnd:], b)
+	d.windowEnd += n
+	return n
+}
+
+// storeHuff will compress and store the currently added data,
+// if enough has been accumulated or we at the end of the stream.
+// Any error that occurred will be in d.err
+func (d *compressor) storeHuff() {
+	if d.windowEnd < len(d.window) && !d.sync || d.windowEnd == 0 {
+		return
+	}
+	d.w.writeBlockHuff(false, d.window[:d.windowEnd], d.sync)
+	d.err = d.w.err
+	d.windowEnd = 0
+}
+
+// storeFast will compress and store the currently added data,
+// if enough has been accumulated or we at the end of the stream.
+// Any error that occurred will be in d.err
+func (d *compressor) storeFast() {
+	// We only compress if we have maxStoreBlockSize.
+	if d.windowEnd < len(d.window) {
+		if !d.sync {
+			return
+		}
+		// Handle extremely small sizes.
+		if d.windowEnd < 128 {
+			if d.windowEnd == 0 {
+				return
+			}
+			if d.windowEnd <= 32 {
+				d.err = d.writeStoredBlock(d.window[:d.windowEnd])
+			} else {
+				d.w.writeBlockHuff(false, d.window[:d.windowEnd], true)
+				d.err = d.w.err
+			}
+			d.tokens.Reset()
+			d.windowEnd = 0
+			d.fast.Reset()
+			return
+		}
+	}
+
+	d.fast.Encode(&d.tokens, d.window[:d.windowEnd])
+	// If we made zero matches, store the block as is.
+	if d.tokens.n == 0 {
+		d.err = d.writeStoredBlock(d.window[:d.windowEnd])
+		// If we removed less than 1/16th, huffman compress the block.
+	} else if int(d.tokens.n) > d.windowEnd-(d.windowEnd>>4) {
+		d.w.writeBlockHuff(false, d.window[:d.windowEnd], d.sync)
+		d.err = d.w.err
+	} else {
+		d.w.writeBlockDynamic(&d.tokens, false, d.window[:d.windowEnd], d.sync)
+		d.err = d.w.err
+	}
+	d.tokens.Reset()
+	d.windowEnd = 0
+}
+
+// write will add input byte to the stream.
+// Unless an error occurs all bytes will be consumed.
+func (d *compressor) write(b []byte) (n int, err error) {
+	if d.err != nil {
+		return 0, d.err
+	}
+	n = len(b)
+	for len(b) > 0 {
+		if d.windowEnd == len(d.window) || d.sync {
+			d.step(d)
+		}
+		b = b[d.fill(d, b):]
+		if d.err != nil {
+			return 0, d.err
+		}
+	}
+	return n, d.err
+}
+
+func (d *compressor) syncFlush() error {
+	d.sync = true
+	if d.err != nil {
+		return d.err
+	}
+	d.step(d)
+	if d.err == nil {
+		d.w.writeStoredHeader(0, false)
+		d.w.flush()
+		d.err = d.w.err
+	}
+	d.sync = false
+	return d.err
+}
+
+func (d *compressor) init(w io.Writer, level int) (err error) {
+	d.w = newHuffmanBitWriter(w)
+
+	switch {
+	case level == NoCompression:
+		d.window = make([]byte, maxStoreBlockSize)
+		d.fill = (*compressor).fillBlock
+		d.step = (*compressor).store
+	case level == ConstantCompression:
+		d.w.logNewTablePenalty = 10
+		d.window = make([]byte, 32<<10)
+		d.fill = (*compressor).fillBlock
+		d.step = (*compressor).storeHuff
+	case level == DefaultCompression:
+		level = 5
+		fallthrough
+	case level >= 1 && level <= 6:
+		d.w.logNewTablePenalty = 7
+		d.fast = newFastEnc(level)
+		d.window = make([]byte, maxStoreBlockSize)
+		d.fill = (*compressor).fillBlock
+		d.step = (*compressor).storeFast
+	case 7 <= level && level <= 9:
+		d.w.logNewTablePenalty = 8
+		d.state = &advancedState{}
+		d.compressionLevel = levels[level]
+		d.initDeflate()
+		d.fill = (*compressor).fillDeflate
+		d.step = (*compressor).deflateLazy
+	case -level >= MinCustomWindowSize && -level <= MaxCustomWindowSize:
+		d.w.logNewTablePenalty = 7
+		d.fast = &fastEncL5Window{maxOffset: int32(-level), cur: maxStoreBlockSize}
+		d.window = make([]byte, maxStoreBlockSize)
+		d.fill = (*compressor).fillBlock
+		d.step = (*compressor).storeFast
+	default:
+		return fmt.Errorf("flate: invalid compression level %d: want value in range [-2, 9]", level)
+	}
+	d.level = level
+	return nil
+}
+
+// reset the state of the compressor.
+func (d *compressor) reset(w io.Writer) {
+	d.w.reset(w)
+	d.sync = false
+	d.err = nil
+	// We only need to reset a few things for Snappy.
+	if d.fast != nil {
+		d.fast.Reset()
+		d.windowEnd = 0
+		d.tokens.Reset()
+		return
+	}
+	switch d.compressionLevel.chain {
+	case 0:
+		// level was NoCompression or ConstantCompression.
+		d.windowEnd = 0
+	default:
+		s := d.state
+		s.chainHead = -1
+		for i := range s.hashHead {
+			s.hashHead[i] = 0
+		}
+		for i := range s.hashPrev {
+			s.hashPrev[i] = 0
+		}
+		s.hashOffset = 1
+		s.index, d.windowEnd = 0, 0
+		d.blockStart, d.byteAvailable = 0, false
+		d.tokens.Reset()
+		s.length = minMatchLength - 1
+		s.offset = 0
+		s.ii = 0
+		s.maxInsertIndex = 0
+	}
+}
+
+func (d *compressor) close() error {
+	if d.err != nil {
+		return d.err
+	}
+	d.sync = true
+	d.step(d)
+	if d.err != nil {
+		return d.err
+	}
+	if d.w.writeStoredHeader(0, true); d.w.err != nil {
+		return d.w.err
+	}
+	d.w.flush()
+	d.w.reset(nil)
+	return d.w.err
+}
+
+// NewWriter returns a new Writer compressing data at the given level.
+// Following zlib, levels range from 1 (BestSpeed) to 9 (BestCompression);
+// higher levels typically run slower but compress more.
+// Level 0 (NoCompression) does not attempt any compression; it only adds the
+// necessary DEFLATE framing.
+// Level -1 (DefaultCompression) uses the default compression level.
+// Level -2 (ConstantCompression) will use Huffman compression only, giving
+// a very fast compression for all types of input, but sacrificing considerable
+// compression efficiency.
+//
+// If level is in the range [-2, 9] then the error returned will be nil.
+// Otherwise the error returned will be non-nil.
+func NewWriter(w io.Writer, level int) (*Writer, error) {
+	var dw Writer
+	if err := dw.d.init(w, level); err != nil {
+		return nil, err
+	}
+	return &dw, nil
+}
+
+// NewWriterDict is like NewWriter but initializes the new
+// Writer with a preset dictionary.  The returned Writer behaves
+// as if the dictionary had been written to it without producing
+// any compressed output.  The compressed data written to w
+// can only be decompressed by a Reader initialized with the
+// same dictionary.
+func NewWriterDict(w io.Writer, level int, dict []byte) (*Writer, error) {
+	zw, err := NewWriter(w, level)
+	if err != nil {
+		return nil, err
+	}
+	zw.d.fillWindow(dict)
+	zw.dict = append(zw.dict, dict...) // duplicate dictionary for Reset method.
+	return zw, err
+}
+
+// MinCustomWindowSize is the minimum window size that can be sent to NewWriterWindow.
+const MinCustomWindowSize = 32
+
+// MaxCustomWindowSize is the maximum custom window that can be sent to NewWriterWindow.
+const MaxCustomWindowSize = windowSize
+
+// NewWriterWindow returns a new Writer compressing data with a custom window size.
+// windowSize must be from MinCustomWindowSize to MaxCustomWindowSize.
+func NewWriterWindow(w io.Writer, windowSize int) (*Writer, error) {
+	if windowSize < MinCustomWindowSize {
+		return nil, errors.New("flate: requested window size less than MinWindowSize")
+	}
+	if windowSize > MaxCustomWindowSize {
+		return nil, errors.New("flate: requested window size bigger than MaxCustomWindowSize")
+	}
+	var dw Writer
+	if err := dw.d.init(w, -windowSize); err != nil {
+		return nil, err
+	}
+	return &dw, nil
+}
+
+// A Writer takes data written to it and writes the compressed
+// form of that data to an underlying writer (see NewWriter).
+type Writer struct {
+	d    compressor
+	dict []byte
+}
+
+// Write writes data to w, which will eventually write the
+// compressed form of data to its underlying writer.
+func (w *Writer) Write(data []byte) (n int, err error) {
+	return w.d.write(data)
+}
+
+// Flush flushes any pending data to the underlying writer.
+// It is useful mainly in compressed network protocols, to ensure that
+// a remote reader has enough data to reconstruct a packet.
+// Flush does not return until the data has been written.
+// Calling Flush when there is no pending data still causes the Writer
+// to emit a sync marker of at least 4 bytes.
+// If the underlying writer returns an error, Flush returns that error.
+//
+// In the terminology of the zlib library, Flush is equivalent to Z_SYNC_FLUSH.
+func (w *Writer) Flush() error {
+	// For more about flushing:
+	// http://www.bolet.org/~pornin/deflate-flush.html
+	return w.d.syncFlush()
+}
+
+// Close flushes and closes the writer.
+func (w *Writer) Close() error {
+	return w.d.close()
+}
+
+// Reset discards the writer's state and makes it equivalent to
+// the result of NewWriter or NewWriterDict called with dst
+// and w's level and dictionary.
+func (w *Writer) Reset(dst io.Writer) {
+	if len(w.dict) > 0 {
+		// w was created with NewWriterDict
+		w.d.reset(dst)
+		if dst != nil {
+			w.d.fillWindow(w.dict)
+		}
+	} else {
+		// w was created with NewWriter
+		w.d.reset(dst)
+	}
+}
+
+// ResetDict discards the writer's state and makes it equivalent to
+// the result of NewWriter or NewWriterDict called with dst
+// and w's level, but sets a specific dictionary.
+func (w *Writer) ResetDict(dst io.Writer, dict []byte) {
+	w.dict = dict
+	w.d.reset(dst)
+	w.d.fillWindow(w.dict)
+}

vendor/github.com/klauspost/compress/flate/dict_decoder.go

@@ -0,0 +1,181 @@
+// Copyright 2016 The Go Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE file.
+
+package flate
+
+// dictDecoder implements the LZ77 sliding dictionary as used in decompression.
+// LZ77 decompresses data through sequences of two forms of commands:
+//
+//   - Literal insertions: Runs of one or more symbols are inserted into the data
+//     stream as is. This is accomplished through the writeByte method for a
+//     single symbol, or combinations of writeSlice/writeMark for multiple symbols.
+//     Any valid stream must start with a literal insertion if no preset dictionary
+//     is used.
+//
+//   - Backward copies: Runs of one or more symbols are copied from previously
+//     emitted data. Backward copies come as the tuple (dist, length) where dist
+//     determines how far back in the stream to copy from and length determines how
+//     many bytes to copy. Note that it is valid for the length to be greater than
+//     the distance. Since LZ77 uses forward copies, that situation is used to
+//     perform a form of run-length encoding on repeated runs of symbols.
+//     The writeCopy and tryWriteCopy are used to implement this command.
+//
+// For performance reasons, this implementation performs little to no sanity
+// checks about the arguments. As such, the invariants documented for each
+// method call must be respected.
+type dictDecoder struct {
+	hist []byte // Sliding window history
+
+	// Invariant: 0 <= rdPos <= wrPos <= len(hist)
+	wrPos int  // Current output position in buffer
+	rdPos int  // Have emitted hist[:rdPos] already
+	full  bool // Has a full window length been written yet?
+}
+
+// init initializes dictDecoder to have a sliding window dictionary of the given
+// size. If a preset dict is provided, it will initialize the dictionary with
+// the contents of dict.
+func (dd *dictDecoder) init(size int, dict []byte) {
+	*dd = dictDecoder{hist: dd.hist}
+
+	if cap(dd.hist) < size {
+		dd.hist = make([]byte, size)
+	}
+	dd.hist = dd.hist[:size]
+
+	if len(dict) > len(dd.hist) {
+		dict = dict[len(dict)-len(dd.hist):]
+	}
+	dd.wrPos = copy(dd.hist, dict)
+	if dd.wrPos == len(dd.hist) {
+		dd.wrPos = 0
+		dd.full = true
+	}
+	dd.rdPos = dd.wrPos
+}
+
+// histSize reports the total amount of historical data in the dictionary.
+func (dd *dictDecoder) histSize() int {
+	if dd.full {
+		return len(dd.hist)
+	}
+	return dd.wrPos
+}
+
+// availRead reports the number of bytes that can be flushed by readFlush.
+func (dd *dictDecoder) availRead() int {
+	return dd.wrPos - dd.rdPos
+}
+
+// availWrite reports the available amount of output buffer space.
+func (dd *dictDecoder) availWrite() int {
+	return len(dd.hist) - dd.wrPos
+}
+
+// writeSlice returns a slice of the available buffer to write data to.
+//
+// This invariant will be kept: len(s) <= availWrite()
+func (dd *dictDecoder) writeSlice() []byte {
+	return dd.hist[dd.wrPos:]
+}
+
+// writeMark advances the writer pointer by cnt.
+//
+// This invariant must be kept: 0 <= cnt <= availWrite()
+func (dd *dictDecoder) writeMark(cnt int) {
+	dd.wrPos += cnt
+}
+
+// writeByte writes a single byte to the dictionary.
+//
+// This invariant must be kept: 0 < availWrite()
+func (dd *dictDecoder) writeByte(c byte) {
+	dd.hist[dd.wrPos] = c
+	dd.wrPos++
+}
+
+// writeCopy copies a string at a given (dist, length) to the output.
+// This returns the number of bytes copied and may be less than the requested
+// length if the available space in the output buffer is too small.
+//
+// This invariant must be kept: 0 < dist <= histSize()
+func (dd *dictDecoder) writeCopy(dist, length int) int {
+	dstBase := dd.wrPos
+	dstPos := dstBase
+	srcPos := dstPos - dist
+	endPos := min(dstPos+length, len(dd.hist))
+
+	// Copy non-overlapping section after destination position.
+	//
+	// This section is non-overlapping in that the copy length for this section
+	// is always less than or equal to the backwards distance. This can occur
+	// if a distance refers to data that wraps-around in the buffer.
+	// Thus, a backwards copy is performed here; that is, the exact bytes in
+	// the source prior to the copy is placed in the destination.
+	if srcPos < 0 {
+		srcPos += len(dd.hist)
+		dstPos += copy(dd.hist[dstPos:endPos], dd.hist[srcPos:])
+		srcPos = 0
+	}
+
+	// Copy possibly overlapping section before destination position.
+	//
+	// This section can overlap if the copy length for this section is larger
+	// than the backwards distance. This is allowed by LZ77 so that repeated
+	// strings can be succinctly represented using (dist, length) pairs.
+	// Thus, a forwards copy is performed here; that is, the bytes copied is
+	// possibly dependent on the resulting bytes in the destination as the copy
+	// progresses along. This is functionally equivalent to the following:
+	//
+	//	for i := 0; i < endPos-dstPos; i++ {
+	//		dd.hist[dstPos+i] = dd.hist[srcPos+i]
+	//	}
+	//	dstPos = endPos
+	//
+	for dstPos < endPos {
+		dstPos += copy(dd.hist[dstPos:endPos], dd.hist[srcPos:dstPos])
+	}
+
+	dd.wrPos = dstPos
+	return dstPos - dstBase
+}
+
+// tryWriteCopy tries to copy a string at a given (distance, length) to the
+// output. This specialized version is optimized for short distances.
+//
+// This method is designed to be inlined for performance reasons.
+//
+// This invariant must be kept: 0 < dist <= histSize()
+func (dd *dictDecoder) tryWriteCopy(dist, length int) int {
+	dstPos := dd.wrPos
+	endPos := dstPos + length
+	if dstPos < dist || endPos > len(dd.hist) {
+		return 0
+	}
+	dstBase := dstPos
+	srcPos := dstPos - dist
+
+	// Copy possibly overlapping section before destination position.
+loop:
+	dstPos += copy(dd.hist[dstPos:endPos], dd.hist[srcPos:dstPos])
+	if dstPos < endPos {
+		goto loop // Avoid for-loop so that this function can be inlined
+	}
+
+	dd.wrPos = dstPos
+	return dstPos - dstBase
+}
+
+// readFlush returns a slice of the historical buffer that is ready to be
+// emitted to the user. The data returned by readFlush must be fully consumed
+// before calling any other dictDecoder methods.
+func (dd *dictDecoder) readFlush() []byte {
+	toRead := dd.hist[dd.rdPos:dd.wrPos]
+	dd.rdPos = dd.wrPos
+	if dd.wrPos == len(dd.hist) {
+		dd.wrPos, dd.rdPos = 0, 0
+		dd.full = true
+	}
+	return toRead
+}

vendor/github.com/klauspost/compress/flate/fast_encoder.go

@@ -0,0 +1,189 @@
+// Copyright 2011 The Snappy-Go Authors. All rights reserved.
+// Modified for deflate by Klaus Post (c) 2015.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE file.
+
+package flate
+
+import (
+	"fmt"
+
+	"github.com/klauspost/compress/internal/le"
+)
+
+type fastEnc interface {
+	Encode(dst *tokens, src []byte)
+	Reset()
+}
+
+func newFastEnc(level int) fastEnc {
+	switch level {
+	case 1:
+		return &fastEncL1{fastGen: fastGen{cur: maxStoreBlockSize}}
+	case 2:
+		return &fastEncL2{fastGen: fastGen{cur: maxStoreBlockSize}}
+	case 3:
+		return &fastEncL3{fastGen: fastGen{cur: maxStoreBlockSize}}
+	case 4:
+		return &fastEncL4{fastGen: fastGen{cur: maxStoreBlockSize}}
+	case 5:
+		return &fastEncL5{fastGen: fastGen{cur: maxStoreBlockSize}}
+	case 6:
+		return &fastEncL6{fastGen: fastGen{cur: maxStoreBlockSize}}
+	default:
+		panic("invalid level specified")
+	}
+}
+
+const (
+	tableBits       = 15             // Bits used in the table
+	tableSize       = 1 << tableBits // Size of the table
+	tableShift      = 32 - tableBits // Right-shift to get the tableBits most significant bits of a uint32.
+	baseMatchOffset = 1              // The smallest match offset
+	baseMatchLength = 3              // The smallest match length per the RFC section 3.2.5
+	maxMatchOffset  = 1 << 15        // The largest match offset
+
+	bTableBits   = 17                                               // Bits used in the big tables
+	bTableSize   = 1 << bTableBits                                  // Size of the table
+	allocHistory = maxStoreBlockSize * 5                            // Size to preallocate for history.
+	bufferReset  = (1 << 31) - allocHistory - maxStoreBlockSize - 1 // Reset the buffer offset when reaching this.
+)
+
+const (
+	prime3bytes = 506832829
+	prime4bytes = 2654435761
+	prime5bytes = 889523592379
+	prime6bytes = 227718039650203
+	prime7bytes = 58295818150454627
+	prime8bytes = 0xcf1bbcdcb7a56463
+)
+
+func load3232(b []byte, i int32) uint32 {
+	return le.Load32(b, i)
+}
+
+func load6432(b []byte, i int32) uint64 {
+	return le.Load64(b, i)
+}
+
+type tableEntry struct {
+	offset int32
+}
+
+// fastGen maintains the table for matches,
+// and the previous byte block for level 2.
+// This is the generic implementation.
+type fastGen struct {
+	hist []byte
+	cur  int32
+}
+
+func (e *fastGen) addBlock(src []byte) int32 {
+	// check if we have space already
+	if len(e.hist)+len(src) > cap(e.hist) {
+		if cap(e.hist) == 0 {
+			e.hist = make([]byte, 0, allocHistory)
+		} else {
+			if cap(e.hist) < maxMatchOffset*2 {
+				panic("unexpected buffer size")
+			}
+			// Move down
+			offset := int32(len(e.hist)) - maxMatchOffset
+			// copy(e.hist[0:maxMatchOffset], e.hist[offset:])
+			*(*[maxMatchOffset]byte)(e.hist) = *(*[maxMatchOffset]byte)(e.hist[offset:])
+			e.cur += offset
+			e.hist = e.hist[:maxMatchOffset]
+		}
+	}
+	s := int32(len(e.hist))
+	e.hist = append(e.hist, src...)
+	return s
+}
+
+type tableEntryPrev struct {
+	Cur  tableEntry
+	Prev tableEntry
+}
+
+// hash7 returns the hash of the lowest 7 bytes of u to fit in a hash table with h bits.
+// Preferably h should be a constant and should always be <64.
+func hash7(u uint64, h uint8) uint32 {
+	return uint32(((u << (64 - 56)) * prime7bytes) >> ((64 - h) & reg8SizeMask64))
+}
+
+// hashLen returns a hash of the lowest mls bytes of with length output bits.
+// mls must be >=3 and <=8. Any other value will return hash for 4 bytes.
+// length should always be < 32.
+// Preferably length and mls should be a constant for inlining.
+func hashLen(u uint64, length, mls uint8) uint32 {
+	switch mls {
+	case 3:
+		return (uint32(u<<8) * prime3bytes) >> (32 - length)
+	case 5:
+		return uint32(((u << (64 - 40)) * prime5bytes) >> (64 - length))
+	case 6:
+		return uint32(((u << (64 - 48)) * prime6bytes) >> (64 - length))
+	case 7:
+		return uint32(((u << (64 - 56)) * prime7bytes) >> (64 - length))
+	case 8:
+		return uint32((u * prime8bytes) >> (64 - length))
+	default:
+		return (uint32(u) * prime4bytes) >> (32 - length)
+	}
+}
+
+// matchlen will return the match length between offsets and t in src.
+// The maximum length returned is maxMatchLength - 4.
+// It is assumed that s > t, that t >=0 and s < len(src).
+func (e *fastGen) matchlen(s, t int, src []byte) int32 {
+	if debugDeflate {
+		if t >= s {
+			panic(fmt.Sprint("t >=s:", t, s))
+		}
+		if int(s) >= len(src) {
+			panic(fmt.Sprint("s >= len(src):", s, len(src)))
+		}
+		if t < 0 {
+			panic(fmt.Sprint("t < 0:", t))
+		}
+		if s-t > maxMatchOffset {
+			panic(fmt.Sprint(s, "-", t, "(", s-t, ") > maxMatchLength (", maxMatchOffset, ")"))
+		}
+	}
+	a := src[s:min(s+maxMatchLength-4, len(src))]
+	b := src[t:]
+	return int32(matchLen(a, b))
+}
+
+// matchlenLong will return the match length between offsets and t in src.
+// It is assumed that s > t, that t >=0 and s < len(src).
+func (e *fastGen) matchlenLong(s, t int, src []byte) int32 {
+	if debugDeflate {
+		if t >= s {
+			panic(fmt.Sprint("t >=s:", t, s))
+		}
+		if int(s) >= len(src) {
+			panic(fmt.Sprint("s >= len(src):", s, len(src)))
+		}
+		if t < 0 {
+			panic(fmt.Sprint("t < 0:", t))
+		}
+		if s-t > maxMatchOffset {
+			panic(fmt.Sprint(s, "-", t, "(", s-t, ") > maxMatchLength (", maxMatchOffset, ")"))
+		}
+	}
+	return int32(matchLen(src[s:], src[t:]))
+}
+
+// Reset the encoding table.
+func (e *fastGen) Reset() {
+	if cap(e.hist) < allocHistory {
+		e.hist = make([]byte, 0, allocHistory)
+	}
+	// We offset current position so everything will be out of reach.
+	// If we are above the buffer reset it will be cleared anyway since len(hist) == 0.
+	if e.cur <= bufferReset {
+		e.cur += maxMatchOffset + int32(len(e.hist))
+	}
+	e.hist = e.hist[:0]
+}

vendor/github.com/klauspost/compress/flate/huffman_code.go

@@ -0,0 +1,417 @@
+// Copyright 2009 The Go Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE file.
+
+package flate
+
+import (
+	"math"
+	"math/bits"
+)
+
+const (
+	maxBitsLimit = 16
+	// number of valid literals
+	literalCount = 286
+)
+
+// hcode is a huffman code with a bit code and bit length.
+type hcode uint32
+
+func (h hcode) len() uint8 {
+	return uint8(h)
+}
+
+func (h hcode) code64() uint64 {
+	return uint64(h >> 8)
+}
+
+func (h hcode) zero() bool {
+	return h == 0
+}
+
+type huffmanEncoder struct {
+	codes    []hcode
+	bitCount [17]int32
+
+	// Allocate a reusable buffer with the longest possible frequency table.
+	// Possible lengths are codegenCodeCount, offsetCodeCount and literalCount.
+	// The largest of these is literalCount, so we allocate for that case.
+	freqcache [literalCount + 1]literalNode
+}
+
+type literalNode struct {
+	literal uint16
+	freq    uint16
+}
+
+// A levelInfo describes the state of the constructed tree for a given depth.
+type levelInfo struct {
+	// Our level.  for better printing
+	level int32
+
+	// The frequency of the last node at this level
+	lastFreq int32
+
+	// The frequency of the next character to add to this level
+	nextCharFreq int32
+
+	// The frequency of the next pair (from level below) to add to this level.
+	// Only valid if the "needed" value of the next lower level is 0.
+	nextPairFreq int32
+
+	// The number of chains remaining to generate for this level before moving
+	// up to the next level
+	needed int32
+}
+
+// set sets the code and length of an hcode.
+func (h *hcode) set(code uint16, length uint8) {
+	*h = hcode(length) | (hcode(code) << 8)
+}
+
+func newhcode(code uint16, length uint8) hcode {
+	return hcode(length) | (hcode(code) << 8)
+}
+
+func reverseBits(number uint16, bitLength byte) uint16 {
+	return bits.Reverse16(number << ((16 - bitLength) & 15))
+}
+
+func maxNode() literalNode { return literalNode{math.MaxUint16, math.MaxUint16} }
+
+func newHuffmanEncoder(size int) *huffmanEncoder {
+	// Make capacity to next power of two.
+	c := uint(bits.Len32(uint32(size - 1)))
+	return &huffmanEncoder{codes: make([]hcode, size, 1<<c)}
+}
+
+// Generates a HuffmanCode corresponding to the fixed literal table
+func generateFixedLiteralEncoding() *huffmanEncoder {
+	h := newHuffmanEncoder(literalCount)
+	codes := h.codes
+	var ch uint16
+	for ch = range uint16(literalCount) {
+		var bits uint16
+		var size uint8
+		switch {
+		case ch < 144:
+			// size 8, 000110000  .. 10111111
+			bits = ch + 48
+			size = 8
+		case ch < 256:
+			// size 9, 110010000 .. 111111111
+			bits = ch + 400 - 144
+			size = 9
+		case ch < 280:
+			// size 7, 0000000 .. 0010111
+			bits = ch - 256
+			size = 7
+		default:
+			// size 8, 11000000 .. 11000111
+			bits = ch + 192 - 280
+			size = 8
+		}
+		codes[ch] = newhcode(reverseBits(bits, size), size)
+	}
+	return h
+}
+
+func generateFixedOffsetEncoding() *huffmanEncoder {
+	h := newHuffmanEncoder(30)
+	codes := h.codes
+	for ch := range codes {
+		codes[ch] = newhcode(reverseBits(uint16(ch), 5), 5)
+	}
+	return h
+}
+
+var fixedLiteralEncoding = generateFixedLiteralEncoding()
+var fixedOffsetEncoding = generateFixedOffsetEncoding()
+
+func (h *huffmanEncoder) bitLength(freq []uint16) int {
+	var total int
+	for i, f := range freq {
+		if f != 0 {
+			total += int(f) * int(h.codes[i].len())
+		}
+	}
+	return total
+}
+
+func (h *huffmanEncoder) bitLengthRaw(b []byte) int {
+	var total int
+	for _, f := range b {
+		total += int(h.codes[f].len())
+	}
+	return total
+}
+
+// canReuseBits returns the number of bits or math.MaxInt32 if the encoder cannot be reused.
+func (h *huffmanEncoder) canReuseBits(freq []uint16) int {
+	var total int
+	for i, f := range freq {
+		if f != 0 {
+			code := h.codes[i]
+			if code.zero() {
+				return math.MaxInt32
+			}
+			total += int(f) * int(code.len())
+		}
+	}
+	return total
+}
+
+// Return the number of literals assigned to each bit size in the Huffman encoding
+//
+// This method is only called when list.length >= 3
+// The cases of 0, 1, and 2 literals are handled by special case code.
+//
+// list  An array of the literals with non-zero frequencies
+//
+//	and their associated frequencies. The array is in order of increasing
+//	frequency, and has as its last element a special element with frequency
+//	MaxInt32
+//
+// maxBits     The maximum number of bits that should be used to encode any literal.
+//
+//	Must be less than 16.
+//
+// return      An integer array in which array[i] indicates the number of literals
+//
+//	that should be encoded in i bits.
+func (h *huffmanEncoder) bitCounts(list []literalNode, maxBits int32) []int32 {
+	if maxBits >= maxBitsLimit {
+		panic("flate: maxBits too large")
+	}
+	n := int32(len(list))
+	list = list[0 : n+1]
+	list[n] = maxNode()
+
+	// The tree can't have greater depth than n - 1, no matter what. This
+	// saves a little bit of work in some small cases
+	if maxBits > n-1 {
+		maxBits = n - 1
+	}
+
+	// Create information about each of the levels.
+	// A bogus "Level 0" whose sole purpose is so that
+	// level1.prev.needed==0.  This makes level1.nextPairFreq
+	// be a legitimate value that never gets chosen.
+	var levels [maxBitsLimit]levelInfo
+	// leafCounts[i] counts the number of literals at the left
+	// of ancestors of the rightmost node at level i.
+	// leafCounts[i][j] is the number of literals at the left
+	// of the level j ancestor.
+	var leafCounts [maxBitsLimit][maxBitsLimit]int32
+
+	// Descending to only have 1 bounds check.
+	l2f := int32(list[2].freq)
+	l1f := int32(list[1].freq)
+	l0f := int32(list[0].freq) + int32(list[1].freq)
+
+	for level := int32(1); level <= maxBits; level++ {
+		// For every level, the first two items are the first two characters.
+		// We initialize the levels as if we had already figured this out.
+		levels[level] = levelInfo{
+			level:        level,
+			lastFreq:     l1f,
+			nextCharFreq: l2f,
+			nextPairFreq: l0f,
+		}
+		leafCounts[level][level] = 2
+		if level == 1 {
+			levels[level].nextPairFreq = math.MaxInt32
+		}
+	}
+
+	// We need a total of 2*n - 2 items at top level and have already generated 2.
+	levels[maxBits].needed = 2*n - 4
+
+	level := uint32(maxBits)
+	for level < 16 {
+		l := &levels[level]
+		if l.nextPairFreq == math.MaxInt32 && l.nextCharFreq == math.MaxInt32 {
+			// We've run out of both leafs and pairs.
+			// End all calculations for this level.
+			// To make sure we never come back to this level or any lower level,
+			// set nextPairFreq impossibly large.
+			l.needed = 0
+			levels[level+1].nextPairFreq = math.MaxInt32
+			level++
+			continue
+		}
+
+		prevFreq := l.lastFreq
+		if l.nextCharFreq < l.nextPairFreq {
+			// The next item on this row is a leaf node.
+			n := leafCounts[level][level] + 1
+			l.lastFreq = l.nextCharFreq
+			// Lower leafCounts are the same of the previous node.
+			leafCounts[level][level] = n
+			e := list[n]
+			if e.literal < math.MaxUint16 {
+				l.nextCharFreq = int32(e.freq)
+			} else {
+				l.nextCharFreq = math.MaxInt32
+			}
+		} else {
+			// The next item on this row is a pair from the previous row.
+			// nextPairFreq isn't valid until we generate two
+			// more values in the level below
+			l.lastFreq = l.nextPairFreq
+			// Take leaf counts from the lower level, except counts[level] remains the same.
+			if true {
+				save := leafCounts[level][level]
+				leafCounts[level] = leafCounts[level-1]
+				leafCounts[level][level] = save
+			} else {
+				copy(leafCounts[level][:level], leafCounts[level-1][:level])
+			}
+			levels[l.level-1].needed = 2
+		}
+
+		if l.needed--; l.needed == 0 {
+			// We've done everything we need to do for this level.
+			// Continue calculating one level up. Fill in nextPairFreq
+			// of that level with the sum of the two nodes we've just calculated on
+			// this level.
+			if l.level == maxBits {
+				// All done!
+				break
+			}
+			levels[l.level+1].nextPairFreq = prevFreq + l.lastFreq
+			level++
+		} else {
+			// If we stole from below, move down temporarily to replenish it.
+			for levels[level-1].needed > 0 {
+				level--
+			}
+		}
+	}
+
+	// Somethings is wrong if at the end, the top level is null or hasn't used
+	// all of the leaves.
+	if leafCounts[maxBits][maxBits] != n {
+		panic("leafCounts[maxBits][maxBits] != n")
+	}
+
+	bitCount := h.bitCount[:maxBits+1]
+	bits := 1
+	counts := &leafCounts[maxBits]
+	for level := maxBits; level > 0; level-- {
+		// chain.leafCount gives the number of literals requiring at least "bits"
+		// bits to encode.
+		bitCount[bits] = counts[level] - counts[level-1]
+		bits++
+	}
+	return bitCount
+}
+
+// Look at the leaves and assign them a bit count and an encoding as specified
+// in RFC 1951 3.2.2
+func (h *huffmanEncoder) assignEncodingAndSize(bitCount []int32, list []literalNode) {
+	code := uint16(0)
+	for n, bits := range bitCount {
+		code <<= 1
+		if n == 0 || bits == 0 {
+			continue
+		}
+		// The literals list[len(list)-bits] .. list[len(list)-bits]
+		// are encoded using "bits" bits, and get the values
+		// code, code + 1, ....  The code values are
+		// assigned in literal order (not frequency order).
+		chunk := list[len(list)-int(bits):]
+
+		sortByLiteral(chunk)
+		for _, node := range chunk {
+			h.codes[node.literal] = newhcode(reverseBits(code, uint8(n)), uint8(n))
+			code++
+		}
+		list = list[0 : len(list)-int(bits)]
+	}
+}
+
+// Update this Huffman Code object to be the minimum code for the specified frequency count.
+//
+// freq  An array of frequencies, in which frequency[i] gives the frequency of literal i.
+// maxBits  The maximum number of bits to use for any literal.
+func (h *huffmanEncoder) generate(freq []uint16, maxBits int32) {
+	list := h.freqcache[:len(freq)+1]
+	codes := h.codes[:len(freq)]
+	// Number of non-zero literals
+	count := 0
+	// Set list to be the set of all non-zero literals and their frequencies
+	for i, f := range freq {
+		if f != 0 {
+			list[count] = literalNode{uint16(i), f}
+			count++
+		} else {
+			codes[i] = 0
+		}
+	}
+	list[count] = literalNode{}
+
+	list = list[:count]
+	if count <= 2 {
+		// Handle the small cases here, because they are awkward for the general case code. With
+		// two or fewer literals, everything has bit length 1.
+		for i, node := range list {
+			// "list" is in order of increasing literal value.
+			h.codes[node.literal].set(uint16(i), 1)
+		}
+		return
+	}
+	sortByFreq(list)
+
+	// Get the number of literals for each bit count
+	bitCount := h.bitCounts(list, maxBits)
+	// And do the assignment
+	h.assignEncodingAndSize(bitCount, list)
+}
+
+// atLeastOne clamps the result between 1 and 15.
+func atLeastOne(v float32) float32 {
+	if v < 1 {
+		return 1
+	}
+	if v > 15 {
+		return 15
+	}
+	return v
+}
+
+func histogram(b []byte, h []uint16) {
+	if true && len(b) >= 8<<10 {
+		// Split for bigger inputs
+		histogramSplit(b, h)
+	} else {
+		h = h[:256]
+		for _, t := range b {
+			h[t]++
+		}
+	}
+}
+
+func histogramSplit(b []byte, h []uint16) {
+	// Tested, and slightly faster than 2-way.
+	// Writing to separate arrays and combining is also slightly slower.
+	h = h[:256]
+	for len(b)&3 != 0 {
+		h[b[0]]++
+		b = b[1:]
+	}
+	n := len(b) / 4
+	x, y, z, w := b[:n], b[n:], b[n+n:], b[n+n+n:]
+	y, z, w = y[:len(x)], z[:len(x)], w[:len(x)]
+	for i, t := range x {
+		v0 := &h[t]
+		v1 := &h[y[i]]
+		v2 := &h[z[i]]
+		v3 := &h[w[i]]
+		*v0++
+		*v1++
+		*v2++
+		*v3++
+	}
+}

vendor/github.com/klauspost/compress/flate/huffman_sortByFreq.go

@@ -0,0 +1,159 @@
+// Copyright 2009 The Go Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE file.
+
+package flate
+
+// Sort sorts data.
+// It makes one call to data.Len to determine n, and O(n*log(n)) calls to
+// data.Less and data.Swap. The sort is not guaranteed to be stable.
+func sortByFreq(data []literalNode) {
+	n := len(data)
+	quickSortByFreq(data, 0, n, maxDepth(n))
+}
+
+func quickSortByFreq(data []literalNode, a, b, maxDepth int) {
+	for b-a > 12 { // Use ShellSort for slices <= 12 elements
+		if maxDepth == 0 {
+			heapSort(data, a, b)
+			return
+		}
+		maxDepth--
+		mlo, mhi := doPivotByFreq(data, a, b)
+		// Avoiding recursion on the larger subproblem guarantees
+		// a stack depth of at most lg(b-a).
+		if mlo-a < b-mhi {
+			quickSortByFreq(data, a, mlo, maxDepth)
+			a = mhi // i.e., quickSortByFreq(data, mhi, b)
+		} else {
+			quickSortByFreq(data, mhi, b, maxDepth)
+			b = mlo // i.e., quickSortByFreq(data, a, mlo)
+		}
+	}
+	if b-a > 1 {
+		// Do ShellSort pass with gap 6
+		// It could be written in this simplified form cause b-a <= 12
+		for i := a + 6; i < b; i++ {
+			if data[i].freq == data[i-6].freq && data[i].literal < data[i-6].literal || data[i].freq < data[i-6].freq {
+				data[i], data[i-6] = data[i-6], data[i]
+			}
+		}
+		insertionSortByFreq(data, a, b)
+	}
+}
+
+func doPivotByFreq(data []literalNode, lo, hi int) (midlo, midhi int) {
+	m := int(uint(lo+hi) >> 1) // Written like this to avoid integer overflow.
+	if hi-lo > 40 {
+		// Tukey's ``Ninther,'' median of three medians of three.
+		s := (hi - lo) / 8
+		medianOfThreeSortByFreq(data, lo, lo+s, lo+2*s)
+		medianOfThreeSortByFreq(data, m, m-s, m+s)
+		medianOfThreeSortByFreq(data, hi-1, hi-1-s, hi-1-2*s)
+	}
+	medianOfThreeSortByFreq(data, lo, m, hi-1)
+
+	// Invariants are:
+	//	data[lo] = pivot (set up by ChoosePivot)
+	//	data[lo < i < a] < pivot
+	//	data[a <= i < b] <= pivot
+	//	data[b <= i < c] unexamined
+	//	data[c <= i < hi-1] > pivot
+	//	data[hi-1] >= pivot
+	pivot := lo
+	a, c := lo+1, hi-1
+
+	for ; a < c && (data[a].freq == data[pivot].freq && data[a].literal < data[pivot].literal || data[a].freq < data[pivot].freq); a++ {
+	}
+	b := a
+	for {
+		for ; b < c && (data[pivot].freq == data[b].freq && data[pivot].literal > data[b].literal || data[pivot].freq > data[b].freq); b++ { // data[b] <= pivot
+		}
+		for ; b < c && (data[pivot].freq == data[c-1].freq && data[pivot].literal < data[c-1].literal || data[pivot].freq < data[c-1].freq); c-- { // data[c-1] > pivot
+		}
+		if b >= c {
+			break
+		}
+		// data[b] > pivot; data[c-1] <= pivot
+		data[b], data[c-1] = data[c-1], data[b]
+		b++
+		c--
+	}
+	// If hi-c<3 then there are duplicates (by property of median of nine).
+	// Let's be a bit more conservative, and set border to 5.
+	protect := hi-c < 5
+	if !protect && hi-c < (hi-lo)/4 {
+		// Lets test some points for equality to pivot
+		dups := 0
+		if data[pivot].freq == data[hi-1].freq && data[pivot].literal > data[hi-1].literal || data[pivot].freq > data[hi-1].freq { // data[hi-1] = pivot
+			data[c], data[hi-1] = data[hi-1], data[c]
+			c++
+			dups++
+		}
+		if data[b-1].freq == data[pivot].freq && data[b-1].literal > data[pivot].literal || data[b-1].freq > data[pivot].freq { // data[b-1] = pivot
+			b--
+			dups++
+		}
+		// m-lo = (hi-lo)/2 > 6
+		// b-lo > (hi-lo)*3/4-1 > 8
+		// ==> m < b ==> data[m] <= pivot
+		if data[m].freq == data[pivot].freq && data[m].literal > data[pivot].literal || data[m].freq > data[pivot].freq { // data[m] = pivot
+			data[m], data[b-1] = data[b-1], data[m]
+			b--
+			dups++
+		}
+		// if at least 2 points are equal to pivot, assume skewed distribution
+		protect = dups > 1
+	}
+	if protect {
+		// Protect against a lot of duplicates
+		// Add invariant:
+		//	data[a <= i < b] unexamined
+		//	data[b <= i < c] = pivot
+		for {
+			for ; a < b && (data[b-1].freq == data[pivot].freq && data[b-1].literal > data[pivot].literal || data[b-1].freq > data[pivot].freq); b-- { // data[b] == pivot
+			}
+			for ; a < b && (data[a].freq == data[pivot].freq && data[a].literal < data[pivot].literal || data[a].freq < data[pivot].freq); a++ { // data[a] < pivot
+			}
+			if a >= b {
+				break
+			}
+			// data[a] == pivot; data[b-1] < pivot
+			data[a], data[b-1] = data[b-1], data[a]
+			a++
+			b--
+		}
+	}
+	// Swap pivot into middle
+	data[pivot], data[b-1] = data[b-1], data[pivot]
+	return b - 1, c
+}
+
+// Insertion sort
+func insertionSortByFreq(data []literalNode, a, b int) {
+	for i := a + 1; i < b; i++ {
+		for j := i; j > a && (data[j].freq == data[j-1].freq && data[j].literal < data[j-1].literal || data[j].freq < data[j-1].freq); j-- {
+			data[j], data[j-1] = data[j-1], data[j]
+		}
+	}
+}
+
+// quickSortByFreq, loosely following Bentley and McIlroy,
+// ``Engineering a Sort Function,'' SP&E November 1993.
+
+// medianOfThreeSortByFreq moves the median of the three values data[m0], data[m1], data[m2] into data[m1].
+func medianOfThreeSortByFreq(data []literalNode, m1, m0, m2 int) {
+	// sort 3 elements
+	if data[m1].freq == data[m0].freq && data[m1].literal < data[m0].literal || data[m1].freq < data[m0].freq {
+		data[m1], data[m0] = data[m0], data[m1]
+	}
+	// data[m0] <= data[m1]
+	if data[m2].freq == data[m1].freq && data[m2].literal < data[m1].literal || data[m2].freq < data[m1].freq {
+		data[m2], data[m1] = data[m1], data[m2]
+		// data[m0] <= data[m2] && data[m1] < data[m2]
+		if data[m1].freq == data[m0].freq && data[m1].literal < data[m0].literal || data[m1].freq < data[m0].freq {
+			data[m1], data[m0] = data[m0], data[m1]
+		}
+	}
+	// now data[m0] <= data[m1] <= data[m2]
+}

vendor/github.com/klauspost/compress/flate/huffman_sortByLiteral.go

@@ -0,0 +1,201 @@
+// Copyright 2009 The Go Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE file.
+
+package flate
+
+// Sort sorts data.
+// It makes one call to data.Len to determine n, and O(n*log(n)) calls to
+// data.Less and data.Swap. The sort is not guaranteed to be stable.
+func sortByLiteral(data []literalNode) {
+	n := len(data)
+	quickSort(data, 0, n, maxDepth(n))
+}
+
+func quickSort(data []literalNode, a, b, maxDepth int) {
+	for b-a > 12 { // Use ShellSort for slices <= 12 elements
+		if maxDepth == 0 {
+			heapSort(data, a, b)
+			return
+		}
+		maxDepth--
+		mlo, mhi := doPivot(data, a, b)
+		// Avoiding recursion on the larger subproblem guarantees
+		// a stack depth of at most lg(b-a).
+		if mlo-a < b-mhi {
+			quickSort(data, a, mlo, maxDepth)
+			a = mhi // i.e., quickSort(data, mhi, b)
+		} else {
+			quickSort(data, mhi, b, maxDepth)
+			b = mlo // i.e., quickSort(data, a, mlo)
+		}
+	}
+	if b-a > 1 {
+		// Do ShellSort pass with gap 6
+		// It could be written in this simplified form cause b-a <= 12
+		for i := a + 6; i < b; i++ {
+			if data[i].literal < data[i-6].literal {
+				data[i], data[i-6] = data[i-6], data[i]
+			}
+		}
+		insertionSort(data, a, b)
+	}
+}
+func heapSort(data []literalNode, a, b int) {
+	first := a
+	lo := 0
+	hi := b - a
+
+	// Build heap with greatest element at top.
+	for i := (hi - 1) / 2; i >= 0; i-- {
+		siftDown(data, i, hi, first)
+	}
+
+	// Pop elements, largest first, into end of data.
+	for i := hi - 1; i >= 0; i-- {
+		data[first], data[first+i] = data[first+i], data[first]
+		siftDown(data, lo, i, first)
+	}
+}
+
+// siftDown implements the heap property on data[lo, hi).
+// first is an offset into the array where the root of the heap lies.
+func siftDown(data []literalNode, lo, hi, first int) {
+	root := lo
+	for {
+		child := 2*root + 1
+		if child >= hi {
+			break
+		}
+		if child+1 < hi && data[first+child].literal < data[first+child+1].literal {
+			child++
+		}
+		if data[first+root].literal > data[first+child].literal {
+			return
+		}
+		data[first+root], data[first+child] = data[first+child], data[first+root]
+		root = child
+	}
+}
+func doPivot(data []literalNode, lo, hi int) (midlo, midhi int) {
+	m := int(uint(lo+hi) >> 1) // Written like this to avoid integer overflow.
+	if hi-lo > 40 {
+		// Tukey's ``Ninther,'' median of three medians of three.
+		s := (hi - lo) / 8
+		medianOfThree(data, lo, lo+s, lo+2*s)
+		medianOfThree(data, m, m-s, m+s)
+		medianOfThree(data, hi-1, hi-1-s, hi-1-2*s)
+	}
+	medianOfThree(data, lo, m, hi-1)
+
+	// Invariants are:
+	//	data[lo] = pivot (set up by ChoosePivot)
+	//	data[lo < i < a] < pivot
+	//	data[a <= i < b] <= pivot
+	//	data[b <= i < c] unexamined
+	//	data[c <= i < hi-1] > pivot
+	//	data[hi-1] >= pivot
+	pivot := lo
+	a, c := lo+1, hi-1
+
+	for ; a < c && data[a].literal < data[pivot].literal; a++ {
+	}
+	b := a
+	for {
+		for ; b < c && data[pivot].literal > data[b].literal; b++ { // data[b] <= pivot
+		}
+		for ; b < c && data[pivot].literal < data[c-1].literal; c-- { // data[c-1] > pivot
+		}
+		if b >= c {
+			break
+		}
+		// data[b] > pivot; data[c-1] <= pivot
+		data[b], data[c-1] = data[c-1], data[b]
+		b++
+		c--
+	}
+	// If hi-c<3 then there are duplicates (by property of median of nine).
+	// Let's be a bit more conservative, and set border to 5.
+	protect := hi-c < 5
+	if !protect && hi-c < (hi-lo)/4 {
+		// Lets test some points for equality to pivot
+		dups := 0
+		if data[pivot].literal > data[hi-1].literal { // data[hi-1] = pivot
+			data[c], data[hi-1] = data[hi-1], data[c]
+			c++
+			dups++
+		}
+		if data[b-1].literal > data[pivot].literal { // data[b-1] = pivot
+			b--
+			dups++
+		}
+		// m-lo = (hi-lo)/2 > 6
+		// b-lo > (hi-lo)*3/4-1 > 8
+		// ==> m < b ==> data[m] <= pivot
+		if data[m].literal > data[pivot].literal { // data[m] = pivot
+			data[m], data[b-1] = data[b-1], data[m]
+			b--
+			dups++
+		}
+		// if at least 2 points are equal to pivot, assume skewed distribution
+		protect = dups > 1
+	}
+	if protect {
+		// Protect against a lot of duplicates
+		// Add invariant:
+		//	data[a <= i < b] unexamined
+		//	data[b <= i < c] = pivot
+		for {
+			for ; a < b && data[b-1].literal > data[pivot].literal; b-- { // data[b] == pivot
+			}
+			for ; a < b && data[a].literal < data[pivot].literal; a++ { // data[a] < pivot
+			}
+			if a >= b {
+				break
+			}
+			// data[a] == pivot; data[b-1] < pivot
+			data[a], data[b-1] = data[b-1], data[a]
+			a++
+			b--
+		}
+	}
+	// Swap pivot into middle
+	data[pivot], data[b-1] = data[b-1], data[pivot]
+	return b - 1, c
+}
+
+// Insertion sort
+func insertionSort(data []literalNode, a, b int) {
+	for i := a + 1; i < b; i++ {
+		for j := i; j > a && data[j].literal < data[j-1].literal; j-- {
+			data[j], data[j-1] = data[j-1], data[j]
+		}
+	}
+}
+
+// maxDepth returns a threshold at which quicksort should switch
+// to heapsort. It returns 2*ceil(lg(n+1)).
+func maxDepth(n int) int {
+	var depth int
+	for i := n; i > 0; i >>= 1 {
+		depth++
+	}
+	return depth * 2
+}
+
+// medianOfThree moves the median of the three values data[m0], data[m1], data[m2] into data[m1].
+func medianOfThree(data []literalNode, m1, m0, m2 int) {
+	// sort 3 elements
+	if data[m1].literal < data[m0].literal {
+		data[m1], data[m0] = data[m0], data[m1]
+	}
+	// data[m0] <= data[m1]
+	if data[m2].literal < data[m1].literal {
+		data[m2], data[m1] = data[m1], data[m2]
+		// data[m0] <= data[m2] && data[m1] < data[m2]
+		if data[m1].literal < data[m0].literal {
+			data[m1], data[m0] = data[m0], data[m1]
+		}
+	}
+	// now data[m0] <= data[m1] <= data[m2]
+}

vendor/github.com/klauspost/compress/flate/inflate.go

@@ -0,0 +1,865 @@
+// Copyright 2009 The Go Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE file.
+
+// Package flate implements the DEFLATE compressed data format, described in
+// RFC 1951.  The gzip and zlib packages implement access to DEFLATE-based file
+// formats.
+package flate
+
+import (
+	"bufio"
+	"compress/flate"
+	"fmt"
+	"io"
+	"math/bits"
+	"sync"
+)
+
+const (
+	maxCodeLen     = 16 // max length of Huffman code
+	maxCodeLenMask = 15 // mask for max length of Huffman code
+	// The next three numbers come from the RFC section 3.2.7, with the
+	// additional proviso in section 3.2.5 which implies that distance codes
+	// 30 and 31 should never occur in compressed data.
+	maxNumLit  = 286
+	maxNumDist = 30
+	numCodes   = 19 // number of codes in Huffman meta-code
+
+	debugDecode = false
+)
+
+// Value of length - 3 and extra bits.
+type lengthExtra struct {
+	length, extra uint8
+}
+
+var decCodeToLen = [32]lengthExtra{{length: 0x0, extra: 0x0}, {length: 0x1, extra: 0x0}, {length: 0x2, extra: 0x0}, {length: 0x3, extra: 0x0}, {length: 0x4, extra: 0x0}, {length: 0x5, extra: 0x0}, {length: 0x6, extra: 0x0}, {length: 0x7, extra: 0x0}, {length: 0x8, extra: 0x1}, {length: 0xa, extra: 0x1}, {length: 0xc, extra: 0x1}, {length: 0xe, extra: 0x1}, {length: 0x10, extra: 0x2}, {length: 0x14, extra: 0x2}, {length: 0x18, extra: 0x2}, {length: 0x1c, extra: 0x2}, {length: 0x20, extra: 0x3}, {length: 0x28, extra: 0x3}, {length: 0x30, extra: 0x3}, {length: 0x38, extra: 0x3}, {length: 0x40, extra: 0x4}, {length: 0x50, extra: 0x4}, {length: 0x60, extra: 0x4}, {length: 0x70, extra: 0x4}, {length: 0x80, extra: 0x5}, {length: 0xa0, extra: 0x5}, {length: 0xc0, extra: 0x5}, {length: 0xe0, extra: 0x5}, {length: 0xff, extra: 0x0}, {length: 0x0, extra: 0x0}, {length: 0x0, extra: 0x0}, {length: 0x0, extra: 0x0}}
+
+var bitMask32 = [32]uint32{
+	0, 1, 3, 7, 0xF, 0x1F, 0x3F, 0x7F, 0xFF,
+	0x1FF, 0x3FF, 0x7FF, 0xFFF, 0x1FFF, 0x3FFF, 0x7FFF, 0xFFFF,
+	0x1ffff, 0x3ffff, 0x7FFFF, 0xfFFFF, 0x1fFFFF, 0x3fFFFF, 0x7fFFFF, 0xffFFFF,
+	0x1ffFFFF, 0x3ffFFFF, 0x7ffFFFF, 0xfffFFFF, 0x1fffFFFF, 0x3fffFFFF, 0x7fffFFFF,
+} // up to 32 bits
+
+// Initialize the fixedHuffmanDecoder only once upon first use.
+var fixedOnce sync.Once
+var fixedHuffmanDecoder huffmanDecoder
+
+// A CorruptInputError reports the presence of corrupt input at a given offset.
+type CorruptInputError = flate.CorruptInputError
+
+// An InternalError reports an error in the flate code itself.
+type InternalError string
+
+func (e InternalError) Error() string { return "flate: internal error: " + string(e) }
+
+// A ReadError reports an error encountered while reading input.
+//
+// Deprecated: No longer returned.
+type ReadError = flate.ReadError
+
+// A WriteError reports an error encountered while writing output.
+//
+// Deprecated: No longer returned.
+type WriteError = flate.WriteError
+
+// Resetter resets a ReadCloser returned by NewReader or NewReaderDict to
+// to switch to a new underlying Reader. This permits reusing a ReadCloser
+// instead of allocating a new one.
+type Resetter interface {
+	// Reset discards any buffered data and resets the Resetter as if it was
+	// newly initialized with the given reader.
+	Reset(r io.Reader, dict []byte) error
+}
+
+// The data structure for decoding Huffman tables is based on that of
+// zlib. There is a lookup table of a fixed bit width (huffmanChunkBits),
+// For codes smaller than the table width, there are multiple entries
+// (each combination of trailing bits has the same value). For codes
+// larger than the table width, the table contains a link to an overflow
+// table. The width of each entry in the link table is the maximum code
+// size minus the chunk width.
+//
+// Note that you can do a lookup in the table even without all bits
+// filled. Since the extra bits are zero, and the DEFLATE Huffman codes
+// have the property that shorter codes come before longer ones, the
+// bit length estimate in the result is a lower bound on the actual
+// number of bits.
+//
+// See the following:
+//	http://www.gzip.org/algorithm.txt
+
+// chunk & 15 is number of bits
+// chunk >> 4 is value, including table link
+
+const (
+	huffmanChunkBits  = 9
+	huffmanNumChunks  = 1 << huffmanChunkBits
+	huffmanCountMask  = 15
+	huffmanValueShift = 4
+)
+
+type huffmanDecoder struct {
+	maxRead  int                       // the maximum number of bits we can read and not overread
+	chunks   *[huffmanNumChunks]uint16 // chunks as described above
+	links    [][]uint16                // overflow links
+	linkMask uint32                    // mask the width of the link table
+}
+
+// Initialize Huffman decoding tables from array of code lengths.
+// Following this function, h is guaranteed to be initialized into a complete
+// tree (i.e., neither over-subscribed nor under-subscribed). The exception is a
+// degenerate case where the tree has only a single symbol with length 1. Empty
+// trees are permitted.
+func (h *huffmanDecoder) init(lengths []int) bool {
+	// Sanity enables additional runtime tests during Huffman
+	// table construction. It's intended to be used during
+	// development to supplement the currently ad-hoc unit tests.
+	const sanity = false
+
+	if h.chunks == nil {
+		h.chunks = new([huffmanNumChunks]uint16)
+	}
+
+	if h.maxRead != 0 {
+		*h = huffmanDecoder{chunks: h.chunks, links: h.links}
+	}
+
+	// Count number of codes of each length,
+	// compute maxRead and max length.
+	var count [maxCodeLen]int
+	var min, max int
+	for _, n := range lengths {
+		if n == 0 {
+			continue
+		}
+		if min == 0 || n < min {
+			min = n
+		}
+		if n > max {
+			max = n
+		}
+		count[n&maxCodeLenMask]++
+	}
+
+	// Empty tree. The decompressor.huffSym function will fail later if the tree
+	// is used. Technically, an empty tree is only valid for the HDIST tree and
+	// not the HCLEN and HLIT tree. However, a stream with an empty HCLEN tree
+	// is guaranteed to fail since it will attempt to use the tree to decode the
+	// codes for the HLIT and HDIST trees. Similarly, an empty HLIT tree is
+	// guaranteed to fail later since the compressed data section must be
+	// composed of at least one symbol (the end-of-block marker).
+	if max == 0 {
+		return true
+	}
+
+	code := 0
+	var nextcode [maxCodeLen]int
+	for i := min; i <= max; i++ {
+		code <<= 1
+		nextcode[i&maxCodeLenMask] = code
+		code += count[i&maxCodeLenMask]
+	}
+
+	// Check that the coding is complete (i.e., that we've
+	// assigned all 2-to-the-max possible bit sequences).
+	// Exception: To be compatible with zlib, we also need to
+	// accept degenerate single-code codings. See also
+	// TestDegenerateHuffmanCoding.
+	if code != 1<<uint(max) && !(code == 1 && max == 1) {
+		if debugDecode {
+			fmt.Println("coding failed, code, max:", code, max, code == 1<<uint(max), code == 1 && max == 1, "(one should be true)")
+		}
+		return false
+	}
+
+	h.maxRead = min
+
+	chunks := h.chunks[:]
+	for i := range chunks {
+		chunks[i] = 0
+	}
+
+	if max > huffmanChunkBits {
+		numLinks := 1 << (uint(max) - huffmanChunkBits)
+		h.linkMask = uint32(numLinks - 1)
+
+		// create link tables
+		link := nextcode[huffmanChunkBits+1] >> 1
+		if cap(h.links) < huffmanNumChunks-link {
+			h.links = make([][]uint16, huffmanNumChunks-link)
+		} else {
+			h.links = h.links[:huffmanNumChunks-link]
+		}
+		for j := uint(link); j < huffmanNumChunks; j++ {
+			reverse := int(bits.Reverse16(uint16(j)))
+			reverse >>= uint(16 - huffmanChunkBits)
+			off := j - uint(link)
+			if sanity && h.chunks[reverse] != 0 {
+				panic("impossible: overwriting existing chunk")
+			}
+			h.chunks[reverse] = uint16(off<<huffmanValueShift | (huffmanChunkBits + 1))
+			if cap(h.links[off]) < numLinks {
+				h.links[off] = make([]uint16, numLinks)
+			} else {
+				h.links[off] = h.links[off][:numLinks]
+			}
+		}
+	} else {
+		h.links = h.links[:0]
+	}
+
+	for i, n := range lengths {
+		if n == 0 {
+			continue
+		}
+		code := nextcode[n]
+		nextcode[n]++
+		chunk := uint16(i<<huffmanValueShift | n)
+		reverse := int(bits.Reverse16(uint16(code)))
+		reverse >>= uint(16 - n)
+		if n <= huffmanChunkBits {
+			for off := reverse; off < len(h.chunks); off += 1 << uint(n) {
+				// We should never need to overwrite
+				// an existing chunk. Also, 0 is
+				// never a valid chunk, because the
+				// lower 4 "count" bits should be
+				// between 1 and 15.
+				if sanity && h.chunks[off] != 0 {
+					panic("impossible: overwriting existing chunk")
+				}
+				h.chunks[off] = chunk
+			}
+		} else {
+			j := reverse & (huffmanNumChunks - 1)
+			if sanity && h.chunks[j]&huffmanCountMask != huffmanChunkBits+1 {
+				// Longer codes should have been
+				// associated with a link table above.
+				panic("impossible: not an indirect chunk")
+			}
+			value := h.chunks[j] >> huffmanValueShift
+			linktab := h.links[value]
+			reverse >>= huffmanChunkBits
+			for off := reverse; off < len(linktab); off += 1 << uint(n-huffmanChunkBits) {
+				if sanity && linktab[off] != 0 {
+					panic("impossible: overwriting existing chunk")
+				}
+				linktab[off] = chunk
+			}
+		}
+	}
+
+	if sanity {
+		// Above we've sanity checked that we never overwrote
+		// an existing entry. Here we additionally check that
+		// we filled the tables completely.
+		for i, chunk := range h.chunks {
+			if chunk == 0 {
+				// As an exception, in the degenerate
+				// single-code case, we allow odd
+				// chunks to be missing.
+				if code == 1 && i%2 == 1 {
+					continue
+				}
+				panic("impossible: missing chunk")
+			}
+		}
+		for _, linktab := range h.links {
+			for _, chunk := range linktab {
+				if chunk == 0 {
+					panic("impossible: missing chunk")
+				}
+			}
+		}
+	}
+
+	return true
+}
+
+// Reader is the actual read interface needed by NewReader.
+// If the passed in io.Reader does not also have ReadByte,
+// the NewReader will introduce its own buffering.
+type Reader interface {
+	io.Reader
+	io.ByteReader
+}
+
+type step uint8
+
+const (
+	copyData step = iota + 1
+	nextBlock
+	huffmanBytesBuffer
+	huffmanBytesReader
+	huffmanBufioReader
+	huffmanStringsReader
+	huffmanGenericReader
+)
+
+// flushMode tells decompressor when to return data
+type flushMode uint8
+
+const (
+	syncFlush    flushMode = iota // return data after sync flush block
+	partialFlush                  // return data after each block
+)
+
+// Decompress state.
+type decompressor struct {
+	// Input source.
+	r       Reader
+	roffset int64
+
+	// Huffman decoders for literal/length, distance.
+	h1, h2 huffmanDecoder
+
+	// Length arrays used to define Huffman codes.
+	bits     *[maxNumLit + maxNumDist]int
+	codebits *[numCodes]int
+
+	// Output history, buffer.
+	dict dictDecoder
+
+	// Next step in the decompression,
+	// and decompression state.
+	step      step
+	stepState int
+	err       error
+	toRead    []byte
+	hl, hd    *huffmanDecoder
+	copyLen   int
+	copyDist  int
+
+	// Temporary buffer (avoids repeated allocation).
+	buf [4]byte
+
+	// Input bits, in top of b.
+	b uint32
+
+	nb    uint
+	final bool
+
+	flushMode flushMode
+}
+
+func (f *decompressor) nextBlock() {
+	for f.nb < 1+2 {
+		if f.err = f.moreBits(); f.err != nil {
+			return
+		}
+	}
+	f.final = f.b&1 == 1
+	f.b >>= 1
+	typ := f.b & 3
+	f.b >>= 2
+	f.nb -= 1 + 2
+	switch typ {
+	case 0:
+		f.dataBlock()
+		if debugDecode {
+			fmt.Println("stored block")
+		}
+	case 1:
+		// compressed, fixed Huffman tables
+		f.hl = &fixedHuffmanDecoder
+		f.hd = nil
+		f.huffmanBlockDecoder()
+		if debugDecode {
+			fmt.Println("predefinied huffman block")
+		}
+	case 2:
+		// compressed, dynamic Huffman tables
+		if f.err = f.readHuffman(); f.err != nil {
+			break
+		}
+		f.hl = &f.h1
+		f.hd = &f.h2
+		f.huffmanBlockDecoder()
+		if debugDecode {
+			fmt.Println("dynamic huffman block")
+		}
+	default:
+		// 3 is reserved.
+		if debugDecode {
+			fmt.Println("reserved data block encountered")
+		}
+		f.err = CorruptInputError(f.roffset)
+	}
+}
+
+func (f *decompressor) Read(b []byte) (int, error) {
+	for {
+		if len(f.toRead) > 0 {
+			n := copy(b, f.toRead)
+			f.toRead = f.toRead[n:]
+			if len(f.toRead) == 0 {
+				return n, f.err
+			}
+			return n, nil
+		}
+		if f.err != nil {
+			return 0, f.err
+		}
+
+		f.doStep()
+
+		if f.err != nil && len(f.toRead) == 0 {
+			f.toRead = f.dict.readFlush() // Flush what's left in case of error
+		}
+	}
+}
+
+// WriteTo implements the io.WriteTo interface for io.Copy and friends.
+func (f *decompressor) WriteTo(w io.Writer) (int64, error) {
+	total := int64(0)
+	flushed := false
+	for {
+		if len(f.toRead) > 0 {
+			n, err := w.Write(f.toRead)
+			total += int64(n)
+			if err != nil {
+				f.err = err
+				return total, err
+			}
+			if n != len(f.toRead) {
+				return total, io.ErrShortWrite
+			}
+			f.toRead = f.toRead[:0]
+		}
+		if f.err != nil && flushed {
+			if f.err == io.EOF {
+				return total, nil
+			}
+			return total, f.err
+		}
+		if f.err == nil {
+			f.doStep()
+		}
+		if len(f.toRead) == 0 && f.err != nil && !flushed {
+			f.toRead = f.dict.readFlush() // Flush what's left in case of error
+			flushed = true
+		}
+	}
+}
+
+func (f *decompressor) Close() error {
+	if f.err == io.EOF {
+		return nil
+	}
+	return f.err
+}
+
+// RFC 1951 section 3.2.7.
+// Compression with dynamic Huffman codes
+
+var codeOrder = [...]int{16, 17, 18, 0, 8, 7, 9, 6, 10, 5, 11, 4, 12, 3, 13, 2, 14, 1, 15}
+
+func (f *decompressor) readHuffman() error {
+	// HLIT[5], HDIST[5], HCLEN[4].
+	for f.nb < 5+5+4 {
+		if err := f.moreBits(); err != nil {
+			return err
+		}
+	}
+	nlit := int(f.b&0x1F) + 257
+	if nlit > maxNumLit {
+		if debugDecode {
+			fmt.Println("nlit > maxNumLit", nlit)
+		}
+		return CorruptInputError(f.roffset)
+	}
+	f.b >>= 5
+	ndist := int(f.b&0x1F) + 1
+	if ndist > maxNumDist {
+		if debugDecode {
+			fmt.Println("ndist > maxNumDist", ndist)
+		}
+		return CorruptInputError(f.roffset)
+	}
+	f.b >>= 5
+	nclen := int(f.b&0xF) + 4
+	// numCodes is 19, so nclen is always valid.
+	f.b >>= 4
+	f.nb -= 5 + 5 + 4
+
+	// (HCLEN+4)*3 bits: code lengths in the magic codeOrder order.
+	for i := range nclen {
+		for f.nb < 3 {
+			if err := f.moreBits(); err != nil {
+				return err
+			}
+		}
+		f.codebits[codeOrder[i]] = int(f.b & 0x7)
+		f.b >>= 3
+		f.nb -= 3
+	}
+	for i := nclen; i < len(codeOrder); i++ {
+		f.codebits[codeOrder[i]] = 0
+	}
+	if !f.h1.init(f.codebits[0:]) {
+		if debugDecode {
+			fmt.Println("init codebits failed")
+		}
+		return CorruptInputError(f.roffset)
+	}
+
+	// HLIT + 257 code lengths, HDIST + 1 code lengths,
+	// using the code length Huffman code.
+	for i, n := 0, nlit+ndist; i < n; {
+		x, err := f.huffSym(&f.h1)
+		if err != nil {
+			return err
+		}
+		if x < 16 {
+			// Actual length.
+			f.bits[i] = x
+			i++
+			continue
+		}
+		// Repeat previous length or zero.
+		var rep int
+		var nb uint
+		var b int
+		switch x {
+		default:
+			return InternalError("unexpected length code")
+		case 16:
+			rep = 3
+			nb = 2
+			if i == 0 {
+				if debugDecode {
+					fmt.Println("i==0")
+				}
+				return CorruptInputError(f.roffset)
+			}
+			b = f.bits[i-1]
+		case 17:
+			rep = 3
+			nb = 3
+			b = 0
+		case 18:
+			rep = 11
+			nb = 7
+			b = 0
+		}
+		for f.nb < nb {
+			if err := f.moreBits(); err != nil {
+				if debugDecode {
+					fmt.Println("morebits:", err)
+				}
+				return err
+			}
+		}
+		rep += int(f.b & uint32(1<<(nb&regSizeMaskUint32)-1))
+		f.b >>= nb & regSizeMaskUint32
+		f.nb -= nb
+		if i+rep > n {
+			if debugDecode {
+				fmt.Println("i+rep > n", i, rep, n)
+			}
+			return CorruptInputError(f.roffset)
+		}
+		for j := 0; j < rep; j++ {
+			f.bits[i] = b
+			i++
+		}
+	}
+
+	if !f.h1.init(f.bits[0:nlit]) || !f.h2.init(f.bits[nlit:nlit+ndist]) {
+		if debugDecode {
+			fmt.Println("init2 failed")
+		}
+		return CorruptInputError(f.roffset)
+	}
+
+	// As an optimization, we can initialize the maxRead bits to read at a time
+	// for the HLIT tree to the length of the EOB marker since we know that
+	// every block must terminate with one. This preserves the property that
+	// we never read any extra bytes after the end of the DEFLATE stream.
+	if f.h1.maxRead < f.bits[endBlockMarker] {
+		f.h1.maxRead = f.bits[endBlockMarker]
+	}
+	if !f.final {
+		// If not the final block, the smallest block possible is
+		// a predefined table, BTYPE=01, with a single EOB marker.
+		// This will take up 3 + 7 bits.
+		f.h1.maxRead += 10
+	}
+
+	return nil
+}
+
+// Copy a single uncompressed data block from input to output.
+func (f *decompressor) dataBlock() {
+	// Uncompressed.
+	// Discard current half-byte.
+	left := (f.nb) & 7
+	f.nb -= left
+	f.b >>= left
+
+	offBytes := f.nb >> 3
+	// Unfilled values will be overwritten.
+	f.buf[0] = uint8(f.b)
+	f.buf[1] = uint8(f.b >> 8)
+	f.buf[2] = uint8(f.b >> 16)
+	f.buf[3] = uint8(f.b >> 24)
+
+	f.roffset += int64(offBytes)
+	f.nb, f.b = 0, 0
+
+	// Length then ones-complement of length.
+	nr, err := io.ReadFull(f.r, f.buf[offBytes:4])
+	f.roffset += int64(nr)
+	if err != nil {
+		f.err = noEOF(err)
+		return
+	}
+	n := uint16(f.buf[0]) | uint16(f.buf[1])<<8
+	nn := uint16(f.buf[2]) | uint16(f.buf[3])<<8
+	if nn != ^n {
+		if debugDecode {
+			ncomp := ^n
+			fmt.Println("uint16(nn) != uint16(^n)", nn, ncomp)
+		}
+		f.err = CorruptInputError(f.roffset)
+		return
+	}
+
+	if n == 0 {
+		if f.flushMode == syncFlush {
+			f.toRead = f.dict.readFlush()
+		}
+
+		f.finishBlock()
+		return
+	}
+
+	f.copyLen = int(n)
+	f.copyData()
+}
+
+// copyData copies f.copyLen bytes from the underlying reader into f.hist.
+// It pauses for reads when f.hist is full.
+func (f *decompressor) copyData() {
+	buf := f.dict.writeSlice()
+	if len(buf) > f.copyLen {
+		buf = buf[:f.copyLen]
+	}
+
+	cnt, err := io.ReadFull(f.r, buf)
+	f.roffset += int64(cnt)
+	f.copyLen -= cnt
+	f.dict.writeMark(cnt)
+	if err != nil {
+		f.err = noEOF(err)
+		return
+	}
+
+	if f.dict.availWrite() == 0 || f.copyLen > 0 {
+		f.toRead = f.dict.readFlush()
+		f.step = copyData
+		return
+	}
+	f.finishBlock()
+}
+
+func (f *decompressor) finishBlock() {
+	if f.final {
+		if f.dict.availRead() > 0 {
+			f.toRead = f.dict.readFlush()
+		}
+
+		f.err = io.EOF
+	} else if f.flushMode == partialFlush && f.dict.availRead() > 0 {
+		f.toRead = f.dict.readFlush()
+	}
+
+	f.step = nextBlock
+}
+
+func (f *decompressor) doStep() {
+	switch f.step {
+	case copyData:
+		f.copyData()
+	case nextBlock:
+		f.nextBlock()
+	case huffmanBytesBuffer:
+		f.huffmanBytesBuffer()
+	case huffmanBytesReader:
+		f.huffmanBytesReader()
+	case huffmanBufioReader:
+		f.huffmanBufioReader()
+	case huffmanStringsReader:
+		f.huffmanStringsReader()
+	case huffmanGenericReader:
+		f.huffmanGenericReader()
+	default:
+		panic("BUG: unexpected step state")
+	}
+}
+
+// noEOF returns err, unless err == io.EOF, in which case it returns io.ErrUnexpectedEOF.
+func noEOF(e error) error {
+	if e == io.EOF {
+		return io.ErrUnexpectedEOF
+	}
+	return e
+}
+
+func (f *decompressor) moreBits() error {
+	c, err := f.r.ReadByte()
+	if err != nil {
+		return noEOF(err)
+	}
+	f.roffset++
+	f.b |= uint32(c) << (f.nb & regSizeMaskUint32)
+	f.nb += 8
+	return nil
+}
+
+// Read the next Huffman-encoded symbol from f according to h.
+func (f *decompressor) huffSym(h *huffmanDecoder) (int, error) {
+	// Since a huffmanDecoder can be empty or be composed of a degenerate tree
+	// with single element, huffSym must error on these two edge cases. In both
+	// cases, the chunks slice will be 0 for the invalid sequence, leading it
+	// satisfy the n == 0 check below.
+	n := uint(h.maxRead)
+	// Optimization. Compiler isn't smart enough to keep f.b,f.nb in registers,
+	// but is smart enough to keep local variables in registers, so use nb and b,
+	// inline call to moreBits and reassign b,nb back to f on return.
+	nb, b := f.nb, f.b
+	for {
+		for nb < n {
+			c, err := f.r.ReadByte()
+			if err != nil {
+				f.b = b
+				f.nb = nb
+				return 0, noEOF(err)
+			}
+			f.roffset++
+			b |= uint32(c) << (nb & regSizeMaskUint32)
+			nb += 8
+		}
+		chunk := h.chunks[b&(huffmanNumChunks-1)]
+		n = uint(chunk & huffmanCountMask)
+		if n > huffmanChunkBits {
+			chunk = h.links[chunk>>huffmanValueShift][(b>>huffmanChunkBits)&h.linkMask]
+			n = uint(chunk & huffmanCountMask)
+		}
+		if n <= nb {
+			if n == 0 {
+				f.b = b
+				f.nb = nb
+				if debugDecode {
+					fmt.Println("huffsym: n==0")
+				}
+				f.err = CorruptInputError(f.roffset)
+				return 0, f.err
+			}
+			f.b = b >> (n & regSizeMaskUint32)
+			f.nb = nb - n
+			return int(chunk >> huffmanValueShift), nil
+		}
+	}
+}
+
+func makeReader(r io.Reader) Reader {
+	if rr, ok := r.(Reader); ok {
+		return rr
+	}
+	return bufio.NewReader(r)
+}
+
+func fixedHuffmanDecoderInit() {
+	fixedOnce.Do(func() {
+		// These come from the RFC section 3.2.6.
+		var bits [288]int
+		for i := range 144 {
+			bits[i] = 8
+		}
+		for i := 144; i < 256; i++ {
+			bits[i] = 9
+		}
+		for i := 256; i < 280; i++ {
+			bits[i] = 7
+		}
+		for i := 280; i < 288; i++ {
+			bits[i] = 8
+		}
+		fixedHuffmanDecoder.init(bits[:])
+	})
+}
+
+func (f *decompressor) Reset(r io.Reader, dict []byte) error {
+	*f = decompressor{
+		r:        makeReader(r),
+		bits:     f.bits,
+		codebits: f.codebits,
+		h1:       f.h1,
+		h2:       f.h2,
+		dict:     f.dict,
+		step:     nextBlock,
+	}
+	f.dict.init(maxMatchOffset, dict)
+	return nil
+}
+
+type ReaderOpt func(*decompressor)
+
+// WithPartialBlock tells decompressor to return after each block,
+// so it can read data written with partial flush
+func WithPartialBlock() ReaderOpt {
+	return func(f *decompressor) {
+		f.flushMode = partialFlush
+	}
+}
+
+// WithDict initializes the reader with a preset dictionary
+func WithDict(dict []byte) ReaderOpt {
+	return func(f *decompressor) {
+		f.dict.init(maxMatchOffset, dict)
+	}
+}
+
+// NewReaderOpts returns new reader with provided options
+func NewReaderOpts(r io.Reader, opts ...ReaderOpt) io.ReadCloser {
+	fixedHuffmanDecoderInit()
+
+	var f decompressor
+	f.r = makeReader(r)
+	f.bits = new([maxNumLit + maxNumDist]int)
+	f.codebits = new([numCodes]int)
+	f.step = nextBlock
+	f.dict.init(maxMatchOffset, nil)
+
+	for _, opt := range opts {
+		opt(&f)
+	}
+
+	return &f
+}
+
+// NewReader returns a new ReadCloser that can be used
+// to read the uncompressed version of r.
+// If r does not also implement io.ByteReader,
+// the decompressor may read more data than necessary from r.
+// It is the caller's responsibility to call Close on the ReadCloser
+// when finished reading.
+//
+// The ReadCloser returned by NewReader also implements Resetter.
+func NewReader(r io.Reader) io.ReadCloser {
+	return NewReaderOpts(r)
+}
+
+// NewReaderDict is like NewReader but initializes the reader
+// with a preset dictionary. The returned Reader behaves as if
+// the uncompressed data stream started with the given dictionary,
+// which has already been read. NewReaderDict is typically used
+// to read data compressed by NewWriterDict.
+//
+// The ReadCloser returned by NewReader also implements Resetter.
+func NewReaderDict(r io.Reader, dict []byte) io.ReadCloser {
+	return NewReaderOpts(r, WithDict(dict))
+}

vendor/github.com/klauspost/compress/flate/level1.go

@@ -0,0 +1,215 @@
+package flate
+
+import (
+	"fmt"
+
+	"github.com/klauspost/compress/internal/le"
+)
+
+// fastGen maintains the table for matches,
+// and the previous byte block for level 2.
+// This is the generic implementation.
+type fastEncL1 struct {
+	fastGen
+	table [tableSize]tableEntry
+}
+
+// EncodeL1 uses a similar algorithm to level 1
+func (e *fastEncL1) Encode(dst *tokens, src []byte) {
+	const (
+		inputMargin            = 12 - 1
+		minNonLiteralBlockSize = 1 + 1 + inputMargin
+		hashBytes              = 5
+	)
+	if debugDeflate && e.cur < 0 {
+		panic(fmt.Sprint("e.cur < 0: ", e.cur))
+	}
+
+	// Protect against e.cur wraparound.
+	for e.cur >= bufferReset {
+		if len(e.hist) == 0 {
+			for i := range e.table[:] {
+				e.table[i] = tableEntry{}
+			}
+			e.cur = maxMatchOffset
+			break
+		}
+		// Shift down everything in the table that isn't already too far away.
+		minOff := e.cur + int32(len(e.hist)) - maxMatchOffset
+		for i := range e.table[:] {
+			v := e.table[i].offset
+			if v <= minOff {
+				v = 0
+			} else {
+				v = v - e.cur + maxMatchOffset
+			}
+			e.table[i].offset = v
+		}
+		e.cur = maxMatchOffset
+	}
+
+	s := e.addBlock(src)
+
+	// This check isn't in the Snappy implementation, but there, the caller
+	// instead of the callee handles this case.
+	if len(src) < minNonLiteralBlockSize {
+		// We do not fill the token table.
+		// This will be picked up by caller.
+		dst.n = uint16(len(src))
+		return
+	}
+
+	// Override src
+	src = e.hist
+	nextEmit := s
+
+	// sLimit is when to stop looking for offset/length copies. The inputMargin
+	// lets us use a fast path for emitLiteral in the main loop, while we are
+	// looking for copies.
+	sLimit := int32(len(src) - inputMargin)
+
+	// nextEmit is where in src the next emitLiteral should start from.
+	cv := load6432(src, s)
+
+	for {
+		const skipLog = 5
+		const doEvery = 2
+
+		nextS := s
+		var candidate tableEntry
+		var t int32
+		for {
+			nextHash := hashLen(cv, tableBits, hashBytes)
+			candidate = e.table[nextHash]
+			nextS = s + doEvery + (s-nextEmit)>>skipLog
+			if nextS > sLimit {
+				goto emitRemainder
+			}
+
+			now := load6432(src, nextS)
+			e.table[nextHash] = tableEntry{offset: s + e.cur}
+			nextHash = hashLen(now, tableBits, hashBytes)
+			t = candidate.offset - e.cur
+			if s-t < maxMatchOffset && uint32(cv) == load3232(src, t) {
+				e.table[nextHash] = tableEntry{offset: nextS + e.cur}
+				break
+			}
+
+			// Do one right away...
+			cv = now
+			s = nextS
+			nextS++
+			candidate = e.table[nextHash]
+			now >>= 8
+			e.table[nextHash] = tableEntry{offset: s + e.cur}
+
+			t = candidate.offset - e.cur
+			if s-t < maxMatchOffset && uint32(cv) == load3232(src, t) {
+				e.table[nextHash] = tableEntry{offset: nextS + e.cur}
+				break
+			}
+			cv = now
+			s = nextS
+		}
+
+		// A 4-byte match has been found. We'll later see if more than 4 bytes
+		// match. But, prior to the match, src[nextEmit:s] are unmatched. Emit
+		// them as literal bytes.
+		for {
+			// Invariant: we have a 4-byte match at s, and no need to emit any
+			// literal bytes prior to s.
+
+			// Extend the 4-byte match as long as possible.
+			l := e.matchlenLong(int(s+4), int(t+4), src) + 4
+
+			// Extend backwards
+			for t > 0 && s > nextEmit && le.Load8(src, t-1) == le.Load8(src, s-1) {
+				s--
+				t--
+				l++
+			}
+			if nextEmit < s {
+				if false {
+					emitLiteral(dst, src[nextEmit:s])
+				} else {
+					for _, v := range src[nextEmit:s] {
+						dst.tokens[dst.n] = token(v)
+						dst.litHist[v]++
+						dst.n++
+					}
+				}
+			}
+
+			// Save the match found
+			if false {
+				dst.AddMatchLong(l, uint32(s-t-baseMatchOffset))
+			} else {
+				// Inlined...
+				xoffset := uint32(s - t - baseMatchOffset)
+				xlength := l
+				oc := offsetCode(xoffset)
+				xoffset |= oc << 16
+				for xlength > 0 {
+					xl := xlength
+					if xl > 258 {
+						if xl > 258+baseMatchLength {
+							xl = 258
+						} else {
+							xl = 258 - baseMatchLength
+						}
+					}
+					xlength -= xl
+					xl -= baseMatchLength
+					dst.extraHist[lengthCodes1[uint8(xl)]]++
+					dst.offHist[oc]++
+					dst.tokens[dst.n] = token(matchType | uint32(xl)<<lengthShift | xoffset)
+					dst.n++
+				}
+			}
+			s += l
+			nextEmit = s
+			if nextS >= s {
+				s = nextS + 1
+			}
+			if s >= sLimit {
+				// Index first pair after match end.
+				if int(s+l+8) < len(src) {
+					cv := load6432(src, s)
+					e.table[hashLen(cv, tableBits, hashBytes)] = tableEntry{offset: s + e.cur}
+				}
+				goto emitRemainder
+			}
+
+			// We could immediately start working at s now, but to improve
+			// compression we first update the hash table at s-2 and at s. If
+			// another emitCopy is not our next move, also calculate nextHash
+			// at s+1. At least on GOARCH=amd64, these three hash calculations
+			// are faster as one load64 call (with some shifts) instead of
+			// three load32 calls.
+			x := load6432(src, s-2)
+			o := e.cur + s - 2
+			prevHash := hashLen(x, tableBits, hashBytes)
+			e.table[prevHash] = tableEntry{offset: o}
+			x >>= 16
+			currHash := hashLen(x, tableBits, hashBytes)
+			candidate = e.table[currHash]
+			e.table[currHash] = tableEntry{offset: o + 2}
+
+			t = candidate.offset - e.cur
+			if s-t > maxMatchOffset || uint32(x) != load3232(src, t) {
+				cv = x >> 8
+				s++
+				break
+			}
+		}
+	}
+
+emitRemainder:
+	if int(nextEmit) < len(src) {
+		// If nothing was added, don't encode literals.
+		if dst.n == 0 {
+			return
+		}
+		emitLiteral(dst, src[nextEmit:])
+	}
+}

vendor/github.com/klauspost/compress/flate/level2.go

@@ -0,0 +1,214 @@
+package flate
+
+import "fmt"
+
+// fastGen maintains the table for matches,
+// and the previous byte block for level 2.
+// This is the generic implementation.
+type fastEncL2 struct {
+	fastGen
+	table [bTableSize]tableEntry
+}
+
+// EncodeL2 uses a similar algorithm to level 1, but is capable
+// of matching across blocks giving better compression at a small slowdown.
+func (e *fastEncL2) Encode(dst *tokens, src []byte) {
+	const (
+		inputMargin            = 12 - 1
+		minNonLiteralBlockSize = 1 + 1 + inputMargin
+		hashBytes              = 5
+	)
+
+	if debugDeflate && e.cur < 0 {
+		panic(fmt.Sprint("e.cur < 0: ", e.cur))
+	}
+
+	// Protect against e.cur wraparound.
+	for e.cur >= bufferReset {
+		if len(e.hist) == 0 {
+			for i := range e.table[:] {
+				e.table[i] = tableEntry{}
+			}
+			e.cur = maxMatchOffset
+			break
+		}
+		// Shift down everything in the table that isn't already too far away.
+		minOff := e.cur + int32(len(e.hist)) - maxMatchOffset
+		for i := range e.table[:] {
+			v := e.table[i].offset
+			if v <= minOff {
+				v = 0
+			} else {
+				v = v - e.cur + maxMatchOffset
+			}
+			e.table[i].offset = v
+		}
+		e.cur = maxMatchOffset
+	}
+
+	s := e.addBlock(src)
+
+	// This check isn't in the Snappy implementation, but there, the caller
+	// instead of the callee handles this case.
+	if len(src) < minNonLiteralBlockSize {
+		// We do not fill the token table.
+		// This will be picked up by caller.
+		dst.n = uint16(len(src))
+		return
+	}
+
+	// Override src
+	src = e.hist
+	nextEmit := s
+
+	// sLimit is when to stop looking for offset/length copies. The inputMargin
+	// lets us use a fast path for emitLiteral in the main loop, while we are
+	// looking for copies.
+	sLimit := int32(len(src) - inputMargin)
+
+	// nextEmit is where in src the next emitLiteral should start from.
+	cv := load6432(src, s)
+	for {
+		// When should we start skipping if we haven't found matches in a long while.
+		const skipLog = 5
+		const doEvery = 2
+
+		nextS := s
+		var candidate tableEntry
+		for {
+			nextHash := hashLen(cv, bTableBits, hashBytes)
+			s = nextS
+			nextS = s + doEvery + (s-nextEmit)>>skipLog
+			if nextS > sLimit {
+				goto emitRemainder
+			}
+			candidate = e.table[nextHash]
+			now := load6432(src, nextS)
+			e.table[nextHash] = tableEntry{offset: s + e.cur}
+			nextHash = hashLen(now, bTableBits, hashBytes)
+
+			offset := s - (candidate.offset - e.cur)
+			if offset < maxMatchOffset && uint32(cv) == load3232(src, candidate.offset-e.cur) {
+				e.table[nextHash] = tableEntry{offset: nextS + e.cur}
+				break
+			}
+
+			// Do one right away...
+			cv = now
+			s = nextS
+			nextS++
+			candidate = e.table[nextHash]
+			now >>= 8
+			e.table[nextHash] = tableEntry{offset: s + e.cur}
+
+			offset = s - (candidate.offset - e.cur)
+			if offset < maxMatchOffset && uint32(cv) == load3232(src, candidate.offset-e.cur) {
+				break
+			}
+			cv = now
+		}
+
+		// A 4-byte match has been found. We'll later see if more than 4 bytes
+		// match. But, prior to the match, src[nextEmit:s] are unmatched. Emit
+		// them as literal bytes.
+
+		// Call emitCopy, and then see if another emitCopy could be our next
+		// move. Repeat until we find no match for the input immediately after
+		// what was consumed by the last emitCopy call.
+		//
+		// If we exit this loop normally then we need to call emitLiteral next,
+		// though we don't yet know how big the literal will be. We handle that
+		// by proceeding to the next iteration of the main loop. We also can
+		// exit this loop via goto if we get close to exhausting the input.
+		for {
+			// Invariant: we have a 4-byte match at s, and no need to emit any
+			// literal bytes prior to s.
+
+			// Extend the 4-byte match as long as possible.
+			t := candidate.offset - e.cur
+			l := e.matchlenLong(int(s+4), int(t+4), src) + 4
+
+			// Extend backwards
+			for t > 0 && s > nextEmit && src[t-1] == src[s-1] {
+				s--
+				t--
+				l++
+			}
+			if nextEmit < s {
+				if false {
+					emitLiteral(dst, src[nextEmit:s])
+				} else {
+					for _, v := range src[nextEmit:s] {
+						dst.tokens[dst.n] = token(v)
+						dst.litHist[v]++
+						dst.n++
+					}
+				}
+			}
+
+			dst.AddMatchLong(l, uint32(s-t-baseMatchOffset))
+			s += l
+			nextEmit = s
+			if nextS >= s {
+				s = nextS + 1
+			}
+
+			if s >= sLimit {
+				// Index first pair after match end.
+				if int(s+l+8) < len(src) {
+					cv := load6432(src, s)
+					e.table[hashLen(cv, bTableBits, hashBytes)] = tableEntry{offset: s + e.cur}
+				}
+				goto emitRemainder
+			}
+
+			// Store every second hash in-between, but offset by 1.
+			for i := s - l + 2; i < s-5; i += 7 {
+				x := load6432(src, i)
+				nextHash := hashLen(x, bTableBits, hashBytes)
+				e.table[nextHash] = tableEntry{offset: e.cur + i}
+				// Skip one
+				x >>= 16
+				nextHash = hashLen(x, bTableBits, hashBytes)
+				e.table[nextHash] = tableEntry{offset: e.cur + i + 2}
+				// Skip one
+				x >>= 16
+				nextHash = hashLen(x, bTableBits, hashBytes)
+				e.table[nextHash] = tableEntry{offset: e.cur + i + 4}
+			}
+
+			// We could immediately start working at s now, but to improve
+			// compression we first update the hash table at s-2 to s. If
+			// another emitCopy is not our next move, also calculate nextHash
+			// at s+1. At least on GOARCH=amd64, these three hash calculations
+			// are faster as one load64 call (with some shifts) instead of
+			// three load32 calls.
+			x := load6432(src, s-2)
+			o := e.cur + s - 2
+			prevHash := hashLen(x, bTableBits, hashBytes)
+			prevHash2 := hashLen(x>>8, bTableBits, hashBytes)
+			e.table[prevHash] = tableEntry{offset: o}
+			e.table[prevHash2] = tableEntry{offset: o + 1}
+			currHash := hashLen(x>>16, bTableBits, hashBytes)
+			candidate = e.table[currHash]
+			e.table[currHash] = tableEntry{offset: o + 2}
+
+			offset := s - (candidate.offset - e.cur)
+			if offset > maxMatchOffset || uint32(x>>16) != load3232(src, candidate.offset-e.cur) {
+				cv = x >> 24
+				s++
+				break
+			}
+		}
+	}
+
+emitRemainder:
+	if int(nextEmit) < len(src) {
+		// If nothing was added, don't encode literals.
+		if dst.n == 0 {
+			return
+		}
+
+		emitLiteral(dst, src[nextEmit:])
+	}
+}

vendor/github.com/klauspost/compress/flate/level3.go

@@ -0,0 +1,241 @@
+package flate
+
+import "fmt"
+
+// fastEncL3
+type fastEncL3 struct {
+	fastGen
+	table [1 << 16]tableEntryPrev
+}
+
+// Encode uses a similar algorithm to level 2, will check up to two candidates.
+func (e *fastEncL3) Encode(dst *tokens, src []byte) {
+	const (
+		inputMargin            = 12 - 1
+		minNonLiteralBlockSize = 1 + 1 + inputMargin
+		tableBits              = 16
+		tableSize              = 1 << tableBits
+		hashBytes              = 5
+	)
+
+	if debugDeflate && e.cur < 0 {
+		panic(fmt.Sprint("e.cur < 0: ", e.cur))
+	}
+
+	// Protect against e.cur wraparound.
+	for e.cur >= bufferReset {
+		if len(e.hist) == 0 {
+			for i := range e.table[:] {
+				e.table[i] = tableEntryPrev{}
+			}
+			e.cur = maxMatchOffset
+			break
+		}
+		// Shift down everything in the table that isn't already too far away.
+		minOff := e.cur + int32(len(e.hist)) - maxMatchOffset
+		for i := range e.table[:] {
+			v := e.table[i]
+			if v.Cur.offset <= minOff {
+				v.Cur.offset = 0
+			} else {
+				v.Cur.offset = v.Cur.offset - e.cur + maxMatchOffset
+			}
+			if v.Prev.offset <= minOff {
+				v.Prev.offset = 0
+			} else {
+				v.Prev.offset = v.Prev.offset - e.cur + maxMatchOffset
+			}
+			e.table[i] = v
+		}
+		e.cur = maxMatchOffset
+	}
+
+	s := e.addBlock(src)
+
+	// Skip if too small.
+	if len(src) < minNonLiteralBlockSize {
+		// We do not fill the token table.
+		// This will be picked up by caller.
+		dst.n = uint16(len(src))
+		return
+	}
+
+	// Override src
+	src = e.hist
+	nextEmit := s
+
+	// sLimit is when to stop looking for offset/length copies. The inputMargin
+	// lets us use a fast path for emitLiteral in the main loop, while we are
+	// looking for copies.
+	sLimit := int32(len(src) - inputMargin)
+
+	// nextEmit is where in src the next emitLiteral should start from.
+	cv := load6432(src, s)
+	for {
+		const skipLog = 7
+		nextS := s
+		var candidate tableEntry
+		for {
+			nextHash := hashLen(cv, tableBits, hashBytes)
+			s = nextS
+			nextS = s + 1 + (s-nextEmit)>>skipLog
+			if nextS > sLimit {
+				goto emitRemainder
+			}
+			candidates := e.table[nextHash]
+			now := load6432(src, nextS)
+
+			// Safe offset distance until s + 4...
+			minOffset := e.cur + s - (maxMatchOffset - 4)
+			e.table[nextHash] = tableEntryPrev{Prev: candidates.Cur, Cur: tableEntry{offset: s + e.cur}}
+
+			// Check both candidates
+			candidate = candidates.Cur
+			if candidate.offset < minOffset {
+				cv = now
+				// Previous will also be invalid, we have nothing.
+				continue
+			}
+
+			if uint32(cv) == load3232(src, candidate.offset-e.cur) {
+				if candidates.Prev.offset < minOffset || uint32(cv) != load3232(src, candidates.Prev.offset-e.cur) {
+					break
+				}
+				// Both match and are valid, pick longest.
+				offset := s - (candidate.offset - e.cur)
+				o2 := s - (candidates.Prev.offset - e.cur)
+				l1, l2 := matchLen(src[s+4:], src[s-offset+4:]), matchLen(src[s+4:], src[s-o2+4:])
+				if l2 > l1 {
+					candidate = candidates.Prev
+				}
+				break
+			} else {
+				// We only check if value mismatches.
+				// Offset will always be invalid in other cases.
+				candidate = candidates.Prev
+				if candidate.offset > minOffset && uint32(cv) == load3232(src, candidate.offset-e.cur) {
+					break
+				}
+			}
+			cv = now
+		}
+
+		// Call emitCopy, and then see if another emitCopy could be our next
+		// move. Repeat until we find no match for the input immediately after
+		// what was consumed by the last emitCopy call.
+		//
+		// If we exit this loop normally then we need to call emitLiteral next,
+		// though we don't yet know how big the literal will be. We handle that
+		// by proceeding to the next iteration of the main loop. We also can
+		// exit this loop via goto if we get close to exhausting the input.
+		for {
+			// Invariant: we have a 4-byte match at s, and no need to emit any
+			// literal bytes prior to s.
+
+			// Extend the 4-byte match as long as possible.
+			//
+			t := candidate.offset - e.cur
+			l := e.matchlenLong(int(s+4), int(t+4), src) + 4
+
+			// Extend backwards
+			for t > 0 && s > nextEmit && src[t-1] == src[s-1] {
+				s--
+				t--
+				l++
+			}
+			if nextEmit < s {
+				if false {
+					emitLiteral(dst, src[nextEmit:s])
+				} else {
+					for _, v := range src[nextEmit:s] {
+						dst.tokens[dst.n] = token(v)
+						dst.litHist[v]++
+						dst.n++
+					}
+				}
+			}
+
+			dst.AddMatchLong(l, uint32(s-t-baseMatchOffset))
+			s += l
+			nextEmit = s
+			if nextS >= s {
+				s = nextS + 1
+			}
+
+			if s >= sLimit {
+				t += l
+				// Index first pair after match end.
+				if int(t+8) < len(src) && t > 0 {
+					cv = load6432(src, t)
+					nextHash := hashLen(cv, tableBits, hashBytes)
+					e.table[nextHash] = tableEntryPrev{
+						Prev: e.table[nextHash].Cur,
+						Cur:  tableEntry{offset: e.cur + t},
+					}
+				}
+				goto emitRemainder
+			}
+
+			// Store every 5th hash in-between.
+			for i := s - l + 2; i < s-5; i += 6 {
+				nextHash := hashLen(load6432(src, i), tableBits, hashBytes)
+				e.table[nextHash] = tableEntryPrev{
+					Prev: e.table[nextHash].Cur,
+					Cur:  tableEntry{offset: e.cur + i}}
+			}
+			// We could immediately start working at s now, but to improve
+			// compression we first update the hash table at s-2 to s.
+			x := load6432(src, s-2)
+			prevHash := hashLen(x, tableBits, hashBytes)
+
+			e.table[prevHash] = tableEntryPrev{
+				Prev: e.table[prevHash].Cur,
+				Cur:  tableEntry{offset: e.cur + s - 2},
+			}
+			x >>= 8
+			prevHash = hashLen(x, tableBits, hashBytes)
+
+			e.table[prevHash] = tableEntryPrev{
+				Prev: e.table[prevHash].Cur,
+				Cur:  tableEntry{offset: e.cur + s - 1},
+			}
+			x >>= 8
+			currHash := hashLen(x, tableBits, hashBytes)
+			candidates := e.table[currHash]
+			cv = x
+			e.table[currHash] = tableEntryPrev{
+				Prev: candidates.Cur,
+				Cur:  tableEntry{offset: s + e.cur},
+			}
+
+			// Check both candidates
+			candidate = candidates.Cur
+			minOffset := e.cur + s - (maxMatchOffset - 4)
+
+			if candidate.offset > minOffset {
+				if uint32(cv) == load3232(src, candidate.offset-e.cur) {
+					// Found a match...
+					continue
+				}
+				candidate = candidates.Prev
+				if candidate.offset > minOffset && uint32(cv) == load3232(src, candidate.offset-e.cur) {
+					// Match at prev...
+					continue
+				}
+			}
+			cv = x >> 8
+			s++
+			break
+		}
+	}
+
+emitRemainder:
+	if int(nextEmit) < len(src) {
+		// If nothing was added, don't encode literals.
+		if dst.n == 0 {
+			return
+		}
+
+		emitLiteral(dst, src[nextEmit:])
+	}
+}

vendor/github.com/klauspost/compress/flate/level4.go

@@ -0,0 +1,221 @@
+package flate
+
+import "fmt"
+
+type fastEncL4 struct {
+	fastGen
+	table  [tableSize]tableEntry
+	bTable [tableSize]tableEntry
+}
+
+func (e *fastEncL4) Encode(dst *tokens, src []byte) {
+	const (
+		inputMargin            = 12 - 1
+		minNonLiteralBlockSize = 1 + 1 + inputMargin
+		hashShortBytes         = 4
+	)
+	if debugDeflate && e.cur < 0 {
+		panic(fmt.Sprint("e.cur < 0: ", e.cur))
+	}
+	// Protect against e.cur wraparound.
+	for e.cur >= bufferReset {
+		if len(e.hist) == 0 {
+			for i := range e.table[:] {
+				e.table[i] = tableEntry{}
+			}
+			for i := range e.bTable[:] {
+				e.bTable[i] = tableEntry{}
+			}
+			e.cur = maxMatchOffset
+			break
+		}
+		// Shift down everything in the table that isn't already too far away.
+		minOff := e.cur + int32(len(e.hist)) - maxMatchOffset
+		for i := range e.table[:] {
+			v := e.table[i].offset
+			if v <= minOff {
+				v = 0
+			} else {
+				v = v - e.cur + maxMatchOffset
+			}
+			e.table[i].offset = v
+		}
+		for i := range e.bTable[:] {
+			v := e.bTable[i].offset
+			if v <= minOff {
+				v = 0
+			} else {
+				v = v - e.cur + maxMatchOffset
+			}
+			e.bTable[i].offset = v
+		}
+		e.cur = maxMatchOffset
+	}
+
+	s := e.addBlock(src)
+
+	// This check isn't in the Snappy implementation, but there, the caller
+	// instead of the callee handles this case.
+	if len(src) < minNonLiteralBlockSize {
+		// We do not fill the token table.
+		// This will be picked up by caller.
+		dst.n = uint16(len(src))
+		return
+	}
+
+	// Override src
+	src = e.hist
+	nextEmit := s
+
+	// sLimit is when to stop looking for offset/length copies. The inputMargin
+	// lets us use a fast path for emitLiteral in the main loop, while we are
+	// looking for copies.
+	sLimit := int32(len(src) - inputMargin)
+
+	// nextEmit is where in src the next emitLiteral should start from.
+	cv := load6432(src, s)
+	for {
+		const skipLog = 6
+		const doEvery = 1
+
+		nextS := s
+		var t int32
+		for {
+			nextHashS := hashLen(cv, tableBits, hashShortBytes)
+			nextHashL := hash7(cv, tableBits)
+
+			s = nextS
+			nextS = s + doEvery + (s-nextEmit)>>skipLog
+			if nextS > sLimit {
+				goto emitRemainder
+			}
+			// Fetch a short+long candidate
+			sCandidate := e.table[nextHashS]
+			lCandidate := e.bTable[nextHashL]
+			next := load6432(src, nextS)
+			entry := tableEntry{offset: s + e.cur}
+			e.table[nextHashS] = entry
+			e.bTable[nextHashL] = entry
+
+			t = lCandidate.offset - e.cur
+			if s-t < maxMatchOffset && uint32(cv) == load3232(src, t) {
+				// We got a long match. Use that.
+				break
+			}
+
+			t = sCandidate.offset - e.cur
+			if s-t < maxMatchOffset && uint32(cv) == load3232(src, t) {
+				// Found a 4 match...
+				lCandidate = e.bTable[hash7(next, tableBits)]
+
+				// If the next long is a candidate, check if we should use that instead...
+				lOff := lCandidate.offset - e.cur
+				if nextS-lOff < maxMatchOffset && load3232(src, lOff) == uint32(next) {
+					l1, l2 := matchLen(src[s+4:], src[t+4:]), matchLen(src[nextS+4:], src[nextS-lOff+4:])
+					if l2 > l1 {
+						s = nextS
+						t = lCandidate.offset - e.cur
+					}
+				}
+				break
+			}
+			cv = next
+		}
+
+		// A 4-byte match has been found. We'll later see if more than 4 bytes
+		// match. But, prior to the match, src[nextEmit:s] are unmatched. Emit
+		// them as literal bytes.
+
+		// Extend the 4-byte match as long as possible.
+		l := e.matchlenLong(int(s+4), int(t+4), src) + 4
+
+		// Extend backwards
+		for t > 0 && s > nextEmit && src[t-1] == src[s-1] {
+			s--
+			t--
+			l++
+		}
+		if nextEmit < s {
+			if false {
+				emitLiteral(dst, src[nextEmit:s])
+			} else {
+				for _, v := range src[nextEmit:s] {
+					dst.tokens[dst.n] = token(v)
+					dst.litHist[v]++
+					dst.n++
+				}
+			}
+		}
+		if debugDeflate {
+			if t >= s {
+				panic("s-t")
+			}
+			if (s - t) > maxMatchOffset {
+				panic(fmt.Sprintln("mmo", t))
+			}
+			if l < baseMatchLength {
+				panic("bml")
+			}
+		}
+
+		dst.AddMatchLong(l, uint32(s-t-baseMatchOffset))
+		s += l
+		nextEmit = s
+		if nextS >= s {
+			s = nextS + 1
+		}
+
+		if s >= sLimit {
+			// Index first pair after match end.
+			if int(s+8) < len(src) {
+				cv := load6432(src, s)
+				e.table[hashLen(cv, tableBits, hashShortBytes)] = tableEntry{offset: s + e.cur}
+				e.bTable[hash7(cv, tableBits)] = tableEntry{offset: s + e.cur}
+			}
+			goto emitRemainder
+		}
+
+		// Store every 3rd hash in-between
+		if true {
+			i := nextS
+			if i < s-1 {
+				cv := load6432(src, i)
+				t := tableEntry{offset: i + e.cur}
+				t2 := tableEntry{offset: t.offset + 1}
+				e.bTable[hash7(cv, tableBits)] = t
+				e.bTable[hash7(cv>>8, tableBits)] = t2
+				e.table[hashLen(cv>>8, tableBits, hashShortBytes)] = t2
+
+				i += 3
+				for ; i < s-1; i += 3 {
+					cv := load6432(src, i)
+					t := tableEntry{offset: i + e.cur}
+					t2 := tableEntry{offset: t.offset + 1}
+					e.bTable[hash7(cv, tableBits)] = t
+					e.bTable[hash7(cv>>8, tableBits)] = t2
+					e.table[hashLen(cv>>8, tableBits, hashShortBytes)] = t2
+				}
+			}
+		}
+
+		// We could immediately start working at s now, but to improve
+		// compression we first update the hash table at s-1 and at s.
+		x := load6432(src, s-1)
+		o := e.cur + s - 1
+		prevHashS := hashLen(x, tableBits, hashShortBytes)
+		prevHashL := hash7(x, tableBits)
+		e.table[prevHashS] = tableEntry{offset: o}
+		e.bTable[prevHashL] = tableEntry{offset: o}
+		cv = x >> 8
+	}
+
+emitRemainder:
+	if int(nextEmit) < len(src) {
+		// If nothing was added, don't encode literals.
+		if dst.n == 0 {
+			return
+		}
+
+		emitLiteral(dst, src[nextEmit:])
+	}
+}

vendor/github.com/klauspost/compress/flate/level5.go

@@ -0,0 +1,705 @@
+package flate
+
+import "fmt"
+
+type fastEncL5 struct {
+	fastGen
+	table  [tableSize]tableEntry
+	bTable [tableSize]tableEntryPrev
+}
+
+func (e *fastEncL5) Encode(dst *tokens, src []byte) {
+	const (
+		inputMargin            = 12 - 1
+		minNonLiteralBlockSize = 1 + 1 + inputMargin
+		hashShortBytes         = 4
+	)
+	if debugDeflate && e.cur < 0 {
+		panic(fmt.Sprint("e.cur < 0: ", e.cur))
+	}
+
+	// Protect against e.cur wraparound.
+	for e.cur >= bufferReset {
+		if len(e.hist) == 0 {
+			for i := range e.table[:] {
+				e.table[i] = tableEntry{}
+			}
+			for i := range e.bTable[:] {
+				e.bTable[i] = tableEntryPrev{}
+			}
+			e.cur = maxMatchOffset
+			break
+		}
+		// Shift down everything in the table that isn't already too far away.
+		minOff := e.cur + int32(len(e.hist)) - maxMatchOffset
+		for i := range e.table[:] {
+			v := e.table[i].offset
+			if v <= minOff {
+				v = 0
+			} else {
+				v = v - e.cur + maxMatchOffset
+			}
+			e.table[i].offset = v
+		}
+		for i := range e.bTable[:] {
+			v := e.bTable[i]
+			if v.Cur.offset <= minOff {
+				v.Cur.offset = 0
+				v.Prev.offset = 0
+			} else {
+				v.Cur.offset = v.Cur.offset - e.cur + maxMatchOffset
+				if v.Prev.offset <= minOff {
+					v.Prev.offset = 0
+				} else {
+					v.Prev.offset = v.Prev.offset - e.cur + maxMatchOffset
+				}
+			}
+			e.bTable[i] = v
+		}
+		e.cur = maxMatchOffset
+	}
+
+	s := e.addBlock(src)
+
+	// This check isn't in the Snappy implementation, but there, the caller
+	// instead of the callee handles this case.
+	if len(src) < minNonLiteralBlockSize {
+		// We do not fill the token table.
+		// This will be picked up by caller.
+		dst.n = uint16(len(src))
+		return
+	}
+
+	// Override src
+	src = e.hist
+	nextEmit := s
+
+	// sLimit is when to stop looking for offset/length copies. The inputMargin
+	// lets us use a fast path for emitLiteral in the main loop, while we are
+	// looking for copies.
+	sLimit := int32(len(src) - inputMargin)
+
+	// nextEmit is where in src the next emitLiteral should start from.
+	cv := load6432(src, s)
+	for {
+		const skipLog = 6
+		const doEvery = 1
+
+		nextS := s
+		var l int32
+		var t int32
+		for {
+			nextHashS := hashLen(cv, tableBits, hashShortBytes)
+			nextHashL := hash7(cv, tableBits)
+
+			s = nextS
+			nextS = s + doEvery + (s-nextEmit)>>skipLog
+			if nextS > sLimit {
+				goto emitRemainder
+			}
+			// Fetch a short+long candidate
+			sCandidate := e.table[nextHashS]
+			lCandidate := e.bTable[nextHashL]
+			next := load6432(src, nextS)
+			entry := tableEntry{offset: s + e.cur}
+			e.table[nextHashS] = entry
+			eLong := &e.bTable[nextHashL]
+			eLong.Cur, eLong.Prev = entry, eLong.Cur
+
+			nextHashS = hashLen(next, tableBits, hashShortBytes)
+			nextHashL = hash7(next, tableBits)
+
+			t = lCandidate.Cur.offset - e.cur
+			if s-t < maxMatchOffset {
+				if uint32(cv) == load3232(src, t) {
+					// Store the next match
+					e.table[nextHashS] = tableEntry{offset: nextS + e.cur}
+					eLong := &e.bTable[nextHashL]
+					eLong.Cur, eLong.Prev = tableEntry{offset: nextS + e.cur}, eLong.Cur
+
+					t2 := lCandidate.Prev.offset - e.cur
+					if s-t2 < maxMatchOffset && uint32(cv) == load3232(src, t2) {
+						l = e.matchlen(int(s+4), int(t+4), src) + 4
+						ml1 := e.matchlen(int(s+4), int(t2+4), src) + 4
+						if ml1 > l {
+							t = t2
+							l = ml1
+							break
+						}
+					}
+					break
+				}
+				t = lCandidate.Prev.offset - e.cur
+				if s-t < maxMatchOffset && uint32(cv) == load3232(src, t) {
+					// Store the next match
+					e.table[nextHashS] = tableEntry{offset: nextS + e.cur}
+					eLong := &e.bTable[nextHashL]
+					eLong.Cur, eLong.Prev = tableEntry{offset: nextS + e.cur}, eLong.Cur
+					break
+				}
+			}
+
+			t = sCandidate.offset - e.cur
+			if s-t < maxMatchOffset && uint32(cv) == load3232(src, t) {
+				// Found a 4 match...
+				l = e.matchlen(int(s+4), int(t+4), src) + 4
+				lCandidate = e.bTable[nextHashL]
+				// Store the next match
+
+				e.table[nextHashS] = tableEntry{offset: nextS + e.cur}
+				eLong := &e.bTable[nextHashL]
+				eLong.Cur, eLong.Prev = tableEntry{offset: nextS + e.cur}, eLong.Cur
+
+				// If the next long is a candidate, use that...
+				t2 := lCandidate.Cur.offset - e.cur
+				if nextS-t2 < maxMatchOffset {
+					if load3232(src, t2) == uint32(next) {
+						ml := e.matchlen(int(nextS+4), int(t2+4), src) + 4
+						if ml > l {
+							t = t2
+							s = nextS
+							l = ml
+							break
+						}
+					}
+					// If the previous long is a candidate, use that...
+					t2 = lCandidate.Prev.offset - e.cur
+					if nextS-t2 < maxMatchOffset && load3232(src, t2) == uint32(next) {
+						ml := e.matchlen(int(nextS+4), int(t2+4), src) + 4
+						if ml > l {
+							t = t2
+							s = nextS
+							l = ml
+							break
+						}
+					}
+				}
+				break
+			}
+			cv = next
+		}
+
+		// A 4-byte match has been found. We'll later see if more than 4 bytes
+		// match. But, prior to the match, src[nextEmit:s] are unmatched. Emit
+		// them as literal bytes.
+
+		if l == 0 {
+			// Extend the 4-byte match as long as possible.
+			l = e.matchlenLong(int(s+4), int(t+4), src) + 4
+		} else if l == maxMatchLength {
+			l += e.matchlenLong(int(s+l), int(t+l), src)
+		}
+
+		// Try to locate a better match by checking the end of best match...
+		if sAt := s + l; l < 30 && sAt < sLimit {
+			// Allow some bytes at the beginning to mismatch.
+			// Sweet spot is 2/3 bytes depending on input.
+			// 3 is only a little better when it is but sometimes a lot worse.
+			// The skipped bytes are tested in Extend backwards,
+			// and still picked up as part of the match if they do.
+			const skipBeginning = 2
+			eLong := e.bTable[hash7(load6432(src, sAt), tableBits)].Cur.offset
+			t2 := eLong - e.cur - l + skipBeginning
+			s2 := s + skipBeginning
+			off := s2 - t2
+			if t2 >= 0 && off < maxMatchOffset && off > 0 {
+				if l2 := e.matchlenLong(int(s2), int(t2), src); l2 > l {
+					t = t2
+					l = l2
+					s = s2
+				}
+			}
+		}
+
+		// Extend backwards
+		for t > 0 && s > nextEmit && src[t-1] == src[s-1] {
+			s--
+			t--
+			l++
+		}
+		if nextEmit < s {
+			if false {
+				emitLiteral(dst, src[nextEmit:s])
+			} else {
+				for _, v := range src[nextEmit:s] {
+					dst.tokens[dst.n] = token(v)
+					dst.litHist[v]++
+					dst.n++
+				}
+			}
+		}
+		if debugDeflate {
+			if t >= s {
+				panic(fmt.Sprintln("s-t", s, t))
+			}
+			if (s - t) > maxMatchOffset {
+				panic(fmt.Sprintln("mmo", s-t))
+			}
+			if l < baseMatchLength {
+				panic("bml")
+			}
+		}
+
+		dst.AddMatchLong(l, uint32(s-t-baseMatchOffset))
+		s += l
+		nextEmit = s
+		if nextS >= s {
+			s = nextS + 1
+		}
+
+		if s >= sLimit {
+			goto emitRemainder
+		}
+
+		// Store every 3rd hash in-between.
+		if true {
+			const hashEvery = 3
+			i := s - l + 1
+			if i < s-1 {
+				cv := load6432(src, i)
+				t := tableEntry{offset: i + e.cur}
+				e.table[hashLen(cv, tableBits, hashShortBytes)] = t
+				eLong := &e.bTable[hash7(cv, tableBits)]
+				eLong.Cur, eLong.Prev = t, eLong.Cur
+
+				// Do an long at i+1
+				cv >>= 8
+				t = tableEntry{offset: t.offset + 1}
+				eLong = &e.bTable[hash7(cv, tableBits)]
+				eLong.Cur, eLong.Prev = t, eLong.Cur
+
+				// We only have enough bits for a short entry at i+2
+				cv >>= 8
+				t = tableEntry{offset: t.offset + 1}
+				e.table[hashLen(cv, tableBits, hashShortBytes)] = t
+
+				// Skip one - otherwise we risk hitting 's'
+				i += 4
+				for ; i < s-1; i += hashEvery {
+					cv := load6432(src, i)
+					t := tableEntry{offset: i + e.cur}
+					t2 := tableEntry{offset: t.offset + 1}
+					eLong := &e.bTable[hash7(cv, tableBits)]
+					eLong.Cur, eLong.Prev = t, eLong.Cur
+					e.table[hashLen(cv>>8, tableBits, hashShortBytes)] = t2
+				}
+			}
+		}
+
+		// We could immediately start working at s now, but to improve
+		// compression we first update the hash table at s-1 and at s.
+		x := load6432(src, s-1)
+		o := e.cur + s - 1
+		prevHashS := hashLen(x, tableBits, hashShortBytes)
+		prevHashL := hash7(x, tableBits)
+		e.table[prevHashS] = tableEntry{offset: o}
+		eLong := &e.bTable[prevHashL]
+		eLong.Cur, eLong.Prev = tableEntry{offset: o}, eLong.Cur
+		cv = x >> 8
+	}
+
+emitRemainder:
+	if int(nextEmit) < len(src) {
+		// If nothing was added, don't encode literals.
+		if dst.n == 0 {
+			return
+		}
+
+		emitLiteral(dst, src[nextEmit:])
+	}
+}
+
+// fastEncL5Window is a level 5 encoder,
+// but with a custom window size.
+type fastEncL5Window struct {
+	hist      []byte
+	cur       int32
+	maxOffset int32
+	table     [tableSize]tableEntry
+	bTable    [tableSize]tableEntryPrev
+}
+
+func (e *fastEncL5Window) Encode(dst *tokens, src []byte) {
+	const (
+		inputMargin            = 12 - 1
+		minNonLiteralBlockSize = 1 + 1 + inputMargin
+		hashShortBytes         = 4
+	)
+	maxMatchOffset := e.maxOffset
+	if debugDeflate && e.cur < 0 {
+		panic(fmt.Sprint("e.cur < 0: ", e.cur))
+	}
+
+	// Protect against e.cur wraparound.
+	for e.cur >= bufferReset {
+		if len(e.hist) == 0 {
+			for i := range e.table[:] {
+				e.table[i] = tableEntry{}
+			}
+			for i := range e.bTable[:] {
+				e.bTable[i] = tableEntryPrev{}
+			}
+			e.cur = maxMatchOffset
+			break
+		}
+		// Shift down everything in the table that isn't already too far away.
+		minOff := e.cur + int32(len(e.hist)) - maxMatchOffset
+		for i := range e.table[:] {
+			v := e.table[i].offset
+			if v <= minOff {
+				v = 0
+			} else {
+				v = v - e.cur + maxMatchOffset
+			}
+			e.table[i].offset = v
+		}
+		for i := range e.bTable[:] {
+			v := e.bTable[i]
+			if v.Cur.offset <= minOff {
+				v.Cur.offset = 0
+				v.Prev.offset = 0
+			} else {
+				v.Cur.offset = v.Cur.offset - e.cur + maxMatchOffset
+				if v.Prev.offset <= minOff {
+					v.Prev.offset = 0
+				} else {
+					v.Prev.offset = v.Prev.offset - e.cur + maxMatchOffset
+				}
+			}
+			e.bTable[i] = v
+		}
+		e.cur = maxMatchOffset
+	}
+
+	s := e.addBlock(src)
+
+	// This check isn't in the Snappy implementation, but there, the caller
+	// instead of the callee handles this case.
+	if len(src) < minNonLiteralBlockSize {
+		// We do not fill the token table.
+		// This will be picked up by caller.
+		dst.n = uint16(len(src))
+		return
+	}
+
+	// Override src
+	src = e.hist
+	nextEmit := s
+
+	// sLimit is when to stop looking for offset/length copies. The inputMargin
+	// lets us use a fast path for emitLiteral in the main loop, while we are
+	// looking for copies.
+	sLimit := int32(len(src) - inputMargin)
+
+	// nextEmit is where in src the next emitLiteral should start from.
+	cv := load6432(src, s)
+	for {
+		const skipLog = 6
+		const doEvery = 1
+
+		nextS := s
+		var l int32
+		var t int32
+		for {
+			nextHashS := hashLen(cv, tableBits, hashShortBytes)
+			nextHashL := hash7(cv, tableBits)
+
+			s = nextS
+			nextS = s + doEvery + (s-nextEmit)>>skipLog
+			if nextS > sLimit {
+				goto emitRemainder
+			}
+			// Fetch a short+long candidate
+			sCandidate := e.table[nextHashS]
+			lCandidate := e.bTable[nextHashL]
+			next := load6432(src, nextS)
+			entry := tableEntry{offset: s + e.cur}
+			e.table[nextHashS] = entry
+			eLong := &e.bTable[nextHashL]
+			eLong.Cur, eLong.Prev = entry, eLong.Cur
+
+			nextHashS = hashLen(next, tableBits, hashShortBytes)
+			nextHashL = hash7(next, tableBits)
+
+			t = lCandidate.Cur.offset - e.cur
+			if s-t < maxMatchOffset {
+				if uint32(cv) == load3232(src, t) {
+					// Store the next match
+					e.table[nextHashS] = tableEntry{offset: nextS + e.cur}
+					eLong := &e.bTable[nextHashL]
+					eLong.Cur, eLong.Prev = tableEntry{offset: nextS + e.cur}, eLong.Cur
+
+					t2 := lCandidate.Prev.offset - e.cur
+					if s-t2 < maxMatchOffset && uint32(cv) == load3232(src, t2) {
+						l = e.matchlen(s+4, t+4, src) + 4
+						ml1 := e.matchlen(s+4, t2+4, src) + 4
+						if ml1 > l {
+							t = t2
+							l = ml1
+							break
+						}
+					}
+					break
+				}
+				t = lCandidate.Prev.offset - e.cur
+				if s-t < maxMatchOffset && uint32(cv) == load3232(src, t) {
+					// Store the next match
+					e.table[nextHashS] = tableEntry{offset: nextS + e.cur}
+					eLong := &e.bTable[nextHashL]
+					eLong.Cur, eLong.Prev = tableEntry{offset: nextS + e.cur}, eLong.Cur
+					break
+				}
+			}
+
+			t = sCandidate.offset - e.cur
+			if s-t < maxMatchOffset && uint32(cv) == load3232(src, t) {
+				// Found a 4 match...
+				l = e.matchlen(s+4, t+4, src) + 4
+				lCandidate = e.bTable[nextHashL]
+				// Store the next match
+
+				e.table[nextHashS] = tableEntry{offset: nextS + e.cur}
+				eLong := &e.bTable[nextHashL]
+				eLong.Cur, eLong.Prev = tableEntry{offset: nextS + e.cur}, eLong.Cur
+
+				// If the next long is a candidate, use that...
+				t2 := lCandidate.Cur.offset - e.cur
+				if nextS-t2 < maxMatchOffset {
+					if load3232(src, t2) == uint32(next) {
+						ml := e.matchlen(nextS+4, t2+4, src) + 4
+						if ml > l {
+							t = t2
+							s = nextS
+							l = ml
+							break
+						}
+					}
+					// If the previous long is a candidate, use that...
+					t2 = lCandidate.Prev.offset - e.cur
+					if nextS-t2 < maxMatchOffset && load3232(src, t2) == uint32(next) {
+						ml := e.matchlen(nextS+4, t2+4, src) + 4
+						if ml > l {
+							t = t2
+							s = nextS
+							l = ml
+							break
+						}
+					}
+				}
+				break
+			}
+			cv = next
+		}
+
+		// A 4-byte match has been found. We'll later see if more than 4 bytes
+		// match. But, prior to the match, src[nextEmit:s] are unmatched. Emit
+		// them as literal bytes.
+
+		if l == 0 {
+			// Extend the 4-byte match as long as possible.
+			l = e.matchlenLong(s+4, t+4, src) + 4
+		} else if l == maxMatchLength {
+			l += e.matchlenLong(s+l, t+l, src)
+		}
+
+		// Try to locate a better match by checking the end of best match...
+		if sAt := s + l; l < 30 && sAt < sLimit {
+			// Allow some bytes at the beginning to mismatch.
+			// Sweet spot is 2/3 bytes depending on input.
+			// 3 is only a little better when it is but sometimes a lot worse.
+			// The skipped bytes are tested in Extend backwards,
+			// and still picked up as part of the match if they do.
+			const skipBeginning = 2
+			eLong := e.bTable[hash7(load6432(src, sAt), tableBits)].Cur.offset
+			t2 := eLong - e.cur - l + skipBeginning
+			s2 := s + skipBeginning
+			off := s2 - t2
+			if t2 >= 0 && off < maxMatchOffset && off > 0 {
+				if l2 := e.matchlenLong(s2, t2, src); l2 > l {
+					t = t2
+					l = l2
+					s = s2
+				}
+			}
+		}
+
+		// Extend backwards
+		for t > 0 && s > nextEmit && src[t-1] == src[s-1] {
+			s--
+			t--
+			l++
+		}
+		if nextEmit < s {
+			if false {
+				emitLiteral(dst, src[nextEmit:s])
+			} else {
+				for _, v := range src[nextEmit:s] {
+					dst.tokens[dst.n] = token(v)
+					dst.litHist[v]++
+					dst.n++
+				}
+			}
+		}
+		if debugDeflate {
+			if t >= s {
+				panic(fmt.Sprintln("s-t", s, t))
+			}
+			if (s - t) > maxMatchOffset {
+				panic(fmt.Sprintln("mmo", s-t))
+			}
+			if l < baseMatchLength {
+				panic("bml")
+			}
+		}
+
+		dst.AddMatchLong(l, uint32(s-t-baseMatchOffset))
+		s += l
+		nextEmit = s
+		if nextS >= s {
+			s = nextS + 1
+		}
+
+		if s >= sLimit {
+			goto emitRemainder
+		}
+
+		// Store every 3rd hash in-between.
+		if true {
+			const hashEvery = 3
+			i := s - l + 1
+			if i < s-1 {
+				cv := load6432(src, i)
+				t := tableEntry{offset: i + e.cur}
+				e.table[hashLen(cv, tableBits, hashShortBytes)] = t
+				eLong := &e.bTable[hash7(cv, tableBits)]
+				eLong.Cur, eLong.Prev = t, eLong.Cur
+
+				// Do an long at i+1
+				cv >>= 8
+				t = tableEntry{offset: t.offset + 1}
+				eLong = &e.bTable[hash7(cv, tableBits)]
+				eLong.Cur, eLong.Prev = t, eLong.Cur
+
+				// We only have enough bits for a short entry at i+2
+				cv >>= 8
+				t = tableEntry{offset: t.offset + 1}
+				e.table[hashLen(cv, tableBits, hashShortBytes)] = t
+
+				// Skip one - otherwise we risk hitting 's'
+				i += 4
+				for ; i < s-1; i += hashEvery {
+					cv := load6432(src, i)
+					t := tableEntry{offset: i + e.cur}
+					t2 := tableEntry{offset: t.offset + 1}
+					eLong := &e.bTable[hash7(cv, tableBits)]
+					eLong.Cur, eLong.Prev = t, eLong.Cur
+					e.table[hashLen(cv>>8, tableBits, hashShortBytes)] = t2
+				}
+			}
+		}
+
+		// We could immediately start working at s now, but to improve
+		// compression we first update the hash table at s-1 and at s.
+		x := load6432(src, s-1)
+		o := e.cur + s - 1
+		prevHashS := hashLen(x, tableBits, hashShortBytes)
+		prevHashL := hash7(x, tableBits)
+		e.table[prevHashS] = tableEntry{offset: o}
+		eLong := &e.bTable[prevHashL]
+		eLong.Cur, eLong.Prev = tableEntry{offset: o}, eLong.Cur
+		cv = x >> 8
+	}
+
+emitRemainder:
+	if int(nextEmit) < len(src) {
+		// If nothing was added, don't encode literals.
+		if dst.n == 0 {
+			return
+		}
+
+		emitLiteral(dst, src[nextEmit:])
+	}
+}
+
+// Reset the encoding table.
+func (e *fastEncL5Window) Reset() {
+	// We keep the same allocs, since we are compressing the same block sizes.
+	if cap(e.hist) < allocHistory {
+		e.hist = make([]byte, 0, allocHistory)
+	}
+
+	// We offset current position so everything will be out of reach.
+	// If we are above the buffer reset it will be cleared anyway since len(hist) == 0.
+	if e.cur <= int32(bufferReset) {
+		e.cur += e.maxOffset + int32(len(e.hist))
+	}
+	e.hist = e.hist[:0]
+}
+
+func (e *fastEncL5Window) addBlock(src []byte) int32 {
+	// check if we have space already
+	maxMatchOffset := e.maxOffset
+
+	if len(e.hist)+len(src) > cap(e.hist) {
+		if cap(e.hist) == 0 {
+			e.hist = make([]byte, 0, allocHistory)
+		} else {
+			if cap(e.hist) < int(maxMatchOffset*2) {
+				panic("unexpected buffer size")
+			}
+			// Move down
+			offset := int32(len(e.hist)) - maxMatchOffset
+			copy(e.hist[0:maxMatchOffset], e.hist[offset:])
+			e.cur += offset
+			e.hist = e.hist[:maxMatchOffset]
+		}
+	}
+	s := int32(len(e.hist))
+	e.hist = append(e.hist, src...)
+	return s
+}
+
+// matchlen will return the match length between offsets and t in src.
+// The maximum length returned is maxMatchLength - 4.
+// It is assumed that s > t, that t >=0 and s < len(src).
+func (e *fastEncL5Window) matchlen(s, t int32, src []byte) int32 {
+	if debugDecode {
+		if t >= s {
+			panic(fmt.Sprint("t >=s:", t, s))
+		}
+		if int(s) >= len(src) {
+			panic(fmt.Sprint("s >= len(src):", s, len(src)))
+		}
+		if t < 0 {
+			panic(fmt.Sprint("t < 0:", t))
+		}
+		if s-t > e.maxOffset {
+			panic(fmt.Sprint(s, "-", t, "(", s-t, ") > maxMatchLength (", maxMatchOffset, ")"))
+		}
+	}
+	s1 := min(int(s)+maxMatchLength-4, len(src))
+
+	// Extend the match to be as long as possible.
+	return int32(matchLen(src[s:s1], src[t:]))
+}
+
+// matchlenLong will return the match length between offsets and t in src.
+// It is assumed that s > t, that t >=0 and s < len(src).
+func (e *fastEncL5Window) matchlenLong(s, t int32, src []byte) int32 {
+	if debugDeflate {
+		if t >= s {
+			panic(fmt.Sprint("t >=s:", t, s))
+		}
+		if int(s) >= len(src) {
+			panic(fmt.Sprint("s >= len(src):", s, len(src)))
+		}
+		if t < 0 {
+			panic(fmt.Sprint("t < 0:", t))
+		}
+		if s-t > e.maxOffset {
+			panic(fmt.Sprint(s, "-", t, "(", s-t, ") > maxMatchLength (", maxMatchOffset, ")"))
+		}
+	}
+	// Extend the match to be as long as possible.
+	return int32(matchLen(src[s:], src[t:]))
+}

vendor/github.com/klauspost/compress/flate/level6.go

@@ -0,0 +1,325 @@
+package flate
+
+import "fmt"
+
+type fastEncL6 struct {
+	fastGen
+	table  [tableSize]tableEntry
+	bTable [tableSize]tableEntryPrev
+}
+
+func (e *fastEncL6) Encode(dst *tokens, src []byte) {
+	const (
+		inputMargin            = 12 - 1
+		minNonLiteralBlockSize = 1 + 1 + inputMargin
+		hashShortBytes         = 4
+	)
+	if debugDeflate && e.cur < 0 {
+		panic(fmt.Sprint("e.cur < 0: ", e.cur))
+	}
+
+	// Protect against e.cur wraparound.
+	for e.cur >= bufferReset {
+		if len(e.hist) == 0 {
+			for i := range e.table[:] {
+				e.table[i] = tableEntry{}
+			}
+			for i := range e.bTable[:] {
+				e.bTable[i] = tableEntryPrev{}
+			}
+			e.cur = maxMatchOffset
+			break
+		}
+		// Shift down everything in the table that isn't already too far away.
+		minOff := e.cur + int32(len(e.hist)) - maxMatchOffset
+		for i := range e.table[:] {
+			v := e.table[i].offset
+			if v <= minOff {
+				v = 0
+			} else {
+				v = v - e.cur + maxMatchOffset
+			}
+			e.table[i].offset = v
+		}
+		for i := range e.bTable[:] {
+			v := e.bTable[i]
+			if v.Cur.offset <= minOff {
+				v.Cur.offset = 0
+				v.Prev.offset = 0
+			} else {
+				v.Cur.offset = v.Cur.offset - e.cur + maxMatchOffset
+				if v.Prev.offset <= minOff {
+					v.Prev.offset = 0
+				} else {
+					v.Prev.offset = v.Prev.offset - e.cur + maxMatchOffset
+				}
+			}
+			e.bTable[i] = v
+		}
+		e.cur = maxMatchOffset
+	}
+
+	s := e.addBlock(src)
+
+	// This check isn't in the Snappy implementation, but there, the caller
+	// instead of the callee handles this case.
+	if len(src) < minNonLiteralBlockSize {
+		// We do not fill the token table.
+		// This will be picked up by caller.
+		dst.n = uint16(len(src))
+		return
+	}
+
+	// Override src
+	src = e.hist
+	nextEmit := s
+
+	// sLimit is when to stop looking for offset/length copies. The inputMargin
+	// lets us use a fast path for emitLiteral in the main loop, while we are
+	// looking for copies.
+	sLimit := int32(len(src) - inputMargin)
+
+	// nextEmit is where in src the next emitLiteral should start from.
+	cv := load6432(src, s)
+	// Repeat MUST be > 1 and within range
+	repeat := int32(1)
+	for {
+		const skipLog = 7
+		const doEvery = 1
+
+		nextS := s
+		var l int32
+		var t int32
+		for {
+			nextHashS := hashLen(cv, tableBits, hashShortBytes)
+			nextHashL := hash7(cv, tableBits)
+			s = nextS
+			nextS = s + doEvery + (s-nextEmit)>>skipLog
+			if nextS > sLimit {
+				goto emitRemainder
+			}
+			// Fetch a short+long candidate
+			sCandidate := e.table[nextHashS]
+			lCandidate := e.bTable[nextHashL]
+			next := load6432(src, nextS)
+			entry := tableEntry{offset: s + e.cur}
+			e.table[nextHashS] = entry
+			eLong := &e.bTable[nextHashL]
+			eLong.Cur, eLong.Prev = entry, eLong.Cur
+
+			// Calculate hashes of 'next'
+			nextHashS = hashLen(next, tableBits, hashShortBytes)
+			nextHashL = hash7(next, tableBits)
+
+			t = lCandidate.Cur.offset - e.cur
+			if s-t < maxMatchOffset {
+				if uint32(cv) == load3232(src, t) {
+					// Long candidate matches at least 4 bytes.
+
+					// Store the next match
+					e.table[nextHashS] = tableEntry{offset: nextS + e.cur}
+					eLong := &e.bTable[nextHashL]
+					eLong.Cur, eLong.Prev = tableEntry{offset: nextS + e.cur}, eLong.Cur
+
+					// Check the previous long candidate as well.
+					t2 := lCandidate.Prev.offset - e.cur
+					if s-t2 < maxMatchOffset && uint32(cv) == load3232(src, t2) {
+						l = e.matchlen(int(s+4), int(t+4), src) + 4
+						ml1 := e.matchlen(int(s+4), int(t2+4), src) + 4
+						if ml1 > l {
+							t = t2
+							l = ml1
+							break
+						}
+					}
+					break
+				}
+				// Current value did not match, but check if previous long value does.
+				t = lCandidate.Prev.offset - e.cur
+				if s-t < maxMatchOffset && uint32(cv) == load3232(src, t) {
+					// Store the next match
+					e.table[nextHashS] = tableEntry{offset: nextS + e.cur}
+					eLong := &e.bTable[nextHashL]
+					eLong.Cur, eLong.Prev = tableEntry{offset: nextS + e.cur}, eLong.Cur
+					break
+				}
+			}
+
+			t = sCandidate.offset - e.cur
+			if s-t < maxMatchOffset && uint32(cv) == load3232(src, t) {
+				// Found a 4 match...
+				l = e.matchlen(int(s+4), int(t+4), src) + 4
+
+				// Look up next long candidate (at nextS)
+				lCandidate = e.bTable[nextHashL]
+
+				// Store the next match
+				e.table[nextHashS] = tableEntry{offset: nextS + e.cur}
+				eLong := &e.bTable[nextHashL]
+				eLong.Cur, eLong.Prev = tableEntry{offset: nextS + e.cur}, eLong.Cur
+
+				// Check repeat at s + repOff
+				const repOff = 1
+				t2 := s - repeat + repOff
+				if load3232(src, t2) == uint32(cv>>(8*repOff)) {
+					ml := e.matchlen(int(s+4+repOff), int(t2+4), src) + 4
+					if ml > l {
+						t = t2
+						l = ml
+						s += repOff
+						// Not worth checking more.
+						break
+					}
+				}
+
+				// If the next long is a candidate, use that...
+				t2 = lCandidate.Cur.offset - e.cur
+				if nextS-t2 < maxMatchOffset {
+					if load3232(src, t2) == uint32(next) {
+						ml := e.matchlen(int(nextS+4), int(t2+4), src) + 4
+						if ml > l {
+							t = t2
+							s = nextS
+							l = ml
+							// This is ok, but check previous as well.
+						}
+					}
+					// If the previous long is a candidate, use that...
+					t2 = lCandidate.Prev.offset - e.cur
+					if nextS-t2 < maxMatchOffset && load3232(src, t2) == uint32(next) {
+						ml := e.matchlen(int(nextS+4), int(t2+4), src) + 4
+						if ml > l {
+							t = t2
+							s = nextS
+							l = ml
+							break
+						}
+					}
+				}
+				break
+			}
+			cv = next
+		}
+
+		// A 4-byte match has been found. We'll later see if more than 4 bytes
+		// match. But, prior to the match, src[nextEmit:s] are unmatched. Emit
+		// them as literal bytes.
+
+		// Extend the 4-byte match as long as possible.
+		if l == 0 {
+			l = e.matchlenLong(int(s+4), int(t+4), src) + 4
+		} else if l == maxMatchLength {
+			l += e.matchlenLong(int(s+l), int(t+l), src)
+		}
+
+		// Try to locate a better match by checking the end-of-match...
+		if sAt := s + l; sAt < sLimit {
+			// Allow some bytes at the beginning to mismatch.
+			// Sweet spot is 2/3 bytes depending on input.
+			// 3 is only a little better when it is but sometimes a lot worse.
+			// The skipped bytes are tested in Extend backwards,
+			// and still picked up as part of the match if they do.
+			const skipBeginning = 2
+			eLong := &e.bTable[hash7(load6432(src, sAt), tableBits)]
+			// Test current
+			t2 := eLong.Cur.offset - e.cur - l + skipBeginning
+			s2 := s + skipBeginning
+			off := s2 - t2
+			if off < maxMatchOffset {
+				if off > 0 && t2 >= 0 {
+					if l2 := e.matchlenLong(int(s2), int(t2), src); l2 > l {
+						t = t2
+						l = l2
+						s = s2
+					}
+				}
+				// Test next:
+				t2 = eLong.Prev.offset - e.cur - l + skipBeginning
+				off := s2 - t2
+				if off > 0 && off < maxMatchOffset && t2 >= 0 {
+					if l2 := e.matchlenLong(int(s2), int(t2), src); l2 > l {
+						t = t2
+						l = l2
+						s = s2
+					}
+				}
+			}
+		}
+
+		// Extend backwards
+		for t > 0 && s > nextEmit && src[t-1] == src[s-1] {
+			s--
+			t--
+			l++
+		}
+		if nextEmit < s {
+			if false {
+				emitLiteral(dst, src[nextEmit:s])
+			} else {
+				for _, v := range src[nextEmit:s] {
+					dst.tokens[dst.n] = token(v)
+					dst.litHist[v]++
+					dst.n++
+				}
+			}
+		}
+		if false {
+			if t >= s {
+				panic(fmt.Sprintln("s-t", s, t))
+			}
+			if (s - t) > maxMatchOffset {
+				panic(fmt.Sprintln("mmo", s-t))
+			}
+			if l < baseMatchLength {
+				panic("bml")
+			}
+		}
+
+		dst.AddMatchLong(l, uint32(s-t-baseMatchOffset))
+		repeat = s - t
+		s += l
+		nextEmit = s
+		if nextS >= s {
+			s = nextS + 1
+		}
+
+		if s >= sLimit {
+			// Index after match end.
+			for i := nextS + 1; i < int32(len(src))-8; i += 2 {
+				cv := load6432(src, i)
+				e.table[hashLen(cv, tableBits, hashShortBytes)] = tableEntry{offset: i + e.cur}
+				eLong := &e.bTable[hash7(cv, tableBits)]
+				eLong.Cur, eLong.Prev = tableEntry{offset: i + e.cur}, eLong.Cur
+			}
+			goto emitRemainder
+		}
+
+		// Store every long hash in-between and every second short.
+		if true {
+			for i := nextS + 1; i < s-1; i += 2 {
+				cv := load6432(src, i)
+				t := tableEntry{offset: i + e.cur}
+				t2 := tableEntry{offset: t.offset + 1}
+				eLong := &e.bTable[hash7(cv, tableBits)]
+				eLong2 := &e.bTable[hash7(cv>>8, tableBits)]
+				e.table[hashLen(cv, tableBits, hashShortBytes)] = t
+				eLong.Cur, eLong.Prev = t, eLong.Cur
+				eLong2.Cur, eLong2.Prev = t2, eLong2.Cur
+			}
+		}
+
+		// We could immediately start working at s now, but to improve
+		// compression we first update the hash table at s-1 and at s.
+		cv = load6432(src, s)
+	}
+
+emitRemainder:
+	if int(nextEmit) < len(src) {
+		// If nothing was added, don't encode literals.
+		if dst.n == 0 {
+			return
+		}
+
+		emitLiteral(dst, src[nextEmit:])
+	}
+}