Backer
HTTP/HTTPS backup server that creates compressed archives on-the-fly from configured directories. Supports multiple compression algorithms: gzip, pgzip (which is parallel gzip), bzip2, zstd, lz4, xz.
How to build it
make # Build binary
make build VERSION=0.1.0 # Build with version
make test # Run tests
make # Clean + build
make upgrade # Update dependencies
That will produce binary named backer. Copy it somewhere in /usr/local/sbin/backer, for example. In data
directory resides config_example.json. Using this file make working config, put it into /etc/backer.json.
Service ready to run.
You can copy systemd unit from contrib directory to /etc/systemd/system and make usual rituals required for systemd.
If you running alpine linux or gentoo, you can use openrc init script from contrib dir.
If you running FreeBSD, use the rc.d script contrib/backer.freebsd and place it in /usr/local/etc/rc.d/.
How to use it
Build, install and configure backer. On your backup storage server you can do
wget --quiet --no-check-certificate --content-disposition --http-user=username --http-password=password https://your_server:8086/archive
The filename and extension depend on your filename_prefix and default_compression settings.
You can also override compression by specifying the archive extension in the URL:
wget --quiet --no-check-certificate --content-disposition --http-user=username --http-password=password https://your_server:8086/archive.xz
wget --quiet --no-check-certificate --content-disposition --http-user=username --http-password=password https://your_server:8086/archive.gz # Uses pgzip
wget --quiet --no-check-certificate --content-disposition --http-user=username --http-password=password https://your_server:8086/archive.bz2
wget --quiet --no-check-certificate --content-disposition --http-user=username --http-password=password https://your_server:8086/archive.zst
wget --quiet --no-check-certificate --content-disposition --http-user=username --http-password=password https://your_server:8086/archive.lz4
Note: /archive.tar.gz always uses pgzip (parallel gzip) for better performance.
Of course, something can go wrong, so you must check http status code after downloading backup.
About config options
All options described in example config.
| Option | Default value | Wtf | Notes |
|---|---|---|---|
| address | "0.0.0.0" | Address at which server listens. | In many cases 0.0.0.0 is okay. |
| port | 8086 | Port at which server binds. | Number picked after intel first commercially successful cpu. |
| cert | — | Path to ssl certificate file. | Required when nohttps is false. |
| key | — | Path to ssl certificates key file. | Required when nohttps is false. |
| nohttps | false | Disable https and use plain http. | For lab experiments or development. |
| location | "/archive" | API endpoint path for backup download. | |
| user | — | Username for basic auth. | Required. |
| password | — | Password for basic auth. | Required. |
| log | stderr | Log output file path. | |
| loglevel | "info" | Verbosity of logs. | Options: error, warn, info, debug. |
| directories | — | Directories to backup. | Required. Array of paths. |
| backup_timeout | 60 | Timeout in minutes for backup streaming. | Range: 1-1440 minutes. |
| compression_level | 9 | Compression level. | 1 (fastest) to 9 (best compression). |
| exclude_patterns | [] | Regex patterns to exclude from backup. | E.g., [".*\\.tmp$", "/node_modules/"]. |
| filename_prefix | "backup" | Prefix for backup filename. | E.g., "mybackup" produces mybackup-20260325-092341.tar.gz. |
| default_compression | "gzip" | Default compression algorithm. | Options: gzip, pgzip, bzip2, zstd, lz4, xz. |
| compression_algorithm | "gzip" | Alias for default_compression. | You should use default_compression instead. |
Compression algorithms
AI suggests this table as expected performance.
| Algorithm | Speed | Compression | Notes |
|---|---|---|---|
| gzip | Fast | Good | Default. Standard, widely compatible. |
| pgzip | Fast | Good | Parallel gzip. Uses multiple CPU cores, slightly larger files. |
| bzip2 | Slow | Better | Pure Go. Good compression, slower than gzip. |
| zstd | Fast | Very good | Good balance of speed and compression. |
| lz4 | Fastest | Moderate | Extremely fast, larger files. |
| xz | Slowest | Best | Best compression, slowest speed. |
In real live with backup of 1 lxc container inside vm powered by 1 core cpu, 768Mb RAM and 100Mbit non-guaranteed network bandwidth, located in near by country I've got these results
| Algorithm | Size | Time | CPU Load | RAM RSS+Shm, Mb | Notes |
|---|---|---|---|---|---|
| None | 1461250551 | 0-2% | 0+0 | original dataset | |
| lz4 | 1097745915 | 6m:12s | 4-6% | 30+6 | multithreaded lz4, pierrec/lz4 |
| zstd | 614015773 | 4m:52s | 40-60% | 60+6 | klauspost/compress/zstd |
| pgzip | 817163857 | 11m:25s | 96-97% | 37+6 | multithreaded (parallel gzip), fully gzip compatible, klauspost/pgzip |
| gzip | 816734188 | 14m:30s | 96% | 18+6 | stdlib compression/gzip |
| bzip2 | 788061284 | 7m:16s | 92-94% | 43+6 | dsnet/compress/bzip2 |
| xz | 512004612 | 13m:53s | 96% | 116+6 | ulikunitz/xz |
Obvously in case of lz4 and zstd the bottleneck was network. And I have to admit, that network definitely was not 100Mbit, but it was pretty hard to measure real bandwidth. Anyway let's pretend that during experiment it was roughly the same.
Best practices
Traditional yada-yada-yada about internet threats.
It is entirely possible to use this utility without https/certificate. But you should not do this, really. Validity of certificate is up to you, it can be even expired, but it should be well secured with reasonable key length and good enough cypher.
And one more thing user and password are mandatory, we should have at least some minimal barriers, right?
Special Thanks
This project was developed with the assistance of opencode (AI co-programmer powered by MIMO). The AI helped with:
- Code implementation and refactoring
- Test coverage
- Documentation
- Bug fixes
- Linting and code style consistency
Big thanks to the AI for doing most of the boring stuff while I was drinking coffee.