eleksir/bunny-bot
0
Fork 0
Code Pull Requests Projects Activity

Global code ovehaul - remove CAS and other, add 15 minute greet check

Browse Source 
- Remove CAS, as it no longer works for us
- Remove other checks
- Implement 15 minute greet timeout check
- Fix chat logos
- Ditch thirdparty logger, use stdlib log facility wrapper for logging
This commit is contained in:
eleksir
2026-10-20 12:20:16 +03:00
parent 2f467af6cf
commit 0c34b6e781
1597 changed files with 6440 additions and 648118 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
.golangci.yml
View File

@@ -2,7 +2,7 @@ version: "2"
run:
go: "1.24"
modules-download-mode: readonly
tests: false
tests: true
allow-parallel-runners: true
linters:
default: none

32
DESIGN.txt
View File

@@ -1,32 +0,0 @@
# How it works.
------------------------------------------------------------------------------------------------------------------------
Autoban uses 2 methods of matching spammers.
1. by querying CAS on member entry or change status from left-to-member.
2. by checking events, collected after 10 seconds since member enters chat.
------------------------------------------------------------------------------------------------------------------------
1.a Events, from new_chat_members being checked versus CAS and if check is positive user being banned.
1.b Events chat_members_updated with fields new_chat_member.status = member and old_chat_member.status = left being
checked versus CAS and if check is positive user being banned.
1.c If user was not banned due to CAS check immediately, it will be checked against CAS second time after 10 minutes.
2.a Uids from message events, from new_chat_members fields stored into NewMembers collection alongside with timestamps
for each user id.
2.b Uids from chat_members_updated with fields new_chat_member.status = member and old_chat_member.status = left are
stored into AppearedUsers collection alongside with timestamps for each user id
2.c Every 10 seconds goroutine runAndBan rakes all events stored in NewMembers and AppearedUsers and compares them.
First it tosses off pairs of events with matching user ids. Then it searches in AppearedUsers events older than 10
seconds since current time and bans them (also puts them into SquashedMembers collection to avoid banning of banned
users). Then it iterates over NewMembers and cleans all events older than 15 seconds.
# Note on chat logging.
------------------------------------------------------------------------------------------------------------------------
Since logging done in synchronous manner (to avoid too many opened descriptors) it can be bottleneck.

76
README.md
View File

@@ -1,31 +1,75 @@
# Bunny telegram bot
Yet another telegram bot. It has one purpose - restrict access of abusers to chat.
Ещё один телеграм-бот.
By now it uses [Combot Anti-Spam System](https://cas.chat/) to detect spammers and its own method of detection of bad
users.
Его задача - банить спаммеров. Обнаружение спаммеров работает единственным способом - если вошедший не отвечает на
приветствие в течение 15 минут, он попадает в бан.
## Build instructions
## Как это собрать?
Bot is not done yet, actually it does not do anything useful.
Понадобится компилятор go версии 1.24 или новее. Воодим в каталоге с исходниками команду
```bash
make
```
## Run instructions
и ждём. Через некоторое время в каталоге должен появиться бмнапник bunny-bot или bunny-bot.exe - это и есть весь наш бот.
You probably don't need it in its current state. But if you want - all is simple. Visit data/config.json, edit for your
needs and run:
## Как это запустить
Надо создать каталог `data/config.json` рядом с бинарником в качестве примера можно использовать `data/config-example.json`:
```json
{
"token": "get_it_from_botfather",
"loglevel": "debug",
"logfile": "/var/log/bunny-bot/bunny-bot.log",
"chat_logs": "chat_logs"
}
```
Затем запускаем сам бинарник:
```bash
./bunny-bot
```
There is openrc init script in contrib directory, you may use it for creating service under alpine linux or
gentoo(untested) with openrc init.
Путь до конфига захардкожен и это `<binary dir>/data/config.json`.
That's it.
### Логгирование чатов
Параметр `chat_logs` опционален и, если он есть и целевой каталог существует и доступен для записи, бот будет логгировать события в чате.
События из чата группируются по дням:
```text
<chat_logs>/<chat id>/<yyyy>/<MM>/<dd>.txt
```
И каждая строка выглядит примерно так:
```text
21:33:07.123 in chat -1001234567890 (title=Bunny Fans username=@bunnybot) user 42 (first_name=Ann last_name=Smith username=@ann) says hello there
21:33:07.123 in chat -1001234567890 (title=Bunny Fans username=@bunnybot) user 42 (first_name=Ann last_name=Smith username=@ann) post video (file_name=clip.mp4 mime=video/mp4 1280x720 duration=15s size=1048576) says look at this reply_to=1
21:33:07.123 in chat -1001234567890 (title=Bunny Fans username=@bunnybot) user 42 (first_name=Ann) user 7 (first_name=Al) joined the chat
21:33:07.123 in chat -1001234567890 (title=Bunny Fans) user 42 (first_name=Ann) status member -> administrator custom_title=Mod by user 7 (first_name=Bob)
```
Временные метки во всех логах проставляются по часовому поясу GMT+0.
### Дополнительные файлы
в каталоге `contrib` есть пример openrc-скрипта для запуска бота в среде alpine linux.
### Signals
| Signal | Effect |
|---|---|
| `SIGINT`, `SIGTERM`, `SIGQUIT` | Аккуратоное запершение работы - закрываются кэши, логи и затем программа завершается. |
| `SIGHUP`, `SIGUSR1` | Переоткрывает лог-файлы, для ротации логов. Процесс продолжает работать. |
N.B.
Бот не откладывает PID-файла, используйте copy-truncate в logrotate.
## Special thanks
@@ -33,16 +77,10 @@ To all active users of chat :)
To maintainers of telegram bot lib [https://github.com/NicoNex/echotron](https://github.com/NicoNex/echotron)
To maintainers of pebble db [https://github.com/cockroachdb/pebble](https://github.com/cockroachdb/pebble)
To maintainers of zap logging facility [https://github.com/uber-go/zap](https://github.com/uber-go/zap)
And kind hello to maintainer of spew dumper [https://github.com/davecgh/go-spew](https://github.com/davecgh/go-spew)
Without you, folks, all this bot-thing become very boring adventure.
## Maybe some more mumble-jumble
## AI Slop warning
Yes, but next time. When bot will be ready for some real action.
For now I can say that bot does not support any of commands.
Последние изменения в боте сделаны с помощью ai.

14
TODO.txt
View File

@@ -1,14 +0,0 @@
Before release we have to do next features:
[ ] option to enable/disable cas api
[ ] option to enable/disable generic spammer detection
[ ] command to leave chat
[ ] command to mute user
[ ] command to unmute user
[ ] command to ban user
[ ] command to unban user
[ ] add ability to show software version
[ ] make use of sailor moon quotes :)
[ ] implement hello phrase with 15 minute timeout. if user does not answer, ban it
[ ] ban people with first message is video or picture/image/photo.
[ ] replace zap with zerolog, to use only one facility in all places.

100
cmd/bunny-bot/main.go
View File

@@ -1,87 +1,83 @@
package main
import (
"bunny-bot/internal/moon"
"fmt"
"os"
"os/signal"
"path/filepath"
"syscall"
"time"
"go.uber.org/zap"
"go.uber.org/zap/zapcore"
"bunny-bot/internal/moon"
"bunny-bot/internal/mylog"
)
// TODO: use https://dev.to/gholami1313/saving-log-messages-to-a-custom-log-file-in-golang-ce5 for
// main decides the process's fate and nothing else: run the bot, and decide the
// exit status from how it ended. An error leaves 1; a signal-driven stop leaves
// 128 + the signal's number (130 for Ctrl+C on this platform) so a supervisor can
// tell a signal stop from a clean one, which is what D33 fixed; a stop nobody
// signalled leaves 0.
//
// chat messages.
// os.Exit is called here and nowhere else, deliberately. A worker goroutine cannot
// set an exit status without ending the process, which skips every deferred
// cleanup on the way out — including the one that closes the log we would be
// complaining about.
func main() {
var err error
if err := run(); err != nil {
// The logs may already be closed, so stderr is the one channel left that
// is certain to be attached.
fmt.Fprintf(os.Stderr, "bunny-bot: %v\n", err)
os.Exit(1)
}
os.Exit(moon.ExitCode())
}
// run is the whole lifecycle, so that every deferred cleanup belongs to a function
// that returns instead of one that exits.
//
// Its single defer covers the bot's entire life: once the log is open, nothing
// after that point returns early.
func run() error {
// Main config.
executablePath, err := os.Executable()
if err != nil {
panic(fmt.Sprintf("Unable to get current executable path: %s", err))
return fmt.Errorf("unable to get current executable path: %w", err)
}
configJSONPath := fmt.Sprintf("%s/data/config.json", filepath.Dir(executablePath))
moon.Cfg, err = moon.ParseConfig(configJSONPath)
if err != nil {
panic(err)
return fmt.Errorf("unable to read the config: %w", err)
}
// Config for Zap logger facility.
var logCfg = zap.NewDevelopmentConfig()
// Define loglevel.
var l zapcore.Level
if err := l.UnmarshalText([]byte(moon.Cfg.LogLevel)); err != nil {
panic("Unknown LogLevel")
// Open the log. ParseConfig has already coerced an unknown level to "info",
// so an error here means the file itself could not be opened — which is
// worth surfacing loudly rather than continuing with no record of anything.
if err := mylog.Init(moon.Cfg.LogLevel, moon.Cfg.LogFile); err != nil {
return fmt.Errorf("unable to open the log: %w", err)
}
logCfg.Level = zap.NewAtomicLevelAt(l)
defer func() { _ = mylog.Close() }()
// Define output file if set in config.
if moon.Cfg.LogFile != "" {
logCfg.OutputPaths[0] = moon.Cfg.LogFile
}
// We do not want to see caller in logs.
logCfg.DisableCaller = true
// We want human-readable timestamp in logs.
logCfg.EncoderConfig.EncodeTime = zapcore.TimeEncoderOfLayout(time.DateTime)
logger := zap.Must(logCfg.Build())
moon.Log = logger.Sugar()
mystdlogger := zap.RedirectStdLog(logger)
defer mystdlogger()
moon.Log.Info("Config read looks good")
mylog.Info("Config read looks good")
// Самое время поставить траппер сигналов.
signal.Notify(moon.SigChan,
syscall.SIGINT,
syscall.SIGTERM,
syscall.SIGQUIT)
moon.InstallSignalHandler()
go moon.SigHandler()
mylog.Info("Signal handler installed")
moon.Log.Info("Singnal handler installed")
moon.Log.Info("Starting RunAndBan() periodic task")
go moon.RunAndBan()
moon.Log.Info("Starting main loop")
mylog.Info("Starting main loop")
// Telegram assigns moon.Tg on this goroutine; the sweeper reads it, so the
// client must be constructed before any goroutine that can call the API.
moon.Telegram(moon.Cfg)
// The update listener has returned, but that is not the same as a finished
// shutdown: a signal handler may be mid-teardown right now. Shutdown blocks
// until the ordered sequence has completed, whichever goroutine started it,
// so the process cannot exit with the log half-closed — and its error is what
// decides the exit status.
return moon.Shutdown()
}
/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

7
data/config-example.json
View File

@@ -1,9 +1,6 @@
{
"name": "Bunny",
"token": "get_it_from_botfather",
"loglevel": "debug",
"logfile": "bunny-bot.log",
"logchats": false,
"csign": ".",
"datadir": "data"
"logfile": "/var/log/bunny-bot/bunny-bot.log",
"chat_logs": "chat_logs"
}

21
data/quotes.json
View File

@@ -1,21 +0,0 @@
{
"Тот, кто не знает силу пешек, не имеет права ходить королевой.",
"Когда ты пройдешь сквозь мрак беспомощности и попадешь в мир надежды, тогда ты преодолеешь все преграды.",
"По-моему, тебе очень хочется стать уткой по-пекински!",
"If a soldier’s pride means hurting one another, I don’t want it.",
"Without the bad times, we wouldn’t appreciate the good times.",
"I’ll just warn you now, I don’t know how to use a computer.",
"Thinking isn’t going to get us anywhere.",
"Вместо того, чтобы произносить речи, лучше бы подумала, что ты будешь делать дальше.",
"Да как ты смела разгадать мой замысел?! Я же целую ночь не спала, пока его придумала. Верни мне мой сон!",
"Не волнуйся, жизнь намного скучнее, чем ты себе её представляешь.",
"Я бы завела котёнка, но у меня уже есть Луна и она жрёт как лошадь.",
"Звезда нашей надежды никогда не исчезнет. Пока она горит, всё будет в порядке. Мы не сможем проиграть",
"Голова с хвостиками… Не вмешивайся в наши дела! Играть в войну — это опасно",
"I don’t have a short attention span, I just… oh look, something shiny!",
"When I’m with you, I feel like I can conquer anything.",
"Moon Prism Power, Make Up!",
"In the name of the moon, I will punish you!",
"I’m not just some pretty face! I’m also the soldier of love and justice, Sailor Moon!",
"Don’t let anyone dim your light. Shine bright like the moon and stars."
}

38
go.mod
View File

@@ -4,48 +4,14 @@ go 1.24.3
tool (
bunny-bot/cmd/bunny-bot
bunny-bot/internal/collection
bunny-bot/internal/moon
)
require (
github.com/NicoNex/echotron/v3 v3.43.0
github.com/NicoNex/echotron/v3 v3.46.0
github.com/akyoto/cache v1.0.6
github.com/cockroachdb/errors v1.12.0
github.com/cockroachdb/pebble v1.1.5
github.com/davecgh/go-spew v1.1.1
github.com/hjson/hjson-go v3.3.0+incompatible
go.uber.org/zap v1.27.0
)
require (
github.com/DataDog/zstd v1.5.7 // indirect
github.com/beorn7/perks v1.0.1 // indirect
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/cockroachdb/fifo v0.0.0-20240816210425-c5d0cb0b6fc0 // indirect
github.com/cockroachdb/logtags v0.0.0-20241215232642-bb51bb14a506 // indirect
github.com/cockroachdb/redact v1.1.6 // indirect
github.com/cockroachdb/tokenbucket v0.0.0-20250429170803-42689b6311bb // indirect
github.com/getsentry/sentry-go v0.33.0 // indirect
github.com/gogo/protobuf v1.3.2 // indirect
github.com/golang/snappy v1.0.0 // indirect
github.com/klauspost/compress v1.18.0 // indirect
github.com/kr/pretty v0.3.1 // indirect
github.com/kr/text v0.2.0 // indirect
github.com/mattn/go-colorable v0.1.13 // indirect
github.com/mattn/go-isatty v0.0.19 // indirect
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
github.com/pkg/errors v0.9.1 // indirect
github.com/prometheus/client_golang v1.22.0 // indirect
github.com/prometheus/client_model v0.6.2 // indirect
github.com/prometheus/common v0.64.0 // indirect
github.com/prometheus/procfs v0.16.1 // indirect
github.com/rogpeppe/go-internal v1.14.1 // indirect
github.com/rs/zerolog v1.34.0 // indirect
go.uber.org/multierr v1.11.0 // indirect
golang.org/x/exp v0.0.0-20250531010427-b6e5de432a8b // indirect
golang.org/x/sys v0.33.0 // indirect
golang.org/x/text v0.26.0 // indirect
golang.org/x/time v0.12.0 // indirect
google.golang.org/protobuf v1.36.6 // indirect
)
require golang.org/x/time v0.12.0 // indirect

123
go.sum
View File

@@ -1,129 +1,10 @@
github.com/DataDog/zstd v1.5.7 h1:ybO8RBeh29qrxIhCA9E8gKY6xfONU9T6G6aP9DTKfLE=
github.com/DataDog/zstd v1.5.7/go.mod h1:g4AWEaM3yOg3HYfnJ3YIawPnVdXJh9QME85blwSAmyw=
github.com/NicoNex/echotron/v3 v3.43.0 h1:efE2spw3mfU0Ev20m0PqqvgMSm0xHzgSxlWAEmC9RC4=
github.com/NicoNex/echotron/v3 v3.43.0/go.mod h1:7LvjveJmezuUOeaoA3nzQduNlSPQYfq219Z+baKY04Q=
github.com/NicoNex/echotron/v3 v3.46.0 h1:IHPhEDU3pbxa3uLnxRz5e247SYEVzIbQS014qvuWFOU=
github.com/NicoNex/echotron/v3 v3.46.0/go.mod h1:7LvjveJmezuUOeaoA3nzQduNlSPQYfq219Z+baKY04Q=
github.com/akyoto/cache v1.0.6 h1:5XGVVYoi2i+DZLLPuVIXtsNIJ/qaAM16XT0LaBaXd2k=
github.com/akyoto/cache v1.0.6/go.mod h1:WfxTRqKhfgAG71Xh6E3WLpjhBtZI37O53G4h5s+3iM4=
github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
github.com/cockroachdb/datadriven v1.0.3-0.20230413201302-be42291fc80f h1:otljaYPt5hWxV3MUfO5dFPFiOXg9CyG5/kCfayTqsJ4=
github.com/cockroachdb/datadriven v1.0.3-0.20230413201302-be42291fc80f/go.mod h1:a9RdTaap04u637JoCzcUoIcDmvwSUtcUFtT/C3kJlTU=
github.com/cockroachdb/errors v1.12.0 h1:d7oCs6vuIMUQRVbi6jWWWEJZahLCfJpnJSVobd1/sUo=
github.com/cockroachdb/errors v1.12.0/go.mod h1:SvzfYNNBshAVbZ8wzNc/UPK3w1vf0dKDUP41ucAIf7g=
github.com/cockroachdb/fifo v0.0.0-20240816210425-c5d0cb0b6fc0 h1:pU88SPhIFid6/k0egdR5V6eALQYq2qbSmukrkgIh/0A=
github.com/cockroachdb/fifo v0.0.0-20240816210425-c5d0cb0b6fc0/go.mod h1:9/y3cnZ5GKakj/H4y9r9GTjCvAFta7KLgSHPJJYc52M=
github.com/cockroachdb/logtags v0.0.0-20241215232642-bb51bb14a506 h1:ASDL+UJcILMqgNeV5jiqR4j+sTuvQNHdf2chuKj1M5k=
github.com/cockroachdb/logtags v0.0.0-20241215232642-bb51bb14a506/go.mod h1:Mw7HqKr2kdtu6aYGn3tPmAftiP3QPX63LdK/zcariIo=
github.com/cockroachdb/pebble v1.1.5 h1:5AAWCBWbat0uE0blr8qzufZP5tBjkRyy/jWe1QWLnvw=
github.com/cockroachdb/pebble v1.1.5/go.mod h1:17wO9el1YEigxkP/YtV8NtCivQDgoCyBg5c4VR/eOWo=
github.com/cockroachdb/redact v1.1.6 h1:zXJBwDZ84xJNlHl1rMyCojqyIxv+7YUpQiJLQ7n4314=
github.com/cockroachdb/redact v1.1.6/go.mod h1:BVNblN9mBWFyMyqK1k3AAiSxhvhfK2oOZZ2lK+dpvRg=
github.com/cockroachdb/tokenbucket v0.0.0-20250429170803-42689b6311bb h1:3bCgBvB8PbJVMX1ouCcSIxvsqKPYM7gs72o0zC76n9g=
github.com/cockroachdb/tokenbucket v0.0.0-20250429170803-42689b6311bb/go.mod h1:7nc4anLGjupUW/PeY5qiNYsdNXj7zopG+eqsS7To5IQ=
github.com/coreos/go-systemd/v22 v22.5.0/go.mod h1:Y58oyj3AT4RCenI/lSvhwexgC+NSVTIJ3seZv2GcEnc=
github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/getsentry/sentry-go v0.33.0 h1:YWyDii0KGVov3xOaamOnF0mjOrqSjBqwv48UEzn7QFg=
github.com/getsentry/sentry-go v0.33.0/go.mod h1:C55omcY9ChRQIUcVcGcs+Zdy4ZpQGvNJ7JYHIoSWOtE=
github.com/go-errors/errors v1.4.2 h1:J6MZopCL4uSllY1OfXM374weqZFFItUbrImctkmUxIA=
github.com/go-errors/errors v1.4.2/go.mod h1:sIVyrIiJhuEF+Pj9Ebtd6P/rEYROXFi3BopGUQ5a5Og=
github.com/godbus/dbus/v5 v5.0.4/go.mod h1:xhWf0FNVPg57R7Z0UbKHbJfkEywrmjJnf7w5xrFpKfA=
github.com/gogo/protobuf v1.3.2 h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q=
github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q=
github.com/golang/snappy v1.0.0 h1:Oy607GVXHs7RtbggtPBnr2RmDArIsAefDwvrdWvRhGs=
github.com/golang/snappy v1.0.0/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q=
github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
github.com/hjson/hjson-go v3.3.0+incompatible h1:Rqr+Ya+0aCJMjaE4s8E9YKvuJLuLVpEvz4ONum52vnI=
github.com/hjson/hjson-go v3.3.0+incompatible/go.mod h1:qsetwF8NlsTsOTwZTApNlTCerV+b2GjYRRcIk4JMFio=
github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8=
github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck=
github.com/klauspost/compress v1.18.0 h1:c/Cqfb0r+Yi+JtIEq73FWXVkRonBlf0CRNYc8Zttxdo=
github.com/klauspost/compress v1.18.0/go.mod h1:2Pp+KzxcywXVXMr50+X0Q/Lsb43OQHYWRCY2AiWywWQ=
github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=
github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk=
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
github.com/mattn/go-colorable v0.1.13 h1:fFA4WZxdEF4tXPZVKMLwD8oUnCTTo08duU7wxecdEvA=
github.com/mattn/go-colorable v0.1.13/go.mod h1:7S9/ev0klgBDR4GtXTXX8a3vIGJpMovkB8vQcUbaXHg=
github.com/mattn/go-isatty v0.0.16/go.mod h1:kYGgaQfpe5nmfYZH+SKPsOc2e4SrIfOl2e/yFXSvRLM=
github.com/mattn/go-isatty v0.0.19 h1:JITubQf0MOLdlGRuRq+jtsDlekdYPia9ZFsB8h/APPA=
github.com/mattn/go-isatty v0.0.19/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y=
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA=
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
github.com/pingcap/errors v0.11.4 h1:lFuQV/oaUMGcD2tqt+01ROSmJs75VG1ToEOkZIZ4nE4=
github.com/pingcap/errors v0.11.4/go.mod h1:Oi8TUi2kEtXXLMJk9l1cGmz20kV3TaQ0usTwv5KuLY8=
github.com/pkg/diff v0.0.0-20210226163009-20ebb0f2a09e/go.mod h1:pJLUxLENpZxwdsKMEsNbx1VGcRFpLqf3715MtcvvzbA=
github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/prometheus/client_golang v1.22.0 h1:rb93p9lokFEsctTys46VnV1kLCDpVZ0a/Y92Vm0Zc6Q=
github.com/prometheus/client_golang v1.22.0/go.mod h1:R7ljNsLXhuQXYZYtw6GAE9AZg8Y7vEW5scdCXrWRXC0=
github.com/prometheus/client_model v0.6.2 h1:oBsgwpGs7iVziMvrGhE53c/GrLUsZdHnqNwqPLxwZyk=
github.com/prometheus/client_model v0.6.2/go.mod h1:y3m2F6Gdpfy6Ut/GBsUqTWZqCUvMVzSfMLjcu6wAwpE=
github.com/prometheus/common v0.64.0 h1:pdZeA+g617P7oGv1CzdTzyeShxAGrTBsolKNOLQPGO4=
github.com/prometheus/common v0.64.0/go.mod h1:0gZns+BLRQ3V6NdaerOhMbwwRbNh9hkGINtQAsP5GS8=
github.com/prometheus/procfs v0.16.1 h1:hZ15bTNuirocR6u0JZ6BAHHmwS1p8B4P6MRqxtzMyRg=
github.com/prometheus/procfs v0.16.1/go.mod h1:teAbpZRB1iIAJYREa1LsoWUXykVXA1KlTmWl8x/U+Is=
github.com/rogpeppe/go-internal v1.9.0/go.mod h1:WtVeX8xhTBvf0smdhujwtBcq4Qrzq/fJaraNFVN+nFs=
github.com/rogpeppe/go-internal v1.14.1 h1:UQB4HGPB6osV0SQTLymcB4TgvyWu6ZyliaW0tI/otEQ=
github.com/rogpeppe/go-internal v1.14.1/go.mod h1:MaRKkUm5W0goXpeCfT7UZI6fk/L7L7so1lCWt35ZSgc=
github.com/rs/xid v1.6.0/go.mod h1:7XoLgs4eV+QndskICGsho+ADou8ySMSjJKDIan90Nz0=
github.com/rs/zerolog v1.34.0 h1:k43nTLIwcTVQAncfCw4KZ2VY6ukYoZaBPNOE8txlOeY=
github.com/rs/zerolog v1.34.0/go.mod h1:bJsvje4Z08ROH4Nhs5iH600c3IkWhwp44iRc54W6wYQ=
github.com/stretchr/testify v1.10.0 h1:Xv5erBjTwe/5IxqUQTdXv5kgmIvbHo3QQyRwhJsOfJA=
github.com/stretchr/testify v1.10.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY=
github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto=
go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE=
go.uber.org/multierr v1.11.0 h1:blXXJkSxSSfBVBlC76pxqeO+LN3aDfLQo+309xJstO0=
go.uber.org/multierr v1.11.0/go.mod h1:20+QtiLqy0Nd6FdQB9TLXag12DsQkrbs3htMFfDN80Y=
go.uber.org/zap v1.27.0 h1:aJMhYGrd5QSmlpLMr2MftRKl7t8J8PTZPA732ud/XR8=
go.uber.org/zap v1.27.0/go.mod h1:GB2qFLM7cTU87MWRP2mPIjqfIDnGu+VIO4V/SdhGo2E=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
golang.org/x/exp v0.0.0-20250531010427-b6e5de432a8b h1:QoALfVG9rhQ/M7vYDScfPdWjGL9dlsVVM5VGh7aKoAA=
golang.org/x/exp v0.0.0-20250531010427-b6e5de432a8b/go.mod h1:U6Lno4MTRCDY+Ba7aCcauB9T60gsv5s4ralQzP72ZoQ=
golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.15.0 h1:KWH3jNZsfyT6xfAfKiz6MRNmd46ByHDYaZ7KSkCtdW8=
golang.org/x/sync v0.15.0/go.mod h1:1dzgHSNfp02xaA81J2MS99Qcpr2w7fw1gpm99rleRqA=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.12.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.33.0 h1:q3i8TbbEz+JRD9ywIRlyRAQbM0qF7hu24q3teo2hbuw=
golang.org/x/sys v0.33.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.26.0 h1:P42AVeLghgTYr4+xUnTRKDMqpar+PtX7KWuNQL21L8M=
golang.org/x/text v0.26.0/go.mod h1:QK15LZJUUQVJxhz7wXgxSy/CJaTFjd0G+YLonydOVQA=
golang.org/x/time v0.12.0 h1:ScB/8o8olJvc+CQPWrK3fPZNfh7qgwCrY0zJmoEQLSE=
golang.org/x/time v0.12.0/go.mod h1:CDIdPxbZBQxdj6cxyCIdrNogrJKMJ7pr37NYpMcMDSg=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20200619180055-7c47624df98f/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE=
golang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
google.golang.org/protobuf v1.36.6 h1:z1NpPI8ku2WgiWnf+t9wTPsn6eP1L7ksHUlkfLvd9xY=
google.golang.org/protobuf v1.36.6/go.mod h1:jduwjTPXsFjZGTmRluh+L6NjiWu7pchiJ2/5YcXBHnY=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=

14
internal/elog/clock.go Normal file
View File

@@ -0,0 +1,14 @@
package elog
import "time"
// nowUTC is the single clock every path and timestamp derives from.
//
// It is not time.Now at each call site: a log line stamped in local time beside a
// directory named for UTC is a bug report waiting to happen, and a file that rolls
// over at midnight local time while its lines are stamped UTC is worse.
func nowUTC() time.Time {
return time.Now().UTC()
}
/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

78
internal/elog/format.go Normal file
View File

@@ -0,0 +1,78 @@
package elog
// The value formatters: small conversions shared by the write path (which builds
// path components) and the renderer (which builds inline fields). They are shared
// by two topics, so they get a file of their own rather than living in one of them
// and making the other look in the wrong place.
import (
"strconv"
"strings"
)
// field renders "key=value", or empty when the value is empty. Empty fields are
// omitted: a bare "username=@" is worse than nothing.
//
// It lives here rather than in render.go because sizeField, a formatter, is one of
// its callers — and a helper two topics use belongs to neither.
func field(key string, value string) string {
if value == "" {
return ""
}
return key + "=" + value
}
// joinFields drops empty fragments and joins the rest with single spaces.
func joinFields(parts []string) string {
kept := make([]string, 0, len(parts))
for _, p := range parts {
if p != "" {
kept = append(kept, p)
}
}
return strings.Join(kept, " ")
}
// itoa renders an int64 for a path component or an inline field.
func itoa(v int64) string {
return strconv.FormatInt(v, 10)
}
// itoaInt renders an int for a rendered field.
func itoaInt(v int) string {
return strconv.Itoa(v)
}
// itoaFloat renders a float for a rendered field.
func itoaFloat(v float64) string {
return strconv.FormatFloat(v, 'g', -1, 64)
}
// atUsername renders a username with the conventional @ prefix, or empty.
func atUsername(name string) string {
if name == "" {
return ""
}
return "@" + name
}
// seconds renders a duration in seconds with its unit.
func seconds(n int) string {
return itoaInt(n) + "s"
}
// sizeField renders a byte count, omitted when unknown. A size of 0 means
// Telegram did not tell us, not that the file was empty.
func sizeField(n int64) string {
if n <= 0 {
return ""
}
return field("size", itoa(n))
}
/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

59
internal/elog/globals.go Normal file
View File

@@ -0,0 +1,59 @@
package elog
import (
"errors"
"sync"
"time"
)
const (
// chatLockIdleTimeout is how long a chat's lock entry may sit unused before it
// is evicted. Without it the registry grows by one entry per chat ever seen.
chatLockIdleTimeout = time.Hour
// stampLayout is the per-line timestamp. GMT+0 with millisecond precision.
stampLayout = "15:04:05.000"
// logFilePerm is the mode for a newly created chat log file.
logFilePerm = 0o644
// logDirPerm is the mode for the chat, year and month directories.
//
// It is deliberately not logFilePerm. A directory needs the execute bit to
// be traversable, and 0o644 has none, so creating the tree with the file mode
// makes every subsequent open inside it fail with EACCES rather than ENOENT
// — a failure reportFailure would swallow, leaving chat logging to silently
// produce no files at all.
//
// The two live in one block on purpose: they are a pair that must never be
// confused with each other, and keeping them adjacent is what makes confusing
// them hard. They are here, in globals.go, because that is where every
// package-level const belongs — the pair argument was weighed against rule 2
// and rule 2 won, once, on the record.
logDirPerm = 0o755
)
var (
// errNotDirectory marks a chat_logs path that exists but is a regular file.
errNotDirectory = errors.New("chat_logs path is not a directory")
// root is the configured chat-logs directory. Empty means chat logging is
// disabled and every Log* call is a no-op.
root string
// failing records that a write error has been reported and not yet cleared,
// so a persistent failure (a full disk) logs once instead of once per event.
failing bool
// stateMu guards root and failing.
stateMu sync.RWMutex
// registryMu guards locks. It is never held while a chat lock is acquired,
// or the whole logger would serialise.
registryMu sync.Mutex
// locks is the per-chat lock registry.
locks = map[int64]*chatLock{}
)
/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

46
internal/elog/init.go Normal file
View File

@@ -0,0 +1,46 @@
package elog
import (
"fmt"
"os"
)
// Init switches chat logging on.
//
// An empty root disables it, and every Log* call becomes a no-op — chat logging
// is optional. A non-empty root must already exist: a path that does not is a
// typo, and silently logging nowhere is the worst possible response to one, so
// this returns an error and moon.ParseConfig treats it as fatal.
func Init(rootDir string) error {
if rootDir == "" {
setRoot("")
return nil
}
info, err := os.Stat(rootDir)
if err != nil {
return fmt.Errorf("chat_logs directory %s is not usable: %w", rootDir, err)
}
if !info.IsDir() {
return fmt.Errorf("%w: %s", errNotDirectory, rootDir)
}
setRoot(rootDir)
return nil
}
// Close disables chat logging.
//
// Nothing is held open between lines, so there are no handles to close; this
// exists so shutdown has a symmetric counterpart to Init and so a misbehaving
// caller can stop writing.
func Close() error {
setRoot("")
return nil
}
/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

127
internal/elog/message.go Normal file
View File

@@ -0,0 +1,127 @@
package elog
import (
"github.com/NicoNex/echotron/v3"
)
// LogMessage records a chat message.
//
// It handles both ordinary messages — text, media, captions, replies — and the
// service messages Telegram delivers as a Message with a field set instead of
// text: joins, leaves, title changes, pins, migrations, forum topics. A message
// with nothing describable is skipped rather than logged as noise.
func LogMessage(msg *echotron.Message) {
logMessage(msg, "")
}
// LogEditedMessage records an edited message. The line is marked "edited" so a
// reader can tell it apart from the original.
func LogEditedMessage(msg *echotron.Message) {
logMessage(msg, "edited ")
}
// logMessage is the shared body of LogMessage and LogEditedMessage.
func logMessage(msg *echotron.Message, prefix string) {
if _, on := chatLogRoot(); !on || msg == nil {
return
}
for _, body := range describeMessage(msg) {
appendLine(msg.Chat.ID, prefix+renderLine(msg.Chat, msg.From, body))
}
}
// describeMessage returns one rendered body per line to write. It returns more
// than one only for a multi-member join, where one line per newcomer keeps each
// fact greppable instead of cramming every newcomer onto one line.
func describeMessage(msg *echotron.Message) []string {
if bodies := describeService(msg); len(bodies) > 0 {
return bodies
}
body := describePayload(msg)
if body == "" {
return nil
}
if msg.ReplyToMessage != nil {
body += " reply_to=" + itoaInt(msg.ReplyToMessage.ID)
}
return []string{body}
}
// describeService renders the service-message fields, newest-first, and returns
// nil when the message is not a service message.
func describeService(msg *echotron.Message) []string {
var bodies []string
// A join with N newcomers writes N lines.
for _, u := range msg.NewChatMembers {
if u == nil {
continue
}
bodies = append(bodies, describeReference(u)+" joined the chat")
}
if msg.LeftChatMember != nil {
bodies = append(bodies, describeReference(msg.LeftChatMember)+" left the chat")
}
if msg.NewChatTitle != "" {
bodies = append(bodies, "chat title changed to "+msg.NewChatTitle)
}
if len(msg.NewChatPhoto) > 0 {
bodies = append(bodies, "chat photo changed")
}
if msg.PinnedMessage != nil {
bodies = append(bodies, "pinned message "+itoaInt(msg.PinnedMessage.ID))
}
if msg.GroupChatCreated {
bodies = append(bodies, "group chat created")
}
if msg.MigrateToChatID != 0 {
bodies = append(bodies, "chat migrated to "+itoa(int64(msg.MigrateToChatID)))
}
if msg.MigrateFromChatID != 0 {
bodies = append(bodies, "chat migrated from "+itoa(int64(msg.MigrateFromChatID)))
}
if msg.ConnectedWebsite != "" {
bodies = append(bodies, "connected website "+msg.ConnectedWebsite)
}
if msg.ForumTopicCreated != nil {
bodies = append(bodies, "forum topic "+msg.ForumTopicCreated.Name+" created")
}
if msg.ForumTopicClosed != nil {
bodies = append(bodies, "forum topic closed")
}
if msg.ForumTopicReopened != nil {
bodies = append(bodies, "forum topic reopened")
}
if msg.GeneralForumTopicHidden != nil {
bodies = append(bodies, "general forum topic hidden")
}
if msg.GeneralForumTopicUnhidden != nil {
bodies = append(bodies, "general forum topic unhidden")
}
if msg.VideoChatScheduled != nil {
bodies = append(bodies, "video chat scheduled")
}
return bodies
}
/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

248
internal/elog/render.go Normal file
View File

@@ -0,0 +1,248 @@
package elog
import (
"fmt"
"strings"
"github.com/NicoNex/echotron/v3"
)
// renderLine assembles one log line:
//
// 21:33:07.123 in chat -1001 (title=…) user 42 (first_name=…) says …
//
// Empty field groups are dropped entirely rather than printed as empty parens.
func renderLine(chat echotron.Chat, user *echotron.User, body string) string {
var b strings.Builder
b.WriteString(nowUTC().Format(stampLayout))
b.WriteString(" in chat ")
b.WriteString(itoa(chat.ID))
if desc := describeChat(&chat); desc != "" {
b.WriteString(" (")
b.WriteString(desc)
b.WriteString(")")
}
if user != nil {
b.WriteString(" user ")
b.WriteString(itoa(user.ID))
if desc := describeUser(user); desc != "" {
b.WriteString(" (")
b.WriteString(desc)
b.WriteString(")")
}
}
if body != "" {
b.WriteString(" ")
b.WriteString(body)
}
return b.String()
}
// describeChat renders a chat's identity fields. Type is deliberately omitted:
// the user-specified format has no slot for it, and which of title / first_name
// is present already distinguishes group from private chat.
func describeChat(c *echotron.Chat) string {
if c == nil {
return ""
}
return joinFields([]string{
field("title", c.Title),
field("username", atUsername(c.Username)),
field("first_name", c.FirstName),
field("last_name", c.LastName),
})
}
// describeUser renders a user's identity fields.
func describeUser(u *echotron.User) string {
if u == nil {
return ""
}
return joinFields([]string{
field("first_name", u.FirstName),
field("last_name", u.LastName),
field("username", atUsername(u.Username)),
})
}
// describePayload renders what a message carries: media first, then a caption,
// because "post photo (…) caption" reads better than the reverse.
func describePayload(m *echotron.Message) string {
kind, details := describeMedia(m)
var b strings.Builder
if kind != "" {
b.WriteString("post ")
b.WriteString(kind)
if details != "" {
b.WriteString(" (")
b.WriteString(details)
b.WriteString(")")
}
}
// A caption accompanies the media rather than replacing it: "post photo (…)"
// alone loses what the user actually said.
if m.Text != "" {
if b.Len() > 0 {
b.WriteString(" ")
}
b.WriteString("says ")
b.WriteString(m.Text)
}
return b.String()
}
// describeMedia returns the media kind and its key info. The first media field
// set on the message wins, in the order listed in the SPEC.
func describeMedia(m *echotron.Message) (string, string) {
switch {
case len(m.Photo) > 0:
// The array is ordered smallest first, so the last entry is the largest.
best := m.Photo[len(m.Photo)-1]
return "photo", joinFields([]string{
dims(best.Width, best.Height),
sizeField(int64(best.FileSize)),
})
case m.Video != nil:
return "video", joinFields([]string{
field("file_name", m.Video.FileName),
field("mime", m.Video.MimeType),
dims(m.Video.Width, m.Video.Height),
field("duration", seconds(m.Video.Duration)),
sizeField(m.Video.FileSize),
})
case m.Animation != nil:
return "animation", joinFields([]string{
field("file_name", m.Animation.FileName),
field("mime", m.Animation.MimeType),
dims(m.Animation.Width, m.Animation.Height),
field("duration", seconds(m.Animation.Duration)),
sizeField(m.Animation.FileSize),
})
case m.Audio != nil:
return "audio", joinFields([]string{
field("title", m.Audio.Title),
field("file_name", m.Audio.FileName),
field("mime", m.Audio.MimeType),
field("duration", seconds(m.Audio.Duration)),
sizeField(m.Audio.FileSize),
})
case m.Voice != nil:
return "voice", joinFields([]string{
field("mime", m.Voice.MimeType),
field("duration", seconds(m.Voice.Duration)),
sizeField(m.Voice.FileSize),
})
case m.VideoNote != nil:
return "video_note", joinFields([]string{
dims(m.VideoNote.Length, m.VideoNote.Length),
field("duration", seconds(m.VideoNote.Duration)),
sizeField(int64(m.VideoNote.FileSize)),
})
case m.Sticker != nil:
return "sticker", joinFields([]string{
field("emoji", m.Sticker.Emoji),
field("set", m.Sticker.SetName),
dims(m.Sticker.Width, m.Sticker.Height),
field("video", itoaBool(m.Sticker.IsVideo)),
field("animated", itoaBool(m.Sticker.IsAnimated)),
})
case m.Document != nil:
return "document", joinFields([]string{
field("file_name", m.Document.FileName),
field("mime", m.Document.MimeType),
sizeField(m.Document.FileSize),
})
case m.Contact != nil:
return "contact", joinFields([]string{
field("phone", m.Contact.PhoneNumber),
field("first_name", m.Contact.FirstName),
field("last_name", m.Contact.LastName),
field("user_id", itoaInt(m.Contact.UserID)),
})
case m.Location != nil:
return "location", joinFields([]string{
field("lat", itoaFloat(m.Location.Latitude)),
field("lon", itoaFloat(m.Location.Longitude)),
})
case m.Venue != nil:
parts := []string{
field("title", m.Venue.Title),
field("address", m.Venue.Address),
}
if m.Venue.Location != nil {
parts = append(parts,
field("lat", itoaFloat(m.Venue.Location.Latitude)),
field("lon", itoaFloat(m.Venue.Location.Longitude)))
}
return "venue", joinFields(parts)
case m.Dice != nil:
return "dice", joinFields([]string{
field("emoji", m.Dice.Emoji),
field("value", itoaInt(m.Dice.Value)),
})
}
return "", ""
}
// dims renders "WxH", or empty when the dimensions are unknown.
func dims(w int, h int) string {
if w <= 0 || h <= 0 {
return ""
}
return itoaInt(w) + "x" + itoaInt(h)
}
// itoaBool renders a boolean for an inline field.
func itoaBool(b bool) string {
if b {
return "true"
}
return "false"
}
// describeReference renders "user 42 (first_name=Ann)", used for the newcomer in
// a service message and for the member in a status change.
func describeReference(u *echotron.User) string {
if u == nil {
return "user unknown"
}
if desc := describeUser(u); desc != "" {
return fmt.Sprintf("user %d (%s)", u.ID, desc)
}
return fmt.Sprintf("user %d", u.ID)
}
/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

24
internal/elog/retraction.go Normal file
View File

@@ -0,0 +1,24 @@
package elog
import (
"github.com/NicoNex/echotron/v3"
)
// LogGreetingRetracted records that we deleted a greeting of our own — one the
// addressed member never answered, because they were banned for the silence.
//
// The log is append-only, so the question's original line stays where it was and
// this line says what became of it. A reader must be able to tell that the room
// asked a question and later removed it, rather than finding a line that
// describes a message which no longer exists.
func LogGreetingRetracted(chat *echotron.Chat, user *echotron.User, messageID int) {
if _, on := chatLogRoot(); !on || chat == nil {
return
}
body := "greeting " + itoaInt(messageID) + " retracted, member banned without answering"
appendLine(chat.ID, renderLine(*chat, user, body))
}
/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

44
internal/elog/root.go Normal file
View File

@@ -0,0 +1,44 @@
package elog
// The chat-logs root: where the tree lives, and whether logging is on at all.
//
// The state itself is in globals.go with the rest of the package-level variables;
// this file holds only the accessors, because "is it on and where" is one question
// with one answer and must not be asked in two pieces.
// ChatLoggingEnabled reports whether chat logging is currently switched on.
//
// Exported so another package can assert the post-shutdown state instead of
// inferring it from the absence of files.
func ChatLoggingEnabled() (string, bool) {
return chatLogRoot()
}
// chatLogRoot returns the configured root and reports whether chat logging is
// on, in a *single* read.
//
// The two must not be asked separately: a Close landing between an enabled()
// check and a rootDir() read would leave the caller with an empty root, and
// filepath.Join("", ...) silently yields a path relative to the working
// directory — chat logs would be written wherever the supervisor happened to
// start the process.
func chatLogRoot() (string, bool) {
stateMu.RLock()
defer stateMu.RUnlock()
return root, root != ""
}
// setRoot switches chat logging on or off.
func setRoot(dir string) {
stateMu.Lock()
defer stateMu.Unlock()
root = dir
if dir == "" {
failing = false
}
}
/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

63
internal/elog/status.go Normal file
View File

@@ -0,0 +1,63 @@
package elog
import (
"strings"
"github.com/NicoNex/echotron/v3"
)
// LogUserStatus records a change to a chat member's status — join, leave, ban,
// promote, demote, restrict — as carried by Update.ChatMember.
func LogUserStatus(ev *echotron.ChatMemberUpdated) {
if _, on := chatLogRoot(); !on || ev == nil {
return
}
who := ev.NewChatMember.User
if who == nil {
who = ev.OldChatMember.User
}
var b strings.Builder
// The subject is already rendered by renderLine's "user <id> (<desc>)"
// prefix, so the body starts at the transition itself.
b.WriteString("status ")
b.WriteString(orUnknown(ev.OldChatMember.Status))
b.WriteString(" -> ")
b.WriteString(orUnknown(ev.NewChatMember.Status))
if ev.NewChatMember.CustomTitle != "" {
b.WriteString(" custom_title=")
b.WriteString(ev.NewChatMember.CustomTitle)
}
b.WriteString(" by ")
b.WriteString(describeUserRef(&ev.From))
appendLine(ev.Chat.ID, renderLine(ev.Chat, who, b.String()))
}
// describeUserRef renders a value-type User reference.
func describeUserRef(u *echotron.User) string {
if u == nil {
return "user unknown"
}
if desc := describeUser(u); desc != "" {
return "user " + itoa(u.ID) + " (" + desc + ")"
}
return "user " + itoa(u.ID)
}
// orUnknown renders a status, or a placeholder when Telegram sent none.
func orUnknown(status string) string {
if status == "" {
return "unknown"
}
return status
}
/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

21
internal/elog/types.go Normal file
View File

@@ -0,0 +1,21 @@
package elog
import (
"sync"
"time"
)
// chatLock serialises writes for one chat so that two goroutines cannot splice
// their lines into the same file.
type chatLock struct {
mu sync.Mutex
// refs counts goroutines holding or waiting for this entry. It is only
// ever read or written while registryMu is held, which is what makes it a
// usable proof of "nobody is holding this" during eviction.
refs int
lastUsed time.Time
}
/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

148
internal/elog/write.go Normal file
View File

@@ -0,0 +1,148 @@
package elog
import (
"os"
"path/filepath"
"time"
"bunny-bot/internal/mylog"
)
// acquire returns the chat's lock with this goroutine holding a reference.
func acquire(chatID int64) *chatLock {
registryMu.Lock()
l, ok := locks[chatID]
if !ok {
l = &chatLock{}
locks[chatID] = l
}
l.refs++
l.lastUsed = time.Now()
evictIdleLocked()
registryMu.Unlock()
l.mu.Lock()
return l
}
// release drops this goroutine's reference and unlocks the chat.
func release(l *chatLock) {
l.mu.Unlock()
registryMu.Lock()
l.refs--
l.lastUsed = time.Now()
registryMu.Unlock()
}
// evictIdleLocked drops registry entries nobody is holding and that have been
// idle for longer than chatLockIdleTimeout.
//
// Safety rests on refs: it is only ever changed while registryMu is held, so
// observing refs == 0 here is a proof that no goroutine holds the entry, not a
// guess. Evicting a held entry would let a second writer take a *different*
// mutex for the same chat, which is precisely the interleaving the lock exists
// to prevent. The caller must hold registryMu.
func evictIdleLocked() {
cutoff := time.Now().Add(-chatLockIdleTimeout)
for id, l := range locks {
if l.refs == 0 && l.lastUsed.Before(cutoff) {
delete(locks, id)
}
}
}
// appendLine writes one line to the chat's log for the day containing now.
//
// The whole protocol is set_mutex -> MkdirAll -> open|create -> write -> close,
// with unset_mutex deferred. Nothing is held open between lines.
func appendLine(chatID int64, line string) {
// One read of both the root and whether logging is on, so a Close racing
// this call can never leave us with an empty root and a relative path.
dir, on := chatLogRoot()
if !on {
return
}
lock := acquire(chatID)
defer release(lock)
// One clock for both the path and the line: using Message.Date for one and
// time.Now() for the other would let a line carry a date its own path
// disagrees with.
now := nowUTC()
dayDir := filepath.Join(dir, itoa(chatID), now.Format("2006"), now.Format("01"))
path := filepath.Join(dayDir, now.Format("02")+".txt")
// MkdirAll creates every missing parent, so this covers the chat directory
// and the year too. No need to pre-create anything.
if err := os.MkdirAll(dayDir, logDirPerm); err != nil {
reportFailure(err)
return
}
f, err := os.OpenFile(path, os.O_CREATE|os.O_APPEND|os.O_WRONLY, logFilePerm)
if err != nil {
reportFailure(err)
return
}
_, writeErr := f.WriteString(line + "\n")
closeErr := f.Close()
if writeErr != nil {
reportFailure(writeErr)
return
}
if closeErr != nil {
reportFailure(closeErr)
return
}
reportRecovered()
}
// reportFailure sends a write failure to the main log, but only the first one
// until a write succeeds again. A full disk would otherwise produce one error
// per chat event and drown the very log meant to explain it.
func reportFailure(err error) {
stateMu.Lock()
already := failing
failing = true
stateMu.Unlock()
if already {
return
}
mylog.Errorf("CHAT LOG WRITE FAILED, further failures suppressed until a write succeeds: %s", err)
}
// reportRecovered clears the suppression after a successful write.
func reportRecovered() {
stateMu.Lock()
was := failing
failing = false
stateMu.Unlock()
if was {
mylog.Info("Chat logging recovered")
}
}
/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

28
internal/moon/animation.go
View File

@@ -1,28 +0,0 @@
package moon
import (
"fmt"
"github.com/NicoNex/echotron/v3"
)
// animation parses message with animation.
func animation(msg *echotron.Update) {
if Cfg.LogChats {
ChatLog(
msg.Message.Chat.ID,
fmt.Sprintf(
"User %s %s (username = %s, id = %d) sent animation to chat %s %s (%d)",
msg.Message.From.FirstName,
msg.Message.From.LastName,
msg.Message.From.Username,
msg.Message.From.ID,
msg.Message.Chat.Type,
msg.Message.Chat.Title,
msg.Message.Chat.ID,
),
)
}
}
/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

100
internal/moon/casChatAPI.go
View File

@@ -1,100 +0,0 @@
package moon
import (
"encoding/json"
"fmt"
"io"
"net/http"
"time"
"github.com/hjson/hjson-go"
)
// CasCheckID check given telegram user id vs CAS spammers base.
// https://api.cas.chat/check?user_id=821871410
func CasCheckID(id int64) (bool, error) {
var (
result bool
err error
url = fmt.Sprintf("https://api.cas.chat/check?user_id=%d", id)
c = &http.Client{
Timeout: 15 * time.Second,
}
banned CasTrue
notBanned CasFalse
)
resp, err := c.Get(url) //nolint: bodyclose, noctx
if err != nil {
return result, fmt.Errorf("unable to query api.cas.chat, %w", err)
}
// Looks like we should clse body as it it recommends in docs: https://pkg.go.dev/net/http .
defer func(Body io.ReadCloser) {
err := Body.Close()
if err != nil {
Log.Errorf("Unable to close response body for request to %s: %s", url, err)
}
}(resp.Body)
// Client should not return with http status code other than 200.
if resp.StatusCode > 200 {
return result, fmt.Errorf( //nolint: err113
"api.cas.chat returns non-200 http status code: %d %s",
resp.StatusCode, resp.Status)
}
body, err := io.ReadAll(resp.Body)
if err != nil {
return result, fmt.Errorf("unable to read response from api.cas.chat, %w", err)
}
// looks like go native json decoder is pretty shitty, so we'll take hjson as less restrictive parser, parse json
// encode it back to struct, then parse struct as encode/json
var tmp map[string]any
err = hjson.Unmarshal(body, &tmp)
// Не удалось распарсить.
if err != nil {
return result, fmt.Errorf("unable to parse response from api.cas.chat via hjson: %w", err)
}
tmpjson, err := json.Marshal(tmp)
// Не удалось преобразовать map-ку в json.
if err != nil {
return result, fmt.Errorf(
"unable to parse back to json response from api.cas.chat parsed by hjson: %w", err)
}
if err := json.Unmarshal(tmpjson, &notBanned); err == nil {
// if ok field in json == false, then user is not in ban. (records about id not found)
if !notBanned.Ok {
return false, err
}
} // At this point we just ignore err != nil, because it *can be* other json type.
if err := json.Unmarshal(body, &banned); err == nil {
// if ok field in json == true, then user is banned. (records about id are found)
if banned.Ok {
Log.Debugf(
"api.cas.chat reported user with id=%d as banned: offences=%d ban_time=%s",
id,
banned.Result.Offenses,
banned.Result.TimeAdded,
)
return true, err
}
return result, fmt.Errorf("api.cas.chat returns strange structure: %s", string(body)) //nolint: err113
} // At this point we can't ignore err != nil because we do not know (yet?) other forms of json reply from cas api.
return result, fmt.Errorf("unable to parse response from api.cas.chat: %s", string(body)) //nolint: err113
}
/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

208
internal/moon/chatMemberUpdated.go
View File

@@ -1,208 +0,0 @@
package moon
import (
"fmt"
"slices"
"time"
"github.com/NicoNex/echotron/v3"
)
// newChatMember parses one of types ChatMemberUpdated event.
// https://core.telegram.org/bots/api#chatmemberupdated
func chatMemberUpdated(msg *echotron.Update) {
AddChat(&msg.ChatMember.Chat)
AddUser(msg.ChatMember.NewChatMember.User)
Log.Infof(
"User %s %s (username = %s, id = %d) in chat %s %s (%d) now %s was %s",
msg.ChatMember.NewChatMember.User.FirstName,
msg.ChatMember.NewChatMember.User.LastName,
msg.ChatMember.NewChatMember.User.Username,
msg.ChatMember.NewChatMember.User.ID,
msg.ChatMember.Chat.Type,
msg.ChatMember.Chat.Title,
msg.ChatMember.Chat.ID,
msg.ChatMember.NewChatMember.Status,
msg.ChatMember.OldChatMember.Status,
)
Log.Debugf(
"Add %s %s (%d) to list of known chats",
msg.ChatMember.Chat.Type,
msg.ChatMember.Chat.Title,
msg.ChatMember.Chat.ID,
)
ChatList = append(ChatList, msg.ChatMember.Chat.ID)
slices.Sort(ChatList)
ChatList = slices.Compact(ChatList)
// Check some other attributes.
switch {
case msg.ChatMember.OldChatMember.Status == "left" && msg.ChatMember.NewChatMember.Status == "member":
Log.Infof(
"Getting info about user %s %s (username = %s, id = %d) in chat %s %s (%d) from LeftMembers DB",
msg.ChatMember.NewChatMember.User.FirstName,
msg.ChatMember.NewChatMember.User.LastName,
msg.ChatMember.NewChatMember.User.Username,
msg.ChatMember.NewChatMember.User.ID,
msg.ChatMember.Chat.Type,
msg.ChatMember.Chat.Title,
msg.ChatMember.Chat.ID,
)
leftTimestampString := Cfg.GetValue(
"LeftMembers",
fmt.Sprintf("%d", msg.ChatMember.Chat.ID),
fmt.Sprintf("%d", msg.ChatMember.NewChatMember.User.ID),
)
// Yes, this member definitely left chat at some point, we have no question to him/her.
if leftTimestampString != "" {
Log.Infof(
"User %s %s (username = %s, id = %d) in chat %s %s (%d) found in from LeftMembers DB",
msg.ChatMember.NewChatMember.User.FirstName,
msg.ChatMember.NewChatMember.User.LastName,
msg.ChatMember.NewChatMember.User.Username,
msg.ChatMember.NewChatMember.User.ID,
msg.ChatMember.Chat.Type,
msg.ChatMember.Chat.Title,
msg.ChatMember.Chat.ID,
)
return
}
Log.Infof(
"User %s %s (username = %s, id = %d) in chat %s %s (%d) not found in LeftMembers DB",
msg.ChatMember.NewChatMember.User.FirstName,
msg.ChatMember.NewChatMember.User.LastName,
msg.ChatMember.NewChatMember.User.Username,
msg.ChatMember.NewChatMember.User.ID,
msg.ChatMember.Chat.Type,
msg.ChatMember.Chat.Title,
msg.ChatMember.Chat.ID,
)
// Spam bots come with these attributes.
// msg.ChatMember.NewChatMember.Status = "member"
// msg.ChatMember.OldChatMember.User != nil
// msg.ChatMember.OldChatMember.Status = "left"
casBanned, err := CasCheckID(msg.ChatMember.NewChatMember.User.ID)
if err != nil {
Log.Errorf(
"Unable to check user %s %s (username = %s, id = %d) in chat %s %s (%d): %s",
msg.ChatMember.NewChatMember.User.FirstName,
msg.ChatMember.NewChatMember.User.LastName,
msg.ChatMember.NewChatMember.User.Username,
msg.ChatMember.NewChatMember.User.ID,
msg.ChatMember.Chat.Type,
msg.ChatMember.Chat.Title,
msg.ChatMember.Chat.ID,
err,
)
} else if casBanned {
// Ban user if cas report it as banned.
Log.Infof(
"Ban user %s %s (username = %s, id = %d) in chat %s %s (%d) by CAS blacklist",
msg.ChatMember.NewChatMember.User.FirstName,
msg.ChatMember.NewChatMember.User.LastName,
msg.ChatMember.NewChatMember.User.Username,
msg.ChatMember.NewChatMember.User.ID,
msg.ChatMember.Chat.Type,
msg.ChatMember.Chat.Title,
msg.ChatMember.Chat.ID,
)
squash(
msg.ChatMember.Chat.ID,
msg.ChatMember.NewChatMember.User.ID,
)
return
}
Log.Infof(
"Storing info about user %s %s (username = %s, id = %d) in chat %s %s (%d) to AppearedMember DB",
msg.ChatMember.NewChatMember.User.FirstName,
msg.ChatMember.NewChatMember.User.LastName,
msg.ChatMember.NewChatMember.User.Username,
msg.ChatMember.NewChatMember.User.ID,
msg.ChatMember.Chat.Type,
msg.ChatMember.Chat.Title,
msg.ChatMember.Chat.ID,
)
key := fmt.Sprintf("%d+%d", msg.ChatMember.Chat.ID, msg.ChatMember.NewChatMember.User.ID)
AppearedMembers.Set(key, time.Now().Unix(), 1*time.Minute)
PendingCASMembers.Set(key, time.Now().Unix(), 30*time.Minute)
case msg.ChatMember.OldChatMember.Status == "member" && msg.ChatMember.NewChatMember.Status == "kicked":
Log.Infof(
"User %s %s (username = %s, id = %d) in chat %s %s (%d) was banned",
msg.ChatMember.NewChatMember.User.FirstName,
msg.ChatMember.NewChatMember.User.LastName,
msg.ChatMember.NewChatMember.User.Username,
msg.ChatMember.NewChatMember.User.ID,
msg.ChatMember.Chat.Type,
msg.ChatMember.Chat.Title,
msg.ChatMember.Chat.ID,
)
key := fmt.Sprintf("%d+%d", msg.ChatMember.Chat.ID, msg.ChatMember.NewChatMember.User.ID)
Log.Debugf(
"Put user %s %s (username = %s, id = %d) in %s %s (%d) into SquashedMembers to debounce ban.",
msg.ChatMember.NewChatMember.User.FirstName,
msg.ChatMember.NewChatMember.User.LastName,
msg.ChatMember.NewChatMember.User.Username,
msg.ChatMember.NewChatMember.User.ID,
msg.ChatMember.Chat.Type,
msg.ChatMember.Chat.Title,
msg.ChatMember.Chat.ID,
)
SquashedMembers.Set(key, true, 30*time.Second)
Log.Debugf(
"Storing info about user %s %s (username = %s, id = %d) in chat %s %s (%d) to BannedMembers DB",
msg.ChatMember.NewChatMember.User.FirstName,
msg.ChatMember.NewChatMember.User.LastName,
msg.ChatMember.NewChatMember.User.Username,
msg.ChatMember.NewChatMember.User.ID,
msg.ChatMember.Chat.Type,
msg.ChatMember.Chat.Title,
msg.ChatMember.Chat.ID,
)
if err := Cfg.SaveKeyValue(
"BannedMembers",
fmt.Sprintf("%d", msg.ChatMember.Chat.ID),
fmt.Sprintf("%d", msg.ChatMember.NewChatMember.User.ID),
fmt.Sprintf("%d", time.Now().Unix()),
); err != nil {
Log.Error(err)
} else {
Log.Infof("Info about banned userid %d saved to BannedMembers DB", msg.ChatMember.Chat.ID)
}
default:
Log.Infof(
"User %s %s (username = %s, id = %d) in chat %s %s (%d) now %s was %s",
msg.ChatMember.NewChatMember.User.FirstName,
msg.ChatMember.NewChatMember.User.LastName,
msg.ChatMember.NewChatMember.User.Username,
msg.ChatMember.NewChatMember.User.ID,
msg.ChatMember.Chat.Type,
msg.ChatMember.Chat.Title,
msg.ChatMember.Chat.ID,
msg.ChatMember.NewChatMember.Status,
msg.ChatMember.OldChatMember.Status,
)
}
}
/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

49
internal/moon/chatlog.go
View File

@@ -1,49 +0,0 @@
package moon
import (
"fmt"
"os"
"path/filepath"
"strconv"
"time"
"github.com/rs/zerolog"
"github.com/rs/zerolog/log"
)
// ChatLog logs message to separate file, placed in basedir(Cfg.LogFile).
func ChatLog(chatid int64, message string) {
var (
filename string
)
if Cfg.LogFile != "" {
filename = fmt.Sprintf("%s/%s", filepath.Dir(Cfg.LogFile), strconv.FormatInt(chatid, 10))
} else {
filename = os.DevNull
}
f, err := os.OpenFile(filename, os.O_APPEND|os.O_WRONLY|os.O_CREATE, 0644)
if err != nil {
fmt.Printf("Unable to open file %s for logging: %s", filename, err)
// TODO: Should we try harder?
return
}
defer f.Close()
log.Logger = log.Output(
zerolog.ConsoleWriter{
Out: f,
NoColor: true,
TimeFormat: time.DateTime,
PartsOrder: []string{"time", "message"},
},
)
log.Info().Msg(message)
}
/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

44
internal/moon/config.go
View File

@@ -5,10 +5,11 @@ import (
"fmt"
"os"
"path/filepath"
"strings"
"bunny-bot/internal/elog"
"bunny-bot/internal/mylog"
"github.com/hjson/hjson-go"
"go.uber.org/zap/zapcore"
)
// parseConfig разбирает и валидирует даденный конфиг.
@@ -66,34 +67,27 @@ func ParseConfig(path string) (MyConfig, error) {
return MyConfig{}, fmt.Errorf("token field in config file %s must be set", path) //nolint: err113
}
if _, err := zapcore.ParseLevel(sampleConfig.LogLevel); err != nil {
if _, err := mylog.ParseLevel(sampleConfig.LogLevel); err != nil {
sampleConfig.LogLevel = "info"
}
// Chat logs are optional, but a configured path that does not exist is a
// typo and must stop startup rather than silently disable logging.
//
// A relative chat_logs is resolved against the directory holding the config
// file, not the process working directory: a daemon's cwd is wherever its
// supervisor happened to start it, so a relative path there would move the
// log directory on every restart under a different supervisor.
if sampleConfig.ChatLogs != "" && !filepath.IsAbs(sampleConfig.ChatLogs) {
sampleConfig.ChatLogs = filepath.Join(filepath.Dir(path), sampleConfig.ChatLogs)
}
if err := elog.Init(sampleConfig.ChatLogs); err != nil {
return MyConfig{}, fmt.Errorf("chat_logs field in config file %s is invalid: %w", path, err)
}
// sampleConfig.Log = "" if not set
if sampleConfig.CSign == "" {
err := fmt.Errorf("csign field in config file %s must be set", path) //nolint: err113
return MyConfig{}, err
}
if sampleConfig.DataDir == "" {
return MyConfig{}, fmt.Errorf("data_dir field in config file %s must be set", path) //nolint: err113
}
if strings.Split(sampleConfig.DataDir, "")[0] != "/" {
executablePath, err := os.Executable()
if err != nil {
err := fmt.Errorf("unable to get current executable path: %w", err)
return MyConfig{}, err
}
sampleConfig.DataDir = fmt.Sprintf("%s/%s", filepath.Dir(executablePath), sampleConfig.DataDir)
}
return sampleConfig, err
}

141
internal/moon/db.go
View File

@@ -1,141 +0,0 @@
package moon
import (
"crypto/sha256"
"errors"
"fmt"
"io/fs"
"os"
"github.com/cockroachdb/errors/oserror"
"github.com/cockroachdb/pebble"
)
// StoreKV saves given key and value in given db.
func StoreKV(db *pebble.DB, key string, value string) error {
var (
kArray = []byte(key)
vArray = []byte(value)
)
err := db.Set(kArray, vArray, pebble.Sync)
if err != nil {
return err
}
err = db.Flush()
return err
}
// FetchV gets value by given key from given db.
func FetchV(db *pebble.DB, key string) (string, error) {
var (
kArray = []byte(key)
vArray []byte
valueString = ""
)
vArray, closer, err := db.Get(kArray)
if err != nil {
return valueString, err
}
valueString = string(vArray)
err = closer.Close()
return valueString, err
}
// GetValue gets setting for chat from given db name.
// TODO: should return string, error and should not log error internally.
func (cfg *MyConfig) GetValue(dataBaseName string, chatID string, setting string) string {
var err error
chatHash := sha256.Sum256([]byte(chatID))
dataBaseDir := fmt.Sprintf("%s/db/%s/%x", cfg.DataDir, dataBaseName, chatHash)
// If db descriptor is not open yet, open it.
if _, ok := DB[dataBaseDir]; !ok {
var options pebble.Options
/*
* By default limit set on megs of data, but not on amount of files thus with defaults we have tons of
* files. Default limit on tomporary files is 500, lets cut it down to 8.
*/
options.L0CompactionFileThreshold = 8
if err := os.MkdirAll(dataBaseDir, os.ModePerm); err != nil {
Log.Errorf("Unable to create db dir %s: %s", dataBaseDir, err)
return ""
}
DB[dataBaseDir], err = pebble.Open(dataBaseDir, &options)
if err != nil {
Log.Errorf("Unable to open db, %s: %s\n", dataBaseDir, err)
return ""
}
}
value, err := FetchV(DB[dataBaseDir], setting)
// If db returns nothing for some reason, just return ampty string.
if err != nil {
switch {
case errors.Is(err, pebble.ErrNotFound):
Log.Debugf("Unable to get value for %s: no record found in db %s", setting, dataBaseDir)
case errors.Is(err, fs.ErrNotExist):
Log.Debugf("Unable to get value for %s: db dir %s does not exist", setting, dataBaseDir)
case errors.Is(err, oserror.ErrNotExist):
Log.Debugf("Unable to get value for %s: db dir %s does not exist", setting, dataBaseDir)
default:
Log.Errorf("Unable to get value for %s in db dir %s: %s", setting, dataBaseDir, err)
}
return ""
}
return value
}
// SaveSetting saves setting name and its value to given db name.
func (cfg *MyConfig) SaveKeyValue(dataBaseName string, chatID string, setting string, value string) error {
var (
chatHash = sha256.Sum256([]byte(chatID))
dataBaseDir = fmt.Sprintf("%s/db/%s/%x", cfg.DataDir, dataBaseName, chatHash)
err error
)
// If db descriptor is not open yet, open it.
if _, ok := DB[dataBaseDir]; !ok {
var options pebble.Options
/*
* By default limit set on megs of data, but not on amount of files thus with defaults we have tons of
* files. Default limit on tomporary files is 500, lets cut it down to 8.
*/
options.L0CompactionFileThreshold = 8
if err := os.MkdirAll(dataBaseDir, os.ModePerm); err != nil {
return fmt.Errorf("unable to create db dir %s: %w", dataBaseDir, err)
}
DB[dataBaseDir], err = pebble.Open(dataBaseDir, &options)
if err != nil {
return fmt.Errorf("unable to open settings db, %s: %w", dataBaseDir, err)
}
}
if err := StoreKV(DB[dataBaseDir], setting, value); err != nil {
return fmt.Errorf("unable to save setting %s in database %s: %w", setting, dataBaseDir, err)
}
return nil
}
/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

50
internal/moon/editedMessage.go
View File

@@ -1,50 +0,0 @@
package moon
import (
"fmt"
"github.com/NicoNex/echotron/v3"
)
// editedMessage handles editedMessage type of update.
func editedMessage(msg *echotron.Update) {
AddUser(msg.EditedMessage.From)
AddChat(&msg.EditedMessage.Chat)
// Edited message can contain all kinds of edited attributes that telegram allows to change in message.
// Not sure is we should handle all of them.
if Cfg.LogChats {
if msg.EditedMessage.Text != "" {
ChatLog(
msg.EditedMessage.Chat.ID,
fmt.Sprintf(
"User %s %s (username = %s, id = %d) in %s %s (%d) edited message: %s",
msg.EditedMessage.From.FirstName,
msg.EditedMessage.From.LastName,
msg.EditedMessage.From.Username,
msg.EditedMessage.From.ID,
msg.EditedMessage.Chat.Type,
msg.EditedMessage.Chat.Title,
msg.EditedMessage.Chat.ID,
msg.EditedMessage.Text,
),
)
} else {
ChatLog(
msg.EditedMessage.Chat.ID,
fmt.Sprintf(
"User %s %s (username = %s, id = %d) in %s %s (%d) edited message",
msg.EditedMessage.From.FirstName,
msg.EditedMessage.From.LastName,
msg.EditedMessage.From.Username,
msg.EditedMessage.From.ID,
msg.EditedMessage.Chat.Type,
msg.EditedMessage.Chat.Title,
msg.EditedMessage.Chat.ID,
),
)
}
}
}
/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

130
internal/moon/entityGuard.go Normal file
View File

@@ -0,0 +1,130 @@
package moon
import (
"bunny-bot/internal/mylog"
"github.com/NicoNex/echotron/v3"
"github.com/davecgh/go-spew/spew"
)
// The Bot API always returns a User/Chat carrying its id, and echotron validates
// nothing: lclient.dispatch (network.go:138) is json.Unmarshal plus a check() that
// reads only the ok/error_code envelope (helpers.go:37). An entity that arrives
// without an id therefore reaches us as a silent zero, with no layer below to
// complain about it. These predicates are the whole of our defence.
//
// They are SILENT. A refusal is a fact, not an incident: reporting happens once, at
// the ingress, where the offending update can be dumped whole. A predicate that logs
// would make one broken update produce one line per entity that happens to meet it.
// usableUser reports whether a user entity is one we can actually address.
//
// The test is against zero and not against zero-or-less on purpose: a chat id is
// legitimately negative for a group or supergroup, so a positivity check would
// refuse every group chat in the project.
func usableUser(u *echotron.User) bool {
return u != nil && u.ID != 0
}
// usableChat reports whether a chat entity is one we can actually address, with the
// same reasoning and the same zero-not-positive rule.
func usableChat(c *echotron.Chat) bool {
return c != nil && c.ID != 0
}
// usableUpdate reports whether an update is worth dispatching at all.
//
// Its branches mirror telegramMsgParser's, in the same order and under the same
// conditions, so a branch added to one and not the other is visible by reading. That
// is not tidiness: the alternative is a validator shaped differently from the
// dispatcher, whose coverage nobody can check by eye.
//
// nil is tolerated only where absence is normal. A message with no author is a
// channel post or a service message, and refusing those would disable chat logging
// for channels. A nil element inside NewChatMembers is different — the array asserts
// a newcomer exists — and so is a nil NewChatMember.User, which asserts who changed.
// A nil OldChatMember.User is tolerated: some transitions have no previous member.
func usableUpdate(u *echotron.Update) bool {
if u == nil {
return false
}
switch {
case u.ChatMember != nil:
return usableChatMember(u.ChatMember)
case u.Message != nil:
return usableMessage(u.Message)
case u.EditedMessage != nil:
return usableMessage(u.EditedMessage)
default:
// Nothing we act on. The dispatcher's spew fallback is where this belongs.
return true
}
}
// usableChatMember checks a membership change.
//
// Its entities are values, not pointers: ChatMemberUpdated.Chat, .From and both
// ChatMember structs can never be nil, so there is no nil test to fall back on and
// the id check is the only thing that can catch a malformed one.
func usableChatMember(cm *echotron.ChatMemberUpdated) bool {
return usableChat(&cm.Chat) &&
usableUser(cm.NewChatMember.User) &&
(cm.OldChatMember.User == nil || usableUser(cm.OldChatMember.User)) &&
usableUser(&cm.From)
}
// usableMessage checks a message, or an edit of one — the two carry the same fields
// and are treated the same way.
//
// Message.Chat is a value: a zero chat id would otherwise reach elog, which builds
// chat_logs/<id>/yyyy/mm/dd.txt from it and would silently create a chat_logs/0
// directory and write every malformed update into it.
func usableMessage(m *echotron.Message) bool {
if !usableChat(&m.Chat) {
return false
}
if m.From != nil && !usableUser(m.From) {
return false
}
if m.LeftChatMember != nil && !usableUser(m.LeftChatMember) {
return false
}
for _, member := range m.NewChatMembers {
if !usableUser(member) {
return false
}
}
// One level deep, and the nested chat only. The nested author is never rendered,
// so a zero there is cosmetic; the level bound exists to keep the work constant.
if m.PinnedMessage != nil && !usableChat(&m.PinnedMessage.Chat) {
return false
}
if m.ReplyToMessage != nil && !usableChat(&m.ReplyToMessage.Chat) {
return false
}
return true
}
// reportBrokenUpdate records an update we are refusing, in the terms it deserves:
// this is not a user being difficult and not a transient API error, it is an
// upstream that broke its own contract.
//
// The whole update is dumped, once. The dump is the point — the condition is
// unreproducible and unqueryable, since there is no id to ask Telegram about — and a
// partial dump of one field out of four would not be evidence of anything.
func reportBrokenUpdate(u *echotron.Update) {
mylog.Error(
"Telegram sent an update with an entity carrying no id, refusing it: nothing in it can be greeted, recorded or banned, and an API that sends one is broken",
)
mylog.Error(spew.Sdump(u))
}
/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

25
internal/moon/exitCode.go Normal file
View File

@@ -0,0 +1,25 @@
package moon
import "os"
// ExitCode reports the status the process should leave with on a clean return
// from run(). It is zero while the bot is running normally, and 128 + the
// signal's number once a termination signal has begun the shutdown — the Unix
// convention a supervisor can use to tell "stopped with a signal" from "stopped
// cleanly", which D33 left permanently 0 before it.
//
// It never reports an error status: a failed shutdown still returns an error
// from run(), and main exits 1 over that, the same as before.
func ExitCode() int {
return int(exitCode.Load())
}
// recordExitCode stores the status the given terminating signal should leave.
// Only the first termination signal matters — the handler goroutine returns
// after one — so a later write cannot pass either.
func recordExitCode(s os.Signal) {
// The value is one of a fixed table of 128+n, never a caller-supplied int.
exitCode.Store(int32(exitCodeFor(s))) //nolint:gosec // G115: exitCodeFor returns only 130, 131, 143 or 0.
}
/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

28
internal/moon/exitCode_unix.go Normal file
View File

@@ -0,0 +1,28 @@
//go:build !windows
package moon
import (
"os"
"syscall"
)
// exitCodeFor maps one terminating signal to the status a stopped process
// should carry: 128 + the signal's number on Unix.
func exitCodeFor(s os.Signal) int {
switch s {
case syscall.SIGINT:
return signalExitBase + 2
case syscall.SIGTERM:
return signalExitBase + 15
case syscall.SIGQUIT:
return signalExitBase + 3
default:
return 0
}
}
/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

18
internal/moon/exitCode_windows.go Normal file
View File

@@ -0,0 +1,18 @@
//go:build windows
package moon
import "os"
// exitCodeFor maps one terminating signal to the status a stopped process
// should carry. Windows has no signal numbers to follow, so it imitates the
// Unix convention: Ctrl+C is an interrupt, and an interrupt is 130.
func exitCodeFor(s os.Signal) int {
if s == os.Interrupt {
return signalExitBase + 2
}
return 0
}
/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

139
internal/moon/globals.go
View File

@@ -2,48 +2,143 @@ package moon
import (
"os"
"strings"
"sync"
"sync/atomic"
"time"
"github.com/NicoNex/echotron/v3"
"github.com/akyoto/cache"
"github.com/cockroachdb/pebble"
"go.uber.org/zap"
)
const (
// greetCacheTTL is how long a join event or a chat message is retained. It
// is deliberately larger than greetReplyWindow so that a sweep occurring at
// the far end of the window still has the data it needs to decide.
greetCacheTTL = 17 * time.Minute
// greetReplyWindow is how long a newcomer has to answer the greeting before
// being treated as a spammer.
greetReplyWindow = 15 * time.Minute
// greetSweepInterval is how often the sweeper runs.
greetSweepInterval = 1 * time.Minute
)
// signalExitBase is the offset a signal's exit status is derived from. The Unix
// convention is 128 + n, where n is the signal number, so SIGINT is 130, SIGTERM
// is 143 and SIGQUIT is 131 — what a shell reports for a process killed by a
// signal, which is exactly what these statuses imitate.
const signalExitBase = 128
// The outcomes of a squash attempt. Names state what happened, not how it was
// detected, so callers never have to reinterpret them.
const (
// banApplied means Telegram accepted the ban.
banApplied banOutcome = iota
// banSuppressed means we did not call the API at all: the debounce already
// holds this user, i.e. we banned them moments ago and they are already out.
banSuppressed
// banFailed means we called the API and it refused. The user is still in the
// chat and was not retried.
banFailed
)
// The outcomes of retracting a greeting a banned member never answered. The three
// cases are distinct because "nothing to delete" and "tried and failed" call for
// different log lines, and only the second is worth an operator's attention.
const (
// nothingToRetract means no remembered greeting id was greater than zero, so
// the API was never called.
nothingToRetract retractOutcome = iota
// greetingRetracted means every remembered greeting was deleted.
greetingRetracted
// greetingRetractFailed means at least one deletion was refused by Telegram.
// The others still went: a failure never aborts the rest.
greetingRetractFailed
)
var (
// htmlReplacer escapes the three characters HTML requires, all three pairs at
// once so no ordering can be got wrong: "&" is never re-escaped by the "<" or
// ">" rules. Created once because a Replacer is immutable and safe to share.
htmlReplacer = strings.NewReplacer("&", "&amp;", "<", "&lt;", ">", "&gt;")
Cfg MyConfig
SigChan = make(chan os.Signal, 1)
Log *zap.SugaredLogger
Tg echotron.API
Tg tgClient
// Мапка с открытыми дескрипторами баз настроек.
DB = make(map[string]*pebble.DB)
// allowedUpdates is the set of update types Telegram is asked to deliver.
//
// It lives here, not inline in Telegram(), because it must stay in step with
// the branches of telegramMsgParser: a type missing from this list makes its
// branch unreachable, and nothing at compile time or in the test suite says
// so. TestEveryDispatchedUpdateTypeIsRequested exists to catch exactly that.
//
// chat_join_request is deliberately absent. The community is open and join
// requests are not moderated; Telegram auto-approves them when the type is
// not requested, which is the intended behaviour.
allowedUpdates = []echotron.UpdateType{
"message",
"edited_message",
"chat_member",
}
// SquashedMembers struct with banned members.
// SquashedMembers holds members recently banned by us, used as a ban debounce.
SquashedMembers = cache.New(1 * time.Minute)
// NewMembers struct with new members appeared via message with new_chat_members field set.
NewMembers = cache.New(1 * time.Minute)
// PendingGreetings holds recorded joins keyed by join timestamp in
// microseconds. Used by the greeting sweeper to find newcomers who never
// answered, and to retract the greetings we asked them.
PendingGreetings = newEventCache[pendingGreeting](1 * time.Minute)
// AppearedMembers struct with new members appeared via ChatMemberUpdated event.
AppearedMembers = cache.New(60 * time.Minute)
// GreetAnswers holds text messages keyed by message timestamp in
// microseconds. Used by the greeting sweeper to find proof that a newcomer
// answered the greeting.
GreetAnswers = newEventCache[*echotron.Message](1 * time.Minute)
// PendingCASMembers contains newly comed members that pass first cas check, but still waiting second cas check.
PendingCASMembers = cache.New(60 * time.Minute)
// closeOnce guards closeCaches: cache.Cache.Close() sends on an unbuffered
// channel and therefore must be called exactly once per cache; calling it
// twice deadlocks. Each cache is guarded so a failure partway through cannot
// turn into a hang on the second attempt.
closeOnce sync.Once
// GreetPending contains people that should answer to bot greet.
GreetPending = cache.New(1 * time.Minute)
// stopRequested is closed once, by shutdown, to ask every goroutine that
// loops to return. One signal for all of them, rather than one per worker:
// several workers can come and go (and tests start them deliberately), and
// a per-worker channel would have to be closed exactly once each, which is a
// panic waiting to happen.
stopRequested = make(chan struct{})
// NoMessagePeople contains people that are never say something in chat.
NoMessagePeople = cache.New(1 * time.Minute)
// workers counts the running background goroutines — the update listener and
// the greeting sweeper. shutdown waits on it, so no worker can be touching a
// cache, the logs or the API while the teardown closes them.
workers sync.WaitGroup
// People cache with users info.
People = cache.New(8 * time.Hour)
// workersMu guards workers and stopping. A worker must register before
// shutdown starts waiting, and a goroutine that tries to start afterwards is
// refused instead of racing WaitGroup.Add against Wait.
workersMu sync.Mutex
stopping bool
// Groups cache with groups info.
Groups = cache.New(8 * time.Hour)
// runningWorkers counts registered, not-yet-finished workers, so a caller
// can tell "nothing started yet" from "started and still going".
runningWorkers atomic.Int32
ChatList = make([]int64, 0)
// exitCode carries the status the process should exit with on a clean
// return from run(): zero while the bot is running and sleeping normally,
// 128 + signal number once a termination signal has begun the shutdown.
// It is an atomic because the handler goroutine writes it and main reads
// it, and plain avoids any question of which goroutine wins.
exitCode atomic.Int32
// shutdownOnce guards shutdown so two termination signals arriving together
// cannot run the sequence twice.
shutdownOnce sync.Once
)
/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

98
internal/moon/greet.go Normal file
View File

@@ -0,0 +1,98 @@
package moon
import (
"bunny-bot/internal/mylog"
"math/rand/v2"
"strings"
"github.com/NicoNex/echotron/v3"
)
// greetings are the templates used to greet a newly joined member. The single
// verb is replaced with the newcomer's computed display name. The fourth one is
// intentionally kept verbatim despite looking like a typo.
var greetings = []string{
"Дарована, %s. Представьтес, пожалуйста, и расскажите, что вас сюда привело.",
"Добро пожаловать в наше скромное коммунити, %s. Представьтес, пожалуйста, и расскажите, что вас сюда привело.",
"Доброе утро, день или вечер, %s. Представьтес, пожалуйста, и расскажите, что вас сюда привело.",
"Наше вам с кисточкой тут, на канальчике, %s. Представьтес, пожалуйста, и расскажите, что вас сюда привело.",
}
// displayName builds the name substituted into a greeting for the given user.
// Preference order: "FirstName LastName", then LastName alone, then FirstName
// alone, then the username.
func displayName(u *echotron.User) string {
if u == nil {
return ""
}
name := strings.TrimSpace(strings.Join([]string{u.FirstName, u.LastName}, " "))
if name != "" {
return name
}
return u.Username
}
// sendGreeting sends a randomly chosen greeting for the given newcomer into the chat.
//
// It returns the id of the greeting it sent and whether it was actually
// delivered. Both matter: a newcomer who never received the question must not be
// judged on their silence, and a newcomer who is banned later must have the
// question taken back down.
//
// The mention makes the greeting a clickable link to the newcomer, and the
// message is sent in HTML mode because that is the only mode in which an
// arbitrary display name can be embedded safely — see greetMention.
func sendGreeting(chatid int64, u *echotron.User) (int, bool) {
// Everything below dereferences u, including the error paths that name the
// user, so an entity with no id is refused here rather than crashing later.
if !usableUser(u) {
return 0, false
}
// math/rand is sufficient here: we pick one of four static templates, not a
// secret or a security-relevant value.
greeting := greetings[rand.IntN(len(greetings))] //nolint: gosec // G404
text := strings.Replace(greeting, "%s", greetingAddressee(u), 1)
res, err := Tg.SendMessage(text, chatid, &echotron.MessageOptions{
ParseMode: echotron.HTML,
})
if err != nil {
mylog.Errorf("GREETING NOT DELIVERED to user %d in chat %d: %s", u.ID, chatid, err)
return 0, false
}
mylog.Debugf("Greeted user %d in chat %d", u.ID, chatid)
// A response carrying no message still means the greeting was delivered; we
// simply have no id to retract later.
return sentMessageID(res), true
}
// greetingAddressee is the name the template's %s is replaced with: a mention of
// the newcomer when they can be mentioned, and their plain display name when they
// cannot, so an unmentionable user is still greeted rather than skipped.
func greetingAddressee(u *echotron.User) string {
if mention := greetMention(u); mention != "" {
return mention
}
return displayName(u)
}
// sentMessageID pulls the id out of a send response, tolerating a response that
// carries no message at all.
func sentMessageID(res echotron.APIResponseMessage) int {
if res.Result == nil {
return 0
}
return res.Result.ID
}
/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

52
internal/moon/greetCache.go Normal file
View File

@@ -0,0 +1,52 @@
package moon
import (
"time"
"github.com/akyoto/cache"
)
// newEventCache creates an eventCache whose expired entries are swept every
// cleaningInterval.
func newEventCache[T any](cleaningInterval time.Duration) *eventCache[T] {
return &eventCache[T]{cache: cache.New(cleaningInterval)}
}
// Set stores value under the microsecond key with the given TTL.
func (m *eventCache[T]) Set(micros int64, value T, ttl time.Duration) {
m.cache.Set(micros, value, ttl)
}
// Snapshot returns a typed copy of all non-expired entries keyed by their
// microsecond timestamp. Callers can range over the returned map without
// performing any interface{} conversions.
func (m *eventCache[T]) Snapshot() map[int64]T {
out := make(map[int64]T)
m.cache.Range(
func(k, v any) bool {
key, ok := k.(int64)
if !ok {
return true
}
value, ok := v.(T)
if !ok {
return true
}
out[key] = value
return true
},
)
return out
}
// Delete removes the entry stored under the given microsecond key.
func (m *eventCache[T]) Delete(micros int64) {
m.cache.Delete(micros)
}
/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

81
internal/moon/greetMention.go Normal file
View File

@@ -0,0 +1,81 @@
package moon
import (
"strconv"
"strings"
"github.com/NicoNex/echotron/v3"
)
// greetMention renders a clickable mention of a newcomer for use inside a
// greeting, or "" when the user cannot be mentioned at all.
//
// The mode is HTML, not markdown, and that is a correctness decision rather than
// a preference: the label is a name Telegram gave us, and MarkdownV2 requires
// escaping eighteen characters — "-" among them — so an ordinary "Анна-Мария"
// would be rejected with a 400. A rejected greeting makes newChatMembers delete
// the join event, which is how one hyphen would disable the only detector. HTML
// needs "&", "<" and ">" escaped, and those essentially never occur in a name.
//
// The mention works without a @username, which is the common case for someone who
// has only just joined a public group.
func greetMention(u *echotron.User) string {
if u == nil || u.ID <= 0 {
return ""
}
var b strings.Builder
b.WriteString(`<a href="tg://user?id=`)
b.WriteString(strconv.FormatInt(u.ID, 10))
b.WriteString(`">`)
for _, r := range mentionLabel(u) {
switch r {
case '&':
b.WriteString("&amp;")
case '<':
b.WriteString("&lt;")
case '>':
b.WriteString("&gt;")
default:
b.WriteRune(r)
}
}
b.WriteString("</a>")
return b.String()
}
// escapeHTML escapes the three characters HTML requires. It exists for callers
// that assemble markup themselves; the greeting path uses greetMention's own
// single pass so the anchor and the label cannot drift apart.
func escapeHTML(s string) string {
return htmlReplacer.Replace(s)
}
// mentionLabel is the text shown inside the mention link, in the preference order
// a reader expects: full name, then either name alone, then the @username, and
// finally the bare id so the link is never empty.
func mentionLabel(u *echotron.User) string {
if name := strings.TrimSpace(u.FirstName + " " + u.LastName); name != "" {
return name
}
if u.LastName != "" {
return u.LastName
}
if u.FirstName != "" {
return u.FirstName
}
if u.Username != "" {
return "@" + u.Username
}
return strconv.FormatInt(u.ID, 10)
}
/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

206
internal/moon/greetSweep.go Normal file
View File

@@ -0,0 +1,206 @@
package moon
import (
"bunny-bot/internal/mylog"
"time"
"github.com/NicoNex/echotron/v3"
)
// SweepGreetings runs until a termination signal stops it. Every
// greetSweepInterval it looks at join events that are at least greetReplyWindow
// old and decides whether each newcomer ever answered the greeting:
//
// - answered (a message from that user exists after the join, within the
// reply window) -> newcomer is legitimate, drop the join event(s);
// - not answered -> newcomer is a spammer, ban them and drop the join
// event(s).
//
// A user may appear in PendingGreetings more than once (rejoins). For every user we
// re-query both caches so that a single answer rescues all of their join events
// and a single unanswered join bans them once.
//
// The stop is checked between passes and never during one, so a sweep in
// progress always finishes and can never be running while the caches are being
// closed underneath it.
func SweepGreetings() {
if !startWorker() {
return
}
defer stopWorker()
timer := time.NewTimer(greetSweepInterval)
defer timer.Stop()
for {
select {
case <-stopRequested:
mylog.Info("Greeting sweeper stopped")
return
case <-timer.C:
sweepGreetings()
timer.Reset(greetSweepInterval)
}
}
}
// sweepGreetings performs a single sweep pass against the current wall clock.
func sweepGreetings() {
sweepGreetingsAt(time.Now())
}
// sweepGreetingsAt performs a single sweep pass as of now. Taking the clock as
// a parameter is what makes the reply-window boundary testable without sleeping
// for 15 minutes.
func sweepGreetingsAt(now time.Time) {
joins := PendingGreetings.Snapshot()
messages := GreetAnswers.Snapshot()
windowMicros := greetReplyWindow.Microseconds()
nowMicros := now.UnixMicro()
// Group stale join events per (chat, user). A user is a distinct identity in
// every chat: being silent in one chat says nothing about another, and
// answering in one chat must not rescue them in another.
staleByMember := make(map[chatUser][]joinRef)
for joinMicros, recorded := range joins {
join := recorded.join
// Not stale yet.
if nowMicros-joinMicros <= windowMicros {
continue
}
// A join event carries the newcomer's user in NewChatMembers.
if join == nil || len(join.NewChatMembers) == 0 {
continue
}
user := join.NewChatMembers[0]
// An entity with no id cannot be banned, and the dump has already been
// taken. Drop the join rather than keeping it: re-reporting the same
// broken update every minute for the rest of its TTL is noise that
// teaches an operator to ignore this log.
if !usableUser(user) {
PendingGreetings.Delete(joinMicros)
continue
}
member := chatUser{chatID: join.Chat.ID, userID: user.ID}
staleByMember[member] = append(staleByMember[member], joinRef{
micros: joinMicros,
chat: join.Chat,
user: user,
greetingID: recorded.greetingID,
})
}
for member, refs := range staleByMember {
// A single answer rescues every join this user has *in this chat*, no
// matter which join the reply was actually answering.
answered := false
for _, ref := range refs {
if userAnswered(member.chatID, member.userID, ref.micros, windowMicros, messages) {
answered = true
break
}
}
if answered {
mylog.Debugf(
"User %d answered the greeting in chat %d, not banning",
member.userID,
member.chatID,
)
} else {
mylog.Infof(
"User %d did not answer the greeting in chat %d within %s, banning as spammer",
member.userID,
member.chatID,
greetReplyWindow,
)
// squash already logs every outcome at the level it deserves: a
// ban that landed is a debug line and a duplicate we deliberately
// suppressed is an info line. Only a refused ban needs anything
// added here, because it is the one case where the user is still
// in the chat and will never be re-judged.
outcome := squash(&refs[0].chat, refs[0].user)
if outcome == banFailed {
mylog.Errorf(
"BAN FAILED for user %d in chat %d — they stayed in the chat and were not re-judged",
member.userID,
member.chatID,
)
} else {
// The user is out of the chat, so every question we asked them
// is a leftover. That includes the debounced repeat: they are
// banned either way, and the room must not be left asking a
// question nobody can answer.
retractAskedGreetings(refs)
}
}
// Drop every join event of this user *in this chat* so the next pass
// does not re-decide them. Joins in their other chats are untouched.
forgetPendingGreetings(member.chatID, member.userID)
}
}
// userAnswered reports whether a message authored by userID in chatID exists
// in messages with a timestamp in (joinMicros, joinMicros+windowMicros].
//
// Both the chat and the author must match: the newcomer must answer in the same
// chat they were greeted in. Messages sent before the join do not count either,
// because the greeting is only sent in response to the join.
func userAnswered(
chatID int64,
userID int64,
joinMicros int64,
windowMicros int64,
messages map[int64]*echotron.Message,
) bool {
for msgMicros, msg := range messages {
if msg == nil || msg.From == nil {
continue
}
if msg.From.ID != userID || msg.Chat.ID != chatID {
continue
}
if msgMicros > joinMicros && msgMicros <= joinMicros+windowMicros {
return true
}
}
return false
}
// forgetPendingGreetings removes every PendingGreetings entry whose newcomer is userID in the
// given chat. Joins by the same user in other chats are deliberately preserved:
// those are independent memberships with their own greetings and their own
// verdicts.
func forgetPendingGreetings(chatID int64, userID int64) {
for joinMicros, recorded := range PendingGreetings.Snapshot() {
join := recorded.join
if join == nil || join.Chat.ID != chatID || len(join.NewChatMembers) == 0 {
continue
}
if join.NewChatMembers[0] != nil && join.NewChatMembers[0].ID == userID {
PendingGreetings.Delete(joinMicros)
}
}
}
/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

37
internal/moon/leftChatMember.go
View File

@@ -1,37 +0,0 @@
package moon
import (
"fmt"
"time"
"github.com/NicoNex/echotron/v3"
)
// leftChatMember handles left_chat_member event.
func leftChatMember(msg *echotron.Update) {
Log.Infof(
"User %s %s (username = %s, id = %d) left %s %s (%d)",
msg.Message.LeftChatMember.FirstName,
msg.Message.LeftChatMember.LastName,
msg.Message.LeftChatMember.Username,
msg.Message.LeftChatMember.ID,
msg.Message.Chat.Type,
msg.Message.Chat.Title,
msg.Message.Chat.ID,
)
// Save people that left chat to pebbledb. To preserve their state.
// Left folks can eventually return and in chat_mamber_updated event their state will be left->member.
if err := Cfg.SaveKeyValue(
"LeftMembers",
fmt.Sprintf("%d", msg.Message.Chat.ID),
fmt.Sprintf("%d", msg.Message.LeftChatMember.ID),
fmt.Sprintf("%d", time.Now().Unix()),
); err != nil {
Log.Error(err)
} else {
Log.Infof("Info about left userid %d saved to left users db", msg.Message.LeftChatMember.ID)
}
}
/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

24
internal/moon/logOnlyMessage.go Normal file
View File

@@ -0,0 +1,24 @@
package moon
import (
"bunny-bot/internal/elog"
"github.com/NicoNex/echotron/v3"
)
// logOnlyMessage writes a message to the chat log and nothing else.
//
// It handles the messages the greeting detector must NOT count as answers: a photo with no
// caption, a sticker, a location, a leave, a title change. They are real events in the room
// and belong in the log — which is also what makes a later ban diagnosable, since a moderator
// can read what the member actually replied with — but a reply with no text is not an
// introduction, and recording it in GreetAnswers would let any attachment rescue a spammer
// from the ban.
//
// The dispatcher has already established that this is neither a join nor a text message; it
// is not re-checked here.
func logOnlyMessage(msg *echotron.Update) {
elog.LogMessage(msg.Message)
}
/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

121
internal/moon/newChatMembers.go
View File

@@ -1,31 +1,61 @@
package moon
import (
"fmt"
"slices"
"time"
"bunny-bot/internal/elog"
"bunny-bot/internal/mylog"
"github.com/NicoNex/echotron/v3"
)
// newChatMembers parses NewChatMember event.
func newChatMembers(msg *echotron.Update) {
// If there is no such chatid, means that no such cache too.
Log.Debugf(
"Add %s %s (%d) to list of known chats",
msg.Message.Chat.Type,
msg.Message.Chat.Title,
// Telegram normally reports one newcomer per service message. If that ever
// stops being true, the sweeper would only judge the first member, so make it
// visible in the logs rather than silently ignoring the rest.
if len(msg.Message.NewChatMembers) > 1 {
mylog.Warnf(
"MULTI-MEMBER JOIN in chat %d: %d newcomers in one event, only the first is moderated",
msg.Message.Chat.ID,
len(msg.Message.NewChatMembers),
)
}
ChatList = append(ChatList, msg.Message.Chat.ID)
slices.Sort(ChatList)
ChatList = slices.Compact(ChatList)
AddChat(&msg.Message.Chat)
// Keep only the newcomers we can actually address. In production the ingress has
// already refused such an update; this filter is the nil-safety that keeps a
// direct caller from recording somebody we cannot name.
newcomers := make([]*echotron.User, 0, len(msg.Message.NewChatMembers))
for _, user := range msg.Message.NewChatMembers {
Log.Infof(
if !usableUser(user) {
continue
}
newcomers = append(newcomers, user)
}
if len(newcomers) == 0 {
return
}
// Remember the join so the greeting sweeper can tell later whether this
// newcomer answered, and so a ban can retract the greeting we sent.
//
// The join is recorded before the greeting exists, so the greeting id starts
// at zero and is filled in once the send returns. A join whose greeting was
// never delivered therefore carries zero, which is exactly the marker
// retractGreetings skips.
joinMicros := time.Now().UnixMicro()
PendingGreetings.Set(joinMicros, pendingGreeting{join: msg.Message}, greetCacheTTL)
// Record the join in the chat log before anything else, so the arrival is on
// record even if the greeting later fails. elog writes one line per newcomer
// itself, so this is called once for the whole event rather than per user.
elog.LogMessage(msg.Message)
for _, user := range newcomers {
mylog.Infof(
"User %s %s (username = %s, id = %d) joined %s %s (%d)",
user.FirstName,
user.LastName,
@@ -36,60 +66,23 @@ func newChatMembers(msg *echotron.Update) {
msg.Message.Chat.ID,
)
AddUser(user)
// Greet the newcomer, so that a spammer who never answers becomes
// detectable. If the greeting could not be delivered the newcomer never
// saw the question, so they must not be judged on their silence.
greetingID, delivered := sendGreeting(msg.Message.Chat.ID, user)
if !delivered {
PendingGreetings.Delete(joinMicros)
casBanned, err := CasCheckID(user.ID)
if err != nil {
Log.Error(err)
return
}
// Ban user if cas report it as banned.
if casBanned {
Log.Infof(
"Ban user %s %s (username = %s, id = %d) in %s %s (%d) by CAS blacklist",
user.FirstName,
user.LastName,
user.Username,
user.ID,
msg.Message.Chat.Type,
msg.Message.Chat.Title,
msg.Message.Chat.ID,
)
squash(
msg.Message.Chat.ID,
user.ID,
)
}
// Do not return on prev. step.
Log.Infof(
"Skip banning user %s %s (username = %s, id = %d) in %s %s (%d) because it was not found in CAS blacklist",
user.FirstName,
user.LastName,
user.Username,
user.ID,
msg.Message.Chat.Type,
msg.Message.Chat.Title,
msg.Message.Chat.ID,
)
Log.Infof(
"Add user %s %s (username = %s, id = %d) in %s %s (%d) to NewMember DB",
user.FirstName,
user.LastName,
user.Username,
user.ID,
msg.Message.Chat.Type,
msg.Message.Chat.Title,
msg.Message.Chat.ID,
)
key := fmt.Sprintf("%d+%d", msg.Message.Chat.ID, user.ID)
NewMembers.Set(key, time.Now().Unix(), 1*time.Minute)
PendingCASMembers.Set(key, time.Now().Unix(), 30*time.Minute)
// Re-record the join carrying the greeting's id. A member who joined
// twice has one recorded join per greeting, so each id is retracted
// separately.
PendingGreetings.Set(joinMicros, pendingGreeting{
join: msg.Message,
greetingID: greetingID,
}, greetCacheTTL)
}
}

50
internal/moon/photo.go
View File

@@ -1,50 +0,0 @@
package moon
import (
"fmt"
"github.com/NicoNex/echotron/v3"
)
// photo handles messages with photos.
func photo(msg *echotron.Update) {
AddChat(&msg.Message.Chat)
AddUser(msg.Message.From)
if Cfg.LogChats {
if msg.Message.Text != "" {
ChatLog(
msg.Message.Chat.ID,
fmt.Sprintf(
"User %s %s (username = %s, id = %d) sent %d photos to %s %s (%d) with text: %s",
msg.Message.From.FirstName,
msg.Message.From.LastName,
msg.Message.From.Username,
msg.Message.From.ID,
len(msg.Message.Photo),
msg.Message.Chat.Type,
msg.Message.Chat.Title,
msg.Message.Chat.ID,
msg.Message.Text,
),
)
} else {
ChatLog(
msg.Message.Chat.ID,
fmt.Sprintf(
"User %s %s (username = %s, id = %d) sent %d photos to %s %s (%d)",
msg.Message.From.FirstName,
msg.Message.From.LastName,
msg.Message.From.Username,
msg.Message.From.ID,
len(msg.Message.Photo),
msg.Message.Chat.Type,
msg.Message.Chat.Title,
msg.Message.Chat.ID,
),
)
}
}
}
/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

68
internal/moon/reaction.go
View File

@@ -1,68 +0,0 @@
package moon
import (
"fmt"
"github.com/NicoNex/echotron/v3"
)
func reaction(msg *echotron.Update) {
AddChat(&msg.MessageReaction.Chat)
AddUser(&msg.MessageReaction.User)
if Cfg.LogChats {
var reactionString string
for _, item := range msg.MessageReaction.NewReaction {
switch item.Type {
case "Emoji":
reactionString += item.Emoji
case "CustomEmoji":
reactionString += item.CustomEmoji
}
}
if reactionString != "" {
ChatLog(
msg.MessageReaction.Chat.ID,
fmt.Sprintf(
"User %s %s (username = %s, id = %d) set reaction %s to chat %s %s (%d)",
msg.MessageReaction.User.FirstName,
msg.MessageReaction.User.LastName,
msg.MessageReaction.User.Username,
msg.MessageReaction.User.ID,
reactionString,
msg.MessageReaction.Chat.Type,
msg.MessageReaction.Chat.Title,
msg.MessageReaction.Chat.ID,
),
)
} else {
for _, item := range msg.MessageReaction.OldReaction {
switch item.Type {
case "Emoji":
reactionString += item.Emoji
case "CustomEmoji":
reactionString += item.CustomEmoji
}
}
ChatLog(
msg.MessageReaction.Chat.ID,
fmt.Sprintf(
"User %s %s (username = %s, id = %d) removed reaction %s to chat %s %s (%d)",
msg.MessageReaction.User.FirstName,
msg.MessageReaction.User.LastName,
msg.MessageReaction.User.Username,
msg.MessageReaction.User.ID,
reactionString,
msg.MessageReaction.Chat.Type,
msg.MessageReaction.Chat.Title,
msg.MessageReaction.Chat.ID,
),
)
}
}
}
/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

24
internal/moon/reactionCount.go
View File

@@ -1,24 +0,0 @@
package moon
import (
"fmt"
"github.com/NicoNex/echotron/v3"
"github.com/davecgh/go-spew/spew"
)
// reactionCount handles reactionCount event.
func reactionCount(msg *echotron.Update) {
ChatLog(
msg.MessageReaction.Chat.ID,
fmt.Sprintf(
"Reaction count event in %s %s (%d): %s",
msg.MessageReaction.Chat.Type,
msg.MessageReaction.Chat.Title,
msg.MessageReaction.Chat.ID,
spew.Sdump(msg),
),
)
}
/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

119
internal/moon/retractGreeting.go Normal file
View File

@@ -0,0 +1,119 @@
package moon
import (
"bunny-bot/internal/elog"
"bunny-bot/internal/mylog"
"github.com/NicoNex/echotron/v3"
)
// retractGreetings deletes the greetings we sent to a member who has just been
// banned for never answering them, so the room is not left asking a question of
// someone who no longer has access.
//
// Ids <= 0 mean "no greeting was ever sent for that join" and are skipped
// silently. A deletion Telegram refuses is logged and the pass continues: the
// greeting is not worth abandoning the rest over, and a failed retraction never
// changes the ban that has already landed.
func retractGreetings(chat *echotron.Chat, user *echotron.User, greetingIDs []int) retractOutcome {
retracted, failed := 0, 0
for _, id := range greetingIDs {
if id <= 0 {
continue
}
if _, err := Tg.DeleteMessage(chat.ID, id); err != nil {
failed++
reportRetractFailure(chat, user, id, err)
continue
}
retracted++
elog.LogGreetingRetracted(chat, user, id)
}
return retractOutcomeFor(retracted, failed)
}
// reportRetractFailure logs a refused deletion. It is a WARN rather than an ERROR
// because the ban it followed already succeeded and nothing depends on the
// greeting being gone.
func reportRetractFailure(chat *echotron.Chat, user *echotron.User, id int, err error) {
mylog.Warnf(
"Left an unanswered greeting in place: cannot delete message %d in chat %d for user %d: %s",
id,
chat.ID,
userIDOf(user),
err,
)
}
// retractAskedGreetings takes down the questions this member was asked and never
// answered. One id per recorded join, so a member who joined twice had two
// greetings and both go.
func retractAskedGreetings(refs []joinRef) {
ids := make([]int, 0, len(refs))
for _, ref := range refs {
ids = append(ids, ref.greetingID)
}
chat, user := refs[0].chat, refs[0].user
switch retractGreetings(&chat, user, ids) {
case greetingRetracted:
mylog.Debugf(
"Deleted %d unanswered greeting(s) asked of user %d in chat %d",
countNamedGreetings(ids),
user.ID,
chat.ID,
)
case greetingRetractFailed:
mylog.Warnf(
"Banned user %d in chat %d, but could not delete every greeting we asked them",
user.ID,
chat.ID,
)
case nothingToRetract:
}
}
// countNamedGreetings counts the ids that actually named a message; the rest
// mean no greeting was ever delivered.
func countNamedGreetings(ids []int) int {
n := 0
for _, id := range ids {
if id > 0 {
n++
}
}
return n
}
// retractOutcomeFor collapses the two counts into the single value callers act on.
func retractOutcomeFor(retracted int, failed int) retractOutcome {
switch {
case retracted > 0 && failed == 0:
return greetingRetracted
case failed > 0:
return greetingRetractFailed
default:
return nothingToRetract
}
}
// userIDOf names a user in a log line without dereferencing a possibly nil one.
func userIDOf(user *echotron.User) int64 {
if user == nil {
return 0
}
return user.ID
}
/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

293
internal/moon/runAndBan.go
View File

@@ -1,293 +0,0 @@
package moon
import (
"fmt"
"slices"
"strconv"
"strings"
"time"
)
func RunAndBan() {
for {
var (
ListAppearedMembers, ListNewMembers []string
)
AppearedMembers.Range(
func(k, _ any) bool {
ListAppearedMembers = append(ListAppearedMembers, k.(string))
return true
},
)
NewMembers.Range(
func(k, _ any) bool {
ListNewMembers = append(ListNewMembers, k.(string))
return true
},
)
for _, AppearedMember := range ListAppearedMembers {
for _, NewMember := range ListNewMembers {
if AppearedMember == NewMember {
AppearedMembers.Delete(AppearedMember)
NewMembers.Delete(NewMember)
}
}
}
currentTime := time.Now().Unix()
ListAppearedMembers = ListAppearedMembers[:0]
ListNewMembers = ListNewMembers[:0]
ListAppearedMembers = slices.Clip(ListAppearedMembers)
ListNewMembers = slices.Clip(ListNewMembers)
AppearedMembers.Range(
func(k, _ any) bool {
ListAppearedMembers = append(ListAppearedMembers, k.(string))
return true
},
)
NewMembers.Range(
func(k, _ any) bool {
ListNewMembers = append(ListNewMembers, k.(string))
return true
},
)
for _, AppearedMember := range ListAppearedMembers {
eventTimestamp, found := AppearedMembers.Get(AppearedMember)
if !found {
continue
}
eventTimeDelta := currentTime - eventTimestamp.(int64)
// Spammer?
if eventTimeDelta > 10 {
item := strings.Split(AppearedMember, "+")
chatid, err := strconv.ParseInt(item[0], 10, 64)
if err != nil {
Log.Errorf("Got broken key in AppearedMembers cache: %s", AppearedMember)
continue
}
userid, err := strconv.ParseInt(item[1], 10, 64)
if err != nil {
Log.Errorf("Got broken key in AppearedMembers cache: %s", AppearedMember)
continue
}
user := GetUser(userid)
chat := GetChat(chatid)
Log.Infof(
"Ban user %s %s (username = %s, id = %d) in chat %s %s (%d) that have no newChatMember message since chatMemberUpdated for more than 10 seconds",
user.FirstName,
user.LastName,
user.Username,
userid,
chat.Type,
chat.Title,
chatid,
)
squash(chatid, userid)
AppearedMembers.Delete(AppearedMember)
}
}
for _, NewMember := range ListNewMembers {
eventTimestamp, found := NewMembers.Get(NewMember)
if !found {
continue
}
eventTimeDelta := currentTime - eventTimestamp.(int64)
// Spammer?
if eventTimeDelta > 10 {
item := strings.Split(NewMember, "+")
chatid, err := strconv.ParseInt(item[0], 10, 64)
if err != nil {
Log.Errorf("Got broken key in AppearedMembers cache: %s", NewMember)
continue
}
userid, err := strconv.ParseInt(item[1], 10, 64)
if err != nil {
Log.Errorf("Got broken key in AppearedMembers cache: %s", NewMember)
continue
}
user := GetUser(userid)
chat := GetChat(chatid)
Log.Debugf("Clean newChatMember event record older than 10 seconds for user user %s %s (username = %s, id = %d) in chat %s %s (%d)",
user.FirstName,
user.LastName,
user.Username,
userid,
chat.Type,
chat.Title,
chatid,
)
NewMembers.Delete(NewMember)
}
}
time.Sleep(10 * time.Second)
}
}
func PeriodicCASCheck() {
for {
var (
ListPendingMembers []string
)
PendingCASMembers.Range(
func(k, _ any) bool {
ListPendingMembers = append(ListPendingMembers, k.(string))
return true
},
)
currentTime := time.Now().Unix()
for _, PendingMember := range ListPendingMembers {
eventTimestamp, found := PendingCASMembers.Get(PendingMember)
if !found {
continue
}
eventTimeDelta := currentTime - eventTimestamp.(int64)
// It's time to check for second time this id if it was banned in CAS.
if eventTimeDelta > 300 {
item := strings.Split(PendingMember, "+")
chatid, err := strconv.ParseInt(item[0], 10, 64)
if err != nil {
Log.Errorf("Got broken key in PendingMembers cache: %s", PendingMember)
continue
}
userid, err := strconv.ParseInt(item[1], 10, 64)
if err != nil {
Log.Errorf("Got broken key in PendingMembers cache: %s", PendingMember)
continue
}
value := Cfg.GetValue(
"BannedMembers",
fmt.Sprintf("%d", chatid),
fmt.Sprintf("%d", userid),
)
user := GetUser(userid)
chat := GetChat(chatid)
if value != "" {
Log.Debugf("User %s %s (username = %s, id = %d) in %s %s (%d) is already banned, skip cas check.",
user.FirstName,
user.LastName,
user.Username,
userid,
chat.Type,
chat.Title,
chatid,
)
Log.Debugf(
"Remove user %s %s (username = %s, id = %d) in %s %s (%d) from PendingCASMembers",
user.FirstName,
user.LastName,
user.Username,
userid,
chat.Type,
chat.Title,
chatid,
)
key := fmt.Sprintf("%d+%d", chatid, userid)
PendingCASMembers.Delete(key)
return
}
Log.Debugf("Checking user %s %s (username = %s, id = %d) in %s %s (%d) for second time in CAS",
user.FirstName,
user.LastName,
user.Username,
userid,
chat.Type,
chat.Title,
chatid,
)
banned, err := CasCheckID(userid)
if err != nil {
Log.Error(err)
}
if banned {
Log.Infof(
"Ban user %s %s (username = %s, id = %d) in %s %s (%d) by CAS blacklist",
user.FirstName,
user.LastName,
user.Username,
userid,
chat.Type,
chat.Title,
chatid,
)
squash(chatid, userid)
}
Log.Debugf(
"Remove user %s %s (username = %s, id = %d) in %s %s (%d) from PendingCASMembers",
user.FirstName,
user.LastName,
user.Username,
userid,
chat.Type,
chat.Title,
chatid,
)
PendingCASMembers.Delete(PendingMember)
}
}
time.Sleep(10 * time.Minute)
}
}
/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

97
internal/moon/shutdown.go Normal file
View File

@@ -0,0 +1,97 @@
package moon
import (
"bunny-bot/internal/elog"
"bunny-bot/internal/mylog"
"fmt"
"github.com/akyoto/cache"
)
// closeCaches shuts down every cache sweeper goroutine.
//
// Safe only once the update listener and the greeting sweeper have stopped:
// cache.Close() reassigns the underlying sync.Map, which is a data race against
// any concurrent reader. shutdown() enforces that order.
func closeCaches() {
closeOnce.Do(func() {
for _, c := range []*cache.Cache{
SquashedMembers,
PendingGreetings.cache, GreetAnswers.cache,
} {
c.Close()
}
})
}
// Shutdown performs the ordered teardown, waits for it to finish, and reports
// whether every stage of it succeeded.
//
// main calls it after Telegram returns, for two reasons. The listener returning is
// not by itself a finished shutdown — a signal handler may be halfway through the
// sequence at that moment, and main returning would end the process underneath it,
// skipping stages and closing the log twice. And if the listener stopped for any
// other reason, nobody would run the sequence at all.
//
// It is safe to call from both goroutines: sync.Once runs the body once and blocks
// every other caller until that run completes, so both callers see the same result.
func Shutdown() error {
return shutdown()
}
// shutdown stops everything in dependency order and waits for each stage.
//
// The order is not cosmetic. Every stage waits for the previous one to have
// finished, so no goroutine is ever using a resource that the next stage is
// closing:
//
// 1. the Telegram update listener — nothing else may add work;
// 2. the greeting sweeper — the last periodic user of the caches;
// 3. the caches — cache.Close() both signals the sweeper goroutines and
// reassigns the underlying sync.Map, so it must run with nothing else
// touching them;
// 4. chat logging, then 5. the main log — so the last log lines, which
// describe stages 1 to 3, still have somewhere to go.
//
// Stages 1 and 2 share one stop signal and one WaitGroup, so a worker that has
// not started costs nothing and a worker that starts late is refused rather than
// racing the wait.
//
// The main log's own failure is reported ahead of the chat log's, because it happened
// last: a chat-log failure may no longer have been writable either.
func shutdown() error {
var result error
shutdownOnce.Do(func() {
// Refuse new workers before signalling, so nothing can register between
// the signal and the wait below.
workersMu.Lock()
stopping = true
workersMu.Unlock()
// Ask every loop to return, then wait for all of them. Nothing may be
// using a cache, a log or the API once the next stages start closing.
close(stopRequested)
workers.Wait()
closeCaches()
// Chat logging first: it writes to the main log, so it must be closed
// before the main log is.
chatErr := elog.Close()
// Since we shutting down, we have noting to do with log sync.
logErr := mylog.Close()
switch {
case logErr != nil:
result = fmt.Errorf("main log: %w", logErr)
case chatErr != nil:
result = fmt.Errorf("chat log: %w", chatErr)
}
})
return result
}
/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

98
internal/moon/sighandler.go
View File

@@ -2,39 +2,89 @@ package moon
import (
"os"
"syscall"
"os/signal"
"bunny-bot/internal/mylog"
)
// SigHandler.handles signals. Mainly closes persistent dbs and quits for good.
func SigHandler() {
for {
var s = <-SigChan
switch s {
case syscall.SIGINT:
Log.Infoln("Got SIGINT, quitting")
case syscall.SIGTERM:
Log.Infoln("Got SIGTERM, quitting")
case syscall.SIGQUIT:
Log.Infoln("Got SIGQUIT, quitting")
// InstallSignalHandler registers every signal the bot cares about and starts the
// handler goroutine. The signal sets differ per platform, so the knowledge lives
// behind terminationSignals and reopenSignals rather than leaking into main.
func InstallSignalHandler() {
signal.Notify(SigChan, terminationSignals()...)
signal.Notify(SigChan, reopenSignals()...)
// Make new iteration since we 've got signal we not interested in.
default:
go SigHandler()
}
// SigHandler handles signals until one asks the process to quit.
//
// It stops the program; it does not end it. os.Exit here would skip main's
// deferred cleanup, and would report success whether or not the teardown actually
// worked — a service manager told "clean shutdown" over a log nobody can write to
// again is worse than one told to restart us. So the handler runs the teardown and
// returns; main decides the exit status from its own call to Shutdown.
func SigHandler() {
for s := range SigChan {
if !handleSignal(s) {
continue
}
if len(DB) > 0 {
Log.Debug("Closing runtime bot settings db")
for _, db := range DB {
_ = db.Close()
}
if err := shutdown(); err != nil {
// The main log is already closed by now, so this is the last line we
// get to write; main still exits non-zero over the same failure.
mylog.Errorf("Shutdown did not complete cleanly: %s", err)
}
// Since we shutting down, we have noting to do with log sync.
_ = Log.Sync()
os.Exit(0)
return
}
}
// handleSignal applies one signal and reports whether it means "shut down".
//
// Reopen signals re-open the log file, which is what a log rotation tool needs
// after it has moved the file aside. Neither of them is expected on Windows,
// and an unrecognised signal is ignored rather than treated as a quit.
func handleSignal(s os.Signal) bool {
if containsSignal(reopenSignals(), s) {
mylog.Infof("Got %s, re-opening the log file", s)
if err := mylog.ReOpen(); err != nil {
// Reopening can legitimately fail — the destination may not be a
// file at all, or the logger may already be closed. Logging is
// best-effort here because the log is exactly what just broke.
mylog.Errorf("Unable to re-open the log file on %s: %s", s, err)
} else {
mylog.Info("Log file re-opened")
}
return false
}
if !containsSignal(terminationSignals(), s) {
// Make new iteration since we 've got signal we not interested in.
return false
}
// The status has to be recorded here, before the teardown runs: main reads
// it after Shutdown returns, and neither of them can reconstruct which
// signal triggered all of this by then.
recordExitCode(s)
mylog.Infof("Got %s, quitting", s)
return true
}
// containsSignal reports whether sig is in the set.
func containsSignal(set []os.Signal, sig os.Signal) bool {
for _, s := range set {
if s == sig {
return true
}
}
return false
}
/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

25
internal/moon/signals_unix.go Normal file
View File

@@ -0,0 +1,25 @@
//go:build !windows
package moon
import (
"os"
"syscall"
)
// terminationSignals are the signals that ask the bot to shut down.
func terminationSignals() []os.Signal {
return []os.Signal{syscall.SIGINT, syscall.SIGTERM, syscall.SIGQUIT}
}
// reopenSignals are the signals that ask the bot to re-open its log file.
//
// SIGHUP is the conventional daemon signal for "re-read your configuration and
// re-open your logs"; SIGUSR1 is the other common choice for log re-opening
// alone. Both are accepted so the bot fits either convention an operator's
// logrotate configuration expects.
func reopenSignals() []os.Signal {
return []os.Signal{syscall.SIGHUP, syscall.SIGUSR1}
}
/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

22
internal/moon/signals_windows.go Normal file
View File

@@ -0,0 +1,22 @@
//go:build windows
package moon
import "os"
// terminationSignals are the signals that ask the bot to shut down.
func terminationSignals() []os.Signal {
return []os.Signal{os.Interrupt}
}
// reopenSignals is empty on Windows.
//
// SIGHUP and SIGUSR1 are Unix signals that Windows never delivers — SIGUSR1 does
// not even exist in the syscall package there — so there is nothing to re-open
// the log for. The empty set keeps handleSignal platform-agnostic: a Windows
// service restarts to rotate its logs instead of signalling.
func reopenSignals() []os.Signal {
return nil
}
/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

114
internal/moon/squash.go
View File

@@ -1,32 +1,47 @@
package moon
import (
"bunny-bot/internal/mylog"
"fmt"
"slices"
"time"
"github.com/NicoNex/echotron/v3"
)
// TODO: refactor to return bool, error
// squash bans given chat member.
func squash(chatid int64, userid int64) {
banUntilDate := time.Now().Unix() + 10
user := GetUser(userid)
chat := GetChat(chatid)
// If there is no such chatid, means that no such cache too.
if len(ChatList) == 0 || !slices.Contains(ChatList, chatid) {
Log.Debugf("Add %s %s (%d) to list of known chats", chat.Type, chat.Title, chat.ID)
ChatList = append(ChatList, chatid)
// squash bans the given chat member and reports what it actually did.
//
// The three outcomes are deliberately distinct. A debounced call is not a
// failure: the user is already banned, which is the outcome we wanted, and the
// caller must not report it as an error.
//
// The user and chat come from the join event that triggered the judgement
// rather than from a cache. That event already carries both in full, so a cache
// would only ever be a second, staler copy of data we are holding — and a miss
// in it would produce a ban logged with blank names.
func squash(chat *echotron.Chat, user *echotron.User) banOutcome {
// A chat or user with no id is an entity we cannot address, not a ban that
// failed. In production the ingress has already refused such an update, so
// reaching here means a caller handed us one directly; the guard is the
// nil-safety that keeps that from becoming a call against chat 0.
if !usableChat(chat) {
return banFailed
}
if !usableUser(user) {
return banFailed
}
// Telegram treats any ban shorter than 30 seconds (and longer than 1 year)
// as permanent, so this deliberately short offset produces a permanent ban.
// Do not "fix" it into a longer duration.
banUntilDate := time.Now().Unix() + 10
chatid, userid := chat.ID, user.ID
key := fmt.Sprintf("%d+%d", chatid, userid)
if _, exist := SquashedMembers.Get(key); exist {
Log.Infof(
mylog.Infof(
"Skip banning user %s %s (username = %s, id = %d) in %s %s (%d), already banned",
user.FirstName,
user.LastName,
@@ -37,17 +52,18 @@ func squash(chatid int64, userid int64) {
chat.ID,
)
return
return banSuppressed
}
if res, err := Tg.BanChatMember(
res, err := Tg.BanChatMember(
chatid,
userid,
&echotron.BanOptions{
UntilDate: int(banUntilDate),
},
); err != nil {
Log.Errorf(
)
if err != nil {
mylog.Errorf(
"Unable to ban user user %s %s (username = %s, id = %d) in %s %s (%d): %s",
user.FirstName,
user.LastName,
@@ -56,11 +72,14 @@ func squash(chatid int64, userid int64) {
chat.Type,
chat.Title,
chat.ID,
res.Description,
err,
)
} else {
return banFailed
}
if !res.Ok {
Log.Errorf(
mylog.Errorf(
"Unable to ban user %s %s (username = %s, id = %d) in %s %s (%d): %s",
user.FirstName,
user.LastName,
@@ -71,8 +90,11 @@ func squash(chatid int64, userid int64) {
chat.ID,
res.Description,
)
} else {
Log.Debugf(
return banFailed
}
mylog.Debugf(
"Banned user %s %s (username = %s, id = %d) in %s %s (%d).",
user.FirstName,
user.LastName,
@@ -83,9 +105,7 @@ func squash(chatid int64, userid int64) {
chat.ID,
)
key := fmt.Sprintf("%d+%d", chatid, userid)
Log.Debugf(
mylog.Debugf(
"Put user %s %s (username = %s, id = %d) in %s %s (%d) into SquashedMembers to debounce ban.",
user.FirstName,
user.LastName,
@@ -98,43 +118,7 @@ func squash(chatid int64, userid int64) {
SquashedMembers.Set(key, true, 30*time.Second)
Log.Debugf(
"Remove user %s %s (username = %s, id = %d) in %s %s (%d) from NewMembers.",
user.FirstName,
user.LastName,
user.Username,
user.ID,
chat.Type,
chat.Title,
chat.ID,
)
NewMembers.Delete(key)
Log.Debugf(
"Remove user %s %s (username = %s, id = %d) in %s %s (%d) from AppearedMembers.",
user.FirstName,
user.LastName,
user.Username,
user.ID,
chat.Type,
chat.Title,
chat.ID,
)
AppearedMembers.Delete(key)
if err := Cfg.SaveKeyValue(
"BannedMembers",
fmt.Sprintf("%d", chatid),
fmt.Sprintf("%d", userid),
fmt.Sprintf("%d", time.Now().Unix()),
); err != nil {
Log.Error(err)
} else {
Log.Infof("Info about banned userid %d saved to BannedMembers db", userid)
}
}
}
return banApplied
}
/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

122
internal/moon/telegram.go
View File

@@ -1,74 +1,110 @@
package moon
import (
"bunny-bot/internal/elog"
"bunny-bot/internal/mylog"
"github.com/NicoNex/echotron/v3"
"github.com/davecgh/go-spew/spew"
)
// Telegram main goroutine that implements bot.
//
// It blocks until a termination signal stops it, so the update listener can be
// shut down in a defined order rather than simply abandoned when the process
// exits.
func Telegram(cfg MyConfig) {
// A signal can arrive during startup; then this must not start at all.
if !startWorker() {
mylog.Info("Telegram not started: shutdown already in progress")
return
}
defer stopWorker()
Tg = echotron.NewAPI(cfg.Token)
Log.Infof("Starting api update poller loop")
mylog.Info("Starting api update poller loop")
// The sweeper calls the API too, so it may only start once Tg exists.
mylog.Info("Starting greeting sweeper periodic task")
go SweepGreetings()
updates := echotron.PollingUpdatesOptions(cfg.Token, true, echotron.UpdateOptions{
AllowedUpdates: allowedUpdates,
})
for {
select {
// Stop between updates, never inside one: the handler currently running
// finishes, so nothing is left half-processed when we return.
case <-stopRequested:
mylog.Info("Telegram update listener stopped")
return
case u, ok := <-updates:
if !ok {
mylog.Info("Telegram update poller ended")
return
}
for u := range echotron.PollingUpdatesOptions(cfg.Token, true, echotron.UpdateOptions{
AllowedUpdates: []echotron.UpdateType{
"message",
"edited_message",
"channel_post",
"edited_channel_post",
"message_reaction",
"message_reaction_count",
"inline_query",
"chosen_inline_result",
"callback_query",
"poll",
"poll_answer",
"my_chat_member",
"chat_member",
"chat_join_request",
"chat_boost",
"removed_chat_boost",
},
}) {
telegramMsgParser(u)
}
}
}
// telegramMsgParser parses events generated by echotron.
func telegramMsgParser(msg *echotron.Update) {
// The one place an update enters the program, and therefore the one place an
// entity with no id is caught. Everything downstream — the caches, the sweeper,
// the bans — may then assume the ids it holds are real.
if !usableUpdate(msg) {
reportBrokenUpdate(msg)
return
}
switch {
case msg.Message != nil && msg.Message.Text != "":
textMessage(msg)
case msg.EditedMessage != nil:
editedMessage(msg)
// Membership status changes: joins, leaves, bans, promotions. Logged only —
// moderation is the sweeper's job, on a timer, not the dispatcher's.
//
// This also catches the membership updates caused by our own bans, so every
// ban leaves a second record in the chat log.
case msg.ChatMember != nil:
chatMemberUpdated(msg)
elog.LogUserStatus(msg.ChatMember)
// New members entered to chat.
// New members entered to chat. Checked before the text case so that a
// join service message is never mistaken for ordinary chatter.
case msg.Message != nil && msg.Message.NewChatMembers != nil:
newChatMembers(msg)
// Chat member leaves chat.
case msg.Message != nil && msg.Message.LeftChatMember != nil:
leftChatMember(msg)
// Ordinary text messages, used as greeting answers.
case msg.Message != nil && msg.Message.Text != "":
textMessage(msg)
case msg.Message != nil && msg.Message.Animation != nil:
animation(msg)
// Everything else that is still a message: a photo with no caption, a
// sticker, a location, a leave, a title change. Logged, and deliberately NOT
// recorded as a greeting answer — an attachment is not an introduction, and
// letting one satisfy the detector would be a one-key bypass for a spammer.
//
// The condition is deliberately just "there is a message": elog already
// renders every media and service field, and already writes nothing for a
// message with no describable content, so emptiness is its decision to make.
// Testing for a field being non-empty here is what lost these messages.
case msg.Message != nil:
logOnlyMessage(msg)
case msg.MessageReaction != nil:
reaction(msg)
case msg.MessageReactionCount != nil:
reactionCount(msg)
case msg.Message != nil && msg.Message.Photo != nil:
photo(msg)
// An edit revises a message already recorded, so it is a chat-log event
// rather than a moderation one.
case msg.EditedMessage != nil:
elog.LogEditedMessage(msg.EditedMessage)
default:
Log.Debug(spew.Sdump(msg))
mylog.Debug(spew.Sdump(msg))
}
}

61
internal/moon/textMessage.go
View File

@@ -1,58 +1,27 @@
package moon
import (
"fmt"
"time"
"bunny-bot/internal/elog"
"github.com/NicoNex/echotron/v3"
)
// textMessage handles text message events.
// textMessage handles plain text messages. It does not moderate anything; it
// only records the message so the greeting sweeper can tell that a newcomer
// answered their greeting.
func textMessage(msg *echotron.Update) {
AddChat(&msg.Message.Chat)
AddUser(msg.Message.From)
// Not sure if we must handle all kinds of these events.
if Cfg.LogChats {
switch {
// Just log it, it seems later here should be ban/mute command set.
case msg.Message != nil && msg.Message.ReplyToMessage == nil:
ChatLog(
msg.Message.Chat.ID,
fmt.Sprintf(
"User %s %s (username = %s, id = %d) in %s %s (%d) says: %s",
msg.Message.From.FirstName,
msg.Message.From.LastName,
msg.Message.From.Username,
msg.Message.From.ID,
msg.Message.Chat.Type,
msg.Message.Chat.Title,
msg.Message.Chat.ID,
msg.Message.Text,
),
)
case msg.Message != nil && msg.Message.ReplyToMessage != nil:
ChatLog(
msg.Message.Chat.ID,
fmt.Sprintf(
"User %s %s (username = %s, id = %d) in %s %s (%d) replies to message %d by %s %s (username = %s, id = %d): %s",
msg.Message.From.FirstName,
msg.Message.From.LastName,
msg.Message.From.Username,
msg.Message.From.ID,
msg.Message.Chat.Type,
msg.Message.Chat.Title,
msg.Message.Chat.ID,
msg.Message.ReplyToMessage.ID,
msg.Message.ReplyToMessage.From.FirstName,
msg.Message.ReplyToMessage.From.LastName,
msg.Message.ReplyToMessage.From.Username,
msg.Message.ReplyToMessage.From.ID,
msg.Message.Text,
),
)
}
if msg.Message == nil || msg.Message.Text == "" {
return
}
// Remember the message so the greeting sweeper can find the answer to a
// greeting.
GreetAnswers.Set(time.Now().UnixMicro(), msg.Message, greetCacheTTL)
// And record it in the chat's own log, which outlives the cache.
elog.LogMessage(msg.Message)
}
/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

84
internal/moon/types.go
View File

@@ -1,40 +1,82 @@
package moon
import (
"os"
"time"
"github.com/NicoNex/echotron/v3"
"github.com/akyoto/cache"
)
// MyConfig describes bot config file format.
type MyConfig struct {
Name string `json:"name,omitempty"`
Token string `json:"token,omitempty"`
LogLevel string `json:"loglevel,omitempty"`
LogFile string `json:"logfile,omitempty"`
LogChats bool `json:"logchats,omitempty"`
CSign string `json:"csign,omitempty"`
DataDir string `json:"datadir,omitempty"`
// ChatLogs is the directory for per-chat, per-day chat event logs. Empty
// disables chat logging. If set, the directory must already exist: a path
// that does not is a typo, and silently logging nowhere is the worst
// possible response to one.
ChatLogs string `json:"chat_logs,omitempty"`
}
// CasFalse describes response from api.cas.chat when user is not blacklisted.
type CasFalse struct {
Ok bool `json:"ok"`
Description string `json:"description"`
// tgClient is the subset of the Telegram API that bunny-bot actually calls.
// It exists so tests can substitute a fake; echotron.API satisfies it.
type tgClient interface {
SendMessage(text string, chatID int64, opts *echotron.MessageOptions) (echotron.APIResponseMessage, error)
BanChatMember(chatID int64, userID int64, opts *echotron.BanOptions) (echotron.APIResponseBool, error)
DeleteMessage(chatID int64, messageID int) (echotron.APIResponseBase, error)
}
// CasFalse describes response from api.cas.chat when user is blacklisted.
type CasTrue struct {
Ok bool `json:"ok"`
Result struct {
Reasons []int `json:"reasons"`
Offenses int `json:"offenses"`
TimeAdded time.Time `json:"time_added"`
} `json:"result"`
// chatUser identifies a user's membership of one specific chat. The same user is
// a different identity in every chat: being silent in one says nothing about
// another, and answering in one must not rescue them in another.
type chatUser struct {
chatID int64
userID int64
}
type Chatlog struct {
LogFilePath string
File *os.File
// banOutcome is what a squash attempt actually did. The three cases are distinct
// on purpose: a duplicate ban is a success, not a failure, and reporting it as
// one makes a working ban look broken in the logs.
type banOutcome int
// eventCache is a typed wrapper over akyoto/cache.Cache that stores values of
// type T keyed by a Unix timestamp in microseconds. It hides the underlying
// sync.Map so callers never deal with interface{} keys/values or per-access type
// conversions.
type eventCache[T any] struct {
cache *cache.Cache
}
// joinRef is one stale join, kept together with everything the verdict and the
// retraction need so neither has to look anything up.
type joinRef struct {
micros int64
// chat and user come from the join event itself, so a ban can be logged with
// the newcomer's real name and the chat's real title instead of a cache
// lookup that may have expired or never been populated.
chat echotron.Chat
user *echotron.User
// greetingID is the question we asked them, deleted once they are banned for
// never answering it. Zero means we never managed to ask.
greetingID int
}
// pendingGreeting is one recorded join: the message that announced it, and the
// id of the greeting we sent in reply.
//
// The greeting id lives here rather than in a second cache, so that recording,
// forgetting and retracting are all one lookup on one key. A zero greeting id
// means "no greeting was ever delivered for this join" — never a failure.
type pendingGreeting struct {
join *echotron.Message
greetingID int
}
// retractOutcome is what a retraction pass actually achieved. The three cases are
// distinct because "nothing to delete" and "tried and failed" call for different
// log lines, and only the second is worth an operator's attention.
type retractOutcome int
/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

76
internal/moon/usersGroups.go
View File

@@ -1,76 +0,0 @@
package moon
import (
"encoding/json"
"time"
"github.com/NicoNex/echotron/v3"
)
// AddUser remebers user info in People cache.
func AddUser(u *echotron.User) {
bytes, err := json.Marshal(u)
if err != nil {
Log.Errorf("Unable to encode user info to json string: %s", err)
}
People.Set(u.ID, bytes, 24*time.Hour)
}
// GetUser returns user info structure.
func GetUser(id int64) *echotron.User {
var u *echotron.User
bytes, ok := People.Get(id)
if !ok {
return u
}
err := json.Unmarshal(bytes.([]byte), &u)
if err != nil {
Log.Errorf("Unable to decode user info json: %s", err)
return u
}
return u
}
// AddChat remebers chat info in Groups cache.
func AddChat(c *echotron.Chat) {
bytes, err := json.Marshal(c)
if err != nil {
Log.Errorf("Unable to encode chat info to json string: %s", err)
}
Groups.Set(c.ID, bytes, 24*time.Hour)
}
// GetChat returns chat info structure.
func GetChat(id int64) *echotron.Chat {
var c *echotron.Chat
bytes, ok := Groups.Get(id)
if !ok {
Log.Errorf("No group %d in Groups cache", id)
return c
}
err := json.Unmarshal(bytes.([]byte), &c)
if err != nil {
Log.Errorf("Unable to decode chat info json: %s", err)
return c
}
return c
}
/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

36
internal/moon/worker.go Normal file
View File

@@ -0,0 +1,36 @@
package moon
// The worker registry: how a background goroutine joins the shutdown sequence,
// and how it leaves it. The state itself lives in globals.go with the rest of the
// package-level variables; this file holds only what a goroutine does with it.
// stopWorker unregisters a background goroutine that has finished.
func stopWorker() {
runningWorkers.Add(-1)
workers.Done()
}
// workerRunning reports whether any registered worker is still running.
func workerRunning() bool {
return runningWorkers.Load() > 0
}
// startWorker registers a background goroutine with the shutdown sequence and
// reports whether it may run. It returns false once shutdown has begun, in
// which case the caller must return without starting: the stop signal is
// already closed and the teardown may already have closed what it would use.
func startWorker() bool {
workersMu.Lock()
defer workersMu.Unlock()
if stopping {
return false
}
workers.Add(1)
runningWorkers.Add(1)
return true
}
/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

71
internal/mylog/globals.go Normal file
View File

@@ -0,0 +1,71 @@
package mylog
import (
"errors"
"io"
"os"
"sync"
)
const (
// levelTrace is the most verbose level, below DEBUG.
levelTrace Level = iota
levelDebug
levelInfo
levelWarn
levelError
levelFatal
)
const (
// stampLayout is the timestamp layout. Sub-second precision is kept on
// purpose: the greeting sweeper reasons over 15-minute windows measured in
// microseconds, and the previous zap configuration truncated to whole
// seconds, which made window-edge debugging needlessly hard.
stampLayout = "2006-01-02 15:04:05.000"
// calldepth is reported to the stdlib logger. mylog never prints caller
// information, so the value only has to be consistent.
calldepth = 2
// filePerm is the mode for a newly created log file.
filePerm = 0o644
)
var (
// errClosed is returned when ReOpen is called on a closed logger.
errClosed = errors.New("cannot reopen a closed logger")
// errUnknownLevel is returned for a log level name mylog does not know.
errUnknownLevel = errors.New("unknown log level")
// levelNames renders each level as the uppercase token written to the log.
levelNames = map[Level]string{
levelTrace: "TRACE",
levelDebug: "DEBUG",
levelInfo: "INFO",
levelWarn: "WARN",
levelError: "ERROR",
levelFatal: "FATAL",
}
// exit is os.Exit, indirected so tests can observe the fatal path.
exit = os.Exit
// lock guards the package-global logger state. It is held for reading across
// the actual write, so that a concurrent Init cannot close the destination
// between the level check and the write.
lock sync.RWMutex
// current is the active logger. It is deliberately never nil: it is created
// working at package initialisation, writing to stderr at INFO. A nil logger
// would be the defect M2 in this codebase, and a valid default removes the
// state entirely rather than guarding against it.
current = newState(levelInfo, io.Writer(os.Stderr), "", nil, false)
// standard is the singleton Logger handed to modules that expect a
// *log.Logger. It is stateless; the package-global current does the work.
standard Logger = &adapter{}
)
/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

59
internal/mylog/level.go Normal file
View File

@@ -0,0 +1,59 @@
package mylog
// The level vocabulary: what a level is called, how a name in free-form config
// text becomes a Level, and how the configured threshold is changed without
// reopening the destination.
import (
"fmt"
"strings"
)
// ParseLevel maps a level name to a Level. Parsing is case-insensitive and
// tolerates surrounding whitespace, because the level comes from free-form user
// text in the config file. An unknown name is an error: Init and SetLevel are
// programming-level calls where a typo should be loud.
func ParseLevel(s string) (Level, error) {
switch strings.ToUpper(strings.TrimSpace(s)) {
case "TRACE":
return levelTrace, nil
case "DEBUG":
return levelDebug, nil
case "INFO":
return levelInfo, nil
case "WARN":
return levelWarn, nil
case "ERROR":
return levelError, nil
case "FATAL":
return levelFatal, nil
default:
return levelInfo, fmt.Errorf("%w %q, want one of trace, debug, info, warn, error, fatal", errUnknownLevel, s)
}
}
// SetLevel changes the level without reopening the destination.
func SetLevel(loglevel string) error {
lvl, err := ParseLevel(loglevel)
if err != nil {
return err
}
lock.Lock()
defer lock.Unlock()
current.level = lvl
return nil
}
// name returns the uppercase token for the level.
func (l Level) name() string {
if n, ok := levelNames[l]; ok {
return n
}
return "UNKNOWN"
}
/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

118
internal/mylog/log.go Normal file
View File

@@ -0,0 +1,118 @@
package mylog
// The destination lifecycle: opening the log, re-opening it for rotation, and
// closing it for good. The level vocabulary lives in level.go, the state in
// state.go, and the act of writing a line in write.go.
import (
"fmt"
"io"
stdlog "log"
"os"
)
// Init opens the log destination and sets the level.
//
// An empty filepath selects stderr. A non-empty path is opened appending, so
// an existing log is never truncated.
//
// Init is re-entrant: any previously open file is closed first, so repeated
// calls do not leak descriptors. If the new destination cannot be opened, Init
// returns an error and leaves the previous destination and level untouched —
// it deliberately does not fall back to stderr, because a silent fallback
// hides a broken log path from the operator.
func Init(loglevel string, filepath string) error {
lvl, err := ParseLevel(loglevel)
if err != nil {
return err
}
lock.Lock()
defer lock.Unlock()
dest, owned := io.Writer(os.Stderr), (*os.File)(nil)
if filepath != "" {
// The path comes from the operator's own config file, not from a
// network peer, and opening it is this function's entire purpose.
file, err := os.OpenFile(filepath, os.O_CREATE|os.O_APPEND|os.O_WRONLY, filePerm) //nolint: gosec // G703
if err != nil {
return fmt.Errorf("unable to open log file %s: %w", filepath, err)
}
dest, owned = file, file
}
swapLocked(lvl, dest, filepath, owned)
// Keep the standard library's default logger pointed at the same
// destination, so third-party stdlib logging does not bypass our file.
stdlog.SetFlags(0)
stdlog.SetOutput(dest)
return nil
}
// ReOpen closes and re-opens the current log file, keeping its level. It exists
// for external rotation: the rotation tool moves the file aside, and ReOpen
// makes the logger pick up the new one.
//
// ReOpen on a closed logger is an error rather than a silent reopen, and on a
// stderr destination it is a no-op, since there is no file to re-open.
func ReOpen() error {
lock.RLock()
closed, path, lvl := current.closed, current.path, current.level
lock.RUnlock()
switch {
case closed:
return errClosed
case path == "":
return nil
default:
return Init(lvl.name(), path)
}
}
// Close closes the log destination. It is idempotent, and it is safe when the
// destination is stderr — stderr is not ours to close.
//
// After Close, logging calls are silently dropped rather than panicking or
// resurrecting the file: Close is terminal.
func Close() error {
lock.Lock()
defer lock.Unlock()
if current.closed {
return nil
}
current.closed = true
if current.owned == nil {
return nil
}
// The stdlib Logger has no Close — it does not own its writer — so the file
// we opened ourselves is closed directly.
file := current.owned
current.owned = nil
if file == nil {
return nil
}
return file.Close()
}
// swapLocked installs a new state, closing any file we previously owned.
// The caller must hold the write lock.
func swapLocked(lvl Level, dest io.Writer, path string, owned *os.File) {
if current.owned != nil {
_ = current.owned.Close()
}
current = newState(lvl, dest, path, owned, false)
}
/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

28
internal/mylog/standard.go Normal file
View File

@@ -0,0 +1,28 @@
package mylog
// The standard-library-compatible Logger: what a module receives instead of
// depending on *log.Logger itself, and where its messages end up.
import "strings"
// Standard returns the Logger for modules that would otherwise depend on the
// standard library's *log.Logger. Messages passed to it are logged at debug
// level into the already-open destination.
//
// Returning the interface rather than the concrete *adapter is the whole point:
// it is the type such modules declare, so hiding it would defeat the
// substitution it exists for.
func Standard() Logger { //nolint:ireturn // deliberate public interface
return standard
}
// Output implements Logger, logging at debug level. The trailing newline of the
// standard library convention is trimmed so that mylog does not append a blank
// line to every message.
func (a *adapter) Output(_ int, s string) error {
write(levelDebug, strings.TrimRight(s, "\r\n"))
return nil
}
/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

52
internal/mylog/state.go Normal file
View File

@@ -0,0 +1,52 @@
package mylog
// The logger state: the one struct every write goes through, how a new one is
// built, and the two accessors that read and replace it under the package lock.
// The state itself lives in globals.go with the rest of the package-level
// variables; this file holds only what a caller does with it.
import (
"io"
stdlog "log"
"os"
)
// newState builds a state whose stdlib logger writes to dest.
func newState(lvl Level, dest io.Writer, path string, owned *os.File, closed bool) *state {
return &state{
level: lvl,
dest: dest,
slog: newStdLogger(dest),
path: path,
owned: owned,
closed: closed,
}
}
// newStdLogger wraps a writer in a standard library logger. Flags are zeroed
// because the line format is rendered by mylog itself.
func newStdLogger(dest io.Writer) *stdLogger {
l := stdlog.New(dest, "", 0)
l.SetFlags(0)
return l
}
// getState returns the active logger state.
func getState() *state {
lock.RLock()
defer lock.RUnlock()
return current
}
// setState replaces the active state wholesale. Tests use it to restore the
// documented before-Init default.
func setState(s *state) {
lock.Lock()
defer lock.Unlock()
current = s
}
/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

43
internal/mylog/types.go Normal file
View File

@@ -0,0 +1,43 @@
package mylog
// Types only. The level constants, the name table and Level.name live in
// globals.go and level.go respectively, because a file named "types" must hold
// nothing that is not a type.
import (
"io"
stdlog "log"
"os"
)
// Level is the severity of a log message. Levels are ordered, and a message is
// written when its own level is at or above the configured level.
type Level int
// stdLogger is the standard library logger doing the actual writing.
type stdLogger = stdlog.Logger
// Logger is the interface for modules that would otherwise depend on the
// standard library's *log.Logger. It is deliberately the smallest method set
// that *log.Logger satisfies, so widening a module's parameter type from
// *log.Logger to Logger keeps existing callers compiling.
type Logger interface {
Output(calldepth int, s string) error
}
// adapter is the Logger handed to such modules. Messages arrive through
// Output and are logged at debug level into the already-open destination.
type adapter struct{}
// state is everything the logger needs to write a line. It is held behind a
// lock so that a concurrent Init cannot close a destination mid-write.
type state struct {
level Level
dest io.Writer
slog *stdLogger
path string
owned *os.File
closed bool
}
/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

63
internal/mylog/wrappers.go Normal file
View File

@@ -0,0 +1,63 @@
package mylog
import (
"fmt"
)
// Trace logs at the most verbose level, below DEBUG.
func Trace(args ...any) { write(levelTrace, fmt.Sprint(args...)) }
// Tracef logs a formatted message at the most verbose level, below DEBUG.
func Tracef(format string, args ...any) { write(levelTrace, fmt.Sprintf(format, args...)) }
// Debug logs at debug level.
func Debug(args ...any) { write(levelDebug, fmt.Sprint(args...)) }
// Debugf logs a formatted message at debug level.
func Debugf(format string, args ...any) { write(levelDebug, fmt.Sprintf(format, args...)) }
// Info logs at info level.
func Info(args ...any) { write(levelInfo, fmt.Sprint(args...)) }
// Infof logs a formatted message at info level.
func Infof(format string, args ...any) { write(levelInfo, fmt.Sprintf(format, args...)) }
// Warn logs at warn level.
func Warn(args ...any) { write(levelWarn, fmt.Sprint(args...)) }
// Warnf logs a formatted message at warn level.
func Warnf(format string, args ...any) { write(levelWarn, fmt.Sprintf(format, args...)) }
// Error logs at error level.
func Error(args ...any) { write(levelError, fmt.Sprint(args...)) }
// Errorf logs a formatted message at error level.
func Errorf(format string, args ...any) { write(levelError, fmt.Sprintf(format, args...)) }
// Fatal logs at fatal level, then exits the process with status 1.
//
// The destination is closed before exiting. os.Exit does not run deferred
// functions, so without this an explicit close a buffered tail could be lost —
// which is precisely the case where the message matters most.
func Fatal(args ...any) {
write(levelFatal, fmt.Sprint(args...))
exitAfterClose(1)
}
// Fatalf logs a formatted message at fatal level, then exits the process with
// status 1. The destination is closed first; see Fatal.
func Fatalf(format string, args ...any) {
write(levelFatal, fmt.Sprintf(format, args...))
exitAfterClose(1)
}
// exitAfterClose closes the logger and then exits, in that order.
func exitAfterClose(code int) {
_ = Close()
exit(code)
}
/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

28
internal/mylog/write.go Normal file
View File

@@ -0,0 +1,28 @@
package mylog
// The one-line emit path: decide whether the line is wanted, render it, write
// it. Everything else in the package exists to reach these two functions.
import "time"
// write renders and writes one line if the level passes the filter and the
// logger is open. The read lock is held across the write so that a concurrent
// Init cannot close the destination underneath us.
func write(lvl Level, msg string) {
lock.RLock()
defer lock.RUnlock()
if current.closed || lvl < current.level {
return
}
_ = current.slog.Output(calldepth, render(lvl, msg))
}
// render builds one log line: a GMT+0 timestamp, the uppercase level, and the
// message verbatim, separated by exactly one space each.
func render(lvl Level, msg string) string {
return time.Now().UTC().Format(stampLayout) + " " + lvl.name() + " " + msg
}
/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

27
vendor/github.com/DataDog/zstd/LICENSE generated vendored
View File

@@ -1,27 +0,0 @@
Simplified BSD License
Copyright (c) 2016, Datadog <info@datadoghq.com>
All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
* Redistributions of source code must retain the above copyright notice,
this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.
* Neither the name of the copyright holder nor the names of its contributors
may be used to endorse or promote products derived from this software
without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

142
vendor/github.com/DataDog/zstd/README.md generated vendored
View File

@@ -1,142 +0,0 @@
# Zstd Go Wrapper
[![CircleCI](https://circleci.com/gh/DataDog/zstd/tree/1.x.svg?style=svg)](https://circleci.com/gh/DataDog/zstd/tree/1.x)
[![GoDoc](https://godoc.org/github.com/DataDog/zstd?status.svg)](https://godoc.org/github.com/DataDog/zstd)
[C Zstd Homepage](https://github.com/facebook/zstd)
The current headers and C files are from *v1.5.7* (Commit
[f8745da](https://github.com/facebook/zstd/releases/tag/v1.5.7)).
## Usage
There are two main APIs:
* simple Compress/Decompress
* streaming API (io.Reader/io.Writer)
The compress/decompress APIs mirror that of lz4, while the streaming API was
designed to be a drop-in replacement for zlib.
### Building against an external libzstd
By default, zstd source code is vendored in this repository and the binding will be built with
the vendored source code bundled.
If you want to build this binding against an external static or shared libzstd library, you can
use the `external_libzstd` build tag. This will look for the libzstd pkg-config file and extract
build and linking parameters from that pkg-config file.
Note that it requires at least libzstd 1.4.0.
```bash
go build -tags external_libzstd
```
### Simple `Compress/Decompress`
```go
// Compress compresses the byte array given in src and writes it to dst.
// If you already have a buffer allocated, you can pass it to prevent allocation
// If not, you can pass nil as dst.
// If the buffer is too small, it will be reallocated, resized, and returned by the function
// If dst is nil, this will allocate the worst case size (CompressBound(src))
Compress(dst, src []byte) ([]byte, error)
```
```go
// CompressLevel is the same as Compress but you can pass another compression level
CompressLevel(dst, src []byte, level int) ([]byte, error)
```
```go
// Decompress will decompress your payload into dst.
// If you already have a buffer allocated, you can pass it to prevent allocation
// If not, you can pass nil as dst (allocates a 4*src size as default).
// If the buffer is too small, it will retry 3 times by doubling the dst size
// After max retries, it will switch to the slower stream API to be sure to be able
// to decompress. Currently switches if compression ratio > 4*2**3=32.
Decompress(dst, src []byte) ([]byte, error)
```
### Stream API
```go
// NewWriter creates a new object that can optionally be initialized with
// a precomputed dictionary. If dict is nil, compress without a dictionary.
// The dictionary array should not be changed during the use of this object.
// You MUST CALL Close() to write the last bytes of a zstd stream and free C objects.
NewWriter(w io.Writer) *Writer
NewWriterLevel(w io.Writer, level int) *Writer
NewWriterLevelDict(w io.Writer, level int, dict []byte) *Writer
// Write compresses the input data and write it to the underlying writer
(w *Writer) Write(p []byte) (int, error)
// Flush writes any unwritten data to the underlying writer
(w *Writer) Flush() error
// Close flushes the buffer and frees C zstd objects
(w *Writer) Close() error
```
```go
// NewReader returns a new io.ReadCloser that will decompress data from the
// underlying reader. If a dictionary is provided to NewReaderDict, it must
// not be modified until Close is called. It is the caller's responsibility
// to call Close, which frees up C objects.
NewReader(r io.Reader) io.ReadCloser
NewReaderDict(r io.Reader, dict []byte) io.ReadCloser
```
### Benchmarks (benchmarked with v0.5.0)
The author of Zstd also wrote lz4. Zstd is intended to occupy a speed/ratio
level similar to what zlib currently provides. In our tests, the can always
be made to be better than zlib by chosing an appropriate level while still
keeping compression and decompression time faster than zlib.
You can run the benchmarks against your own payloads by using the Go benchmarks tool.
Just export your payload filepath as the `PAYLOAD` environment variable and run the benchmarks:
```go
go test -bench .
```
Compression of a 7Mb pdf zstd (this wrapper) vs [czlib](https://github.com/DataDog/czlib):
```
BenchmarkCompression 5 221056624 ns/op 67.34 MB/s
BenchmarkDecompression 100 18370416 ns/op 810.32 MB/s
BenchmarkFzlibCompress 2 610156603 ns/op 24.40 MB/s
BenchmarkFzlibDecompress 20 81195246 ns/op 183.33 MB/s
```
Ratio is also better by a margin of ~20%.
Compression speed is always better than zlib on all the payloads we tested;
However, [czlib](https://github.com/DataDog/czlib) has optimisations that make it
faster at decompressiong small payloads:
```
Testing with size: 11... czlib: 8.97 MB/s, zstd: 3.26 MB/s
Testing with size: 27... czlib: 23.3 MB/s, zstd: 8.22 MB/s
Testing with size: 62... czlib: 31.6 MB/s, zstd: 19.49 MB/s
Testing with size: 141... czlib: 74.54 MB/s, zstd: 42.55 MB/s
Testing with size: 323... czlib: 155.14 MB/s, zstd: 99.39 MB/s
Testing with size: 739... czlib: 235.9 MB/s, zstd: 216.45 MB/s
Testing with size: 1689... czlib: 116.45 MB/s, zstd: 345.64 MB/s
Testing with size: 3858... czlib: 176.39 MB/s, zstd: 617.56 MB/s
Testing with size: 8811... czlib: 254.11 MB/s, zstd: 824.34 MB/s
Testing with size: 20121... czlib: 197.43 MB/s, zstd: 1339.11 MB/s
Testing with size: 45951... czlib: 201.62 MB/s, zstd: 1951.57 MB/s
```
zstd starts to shine with payloads > 1KB
### Stability - Current state: STABLE
The C library seems to be pretty stable and according to the author has been tested and fuzzed.
For the Go wrapper, the test cover most usual cases and we have succesfully tested it on all staging and prod data.

30
vendor/github.com/DataDog/zstd/ZSTD_LICENSE generated vendored
View File

@@ -1,30 +0,0 @@
BSD License
For Zstandard software
Copyright (c) Meta Platforms, Inc. and affiliates. All rights reserved.
Redistribution and use in source and binary forms, with or without modification,
are permitted provided that the following conditions are met:
* Redistributions of source code must retain the above copyright notice, this
list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.
* Neither the name Facebook, nor Meta, nor the names of its contributors may
be used to endorse or promote products derived from this software without
specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR
ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

58
vendor/github.com/DataDog/zstd/allocations.h generated vendored
View File

@@ -1,58 +0,0 @@
#ifndef USE_EXTERNAL_ZSTD
/*
* Copyright (c) Meta Platforms, Inc. and affiliates.
* All rights reserved.
*
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
* You may select, at your option, one of the above-listed licenses.
*/
/* This file provides custom allocation primitives
*/
#define ZSTD_DEPS_NEED_MALLOC
#include "zstd_deps.h" /* ZSTD_malloc, ZSTD_calloc, ZSTD_free, ZSTD_memset */
#include "compiler.h" /* MEM_STATIC */
#define ZSTD_STATIC_LINKING_ONLY
#include "zstd.h" /* ZSTD_customMem */
#ifndef ZSTD_ALLOCATIONS_H
#define ZSTD_ALLOCATIONS_H
/* custom memory allocation functions */
MEM_STATIC void* ZSTD_customMalloc(size_t size, ZSTD_customMem customMem)
{
if (customMem.customAlloc)
return customMem.customAlloc(customMem.opaque, size);
return ZSTD_malloc(size);
}
MEM_STATIC void* ZSTD_customCalloc(size_t size, ZSTD_customMem customMem)
{
if (customMem.customAlloc) {
/* calloc implemented as malloc+memset;
* not as efficient as calloc, but next best guess for custom malloc */
void* const ptr = customMem.customAlloc(customMem.opaque, size);
ZSTD_memset(ptr, 0, size);
return ptr;
}
return ZSTD_calloc(1, size);
}
MEM_STATIC void ZSTD_customFree(void* ptr, ZSTD_customMem customMem)
{
if (ptr!=NULL) {
if (customMem.customFree)
customMem.customFree(customMem.opaque, ptr);
else
ZSTD_free(ptr);
}
}
#endif /* ZSTD_ALLOCATIONS_H */
#endif /* USE_EXTERNAL_ZSTD */

208
vendor/github.com/DataDog/zstd/bits.h generated vendored
View File

@@ -1,208 +0,0 @@
#ifndef USE_EXTERNAL_ZSTD
/*
* Copyright (c) Meta Platforms, Inc. and affiliates.
* All rights reserved.
*
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
* You may select, at your option, one of the above-listed licenses.
*/
#ifndef ZSTD_BITS_H
#define ZSTD_BITS_H
#include "mem.h"
MEM_STATIC unsigned ZSTD_countTrailingZeros32_fallback(U32 val)
{
assert(val != 0);
{
static const U32 DeBruijnBytePos[32] = {0, 1, 28, 2, 29, 14, 24, 3,
30, 22, 20, 15, 25, 17, 4, 8,
31, 27, 13, 23, 21, 19, 16, 7,
26, 12, 18, 6, 11, 5, 10, 9};
return DeBruijnBytePos[((U32) ((val & -(S32) val) * 0x077CB531U)) >> 27];
}
}
MEM_STATIC unsigned ZSTD_countTrailingZeros32(U32 val)
{
assert(val != 0);
#if defined(_MSC_VER)
# if STATIC_BMI2
return (unsigned)_tzcnt_u32(val);
# else
if (val != 0) {
unsigned long r;
_BitScanForward(&r, val);
return (unsigned)r;
} else {
__assume(0); /* Should not reach this code path */
}
# endif
#elif defined(__GNUC__) && (__GNUC__ >= 4)
return (unsigned)__builtin_ctz(val);
#elif defined(__ICCARM__)
return (unsigned)__builtin_ctz(val);
#else
return ZSTD_countTrailingZeros32_fallback(val);
#endif
}
MEM_STATIC unsigned ZSTD_countLeadingZeros32_fallback(U32 val)
{
assert(val != 0);
{
static const U32 DeBruijnClz[32] = {0, 9, 1, 10, 13, 21, 2, 29,
11, 14, 16, 18, 22, 25, 3, 30,
8, 12, 20, 28, 15, 17, 24, 7,
19, 27, 23, 6, 26, 5, 4, 31};
val |= val >> 1;
val |= val >> 2;
val |= val >> 4;
val |= val >> 8;
val |= val >> 16;
return 31 - DeBruijnClz[(val * 0x07C4ACDDU) >> 27];
}
}
MEM_STATIC unsigned ZSTD_countLeadingZeros32(U32 val)
{
assert(val != 0);
#if defined(_MSC_VER)
# if STATIC_BMI2
return (unsigned)_lzcnt_u32(val);
# else
if (val != 0) {
unsigned long r;
_BitScanReverse(&r, val);
return (unsigned)(31 - r);
} else {
__assume(0); /* Should not reach this code path */
}
# endif
#elif defined(__GNUC__) && (__GNUC__ >= 4)
return (unsigned)__builtin_clz(val);
#elif defined(__ICCARM__)
return (unsigned)__builtin_clz(val);
#else
return ZSTD_countLeadingZeros32_fallback(val);
#endif
}
MEM_STATIC unsigned ZSTD_countTrailingZeros64(U64 val)
{
assert(val != 0);
#if defined(_MSC_VER) && defined(_WIN64)
# if STATIC_BMI2
return (unsigned)_tzcnt_u64(val);
# else
if (val != 0) {
unsigned long r;
_BitScanForward64(&r, val);
return (unsigned)r;
} else {
__assume(0); /* Should not reach this code path */
}
# endif
#elif defined(__GNUC__) && (__GNUC__ >= 4) && defined(__LP64__)
return (unsigned)__builtin_ctzll(val);
#elif defined(__ICCARM__)
return (unsigned)__builtin_ctzll(val);
#else
{
U32 mostSignificantWord = (U32)(val >> 32);
U32 leastSignificantWord = (U32)val;
if (leastSignificantWord == 0) {
return 32 + ZSTD_countTrailingZeros32(mostSignificantWord);
} else {
return ZSTD_countTrailingZeros32(leastSignificantWord);
}
}
#endif
}
MEM_STATIC unsigned ZSTD_countLeadingZeros64(U64 val)
{
assert(val != 0);
#if defined(_MSC_VER) && defined(_WIN64)
# if STATIC_BMI2
return (unsigned)_lzcnt_u64(val);
# else
if (val != 0) {
unsigned long r;
_BitScanReverse64(&r, val);
return (unsigned)(63 - r);
} else {
__assume(0); /* Should not reach this code path */
}
# endif
#elif defined(__GNUC__) && (__GNUC__ >= 4)
return (unsigned)(__builtin_clzll(val));
#elif defined(__ICCARM__)
return (unsigned)(__builtin_clzll(val));
#else
{
U32 mostSignificantWord = (U32)(val >> 32);
U32 leastSignificantWord = (U32)val;
if (mostSignificantWord == 0) {
return 32 + ZSTD_countLeadingZeros32(leastSignificantWord);
} else {
return ZSTD_countLeadingZeros32(mostSignificantWord);
}
}
#endif
}
MEM_STATIC unsigned ZSTD_NbCommonBytes(size_t val)
{
if (MEM_isLittleEndian()) {
if (MEM_64bits()) {
return ZSTD_countTrailingZeros64((U64)val) >> 3;
} else {
return ZSTD_countTrailingZeros32((U32)val) >> 3;
}
} else { /* Big Endian CPU */
if (MEM_64bits()) {
return ZSTD_countLeadingZeros64((U64)val) >> 3;
} else {
return ZSTD_countLeadingZeros32((U32)val) >> 3;
}
}
}
MEM_STATIC unsigned ZSTD_highbit32(U32 val) /* compress, dictBuilder, decodeCorpus */
{
assert(val != 0);
return 31 - ZSTD_countLeadingZeros32(val);
}
/* ZSTD_rotateRight_*():
* Rotates a bitfield to the right by "count" bits.
* https://en.wikipedia.org/w/index.php?title=Circular_shift&oldid=991635599#Implementing_circular_shifts
*/
MEM_STATIC
U64 ZSTD_rotateRight_U64(U64 const value, U32 count) {
assert(count < 64);
count &= 0x3F; /* for fickle pattern recognition */
return (value >> count) | (U64)(value << ((0U - count) & 0x3F));
}
MEM_STATIC
U32 ZSTD_rotateRight_U32(U32 const value, U32 count) {
assert(count < 32);
count &= 0x1F; /* for fickle pattern recognition */
return (value >> count) | (U32)(value << ((0U - count) & 0x1F));
}
MEM_STATIC
U16 ZSTD_rotateRight_U16(U16 const value, U32 count) {
assert(count < 16);
count &= 0x0F; /* for fickle pattern recognition */
return (value >> count) | (U16)(value << ((0U - count) & 0x0F));
}
#endif /* ZSTD_BITS_H */
#endif /* USE_EXTERNAL_ZSTD */

457
vendor/github.com/DataDog/zstd/bitstream.h generated vendored
View File

@@ -1,457 +0,0 @@
#ifndef USE_EXTERNAL_ZSTD
/* ******************************************************************
* bitstream
* Part of FSE library
* Copyright (c) Meta Platforms, Inc. and affiliates.
*
* You can contact the author at :
* - Source repository : https://github.com/Cyan4973/FiniteStateEntropy
*
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
* You may select, at your option, one of the above-listed licenses.
****************************************************************** */
#ifndef BITSTREAM_H_MODULE
#define BITSTREAM_H_MODULE
/*
* This API consists of small unitary functions, which must be inlined for best performance.
* Since link-time-optimization is not available for all compilers,
* these functions are defined into a .h to be included.
*/
/*-****************************************
* Dependencies
******************************************/
#include "mem.h" /* unaligned access routines */
#include "compiler.h" /* UNLIKELY() */
#include "debug.h" /* assert(), DEBUGLOG(), RAWLOG() */
#include "error_private.h" /* error codes and messages */
#include "bits.h" /* ZSTD_highbit32 */
/*=========================================
* Target specific
=========================================*/
#ifndef ZSTD_NO_INTRINSICS
# if (defined(__BMI__) || defined(__BMI2__)) && defined(__GNUC__)
# include <immintrin.h> /* support for bextr (experimental)/bzhi */
# elif defined(__ICCARM__)
# include <intrinsics.h>
# endif
#endif
#define STREAM_ACCUMULATOR_MIN_32 25
#define STREAM_ACCUMULATOR_MIN_64 57
#define STREAM_ACCUMULATOR_MIN ((U32)(MEM_32bits() ? STREAM_ACCUMULATOR_MIN_32 : STREAM_ACCUMULATOR_MIN_64))
/*-******************************************
* bitStream encoding API (write forward)
********************************************/
typedef size_t BitContainerType;
/* bitStream can mix input from multiple sources.
* A critical property of these streams is that they encode and decode in **reverse** direction.
* So the first bit sequence you add will be the last to be read, like a LIFO stack.
*/
typedef struct {
BitContainerType bitContainer;
unsigned bitPos;
char* startPtr;
char* ptr;
char* endPtr;
} BIT_CStream_t;
MEM_STATIC size_t BIT_initCStream(BIT_CStream_t* bitC, void* dstBuffer, size_t dstCapacity);
MEM_STATIC void BIT_addBits(BIT_CStream_t* bitC, BitContainerType value, unsigned nbBits);
MEM_STATIC void BIT_flushBits(BIT_CStream_t* bitC);
MEM_STATIC size_t BIT_closeCStream(BIT_CStream_t* bitC);
/* Start with initCStream, providing the size of buffer to write into.
* bitStream will never write outside of this buffer.
* `dstCapacity` must be >= sizeof(bitD->bitContainer), otherwise @return will be an error code.
*
* bits are first added to a local register.
* Local register is BitContainerType, 64-bits on 64-bits systems, or 32-bits on 32-bits systems.
* Writing data into memory is an explicit operation, performed by the flushBits function.
* Hence keep track how many bits are potentially stored into local register to avoid register overflow.
* After a flushBits, a maximum of 7 bits might still be stored into local register.
*
* Avoid storing elements of more than 24 bits if you want compatibility with 32-bits bitstream readers.
*
* Last operation is to close the bitStream.
* The function returns the final size of CStream in bytes.
* If data couldn't fit into `dstBuffer`, it will return a 0 ( == not storable)
*/
/*-********************************************
* bitStream decoding API (read backward)
**********************************************/
typedef struct {
BitContainerType bitContainer;
unsigned bitsConsumed;
const char* ptr;
const char* start;
const char* limitPtr;
} BIT_DStream_t;
typedef enum { BIT_DStream_unfinished = 0, /* fully refilled */
BIT_DStream_endOfBuffer = 1, /* still some bits left in bitstream */
BIT_DStream_completed = 2, /* bitstream entirely consumed, bit-exact */
BIT_DStream_overflow = 3 /* user requested more bits than present in bitstream */
} BIT_DStream_status; /* result of BIT_reloadDStream() */
MEM_STATIC size_t BIT_initDStream(BIT_DStream_t* bitD, const void* srcBuffer, size_t srcSize);
MEM_STATIC BitContainerType BIT_readBits(BIT_DStream_t* bitD, unsigned nbBits);
MEM_STATIC BIT_DStream_status BIT_reloadDStream(BIT_DStream_t* bitD);
MEM_STATIC unsigned BIT_endOfDStream(const BIT_DStream_t* bitD);
/* Start by invoking BIT_initDStream().
* A chunk of the bitStream is then stored into a local register.
* Local register size is 64-bits on 64-bits systems, 32-bits on 32-bits systems (BitContainerType).
* You can then retrieve bitFields stored into the local register, **in reverse order**.
* Local register is explicitly reloaded from memory by the BIT_reloadDStream() method.
* A reload guarantee a minimum of ((8*sizeof(bitD->bitContainer))-7) bits when its result is BIT_DStream_unfinished.
* Otherwise, it can be less than that, so proceed accordingly.
* Checking if DStream has reached its end can be performed with BIT_endOfDStream().
*/
/*-****************************************
* unsafe API
******************************************/
MEM_STATIC void BIT_addBitsFast(BIT_CStream_t* bitC, BitContainerType value, unsigned nbBits);
/* faster, but works only if value is "clean", meaning all high bits above nbBits are 0 */
MEM_STATIC void BIT_flushBitsFast(BIT_CStream_t* bitC);
/* unsafe version; does not check buffer overflow */
MEM_STATIC size_t BIT_readBitsFast(BIT_DStream_t* bitD, unsigned nbBits);
/* faster, but works only if nbBits >= 1 */
/*===== Local Constants =====*/
static const unsigned BIT_mask[] = {
0, 1, 3, 7, 0xF, 0x1F,
0x3F, 0x7F, 0xFF, 0x1FF, 0x3FF, 0x7FF,
0xFFF, 0x1FFF, 0x3FFF, 0x7FFF, 0xFFFF, 0x1FFFF,
0x3FFFF, 0x7FFFF, 0xFFFFF, 0x1FFFFF, 0x3FFFFF, 0x7FFFFF,
0xFFFFFF, 0x1FFFFFF, 0x3FFFFFF, 0x7FFFFFF, 0xFFFFFFF, 0x1FFFFFFF,
0x3FFFFFFF, 0x7FFFFFFF}; /* up to 31 bits */
#define BIT_MASK_SIZE (sizeof(BIT_mask) / sizeof(BIT_mask[0]))
/*-**************************************************************
* bitStream encoding
****************************************************************/
/*! BIT_initCStream() :
* `dstCapacity` must be > sizeof(size_t)
* @return : 0 if success,
* otherwise an error code (can be tested using ERR_isError()) */
MEM_STATIC size_t BIT_initCStream(BIT_CStream_t* bitC,
void* startPtr, size_t dstCapacity)
{
bitC->bitContainer = 0;
bitC->bitPos = 0;
bitC->startPtr = (char*)startPtr;
bitC->ptr = bitC->startPtr;
bitC->endPtr = bitC->startPtr + dstCapacity - sizeof(bitC->bitContainer);
if (dstCapacity <= sizeof(bitC->bitContainer)) return ERROR(dstSize_tooSmall);
return 0;
}
FORCE_INLINE_TEMPLATE BitContainerType BIT_getLowerBits(BitContainerType bitContainer, U32 const nbBits)
{
#if STATIC_BMI2 && !defined(ZSTD_NO_INTRINSICS)
# if (defined(__x86_64__) || defined(_M_X64)) && !defined(__ILP32__)
return _bzhi_u64(bitContainer, nbBits);
# else
DEBUG_STATIC_ASSERT(sizeof(bitContainer) == sizeof(U32));
return _bzhi_u32(bitContainer, nbBits);
# endif
#else
assert(nbBits < BIT_MASK_SIZE);
return bitContainer & BIT_mask[nbBits];
#endif
}
/*! BIT_addBits() :
* can add up to 31 bits into `bitC`.
* Note : does not check for register overflow ! */
MEM_STATIC void BIT_addBits(BIT_CStream_t* bitC,
BitContainerType value, unsigned nbBits)
{
DEBUG_STATIC_ASSERT(BIT_MASK_SIZE == 32);
assert(nbBits < BIT_MASK_SIZE);
assert(nbBits + bitC->bitPos < sizeof(bitC->bitContainer) * 8);
bitC->bitContainer |= BIT_getLowerBits(value, nbBits) << bitC->bitPos;
bitC->bitPos += nbBits;
}
/*! BIT_addBitsFast() :
* works only if `value` is _clean_,
* meaning all high bits above nbBits are 0 */
MEM_STATIC void BIT_addBitsFast(BIT_CStream_t* bitC,
BitContainerType value, unsigned nbBits)
{
assert((value>>nbBits) == 0);
assert(nbBits + bitC->bitPos < sizeof(bitC->bitContainer) * 8);
bitC->bitContainer |= value << bitC->bitPos;
bitC->bitPos += nbBits;
}
/*! BIT_flushBitsFast() :
* assumption : bitContainer has not overflowed
* unsafe version; does not check buffer overflow */
MEM_STATIC void BIT_flushBitsFast(BIT_CStream_t* bitC)
{
size_t const nbBytes = bitC->bitPos >> 3;
assert(bitC->bitPos < sizeof(bitC->bitContainer) * 8);
assert(bitC->ptr <= bitC->endPtr);
MEM_writeLEST(bitC->ptr, bitC->bitContainer);
bitC->ptr += nbBytes;
bitC->bitPos &= 7;
bitC->bitContainer >>= nbBytes*8;
}
/*! BIT_flushBits() :
* assumption : bitContainer has not overflowed
* safe version; check for buffer overflow, and prevents it.
* note : does not signal buffer overflow.
* overflow will be revealed later on using BIT_closeCStream() */
MEM_STATIC void BIT_flushBits(BIT_CStream_t* bitC)
{
size_t const nbBytes = bitC->bitPos >> 3;
assert(bitC->bitPos < sizeof(bitC->bitContainer) * 8);
assert(bitC->ptr <= bitC->endPtr);
MEM_writeLEST(bitC->ptr, bitC->bitContainer);
bitC->ptr += nbBytes;
if (bitC->ptr > bitC->endPtr) bitC->ptr = bitC->endPtr;
bitC->bitPos &= 7;
bitC->bitContainer >>= nbBytes*8;
}
/*! BIT_closeCStream() :
* @return : size of CStream, in bytes,
* or 0 if it could not fit into dstBuffer */
MEM_STATIC size_t BIT_closeCStream(BIT_CStream_t* bitC)
{
BIT_addBitsFast(bitC, 1, 1); /* endMark */
BIT_flushBits(bitC);
if (bitC->ptr >= bitC->endPtr) return 0; /* overflow detected */
return (size_t)(bitC->ptr - bitC->startPtr) + (bitC->bitPos > 0);
}
/*-********************************************************
* bitStream decoding
**********************************************************/
/*! BIT_initDStream() :
* Initialize a BIT_DStream_t.
* `bitD` : a pointer to an already allocated BIT_DStream_t structure.
* `srcSize` must be the *exact* size of the bitStream, in bytes.
* @return : size of stream (== srcSize), or an errorCode if a problem is detected
*/
MEM_STATIC size_t BIT_initDStream(BIT_DStream_t* bitD, const void* srcBuffer, size_t srcSize)
{
if (srcSize < 1) { ZSTD_memset(bitD, 0, sizeof(*bitD)); return ERROR(srcSize_wrong); }
bitD->start = (const char*)srcBuffer;
bitD->limitPtr = bitD->start + sizeof(bitD->bitContainer);
if (srcSize >= sizeof(bitD->bitContainer)) { /* normal case */
bitD->ptr = (const char*)srcBuffer + srcSize - sizeof(bitD->bitContainer);
bitD->bitContainer = MEM_readLEST(bitD->ptr);
{ BYTE const lastByte = ((const BYTE*)srcBuffer)[srcSize-1];
bitD->bitsConsumed = lastByte ? 8 - ZSTD_highbit32(lastByte) : 0; /* ensures bitsConsumed is always set */
if (lastByte == 0) return ERROR(GENERIC); /* endMark not present */ }
} else {
bitD->ptr = bitD->start;
bitD->bitContainer = *(const BYTE*)(bitD->start);
switch(srcSize)
{
case 7: bitD->bitContainer += (BitContainerType)(((const BYTE*)(srcBuffer))[6]) << (sizeof(bitD->bitContainer)*8 - 16);
ZSTD_FALLTHROUGH;
case 6: bitD->bitContainer += (BitContainerType)(((const BYTE*)(srcBuffer))[5]) << (sizeof(bitD->bitContainer)*8 - 24);
ZSTD_FALLTHROUGH;
case 5: bitD->bitContainer += (BitContainerType)(((const BYTE*)(srcBuffer))[4]) << (sizeof(bitD->bitContainer)*8 - 32);
ZSTD_FALLTHROUGH;
case 4: bitD->bitContainer += (BitContainerType)(((const BYTE*)(srcBuffer))[3]) << 24;
ZSTD_FALLTHROUGH;
case 3: bitD->bitContainer += (BitContainerType)(((const BYTE*)(srcBuffer))[2]) << 16;
ZSTD_FALLTHROUGH;
case 2: bitD->bitContainer += (BitContainerType)(((const BYTE*)(srcBuffer))[1]) << 8;
ZSTD_FALLTHROUGH;
default: break;
}
{ BYTE const lastByte = ((const BYTE*)srcBuffer)[srcSize-1];
bitD->bitsConsumed = lastByte ? 8 - ZSTD_highbit32(lastByte) : 0;
if (lastByte == 0) return ERROR(corruption_detected); /* endMark not present */
}
bitD->bitsConsumed += (U32)(sizeof(bitD->bitContainer) - srcSize)*8;
}
return srcSize;
}
FORCE_INLINE_TEMPLATE BitContainerType BIT_getUpperBits(BitContainerType bitContainer, U32 const start)
{
return bitContainer >> start;
}
FORCE_INLINE_TEMPLATE BitContainerType BIT_getMiddleBits(BitContainerType bitContainer, U32 const start, U32 const nbBits)
{
U32 const regMask = sizeof(bitContainer)*8 - 1;
/* if start > regMask, bitstream is corrupted, and result is undefined */
assert(nbBits < BIT_MASK_SIZE);
/* x86 transform & ((1 << nbBits) - 1) to bzhi instruction, it is better
* than accessing memory. When bmi2 instruction is not present, we consider
* such cpus old (pre-Haswell, 2013) and their performance is not of that
* importance.
*/
#if defined(__x86_64__) || defined(_M_X64)
return (bitContainer >> (start & regMask)) & ((((U64)1) << nbBits) - 1);
#else
return (bitContainer >> (start & regMask)) & BIT_mask[nbBits];
#endif
}
/*! BIT_lookBits() :
* Provides next n bits from local register.
* local register is not modified.
* On 32-bits, maxNbBits==24.
* On 64-bits, maxNbBits==56.
* @return : value extracted */
FORCE_INLINE_TEMPLATE BitContainerType BIT_lookBits(const BIT_DStream_t* bitD, U32 nbBits)
{
/* arbitrate between double-shift and shift+mask */
#if 1
/* if bitD->bitsConsumed + nbBits > sizeof(bitD->bitContainer)*8,
* bitstream is likely corrupted, and result is undefined */
return BIT_getMiddleBits(bitD->bitContainer, (sizeof(bitD->bitContainer)*8) - bitD->bitsConsumed - nbBits, nbBits);
#else
/* this code path is slower on my os-x laptop */
U32 const regMask = sizeof(bitD->bitContainer)*8 - 1;
return ((bitD->bitContainer << (bitD->bitsConsumed & regMask)) >> 1) >> ((regMask-nbBits) & regMask);
#endif
}
/*! BIT_lookBitsFast() :
* unsafe version; only works if nbBits >= 1 */
MEM_STATIC BitContainerType BIT_lookBitsFast(const BIT_DStream_t* bitD, U32 nbBits)
{
U32 const regMask = sizeof(bitD->bitContainer)*8 - 1;
assert(nbBits >= 1);
return (bitD->bitContainer << (bitD->bitsConsumed & regMask)) >> (((regMask+1)-nbBits) & regMask);
}
FORCE_INLINE_TEMPLATE void BIT_skipBits(BIT_DStream_t* bitD, U32 nbBits)
{
bitD->bitsConsumed += nbBits;
}
/*! BIT_readBits() :
* Read (consume) next n bits from local register and update.
* Pay attention to not read more than nbBits contained into local register.
* @return : extracted value. */
FORCE_INLINE_TEMPLATE BitContainerType BIT_readBits(BIT_DStream_t* bitD, unsigned nbBits)
{
BitContainerType const value = BIT_lookBits(bitD, nbBits);
BIT_skipBits(bitD, nbBits);
return value;
}
/*! BIT_readBitsFast() :
* unsafe version; only works if nbBits >= 1 */
MEM_STATIC BitContainerType BIT_readBitsFast(BIT_DStream_t* bitD, unsigned nbBits)
{
BitContainerType const value = BIT_lookBitsFast(bitD, nbBits);
assert(nbBits >= 1);
BIT_skipBits(bitD, nbBits);
return value;
}
/*! BIT_reloadDStream_internal() :
* Simple variant of BIT_reloadDStream(), with two conditions:
* 1. bitstream is valid : bitsConsumed <= sizeof(bitD->bitContainer)*8
* 2. look window is valid after shifted down : bitD->ptr >= bitD->start
*/
MEM_STATIC BIT_DStream_status BIT_reloadDStream_internal(BIT_DStream_t* bitD)
{
assert(bitD->bitsConsumed <= sizeof(bitD->bitContainer)*8);
bitD->ptr -= bitD->bitsConsumed >> 3;
assert(bitD->ptr >= bitD->start);
bitD->bitsConsumed &= 7;
bitD->bitContainer = MEM_readLEST(bitD->ptr);
return BIT_DStream_unfinished;
}
/*! BIT_reloadDStreamFast() :
* Similar to BIT_reloadDStream(), but with two differences:
* 1. bitsConsumed <= sizeof(bitD->bitContainer)*8 must hold!
* 2. Returns BIT_DStream_overflow when bitD->ptr < bitD->limitPtr, at this
* point you must use BIT_reloadDStream() to reload.
*/
MEM_STATIC BIT_DStream_status BIT_reloadDStreamFast(BIT_DStream_t* bitD)
{
if (UNLIKELY(bitD->ptr < bitD->limitPtr))
return BIT_DStream_overflow;
return BIT_reloadDStream_internal(bitD);
}
/*! BIT_reloadDStream() :
* Refill `bitD` from buffer previously set in BIT_initDStream() .
* This function is safe, it guarantees it will not never beyond src buffer.
* @return : status of `BIT_DStream_t` internal register.
* when status == BIT_DStream_unfinished, internal register is filled with at least 25 or 57 bits */
FORCE_INLINE_TEMPLATE BIT_DStream_status BIT_reloadDStream(BIT_DStream_t* bitD)
{
/* note : once in overflow mode, a bitstream remains in this mode until it's reset */
if (UNLIKELY(bitD->bitsConsumed > (sizeof(bitD->bitContainer)*8))) {
static const BitContainerType zeroFilled = 0;
bitD->ptr = (const char*)&zeroFilled; /* aliasing is allowed for char */
/* overflow detected, erroneous scenario or end of stream: no update */
return BIT_DStream_overflow;
}
assert(bitD->ptr >= bitD->start);
if (bitD->ptr >= bitD->limitPtr) {
return BIT_reloadDStream_internal(bitD);
}
if (bitD->ptr == bitD->start) {
/* reached end of bitStream => no update */
if (bitD->bitsConsumed < sizeof(bitD->bitContainer)*8) return BIT_DStream_endOfBuffer;
return BIT_DStream_completed;
}
/* start < ptr < limitPtr => cautious update */
{ U32 nbBytes = bitD->bitsConsumed >> 3;
BIT_DStream_status result = BIT_DStream_unfinished;
if (bitD->ptr - nbBytes < bitD->start) {
nbBytes = (U32)(bitD->ptr - bitD->start); /* ptr > start */
result = BIT_DStream_endOfBuffer;
}
bitD->ptr -= nbBytes;
bitD->bitsConsumed -= nbBytes*8;
bitD->bitContainer = MEM_readLEST(bitD->ptr); /* reminder : srcSize > sizeof(bitD->bitContainer), otherwise bitD->ptr == bitD->start */
return result;
}
}
/*! BIT_endOfDStream() :
* @return : 1 if DStream has _exactly_ reached its end (all bits consumed).
*/
MEM_STATIC unsigned BIT_endOfDStream(const BIT_DStream_t* DStream)
{
return ((DStream->ptr == DStream->start) && (DStream->bitsConsumed == sizeof(DStream->bitContainer)*8));
}
#endif /* BITSTREAM_H_MODULE */
#endif /* USE_EXTERNAL_ZSTD */

137
vendor/github.com/DataDog/zstd/clevels.h generated vendored
View File

@@ -1,137 +0,0 @@
#ifndef USE_EXTERNAL_ZSTD
/*
* Copyright (c) Meta Platforms, Inc. and affiliates.
* All rights reserved.
*
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
* You may select, at your option, one of the above-listed licenses.
*/
#ifndef ZSTD_CLEVELS_H
#define ZSTD_CLEVELS_H
#define ZSTD_STATIC_LINKING_ONLY /* ZSTD_compressionParameters */
#include "zstd.h"
/*-===== Pre-defined compression levels =====-*/
#define ZSTD_MAX_CLEVEL 22
#ifdef __GNUC__
__attribute__((__unused__))
#endif
static const ZSTD_compressionParameters ZSTD_defaultCParameters[4][ZSTD_MAX_CLEVEL+1] = {
{ /* "default" - for any srcSize > 256 KB */
/* W, C, H, S, L, TL, strat */
{ 19, 12, 13, 1, 6, 1, ZSTD_fast }, /* base for negative levels */
{ 19, 13, 14, 1, 7, 0, ZSTD_fast }, /* level 1 */
{ 20, 15, 16, 1, 6, 0, ZSTD_fast }, /* level 2 */
{ 21, 16, 17, 1, 5, 0, ZSTD_dfast }, /* level 3 */
{ 21, 18, 18, 1, 5, 0, ZSTD_dfast }, /* level 4 */
{ 21, 18, 19, 3, 5, 2, ZSTD_greedy }, /* level 5 */
{ 21, 18, 19, 3, 5, 4, ZSTD_lazy }, /* level 6 */
{ 21, 19, 20, 4, 5, 8, ZSTD_lazy }, /* level 7 */
{ 21, 19, 20, 4, 5, 16, ZSTD_lazy2 }, /* level 8 */
{ 22, 20, 21, 4, 5, 16, ZSTD_lazy2 }, /* level 9 */
{ 22, 21, 22, 5, 5, 16, ZSTD_lazy2 }, /* level 10 */
{ 22, 21, 22, 6, 5, 16, ZSTD_lazy2 }, /* level 11 */
{ 22, 22, 23, 6, 5, 32, ZSTD_lazy2 }, /* level 12 */
{ 22, 22, 22, 4, 5, 32, ZSTD_btlazy2 }, /* level 13 */
{ 22, 22, 23, 5, 5, 32, ZSTD_btlazy2 }, /* level 14 */
{ 22, 23, 23, 6, 5, 32, ZSTD_btlazy2 }, /* level 15 */
{ 22, 22, 22, 5, 5, 48, ZSTD_btopt }, /* level 16 */
{ 23, 23, 22, 5, 4, 64, ZSTD_btopt }, /* level 17 */
{ 23, 23, 22, 6, 3, 64, ZSTD_btultra }, /* level 18 */
{ 23, 24, 22, 7, 3,256, ZSTD_btultra2}, /* level 19 */
{ 25, 25, 23, 7, 3,256, ZSTD_btultra2}, /* level 20 */
{ 26, 26, 24, 7, 3,512, ZSTD_btultra2}, /* level 21 */
{ 27, 27, 25, 9, 3,999, ZSTD_btultra2}, /* level 22 */
},
{ /* for srcSize <= 256 KB */
/* W, C, H, S, L, T, strat */
{ 18, 12, 13, 1, 5, 1, ZSTD_fast }, /* base for negative levels */
{ 18, 13, 14, 1, 6, 0, ZSTD_fast }, /* level 1 */
{ 18, 14, 14, 1, 5, 0, ZSTD_dfast }, /* level 2 */
{ 18, 16, 16, 1, 4, 0, ZSTD_dfast }, /* level 3 */
{ 18, 16, 17, 3, 5, 2, ZSTD_greedy }, /* level 4.*/
{ 18, 17, 18, 5, 5, 2, ZSTD_greedy }, /* level 5.*/
{ 18, 18, 19, 3, 5, 4, ZSTD_lazy }, /* level 6.*/
{ 18, 18, 19, 4, 4, 4, ZSTD_lazy }, /* level 7 */
{ 18, 18, 19, 4, 4, 8, ZSTD_lazy2 }, /* level 8 */
{ 18, 18, 19, 5, 4, 8, ZSTD_lazy2 }, /* level 9 */
{ 18, 18, 19, 6, 4, 8, ZSTD_lazy2 }, /* level 10 */
{ 18, 18, 19, 5, 4, 12, ZSTD_btlazy2 }, /* level 11.*/
{ 18, 19, 19, 7, 4, 12, ZSTD_btlazy2 }, /* level 12.*/
{ 18, 18, 19, 4, 4, 16, ZSTD_btopt }, /* level 13 */
{ 18, 18, 19, 4, 3, 32, ZSTD_btopt }, /* level 14.*/
{ 18, 18, 19, 6, 3,128, ZSTD_btopt }, /* level 15.*/
{ 18, 19, 19, 6, 3,128, ZSTD_btultra }, /* level 16.*/
{ 18, 19, 19, 8, 3,256, ZSTD_btultra }, /* level 17.*/
{ 18, 19, 19, 6, 3,128, ZSTD_btultra2}, /* level 18.*/
{ 18, 19, 19, 8, 3,256, ZSTD_btultra2}, /* level 19.*/
{ 18, 19, 19, 10, 3,512, ZSTD_btultra2}, /* level 20.*/
{ 18, 19, 19, 12, 3,512, ZSTD_btultra2}, /* level 21.*/
{ 18, 19, 19, 13, 3,999, ZSTD_btultra2}, /* level 22.*/
},
{ /* for srcSize <= 128 KB */
/* W, C, H, S, L, T, strat */
{ 17, 12, 12, 1, 5, 1, ZSTD_fast }, /* base for negative levels */
{ 17, 12, 13, 1, 6, 0, ZSTD_fast }, /* level 1 */
{ 17, 13, 15, 1, 5, 0, ZSTD_fast }, /* level 2 */
{ 17, 15, 16, 2, 5, 0, ZSTD_dfast }, /* level 3 */
{ 17, 17, 17, 2, 4, 0, ZSTD_dfast }, /* level 4 */
{ 17, 16, 17, 3, 4, 2, ZSTD_greedy }, /* level 5 */
{ 17, 16, 17, 3, 4, 4, ZSTD_lazy }, /* level 6 */
{ 17, 16, 17, 3, 4, 8, ZSTD_lazy2 }, /* level 7 */
{ 17, 16, 17, 4, 4, 8, ZSTD_lazy2 }, /* level 8 */
{ 17, 16, 17, 5, 4, 8, ZSTD_lazy2 }, /* level 9 */
{ 17, 16, 17, 6, 4, 8, ZSTD_lazy2 }, /* level 10 */
{ 17, 17, 17, 5, 4, 8, ZSTD_btlazy2 }, /* level 11 */
{ 17, 18, 17, 7, 4, 12, ZSTD_btlazy2 }, /* level 12 */
{ 17, 18, 17, 3, 4, 12, ZSTD_btopt }, /* level 13.*/
{ 17, 18, 17, 4, 3, 32, ZSTD_btopt }, /* level 14.*/
{ 17, 18, 17, 6, 3,256, ZSTD_btopt }, /* level 15.*/
{ 17, 18, 17, 6, 3,128, ZSTD_btultra }, /* level 16.*/
{ 17, 18, 17, 8, 3,256, ZSTD_btultra }, /* level 17.*/
{ 17, 18, 17, 10, 3,512, ZSTD_btultra }, /* level 18.*/
{ 17, 18, 17, 5, 3,256, ZSTD_btultra2}, /* level 19.*/
{ 17, 18, 17, 7, 3,512, ZSTD_btultra2}, /* level 20.*/
{ 17, 18, 17, 9, 3,512, ZSTD_btultra2}, /* level 21.*/
{ 17, 18, 17, 11, 3,999, ZSTD_btultra2}, /* level 22.*/
},
{ /* for srcSize <= 16 KB */
/* W, C, H, S, L, T, strat */
{ 14, 12, 13, 1, 5, 1, ZSTD_fast }, /* base for negative levels */
{ 14, 14, 15, 1, 5, 0, ZSTD_fast }, /* level 1 */
{ 14, 14, 15, 1, 4, 0, ZSTD_fast }, /* level 2 */
{ 14, 14, 15, 2, 4, 0, ZSTD_dfast }, /* level 3 */
{ 14, 14, 14, 4, 4, 2, ZSTD_greedy }, /* level 4 */
{ 14, 14, 14, 3, 4, 4, ZSTD_lazy }, /* level 5.*/
{ 14, 14, 14, 4, 4, 8, ZSTD_lazy2 }, /* level 6 */
{ 14, 14, 14, 6, 4, 8, ZSTD_lazy2 }, /* level 7 */
{ 14, 14, 14, 8, 4, 8, ZSTD_lazy2 }, /* level 8.*/
{ 14, 15, 14, 5, 4, 8, ZSTD_btlazy2 }, /* level 9.*/
{ 14, 15, 14, 9, 4, 8, ZSTD_btlazy2 }, /* level 10.*/
{ 14, 15, 14, 3, 4, 12, ZSTD_btopt }, /* level 11.*/
{ 14, 15, 14, 4, 3, 24, ZSTD_btopt }, /* level 12.*/
{ 14, 15, 14, 5, 3, 32, ZSTD_btultra }, /* level 13.*/
{ 14, 15, 15, 6, 3, 64, ZSTD_btultra }, /* level 14.*/
{ 14, 15, 15, 7, 3,256, ZSTD_btultra }, /* level 15.*/
{ 14, 15, 15, 5, 3, 48, ZSTD_btultra2}, /* level 16.*/
{ 14, 15, 15, 6, 3,128, ZSTD_btultra2}, /* level 17.*/
{ 14, 15, 15, 7, 3,256, ZSTD_btultra2}, /* level 18.*/
{ 14, 15, 15, 8, 3,256, ZSTD_btultra2}, /* level 19.*/
{ 14, 15, 15, 8, 3,512, ZSTD_btultra2}, /* level 20.*/
{ 14, 15, 15, 9, 3,512, ZSTD_btultra2}, /* level 21.*/
{ 14, 15, 15, 10, 3,999, ZSTD_btultra2}, /* level 22.*/
},
};
#endif /* ZSTD_CLEVELS_H */
#endif /* USE_EXTERNAL_ZSTD */

467
vendor/github.com/DataDog/zstd/compiler.h generated vendored
View File

@@ -1,467 +0,0 @@
#ifndef USE_EXTERNAL_ZSTD
/*
* Copyright (c) Meta Platforms, Inc. and affiliates.
* All rights reserved.
*
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
* You may select, at your option, one of the above-listed licenses.
*/
#ifndef ZSTD_COMPILER_H
#define ZSTD_COMPILER_H
#include <stddef.h>
#include "portability_macros.h"
/*-*******************************************************
* Compiler specifics
*********************************************************/
/* force inlining */
#if !defined(ZSTD_NO_INLINE)
#if (defined(__GNUC__) && !defined(__STRICT_ANSI__)) || defined(__cplusplus) || defined(__STDC_VERSION__) && __STDC_VERSION__ >= 199901L /* C99 */
# define INLINE_KEYWORD inline
#else
# define INLINE_KEYWORD
#endif
#if defined(__GNUC__) || defined(__IAR_SYSTEMS_ICC__)
# define FORCE_INLINE_ATTR __attribute__((always_inline))
#elif defined(_MSC_VER)
# define FORCE_INLINE_ATTR __forceinline
#else
# define FORCE_INLINE_ATTR
#endif
#else
#define INLINE_KEYWORD
#define FORCE_INLINE_ATTR
#endif
/**
On MSVC qsort requires that functions passed into it use the __cdecl calling conversion(CC).
This explicitly marks such functions as __cdecl so that the code will still compile
if a CC other than __cdecl has been made the default.
*/
#if defined(_MSC_VER)
# define WIN_CDECL __cdecl
#else
# define WIN_CDECL
#endif
/* UNUSED_ATTR tells the compiler it is okay if the function is unused. */
#if defined(__GNUC__) || defined(__IAR_SYSTEMS_ICC__)
# define UNUSED_ATTR __attribute__((unused))
#else
# define UNUSED_ATTR
#endif
/**
* FORCE_INLINE_TEMPLATE is used to define C "templates", which take constant
* parameters. They must be inlined for the compiler to eliminate the constant
* branches.
*/
#define FORCE_INLINE_TEMPLATE static INLINE_KEYWORD FORCE_INLINE_ATTR UNUSED_ATTR
/**
* HINT_INLINE is used to help the compiler generate better code. It is *not*
* used for "templates", so it can be tweaked based on the compilers
* performance.
*
* gcc-4.8 and gcc-4.9 have been shown to benefit from leaving off the
* always_inline attribute.
*
* clang up to 5.0.0 (trunk) benefit tremendously from the always_inline
* attribute.
*/
#if !defined(__clang__) && defined(__GNUC__) && __GNUC__ >= 4 && __GNUC_MINOR__ >= 8 && __GNUC__ < 5
# define HINT_INLINE static INLINE_KEYWORD
#else
# define HINT_INLINE FORCE_INLINE_TEMPLATE
#endif
/* "soft" inline :
* The compiler is free to select if it's a good idea to inline or not.
* The main objective is to silence compiler warnings
* when a defined function in included but not used.
*
* Note : this macro is prefixed `MEM_` because it used to be provided by `mem.h` unit.
* Updating the prefix is probably preferable, but requires a fairly large codemod,
* since this name is used everywhere.
*/
#ifndef MEM_STATIC /* already defined in Linux Kernel mem.h */
#if defined(__GNUC__)
# define MEM_STATIC static __inline UNUSED_ATTR
#elif defined(__IAR_SYSTEMS_ICC__)
# define MEM_STATIC static inline UNUSED_ATTR
#elif defined (__cplusplus) || (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */)
# define MEM_STATIC static inline
#elif defined(_MSC_VER)
# define MEM_STATIC static __inline
#else
# define MEM_STATIC static /* this version may generate warnings for unused static functions; disable the relevant warning */
#endif
#endif
/* force no inlining */
#ifdef _MSC_VER
# define FORCE_NOINLINE static __declspec(noinline)
#else
# if defined(__GNUC__) || defined(__IAR_SYSTEMS_ICC__)
# define FORCE_NOINLINE static __attribute__((__noinline__))
# else
# define FORCE_NOINLINE static
# endif
#endif
/* target attribute */
#if defined(__GNUC__) || defined(__IAR_SYSTEMS_ICC__)
# define TARGET_ATTRIBUTE(target) __attribute__((__target__(target)))
#else
# define TARGET_ATTRIBUTE(target)
#endif
/* Target attribute for BMI2 dynamic dispatch.
* Enable lzcnt, bmi, and bmi2.
* We test for bmi1 & bmi2. lzcnt is included in bmi1.
*/
#define BMI2_TARGET_ATTRIBUTE TARGET_ATTRIBUTE("lzcnt,bmi,bmi2")
/* prefetch
* can be disabled, by declaring NO_PREFETCH build macro */
#if defined(NO_PREFETCH)
# define PREFETCH_L1(ptr) do { (void)(ptr); } while (0) /* disabled */
# define PREFETCH_L2(ptr) do { (void)(ptr); } while (0) /* disabled */
#else
# if defined(_MSC_VER) && (defined(_M_X64) || defined(_M_I86)) && !defined(_M_ARM64EC) /* _mm_prefetch() is not defined outside of x86/x64 */
# include <mmintrin.h> /* https://msdn.microsoft.com/fr-fr/library/84szxsww(v=vs.90).aspx */
# define PREFETCH_L1(ptr) _mm_prefetch((const char*)(ptr), _MM_HINT_T0)
# define PREFETCH_L2(ptr) _mm_prefetch((const char*)(ptr), _MM_HINT_T1)
# elif defined(__GNUC__) && ( (__GNUC__ >= 4) || ( (__GNUC__ == 3) && (__GNUC_MINOR__ >= 1) ) )
# define PREFETCH_L1(ptr) __builtin_prefetch((ptr), 0 /* rw==read */, 3 /* locality */)
# define PREFETCH_L2(ptr) __builtin_prefetch((ptr), 0 /* rw==read */, 2 /* locality */)
# elif defined(__aarch64__)
# define PREFETCH_L1(ptr) do { __asm__ __volatile__("prfm pldl1keep, %0" ::"Q"(*(ptr))); } while (0)
# define PREFETCH_L2(ptr) do { __asm__ __volatile__("prfm pldl2keep, %0" ::"Q"(*(ptr))); } while (0)
# else
# define PREFETCH_L1(ptr) do { (void)(ptr); } while (0) /* disabled */
# define PREFETCH_L2(ptr) do { (void)(ptr); } while (0) /* disabled */
# endif
#endif /* NO_PREFETCH */
#define CACHELINE_SIZE 64
#define PREFETCH_AREA(p, s) \
do { \
const char* const _ptr = (const char*)(p); \
size_t const _size = (size_t)(s); \
size_t _pos; \
for (_pos=0; _pos<_size; _pos+=CACHELINE_SIZE) { \
PREFETCH_L2(_ptr + _pos); \
} \
} while (0)
/* vectorization
* older GCC (pre gcc-4.3 picked as the cutoff) uses a different syntax,
* and some compilers, like Intel ICC and MCST LCC, do not support it at all. */
#if !defined(__INTEL_COMPILER) && !defined(__clang__) && defined(__GNUC__) && !defined(__LCC__)
# if (__GNUC__ == 4 && __GNUC_MINOR__ > 3) || (__GNUC__ >= 5)
# define DONT_VECTORIZE __attribute__((optimize("no-tree-vectorize")))
# else
# define DONT_VECTORIZE _Pragma("GCC optimize(\"no-tree-vectorize\")")
# endif
#else
# define DONT_VECTORIZE
#endif
/* Tell the compiler that a branch is likely or unlikely.
* Only use these macros if it causes the compiler to generate better code.
* If you can remove a LIKELY/UNLIKELY annotation without speed changes in gcc
* and clang, please do.
*/
#if defined(__GNUC__)
#define LIKELY(x) (__builtin_expect((x), 1))
#define UNLIKELY(x) (__builtin_expect((x), 0))
#else
#define LIKELY(x) (x)
#define UNLIKELY(x) (x)
#endif
#if __has_builtin(__builtin_unreachable) || (defined(__GNUC__) && (__GNUC__ > 4 || (__GNUC__ == 4 && __GNUC_MINOR__ >= 5)))
# define ZSTD_UNREACHABLE do { assert(0), __builtin_unreachable(); } while (0)
#else
# define ZSTD_UNREACHABLE do { assert(0); } while (0)
#endif
/* disable warnings */
#ifdef _MSC_VER /* Visual Studio */
# include <intrin.h> /* For Visual 2005 */
# pragma warning(disable : 4100) /* disable: C4100: unreferenced formal parameter */
# pragma warning(disable : 4127) /* disable: C4127: conditional expression is constant */
# pragma warning(disable : 4204) /* disable: C4204: non-constant aggregate initializer */
# pragma warning(disable : 4214) /* disable: C4214: non-int bitfields */
# pragma warning(disable : 4324) /* disable: C4324: padded structure */
#endif
/* compile time determination of SIMD support */
#if !defined(ZSTD_NO_INTRINSICS)
# if defined(__AVX2__)
# define ZSTD_ARCH_X86_AVX2
# endif
# if defined(__SSE2__) || defined(_M_X64) || (defined (_M_IX86) && defined(_M_IX86_FP) && (_M_IX86_FP >= 2))
# define ZSTD_ARCH_X86_SSE2
# endif
# if defined(__ARM_NEON) || defined(_M_ARM64)
# define ZSTD_ARCH_ARM_NEON
# endif
#
# if defined(ZSTD_ARCH_X86_AVX2)
# include <immintrin.h>
# endif
# if defined(ZSTD_ARCH_X86_SSE2)
# include <emmintrin.h>
# elif defined(ZSTD_ARCH_ARM_NEON)
# include <arm_neon.h>
# endif
#endif
/* C-language Attributes are added in C23. */
#if defined(__STDC_VERSION__) && (__STDC_VERSION__ > 201710L) && defined(__has_c_attribute)
# define ZSTD_HAS_C_ATTRIBUTE(x) __has_c_attribute(x)
#else
# define ZSTD_HAS_C_ATTRIBUTE(x) 0
#endif
/* Only use C++ attributes in C++. Some compilers report support for C++
* attributes when compiling with C.
*/
#if defined(__cplusplus) && defined(__has_cpp_attribute)
# define ZSTD_HAS_CPP_ATTRIBUTE(x) __has_cpp_attribute(x)
#else
# define ZSTD_HAS_CPP_ATTRIBUTE(x) 0
#endif
/* Define ZSTD_FALLTHROUGH macro for annotating switch case with the 'fallthrough' attribute.
* - C23: https://en.cppreference.com/w/c/language/attributes/fallthrough
* - CPP17: https://en.cppreference.com/w/cpp/language/attributes/fallthrough
* - Else: __attribute__((__fallthrough__))
*/
#ifndef ZSTD_FALLTHROUGH
# if ZSTD_HAS_C_ATTRIBUTE(fallthrough)
# define ZSTD_FALLTHROUGH [[fallthrough]]
# elif ZSTD_HAS_CPP_ATTRIBUTE(fallthrough)
# define ZSTD_FALLTHROUGH [[fallthrough]]
# elif __has_attribute(__fallthrough__)
/* Leading semicolon is to satisfy gcc-11 with -pedantic. Without the semicolon
* gcc complains about: a label can only be part of a statement and a declaration is not a statement.
*/
# define ZSTD_FALLTHROUGH ; __attribute__((__fallthrough__))
# else
# define ZSTD_FALLTHROUGH
# endif
#endif
/*-**************************************************************
* Alignment
*****************************************************************/
/* @return 1 if @u is a 2^n value, 0 otherwise
* useful to check a value is valid for alignment restrictions */
MEM_STATIC int ZSTD_isPower2(size_t u) {
return (u & (u-1)) == 0;
}
/* this test was initially positioned in mem.h,
* but this file is removed (or replaced) for linux kernel
* so it's now hosted in compiler.h,
* which remains valid for both user & kernel spaces.
*/
#ifndef ZSTD_ALIGNOF
# if defined(__GNUC__) || defined(_MSC_VER)
/* covers gcc, clang & MSVC */
/* note : this section must come first, before C11,
* due to a limitation in the kernel source generator */
# define ZSTD_ALIGNOF(T) __alignof(T)
# elif defined(__STDC_VERSION__) && (__STDC_VERSION__ >= 201112L)
/* C11 support */
# include <stdalign.h>
# define ZSTD_ALIGNOF(T) alignof(T)
# else
/* No known support for alignof() - imperfect backup */
# define ZSTD_ALIGNOF(T) (sizeof(void*) < sizeof(T) ? sizeof(void*) : sizeof(T))
# endif
#endif /* ZSTD_ALIGNOF */
#ifndef ZSTD_ALIGNED
/* C90-compatible alignment macro (GCC/Clang). Adjust for other compilers if needed. */
# if defined(__GNUC__) || defined(__clang__)
# define ZSTD_ALIGNED(a) __attribute__((aligned(a)))
# elif defined(__STDC_VERSION__) && (__STDC_VERSION__ >= 201112L) /* C11 */
# define ZSTD_ALIGNED(a) _Alignas(a)
#elif defined(_MSC_VER)
# define ZSTD_ALIGNED(n) __declspec(align(n))
# else
/* this compiler will require its own alignment instruction */
# define ZSTD_ALIGNED(...)
# endif
#endif /* ZSTD_ALIGNED */
/*-**************************************************************
* Sanitizer
*****************************************************************/
/**
* Zstd relies on pointer overflow in its decompressor.
* We add this attribute to functions that rely on pointer overflow.
*/
#ifndef ZSTD_ALLOW_POINTER_OVERFLOW_ATTR
# if __has_attribute(no_sanitize)
# if !defined(__clang__) && defined(__GNUC__) && __GNUC__ < 8
/* gcc < 8 only has signed-integer-overlow which triggers on pointer overflow */
# define ZSTD_ALLOW_POINTER_OVERFLOW_ATTR __attribute__((no_sanitize("signed-integer-overflow")))
# else
/* older versions of clang [3.7, 5.0) will warn that pointer-overflow is ignored. */
# define ZSTD_ALLOW_POINTER_OVERFLOW_ATTR __attribute__((no_sanitize("pointer-overflow")))
# endif
# else
# define ZSTD_ALLOW_POINTER_OVERFLOW_ATTR
# endif
#endif
/**
* Helper function to perform a wrapped pointer difference without triggering
* UBSAN.
*
* @returns lhs - rhs with wrapping
*/
MEM_STATIC
ZSTD_ALLOW_POINTER_OVERFLOW_ATTR
ptrdiff_t ZSTD_wrappedPtrDiff(unsigned char const* lhs, unsigned char const* rhs)
{
return lhs - rhs;
}
/**
* Helper function to perform a wrapped pointer add without triggering UBSAN.
*
* @return ptr + add with wrapping
*/
MEM_STATIC
ZSTD_ALLOW_POINTER_OVERFLOW_ATTR
unsigned char const* ZSTD_wrappedPtrAdd(unsigned char const* ptr, ptrdiff_t add)
{
return ptr + add;
}
/**
* Helper function to perform a wrapped pointer subtraction without triggering
* UBSAN.
*
* @return ptr - sub with wrapping
*/
MEM_STATIC
ZSTD_ALLOW_POINTER_OVERFLOW_ATTR
unsigned char const* ZSTD_wrappedPtrSub(unsigned char const* ptr, ptrdiff_t sub)
{
return ptr - sub;
}
/**
* Helper function to add to a pointer that works around C's undefined behavior
* of adding 0 to NULL.
*
* @returns `ptr + add` except it defines `NULL + 0 == NULL`.
*/
MEM_STATIC
unsigned char* ZSTD_maybeNullPtrAdd(unsigned char* ptr, ptrdiff_t add)
{
return add > 0 ? ptr + add : ptr;
}
/* Issue #3240 reports an ASAN failure on an llvm-mingw build. Out of an
* abundance of caution, disable our custom poisoning on mingw. */
#ifdef __MINGW32__
#ifndef ZSTD_ASAN_DONT_POISON_WORKSPACE
#define ZSTD_ASAN_DONT_POISON_WORKSPACE 1
#endif
#ifndef ZSTD_MSAN_DONT_POISON_WORKSPACE
#define ZSTD_MSAN_DONT_POISON_WORKSPACE 1
#endif
#endif
#if ZSTD_MEMORY_SANITIZER && !defined(ZSTD_MSAN_DONT_POISON_WORKSPACE)
/* Not all platforms that support msan provide sanitizers/msan_interface.h.
* We therefore declare the functions we need ourselves, rather than trying to
* include the header file... */
#include <stddef.h> /* size_t */
#define ZSTD_DEPS_NEED_STDINT
#include "zstd_deps.h" /* intptr_t */
/* Make memory region fully initialized (without changing its contents). */
void __msan_unpoison(const volatile void *a, size_t size);
/* Make memory region fully uninitialized (without changing its contents).
This is a legacy interface that does not update origin information. Use
__msan_allocated_memory() instead. */
void __msan_poison(const volatile void *a, size_t size);
/* Returns the offset of the first (at least partially) poisoned byte in the
memory range, or -1 if the whole range is good. */
intptr_t __msan_test_shadow(const volatile void *x, size_t size);
/* Print shadow and origin for the memory range to stderr in a human-readable
format. */
void __msan_print_shadow(const volatile void *x, size_t size);
#endif
#if ZSTD_ADDRESS_SANITIZER && !defined(ZSTD_ASAN_DONT_POISON_WORKSPACE)
/* Not all platforms that support asan provide sanitizers/asan_interface.h.
* We therefore declare the functions we need ourselves, rather than trying to
* include the header file... */
#include <stddef.h> /* size_t */
/**
* Marks a memory region (<c>[addr, addr+size)</c>) as unaddressable.
*
* This memory must be previously allocated by your program. Instrumented
* code is forbidden from accessing addresses in this region until it is
* unpoisoned. This function is not guaranteed to poison the entire region -
* it could poison only a subregion of <c>[addr, addr+size)</c> due to ASan
* alignment restrictions.
*
* \note This function is not thread-safe because no two threads can poison or
* unpoison memory in the same memory region simultaneously.
*
* \param addr Start of memory region.
* \param size Size of memory region. */
void __asan_poison_memory_region(void const volatile *addr, size_t size);
/**
* Marks a memory region (<c>[addr, addr+size)</c>) as addressable.
*
* This memory must be previously allocated by your program. Accessing
* addresses in this region is allowed until this region is poisoned again.
* This function could unpoison a super-region of <c>[addr, addr+size)</c> due
* to ASan alignment restrictions.
*
* \note This function is not thread-safe because no two threads can
* poison or unpoison memory in the same memory region simultaneously.
*
* \param addr Start of memory region.
* \param size Size of memory region. */
void __asan_unpoison_memory_region(void const volatile *addr, size_t size);
#endif
#endif /* ZSTD_COMPILER_H */
#endif /* USE_EXTERNAL_ZSTD */

1305
vendor/github.com/DataDog/zstd/cover.c generated vendored
View File

File diff suppressed because it is too large Load Diff

155
vendor/github.com/DataDog/zstd/cover.h generated vendored
View File

@@ -1,155 +0,0 @@
#ifndef USE_EXTERNAL_ZSTD
/*
* Copyright (c) Meta Platforms, Inc. and affiliates.
* All rights reserved.
*
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
* You may select, at your option, one of the above-listed licenses.
*/
#ifndef ZDICT_STATIC_LINKING_ONLY
# define ZDICT_STATIC_LINKING_ONLY
#endif
#include "threading.h" /* ZSTD_pthread_mutex_t */
#include "mem.h" /* U32, BYTE */
#include "zdict.h"
/**
* COVER_best_t is used for two purposes:
* 1. Synchronizing threads.
* 2. Saving the best parameters and dictionary.
*
* All of the methods except COVER_best_init() are thread safe if zstd is
* compiled with multithreaded support.
*/
typedef struct COVER_best_s {
ZSTD_pthread_mutex_t mutex;
ZSTD_pthread_cond_t cond;
size_t liveJobs;
void *dict;
size_t dictSize;
ZDICT_cover_params_t parameters;
size_t compressedSize;
} COVER_best_t;
/**
* A segment is a range in the source as well as the score of the segment.
*/
typedef struct {
U32 begin;
U32 end;
U32 score;
} COVER_segment_t;
/**
*Number of epochs and size of each epoch.
*/
typedef struct {
U32 num;
U32 size;
} COVER_epoch_info_t;
/**
* Struct used for the dictionary selection function.
*/
typedef struct COVER_dictSelection {
BYTE* dictContent;
size_t dictSize;
size_t totalCompressedSize;
} COVER_dictSelection_t;
/**
* Computes the number of epochs and the size of each epoch.
* We will make sure that each epoch gets at least 10 * k bytes.
*
* The COVER algorithms divide the data up into epochs of equal size and
* select one segment from each epoch.
*
* @param maxDictSize The maximum allowed dictionary size.
* @param nbDmers The number of dmers we are training on.
* @param k The parameter k (segment size).
* @param passes The target number of passes over the dmer corpus.
* More passes means a better dictionary.
*/
COVER_epoch_info_t COVER_computeEpochs(U32 maxDictSize, U32 nbDmers,
U32 k, U32 passes);
/**
* Warns the user when their corpus is too small.
*/
void COVER_warnOnSmallCorpus(size_t maxDictSize, size_t nbDmers, int displayLevel);
/**
* Checks total compressed size of a dictionary
*/
size_t COVER_checkTotalCompressedSize(const ZDICT_cover_params_t parameters,
const size_t *samplesSizes, const BYTE *samples,
size_t *offsets,
size_t nbTrainSamples, size_t nbSamples,
BYTE *const dict, size_t dictBufferCapacity);
/**
* Returns the sum of the sample sizes.
*/
size_t COVER_sum(const size_t *samplesSizes, unsigned nbSamples) ;
/**
* Initialize the `COVER_best_t`.
*/
void COVER_best_init(COVER_best_t *best);
/**
* Wait until liveJobs == 0.
*/
void COVER_best_wait(COVER_best_t *best);
/**
* Call COVER_best_wait() and then destroy the COVER_best_t.
*/
void COVER_best_destroy(COVER_best_t *best);
/**
* Called when a thread is about to be launched.
* Increments liveJobs.
*/
void COVER_best_start(COVER_best_t *best);
/**
* Called when a thread finishes executing, both on error or success.
* Decrements liveJobs and signals any waiting threads if liveJobs == 0.
* If this dictionary is the best so far save it and its parameters.
*/
void COVER_best_finish(COVER_best_t *best, ZDICT_cover_params_t parameters,
COVER_dictSelection_t selection);
/**
* Error function for COVER_selectDict function. Checks if the return
* value is an error.
*/
unsigned COVER_dictSelectionIsError(COVER_dictSelection_t selection);
/**
* Error function for COVER_selectDict function. Returns a struct where
* return.totalCompressedSize is a ZSTD error.
*/
COVER_dictSelection_t COVER_dictSelectionError(size_t error);
/**
* Always call after selectDict is called to free up used memory from
* newly created dictionary.
*/
void COVER_dictSelectionFree(COVER_dictSelection_t selection);
/**
* Called to finalize the dictionary and select one based on whether or not
* the shrink-dict flag was enabled. If enabled the dictionary used is the
* smallest dictionary within a specified regression of the compressed size
* from the largest dictionary.
*/
COVER_dictSelection_t COVER_selectDict(BYTE* customDictContent, size_t dictBufferCapacity,
size_t dictContentSize, const BYTE* samplesBuffer, const size_t* samplesSizes, unsigned nbFinalizeSamples,
size_t nbCheckSamples, size_t nbSamples, ZDICT_cover_params_t params, size_t* offsets, size_t totalCompressedSize);
#endif /* USE_EXTERNAL_ZSTD */

252
vendor/github.com/DataDog/zstd/cpu.h generated vendored
View File

@@ -1,252 +0,0 @@
#ifndef USE_EXTERNAL_ZSTD
/*
* Copyright (c) Meta Platforms, Inc. and affiliates.
* All rights reserved.
*
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
* You may select, at your option, one of the above-listed licenses.
*/
#ifndef ZSTD_COMMON_CPU_H
#define ZSTD_COMMON_CPU_H
/**
* Implementation taken from folly/CpuId.h
* https://github.com/facebook/folly/blob/master/folly/CpuId.h
*/
#include "mem.h"
#ifdef _MSC_VER
#include <intrin.h>
#endif
typedef struct {
U32 f1c;
U32 f1d;
U32 f7b;
U32 f7c;
} ZSTD_cpuid_t;
MEM_STATIC ZSTD_cpuid_t ZSTD_cpuid(void) {
U32 f1c = 0;
U32 f1d = 0;
U32 f7b = 0;
U32 f7c = 0;
#if defined(_MSC_VER) && (defined(_M_X64) || defined(_M_IX86))
#if !defined(_M_X64) || !defined(__clang__) || __clang_major__ >= 16
int reg[4];
__cpuid((int*)reg, 0);
{
int const n = reg[0];
if (n >= 1) {
__cpuid((int*)reg, 1);
f1c = (U32)reg[2];
f1d = (U32)reg[3];
}
if (n >= 7) {
__cpuidex((int*)reg, 7, 0);
f7b = (U32)reg[1];
f7c = (U32)reg[2];
}
}
#else
/* Clang compiler has a bug (fixed in https://reviews.llvm.org/D101338) in
* which the `__cpuid` intrinsic does not save and restore `rbx` as it needs
* to due to being a reserved register. So in that case, do the `cpuid`
* ourselves. Clang supports inline assembly anyway.
*/
U32 n;
__asm__(
"pushq %%rbx\n\t"
"cpuid\n\t"
"popq %%rbx\n\t"
: "=a"(n)
: "a"(0)
: "rcx", "rdx");
if (n >= 1) {
U32 f1a;
__asm__(
"pushq %%rbx\n\t"
"cpuid\n\t"
"popq %%rbx\n\t"
: "=a"(f1a), "=c"(f1c), "=d"(f1d)
: "a"(1)
:);
}
if (n >= 7) {
__asm__(
"pushq %%rbx\n\t"
"cpuid\n\t"
"movq %%rbx, %%rax\n\t"
"popq %%rbx"
: "=a"(f7b), "=c"(f7c)
: "a"(7), "c"(0)
: "rdx");
}
#endif
#elif defined(__i386__) && defined(__PIC__) && !defined(__clang__) && defined(__GNUC__)
/* The following block like the normal cpuid branch below, but gcc
* reserves ebx for use of its pic register so we must specially
* handle the save and restore to avoid clobbering the register
*/
U32 n;
__asm__(
"pushl %%ebx\n\t"
"cpuid\n\t"
"popl %%ebx\n\t"
: "=a"(n)
: "a"(0)
: "ecx", "edx");
if (n >= 1) {
U32 f1a;
__asm__(
"pushl %%ebx\n\t"
"cpuid\n\t"
"popl %%ebx\n\t"
: "=a"(f1a), "=c"(f1c), "=d"(f1d)
: "a"(1));
}
if (n >= 7) {
__asm__(
"pushl %%ebx\n\t"
"cpuid\n\t"
"movl %%ebx, %%eax\n\t"
"popl %%ebx"
: "=a"(f7b), "=c"(f7c)
: "a"(7), "c"(0)
: "edx");
}
#elif defined(__x86_64__) || defined(_M_X64) || defined(__i386__)
U32 n;
__asm__("cpuid" : "=a"(n) : "a"(0) : "ebx", "ecx", "edx");
if (n >= 1) {
U32 f1a;
__asm__("cpuid" : "=a"(f1a), "=c"(f1c), "=d"(f1d) : "a"(1) : "ebx");
}
if (n >= 7) {
U32 f7a;
__asm__("cpuid"
: "=a"(f7a), "=b"(f7b), "=c"(f7c)
: "a"(7), "c"(0)
: "edx");
}
#endif
{
ZSTD_cpuid_t cpuid;
cpuid.f1c = f1c;
cpuid.f1d = f1d;
cpuid.f7b = f7b;
cpuid.f7c = f7c;
return cpuid;
}
}
#define X(name, r, bit) \
MEM_STATIC int ZSTD_cpuid_##name(ZSTD_cpuid_t const cpuid) { \
return ((cpuid.r) & (1U << bit)) != 0; \
}
/* cpuid(1): Processor Info and Feature Bits. */
#define C(name, bit) X(name, f1c, bit)
C(sse3, 0)
C(pclmuldq, 1)
C(dtes64, 2)
C(monitor, 3)
C(dscpl, 4)
C(vmx, 5)
C(smx, 6)
C(eist, 7)
C(tm2, 8)
C(ssse3, 9)
C(cnxtid, 10)
C(fma, 12)
C(cx16, 13)
C(xtpr, 14)
C(pdcm, 15)
C(pcid, 17)
C(dca, 18)
C(sse41, 19)
C(sse42, 20)
C(x2apic, 21)
C(movbe, 22)
C(popcnt, 23)
C(tscdeadline, 24)
C(aes, 25)
C(xsave, 26)
C(osxsave, 27)
C(avx, 28)
C(f16c, 29)
C(rdrand, 30)
#undef C
#define D(name, bit) X(name, f1d, bit)
D(fpu, 0)
D(vme, 1)
D(de, 2)
D(pse, 3)
D(tsc, 4)
D(msr, 5)
D(pae, 6)
D(mce, 7)
D(cx8, 8)
D(apic, 9)
D(sep, 11)
D(mtrr, 12)
D(pge, 13)
D(mca, 14)
D(cmov, 15)
D(pat, 16)
D(pse36, 17)
D(psn, 18)
D(clfsh, 19)
D(ds, 21)
D(acpi, 22)
D(mmx, 23)
D(fxsr, 24)
D(sse, 25)
D(sse2, 26)
D(ss, 27)
D(htt, 28)
D(tm, 29)
D(pbe, 31)
#undef D
/* cpuid(7): Extended Features. */
#define B(name, bit) X(name, f7b, bit)
B(bmi1, 3)
B(hle, 4)
B(avx2, 5)
B(smep, 7)
B(bmi2, 8)
B(erms, 9)
B(invpcid, 10)
B(rtm, 11)
B(mpx, 14)
B(avx512f, 16)
B(avx512dq, 17)
B(rdseed, 18)
B(adx, 19)
B(smap, 20)
B(avx512ifma, 21)
B(pcommit, 22)
B(clflushopt, 23)
B(clwb, 24)
B(avx512pf, 26)
B(avx512er, 27)
B(avx512cd, 28)
B(sha, 29)
B(avx512bw, 30)
B(avx512vl, 31)
#undef B
#define C(name, bit) X(name, f7c, bit)
C(prefetchwt1, 0)
C(avx512vbmi, 1)
#undef C
#undef X
#endif /* ZSTD_COMMON_CPU_H */
#endif /* USE_EXTERNAL_ZSTD */

33
vendor/github.com/DataDog/zstd/debug.c generated vendored
View File

@@ -1,33 +0,0 @@
#ifndef USE_EXTERNAL_ZSTD
/* ******************************************************************
* debug
* Part of FSE library
* Copyright (c) Meta Platforms, Inc. and affiliates.
*
* You can contact the author at :
* - Source repository : https://github.com/Cyan4973/FiniteStateEntropy
*
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
* You may select, at your option, one of the above-listed licenses.
****************************************************************** */
/*
* This module only hosts one global variable
* which can be used to dynamically influence the verbosity of traces,
* such as DEBUGLOG and RAWLOG
*/
#include "debug.h"
#if !defined(ZSTD_LINUX_KERNEL) || (DEBUGLEVEL>=2)
/* We only use this when DEBUGLEVEL>=2, but we get -Werror=pedantic errors if a
* translation unit is empty. So remove this from Linux kernel builds, but
* otherwise just leave it in.
*/
int g_debuglevel = DEBUGLEVEL;
#endif
#endif /* USE_EXTERNAL_ZSTD */

110
vendor/github.com/DataDog/zstd/debug.h generated vendored
View File

@@ -1,110 +0,0 @@
#ifndef USE_EXTERNAL_ZSTD
/* ******************************************************************
* debug
* Part of FSE library
* Copyright (c) Meta Platforms, Inc. and affiliates.
*
* You can contact the author at :
* - Source repository : https://github.com/Cyan4973/FiniteStateEntropy
*
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
* You may select, at your option, one of the above-listed licenses.
****************************************************************** */
/*
* The purpose of this header is to enable debug functions.
* They regroup assert(), DEBUGLOG() and RAWLOG() for run-time,
* and DEBUG_STATIC_ASSERT() for compile-time.
*
* By default, DEBUGLEVEL==0, which means run-time debug is disabled.
*
* Level 1 enables assert() only.
* Starting level 2, traces can be generated and pushed to stderr.
* The higher the level, the more verbose the traces.
*
* It's possible to dynamically adjust level using variable g_debug_level,
* which is only declared if DEBUGLEVEL>=2,
* and is a global variable, not multi-thread protected (use with care)
*/
#ifndef DEBUG_H_12987983217
#define DEBUG_H_12987983217
/* static assert is triggered at compile time, leaving no runtime artefact.
* static assert only works with compile-time constants.
* Also, this variant can only be used inside a function. */
#define DEBUG_STATIC_ASSERT(c) (void)sizeof(char[(c) ? 1 : -1])
/* DEBUGLEVEL is expected to be defined externally,
* typically through compiler command line.
* Value must be a number. */
#ifndef DEBUGLEVEL
# define DEBUGLEVEL 0
#endif
/* recommended values for DEBUGLEVEL :
* 0 : release mode, no debug, all run-time checks disabled
* 1 : enables assert() only, no display
* 2 : reserved, for currently active debug path
* 3 : events once per object lifetime (CCtx, CDict, etc.)
* 4 : events once per frame
* 5 : events once per block
* 6 : events once per sequence (verbose)
* 7+: events at every position (*very* verbose)
*
* It's generally inconvenient to output traces > 5.
* In which case, it's possible to selectively trigger high verbosity levels
* by modifying g_debug_level.
*/
#if (DEBUGLEVEL>=1)
# define ZSTD_DEPS_NEED_ASSERT
# include "zstd_deps.h"
#else
# ifndef assert /* assert may be already defined, due to prior #include <assert.h> */
# define assert(condition) ((void)0) /* disable assert (default) */
# endif
#endif
#if (DEBUGLEVEL>=2)
# define ZSTD_DEPS_NEED_IO
# include "zstd_deps.h"
extern int g_debuglevel; /* the variable is only declared,
it actually lives in debug.c,
and is shared by the whole process.
It's not thread-safe.
It's useful when enabling very verbose levels
on selective conditions (such as position in src) */
# define RAWLOG(l, ...) \
do { \
if (l<=g_debuglevel) { \
ZSTD_DEBUG_PRINT(__VA_ARGS__); \
} \
} while (0)
#define STRINGIFY(x) #x
#define TOSTRING(x) STRINGIFY(x)
#define LINE_AS_STRING TOSTRING(__LINE__)
# define DEBUGLOG(l, ...) \
do { \
if (l<=g_debuglevel) { \
ZSTD_DEBUG_PRINT(__FILE__ ":" LINE_AS_STRING ": " __VA_ARGS__); \
ZSTD_DEBUG_PRINT(" \n"); \
} \
} while (0)
#else
# define RAWLOG(l, ...) do { } while (0) /* disabled */
# define DEBUGLOG(l, ...) do { } while (0) /* disabled */
#endif
#endif /* DEBUG_H_12987983217 */
#endif /* USE_EXTERNAL_ZSTD */

1916
vendor/github.com/DataDog/zstd/divsufsort.c generated vendored
View File

File diff suppressed because it is too large Load Diff

60
vendor/github.com/DataDog/zstd/divsufsort.h generated vendored
View File

@@ -1,60 +0,0 @@
#ifndef USE_EXTERNAL_ZSTD
/*
* divsufsort.h for libdivsufsort-lite
* Copyright (c) 2003-2008 Yuta Mori All Rights Reserved.
*
* Permission is hereby granted, free of charge, to any person
* obtaining a copy of this software and associated documentation
* files (the "Software"), to deal in the Software without
* restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell
* copies of the Software, and to permit persons to whom the
* Software is furnished to do so, subject to the following
* conditions:
*
* The above copyright notice and this permission notice shall be
* included in all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
* EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
* OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
* NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
* HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
* WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
* FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
* OTHER DEALINGS IN THE SOFTWARE.
*/
#ifndef _DIVSUFSORT_H
#define _DIVSUFSORT_H 1
/*- Prototypes -*/
/**
* Constructs the suffix array of a given string.
* @param T [0..n-1] The input string.
* @param SA [0..n-1] The output array of suffixes.
* @param n The length of the given string.
* @param openMP enables OpenMP optimization.
* @return 0 if no error occurred, -1 or -2 otherwise.
*/
int
divsufsort(const unsigned char *T, int *SA, int n, int openMP);
/**
* Constructs the burrows-wheeler transformed string of a given string.
* @param T [0..n-1] The input string.
* @param U [0..n-1] The output string. (can be T)
* @param A [0..n-1] The temporary array. (can be NULL)
* @param n The length of the given string.
* @param num_indexes The length of secondary indexes array. (can be NULL)
* @param indexes The secondary indexes array. (can be NULL)
* @param openMP enables OpenMP optimization.
* @return The primary index if no error occurred, -1 or -2 otherwise.
*/
int
divbwt(const unsigned char *T, unsigned char *U, int *A, int n, unsigned char * num_indexes, int * indexes, int openMP);
#endif /* _DIVSUFSORT_H */
#endif /* USE_EXTERNAL_ZSTD */

343
vendor/github.com/DataDog/zstd/entropy_common.c generated vendored
View File

@@ -1,343 +0,0 @@
#ifndef USE_EXTERNAL_ZSTD
/* ******************************************************************
* Common functions of New Generation Entropy library
* Copyright (c) Meta Platforms, Inc. and affiliates.
*
* You can contact the author at :
* - FSE+HUF source repository : https://github.com/Cyan4973/FiniteStateEntropy
* - Public forum : https://groups.google.com/forum/#!forum/lz4c
*
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
* You may select, at your option, one of the above-listed licenses.
****************************************************************** */
/* *************************************
* Dependencies
***************************************/
#include "mem.h"
#include "error_private.h" /* ERR_*, ERROR */
#define FSE_STATIC_LINKING_ONLY /* FSE_MIN_TABLELOG */
#include "fse.h"
#include "huf.h"
#include "bits.h" /* ZSDT_highbit32, ZSTD_countTrailingZeros32 */
/*=== Version ===*/
unsigned FSE_versionNumber(void) { return FSE_VERSION_NUMBER; }
/*=== Error Management ===*/
unsigned FSE_isError(size_t code) { return ERR_isError(code); }
const char* FSE_getErrorName(size_t code) { return ERR_getErrorName(code); }
unsigned HUF_isError(size_t code) { return ERR_isError(code); }
const char* HUF_getErrorName(size_t code) { return ERR_getErrorName(code); }
/*-**************************************************************
* FSE NCount encoding-decoding
****************************************************************/
FORCE_INLINE_TEMPLATE
size_t FSE_readNCount_body(short* normalizedCounter, unsigned* maxSVPtr, unsigned* tableLogPtr,
const void* headerBuffer, size_t hbSize)
{
const BYTE* const istart = (const BYTE*) headerBuffer;
const BYTE* const iend = istart + hbSize;
const BYTE* ip = istart;
int nbBits;
int remaining;
int threshold;
U32 bitStream;
int bitCount;
unsigned charnum = 0;
unsigned const maxSV1 = *maxSVPtr + 1;
int previous0 = 0;
if (hbSize < 8) {
/* This function only works when hbSize >= 8 */
char buffer[8] = {0};
ZSTD_memcpy(buffer, headerBuffer, hbSize);
{ size_t const countSize = FSE_readNCount(normalizedCounter, maxSVPtr, tableLogPtr,
buffer, sizeof(buffer));
if (FSE_isError(countSize)) return countSize;
if (countSize > hbSize) return ERROR(corruption_detected);
return countSize;
} }
assert(hbSize >= 8);
/* init */
ZSTD_memset(normalizedCounter, 0, (*maxSVPtr+1) * sizeof(normalizedCounter[0])); /* all symbols not present in NCount have a frequency of 0 */
bitStream = MEM_readLE32(ip);
nbBits = (bitStream & 0xF) + FSE_MIN_TABLELOG; /* extract tableLog */
if (nbBits > FSE_TABLELOG_ABSOLUTE_MAX) return ERROR(tableLog_tooLarge);
bitStream >>= 4;
bitCount = 4;
*tableLogPtr = nbBits;
remaining = (1<<nbBits)+1;
threshold = 1<<nbBits;
nbBits++;
for (;;) {
if (previous0) {
/* Count the number of repeats. Each time the
* 2-bit repeat code is 0b11 there is another
* repeat.
* Avoid UB by setting the high bit to 1.
*/
int repeats = ZSTD_countTrailingZeros32(~bitStream | 0x80000000) >> 1;
while (repeats >= 12) {
charnum += 3 * 12;
if (LIKELY(ip <= iend-7)) {
ip += 3;
} else {
bitCount -= (int)(8 * (iend - 7 - ip));
bitCount &= 31;
ip = iend - 4;
}
bitStream = MEM_readLE32(ip) >> bitCount;
repeats = ZSTD_countTrailingZeros32(~bitStream | 0x80000000) >> 1;
}
charnum += 3 * repeats;
bitStream >>= 2 * repeats;
bitCount += 2 * repeats;
/* Add the final repeat which isn't 0b11. */
assert((bitStream & 3) < 3);
charnum += bitStream & 3;
bitCount += 2;
/* This is an error, but break and return an error
* at the end, because returning out of a loop makes
* it harder for the compiler to optimize.
*/
if (charnum >= maxSV1) break;
/* We don't need to set the normalized count to 0
* because we already memset the whole buffer to 0.
*/
if (LIKELY(ip <= iend-7) || (ip + (bitCount>>3) <= iend-4)) {
assert((bitCount >> 3) <= 3); /* For first condition to work */
ip += bitCount>>3;
bitCount &= 7;
} else {
bitCount -= (int)(8 * (iend - 4 - ip));
bitCount &= 31;
ip = iend - 4;
}
bitStream = MEM_readLE32(ip) >> bitCount;
}
{
int const max = (2*threshold-1) - remaining;
int count;
if ((bitStream & (threshold-1)) < (U32)max) {
count = bitStream & (threshold-1);
bitCount += nbBits-1;
} else {
count = bitStream & (2*threshold-1);
if (count >= threshold) count -= max;
bitCount += nbBits;
}
count--; /* extra accuracy */
/* When it matters (small blocks), this is a
* predictable branch, because we don't use -1.
*/
if (count >= 0) {
remaining -= count;
} else {
assert(count == -1);
remaining += count;
}
normalizedCounter[charnum++] = (short)count;
previous0 = !count;
assert(threshold > 1);
if (remaining < threshold) {
/* This branch can be folded into the
* threshold update condition because we
* know that threshold > 1.
*/
if (remaining <= 1) break;
nbBits = ZSTD_highbit32(remaining) + 1;
threshold = 1 << (nbBits - 1);
}
if (charnum >= maxSV1) break;
if (LIKELY(ip <= iend-7) || (ip + (bitCount>>3) <= iend-4)) {
ip += bitCount>>3;
bitCount &= 7;
} else {
bitCount -= (int)(8 * (iend - 4 - ip));
bitCount &= 31;
ip = iend - 4;
}
bitStream = MEM_readLE32(ip) >> bitCount;
} }
if (remaining != 1) return ERROR(corruption_detected);
/* Only possible when there are too many zeros. */
if (charnum > maxSV1) return ERROR(maxSymbolValue_tooSmall);
if (bitCount > 32) return ERROR(corruption_detected);
*maxSVPtr = charnum-1;
ip += (bitCount+7)>>3;
return ip-istart;
}
/* Avoids the FORCE_INLINE of the _body() function. */
static size_t FSE_readNCount_body_default(
short* normalizedCounter, unsigned* maxSVPtr, unsigned* tableLogPtr,
const void* headerBuffer, size_t hbSize)
{
return FSE_readNCount_body(normalizedCounter, maxSVPtr, tableLogPtr, headerBuffer, hbSize);
}
#if DYNAMIC_BMI2
BMI2_TARGET_ATTRIBUTE static size_t FSE_readNCount_body_bmi2(
short* normalizedCounter, unsigned* maxSVPtr, unsigned* tableLogPtr,
const void* headerBuffer, size_t hbSize)
{
return FSE_readNCount_body(normalizedCounter, maxSVPtr, tableLogPtr, headerBuffer, hbSize);
}
#endif
size_t FSE_readNCount_bmi2(
short* normalizedCounter, unsigned* maxSVPtr, unsigned* tableLogPtr,
const void* headerBuffer, size_t hbSize, int bmi2)
{
#if DYNAMIC_BMI2
if (bmi2) {
return FSE_readNCount_body_bmi2(normalizedCounter, maxSVPtr, tableLogPtr, headerBuffer, hbSize);
}
#endif
(void)bmi2;
return FSE_readNCount_body_default(normalizedCounter, maxSVPtr, tableLogPtr, headerBuffer, hbSize);
}
size_t FSE_readNCount(
short* normalizedCounter, unsigned* maxSVPtr, unsigned* tableLogPtr,
const void* headerBuffer, size_t hbSize)
{
return FSE_readNCount_bmi2(normalizedCounter, maxSVPtr, tableLogPtr, headerBuffer, hbSize, /* bmi2 */ 0);
}
/*! HUF_readStats() :
Read compact Huffman tree, saved by HUF_writeCTable().
`huffWeight` is destination buffer.
`rankStats` is assumed to be a table of at least HUF_TABLELOG_MAX U32.
@return : size read from `src` , or an error Code .
Note : Needed by HUF_readCTable() and HUF_readDTableX?() .
*/
size_t HUF_readStats(BYTE* huffWeight, size_t hwSize, U32* rankStats,
U32* nbSymbolsPtr, U32* tableLogPtr,
const void* src, size_t srcSize)
{
U32 wksp[HUF_READ_STATS_WORKSPACE_SIZE_U32];
return HUF_readStats_wksp(huffWeight, hwSize, rankStats, nbSymbolsPtr, tableLogPtr, src, srcSize, wksp, sizeof(wksp), /* flags */ 0);
}
FORCE_INLINE_TEMPLATE size_t
HUF_readStats_body(BYTE* huffWeight, size_t hwSize, U32* rankStats,
U32* nbSymbolsPtr, U32* tableLogPtr,
const void* src, size_t srcSize,
void* workSpace, size_t wkspSize,
int bmi2)
{
U32 weightTotal;
const BYTE* ip = (const BYTE*) src;
size_t iSize;
size_t oSize;
if (!srcSize) return ERROR(srcSize_wrong);
iSize = ip[0];
/* ZSTD_memset(huffWeight, 0, hwSize); *//* is not necessary, even though some analyzer complain ... */
if (iSize >= 128) { /* special header */
oSize = iSize - 127;
iSize = ((oSize+1)/2);
if (iSize+1 > srcSize) return ERROR(srcSize_wrong);
if (oSize >= hwSize) return ERROR(corruption_detected);
ip += 1;
{ U32 n;
for (n=0; n<oSize; n+=2) {
huffWeight[n] = ip[n/2] >> 4;
huffWeight[n+1] = ip[n/2] & 15;
} } }
else { /* header compressed with FSE (normal case) */
if (iSize+1 > srcSize) return ERROR(srcSize_wrong);
/* max (hwSize-1) values decoded, as last one is implied */
oSize = FSE_decompress_wksp_bmi2(huffWeight, hwSize-1, ip+1, iSize, 6, workSpace, wkspSize, bmi2);
if (FSE_isError(oSize)) return oSize;
}
/* collect weight stats */
ZSTD_memset(rankStats, 0, (HUF_TABLELOG_MAX + 1) * sizeof(U32));
weightTotal = 0;
{ U32 n; for (n=0; n<oSize; n++) {
if (huffWeight[n] > HUF_TABLELOG_MAX) return ERROR(corruption_detected);
rankStats[huffWeight[n]]++;
weightTotal += (1 << huffWeight[n]) >> 1;
} }
if (weightTotal == 0) return ERROR(corruption_detected);
/* get last non-null symbol weight (implied, total must be 2^n) */
{ U32 const tableLog = ZSTD_highbit32(weightTotal) + 1;
if (tableLog > HUF_TABLELOG_MAX) return ERROR(corruption_detected);
*tableLogPtr = tableLog;
/* determine last weight */
{ U32 const total = 1 << tableLog;
U32 const rest = total - weightTotal;
U32 const verif = 1 << ZSTD_highbit32(rest);
U32 const lastWeight = ZSTD_highbit32(rest) + 1;
if (verif != rest) return ERROR(corruption_detected); /* last value must be a clean power of 2 */
huffWeight[oSize] = (BYTE)lastWeight;
rankStats[lastWeight]++;
} }
/* check tree construction validity */
if ((rankStats[1] < 2) || (rankStats[1] & 1)) return ERROR(corruption_detected); /* by construction : at least 2 elts of rank 1, must be even */
/* results */
*nbSymbolsPtr = (U32)(oSize+1);
return iSize+1;
}
/* Avoids the FORCE_INLINE of the _body() function. */
static size_t HUF_readStats_body_default(BYTE* huffWeight, size_t hwSize, U32* rankStats,
U32* nbSymbolsPtr, U32* tableLogPtr,
const void* src, size_t srcSize,
void* workSpace, size_t wkspSize)
{
return HUF_readStats_body(huffWeight, hwSize, rankStats, nbSymbolsPtr, tableLogPtr, src, srcSize, workSpace, wkspSize, 0);
}
#if DYNAMIC_BMI2
static BMI2_TARGET_ATTRIBUTE size_t HUF_readStats_body_bmi2(BYTE* huffWeight, size_t hwSize, U32* rankStats,
U32* nbSymbolsPtr, U32* tableLogPtr,
const void* src, size_t srcSize,
void* workSpace, size_t wkspSize)
{
return HUF_readStats_body(huffWeight, hwSize, rankStats, nbSymbolsPtr, tableLogPtr, src, srcSize, workSpace, wkspSize, 1);
}
#endif
size_t HUF_readStats_wksp(BYTE* huffWeight, size_t hwSize, U32* rankStats,
U32* nbSymbolsPtr, U32* tableLogPtr,
const void* src, size_t srcSize,
void* workSpace, size_t wkspSize,
int flags)
{
#if DYNAMIC_BMI2
if (flags & HUF_flags_bmi2) {
return HUF_readStats_body_bmi2(huffWeight, hwSize, rankStats, nbSymbolsPtr, tableLogPtr, src, srcSize, workSpace, wkspSize);
}
#endif
(void)flags;
return HUF_readStats_body_default(huffWeight, hwSize, rankStats, nbSymbolsPtr, tableLogPtr, src, srcSize, workSpace, wkspSize);
}
#endif /* USE_EXTERNAL_ZSTD */

67
vendor/github.com/DataDog/zstd/error_private.c generated vendored
View File

@@ -1,67 +0,0 @@
#ifndef USE_EXTERNAL_ZSTD
/*
* Copyright (c) Meta Platforms, Inc. and affiliates.
* All rights reserved.
*
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
* You may select, at your option, one of the above-listed licenses.
*/
/* The purpose of this file is to have a single list of error strings embedded in binary */
#include "error_private.h"
const char* ERR_getErrorString(ERR_enum code)
{
#ifdef ZSTD_STRIP_ERROR_STRINGS
(void)code;
return "Error strings stripped";
#else
static const char* const notErrorCode = "Unspecified error code";
switch( code )
{
case PREFIX(no_error): return "No error detected";
case PREFIX(GENERIC): return "Error (generic)";
case PREFIX(prefix_unknown): return "Unknown frame descriptor";
case PREFIX(version_unsupported): return "Version not supported";
case PREFIX(frameParameter_unsupported): return "Unsupported frame parameter";
case PREFIX(frameParameter_windowTooLarge): return "Frame requires too much memory for decoding";
case PREFIX(corruption_detected): return "Data corruption detected";
case PREFIX(checksum_wrong): return "Restored data doesn't match checksum";
case PREFIX(literals_headerWrong): return "Header of Literals' block doesn't respect format specification";
case PREFIX(parameter_unsupported): return "Unsupported parameter";
case PREFIX(parameter_combination_unsupported): return "Unsupported combination of parameters";
case PREFIX(parameter_outOfBound): return "Parameter is out of bound";
case PREFIX(init_missing): return "Context should be init first";
case PREFIX(memory_allocation): return "Allocation error : not enough memory";
case PREFIX(workSpace_tooSmall): return "workSpace buffer is not large enough";
case PREFIX(stage_wrong): return "Operation not authorized at current processing stage";
case PREFIX(tableLog_tooLarge): return "tableLog requires too much memory : unsupported";
case PREFIX(maxSymbolValue_tooLarge): return "Unsupported max Symbol Value : too large";
case PREFIX(maxSymbolValue_tooSmall): return "Specified maxSymbolValue is too small";
case PREFIX(cannotProduce_uncompressedBlock): return "This mode cannot generate an uncompressed block";
case PREFIX(stabilityCondition_notRespected): return "pledged buffer stability condition is not respected";
case PREFIX(dictionary_corrupted): return "Dictionary is corrupted";
case PREFIX(dictionary_wrong): return "Dictionary mismatch";
case PREFIX(dictionaryCreation_failed): return "Cannot create Dictionary from provided samples";
case PREFIX(dstSize_tooSmall): return "Destination buffer is too small";
case PREFIX(srcSize_wrong): return "Src size is incorrect";
case PREFIX(dstBuffer_null): return "Operation on NULL destination buffer";
case PREFIX(noForwardProgress_destFull): return "Operation made no progress over multiple calls, due to output buffer being full";
case PREFIX(noForwardProgress_inputEmpty): return "Operation made no progress over multiple calls, due to input being empty";
/* following error codes are not stable and may be removed or changed in a future version */
case PREFIX(frameIndex_tooLarge): return "Frame index is too large";
case PREFIX(seekableIO): return "An I/O error occurred when reading/seeking";
case PREFIX(dstBuffer_wrong): return "Destination buffer is wrong";
case PREFIX(srcBuffer_wrong): return "Source buffer is wrong";
case PREFIX(sequenceProducer_failed): return "Block-level external sequence producer returned an error code";
case PREFIX(externalSequences_invalid): return "External sequences are not valid";
case PREFIX(maxCode):
default: return notErrorCode;
}
#endif
}
#endif /* USE_EXTERNAL_ZSTD */

161
vendor/github.com/DataDog/zstd/error_private.h generated vendored
View File

@@ -1,161 +0,0 @@
#ifndef USE_EXTERNAL_ZSTD
/*
* Copyright (c) Meta Platforms, Inc. and affiliates.
* All rights reserved.
*
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
* You may select, at your option, one of the above-listed licenses.
*/
/* Note : this module is expected to remain private, do not expose it */
#ifndef ERROR_H_MODULE
#define ERROR_H_MODULE
/* ****************************************
* Dependencies
******************************************/
#include "zstd_errors.h" /* enum list */
#include "compiler.h"
#include "debug.h"
#include "zstd_deps.h" /* size_t */
/* ****************************************
* Compiler-specific
******************************************/
#if defined(__GNUC__)
# define ERR_STATIC static __attribute__((unused))
#elif defined (__cplusplus) || (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */)
# define ERR_STATIC static inline
#elif defined(_MSC_VER)
# define ERR_STATIC static __inline
#else
# define ERR_STATIC static /* this version may generate warnings for unused static functions; disable the relevant warning */
#endif
/*-****************************************
* Customization (error_public.h)
******************************************/
typedef ZSTD_ErrorCode ERR_enum;
#define PREFIX(name) ZSTD_error_##name
/*-****************************************
* Error codes handling
******************************************/
#undef ERROR /* already defined on Visual Studio */
#define ERROR(name) ZSTD_ERROR(name)
#define ZSTD_ERROR(name) ((size_t)-PREFIX(name))
ERR_STATIC unsigned ERR_isError(size_t code) { return (code > ERROR(maxCode)); }
ERR_STATIC ERR_enum ERR_getErrorCode(size_t code) { if (!ERR_isError(code)) return (ERR_enum)0; return (ERR_enum) (0-code); }
/* check and forward error code */
#define CHECK_V_F(e, f) \
size_t const e = f; \
do { \
if (ERR_isError(e)) \
return e; \
} while (0)
#define CHECK_F(f) do { CHECK_V_F(_var_err__, f); } while (0)
/*-****************************************
* Error Strings
******************************************/
const char* ERR_getErrorString(ERR_enum code); /* error_private.c */
ERR_STATIC const char* ERR_getErrorName(size_t code)
{
return ERR_getErrorString(ERR_getErrorCode(code));
}
/**
* Ignore: this is an internal helper.
*
* This is a helper function to help force C99-correctness during compilation.
* Under strict compilation modes, variadic macro arguments can't be empty.
* However, variadic function arguments can be. Using a function therefore lets
* us statically check that at least one (string) argument was passed,
* independent of the compilation flags.
*/
static INLINE_KEYWORD UNUSED_ATTR
void _force_has_format_string(const char *format, ...) {
(void)format;
}
/**
* Ignore: this is an internal helper.
*
* We want to force this function invocation to be syntactically correct, but
* we don't want to force runtime evaluation of its arguments.
*/
#define _FORCE_HAS_FORMAT_STRING(...) \
do { \
if (0) { \
_force_has_format_string(__VA_ARGS__); \
} \
} while (0)
#define ERR_QUOTE(str) #str
/**
* Return the specified error if the condition evaluates to true.
*
* In debug modes, prints additional information.
* In order to do that (particularly, printing the conditional that failed),
* this can't just wrap RETURN_ERROR().
*/
#define RETURN_ERROR_IF(cond, err, ...) \
do { \
if (cond) { \
RAWLOG(3, "%s:%d: ERROR!: check %s failed, returning %s", \
__FILE__, __LINE__, ERR_QUOTE(cond), ERR_QUOTE(ERROR(err))); \
_FORCE_HAS_FORMAT_STRING(__VA_ARGS__); \
RAWLOG(3, ": " __VA_ARGS__); \
RAWLOG(3, "\n"); \
return ERROR(err); \
} \
} while (0)
/**
* Unconditionally return the specified error.
*
* In debug modes, prints additional information.
*/
#define RETURN_ERROR(err, ...) \
do { \
RAWLOG(3, "%s:%d: ERROR!: unconditional check failed, returning %s", \
__FILE__, __LINE__, ERR_QUOTE(ERROR(err))); \
_FORCE_HAS_FORMAT_STRING(__VA_ARGS__); \
RAWLOG(3, ": " __VA_ARGS__); \
RAWLOG(3, "\n"); \
return ERROR(err); \
} while(0)
/**
* If the provided expression evaluates to an error code, returns that error code.
*
* In debug modes, prints additional information.
*/
#define FORWARD_IF_ERROR(err, ...) \
do { \
size_t const err_code = (err); \
if (ERR_isError(err_code)) { \
RAWLOG(3, "%s:%d: ERROR!: forwarding error in %s: %s", \
__FILE__, __LINE__, ERR_QUOTE(err), ERR_getErrorName(err_code)); \
_FORCE_HAS_FORMAT_STRING(__VA_ARGS__); \
RAWLOG(3, ": " __VA_ARGS__); \
RAWLOG(3, "\n"); \
return err_code; \
} \
} while(0)
#endif /* ERROR_H_MODULE */
#endif /* USE_EXTERNAL_ZSTD */

34
vendor/github.com/DataDog/zstd/errors.go generated vendored
View File

@@ -1,34 +0,0 @@
package zstd
/*
#include "zstd.h"
*/
import "C"
// ErrorCode is an error returned by the zstd library.
type ErrorCode int
// Error returns the error string given by zstd
func (e ErrorCode) Error() string {
return C.GoString(C.ZSTD_getErrorName(C.size_t(e)))
}
func cIsError(code int) bool {
return int(C.ZSTD_isError(C.size_t(code))) != 0
}
// getError returns an error for the return code, or nil if it's not an error
func getError(code int) error {
if code < 0 && cIsError(code) {
return ErrorCode(code)
}
return nil
}
// IsDstSizeTooSmallError returns whether the error correspond to zstd standard sDstSizeTooSmall error
func IsDstSizeTooSmallError(e error) bool {
if e != nil && e.Error() == "Destination buffer is too small" {
return true
}
return false
}

14
vendor/github.com/DataDog/zstd/external_zstd.go generated vendored
View File

@@ -1,14 +0,0 @@
//go:build external_libzstd
// +build external_libzstd
package zstd
// #cgo CFLAGS: -DUSE_EXTERNAL_ZSTD
// #cgo pkg-config: libzstd
/*
#include<zstd.h>
#if ZSTD_VERSION_NUMBER < 10400
#error "ZSTD version >= 1.4 is required"
#endif
*/
import "C"

769
vendor/github.com/DataDog/zstd/fastcover.c generated vendored
View File

@@ -1,769 +0,0 @@
#ifndef USE_EXTERNAL_ZSTD
/*
* Copyright (c) Meta Platforms, Inc. and affiliates.
* All rights reserved.
*
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
* You may select, at your option, one of the above-listed licenses.
*/
/*-*************************************
* Dependencies
***************************************/
#include <stdio.h> /* fprintf */
#include <stdlib.h> /* malloc, free, qsort */
#include <string.h> /* memset */
#include <time.h> /* clock */
#ifndef ZDICT_STATIC_LINKING_ONLY
# define ZDICT_STATIC_LINKING_ONLY
#endif
#include "mem.h" /* read */
#include "pool.h"
#include "threading.h"
#include "zstd_internal.h" /* includes zstd.h */
#include "zstd_compress_internal.h" /* ZSTD_hash*() */
#include "zdict.h"
#include "cover.h"
/*-*************************************
* Constants
***************************************/
/**
* There are 32bit indexes used to ref samples, so limit samples size to 4GB
* on 64bit builds.
* For 32bit builds we choose 1 GB.
* Most 32bit platforms have 2GB user-mode addressable space and we allocate a large
* contiguous buffer, so 1GB is already a high limit.
*/
#define FASTCOVER_MAX_SAMPLES_SIZE (sizeof(size_t) == 8 ? ((unsigned)-1) : ((unsigned)1 GB))
#define FASTCOVER_MAX_F 31
#define FASTCOVER_MAX_ACCEL 10
#define FASTCOVER_DEFAULT_SPLITPOINT 0.75
#define DEFAULT_F 20
#define DEFAULT_ACCEL 1
/*-*************************************
* Console display
***************************************/
#ifndef LOCALDISPLAYLEVEL
static int g_displayLevel = 0;
#endif
#undef DISPLAY
#define DISPLAY(...) \
{ \
fprintf(stderr, __VA_ARGS__); \
fflush(stderr); \
}
#undef LOCALDISPLAYLEVEL
#define LOCALDISPLAYLEVEL(displayLevel, l, ...) \
if (displayLevel >= l) { \
DISPLAY(__VA_ARGS__); \
} /* 0 : no display; 1: errors; 2: default; 3: details; 4: debug */
#undef DISPLAYLEVEL
#define DISPLAYLEVEL(l, ...) LOCALDISPLAYLEVEL(g_displayLevel, l, __VA_ARGS__)
#ifndef LOCALDISPLAYUPDATE
static const clock_t g_refreshRate = CLOCKS_PER_SEC * 15 / 100;
static clock_t g_time = 0;
#endif
#undef LOCALDISPLAYUPDATE
#define LOCALDISPLAYUPDATE(displayLevel, l, ...) \
if (displayLevel >= l) { \
if ((clock() - g_time > g_refreshRate) || (displayLevel >= 4)) { \
g_time = clock(); \
DISPLAY(__VA_ARGS__); \
} \
}
#undef DISPLAYUPDATE
#define DISPLAYUPDATE(l, ...) LOCALDISPLAYUPDATE(g_displayLevel, l, __VA_ARGS__)
/*-*************************************
* Hash Functions
***************************************/
/**
* Hash the d-byte value pointed to by p and mod 2^f into the frequency vector
*/
static size_t FASTCOVER_hashPtrToIndex(const void* p, U32 f, unsigned d) {
if (d == 6) {
return ZSTD_hash6Ptr(p, f);
}
return ZSTD_hash8Ptr(p, f);
}
/*-*************************************
* Acceleration
***************************************/
typedef struct {
unsigned finalize; /* Percentage of training samples used for ZDICT_finalizeDictionary */
unsigned skip; /* Number of dmer skipped between each dmer counted in computeFrequency */
} FASTCOVER_accel_t;
static const FASTCOVER_accel_t FASTCOVER_defaultAccelParameters[FASTCOVER_MAX_ACCEL+1] = {
{ 100, 0 }, /* accel = 0, should not happen because accel = 0 defaults to accel = 1 */
{ 100, 0 }, /* accel = 1 */
{ 50, 1 }, /* accel = 2 */
{ 34, 2 }, /* accel = 3 */
{ 25, 3 }, /* accel = 4 */
{ 20, 4 }, /* accel = 5 */
{ 17, 5 }, /* accel = 6 */
{ 14, 6 }, /* accel = 7 */
{ 13, 7 }, /* accel = 8 */
{ 11, 8 }, /* accel = 9 */
{ 10, 9 }, /* accel = 10 */
};
/*-*************************************
* Context
***************************************/
typedef struct {
const BYTE *samples;
size_t *offsets;
const size_t *samplesSizes;
size_t nbSamples;
size_t nbTrainSamples;
size_t nbTestSamples;
size_t nbDmers;
U32 *freqs;
unsigned d;
unsigned f;
FASTCOVER_accel_t accelParams;
} FASTCOVER_ctx_t;
/*-*************************************
* Helper functions
***************************************/
/**
* Selects the best segment in an epoch.
* Segments of are scored according to the function:
*
* Let F(d) be the frequency of all dmers with hash value d.
* Let S_i be hash value of the dmer at position i of segment S which has length k.
*
* Score(S) = F(S_1) + F(S_2) + ... + F(S_{k-d+1})
*
* Once the dmer with hash value d is in the dictionary we set F(d) = 0.
*/
static COVER_segment_t FASTCOVER_selectSegment(const FASTCOVER_ctx_t *ctx,
U32 *freqs, U32 begin, U32 end,
ZDICT_cover_params_t parameters,
U16* segmentFreqs) {
/* Constants */
const U32 k = parameters.k;
const U32 d = parameters.d;
const U32 f = ctx->f;
const U32 dmersInK = k - d + 1;
/* Try each segment (activeSegment) and save the best (bestSegment) */
COVER_segment_t bestSegment = {0, 0, 0};
COVER_segment_t activeSegment;
/* Reset the activeDmers in the segment */
/* The activeSegment starts at the beginning of the epoch. */
activeSegment.begin = begin;
activeSegment.end = begin;
activeSegment.score = 0;
/* Slide the activeSegment through the whole epoch.
* Save the best segment in bestSegment.
*/
while (activeSegment.end < end) {
/* Get hash value of current dmer */
const size_t idx = FASTCOVER_hashPtrToIndex(ctx->samples + activeSegment.end, f, d);
/* Add frequency of this index to score if this is the first occurrence of index in active segment */
if (segmentFreqs[idx] == 0) {
activeSegment.score += freqs[idx];
}
/* Increment end of segment and segmentFreqs*/
activeSegment.end += 1;
segmentFreqs[idx] += 1;
/* If the window is now too large, drop the first position */
if (activeSegment.end - activeSegment.begin == dmersInK + 1) {
/* Get hash value of the dmer to be eliminated from active segment */
const size_t delIndex = FASTCOVER_hashPtrToIndex(ctx->samples + activeSegment.begin, f, d);
segmentFreqs[delIndex] -= 1;
/* Subtract frequency of this index from score if this is the last occurrence of this index in active segment */
if (segmentFreqs[delIndex] == 0) {
activeSegment.score -= freqs[delIndex];
}
/* Increment start of segment */
activeSegment.begin += 1;
}
/* If this segment is the best so far save it */
if (activeSegment.score > bestSegment.score) {
bestSegment = activeSegment;
}
}
/* Zero out rest of segmentFreqs array */
while (activeSegment.begin < end) {
const size_t delIndex = FASTCOVER_hashPtrToIndex(ctx->samples + activeSegment.begin, f, d);
segmentFreqs[delIndex] -= 1;
activeSegment.begin += 1;
}
{
/* Zero the frequency of hash value of each dmer covered by the chosen segment. */
U32 pos;
for (pos = bestSegment.begin; pos != bestSegment.end; ++pos) {
const size_t i = FASTCOVER_hashPtrToIndex(ctx->samples + pos, f, d);
freqs[i] = 0;
}
}
return bestSegment;
}
static int FASTCOVER_checkParameters(ZDICT_cover_params_t parameters,
size_t maxDictSize, unsigned f,
unsigned accel) {
/* k, d, and f are required parameters */
if (parameters.d == 0 || parameters.k == 0) {
return 0;
}
/* d has to be 6 or 8 */
if (parameters.d != 6 && parameters.d != 8) {
return 0;
}
/* k <= maxDictSize */
if (parameters.k > maxDictSize) {
return 0;
}
/* d <= k */
if (parameters.d > parameters.k) {
return 0;
}
/* 0 < f <= FASTCOVER_MAX_F*/
if (f > FASTCOVER_MAX_F || f == 0) {
return 0;
}
/* 0 < splitPoint <= 1 */
if (parameters.splitPoint <= 0 || parameters.splitPoint > 1) {
return 0;
}
/* 0 < accel <= 10 */
if (accel > 10 || accel == 0) {
return 0;
}
return 1;
}
/**
* Clean up a context initialized with `FASTCOVER_ctx_init()`.
*/
static void
FASTCOVER_ctx_destroy(FASTCOVER_ctx_t* ctx)
{
if (!ctx) return;
free(ctx->freqs);
ctx->freqs = NULL;
free(ctx->offsets);
ctx->offsets = NULL;
}
/**
* Calculate for frequency of hash value of each dmer in ctx->samples
*/
static void
FASTCOVER_computeFrequency(U32* freqs, const FASTCOVER_ctx_t* ctx)
{
const unsigned f = ctx->f;
const unsigned d = ctx->d;
const unsigned skip = ctx->accelParams.skip;
const unsigned readLength = MAX(d, 8);
size_t i;
assert(ctx->nbTrainSamples >= 5);
assert(ctx->nbTrainSamples <= ctx->nbSamples);
for (i = 0; i < ctx->nbTrainSamples; i++) {
size_t start = ctx->offsets[i]; /* start of current dmer */
size_t const currSampleEnd = ctx->offsets[i+1];
while (start + readLength <= currSampleEnd) {
const size_t dmerIndex = FASTCOVER_hashPtrToIndex(ctx->samples + start, f, d);
freqs[dmerIndex]++;
start = start + skip + 1;
}
}
}
/**
* Prepare a context for dictionary building.
* The context is only dependent on the parameter `d` and can be used multiple
* times.
* Returns 0 on success or error code on error.
* The context must be destroyed with `FASTCOVER_ctx_destroy()`.
*/
static size_t
FASTCOVER_ctx_init(FASTCOVER_ctx_t* ctx,
const void* samplesBuffer,
const size_t* samplesSizes, unsigned nbSamples,
unsigned d, double splitPoint, unsigned f,
FASTCOVER_accel_t accelParams)
{
const BYTE* const samples = (const BYTE*)samplesBuffer;
const size_t totalSamplesSize = COVER_sum(samplesSizes, nbSamples);
/* Split samples into testing and training sets */
const unsigned nbTrainSamples = splitPoint < 1.0 ? (unsigned)((double)nbSamples * splitPoint) : nbSamples;
const unsigned nbTestSamples = splitPoint < 1.0 ? nbSamples - nbTrainSamples : nbSamples;
const size_t trainingSamplesSize = splitPoint < 1.0 ? COVER_sum(samplesSizes, nbTrainSamples) : totalSamplesSize;
const size_t testSamplesSize = splitPoint < 1.0 ? COVER_sum(samplesSizes + nbTrainSamples, nbTestSamples) : totalSamplesSize;
/* Checks */
if (totalSamplesSize < MAX(d, sizeof(U64)) ||
totalSamplesSize >= (size_t)FASTCOVER_MAX_SAMPLES_SIZE) {
DISPLAYLEVEL(1, "Total samples size is too large (%u MB), maximum size is %u MB\n",
(unsigned)(totalSamplesSize >> 20), (FASTCOVER_MAX_SAMPLES_SIZE >> 20));
return ERROR(srcSize_wrong);
}
/* Check if there are at least 5 training samples */
if (nbTrainSamples < 5) {
DISPLAYLEVEL(1, "Total number of training samples is %u and is invalid\n", nbTrainSamples);
return ERROR(srcSize_wrong);
}
/* Check if there's testing sample */
if (nbTestSamples < 1) {
DISPLAYLEVEL(1, "Total number of testing samples is %u and is invalid.\n", nbTestSamples);
return ERROR(srcSize_wrong);
}
/* Zero the context */
memset(ctx, 0, sizeof(*ctx));
DISPLAYLEVEL(2, "Training on %u samples of total size %u\n", nbTrainSamples,
(unsigned)trainingSamplesSize);
DISPLAYLEVEL(2, "Testing on %u samples of total size %u\n", nbTestSamples,
(unsigned)testSamplesSize);
ctx->samples = samples;
ctx->samplesSizes = samplesSizes;
ctx->nbSamples = nbSamples;
ctx->nbTrainSamples = nbTrainSamples;
ctx->nbTestSamples = nbTestSamples;
ctx->nbDmers = trainingSamplesSize - MAX(d, sizeof(U64)) + 1;
ctx->d = d;
ctx->f = f;
ctx->accelParams = accelParams;
/* The offsets of each file */
ctx->offsets = (size_t*)calloc((nbSamples + 1), sizeof(size_t));
if (ctx->offsets == NULL) {
DISPLAYLEVEL(1, "Failed to allocate scratch buffers \n");
FASTCOVER_ctx_destroy(ctx);
return ERROR(memory_allocation);
}
/* Fill offsets from the samplesSizes */
{ U32 i;
ctx->offsets[0] = 0;
assert(nbSamples >= 5);
for (i = 1; i <= nbSamples; ++i) {
ctx->offsets[i] = ctx->offsets[i - 1] + samplesSizes[i - 1];
}
}
/* Initialize frequency array of size 2^f */
ctx->freqs = (U32*)calloc(((U64)1 << f), sizeof(U32));
if (ctx->freqs == NULL) {
DISPLAYLEVEL(1, "Failed to allocate frequency table \n");
FASTCOVER_ctx_destroy(ctx);
return ERROR(memory_allocation);
}
DISPLAYLEVEL(2, "Computing frequencies\n");
FASTCOVER_computeFrequency(ctx->freqs, ctx);
return 0;
}
/**
* Given the prepared context build the dictionary.
*/
static size_t
FASTCOVER_buildDictionary(const FASTCOVER_ctx_t* ctx,
U32* freqs,
void* dictBuffer, size_t dictBufferCapacity,
ZDICT_cover_params_t parameters,
U16* segmentFreqs)
{
BYTE *const dict = (BYTE *)dictBuffer;
size_t tail = dictBufferCapacity;
/* Divide the data into epochs. We will select one segment from each epoch. */
const COVER_epoch_info_t epochs = COVER_computeEpochs(
(U32)dictBufferCapacity, (U32)ctx->nbDmers, parameters.k, 1);
const size_t maxZeroScoreRun = 10;
size_t zeroScoreRun = 0;
size_t epoch;
DISPLAYLEVEL(2, "Breaking content into %u epochs of size %u\n",
(U32)epochs.num, (U32)epochs.size);
/* Loop through the epochs until there are no more segments or the dictionary
* is full.
*/
for (epoch = 0; tail > 0; epoch = (epoch + 1) % epochs.num) {
const U32 epochBegin = (U32)(epoch * epochs.size);
const U32 epochEnd = epochBegin + epochs.size;
size_t segmentSize;
/* Select a segment */
COVER_segment_t segment = FASTCOVER_selectSegment(
ctx, freqs, epochBegin, epochEnd, parameters, segmentFreqs);
/* If the segment covers no dmers, then we are out of content.
* There may be new content in other epochs, for continue for some time.
*/
if (segment.score == 0) {
if (++zeroScoreRun >= maxZeroScoreRun) {
break;
}
continue;
}
zeroScoreRun = 0;
/* Trim the segment if necessary and if it is too small then we are done */
segmentSize = MIN(segment.end - segment.begin + parameters.d - 1, tail);
if (segmentSize < parameters.d) {
break;
}
/* We fill the dictionary from the back to allow the best segments to be
* referenced with the smallest offsets.
*/
tail -= segmentSize;
memcpy(dict + tail, ctx->samples + segment.begin, segmentSize);
DISPLAYUPDATE(
2, "\r%u%% ",
(unsigned)(((dictBufferCapacity - tail) * 100) / dictBufferCapacity));
}
DISPLAYLEVEL(2, "\r%79s\r", "");
return tail;
}
/**
* Parameters for FASTCOVER_tryParameters().
*/
typedef struct FASTCOVER_tryParameters_data_s {
const FASTCOVER_ctx_t* ctx;
COVER_best_t* best;
size_t dictBufferCapacity;
ZDICT_cover_params_t parameters;
} FASTCOVER_tryParameters_data_t;
/**
* Tries a set of parameters and updates the COVER_best_t with the results.
* This function is thread safe if zstd is compiled with multithreaded support.
* It takes its parameters as an *OWNING* opaque pointer to support threading.
*/
static void FASTCOVER_tryParameters(void* opaque)
{
/* Save parameters as local variables */
FASTCOVER_tryParameters_data_t *const data = (FASTCOVER_tryParameters_data_t*)opaque;
const FASTCOVER_ctx_t *const ctx = data->ctx;
const ZDICT_cover_params_t parameters = data->parameters;
size_t dictBufferCapacity = data->dictBufferCapacity;
size_t totalCompressedSize = ERROR(GENERIC);
/* Initialize array to keep track of frequency of dmer within activeSegment */
U16* segmentFreqs = (U16*)calloc(((U64)1 << ctx->f), sizeof(U16));
/* Allocate space for hash table, dict, and freqs */
BYTE *const dict = (BYTE*)malloc(dictBufferCapacity);
COVER_dictSelection_t selection = COVER_dictSelectionError(ERROR(GENERIC));
U32* freqs = (U32*) malloc(((U64)1 << ctx->f) * sizeof(U32));
if (!segmentFreqs || !dict || !freqs) {
DISPLAYLEVEL(1, "Failed to allocate buffers: out of memory\n");
goto _cleanup;
}
/* Copy the frequencies because we need to modify them */
memcpy(freqs, ctx->freqs, ((U64)1 << ctx->f) * sizeof(U32));
/* Build the dictionary */
{ const size_t tail = FASTCOVER_buildDictionary(ctx, freqs, dict, dictBufferCapacity,
parameters, segmentFreqs);
const unsigned nbFinalizeSamples = (unsigned)(ctx->nbTrainSamples * ctx->accelParams.finalize / 100);
selection = COVER_selectDict(dict + tail, dictBufferCapacity, dictBufferCapacity - tail,
ctx->samples, ctx->samplesSizes, nbFinalizeSamples, ctx->nbTrainSamples, ctx->nbSamples, parameters, ctx->offsets,
totalCompressedSize);
if (COVER_dictSelectionIsError(selection)) {
DISPLAYLEVEL(1, "Failed to select dictionary\n");
goto _cleanup;
}
}
_cleanup:
free(dict);
COVER_best_finish(data->best, parameters, selection);
free(data);
free(segmentFreqs);
COVER_dictSelectionFree(selection);
free(freqs);
}
static void
FASTCOVER_convertToCoverParams(ZDICT_fastCover_params_t fastCoverParams,
ZDICT_cover_params_t* coverParams)
{
coverParams->k = fastCoverParams.k;
coverParams->d = fastCoverParams.d;
coverParams->steps = fastCoverParams.steps;
coverParams->nbThreads = fastCoverParams.nbThreads;
coverParams->splitPoint = fastCoverParams.splitPoint;
coverParams->zParams = fastCoverParams.zParams;
coverParams->shrinkDict = fastCoverParams.shrinkDict;
}
static void
FASTCOVER_convertToFastCoverParams(ZDICT_cover_params_t coverParams,
ZDICT_fastCover_params_t* fastCoverParams,
unsigned f, unsigned accel)
{
fastCoverParams->k = coverParams.k;
fastCoverParams->d = coverParams.d;
fastCoverParams->steps = coverParams.steps;
fastCoverParams->nbThreads = coverParams.nbThreads;
fastCoverParams->splitPoint = coverParams.splitPoint;
fastCoverParams->f = f;
fastCoverParams->accel = accel;
fastCoverParams->zParams = coverParams.zParams;
fastCoverParams->shrinkDict = coverParams.shrinkDict;
}
ZDICTLIB_STATIC_API size_t
ZDICT_trainFromBuffer_fastCover(void* dictBuffer, size_t dictBufferCapacity,
const void* samplesBuffer,
const size_t* samplesSizes, unsigned nbSamples,
ZDICT_fastCover_params_t parameters)
{
BYTE* const dict = (BYTE*)dictBuffer;
FASTCOVER_ctx_t ctx;
ZDICT_cover_params_t coverParams;
FASTCOVER_accel_t accelParams;
/* Initialize global data */
g_displayLevel = (int)parameters.zParams.notificationLevel;
/* Assign splitPoint and f if not provided */
parameters.splitPoint = 1.0;
parameters.f = parameters.f == 0 ? DEFAULT_F : parameters.f;
parameters.accel = parameters.accel == 0 ? DEFAULT_ACCEL : parameters.accel;
/* Convert to cover parameter */
memset(&coverParams, 0 , sizeof(coverParams));
FASTCOVER_convertToCoverParams(parameters, &coverParams);
/* Checks */
if (!FASTCOVER_checkParameters(coverParams, dictBufferCapacity, parameters.f,
parameters.accel)) {
DISPLAYLEVEL(1, "FASTCOVER parameters incorrect\n");
return ERROR(parameter_outOfBound);
}
if (nbSamples == 0) {
DISPLAYLEVEL(1, "FASTCOVER must have at least one input file\n");
return ERROR(srcSize_wrong);
}
if (dictBufferCapacity < ZDICT_DICTSIZE_MIN) {
DISPLAYLEVEL(1, "dictBufferCapacity must be at least %u\n",
ZDICT_DICTSIZE_MIN);
return ERROR(dstSize_tooSmall);
}
/* Assign corresponding FASTCOVER_accel_t to accelParams*/
accelParams = FASTCOVER_defaultAccelParameters[parameters.accel];
/* Initialize context */
{
size_t const initVal = FASTCOVER_ctx_init(&ctx, samplesBuffer, samplesSizes, nbSamples,
coverParams.d, parameters.splitPoint, parameters.f,
accelParams);
if (ZSTD_isError(initVal)) {
DISPLAYLEVEL(1, "Failed to initialize context\n");
return initVal;
}
}
COVER_warnOnSmallCorpus(dictBufferCapacity, ctx.nbDmers, g_displayLevel);
/* Build the dictionary */
DISPLAYLEVEL(2, "Building dictionary\n");
{
/* Initialize array to keep track of frequency of dmer within activeSegment */
U16* segmentFreqs = (U16 *)calloc(((U64)1 << parameters.f), sizeof(U16));
const size_t tail = FASTCOVER_buildDictionary(&ctx, ctx.freqs, dictBuffer,
dictBufferCapacity, coverParams, segmentFreqs);
const unsigned nbFinalizeSamples = (unsigned)(ctx.nbTrainSamples * ctx.accelParams.finalize / 100);
const size_t dictionarySize = ZDICT_finalizeDictionary(
dict, dictBufferCapacity, dict + tail, dictBufferCapacity - tail,
samplesBuffer, samplesSizes, nbFinalizeSamples, coverParams.zParams);
if (!ZSTD_isError(dictionarySize)) {
DISPLAYLEVEL(2, "Constructed dictionary of size %u\n",
(unsigned)dictionarySize);
}
FASTCOVER_ctx_destroy(&ctx);
free(segmentFreqs);
return dictionarySize;
}
}
ZDICTLIB_STATIC_API size_t
ZDICT_optimizeTrainFromBuffer_fastCover(
void* dictBuffer, size_t dictBufferCapacity,
const void* samplesBuffer,
const size_t* samplesSizes, unsigned nbSamples,
ZDICT_fastCover_params_t* parameters)
{
ZDICT_cover_params_t coverParams;
FASTCOVER_accel_t accelParams;
/* constants */
const unsigned nbThreads = parameters->nbThreads;
const double splitPoint =
parameters->splitPoint <= 0.0 ? FASTCOVER_DEFAULT_SPLITPOINT : parameters->splitPoint;
const unsigned kMinD = parameters->d == 0 ? 6 : parameters->d;
const unsigned kMaxD = parameters->d == 0 ? 8 : parameters->d;
const unsigned kMinK = parameters->k == 0 ? 50 : parameters->k;
const unsigned kMaxK = parameters->k == 0 ? 2000 : parameters->k;
const unsigned kSteps = parameters->steps == 0 ? 40 : parameters->steps;
const unsigned kStepSize = MAX((kMaxK - kMinK) / kSteps, 1);
const unsigned kIterations =
(1 + (kMaxD - kMinD) / 2) * (1 + (kMaxK - kMinK) / kStepSize);
const unsigned f = parameters->f == 0 ? DEFAULT_F : parameters->f;
const unsigned accel = parameters->accel == 0 ? DEFAULT_ACCEL : parameters->accel;
const unsigned shrinkDict = 0;
/* Local variables */
const int displayLevel = (int)parameters->zParams.notificationLevel;
unsigned iteration = 1;
unsigned d;
unsigned k;
COVER_best_t best;
POOL_ctx *pool = NULL;
int warned = 0;
/* Checks */
if (splitPoint <= 0 || splitPoint > 1) {
LOCALDISPLAYLEVEL(displayLevel, 1, "Incorrect splitPoint\n");
return ERROR(parameter_outOfBound);
}
if (accel == 0 || accel > FASTCOVER_MAX_ACCEL) {
LOCALDISPLAYLEVEL(displayLevel, 1, "Incorrect accel\n");
return ERROR(parameter_outOfBound);
}
if (kMinK < kMaxD || kMaxK < kMinK) {
LOCALDISPLAYLEVEL(displayLevel, 1, "Incorrect k\n");
return ERROR(parameter_outOfBound);
}
if (nbSamples == 0) {
LOCALDISPLAYLEVEL(displayLevel, 1, "FASTCOVER must have at least one input file\n");
return ERROR(srcSize_wrong);
}
if (dictBufferCapacity < ZDICT_DICTSIZE_MIN) {
LOCALDISPLAYLEVEL(displayLevel, 1, "dictBufferCapacity must be at least %u\n",
ZDICT_DICTSIZE_MIN);
return ERROR(dstSize_tooSmall);
}
if (nbThreads > 1) {
pool = POOL_create(nbThreads, 1);
if (!pool) {
return ERROR(memory_allocation);
}
}
/* Initialization */
COVER_best_init(&best);
memset(&coverParams, 0 , sizeof(coverParams));
FASTCOVER_convertToCoverParams(*parameters, &coverParams);
accelParams = FASTCOVER_defaultAccelParameters[accel];
/* Turn down global display level to clean up display at level 2 and below */
g_displayLevel = displayLevel == 0 ? 0 : displayLevel - 1;
/* Loop through d first because each new value needs a new context */
LOCALDISPLAYLEVEL(displayLevel, 2, "Trying %u different sets of parameters\n",
kIterations);
for (d = kMinD; d <= kMaxD; d += 2) {
/* Initialize the context for this value of d */
FASTCOVER_ctx_t ctx;
LOCALDISPLAYLEVEL(displayLevel, 3, "d=%u\n", d);
{
size_t const initVal = FASTCOVER_ctx_init(&ctx, samplesBuffer, samplesSizes, nbSamples, d, splitPoint, f, accelParams);
if (ZSTD_isError(initVal)) {
LOCALDISPLAYLEVEL(displayLevel, 1, "Failed to initialize context\n");
COVER_best_destroy(&best);
POOL_free(pool);
return initVal;
}
}
if (!warned) {
COVER_warnOnSmallCorpus(dictBufferCapacity, ctx.nbDmers, displayLevel);
warned = 1;
}
/* Loop through k reusing the same context */
for (k = kMinK; k <= kMaxK; k += kStepSize) {
/* Prepare the arguments */
FASTCOVER_tryParameters_data_t *data = (FASTCOVER_tryParameters_data_t *)malloc(
sizeof(FASTCOVER_tryParameters_data_t));
LOCALDISPLAYLEVEL(displayLevel, 3, "k=%u\n", k);
if (!data) {
LOCALDISPLAYLEVEL(displayLevel, 1, "Failed to allocate parameters\n");
COVER_best_destroy(&best);
FASTCOVER_ctx_destroy(&ctx);
POOL_free(pool);
return ERROR(memory_allocation);
}
data->ctx = &ctx;
data->best = &best;
data->dictBufferCapacity = dictBufferCapacity;
data->parameters = coverParams;
data->parameters.k = k;
data->parameters.d = d;
data->parameters.splitPoint = splitPoint;
data->parameters.steps = kSteps;
data->parameters.shrinkDict = shrinkDict;
data->parameters.zParams.notificationLevel = (unsigned)g_displayLevel;
/* Check the parameters */
if (!FASTCOVER_checkParameters(data->parameters, dictBufferCapacity,
data->ctx->f, accel)) {
DISPLAYLEVEL(1, "FASTCOVER parameters incorrect\n");
free(data);
continue;
}
/* Call the function and pass ownership of data to it */
COVER_best_start(&best);
if (pool) {
POOL_add(pool, &FASTCOVER_tryParameters, data);
} else {
FASTCOVER_tryParameters(data);
}
/* Print status */
LOCALDISPLAYUPDATE(displayLevel, 2, "\r%u%% ",
(unsigned)((iteration * 100) / kIterations));
++iteration;
}
COVER_best_wait(&best);
FASTCOVER_ctx_destroy(&ctx);
}
LOCALDISPLAYLEVEL(displayLevel, 2, "\r%79s\r", "");
/* Fill the output buffer and parameters with output of the best parameters */
{
const size_t dictSize = best.dictSize;
if (ZSTD_isError(best.compressedSize)) {
const size_t compressedSize = best.compressedSize;
COVER_best_destroy(&best);
POOL_free(pool);
return compressedSize;
}
FASTCOVER_convertToFastCoverParams(best.parameters, parameters, f, accel);
memcpy(dictBuffer, best.dict, dictSize);
COVER_best_destroy(&best);
POOL_free(pool);
return dictSize;
}
}
#endif /* USE_EXTERNAL_ZSTD */

628
vendor/github.com/DataDog/zstd/fse.h generated vendored
View File

@@ -1,628 +0,0 @@
#ifndef USE_EXTERNAL_ZSTD
/* ******************************************************************
* FSE : Finite State Entropy codec
* Public Prototypes declaration
* Copyright (c) Meta Platforms, Inc. and affiliates.
*
* You can contact the author at :
* - Source repository : https://github.com/Cyan4973/FiniteStateEntropy
*
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
* You may select, at your option, one of the above-listed licenses.
****************************************************************** */
#ifndef FSE_H
#define FSE_H
/*-*****************************************
* Dependencies
******************************************/
#include "zstd_deps.h" /* size_t, ptrdiff_t */
/*-*****************************************
* FSE_PUBLIC_API : control library symbols visibility
******************************************/
#if defined(FSE_DLL_EXPORT) && (FSE_DLL_EXPORT==1) && defined(__GNUC__) && (__GNUC__ >= 4)
# define FSE_PUBLIC_API __attribute__ ((visibility ("default")))
#elif defined(FSE_DLL_EXPORT) && (FSE_DLL_EXPORT==1) /* Visual expected */
# define FSE_PUBLIC_API __declspec(dllexport)
#elif defined(FSE_DLL_IMPORT) && (FSE_DLL_IMPORT==1)
# define FSE_PUBLIC_API __declspec(dllimport) /* It isn't required but allows to generate better code, saving a function pointer load from the IAT and an indirect jump.*/
#else
# define FSE_PUBLIC_API
#endif
/*------ Version ------*/
#define FSE_VERSION_MAJOR 0
#define FSE_VERSION_MINOR 9
#define FSE_VERSION_RELEASE 0
#define FSE_LIB_VERSION FSE_VERSION_MAJOR.FSE_VERSION_MINOR.FSE_VERSION_RELEASE
#define FSE_QUOTE(str) #str
#define FSE_EXPAND_AND_QUOTE(str) FSE_QUOTE(str)
#define FSE_VERSION_STRING FSE_EXPAND_AND_QUOTE(FSE_LIB_VERSION)
#define FSE_VERSION_NUMBER (FSE_VERSION_MAJOR *100*100 + FSE_VERSION_MINOR *100 + FSE_VERSION_RELEASE)
FSE_PUBLIC_API unsigned FSE_versionNumber(void); /**< library version number; to be used when checking dll version */
/*-*****************************************
* Tool functions
******************************************/
FSE_PUBLIC_API size_t FSE_compressBound(size_t size); /* maximum compressed size */
/* Error Management */
FSE_PUBLIC_API unsigned FSE_isError(size_t code); /* tells if a return value is an error code */
FSE_PUBLIC_API const char* FSE_getErrorName(size_t code); /* provides error code string (useful for debugging) */
/*-*****************************************
* FSE detailed API
******************************************/
/*!
FSE_compress() does the following:
1. count symbol occurrence from source[] into table count[] (see hist.h)
2. normalize counters so that sum(count[]) == Power_of_2 (2^tableLog)
3. save normalized counters to memory buffer using writeNCount()
4. build encoding table 'CTable' from normalized counters
5. encode the data stream using encoding table 'CTable'
FSE_decompress() does the following:
1. read normalized counters with readNCount()
2. build decoding table 'DTable' from normalized counters
3. decode the data stream using decoding table 'DTable'
The following API allows targeting specific sub-functions for advanced tasks.
For example, it's possible to compress several blocks using the same 'CTable',
or to save and provide normalized distribution using external method.
*/
/* *** COMPRESSION *** */
/*! FSE_optimalTableLog():
dynamically downsize 'tableLog' when conditions are met.
It saves CPU time, by using smaller tables, while preserving or even improving compression ratio.
@return : recommended tableLog (necessarily <= 'maxTableLog') */
FSE_PUBLIC_API unsigned FSE_optimalTableLog(unsigned maxTableLog, size_t srcSize, unsigned maxSymbolValue);
/*! FSE_normalizeCount():
normalize counts so that sum(count[]) == Power_of_2 (2^tableLog)
'normalizedCounter' is a table of short, of minimum size (maxSymbolValue+1).
useLowProbCount is a boolean parameter which trades off compressed size for
faster header decoding. When it is set to 1, the compressed data will be slightly
smaller. And when it is set to 0, FSE_readNCount() and FSE_buildDTable() will be
faster. If you are compressing a small amount of data (< 2 KB) then useLowProbCount=0
is a good default, since header deserialization makes a big speed difference.
Otherwise, useLowProbCount=1 is a good default, since the speed difference is small.
@return : tableLog,
or an errorCode, which can be tested using FSE_isError() */
FSE_PUBLIC_API size_t FSE_normalizeCount(short* normalizedCounter, unsigned tableLog,
const unsigned* count, size_t srcSize, unsigned maxSymbolValue, unsigned useLowProbCount);
/*! FSE_NCountWriteBound():
Provides the maximum possible size of an FSE normalized table, given 'maxSymbolValue' and 'tableLog'.
Typically useful for allocation purpose. */
FSE_PUBLIC_API size_t FSE_NCountWriteBound(unsigned maxSymbolValue, unsigned tableLog);
/*! FSE_writeNCount():
Compactly save 'normalizedCounter' into 'buffer'.
@return : size of the compressed table,
or an errorCode, which can be tested using FSE_isError(). */
FSE_PUBLIC_API size_t FSE_writeNCount (void* buffer, size_t bufferSize,
const short* normalizedCounter,
unsigned maxSymbolValue, unsigned tableLog);
/*! Constructor and Destructor of FSE_CTable.
Note that FSE_CTable size depends on 'tableLog' and 'maxSymbolValue' */
typedef unsigned FSE_CTable; /* don't allocate that. It's only meant to be more restrictive than void* */
/*! FSE_buildCTable():
Builds `ct`, which must be already allocated, using FSE_createCTable().
@return : 0, or an errorCode, which can be tested using FSE_isError() */
FSE_PUBLIC_API size_t FSE_buildCTable(FSE_CTable* ct, const short* normalizedCounter, unsigned maxSymbolValue, unsigned tableLog);
/*! FSE_compress_usingCTable():
Compress `src` using `ct` into `dst` which must be already allocated.
@return : size of compressed data (<= `dstCapacity`),
or 0 if compressed data could not fit into `dst`,
or an errorCode, which can be tested using FSE_isError() */
FSE_PUBLIC_API size_t FSE_compress_usingCTable (void* dst, size_t dstCapacity, const void* src, size_t srcSize, const FSE_CTable* ct);
/*!
Tutorial :
----------
The first step is to count all symbols. FSE_count() does this job very fast.
Result will be saved into 'count', a table of unsigned int, which must be already allocated, and have 'maxSymbolValuePtr[0]+1' cells.
'src' is a table of bytes of size 'srcSize'. All values within 'src' MUST be <= maxSymbolValuePtr[0]
maxSymbolValuePtr[0] will be updated, with its real value (necessarily <= original value)
FSE_count() will return the number of occurrence of the most frequent symbol.
This can be used to know if there is a single symbol within 'src', and to quickly evaluate its compressibility.
If there is an error, the function will return an ErrorCode (which can be tested using FSE_isError()).
The next step is to normalize the frequencies.
FSE_normalizeCount() will ensure that sum of frequencies is == 2 ^'tableLog'.
It also guarantees a minimum of 1 to any Symbol with frequency >= 1.
You can use 'tableLog'==0 to mean "use default tableLog value".
If you are unsure of which tableLog value to use, you can ask FSE_optimalTableLog(),
which will provide the optimal valid tableLog given sourceSize, maxSymbolValue, and a user-defined maximum (0 means "default").
The result of FSE_normalizeCount() will be saved into a table,
called 'normalizedCounter', which is a table of signed short.
'normalizedCounter' must be already allocated, and have at least 'maxSymbolValue+1' cells.
The return value is tableLog if everything proceeded as expected.
It is 0 if there is a single symbol within distribution.
If there is an error (ex: invalid tableLog value), the function will return an ErrorCode (which can be tested using FSE_isError()).
'normalizedCounter' can be saved in a compact manner to a memory area using FSE_writeNCount().
'buffer' must be already allocated.
For guaranteed success, buffer size must be at least FSE_headerBound().
The result of the function is the number of bytes written into 'buffer'.
If there is an error, the function will return an ErrorCode (which can be tested using FSE_isError(); ex : buffer size too small).
'normalizedCounter' can then be used to create the compression table 'CTable'.
The space required by 'CTable' must be already allocated, using FSE_createCTable().
You can then use FSE_buildCTable() to fill 'CTable'.
If there is an error, both functions will return an ErrorCode (which can be tested using FSE_isError()).
'CTable' can then be used to compress 'src', with FSE_compress_usingCTable().
Similar to FSE_count(), the convention is that 'src' is assumed to be a table of char of size 'srcSize'
The function returns the size of compressed data (without header), necessarily <= `dstCapacity`.
If it returns '0', compressed data could not fit into 'dst'.
If there is an error, the function will return an ErrorCode (which can be tested using FSE_isError()).
*/
/* *** DECOMPRESSION *** */
/*! FSE_readNCount():
Read compactly saved 'normalizedCounter' from 'rBuffer'.
@return : size read from 'rBuffer',
or an errorCode, which can be tested using FSE_isError().
maxSymbolValuePtr[0] and tableLogPtr[0] will also be updated with their respective values */
FSE_PUBLIC_API size_t FSE_readNCount (short* normalizedCounter,
unsigned* maxSymbolValuePtr, unsigned* tableLogPtr,
const void* rBuffer, size_t rBuffSize);
/*! FSE_readNCount_bmi2():
* Same as FSE_readNCount() but pass bmi2=1 when your CPU supports BMI2 and 0 otherwise.
*/
FSE_PUBLIC_API size_t FSE_readNCount_bmi2(short* normalizedCounter,
unsigned* maxSymbolValuePtr, unsigned* tableLogPtr,
const void* rBuffer, size_t rBuffSize, int bmi2);
typedef unsigned FSE_DTable; /* don't allocate that. It's just a way to be more restrictive than void* */
/*!
Tutorial :
----------
(Note : these functions only decompress FSE-compressed blocks.
If block is uncompressed, use memcpy() instead
If block is a single repeated byte, use memset() instead )
The first step is to obtain the normalized frequencies of symbols.
This can be performed by FSE_readNCount() if it was saved using FSE_writeNCount().
'normalizedCounter' must be already allocated, and have at least 'maxSymbolValuePtr[0]+1' cells of signed short.
In practice, that means it's necessary to know 'maxSymbolValue' beforehand,
or size the table to handle worst case situations (typically 256).
FSE_readNCount() will provide 'tableLog' and 'maxSymbolValue'.
The result of FSE_readNCount() is the number of bytes read from 'rBuffer'.
Note that 'rBufferSize' must be at least 4 bytes, even if useful information is less than that.
If there is an error, the function will return an error code, which can be tested using FSE_isError().
The next step is to build the decompression tables 'FSE_DTable' from 'normalizedCounter'.
This is performed by the function FSE_buildDTable().
The space required by 'FSE_DTable' must be already allocated using FSE_createDTable().
If there is an error, the function will return an error code, which can be tested using FSE_isError().
`FSE_DTable` can then be used to decompress `cSrc`, with FSE_decompress_usingDTable().
`cSrcSize` must be strictly correct, otherwise decompression will fail.
FSE_decompress_usingDTable() result will tell how many bytes were regenerated (<=`dstCapacity`).
If there is an error, the function will return an error code, which can be tested using FSE_isError(). (ex: dst buffer too small)
*/
#endif /* FSE_H */
#if defined(FSE_STATIC_LINKING_ONLY) && !defined(FSE_H_FSE_STATIC_LINKING_ONLY)
#define FSE_H_FSE_STATIC_LINKING_ONLY
#include "bitstream.h"
/* *****************************************
* Static allocation
*******************************************/
/* FSE buffer bounds */
#define FSE_NCOUNTBOUND 512
#define FSE_BLOCKBOUND(size) ((size) + ((size)>>7) + 4 /* fse states */ + sizeof(size_t) /* bitContainer */)
#define FSE_COMPRESSBOUND(size) (FSE_NCOUNTBOUND + FSE_BLOCKBOUND(size)) /* Macro version, useful for static allocation */
/* It is possible to statically allocate FSE CTable/DTable as a table of FSE_CTable/FSE_DTable using below macros */
#define FSE_CTABLE_SIZE_U32(maxTableLog, maxSymbolValue) (1 + (1<<((maxTableLog)-1)) + (((maxSymbolValue)+1)*2))
#define FSE_DTABLE_SIZE_U32(maxTableLog) (1 + (1<<(maxTableLog)))
/* or use the size to malloc() space directly. Pay attention to alignment restrictions though */
#define FSE_CTABLE_SIZE(maxTableLog, maxSymbolValue) (FSE_CTABLE_SIZE_U32(maxTableLog, maxSymbolValue) * sizeof(FSE_CTable))
#define FSE_DTABLE_SIZE(maxTableLog) (FSE_DTABLE_SIZE_U32(maxTableLog) * sizeof(FSE_DTable))
/* *****************************************
* FSE advanced API
***************************************** */
unsigned FSE_optimalTableLog_internal(unsigned maxTableLog, size_t srcSize, unsigned maxSymbolValue, unsigned minus);
/**< same as FSE_optimalTableLog(), which used `minus==2` */
size_t FSE_buildCTable_rle (FSE_CTable* ct, unsigned char symbolValue);
/**< build a fake FSE_CTable, designed to compress always the same symbolValue */
/* FSE_buildCTable_wksp() :
* Same as FSE_buildCTable(), but using an externally allocated scratch buffer (`workSpace`).
* `wkspSize` must be >= `FSE_BUILD_CTABLE_WORKSPACE_SIZE_U32(maxSymbolValue, tableLog)` of `unsigned`.
* See FSE_buildCTable_wksp() for breakdown of workspace usage.
*/
#define FSE_BUILD_CTABLE_WORKSPACE_SIZE_U32(maxSymbolValue, tableLog) (((maxSymbolValue + 2) + (1ull << (tableLog)))/2 + sizeof(U64)/sizeof(U32) /* additional 8 bytes for potential table overwrite */)
#define FSE_BUILD_CTABLE_WORKSPACE_SIZE(maxSymbolValue, tableLog) (sizeof(unsigned) * FSE_BUILD_CTABLE_WORKSPACE_SIZE_U32(maxSymbolValue, tableLog))
size_t FSE_buildCTable_wksp(FSE_CTable* ct, const short* normalizedCounter, unsigned maxSymbolValue, unsigned tableLog, void* workSpace, size_t wkspSize);
#define FSE_BUILD_DTABLE_WKSP_SIZE(maxTableLog, maxSymbolValue) (sizeof(short) * (maxSymbolValue + 1) + (1ULL << maxTableLog) + 8)
#define FSE_BUILD_DTABLE_WKSP_SIZE_U32(maxTableLog, maxSymbolValue) ((FSE_BUILD_DTABLE_WKSP_SIZE(maxTableLog, maxSymbolValue) + sizeof(unsigned) - 1) / sizeof(unsigned))
FSE_PUBLIC_API size_t FSE_buildDTable_wksp(FSE_DTable* dt, const short* normalizedCounter, unsigned maxSymbolValue, unsigned tableLog, void* workSpace, size_t wkspSize);
/**< Same as FSE_buildDTable(), using an externally allocated `workspace` produced with `FSE_BUILD_DTABLE_WKSP_SIZE_U32(maxSymbolValue)` */
#define FSE_DECOMPRESS_WKSP_SIZE_U32(maxTableLog, maxSymbolValue) (FSE_DTABLE_SIZE_U32(maxTableLog) + 1 + FSE_BUILD_DTABLE_WKSP_SIZE_U32(maxTableLog, maxSymbolValue) + (FSE_MAX_SYMBOL_VALUE + 1) / 2 + 1)
#define FSE_DECOMPRESS_WKSP_SIZE(maxTableLog, maxSymbolValue) (FSE_DECOMPRESS_WKSP_SIZE_U32(maxTableLog, maxSymbolValue) * sizeof(unsigned))
size_t FSE_decompress_wksp_bmi2(void* dst, size_t dstCapacity, const void* cSrc, size_t cSrcSize, unsigned maxLog, void* workSpace, size_t wkspSize, int bmi2);
/**< same as FSE_decompress(), using an externally allocated `workSpace` produced with `FSE_DECOMPRESS_WKSP_SIZE_U32(maxLog, maxSymbolValue)`.
* Set bmi2 to 1 if your CPU supports BMI2 or 0 if it doesn't */
typedef enum {
FSE_repeat_none, /**< Cannot use the previous table */
FSE_repeat_check, /**< Can use the previous table but it must be checked */
FSE_repeat_valid /**< Can use the previous table and it is assumed to be valid */
} FSE_repeat;
/* *****************************************
* FSE symbol compression API
*******************************************/
/*!
This API consists of small unitary functions, which highly benefit from being inlined.
Hence their body are included in next section.
*/
typedef struct {
ptrdiff_t value;
const void* stateTable;
const void* symbolTT;
unsigned stateLog;
} FSE_CState_t;
static void FSE_initCState(FSE_CState_t* CStatePtr, const FSE_CTable* ct);
static void FSE_encodeSymbol(BIT_CStream_t* bitC, FSE_CState_t* CStatePtr, unsigned symbol);
static void FSE_flushCState(BIT_CStream_t* bitC, const FSE_CState_t* CStatePtr);
/**<
These functions are inner components of FSE_compress_usingCTable().
They allow the creation of custom streams, mixing multiple tables and bit sources.
A key property to keep in mind is that encoding and decoding are done **in reverse direction**.
So the first symbol you will encode is the last you will decode, like a LIFO stack.
You will need a few variables to track your CStream. They are :
FSE_CTable ct; // Provided by FSE_buildCTable()
BIT_CStream_t bitStream; // bitStream tracking structure
FSE_CState_t state; // State tracking structure (can have several)
The first thing to do is to init bitStream and state.
size_t errorCode = BIT_initCStream(&bitStream, dstBuffer, maxDstSize);
FSE_initCState(&state, ct);
Note that BIT_initCStream() can produce an error code, so its result should be tested, using FSE_isError();
You can then encode your input data, byte after byte.
FSE_encodeSymbol() outputs a maximum of 'tableLog' bits at a time.
Remember decoding will be done in reverse direction.
FSE_encodeByte(&bitStream, &state, symbol);
At any time, you can also add any bit sequence.
Note : maximum allowed nbBits is 25, for compatibility with 32-bits decoders
BIT_addBits(&bitStream, bitField, nbBits);
The above methods don't commit data to memory, they just store it into local register, for speed.
Local register size is 64-bits on 64-bits systems, 32-bits on 32-bits systems (size_t).
Writing data to memory is a manual operation, performed by the flushBits function.
BIT_flushBits(&bitStream);
Your last FSE encoding operation shall be to flush your last state value(s).
FSE_flushState(&bitStream, &state);
Finally, you must close the bitStream.
The function returns the size of CStream in bytes.
If data couldn't fit into dstBuffer, it will return a 0 ( == not compressible)
If there is an error, it returns an errorCode (which can be tested using FSE_isError()).
size_t size = BIT_closeCStream(&bitStream);
*/
/* *****************************************
* FSE symbol decompression API
*******************************************/
typedef struct {
size_t state;
const void* table; /* precise table may vary, depending on U16 */
} FSE_DState_t;
static void FSE_initDState(FSE_DState_t* DStatePtr, BIT_DStream_t* bitD, const FSE_DTable* dt);
static unsigned char FSE_decodeSymbol(FSE_DState_t* DStatePtr, BIT_DStream_t* bitD);
static unsigned FSE_endOfDState(const FSE_DState_t* DStatePtr);
/**<
Let's now decompose FSE_decompress_usingDTable() into its unitary components.
You will decode FSE-encoded symbols from the bitStream,
and also any other bitFields you put in, **in reverse order**.
You will need a few variables to track your bitStream. They are :
BIT_DStream_t DStream; // Stream context
FSE_DState_t DState; // State context. Multiple ones are possible
FSE_DTable* DTablePtr; // Decoding table, provided by FSE_buildDTable()
The first thing to do is to init the bitStream.
errorCode = BIT_initDStream(&DStream, srcBuffer, srcSize);
You should then retrieve your initial state(s)
(in reverse flushing order if you have several ones) :
errorCode = FSE_initDState(&DState, &DStream, DTablePtr);
You can then decode your data, symbol after symbol.
For information the maximum number of bits read by FSE_decodeSymbol() is 'tableLog'.
Keep in mind that symbols are decoded in reverse order, like a LIFO stack (last in, first out).
unsigned char symbol = FSE_decodeSymbol(&DState, &DStream);
You can retrieve any bitfield you eventually stored into the bitStream (in reverse order)
Note : maximum allowed nbBits is 25, for 32-bits compatibility
size_t bitField = BIT_readBits(&DStream, nbBits);
All above operations only read from local register (which size depends on size_t).
Refueling the register from memory is manually performed by the reload method.
endSignal = FSE_reloadDStream(&DStream);
BIT_reloadDStream() result tells if there is still some more data to read from DStream.
BIT_DStream_unfinished : there is still some data left into the DStream.
BIT_DStream_endOfBuffer : Dstream reached end of buffer. Its container may no longer be completely filled.
BIT_DStream_completed : Dstream reached its exact end, corresponding in general to decompression completed.
BIT_DStream_tooFar : Dstream went too far. Decompression result is corrupted.
When reaching end of buffer (BIT_DStream_endOfBuffer), progress slowly, notably if you decode multiple symbols per loop,
to properly detect the exact end of stream.
After each decoded symbol, check if DStream is fully consumed using this simple test :
BIT_reloadDStream(&DStream) >= BIT_DStream_completed
When it's done, verify decompression is fully completed, by checking both DStream and the relevant states.
Checking if DStream has reached its end is performed by :
BIT_endOfDStream(&DStream);
Check also the states. There might be some symbols left there, if some high probability ones (>50%) are possible.
FSE_endOfDState(&DState);
*/
/* *****************************************
* FSE unsafe API
*******************************************/
static unsigned char FSE_decodeSymbolFast(FSE_DState_t* DStatePtr, BIT_DStream_t* bitD);
/* faster, but works only if nbBits is always >= 1 (otherwise, result will be corrupted) */
/* *****************************************
* Implementation of inlined functions
*******************************************/
typedef struct {
int deltaFindState;
U32 deltaNbBits;
} FSE_symbolCompressionTransform; /* total 8 bytes */
MEM_STATIC void FSE_initCState(FSE_CState_t* statePtr, const FSE_CTable* ct)
{
const void* ptr = ct;
const U16* u16ptr = (const U16*) ptr;
const U32 tableLog = MEM_read16(ptr);
statePtr->value = (ptrdiff_t)1<<tableLog;
statePtr->stateTable = u16ptr+2;
statePtr->symbolTT = ct + 1 + (tableLog ? (1<<(tableLog-1)) : 1);
statePtr->stateLog = tableLog;
}
/*! FSE_initCState2() :
* Same as FSE_initCState(), but the first symbol to include (which will be the last to be read)
* uses the smallest state value possible, saving the cost of this symbol */
MEM_STATIC void FSE_initCState2(FSE_CState_t* statePtr, const FSE_CTable* ct, U32 symbol)
{
FSE_initCState(statePtr, ct);
{ const FSE_symbolCompressionTransform symbolTT = ((const FSE_symbolCompressionTransform*)(statePtr->symbolTT))[symbol];
const U16* stateTable = (const U16*)(statePtr->stateTable);
U32 nbBitsOut = (U32)((symbolTT.deltaNbBits + (1<<15)) >> 16);
statePtr->value = (nbBitsOut << 16) - symbolTT.deltaNbBits;
statePtr->value = stateTable[(statePtr->value >> nbBitsOut) + symbolTT.deltaFindState];
}
}
MEM_STATIC void FSE_encodeSymbol(BIT_CStream_t* bitC, FSE_CState_t* statePtr, unsigned symbol)
{
FSE_symbolCompressionTransform const symbolTT = ((const FSE_symbolCompressionTransform*)(statePtr->symbolTT))[symbol];
const U16* const stateTable = (const U16*)(statePtr->stateTable);
U32 const nbBitsOut = (U32)((statePtr->value + symbolTT.deltaNbBits) >> 16);
BIT_addBits(bitC, (BitContainerType)statePtr->value, nbBitsOut);
statePtr->value = stateTable[ (statePtr->value >> nbBitsOut) + symbolTT.deltaFindState];
}
MEM_STATIC void FSE_flushCState(BIT_CStream_t* bitC, const FSE_CState_t* statePtr)
{
BIT_addBits(bitC, (BitContainerType)statePtr->value, statePtr->stateLog);
BIT_flushBits(bitC);
}
/* FSE_getMaxNbBits() :
* Approximate maximum cost of a symbol, in bits.
* Fractional get rounded up (i.e. a symbol with a normalized frequency of 3 gives the same result as a frequency of 2)
* note 1 : assume symbolValue is valid (<= maxSymbolValue)
* note 2 : if freq[symbolValue]==0, @return a fake cost of tableLog+1 bits */
MEM_STATIC U32 FSE_getMaxNbBits(const void* symbolTTPtr, U32 symbolValue)
{
const FSE_symbolCompressionTransform* symbolTT = (const FSE_symbolCompressionTransform*) symbolTTPtr;
return (symbolTT[symbolValue].deltaNbBits + ((1<<16)-1)) >> 16;
}
/* FSE_bitCost() :
* Approximate symbol cost, as fractional value, using fixed-point format (accuracyLog fractional bits)
* note 1 : assume symbolValue is valid (<= maxSymbolValue)
* note 2 : if freq[symbolValue]==0, @return a fake cost of tableLog+1 bits */
MEM_STATIC U32 FSE_bitCost(const void* symbolTTPtr, U32 tableLog, U32 symbolValue, U32 accuracyLog)
{
const FSE_symbolCompressionTransform* symbolTT = (const FSE_symbolCompressionTransform*) symbolTTPtr;
U32 const minNbBits = symbolTT[symbolValue].deltaNbBits >> 16;
U32 const threshold = (minNbBits+1) << 16;
assert(tableLog < 16);
assert(accuracyLog < 31-tableLog); /* ensure enough room for renormalization double shift */
{ U32 const tableSize = 1 << tableLog;
U32 const deltaFromThreshold = threshold - (symbolTT[symbolValue].deltaNbBits + tableSize);
U32 const normalizedDeltaFromThreshold = (deltaFromThreshold << accuracyLog) >> tableLog; /* linear interpolation (very approximate) */
U32 const bitMultiplier = 1 << accuracyLog;
assert(symbolTT[symbolValue].deltaNbBits + tableSize <= threshold);
assert(normalizedDeltaFromThreshold <= bitMultiplier);
return (minNbBits+1)*bitMultiplier - normalizedDeltaFromThreshold;
}
}
/* ====== Decompression ====== */
typedef struct {
U16 tableLog;
U16 fastMode;
} FSE_DTableHeader; /* sizeof U32 */
typedef struct
{
unsigned short newState;
unsigned char symbol;
unsigned char nbBits;
} FSE_decode_t; /* size == U32 */
MEM_STATIC void FSE_initDState(FSE_DState_t* DStatePtr, BIT_DStream_t* bitD, const FSE_DTable* dt)
{
const void* ptr = dt;
const FSE_DTableHeader* const DTableH = (const FSE_DTableHeader*)ptr;
DStatePtr->state = BIT_readBits(bitD, DTableH->tableLog);
BIT_reloadDStream(bitD);
DStatePtr->table = dt + 1;
}
MEM_STATIC BYTE FSE_peekSymbol(const FSE_DState_t* DStatePtr)
{
FSE_decode_t const DInfo = ((const FSE_decode_t*)(DStatePtr->table))[DStatePtr->state];
return DInfo.symbol;
}
MEM_STATIC void FSE_updateState(FSE_DState_t* DStatePtr, BIT_DStream_t* bitD)
{
FSE_decode_t const DInfo = ((const FSE_decode_t*)(DStatePtr->table))[DStatePtr->state];
U32 const nbBits = DInfo.nbBits;
size_t const lowBits = BIT_readBits(bitD, nbBits);
DStatePtr->state = DInfo.newState + lowBits;
}
MEM_STATIC BYTE FSE_decodeSymbol(FSE_DState_t* DStatePtr, BIT_DStream_t* bitD)
{
FSE_decode_t const DInfo = ((const FSE_decode_t*)(DStatePtr->table))[DStatePtr->state];
U32 const nbBits = DInfo.nbBits;
BYTE const symbol = DInfo.symbol;
size_t const lowBits = BIT_readBits(bitD, nbBits);
DStatePtr->state = DInfo.newState + lowBits;
return symbol;
}
/*! FSE_decodeSymbolFast() :
unsafe, only works if no symbol has a probability > 50% */
MEM_STATIC BYTE FSE_decodeSymbolFast(FSE_DState_t* DStatePtr, BIT_DStream_t* bitD)
{
FSE_decode_t const DInfo = ((const FSE_decode_t*)(DStatePtr->table))[DStatePtr->state];
U32 const nbBits = DInfo.nbBits;
BYTE const symbol = DInfo.symbol;
size_t const lowBits = BIT_readBitsFast(bitD, nbBits);
DStatePtr->state = DInfo.newState + lowBits;
return symbol;
}
MEM_STATIC unsigned FSE_endOfDState(const FSE_DState_t* DStatePtr)
{
return DStatePtr->state == 0;
}
#ifndef FSE_COMMONDEFS_ONLY
/* **************************************************************
* Tuning parameters
****************************************************************/
/*!MEMORY_USAGE :
* Memory usage formula : N->2^N Bytes (examples : 10 -> 1KB; 12 -> 4KB ; 16 -> 64KB; 20 -> 1MB; etc.)
* Increasing memory usage improves compression ratio
* Reduced memory usage can improve speed, due to cache effect
* Recommended max value is 14, for 16KB, which nicely fits into Intel x86 L1 cache */
#ifndef FSE_MAX_MEMORY_USAGE
# define FSE_MAX_MEMORY_USAGE 14
#endif
#ifndef FSE_DEFAULT_MEMORY_USAGE
# define FSE_DEFAULT_MEMORY_USAGE 13
#endif
#if (FSE_DEFAULT_MEMORY_USAGE > FSE_MAX_MEMORY_USAGE)
# error "FSE_DEFAULT_MEMORY_USAGE must be <= FSE_MAX_MEMORY_USAGE"
#endif
/*!FSE_MAX_SYMBOL_VALUE :
* Maximum symbol value authorized.
* Required for proper stack allocation */
#ifndef FSE_MAX_SYMBOL_VALUE
# define FSE_MAX_SYMBOL_VALUE 255
#endif
/* **************************************************************
* template functions type & suffix
****************************************************************/
#define FSE_FUNCTION_TYPE BYTE
#define FSE_FUNCTION_EXTENSION
#define FSE_DECODE_TYPE FSE_decode_t
#endif /* !FSE_COMMONDEFS_ONLY */
/* ***************************************************************
* Constants
*****************************************************************/
#define FSE_MAX_TABLELOG (FSE_MAX_MEMORY_USAGE-2)
#define FSE_MAX_TABLESIZE (1U<<FSE_MAX_TABLELOG)
#define FSE_MAXTABLESIZE_MASK (FSE_MAX_TABLESIZE-1)
#define FSE_DEFAULT_TABLELOG (FSE_DEFAULT_MEMORY_USAGE-2)
#define FSE_MIN_TABLELOG 5
#define FSE_TABLELOG_ABSOLUTE_MAX 15
#if FSE_MAX_TABLELOG > FSE_TABLELOG_ABSOLUTE_MAX
# error "FSE_MAX_TABLELOG > FSE_TABLELOG_ABSOLUTE_MAX is not supported"
#endif
#define FSE_TABLESTEP(tableSize) (((tableSize)>>1) + ((tableSize)>>3) + 3)
#endif /* FSE_STATIC_LINKING_ONLY */
#endif /* USE_EXTERNAL_ZSTD */

628
vendor/github.com/DataDog/zstd/fse_compress.c generated vendored
View File

@@ -1,628 +0,0 @@
#ifndef USE_EXTERNAL_ZSTD
/* ******************************************************************
* FSE : Finite State Entropy encoder
* Copyright (c) Meta Platforms, Inc. and affiliates.
*
* You can contact the author at :
* - FSE source repository : https://github.com/Cyan4973/FiniteStateEntropy
* - Public forum : https://groups.google.com/forum/#!forum/lz4c
*
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
* You may select, at your option, one of the above-listed licenses.
****************************************************************** */
/* **************************************************************
* Includes
****************************************************************/
#include "compiler.h"
#include "mem.h" /* U32, U16, etc. */
#include "debug.h" /* assert, DEBUGLOG */
#include "hist.h" /* HIST_count_wksp */
#include "bitstream.h"
#define FSE_STATIC_LINKING_ONLY
#include "fse.h"
#include "error_private.h"
#define ZSTD_DEPS_NEED_MALLOC
#define ZSTD_DEPS_NEED_MATH64
#include "zstd_deps.h" /* ZSTD_memset */
#include "bits.h" /* ZSTD_highbit32 */
/* **************************************************************
* Error Management
****************************************************************/
#define FSE_isError ERR_isError
/* **************************************************************
* Templates
****************************************************************/
/*
designed to be included
for type-specific functions (template emulation in C)
Objective is to write these functions only once, for improved maintenance
*/
/* safety checks */
#ifndef FSE_FUNCTION_EXTENSION
# error "FSE_FUNCTION_EXTENSION must be defined"
#endif
#ifndef FSE_FUNCTION_TYPE
# error "FSE_FUNCTION_TYPE must be defined"
#endif
/* Function names */
#define FSE_CAT(X,Y) X##Y
#define FSE_FUNCTION_NAME(X,Y) FSE_CAT(X,Y)
#define FSE_TYPE_NAME(X,Y) FSE_CAT(X,Y)
/* Function templates */
/* FSE_buildCTable_wksp() :
* Same as FSE_buildCTable(), but using an externally allocated scratch buffer (`workSpace`).
* wkspSize should be sized to handle worst case situation, which is `1<<max_tableLog * sizeof(FSE_FUNCTION_TYPE)`
* workSpace must also be properly aligned with FSE_FUNCTION_TYPE requirements
*/
size_t FSE_buildCTable_wksp(FSE_CTable* ct,
const short* normalizedCounter, unsigned maxSymbolValue, unsigned tableLog,
void* workSpace, size_t wkspSize)
{
U32 const tableSize = 1 << tableLog;
U32 const tableMask = tableSize - 1;
void* const ptr = ct;
U16* const tableU16 = ( (U16*) ptr) + 2;
void* const FSCT = ((U32*)ptr) + 1 /* header */ + (tableLog ? tableSize>>1 : 1) ;
FSE_symbolCompressionTransform* const symbolTT = (FSE_symbolCompressionTransform*) (FSCT);
U32 const step = FSE_TABLESTEP(tableSize);
U32 const maxSV1 = maxSymbolValue+1;
U16* cumul = (U16*)workSpace; /* size = maxSV1 */
FSE_FUNCTION_TYPE* const tableSymbol = (FSE_FUNCTION_TYPE*)(cumul + (maxSV1+1)); /* size = tableSize */
U32 highThreshold = tableSize-1;
assert(((size_t)workSpace & 1) == 0); /* Must be 2 bytes-aligned */
if (FSE_BUILD_CTABLE_WORKSPACE_SIZE(maxSymbolValue, tableLog) > wkspSize) return ERROR(tableLog_tooLarge);
/* CTable header */
tableU16[-2] = (U16) tableLog;
tableU16[-1] = (U16) maxSymbolValue;
assert(tableLog < 16); /* required for threshold strategy to work */
/* For explanations on how to distribute symbol values over the table :
* https://fastcompression.blogspot.fr/2014/02/fse-distributing-symbol-values.html */
#ifdef __clang_analyzer__
ZSTD_memset(tableSymbol, 0, sizeof(*tableSymbol) * tableSize); /* useless initialization, just to keep scan-build happy */
#endif
/* symbol start positions */
{ U32 u;
cumul[0] = 0;
for (u=1; u <= maxSV1; u++) {
if (normalizedCounter[u-1]==-1) { /* Low proba symbol */
cumul[u] = cumul[u-1] + 1;
tableSymbol[highThreshold--] = (FSE_FUNCTION_TYPE)(u-1);
} else {
assert(normalizedCounter[u-1] >= 0);
cumul[u] = cumul[u-1] + (U16)normalizedCounter[u-1];
assert(cumul[u] >= cumul[u-1]); /* no overflow */
} }
cumul[maxSV1] = (U16)(tableSize+1);
}
/* Spread symbols */
if (highThreshold == tableSize - 1) {
/* Case for no low prob count symbols. Lay down 8 bytes at a time
* to reduce branch misses since we are operating on a small block
*/
BYTE* const spread = tableSymbol + tableSize; /* size = tableSize + 8 (may write beyond tableSize) */
{ U64 const add = 0x0101010101010101ull;
size_t pos = 0;
U64 sv = 0;
U32 s;
for (s=0; s<maxSV1; ++s, sv += add) {
int i;
int const n = normalizedCounter[s];
MEM_write64(spread + pos, sv);
for (i = 8; i < n; i += 8) {
MEM_write64(spread + pos + i, sv);
}
assert(n>=0);
pos += (size_t)n;
}
}
/* Spread symbols across the table. Lack of lowprob symbols means that
* we don't need variable sized inner loop, so we can unroll the loop and
* reduce branch misses.
*/
{ size_t position = 0;
size_t s;
size_t const unroll = 2; /* Experimentally determined optimal unroll */
assert(tableSize % unroll == 0); /* FSE_MIN_TABLELOG is 5 */
for (s = 0; s < (size_t)tableSize; s += unroll) {
size_t u;
for (u = 0; u < unroll; ++u) {
size_t const uPosition = (position + (u * step)) & tableMask;
tableSymbol[uPosition] = spread[s + u];
}
position = (position + (unroll * step)) & tableMask;
}
assert(position == 0); /* Must have initialized all positions */
}
} else {
U32 position = 0;
U32 symbol;
for (symbol=0; symbol<maxSV1; symbol++) {
int nbOccurrences;
int const freq = normalizedCounter[symbol];
for (nbOccurrences=0; nbOccurrences<freq; nbOccurrences++) {
tableSymbol[position] = (FSE_FUNCTION_TYPE)symbol;
position = (position + step) & tableMask;
while (position > highThreshold)
position = (position + step) & tableMask; /* Low proba area */
} }
assert(position==0); /* Must have initialized all positions */
}
/* Build table */
{ U32 u; for (u=0; u<tableSize; u++) {
FSE_FUNCTION_TYPE s = tableSymbol[u]; /* note : static analyzer may not understand tableSymbol is properly initialized */
tableU16[cumul[s]++] = (U16) (tableSize+u); /* TableU16 : sorted by symbol order; gives next state value */
} }
/* Build Symbol Transformation Table */
{ unsigned total = 0;
unsigned s;
for (s=0; s<=maxSymbolValue; s++) {
switch (normalizedCounter[s])
{
case 0:
/* filling nonetheless, for compatibility with FSE_getMaxNbBits() */
symbolTT[s].deltaNbBits = ((tableLog+1) << 16) - (1<<tableLog);
break;
case -1:
case 1:
symbolTT[s].deltaNbBits = (tableLog << 16) - (1<<tableLog);
assert(total <= INT_MAX);
symbolTT[s].deltaFindState = (int)(total - 1);
total ++;
break;
default :
assert(normalizedCounter[s] > 1);
{ U32 const maxBitsOut = tableLog - ZSTD_highbit32 ((U32)normalizedCounter[s]-1);
U32 const minStatePlus = (U32)normalizedCounter[s] << maxBitsOut;
symbolTT[s].deltaNbBits = (maxBitsOut << 16) - minStatePlus;
symbolTT[s].deltaFindState = (int)(total - (unsigned)normalizedCounter[s]);
total += (unsigned)normalizedCounter[s];
} } } }
#if 0 /* debug : symbol costs */
DEBUGLOG(5, "\n --- table statistics : ");
{ U32 symbol;
for (symbol=0; symbol<=maxSymbolValue; symbol++) {
DEBUGLOG(5, "%3u: w=%3i, maxBits=%u, fracBits=%.2f",
symbol, normalizedCounter[symbol],
FSE_getMaxNbBits(symbolTT, symbol),
(double)FSE_bitCost(symbolTT, tableLog, symbol, 8) / 256);
} }
#endif
return 0;
}
#ifndef FSE_COMMONDEFS_ONLY
/*-**************************************************************
* FSE NCount encoding
****************************************************************/
size_t FSE_NCountWriteBound(unsigned maxSymbolValue, unsigned tableLog)
{
size_t const maxHeaderSize = (((maxSymbolValue+1) * tableLog
+ 4 /* bitCount initialized at 4 */
+ 2 /* first two symbols may use one additional bit each */) / 8)
+ 1 /* round up to whole nb bytes */
+ 2 /* additional two bytes for bitstream flush */;
return maxSymbolValue ? maxHeaderSize : FSE_NCOUNTBOUND; /* maxSymbolValue==0 ? use default */
}
static size_t
FSE_writeNCount_generic (void* header, size_t headerBufferSize,
const short* normalizedCounter, unsigned maxSymbolValue, unsigned tableLog,
unsigned writeIsSafe)
{
BYTE* const ostart = (BYTE*) header;
BYTE* out = ostart;
BYTE* const oend = ostart + headerBufferSize;
int nbBits;
const int tableSize = 1 << tableLog;
int remaining;
int threshold;
U32 bitStream = 0;
int bitCount = 0;
unsigned symbol = 0;
unsigned const alphabetSize = maxSymbolValue + 1;
int previousIs0 = 0;
/* Table Size */
bitStream += (tableLog-FSE_MIN_TABLELOG) << bitCount;
bitCount += 4;
/* Init */
remaining = tableSize+1; /* +1 for extra accuracy */
threshold = tableSize;
nbBits = (int)tableLog+1;
while ((symbol < alphabetSize) && (remaining>1)) { /* stops at 1 */
if (previousIs0) {
unsigned start = symbol;
while ((symbol < alphabetSize) && !normalizedCounter[symbol]) symbol++;
if (symbol == alphabetSize) break; /* incorrect distribution */
while (symbol >= start+24) {
start+=24;
bitStream += 0xFFFFU << bitCount;
if ((!writeIsSafe) && (out > oend-2))
return ERROR(dstSize_tooSmall); /* Buffer overflow */
out[0] = (BYTE) bitStream;
out[1] = (BYTE)(bitStream>>8);
out+=2;
bitStream>>=16;
}
while (symbol >= start+3) {
start+=3;
bitStream += 3U << bitCount;
bitCount += 2;
}
bitStream += (symbol-start) << bitCount;
bitCount += 2;
if (bitCount>16) {
if ((!writeIsSafe) && (out > oend - 2))
return ERROR(dstSize_tooSmall); /* Buffer overflow */
out[0] = (BYTE)bitStream;
out[1] = (BYTE)(bitStream>>8);
out += 2;
bitStream >>= 16;
bitCount -= 16;
} }
{ int count = normalizedCounter[symbol++];
int const max = (2*threshold-1) - remaining;
remaining -= count < 0 ? -count : count;
count++; /* +1 for extra accuracy */
if (count>=threshold)
count += max; /* [0..max[ [max..threshold[ (...) [threshold+max 2*threshold[ */
bitStream += (U32)count << bitCount;
bitCount += nbBits;
bitCount -= (count<max);
previousIs0 = (count==1);
if (remaining<1) return ERROR(GENERIC);
while (remaining<threshold) { nbBits--; threshold>>=1; }
}
if (bitCount>16) {
if ((!writeIsSafe) && (out > oend - 2))
return ERROR(dstSize_tooSmall); /* Buffer overflow */
out[0] = (BYTE)bitStream;
out[1] = (BYTE)(bitStream>>8);
out += 2;
bitStream >>= 16;
bitCount -= 16;
} }
if (remaining != 1)
return ERROR(GENERIC); /* incorrect normalized distribution */
assert(symbol <= alphabetSize);
/* flush remaining bitStream */
if ((!writeIsSafe) && (out > oend - 2))
return ERROR(dstSize_tooSmall); /* Buffer overflow */
out[0] = (BYTE)bitStream;
out[1] = (BYTE)(bitStream>>8);
out+= (bitCount+7) /8;
assert(out >= ostart);
return (size_t)(out-ostart);
}
size_t FSE_writeNCount (void* buffer, size_t bufferSize,
const short* normalizedCounter, unsigned maxSymbolValue, unsigned tableLog)
{
if (tableLog > FSE_MAX_TABLELOG) return ERROR(tableLog_tooLarge); /* Unsupported */
if (tableLog < FSE_MIN_TABLELOG) return ERROR(GENERIC); /* Unsupported */
if (bufferSize < FSE_NCountWriteBound(maxSymbolValue, tableLog))
return FSE_writeNCount_generic(buffer, bufferSize, normalizedCounter, maxSymbolValue, tableLog, 0);
return FSE_writeNCount_generic(buffer, bufferSize, normalizedCounter, maxSymbolValue, tableLog, 1 /* write in buffer is safe */);
}
/*-**************************************************************
* FSE Compression Code
****************************************************************/
/* provides the minimum logSize to safely represent a distribution */
static unsigned FSE_minTableLog(size_t srcSize, unsigned maxSymbolValue)
{
U32 minBitsSrc = ZSTD_highbit32((U32)(srcSize)) + 1;
U32 minBitsSymbols = ZSTD_highbit32(maxSymbolValue) + 2;
U32 minBits = minBitsSrc < minBitsSymbols ? minBitsSrc : minBitsSymbols;
assert(srcSize > 1); /* Not supported, RLE should be used instead */
return minBits;
}
unsigned FSE_optimalTableLog_internal(unsigned maxTableLog, size_t srcSize, unsigned maxSymbolValue, unsigned minus)
{
U32 maxBitsSrc = ZSTD_highbit32((U32)(srcSize - 1)) - minus;
U32 tableLog = maxTableLog;
U32 minBits = FSE_minTableLog(srcSize, maxSymbolValue);
assert(srcSize > 1); /* Not supported, RLE should be used instead */
if (tableLog==0) tableLog = FSE_DEFAULT_TABLELOG;
if (maxBitsSrc < tableLog) tableLog = maxBitsSrc; /* Accuracy can be reduced */
if (minBits > tableLog) tableLog = minBits; /* Need a minimum to safely represent all symbol values */
if (tableLog < FSE_MIN_TABLELOG) tableLog = FSE_MIN_TABLELOG;
if (tableLog > FSE_MAX_TABLELOG) tableLog = FSE_MAX_TABLELOG;
return tableLog;
}
unsigned FSE_optimalTableLog(unsigned maxTableLog, size_t srcSize, unsigned maxSymbolValue)
{
return FSE_optimalTableLog_internal(maxTableLog, srcSize, maxSymbolValue, 2);
}
/* Secondary normalization method.
To be used when primary method fails. */
static size_t FSE_normalizeM2(short* norm, U32 tableLog, const unsigned* count, size_t total, U32 maxSymbolValue, short lowProbCount)
{
short const NOT_YET_ASSIGNED = -2;
U32 s;
U32 distributed = 0;
U32 ToDistribute;
/* Init */
U32 const lowThreshold = (U32)(total >> tableLog);
U32 lowOne = (U32)((total * 3) >> (tableLog + 1));
for (s=0; s<=maxSymbolValue; s++) {
if (count[s] == 0) {
norm[s]=0;
continue;
}
if (count[s] <= lowThreshold) {
norm[s] = lowProbCount;
distributed++;
total -= count[s];
continue;
}
if (count[s] <= lowOne) {
norm[s] = 1;
distributed++;
total -= count[s];
continue;
}
norm[s]=NOT_YET_ASSIGNED;
}
ToDistribute = (1 << tableLog) - distributed;
if (ToDistribute == 0)
return 0;
if ((total / ToDistribute) > lowOne) {
/* risk of rounding to zero */
lowOne = (U32)((total * 3) / (ToDistribute * 2));
for (s=0; s<=maxSymbolValue; s++) {
if ((norm[s] == NOT_YET_ASSIGNED) && (count[s] <= lowOne)) {
norm[s] = 1;
distributed++;
total -= count[s];
continue;
} }
ToDistribute = (1 << tableLog) - distributed;
}
if (distributed == maxSymbolValue+1) {
/* all values are pretty poor;
probably incompressible data (should have already been detected);
find max, then give all remaining points to max */
U32 maxV = 0, maxC = 0;
for (s=0; s<=maxSymbolValue; s++)
if (count[s] > maxC) { maxV=s; maxC=count[s]; }
norm[maxV] += (short)ToDistribute;
return 0;
}
if (total == 0) {
/* all of the symbols were low enough for the lowOne or lowThreshold */
for (s=0; ToDistribute > 0; s = (s+1)%(maxSymbolValue+1))
if (norm[s] > 0) { ToDistribute--; norm[s]++; }
return 0;
}
{ U64 const vStepLog = 62 - tableLog;
U64 const mid = (1ULL << (vStepLog-1)) - 1;
U64 const rStep = ZSTD_div64((((U64)1<<vStepLog) * ToDistribute) + mid, (U32)total); /* scale on remaining */
U64 tmpTotal = mid;
for (s=0; s<=maxSymbolValue; s++) {
if (norm[s]==NOT_YET_ASSIGNED) {
U64 const end = tmpTotal + (count[s] * rStep);
U32 const sStart = (U32)(tmpTotal >> vStepLog);
U32 const sEnd = (U32)(end >> vStepLog);
U32 const weight = sEnd - sStart;
if (weight < 1)
return ERROR(GENERIC);
norm[s] = (short)weight;
tmpTotal = end;
} } }
return 0;
}
size_t FSE_normalizeCount (short* normalizedCounter, unsigned tableLog,
const unsigned* count, size_t total,
unsigned maxSymbolValue, unsigned useLowProbCount)
{
/* Sanity checks */
if (tableLog==0) tableLog = FSE_DEFAULT_TABLELOG;
if (tableLog < FSE_MIN_TABLELOG) return ERROR(GENERIC); /* Unsupported size */
if (tableLog > FSE_MAX_TABLELOG) return ERROR(tableLog_tooLarge); /* Unsupported size */
if (tableLog < FSE_minTableLog(total, maxSymbolValue)) return ERROR(GENERIC); /* Too small tableLog, compression potentially impossible */
{ static U32 const rtbTable[] = { 0, 473195, 504333, 520860, 550000, 700000, 750000, 830000 };
short const lowProbCount = useLowProbCount ? -1 : 1;
U64 const scale = 62 - tableLog;
U64 const step = ZSTD_div64((U64)1<<62, (U32)total); /* <== here, one division ! */
U64 const vStep = 1ULL<<(scale-20);
int stillToDistribute = 1<<tableLog;
unsigned s;
unsigned largest=0;
short largestP=0;
U32 lowThreshold = (U32)(total >> tableLog);
for (s=0; s<=maxSymbolValue; s++) {
if (count[s] == total) return 0; /* rle special case */
if (count[s] == 0) { normalizedCounter[s]=0; continue; }
if (count[s] <= lowThreshold) {
normalizedCounter[s] = lowProbCount;
stillToDistribute--;
} else {
short proba = (short)((count[s]*step) >> scale);
if (proba<8) {
U64 restToBeat = vStep * rtbTable[proba];
proba += (count[s]*step) - ((U64)proba<<scale) > restToBeat;
}
if (proba > largestP) { largestP=proba; largest=s; }
normalizedCounter[s] = proba;
stillToDistribute -= proba;
} }
if (-stillToDistribute >= (normalizedCounter[largest] >> 1)) {
/* corner case, need another normalization method */
size_t const errorCode = FSE_normalizeM2(normalizedCounter, tableLog, count, total, maxSymbolValue, lowProbCount);
if (FSE_isError(errorCode)) return errorCode;
}
else normalizedCounter[largest] += (short)stillToDistribute;
}
#if 0
{ /* Print Table (debug) */
U32 s;
U32 nTotal = 0;
for (s=0; s<=maxSymbolValue; s++)
RAWLOG(2, "%3i: %4i \n", s, normalizedCounter[s]);
for (s=0; s<=maxSymbolValue; s++)
nTotal += abs(normalizedCounter[s]);
if (nTotal != (1U<<tableLog))
RAWLOG(2, "Warning !!! Total == %u != %u !!!", nTotal, 1U<<tableLog);
getchar();
}
#endif
return tableLog;
}
/* fake FSE_CTable, for rle input (always same symbol) */
size_t FSE_buildCTable_rle (FSE_CTable* ct, BYTE symbolValue)
{
void* ptr = ct;
U16* tableU16 = ( (U16*) ptr) + 2;
void* FSCTptr = (U32*)ptr + 2;
FSE_symbolCompressionTransform* symbolTT = (FSE_symbolCompressionTransform*) FSCTptr;
/* header */
tableU16[-2] = (U16) 0;
tableU16[-1] = (U16) symbolValue;
/* Build table */
tableU16[0] = 0;
tableU16[1] = 0; /* just in case */
/* Build Symbol Transformation Table */
symbolTT[symbolValue].deltaNbBits = 0;
symbolTT[symbolValue].deltaFindState = 0;
return 0;
}
static size_t FSE_compress_usingCTable_generic (void* dst, size_t dstSize,
const void* src, size_t srcSize,
const FSE_CTable* ct, const unsigned fast)
{
const BYTE* const istart = (const BYTE*) src;
const BYTE* const iend = istart + srcSize;
const BYTE* ip=iend;
BIT_CStream_t bitC;
FSE_CState_t CState1, CState2;
/* init */
if (srcSize <= 2) return 0;
{ size_t const initError = BIT_initCStream(&bitC, dst, dstSize);
if (FSE_isError(initError)) return 0; /* not enough space available to write a bitstream */ }
#define FSE_FLUSHBITS(s) (fast ? BIT_flushBitsFast(s) : BIT_flushBits(s))
if (srcSize & 1) {
FSE_initCState2(&CState1, ct, *--ip);
FSE_initCState2(&CState2, ct, *--ip);
FSE_encodeSymbol(&bitC, &CState1, *--ip);
FSE_FLUSHBITS(&bitC);
} else {
FSE_initCState2(&CState2, ct, *--ip);
FSE_initCState2(&CState1, ct, *--ip);
}
/* join to mod 4 */
srcSize -= 2;
if ((sizeof(bitC.bitContainer)*8 > FSE_MAX_TABLELOG*4+7 ) && (srcSize & 2)) { /* test bit 2 */
FSE_encodeSymbol(&bitC, &CState2, *--ip);
FSE_encodeSymbol(&bitC, &CState1, *--ip);
FSE_FLUSHBITS(&bitC);
}
/* 2 or 4 encoding per loop */
while ( ip>istart ) {
FSE_encodeSymbol(&bitC, &CState2, *--ip);
if (sizeof(bitC.bitContainer)*8 < FSE_MAX_TABLELOG*2+7 ) /* this test must be static */
FSE_FLUSHBITS(&bitC);
FSE_encodeSymbol(&bitC, &CState1, *--ip);
if (sizeof(bitC.bitContainer)*8 > FSE_MAX_TABLELOG*4+7 ) { /* this test must be static */
FSE_encodeSymbol(&bitC, &CState2, *--ip);
FSE_encodeSymbol(&bitC, &CState1, *--ip);
}
FSE_FLUSHBITS(&bitC);
}
FSE_flushCState(&bitC, &CState2);
FSE_flushCState(&bitC, &CState1);
return BIT_closeCStream(&bitC);
}
size_t FSE_compress_usingCTable (void* dst, size_t dstSize,
const void* src, size_t srcSize,
const FSE_CTable* ct)
{
unsigned const fast = (dstSize >= FSE_BLOCKBOUND(srcSize));
if (fast)
return FSE_compress_usingCTable_generic(dst, dstSize, src, srcSize, ct, 1);
else
return FSE_compress_usingCTable_generic(dst, dstSize, src, srcSize, ct, 0);
}
size_t FSE_compressBound(size_t size) { return FSE_COMPRESSBOUND(size); }
#endif /* FSE_COMMONDEFS_ONLY */
#endif /* USE_EXTERNAL_ZSTD */

318
vendor/github.com/DataDog/zstd/fse_decompress.c generated vendored
View File

@@ -1,318 +0,0 @@
#ifndef USE_EXTERNAL_ZSTD
/* ******************************************************************
* FSE : Finite State Entropy decoder
* Copyright (c) Meta Platforms, Inc. and affiliates.
*
* You can contact the author at :
* - FSE source repository : https://github.com/Cyan4973/FiniteStateEntropy
* - Public forum : https://groups.google.com/forum/#!forum/lz4c
*
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
* You may select, at your option, one of the above-listed licenses.
****************************************************************** */
/* **************************************************************
* Includes
****************************************************************/
#include "debug.h" /* assert */
#include "bitstream.h"
#include "compiler.h"
#define FSE_STATIC_LINKING_ONLY
#include "fse.h"
#include "error_private.h"
#include "zstd_deps.h" /* ZSTD_memcpy */
#include "bits.h" /* ZSTD_highbit32 */
/* **************************************************************
* Error Management
****************************************************************/
#define FSE_isError ERR_isError
#define FSE_STATIC_ASSERT(c) DEBUG_STATIC_ASSERT(c) /* use only *after* variable declarations */
/* **************************************************************
* Templates
****************************************************************/
/*
designed to be included
for type-specific functions (template emulation in C)
Objective is to write these functions only once, for improved maintenance
*/
/* safety checks */
#ifndef FSE_FUNCTION_EXTENSION
# error "FSE_FUNCTION_EXTENSION must be defined"
#endif
#ifndef FSE_FUNCTION_TYPE
# error "FSE_FUNCTION_TYPE must be defined"
#endif
/* Function names */
#define FSE_CAT(X,Y) X##Y
#define FSE_FUNCTION_NAME(X,Y) FSE_CAT(X,Y)
#define FSE_TYPE_NAME(X,Y) FSE_CAT(X,Y)
static size_t FSE_buildDTable_internal(FSE_DTable* dt, const short* normalizedCounter, unsigned maxSymbolValue, unsigned tableLog, void* workSpace, size_t wkspSize)
{
void* const tdPtr = dt+1; /* because *dt is unsigned, 32-bits aligned on 32-bits */
FSE_DECODE_TYPE* const tableDecode = (FSE_DECODE_TYPE*) (tdPtr);
U16* symbolNext = (U16*)workSpace;
BYTE* spread = (BYTE*)(symbolNext + maxSymbolValue + 1);
U32 const maxSV1 = maxSymbolValue + 1;
U32 const tableSize = 1 << tableLog;
U32 highThreshold = tableSize-1;
/* Sanity Checks */
if (FSE_BUILD_DTABLE_WKSP_SIZE(tableLog, maxSymbolValue) > wkspSize) return ERROR(maxSymbolValue_tooLarge);
if (maxSymbolValue > FSE_MAX_SYMBOL_VALUE) return ERROR(maxSymbolValue_tooLarge);
if (tableLog > FSE_MAX_TABLELOG) return ERROR(tableLog_tooLarge);
/* Init, lay down lowprob symbols */
{ FSE_DTableHeader DTableH;
DTableH.tableLog = (U16)tableLog;
DTableH.fastMode = 1;
{ S16 const largeLimit= (S16)(1 << (tableLog-1));
U32 s;
for (s=0; s<maxSV1; s++) {
if (normalizedCounter[s]==-1) {
tableDecode[highThreshold--].symbol = (FSE_FUNCTION_TYPE)s;
symbolNext[s] = 1;
} else {
if (normalizedCounter[s] >= largeLimit) DTableH.fastMode=0;
symbolNext[s] = (U16)normalizedCounter[s];
} } }
ZSTD_memcpy(dt, &DTableH, sizeof(DTableH));
}
/* Spread symbols */
if (highThreshold == tableSize - 1) {
size_t const tableMask = tableSize-1;
size_t const step = FSE_TABLESTEP(tableSize);
/* First lay down the symbols in order.
* We use a uint64_t to lay down 8 bytes at a time. This reduces branch
* misses since small blocks generally have small table logs, so nearly
* all symbols have counts <= 8. We ensure we have 8 bytes at the end of
* our buffer to handle the over-write.
*/
{ U64 const add = 0x0101010101010101ull;
size_t pos = 0;
U64 sv = 0;
U32 s;
for (s=0; s<maxSV1; ++s, sv += add) {
int i;
int const n = normalizedCounter[s];
MEM_write64(spread + pos, sv);
for (i = 8; i < n; i += 8) {
MEM_write64(spread + pos + i, sv);
}
pos += (size_t)n;
} }
/* Now we spread those positions across the table.
* The benefit of doing it in two stages is that we avoid the
* variable size inner loop, which caused lots of branch misses.
* Now we can run through all the positions without any branch misses.
* We unroll the loop twice, since that is what empirically worked best.
*/
{
size_t position = 0;
size_t s;
size_t const unroll = 2;
assert(tableSize % unroll == 0); /* FSE_MIN_TABLELOG is 5 */
for (s = 0; s < (size_t)tableSize; s += unroll) {
size_t u;
for (u = 0; u < unroll; ++u) {
size_t const uPosition = (position + (u * step)) & tableMask;
tableDecode[uPosition].symbol = spread[s + u];
}
position = (position + (unroll * step)) & tableMask;
}
assert(position == 0);
}
} else {
U32 const tableMask = tableSize-1;
U32 const step = FSE_TABLESTEP(tableSize);
U32 s, position = 0;
for (s=0; s<maxSV1; s++) {
int i;
for (i=0; i<normalizedCounter[s]; i++) {
tableDecode[position].symbol = (FSE_FUNCTION_TYPE)s;
position = (position + step) & tableMask;
while (position > highThreshold) position = (position + step) & tableMask; /* lowprob area */
} }
if (position!=0) return ERROR(GENERIC); /* position must reach all cells once, otherwise normalizedCounter is incorrect */
}
/* Build Decoding table */
{ U32 u;
for (u=0; u<tableSize; u++) {
FSE_FUNCTION_TYPE const symbol = (FSE_FUNCTION_TYPE)(tableDecode[u].symbol);
U32 const nextState = symbolNext[symbol]++;
tableDecode[u].nbBits = (BYTE) (tableLog - ZSTD_highbit32(nextState) );
tableDecode[u].newState = (U16) ( (nextState << tableDecode[u].nbBits) - tableSize);
} }
return 0;
}
size_t FSE_buildDTable_wksp(FSE_DTable* dt, const short* normalizedCounter, unsigned maxSymbolValue, unsigned tableLog, void* workSpace, size_t wkspSize)
{
return FSE_buildDTable_internal(dt, normalizedCounter, maxSymbolValue, tableLog, workSpace, wkspSize);
}
#ifndef FSE_COMMONDEFS_ONLY
/*-*******************************************************
* Decompression (Byte symbols)
*********************************************************/
FORCE_INLINE_TEMPLATE size_t FSE_decompress_usingDTable_generic(
void* dst, size_t maxDstSize,
const void* cSrc, size_t cSrcSize,
const FSE_DTable* dt, const unsigned fast)
{
BYTE* const ostart = (BYTE*) dst;
BYTE* op = ostart;
BYTE* const omax = op + maxDstSize;
BYTE* const olimit = omax-3;
BIT_DStream_t bitD;
FSE_DState_t state1;
FSE_DState_t state2;
/* Init */
CHECK_F(BIT_initDStream(&bitD, cSrc, cSrcSize));
FSE_initDState(&state1, &bitD, dt);
FSE_initDState(&state2, &bitD, dt);
RETURN_ERROR_IF(BIT_reloadDStream(&bitD)==BIT_DStream_overflow, corruption_detected, "");
#define FSE_GETSYMBOL(statePtr) fast ? FSE_decodeSymbolFast(statePtr, &bitD) : FSE_decodeSymbol(statePtr, &bitD)
/* 4 symbols per loop */
for ( ; (BIT_reloadDStream(&bitD)==BIT_DStream_unfinished) & (op<olimit) ; op+=4) {
op[0] = FSE_GETSYMBOL(&state1);
if (FSE_MAX_TABLELOG*2+7 > sizeof(bitD.bitContainer)*8) /* This test must be static */
BIT_reloadDStream(&bitD);
op[1] = FSE_GETSYMBOL(&state2);
if (FSE_MAX_TABLELOG*4+7 > sizeof(bitD.bitContainer)*8) /* This test must be static */
{ if (BIT_reloadDStream(&bitD) > BIT_DStream_unfinished) { op+=2; break; } }
op[2] = FSE_GETSYMBOL(&state1);
if (FSE_MAX_TABLELOG*2+7 > sizeof(bitD.bitContainer)*8) /* This test must be static */
BIT_reloadDStream(&bitD);
op[3] = FSE_GETSYMBOL(&state2);
}
/* tail */
/* note : BIT_reloadDStream(&bitD) >= FSE_DStream_partiallyFilled; Ends at exactly BIT_DStream_completed */
while (1) {
if (op>(omax-2)) return ERROR(dstSize_tooSmall);
*op++ = FSE_GETSYMBOL(&state1);
if (BIT_reloadDStream(&bitD)==BIT_DStream_overflow) {
*op++ = FSE_GETSYMBOL(&state2);
break;
}
if (op>(omax-2)) return ERROR(dstSize_tooSmall);
*op++ = FSE_GETSYMBOL(&state2);
if (BIT_reloadDStream(&bitD)==BIT_DStream_overflow) {
*op++ = FSE_GETSYMBOL(&state1);
break;
} }
assert(op >= ostart);
return (size_t)(op-ostart);
}
typedef struct {
short ncount[FSE_MAX_SYMBOL_VALUE + 1];
} FSE_DecompressWksp;
FORCE_INLINE_TEMPLATE size_t FSE_decompress_wksp_body(
void* dst, size_t dstCapacity,
const void* cSrc, size_t cSrcSize,
unsigned maxLog, void* workSpace, size_t wkspSize,
int bmi2)
{
const BYTE* const istart = (const BYTE*)cSrc;
const BYTE* ip = istart;
unsigned tableLog;
unsigned maxSymbolValue = FSE_MAX_SYMBOL_VALUE;
FSE_DecompressWksp* const wksp = (FSE_DecompressWksp*)workSpace;
size_t const dtablePos = sizeof(FSE_DecompressWksp) / sizeof(FSE_DTable);
FSE_DTable* const dtable = (FSE_DTable*)workSpace + dtablePos;
FSE_STATIC_ASSERT((FSE_MAX_SYMBOL_VALUE + 1) % 2 == 0);
if (wkspSize < sizeof(*wksp)) return ERROR(GENERIC);
/* correct offset to dtable depends on this property */
FSE_STATIC_ASSERT(sizeof(FSE_DecompressWksp) % sizeof(FSE_DTable) == 0);
/* normal FSE decoding mode */
{ size_t const NCountLength =
FSE_readNCount_bmi2(wksp->ncount, &maxSymbolValue, &tableLog, istart, cSrcSize, bmi2);
if (FSE_isError(NCountLength)) return NCountLength;
if (tableLog > maxLog) return ERROR(tableLog_tooLarge);
assert(NCountLength <= cSrcSize);
ip += NCountLength;
cSrcSize -= NCountLength;
}
if (FSE_DECOMPRESS_WKSP_SIZE(tableLog, maxSymbolValue) > wkspSize) return ERROR(tableLog_tooLarge);
assert(sizeof(*wksp) + FSE_DTABLE_SIZE(tableLog) <= wkspSize);
workSpace = (BYTE*)workSpace + sizeof(*wksp) + FSE_DTABLE_SIZE(tableLog);
wkspSize -= sizeof(*wksp) + FSE_DTABLE_SIZE(tableLog);
CHECK_F( FSE_buildDTable_internal(dtable, wksp->ncount, maxSymbolValue, tableLog, workSpace, wkspSize) );
{
const void* ptr = dtable;
const FSE_DTableHeader* DTableH = (const FSE_DTableHeader*)ptr;
const U32 fastMode = DTableH->fastMode;
/* select fast mode (static) */
if (fastMode) return FSE_decompress_usingDTable_generic(dst, dstCapacity, ip, cSrcSize, dtable, 1);
return FSE_decompress_usingDTable_generic(dst, dstCapacity, ip, cSrcSize, dtable, 0);
}
}
/* Avoids the FORCE_INLINE of the _body() function. */
static size_t FSE_decompress_wksp_body_default(void* dst, size_t dstCapacity, const void* cSrc, size_t cSrcSize, unsigned maxLog, void* workSpace, size_t wkspSize)
{
return FSE_decompress_wksp_body(dst, dstCapacity, cSrc, cSrcSize, maxLog, workSpace, wkspSize, 0);
}
#if DYNAMIC_BMI2
BMI2_TARGET_ATTRIBUTE static size_t FSE_decompress_wksp_body_bmi2(void* dst, size_t dstCapacity, const void* cSrc, size_t cSrcSize, unsigned maxLog, void* workSpace, size_t wkspSize)
{
return FSE_decompress_wksp_body(dst, dstCapacity, cSrc, cSrcSize, maxLog, workSpace, wkspSize, 1);
}
#endif
size_t FSE_decompress_wksp_bmi2(void* dst, size_t dstCapacity, const void* cSrc, size_t cSrcSize, unsigned maxLog, void* workSpace, size_t wkspSize, int bmi2)
{
#if DYNAMIC_BMI2
if (bmi2) {
return FSE_decompress_wksp_body_bmi2(dst, dstCapacity, cSrc, cSrcSize, maxLog, workSpace, wkspSize);
}
#endif
(void)bmi2;
return FSE_decompress_wksp_body_default(dst, dstCapacity, cSrc, cSrcSize, maxLog, workSpace, wkspSize);
}
#endif /* FSE_COMMONDEFS_ONLY */
#endif /* USE_EXTERNAL_ZSTD */

194
vendor/github.com/DataDog/zstd/hist.c generated vendored
View File

@@ -1,194 +0,0 @@
#ifndef USE_EXTERNAL_ZSTD
/* ******************************************************************
* hist : Histogram functions
* part of Finite State Entropy project
* Copyright (c) Meta Platforms, Inc. and affiliates.
*
* You can contact the author at :
* - FSE source repository : https://github.com/Cyan4973/FiniteStateEntropy
* - Public forum : https://groups.google.com/forum/#!forum/lz4c
*
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
* You may select, at your option, one of the above-listed licenses.
****************************************************************** */
/* --- dependencies --- */
#include "mem.h" /* U32, BYTE, etc. */
#include "debug.h" /* assert, DEBUGLOG */
#include "error_private.h" /* ERROR */
#include "hist.h"
/* --- Error management --- */
unsigned HIST_isError(size_t code) { return ERR_isError(code); }
/*-**************************************************************
* Histogram functions
****************************************************************/
void HIST_add(unsigned* count, const void* src, size_t srcSize)
{
const BYTE* ip = (const BYTE*)src;
const BYTE* const end = ip + srcSize;
while (ip<end) {
count[*ip++]++;
}
}
unsigned HIST_count_simple(unsigned* count, unsigned* maxSymbolValuePtr,
const void* src, size_t srcSize)
{
const BYTE* ip = (const BYTE*)src;
const BYTE* const end = ip + srcSize;
unsigned maxSymbolValue = *maxSymbolValuePtr;
unsigned largestCount=0;
ZSTD_memset(count, 0, (maxSymbolValue+1) * sizeof(*count));
if (srcSize==0) { *maxSymbolValuePtr = 0; return 0; }
while (ip<end) {
assert(*ip <= maxSymbolValue);
count[*ip++]++;
}
while (!count[maxSymbolValue]) maxSymbolValue--;
*maxSymbolValuePtr = maxSymbolValue;
{ U32 s;
for (s=0; s<=maxSymbolValue; s++)
if (count[s] > largestCount) largestCount = count[s];
}
return largestCount;
}
typedef enum { trustInput, checkMaxSymbolValue } HIST_checkInput_e;
/* HIST_count_parallel_wksp() :
* store histogram into 4 intermediate tables, recombined at the end.
* this design makes better use of OoO cpus,
* and is noticeably faster when some values are heavily repeated.
* But it needs some additional workspace for intermediate tables.
* `workSpace` must be a U32 table of size >= HIST_WKSP_SIZE_U32.
* @return : largest histogram frequency,
* or an error code (notably when histogram's alphabet is larger than *maxSymbolValuePtr) */
static size_t HIST_count_parallel_wksp(
unsigned* count, unsigned* maxSymbolValuePtr,
const void* source, size_t sourceSize,
HIST_checkInput_e check,
U32* const workSpace)
{
const BYTE* ip = (const BYTE*)source;
const BYTE* const iend = ip+sourceSize;
size_t const countSize = (*maxSymbolValuePtr + 1) * sizeof(*count);
unsigned max=0;
U32* const Counting1 = workSpace;
U32* const Counting2 = Counting1 + 256;
U32* const Counting3 = Counting2 + 256;
U32* const Counting4 = Counting3 + 256;
/* safety checks */
assert(*maxSymbolValuePtr <= 255);
if (!sourceSize) {
ZSTD_memset(count, 0, countSize);
*maxSymbolValuePtr = 0;
return 0;
}
ZSTD_memset(workSpace, 0, 4*256*sizeof(unsigned));
/* by stripes of 16 bytes */
{ U32 cached = MEM_read32(ip); ip += 4;
while (ip < iend-15) {
U32 c = cached; cached = MEM_read32(ip); ip += 4;
Counting1[(BYTE) c ]++;
Counting2[(BYTE)(c>>8) ]++;
Counting3[(BYTE)(c>>16)]++;
Counting4[ c>>24 ]++;
c = cached; cached = MEM_read32(ip); ip += 4;
Counting1[(BYTE) c ]++;
Counting2[(BYTE)(c>>8) ]++;
Counting3[(BYTE)(c>>16)]++;
Counting4[ c>>24 ]++;
c = cached; cached = MEM_read32(ip); ip += 4;
Counting1[(BYTE) c ]++;
Counting2[(BYTE)(c>>8) ]++;
Counting3[(BYTE)(c>>16)]++;
Counting4[ c>>24 ]++;
c = cached; cached = MEM_read32(ip); ip += 4;
Counting1[(BYTE) c ]++;
Counting2[(BYTE)(c>>8) ]++;
Counting3[(BYTE)(c>>16)]++;
Counting4[ c>>24 ]++;
}
ip-=4;
}
/* finish last symbols */
while (ip<iend) Counting1[*ip++]++;
{ U32 s;
for (s=0; s<256; s++) {
Counting1[s] += Counting2[s] + Counting3[s] + Counting4[s];
if (Counting1[s] > max) max = Counting1[s];
} }
{ unsigned maxSymbolValue = 255;
while (!Counting1[maxSymbolValue]) maxSymbolValue--;
if (check && maxSymbolValue > *maxSymbolValuePtr) return ERROR(maxSymbolValue_tooSmall);
*maxSymbolValuePtr = maxSymbolValue;
ZSTD_memmove(count, Counting1, countSize); /* in case count & Counting1 are overlapping */
}
return (size_t)max;
}
/* HIST_countFast_wksp() :
* Same as HIST_countFast(), but using an externally provided scratch buffer.
* `workSpace` is a writable buffer which must be 4-bytes aligned,
* `workSpaceSize` must be >= HIST_WKSP_SIZE
*/
size_t HIST_countFast_wksp(unsigned* count, unsigned* maxSymbolValuePtr,
const void* source, size_t sourceSize,
void* workSpace, size_t workSpaceSize)
{
if (sourceSize < 1500) /* heuristic threshold */
return HIST_count_simple(count, maxSymbolValuePtr, source, sourceSize);
if ((size_t)workSpace & 3) return ERROR(GENERIC); /* must be aligned on 4-bytes boundaries */
if (workSpaceSize < HIST_WKSP_SIZE) return ERROR(workSpace_tooSmall);
return HIST_count_parallel_wksp(count, maxSymbolValuePtr, source, sourceSize, trustInput, (U32*)workSpace);
}
/* HIST_count_wksp() :
* Same as HIST_count(), but using an externally provided scratch buffer.
* `workSpace` size must be table of >= HIST_WKSP_SIZE_U32 unsigned */
size_t HIST_count_wksp(unsigned* count, unsigned* maxSymbolValuePtr,
const void* source, size_t sourceSize,
void* workSpace, size_t workSpaceSize)
{
if ((size_t)workSpace & 3) return ERROR(GENERIC); /* must be aligned on 4-bytes boundaries */
if (workSpaceSize < HIST_WKSP_SIZE) return ERROR(workSpace_tooSmall);
if (*maxSymbolValuePtr < 255)
return HIST_count_parallel_wksp(count, maxSymbolValuePtr, source, sourceSize, checkMaxSymbolValue, (U32*)workSpace);
*maxSymbolValuePtr = 255;
return HIST_countFast_wksp(count, maxSymbolValuePtr, source, sourceSize, workSpace, workSpaceSize);
}
#ifndef ZSTD_NO_UNUSED_FUNCTIONS
/* fast variant (unsafe : won't check if src contains values beyond count[] limit) */
size_t HIST_countFast(unsigned* count, unsigned* maxSymbolValuePtr,
const void* source, size_t sourceSize)
{
unsigned tmpCounters[HIST_WKSP_SIZE_U32];
return HIST_countFast_wksp(count, maxSymbolValuePtr, source, sourceSize, tmpCounters, sizeof(tmpCounters));
}
size_t HIST_count(unsigned* count, unsigned* maxSymbolValuePtr,
const void* src, size_t srcSize)
{
unsigned tmpCounters[HIST_WKSP_SIZE_U32];
return HIST_count_wksp(count, maxSymbolValuePtr, src, srcSize, tmpCounters, sizeof(tmpCounters));
}
#endif
#endif /* USE_EXTERNAL_ZSTD */

85
vendor/github.com/DataDog/zstd/hist.h generated vendored
View File

@@ -1,85 +0,0 @@
#ifndef USE_EXTERNAL_ZSTD
/* ******************************************************************
* hist : Histogram functions
* part of Finite State Entropy project
* Copyright (c) Meta Platforms, Inc. and affiliates.
*
* You can contact the author at :
* - FSE source repository : https://github.com/Cyan4973/FiniteStateEntropy
* - Public forum : https://groups.google.com/forum/#!forum/lz4c
*
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
* You may select, at your option, one of the above-listed licenses.
****************************************************************** */
/* --- dependencies --- */
#include "zstd_deps.h" /* size_t */
/* --- simple histogram functions --- */
/*! HIST_count():
* Provides the precise count of each byte within a table 'count'.
* 'count' is a table of unsigned int, of minimum size (*maxSymbolValuePtr+1).
* Updates *maxSymbolValuePtr with actual largest symbol value detected.
* @return : count of the most frequent symbol (which isn't identified).
* or an error code, which can be tested using HIST_isError().
* note : if return == srcSize, there is only one symbol.
*/
size_t HIST_count(unsigned* count, unsigned* maxSymbolValuePtr,
const void* src, size_t srcSize);
unsigned HIST_isError(size_t code); /**< tells if a return value is an error code */
/* --- advanced histogram functions --- */
#define HIST_WKSP_SIZE_U32 1024
#define HIST_WKSP_SIZE (HIST_WKSP_SIZE_U32 * sizeof(unsigned))
/** HIST_count_wksp() :
* Same as HIST_count(), but using an externally provided scratch buffer.
* Benefit is this function will use very little stack space.
* `workSpace` is a writable buffer which must be 4-bytes aligned,
* `workSpaceSize` must be >= HIST_WKSP_SIZE
*/
size_t HIST_count_wksp(unsigned* count, unsigned* maxSymbolValuePtr,
const void* src, size_t srcSize,
void* workSpace, size_t workSpaceSize);
/** HIST_countFast() :
* same as HIST_count(), but blindly trusts that all byte values within src are <= *maxSymbolValuePtr.
* This function is unsafe, and will segfault if any value within `src` is `> *maxSymbolValuePtr`
*/
size_t HIST_countFast(unsigned* count, unsigned* maxSymbolValuePtr,
const void* src, size_t srcSize);
/** HIST_countFast_wksp() :
* Same as HIST_countFast(), but using an externally provided scratch buffer.
* `workSpace` is a writable buffer which must be 4-bytes aligned,
* `workSpaceSize` must be >= HIST_WKSP_SIZE
*/
size_t HIST_countFast_wksp(unsigned* count, unsigned* maxSymbolValuePtr,
const void* src, size_t srcSize,
void* workSpace, size_t workSpaceSize);
/*! HIST_count_simple() :
* Same as HIST_countFast(), this function is unsafe,
* and will segfault if any value within `src` is `> *maxSymbolValuePtr`.
* It is also a bit slower for large inputs.
* However, it does not need any additional memory (not even on stack).
* @return : count of the most frequent symbol.
* Note this function doesn't produce any error (i.e. it must succeed).
*/
unsigned HIST_count_simple(unsigned* count, unsigned* maxSymbolValuePtr,
const void* src, size_t srcSize);
/*! HIST_add() :
* Lowest level: just add nb of occurrences of characters from @src into @count.
* @count is not reset. @count array is presumed large enough (i.e. 1 KB).
@ This function does not need any additional stack memory.
*/
void HIST_add(unsigned* count, const void* src, size_t srcSize);
#endif /* USE_EXTERNAL_ZSTD */

280
vendor/github.com/DataDog/zstd/huf.h generated vendored
View File

@@ -1,280 +0,0 @@
#ifndef USE_EXTERNAL_ZSTD
/* ******************************************************************
* huff0 huffman codec,
* part of Finite State Entropy library
* Copyright (c) Meta Platforms, Inc. and affiliates.
*
* You can contact the author at :
* - Source repository : https://github.com/Cyan4973/FiniteStateEntropy
*
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
* You may select, at your option, one of the above-listed licenses.
****************************************************************** */
#ifndef HUF_H_298734234
#define HUF_H_298734234
/* *** Dependencies *** */
#include "zstd_deps.h" /* size_t */
#include "mem.h" /* U32 */
#define FSE_STATIC_LINKING_ONLY
#include "fse.h"
/* *** Tool functions *** */
#define HUF_BLOCKSIZE_MAX (128 * 1024) /**< maximum input size for a single block compressed with HUF_compress */
size_t HUF_compressBound(size_t size); /**< maximum compressed size (worst case) */
/* Error Management */
unsigned HUF_isError(size_t code); /**< tells if a return value is an error code */
const char* HUF_getErrorName(size_t code); /**< provides error code string (useful for debugging) */
#define HUF_WORKSPACE_SIZE ((8 << 10) + 512 /* sorting scratch space */)
#define HUF_WORKSPACE_SIZE_U64 (HUF_WORKSPACE_SIZE / sizeof(U64))
/* *** Constants *** */
#define HUF_TABLELOG_MAX 12 /* max runtime value of tableLog (due to static allocation); can be modified up to HUF_TABLELOG_ABSOLUTEMAX */
#define HUF_TABLELOG_DEFAULT 11 /* default tableLog value when none specified */
#define HUF_SYMBOLVALUE_MAX 255
#define HUF_TABLELOG_ABSOLUTEMAX 12 /* absolute limit of HUF_MAX_TABLELOG. Beyond that value, code does not work */
#if (HUF_TABLELOG_MAX > HUF_TABLELOG_ABSOLUTEMAX)
# error "HUF_TABLELOG_MAX is too large !"
#endif
/* ****************************************
* Static allocation
******************************************/
/* HUF buffer bounds */
#define HUF_CTABLEBOUND 129
#define HUF_BLOCKBOUND(size) (size + (size>>8) + 8) /* only true when incompressible is pre-filtered with fast heuristic */
#define HUF_COMPRESSBOUND(size) (HUF_CTABLEBOUND + HUF_BLOCKBOUND(size)) /* Macro version, useful for static allocation */
/* static allocation of HUF's Compression Table */
/* this is a private definition, just exposed for allocation and strict aliasing purpose. never EVER access its members directly */
typedef size_t HUF_CElt; /* consider it an incomplete type */
#define HUF_CTABLE_SIZE_ST(maxSymbolValue) ((maxSymbolValue)+2) /* Use tables of size_t, for proper alignment */
#define HUF_CTABLE_SIZE(maxSymbolValue) (HUF_CTABLE_SIZE_ST(maxSymbolValue) * sizeof(size_t))
#define HUF_CREATE_STATIC_CTABLE(name, maxSymbolValue) \
HUF_CElt name[HUF_CTABLE_SIZE_ST(maxSymbolValue)] /* no final ; */
/* static allocation of HUF's DTable */
typedef U32 HUF_DTable;
#define HUF_DTABLE_SIZE(maxTableLog) (1 + (1<<(maxTableLog)))
#define HUF_CREATE_STATIC_DTABLEX1(DTable, maxTableLog) \
HUF_DTable DTable[HUF_DTABLE_SIZE((maxTableLog)-1)] = { ((U32)((maxTableLog)-1) * 0x01000001) }
#define HUF_CREATE_STATIC_DTABLEX2(DTable, maxTableLog) \
HUF_DTable DTable[HUF_DTABLE_SIZE(maxTableLog)] = { ((U32)(maxTableLog) * 0x01000001) }
/* ****************************************
* Advanced decompression functions
******************************************/
/**
* Huffman flags bitset.
* For all flags, 0 is the default value.
*/
typedef enum {
/**
* If compiled with DYNAMIC_BMI2: Set flag only if the CPU supports BMI2 at runtime.
* Otherwise: Ignored.
*/
HUF_flags_bmi2 = (1 << 0),
/**
* If set: Test possible table depths to find the one that produces the smallest header + encoded size.
* If unset: Use heuristic to find the table depth.
*/
HUF_flags_optimalDepth = (1 << 1),
/**
* If set: If the previous table can encode the input, always reuse the previous table.
* If unset: If the previous table can encode the input, reuse the previous table if it results in a smaller output.
*/
HUF_flags_preferRepeat = (1 << 2),
/**
* If set: Sample the input and check if the sample is uncompressible, if it is then don't attempt to compress.
* If unset: Always histogram the entire input.
*/
HUF_flags_suspectUncompressible = (1 << 3),
/**
* If set: Don't use assembly implementations
* If unset: Allow using assembly implementations
*/
HUF_flags_disableAsm = (1 << 4),
/**
* If set: Don't use the fast decoding loop, always use the fallback decoding loop.
* If unset: Use the fast decoding loop when possible.
*/
HUF_flags_disableFast = (1 << 5)
} HUF_flags_e;
/* ****************************************
* HUF detailed API
* ****************************************/
#define HUF_OPTIMAL_DEPTH_THRESHOLD ZSTD_btultra
/*! HUF_compress() does the following:
* 1. count symbol occurrence from source[] into table count[] using FSE_count() (exposed within "fse.h")
* 2. (optional) refine tableLog using HUF_optimalTableLog()
* 3. build Huffman table from count using HUF_buildCTable()
* 4. save Huffman table to memory buffer using HUF_writeCTable()
* 5. encode the data stream using HUF_compress4X_usingCTable()
*
* The following API allows targeting specific sub-functions for advanced tasks.
* For example, it's possible to compress several blocks using the same 'CTable',
* or to save and regenerate 'CTable' using external methods.
*/
unsigned HUF_minTableLog(unsigned symbolCardinality);
unsigned HUF_cardinality(const unsigned* count, unsigned maxSymbolValue);
unsigned HUF_optimalTableLog(unsigned maxTableLog, size_t srcSize, unsigned maxSymbolValue, void* workSpace,
size_t wkspSize, HUF_CElt* table, const unsigned* count, int flags); /* table is used as scratch space for building and testing tables, not a return value */
size_t HUF_writeCTable_wksp(void* dst, size_t maxDstSize, const HUF_CElt* CTable, unsigned maxSymbolValue, unsigned huffLog, void* workspace, size_t workspaceSize);
size_t HUF_compress4X_usingCTable(void* dst, size_t dstSize, const void* src, size_t srcSize, const HUF_CElt* CTable, int flags);
size_t HUF_estimateCompressedSize(const HUF_CElt* CTable, const unsigned* count, unsigned maxSymbolValue);
int HUF_validateCTable(const HUF_CElt* CTable, const unsigned* count, unsigned maxSymbolValue);
typedef enum {
HUF_repeat_none, /**< Cannot use the previous table */
HUF_repeat_check, /**< Can use the previous table but it must be checked. Note : The previous table must have been constructed by HUF_compress{1, 4}X_repeat */
HUF_repeat_valid /**< Can use the previous table and it is assumed to be valid */
} HUF_repeat;
/** HUF_compress4X_repeat() :
* Same as HUF_compress4X_wksp(), but considers using hufTable if *repeat != HUF_repeat_none.
* If it uses hufTable it does not modify hufTable or repeat.
* If it doesn't, it sets *repeat = HUF_repeat_none, and it sets hufTable to the table used.
* If preferRepeat then the old table will always be used if valid.
* If suspectUncompressible then some sampling checks will be run to potentially skip huffman coding */
size_t HUF_compress4X_repeat(void* dst, size_t dstSize,
const void* src, size_t srcSize,
unsigned maxSymbolValue, unsigned tableLog,
void* workSpace, size_t wkspSize, /**< `workSpace` must be aligned on 4-bytes boundaries, `wkspSize` must be >= HUF_WORKSPACE_SIZE */
HUF_CElt* hufTable, HUF_repeat* repeat, int flags);
/** HUF_buildCTable_wksp() :
* Same as HUF_buildCTable(), but using externally allocated scratch buffer.
* `workSpace` must be aligned on 4-bytes boundaries, and its size must be >= HUF_CTABLE_WORKSPACE_SIZE.
*/
#define HUF_CTABLE_WORKSPACE_SIZE_U32 ((4 * (HUF_SYMBOLVALUE_MAX + 1)) + 192)
#define HUF_CTABLE_WORKSPACE_SIZE (HUF_CTABLE_WORKSPACE_SIZE_U32 * sizeof(unsigned))
size_t HUF_buildCTable_wksp (HUF_CElt* tree,
const unsigned* count, U32 maxSymbolValue, U32 maxNbBits,
void* workSpace, size_t wkspSize);
/*! HUF_readStats() :
* Read compact Huffman tree, saved by HUF_writeCTable().
* `huffWeight` is destination buffer.
* @return : size read from `src` , or an error Code .
* Note : Needed by HUF_readCTable() and HUF_readDTableXn() . */
size_t HUF_readStats(BYTE* huffWeight, size_t hwSize,
U32* rankStats, U32* nbSymbolsPtr, U32* tableLogPtr,
const void* src, size_t srcSize);
/*! HUF_readStats_wksp() :
* Same as HUF_readStats() but takes an external workspace which must be
* 4-byte aligned and its size must be >= HUF_READ_STATS_WORKSPACE_SIZE.
* If the CPU has BMI2 support, pass bmi2=1, otherwise pass bmi2=0.
*/
#define HUF_READ_STATS_WORKSPACE_SIZE_U32 FSE_DECOMPRESS_WKSP_SIZE_U32(6, HUF_TABLELOG_MAX-1)
#define HUF_READ_STATS_WORKSPACE_SIZE (HUF_READ_STATS_WORKSPACE_SIZE_U32 * sizeof(unsigned))
size_t HUF_readStats_wksp(BYTE* huffWeight, size_t hwSize,
U32* rankStats, U32* nbSymbolsPtr, U32* tableLogPtr,
const void* src, size_t srcSize,
void* workspace, size_t wkspSize,
int flags);
/** HUF_readCTable() :
* Loading a CTable saved with HUF_writeCTable() */
size_t HUF_readCTable (HUF_CElt* CTable, unsigned* maxSymbolValuePtr, const void* src, size_t srcSize, unsigned *hasZeroWeights);
/** HUF_getNbBitsFromCTable() :
* Read nbBits from CTable symbolTable, for symbol `symbolValue` presumed <= HUF_SYMBOLVALUE_MAX
* Note 1 : If symbolValue > HUF_readCTableHeader(symbolTable).maxSymbolValue, returns 0
* Note 2 : is not inlined, as HUF_CElt definition is private
*/
U32 HUF_getNbBitsFromCTable(const HUF_CElt* symbolTable, U32 symbolValue);
typedef struct {
BYTE tableLog;
BYTE maxSymbolValue;
BYTE unused[sizeof(size_t) - 2];
} HUF_CTableHeader;
/** HUF_readCTableHeader() :
* @returns The header from the CTable specifying the tableLog and the maxSymbolValue.
*/
HUF_CTableHeader HUF_readCTableHeader(HUF_CElt const* ctable);
/*
* HUF_decompress() does the following:
* 1. select the decompression algorithm (X1, X2) based on pre-computed heuristics
* 2. build Huffman table from save, using HUF_readDTableX?()
* 3. decode 1 or 4 segments in parallel using HUF_decompress?X?_usingDTable()
*/
/** HUF_selectDecoder() :
* Tells which decoder is likely to decode faster,
* based on a set of pre-computed metrics.
* @return : 0==HUF_decompress4X1, 1==HUF_decompress4X2 .
* Assumption : 0 < dstSize <= 128 KB */
U32 HUF_selectDecoder (size_t dstSize, size_t cSrcSize);
/**
* The minimum workspace size for the `workSpace` used in
* HUF_readDTableX1_wksp() and HUF_readDTableX2_wksp().
*
* The space used depends on HUF_TABLELOG_MAX, ranging from ~1500 bytes when
* HUF_TABLE_LOG_MAX=12 to ~1850 bytes when HUF_TABLE_LOG_MAX=15.
* Buffer overflow errors may potentially occur if code modifications result in
* a required workspace size greater than that specified in the following
* macro.
*/
#define HUF_DECOMPRESS_WORKSPACE_SIZE ((2 << 10) + (1 << 9))
#define HUF_DECOMPRESS_WORKSPACE_SIZE_U32 (HUF_DECOMPRESS_WORKSPACE_SIZE / sizeof(U32))
/* ====================== */
/* single stream variants */
/* ====================== */
size_t HUF_compress1X_usingCTable(void* dst, size_t dstSize, const void* src, size_t srcSize, const HUF_CElt* CTable, int flags);
/** HUF_compress1X_repeat() :
* Same as HUF_compress1X_wksp(), but considers using hufTable if *repeat != HUF_repeat_none.
* If it uses hufTable it does not modify hufTable or repeat.
* If it doesn't, it sets *repeat = HUF_repeat_none, and it sets hufTable to the table used.
* If preferRepeat then the old table will always be used if valid.
* If suspectUncompressible then some sampling checks will be run to potentially skip huffman coding */
size_t HUF_compress1X_repeat(void* dst, size_t dstSize,
const void* src, size_t srcSize,
unsigned maxSymbolValue, unsigned tableLog,
void* workSpace, size_t wkspSize, /**< `workSpace` must be aligned on 4-bytes boundaries, `wkspSize` must be >= HUF_WORKSPACE_SIZE */
HUF_CElt* hufTable, HUF_repeat* repeat, int flags);
size_t HUF_decompress1X_DCtx_wksp(HUF_DTable* dctx, void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize, void* workSpace, size_t wkspSize, int flags);
#ifndef HUF_FORCE_DECOMPRESS_X1
size_t HUF_decompress1X2_DCtx_wksp(HUF_DTable* dctx, void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize, void* workSpace, size_t wkspSize, int flags); /**< double-symbols decoder */
#endif
/* BMI2 variants.
* If the CPU has BMI2 support, pass bmi2=1, otherwise pass bmi2=0.
*/
size_t HUF_decompress1X_usingDTable(void* dst, size_t maxDstSize, const void* cSrc, size_t cSrcSize, const HUF_DTable* DTable, int flags);
#ifndef HUF_FORCE_DECOMPRESS_X2
size_t HUF_decompress1X1_DCtx_wksp(HUF_DTable* dctx, void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize, void* workSpace, size_t wkspSize, int flags);
#endif
size_t HUF_decompress4X_usingDTable(void* dst, size_t maxDstSize, const void* cSrc, size_t cSrcSize, const HUF_DTable* DTable, int flags);
size_t HUF_decompress4X_hufOnly_wksp(HUF_DTable* dctx, void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize, void* workSpace, size_t wkspSize, int flags);
#ifndef HUF_FORCE_DECOMPRESS_X2
size_t HUF_readDTableX1_wksp(HUF_DTable* DTable, const void* src, size_t srcSize, void* workSpace, size_t wkspSize, int flags);
#endif
#ifndef HUF_FORCE_DECOMPRESS_X1
size_t HUF_readDTableX2_wksp(HUF_DTable* DTable, const void* src, size_t srcSize, void* workSpace, size_t wkspSize, int flags);
#endif
#endif /* HUF_H_298734234 */
#endif /* USE_EXTERNAL_ZSTD */

1467
vendor/github.com/DataDog/zstd/huf_compress.c generated vendored
View File

File diff suppressed because it is too large Load Diff

1947
vendor/github.com/DataDog/zstd/huf_decompress.c generated vendored
View File

File diff suppressed because it is too large Load Diff

602
vendor/github.com/DataDog/zstd/huf_decompress_amd64.S generated vendored
View File

@@ -1,602 +0,0 @@
/*
* Copyright (c) Meta Platforms, Inc. and affiliates.
* All rights reserved.
*
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
* You may select, at your option, one of the above-listed licenses.
*/
#include "portability_macros.h"
#if defined(__ELF__) && defined(__GNUC__)
/* Stack marking
* ref: https://wiki.gentoo.org/wiki/Hardened/GNU_stack_quickstart
*/
.section .note.GNU-stack,"",%progbits
#if defined(__aarch64__)
/* Mark that this assembly supports BTI & PAC, because it is empty for aarch64.
* See: https://github.com/facebook/zstd/issues/3841
* See: https://gcc.godbolt.org/z/sqr5T4ffK
* See: https://lore.kernel.org/linux-arm-kernel/20200429211641.9279-8-broonie@kernel.org/
* See: https://reviews.llvm.org/D62609
*/
.pushsection .note.gnu.property, "a"
.p2align 3
.long 4 /* size of the name - "GNU\0" */
.long 0x10 /* size of descriptor */
.long 0x5 /* NT_GNU_PROPERTY_TYPE_0 */
.asciz "GNU"
.long 0xc0000000 /* pr_type - GNU_PROPERTY_AARCH64_FEATURE_1_AND */
.long 4 /* pr_datasz - 4 bytes */
.long 3 /* pr_data - GNU_PROPERTY_AARCH64_FEATURE_1_BTI | GNU_PROPERTY_AARCH64_FEATURE_1_PAC */
.p2align 3 /* pr_padding - bring everything to 8 byte alignment */
.popsection
#endif
#endif
#if ZSTD_ENABLE_ASM_X86_64_BMI2
/* Calling convention:
*
* %rdi (or %rcx on Windows) contains the first argument: HUF_DecompressAsmArgs*.
* %rbp isn't maintained (no frame pointer).
* %rsp contains the stack pointer that grows down.
* No red-zone is assumed, only addresses >= %rsp are used.
* All register contents are preserved.
*/
ZSTD_HIDE_ASM_FUNCTION(HUF_decompress4X1_usingDTable_internal_fast_asm_loop)
ZSTD_HIDE_ASM_FUNCTION(HUF_decompress4X2_usingDTable_internal_fast_asm_loop)
ZSTD_HIDE_ASM_FUNCTION(_HUF_decompress4X2_usingDTable_internal_fast_asm_loop)
ZSTD_HIDE_ASM_FUNCTION(_HUF_decompress4X1_usingDTable_internal_fast_asm_loop)
.global HUF_decompress4X1_usingDTable_internal_fast_asm_loop
.global HUF_decompress4X2_usingDTable_internal_fast_asm_loop
.global _HUF_decompress4X1_usingDTable_internal_fast_asm_loop
.global _HUF_decompress4X2_usingDTable_internal_fast_asm_loop
.text
/* Sets up register mappings for clarity.
* op[], bits[], dtable & ip[0] each get their own register.
* ip[1,2,3] & olimit alias var[].
* %rax is a scratch register.
*/
#define op0 rsi
#define op1 rbx
#define op2 rcx
#define op3 rdi
#define ip0 r8
#define ip1 r9
#define ip2 r10
#define ip3 r11
#define bits0 rbp
#define bits1 rdx
#define bits2 r12
#define bits3 r13
#define dtable r14
#define olimit r15
/* var[] aliases ip[1,2,3] & olimit
* ip[1,2,3] are saved every iteration.
* olimit is only used in compute_olimit.
*/
#define var0 r15
#define var1 r9
#define var2 r10
#define var3 r11
/* 32-bit var registers */
#define vard0 r15d
#define vard1 r9d
#define vard2 r10d
#define vard3 r11d
/* Calls X(N) for each stream 0, 1, 2, 3. */
#define FOR_EACH_STREAM(X) \
X(0); \
X(1); \
X(2); \
X(3)
/* Calls X(N, idx) for each stream 0, 1, 2, 3. */
#define FOR_EACH_STREAM_WITH_INDEX(X, idx) \
X(0, idx); \
X(1, idx); \
X(2, idx); \
X(3, idx)
/* Define both _HUF_* & HUF_* symbols because MacOS
* C symbols are prefixed with '_' & Linux symbols aren't.
*/
_HUF_decompress4X1_usingDTable_internal_fast_asm_loop:
HUF_decompress4X1_usingDTable_internal_fast_asm_loop:
ZSTD_CET_ENDBRANCH
/* Save all registers - even if they are callee saved for simplicity. */
push %rax
push %rbx
push %rcx
push %rdx
push %rbp
push %rsi
push %rdi
push %r8
push %r9
push %r10
push %r11
push %r12
push %r13
push %r14
push %r15
/* Read HUF_DecompressAsmArgs* args from %rax */
#if defined(_WIN32)
movq %rcx, %rax
#else
movq %rdi, %rax
#endif
movq 0(%rax), %ip0
movq 8(%rax), %ip1
movq 16(%rax), %ip2
movq 24(%rax), %ip3
movq 32(%rax), %op0
movq 40(%rax), %op1
movq 48(%rax), %op2
movq 56(%rax), %op3
movq 64(%rax), %bits0
movq 72(%rax), %bits1
movq 80(%rax), %bits2
movq 88(%rax), %bits3
movq 96(%rax), %dtable
push %rax /* argument */
push 104(%rax) /* ilowest */
push 112(%rax) /* oend */
push %olimit /* olimit space */
subq $24, %rsp
.L_4X1_compute_olimit:
/* Computes how many iterations we can do safely
* %r15, %rax may be clobbered
* rbx, rdx must be saved
* op3 & ip0 mustn't be clobbered
*/
movq %rbx, 0(%rsp)
movq %rdx, 8(%rsp)
movq 32(%rsp), %rax /* rax = oend */
subq %op3, %rax /* rax = oend - op3 */
/* r15 = (oend - op3) / 5 */
movabsq $-3689348814741910323, %rdx
mulq %rdx
movq %rdx, %r15
shrq $2, %r15
movq %ip0, %rax /* rax = ip0 */
movq 40(%rsp), %rdx /* rdx = ilowest */
subq %rdx, %rax /* rax = ip0 - ilowest */
movq %rax, %rbx /* rbx = ip0 - ilowest */
/* rdx = (ip0 - ilowest) / 7 */
movabsq $2635249153387078803, %rdx
mulq %rdx
subq %rdx, %rbx
shrq %rbx
addq %rbx, %rdx
shrq $2, %rdx
/* r15 = min(%rdx, %r15) */
cmpq %rdx, %r15
cmova %rdx, %r15
/* r15 = r15 * 5 */
leaq (%r15, %r15, 4), %r15
/* olimit = op3 + r15 */
addq %op3, %olimit
movq 8(%rsp), %rdx
movq 0(%rsp), %rbx
/* If (op3 + 20 > olimit) */
movq %op3, %rax /* rax = op3 */
cmpq %rax, %olimit /* op3 == olimit */
je .L_4X1_exit
/* If (ip1 < ip0) go to exit */
cmpq %ip0, %ip1
jb .L_4X1_exit
/* If (ip2 < ip1) go to exit */
cmpq %ip1, %ip2
jb .L_4X1_exit
/* If (ip3 < ip2) go to exit */
cmpq %ip2, %ip3
jb .L_4X1_exit
/* Reads top 11 bits from bits[n]
* Loads dt[bits[n]] into var[n]
*/
#define GET_NEXT_DELT(n) \
movq $53, %var##n; \
shrxq %var##n, %bits##n, %var##n; \
movzwl (%dtable,%var##n,2),%vard##n
/* var[n] must contain the DTable entry computed with GET_NEXT_DELT
* Moves var[n] to %rax
* bits[n] <<= var[n] & 63
* op[n][idx] = %rax >> 8
* %ah is a way to access bits [8, 16) of %rax
*/
#define DECODE_FROM_DELT(n, idx) \
movq %var##n, %rax; \
shlxq %var##n, %bits##n, %bits##n; \
movb %ah, idx(%op##n)
/* Assumes GET_NEXT_DELT has been called.
* Calls DECODE_FROM_DELT then GET_NEXT_DELT
*/
#define DECODE_AND_GET_NEXT(n, idx) \
DECODE_FROM_DELT(n, idx); \
GET_NEXT_DELT(n) \
/* // ctz & nbBytes is stored in bits[n]
* // nbBits is stored in %rax
* ctz = CTZ[bits[n]]
* nbBits = ctz & 7
* nbBytes = ctz >> 3
* op[n] += 5
* ip[n] -= nbBytes
* // Note: x86-64 is little-endian ==> no bswap
* bits[n] = MEM_readST(ip[n]) | 1
* bits[n] <<= nbBits
*/
#define RELOAD_BITS(n) \
bsfq %bits##n, %bits##n; \
movq %bits##n, %rax; \
andq $7, %rax; \
shrq $3, %bits##n; \
leaq 5(%op##n), %op##n; \
subq %bits##n, %ip##n; \
movq (%ip##n), %bits##n; \
orq $1, %bits##n; \
shlx %rax, %bits##n, %bits##n
/* Store clobbered variables on the stack */
movq %olimit, 24(%rsp)
movq %ip1, 0(%rsp)
movq %ip2, 8(%rsp)
movq %ip3, 16(%rsp)
/* Call GET_NEXT_DELT for each stream */
FOR_EACH_STREAM(GET_NEXT_DELT)
.p2align 6
.L_4X1_loop_body:
/* Decode 5 symbols in each of the 4 streams (20 total)
* Must have called GET_NEXT_DELT for each stream
*/
FOR_EACH_STREAM_WITH_INDEX(DECODE_AND_GET_NEXT, 0)
FOR_EACH_STREAM_WITH_INDEX(DECODE_AND_GET_NEXT, 1)
FOR_EACH_STREAM_WITH_INDEX(DECODE_AND_GET_NEXT, 2)
FOR_EACH_STREAM_WITH_INDEX(DECODE_AND_GET_NEXT, 3)
FOR_EACH_STREAM_WITH_INDEX(DECODE_FROM_DELT, 4)
/* Load ip[1,2,3] from stack (var[] aliases them)
* ip[] is needed for RELOAD_BITS
* Each will be stored back to the stack after RELOAD
*/
movq 0(%rsp), %ip1
movq 8(%rsp), %ip2
movq 16(%rsp), %ip3
/* Reload each stream & fetch the next table entry
* to prepare for the next iteration
*/
RELOAD_BITS(0)
GET_NEXT_DELT(0)
RELOAD_BITS(1)
movq %ip1, 0(%rsp)
GET_NEXT_DELT(1)
RELOAD_BITS(2)
movq %ip2, 8(%rsp)
GET_NEXT_DELT(2)
RELOAD_BITS(3)
movq %ip3, 16(%rsp)
GET_NEXT_DELT(3)
/* If op3 < olimit: continue the loop */
cmp %op3, 24(%rsp)
ja .L_4X1_loop_body
/* Reload ip[1,2,3] from stack */
movq 0(%rsp), %ip1
movq 8(%rsp), %ip2
movq 16(%rsp), %ip3
/* Re-compute olimit */
jmp .L_4X1_compute_olimit
#undef GET_NEXT_DELT
#undef DECODE_FROM_DELT
#undef DECODE
#undef RELOAD_BITS
.L_4X1_exit:
addq $24, %rsp
/* Restore stack (oend & olimit) */
pop %rax /* olimit */
pop %rax /* oend */
pop %rax /* ilowest */
pop %rax /* arg */
/* Save ip / op / bits */
movq %ip0, 0(%rax)
movq %ip1, 8(%rax)
movq %ip2, 16(%rax)
movq %ip3, 24(%rax)
movq %op0, 32(%rax)
movq %op1, 40(%rax)
movq %op2, 48(%rax)
movq %op3, 56(%rax)
movq %bits0, 64(%rax)
movq %bits1, 72(%rax)
movq %bits2, 80(%rax)
movq %bits3, 88(%rax)
/* Restore registers */
pop %r15
pop %r14
pop %r13
pop %r12
pop %r11
pop %r10
pop %r9
pop %r8
pop %rdi
pop %rsi
pop %rbp
pop %rdx
pop %rcx
pop %rbx
pop %rax
ret
_HUF_decompress4X2_usingDTable_internal_fast_asm_loop:
HUF_decompress4X2_usingDTable_internal_fast_asm_loop:
ZSTD_CET_ENDBRANCH
/* Save all registers - even if they are callee saved for simplicity. */
push %rax
push %rbx
push %rcx
push %rdx
push %rbp
push %rsi
push %rdi
push %r8
push %r9
push %r10
push %r11
push %r12
push %r13
push %r14
push %r15
/* Read HUF_DecompressAsmArgs* args from %rax */
#if defined(_WIN32)
movq %rcx, %rax
#else
movq %rdi, %rax
#endif
movq 0(%rax), %ip0
movq 8(%rax), %ip1
movq 16(%rax), %ip2
movq 24(%rax), %ip3
movq 32(%rax), %op0
movq 40(%rax), %op1
movq 48(%rax), %op2
movq 56(%rax), %op3
movq 64(%rax), %bits0
movq 72(%rax), %bits1
movq 80(%rax), %bits2
movq 88(%rax), %bits3
movq 96(%rax), %dtable
push %rax /* argument */
push %rax /* olimit */
push 104(%rax) /* ilowest */
movq 112(%rax), %rax
push %rax /* oend3 */
movq %op3, %rax
push %rax /* oend2 */
movq %op2, %rax
push %rax /* oend1 */
movq %op1, %rax
push %rax /* oend0 */
/* Scratch space */
subq $8, %rsp
.L_4X2_compute_olimit:
/* Computes how many iterations we can do safely
* %r15, %rax may be clobbered
* rdx must be saved
* op[1,2,3,4] & ip0 mustn't be clobbered
*/
movq %rdx, 0(%rsp)
/* We can consume up to 7 input bytes each iteration. */
movq %ip0, %rax /* rax = ip0 */
movq 40(%rsp), %rdx /* rdx = ilowest */
subq %rdx, %rax /* rax = ip0 - ilowest */
movq %rax, %r15 /* r15 = ip0 - ilowest */
/* rdx = rax / 7 */
movabsq $2635249153387078803, %rdx
mulq %rdx
subq %rdx, %r15
shrq %r15
addq %r15, %rdx
shrq $2, %rdx
/* r15 = (ip0 - ilowest) / 7 */
movq %rdx, %r15
/* r15 = min(r15, min(oend0 - op0, oend1 - op1, oend2 - op2, oend3 - op3) / 10) */
movq 8(%rsp), %rax /* rax = oend0 */
subq %op0, %rax /* rax = oend0 - op0 */
movq 16(%rsp), %rdx /* rdx = oend1 */
subq %op1, %rdx /* rdx = oend1 - op1 */
cmpq %rax, %rdx
cmova %rax, %rdx /* rdx = min(%rdx, %rax) */
movq 24(%rsp), %rax /* rax = oend2 */
subq %op2, %rax /* rax = oend2 - op2 */
cmpq %rax, %rdx
cmova %rax, %rdx /* rdx = min(%rdx, %rax) */
movq 32(%rsp), %rax /* rax = oend3 */
subq %op3, %rax /* rax = oend3 - op3 */
cmpq %rax, %rdx
cmova %rax, %rdx /* rdx = min(%rdx, %rax) */
movabsq $-3689348814741910323, %rax
mulq %rdx
shrq $3, %rdx /* rdx = rdx / 10 */
/* r15 = min(%rdx, %r15) */
cmpq %rdx, %r15
cmova %rdx, %r15
/* olimit = op3 + 5 * r15 */
movq %r15, %rax
leaq (%op3, %rax, 4), %olimit
addq %rax, %olimit
movq 0(%rsp), %rdx
/* If (op3 + 10 > olimit) */
movq %op3, %rax /* rax = op3 */
cmpq %rax, %olimit /* op3 == olimit */
je .L_4X2_exit
/* If (ip1 < ip0) go to exit */
cmpq %ip0, %ip1
jb .L_4X2_exit
/* If (ip2 < ip1) go to exit */
cmpq %ip1, %ip2
jb .L_4X2_exit
/* If (ip3 < ip2) go to exit */
cmpq %ip2, %ip3
jb .L_4X2_exit
#define DECODE(n, idx) \
movq %bits##n, %rax; \
shrq $53, %rax; \
movzwl 0(%dtable,%rax,4),%r8d; \
movzbl 2(%dtable,%rax,4),%r15d; \
movzbl 3(%dtable,%rax,4),%eax; \
movw %r8w, (%op##n); \
shlxq %r15, %bits##n, %bits##n; \
addq %rax, %op##n
#define RELOAD_BITS(n) \
bsfq %bits##n, %bits##n; \
movq %bits##n, %rax; \
shrq $3, %bits##n; \
andq $7, %rax; \
subq %bits##n, %ip##n; \
movq (%ip##n), %bits##n; \
orq $1, %bits##n; \
shlxq %rax, %bits##n, %bits##n
movq %olimit, 48(%rsp)
.p2align 6
.L_4X2_loop_body:
/* We clobber r8, so store it on the stack */
movq %r8, 0(%rsp)
/* Decode 5 symbols from each of the 4 streams (20 symbols total). */
FOR_EACH_STREAM_WITH_INDEX(DECODE, 0)
FOR_EACH_STREAM_WITH_INDEX(DECODE, 1)
FOR_EACH_STREAM_WITH_INDEX(DECODE, 2)
FOR_EACH_STREAM_WITH_INDEX(DECODE, 3)
FOR_EACH_STREAM_WITH_INDEX(DECODE, 4)
/* Reload r8 */
movq 0(%rsp), %r8
FOR_EACH_STREAM(RELOAD_BITS)
cmp %op3, 48(%rsp)
ja .L_4X2_loop_body
jmp .L_4X2_compute_olimit
#undef DECODE
#undef RELOAD_BITS
.L_4X2_exit:
addq $8, %rsp
/* Restore stack (oend & olimit) */
pop %rax /* oend0 */
pop %rax /* oend1 */
pop %rax /* oend2 */
pop %rax /* oend3 */
pop %rax /* ilowest */
pop %rax /* olimit */
pop %rax /* arg */
/* Save ip / op / bits */
movq %ip0, 0(%rax)
movq %ip1, 8(%rax)
movq %ip2, 16(%rax)
movq %ip3, 24(%rax)
movq %op0, 32(%rax)
movq %op1, 40(%rax)
movq %op2, 48(%rax)
movq %op3, 56(%rax)
movq %bits0, 64(%rax)
movq %bits1, 72(%rax)
movq %bits2, 80(%rax)
movq %bits3, 88(%rax)
/* Restore registers */
pop %r15
pop %r14
pop %r13
pop %r12
pop %r11
pop %r10
pop %r9
pop %r8
pop %rdi
pop %rsi
pop %rbp
pop %rdx
pop %rcx
pop %rbx
pop %rax
ret
#endif

425
vendor/github.com/DataDog/zstd/mem.h generated vendored
View File

@@ -1,425 +0,0 @@
#ifndef USE_EXTERNAL_ZSTD
/*
* Copyright (c) Meta Platforms, Inc. and affiliates.
* All rights reserved.
*
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
* You may select, at your option, one of the above-listed licenses.
*/
#ifndef MEM_H_MODULE
#define MEM_H_MODULE
/*-****************************************
* Dependencies
******************************************/
#include <stddef.h> /* size_t, ptrdiff_t */
#include "compiler.h" /* __has_builtin */
#include "debug.h" /* DEBUG_STATIC_ASSERT */
#include "zstd_deps.h" /* ZSTD_memcpy */
/*-****************************************
* Compiler specifics
******************************************/
#if defined(_MSC_VER) /* Visual Studio */
# include <stdlib.h> /* _byteswap_ulong */
# include <intrin.h> /* _byteswap_* */
#elif defined(__ICCARM__)
# include <intrinsics.h>
#endif
/*-**************************************************************
* Basic Types
*****************************************************************/
#if !defined (__VMS) && (defined (__cplusplus) || (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */) )
# if defined(_AIX)
# include <inttypes.h>
# else
# include <stdint.h> /* intptr_t */
# endif
typedef uint8_t BYTE;
typedef uint8_t U8;
typedef int8_t S8;
typedef uint16_t U16;
typedef int16_t S16;
typedef uint32_t U32;
typedef int32_t S32;
typedef uint64_t U64;
typedef int64_t S64;
#else
# include <limits.h>
#if CHAR_BIT != 8
# error "this implementation requires char to be exactly 8-bit type"
#endif
typedef unsigned char BYTE;
typedef unsigned char U8;
typedef signed char S8;
#if USHRT_MAX != 65535
# error "this implementation requires short to be exactly 16-bit type"
#endif
typedef unsigned short U16;
typedef signed short S16;
#if UINT_MAX != 4294967295
# error "this implementation requires int to be exactly 32-bit type"
#endif
typedef unsigned int U32;
typedef signed int S32;
/* note : there are no limits defined for long long type in C90.
* limits exist in C99, however, in such case, <stdint.h> is preferred */
typedef unsigned long long U64;
typedef signed long long S64;
#endif
/*-**************************************************************
* Memory I/O API
*****************************************************************/
/*=== Static platform detection ===*/
MEM_STATIC unsigned MEM_32bits(void);
MEM_STATIC unsigned MEM_64bits(void);
MEM_STATIC unsigned MEM_isLittleEndian(void);
/*=== Native unaligned read/write ===*/
MEM_STATIC U16 MEM_read16(const void* memPtr);
MEM_STATIC U32 MEM_read32(const void* memPtr);
MEM_STATIC U64 MEM_read64(const void* memPtr);
MEM_STATIC size_t MEM_readST(const void* memPtr);
MEM_STATIC void MEM_write16(void* memPtr, U16 value);
MEM_STATIC void MEM_write32(void* memPtr, U32 value);
MEM_STATIC void MEM_write64(void* memPtr, U64 value);
/*=== Little endian unaligned read/write ===*/
MEM_STATIC U16 MEM_readLE16(const void* memPtr);
MEM_STATIC U32 MEM_readLE24(const void* memPtr);
MEM_STATIC U32 MEM_readLE32(const void* memPtr);
MEM_STATIC U64 MEM_readLE64(const void* memPtr);
MEM_STATIC size_t MEM_readLEST(const void* memPtr);
MEM_STATIC void MEM_writeLE16(void* memPtr, U16 val);
MEM_STATIC void MEM_writeLE24(void* memPtr, U32 val);
MEM_STATIC void MEM_writeLE32(void* memPtr, U32 val32);
MEM_STATIC void MEM_writeLE64(void* memPtr, U64 val64);
MEM_STATIC void MEM_writeLEST(void* memPtr, size_t val);
/*=== Big endian unaligned read/write ===*/
MEM_STATIC U32 MEM_readBE32(const void* memPtr);
MEM_STATIC U64 MEM_readBE64(const void* memPtr);
MEM_STATIC size_t MEM_readBEST(const void* memPtr);
MEM_STATIC void MEM_writeBE32(void* memPtr, U32 val32);
MEM_STATIC void MEM_writeBE64(void* memPtr, U64 val64);
MEM_STATIC void MEM_writeBEST(void* memPtr, size_t val);
/*=== Byteswap ===*/
MEM_STATIC U32 MEM_swap32(U32 in);
MEM_STATIC U64 MEM_swap64(U64 in);
MEM_STATIC size_t MEM_swapST(size_t in);
/*-**************************************************************
* Memory I/O Implementation
*****************************************************************/
/* MEM_FORCE_MEMORY_ACCESS : For accessing unaligned memory:
* Method 0 : always use `memcpy()`. Safe and portable.
* Method 1 : Use compiler extension to set unaligned access.
* Method 2 : direct access. This method is portable but violate C standard.
* It can generate buggy code on targets depending on alignment.
* Default : method 1 if supported, else method 0
*/
#ifndef MEM_FORCE_MEMORY_ACCESS /* can be defined externally, on command line for example */
# ifdef __GNUC__
# define MEM_FORCE_MEMORY_ACCESS 1
# endif
#endif
MEM_STATIC unsigned MEM_32bits(void) { return sizeof(size_t)==4; }
MEM_STATIC unsigned MEM_64bits(void) { return sizeof(size_t)==8; }
MEM_STATIC unsigned MEM_isLittleEndian(void)
{
#if defined(__BYTE_ORDER__) && defined(__ORDER_LITTLE_ENDIAN__) && (__BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__)
return 1;
#elif defined(__BYTE_ORDER__) && defined(__ORDER_BIG_ENDIAN__) && (__BYTE_ORDER__ == __ORDER_BIG_ENDIAN__)
return 0;
#elif defined(__clang__) && __LITTLE_ENDIAN__
return 1;
#elif defined(__clang__) && __BIG_ENDIAN__
return 0;
#elif defined(_MSC_VER) && (_M_X64 || _M_IX86)
return 1;
#elif defined(__DMC__) && defined(_M_IX86)
return 1;
#elif defined(__IAR_SYSTEMS_ICC__) && __LITTLE_ENDIAN__
return 1;
#else
const union { U32 u; BYTE c[4]; } one = { 1 }; /* don't use static : performance detrimental */
return one.c[0];
#endif
}
#if defined(MEM_FORCE_MEMORY_ACCESS) && (MEM_FORCE_MEMORY_ACCESS==2)
/* violates C standard, by lying on structure alignment.
Only use if no other choice to achieve best performance on target platform */
MEM_STATIC U16 MEM_read16(const void* memPtr) { return *(const U16*) memPtr; }
MEM_STATIC U32 MEM_read32(const void* memPtr) { return *(const U32*) memPtr; }
MEM_STATIC U64 MEM_read64(const void* memPtr) { return *(const U64*) memPtr; }
MEM_STATIC size_t MEM_readST(const void* memPtr) { return *(const size_t*) memPtr; }
MEM_STATIC void MEM_write16(void* memPtr, U16 value) { *(U16*)memPtr = value; }
MEM_STATIC void MEM_write32(void* memPtr, U32 value) { *(U32*)memPtr = value; }
MEM_STATIC void MEM_write64(void* memPtr, U64 value) { *(U64*)memPtr = value; }
#elif defined(MEM_FORCE_MEMORY_ACCESS) && (MEM_FORCE_MEMORY_ACCESS==1)
typedef __attribute__((aligned(1))) U16 unalign16;
typedef __attribute__((aligned(1))) U32 unalign32;
typedef __attribute__((aligned(1))) U64 unalign64;
typedef __attribute__((aligned(1))) size_t unalignArch;
MEM_STATIC U16 MEM_read16(const void* ptr) { return *(const unalign16*)ptr; }
MEM_STATIC U32 MEM_read32(const void* ptr) { return *(const unalign32*)ptr; }
MEM_STATIC U64 MEM_read64(const void* ptr) { return *(const unalign64*)ptr; }
MEM_STATIC size_t MEM_readST(const void* ptr) { return *(const unalignArch*)ptr; }
MEM_STATIC void MEM_write16(void* memPtr, U16 value) { *(unalign16*)memPtr = value; }
MEM_STATIC void MEM_write32(void* memPtr, U32 value) { *(unalign32*)memPtr = value; }
MEM_STATIC void MEM_write64(void* memPtr, U64 value) { *(unalign64*)memPtr = value; }
#else
/* default method, safe and standard.
can sometimes prove slower */
MEM_STATIC U16 MEM_read16(const void* memPtr)
{
U16 val; ZSTD_memcpy(&val, memPtr, sizeof(val)); return val;
}
MEM_STATIC U32 MEM_read32(const void* memPtr)
{
U32 val; ZSTD_memcpy(&val, memPtr, sizeof(val)); return val;
}
MEM_STATIC U64 MEM_read64(const void* memPtr)
{
U64 val; ZSTD_memcpy(&val, memPtr, sizeof(val)); return val;
}
MEM_STATIC size_t MEM_readST(const void* memPtr)
{
size_t val; ZSTD_memcpy(&val, memPtr, sizeof(val)); return val;
}
MEM_STATIC void MEM_write16(void* memPtr, U16 value)
{
ZSTD_memcpy(memPtr, &value, sizeof(value));
}
MEM_STATIC void MEM_write32(void* memPtr, U32 value)
{
ZSTD_memcpy(memPtr, &value, sizeof(value));
}
MEM_STATIC void MEM_write64(void* memPtr, U64 value)
{
ZSTD_memcpy(memPtr, &value, sizeof(value));
}
#endif /* MEM_FORCE_MEMORY_ACCESS */
MEM_STATIC U32 MEM_swap32_fallback(U32 in)
{
return ((in << 24) & 0xff000000 ) |
((in << 8) & 0x00ff0000 ) |
((in >> 8) & 0x0000ff00 ) |
((in >> 24) & 0x000000ff );
}
MEM_STATIC U32 MEM_swap32(U32 in)
{
#if defined(_MSC_VER) /* Visual Studio */
return _byteswap_ulong(in);
#elif (defined (__GNUC__) && (__GNUC__ * 100 + __GNUC_MINOR__ >= 403)) \
|| (defined(__clang__) && __has_builtin(__builtin_bswap32))
return __builtin_bswap32(in);
#elif defined(__ICCARM__)
return __REV(in);
#else
return MEM_swap32_fallback(in);
#endif
}
MEM_STATIC U64 MEM_swap64_fallback(U64 in)
{
return ((in << 56) & 0xff00000000000000ULL) |
((in << 40) & 0x00ff000000000000ULL) |
((in << 24) & 0x0000ff0000000000ULL) |
((in << 8) & 0x000000ff00000000ULL) |
((in >> 8) & 0x00000000ff000000ULL) |
((in >> 24) & 0x0000000000ff0000ULL) |
((in >> 40) & 0x000000000000ff00ULL) |
((in >> 56) & 0x00000000000000ffULL);
}
MEM_STATIC U64 MEM_swap64(U64 in)
{
#if defined(_MSC_VER) /* Visual Studio */
return _byteswap_uint64(in);
#elif (defined (__GNUC__) && (__GNUC__ * 100 + __GNUC_MINOR__ >= 403)) \
|| (defined(__clang__) && __has_builtin(__builtin_bswap64))
return __builtin_bswap64(in);
#else
return MEM_swap64_fallback(in);
#endif
}
MEM_STATIC size_t MEM_swapST(size_t in)
{
if (MEM_32bits())
return (size_t)MEM_swap32((U32)in);
else
return (size_t)MEM_swap64((U64)in);
}
/*=== Little endian r/w ===*/
MEM_STATIC U16 MEM_readLE16(const void* memPtr)
{
if (MEM_isLittleEndian())
return MEM_read16(memPtr);
else {
const BYTE* p = (const BYTE*)memPtr;
return (U16)(p[0] + (p[1]<<8));
}
}
MEM_STATIC void MEM_writeLE16(void* memPtr, U16 val)
{
if (MEM_isLittleEndian()) {
MEM_write16(memPtr, val);
} else {
BYTE* p = (BYTE*)memPtr;
p[0] = (BYTE)val;
p[1] = (BYTE)(val>>8);
}
}
MEM_STATIC U32 MEM_readLE24(const void* memPtr)
{
return (U32)MEM_readLE16(memPtr) + ((U32)(((const BYTE*)memPtr)[2]) << 16);
}
MEM_STATIC void MEM_writeLE24(void* memPtr, U32 val)
{
MEM_writeLE16(memPtr, (U16)val);
((BYTE*)memPtr)[2] = (BYTE)(val>>16);
}
MEM_STATIC U32 MEM_readLE32(const void* memPtr)
{
if (MEM_isLittleEndian())
return MEM_read32(memPtr);
else
return MEM_swap32(MEM_read32(memPtr));
}
MEM_STATIC void MEM_writeLE32(void* memPtr, U32 val32)
{
if (MEM_isLittleEndian())
MEM_write32(memPtr, val32);
else
MEM_write32(memPtr, MEM_swap32(val32));
}
MEM_STATIC U64 MEM_readLE64(const void* memPtr)
{
if (MEM_isLittleEndian())
return MEM_read64(memPtr);
else
return MEM_swap64(MEM_read64(memPtr));
}
MEM_STATIC void MEM_writeLE64(void* memPtr, U64 val64)
{
if (MEM_isLittleEndian())
MEM_write64(memPtr, val64);
else
MEM_write64(memPtr, MEM_swap64(val64));
}
MEM_STATIC size_t MEM_readLEST(const void* memPtr)
{
if (MEM_32bits())
return (size_t)MEM_readLE32(memPtr);
else
return (size_t)MEM_readLE64(memPtr);
}
MEM_STATIC void MEM_writeLEST(void* memPtr, size_t val)
{
if (MEM_32bits())
MEM_writeLE32(memPtr, (U32)val);
else
MEM_writeLE64(memPtr, (U64)val);
}
/*=== Big endian r/w ===*/
MEM_STATIC U32 MEM_readBE32(const void* memPtr)
{
if (MEM_isLittleEndian())
return MEM_swap32(MEM_read32(memPtr));
else
return MEM_read32(memPtr);
}
MEM_STATIC void MEM_writeBE32(void* memPtr, U32 val32)
{
if (MEM_isLittleEndian())
MEM_write32(memPtr, MEM_swap32(val32));
else
MEM_write32(memPtr, val32);
}
MEM_STATIC U64 MEM_readBE64(const void* memPtr)
{
if (MEM_isLittleEndian())
return MEM_swap64(MEM_read64(memPtr));
else
return MEM_read64(memPtr);
}
MEM_STATIC void MEM_writeBE64(void* memPtr, U64 val64)
{
if (MEM_isLittleEndian())
MEM_write64(memPtr, MEM_swap64(val64));
else
MEM_write64(memPtr, val64);
}
MEM_STATIC size_t MEM_readBEST(const void* memPtr)
{
if (MEM_32bits())
return (size_t)MEM_readBE32(memPtr);
else
return (size_t)MEM_readBE64(memPtr);
}
MEM_STATIC void MEM_writeBEST(void* memPtr, size_t val)
{
if (MEM_32bits())
MEM_writeBE32(memPtr, (U32)val);
else
MEM_writeBE64(memPtr, (U64)val);
}
/* code only tested on 32 and 64 bits systems */
MEM_STATIC void MEM_check(void) { DEBUG_STATIC_ASSERT((sizeof(size_t)==4) || (sizeof(size_t)==8)); }
#endif /* MEM_H_MODULE */
#endif /* USE_EXTERNAL_ZSTD */

374
vendor/github.com/DataDog/zstd/pool.c generated vendored
View File

@@ -1,374 +0,0 @@
#ifndef USE_EXTERNAL_ZSTD
/*
* Copyright (c) Meta Platforms, Inc. and affiliates.
* All rights reserved.
*
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
* You may select, at your option, one of the above-listed licenses.
*/
/* ====== Dependencies ======= */
#include "allocations.h" /* ZSTD_customCalloc, ZSTD_customFree */
#include "zstd_deps.h" /* size_t */
#include "debug.h" /* assert */
#include "pool.h"
/* ====== Compiler specifics ====== */
#if defined(_MSC_VER)
# pragma warning(disable : 4204) /* disable: C4204: non-constant aggregate initializer */
#endif
#ifdef ZSTD_MULTITHREAD
#include "threading.h" /* pthread adaptation */
/* A job is a function and an opaque argument */
typedef struct POOL_job_s {
POOL_function function;
void *opaque;
} POOL_job;
struct POOL_ctx_s {
ZSTD_customMem customMem;
/* Keep track of the threads */
ZSTD_pthread_t* threads;
size_t threadCapacity;
size_t threadLimit;
/* The queue is a circular buffer */
POOL_job *queue;
size_t queueHead;
size_t queueTail;
size_t queueSize;
/* The number of threads working on jobs */
size_t numThreadsBusy;
/* Indicates if the queue is empty */
int queueEmpty;
/* The mutex protects the queue */
ZSTD_pthread_mutex_t queueMutex;
/* Condition variable for pushers to wait on when the queue is full */
ZSTD_pthread_cond_t queuePushCond;
/* Condition variables for poppers to wait on when the queue is empty */
ZSTD_pthread_cond_t queuePopCond;
/* Indicates if the queue is shutting down */
int shutdown;
};
/* POOL_thread() :
* Work thread for the thread pool.
* Waits for jobs and executes them.
* @returns : NULL on failure else non-null.
*/
static void* POOL_thread(void* opaque) {
POOL_ctx* const ctx = (POOL_ctx*)opaque;
if (!ctx) { return NULL; }
for (;;) {
/* Lock the mutex and wait for a non-empty queue or until shutdown */
ZSTD_pthread_mutex_lock(&ctx->queueMutex);
while ( ctx->queueEmpty
|| (ctx->numThreadsBusy >= ctx->threadLimit) ) {
if (ctx->shutdown) {
/* even if !queueEmpty, (possible if numThreadsBusy >= threadLimit),
* a few threads will be shutdown while !queueEmpty,
* but enough threads will remain active to finish the queue */
ZSTD_pthread_mutex_unlock(&ctx->queueMutex);
return opaque;
}
ZSTD_pthread_cond_wait(&ctx->queuePopCond, &ctx->queueMutex);
}
/* Pop a job off the queue */
{ POOL_job const job = ctx->queue[ctx->queueHead];
ctx->queueHead = (ctx->queueHead + 1) % ctx->queueSize;
ctx->numThreadsBusy++;
ctx->queueEmpty = (ctx->queueHead == ctx->queueTail);
/* Unlock the mutex, signal a pusher, and run the job */
ZSTD_pthread_cond_signal(&ctx->queuePushCond);
ZSTD_pthread_mutex_unlock(&ctx->queueMutex);
job.function(job.opaque);
/* If the intended queue size was 0, signal after finishing job */
ZSTD_pthread_mutex_lock(&ctx->queueMutex);
ctx->numThreadsBusy--;
ZSTD_pthread_cond_signal(&ctx->queuePushCond);
ZSTD_pthread_mutex_unlock(&ctx->queueMutex);
}
} /* for (;;) */
assert(0); /* Unreachable */
}
/* ZSTD_createThreadPool() : public access point */
POOL_ctx* ZSTD_createThreadPool(size_t numThreads) {
return POOL_create (numThreads, 0);
}
POOL_ctx* POOL_create(size_t numThreads, size_t queueSize) {
return POOL_create_advanced(numThreads, queueSize, ZSTD_defaultCMem);
}
POOL_ctx* POOL_create_advanced(size_t numThreads, size_t queueSize,
ZSTD_customMem customMem)
{
POOL_ctx* ctx;
/* Check parameters */
if (!numThreads) { return NULL; }
/* Allocate the context and zero initialize */
ctx = (POOL_ctx*)ZSTD_customCalloc(sizeof(POOL_ctx), customMem);
if (!ctx) { return NULL; }
/* Initialize the job queue.
* It needs one extra space since one space is wasted to differentiate
* empty and full queues.
*/
ctx->queueSize = queueSize + 1;
ctx->queue = (POOL_job*)ZSTD_customCalloc(ctx->queueSize * sizeof(POOL_job), customMem);
ctx->queueHead = 0;
ctx->queueTail = 0;
ctx->numThreadsBusy = 0;
ctx->queueEmpty = 1;
{
int error = 0;
error |= ZSTD_pthread_mutex_init(&ctx->queueMutex, NULL);
error |= ZSTD_pthread_cond_init(&ctx->queuePushCond, NULL);
error |= ZSTD_pthread_cond_init(&ctx->queuePopCond, NULL);
if (error) { POOL_free(ctx); return NULL; }
}
ctx->shutdown = 0;
/* Allocate space for the thread handles */
ctx->threads = (ZSTD_pthread_t*)ZSTD_customCalloc(numThreads * sizeof(ZSTD_pthread_t), customMem);
ctx->threadCapacity = 0;
ctx->customMem = customMem;
/* Check for errors */
if (!ctx->threads || !ctx->queue) { POOL_free(ctx); return NULL; }
/* Initialize the threads */
{ size_t i;
for (i = 0; i < numThreads; ++i) {
if (ZSTD_pthread_create(&ctx->threads[i], NULL, &POOL_thread, ctx)) {
ctx->threadCapacity = i;
POOL_free(ctx);
return NULL;
} }
ctx->threadCapacity = numThreads;
ctx->threadLimit = numThreads;
}
return ctx;
}
/*! POOL_join() :
Shutdown the queue, wake any sleeping threads, and join all of the threads.
*/
static void POOL_join(POOL_ctx* ctx) {
/* Shut down the queue */
ZSTD_pthread_mutex_lock(&ctx->queueMutex);
ctx->shutdown = 1;
ZSTD_pthread_mutex_unlock(&ctx->queueMutex);
/* Wake up sleeping threads */
ZSTD_pthread_cond_broadcast(&ctx->queuePushCond);
ZSTD_pthread_cond_broadcast(&ctx->queuePopCond);
/* Join all of the threads */
{ size_t i;
for (i = 0; i < ctx->threadCapacity; ++i) {
ZSTD_pthread_join(ctx->threads[i]); /* note : could fail */
} }
}
void POOL_free(POOL_ctx *ctx) {
if (!ctx) { return; }
POOL_join(ctx);
ZSTD_pthread_mutex_destroy(&ctx->queueMutex);
ZSTD_pthread_cond_destroy(&ctx->queuePushCond);
ZSTD_pthread_cond_destroy(&ctx->queuePopCond);
ZSTD_customFree(ctx->queue, ctx->customMem);
ZSTD_customFree(ctx->threads, ctx->customMem);
ZSTD_customFree(ctx, ctx->customMem);
}
/*! POOL_joinJobs() :
* Waits for all queued jobs to finish executing.
*/
void POOL_joinJobs(POOL_ctx* ctx) {
ZSTD_pthread_mutex_lock(&ctx->queueMutex);
while(!ctx->queueEmpty || ctx->numThreadsBusy > 0) {
ZSTD_pthread_cond_wait(&ctx->queuePushCond, &ctx->queueMutex);
}
ZSTD_pthread_mutex_unlock(&ctx->queueMutex);
}
void ZSTD_freeThreadPool (ZSTD_threadPool* pool) {
POOL_free (pool);
}
size_t POOL_sizeof(const POOL_ctx* ctx) {
if (ctx==NULL) return 0; /* supports sizeof NULL */
return sizeof(*ctx)
+ ctx->queueSize * sizeof(POOL_job)
+ ctx->threadCapacity * sizeof(ZSTD_pthread_t);
}
/* @return : 0 on success, 1 on error */
static int POOL_resize_internal(POOL_ctx* ctx, size_t numThreads)
{
if (numThreads <= ctx->threadCapacity) {
if (!numThreads) return 1;
ctx->threadLimit = numThreads;
return 0;
}
/* numThreads > threadCapacity */
{ ZSTD_pthread_t* const threadPool = (ZSTD_pthread_t*)ZSTD_customCalloc(numThreads * sizeof(ZSTD_pthread_t), ctx->customMem);
if (!threadPool) return 1;
/* replace existing thread pool */
ZSTD_memcpy(threadPool, ctx->threads, ctx->threadCapacity * sizeof(ZSTD_pthread_t));
ZSTD_customFree(ctx->threads, ctx->customMem);
ctx->threads = threadPool;
/* Initialize additional threads */
{ size_t threadId;
for (threadId = ctx->threadCapacity; threadId < numThreads; ++threadId) {
if (ZSTD_pthread_create(&threadPool[threadId], NULL, &POOL_thread, ctx)) {
ctx->threadCapacity = threadId;
return 1;
} }
} }
/* successfully expanded */
ctx->threadCapacity = numThreads;
ctx->threadLimit = numThreads;
return 0;
}
/* @return : 0 on success, 1 on error */
int POOL_resize(POOL_ctx* ctx, size_t numThreads)
{
int result;
if (ctx==NULL) return 1;
ZSTD_pthread_mutex_lock(&ctx->queueMutex);
result = POOL_resize_internal(ctx, numThreads);
ZSTD_pthread_cond_broadcast(&ctx->queuePopCond);
ZSTD_pthread_mutex_unlock(&ctx->queueMutex);
return result;
}
/**
* Returns 1 if the queue is full and 0 otherwise.
*
* When queueSize is 1 (pool was created with an intended queueSize of 0),
* then a queue is empty if there is a thread free _and_ no job is waiting.
*/
static int isQueueFull(POOL_ctx const* ctx) {
if (ctx->queueSize > 1) {
return ctx->queueHead == ((ctx->queueTail + 1) % ctx->queueSize);
} else {
return (ctx->numThreadsBusy == ctx->threadLimit) ||
!ctx->queueEmpty;
}
}
static void
POOL_add_internal(POOL_ctx* ctx, POOL_function function, void *opaque)
{
POOL_job job;
job.function = function;
job.opaque = opaque;
assert(ctx != NULL);
if (ctx->shutdown) return;
ctx->queueEmpty = 0;
ctx->queue[ctx->queueTail] = job;
ctx->queueTail = (ctx->queueTail + 1) % ctx->queueSize;
ZSTD_pthread_cond_signal(&ctx->queuePopCond);
}
void POOL_add(POOL_ctx* ctx, POOL_function function, void* opaque)
{
assert(ctx != NULL);
ZSTD_pthread_mutex_lock(&ctx->queueMutex);
/* Wait until there is space in the queue for the new job */
while (isQueueFull(ctx) && (!ctx->shutdown)) {
ZSTD_pthread_cond_wait(&ctx->queuePushCond, &ctx->queueMutex);
}
POOL_add_internal(ctx, function, opaque);
ZSTD_pthread_mutex_unlock(&ctx->queueMutex);
}
int POOL_tryAdd(POOL_ctx* ctx, POOL_function function, void* opaque)
{
assert(ctx != NULL);
ZSTD_pthread_mutex_lock(&ctx->queueMutex);
if (isQueueFull(ctx)) {
ZSTD_pthread_mutex_unlock(&ctx->queueMutex);
return 0;
}
POOL_add_internal(ctx, function, opaque);
ZSTD_pthread_mutex_unlock(&ctx->queueMutex);
return 1;
}
#else /* ZSTD_MULTITHREAD not defined */
/* ========================== */
/* No multi-threading support */
/* ========================== */
/* We don't need any data, but if it is empty, malloc() might return NULL. */
struct POOL_ctx_s {
int dummy;
};
static POOL_ctx g_poolCtx;
POOL_ctx* POOL_create(size_t numThreads, size_t queueSize) {
return POOL_create_advanced(numThreads, queueSize, ZSTD_defaultCMem);
}
POOL_ctx*
POOL_create_advanced(size_t numThreads, size_t queueSize, ZSTD_customMem customMem)
{
(void)numThreads;
(void)queueSize;
(void)customMem;
return &g_poolCtx;
}
void POOL_free(POOL_ctx* ctx) {
assert(!ctx || ctx == &g_poolCtx);
(void)ctx;
}
void POOL_joinJobs(POOL_ctx* ctx){
assert(!ctx || ctx == &g_poolCtx);
(void)ctx;
}
int POOL_resize(POOL_ctx* ctx, size_t numThreads) {
(void)ctx; (void)numThreads;
return 0;
}
void POOL_add(POOL_ctx* ctx, POOL_function function, void* opaque) {
(void)ctx;
function(opaque);
}
int POOL_tryAdd(POOL_ctx* ctx, POOL_function function, void* opaque) {
(void)ctx;
function(opaque);
return 1;
}
size_t POOL_sizeof(const POOL_ctx* ctx) {
if (ctx==NULL) return 0; /* supports sizeof NULL */
assert(ctx == &g_poolCtx);
return sizeof(*ctx);
}
#endif /* ZSTD_MULTITHREAD */
#endif /* USE_EXTERNAL_ZSTD */

84
vendor/github.com/DataDog/zstd/pool.h generated vendored
View File

@@ -1,84 +0,0 @@
#ifndef USE_EXTERNAL_ZSTD
/*
* Copyright (c) Meta Platforms, Inc. and affiliates.
* All rights reserved.
*
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
* You may select, at your option, one of the above-listed licenses.
*/
#ifndef POOL_H
#define POOL_H
#include "zstd_deps.h"
#define ZSTD_STATIC_LINKING_ONLY /* ZSTD_customMem */
#include "zstd.h"
typedef struct POOL_ctx_s POOL_ctx;
/*! POOL_create() :
* Create a thread pool with at most `numThreads` threads.
* `numThreads` must be at least 1.
* The maximum number of queued jobs before blocking is `queueSize`.
* @return : POOL_ctx pointer on success, else NULL.
*/
POOL_ctx* POOL_create(size_t numThreads, size_t queueSize);
POOL_ctx* POOL_create_advanced(size_t numThreads, size_t queueSize,
ZSTD_customMem customMem);
/*! POOL_free() :
* Free a thread pool returned by POOL_create().
*/
void POOL_free(POOL_ctx* ctx);
/*! POOL_joinJobs() :
* Waits for all queued jobs to finish executing.
*/
void POOL_joinJobs(POOL_ctx* ctx);
/*! POOL_resize() :
* Expands or shrinks pool's number of threads.
* This is more efficient than releasing + creating a new context,
* since it tries to preserve and reuse existing threads.
* `numThreads` must be at least 1.
* @return : 0 when resize was successful,
* !0 (typically 1) if there is an error.
* note : only numThreads can be resized, queueSize remains unchanged.
*/
int POOL_resize(POOL_ctx* ctx, size_t numThreads);
/*! POOL_sizeof() :
* @return threadpool memory usage
* note : compatible with NULL (returns 0 in this case)
*/
size_t POOL_sizeof(const POOL_ctx* ctx);
/*! POOL_function :
* The function type that can be added to a thread pool.
*/
typedef void (*POOL_function)(void*);
/*! POOL_add() :
* Add the job `function(opaque)` to the thread pool. `ctx` must be valid.
* Possibly blocks until there is room in the queue.
* Note : The function may be executed asynchronously,
* therefore, `opaque` must live until function has been completed.
*/
void POOL_add(POOL_ctx* ctx, POOL_function function, void* opaque);
/*! POOL_tryAdd() :
* Add the job `function(opaque)` to thread pool _if_ a queue slot is available.
* Returns immediately even if not (does not block).
* @return : 1 if successful, 0 if not.
*/
int POOL_tryAdd(POOL_ctx* ctx, POOL_function function, void* opaque);
#endif
#endif /* USE_EXTERNAL_ZSTD */

174
vendor/github.com/DataDog/zstd/portability_macros.h generated vendored
View File

@@ -1,174 +0,0 @@
#ifndef USE_EXTERNAL_ZSTD
/*
* Copyright (c) Meta Platforms, Inc. and affiliates.
* All rights reserved.
*
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
* You may select, at your option, one of the above-listed licenses.
*/
#ifndef ZSTD_PORTABILITY_MACROS_H
#define ZSTD_PORTABILITY_MACROS_H
/**
* This header file contains macro definitions to support portability.
* This header is shared between C and ASM code, so it MUST only
* contain macro definitions. It MUST not contain any C code.
*
* This header ONLY defines macros to detect platforms/feature support.
*
*/
/* compat. with non-clang compilers */
#ifndef __has_attribute
#define __has_attribute(x) 0
#endif
/* compat. with non-clang compilers */
#ifndef __has_builtin
# define __has_builtin(x) 0
#endif
/* compat. with non-clang compilers */
#ifndef __has_feature
# define __has_feature(x) 0
#endif
/* detects whether we are being compiled under msan */
#ifndef ZSTD_MEMORY_SANITIZER
# if __has_feature(memory_sanitizer)
# define ZSTD_MEMORY_SANITIZER 1
# else
# define ZSTD_MEMORY_SANITIZER 0
# endif
#endif
/* detects whether we are being compiled under asan */
#ifndef ZSTD_ADDRESS_SANITIZER
# if __has_feature(address_sanitizer)
# define ZSTD_ADDRESS_SANITIZER 1
# elif defined(__SANITIZE_ADDRESS__)
# define ZSTD_ADDRESS_SANITIZER 1
# else
# define ZSTD_ADDRESS_SANITIZER 0
# endif
#endif
/* detects whether we are being compiled under dfsan */
#ifndef ZSTD_DATAFLOW_SANITIZER
# if __has_feature(dataflow_sanitizer)
# define ZSTD_DATAFLOW_SANITIZER 1
# else
# define ZSTD_DATAFLOW_SANITIZER 0
# endif
#endif
/* Mark the internal assembly functions as hidden */
#ifdef __ELF__
# define ZSTD_HIDE_ASM_FUNCTION(func) .hidden func
#elif defined(__APPLE__)
# define ZSTD_HIDE_ASM_FUNCTION(func) .private_extern func
#else
# define ZSTD_HIDE_ASM_FUNCTION(func)
#endif
/* Compile time determination of BMI2 support */
#ifndef STATIC_BMI2
# if defined(__BMI2__)
# define STATIC_BMI2 1
# elif defined(_MSC_VER) && defined(__AVX2__)
# define STATIC_BMI2 1 /* MSVC does not have a BMI2 specific flag, but every CPU that supports AVX2 also supports BMI2 */
# endif
#endif
#ifndef STATIC_BMI2
# define STATIC_BMI2 0
#endif
/* Enable runtime BMI2 dispatch based on the CPU.
* Enabled for clang & gcc >=4.8 on x86 when BMI2 isn't enabled by default.
*/
#ifndef DYNAMIC_BMI2
# if ((defined(__clang__) && __has_attribute(__target__)) \
|| (defined(__GNUC__) \
&& (__GNUC__ >= 5 || (__GNUC__ == 4 && __GNUC_MINOR__ >= 8)))) \
&& (defined(__i386__) || defined(__x86_64__) || defined(_M_IX86) || defined(_M_X64)) \
&& !defined(__BMI2__)
# define DYNAMIC_BMI2 1
# else
# define DYNAMIC_BMI2 0
# endif
#endif
/**
* Only enable assembly for GNU C compatible compilers,
* because other platforms may not support GAS assembly syntax.
*
* Only enable assembly for Linux / MacOS / Win32, other platforms may
* work, but they haven't been tested. This could likely be
* extended to BSD systems.
*
* Disable assembly when MSAN is enabled, because MSAN requires
* 100% of code to be instrumented to work.
*/
#if defined(__GNUC__)
# if defined(__linux__) || defined(__linux) || defined(__APPLE__) || defined(_WIN32)
# if ZSTD_MEMORY_SANITIZER
# define ZSTD_ASM_SUPPORTED 0
# elif ZSTD_DATAFLOW_SANITIZER
# define ZSTD_ASM_SUPPORTED 0
# else
# define ZSTD_ASM_SUPPORTED 1
# endif
# else
# define ZSTD_ASM_SUPPORTED 0
# endif
#else
# define ZSTD_ASM_SUPPORTED 0
#endif
/**
* Determines whether we should enable assembly for x86-64
* with BMI2.
*
* Enable if all of the following conditions hold:
* - ASM hasn't been explicitly disabled by defining ZSTD_DISABLE_ASM
* - Assembly is supported
* - We are compiling for x86-64 and either:
* - DYNAMIC_BMI2 is enabled
* - BMI2 is supported at compile time
*/
#if !defined(ZSTD_DISABLE_ASM) && \
ZSTD_ASM_SUPPORTED && \
defined(__x86_64__) && \
(DYNAMIC_BMI2 || defined(__BMI2__))
# define ZSTD_ENABLE_ASM_X86_64_BMI2 1
#else
# define ZSTD_ENABLE_ASM_X86_64_BMI2 0
#endif
/*
* For x86 ELF targets, add .note.gnu.property section for Intel CET in
* assembly sources when CET is enabled.
*
* Additionally, any function that may be called indirectly must begin
* with ZSTD_CET_ENDBRANCH.
*/
#if defined(__ELF__) && (defined(__x86_64__) || defined(__i386__)) \
&& defined(__has_include)
# if __has_include(<cet.h>)
# include <cet.h>
# define ZSTD_CET_ENDBRANCH _CET_ENDBR
# endif
#endif
#ifndef ZSTD_CET_ENDBRANCH
# define ZSTD_CET_ENDBRANCH
#endif
#endif /* ZSTD_PORTABILITY_MACROS_H */
#endif /* USE_EXTERNAL_ZSTD */

185
vendor/github.com/DataDog/zstd/threading.c generated vendored
View File

@@ -1,185 +0,0 @@
#ifndef USE_EXTERNAL_ZSTD
/**
* Copyright (c) 2016 Tino Reichardt
* All rights reserved.
*
* You can contact the author at:
* - zstdmt source repository: https://github.com/mcmilk/zstdmt
*
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
* You may select, at your option, one of the above-listed licenses.
*/
/**
* This file will hold wrapper for systems, which do not support pthreads
*/
#include "threading.h"
/* create fake symbol to avoid empty translation unit warning */
int g_ZSTD_threading_useless_symbol;
#if defined(ZSTD_MULTITHREAD) && defined(_WIN32)
/**
* Windows minimalist Pthread Wrapper
*/
/* === Dependencies === */
#include <process.h>
#include <errno.h>
/* === Implementation === */
typedef struct {
void* (*start_routine)(void*);
void* arg;
int initialized;
ZSTD_pthread_cond_t initialized_cond;
ZSTD_pthread_mutex_t initialized_mutex;
} ZSTD_thread_params_t;
static unsigned __stdcall worker(void *arg)
{
void* (*start_routine)(void*);
void* thread_arg;
/* Initialized thread_arg and start_routine and signal main thread that we don't need it
* to wait any longer.
*/
{
ZSTD_thread_params_t* thread_param = (ZSTD_thread_params_t*)arg;
thread_arg = thread_param->arg;
start_routine = thread_param->start_routine;
/* Signal main thread that we are running and do not depend on its memory anymore */
ZSTD_pthread_mutex_lock(&thread_param->initialized_mutex);
thread_param->initialized = 1;
ZSTD_pthread_cond_signal(&thread_param->initialized_cond);
ZSTD_pthread_mutex_unlock(&thread_param->initialized_mutex);
}
start_routine(thread_arg);
return 0;
}
int ZSTD_pthread_create(ZSTD_pthread_t* thread, const void* unused,
void* (*start_routine) (void*), void* arg)
{
ZSTD_thread_params_t thread_param;
(void)unused;
if (thread==NULL) return -1;
*thread = NULL;
thread_param.start_routine = start_routine;
thread_param.arg = arg;
thread_param.initialized = 0;
/* Setup thread initialization synchronization */
if(ZSTD_pthread_cond_init(&thread_param.initialized_cond, NULL)) {
/* Should never happen on Windows */
return -1;
}
if(ZSTD_pthread_mutex_init(&thread_param.initialized_mutex, NULL)) {
/* Should never happen on Windows */
ZSTD_pthread_cond_destroy(&thread_param.initialized_cond);
return -1;
}
/* Spawn thread */
*thread = (HANDLE)_beginthreadex(NULL, 0, worker, &thread_param, 0, NULL);
if (*thread==NULL) {
ZSTD_pthread_mutex_destroy(&thread_param.initialized_mutex);
ZSTD_pthread_cond_destroy(&thread_param.initialized_cond);
return errno;
}
/* Wait for thread to be initialized */
ZSTD_pthread_mutex_lock(&thread_param.initialized_mutex);
while(!thread_param.initialized) {
ZSTD_pthread_cond_wait(&thread_param.initialized_cond, &thread_param.initialized_mutex);
}
ZSTD_pthread_mutex_unlock(&thread_param.initialized_mutex);
ZSTD_pthread_mutex_destroy(&thread_param.initialized_mutex);
ZSTD_pthread_cond_destroy(&thread_param.initialized_cond);
return 0;
}
int ZSTD_pthread_join(ZSTD_pthread_t thread)
{
DWORD result;
if (!thread) return 0;
result = WaitForSingleObject(thread, INFINITE);
CloseHandle(thread);
switch (result) {
case WAIT_OBJECT_0:
return 0;
case WAIT_ABANDONED:
return EINVAL;
default:
return GetLastError();
}
}
#endif /* ZSTD_MULTITHREAD */
#if defined(ZSTD_MULTITHREAD) && DEBUGLEVEL >= 1 && !defined(_WIN32)
#define ZSTD_DEPS_NEED_MALLOC
#include "zstd_deps.h"
int ZSTD_pthread_mutex_init(ZSTD_pthread_mutex_t* mutex, pthread_mutexattr_t const* attr)
{
assert(mutex != NULL);
*mutex = (pthread_mutex_t*)ZSTD_malloc(sizeof(pthread_mutex_t));
if (!*mutex)
return 1;
return pthread_mutex_init(*mutex, attr);
}
int ZSTD_pthread_mutex_destroy(ZSTD_pthread_mutex_t* mutex)
{
assert(mutex != NULL);
if (!*mutex)
return 0;
{
int const ret = pthread_mutex_destroy(*mutex);
ZSTD_free(*mutex);
return ret;
}
}
int ZSTD_pthread_cond_init(ZSTD_pthread_cond_t* cond, pthread_condattr_t const* attr)
{
assert(cond != NULL);
*cond = (pthread_cond_t*)ZSTD_malloc(sizeof(pthread_cond_t));
if (!*cond)
return 1;
return pthread_cond_init(*cond, attr);
}
int ZSTD_pthread_cond_destroy(ZSTD_pthread_cond_t* cond)
{
assert(cond != NULL);
if (!*cond)
return 0;
{
int const ret = pthread_cond_destroy(*cond);
ZSTD_free(*cond);
return ret;
}
}
#endif
#endif /* USE_EXTERNAL_ZSTD */

145
vendor/github.com/DataDog/zstd/threading.h generated vendored
View File

@@ -1,145 +0,0 @@
#ifndef USE_EXTERNAL_ZSTD
/**
* Copyright (c) 2016 Tino Reichardt
* All rights reserved.
*
* You can contact the author at:
* - zstdmt source repository: https://github.com/mcmilk/zstdmt
*
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
* You may select, at your option, one of the above-listed licenses.
*/
#ifndef THREADING_H_938743
#define THREADING_H_938743
#include "debug.h"
#if defined(ZSTD_MULTITHREAD) && defined(_WIN32)
/**
* Windows minimalist Pthread Wrapper
*/
#ifdef WINVER
# undef WINVER
#endif
#define WINVER 0x0600
#ifdef _WIN32_WINNT
# undef _WIN32_WINNT
#endif
#define _WIN32_WINNT 0x0600
#ifndef WIN32_LEAN_AND_MEAN
# define WIN32_LEAN_AND_MEAN
#endif
#undef ERROR /* reported already defined on VS 2015 (Rich Geldreich) */
#include <windows.h>
#undef ERROR
#define ERROR(name) ZSTD_ERROR(name)
/* mutex */
#define ZSTD_pthread_mutex_t CRITICAL_SECTION
#define ZSTD_pthread_mutex_init(a, b) ((void)(b), InitializeCriticalSection((a)), 0)
#define ZSTD_pthread_mutex_destroy(a) DeleteCriticalSection((a))
#define ZSTD_pthread_mutex_lock(a) EnterCriticalSection((a))
#define ZSTD_pthread_mutex_unlock(a) LeaveCriticalSection((a))
/* condition variable */
#define ZSTD_pthread_cond_t CONDITION_VARIABLE
#define ZSTD_pthread_cond_init(a, b) ((void)(b), InitializeConditionVariable((a)), 0)
#define ZSTD_pthread_cond_destroy(a) ((void)(a))
#define ZSTD_pthread_cond_wait(a, b) SleepConditionVariableCS((a), (b), INFINITE)
#define ZSTD_pthread_cond_signal(a) WakeConditionVariable((a))
#define ZSTD_pthread_cond_broadcast(a) WakeAllConditionVariable((a))
/* ZSTD_pthread_create() and ZSTD_pthread_join() */
typedef HANDLE ZSTD_pthread_t;
int ZSTD_pthread_create(ZSTD_pthread_t* thread, const void* unused,
void* (*start_routine) (void*), void* arg);
int ZSTD_pthread_join(ZSTD_pthread_t thread);
/**
* add here more wrappers as required
*/
#elif defined(ZSTD_MULTITHREAD) /* posix assumed ; need a better detection method */
/* === POSIX Systems === */
# include <pthread.h>
#if DEBUGLEVEL < 1
#define ZSTD_pthread_mutex_t pthread_mutex_t
#define ZSTD_pthread_mutex_init(a, b) pthread_mutex_init((a), (b))
#define ZSTD_pthread_mutex_destroy(a) pthread_mutex_destroy((a))
#define ZSTD_pthread_mutex_lock(a) pthread_mutex_lock((a))
#define ZSTD_pthread_mutex_unlock(a) pthread_mutex_unlock((a))
#define ZSTD_pthread_cond_t pthread_cond_t
#define ZSTD_pthread_cond_init(a, b) pthread_cond_init((a), (b))
#define ZSTD_pthread_cond_destroy(a) pthread_cond_destroy((a))
#define ZSTD_pthread_cond_wait(a, b) pthread_cond_wait((a), (b))
#define ZSTD_pthread_cond_signal(a) pthread_cond_signal((a))
#define ZSTD_pthread_cond_broadcast(a) pthread_cond_broadcast((a))
#define ZSTD_pthread_t pthread_t
#define ZSTD_pthread_create(a, b, c, d) pthread_create((a), (b), (c), (d))
#define ZSTD_pthread_join(a) pthread_join((a),NULL)
#else /* DEBUGLEVEL >= 1 */
/* Debug implementation of threading.
* In this implementation we use pointers for mutexes and condition variables.
* This way, if we forget to init/destroy them the program will crash or ASAN
* will report leaks.
*/
#define ZSTD_pthread_mutex_t pthread_mutex_t*
int ZSTD_pthread_mutex_init(ZSTD_pthread_mutex_t* mutex, pthread_mutexattr_t const* attr);
int ZSTD_pthread_mutex_destroy(ZSTD_pthread_mutex_t* mutex);
#define ZSTD_pthread_mutex_lock(a) pthread_mutex_lock(*(a))
#define ZSTD_pthread_mutex_unlock(a) pthread_mutex_unlock(*(a))
#define ZSTD_pthread_cond_t pthread_cond_t*
int ZSTD_pthread_cond_init(ZSTD_pthread_cond_t* cond, pthread_condattr_t const* attr);
int ZSTD_pthread_cond_destroy(ZSTD_pthread_cond_t* cond);
#define ZSTD_pthread_cond_wait(a, b) pthread_cond_wait(*(a), *(b))
#define ZSTD_pthread_cond_signal(a) pthread_cond_signal(*(a))
#define ZSTD_pthread_cond_broadcast(a) pthread_cond_broadcast(*(a))
#define ZSTD_pthread_t pthread_t
#define ZSTD_pthread_create(a, b, c, d) pthread_create((a), (b), (c), (d))
#define ZSTD_pthread_join(a) pthread_join((a),NULL)
#endif
#else /* ZSTD_MULTITHREAD not defined */
/* No multithreading support */
typedef int ZSTD_pthread_mutex_t;
#define ZSTD_pthread_mutex_init(a, b) ((void)(a), (void)(b), 0)
#define ZSTD_pthread_mutex_destroy(a) ((void)(a))
#define ZSTD_pthread_mutex_lock(a) ((void)(a))
#define ZSTD_pthread_mutex_unlock(a) ((void)(a))
typedef int ZSTD_pthread_cond_t;
#define ZSTD_pthread_cond_init(a, b) ((void)(a), (void)(b), 0)
#define ZSTD_pthread_cond_destroy(a) ((void)(a))
#define ZSTD_pthread_cond_wait(a, b) ((void)(a), (void)(b))
#define ZSTD_pthread_cond_signal(a) ((void)(a))
#define ZSTD_pthread_cond_broadcast(a) ((void)(a))
/* do not use ZSTD_pthread_t */
#endif /* ZSTD_MULTITHREAD */
#endif /* THREADING_H_938743 */
#endif /* USE_EXTERNAL_ZSTD */

19
vendor/github.com/DataDog/zstd/travis_test_32.sh generated vendored
View File

@@ -1,19 +0,0 @@
#!/bin/bash
# Get utilities
#yum -y -q -e 0 install wget tar unzip gcc
apt-get update
apt-get -y install wget tar unzip gcc
# Get Go
wget -q https://dl.google.com/go/go1.13.linux-386.tar.gz
tar -C /usr/local -xzf go1.13.linux-386.tar.gz
export PATH=$PATH:/usr/local/go/bin
# Get payload
wget -q https://github.com/DataDog/zstd/files/2246767/mr.zip
unzip mr.zip
# Build and run tests
go build
DISABLE_BIG_TESTS=1 PAYLOAD=$(pwd)/mr go test -v
DISABLE_BIG_TESTS=1 PAYLOAD=$(pwd)/mr go test -bench .

56
vendor/github.com/DataDog/zstd/update.txt generated vendored
View File

@@ -1,56 +0,0 @@
./lib/common/bitstream.h
./lib/common/compiler.h
./lib/compress/zstd_compress_internal.h
./lib/compress/zstd_fast.h
./lib/compress/zstd_double_fast.h
./lib/compress/zstd_lazy.h
./lib/compress/zstd_ldm.h
./lib/dictBuilder/cover.c
./lib/dictBuilder/divsufsort.c
./lib/dictBuilder/divsufsort.h
./lib/common/entropy_common.c
./lib/common/error_private.c
./lib/common/error_private.h
./lib/compress/fse_compress.c
./lib/common/fse_decompress.c
./lib/common/fse.h
./lib/compress/huf_compress.c
./lib/decompress/huf_decompress.c
./lib/common/huf.h
./lib/common/mem.h
./lib/common/pool.c
./lib/common/pool.h
./lib/common/threading.c
./lib/common/threading.h
./lib/common/xxhash.c
./lib/common/xxhash.h
./lib/deprecated/zbuff_common.c
./lib/deprecated/zbuff_compress.c
./lib/deprecated/zbuff_decompress.c
./lib/deprecated/zbuff.h
./lib/dictBuilder/zdict.c
./lib/dictBuilder/zdict.h
./lib/common/zstd_common.c
./lib/compress/zstd_compress.c
./lib/decompress/zstd_decompress.c
./lib/common/zstd_errors.h
./lib/zstd.h
./lib/common/zstd_internal.h
./lib/legacy/zstd_legacy.h
./lib/compress/zstd_opt.c
./lib/compress/zstd_opt.h
./lib/legacy/zstd_v01.c
./lib/legacy/zstd_v01.h
./lib/legacy/zstd_v02.c
./lib/legacy/zstd_v02.h
./lib/legacy/zstd_v03.c
./lib/legacy/zstd_v03.h
./lib/legacy/zstd_v04.c
./lib/legacy/zstd_v04.h
./lib/legacy/zstd_v05.c
./lib/legacy/zstd_v05.h
./lib/legacy/zstd_v06.c
./lib/legacy/zstd_v06.h
./lib/legacy/zstd_v07.c
./lib/legacy/zstd_v07.h

Some files were not shown because too many files have changed in this diff Show More