Global code ovehaul - remove CAS and other, add 15 minute greet check

- Remove CAS, as it no longer works for us
- Remove other checks
- Implement 15 minute greet timeout check
- Fix chat logos
- Ditch thirdparty logger, use stdlib log facility wrapper for logging

.golangci.yml

@@ -2,7 +2,7 @@ version: "2"
 run:
   go: "1.24"
   modules-download-mode: readonly
-  tests: false
+  tests: true
   allow-parallel-runners: true
 linters:
   default: none

DESIGN.txt

@@ -1,32 +0,0 @@
-# How it works.
-------------------------------------------------------------------------------------------------------------------------
-
-Autoban uses 2 methods of matching spammers.
-
-1. by querying CAS on member entry or change status from left-to-member.
-2. by checking events, collected after 10 seconds since member enters chat.
-------------------------------------------------------------------------------------------------------------------------
-
-1.a Events, from new_chat_members being checked versus CAS and if check is positive user being banned.
-
-1.b Events chat_members_updated with fields new_chat_member.status = member and old_chat_member.status = left being
-    checked versus CAS and if check is positive user being banned.
-
-1.c If user was not banned due to CAS check immediately, it will be checked against CAS second time after 10 minutes.
-
-2.a Uids from message events, from new_chat_members fields stored into NewMembers collection alongside with timestamps
-    for each user id.
-
-2.b Uids from chat_members_updated with fields new_chat_member.status = member and old_chat_member.status = left are
-    stored into AppearedUsers collection alongside with timestamps for each user id
-
-2.c Every 10 seconds goroutine runAndBan rakes all events stored in NewMembers and AppearedUsers and compares them.
-    First it tosses off pairs of events with matching user ids. Then it searches in AppearedUsers events older than 10
-    seconds since current time and bans them (also puts them into SquashedMembers collection to avoid banning of banned
-    users). Then it iterates over NewMembers and cleans all events older than 15 seconds.
-
-
-# Note on chat logging.
-------------------------------------------------------------------------------------------------------------------------
-
-Since logging done in synchronous manner (to avoid too many opened descriptors) it can be bottleneck.

README.md

@@ -1,31 +1,75 @@
 # Bunny telegram bot
 
-Yet another telegram bot. It has one purpose - restrict access of abusers to chat.
+Ещё один телеграм-бот.
 
-By now it uses [Combot Anti-Spam System](https://cas.chat/) to detect spammers and its own method of detection of bad
+Его задача - банить спаммеров. Обнаружение спаммеров работает единственным способом - если вошедший не отвечает на
-users.
+приветствие в течение 15 минут, он попадает в бан.
 
-## Build instructions
+## Как это собрать?
 
-Bot is not done yet, actually it does not do anything useful.
+Понадобится компилятор go версии 1.24 или новее. Воодим в каталоге с исходниками команду
 
 ```bash
 make
 ```
 
-## Run instructions
+и ждём. Через некоторое время в каталоге должен появиться бмнапник bunny-bot или bunny-bot.exe - это и есть весь наш бот.
 
-You probably don't need it in its current state. But if you want - all is simple. Visit data/config.json, edit for your
+## Как это запустить
-needs and run:
+
+Надо создать каталог `data/config.json` рядом с бинарником в качестве примера можно использовать `data/config-example.json`:
+
+```json
+{
+  "token": "get_it_from_botfather",
+  "loglevel": "debug",
+  "logfile": "/var/log/bunny-bot/bunny-bot.log",
+  "chat_logs": "chat_logs"
+}
+```
+
+Затем запускаем сам бинарник:
 
 ```bash
 ./bunny-bot
 ```
 
-There is openrc init script in contrib directory, you may use it for creating service under alpine linux or
+Путь до конфига захардкожен и это `<binary dir>/data/config.json`.
-gentoo(untested) with openrc init.
 
-That's it.
+### Логгирование чатов
+
+Параметр `chat_logs` опционален и, если он есть и целевой каталог существует и доступен для записи, бот будет логгировать события в чате.
+
+События из чата группируются по дням:
+
+```text
+<chat_logs>/<chat id>/<yyyy>/<MM>/<dd>.txt
+```
+
+И каждая строка выглядит примерно так:
+
+```text
+21:33:07.123 in chat -1001234567890 (title=Bunny Fans username=@bunnybot) user 42 (first_name=Ann last_name=Smith username=@ann) says hello there
+21:33:07.123 in chat -1001234567890 (title=Bunny Fans username=@bunnybot) user 42 (first_name=Ann last_name=Smith username=@ann) post video (file_name=clip.mp4 mime=video/mp4 1280x720 duration=15s size=1048576) says look at this reply_to=1
+21:33:07.123 in chat -1001234567890 (title=Bunny Fans username=@bunnybot) user 42 (first_name=Ann) user 7 (first_name=Al) joined the chat
+21:33:07.123 in chat -1001234567890 (title=Bunny Fans) user 42 (first_name=Ann) status member -> administrator custom_title=Mod by user 7 (first_name=Bob)
+```
+
+Временные метки во всех логах проставляются по часовому поясу GMT+0.
+
+### Дополнительные файлы
+
+в каталоге `contrib` есть пример openrc-скрипта для запуска бота в среде alpine linux.
+
+### Signals
+
+| Signal | Effect |
+|---|---|
+| `SIGINT`, `SIGTERM`, `SIGQUIT` | Аккуратоное запершение работы - закрываются кэши, логи и затем программа завершается. |
+| `SIGHUP`, `SIGUSR1` | Переоткрывает лог-файлы, для ротации логов. Процесс продолжает работать. |
+
+N.B.
+Бот не откладывает PID-файла, используйте copy-truncate в logrotate.
 
 ## Special thanks
 
@@ -33,16 +77,10 @@ To all active users of chat :)
 
 To maintainers of telegram bot lib [https://github.com/NicoNex/echotron](https://github.com/NicoNex/echotron)
 
-To maintainers of pebble db [https://github.com/cockroachdb/pebble](https://github.com/cockroachdb/pebble)
-
-To maintainers of zap logging facility [https://github.com/uber-go/zap](https://github.com/uber-go/zap)
-
 And kind hello to maintainer of spew dumper [https://github.com/davecgh/go-spew](https://github.com/davecgh/go-spew)
 
 Without you, folks, all this bot-thing become very boring adventure.
 
-## Maybe some more mumble-jumble
+## AI Slop warning
 
-Yes, but next time. When bot will be ready for some real action.
+Последние изменения в боте сделаны с помощью ai.
-
-For now I can say that bot does not support any of commands.

TODO.txt

@@ -1,14 +0,0 @@
-Before release we have to do next features:
-
-[ ] option to enable/disable cas api
-[ ] option to enable/disable generic spammer detection
-[ ] command to leave chat
-[ ] command to mute user
-[ ] command to unmute user
-[ ] command to ban user
-[ ] command to unban user
-[ ] add ability to show software version
-[ ] make use of sailor moon quotes :)
-[ ] implement hello phrase with 15 minute timeout. if user does not answer, ban it
-[ ] ban people with first message is video or picture/image/photo.
-[ ] replace zap with zerolog, to use only one facility in all places.

cmd/bunny-bot/main.go

@@ -1,87 +1,83 @@
 package main
 
 import (
-	"bunny-bot/internal/moon"
 	"fmt"
 	"os"
-	"os/signal"
 	"path/filepath"
-	"syscall"
-	"time"
 
-	"go.uber.org/zap"
+	"bunny-bot/internal/moon"
-	"go.uber.org/zap/zapcore"
+	"bunny-bot/internal/mylog"
 )
 
-// TODO: use https://dev.to/gholami1313/saving-log-messages-to-a-custom-log-file-in-golang-ce5 for
+// main decides the process's fate and nothing else: run the bot, and decide the
+// exit status from how it ended. An error leaves 1; a signal-driven stop leaves
+// 128 + the signal's number (130 for Ctrl+C on this platform) so a supervisor can
+// tell a signal stop from a clean one, which is what D33 fixed; a stop nobody
+// signalled leaves 0.
 //
-//	chat messages.
+// os.Exit is called here and nowhere else, deliberately. A worker goroutine cannot
+// set an exit status without ending the process, which skips every deferred
+// cleanup on the way out — including the one that closes the log we would be
+// complaining about.
 func main() {
-	var err error
+	if err := run(); err != nil {
+		// The logs may already be closed, so stderr is the one channel left that
+		// is certain to be attached.
+		fmt.Fprintf(os.Stderr, "bunny-bot: %v\n", err)
 
+		os.Exit(1)
+	}
+
+	os.Exit(moon.ExitCode())
+}
+
+// run is the whole lifecycle, so that every deferred cleanup belongs to a function
+// that returns instead of one that exits.
+//
+// Its single defer covers the bot's entire life: once the log is open, nothing
+// after that point returns early.
+func run() error {
 	// Main config.
 	executablePath, err := os.Executable()
-
 	if err != nil {
-		panic(fmt.Sprintf("Unable to get current executable path: %s", err))
+		return fmt.Errorf("unable to get current executable path: %w", err)
 	}
 
 	configJSONPath := fmt.Sprintf("%s/data/config.json", filepath.Dir(executablePath))
 
 	moon.Cfg, err = moon.ParseConfig(configJSONPath)
-
 	if err != nil {
-		panic(err)
+		return fmt.Errorf("unable to read the config: %w", err)
 	}
 
-	// Config for Zap logger facility.
+	// Open the log. ParseConfig has already coerced an unknown level to "info",
-	var logCfg = zap.NewDevelopmentConfig()
+	// so an error here means the file itself could not be opened — which is
-
+	// worth surfacing loudly rather than continuing with no record of anything.
-	// Define loglevel.
+	if err := mylog.Init(moon.Cfg.LogLevel, moon.Cfg.LogFile); err != nil {
-	var l zapcore.Level
+		return fmt.Errorf("unable to open the log: %w", err)
-
-	if err := l.UnmarshalText([]byte(moon.Cfg.LogLevel)); err != nil {
-		panic("Unknown LogLevel")
 	}
 
-	logCfg.Level = zap.NewAtomicLevelAt(l)
+	defer func() { _ = mylog.Close() }()
 
-	// Define output file if set in config.
+	mylog.Info("Config read looks good")
-	if moon.Cfg.LogFile != "" {
-		logCfg.OutputPaths[0] = moon.Cfg.LogFile
-	}
-
-	// We do not want to see caller in logs.
-	logCfg.DisableCaller = true
-
-	// We want human-readable timestamp in logs.
-	logCfg.EncoderConfig.EncodeTime = zapcore.TimeEncoderOfLayout(time.DateTime)
-
-	logger := zap.Must(logCfg.Build())
-	moon.Log = logger.Sugar()
-
-	mystdlogger := zap.RedirectStdLog(logger)
-	defer mystdlogger()
-
-	moon.Log.Info("Config read looks good")
 
 	// Самое время поставить траппер сигналов.
-	signal.Notify(moon.SigChan,
+	moon.InstallSignalHandler()
-		syscall.SIGINT,
-		syscall.SIGTERM,
-		syscall.SIGQUIT)
 
-	go moon.SigHandler()
+	mylog.Info("Signal handler installed")
 
-	moon.Log.Info("Singnal handler installed")
+	mylog.Info("Starting main loop")
-
-	moon.Log.Info("Starting RunAndBan() periodic task")
-
-	go moon.RunAndBan()
-
-	moon.Log.Info("Starting main loop")
 
+	// Telegram assigns moon.Tg on this goroutine; the sweeper reads it, so the
+	// client must be constructed before any goroutine that can call the API.
 	moon.Telegram(moon.Cfg)
+
+	// The update listener has returned, but that is not the same as a finished
+	// shutdown: a signal handler may be mid-teardown right now. Shutdown blocks
+	// until the ordered sequence has completed, whichever goroutine started it,
+	// so the process cannot exit with the log half-closed — and its error is what
+	// decides the exit status.
+	return moon.Shutdown()
 }
 
 /* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

data/config-example.json

@@ -1,9 +1,6 @@
 {
-	"name": "Bunny",
 	"token": "get_it_from_botfather",
 	"loglevel": "debug",
-        "logfile": "bunny-bot.log",
+	"logfile": "/var/log/bunny-bot/bunny-bot.log",
-        "logchats": false,
+	"chat_logs": "chat_logs"
-	"csign": ".",
-	"datadir": "data"
 }

data/quotes.json

@@ -1,21 +0,0 @@
-{
-"Тот, кто не знает силу пешек, не имеет права ходить королевой.",
-"Когда ты пройдешь сквозь мрак беспомощности и попадешь в мир надежды, тогда ты преодолеешь все преграды.",
-"По-моему, тебе очень хочется стать уткой по-пекински!",
-"If a soldier’s pride means hurting one another, I don’t want it.",
-"Without the bad times, we wouldn’t appreciate the good times.",
-"I’ll just warn you now, I don’t know how to use a computer.",
-"Thinking isn’t going to get us anywhere.",
-"Вместо того, чтобы произносить речи, лучше бы подумала, что ты будешь делать дальше.",
-"Да как ты смела разгадать мой замысел?! Я же целую ночь не спала, пока его придумала. Верни мне мой сон!",
-"Не волнуйся, жизнь намного скучнее, чем ты себе её представляешь.",
-"Я бы завела котёнка, но у меня уже есть Луна и она жрёт как лошадь.",
-"Звезда нашей надежды никогда не исчезнет. Пока она горит, всё будет в порядке. Мы не сможем проиграть",
-"Голова с хвостиками… Не вмешивайся в наши дела! Играть в войну — это опасно",
-"I don’t have a short attention span, I just… oh look, something shiny!",
-"When I’m with you, I feel like I can conquer anything.",
-"Moon Prism Power, Make Up!",
-"In the name of the moon, I will punish you!",
-"I’m not just some pretty face! I’m also the soldier of love and justice, Sailor Moon!",
-"Don’t let anyone dim your light. Shine bright like the moon and stars."
-}

go.mod

@@ -4,48 +4,14 @@ go 1.24.3
 
 tool (
 	bunny-bot/cmd/bunny-bot
-	bunny-bot/internal/collection
 	bunny-bot/internal/moon
 )
 
 require (
-	github.com/NicoNex/echotron/v3 v3.43.0
+	github.com/NicoNex/echotron/v3 v3.46.0
 	github.com/akyoto/cache v1.0.6
-	github.com/cockroachdb/errors v1.12.0
-	github.com/cockroachdb/pebble v1.1.5
 	github.com/davecgh/go-spew v1.1.1
 	github.com/hjson/hjson-go v3.3.0+incompatible
-	go.uber.org/zap v1.27.0
 )
 
-require (
+require golang.org/x/time v0.12.0 // indirect
-	github.com/DataDog/zstd v1.5.7 // indirect
-	github.com/beorn7/perks v1.0.1 // indirect
-	github.com/cespare/xxhash/v2 v2.3.0 // indirect
-	github.com/cockroachdb/fifo v0.0.0-20240816210425-c5d0cb0b6fc0 // indirect
-	github.com/cockroachdb/logtags v0.0.0-20241215232642-bb51bb14a506 // indirect
-	github.com/cockroachdb/redact v1.1.6 // indirect
-	github.com/cockroachdb/tokenbucket v0.0.0-20250429170803-42689b6311bb // indirect
-	github.com/getsentry/sentry-go v0.33.0 // indirect
-	github.com/gogo/protobuf v1.3.2 // indirect
-	github.com/golang/snappy v1.0.0 // indirect
-	github.com/klauspost/compress v1.18.0 // indirect
-	github.com/kr/pretty v0.3.1 // indirect
-	github.com/kr/text v0.2.0 // indirect
-	github.com/mattn/go-colorable v0.1.13 // indirect
-	github.com/mattn/go-isatty v0.0.19 // indirect
-	github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
-	github.com/pkg/errors v0.9.1 // indirect
-	github.com/prometheus/client_golang v1.22.0 // indirect
-	github.com/prometheus/client_model v0.6.2 // indirect
-	github.com/prometheus/common v0.64.0 // indirect
-	github.com/prometheus/procfs v0.16.1 // indirect
-	github.com/rogpeppe/go-internal v1.14.1 // indirect
-	github.com/rs/zerolog v1.34.0 // indirect
-	go.uber.org/multierr v1.11.0 // indirect
-	golang.org/x/exp v0.0.0-20250531010427-b6e5de432a8b // indirect
-	golang.org/x/sys v0.33.0 // indirect
-	golang.org/x/text v0.26.0 // indirect
-	golang.org/x/time v0.12.0 // indirect
-	google.golang.org/protobuf v1.36.6 // indirect
-)

go.sum

@@ -1,129 +1,10 @@
-github.com/DataDog/zstd v1.5.7 h1:ybO8RBeh29qrxIhCA9E8gKY6xfONU9T6G6aP9DTKfLE=
+github.com/NicoNex/echotron/v3 v3.46.0 h1:IHPhEDU3pbxa3uLnxRz5e247SYEVzIbQS014qvuWFOU=
-github.com/DataDog/zstd v1.5.7/go.mod h1:g4AWEaM3yOg3HYfnJ3YIawPnVdXJh9QME85blwSAmyw=
+github.com/NicoNex/echotron/v3 v3.46.0/go.mod h1:7LvjveJmezuUOeaoA3nzQduNlSPQYfq219Z+baKY04Q=
-github.com/NicoNex/echotron/v3 v3.43.0 h1:efE2spw3mfU0Ev20m0PqqvgMSm0xHzgSxlWAEmC9RC4=
-github.com/NicoNex/echotron/v3 v3.43.0/go.mod h1:7LvjveJmezuUOeaoA3nzQduNlSPQYfq219Z+baKY04Q=
 github.com/akyoto/cache v1.0.6 h1:5XGVVYoi2i+DZLLPuVIXtsNIJ/qaAM16XT0LaBaXd2k=
 github.com/akyoto/cache v1.0.6/go.mod h1:WfxTRqKhfgAG71Xh6E3WLpjhBtZI37O53G4h5s+3iM4=
-github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
-github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
-github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
-github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
-github.com/cockroachdb/datadriven v1.0.3-0.20230413201302-be42291fc80f h1:otljaYPt5hWxV3MUfO5dFPFiOXg9CyG5/kCfayTqsJ4=
-github.com/cockroachdb/datadriven v1.0.3-0.20230413201302-be42291fc80f/go.mod h1:a9RdTaap04u637JoCzcUoIcDmvwSUtcUFtT/C3kJlTU=
-github.com/cockroachdb/errors v1.12.0 h1:d7oCs6vuIMUQRVbi6jWWWEJZahLCfJpnJSVobd1/sUo=
-github.com/cockroachdb/errors v1.12.0/go.mod h1:SvzfYNNBshAVbZ8wzNc/UPK3w1vf0dKDUP41ucAIf7g=
-github.com/cockroachdb/fifo v0.0.0-20240816210425-c5d0cb0b6fc0 h1:pU88SPhIFid6/k0egdR5V6eALQYq2qbSmukrkgIh/0A=
-github.com/cockroachdb/fifo v0.0.0-20240816210425-c5d0cb0b6fc0/go.mod h1:9/y3cnZ5GKakj/H4y9r9GTjCvAFta7KLgSHPJJYc52M=
-github.com/cockroachdb/logtags v0.0.0-20241215232642-bb51bb14a506 h1:ASDL+UJcILMqgNeV5jiqR4j+sTuvQNHdf2chuKj1M5k=
-github.com/cockroachdb/logtags v0.0.0-20241215232642-bb51bb14a506/go.mod h1:Mw7HqKr2kdtu6aYGn3tPmAftiP3QPX63LdK/zcariIo=
-github.com/cockroachdb/pebble v1.1.5 h1:5AAWCBWbat0uE0blr8qzufZP5tBjkRyy/jWe1QWLnvw=
-github.com/cockroachdb/pebble v1.1.5/go.mod h1:17wO9el1YEigxkP/YtV8NtCivQDgoCyBg5c4VR/eOWo=
-github.com/cockroachdb/redact v1.1.6 h1:zXJBwDZ84xJNlHl1rMyCojqyIxv+7YUpQiJLQ7n4314=
-github.com/cockroachdb/redact v1.1.6/go.mod h1:BVNblN9mBWFyMyqK1k3AAiSxhvhfK2oOZZ2lK+dpvRg=
-github.com/cockroachdb/tokenbucket v0.0.0-20250429170803-42689b6311bb h1:3bCgBvB8PbJVMX1ouCcSIxvsqKPYM7gs72o0zC76n9g=
-github.com/cockroachdb/tokenbucket v0.0.0-20250429170803-42689b6311bb/go.mod h1:7nc4anLGjupUW/PeY5qiNYsdNXj7zopG+eqsS7To5IQ=
-github.com/coreos/go-systemd/v22 v22.5.0/go.mod h1:Y58oyj3AT4RCenI/lSvhwexgC+NSVTIJ3seZv2GcEnc=
-github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E=
 github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
 github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
-github.com/getsentry/sentry-go v0.33.0 h1:YWyDii0KGVov3xOaamOnF0mjOrqSjBqwv48UEzn7QFg=
-github.com/getsentry/sentry-go v0.33.0/go.mod h1:C55omcY9ChRQIUcVcGcs+Zdy4ZpQGvNJ7JYHIoSWOtE=
-github.com/go-errors/errors v1.4.2 h1:J6MZopCL4uSllY1OfXM374weqZFFItUbrImctkmUxIA=
-github.com/go-errors/errors v1.4.2/go.mod h1:sIVyrIiJhuEF+Pj9Ebtd6P/rEYROXFi3BopGUQ5a5Og=
-github.com/godbus/dbus/v5 v5.0.4/go.mod h1:xhWf0FNVPg57R7Z0UbKHbJfkEywrmjJnf7w5xrFpKfA=
-github.com/gogo/protobuf v1.3.2 h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q=
-github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q=
-github.com/golang/snappy v1.0.0 h1:Oy607GVXHs7RtbggtPBnr2RmDArIsAefDwvrdWvRhGs=
-github.com/golang/snappy v1.0.0/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q=
-github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
-github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
 github.com/hjson/hjson-go v3.3.0+incompatible h1:Rqr+Ya+0aCJMjaE4s8E9YKvuJLuLVpEvz4ONum52vnI=
 github.com/hjson/hjson-go v3.3.0+incompatible/go.mod h1:qsetwF8NlsTsOTwZTApNlTCerV+b2GjYRRcIk4JMFio=
-github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8=
-github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck=
-github.com/klauspost/compress v1.18.0 h1:c/Cqfb0r+Yi+JtIEq73FWXVkRonBlf0CRNYc8Zttxdo=
-github.com/klauspost/compress v1.18.0/go.mod h1:2Pp+KzxcywXVXMr50+X0Q/Lsb43OQHYWRCY2AiWywWQ=
-github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=
-github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk=
-github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
-github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
-github.com/mattn/go-colorable v0.1.13 h1:fFA4WZxdEF4tXPZVKMLwD8oUnCTTo08duU7wxecdEvA=
-github.com/mattn/go-colorable v0.1.13/go.mod h1:7S9/ev0klgBDR4GtXTXX8a3vIGJpMovkB8vQcUbaXHg=
-github.com/mattn/go-isatty v0.0.16/go.mod h1:kYGgaQfpe5nmfYZH+SKPsOc2e4SrIfOl2e/yFXSvRLM=
-github.com/mattn/go-isatty v0.0.19 h1:JITubQf0MOLdlGRuRq+jtsDlekdYPia9ZFsB8h/APPA=
-github.com/mattn/go-isatty v0.0.19/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y=
-github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA=
-github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
-github.com/pingcap/errors v0.11.4 h1:lFuQV/oaUMGcD2tqt+01ROSmJs75VG1ToEOkZIZ4nE4=
-github.com/pingcap/errors v0.11.4/go.mod h1:Oi8TUi2kEtXXLMJk9l1cGmz20kV3TaQ0usTwv5KuLY8=
-github.com/pkg/diff v0.0.0-20210226163009-20ebb0f2a09e/go.mod h1:pJLUxLENpZxwdsKMEsNbx1VGcRFpLqf3715MtcvvzbA=
-github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
-github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
-github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
-github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
-github.com/prometheus/client_golang v1.22.0 h1:rb93p9lokFEsctTys46VnV1kLCDpVZ0a/Y92Vm0Zc6Q=
-github.com/prometheus/client_golang v1.22.0/go.mod h1:R7ljNsLXhuQXYZYtw6GAE9AZg8Y7vEW5scdCXrWRXC0=
-github.com/prometheus/client_model v0.6.2 h1:oBsgwpGs7iVziMvrGhE53c/GrLUsZdHnqNwqPLxwZyk=
-github.com/prometheus/client_model v0.6.2/go.mod h1:y3m2F6Gdpfy6Ut/GBsUqTWZqCUvMVzSfMLjcu6wAwpE=
-github.com/prometheus/common v0.64.0 h1:pdZeA+g617P7oGv1CzdTzyeShxAGrTBsolKNOLQPGO4=
-github.com/prometheus/common v0.64.0/go.mod h1:0gZns+BLRQ3V6NdaerOhMbwwRbNh9hkGINtQAsP5GS8=
-github.com/prometheus/procfs v0.16.1 h1:hZ15bTNuirocR6u0JZ6BAHHmwS1p8B4P6MRqxtzMyRg=
-github.com/prometheus/procfs v0.16.1/go.mod h1:teAbpZRB1iIAJYREa1LsoWUXykVXA1KlTmWl8x/U+Is=
-github.com/rogpeppe/go-internal v1.9.0/go.mod h1:WtVeX8xhTBvf0smdhujwtBcq4Qrzq/fJaraNFVN+nFs=
-github.com/rogpeppe/go-internal v1.14.1 h1:UQB4HGPB6osV0SQTLymcB4TgvyWu6ZyliaW0tI/otEQ=
-github.com/rogpeppe/go-internal v1.14.1/go.mod h1:MaRKkUm5W0goXpeCfT7UZI6fk/L7L7so1lCWt35ZSgc=
-github.com/rs/xid v1.6.0/go.mod h1:7XoLgs4eV+QndskICGsho+ADou8ySMSjJKDIan90Nz0=
-github.com/rs/zerolog v1.34.0 h1:k43nTLIwcTVQAncfCw4KZ2VY6ukYoZaBPNOE8txlOeY=
-github.com/rs/zerolog v1.34.0/go.mod h1:bJsvje4Z08ROH4Nhs5iH600c3IkWhwp44iRc54W6wYQ=
-github.com/stretchr/testify v1.10.0 h1:Xv5erBjTwe/5IxqUQTdXv5kgmIvbHo3QQyRwhJsOfJA=
-github.com/stretchr/testify v1.10.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY=
-github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
-github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
-go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto=
-go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE=
-go.uber.org/multierr v1.11.0 h1:blXXJkSxSSfBVBlC76pxqeO+LN3aDfLQo+309xJstO0=
-go.uber.org/multierr v1.11.0/go.mod h1:20+QtiLqy0Nd6FdQB9TLXag12DsQkrbs3htMFfDN80Y=
-go.uber.org/zap v1.27.0 h1:aJMhYGrd5QSmlpLMr2MftRKl7t8J8PTZPA732ud/XR8=
-go.uber.org/zap v1.27.0/go.mod h1:GB2qFLM7cTU87MWRP2mPIjqfIDnGu+VIO4V/SdhGo2E=
-golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
-golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
-golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
-golang.org/x/exp v0.0.0-20250531010427-b6e5de432a8b h1:QoALfVG9rhQ/M7vYDScfPdWjGL9dlsVVM5VGh7aKoAA=
-golang.org/x/exp v0.0.0-20250531010427-b6e5de432a8b/go.mod h1:U6Lno4MTRCDY+Ba7aCcauB9T60gsv5s4ralQzP72ZoQ=
-golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
-golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
-golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
-golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
-golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
-golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
-golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
-golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
-golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
-golang.org/x/sync v0.15.0 h1:KWH3jNZsfyT6xfAfKiz6MRNmd46ByHDYaZ7KSkCtdW8=
-golang.org/x/sync v0.15.0/go.mod h1:1dzgHSNfp02xaA81J2MS99Qcpr2w7fw1gpm99rleRqA=
-golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
-golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
-golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
-golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
-golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
-golang.org/x/sys v0.12.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
-golang.org/x/sys v0.33.0 h1:q3i8TbbEz+JRD9ywIRlyRAQbM0qF7hu24q3teo2hbuw=
-golang.org/x/sys v0.33.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k=
-golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
-golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
-golang.org/x/text v0.26.0 h1:P42AVeLghgTYr4+xUnTRKDMqpar+PtX7KWuNQL21L8M=
-golang.org/x/text v0.26.0/go.mod h1:QK15LZJUUQVJxhz7wXgxSy/CJaTFjd0G+YLonydOVQA=
 golang.org/x/time v0.12.0 h1:ScB/8o8olJvc+CQPWrK3fPZNfh7qgwCrY0zJmoEQLSE=
 golang.org/x/time v0.12.0/go.mod h1:CDIdPxbZBQxdj6cxyCIdrNogrJKMJ7pr37NYpMcMDSg=
-golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
-golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
-golang.org/x/tools v0.0.0-20200619180055-7c47624df98f/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE=
-golang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA=
-golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
-golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
-golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
-golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
-google.golang.org/protobuf v1.36.6 h1:z1NpPI8ku2WgiWnf+t9wTPsn6eP1L7ksHUlkfLvd9xY=
-google.golang.org/protobuf v1.36.6/go.mod h1:jduwjTPXsFjZGTmRluh+L6NjiWu7pchiJ2/5YcXBHnY=
-gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
-gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=

internal/elog/clock.go

@@ -0,0 +1,14 @@
+package elog
+
+import "time"
+
+// nowUTC is the single clock every path and timestamp derives from.
+//
+// It is not time.Now at each call site: a log line stamped in local time beside a
+// directory named for UTC is a bug report waiting to happen, and a file that rolls
+// over at midnight local time while its lines are stamped UTC is worse.
+func nowUTC() time.Time {
+	return time.Now().UTC()
+}
+
+/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

internal/elog/format.go

@@ -0,0 +1,78 @@
+package elog
+
+// The value formatters: small conversions shared by the write path (which builds
+// path components) and the renderer (which builds inline fields). They are shared
+// by two topics, so they get a file of their own rather than living in one of them
+// and making the other look in the wrong place.
+
+import (
+	"strconv"
+	"strings"
+)
+
+// field renders "key=value", or empty when the value is empty. Empty fields are
+// omitted: a bare "username=@" is worse than nothing.
+//
+// It lives here rather than in render.go because sizeField, a formatter, is one of
+// its callers — and a helper two topics use belongs to neither.
+func field(key string, value string) string {
+	if value == "" {
+		return ""
+	}
+
+	return key + "=" + value
+}
+
+// joinFields drops empty fragments and joins the rest with single spaces.
+func joinFields(parts []string) string {
+	kept := make([]string, 0, len(parts))
+
+	for _, p := range parts {
+		if p != "" {
+			kept = append(kept, p)
+		}
+	}
+
+	return strings.Join(kept, " ")
+}
+
+// itoa renders an int64 for a path component or an inline field.
+func itoa(v int64) string {
+	return strconv.FormatInt(v, 10)
+}
+
+// itoaInt renders an int for a rendered field.
+func itoaInt(v int) string {
+	return strconv.Itoa(v)
+}
+
+// itoaFloat renders a float for a rendered field.
+func itoaFloat(v float64) string {
+	return strconv.FormatFloat(v, 'g', -1, 64)
+}
+
+// atUsername renders a username with the conventional @ prefix, or empty.
+func atUsername(name string) string {
+	if name == "" {
+		return ""
+	}
+
+	return "@" + name
+}
+
+// seconds renders a duration in seconds with its unit.
+func seconds(n int) string {
+	return itoaInt(n) + "s"
+}
+
+// sizeField renders a byte count, omitted when unknown. A size of 0 means
+// Telegram did not tell us, not that the file was empty.
+func sizeField(n int64) string {
+	if n <= 0 {
+		return ""
+	}
+
+	return field("size", itoa(n))
+}
+
+/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

internal/elog/globals.go

@@ -0,0 +1,59 @@
+package elog
+
+import (
+	"errors"
+	"sync"
+	"time"
+)
+
+const (
+	// chatLockIdleTimeout is how long a chat's lock entry may sit unused before it
+	// is evicted. Without it the registry grows by one entry per chat ever seen.
+	chatLockIdleTimeout = time.Hour
+
+	// stampLayout is the per-line timestamp. GMT+0 with millisecond precision.
+	stampLayout = "15:04:05.000"
+
+	// logFilePerm is the mode for a newly created chat log file.
+	logFilePerm = 0o644
+
+	// logDirPerm is the mode for the chat, year and month directories.
+	//
+	// It is deliberately not logFilePerm. A directory needs the execute bit to
+	// be traversable, and 0o644 has none, so creating the tree with the file mode
+	// makes every subsequent open inside it fail with EACCES rather than ENOENT
+	// — a failure reportFailure would swallow, leaving chat logging to silently
+	// produce no files at all.
+	//
+	// The two live in one block on purpose: they are a pair that must never be
+	// confused with each other, and keeping them adjacent is what makes confusing
+	// them hard. They are here, in globals.go, because that is where every
+	// package-level const belongs — the pair argument was weighed against rule 2
+	// and rule 2 won, once, on the record.
+	logDirPerm = 0o755
+)
+
+var (
+	// errNotDirectory marks a chat_logs path that exists but is a regular file.
+	errNotDirectory = errors.New("chat_logs path is not a directory")
+
+	// root is the configured chat-logs directory. Empty means chat logging is
+	// disabled and every Log* call is a no-op.
+	root string
+
+	// failing records that a write error has been reported and not yet cleared,
+	// so a persistent failure (a full disk) logs once instead of once per event.
+	failing bool
+
+	// stateMu guards root and failing.
+	stateMu sync.RWMutex
+
+	// registryMu guards locks. It is never held while a chat lock is acquired,
+	// or the whole logger would serialise.
+	registryMu sync.Mutex
+
+	// locks is the per-chat lock registry.
+	locks = map[int64]*chatLock{}
+)
+
+/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

internal/elog/init.go

@@ -0,0 +1,46 @@
+package elog
+
+import (
+	"fmt"
+	"os"
+)
+
+// Init switches chat logging on.
+//
+// An empty root disables it, and every Log* call becomes a no-op — chat logging
+// is optional. A non-empty root must already exist: a path that does not is a
+// typo, and silently logging nowhere is the worst possible response to one, so
+// this returns an error and moon.ParseConfig treats it as fatal.
+func Init(rootDir string) error {
+	if rootDir == "" {
+		setRoot("")
+
+		return nil
+	}
+
+	info, err := os.Stat(rootDir)
+	if err != nil {
+		return fmt.Errorf("chat_logs directory %s is not usable: %w", rootDir, err)
+	}
+
+	if !info.IsDir() {
+		return fmt.Errorf("%w: %s", errNotDirectory, rootDir)
+	}
+
+	setRoot(rootDir)
+
+	return nil
+}
+
+// Close disables chat logging.
+//
+// Nothing is held open between lines, so there are no handles to close; this
+// exists so shutdown has a symmetric counterpart to Init and so a misbehaving
+// caller can stop writing.
+func Close() error {
+	setRoot("")
+
+	return nil
+}
+
+/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

internal/elog/message.go

@@ -0,0 +1,127 @@
+package elog
+
+import (
+	"github.com/NicoNex/echotron/v3"
+)
+
+// LogMessage records a chat message.
+//
+// It handles both ordinary messages — text, media, captions, replies — and the
+// service messages Telegram delivers as a Message with a field set instead of
+// text: joins, leaves, title changes, pins, migrations, forum topics. A message
+// with nothing describable is skipped rather than logged as noise.
+func LogMessage(msg *echotron.Message) {
+	logMessage(msg, "")
+}
+
+// LogEditedMessage records an edited message. The line is marked "edited" so a
+// reader can tell it apart from the original.
+func LogEditedMessage(msg *echotron.Message) {
+	logMessage(msg, "edited ")
+}
+
+// logMessage is the shared body of LogMessage and LogEditedMessage.
+func logMessage(msg *echotron.Message, prefix string) {
+	if _, on := chatLogRoot(); !on || msg == nil {
+		return
+	}
+
+	for _, body := range describeMessage(msg) {
+		appendLine(msg.Chat.ID, prefix+renderLine(msg.Chat, msg.From, body))
+	}
+}
+
+// describeMessage returns one rendered body per line to write. It returns more
+// than one only for a multi-member join, where one line per newcomer keeps each
+// fact greppable instead of cramming every newcomer onto one line.
+func describeMessage(msg *echotron.Message) []string {
+	if bodies := describeService(msg); len(bodies) > 0 {
+		return bodies
+	}
+
+	body := describePayload(msg)
+	if body == "" {
+		return nil
+	}
+
+	if msg.ReplyToMessage != nil {
+		body += " reply_to=" + itoaInt(msg.ReplyToMessage.ID)
+	}
+
+	return []string{body}
+}
+
+// describeService renders the service-message fields, newest-first, and returns
+// nil when the message is not a service message.
+func describeService(msg *echotron.Message) []string {
+	var bodies []string
+
+	// A join with N newcomers writes N lines.
+	for _, u := range msg.NewChatMembers {
+		if u == nil {
+			continue
+		}
+
+		bodies = append(bodies, describeReference(u)+" joined the chat")
+	}
+
+	if msg.LeftChatMember != nil {
+		bodies = append(bodies, describeReference(msg.LeftChatMember)+" left the chat")
+	}
+
+	if msg.NewChatTitle != "" {
+		bodies = append(bodies, "chat title changed to "+msg.NewChatTitle)
+	}
+
+	if len(msg.NewChatPhoto) > 0 {
+		bodies = append(bodies, "chat photo changed")
+	}
+
+	if msg.PinnedMessage != nil {
+		bodies = append(bodies, "pinned message "+itoaInt(msg.PinnedMessage.ID))
+	}
+
+	if msg.GroupChatCreated {
+		bodies = append(bodies, "group chat created")
+	}
+
+	if msg.MigrateToChatID != 0 {
+		bodies = append(bodies, "chat migrated to "+itoa(int64(msg.MigrateToChatID)))
+	}
+
+	if msg.MigrateFromChatID != 0 {
+		bodies = append(bodies, "chat migrated from "+itoa(int64(msg.MigrateFromChatID)))
+	}
+
+	if msg.ConnectedWebsite != "" {
+		bodies = append(bodies, "connected website "+msg.ConnectedWebsite)
+	}
+
+	if msg.ForumTopicCreated != nil {
+		bodies = append(bodies, "forum topic "+msg.ForumTopicCreated.Name+" created")
+	}
+
+	if msg.ForumTopicClosed != nil {
+		bodies = append(bodies, "forum topic closed")
+	}
+
+	if msg.ForumTopicReopened != nil {
+		bodies = append(bodies, "forum topic reopened")
+	}
+
+	if msg.GeneralForumTopicHidden != nil {
+		bodies = append(bodies, "general forum topic hidden")
+	}
+
+	if msg.GeneralForumTopicUnhidden != nil {
+		bodies = append(bodies, "general forum topic unhidden")
+	}
+
+	if msg.VideoChatScheduled != nil {
+		bodies = append(bodies, "video chat scheduled")
+	}
+
+	return bodies
+}
+
+/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

internal/elog/render.go

@@ -0,0 +1,248 @@
+package elog
+
+import (
+	"fmt"
+	"strings"
+
+	"github.com/NicoNex/echotron/v3"
+)
+
+// renderLine assembles one log line:
+//
+//	21:33:07.123 in chat -1001 (title=…) user 42 (first_name=…) says …
+//
+// Empty field groups are dropped entirely rather than printed as empty parens.
+func renderLine(chat echotron.Chat, user *echotron.User, body string) string {
+	var b strings.Builder
+
+	b.WriteString(nowUTC().Format(stampLayout))
+	b.WriteString(" in chat ")
+	b.WriteString(itoa(chat.ID))
+
+	if desc := describeChat(&chat); desc != "" {
+		b.WriteString(" (")
+		b.WriteString(desc)
+		b.WriteString(")")
+	}
+
+	if user != nil {
+		b.WriteString(" user ")
+		b.WriteString(itoa(user.ID))
+
+		if desc := describeUser(user); desc != "" {
+			b.WriteString(" (")
+			b.WriteString(desc)
+			b.WriteString(")")
+		}
+	}
+
+	if body != "" {
+		b.WriteString(" ")
+		b.WriteString(body)
+	}
+
+	return b.String()
+}
+
+// describeChat renders a chat's identity fields. Type is deliberately omitted:
+// the user-specified format has no slot for it, and which of title / first_name
+// is present already distinguishes group from private chat.
+func describeChat(c *echotron.Chat) string {
+	if c == nil {
+		return ""
+	}
+
+	return joinFields([]string{
+		field("title", c.Title),
+		field("username", atUsername(c.Username)),
+		field("first_name", c.FirstName),
+		field("last_name", c.LastName),
+	})
+}
+
+// describeUser renders a user's identity fields.
+func describeUser(u *echotron.User) string {
+	if u == nil {
+		return ""
+	}
+
+	return joinFields([]string{
+		field("first_name", u.FirstName),
+		field("last_name", u.LastName),
+		field("username", atUsername(u.Username)),
+	})
+}
+
+// describePayload renders what a message carries: media first, then a caption,
+// because "post photo (…) caption" reads better than the reverse.
+func describePayload(m *echotron.Message) string {
+	kind, details := describeMedia(m)
+
+	var b strings.Builder
+
+	if kind != "" {
+		b.WriteString("post ")
+		b.WriteString(kind)
+
+		if details != "" {
+			b.WriteString(" (")
+			b.WriteString(details)
+			b.WriteString(")")
+		}
+	}
+
+	// A caption accompanies the media rather than replacing it: "post photo (…)"
+	// alone loses what the user actually said.
+	if m.Text != "" {
+		if b.Len() > 0 {
+			b.WriteString(" ")
+		}
+
+		b.WriteString("says ")
+		b.WriteString(m.Text)
+	}
+
+	return b.String()
+}
+
+// describeMedia returns the media kind and its key info. The first media field
+// set on the message wins, in the order listed in the SPEC.
+func describeMedia(m *echotron.Message) (string, string) {
+	switch {
+	case len(m.Photo) > 0:
+		// The array is ordered smallest first, so the last entry is the largest.
+		best := m.Photo[len(m.Photo)-1]
+
+		return "photo", joinFields([]string{
+			dims(best.Width, best.Height),
+			sizeField(int64(best.FileSize)),
+		})
+
+	case m.Video != nil:
+		return "video", joinFields([]string{
+			field("file_name", m.Video.FileName),
+			field("mime", m.Video.MimeType),
+			dims(m.Video.Width, m.Video.Height),
+			field("duration", seconds(m.Video.Duration)),
+			sizeField(m.Video.FileSize),
+		})
+
+	case m.Animation != nil:
+		return "animation", joinFields([]string{
+			field("file_name", m.Animation.FileName),
+			field("mime", m.Animation.MimeType),
+			dims(m.Animation.Width, m.Animation.Height),
+			field("duration", seconds(m.Animation.Duration)),
+			sizeField(m.Animation.FileSize),
+		})
+
+	case m.Audio != nil:
+		return "audio", joinFields([]string{
+			field("title", m.Audio.Title),
+			field("file_name", m.Audio.FileName),
+			field("mime", m.Audio.MimeType),
+			field("duration", seconds(m.Audio.Duration)),
+			sizeField(m.Audio.FileSize),
+		})
+
+	case m.Voice != nil:
+		return "voice", joinFields([]string{
+			field("mime", m.Voice.MimeType),
+			field("duration", seconds(m.Voice.Duration)),
+			sizeField(m.Voice.FileSize),
+		})
+
+	case m.VideoNote != nil:
+		return "video_note", joinFields([]string{
+			dims(m.VideoNote.Length, m.VideoNote.Length),
+			field("duration", seconds(m.VideoNote.Duration)),
+			sizeField(int64(m.VideoNote.FileSize)),
+		})
+
+	case m.Sticker != nil:
+		return "sticker", joinFields([]string{
+			field("emoji", m.Sticker.Emoji),
+			field("set", m.Sticker.SetName),
+			dims(m.Sticker.Width, m.Sticker.Height),
+			field("video", itoaBool(m.Sticker.IsVideo)),
+			field("animated", itoaBool(m.Sticker.IsAnimated)),
+		})
+
+	case m.Document != nil:
+		return "document", joinFields([]string{
+			field("file_name", m.Document.FileName),
+			field("mime", m.Document.MimeType),
+			sizeField(m.Document.FileSize),
+		})
+
+	case m.Contact != nil:
+		return "contact", joinFields([]string{
+			field("phone", m.Contact.PhoneNumber),
+			field("first_name", m.Contact.FirstName),
+			field("last_name", m.Contact.LastName),
+			field("user_id", itoaInt(m.Contact.UserID)),
+		})
+
+	case m.Location != nil:
+		return "location", joinFields([]string{
+			field("lat", itoaFloat(m.Location.Latitude)),
+			field("lon", itoaFloat(m.Location.Longitude)),
+		})
+
+	case m.Venue != nil:
+		parts := []string{
+			field("title", m.Venue.Title),
+			field("address", m.Venue.Address),
+		}
+
+		if m.Venue.Location != nil {
+			parts = append(parts,
+				field("lat", itoaFloat(m.Venue.Location.Latitude)),
+				field("lon", itoaFloat(m.Venue.Location.Longitude)))
+		}
+
+		return "venue", joinFields(parts)
+
+	case m.Dice != nil:
+		return "dice", joinFields([]string{
+			field("emoji", m.Dice.Emoji),
+			field("value", itoaInt(m.Dice.Value)),
+		})
+	}
+
+	return "", ""
+}
+
+// dims renders "WxH", or empty when the dimensions are unknown.
+func dims(w int, h int) string {
+	if w <= 0 || h <= 0 {
+		return ""
+	}
+
+	return itoaInt(w) + "x" + itoaInt(h)
+}
+
+// itoaBool renders a boolean for an inline field.
+func itoaBool(b bool) string {
+	if b {
+		return "true"
+	}
+
+	return "false"
+}
+
+// describeReference renders "user 42 (first_name=Ann)", used for the newcomer in
+// a service message and for the member in a status change.
+func describeReference(u *echotron.User) string {
+	if u == nil {
+		return "user unknown"
+	}
+
+	if desc := describeUser(u); desc != "" {
+		return fmt.Sprintf("user %d (%s)", u.ID, desc)
+	}
+
+	return fmt.Sprintf("user %d", u.ID)
+}
+
+/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

internal/elog/retraction.go

@@ -0,0 +1,24 @@
+package elog
+
+import (
+	"github.com/NicoNex/echotron/v3"
+)
+
+// LogGreetingRetracted records that we deleted a greeting of our own — one the
+// addressed member never answered, because they were banned for the silence.
+//
+// The log is append-only, so the question's original line stays where it was and
+// this line says what became of it. A reader must be able to tell that the room
+// asked a question and later removed it, rather than finding a line that
+// describes a message which no longer exists.
+func LogGreetingRetracted(chat *echotron.Chat, user *echotron.User, messageID int) {
+	if _, on := chatLogRoot(); !on || chat == nil {
+		return
+	}
+
+	body := "greeting " + itoaInt(messageID) + " retracted, member banned without answering"
+
+	appendLine(chat.ID, renderLine(*chat, user, body))
+}
+
+/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

internal/elog/root.go

@@ -0,0 +1,44 @@
+package elog
+
+// The chat-logs root: where the tree lives, and whether logging is on at all.
+//
+// The state itself is in globals.go with the rest of the package-level variables;
+// this file holds only the accessors, because "is it on and where" is one question
+// with one answer and must not be asked in two pieces.
+
+// ChatLoggingEnabled reports whether chat logging is currently switched on.
+//
+// Exported so another package can assert the post-shutdown state instead of
+// inferring it from the absence of files.
+func ChatLoggingEnabled() (string, bool) {
+	return chatLogRoot()
+}
+
+// chatLogRoot returns the configured root and reports whether chat logging is
+// on, in a *single* read.
+//
+// The two must not be asked separately: a Close landing between an enabled()
+// check and a rootDir() read would leave the caller with an empty root, and
+// filepath.Join("", ...) silently yields a path relative to the working
+// directory — chat logs would be written wherever the supervisor happened to
+// start the process.
+func chatLogRoot() (string, bool) {
+	stateMu.RLock()
+	defer stateMu.RUnlock()
+
+	return root, root != ""
+}
+
+// setRoot switches chat logging on or off.
+func setRoot(dir string) {
+	stateMu.Lock()
+	defer stateMu.Unlock()
+
+	root = dir
+
+	if dir == "" {
+		failing = false
+	}
+}
+
+/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

internal/elog/status.go

@@ -0,0 +1,63 @@
+package elog
+
+import (
+	"strings"
+
+	"github.com/NicoNex/echotron/v3"
+)
+
+// LogUserStatus records a change to a chat member's status — join, leave, ban,
+// promote, demote, restrict — as carried by Update.ChatMember.
+func LogUserStatus(ev *echotron.ChatMemberUpdated) {
+	if _, on := chatLogRoot(); !on || ev == nil {
+		return
+	}
+
+	who := ev.NewChatMember.User
+	if who == nil {
+		who = ev.OldChatMember.User
+	}
+
+	var b strings.Builder
+
+	// The subject is already rendered by renderLine's "user <id> (<desc>)"
+	// prefix, so the body starts at the transition itself.
+	b.WriteString("status ")
+	b.WriteString(orUnknown(ev.OldChatMember.Status))
+	b.WriteString(" -> ")
+	b.WriteString(orUnknown(ev.NewChatMember.Status))
+
+	if ev.NewChatMember.CustomTitle != "" {
+		b.WriteString(" custom_title=")
+		b.WriteString(ev.NewChatMember.CustomTitle)
+	}
+
+	b.WriteString(" by ")
+	b.WriteString(describeUserRef(&ev.From))
+
+	appendLine(ev.Chat.ID, renderLine(ev.Chat, who, b.String()))
+}
+
+// describeUserRef renders a value-type User reference.
+func describeUserRef(u *echotron.User) string {
+	if u == nil {
+		return "user unknown"
+	}
+
+	if desc := describeUser(u); desc != "" {
+		return "user " + itoa(u.ID) + " (" + desc + ")"
+	}
+
+	return "user " + itoa(u.ID)
+}
+
+// orUnknown renders a status, or a placeholder when Telegram sent none.
+func orUnknown(status string) string {
+	if status == "" {
+		return "unknown"
+	}
+
+	return status
+}
+
+/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

internal/elog/types.go

@@ -0,0 +1,21 @@
+package elog
+
+import (
+	"sync"
+	"time"
+)
+
+// chatLock serialises writes for one chat so that two goroutines cannot splice
+// their lines into the same file.
+type chatLock struct {
+	mu sync.Mutex
+
+	// refs counts goroutines holding or waiting for this entry. It is only
+	// ever read or written while registryMu is held, which is what makes it a
+	// usable proof of "nobody is holding this" during eviction.
+	refs int
+
+	lastUsed time.Time
+}
+
+/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

internal/elog/write.go

@@ -0,0 +1,148 @@
+package elog
+
+import (
+	"os"
+	"path/filepath"
+	"time"
+
+	"bunny-bot/internal/mylog"
+)
+
+// acquire returns the chat's lock with this goroutine holding a reference.
+func acquire(chatID int64) *chatLock {
+	registryMu.Lock()
+
+	l, ok := locks[chatID]
+	if !ok {
+		l = &chatLock{}
+		locks[chatID] = l
+	}
+
+	l.refs++
+	l.lastUsed = time.Now()
+
+	evictIdleLocked()
+
+	registryMu.Unlock()
+
+	l.mu.Lock()
+
+	return l
+}
+
+// release drops this goroutine's reference and unlocks the chat.
+func release(l *chatLock) {
+	l.mu.Unlock()
+
+	registryMu.Lock()
+
+	l.refs--
+	l.lastUsed = time.Now()
+
+	registryMu.Unlock()
+}
+
+// evictIdleLocked drops registry entries nobody is holding and that have been
+// idle for longer than chatLockIdleTimeout.
+//
+// Safety rests on refs: it is only ever changed while registryMu is held, so
+// observing refs == 0 here is a proof that no goroutine holds the entry, not a
+// guess. Evicting a held entry would let a second writer take a *different*
+// mutex for the same chat, which is precisely the interleaving the lock exists
+// to prevent. The caller must hold registryMu.
+func evictIdleLocked() {
+	cutoff := time.Now().Add(-chatLockIdleTimeout)
+
+	for id, l := range locks {
+		if l.refs == 0 && l.lastUsed.Before(cutoff) {
+			delete(locks, id)
+		}
+	}
+}
+
+// appendLine writes one line to the chat's log for the day containing now.
+//
+// The whole protocol is set_mutex -> MkdirAll -> open|create -> write -> close,
+// with unset_mutex deferred. Nothing is held open between lines.
+func appendLine(chatID int64, line string) {
+	// One read of both the root and whether logging is on, so a Close racing
+	// this call can never leave us with an empty root and a relative path.
+	dir, on := chatLogRoot()
+	if !on {
+		return
+	}
+
+	lock := acquire(chatID)
+
+	defer release(lock)
+
+	// One clock for both the path and the line: using Message.Date for one and
+	// time.Now() for the other would let a line carry a date its own path
+	// disagrees with.
+	now := nowUTC()
+
+	dayDir := filepath.Join(dir, itoa(chatID), now.Format("2006"), now.Format("01"))
+	path := filepath.Join(dayDir, now.Format("02")+".txt")
+
+	// MkdirAll creates every missing parent, so this covers the chat directory
+	// and the year too. No need to pre-create anything.
+	if err := os.MkdirAll(dayDir, logDirPerm); err != nil {
+		reportFailure(err)
+
+		return
+	}
+
+	f, err := os.OpenFile(path, os.O_CREATE|os.O_APPEND|os.O_WRONLY, logFilePerm)
+	if err != nil {
+		reportFailure(err)
+
+		return
+	}
+
+	_, writeErr := f.WriteString(line + "\n")
+	closeErr := f.Close()
+
+	if writeErr != nil {
+		reportFailure(writeErr)
+
+		return
+	}
+
+	if closeErr != nil {
+		reportFailure(closeErr)
+
+		return
+	}
+
+	reportRecovered()
+}
+
+// reportFailure sends a write failure to the main log, but only the first one
+// until a write succeeds again. A full disk would otherwise produce one error
+// per chat event and drown the very log meant to explain it.
+func reportFailure(err error) {
+	stateMu.Lock()
+	already := failing
+	failing = true
+	stateMu.Unlock()
+
+	if already {
+		return
+	}
+
+	mylog.Errorf("CHAT LOG WRITE FAILED, further failures suppressed until a write succeeds: %s", err)
+}
+
+// reportRecovered clears the suppression after a successful write.
+func reportRecovered() {
+	stateMu.Lock()
+	was := failing
+	failing = false
+	stateMu.Unlock()
+
+	if was {
+		mylog.Info("Chat logging recovered")
+	}
+}
+
+/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

internal/moon/animation.go

@@ -1,28 +0,0 @@
-package moon
-
-import (
-	"fmt"
-
-	"github.com/NicoNex/echotron/v3"
-)
-
-// animation parses message with animation.
-func animation(msg *echotron.Update) {
-	if Cfg.LogChats {
-		ChatLog(
-			msg.Message.Chat.ID,
-			fmt.Sprintf(
-				"User %s %s (username = %s, id = %d) sent animation to chat %s %s (%d)",
-				msg.Message.From.FirstName,
-				msg.Message.From.LastName,
-				msg.Message.From.Username,
-				msg.Message.From.ID,
-				msg.Message.Chat.Type,
-				msg.Message.Chat.Title,
-				msg.Message.Chat.ID,
-			),
-		)
-	}
-}
-
-/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

internal/moon/casChatAPI.go

@@ -1,100 +0,0 @@
-package moon
-
-import (
-	"encoding/json"
-	"fmt"
-	"io"
-	"net/http"
-	"time"
-
-	"github.com/hjson/hjson-go"
-)
-
-// CasCheckID check given telegram user id vs CAS spammers base.
-// https://api.cas.chat/check?user_id=821871410
-func CasCheckID(id int64) (bool, error) {
-	var (
-		result bool
-		err    error
-		url    = fmt.Sprintf("https://api.cas.chat/check?user_id=%d", id)
-		c      = &http.Client{
-			Timeout: 15 * time.Second,
-		}
-		banned    CasTrue
-		notBanned CasFalse
-	)
-
-	resp, err := c.Get(url) //nolint: bodyclose, noctx
-
-	if err != nil {
-		return result, fmt.Errorf("unable to query api.cas.chat, %w", err)
-	}
-
-	// Looks like we should clse body as it it recommends in docs: https://pkg.go.dev/net/http .
-	defer func(Body io.ReadCloser) {
-		err := Body.Close()
-
-		if err != nil {
-			Log.Errorf("Unable to close response body for request to %s: %s", url, err)
-		}
-	}(resp.Body)
-
-	// Client should not return with http status code other than 200.
-	if resp.StatusCode > 200 {
-		return result, fmt.Errorf( //nolint: err113
-			"api.cas.chat returns non-200 http status code: %d %s",
-			resp.StatusCode, resp.Status)
-	}
-
-	body, err := io.ReadAll(resp.Body)
-
-	if err != nil {
-		return result, fmt.Errorf("unable to read response from api.cas.chat, %w", err)
-	}
-
-	// looks like go native json decoder is pretty shitty, so we'll take hjson as less restrictive parser, parse json
-	// encode it back to struct, then parse struct as encode/json
-
-	var tmp map[string]any
-	err = hjson.Unmarshal(body, &tmp)
-
-	// Не удалось распарсить.
-	if err != nil {
-		return result, fmt.Errorf("unable to parse response from api.cas.chat via hjson: %w", err)
-	}
-
-	tmpjson, err := json.Marshal(tmp)
-
-	// Не удалось преобразовать map-ку в json.
-	if err != nil {
-		return result, fmt.Errorf(
-			"unable to parse back to json response from api.cas.chat parsed by hjson: %w", err)
-	}
-
-	if err := json.Unmarshal(tmpjson, &notBanned); err == nil {
-		// if ok field in json == false, then user is not in ban. (records about id not found)
-		if !notBanned.Ok {
-			return false, err
-		}
-	} // At this point we just ignore err != nil, because it *can be* other json type.
-
-	if err := json.Unmarshal(body, &banned); err == nil {
-		// if ok field in json == true, then user is banned. (records about id are found)
-		if banned.Ok {
-			Log.Debugf(
-				"api.cas.chat reported user with id=%d as banned: offences=%d ban_time=%s",
-				id,
-				banned.Result.Offenses,
-				banned.Result.TimeAdded,
-			)
-
-			return true, err
-		}
-
-		return result, fmt.Errorf("api.cas.chat returns strange structure: %s", string(body)) //nolint: err113
-	} // At this point we can't ignore err != nil because we do not know (yet?) other forms of json reply from cas api.
-
-	return result, fmt.Errorf("unable to parse response from api.cas.chat: %s", string(body)) //nolint: err113
-}
-
-/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

internal/moon/chatMemberUpdated.go

@@ -1,208 +0,0 @@
-package moon
-
-import (
-	"fmt"
-	"slices"
-	"time"
-
-	"github.com/NicoNex/echotron/v3"
-)
-
-// newChatMember parses one of types ChatMemberUpdated event.
-// https://core.telegram.org/bots/api#chatmemberupdated
-func chatMemberUpdated(msg *echotron.Update) {
-	AddChat(&msg.ChatMember.Chat)
-	AddUser(msg.ChatMember.NewChatMember.User)
-
-	Log.Infof(
-		"User %s %s (username = %s, id = %d) in chat %s %s (%d) now %s was %s",
-		msg.ChatMember.NewChatMember.User.FirstName,
-		msg.ChatMember.NewChatMember.User.LastName,
-		msg.ChatMember.NewChatMember.User.Username,
-		msg.ChatMember.NewChatMember.User.ID,
-		msg.ChatMember.Chat.Type,
-		msg.ChatMember.Chat.Title,
-		msg.ChatMember.Chat.ID,
-		msg.ChatMember.NewChatMember.Status,
-		msg.ChatMember.OldChatMember.Status,
-	)
-
-	Log.Debugf(
-		"Add %s %s (%d) to list of known chats",
-		msg.ChatMember.Chat.Type,
-		msg.ChatMember.Chat.Title,
-		msg.ChatMember.Chat.ID,
-	)
-
-	ChatList = append(ChatList, msg.ChatMember.Chat.ID)
-	slices.Sort(ChatList)
-	ChatList = slices.Compact(ChatList)
-
-	// Check some other attributes.
-	switch {
-	case msg.ChatMember.OldChatMember.Status == "left" && msg.ChatMember.NewChatMember.Status == "member":
-		Log.Infof(
-			"Getting info about user %s %s (username = %s, id = %d) in chat %s %s (%d) from LeftMembers DB",
-			msg.ChatMember.NewChatMember.User.FirstName,
-			msg.ChatMember.NewChatMember.User.LastName,
-			msg.ChatMember.NewChatMember.User.Username,
-			msg.ChatMember.NewChatMember.User.ID,
-			msg.ChatMember.Chat.Type,
-			msg.ChatMember.Chat.Title,
-			msg.ChatMember.Chat.ID,
-		)
-
-		leftTimestampString := Cfg.GetValue(
-			"LeftMembers",
-			fmt.Sprintf("%d", msg.ChatMember.Chat.ID),
-			fmt.Sprintf("%d", msg.ChatMember.NewChatMember.User.ID),
-		)
-
-		// Yes, this member definitely left chat at some point, we have no question to him/her.
-		if leftTimestampString != "" {
-			Log.Infof(
-				"User %s %s (username = %s, id = %d) in chat %s %s (%d) found in from LeftMembers DB",
-				msg.ChatMember.NewChatMember.User.FirstName,
-				msg.ChatMember.NewChatMember.User.LastName,
-				msg.ChatMember.NewChatMember.User.Username,
-				msg.ChatMember.NewChatMember.User.ID,
-				msg.ChatMember.Chat.Type,
-				msg.ChatMember.Chat.Title,
-				msg.ChatMember.Chat.ID,
-			)
-
-			return
-		}
-
-		Log.Infof(
-			"User %s %s (username = %s, id = %d) in chat %s %s (%d) not found in LeftMembers DB",
-			msg.ChatMember.NewChatMember.User.FirstName,
-			msg.ChatMember.NewChatMember.User.LastName,
-			msg.ChatMember.NewChatMember.User.Username,
-			msg.ChatMember.NewChatMember.User.ID,
-			msg.ChatMember.Chat.Type,
-			msg.ChatMember.Chat.Title,
-			msg.ChatMember.Chat.ID,
-		)
-
-		// Spam bots come with these attributes.
-		// msg.ChatMember.NewChatMember.Status = "member"
-		// msg.ChatMember.OldChatMember.User != nil
-		// msg.ChatMember.OldChatMember.Status = "left"
-
-		casBanned, err := CasCheckID(msg.ChatMember.NewChatMember.User.ID)
-
-		if err != nil {
-			Log.Errorf(
-				"Unable to check user %s %s (username = %s, id = %d) in chat %s %s (%d): %s",
-				msg.ChatMember.NewChatMember.User.FirstName,
-				msg.ChatMember.NewChatMember.User.LastName,
-				msg.ChatMember.NewChatMember.User.Username,
-				msg.ChatMember.NewChatMember.User.ID,
-				msg.ChatMember.Chat.Type,
-				msg.ChatMember.Chat.Title,
-				msg.ChatMember.Chat.ID,
-				err,
-			)
-		} else if casBanned {
-			// Ban user if cas report it as banned.
-			Log.Infof(
-				"Ban user %s %s (username = %s, id = %d) in chat %s %s (%d) by CAS blacklist",
-				msg.ChatMember.NewChatMember.User.FirstName,
-				msg.ChatMember.NewChatMember.User.LastName,
-				msg.ChatMember.NewChatMember.User.Username,
-				msg.ChatMember.NewChatMember.User.ID,
-				msg.ChatMember.Chat.Type,
-				msg.ChatMember.Chat.Title,
-				msg.ChatMember.Chat.ID,
-			)
-
-			squash(
-				msg.ChatMember.Chat.ID,
-				msg.ChatMember.NewChatMember.User.ID,
-			)
-
-			return
-		}
-
-		Log.Infof(
-			"Storing info about user %s %s (username = %s, id = %d) in chat %s %s (%d) to AppearedMember DB",
-			msg.ChatMember.NewChatMember.User.FirstName,
-			msg.ChatMember.NewChatMember.User.LastName,
-			msg.ChatMember.NewChatMember.User.Username,
-			msg.ChatMember.NewChatMember.User.ID,
-			msg.ChatMember.Chat.Type,
-			msg.ChatMember.Chat.Title,
-			msg.ChatMember.Chat.ID,
-		)
-
-		key := fmt.Sprintf("%d+%d", msg.ChatMember.Chat.ID, msg.ChatMember.NewChatMember.User.ID)
-		AppearedMembers.Set(key, time.Now().Unix(), 1*time.Minute)
-		PendingCASMembers.Set(key, time.Now().Unix(), 30*time.Minute)
-
-	case msg.ChatMember.OldChatMember.Status == "member" && msg.ChatMember.NewChatMember.Status == "kicked":
-		Log.Infof(
-			"User %s %s (username = %s, id = %d) in chat %s %s (%d) was banned",
-			msg.ChatMember.NewChatMember.User.FirstName,
-			msg.ChatMember.NewChatMember.User.LastName,
-			msg.ChatMember.NewChatMember.User.Username,
-			msg.ChatMember.NewChatMember.User.ID,
-			msg.ChatMember.Chat.Type,
-			msg.ChatMember.Chat.Title,
-			msg.ChatMember.Chat.ID,
-		)
-
-		key := fmt.Sprintf("%d+%d", msg.ChatMember.Chat.ID, msg.ChatMember.NewChatMember.User.ID)
-
-		Log.Debugf(
-			"Put user %s %s (username = %s, id = %d) in %s %s (%d) into SquashedMembers to debounce ban.",
-			msg.ChatMember.NewChatMember.User.FirstName,
-			msg.ChatMember.NewChatMember.User.LastName,
-			msg.ChatMember.NewChatMember.User.Username,
-			msg.ChatMember.NewChatMember.User.ID,
-			msg.ChatMember.Chat.Type,
-			msg.ChatMember.Chat.Title,
-			msg.ChatMember.Chat.ID,
-		)
-
-		SquashedMembers.Set(key, true, 30*time.Second)
-
-		Log.Debugf(
-			"Storing info about user %s %s (username = %s, id = %d) in chat %s %s (%d) to BannedMembers DB",
-			msg.ChatMember.NewChatMember.User.FirstName,
-			msg.ChatMember.NewChatMember.User.LastName,
-			msg.ChatMember.NewChatMember.User.Username,
-			msg.ChatMember.NewChatMember.User.ID,
-			msg.ChatMember.Chat.Type,
-			msg.ChatMember.Chat.Title,
-			msg.ChatMember.Chat.ID,
-		)
-
-		if err := Cfg.SaveKeyValue(
-			"BannedMembers",
-			fmt.Sprintf("%d", msg.ChatMember.Chat.ID),
-			fmt.Sprintf("%d", msg.ChatMember.NewChatMember.User.ID),
-			fmt.Sprintf("%d", time.Now().Unix()),
-		); err != nil {
-			Log.Error(err)
-		} else {
-			Log.Infof("Info about banned userid %d saved to BannedMembers DB", msg.ChatMember.Chat.ID)
-		}
-
-	default:
-		Log.Infof(
-			"User %s %s (username = %s, id = %d) in chat %s %s (%d) now %s was %s",
-			msg.ChatMember.NewChatMember.User.FirstName,
-			msg.ChatMember.NewChatMember.User.LastName,
-			msg.ChatMember.NewChatMember.User.Username,
-			msg.ChatMember.NewChatMember.User.ID,
-			msg.ChatMember.Chat.Type,
-			msg.ChatMember.Chat.Title,
-			msg.ChatMember.Chat.ID,
-			msg.ChatMember.NewChatMember.Status,
-			msg.ChatMember.OldChatMember.Status,
-		)
-	}
-}
-
-/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

internal/moon/chatlog.go

@@ -1,49 +0,0 @@
-package moon
-
-import (
-	"fmt"
-	"os"
-	"path/filepath"
-	"strconv"
-	"time"
-
-	"github.com/rs/zerolog"
-	"github.com/rs/zerolog/log"
-)
-
-// ChatLog logs message to separate file, placed in basedir(Cfg.LogFile).
-func ChatLog(chatid int64, message string) {
-	var (
-		filename string
-	)
-
-	if Cfg.LogFile != "" {
-		filename = fmt.Sprintf("%s/%s", filepath.Dir(Cfg.LogFile), strconv.FormatInt(chatid, 10))
-	} else {
-		filename = os.DevNull
-	}
-
-	f, err := os.OpenFile(filename, os.O_APPEND|os.O_WRONLY|os.O_CREATE, 0644)
-
-	if err != nil {
-		fmt.Printf("Unable to open file %s for logging: %s", filename, err)
-
-		// TODO: Should we try harder?
-		return
-	}
-
-	defer f.Close()
-
-	log.Logger = log.Output(
-		zerolog.ConsoleWriter{
-			Out:        f,
-			NoColor:    true,
-			TimeFormat: time.DateTime,
-			PartsOrder: []string{"time", "message"},
-		},
-	)
-
-	log.Info().Msg(message)
-}
-
-/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

internal/moon/config.go

@@ -5,10 +5,11 @@ import (
 	"fmt"
 	"os"
 	"path/filepath"
-	"strings"
+
+	"bunny-bot/internal/elog"
+	"bunny-bot/internal/mylog"
 
 	"github.com/hjson/hjson-go"
-	"go.uber.org/zap/zapcore"
 )
 
 // parseConfig разбирает и валидирует даденный конфиг.
@@ -66,34 +67,27 @@ func ParseConfig(path string) (MyConfig, error) {
 		return MyConfig{}, fmt.Errorf("token field in config file %s must be set", path) //nolint: err113
 	}
 
-	if _, err := zapcore.ParseLevel(sampleConfig.LogLevel); err != nil {
+	if _, err := mylog.ParseLevel(sampleConfig.LogLevel); err != nil {
 		sampleConfig.LogLevel = "info"
 	}
 
+	// Chat logs are optional, but a configured path that does not exist is a
+	// typo and must stop startup rather than silently disable logging.
+	//
+	// A relative chat_logs is resolved against the directory holding the config
+	// file, not the process working directory: a daemon's cwd is wherever its
+	// supervisor happened to start it, so a relative path there would move the
+	// log directory on every restart under a different supervisor.
+	if sampleConfig.ChatLogs != "" && !filepath.IsAbs(sampleConfig.ChatLogs) {
+		sampleConfig.ChatLogs = filepath.Join(filepath.Dir(path), sampleConfig.ChatLogs)
+	}
+
+	if err := elog.Init(sampleConfig.ChatLogs); err != nil {
+		return MyConfig{}, fmt.Errorf("chat_logs field in config file %s is invalid: %w", path, err)
+	}
+
 	// sampleConfig.Log = "" if not set
 
-	if sampleConfig.CSign == "" {
-		err := fmt.Errorf("csign field in config file %s must be set", path) //nolint: err113
-
-		return MyConfig{}, err
-	}
-
-	if sampleConfig.DataDir == "" {
-		return MyConfig{}, fmt.Errorf("data_dir field in config file %s must be set", path) //nolint: err113
-	}
-
-	if strings.Split(sampleConfig.DataDir, "")[0] != "/" {
-		executablePath, err := os.Executable()
-
-		if err != nil {
-			err := fmt.Errorf("unable to get current executable path: %w", err)
-
-			return MyConfig{}, err
-		}
-
-		sampleConfig.DataDir = fmt.Sprintf("%s/%s", filepath.Dir(executablePath), sampleConfig.DataDir)
-	}
-
 	return sampleConfig, err
 }
 

internal/moon/db.go

@@ -1,141 +0,0 @@
-package moon
-
-import (
-	"crypto/sha256"
-	"errors"
-	"fmt"
-	"io/fs"
-	"os"
-
-	"github.com/cockroachdb/errors/oserror"
-	"github.com/cockroachdb/pebble"
-)
-
-// StoreKV saves given  key and value in given db.
-func StoreKV(db *pebble.DB, key string, value string) error {
-	var (
-		kArray = []byte(key)
-		vArray = []byte(value)
-	)
-
-	err := db.Set(kArray, vArray, pebble.Sync)
-
-	if err != nil {
-		return err
-	}
-
-	err = db.Flush()
-
-	return err
-}
-
-// FetchV gets value by given key from given db.
-func FetchV(db *pebble.DB, key string) (string, error) {
-	var (
-		kArray = []byte(key)
-
-		vArray      []byte
-		valueString = ""
-	)
-
-	vArray, closer, err := db.Get(kArray)
-
-	if err != nil {
-		return valueString, err
-	}
-
-	valueString = string(vArray)
-	err = closer.Close()
-
-	return valueString, err
-}
-
-// GetValue gets setting for chat from given db name.
-// TODO: should return string, error and should not log error internally.
-func (cfg *MyConfig) GetValue(dataBaseName string, chatID string, setting string) string {
-	var err error
-
-	chatHash := sha256.Sum256([]byte(chatID))
-	dataBaseDir := fmt.Sprintf("%s/db/%s/%x", cfg.DataDir, dataBaseName, chatHash)
-
-	// If db descriptor is not open yet, open it.
-	if _, ok := DB[dataBaseDir]; !ok {
-		var options pebble.Options
-		/*
-		 * By default limit set on megs of data, but not on amount of files thus with defaults we have tons of
-		 * files. Default limit on tomporary files is 500, lets cut it down to 8.
-		 */
-		options.L0CompactionFileThreshold = 8
-
-		if err := os.MkdirAll(dataBaseDir, os.ModePerm); err != nil {
-			Log.Errorf("Unable to create db dir %s: %s", dataBaseDir, err)
-
-			return ""
-		}
-
-		DB[dataBaseDir], err = pebble.Open(dataBaseDir, &options)
-
-		if err != nil {
-			Log.Errorf("Unable to open db, %s: %s\n", dataBaseDir, err)
-
-			return ""
-		}
-	}
-
-	value, err := FetchV(DB[dataBaseDir], setting)
-
-	// If db returns nothing for some reason, just return ampty string.
-	if err != nil {
-		switch {
-		case errors.Is(err, pebble.ErrNotFound):
-			Log.Debugf("Unable to get value for %s: no record found in db %s", setting, dataBaseDir)
-		case errors.Is(err, fs.ErrNotExist):
-			Log.Debugf("Unable to get value for %s: db dir %s does not exist", setting, dataBaseDir)
-		case errors.Is(err, oserror.ErrNotExist):
-			Log.Debugf("Unable to get value for %s: db dir %s does not exist", setting, dataBaseDir)
-		default:
-			Log.Errorf("Unable to get value for %s in db dir %s: %s", setting, dataBaseDir, err)
-		}
-
-		return ""
-	}
-
-	return value
-}
-
-// SaveSetting saves setting name and its value to given db name.
-func (cfg *MyConfig) SaveKeyValue(dataBaseName string, chatID string, setting string, value string) error {
-	var (
-		chatHash    = sha256.Sum256([]byte(chatID))
-		dataBaseDir = fmt.Sprintf("%s/db/%s/%x", cfg.DataDir, dataBaseName, chatHash)
-		err         error
-	)
-
-	// If db descriptor is not open yet, open it.
-	if _, ok := DB[dataBaseDir]; !ok {
-		var options pebble.Options
-		/*
-		 * By default limit set on megs of data, but not on amount of files thus with defaults we have tons of
-		 * files. Default limit on tomporary files is 500, lets cut it down to 8.
-		 */
-		options.L0CompactionFileThreshold = 8
-
-		if err := os.MkdirAll(dataBaseDir, os.ModePerm); err != nil {
-			return fmt.Errorf("unable to create db dir %s: %w", dataBaseDir, err)
-		}
-
-		DB[dataBaseDir], err = pebble.Open(dataBaseDir, &options)
-
-		if err != nil {
-			return fmt.Errorf("unable to open settings db, %s: %w", dataBaseDir, err)
-		}
-	}
-
-	if err := StoreKV(DB[dataBaseDir], setting, value); err != nil {
-		return fmt.Errorf("unable to save setting %s in database %s: %w", setting, dataBaseDir, err)
-	}
-
-	return nil
-}
-
-/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

internal/moon/editedMessage.go

@@ -1,50 +0,0 @@
-package moon
-
-import (
-	"fmt"
-
-	"github.com/NicoNex/echotron/v3"
-)
-
-// editedMessage handles editedMessage type of update.
-func editedMessage(msg *echotron.Update) {
-	AddUser(msg.EditedMessage.From)
-	AddChat(&msg.EditedMessage.Chat)
-
-	// Edited message can contain all kinds of edited attributes that telegram allows to change in message.
-	// Not sure is we should handle all of them.
-	if Cfg.LogChats {
-		if msg.EditedMessage.Text != "" {
-			ChatLog(
-				msg.EditedMessage.Chat.ID,
-				fmt.Sprintf(
-					"User %s %s (username = %s, id = %d) in %s %s (%d) edited message: %s",
-					msg.EditedMessage.From.FirstName,
-					msg.EditedMessage.From.LastName,
-					msg.EditedMessage.From.Username,
-					msg.EditedMessage.From.ID,
-					msg.EditedMessage.Chat.Type,
-					msg.EditedMessage.Chat.Title,
-					msg.EditedMessage.Chat.ID,
-					msg.EditedMessage.Text,
-				),
-			)
-		} else {
-			ChatLog(
-				msg.EditedMessage.Chat.ID,
-				fmt.Sprintf(
-					"User %s %s (username = %s, id = %d) in %s %s (%d) edited message",
-					msg.EditedMessage.From.FirstName,
-					msg.EditedMessage.From.LastName,
-					msg.EditedMessage.From.Username,
-					msg.EditedMessage.From.ID,
-					msg.EditedMessage.Chat.Type,
-					msg.EditedMessage.Chat.Title,
-					msg.EditedMessage.Chat.ID,
-				),
-			)
-		}
-	}
-}
-
-/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

internal/moon/entityGuard.go

@@ -0,0 +1,130 @@
+package moon
+
+import (
+	"bunny-bot/internal/mylog"
+
+	"github.com/NicoNex/echotron/v3"
+	"github.com/davecgh/go-spew/spew"
+)
+
+// The Bot API always returns a User/Chat carrying its id, and echotron validates
+// nothing: lclient.dispatch (network.go:138) is json.Unmarshal plus a check() that
+// reads only the ok/error_code envelope (helpers.go:37). An entity that arrives
+// without an id therefore reaches us as a silent zero, with no layer below to
+// complain about it. These predicates are the whole of our defence.
+//
+// They are SILENT. A refusal is a fact, not an incident: reporting happens once, at
+// the ingress, where the offending update can be dumped whole. A predicate that logs
+// would make one broken update produce one line per entity that happens to meet it.
+
+// usableUser reports whether a user entity is one we can actually address.
+//
+// The test is against zero and not against zero-or-less on purpose: a chat id is
+// legitimately negative for a group or supergroup, so a positivity check would
+// refuse every group chat in the project.
+func usableUser(u *echotron.User) bool {
+	return u != nil && u.ID != 0
+}
+
+// usableChat reports whether a chat entity is one we can actually address, with the
+// same reasoning and the same zero-not-positive rule.
+func usableChat(c *echotron.Chat) bool {
+	return c != nil && c.ID != 0
+}
+
+// usableUpdate reports whether an update is worth dispatching at all.
+//
+// Its branches mirror telegramMsgParser's, in the same order and under the same
+// conditions, so a branch added to one and not the other is visible by reading. That
+// is not tidiness: the alternative is a validator shaped differently from the
+// dispatcher, whose coverage nobody can check by eye.
+//
+// nil is tolerated only where absence is normal. A message with no author is a
+// channel post or a service message, and refusing those would disable chat logging
+// for channels. A nil element inside NewChatMembers is different — the array asserts
+// a newcomer exists — and so is a nil NewChatMember.User, which asserts who changed.
+// A nil OldChatMember.User is tolerated: some transitions have no previous member.
+func usableUpdate(u *echotron.Update) bool {
+	if u == nil {
+		return false
+	}
+
+	switch {
+	case u.ChatMember != nil:
+		return usableChatMember(u.ChatMember)
+	case u.Message != nil:
+		return usableMessage(u.Message)
+	case u.EditedMessage != nil:
+		return usableMessage(u.EditedMessage)
+	default:
+		// Nothing we act on. The dispatcher's spew fallback is where this belongs.
+		return true
+	}
+}
+
+// usableChatMember checks a membership change.
+//
+// Its entities are values, not pointers: ChatMemberUpdated.Chat, .From and both
+// ChatMember structs can never be nil, so there is no nil test to fall back on and
+// the id check is the only thing that can catch a malformed one.
+func usableChatMember(cm *echotron.ChatMemberUpdated) bool {
+	return usableChat(&cm.Chat) &&
+		usableUser(cm.NewChatMember.User) &&
+		(cm.OldChatMember.User == nil || usableUser(cm.OldChatMember.User)) &&
+		usableUser(&cm.From)
+}
+
+// usableMessage checks a message, or an edit of one — the two carry the same fields
+// and are treated the same way.
+//
+// Message.Chat is a value: a zero chat id would otherwise reach elog, which builds
+// chat_logs/<id>/yyyy/mm/dd.txt from it and would silently create a chat_logs/0
+// directory and write every malformed update into it.
+func usableMessage(m *echotron.Message) bool {
+	if !usableChat(&m.Chat) {
+		return false
+	}
+
+	if m.From != nil && !usableUser(m.From) {
+		return false
+	}
+
+	if m.LeftChatMember != nil && !usableUser(m.LeftChatMember) {
+		return false
+	}
+
+	for _, member := range m.NewChatMembers {
+		if !usableUser(member) {
+			return false
+		}
+	}
+
+	// One level deep, and the nested chat only. The nested author is never rendered,
+	// so a zero there is cosmetic; the level bound exists to keep the work constant.
+	if m.PinnedMessage != nil && !usableChat(&m.PinnedMessage.Chat) {
+		return false
+	}
+
+	if m.ReplyToMessage != nil && !usableChat(&m.ReplyToMessage.Chat) {
+		return false
+	}
+
+	return true
+}
+
+// reportBrokenUpdate records an update we are refusing, in the terms it deserves:
+// this is not a user being difficult and not a transient API error, it is an
+// upstream that broke its own contract.
+//
+// The whole update is dumped, once. The dump is the point — the condition is
+// unreproducible and unqueryable, since there is no id to ask Telegram about — and a
+// partial dump of one field out of four would not be evidence of anything.
+func reportBrokenUpdate(u *echotron.Update) {
+	mylog.Error(
+		"Telegram sent an update with an entity carrying no id, refusing it: nothing in it can be greeted, recorded or banned, and an API that sends one is broken",
+	)
+
+	mylog.Error(spew.Sdump(u))
+}
+
+/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

internal/moon/exitCode.go

@@ -0,0 +1,25 @@
+package moon
+
+import "os"
+
+// ExitCode reports the status the process should leave with on a clean return
+// from run(). It is zero while the bot is running normally, and 128 + the
+// signal's number once a termination signal has begun the shutdown — the Unix
+// convention a supervisor can use to tell "stopped with a signal" from "stopped
+// cleanly", which D33 left permanently 0 before it.
+//
+// It never reports an error status: a failed shutdown still returns an error
+// from run(), and main exits 1 over that, the same as before.
+func ExitCode() int {
+	return int(exitCode.Load())
+}
+
+// recordExitCode stores the status the given terminating signal should leave.
+// Only the first termination signal matters — the handler goroutine returns
+// after one — so a later write cannot pass either.
+func recordExitCode(s os.Signal) {
+	// The value is one of a fixed table of 128+n, never a caller-supplied int.
+	exitCode.Store(int32(exitCodeFor(s))) //nolint:gosec // G115: exitCodeFor returns only 130, 131, 143 or 0.
+}
+
+/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

internal/moon/exitCode_unix.go

@@ -0,0 +1,28 @@
+//go:build !windows
+
+package moon
+
+import (
+	"os"
+	"syscall"
+)
+
+// exitCodeFor maps one terminating signal to the status a stopped process
+// should carry: 128 + the signal's number on Unix.
+func exitCodeFor(s os.Signal) int {
+	switch s {
+	case syscall.SIGINT:
+		return signalExitBase + 2
+
+	case syscall.SIGTERM:
+		return signalExitBase + 15
+
+	case syscall.SIGQUIT:
+		return signalExitBase + 3
+
+	default:
+		return 0
+	}
+}
+
+/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

internal/moon/exitCode_windows.go

@@ -0,0 +1,18 @@
+//go:build windows
+
+package moon
+
+import "os"
+
+// exitCodeFor maps one terminating signal to the status a stopped process
+// should carry. Windows has no signal numbers to follow, so it imitates the
+// Unix convention: Ctrl+C is an interrupt, and an interrupt is 130.
+func exitCodeFor(s os.Signal) int {
+	if s == os.Interrupt {
+		return signalExitBase + 2
+	}
+
+	return 0
+}
+
+/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

internal/moon/globals.go

@@ -2,48 +2,143 @@ package moon
 
 import (
 	"os"
+	"strings"
+	"sync"
+	"sync/atomic"
 	"time"
 
 	"github.com/NicoNex/echotron/v3"
 	"github.com/akyoto/cache"
-	"github.com/cockroachdb/pebble"
+)
-	"go.uber.org/zap"
+
+const (
+	// greetCacheTTL is how long a join event or a chat message is retained. It
+	// is deliberately larger than greetReplyWindow so that a sweep occurring at
+	// the far end of the window still has the data it needs to decide.
+	greetCacheTTL = 17 * time.Minute
+
+	// greetReplyWindow is how long a newcomer has to answer the greeting before
+	// being treated as a spammer.
+	greetReplyWindow = 15 * time.Minute
+
+	// greetSweepInterval is how often the sweeper runs.
+	greetSweepInterval = 1 * time.Minute
+)
+
+// signalExitBase is the offset a signal's exit status is derived from. The Unix
+// convention is 128 + n, where n is the signal number, so SIGINT is 130, SIGTERM
+// is 143 and SIGQUIT is 131 — what a shell reports for a process killed by a
+// signal, which is exactly what these statuses imitate.
+const signalExitBase = 128
+
+// The outcomes of a squash attempt. Names state what happened, not how it was
+// detected, so callers never have to reinterpret them.
+const (
+	// banApplied means Telegram accepted the ban.
+	banApplied banOutcome = iota
+
+	// banSuppressed means we did not call the API at all: the debounce already
+	// holds this user, i.e. we banned them moments ago and they are already out.
+	banSuppressed
+
+	// banFailed means we called the API and it refused. The user is still in the
+	// chat and was not retried.
+	banFailed
+)
+
+// The outcomes of retracting a greeting a banned member never answered. The three
+// cases are distinct because "nothing to delete" and "tried and failed" call for
+// different log lines, and only the second is worth an operator's attention.
+const (
+	// nothingToRetract means no remembered greeting id was greater than zero, so
+	// the API was never called.
+	nothingToRetract retractOutcome = iota
+
+	// greetingRetracted means every remembered greeting was deleted.
+	greetingRetracted
+
+	// greetingRetractFailed means at least one deletion was refused by Telegram.
+	// The others still went: a failure never aborts the rest.
+	greetingRetractFailed
 )
 
 var (
+	// htmlReplacer escapes the three characters HTML requires, all three pairs at
+	// once so no ordering can be got wrong: "&" is never re-escaped by the "<" or
+	// ">" rules. Created once because a Replacer is immutable and safe to share.
+	htmlReplacer = strings.NewReplacer("&", "&amp;", "<", "&lt;", ">", "&gt;")
+
 	Cfg     MyConfig
 	SigChan = make(chan os.Signal, 1)
-	Log     *zap.SugaredLogger
+	Tg      tgClient
-	Tg      echotron.API
 
-	// Мапка с открытыми дескрипторами баз настроек.
+	// allowedUpdates is the set of update types Telegram is asked to deliver.
-	DB = make(map[string]*pebble.DB)
+	//
+	// It lives here, not inline in Telegram(), because it must stay in step with
+	// the branches of telegramMsgParser: a type missing from this list makes its
+	// branch unreachable, and nothing at compile time or in the test suite says
+	// so. TestEveryDispatchedUpdateTypeIsRequested exists to catch exactly that.
+	//
+	// chat_join_request is deliberately absent. The community is open and join
+	// requests are not moderated; Telegram auto-approves them when the type is
+	// not requested, which is the intended behaviour.
+	allowedUpdates = []echotron.UpdateType{
+		"message",
+		"edited_message",
+		"chat_member",
+	}
 
-	// SquashedMembers struct with banned members.
+	// SquashedMembers holds members recently banned by us, used as a ban debounce.
 	SquashedMembers = cache.New(1 * time.Minute)
 
-	// NewMembers struct with new members appeared via message with new_chat_members field set.
+	// PendingGreetings holds recorded joins keyed by join timestamp in
-	NewMembers = cache.New(1 * time.Minute)
+	// microseconds. Used by the greeting sweeper to find newcomers who never
+	// answered, and to retract the greetings we asked them.
+	PendingGreetings = newEventCache[pendingGreeting](1 * time.Minute)
 
-	// AppearedMembers struct with new members appeared via ChatMemberUpdated event.
+	// GreetAnswers holds text messages keyed by message timestamp in
-	AppearedMembers = cache.New(60 * time.Minute)
+	// microseconds. Used by the greeting sweeper to find proof that a newcomer
+	// answered the greeting.
+	GreetAnswers = newEventCache[*echotron.Message](1 * time.Minute)
 
-	// PendingCASMembers contains newly comed members that pass first cas check, but still waiting second cas check.
+	// closeOnce guards closeCaches: cache.Cache.Close() sends on an unbuffered
-	PendingCASMembers = cache.New(60 * time.Minute)
+	// channel and therefore must be called exactly once per cache; calling it
+	// twice deadlocks. Each cache is guarded so a failure partway through cannot
+	// turn into a hang on the second attempt.
+	closeOnce sync.Once
 
-	// GreetPending contains people that should answer to bot greet.
+	// stopRequested is closed once, by shutdown, to ask every goroutine that
-	GreetPending = cache.New(1 * time.Minute)
+	// loops to return. One signal for all of them, rather than one per worker:
+	// several workers can come and go (and tests start them deliberately), and
+	// a per-worker channel would have to be closed exactly once each, which is a
+	// panic waiting to happen.
+	stopRequested = make(chan struct{})
 
-	// NoMessagePeople contains people that are never say something in chat.
+	// workers counts the running background goroutines — the update listener and
-	NoMessagePeople = cache.New(1 * time.Minute)
+	// the greeting sweeper. shutdown waits on it, so no worker can be touching a
+	// cache, the logs or the API while the teardown closes them.
+	workers sync.WaitGroup
 
-	// People cache with users info.
+	// workersMu guards workers and stopping. A worker must register before
-	People = cache.New(8 * time.Hour)
+	// shutdown starts waiting, and a goroutine that tries to start afterwards is
+	// refused instead of racing WaitGroup.Add against Wait.
+	workersMu sync.Mutex
+	stopping  bool
 
-	// Groups cache with groups info.
+	// runningWorkers counts registered, not-yet-finished workers, so a caller
-	Groups = cache.New(8 * time.Hour)
+	// can tell "nothing started yet" from "started and still going".
+	runningWorkers atomic.Int32
 
-	ChatList = make([]int64, 0)
+	// exitCode carries the status the process should exit with on a clean
+	// return from run(): zero while the bot is running and sleeping normally,
+	// 128 + signal number once a termination signal has begun the shutdown.
+	// It is an atomic because the handler goroutine writes it and main reads
+	// it, and plain avoids any question of which goroutine wins.
+	exitCode atomic.Int32
+
+	// shutdownOnce guards shutdown so two termination signals arriving together
+	// cannot run the sequence twice.
+	shutdownOnce sync.Once
 )
 
 /* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

internal/moon/greet.go

@@ -0,0 +1,98 @@
+package moon
+
+import (
+	"bunny-bot/internal/mylog"
+	"math/rand/v2"
+	"strings"
+
+	"github.com/NicoNex/echotron/v3"
+)
+
+// greetings are the templates used to greet a newly joined member. The single
+// verb is replaced with the newcomer's computed display name. The fourth one is
+// intentionally kept verbatim despite looking like a typo.
+var greetings = []string{
+	"Дарована, %s. Представьтес, пожалуйста, и расскажите, что вас сюда привело.",
+	"Добро пожаловать в наше скромное коммунити, %s. Представьтес, пожалуйста, и расскажите, что вас сюда привело.",
+	"Доброе утро, день или вечер, %s. Представьтес, пожалуйста, и расскажите, что вас сюда привело.",
+	"Наше вам с кисточкой тут, на канальчике, %s. Представьтес, пожалуйста, и расскажите, что вас сюда привело.",
+}
+
+// displayName builds the name substituted into a greeting for the given user.
+// Preference order: "FirstName LastName", then LastName alone, then FirstName
+// alone, then the username.
+func displayName(u *echotron.User) string {
+	if u == nil {
+		return ""
+	}
+
+	name := strings.TrimSpace(strings.Join([]string{u.FirstName, u.LastName}, " "))
+
+	if name != "" {
+		return name
+	}
+
+	return u.Username
+}
+
+// sendGreeting sends a randomly chosen greeting for the given newcomer into the chat.
+//
+// It returns the id of the greeting it sent and whether it was actually
+// delivered. Both matter: a newcomer who never received the question must not be
+// judged on their silence, and a newcomer who is banned later must have the
+// question taken back down.
+//
+// The mention makes the greeting a clickable link to the newcomer, and the
+// message is sent in HTML mode because that is the only mode in which an
+// arbitrary display name can be embedded safely — see greetMention.
+func sendGreeting(chatid int64, u *echotron.User) (int, bool) {
+	// Everything below dereferences u, including the error paths that name the
+	// user, so an entity with no id is refused here rather than crashing later.
+	if !usableUser(u) {
+		return 0, false
+	}
+
+	// math/rand is sufficient here: we pick one of four static templates, not a
+	// secret or a security-relevant value.
+	greeting := greetings[rand.IntN(len(greetings))] //nolint: gosec // G404
+
+	text := strings.Replace(greeting, "%s", greetingAddressee(u), 1)
+
+	res, err := Tg.SendMessage(text, chatid, &echotron.MessageOptions{
+		ParseMode: echotron.HTML,
+	})
+	if err != nil {
+		mylog.Errorf("GREETING NOT DELIVERED to user %d in chat %d: %s", u.ID, chatid, err)
+
+		return 0, false
+	}
+
+	mylog.Debugf("Greeted user %d in chat %d", u.ID, chatid)
+
+	// A response carrying no message still means the greeting was delivered; we
+	// simply have no id to retract later.
+	return sentMessageID(res), true
+}
+
+// greetingAddressee is the name the template's %s is replaced with: a mention of
+// the newcomer when they can be mentioned, and their plain display name when they
+// cannot, so an unmentionable user is still greeted rather than skipped.
+func greetingAddressee(u *echotron.User) string {
+	if mention := greetMention(u); mention != "" {
+		return mention
+	}
+
+	return displayName(u)
+}
+
+// sentMessageID pulls the id out of a send response, tolerating a response that
+// carries no message at all.
+func sentMessageID(res echotron.APIResponseMessage) int {
+	if res.Result == nil {
+		return 0
+	}
+
+	return res.Result.ID
+}
+
+/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

internal/moon/greetCache.go

@@ -0,0 +1,52 @@
+package moon
+
+import (
+	"time"
+
+	"github.com/akyoto/cache"
+)
+
+// newEventCache creates an eventCache whose expired entries are swept every
+// cleaningInterval.
+func newEventCache[T any](cleaningInterval time.Duration) *eventCache[T] {
+	return &eventCache[T]{cache: cache.New(cleaningInterval)}
+}
+
+// Set stores value under the microsecond key with the given TTL.
+func (m *eventCache[T]) Set(micros int64, value T, ttl time.Duration) {
+	m.cache.Set(micros, value, ttl)
+}
+
+// Snapshot returns a typed copy of all non-expired entries keyed by their
+// microsecond timestamp. Callers can range over the returned map without
+// performing any interface{} conversions.
+func (m *eventCache[T]) Snapshot() map[int64]T {
+	out := make(map[int64]T)
+
+	m.cache.Range(
+		func(k, v any) bool {
+			key, ok := k.(int64)
+			if !ok {
+				return true
+			}
+
+			value, ok := v.(T)
+			if !ok {
+				return true
+			}
+
+			out[key] = value
+
+			return true
+		},
+	)
+
+	return out
+}
+
+// Delete removes the entry stored under the given microsecond key.
+func (m *eventCache[T]) Delete(micros int64) {
+	m.cache.Delete(micros)
+}
+
+/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

internal/moon/greetMention.go

@@ -0,0 +1,81 @@
+package moon
+
+import (
+	"strconv"
+	"strings"
+
+	"github.com/NicoNex/echotron/v3"
+)
+
+// greetMention renders a clickable mention of a newcomer for use inside a
+// greeting, or "" when the user cannot be mentioned at all.
+//
+// The mode is HTML, not markdown, and that is a correctness decision rather than
+// a preference: the label is a name Telegram gave us, and MarkdownV2 requires
+// escaping eighteen characters — "-" among them — so an ordinary "Анна-Мария"
+// would be rejected with a 400. A rejected greeting makes newChatMembers delete
+// the join event, which is how one hyphen would disable the only detector. HTML
+// needs "&", "<" and ">" escaped, and those essentially never occur in a name.
+//
+// The mention works without a @username, which is the common case for someone who
+// has only just joined a public group.
+func greetMention(u *echotron.User) string {
+	if u == nil || u.ID <= 0 {
+		return ""
+	}
+
+	var b strings.Builder
+
+	b.WriteString(`<a href="tg://user?id=`)
+	b.WriteString(strconv.FormatInt(u.ID, 10))
+	b.WriteString(`">`)
+
+	for _, r := range mentionLabel(u) {
+		switch r {
+		case '&':
+			b.WriteString("&amp;")
+		case '<':
+			b.WriteString("&lt;")
+		case '>':
+			b.WriteString("&gt;")
+		default:
+			b.WriteRune(r)
+		}
+	}
+
+	b.WriteString("</a>")
+
+	return b.String()
+}
+
+// escapeHTML escapes the three characters HTML requires. It exists for callers
+// that assemble markup themselves; the greeting path uses greetMention's own
+// single pass so the anchor and the label cannot drift apart.
+func escapeHTML(s string) string {
+	return htmlReplacer.Replace(s)
+}
+
+// mentionLabel is the text shown inside the mention link, in the preference order
+// a reader expects: full name, then either name alone, then the @username, and
+// finally the bare id so the link is never empty.
+func mentionLabel(u *echotron.User) string {
+	if name := strings.TrimSpace(u.FirstName + " " + u.LastName); name != "" {
+		return name
+	}
+
+	if u.LastName != "" {
+		return u.LastName
+	}
+
+	if u.FirstName != "" {
+		return u.FirstName
+	}
+
+	if u.Username != "" {
+		return "@" + u.Username
+	}
+
+	return strconv.FormatInt(u.ID, 10)
+}
+
+/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

internal/moon/greetSweep.go

@@ -0,0 +1,206 @@
+package moon
+
+import (
+	"bunny-bot/internal/mylog"
+	"time"
+
+	"github.com/NicoNex/echotron/v3"
+)
+
+// SweepGreetings runs until a termination signal stops it. Every
+// greetSweepInterval it looks at join events that are at least greetReplyWindow
+// old and decides whether each newcomer ever answered the greeting:
+//
+//   - answered (a message from that user exists after the join, within the
+//     reply window) -> newcomer is legitimate, drop the join event(s);
+//   - not answered -> newcomer is a spammer, ban them and drop the join
+//     event(s).
+//
+// A user may appear in PendingGreetings more than once (rejoins). For every user we
+// re-query both caches so that a single answer rescues all of their join events
+// and a single unanswered join bans them once.
+//
+// The stop is checked between passes and never during one, so a sweep in
+// progress always finishes and can never be running while the caches are being
+// closed underneath it.
+func SweepGreetings() {
+	if !startWorker() {
+		return
+	}
+
+	defer stopWorker()
+
+	timer := time.NewTimer(greetSweepInterval)
+	defer timer.Stop()
+
+	for {
+		select {
+		case <-stopRequested:
+			mylog.Info("Greeting sweeper stopped")
+
+			return
+
+		case <-timer.C:
+			sweepGreetings()
+
+			timer.Reset(greetSweepInterval)
+		}
+	}
+}
+
+// sweepGreetings performs a single sweep pass against the current wall clock.
+func sweepGreetings() {
+	sweepGreetingsAt(time.Now())
+}
+
+// sweepGreetingsAt performs a single sweep pass as of now. Taking the clock as
+// a parameter is what makes the reply-window boundary testable without sleeping
+// for 15 minutes.
+func sweepGreetingsAt(now time.Time) {
+	joins := PendingGreetings.Snapshot()
+	messages := GreetAnswers.Snapshot()
+
+	windowMicros := greetReplyWindow.Microseconds()
+	nowMicros := now.UnixMicro()
+
+	// Group stale join events per (chat, user). A user is a distinct identity in
+	// every chat: being silent in one chat says nothing about another, and
+	// answering in one chat must not rescue them in another.
+	staleByMember := make(map[chatUser][]joinRef)
+
+	for joinMicros, recorded := range joins {
+		join := recorded.join
+
+		// Not stale yet.
+		if nowMicros-joinMicros <= windowMicros {
+			continue
+		}
+
+		// A join event carries the newcomer's user in NewChatMembers.
+		if join == nil || len(join.NewChatMembers) == 0 {
+			continue
+		}
+
+		user := join.NewChatMembers[0]
+
+		// An entity with no id cannot be banned, and the dump has already been
+		// taken. Drop the join rather than keeping it: re-reporting the same
+		// broken update every minute for the rest of its TTL is noise that
+		// teaches an operator to ignore this log.
+		if !usableUser(user) {
+			PendingGreetings.Delete(joinMicros)
+
+			continue
+		}
+
+		member := chatUser{chatID: join.Chat.ID, userID: user.ID}
+		staleByMember[member] = append(staleByMember[member], joinRef{
+			micros:     joinMicros,
+			chat:       join.Chat,
+			user:       user,
+			greetingID: recorded.greetingID,
+		})
+	}
+
+	for member, refs := range staleByMember {
+		// A single answer rescues every join this user has *in this chat*, no
+		// matter which join the reply was actually answering.
+		answered := false
+
+		for _, ref := range refs {
+			if userAnswered(member.chatID, member.userID, ref.micros, windowMicros, messages) {
+				answered = true
+
+				break
+			}
+		}
+
+		if answered {
+			mylog.Debugf(
+				"User %d answered the greeting in chat %d, not banning",
+				member.userID,
+				member.chatID,
+			)
+		} else {
+			mylog.Infof(
+				"User %d did not answer the greeting in chat %d within %s, banning as spammer",
+				member.userID,
+				member.chatID,
+				greetReplyWindow,
+			)
+
+			// squash already logs every outcome at the level it deserves: a
+			// ban that landed is a debug line and a duplicate we deliberately
+			// suppressed is an info line. Only a refused ban needs anything
+			// added here, because it is the one case where the user is still
+			// in the chat and will never be re-judged.
+			outcome := squash(&refs[0].chat, refs[0].user)
+			if outcome == banFailed {
+				mylog.Errorf(
+					"BAN FAILED for user %d in chat %d — they stayed in the chat and were not re-judged",
+					member.userID,
+					member.chatID,
+				)
+			} else {
+				// The user is out of the chat, so every question we asked them
+				// is a leftover. That includes the debounced repeat: they are
+				// banned either way, and the room must not be left asking a
+				// question nobody can answer.
+				retractAskedGreetings(refs)
+			}
+		}
+
+		// Drop every join event of this user *in this chat* so the next pass
+		// does not re-decide them. Joins in their other chats are untouched.
+		forgetPendingGreetings(member.chatID, member.userID)
+	}
+}
+
+// userAnswered reports whether a message authored by userID in chatID exists
+// in messages with a timestamp in (joinMicros, joinMicros+windowMicros].
+//
+// Both the chat and the author must match: the newcomer must answer in the same
+// chat they were greeted in. Messages sent before the join do not count either,
+// because the greeting is only sent in response to the join.
+func userAnswered(
+	chatID int64,
+	userID int64,
+	joinMicros int64,
+	windowMicros int64,
+	messages map[int64]*echotron.Message,
+) bool {
+	for msgMicros, msg := range messages {
+		if msg == nil || msg.From == nil {
+			continue
+		}
+
+		if msg.From.ID != userID || msg.Chat.ID != chatID {
+			continue
+		}
+
+		if msgMicros > joinMicros && msgMicros <= joinMicros+windowMicros {
+			return true
+		}
+	}
+
+	return false
+}
+
+// forgetPendingGreetings removes every PendingGreetings entry whose newcomer is userID in the
+// given chat. Joins by the same user in other chats are deliberately preserved:
+// those are independent memberships with their own greetings and their own
+// verdicts.
+func forgetPendingGreetings(chatID int64, userID int64) {
+	for joinMicros, recorded := range PendingGreetings.Snapshot() {
+		join := recorded.join
+		if join == nil || join.Chat.ID != chatID || len(join.NewChatMembers) == 0 {
+			continue
+		}
+
+		if join.NewChatMembers[0] != nil && join.NewChatMembers[0].ID == userID {
+			PendingGreetings.Delete(joinMicros)
+		}
+	}
+}
+
+/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

internal/moon/leftChatMember.go

@@ -1,37 +0,0 @@
-package moon
-
-import (
-	"fmt"
-	"time"
-
-	"github.com/NicoNex/echotron/v3"
-)
-
-// leftChatMember handles left_chat_member event.
-func leftChatMember(msg *echotron.Update) {
-	Log.Infof(
-		"User %s %s (username = %s, id = %d) left %s %s (%d)",
-		msg.Message.LeftChatMember.FirstName,
-		msg.Message.LeftChatMember.LastName,
-		msg.Message.LeftChatMember.Username,
-		msg.Message.LeftChatMember.ID,
-		msg.Message.Chat.Type,
-		msg.Message.Chat.Title,
-		msg.Message.Chat.ID,
-	)
-
-	// Save people that left chat to pebbledb. To preserve their state.
-	// Left folks can eventually return and in chat_mamber_updated event their state will be left->member.
-	if err := Cfg.SaveKeyValue(
-		"LeftMembers",
-		fmt.Sprintf("%d", msg.Message.Chat.ID),
-		fmt.Sprintf("%d", msg.Message.LeftChatMember.ID),
-		fmt.Sprintf("%d", time.Now().Unix()),
-	); err != nil {
-		Log.Error(err)
-	} else {
-		Log.Infof("Info about left userid %d saved to left users db", msg.Message.LeftChatMember.ID)
-	}
-}
-
-/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

internal/moon/logOnlyMessage.go

@@ -0,0 +1,24 @@
+package moon
+
+import (
+	"bunny-bot/internal/elog"
+
+	"github.com/NicoNex/echotron/v3"
+)
+
+// logOnlyMessage writes a message to the chat log and nothing else.
+//
+// It handles the messages the greeting detector must NOT count as answers: a photo with no
+// caption, a sticker, a location, a leave, a title change. They are real events in the room
+// and belong in the log — which is also what makes a later ban diagnosable, since a moderator
+// can read what the member actually replied with — but a reply with no text is not an
+// introduction, and recording it in GreetAnswers would let any attachment rescue a spammer
+// from the ban.
+//
+// The dispatcher has already established that this is neither a join nor a text message; it
+// is not re-checked here.
+func logOnlyMessage(msg *echotron.Update) {
+	elog.LogMessage(msg.Message)
+}
+
+/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

internal/moon/newChatMembers.go

@@ -1,31 +1,61 @@
 package moon
 
 import (
-	"fmt"
-	"slices"
 	"time"
 
+	"bunny-bot/internal/elog"
+	"bunny-bot/internal/mylog"
+
 	"github.com/NicoNex/echotron/v3"
 )
 
 // newChatMembers parses NewChatMember event.
 func newChatMembers(msg *echotron.Update) {
-	// If there is no such chatid, means that no such cache too.
+	// Telegram normally reports one newcomer per service message. If that ever
-	Log.Debugf(
+	// stops being true, the sweeper would only judge the first member, so make it
-		"Add %s %s (%d) to list of known chats",
+	// visible in the logs rather than silently ignoring the rest.
-		msg.Message.Chat.Type,
+	if len(msg.Message.NewChatMembers) > 1 {
-		msg.Message.Chat.Title,
+		mylog.Warnf(
-		msg.Message.Chat.ID,
+			"MULTI-MEMBER JOIN in chat %d: %d newcomers in one event, only the first is moderated",
-	)
+			msg.Message.Chat.ID,
+			len(msg.Message.NewChatMembers),
+		)
+	}
 
-	ChatList = append(ChatList, msg.Message.Chat.ID)
+	// Keep only the newcomers we can actually address. In production the ingress has
-	slices.Sort(ChatList)
+	// already refused such an update; this filter is the nil-safety that keeps a
-	ChatList = slices.Compact(ChatList)
+	// direct caller from recording somebody we cannot name.
-
+	newcomers := make([]*echotron.User, 0, len(msg.Message.NewChatMembers))
-	AddChat(&msg.Message.Chat)
 
 	for _, user := range msg.Message.NewChatMembers {
-		Log.Infof(
+		if !usableUser(user) {
+			continue
+		}
+
+		newcomers = append(newcomers, user)
+	}
+
+	if len(newcomers) == 0 {
+		return
+	}
+
+	// Remember the join so the greeting sweeper can tell later whether this
+	// newcomer answered, and so a ban can retract the greeting we sent.
+	//
+	// The join is recorded before the greeting exists, so the greeting id starts
+	// at zero and is filled in once the send returns. A join whose greeting was
+	// never delivered therefore carries zero, which is exactly the marker
+	// retractGreetings skips.
+	joinMicros := time.Now().UnixMicro()
+	PendingGreetings.Set(joinMicros, pendingGreeting{join: msg.Message}, greetCacheTTL)
+
+	// Record the join in the chat log before anything else, so the arrival is on
+	// record even if the greeting later fails. elog writes one line per newcomer
+	// itself, so this is called once for the whole event rather than per user.
+	elog.LogMessage(msg.Message)
+
+	for _, user := range newcomers {
+		mylog.Infof(
 			"User %s %s (username = %s, id = %d) joined %s %s (%d)",
 			user.FirstName,
 			user.LastName,
@@ -36,60 +66,23 @@ func newChatMembers(msg *echotron.Update) {
 			msg.Message.Chat.ID,
 		)
 
-		AddUser(user)
+		// Greet the newcomer, so that a spammer who never answers becomes
+		// detectable. If the greeting could not be delivered the newcomer never
+		// saw the question, so they must not be judged on their silence.
+		greetingID, delivered := sendGreeting(msg.Message.Chat.ID, user)
+		if !delivered {
+			PendingGreetings.Delete(joinMicros)
 
-		casBanned, err := CasCheckID(user.ID)
+			return
-
-		if err != nil {
-			Log.Error(err)
 		}
 
-		// Ban user if cas report it as banned.
+		// Re-record the join carrying the greeting's id. A member who joined
-		if casBanned {
+		// twice has one recorded join per greeting, so each id is retracted
-			Log.Infof(
+		// separately.
-				"Ban user %s %s (username = %s, id = %d) in %s %s (%d) by CAS blacklist",
+		PendingGreetings.Set(joinMicros, pendingGreeting{
-				user.FirstName,
+			join:       msg.Message,
-				user.LastName,
+			greetingID: greetingID,
-				user.Username,
+		}, greetCacheTTL)
-				user.ID,
-				msg.Message.Chat.Type,
-				msg.Message.Chat.Title,
-				msg.Message.Chat.ID,
-			)
-
-			squash(
-				msg.Message.Chat.ID,
-				user.ID,
-			)
-		}
-
-		// Do not return on prev. step.
-
-		Log.Infof(
-			"Skip banning user %s %s (username = %s, id = %d) in %s %s (%d) because it was not found in CAS blacklist",
-			user.FirstName,
-			user.LastName,
-			user.Username,
-			user.ID,
-			msg.Message.Chat.Type,
-			msg.Message.Chat.Title,
-			msg.Message.Chat.ID,
-		)
-
-		Log.Infof(
-			"Add user %s %s (username = %s, id = %d) in %s %s (%d) to NewMember DB",
-			user.FirstName,
-			user.LastName,
-			user.Username,
-			user.ID,
-			msg.Message.Chat.Type,
-			msg.Message.Chat.Title,
-			msg.Message.Chat.ID,
-		)
-
-		key := fmt.Sprintf("%d+%d", msg.Message.Chat.ID, user.ID)
-		NewMembers.Set(key, time.Now().Unix(), 1*time.Minute)
-		PendingCASMembers.Set(key, time.Now().Unix(), 30*time.Minute)
 	}
 }
 

internal/moon/photo.go

@@ -1,50 +0,0 @@
-package moon
-
-import (
-	"fmt"
-
-	"github.com/NicoNex/echotron/v3"
-)
-
-// photo handles messages with photos.
-func photo(msg *echotron.Update) {
-	AddChat(&msg.Message.Chat)
-	AddUser(msg.Message.From)
-
-	if Cfg.LogChats {
-		if msg.Message.Text != "" {
-			ChatLog(
-				msg.Message.Chat.ID,
-				fmt.Sprintf(
-					"User %s %s (username = %s, id = %d) sent %d photos to %s %s (%d) with text: %s",
-					msg.Message.From.FirstName,
-					msg.Message.From.LastName,
-					msg.Message.From.Username,
-					msg.Message.From.ID,
-					len(msg.Message.Photo),
-					msg.Message.Chat.Type,
-					msg.Message.Chat.Title,
-					msg.Message.Chat.ID,
-					msg.Message.Text,
-				),
-			)
-		} else {
-			ChatLog(
-				msg.Message.Chat.ID,
-				fmt.Sprintf(
-					"User %s %s (username = %s, id = %d) sent %d photos to %s %s (%d)",
-					msg.Message.From.FirstName,
-					msg.Message.From.LastName,
-					msg.Message.From.Username,
-					msg.Message.From.ID,
-					len(msg.Message.Photo),
-					msg.Message.Chat.Type,
-					msg.Message.Chat.Title,
-					msg.Message.Chat.ID,
-				),
-			)
-		}
-	}
-}
-
-/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

internal/moon/reaction.go

@@ -1,68 +0,0 @@
-package moon
-
-import (
-	"fmt"
-
-	"github.com/NicoNex/echotron/v3"
-)
-
-func reaction(msg *echotron.Update) {
-	AddChat(&msg.MessageReaction.Chat)
-	AddUser(&msg.MessageReaction.User)
-
-	if Cfg.LogChats {
-		var reactionString string
-
-		for _, item := range msg.MessageReaction.NewReaction {
-			switch item.Type {
-			case "Emoji":
-				reactionString += item.Emoji
-			case "CustomEmoji":
-				reactionString += item.CustomEmoji
-			}
-		}
-
-		if reactionString != "" {
-			ChatLog(
-				msg.MessageReaction.Chat.ID,
-				fmt.Sprintf(
-					"User %s %s (username = %s, id = %d) set reaction %s to chat %s %s (%d)",
-					msg.MessageReaction.User.FirstName,
-					msg.MessageReaction.User.LastName,
-					msg.MessageReaction.User.Username,
-					msg.MessageReaction.User.ID,
-					reactionString,
-					msg.MessageReaction.Chat.Type,
-					msg.MessageReaction.Chat.Title,
-					msg.MessageReaction.Chat.ID,
-				),
-			)
-		} else {
-			for _, item := range msg.MessageReaction.OldReaction {
-				switch item.Type {
-				case "Emoji":
-					reactionString += item.Emoji
-				case "CustomEmoji":
-					reactionString += item.CustomEmoji
-				}
-			}
-
-			ChatLog(
-				msg.MessageReaction.Chat.ID,
-				fmt.Sprintf(
-					"User %s %s (username = %s, id = %d) removed reaction %s to chat %s %s (%d)",
-					msg.MessageReaction.User.FirstName,
-					msg.MessageReaction.User.LastName,
-					msg.MessageReaction.User.Username,
-					msg.MessageReaction.User.ID,
-					reactionString,
-					msg.MessageReaction.Chat.Type,
-					msg.MessageReaction.Chat.Title,
-					msg.MessageReaction.Chat.ID,
-				),
-			)
-		}
-	}
-}
-
-/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

internal/moon/reactionCount.go

@@ -1,24 +0,0 @@
-package moon
-
-import (
-	"fmt"
-
-	"github.com/NicoNex/echotron/v3"
-	"github.com/davecgh/go-spew/spew"
-)
-
-// reactionCount handles reactionCount event.
-func reactionCount(msg *echotron.Update) {
-	ChatLog(
-		msg.MessageReaction.Chat.ID,
-		fmt.Sprintf(
-			"Reaction count event in %s %s (%d): %s",
-			msg.MessageReaction.Chat.Type,
-			msg.MessageReaction.Chat.Title,
-			msg.MessageReaction.Chat.ID,
-			spew.Sdump(msg),
-		),
-	)
-}
-
-/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

internal/moon/retractGreeting.go

@@ -0,0 +1,119 @@
+package moon
+
+import (
+	"bunny-bot/internal/elog"
+	"bunny-bot/internal/mylog"
+
+	"github.com/NicoNex/echotron/v3"
+)
+
+// retractGreetings deletes the greetings we sent to a member who has just been
+// banned for never answering them, so the room is not left asking a question of
+// someone who no longer has access.
+//
+// Ids <= 0 mean "no greeting was ever sent for that join" and are skipped
+// silently. A deletion Telegram refuses is logged and the pass continues: the
+// greeting is not worth abandoning the rest over, and a failed retraction never
+// changes the ban that has already landed.
+func retractGreetings(chat *echotron.Chat, user *echotron.User, greetingIDs []int) retractOutcome {
+	retracted, failed := 0, 0
+
+	for _, id := range greetingIDs {
+		if id <= 0 {
+			continue
+		}
+
+		if _, err := Tg.DeleteMessage(chat.ID, id); err != nil {
+			failed++
+
+			reportRetractFailure(chat, user, id, err)
+
+			continue
+		}
+
+		retracted++
+
+		elog.LogGreetingRetracted(chat, user, id)
+	}
+
+	return retractOutcomeFor(retracted, failed)
+}
+
+// reportRetractFailure logs a refused deletion. It is a WARN rather than an ERROR
+// because the ban it followed already succeeded and nothing depends on the
+// greeting being gone.
+func reportRetractFailure(chat *echotron.Chat, user *echotron.User, id int, err error) {
+	mylog.Warnf(
+		"Left an unanswered greeting in place: cannot delete message %d in chat %d for user %d: %s",
+		id,
+		chat.ID,
+		userIDOf(user),
+		err,
+	)
+}
+
+// retractAskedGreetings takes down the questions this member was asked and never
+// answered. One id per recorded join, so a member who joined twice had two
+// greetings and both go.
+func retractAskedGreetings(refs []joinRef) {
+	ids := make([]int, 0, len(refs))
+	for _, ref := range refs {
+		ids = append(ids, ref.greetingID)
+	}
+
+	chat, user := refs[0].chat, refs[0].user
+
+	switch retractGreetings(&chat, user, ids) {
+	case greetingRetracted:
+		mylog.Debugf(
+			"Deleted %d unanswered greeting(s) asked of user %d in chat %d",
+			countNamedGreetings(ids),
+			user.ID,
+			chat.ID,
+		)
+	case greetingRetractFailed:
+		mylog.Warnf(
+			"Banned user %d in chat %d, but could not delete every greeting we asked them",
+			user.ID,
+			chat.ID,
+		)
+	case nothingToRetract:
+	}
+}
+
+// countNamedGreetings counts the ids that actually named a message; the rest
+// mean no greeting was ever delivered.
+func countNamedGreetings(ids []int) int {
+	n := 0
+
+	for _, id := range ids {
+		if id > 0 {
+			n++
+		}
+	}
+
+	return n
+}
+
+// retractOutcomeFor collapses the two counts into the single value callers act on.
+func retractOutcomeFor(retracted int, failed int) retractOutcome {
+	switch {
+	case retracted > 0 && failed == 0:
+		return greetingRetracted
+	case failed > 0:
+		return greetingRetractFailed
+	default:
+		return nothingToRetract
+	}
+}
+
+// userIDOf names a user in a log line without dereferencing a possibly nil one.
+func userIDOf(user *echotron.User) int64 {
+	if user == nil {
+		return 0
+	}
+
+	return user.ID
+}
+
+/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

internal/moon/runAndBan.go

@@ -1,293 +0,0 @@
-package moon
-
-import (
-	"fmt"
-	"slices"
-	"strconv"
-	"strings"
-	"time"
-)
-
-func RunAndBan() {
-	for {
-		var (
-			ListAppearedMembers, ListNewMembers []string
-		)
-
-		AppearedMembers.Range(
-			func(k, _ any) bool {
-				ListAppearedMembers = append(ListAppearedMembers, k.(string))
-
-				return true
-			},
-		)
-
-		NewMembers.Range(
-			func(k, _ any) bool {
-				ListNewMembers = append(ListNewMembers, k.(string))
-
-				return true
-			},
-		)
-
-		for _, AppearedMember := range ListAppearedMembers {
-			for _, NewMember := range ListNewMembers {
-				if AppearedMember == NewMember {
-					AppearedMembers.Delete(AppearedMember)
-					NewMembers.Delete(NewMember)
-				}
-			}
-		}
-
-		currentTime := time.Now().Unix()
-
-		ListAppearedMembers = ListAppearedMembers[:0]
-		ListNewMembers = ListNewMembers[:0]
-		ListAppearedMembers = slices.Clip(ListAppearedMembers)
-		ListNewMembers = slices.Clip(ListNewMembers)
-
-		AppearedMembers.Range(
-			func(k, _ any) bool {
-				ListAppearedMembers = append(ListAppearedMembers, k.(string))
-
-				return true
-			},
-		)
-
-		NewMembers.Range(
-			func(k, _ any) bool {
-				ListNewMembers = append(ListNewMembers, k.(string))
-
-				return true
-			},
-		)
-
-		for _, AppearedMember := range ListAppearedMembers {
-			eventTimestamp, found := AppearedMembers.Get(AppearedMember)
-
-			if !found {
-				continue
-			}
-
-			eventTimeDelta := currentTime - eventTimestamp.(int64)
-
-			// Spammer?
-			if eventTimeDelta > 10 {
-				item := strings.Split(AppearedMember, "+")
-				chatid, err := strconv.ParseInt(item[0], 10, 64)
-
-				if err != nil {
-					Log.Errorf("Got broken key in AppearedMembers cache: %s", AppearedMember)
-
-					continue
-				}
-
-				userid, err := strconv.ParseInt(item[1], 10, 64)
-
-				if err != nil {
-					Log.Errorf("Got broken key in AppearedMembers cache: %s", AppearedMember)
-
-					continue
-				}
-
-				user := GetUser(userid)
-				chat := GetChat(chatid)
-
-				Log.Infof(
-					"Ban user %s %s (username = %s, id = %d) in chat %s %s (%d) that have no newChatMember message since chatMemberUpdated for more than 10 seconds",
-					user.FirstName,
-					user.LastName,
-					user.Username,
-					userid,
-					chat.Type,
-					chat.Title,
-					chatid,
-				)
-
-				squash(chatid, userid)
-
-				AppearedMembers.Delete(AppearedMember)
-			}
-		}
-
-		for _, NewMember := range ListNewMembers {
-			eventTimestamp, found := NewMembers.Get(NewMember)
-
-			if !found {
-				continue
-			}
-
-			eventTimeDelta := currentTime - eventTimestamp.(int64)
-
-			// Spammer?
-			if eventTimeDelta > 10 {
-				item := strings.Split(NewMember, "+")
-				chatid, err := strconv.ParseInt(item[0], 10, 64)
-
-				if err != nil {
-					Log.Errorf("Got broken key in AppearedMembers cache: %s", NewMember)
-
-					continue
-				}
-
-				userid, err := strconv.ParseInt(item[1], 10, 64)
-
-				if err != nil {
-					Log.Errorf("Got broken key in AppearedMembers cache: %s", NewMember)
-
-					continue
-				}
-
-				user := GetUser(userid)
-				chat := GetChat(chatid)
-
-				Log.Debugf("Clean newChatMember event record older than 10 seconds for user user %s %s (username = %s, id = %d) in chat %s %s (%d)",
-					user.FirstName,
-					user.LastName,
-					user.Username,
-					userid,
-					chat.Type,
-					chat.Title,
-					chatid,
-				)
-
-				NewMembers.Delete(NewMember)
-			}
-		}
-
-		time.Sleep(10 * time.Second)
-	}
-}
-
-func PeriodicCASCheck() {
-	for {
-		var (
-			ListPendingMembers []string
-		)
-
-		PendingCASMembers.Range(
-			func(k, _ any) bool {
-				ListPendingMembers = append(ListPendingMembers, k.(string))
-
-				return true
-			},
-		)
-
-		currentTime := time.Now().Unix()
-
-		for _, PendingMember := range ListPendingMembers {
-			eventTimestamp, found := PendingCASMembers.Get(PendingMember)
-
-			if !found {
-				continue
-			}
-
-			eventTimeDelta := currentTime - eventTimestamp.(int64)
-
-			// It's time to check for second time this id if it was banned in CAS.
-			if eventTimeDelta > 300 {
-				item := strings.Split(PendingMember, "+")
-				chatid, err := strconv.ParseInt(item[0], 10, 64)
-
-				if err != nil {
-					Log.Errorf("Got broken key in PendingMembers cache: %s", PendingMember)
-
-					continue
-				}
-
-				userid, err := strconv.ParseInt(item[1], 10, 64)
-
-				if err != nil {
-					Log.Errorf("Got broken key in PendingMembers cache: %s", PendingMember)
-
-					continue
-				}
-
-				value := Cfg.GetValue(
-					"BannedMembers",
-					fmt.Sprintf("%d", chatid),
-					fmt.Sprintf("%d", userid),
-				)
-
-				user := GetUser(userid)
-				chat := GetChat(chatid)
-
-				if value != "" {
-					Log.Debugf("User %s %s (username = %s, id = %d) in %s %s (%d) is already banned, skip cas check.",
-						user.FirstName,
-						user.LastName,
-						user.Username,
-						userid,
-						chat.Type,
-						chat.Title,
-						chatid,
-					)
-
-					Log.Debugf(
-						"Remove user %s %s (username = %s, id = %d) in %s %s (%d) from PendingCASMembers",
-						user.FirstName,
-						user.LastName,
-						user.Username,
-						userid,
-						chat.Type,
-						chat.Title,
-						chatid,
-					)
-
-					key := fmt.Sprintf("%d+%d", chatid, userid)
-					PendingCASMembers.Delete(key)
-
-					return
-				}
-
-				Log.Debugf("Checking user %s %s (username = %s, id = %d) in %s %s (%d) for second time in CAS",
-					user.FirstName,
-					user.LastName,
-					user.Username,
-					userid,
-					chat.Type,
-					chat.Title,
-					chatid,
-				)
-
-				banned, err := CasCheckID(userid)
-
-				if err != nil {
-					Log.Error(err)
-				}
-
-				if banned {
-					Log.Infof(
-						"Ban user %s %s (username = %s, id = %d) in %s %s (%d) by CAS blacklist",
-						user.FirstName,
-						user.LastName,
-						user.Username,
-						userid,
-						chat.Type,
-						chat.Title,
-						chatid,
-					)
-
-					squash(chatid, userid)
-				}
-
-				Log.Debugf(
-					"Remove user %s %s (username = %s, id = %d) in %s %s (%d) from PendingCASMembers",
-					user.FirstName,
-					user.LastName,
-					user.Username,
-					userid,
-					chat.Type,
-					chat.Title,
-					chatid,
-				)
-
-				PendingCASMembers.Delete(PendingMember)
-			}
-		}
-
-		time.Sleep(10 * time.Minute)
-	}
-}
-
-/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

internal/moon/shutdown.go

@@ -0,0 +1,97 @@
+package moon
+
+import (
+	"bunny-bot/internal/elog"
+	"bunny-bot/internal/mylog"
+	"fmt"
+
+	"github.com/akyoto/cache"
+)
+
+// closeCaches shuts down every cache sweeper goroutine.
+//
+// Safe only once the update listener and the greeting sweeper have stopped:
+// cache.Close() reassigns the underlying sync.Map, which is a data race against
+// any concurrent reader. shutdown() enforces that order.
+func closeCaches() {
+	closeOnce.Do(func() {
+		for _, c := range []*cache.Cache{
+			SquashedMembers,
+			PendingGreetings.cache, GreetAnswers.cache,
+		} {
+			c.Close()
+		}
+	})
+}
+
+// Shutdown performs the ordered teardown, waits for it to finish, and reports
+// whether every stage of it succeeded.
+//
+// main calls it after Telegram returns, for two reasons. The listener returning is
+// not by itself a finished shutdown — a signal handler may be halfway through the
+// sequence at that moment, and main returning would end the process underneath it,
+// skipping stages and closing the log twice. And if the listener stopped for any
+// other reason, nobody would run the sequence at all.
+//
+// It is safe to call from both goroutines: sync.Once runs the body once and blocks
+// every other caller until that run completes, so both callers see the same result.
+func Shutdown() error {
+	return shutdown()
+}
+
+// shutdown stops everything in dependency order and waits for each stage.
+//
+// The order is not cosmetic. Every stage waits for the previous one to have
+// finished, so no goroutine is ever using a resource that the next stage is
+// closing:
+//
+//  1. the Telegram update listener — nothing else may add work;
+//  2. the greeting sweeper — the last periodic user of the caches;
+//  3. the caches — cache.Close() both signals the sweeper goroutines and
+//     reassigns the underlying sync.Map, so it must run with nothing else
+//     touching them;
+//  4. chat logging, then 5. the main log — so the last log lines, which
+//     describe stages 1 to 3, still have somewhere to go.
+//
+// Stages 1 and 2 share one stop signal and one WaitGroup, so a worker that has
+// not started costs nothing and a worker that starts late is refused rather than
+// racing the wait.
+//
+// The main log's own failure is reported ahead of the chat log's, because it happened
+// last: a chat-log failure may no longer have been writable either.
+func shutdown() error {
+	var result error
+
+	shutdownOnce.Do(func() {
+		// Refuse new workers before signalling, so nothing can register between
+		// the signal and the wait below.
+		workersMu.Lock()
+		stopping = true
+		workersMu.Unlock()
+
+		// Ask every loop to return, then wait for all of them. Nothing may be
+		// using a cache, a log or the API once the next stages start closing.
+		close(stopRequested)
+		workers.Wait()
+
+		closeCaches()
+
+		// Chat logging first: it writes to the main log, so it must be closed
+		// before the main log is.
+		chatErr := elog.Close()
+
+		// Since we shutting down, we have noting to do with log sync.
+		logErr := mylog.Close()
+
+		switch {
+		case logErr != nil:
+			result = fmt.Errorf("main log: %w", logErr)
+		case chatErr != nil:
+			result = fmt.Errorf("chat log: %w", chatErr)
+		}
+	})
+
+	return result
+}
+
+/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

internal/moon/sighandler.go

@@ -2,39 +2,89 @@ package moon
 
 import (
 	"os"
-	"syscall"
+	"os/signal"
+
+	"bunny-bot/internal/mylog"
 )
 
-// SigHandler.handles signals. Mainly closes persistent dbs and quits for good.
+// InstallSignalHandler registers every signal the bot cares about and starts the
-func SigHandler() {
+// handler goroutine. The signal sets differ per platform, so the knowledge lives
-	for {
+// behind terminationSignals and reopenSignals rather than leaking into main.
-		var s = <-SigChan
+func InstallSignalHandler() {
-		switch s {
+	signal.Notify(SigChan, terminationSignals()...)
-		case syscall.SIGINT:
+	signal.Notify(SigChan, reopenSignals()...)
-			Log.Infoln("Got SIGINT, quitting")
-		case syscall.SIGTERM:
-			Log.Infoln("Got SIGTERM, quitting")
-		case syscall.SIGQUIT:
-			Log.Infoln("Got SIGQUIT, quitting")
 
-		// Make new iteration since we 've got signal we not interested in.
+	go SigHandler()
-		default:
+}
+
+// SigHandler handles signals until one asks the process to quit.
+//
+// It stops the program; it does not end it. os.Exit here would skip main's
+// deferred cleanup, and would report success whether or not the teardown actually
+// worked — a service manager told "clean shutdown" over a log nobody can write to
+// again is worse than one told to restart us. So the handler runs the teardown and
+// returns; main decides the exit status from its own call to Shutdown.
+func SigHandler() {
+	for s := range SigChan {
+		if !handleSignal(s) {
 			continue
 		}
 
-		if len(DB) > 0 {
+		if err := shutdown(); err != nil {
-			Log.Debug("Closing runtime bot settings db")
+			// The main log is already closed by now, so this is the last line we
-
+			// get to write; main still exits non-zero over the same failure.
-			for _, db := range DB {
+			mylog.Errorf("Shutdown did not complete cleanly: %s", err)
-				_ = db.Close()
-			}
 		}
 
-		// Since we shutting down, we have noting to do with log sync.
+		return
-		_ = Log.Sync()
-
-		os.Exit(0)
 	}
 }
 
+// handleSignal applies one signal and reports whether it means "shut down".
+//
+// Reopen signals re-open the log file, which is what a log rotation tool needs
+// after it has moved the file aside. Neither of them is expected on Windows,
+// and an unrecognised signal is ignored rather than treated as a quit.
+func handleSignal(s os.Signal) bool {
+	if containsSignal(reopenSignals(), s) {
+		mylog.Infof("Got %s, re-opening the log file", s)
+
+		if err := mylog.ReOpen(); err != nil {
+			// Reopening can legitimately fail — the destination may not be a
+			// file at all, or the logger may already be closed. Logging is
+			// best-effort here because the log is exactly what just broke.
+			mylog.Errorf("Unable to re-open the log file on %s: %s", s, err)
+		} else {
+			mylog.Info("Log file re-opened")
+		}
+
+		return false
+	}
+
+	if !containsSignal(terminationSignals(), s) {
+		// Make new iteration since we 've got signal we not interested in.
+		return false
+	}
+
+	// The status has to be recorded here, before the teardown runs: main reads
+	// it after Shutdown returns, and neither of them can reconstruct which
+	// signal triggered all of this by then.
+	recordExitCode(s)
+
+	mylog.Infof("Got %s, quitting", s)
+
+	return true
+}
+
+// containsSignal reports whether sig is in the set.
+func containsSignal(set []os.Signal, sig os.Signal) bool {
+	for _, s := range set {
+		if s == sig {
+			return true
+		}
+	}
+
+	return false
+}
+
 /* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

internal/moon/signals_unix.go

@@ -0,0 +1,25 @@
+//go:build !windows
+
+package moon
+
+import (
+	"os"
+	"syscall"
+)
+
+// terminationSignals are the signals that ask the bot to shut down.
+func terminationSignals() []os.Signal {
+	return []os.Signal{syscall.SIGINT, syscall.SIGTERM, syscall.SIGQUIT}
+}
+
+// reopenSignals are the signals that ask the bot to re-open its log file.
+//
+// SIGHUP is the conventional daemon signal for "re-read your configuration and
+// re-open your logs"; SIGUSR1 is the other common choice for log re-opening
+// alone. Both are accepted so the bot fits either convention an operator's
+// logrotate configuration expects.
+func reopenSignals() []os.Signal {
+	return []os.Signal{syscall.SIGHUP, syscall.SIGUSR1}
+}
+
+/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

internal/moon/signals_windows.go

@@ -0,0 +1,22 @@
+//go:build windows
+
+package moon
+
+import "os"
+
+// terminationSignals are the signals that ask the bot to shut down.
+func terminationSignals() []os.Signal {
+	return []os.Signal{os.Interrupt}
+}
+
+// reopenSignals is empty on Windows.
+//
+// SIGHUP and SIGUSR1 are Unix signals that Windows never delivers — SIGUSR1 does
+// not even exist in the syscall package there — so there is nothing to re-open
+// the log for. The empty set keeps handleSignal platform-agnostic: a Windows
+// service restarts to rotate its logs instead of signalling.
+func reopenSignals() []os.Signal {
+	return nil
+}
+
+/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

internal/moon/squash.go

@@ -1,32 +1,47 @@
 package moon
 
 import (
+	"bunny-bot/internal/mylog"
 	"fmt"
-	"slices"
 	"time"
 
 	"github.com/NicoNex/echotron/v3"
 )
 
-// TODO: refactor to return bool, error
+// squash bans the given chat member and reports what it actually did.
-
+//
-// squash bans given chat member.
+// The three outcomes are deliberately distinct. A debounced call is not a
-func squash(chatid int64, userid int64) {
+// failure: the user is already banned, which is the outcome we wanted, and the
-	banUntilDate := time.Now().Unix() + 10
+// caller must not report it as an error.
-
+//
-	user := GetUser(userid)
+// The user and chat come from the join event that triggered the judgement
-	chat := GetChat(chatid)
+// rather than from a cache. That event already carries both in full, so a cache
-
+// would only ever be a second, staler copy of data we are holding — and a miss
-	// If there is no such chatid, means that no such cache too.
+// in it would produce a ban logged with blank names.
-	if len(ChatList) == 0 || !slices.Contains(ChatList, chatid) {
+func squash(chat *echotron.Chat, user *echotron.User) banOutcome {
-		Log.Debugf("Add %s %s (%d) to list of known chats", chat.Type, chat.Title, chat.ID)
+	// A chat or user with no id is an entity we cannot address, not a ban that
-
+	// failed. In production the ingress has already refused such an update, so
-		ChatList = append(ChatList, chatid)
+	// reaching here means a caller handed us one directly; the guard is the
+	// nil-safety that keeps that from becoming a call against chat 0.
+	if !usableChat(chat) {
+		return banFailed
 	}
 
+	if !usableUser(user) {
+		return banFailed
+	}
+
+	// Telegram treats any ban shorter than 30 seconds (and longer than 1 year)
+	// as permanent, so this deliberately short offset produces a permanent ban.
+	// Do not "fix" it into a longer duration.
+	banUntilDate := time.Now().Unix() + 10
+
+	chatid, userid := chat.ID, user.ID
+
 	key := fmt.Sprintf("%d+%d", chatid, userid)
+
 	if _, exist := SquashedMembers.Get(key); exist {
-		Log.Infof(
+		mylog.Infof(
 			"Skip banning user %s %s (username = %s, id = %d) in %s %s (%d), already banned",
 			user.FirstName,
 			user.LastName,
@@ -37,17 +52,18 @@ func squash(chatid int64, userid int64) {
 			chat.ID,
 		)
 
-		return
+		return banSuppressed
 	}
 
-	if res, err := Tg.BanChatMember(
+	res, err := Tg.BanChatMember(
 		chatid,
 		userid,
 		&echotron.BanOptions{
 			UntilDate: int(banUntilDate),
 		},
-	); err != nil {
+	)
-		Log.Errorf(
+	if err != nil {
+		mylog.Errorf(
 			"Unable to ban user user %s %s (username = %s, id = %d) in %s %s (%d): %s",
 			user.FirstName,
 			user.LastName,
@@ -56,85 +72,53 @@ func squash(chatid int64, userid int64) {
 			chat.Type,
 			chat.Title,
 			chat.ID,
+			err,
+		)
+
+		return banFailed
+	}
+
+	if !res.Ok {
+		mylog.Errorf(
+			"Unable to ban user %s %s (username = %s, id = %d) in %s %s (%d): %s",
+			user.FirstName,
+			user.LastName,
+			user.Username,
+			user.ID,
+			chat.Type,
+			chat.Title,
+			chat.ID,
 			res.Description,
 		)
-	} else {
-		if !res.Ok {
-			Log.Errorf(
-				"Unable to ban user %s %s (username = %s, id = %d) in %s %s (%d): %s",
-				user.FirstName,
-				user.LastName,
-				user.Username,
-				user.ID,
-				chat.Type,
-				chat.Title,
-				chat.ID,
-				res.Description,
-			)
-		} else {
-			Log.Debugf(
-				"Banned user %s %s (username = %s, id = %d) in %s %s (%d).",
-				user.FirstName,
-				user.LastName,
-				user.Username,
-				user.ID,
-				chat.Type,
-				chat.Title,
-				chat.ID,
-			)
 
-			key := fmt.Sprintf("%d+%d", chatid, userid)
+		return banFailed
-
-			Log.Debugf(
-				"Put user %s %s (username = %s, id = %d) in %s %s (%d) into SquashedMembers to debounce ban.",
-				user.FirstName,
-				user.LastName,
-				user.Username,
-				user.ID,
-				chat.Type,
-				chat.Title,
-				chat.ID,
-			)
-
-			SquashedMembers.Set(key, true, 30*time.Second)
-
-			Log.Debugf(
-				"Remove user %s %s (username = %s, id = %d) in %s %s (%d) from NewMembers.",
-				user.FirstName,
-				user.LastName,
-				user.Username,
-				user.ID,
-				chat.Type,
-				chat.Title,
-				chat.ID,
-			)
-
-			NewMembers.Delete(key)
-
-			Log.Debugf(
-				"Remove user %s %s (username = %s, id = %d) in %s %s (%d) from AppearedMembers.",
-				user.FirstName,
-				user.LastName,
-				user.Username,
-				user.ID,
-				chat.Type,
-				chat.Title,
-				chat.ID,
-			)
-			AppearedMembers.Delete(key)
-
-			if err := Cfg.SaveKeyValue(
-				"BannedMembers",
-				fmt.Sprintf("%d", chatid),
-				fmt.Sprintf("%d", userid),
-				fmt.Sprintf("%d", time.Now().Unix()),
-			); err != nil {
-				Log.Error(err)
-			} else {
-				Log.Infof("Info about banned userid %d saved to BannedMembers db", userid)
-			}
-		}
 	}
+
+	mylog.Debugf(
+		"Banned user %s %s (username = %s, id = %d) in %s %s (%d).",
+		user.FirstName,
+		user.LastName,
+		user.Username,
+		user.ID,
+		chat.Type,
+		chat.Title,
+		chat.ID,
+	)
+
+	mylog.Debugf(
+		"Put user %s %s (username = %s, id = %d) in %s %s (%d) into SquashedMembers to debounce ban.",
+		user.FirstName,
+		user.LastName,
+		user.Username,
+		user.ID,
+		chat.Type,
+		chat.Title,
+		chat.ID,
+	)
+
+	SquashedMembers.Set(key, true, 30*time.Second)
+
+	return banApplied
 }
 
 /* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

internal/moon/telegram.go

@@ -1,74 +1,110 @@
 package moon
 
 import (
+	"bunny-bot/internal/elog"
+	"bunny-bot/internal/mylog"
+
 	"github.com/NicoNex/echotron/v3"
 	"github.com/davecgh/go-spew/spew"
 )
 
 // Telegram main goroutine that implements bot.
+//
+// It blocks until a termination signal stops it, so the update listener can be
+// shut down in a defined order rather than simply abandoned when the process
+// exits.
 func Telegram(cfg MyConfig) {
+	// A signal can arrive during startup; then this must not start at all.
+	if !startWorker() {
+		mylog.Info("Telegram not started: shutdown already in progress")
+
+		return
+	}
+
+	defer stopWorker()
+
 	Tg = echotron.NewAPI(cfg.Token)
 
-	Log.Infof("Starting api update poller loop")
+	mylog.Info("Starting api update poller loop")
 
-	for u := range echotron.PollingUpdatesOptions(cfg.Token, true, echotron.UpdateOptions{
+	// The sweeper calls the API too, so it may only start once Tg exists.
-		AllowedUpdates: []echotron.UpdateType{
+	mylog.Info("Starting greeting sweeper periodic task")
-			"message",
+
-			"edited_message",
+	go SweepGreetings()
-			"channel_post",
+
-			"edited_channel_post",
+	updates := echotron.PollingUpdatesOptions(cfg.Token, true, echotron.UpdateOptions{
-			"message_reaction",
+		AllowedUpdates: allowedUpdates,
-			"message_reaction_count",
+	})
-			"inline_query",
+
-			"chosen_inline_result",
+	for {
-			"callback_query",
+		select {
-			"poll",
+		// Stop between updates, never inside one: the handler currently running
-			"poll_answer",
+		// finishes, so nothing is left half-processed when we return.
-			"my_chat_member",
+		case <-stopRequested:
-			"chat_member",
+			mylog.Info("Telegram update listener stopped")
-			"chat_join_request",
+
-			"chat_boost",
+			return
-			"removed_chat_boost",
+
-		},
+		case u, ok := <-updates:
-	}) {
+			if !ok {
-		telegramMsgParser(u)
+				mylog.Info("Telegram update poller ended")
+
+				return
+			}
+
+			telegramMsgParser(u)
+		}
 	}
 }
 
 // telegramMsgParser parses events generated by echotron.
 func telegramMsgParser(msg *echotron.Update) {
+	// The one place an update enters the program, and therefore the one place an
+	// entity with no id is caught. Everything downstream — the caches, the sweeper,
+	// the bans — may then assume the ids it holds are real.
+	if !usableUpdate(msg) {
+		reportBrokenUpdate(msg)
+
+		return
+	}
+
 	switch {
-	case msg.Message != nil && msg.Message.Text != "":
+	// Membership status changes: joins, leaves, bans, promotions. Logged only —
-		textMessage(msg)
+	// moderation is the sweeper's job, on a timer, not the dispatcher's.
-
+	//
-	case msg.EditedMessage != nil:
+	// This also catches the membership updates caused by our own bans, so every
-		editedMessage(msg)
+	// ban leaves a second record in the chat log.
-
 	case msg.ChatMember != nil:
-		chatMemberUpdated(msg)
+		elog.LogUserStatus(msg.ChatMember)
 
-	// New members entered to chat.
+	// New members entered to chat. Checked before the text case so that a
+	// join service message is never mistaken for ordinary chatter.
 	case msg.Message != nil && msg.Message.NewChatMembers != nil:
 		newChatMembers(msg)
 
-	// Chat member leaves chat.
+	// Ordinary text messages, used as greeting answers.
-	case msg.Message != nil && msg.Message.LeftChatMember != nil:
+	case msg.Message != nil && msg.Message.Text != "":
-		leftChatMember(msg)
+		textMessage(msg)
 
-	case msg.Message != nil && msg.Message.Animation != nil:
+	// Everything else that is still a message: a photo with no caption, a
-		animation(msg)
+	// sticker, a location, a leave, a title change. Logged, and deliberately NOT
+	// recorded as a greeting answer — an attachment is not an introduction, and
+	// letting one satisfy the detector would be a one-key bypass for a spammer.
+	//
+	// The condition is deliberately just "there is a message": elog already
+	// renders every media and service field, and already writes nothing for a
+	// message with no describable content, so emptiness is its decision to make.
+	// Testing for a field being non-empty here is what lost these messages.
+	case msg.Message != nil:
+		logOnlyMessage(msg)
 
-	case msg.MessageReaction != nil:
+	// An edit revises a message already recorded, so it is a chat-log event
-		reaction(msg)
+	// rather than a moderation one.
-
+	case msg.EditedMessage != nil:
-	case msg.MessageReactionCount != nil:
+		elog.LogEditedMessage(msg.EditedMessage)
-		reactionCount(msg)
-
-	case msg.Message != nil && msg.Message.Photo != nil:
-		photo(msg)
 
 	default:
-		Log.Debug(spew.Sdump(msg))
+		mylog.Debug(spew.Sdump(msg))
 	}
 }
 

internal/moon/textMessage.go

@@ -1,58 +1,27 @@
 package moon
 
 import (
-	"fmt"
+	"time"
+
+	"bunny-bot/internal/elog"
 
 	"github.com/NicoNex/echotron/v3"
 )
 
-// textMessage handles text message events.
+// textMessage handles plain text messages. It does not moderate anything; it
+// only records the message so the greeting sweeper can tell that a newcomer
+// answered their greeting.
 func textMessage(msg *echotron.Update) {
-	AddChat(&msg.Message.Chat)
+	if msg.Message == nil || msg.Message.Text == "" {
-	AddUser(msg.Message.From)
+		return
-
-	// Not sure if we must handle all kinds of these events.
-	if Cfg.LogChats {
-		switch {
-		// Just log it, it seems later here should be ban/mute command set.
-		case msg.Message != nil && msg.Message.ReplyToMessage == nil:
-			ChatLog(
-				msg.Message.Chat.ID,
-				fmt.Sprintf(
-					"User %s %s (username = %s, id = %d) in %s %s (%d) says: %s",
-					msg.Message.From.FirstName,
-					msg.Message.From.LastName,
-					msg.Message.From.Username,
-					msg.Message.From.ID,
-					msg.Message.Chat.Type,
-					msg.Message.Chat.Title,
-					msg.Message.Chat.ID,
-					msg.Message.Text,
-				),
-			)
-
-		case msg.Message != nil && msg.Message.ReplyToMessage != nil:
-			ChatLog(
-				msg.Message.Chat.ID,
-				fmt.Sprintf(
-					"User %s %s (username = %s, id = %d) in %s %s (%d) replies to message %d by %s %s (username = %s, id = %d): %s",
-					msg.Message.From.FirstName,
-					msg.Message.From.LastName,
-					msg.Message.From.Username,
-					msg.Message.From.ID,
-					msg.Message.Chat.Type,
-					msg.Message.Chat.Title,
-					msg.Message.Chat.ID,
-					msg.Message.ReplyToMessage.ID,
-					msg.Message.ReplyToMessage.From.FirstName,
-					msg.Message.ReplyToMessage.From.LastName,
-					msg.Message.ReplyToMessage.From.Username,
-					msg.Message.ReplyToMessage.From.ID,
-					msg.Message.Text,
-				),
-			)
-		}
 	}
+
+	// Remember the message so the greeting sweeper can find the answer to a
+	// greeting.
+	GreetAnswers.Set(time.Now().UnixMicro(), msg.Message, greetCacheTTL)
+
+	// And record it in the chat's own log, which outlives the cache.
+	elog.LogMessage(msg.Message)
 }
 
 /* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

internal/moon/types.go

@@ -1,40 +1,82 @@
 package moon
 
 import (
-	"os"
+	"github.com/NicoNex/echotron/v3"
-	"time"
+	"github.com/akyoto/cache"
 )
 
 // MyConfig describes bot config file format.
 type MyConfig struct {
-	Name     string `json:"name,omitempty"`
 	Token    string `json:"token,omitempty"`
 	LogLevel string `json:"loglevel,omitempty"`
 	LogFile  string `json:"logfile,omitempty"`
-	LogChats bool   `json:"logchats,omitempty"`
+
-	CSign    string `json:"csign,omitempty"`
+	// ChatLogs is the directory for per-chat, per-day chat event logs. Empty
-	DataDir  string `json:"datadir,omitempty"`
+	// disables chat logging. If set, the directory must already exist: a path
+	// that does not is a typo, and silently logging nowhere is the worst
+	// possible response to one.
+	ChatLogs string `json:"chat_logs,omitempty"`
 }
 
-// CasFalse describes response from api.cas.chat when user is not blacklisted.
+// tgClient is the subset of the Telegram API that bunny-bot actually calls.
-type CasFalse struct {
+// It exists so tests can substitute a fake; echotron.API satisfies it.
-	Ok          bool   `json:"ok"`
+type tgClient interface {
-	Description string `json:"description"`
+	SendMessage(text string, chatID int64, opts *echotron.MessageOptions) (echotron.APIResponseMessage, error)
+	BanChatMember(chatID int64, userID int64, opts *echotron.BanOptions) (echotron.APIResponseBool, error)
+	DeleteMessage(chatID int64, messageID int) (echotron.APIResponseBase, error)
 }
 
-// CasFalse describes response from api.cas.chat when user is blacklisted.
+// chatUser identifies a user's membership of one specific chat. The same user is
-type CasTrue struct {
+// a different identity in every chat: being silent in one says nothing about
-	Ok     bool `json:"ok"`
+// another, and answering in one must not rescue them in another.
-	Result struct {
+type chatUser struct {
-		Reasons   []int     `json:"reasons"`
+	chatID int64
-		Offenses  int       `json:"offenses"`
+	userID int64
-		TimeAdded time.Time `json:"time_added"`
-	} `json:"result"`
 }
 
-type Chatlog struct {
+// banOutcome is what a squash attempt actually did. The three cases are distinct
-	LogFilePath string
+// on purpose: a duplicate ban is a success, not a failure, and reporting it as
-	File        *os.File
+// one makes a working ban look broken in the logs.
+type banOutcome int
+
+// eventCache is a typed wrapper over akyoto/cache.Cache that stores values of
+// type T keyed by a Unix timestamp in microseconds. It hides the underlying
+// sync.Map so callers never deal with interface{} keys/values or per-access type
+// conversions.
+type eventCache[T any] struct {
+	cache *cache.Cache
 }
 
+// joinRef is one stale join, kept together with everything the verdict and the
+// retraction need so neither has to look anything up.
+type joinRef struct {
+	micros int64
+
+	// chat and user come from the join event itself, so a ban can be logged with
+	// the newcomer's real name and the chat's real title instead of a cache
+	// lookup that may have expired or never been populated.
+	chat echotron.Chat
+	user *echotron.User
+
+	// greetingID is the question we asked them, deleted once they are banned for
+	// never answering it. Zero means we never managed to ask.
+	greetingID int
+}
+
+// pendingGreeting is one recorded join: the message that announced it, and the
+// id of the greeting we sent in reply.
+//
+// The greeting id lives here rather than in a second cache, so that recording,
+// forgetting and retracting are all one lookup on one key. A zero greeting id
+// means "no greeting was ever delivered for this join" — never a failure.
+type pendingGreeting struct {
+	join       *echotron.Message
+	greetingID int
+}
+
+// retractOutcome is what a retraction pass actually achieved. The three cases are
+// distinct because "nothing to delete" and "tried and failed" call for different
+// log lines, and only the second is worth an operator's attention.
+type retractOutcome int
+
 /* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

internal/moon/usersGroups.go

@@ -1,76 +0,0 @@
-package moon
-
-import (
-	"encoding/json"
-	"time"
-
-	"github.com/NicoNex/echotron/v3"
-)
-
-// AddUser remebers user info in People cache.
-func AddUser(u *echotron.User) {
-	bytes, err := json.Marshal(u)
-
-	if err != nil {
-		Log.Errorf("Unable to encode user info to json string: %s", err)
-	}
-
-	People.Set(u.ID, bytes, 24*time.Hour)
-}
-
-// GetUser returns user info structure.
-func GetUser(id int64) *echotron.User {
-	var u *echotron.User
-
-	bytes, ok := People.Get(id)
-
-	if !ok {
-		return u
-	}
-
-	err := json.Unmarshal(bytes.([]byte), &u)
-
-	if err != nil {
-		Log.Errorf("Unable to decode user info json: %s", err)
-
-		return u
-	}
-
-	return u
-}
-
-// AddChat remebers chat info in Groups cache.
-func AddChat(c *echotron.Chat) {
-	bytes, err := json.Marshal(c)
-
-	if err != nil {
-		Log.Errorf("Unable to encode chat info to json string: %s", err)
-	}
-
-	Groups.Set(c.ID, bytes, 24*time.Hour)
-}
-
-// GetChat returns chat info structure.
-func GetChat(id int64) *echotron.Chat {
-	var c *echotron.Chat
-
-	bytes, ok := Groups.Get(id)
-
-	if !ok {
-		Log.Errorf("No group %d in Groups cache", id)
-
-		return c
-	}
-
-	err := json.Unmarshal(bytes.([]byte), &c)
-
-	if err != nil {
-		Log.Errorf("Unable to decode chat info json: %s", err)
-
-		return c
-	}
-
-	return c
-}
-
-/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

internal/moon/worker.go

@@ -0,0 +1,36 @@
+package moon
+
+// The worker registry: how a background goroutine joins the shutdown sequence,
+// and how it leaves it. The state itself lives in globals.go with the rest of the
+// package-level variables; this file holds only what a goroutine does with it.
+
+// stopWorker unregisters a background goroutine that has finished.
+func stopWorker() {
+	runningWorkers.Add(-1)
+	workers.Done()
+}
+
+// workerRunning reports whether any registered worker is still running.
+func workerRunning() bool {
+	return runningWorkers.Load() > 0
+}
+
+// startWorker registers a background goroutine with the shutdown sequence and
+// reports whether it may run. It returns false once shutdown has begun, in
+// which case the caller must return without starting: the stop signal is
+// already closed and the teardown may already have closed what it would use.
+func startWorker() bool {
+	workersMu.Lock()
+	defer workersMu.Unlock()
+
+	if stopping {
+		return false
+	}
+
+	workers.Add(1)
+	runningWorkers.Add(1)
+
+	return true
+}
+
+/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

internal/mylog/globals.go

@@ -0,0 +1,71 @@
+package mylog
+
+import (
+	"errors"
+	"io"
+	"os"
+	"sync"
+)
+
+const (
+	// levelTrace is the most verbose level, below DEBUG.
+	levelTrace Level = iota
+	levelDebug
+	levelInfo
+	levelWarn
+	levelError
+	levelFatal
+)
+
+const (
+	// stampLayout is the timestamp layout. Sub-second precision is kept on
+	// purpose: the greeting sweeper reasons over 15-minute windows measured in
+	// microseconds, and the previous zap configuration truncated to whole
+	// seconds, which made window-edge debugging needlessly hard.
+	stampLayout = "2006-01-02 15:04:05.000"
+
+	// calldepth is reported to the stdlib logger. mylog never prints caller
+	// information, so the value only has to be consistent.
+	calldepth = 2
+
+	// filePerm is the mode for a newly created log file.
+	filePerm = 0o644
+)
+
+var (
+	// errClosed is returned when ReOpen is called on a closed logger.
+	errClosed = errors.New("cannot reopen a closed logger")
+
+	// errUnknownLevel is returned for a log level name mylog does not know.
+	errUnknownLevel = errors.New("unknown log level")
+
+	// levelNames renders each level as the uppercase token written to the log.
+	levelNames = map[Level]string{
+		levelTrace: "TRACE",
+		levelDebug: "DEBUG",
+		levelInfo:  "INFO",
+		levelWarn:  "WARN",
+		levelError: "ERROR",
+		levelFatal: "FATAL",
+	}
+
+	// exit is os.Exit, indirected so tests can observe the fatal path.
+	exit = os.Exit
+
+	// lock guards the package-global logger state. It is held for reading across
+	// the actual write, so that a concurrent Init cannot close the destination
+	// between the level check and the write.
+	lock sync.RWMutex
+
+	// current is the active logger. It is deliberately never nil: it is created
+	// working at package initialisation, writing to stderr at INFO. A nil logger
+	// would be the defect M2 in this codebase, and a valid default removes the
+	// state entirely rather than guarding against it.
+	current = newState(levelInfo, io.Writer(os.Stderr), "", nil, false)
+
+	// standard is the singleton Logger handed to modules that expect a
+	// *log.Logger. It is stateless; the package-global current does the work.
+	standard Logger = &adapter{}
+)
+
+/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

internal/mylog/level.go

@@ -0,0 +1,59 @@
+package mylog
+
+// The level vocabulary: what a level is called, how a name in free-form config
+// text becomes a Level, and how the configured threshold is changed without
+// reopening the destination.
+
+import (
+	"fmt"
+	"strings"
+)
+
+// ParseLevel maps a level name to a Level. Parsing is case-insensitive and
+// tolerates surrounding whitespace, because the level comes from free-form user
+// text in the config file. An unknown name is an error: Init and SetLevel are
+// programming-level calls where a typo should be loud.
+func ParseLevel(s string) (Level, error) {
+	switch strings.ToUpper(strings.TrimSpace(s)) {
+	case "TRACE":
+		return levelTrace, nil
+	case "DEBUG":
+		return levelDebug, nil
+	case "INFO":
+		return levelInfo, nil
+	case "WARN":
+		return levelWarn, nil
+	case "ERROR":
+		return levelError, nil
+	case "FATAL":
+		return levelFatal, nil
+	default:
+		return levelInfo, fmt.Errorf("%w %q, want one of trace, debug, info, warn, error, fatal", errUnknownLevel, s)
+	}
+}
+
+// SetLevel changes the level without reopening the destination.
+func SetLevel(loglevel string) error {
+	lvl, err := ParseLevel(loglevel)
+	if err != nil {
+		return err
+	}
+
+	lock.Lock()
+	defer lock.Unlock()
+
+	current.level = lvl
+
+	return nil
+}
+
+// name returns the uppercase token for the level.
+func (l Level) name() string {
+	if n, ok := levelNames[l]; ok {
+		return n
+	}
+
+	return "UNKNOWN"
+}
+
+/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

internal/mylog/log.go

@@ -0,0 +1,118 @@
+package mylog
+
+// The destination lifecycle: opening the log, re-opening it for rotation, and
+// closing it for good. The level vocabulary lives in level.go, the state in
+// state.go, and the act of writing a line in write.go.
+
+import (
+	"fmt"
+	"io"
+	stdlog "log"
+	"os"
+)
+
+// Init opens the log destination and sets the level.
+//
+// An empty filepath selects stderr. A non-empty path is opened appending, so
+// an existing log is never truncated.
+//
+// Init is re-entrant: any previously open file is closed first, so repeated
+// calls do not leak descriptors. If the new destination cannot be opened, Init
+// returns an error and leaves the previous destination and level untouched —
+// it deliberately does not fall back to stderr, because a silent fallback
+// hides a broken log path from the operator.
+func Init(loglevel string, filepath string) error {
+	lvl, err := ParseLevel(loglevel)
+	if err != nil {
+		return err
+	}
+
+	lock.Lock()
+	defer lock.Unlock()
+
+	dest, owned := io.Writer(os.Stderr), (*os.File)(nil)
+
+	if filepath != "" {
+		// The path comes from the operator's own config file, not from a
+		// network peer, and opening it is this function's entire purpose.
+		file, err := os.OpenFile(filepath, os.O_CREATE|os.O_APPEND|os.O_WRONLY, filePerm) //nolint: gosec // G703
+		if err != nil {
+			return fmt.Errorf("unable to open log file %s: %w", filepath, err)
+		}
+
+		dest, owned = file, file
+	}
+
+	swapLocked(lvl, dest, filepath, owned)
+
+	// Keep the standard library's default logger pointed at the same
+	// destination, so third-party stdlib logging does not bypass our file.
+	stdlog.SetFlags(0)
+	stdlog.SetOutput(dest)
+
+	return nil
+}
+
+// ReOpen closes and re-opens the current log file, keeping its level. It exists
+// for external rotation: the rotation tool moves the file aside, and ReOpen
+// makes the logger pick up the new one.
+//
+// ReOpen on a closed logger is an error rather than a silent reopen, and on a
+// stderr destination it is a no-op, since there is no file to re-open.
+func ReOpen() error {
+	lock.RLock()
+	closed, path, lvl := current.closed, current.path, current.level
+	lock.RUnlock()
+
+	switch {
+	case closed:
+		return errClosed
+	case path == "":
+		return nil
+	default:
+		return Init(lvl.name(), path)
+	}
+}
+
+// Close closes the log destination. It is idempotent, and it is safe when the
+// destination is stderr — stderr is not ours to close.
+//
+// After Close, logging calls are silently dropped rather than panicking or
+// resurrecting the file: Close is terminal.
+func Close() error {
+	lock.Lock()
+	defer lock.Unlock()
+
+	if current.closed {
+		return nil
+	}
+
+	current.closed = true
+
+	if current.owned == nil {
+		return nil
+	}
+
+	// The stdlib Logger has no Close — it does not own its writer — so the file
+	// we opened ourselves is closed directly.
+	file := current.owned
+	current.owned = nil
+
+	if file == nil {
+		return nil
+	}
+
+	return file.Close()
+}
+
+// swapLocked installs a new state, closing any file we previously owned.
+// The caller must hold the write lock.
+func swapLocked(lvl Level, dest io.Writer, path string, owned *os.File) {
+	if current.owned != nil {
+		_ = current.owned.Close()
+	}
+
+	current = newState(lvl, dest, path, owned, false)
+}
+
+/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

internal/mylog/standard.go

@@ -0,0 +1,28 @@
+package mylog
+
+// The standard-library-compatible Logger: what a module receives instead of
+// depending on *log.Logger itself, and where its messages end up.
+
+import "strings"
+
+// Standard returns the Logger for modules that would otherwise depend on the
+// standard library's *log.Logger. Messages passed to it are logged at debug
+// level into the already-open destination.
+//
+// Returning the interface rather than the concrete *adapter is the whole point:
+// it is the type such modules declare, so hiding it would defeat the
+// substitution it exists for.
+func Standard() Logger { //nolint:ireturn // deliberate public interface
+	return standard
+}
+
+// Output implements Logger, logging at debug level. The trailing newline of the
+// standard library convention is trimmed so that mylog does not append a blank
+// line to every message.
+func (a *adapter) Output(_ int, s string) error {
+	write(levelDebug, strings.TrimRight(s, "\r\n"))
+
+	return nil
+}
+
+/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

internal/mylog/state.go

@@ -0,0 +1,52 @@
+package mylog
+
+// The logger state: the one struct every write goes through, how a new one is
+// built, and the two accessors that read and replace it under the package lock.
+// The state itself lives in globals.go with the rest of the package-level
+// variables; this file holds only what a caller does with it.
+
+import (
+	"io"
+	stdlog "log"
+	"os"
+)
+
+// newState builds a state whose stdlib logger writes to dest.
+func newState(lvl Level, dest io.Writer, path string, owned *os.File, closed bool) *state {
+	return &state{
+		level:  lvl,
+		dest:   dest,
+		slog:   newStdLogger(dest),
+		path:   path,
+		owned:  owned,
+		closed: closed,
+	}
+}
+
+// newStdLogger wraps a writer in a standard library logger. Flags are zeroed
+// because the line format is rendered by mylog itself.
+func newStdLogger(dest io.Writer) *stdLogger {
+	l := stdlog.New(dest, "", 0)
+	l.SetFlags(0)
+
+	return l
+}
+
+// getState returns the active logger state.
+func getState() *state {
+	lock.RLock()
+	defer lock.RUnlock()
+
+	return current
+}
+
+// setState replaces the active state wholesale. Tests use it to restore the
+// documented before-Init default.
+func setState(s *state) {
+	lock.Lock()
+	defer lock.Unlock()
+
+	current = s
+}
+
+/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

internal/mylog/types.go

@@ -0,0 +1,43 @@
+package mylog
+
+// Types only. The level constants, the name table and Level.name live in
+// globals.go and level.go respectively, because a file named "types" must hold
+// nothing that is not a type.
+
+import (
+	"io"
+	stdlog "log"
+	"os"
+)
+
+// Level is the severity of a log message. Levels are ordered, and a message is
+// written when its own level is at or above the configured level.
+type Level int
+
+// stdLogger is the standard library logger doing the actual writing.
+type stdLogger = stdlog.Logger
+
+// Logger is the interface for modules that would otherwise depend on the
+// standard library's *log.Logger. It is deliberately the smallest method set
+// that *log.Logger satisfies, so widening a module's parameter type from
+// *log.Logger to Logger keeps existing callers compiling.
+type Logger interface {
+	Output(calldepth int, s string) error
+}
+
+// adapter is the Logger handed to such modules. Messages arrive through
+// Output and are logged at debug level into the already-open destination.
+type adapter struct{}
+
+// state is everything the logger needs to write a line. It is held behind a
+// lock so that a concurrent Init cannot close a destination mid-write.
+type state struct {
+	level  Level
+	dest   io.Writer
+	slog   *stdLogger
+	path   string
+	owned  *os.File
+	closed bool
+}
+
+/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

internal/mylog/wrappers.go

@@ -0,0 +1,63 @@
+package mylog
+
+import (
+	"fmt"
+)
+
+// Trace logs at the most verbose level, below DEBUG.
+func Trace(args ...any) { write(levelTrace, fmt.Sprint(args...)) }
+
+// Tracef logs a formatted message at the most verbose level, below DEBUG.
+func Tracef(format string, args ...any) { write(levelTrace, fmt.Sprintf(format, args...)) }
+
+// Debug logs at debug level.
+func Debug(args ...any) { write(levelDebug, fmt.Sprint(args...)) }
+
+// Debugf logs a formatted message at debug level.
+func Debugf(format string, args ...any) { write(levelDebug, fmt.Sprintf(format, args...)) }
+
+// Info logs at info level.
+func Info(args ...any) { write(levelInfo, fmt.Sprint(args...)) }
+
+// Infof logs a formatted message at info level.
+func Infof(format string, args ...any) { write(levelInfo, fmt.Sprintf(format, args...)) }
+
+// Warn logs at warn level.
+func Warn(args ...any) { write(levelWarn, fmt.Sprint(args...)) }
+
+// Warnf logs a formatted message at warn level.
+func Warnf(format string, args ...any) { write(levelWarn, fmt.Sprintf(format, args...)) }
+
+// Error logs at error level.
+func Error(args ...any) { write(levelError, fmt.Sprint(args...)) }
+
+// Errorf logs a formatted message at error level.
+func Errorf(format string, args ...any) { write(levelError, fmt.Sprintf(format, args...)) }
+
+// Fatal logs at fatal level, then exits the process with status 1.
+//
+// The destination is closed before exiting. os.Exit does not run deferred
+// functions, so without this an explicit close a buffered tail could be lost —
+// which is precisely the case where the message matters most.
+func Fatal(args ...any) {
+	write(levelFatal, fmt.Sprint(args...))
+
+	exitAfterClose(1)
+}
+
+// Fatalf logs a formatted message at fatal level, then exits the process with
+// status 1. The destination is closed first; see Fatal.
+func Fatalf(format string, args ...any) {
+	write(levelFatal, fmt.Sprintf(format, args...))
+
+	exitAfterClose(1)
+}
+
+// exitAfterClose closes the logger and then exits, in that order.
+func exitAfterClose(code int) {
+	_ = Close()
+
+	exit(code)
+}
+
+/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

internal/mylog/write.go

@@ -0,0 +1,28 @@
+package mylog
+
+// The one-line emit path: decide whether the line is wanted, render it, write
+// it. Everything else in the package exists to reach these two functions.
+
+import "time"
+
+// write renders and writes one line if the level passes the filter and the
+// logger is open. The read lock is held across the write so that a concurrent
+// Init cannot close the destination underneath us.
+func write(lvl Level, msg string) {
+	lock.RLock()
+	defer lock.RUnlock()
+
+	if current.closed || lvl < current.level {
+		return
+	}
+
+	_ = current.slog.Output(calldepth, render(lvl, msg))
+}
+
+// render builds one log line: a GMT+0 timestamp, the uppercase level, and the
+// message verbatim, separated by exactly one space each.
+func render(lvl Level, msg string) string {
+	return time.Now().UTC().Format(stampLayout) + " " + lvl.name() + " " + msg
+}
+
+/* vim: set ft=go noet ai ts=4 sw=4 sts=4: */

vendor/github.com/DataDog/zstd/LICENSE

@@ -1,27 +0,0 @@
-Simplified BSD License
-
-Copyright (c) 2016, Datadog <info@datadoghq.com>
-All rights reserved.
-
-Redistribution and use in source and binary forms, with or without
-modification, are permitted provided that the following conditions are met:
-
-    * Redistributions of source code must retain the above copyright notice,
-      this list of conditions and the following disclaimer.
-    * Redistributions in binary form must reproduce the above copyright notice,
-      this list of conditions and the following disclaimer in the documentation
-      and/or other materials provided with the distribution.
-    * Neither the name of the copyright holder nor the names of its contributors
-      may be used to endorse or promote products derived from this software
-      without specific prior written permission.
-
-THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
-AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
-IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
-DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE
-FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
-DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
-SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
-CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
-OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
-OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

vendor/github.com/DataDog/zstd/README.md

@@ -1,142 +0,0 @@
-# Zstd Go Wrapper
-
-[![CircleCI](https://circleci.com/gh/DataDog/zstd/tree/1.x.svg?style=svg)](https://circleci.com/gh/DataDog/zstd/tree/1.x)
-[![GoDoc](https://godoc.org/github.com/DataDog/zstd?status.svg)](https://godoc.org/github.com/DataDog/zstd)
-
-
-[C Zstd Homepage](https://github.com/facebook/zstd)
-
-The current headers and C files are from *v1.5.7* (Commit
-[f8745da](https://github.com/facebook/zstd/releases/tag/v1.5.7)).
-
-## Usage
-
-There are two main APIs:
-
-* simple Compress/Decompress
-* streaming API (io.Reader/io.Writer)
-
-The compress/decompress APIs mirror that of lz4, while the streaming API was
-designed to be a drop-in replacement for zlib.
-
-### Building against an external libzstd
-
-By default, zstd source code is vendored in this repository and the binding will be built with
-the vendored source code bundled.
-
-If you want to build this binding against an external static or shared libzstd library, you can
-use the `external_libzstd` build tag. This will look for the libzstd pkg-config file and extract
-build and linking parameters from that pkg-config file.
-
-Note that it requires at least libzstd 1.4.0.
-
-```bash
-go build -tags external_libzstd
-```
-
-### Simple `Compress/Decompress`
-
-
-```go
-// Compress compresses the byte array given in src and writes it to dst.
-// If you already have a buffer allocated, you can pass it to prevent allocation
-// If not, you can pass nil as dst.
-// If the buffer is too small, it will be reallocated, resized, and returned by the function
-// If dst is nil, this will allocate the worst case size (CompressBound(src))
-Compress(dst, src []byte) ([]byte, error)
-```
-
-```go
-// CompressLevel is the same as Compress but you can pass another compression level
-CompressLevel(dst, src []byte, level int) ([]byte, error)
-```
-
-```go
-// Decompress will decompress your payload into dst.
-// If you already have a buffer allocated, you can pass it to prevent allocation
-// If not, you can pass nil as dst (allocates a 4*src size as default).
-// If the buffer is too small, it will retry 3 times by doubling the dst size
-// After max retries, it will switch to the slower stream API to be sure to be able
-// to decompress. Currently switches if compression ratio > 4*2**3=32.
-Decompress(dst, src []byte) ([]byte, error)
-```
-
-### Stream API
-
-```go
-// NewWriter creates a new object that can optionally be initialized with
-// a precomputed dictionary. If dict is nil, compress without a dictionary.
-// The dictionary array should not be changed during the use of this object.
-// You MUST CALL Close() to write the last bytes of a zstd stream and free C objects.
-NewWriter(w io.Writer) *Writer
-NewWriterLevel(w io.Writer, level int) *Writer
-NewWriterLevelDict(w io.Writer, level int, dict []byte) *Writer
-
-// Write compresses the input data and write it to the underlying writer
-(w *Writer) Write(p []byte) (int, error)
-
-// Flush writes any unwritten data to the underlying writer
-(w *Writer) Flush() error
-
-// Close flushes the buffer and frees C zstd objects
-(w *Writer) Close() error
-```
-
-```go
-// NewReader returns a new io.ReadCloser that will decompress data from the
-// underlying reader.  If a dictionary is provided to NewReaderDict, it must
-// not be modified until Close is called.  It is the caller's responsibility
-// to call Close, which frees up C objects.
-NewReader(r io.Reader) io.ReadCloser
-NewReaderDict(r io.Reader, dict []byte) io.ReadCloser
-```
-
-### Benchmarks (benchmarked with v0.5.0)
-
-The author of Zstd also wrote lz4. Zstd is intended to occupy a speed/ratio
-level similar to what zlib currently provides.  In our tests, the can always
-be made to be better than zlib by chosing an appropriate level while still
-keeping compression and decompression time faster than zlib.
-
-You can run the benchmarks against your own payloads by using the Go benchmarks tool.
-Just export your payload filepath as the `PAYLOAD` environment variable and run the benchmarks:
-
-```go
-go test -bench .
-```
-
-Compression of a 7Mb pdf zstd (this wrapper) vs [czlib](https://github.com/DataDog/czlib):
-```
-BenchmarkCompression               5     221056624 ns/op      67.34 MB/s
-BenchmarkDecompression           100      18370416 ns/op     810.32 MB/s
-
-BenchmarkFzlibCompress             2     610156603 ns/op      24.40 MB/s
-BenchmarkFzlibDecompress          20      81195246 ns/op     183.33 MB/s
-```
-
-Ratio is also better by a margin of ~20%.
-Compression speed is always better than zlib on all the payloads we tested;
-However, [czlib](https://github.com/DataDog/czlib) has optimisations that make it
-faster at decompressiong small payloads:
-
-```
-Testing with size: 11... czlib: 8.97 MB/s, zstd: 3.26 MB/s
-Testing with size: 27... czlib: 23.3 MB/s, zstd: 8.22 MB/s
-Testing with size: 62... czlib: 31.6 MB/s, zstd: 19.49 MB/s
-Testing with size: 141... czlib: 74.54 MB/s, zstd: 42.55 MB/s
-Testing with size: 323... czlib: 155.14 MB/s, zstd: 99.39 MB/s
-Testing with size: 739... czlib: 235.9 MB/s, zstd: 216.45 MB/s
-Testing with size: 1689... czlib: 116.45 MB/s, zstd: 345.64 MB/s
-Testing with size: 3858... czlib: 176.39 MB/s, zstd: 617.56 MB/s
-Testing with size: 8811... czlib: 254.11 MB/s, zstd: 824.34 MB/s
-Testing with size: 20121... czlib: 197.43 MB/s, zstd: 1339.11 MB/s
-Testing with size: 45951... czlib: 201.62 MB/s, zstd: 1951.57 MB/s
-```
-
-zstd starts to shine with payloads > 1KB
-
-### Stability - Current state: STABLE
-
-The C library seems to be pretty stable and according to the author has been tested and fuzzed.
-
-For the Go wrapper, the test cover most usual cases and we have succesfully tested it on all staging and prod data.

vendor/github.com/DataDog/zstd/ZSTD_LICENSE

@@ -1,30 +0,0 @@
-BSD License
-
-For Zstandard software
-
-Copyright (c) Meta Platforms, Inc. and affiliates. All rights reserved.
-
-Redistribution and use in source and binary forms, with or without modification,
-are permitted provided that the following conditions are met:
-
- * Redistributions of source code must retain the above copyright notice, this
-   list of conditions and the following disclaimer.
-
- * Redistributions in binary form must reproduce the above copyright notice,
-   this list of conditions and the following disclaimer in the documentation
-   and/or other materials provided with the distribution.
-
- * Neither the name Facebook, nor Meta, nor the names of its contributors may
-   be used to endorse or promote products derived from this software without
-   specific prior written permission.
-
-THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
-ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
-WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
-DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR
-ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
-(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
-LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
-ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
-(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
-SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

vendor/github.com/DataDog/zstd/allocations.h

@@ -1,58 +0,0 @@
-#ifndef USE_EXTERNAL_ZSTD
-/*
- * Copyright (c) Meta Platforms, Inc. and affiliates.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- * You may select, at your option, one of the above-listed licenses.
- */
-
-/* This file provides custom allocation primitives
- */
-
-#define ZSTD_DEPS_NEED_MALLOC
-#include "zstd_deps.h"   /* ZSTD_malloc, ZSTD_calloc, ZSTD_free, ZSTD_memset */
-
-#include "compiler.h" /* MEM_STATIC */
-#define ZSTD_STATIC_LINKING_ONLY
-#include "zstd.h" /* ZSTD_customMem */
-
-#ifndef ZSTD_ALLOCATIONS_H
-#define ZSTD_ALLOCATIONS_H
-
-/* custom memory allocation functions */
-
-MEM_STATIC void* ZSTD_customMalloc(size_t size, ZSTD_customMem customMem)
-{
-    if (customMem.customAlloc)
-        return customMem.customAlloc(customMem.opaque, size);
-    return ZSTD_malloc(size);
-}
-
-MEM_STATIC void* ZSTD_customCalloc(size_t size, ZSTD_customMem customMem)
-{
-    if (customMem.customAlloc) {
-        /* calloc implemented as malloc+memset;
-         * not as efficient as calloc, but next best guess for custom malloc */
-        void* const ptr = customMem.customAlloc(customMem.opaque, size);
-        ZSTD_memset(ptr, 0, size);
-        return ptr;
-    }
-    return ZSTD_calloc(1, size);
-}
-
-MEM_STATIC void ZSTD_customFree(void* ptr, ZSTD_customMem customMem)
-{
-    if (ptr!=NULL) {
-        if (customMem.customFree)
-            customMem.customFree(customMem.opaque, ptr);
-        else
-            ZSTD_free(ptr);
-    }
-}
-
-#endif /* ZSTD_ALLOCATIONS_H */
-
-#endif /* USE_EXTERNAL_ZSTD */

vendor/github.com/DataDog/zstd/bits.h

@@ -1,208 +0,0 @@
-#ifndef USE_EXTERNAL_ZSTD
-/*
- * Copyright (c) Meta Platforms, Inc. and affiliates.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- * You may select, at your option, one of the above-listed licenses.
- */
-
-#ifndef ZSTD_BITS_H
-#define ZSTD_BITS_H
-
-#include "mem.h"
-
-MEM_STATIC unsigned ZSTD_countTrailingZeros32_fallback(U32 val)
-{
-    assert(val != 0);
-    {
-        static const U32 DeBruijnBytePos[32] = {0, 1, 28, 2, 29, 14, 24, 3,
-                                                30, 22, 20, 15, 25, 17, 4, 8,
-                                                31, 27, 13, 23, 21, 19, 16, 7,
-                                                26, 12, 18, 6, 11, 5, 10, 9};
-        return DeBruijnBytePos[((U32) ((val & -(S32) val) * 0x077CB531U)) >> 27];
-    }
-}
-
-MEM_STATIC unsigned ZSTD_countTrailingZeros32(U32 val)
-{
-    assert(val != 0);
-#if defined(_MSC_VER)
-#  if STATIC_BMI2
-    return (unsigned)_tzcnt_u32(val);
-#  else
-    if (val != 0) {
-        unsigned long r;
-        _BitScanForward(&r, val);
-        return (unsigned)r;
-    } else {
-        __assume(0); /* Should not reach this code path */
-    }
-#  endif
-#elif defined(__GNUC__) && (__GNUC__ >= 4)
-    return (unsigned)__builtin_ctz(val);
-#elif defined(__ICCARM__)
-    return (unsigned)__builtin_ctz(val);
-#else
-    return ZSTD_countTrailingZeros32_fallback(val);
-#endif
-}
-
-MEM_STATIC unsigned ZSTD_countLeadingZeros32_fallback(U32 val)
-{
-    assert(val != 0);
-    {
-        static const U32 DeBruijnClz[32] = {0, 9, 1, 10, 13, 21, 2, 29,
-                                            11, 14, 16, 18, 22, 25, 3, 30,
-                                            8, 12, 20, 28, 15, 17, 24, 7,
-                                            19, 27, 23, 6, 26, 5, 4, 31};
-        val |= val >> 1;
-        val |= val >> 2;
-        val |= val >> 4;
-        val |= val >> 8;
-        val |= val >> 16;
-        return 31 - DeBruijnClz[(val * 0x07C4ACDDU) >> 27];
-    }
-}
-
-MEM_STATIC unsigned ZSTD_countLeadingZeros32(U32 val)
-{
-    assert(val != 0);
-#if defined(_MSC_VER)
-#  if STATIC_BMI2
-    return (unsigned)_lzcnt_u32(val);
-#  else
-    if (val != 0) {
-        unsigned long r;
-        _BitScanReverse(&r, val);
-        return (unsigned)(31 - r);
-    } else {
-        __assume(0); /* Should not reach this code path */
-    }
-#  endif
-#elif defined(__GNUC__) && (__GNUC__ >= 4)
-    return (unsigned)__builtin_clz(val);
-#elif defined(__ICCARM__)
-    return (unsigned)__builtin_clz(val);
-#else
-    return ZSTD_countLeadingZeros32_fallback(val);
-#endif
-}
-
-MEM_STATIC unsigned ZSTD_countTrailingZeros64(U64 val)
-{
-    assert(val != 0);
-#if defined(_MSC_VER) && defined(_WIN64)
-#  if STATIC_BMI2
-    return (unsigned)_tzcnt_u64(val);
-#  else
-    if (val != 0) {
-        unsigned long r;
-        _BitScanForward64(&r, val);
-        return (unsigned)r;
-    } else {
-        __assume(0); /* Should not reach this code path */
-    }
-#  endif
-#elif defined(__GNUC__) && (__GNUC__ >= 4) && defined(__LP64__)
-    return (unsigned)__builtin_ctzll(val);
-#elif defined(__ICCARM__)
-    return (unsigned)__builtin_ctzll(val);
-#else
-    {
-        U32 mostSignificantWord = (U32)(val >> 32);
-        U32 leastSignificantWord = (U32)val;
-        if (leastSignificantWord == 0) {
-            return 32 + ZSTD_countTrailingZeros32(mostSignificantWord);
-        } else {
-            return ZSTD_countTrailingZeros32(leastSignificantWord);
-        }
-    }
-#endif
-}
-
-MEM_STATIC unsigned ZSTD_countLeadingZeros64(U64 val)
-{
-    assert(val != 0);
-#if defined(_MSC_VER) && defined(_WIN64)
-#  if STATIC_BMI2
-    return (unsigned)_lzcnt_u64(val);
-#  else
-    if (val != 0) {
-        unsigned long r;
-        _BitScanReverse64(&r, val);
-        return (unsigned)(63 - r);
-    } else {
-        __assume(0); /* Should not reach this code path */
-    }
-#  endif
-#elif defined(__GNUC__) && (__GNUC__ >= 4)
-    return (unsigned)(__builtin_clzll(val));
-#elif defined(__ICCARM__)
-    return (unsigned)(__builtin_clzll(val));
-#else
-    {
-        U32 mostSignificantWord = (U32)(val >> 32);
-        U32 leastSignificantWord = (U32)val;
-        if (mostSignificantWord == 0) {
-            return 32 + ZSTD_countLeadingZeros32(leastSignificantWord);
-        } else {
-            return ZSTD_countLeadingZeros32(mostSignificantWord);
-        }
-    }
-#endif
-}
-
-MEM_STATIC unsigned ZSTD_NbCommonBytes(size_t val)
-{
-    if (MEM_isLittleEndian()) {
-        if (MEM_64bits()) {
-            return ZSTD_countTrailingZeros64((U64)val) >> 3;
-        } else {
-            return ZSTD_countTrailingZeros32((U32)val) >> 3;
-        }
-    } else {  /* Big Endian CPU */
-        if (MEM_64bits()) {
-            return ZSTD_countLeadingZeros64((U64)val) >> 3;
-        } else {
-            return ZSTD_countLeadingZeros32((U32)val) >> 3;
-        }
-    }
-}
-
-MEM_STATIC unsigned ZSTD_highbit32(U32 val)   /* compress, dictBuilder, decodeCorpus */
-{
-    assert(val != 0);
-    return 31 - ZSTD_countLeadingZeros32(val);
-}
-
-/* ZSTD_rotateRight_*():
- * Rotates a bitfield to the right by "count" bits.
- * https://en.wikipedia.org/w/index.php?title=Circular_shift&oldid=991635599#Implementing_circular_shifts
- */
-MEM_STATIC
-U64 ZSTD_rotateRight_U64(U64 const value, U32 count) {
-    assert(count < 64);
-    count &= 0x3F; /* for fickle pattern recognition */
-    return (value >> count) | (U64)(value << ((0U - count) & 0x3F));
-}
-
-MEM_STATIC
-U32 ZSTD_rotateRight_U32(U32 const value, U32 count) {
-    assert(count < 32);
-    count &= 0x1F; /* for fickle pattern recognition */
-    return (value >> count) | (U32)(value << ((0U - count) & 0x1F));
-}
-
-MEM_STATIC
-U16 ZSTD_rotateRight_U16(U16 const value, U32 count) {
-    assert(count < 16);
-    count &= 0x0F; /* for fickle pattern recognition */
-    return (value >> count) | (U16)(value << ((0U - count) & 0x0F));
-}
-
-#endif /* ZSTD_BITS_H */
-
-#endif /* USE_EXTERNAL_ZSTD */

vendor/github.com/DataDog/zstd/bitstream.h

@@ -1,457 +0,0 @@
-#ifndef USE_EXTERNAL_ZSTD
-/* ******************************************************************
- * bitstream
- * Part of FSE library
- * Copyright (c) Meta Platforms, Inc. and affiliates.
- *
- * You can contact the author at :
- * - Source repository : https://github.com/Cyan4973/FiniteStateEntropy
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- * You may select, at your option, one of the above-listed licenses.
-****************************************************************** */
-#ifndef BITSTREAM_H_MODULE
-#define BITSTREAM_H_MODULE
-
-/*
-*  This API consists of small unitary functions, which must be inlined for best performance.
-*  Since link-time-optimization is not available for all compilers,
-*  these functions are defined into a .h to be included.
-*/
-
-/*-****************************************
-*  Dependencies
-******************************************/
-#include "mem.h"            /* unaligned access routines */
-#include "compiler.h"       /* UNLIKELY() */
-#include "debug.h"          /* assert(), DEBUGLOG(), RAWLOG() */
-#include "error_private.h"  /* error codes and messages */
-#include "bits.h"           /* ZSTD_highbit32 */
-
-/*=========================================
-*  Target specific
-=========================================*/
-#ifndef ZSTD_NO_INTRINSICS
-#  if (defined(__BMI__) || defined(__BMI2__)) && defined(__GNUC__)
-#    include <immintrin.h>   /* support for bextr (experimental)/bzhi */
-#  elif defined(__ICCARM__)
-#    include <intrinsics.h>
-#  endif
-#endif
-
-#define STREAM_ACCUMULATOR_MIN_32  25
-#define STREAM_ACCUMULATOR_MIN_64  57
-#define STREAM_ACCUMULATOR_MIN    ((U32)(MEM_32bits() ? STREAM_ACCUMULATOR_MIN_32 : STREAM_ACCUMULATOR_MIN_64))
-
-
-/*-******************************************
-*  bitStream encoding API (write forward)
-********************************************/
-typedef size_t BitContainerType;
-/* bitStream can mix input from multiple sources.
- * A critical property of these streams is that they encode and decode in **reverse** direction.
- * So the first bit sequence you add will be the last to be read, like a LIFO stack.
- */
-typedef struct {
-    BitContainerType bitContainer;
-    unsigned bitPos;
-    char*  startPtr;
-    char*  ptr;
-    char*  endPtr;
-} BIT_CStream_t;
-
-MEM_STATIC size_t BIT_initCStream(BIT_CStream_t* bitC, void* dstBuffer, size_t dstCapacity);
-MEM_STATIC void   BIT_addBits(BIT_CStream_t* bitC, BitContainerType value, unsigned nbBits);
-MEM_STATIC void   BIT_flushBits(BIT_CStream_t* bitC);
-MEM_STATIC size_t BIT_closeCStream(BIT_CStream_t* bitC);
-
-/* Start with initCStream, providing the size of buffer to write into.
-*  bitStream will never write outside of this buffer.
-*  `dstCapacity` must be >= sizeof(bitD->bitContainer), otherwise @return will be an error code.
-*
-*  bits are first added to a local register.
-*  Local register is BitContainerType, 64-bits on 64-bits systems, or 32-bits on 32-bits systems.
-*  Writing data into memory is an explicit operation, performed by the flushBits function.
-*  Hence keep track how many bits are potentially stored into local register to avoid register overflow.
-*  After a flushBits, a maximum of 7 bits might still be stored into local register.
-*
-*  Avoid storing elements of more than 24 bits if you want compatibility with 32-bits bitstream readers.
-*
-*  Last operation is to close the bitStream.
-*  The function returns the final size of CStream in bytes.
-*  If data couldn't fit into `dstBuffer`, it will return a 0 ( == not storable)
-*/
-
-
-/*-********************************************
-*  bitStream decoding API (read backward)
-**********************************************/
-typedef struct {
-    BitContainerType bitContainer;
-    unsigned bitsConsumed;
-    const char* ptr;
-    const char* start;
-    const char* limitPtr;
-} BIT_DStream_t;
-
-typedef enum { BIT_DStream_unfinished = 0,  /* fully refilled */
-               BIT_DStream_endOfBuffer = 1, /* still some bits left in bitstream */
-               BIT_DStream_completed = 2,   /* bitstream entirely consumed, bit-exact */
-               BIT_DStream_overflow = 3     /* user requested more bits than present in bitstream */
-    } BIT_DStream_status;  /* result of BIT_reloadDStream() */
-
-MEM_STATIC size_t   BIT_initDStream(BIT_DStream_t* bitD, const void* srcBuffer, size_t srcSize);
-MEM_STATIC BitContainerType BIT_readBits(BIT_DStream_t* bitD, unsigned nbBits);
-MEM_STATIC BIT_DStream_status BIT_reloadDStream(BIT_DStream_t* bitD);
-MEM_STATIC unsigned BIT_endOfDStream(const BIT_DStream_t* bitD);
-
-
-/* Start by invoking BIT_initDStream().
-*  A chunk of the bitStream is then stored into a local register.
-*  Local register size is 64-bits on 64-bits systems, 32-bits on 32-bits systems (BitContainerType).
-*  You can then retrieve bitFields stored into the local register, **in reverse order**.
-*  Local register is explicitly reloaded from memory by the BIT_reloadDStream() method.
-*  A reload guarantee a minimum of ((8*sizeof(bitD->bitContainer))-7) bits when its result is BIT_DStream_unfinished.
-*  Otherwise, it can be less than that, so proceed accordingly.
-*  Checking if DStream has reached its end can be performed with BIT_endOfDStream().
-*/
-
-
-/*-****************************************
-*  unsafe API
-******************************************/
-MEM_STATIC void BIT_addBitsFast(BIT_CStream_t* bitC, BitContainerType value, unsigned nbBits);
-/* faster, but works only if value is "clean", meaning all high bits above nbBits are 0 */
-
-MEM_STATIC void BIT_flushBitsFast(BIT_CStream_t* bitC);
-/* unsafe version; does not check buffer overflow */
-
-MEM_STATIC size_t BIT_readBitsFast(BIT_DStream_t* bitD, unsigned nbBits);
-/* faster, but works only if nbBits >= 1 */
-
-/*=====    Local Constants   =====*/
-static const unsigned BIT_mask[] = {
-    0,          1,         3,         7,         0xF,       0x1F,
-    0x3F,       0x7F,      0xFF,      0x1FF,     0x3FF,     0x7FF,
-    0xFFF,      0x1FFF,    0x3FFF,    0x7FFF,    0xFFFF,    0x1FFFF,
-    0x3FFFF,    0x7FFFF,   0xFFFFF,   0x1FFFFF,  0x3FFFFF,  0x7FFFFF,
-    0xFFFFFF,   0x1FFFFFF, 0x3FFFFFF, 0x7FFFFFF, 0xFFFFFFF, 0x1FFFFFFF,
-    0x3FFFFFFF, 0x7FFFFFFF}; /* up to 31 bits */
-#define BIT_MASK_SIZE (sizeof(BIT_mask) / sizeof(BIT_mask[0]))
-
-/*-**************************************************************
-*  bitStream encoding
-****************************************************************/
-/*! BIT_initCStream() :
- *  `dstCapacity` must be > sizeof(size_t)
- *  @return : 0 if success,
- *            otherwise an error code (can be tested using ERR_isError()) */
-MEM_STATIC size_t BIT_initCStream(BIT_CStream_t* bitC,
-                                  void* startPtr, size_t dstCapacity)
-{
-    bitC->bitContainer = 0;
-    bitC->bitPos = 0;
-    bitC->startPtr = (char*)startPtr;
-    bitC->ptr = bitC->startPtr;
-    bitC->endPtr = bitC->startPtr + dstCapacity - sizeof(bitC->bitContainer);
-    if (dstCapacity <= sizeof(bitC->bitContainer)) return ERROR(dstSize_tooSmall);
-    return 0;
-}
-
-FORCE_INLINE_TEMPLATE BitContainerType BIT_getLowerBits(BitContainerType bitContainer, U32 const nbBits)
-{
-#if STATIC_BMI2 && !defined(ZSTD_NO_INTRINSICS)
-#  if (defined(__x86_64__) || defined(_M_X64)) && !defined(__ILP32__)
-    return _bzhi_u64(bitContainer, nbBits);
-#  else
-    DEBUG_STATIC_ASSERT(sizeof(bitContainer) == sizeof(U32));
-    return _bzhi_u32(bitContainer, nbBits);
-#  endif
-#else
-    assert(nbBits < BIT_MASK_SIZE);
-    return bitContainer & BIT_mask[nbBits];
-#endif
-}
-
-/*! BIT_addBits() :
- *  can add up to 31 bits into `bitC`.
- *  Note : does not check for register overflow ! */
-MEM_STATIC void BIT_addBits(BIT_CStream_t* bitC,
-                            BitContainerType value, unsigned nbBits)
-{
-    DEBUG_STATIC_ASSERT(BIT_MASK_SIZE == 32);
-    assert(nbBits < BIT_MASK_SIZE);
-    assert(nbBits + bitC->bitPos < sizeof(bitC->bitContainer) * 8);
-    bitC->bitContainer |= BIT_getLowerBits(value, nbBits) << bitC->bitPos;
-    bitC->bitPos += nbBits;
-}
-
-/*! BIT_addBitsFast() :
- *  works only if `value` is _clean_,
- *  meaning all high bits above nbBits are 0 */
-MEM_STATIC void BIT_addBitsFast(BIT_CStream_t* bitC,
-                                BitContainerType value, unsigned nbBits)
-{
-    assert((value>>nbBits) == 0);
-    assert(nbBits + bitC->bitPos < sizeof(bitC->bitContainer) * 8);
-    bitC->bitContainer |= value << bitC->bitPos;
-    bitC->bitPos += nbBits;
-}
-
-/*! BIT_flushBitsFast() :
- *  assumption : bitContainer has not overflowed
- *  unsafe version; does not check buffer overflow */
-MEM_STATIC void BIT_flushBitsFast(BIT_CStream_t* bitC)
-{
-    size_t const nbBytes = bitC->bitPos >> 3;
-    assert(bitC->bitPos < sizeof(bitC->bitContainer) * 8);
-    assert(bitC->ptr <= bitC->endPtr);
-    MEM_writeLEST(bitC->ptr, bitC->bitContainer);
-    bitC->ptr += nbBytes;
-    bitC->bitPos &= 7;
-    bitC->bitContainer >>= nbBytes*8;
-}
-
-/*! BIT_flushBits() :
- *  assumption : bitContainer has not overflowed
- *  safe version; check for buffer overflow, and prevents it.
- *  note : does not signal buffer overflow.
- *  overflow will be revealed later on using BIT_closeCStream() */
-MEM_STATIC void BIT_flushBits(BIT_CStream_t* bitC)
-{
-    size_t const nbBytes = bitC->bitPos >> 3;
-    assert(bitC->bitPos < sizeof(bitC->bitContainer) * 8);
-    assert(bitC->ptr <= bitC->endPtr);
-    MEM_writeLEST(bitC->ptr, bitC->bitContainer);
-    bitC->ptr += nbBytes;
-    if (bitC->ptr > bitC->endPtr) bitC->ptr = bitC->endPtr;
-    bitC->bitPos &= 7;
-    bitC->bitContainer >>= nbBytes*8;
-}
-
-/*! BIT_closeCStream() :
- *  @return : size of CStream, in bytes,
- *            or 0 if it could not fit into dstBuffer */
-MEM_STATIC size_t BIT_closeCStream(BIT_CStream_t* bitC)
-{
-    BIT_addBitsFast(bitC, 1, 1);   /* endMark */
-    BIT_flushBits(bitC);
-    if (bitC->ptr >= bitC->endPtr) return 0; /* overflow detected */
-    return (size_t)(bitC->ptr - bitC->startPtr) + (bitC->bitPos > 0);
-}
-
-
-/*-********************************************************
-*  bitStream decoding
-**********************************************************/
-/*! BIT_initDStream() :
- *  Initialize a BIT_DStream_t.
- * `bitD` : a pointer to an already allocated BIT_DStream_t structure.
- * `srcSize` must be the *exact* size of the bitStream, in bytes.
- * @return : size of stream (== srcSize), or an errorCode if a problem is detected
- */
-MEM_STATIC size_t BIT_initDStream(BIT_DStream_t* bitD, const void* srcBuffer, size_t srcSize)
-{
-    if (srcSize < 1) { ZSTD_memset(bitD, 0, sizeof(*bitD)); return ERROR(srcSize_wrong); }
-
-    bitD->start = (const char*)srcBuffer;
-    bitD->limitPtr = bitD->start + sizeof(bitD->bitContainer);
-
-    if (srcSize >=  sizeof(bitD->bitContainer)) {  /* normal case */
-        bitD->ptr   = (const char*)srcBuffer + srcSize - sizeof(bitD->bitContainer);
-        bitD->bitContainer = MEM_readLEST(bitD->ptr);
-        { BYTE const lastByte = ((const BYTE*)srcBuffer)[srcSize-1];
-          bitD->bitsConsumed = lastByte ? 8 - ZSTD_highbit32(lastByte) : 0;  /* ensures bitsConsumed is always set */
-          if (lastByte == 0) return ERROR(GENERIC); /* endMark not present */ }
-    } else {
-        bitD->ptr   = bitD->start;
-        bitD->bitContainer = *(const BYTE*)(bitD->start);
-        switch(srcSize)
-        {
-        case 7: bitD->bitContainer += (BitContainerType)(((const BYTE*)(srcBuffer))[6]) << (sizeof(bitD->bitContainer)*8 - 16);
-                ZSTD_FALLTHROUGH;
-
-        case 6: bitD->bitContainer += (BitContainerType)(((const BYTE*)(srcBuffer))[5]) << (sizeof(bitD->bitContainer)*8 - 24);
-                ZSTD_FALLTHROUGH;
-
-        case 5: bitD->bitContainer += (BitContainerType)(((const BYTE*)(srcBuffer))[4]) << (sizeof(bitD->bitContainer)*8 - 32);
-                ZSTD_FALLTHROUGH;
-
-        case 4: bitD->bitContainer += (BitContainerType)(((const BYTE*)(srcBuffer))[3]) << 24;
-                ZSTD_FALLTHROUGH;
-
-        case 3: bitD->bitContainer += (BitContainerType)(((const BYTE*)(srcBuffer))[2]) << 16;
-                ZSTD_FALLTHROUGH;
-
-        case 2: bitD->bitContainer += (BitContainerType)(((const BYTE*)(srcBuffer))[1]) <<  8;
-                ZSTD_FALLTHROUGH;
-
-        default: break;
-        }
-        {   BYTE const lastByte = ((const BYTE*)srcBuffer)[srcSize-1];
-            bitD->bitsConsumed = lastByte ? 8 - ZSTD_highbit32(lastByte) : 0;
-            if (lastByte == 0) return ERROR(corruption_detected);  /* endMark not present */
-        }
-        bitD->bitsConsumed += (U32)(sizeof(bitD->bitContainer) - srcSize)*8;
-    }
-
-    return srcSize;
-}
-
-FORCE_INLINE_TEMPLATE BitContainerType BIT_getUpperBits(BitContainerType bitContainer, U32 const start)
-{
-    return bitContainer >> start;
-}
-
-FORCE_INLINE_TEMPLATE BitContainerType BIT_getMiddleBits(BitContainerType bitContainer, U32 const start, U32 const nbBits)
-{
-    U32 const regMask = sizeof(bitContainer)*8 - 1;
-    /* if start > regMask, bitstream is corrupted, and result is undefined */
-    assert(nbBits < BIT_MASK_SIZE);
-    /* x86 transform & ((1 << nbBits) - 1) to bzhi instruction, it is better
-     * than accessing memory. When bmi2 instruction is not present, we consider
-     * such cpus old (pre-Haswell, 2013) and their performance is not of that
-     * importance.
-     */
-#if defined(__x86_64__) || defined(_M_X64)
-    return (bitContainer >> (start & regMask)) & ((((U64)1) << nbBits) - 1);
-#else
-    return (bitContainer >> (start & regMask)) & BIT_mask[nbBits];
-#endif
-}
-
-/*! BIT_lookBits() :
- *  Provides next n bits from local register.
- *  local register is not modified.
- *  On 32-bits, maxNbBits==24.
- *  On 64-bits, maxNbBits==56.
- * @return : value extracted */
-FORCE_INLINE_TEMPLATE BitContainerType BIT_lookBits(const BIT_DStream_t*  bitD, U32 nbBits)
-{
-    /* arbitrate between double-shift and shift+mask */
-#if 1
-    /* if bitD->bitsConsumed + nbBits > sizeof(bitD->bitContainer)*8,
-     * bitstream is likely corrupted, and result is undefined */
-    return BIT_getMiddleBits(bitD->bitContainer, (sizeof(bitD->bitContainer)*8) - bitD->bitsConsumed - nbBits, nbBits);
-#else
-    /* this code path is slower on my os-x laptop */
-    U32 const regMask = sizeof(bitD->bitContainer)*8 - 1;
-    return ((bitD->bitContainer << (bitD->bitsConsumed & regMask)) >> 1) >> ((regMask-nbBits) & regMask);
-#endif
-}
-
-/*! BIT_lookBitsFast() :
- *  unsafe version; only works if nbBits >= 1 */
-MEM_STATIC BitContainerType BIT_lookBitsFast(const BIT_DStream_t* bitD, U32 nbBits)
-{
-    U32 const regMask = sizeof(bitD->bitContainer)*8 - 1;
-    assert(nbBits >= 1);
-    return (bitD->bitContainer << (bitD->bitsConsumed & regMask)) >> (((regMask+1)-nbBits) & regMask);
-}
-
-FORCE_INLINE_TEMPLATE void BIT_skipBits(BIT_DStream_t* bitD, U32 nbBits)
-{
-    bitD->bitsConsumed += nbBits;
-}
-
-/*! BIT_readBits() :
- *  Read (consume) next n bits from local register and update.
- *  Pay attention to not read more than nbBits contained into local register.
- * @return : extracted value. */
-FORCE_INLINE_TEMPLATE BitContainerType BIT_readBits(BIT_DStream_t* bitD, unsigned nbBits)
-{
-    BitContainerType const value = BIT_lookBits(bitD, nbBits);
-    BIT_skipBits(bitD, nbBits);
-    return value;
-}
-
-/*! BIT_readBitsFast() :
- *  unsafe version; only works if nbBits >= 1 */
-MEM_STATIC BitContainerType BIT_readBitsFast(BIT_DStream_t* bitD, unsigned nbBits)
-{
-    BitContainerType const value = BIT_lookBitsFast(bitD, nbBits);
-    assert(nbBits >= 1);
-    BIT_skipBits(bitD, nbBits);
-    return value;
-}
-
-/*! BIT_reloadDStream_internal() :
- *  Simple variant of BIT_reloadDStream(), with two conditions:
- *  1. bitstream is valid : bitsConsumed <= sizeof(bitD->bitContainer)*8
- *  2. look window is valid after shifted down : bitD->ptr >= bitD->start
- */
-MEM_STATIC BIT_DStream_status BIT_reloadDStream_internal(BIT_DStream_t* bitD)
-{
-    assert(bitD->bitsConsumed <= sizeof(bitD->bitContainer)*8);
-    bitD->ptr -= bitD->bitsConsumed >> 3;
-    assert(bitD->ptr >= bitD->start);
-    bitD->bitsConsumed &= 7;
-    bitD->bitContainer = MEM_readLEST(bitD->ptr);
-    return BIT_DStream_unfinished;
-}
-
-/*! BIT_reloadDStreamFast() :
- *  Similar to BIT_reloadDStream(), but with two differences:
- *  1. bitsConsumed <= sizeof(bitD->bitContainer)*8 must hold!
- *  2. Returns BIT_DStream_overflow when bitD->ptr < bitD->limitPtr, at this
- *     point you must use BIT_reloadDStream() to reload.
- */
-MEM_STATIC BIT_DStream_status BIT_reloadDStreamFast(BIT_DStream_t* bitD)
-{
-    if (UNLIKELY(bitD->ptr < bitD->limitPtr))
-        return BIT_DStream_overflow;
-    return BIT_reloadDStream_internal(bitD);
-}
-
-/*! BIT_reloadDStream() :
- *  Refill `bitD` from buffer previously set in BIT_initDStream() .
- *  This function is safe, it guarantees it will not never beyond src buffer.
- * @return : status of `BIT_DStream_t` internal register.
- *           when status == BIT_DStream_unfinished, internal register is filled with at least 25 or 57 bits */
-FORCE_INLINE_TEMPLATE BIT_DStream_status BIT_reloadDStream(BIT_DStream_t* bitD)
-{
-    /* note : once in overflow mode, a bitstream remains in this mode until it's reset */
-    if (UNLIKELY(bitD->bitsConsumed > (sizeof(bitD->bitContainer)*8))) {
-        static const BitContainerType zeroFilled = 0;
-        bitD->ptr = (const char*)&zeroFilled; /* aliasing is allowed for char */
-        /* overflow detected, erroneous scenario or end of stream: no update */
-        return BIT_DStream_overflow;
-    }
-
-    assert(bitD->ptr >= bitD->start);
-
-    if (bitD->ptr >= bitD->limitPtr) {
-        return BIT_reloadDStream_internal(bitD);
-    }
-    if (bitD->ptr == bitD->start) {
-        /* reached end of bitStream => no update */
-        if (bitD->bitsConsumed < sizeof(bitD->bitContainer)*8) return BIT_DStream_endOfBuffer;
-        return BIT_DStream_completed;
-    }
-    /* start < ptr < limitPtr => cautious update */
-    {   U32 nbBytes = bitD->bitsConsumed >> 3;
-        BIT_DStream_status result = BIT_DStream_unfinished;
-        if (bitD->ptr - nbBytes < bitD->start) {
-            nbBytes = (U32)(bitD->ptr - bitD->start);  /* ptr > start */
-            result = BIT_DStream_endOfBuffer;
-        }
-        bitD->ptr -= nbBytes;
-        bitD->bitsConsumed -= nbBytes*8;
-        bitD->bitContainer = MEM_readLEST(bitD->ptr);   /* reminder : srcSize > sizeof(bitD->bitContainer), otherwise bitD->ptr == bitD->start */
-        return result;
-    }
-}
-
-/*! BIT_endOfDStream() :
- * @return : 1 if DStream has _exactly_ reached its end (all bits consumed).
- */
-MEM_STATIC unsigned BIT_endOfDStream(const BIT_DStream_t* DStream)
-{
-    return ((DStream->ptr == DStream->start) && (DStream->bitsConsumed == sizeof(DStream->bitContainer)*8));
-}
-
-#endif /* BITSTREAM_H_MODULE */
-
-#endif /* USE_EXTERNAL_ZSTD */

vendor/github.com/DataDog/zstd/clevels.h

@@ -1,137 +0,0 @@
-#ifndef USE_EXTERNAL_ZSTD
-/*
- * Copyright (c) Meta Platforms, Inc. and affiliates.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- * You may select, at your option, one of the above-listed licenses.
- */
-
-#ifndef ZSTD_CLEVELS_H
-#define ZSTD_CLEVELS_H
-
-#define ZSTD_STATIC_LINKING_ONLY  /* ZSTD_compressionParameters  */
-#include "zstd.h"
-
-/*-=====  Pre-defined compression levels  =====-*/
-
-#define ZSTD_MAX_CLEVEL     22
-
-#ifdef __GNUC__
-__attribute__((__unused__))
-#endif
-
-static const ZSTD_compressionParameters ZSTD_defaultCParameters[4][ZSTD_MAX_CLEVEL+1] = {
-{   /* "default" - for any srcSize > 256 KB */
-    /* W,  C,  H,  S,  L, TL, strat */
-    { 19, 12, 13,  1,  6,  1, ZSTD_fast    },  /* base for negative levels */
-    { 19, 13, 14,  1,  7,  0, ZSTD_fast    },  /* level  1 */
-    { 20, 15, 16,  1,  6,  0, ZSTD_fast    },  /* level  2 */
-    { 21, 16, 17,  1,  5,  0, ZSTD_dfast   },  /* level  3 */
-    { 21, 18, 18,  1,  5,  0, ZSTD_dfast   },  /* level  4 */
-    { 21, 18, 19,  3,  5,  2, ZSTD_greedy  },  /* level  5 */
-    { 21, 18, 19,  3,  5,  4, ZSTD_lazy    },  /* level  6 */
-    { 21, 19, 20,  4,  5,  8, ZSTD_lazy    },  /* level  7 */
-    { 21, 19, 20,  4,  5, 16, ZSTD_lazy2   },  /* level  8 */
-    { 22, 20, 21,  4,  5, 16, ZSTD_lazy2   },  /* level  9 */
-    { 22, 21, 22,  5,  5, 16, ZSTD_lazy2   },  /* level 10 */
-    { 22, 21, 22,  6,  5, 16, ZSTD_lazy2   },  /* level 11 */
-    { 22, 22, 23,  6,  5, 32, ZSTD_lazy2   },  /* level 12 */
-    { 22, 22, 22,  4,  5, 32, ZSTD_btlazy2 },  /* level 13 */
-    { 22, 22, 23,  5,  5, 32, ZSTD_btlazy2 },  /* level 14 */
-    { 22, 23, 23,  6,  5, 32, ZSTD_btlazy2 },  /* level 15 */
-    { 22, 22, 22,  5,  5, 48, ZSTD_btopt   },  /* level 16 */
-    { 23, 23, 22,  5,  4, 64, ZSTD_btopt   },  /* level 17 */
-    { 23, 23, 22,  6,  3, 64, ZSTD_btultra },  /* level 18 */
-    { 23, 24, 22,  7,  3,256, ZSTD_btultra2},  /* level 19 */
-    { 25, 25, 23,  7,  3,256, ZSTD_btultra2},  /* level 20 */
-    { 26, 26, 24,  7,  3,512, ZSTD_btultra2},  /* level 21 */
-    { 27, 27, 25,  9,  3,999, ZSTD_btultra2},  /* level 22 */
-},
-{   /* for srcSize <= 256 KB */
-    /* W,  C,  H,  S,  L,  T, strat */
-    { 18, 12, 13,  1,  5,  1, ZSTD_fast    },  /* base for negative levels */
-    { 18, 13, 14,  1,  6,  0, ZSTD_fast    },  /* level  1 */
-    { 18, 14, 14,  1,  5,  0, ZSTD_dfast   },  /* level  2 */
-    { 18, 16, 16,  1,  4,  0, ZSTD_dfast   },  /* level  3 */
-    { 18, 16, 17,  3,  5,  2, ZSTD_greedy  },  /* level  4.*/
-    { 18, 17, 18,  5,  5,  2, ZSTD_greedy  },  /* level  5.*/
-    { 18, 18, 19,  3,  5,  4, ZSTD_lazy    },  /* level  6.*/
-    { 18, 18, 19,  4,  4,  4, ZSTD_lazy    },  /* level  7 */
-    { 18, 18, 19,  4,  4,  8, ZSTD_lazy2   },  /* level  8 */
-    { 18, 18, 19,  5,  4,  8, ZSTD_lazy2   },  /* level  9 */
-    { 18, 18, 19,  6,  4,  8, ZSTD_lazy2   },  /* level 10 */
-    { 18, 18, 19,  5,  4, 12, ZSTD_btlazy2 },  /* level 11.*/
-    { 18, 19, 19,  7,  4, 12, ZSTD_btlazy2 },  /* level 12.*/
-    { 18, 18, 19,  4,  4, 16, ZSTD_btopt   },  /* level 13 */
-    { 18, 18, 19,  4,  3, 32, ZSTD_btopt   },  /* level 14.*/
-    { 18, 18, 19,  6,  3,128, ZSTD_btopt   },  /* level 15.*/
-    { 18, 19, 19,  6,  3,128, ZSTD_btultra },  /* level 16.*/
-    { 18, 19, 19,  8,  3,256, ZSTD_btultra },  /* level 17.*/
-    { 18, 19, 19,  6,  3,128, ZSTD_btultra2},  /* level 18.*/
-    { 18, 19, 19,  8,  3,256, ZSTD_btultra2},  /* level 19.*/
-    { 18, 19, 19, 10,  3,512, ZSTD_btultra2},  /* level 20.*/
-    { 18, 19, 19, 12,  3,512, ZSTD_btultra2},  /* level 21.*/
-    { 18, 19, 19, 13,  3,999, ZSTD_btultra2},  /* level 22.*/
-},
-{   /* for srcSize <= 128 KB */
-    /* W,  C,  H,  S,  L,  T, strat */
-    { 17, 12, 12,  1,  5,  1, ZSTD_fast    },  /* base for negative levels */
-    { 17, 12, 13,  1,  6,  0, ZSTD_fast    },  /* level  1 */
-    { 17, 13, 15,  1,  5,  0, ZSTD_fast    },  /* level  2 */
-    { 17, 15, 16,  2,  5,  0, ZSTD_dfast   },  /* level  3 */
-    { 17, 17, 17,  2,  4,  0, ZSTD_dfast   },  /* level  4 */
-    { 17, 16, 17,  3,  4,  2, ZSTD_greedy  },  /* level  5 */
-    { 17, 16, 17,  3,  4,  4, ZSTD_lazy    },  /* level  6 */
-    { 17, 16, 17,  3,  4,  8, ZSTD_lazy2   },  /* level  7 */
-    { 17, 16, 17,  4,  4,  8, ZSTD_lazy2   },  /* level  8 */
-    { 17, 16, 17,  5,  4,  8, ZSTD_lazy2   },  /* level  9 */
-    { 17, 16, 17,  6,  4,  8, ZSTD_lazy2   },  /* level 10 */
-    { 17, 17, 17,  5,  4,  8, ZSTD_btlazy2 },  /* level 11 */
-    { 17, 18, 17,  7,  4, 12, ZSTD_btlazy2 },  /* level 12 */
-    { 17, 18, 17,  3,  4, 12, ZSTD_btopt   },  /* level 13.*/
-    { 17, 18, 17,  4,  3, 32, ZSTD_btopt   },  /* level 14.*/
-    { 17, 18, 17,  6,  3,256, ZSTD_btopt   },  /* level 15.*/
-    { 17, 18, 17,  6,  3,128, ZSTD_btultra },  /* level 16.*/
-    { 17, 18, 17,  8,  3,256, ZSTD_btultra },  /* level 17.*/
-    { 17, 18, 17, 10,  3,512, ZSTD_btultra },  /* level 18.*/
-    { 17, 18, 17,  5,  3,256, ZSTD_btultra2},  /* level 19.*/
-    { 17, 18, 17,  7,  3,512, ZSTD_btultra2},  /* level 20.*/
-    { 17, 18, 17,  9,  3,512, ZSTD_btultra2},  /* level 21.*/
-    { 17, 18, 17, 11,  3,999, ZSTD_btultra2},  /* level 22.*/
-},
-{   /* for srcSize <= 16 KB */
-    /* W,  C,  H,  S,  L,  T, strat */
-    { 14, 12, 13,  1,  5,  1, ZSTD_fast    },  /* base for negative levels */
-    { 14, 14, 15,  1,  5,  0, ZSTD_fast    },  /* level  1 */
-    { 14, 14, 15,  1,  4,  0, ZSTD_fast    },  /* level  2 */
-    { 14, 14, 15,  2,  4,  0, ZSTD_dfast   },  /* level  3 */
-    { 14, 14, 14,  4,  4,  2, ZSTD_greedy  },  /* level  4 */
-    { 14, 14, 14,  3,  4,  4, ZSTD_lazy    },  /* level  5.*/
-    { 14, 14, 14,  4,  4,  8, ZSTD_lazy2   },  /* level  6 */
-    { 14, 14, 14,  6,  4,  8, ZSTD_lazy2   },  /* level  7 */
-    { 14, 14, 14,  8,  4,  8, ZSTD_lazy2   },  /* level  8.*/
-    { 14, 15, 14,  5,  4,  8, ZSTD_btlazy2 },  /* level  9.*/
-    { 14, 15, 14,  9,  4,  8, ZSTD_btlazy2 },  /* level 10.*/
-    { 14, 15, 14,  3,  4, 12, ZSTD_btopt   },  /* level 11.*/
-    { 14, 15, 14,  4,  3, 24, ZSTD_btopt   },  /* level 12.*/
-    { 14, 15, 14,  5,  3, 32, ZSTD_btultra },  /* level 13.*/
-    { 14, 15, 15,  6,  3, 64, ZSTD_btultra },  /* level 14.*/
-    { 14, 15, 15,  7,  3,256, ZSTD_btultra },  /* level 15.*/
-    { 14, 15, 15,  5,  3, 48, ZSTD_btultra2},  /* level 16.*/
-    { 14, 15, 15,  6,  3,128, ZSTD_btultra2},  /* level 17.*/
-    { 14, 15, 15,  7,  3,256, ZSTD_btultra2},  /* level 18.*/
-    { 14, 15, 15,  8,  3,256, ZSTD_btultra2},  /* level 19.*/
-    { 14, 15, 15,  8,  3,512, ZSTD_btultra2},  /* level 20.*/
-    { 14, 15, 15,  9,  3,512, ZSTD_btultra2},  /* level 21.*/
-    { 14, 15, 15, 10,  3,999, ZSTD_btultra2},  /* level 22.*/
-},
-};
-
-
-
-#endif  /* ZSTD_CLEVELS_H */
-
-#endif /* USE_EXTERNAL_ZSTD */

vendor/github.com/DataDog/zstd/compiler.h

@@ -1,467 +0,0 @@
-#ifndef USE_EXTERNAL_ZSTD
-/*
- * Copyright (c) Meta Platforms, Inc. and affiliates.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- * You may select, at your option, one of the above-listed licenses.
- */
-
-#ifndef ZSTD_COMPILER_H
-#define ZSTD_COMPILER_H
-
-#include <stddef.h>
-
-#include "portability_macros.h"
-
-/*-*******************************************************
-*  Compiler specifics
-*********************************************************/
-/* force inlining */
-
-#if !defined(ZSTD_NO_INLINE)
-#if (defined(__GNUC__) && !defined(__STRICT_ANSI__)) || defined(__cplusplus) || defined(__STDC_VERSION__) && __STDC_VERSION__ >= 199901L   /* C99 */
-#  define INLINE_KEYWORD inline
-#else
-#  define INLINE_KEYWORD
-#endif
-
-#if defined(__GNUC__) || defined(__IAR_SYSTEMS_ICC__)
-#  define FORCE_INLINE_ATTR __attribute__((always_inline))
-#elif defined(_MSC_VER)
-#  define FORCE_INLINE_ATTR __forceinline
-#else
-#  define FORCE_INLINE_ATTR
-#endif
-
-#else
-
-#define INLINE_KEYWORD
-#define FORCE_INLINE_ATTR
-
-#endif
-
-/**
-  On MSVC qsort requires that functions passed into it use the __cdecl calling conversion(CC).
-  This explicitly marks such functions as __cdecl so that the code will still compile
-  if a CC other than __cdecl has been made the default.
-*/
-#if  defined(_MSC_VER)
-#  define WIN_CDECL __cdecl
-#else
-#  define WIN_CDECL
-#endif
-
-/* UNUSED_ATTR tells the compiler it is okay if the function is unused. */
-#if defined(__GNUC__) || defined(__IAR_SYSTEMS_ICC__)
-#  define UNUSED_ATTR __attribute__((unused))
-#else
-#  define UNUSED_ATTR
-#endif
-
-/**
- * FORCE_INLINE_TEMPLATE is used to define C "templates", which take constant
- * parameters. They must be inlined for the compiler to eliminate the constant
- * branches.
- */
-#define FORCE_INLINE_TEMPLATE static INLINE_KEYWORD FORCE_INLINE_ATTR UNUSED_ATTR
-/**
- * HINT_INLINE is used to help the compiler generate better code. It is *not*
- * used for "templates", so it can be tweaked based on the compilers
- * performance.
- *
- * gcc-4.8 and gcc-4.9 have been shown to benefit from leaving off the
- * always_inline attribute.
- *
- * clang up to 5.0.0 (trunk) benefit tremendously from the always_inline
- * attribute.
- */
-#if !defined(__clang__) && defined(__GNUC__) && __GNUC__ >= 4 && __GNUC_MINOR__ >= 8 && __GNUC__ < 5
-#  define HINT_INLINE static INLINE_KEYWORD
-#else
-#  define HINT_INLINE FORCE_INLINE_TEMPLATE
-#endif
-
-/* "soft" inline :
- * The compiler is free to select if it's a good idea to inline or not.
- * The main objective is to silence compiler warnings
- * when a defined function in included but not used.
- *
- * Note : this macro is prefixed `MEM_` because it used to be provided by `mem.h` unit.
- * Updating the prefix is probably preferable, but requires a fairly large codemod,
- * since this name is used everywhere.
- */
-#ifndef MEM_STATIC  /* already defined in Linux Kernel mem.h */
-#if defined(__GNUC__)
-#  define MEM_STATIC static __inline UNUSED_ATTR
-#elif defined(__IAR_SYSTEMS_ICC__)
-#  define MEM_STATIC static inline UNUSED_ATTR
-#elif defined (__cplusplus) || (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */)
-#  define MEM_STATIC static inline
-#elif defined(_MSC_VER)
-#  define MEM_STATIC static __inline
-#else
-#  define MEM_STATIC static  /* this version may generate warnings for unused static functions; disable the relevant warning */
-#endif
-#endif
-
-/* force no inlining */
-#ifdef _MSC_VER
-#  define FORCE_NOINLINE static __declspec(noinline)
-#else
-#  if defined(__GNUC__) || defined(__IAR_SYSTEMS_ICC__)
-#    define FORCE_NOINLINE static __attribute__((__noinline__))
-#  else
-#    define FORCE_NOINLINE static
-#  endif
-#endif
-
-
-/* target attribute */
-#if defined(__GNUC__) || defined(__IAR_SYSTEMS_ICC__)
-#  define TARGET_ATTRIBUTE(target) __attribute__((__target__(target)))
-#else
-#  define TARGET_ATTRIBUTE(target)
-#endif
-
-/* Target attribute for BMI2 dynamic dispatch.
- * Enable lzcnt, bmi, and bmi2.
- * We test for bmi1 & bmi2. lzcnt is included in bmi1.
- */
-#define BMI2_TARGET_ATTRIBUTE TARGET_ATTRIBUTE("lzcnt,bmi,bmi2")
-
-/* prefetch
- * can be disabled, by declaring NO_PREFETCH build macro */
-#if defined(NO_PREFETCH)
-#  define PREFETCH_L1(ptr)  do { (void)(ptr); } while (0)  /* disabled */
-#  define PREFETCH_L2(ptr)  do { (void)(ptr); } while (0)  /* disabled */
-#else
-#  if defined(_MSC_VER) && (defined(_M_X64) || defined(_M_I86)) && !defined(_M_ARM64EC)  /* _mm_prefetch() is not defined outside of x86/x64 */
-#    include <mmintrin.h>   /* https://msdn.microsoft.com/fr-fr/library/84szxsww(v=vs.90).aspx */
-#    define PREFETCH_L1(ptr)  _mm_prefetch((const char*)(ptr), _MM_HINT_T0)
-#    define PREFETCH_L2(ptr)  _mm_prefetch((const char*)(ptr), _MM_HINT_T1)
-#  elif defined(__GNUC__) && ( (__GNUC__ >= 4) || ( (__GNUC__ == 3) && (__GNUC_MINOR__ >= 1) ) )
-#    define PREFETCH_L1(ptr)  __builtin_prefetch((ptr), 0 /* rw==read */, 3 /* locality */)
-#    define PREFETCH_L2(ptr)  __builtin_prefetch((ptr), 0 /* rw==read */, 2 /* locality */)
-#  elif defined(__aarch64__)
-#    define PREFETCH_L1(ptr)  do { __asm__ __volatile__("prfm pldl1keep, %0" ::"Q"(*(ptr))); } while (0)
-#    define PREFETCH_L2(ptr)  do { __asm__ __volatile__("prfm pldl2keep, %0" ::"Q"(*(ptr))); } while (0)
-#  else
-#    define PREFETCH_L1(ptr) do { (void)(ptr); } while (0)  /* disabled */
-#    define PREFETCH_L2(ptr) do { (void)(ptr); } while (0)  /* disabled */
-#  endif
-#endif  /* NO_PREFETCH */
-
-#define CACHELINE_SIZE 64
-
-#define PREFETCH_AREA(p, s)                              \
-    do {                                                 \
-        const char* const _ptr = (const char*)(p);       \
-        size_t const _size = (size_t)(s);                \
-        size_t _pos;                                     \
-        for (_pos=0; _pos<_size; _pos+=CACHELINE_SIZE) { \
-            PREFETCH_L2(_ptr + _pos);                    \
-        }                                                \
-    } while (0)
-
-/* vectorization
- * older GCC (pre gcc-4.3 picked as the cutoff) uses a different syntax,
- * and some compilers, like Intel ICC and MCST LCC, do not support it at all. */
-#if !defined(__INTEL_COMPILER) && !defined(__clang__) && defined(__GNUC__) && !defined(__LCC__)
-#  if (__GNUC__ == 4 && __GNUC_MINOR__ > 3) || (__GNUC__ >= 5)
-#    define DONT_VECTORIZE __attribute__((optimize("no-tree-vectorize")))
-#  else
-#    define DONT_VECTORIZE _Pragma("GCC optimize(\"no-tree-vectorize\")")
-#  endif
-#else
-#  define DONT_VECTORIZE
-#endif
-
-/* Tell the compiler that a branch is likely or unlikely.
- * Only use these macros if it causes the compiler to generate better code.
- * If you can remove a LIKELY/UNLIKELY annotation without speed changes in gcc
- * and clang, please do.
- */
-#if defined(__GNUC__)
-#define LIKELY(x) (__builtin_expect((x), 1))
-#define UNLIKELY(x) (__builtin_expect((x), 0))
-#else
-#define LIKELY(x) (x)
-#define UNLIKELY(x) (x)
-#endif
-
-#if __has_builtin(__builtin_unreachable) || (defined(__GNUC__) && (__GNUC__ > 4 || (__GNUC__ == 4 && __GNUC_MINOR__ >= 5)))
-#  define ZSTD_UNREACHABLE do { assert(0), __builtin_unreachable(); } while (0)
-#else
-#  define ZSTD_UNREACHABLE do { assert(0); } while (0)
-#endif
-
-/* disable warnings */
-#ifdef _MSC_VER    /* Visual Studio */
-#  include <intrin.h>                    /* For Visual 2005 */
-#  pragma warning(disable : 4100)        /* disable: C4100: unreferenced formal parameter */
-#  pragma warning(disable : 4127)        /* disable: C4127: conditional expression is constant */
-#  pragma warning(disable : 4204)        /* disable: C4204: non-constant aggregate initializer */
-#  pragma warning(disable : 4214)        /* disable: C4214: non-int bitfields */
-#  pragma warning(disable : 4324)        /* disable: C4324: padded structure */
-#endif
-
-/* compile time determination of SIMD support */
-#if !defined(ZSTD_NO_INTRINSICS)
-#  if defined(__AVX2__)
-#    define ZSTD_ARCH_X86_AVX2
-#  endif
-#  if defined(__SSE2__) || defined(_M_X64) || (defined (_M_IX86) && defined(_M_IX86_FP) && (_M_IX86_FP >= 2))
-#    define ZSTD_ARCH_X86_SSE2
-#  endif
-#  if defined(__ARM_NEON) || defined(_M_ARM64)
-#    define ZSTD_ARCH_ARM_NEON
-#  endif
-#
-#  if defined(ZSTD_ARCH_X86_AVX2)
-#    include <immintrin.h>
-#  endif
-#  if defined(ZSTD_ARCH_X86_SSE2)
-#    include <emmintrin.h>
-#  elif defined(ZSTD_ARCH_ARM_NEON)
-#    include <arm_neon.h>
-#  endif
-#endif
-
-/* C-language Attributes are added in C23. */
-#if defined(__STDC_VERSION__) && (__STDC_VERSION__ > 201710L) && defined(__has_c_attribute)
-# define ZSTD_HAS_C_ATTRIBUTE(x) __has_c_attribute(x)
-#else
-# define ZSTD_HAS_C_ATTRIBUTE(x) 0
-#endif
-
-/* Only use C++ attributes in C++. Some compilers report support for C++
- * attributes when compiling with C.
- */
-#if defined(__cplusplus) && defined(__has_cpp_attribute)
-# define ZSTD_HAS_CPP_ATTRIBUTE(x) __has_cpp_attribute(x)
-#else
-# define ZSTD_HAS_CPP_ATTRIBUTE(x) 0
-#endif
-
-/* Define ZSTD_FALLTHROUGH macro for annotating switch case with the 'fallthrough' attribute.
- * - C23: https://en.cppreference.com/w/c/language/attributes/fallthrough
- * - CPP17: https://en.cppreference.com/w/cpp/language/attributes/fallthrough
- * - Else: __attribute__((__fallthrough__))
- */
-#ifndef ZSTD_FALLTHROUGH
-# if ZSTD_HAS_C_ATTRIBUTE(fallthrough)
-#  define ZSTD_FALLTHROUGH [[fallthrough]]
-# elif ZSTD_HAS_CPP_ATTRIBUTE(fallthrough)
-#  define ZSTD_FALLTHROUGH [[fallthrough]]
-# elif __has_attribute(__fallthrough__)
-/* Leading semicolon is to satisfy gcc-11 with -pedantic. Without the semicolon
- * gcc complains about: a label can only be part of a statement and a declaration is not a statement.
- */
-#  define ZSTD_FALLTHROUGH ; __attribute__((__fallthrough__))
-# else
-#  define ZSTD_FALLTHROUGH
-# endif
-#endif
-
-/*-**************************************************************
-*  Alignment
-*****************************************************************/
-
-/* @return 1 if @u is a 2^n value, 0 otherwise
- * useful to check a value is valid for alignment restrictions */
-MEM_STATIC int ZSTD_isPower2(size_t u) {
-    return (u & (u-1)) == 0;
-}
-
-/* this test was initially positioned in mem.h,
- * but this file is removed (or replaced) for linux kernel
- * so it's now hosted in compiler.h,
- * which remains valid for both user & kernel spaces.
- */
-
-#ifndef ZSTD_ALIGNOF
-# if defined(__GNUC__) || defined(_MSC_VER)
-/* covers gcc, clang & MSVC */
-/* note : this section must come first, before C11,
- * due to a limitation in the kernel source generator */
-#  define ZSTD_ALIGNOF(T) __alignof(T)
-
-# elif defined(__STDC_VERSION__) && (__STDC_VERSION__ >= 201112L)
-/* C11 support */
-#  include <stdalign.h>
-#  define ZSTD_ALIGNOF(T) alignof(T)
-
-# else
-/* No known support for alignof() - imperfect backup */
-#  define ZSTD_ALIGNOF(T) (sizeof(void*) < sizeof(T) ? sizeof(void*) : sizeof(T))
-
-# endif
-#endif /* ZSTD_ALIGNOF */
-
-#ifndef ZSTD_ALIGNED
-/* C90-compatible alignment macro (GCC/Clang). Adjust for other compilers if needed. */
-# if defined(__GNUC__) || defined(__clang__)
-#  define ZSTD_ALIGNED(a) __attribute__((aligned(a)))
-# elif defined(__STDC_VERSION__) && (__STDC_VERSION__ >= 201112L) /* C11 */
-#  define ZSTD_ALIGNED(a) _Alignas(a)
-#elif defined(_MSC_VER)
-#  define ZSTD_ALIGNED(n) __declspec(align(n))
-# else
-   /* this compiler will require its own alignment instruction */
-#  define ZSTD_ALIGNED(...)
-# endif
-#endif /* ZSTD_ALIGNED */
-
-
-/*-**************************************************************
-*  Sanitizer
-*****************************************************************/
-
-/**
- * Zstd relies on pointer overflow in its decompressor.
- * We add this attribute to functions that rely on pointer overflow.
- */
-#ifndef ZSTD_ALLOW_POINTER_OVERFLOW_ATTR
-#  if __has_attribute(no_sanitize)
-#    if !defined(__clang__) && defined(__GNUC__) && __GNUC__ < 8
-       /* gcc < 8 only has signed-integer-overlow which triggers on pointer overflow */
-#      define ZSTD_ALLOW_POINTER_OVERFLOW_ATTR __attribute__((no_sanitize("signed-integer-overflow")))
-#    else
-       /* older versions of clang [3.7, 5.0) will warn that pointer-overflow is ignored. */
-#      define ZSTD_ALLOW_POINTER_OVERFLOW_ATTR __attribute__((no_sanitize("pointer-overflow")))
-#    endif
-#  else
-#    define ZSTD_ALLOW_POINTER_OVERFLOW_ATTR
-#  endif
-#endif
-
-/**
- * Helper function to perform a wrapped pointer difference without triggering
- * UBSAN.
- *
- * @returns lhs - rhs with wrapping
- */
-MEM_STATIC
-ZSTD_ALLOW_POINTER_OVERFLOW_ATTR
-ptrdiff_t ZSTD_wrappedPtrDiff(unsigned char const* lhs, unsigned char const* rhs)
-{
-    return lhs - rhs;
-}
-
-/**
- * Helper function to perform a wrapped pointer add without triggering UBSAN.
- *
- * @return ptr + add with wrapping
- */
-MEM_STATIC
-ZSTD_ALLOW_POINTER_OVERFLOW_ATTR
-unsigned char const* ZSTD_wrappedPtrAdd(unsigned char const* ptr, ptrdiff_t add)
-{
-    return ptr + add;
-}
-
-/**
- * Helper function to perform a wrapped pointer subtraction without triggering
- * UBSAN.
- *
- * @return ptr - sub with wrapping
- */
-MEM_STATIC
-ZSTD_ALLOW_POINTER_OVERFLOW_ATTR
-unsigned char const* ZSTD_wrappedPtrSub(unsigned char const* ptr, ptrdiff_t sub)
-{
-    return ptr - sub;
-}
-
-/**
- * Helper function to add to a pointer that works around C's undefined behavior
- * of adding 0 to NULL.
- *
- * @returns `ptr + add` except it defines `NULL + 0 == NULL`.
- */
-MEM_STATIC
-unsigned char* ZSTD_maybeNullPtrAdd(unsigned char* ptr, ptrdiff_t add)
-{
-    return add > 0 ? ptr + add : ptr;
-}
-
-/* Issue #3240 reports an ASAN failure on an llvm-mingw build. Out of an
- * abundance of caution, disable our custom poisoning on mingw. */
-#ifdef __MINGW32__
-#ifndef ZSTD_ASAN_DONT_POISON_WORKSPACE
-#define ZSTD_ASAN_DONT_POISON_WORKSPACE 1
-#endif
-#ifndef ZSTD_MSAN_DONT_POISON_WORKSPACE
-#define ZSTD_MSAN_DONT_POISON_WORKSPACE 1
-#endif
-#endif
-
-#if ZSTD_MEMORY_SANITIZER && !defined(ZSTD_MSAN_DONT_POISON_WORKSPACE)
-/* Not all platforms that support msan provide sanitizers/msan_interface.h.
- * We therefore declare the functions we need ourselves, rather than trying to
- * include the header file... */
-#include <stddef.h>  /* size_t */
-#define ZSTD_DEPS_NEED_STDINT
-#include "zstd_deps.h"  /* intptr_t */
-
-/* Make memory region fully initialized (without changing its contents). */
-void __msan_unpoison(const volatile void *a, size_t size);
-
-/* Make memory region fully uninitialized (without changing its contents).
-   This is a legacy interface that does not update origin information. Use
-   __msan_allocated_memory() instead. */
-void __msan_poison(const volatile void *a, size_t size);
-
-/* Returns the offset of the first (at least partially) poisoned byte in the
-   memory range, or -1 if the whole range is good. */
-intptr_t __msan_test_shadow(const volatile void *x, size_t size);
-
-/* Print shadow and origin for the memory range to stderr in a human-readable
-   format. */
-void __msan_print_shadow(const volatile void *x, size_t size);
-#endif
-
-#if ZSTD_ADDRESS_SANITIZER && !defined(ZSTD_ASAN_DONT_POISON_WORKSPACE)
-/* Not all platforms that support asan provide sanitizers/asan_interface.h.
- * We therefore declare the functions we need ourselves, rather than trying to
- * include the header file... */
-#include <stddef.h>  /* size_t */
-
-/**
- * Marks a memory region (<c>[addr, addr+size)</c>) as unaddressable.
- *
- * This memory must be previously allocated by your program. Instrumented
- * code is forbidden from accessing addresses in this region until it is
- * unpoisoned. This function is not guaranteed to poison the entire region -
- * it could poison only a subregion of <c>[addr, addr+size)</c> due to ASan
- * alignment restrictions.
- *
- * \note This function is not thread-safe because no two threads can poison or
- * unpoison memory in the same memory region simultaneously.
- *
- * \param addr Start of memory region.
- * \param size Size of memory region. */
-void __asan_poison_memory_region(void const volatile *addr, size_t size);
-
-/**
- * Marks a memory region (<c>[addr, addr+size)</c>) as addressable.
- *
- * This memory must be previously allocated by your program. Accessing
- * addresses in this region is allowed until this region is poisoned again.
- * This function could unpoison a super-region of <c>[addr, addr+size)</c> due
- * to ASan alignment restrictions.
- *
- * \note This function is not thread-safe because no two threads can
- * poison or unpoison memory in the same memory region simultaneously.
- *
- * \param addr Start of memory region.
- * \param size Size of memory region. */
-void __asan_unpoison_memory_region(void const volatile *addr, size_t size);
-#endif
-
-#endif /* ZSTD_COMPILER_H */
-
-#endif /* USE_EXTERNAL_ZSTD */

vendor/github.com/DataDog/zstd/cover.h

@@ -1,155 +0,0 @@
-#ifndef USE_EXTERNAL_ZSTD
-/*
- * Copyright (c) Meta Platforms, Inc. and affiliates.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- * You may select, at your option, one of the above-listed licenses.
- */
-
-#ifndef ZDICT_STATIC_LINKING_ONLY
-#  define ZDICT_STATIC_LINKING_ONLY
-#endif
-
-#include "threading.h" /* ZSTD_pthread_mutex_t */
-#include "mem.h"   /* U32, BYTE */
-#include "zdict.h"
-
-/**
- * COVER_best_t is used for two purposes:
- * 1. Synchronizing threads.
- * 2. Saving the best parameters and dictionary.
- *
- * All of the methods except COVER_best_init() are thread safe if zstd is
- * compiled with multithreaded support.
- */
-typedef struct COVER_best_s {
-  ZSTD_pthread_mutex_t mutex;
-  ZSTD_pthread_cond_t cond;
-  size_t liveJobs;
-  void *dict;
-  size_t dictSize;
-  ZDICT_cover_params_t parameters;
-  size_t compressedSize;
-} COVER_best_t;
-
-/**
- * A segment is a range in the source as well as the score of the segment.
- */
-typedef struct {
-  U32 begin;
-  U32 end;
-  U32 score;
-} COVER_segment_t;
-
-/**
- *Number of epochs and size of each epoch.
- */
-typedef struct {
-  U32 num;
-  U32 size;
-} COVER_epoch_info_t;
-
-/**
- * Struct used for the dictionary selection function.
- */
-typedef struct COVER_dictSelection {
-  BYTE* dictContent;
-  size_t dictSize;
-  size_t totalCompressedSize;
-} COVER_dictSelection_t;
-
-/**
- * Computes the number of epochs and the size of each epoch.
- * We will make sure that each epoch gets at least 10 * k bytes.
- *
- * The COVER algorithms divide the data up into epochs of equal size and
- * select one segment from each epoch.
- *
- * @param maxDictSize The maximum allowed dictionary size.
- * @param nbDmers     The number of dmers we are training on.
- * @param k           The parameter k (segment size).
- * @param passes      The target number of passes over the dmer corpus.
- *                    More passes means a better dictionary.
- */
-COVER_epoch_info_t COVER_computeEpochs(U32 maxDictSize, U32 nbDmers,
-                                       U32 k, U32 passes);
-
-/**
- * Warns the user when their corpus is too small.
- */
-void COVER_warnOnSmallCorpus(size_t maxDictSize, size_t nbDmers, int displayLevel);
-
-/**
- *  Checks total compressed size of a dictionary
- */
-size_t COVER_checkTotalCompressedSize(const ZDICT_cover_params_t parameters,
-                                      const size_t *samplesSizes, const BYTE *samples,
-                                      size_t *offsets,
-                                      size_t nbTrainSamples, size_t nbSamples,
-                                      BYTE *const dict, size_t dictBufferCapacity);
-
-/**
- * Returns the sum of the sample sizes.
- */
-size_t COVER_sum(const size_t *samplesSizes, unsigned nbSamples) ;
-
-/**
- * Initialize the `COVER_best_t`.
- */
-void COVER_best_init(COVER_best_t *best);
-
-/**
- * Wait until liveJobs == 0.
- */
-void COVER_best_wait(COVER_best_t *best);
-
-/**
- * Call COVER_best_wait() and then destroy the COVER_best_t.
- */
-void COVER_best_destroy(COVER_best_t *best);
-
-/**
- * Called when a thread is about to be launched.
- * Increments liveJobs.
- */
-void COVER_best_start(COVER_best_t *best);
-
-/**
- * Called when a thread finishes executing, both on error or success.
- * Decrements liveJobs and signals any waiting threads if liveJobs == 0.
- * If this dictionary is the best so far save it and its parameters.
- */
-void COVER_best_finish(COVER_best_t *best, ZDICT_cover_params_t parameters,
-                       COVER_dictSelection_t selection);
-/**
- * Error function for COVER_selectDict function. Checks if the return
- * value is an error.
- */
-unsigned COVER_dictSelectionIsError(COVER_dictSelection_t selection);
-
- /**
-  * Error function for COVER_selectDict function. Returns a struct where
-  * return.totalCompressedSize is a ZSTD error.
-  */
-COVER_dictSelection_t COVER_dictSelectionError(size_t error);
-
-/**
- * Always call after selectDict is called to free up used memory from
- * newly created dictionary.
- */
-void COVER_dictSelectionFree(COVER_dictSelection_t selection);
-
-/**
- * Called to finalize the dictionary and select one based on whether or not
- * the shrink-dict flag was enabled. If enabled the dictionary used is the
- * smallest dictionary within a specified regression of the compressed size
- * from the largest dictionary.
- */
- COVER_dictSelection_t COVER_selectDict(BYTE* customDictContent, size_t dictBufferCapacity,
-                       size_t dictContentSize, const BYTE* samplesBuffer, const size_t* samplesSizes, unsigned nbFinalizeSamples,
-                       size_t nbCheckSamples, size_t nbSamples, ZDICT_cover_params_t params, size_t* offsets, size_t totalCompressedSize);
-
-#endif /* USE_EXTERNAL_ZSTD */

vendor/github.com/DataDog/zstd/cpu.h

@@ -1,252 +0,0 @@
-#ifndef USE_EXTERNAL_ZSTD
-/*
- * Copyright (c) Meta Platforms, Inc. and affiliates.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- * You may select, at your option, one of the above-listed licenses.
- */
-
-#ifndef ZSTD_COMMON_CPU_H
-#define ZSTD_COMMON_CPU_H
-
-/**
- * Implementation taken from folly/CpuId.h
- * https://github.com/facebook/folly/blob/master/folly/CpuId.h
- */
-
-#include "mem.h"
-
-#ifdef _MSC_VER
-#include <intrin.h>
-#endif
-
-typedef struct {
-    U32 f1c;
-    U32 f1d;
-    U32 f7b;
-    U32 f7c;
-} ZSTD_cpuid_t;
-
-MEM_STATIC ZSTD_cpuid_t ZSTD_cpuid(void) {
-    U32 f1c = 0;
-    U32 f1d = 0;
-    U32 f7b = 0;
-    U32 f7c = 0;
-#if defined(_MSC_VER) && (defined(_M_X64) || defined(_M_IX86))
-#if !defined(_M_X64) || !defined(__clang__) || __clang_major__ >= 16
-    int reg[4];
-    __cpuid((int*)reg, 0);
-    {
-        int const n = reg[0];
-        if (n >= 1) {
-            __cpuid((int*)reg, 1);
-            f1c = (U32)reg[2];
-            f1d = (U32)reg[3];
-        }
-        if (n >= 7) {
-            __cpuidex((int*)reg, 7, 0);
-            f7b = (U32)reg[1];
-            f7c = (U32)reg[2];
-        }
-    }
-#else
-    /* Clang compiler has a bug (fixed in https://reviews.llvm.org/D101338) in
-     * which the `__cpuid` intrinsic does not save and restore `rbx` as it needs
-     * to due to being a reserved register. So in that case, do the `cpuid`
-     * ourselves. Clang supports inline assembly anyway.
-     */
-    U32 n;
-    __asm__(
-        "pushq %%rbx\n\t"
-        "cpuid\n\t"
-        "popq %%rbx\n\t"
-        : "=a"(n)
-        : "a"(0)
-        : "rcx", "rdx");
-    if (n >= 1) {
-      U32 f1a;
-      __asm__(
-          "pushq %%rbx\n\t"
-          "cpuid\n\t"
-          "popq %%rbx\n\t"
-          : "=a"(f1a), "=c"(f1c), "=d"(f1d)
-          : "a"(1)
-          :);
-    }
-    if (n >= 7) {
-      __asm__(
-          "pushq %%rbx\n\t"
-          "cpuid\n\t"
-          "movq %%rbx, %%rax\n\t"
-          "popq %%rbx"
-          : "=a"(f7b), "=c"(f7c)
-          : "a"(7), "c"(0)
-          : "rdx");
-    }
-#endif
-#elif defined(__i386__) && defined(__PIC__) && !defined(__clang__) && defined(__GNUC__)
-    /* The following block like the normal cpuid branch below, but gcc
-     * reserves ebx for use of its pic register so we must specially
-     * handle the save and restore to avoid clobbering the register
-     */
-    U32 n;
-    __asm__(
-        "pushl %%ebx\n\t"
-        "cpuid\n\t"
-        "popl %%ebx\n\t"
-        : "=a"(n)
-        : "a"(0)
-        : "ecx", "edx");
-    if (n >= 1) {
-      U32 f1a;
-      __asm__(
-          "pushl %%ebx\n\t"
-          "cpuid\n\t"
-          "popl %%ebx\n\t"
-          : "=a"(f1a), "=c"(f1c), "=d"(f1d)
-          : "a"(1));
-    }
-    if (n >= 7) {
-      __asm__(
-          "pushl %%ebx\n\t"
-          "cpuid\n\t"
-          "movl %%ebx, %%eax\n\t"
-          "popl %%ebx"
-          : "=a"(f7b), "=c"(f7c)
-          : "a"(7), "c"(0)
-          : "edx");
-    }
-#elif defined(__x86_64__) || defined(_M_X64) || defined(__i386__)
-    U32 n;
-    __asm__("cpuid" : "=a"(n) : "a"(0) : "ebx", "ecx", "edx");
-    if (n >= 1) {
-      U32 f1a;
-      __asm__("cpuid" : "=a"(f1a), "=c"(f1c), "=d"(f1d) : "a"(1) : "ebx");
-    }
-    if (n >= 7) {
-      U32 f7a;
-      __asm__("cpuid"
-              : "=a"(f7a), "=b"(f7b), "=c"(f7c)
-              : "a"(7), "c"(0)
-              : "edx");
-    }
-#endif
-    {
-        ZSTD_cpuid_t cpuid;
-        cpuid.f1c = f1c;
-        cpuid.f1d = f1d;
-        cpuid.f7b = f7b;
-        cpuid.f7c = f7c;
-        return cpuid;
-    }
-}
-
-#define X(name, r, bit)                                                        \
-  MEM_STATIC int ZSTD_cpuid_##name(ZSTD_cpuid_t const cpuid) {                 \
-    return ((cpuid.r) & (1U << bit)) != 0;                                     \
-  }
-
-/* cpuid(1): Processor Info and Feature Bits. */
-#define C(name, bit) X(name, f1c, bit)
-  C(sse3, 0)
-  C(pclmuldq, 1)
-  C(dtes64, 2)
-  C(monitor, 3)
-  C(dscpl, 4)
-  C(vmx, 5)
-  C(smx, 6)
-  C(eist, 7)
-  C(tm2, 8)
-  C(ssse3, 9)
-  C(cnxtid, 10)
-  C(fma, 12)
-  C(cx16, 13)
-  C(xtpr, 14)
-  C(pdcm, 15)
-  C(pcid, 17)
-  C(dca, 18)
-  C(sse41, 19)
-  C(sse42, 20)
-  C(x2apic, 21)
-  C(movbe, 22)
-  C(popcnt, 23)
-  C(tscdeadline, 24)
-  C(aes, 25)
-  C(xsave, 26)
-  C(osxsave, 27)
-  C(avx, 28)
-  C(f16c, 29)
-  C(rdrand, 30)
-#undef C
-#define D(name, bit) X(name, f1d, bit)
-  D(fpu, 0)
-  D(vme, 1)
-  D(de, 2)
-  D(pse, 3)
-  D(tsc, 4)
-  D(msr, 5)
-  D(pae, 6)
-  D(mce, 7)
-  D(cx8, 8)
-  D(apic, 9)
-  D(sep, 11)
-  D(mtrr, 12)
-  D(pge, 13)
-  D(mca, 14)
-  D(cmov, 15)
-  D(pat, 16)
-  D(pse36, 17)
-  D(psn, 18)
-  D(clfsh, 19)
-  D(ds, 21)
-  D(acpi, 22)
-  D(mmx, 23)
-  D(fxsr, 24)
-  D(sse, 25)
-  D(sse2, 26)
-  D(ss, 27)
-  D(htt, 28)
-  D(tm, 29)
-  D(pbe, 31)
-#undef D
-
-/* cpuid(7): Extended Features. */
-#define B(name, bit) X(name, f7b, bit)
-  B(bmi1, 3)
-  B(hle, 4)
-  B(avx2, 5)
-  B(smep, 7)
-  B(bmi2, 8)
-  B(erms, 9)
-  B(invpcid, 10)
-  B(rtm, 11)
-  B(mpx, 14)
-  B(avx512f, 16)
-  B(avx512dq, 17)
-  B(rdseed, 18)
-  B(adx, 19)
-  B(smap, 20)
-  B(avx512ifma, 21)
-  B(pcommit, 22)
-  B(clflushopt, 23)
-  B(clwb, 24)
-  B(avx512pf, 26)
-  B(avx512er, 27)
-  B(avx512cd, 28)
-  B(sha, 29)
-  B(avx512bw, 30)
-  B(avx512vl, 31)
-#undef B
-#define C(name, bit) X(name, f7c, bit)
-  C(prefetchwt1, 0)
-  C(avx512vbmi, 1)
-#undef C
-
-#undef X
-
-#endif /* ZSTD_COMMON_CPU_H */
-
-#endif /* USE_EXTERNAL_ZSTD */

vendor/github.com/DataDog/zstd/debug.c

@@ -1,33 +0,0 @@
-#ifndef USE_EXTERNAL_ZSTD
-/* ******************************************************************
- * debug
- * Part of FSE library
- * Copyright (c) Meta Platforms, Inc. and affiliates.
- *
- * You can contact the author at :
- * - Source repository : https://github.com/Cyan4973/FiniteStateEntropy
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- * You may select, at your option, one of the above-listed licenses.
-****************************************************************** */
-
-
-/*
- * This module only hosts one global variable
- * which can be used to dynamically influence the verbosity of traces,
- * such as DEBUGLOG and RAWLOG
- */
-
-#include "debug.h"
-
-#if !defined(ZSTD_LINUX_KERNEL) || (DEBUGLEVEL>=2)
-/* We only use this when DEBUGLEVEL>=2, but we get -Werror=pedantic errors if a
- * translation unit is empty. So remove this from Linux kernel builds, but
- * otherwise just leave it in.
- */
-int g_debuglevel = DEBUGLEVEL;
-#endif
-
-#endif /* USE_EXTERNAL_ZSTD */

vendor/github.com/DataDog/zstd/debug.h

@@ -1,110 +0,0 @@
-#ifndef USE_EXTERNAL_ZSTD
-/* ******************************************************************
- * debug
- * Part of FSE library
- * Copyright (c) Meta Platforms, Inc. and affiliates.
- *
- * You can contact the author at :
- * - Source repository : https://github.com/Cyan4973/FiniteStateEntropy
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- * You may select, at your option, one of the above-listed licenses.
-****************************************************************** */
-
-
-/*
- * The purpose of this header is to enable debug functions.
- * They regroup assert(), DEBUGLOG() and RAWLOG() for run-time,
- * and DEBUG_STATIC_ASSERT() for compile-time.
- *
- * By default, DEBUGLEVEL==0, which means run-time debug is disabled.
- *
- * Level 1 enables assert() only.
- * Starting level 2, traces can be generated and pushed to stderr.
- * The higher the level, the more verbose the traces.
- *
- * It's possible to dynamically adjust level using variable g_debug_level,
- * which is only declared if DEBUGLEVEL>=2,
- * and is a global variable, not multi-thread protected (use with care)
- */
-
-#ifndef DEBUG_H_12987983217
-#define DEBUG_H_12987983217
-
-
-/* static assert is triggered at compile time, leaving no runtime artefact.
- * static assert only works with compile-time constants.
- * Also, this variant can only be used inside a function. */
-#define DEBUG_STATIC_ASSERT(c) (void)sizeof(char[(c) ? 1 : -1])
-
-
-/* DEBUGLEVEL is expected to be defined externally,
- * typically through compiler command line.
- * Value must be a number. */
-#ifndef DEBUGLEVEL
-#  define DEBUGLEVEL 0
-#endif
-
-
-/* recommended values for DEBUGLEVEL :
- * 0 : release mode, no debug, all run-time checks disabled
- * 1 : enables assert() only, no display
- * 2 : reserved, for currently active debug path
- * 3 : events once per object lifetime (CCtx, CDict, etc.)
- * 4 : events once per frame
- * 5 : events once per block
- * 6 : events once per sequence (verbose)
- * 7+: events at every position (*very* verbose)
- *
- * It's generally inconvenient to output traces > 5.
- * In which case, it's possible to selectively trigger high verbosity levels
- * by modifying g_debug_level.
- */
-
-#if (DEBUGLEVEL>=1)
-#  define ZSTD_DEPS_NEED_ASSERT
-#  include "zstd_deps.h"
-#else
-#  ifndef assert   /* assert may be already defined, due to prior #include <assert.h> */
-#    define assert(condition) ((void)0)   /* disable assert (default) */
-#  endif
-#endif
-
-#if (DEBUGLEVEL>=2)
-#  define ZSTD_DEPS_NEED_IO
-#  include "zstd_deps.h"
-extern int g_debuglevel; /* the variable is only declared,
-                            it actually lives in debug.c,
-                            and is shared by the whole process.
-                            It's not thread-safe.
-                            It's useful when enabling very verbose levels
-                            on selective conditions (such as position in src) */
-
-#  define RAWLOG(l, ...)                   \
-    do {                                   \
-        if (l<=g_debuglevel) {             \
-            ZSTD_DEBUG_PRINT(__VA_ARGS__); \
-        }                                  \
-    } while (0)
-
-#define STRINGIFY(x) #x
-#define TOSTRING(x) STRINGIFY(x)
-#define LINE_AS_STRING TOSTRING(__LINE__)
-
-#  define DEBUGLOG(l, ...)                               \
-    do {                                                 \
-        if (l<=g_debuglevel) {                           \
-            ZSTD_DEBUG_PRINT(__FILE__ ":" LINE_AS_STRING ": " __VA_ARGS__); \
-            ZSTD_DEBUG_PRINT(" \n");                     \
-        }                                                \
-    } while (0)
-#else
-#  define RAWLOG(l, ...)   do { } while (0)    /* disabled */
-#  define DEBUGLOG(l, ...) do { } while (0)    /* disabled */
-#endif
-
-#endif /* DEBUG_H_12987983217 */
-
-#endif /* USE_EXTERNAL_ZSTD */

vendor/github.com/DataDog/zstd/divsufsort.h

@@ -1,60 +0,0 @@
-#ifndef USE_EXTERNAL_ZSTD
-/*
- * divsufsort.h for libdivsufsort-lite
- * Copyright (c) 2003-2008 Yuta Mori All Rights Reserved.
- *
- * Permission is hereby granted, free of charge, to any person
- * obtaining a copy of this software and associated documentation
- * files (the "Software"), to deal in the Software without
- * restriction, including without limitation the rights to use,
- * copy, modify, merge, publish, distribute, sublicense, and/or sell
- * copies of the Software, and to permit persons to whom the
- * Software is furnished to do so, subject to the following
- * conditions:
- *
- * The above copyright notice and this permission notice shall be
- * included in all copies or substantial portions of the Software.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
- * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
- * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
- * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
- * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
- * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
- * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
- * OTHER DEALINGS IN THE SOFTWARE.
- */
-
-#ifndef _DIVSUFSORT_H
-#define _DIVSUFSORT_H 1
-
-/*- Prototypes -*/
-
-/**
- * Constructs the suffix array of a given string.
- * @param T [0..n-1] The input string.
- * @param SA [0..n-1] The output array of suffixes.
- * @param n The length of the given string.
- * @param openMP enables OpenMP optimization.
- * @return 0 if no error occurred, -1 or -2 otherwise.
- */
-int
-divsufsort(const unsigned char *T, int *SA, int n, int openMP);
-
-/**
- * Constructs the burrows-wheeler transformed string of a given string.
- * @param T [0..n-1] The input string.
- * @param U [0..n-1] The output string. (can be T)
- * @param A [0..n-1] The temporary array. (can be NULL)
- * @param n The length of the given string.
- * @param num_indexes The length of secondary indexes array. (can be NULL)
- * @param indexes The secondary indexes array. (can be NULL)
- * @param openMP enables OpenMP optimization.
- * @return The primary index if no error occurred, -1 or -2 otherwise.
- */
-int
-divbwt(const unsigned char *T, unsigned char *U, int *A, int n, unsigned char * num_indexes, int * indexes, int openMP);
-
-#endif /* _DIVSUFSORT_H */
-
-#endif /* USE_EXTERNAL_ZSTD */

vendor/github.com/DataDog/zstd/entropy_common.c

@@ -1,343 +0,0 @@
-#ifndef USE_EXTERNAL_ZSTD
-/* ******************************************************************
- * Common functions of New Generation Entropy library
- * Copyright (c) Meta Platforms, Inc. and affiliates.
- *
- *  You can contact the author at :
- *  - FSE+HUF source repository : https://github.com/Cyan4973/FiniteStateEntropy
- *  - Public forum : https://groups.google.com/forum/#!forum/lz4c
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- * You may select, at your option, one of the above-listed licenses.
-****************************************************************** */
-
-/* *************************************
-*  Dependencies
-***************************************/
-#include "mem.h"
-#include "error_private.h"       /* ERR_*, ERROR */
-#define FSE_STATIC_LINKING_ONLY  /* FSE_MIN_TABLELOG */
-#include "fse.h"
-#include "huf.h"
-#include "bits.h"                /* ZSDT_highbit32, ZSTD_countTrailingZeros32 */
-
-
-/*===   Version   ===*/
-unsigned FSE_versionNumber(void) { return FSE_VERSION_NUMBER; }
-
-
-/*===   Error Management   ===*/
-unsigned FSE_isError(size_t code) { return ERR_isError(code); }
-const char* FSE_getErrorName(size_t code) { return ERR_getErrorName(code); }
-
-unsigned HUF_isError(size_t code) { return ERR_isError(code); }
-const char* HUF_getErrorName(size_t code) { return ERR_getErrorName(code); }
-
-
-/*-**************************************************************
-*  FSE NCount encoding-decoding
-****************************************************************/
-FORCE_INLINE_TEMPLATE
-size_t FSE_readNCount_body(short* normalizedCounter, unsigned* maxSVPtr, unsigned* tableLogPtr,
-                           const void* headerBuffer, size_t hbSize)
-{
-    const BYTE* const istart = (const BYTE*) headerBuffer;
-    const BYTE* const iend = istart + hbSize;
-    const BYTE* ip = istart;
-    int nbBits;
-    int remaining;
-    int threshold;
-    U32 bitStream;
-    int bitCount;
-    unsigned charnum = 0;
-    unsigned const maxSV1 = *maxSVPtr + 1;
-    int previous0 = 0;
-
-    if (hbSize < 8) {
-        /* This function only works when hbSize >= 8 */
-        char buffer[8] = {0};
-        ZSTD_memcpy(buffer, headerBuffer, hbSize);
-        {   size_t const countSize = FSE_readNCount(normalizedCounter, maxSVPtr, tableLogPtr,
-                                                    buffer, sizeof(buffer));
-            if (FSE_isError(countSize)) return countSize;
-            if (countSize > hbSize) return ERROR(corruption_detected);
-            return countSize;
-    }   }
-    assert(hbSize >= 8);
-
-    /* init */
-    ZSTD_memset(normalizedCounter, 0, (*maxSVPtr+1) * sizeof(normalizedCounter[0]));   /* all symbols not present in NCount have a frequency of 0 */
-    bitStream = MEM_readLE32(ip);
-    nbBits = (bitStream & 0xF) + FSE_MIN_TABLELOG;   /* extract tableLog */
-    if (nbBits > FSE_TABLELOG_ABSOLUTE_MAX) return ERROR(tableLog_tooLarge);
-    bitStream >>= 4;
-    bitCount = 4;
-    *tableLogPtr = nbBits;
-    remaining = (1<<nbBits)+1;
-    threshold = 1<<nbBits;
-    nbBits++;
-
-    for (;;) {
-        if (previous0) {
-            /* Count the number of repeats. Each time the
-             * 2-bit repeat code is 0b11 there is another
-             * repeat.
-             * Avoid UB by setting the high bit to 1.
-             */
-            int repeats = ZSTD_countTrailingZeros32(~bitStream | 0x80000000) >> 1;
-            while (repeats >= 12) {
-                charnum += 3 * 12;
-                if (LIKELY(ip <= iend-7)) {
-                    ip += 3;
-                } else {
-                    bitCount -= (int)(8 * (iend - 7 - ip));
-                    bitCount &= 31;
-                    ip = iend - 4;
-                }
-                bitStream = MEM_readLE32(ip) >> bitCount;
-                repeats = ZSTD_countTrailingZeros32(~bitStream | 0x80000000) >> 1;
-            }
-            charnum += 3 * repeats;
-            bitStream >>= 2 * repeats;
-            bitCount += 2 * repeats;
-
-            /* Add the final repeat which isn't 0b11. */
-            assert((bitStream & 3) < 3);
-            charnum += bitStream & 3;
-            bitCount += 2;
-
-            /* This is an error, but break and return an error
-             * at the end, because returning out of a loop makes
-             * it harder for the compiler to optimize.
-             */
-            if (charnum >= maxSV1) break;
-
-            /* We don't need to set the normalized count to 0
-             * because we already memset the whole buffer to 0.
-             */
-
-            if (LIKELY(ip <= iend-7) || (ip + (bitCount>>3) <= iend-4)) {
-                assert((bitCount >> 3) <= 3); /* For first condition to work */
-                ip += bitCount>>3;
-                bitCount &= 7;
-            } else {
-                bitCount -= (int)(8 * (iend - 4 - ip));
-                bitCount &= 31;
-                ip = iend - 4;
-            }
-            bitStream = MEM_readLE32(ip) >> bitCount;
-        }
-        {
-            int const max = (2*threshold-1) - remaining;
-            int count;
-
-            if ((bitStream & (threshold-1)) < (U32)max) {
-                count = bitStream & (threshold-1);
-                bitCount += nbBits-1;
-            } else {
-                count = bitStream & (2*threshold-1);
-                if (count >= threshold) count -= max;
-                bitCount += nbBits;
-            }
-
-            count--;   /* extra accuracy */
-            /* When it matters (small blocks), this is a
-             * predictable branch, because we don't use -1.
-             */
-            if (count >= 0) {
-                remaining -= count;
-            } else {
-                assert(count == -1);
-                remaining += count;
-            }
-            normalizedCounter[charnum++] = (short)count;
-            previous0 = !count;
-
-            assert(threshold > 1);
-            if (remaining < threshold) {
-                /* This branch can be folded into the
-                 * threshold update condition because we
-                 * know that threshold > 1.
-                 */
-                if (remaining <= 1) break;
-                nbBits = ZSTD_highbit32(remaining) + 1;
-                threshold = 1 << (nbBits - 1);
-            }
-            if (charnum >= maxSV1) break;
-
-            if (LIKELY(ip <= iend-7) || (ip + (bitCount>>3) <= iend-4)) {
-                ip += bitCount>>3;
-                bitCount &= 7;
-            } else {
-                bitCount -= (int)(8 * (iend - 4 - ip));
-                bitCount &= 31;
-                ip = iend - 4;
-            }
-            bitStream = MEM_readLE32(ip) >> bitCount;
-    }   }
-    if (remaining != 1) return ERROR(corruption_detected);
-    /* Only possible when there are too many zeros. */
-    if (charnum > maxSV1) return ERROR(maxSymbolValue_tooSmall);
-    if (bitCount > 32) return ERROR(corruption_detected);
-    *maxSVPtr = charnum-1;
-
-    ip += (bitCount+7)>>3;
-    return ip-istart;
-}
-
-/* Avoids the FORCE_INLINE of the _body() function. */
-static size_t FSE_readNCount_body_default(
-        short* normalizedCounter, unsigned* maxSVPtr, unsigned* tableLogPtr,
-        const void* headerBuffer, size_t hbSize)
-{
-    return FSE_readNCount_body(normalizedCounter, maxSVPtr, tableLogPtr, headerBuffer, hbSize);
-}
-
-#if DYNAMIC_BMI2
-BMI2_TARGET_ATTRIBUTE static size_t FSE_readNCount_body_bmi2(
-        short* normalizedCounter, unsigned* maxSVPtr, unsigned* tableLogPtr,
-        const void* headerBuffer, size_t hbSize)
-{
-    return FSE_readNCount_body(normalizedCounter, maxSVPtr, tableLogPtr, headerBuffer, hbSize);
-}
-#endif
-
-size_t FSE_readNCount_bmi2(
-        short* normalizedCounter, unsigned* maxSVPtr, unsigned* tableLogPtr,
-        const void* headerBuffer, size_t hbSize, int bmi2)
-{
-#if DYNAMIC_BMI2
-    if (bmi2) {
-        return FSE_readNCount_body_bmi2(normalizedCounter, maxSVPtr, tableLogPtr, headerBuffer, hbSize);
-    }
-#endif
-    (void)bmi2;
-    return FSE_readNCount_body_default(normalizedCounter, maxSVPtr, tableLogPtr, headerBuffer, hbSize);
-}
-
-size_t FSE_readNCount(
-        short* normalizedCounter, unsigned* maxSVPtr, unsigned* tableLogPtr,
-        const void* headerBuffer, size_t hbSize)
-{
-    return FSE_readNCount_bmi2(normalizedCounter, maxSVPtr, tableLogPtr, headerBuffer, hbSize, /* bmi2 */ 0);
-}
-
-
-/*! HUF_readStats() :
-    Read compact Huffman tree, saved by HUF_writeCTable().
-    `huffWeight` is destination buffer.
-    `rankStats` is assumed to be a table of at least HUF_TABLELOG_MAX U32.
-    @return : size read from `src` , or an error Code .
-    Note : Needed by HUF_readCTable() and HUF_readDTableX?() .
-*/
-size_t HUF_readStats(BYTE* huffWeight, size_t hwSize, U32* rankStats,
-                     U32* nbSymbolsPtr, U32* tableLogPtr,
-                     const void* src, size_t srcSize)
-{
-    U32 wksp[HUF_READ_STATS_WORKSPACE_SIZE_U32];
-    return HUF_readStats_wksp(huffWeight, hwSize, rankStats, nbSymbolsPtr, tableLogPtr, src, srcSize, wksp, sizeof(wksp), /* flags */ 0);
-}
-
-FORCE_INLINE_TEMPLATE size_t
-HUF_readStats_body(BYTE* huffWeight, size_t hwSize, U32* rankStats,
-                   U32* nbSymbolsPtr, U32* tableLogPtr,
-                   const void* src, size_t srcSize,
-                   void* workSpace, size_t wkspSize,
-                   int bmi2)
-{
-    U32 weightTotal;
-    const BYTE* ip = (const BYTE*) src;
-    size_t iSize;
-    size_t oSize;
-
-    if (!srcSize) return ERROR(srcSize_wrong);
-    iSize = ip[0];
-    /* ZSTD_memset(huffWeight, 0, hwSize);   *//* is not necessary, even though some analyzer complain ... */
-
-    if (iSize >= 128) {  /* special header */
-        oSize = iSize - 127;
-        iSize = ((oSize+1)/2);
-        if (iSize+1 > srcSize) return ERROR(srcSize_wrong);
-        if (oSize >= hwSize) return ERROR(corruption_detected);
-        ip += 1;
-        {   U32 n;
-            for (n=0; n<oSize; n+=2) {
-                huffWeight[n]   = ip[n/2] >> 4;
-                huffWeight[n+1] = ip[n/2] & 15;
-    }   }   }
-    else  {   /* header compressed with FSE (normal case) */
-        if (iSize+1 > srcSize) return ERROR(srcSize_wrong);
-        /* max (hwSize-1) values decoded, as last one is implied */
-        oSize = FSE_decompress_wksp_bmi2(huffWeight, hwSize-1, ip+1, iSize, 6, workSpace, wkspSize, bmi2);
-        if (FSE_isError(oSize)) return oSize;
-    }
-
-    /* collect weight stats */
-    ZSTD_memset(rankStats, 0, (HUF_TABLELOG_MAX + 1) * sizeof(U32));
-    weightTotal = 0;
-    {   U32 n; for (n=0; n<oSize; n++) {
-            if (huffWeight[n] > HUF_TABLELOG_MAX) return ERROR(corruption_detected);
-            rankStats[huffWeight[n]]++;
-            weightTotal += (1 << huffWeight[n]) >> 1;
-    }   }
-    if (weightTotal == 0) return ERROR(corruption_detected);
-
-    /* get last non-null symbol weight (implied, total must be 2^n) */
-    {   U32 const tableLog = ZSTD_highbit32(weightTotal) + 1;
-        if (tableLog > HUF_TABLELOG_MAX) return ERROR(corruption_detected);
-        *tableLogPtr = tableLog;
-        /* determine last weight */
-        {   U32 const total = 1 << tableLog;
-            U32 const rest = total - weightTotal;
-            U32 const verif = 1 << ZSTD_highbit32(rest);
-            U32 const lastWeight = ZSTD_highbit32(rest) + 1;
-            if (verif != rest) return ERROR(corruption_detected);    /* last value must be a clean power of 2 */
-            huffWeight[oSize] = (BYTE)lastWeight;
-            rankStats[lastWeight]++;
-    }   }
-
-    /* check tree construction validity */
-    if ((rankStats[1] < 2) || (rankStats[1] & 1)) return ERROR(corruption_detected);   /* by construction : at least 2 elts of rank 1, must be even */
-
-    /* results */
-    *nbSymbolsPtr = (U32)(oSize+1);
-    return iSize+1;
-}
-
-/* Avoids the FORCE_INLINE of the _body() function. */
-static size_t HUF_readStats_body_default(BYTE* huffWeight, size_t hwSize, U32* rankStats,
-                     U32* nbSymbolsPtr, U32* tableLogPtr,
-                     const void* src, size_t srcSize,
-                     void* workSpace, size_t wkspSize)
-{
-    return HUF_readStats_body(huffWeight, hwSize, rankStats, nbSymbolsPtr, tableLogPtr, src, srcSize, workSpace, wkspSize, 0);
-}
-
-#if DYNAMIC_BMI2
-static BMI2_TARGET_ATTRIBUTE size_t HUF_readStats_body_bmi2(BYTE* huffWeight, size_t hwSize, U32* rankStats,
-                     U32* nbSymbolsPtr, U32* tableLogPtr,
-                     const void* src, size_t srcSize,
-                     void* workSpace, size_t wkspSize)
-{
-    return HUF_readStats_body(huffWeight, hwSize, rankStats, nbSymbolsPtr, tableLogPtr, src, srcSize, workSpace, wkspSize, 1);
-}
-#endif
-
-size_t HUF_readStats_wksp(BYTE* huffWeight, size_t hwSize, U32* rankStats,
-                     U32* nbSymbolsPtr, U32* tableLogPtr,
-                     const void* src, size_t srcSize,
-                     void* workSpace, size_t wkspSize,
-                     int flags)
-{
-#if DYNAMIC_BMI2
-    if (flags & HUF_flags_bmi2) {
-        return HUF_readStats_body_bmi2(huffWeight, hwSize, rankStats, nbSymbolsPtr, tableLogPtr, src, srcSize, workSpace, wkspSize);
-    }
-#endif
-    (void)flags;
-    return HUF_readStats_body_default(huffWeight, hwSize, rankStats, nbSymbolsPtr, tableLogPtr, src, srcSize, workSpace, wkspSize);
-}
-
-#endif /* USE_EXTERNAL_ZSTD */

vendor/github.com/DataDog/zstd/error_private.c

@@ -1,67 +0,0 @@
-#ifndef USE_EXTERNAL_ZSTD
-/*
- * Copyright (c) Meta Platforms, Inc. and affiliates.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- * You may select, at your option, one of the above-listed licenses.
- */
-
-/* The purpose of this file is to have a single list of error strings embedded in binary */
-
-#include "error_private.h"
-
-const char* ERR_getErrorString(ERR_enum code)
-{
-#ifdef ZSTD_STRIP_ERROR_STRINGS
-    (void)code;
-    return "Error strings stripped";
-#else
-    static const char* const notErrorCode = "Unspecified error code";
-    switch( code )
-    {
-    case PREFIX(no_error): return "No error detected";
-    case PREFIX(GENERIC):  return "Error (generic)";
-    case PREFIX(prefix_unknown): return "Unknown frame descriptor";
-    case PREFIX(version_unsupported): return "Version not supported";
-    case PREFIX(frameParameter_unsupported): return "Unsupported frame parameter";
-    case PREFIX(frameParameter_windowTooLarge): return "Frame requires too much memory for decoding";
-    case PREFIX(corruption_detected): return "Data corruption detected";
-    case PREFIX(checksum_wrong): return "Restored data doesn't match checksum";
-    case PREFIX(literals_headerWrong): return "Header of Literals' block doesn't respect format specification";
-    case PREFIX(parameter_unsupported): return "Unsupported parameter";
-    case PREFIX(parameter_combination_unsupported): return "Unsupported combination of parameters";
-    case PREFIX(parameter_outOfBound): return "Parameter is out of bound";
-    case PREFIX(init_missing): return "Context should be init first";
-    case PREFIX(memory_allocation): return "Allocation error : not enough memory";
-    case PREFIX(workSpace_tooSmall): return "workSpace buffer is not large enough";
-    case PREFIX(stage_wrong): return "Operation not authorized at current processing stage";
-    case PREFIX(tableLog_tooLarge): return "tableLog requires too much memory : unsupported";
-    case PREFIX(maxSymbolValue_tooLarge): return "Unsupported max Symbol Value : too large";
-    case PREFIX(maxSymbolValue_tooSmall): return "Specified maxSymbolValue is too small";
-    case PREFIX(cannotProduce_uncompressedBlock): return "This mode cannot generate an uncompressed block";
-    case PREFIX(stabilityCondition_notRespected): return "pledged buffer stability condition is not respected";
-    case PREFIX(dictionary_corrupted): return "Dictionary is corrupted";
-    case PREFIX(dictionary_wrong): return "Dictionary mismatch";
-    case PREFIX(dictionaryCreation_failed): return "Cannot create Dictionary from provided samples";
-    case PREFIX(dstSize_tooSmall): return "Destination buffer is too small";
-    case PREFIX(srcSize_wrong): return "Src size is incorrect";
-    case PREFIX(dstBuffer_null): return "Operation on NULL destination buffer";
-    case PREFIX(noForwardProgress_destFull): return "Operation made no progress over multiple calls, due to output buffer being full";
-    case PREFIX(noForwardProgress_inputEmpty): return "Operation made no progress over multiple calls, due to input being empty";
-        /* following error codes are not stable and may be removed or changed in a future version */
-    case PREFIX(frameIndex_tooLarge): return "Frame index is too large";
-    case PREFIX(seekableIO): return "An I/O error occurred when reading/seeking";
-    case PREFIX(dstBuffer_wrong): return "Destination buffer is wrong";
-    case PREFIX(srcBuffer_wrong): return "Source buffer is wrong";
-    case PREFIX(sequenceProducer_failed): return "Block-level external sequence producer returned an error code";
-    case PREFIX(externalSequences_invalid): return "External sequences are not valid";
-    case PREFIX(maxCode):
-    default: return notErrorCode;
-    }
-#endif
-}
-
-#endif /* USE_EXTERNAL_ZSTD */

vendor/github.com/DataDog/zstd/error_private.h

@@ -1,161 +0,0 @@
-#ifndef USE_EXTERNAL_ZSTD
-/*
- * Copyright (c) Meta Platforms, Inc. and affiliates.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- * You may select, at your option, one of the above-listed licenses.
- */
-
-/* Note : this module is expected to remain private, do not expose it */
-
-#ifndef ERROR_H_MODULE
-#define ERROR_H_MODULE
-
-/* ****************************************
-*  Dependencies
-******************************************/
-#include "zstd_errors.h"  /* enum list */
-#include "compiler.h"
-#include "debug.h"
-#include "zstd_deps.h"       /* size_t */
-
-/* ****************************************
-*  Compiler-specific
-******************************************/
-#if defined(__GNUC__)
-#  define ERR_STATIC static __attribute__((unused))
-#elif defined (__cplusplus) || (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */)
-#  define ERR_STATIC static inline
-#elif defined(_MSC_VER)
-#  define ERR_STATIC static __inline
-#else
-#  define ERR_STATIC static  /* this version may generate warnings for unused static functions; disable the relevant warning */
-#endif
-
-
-/*-****************************************
-*  Customization (error_public.h)
-******************************************/
-typedef ZSTD_ErrorCode ERR_enum;
-#define PREFIX(name) ZSTD_error_##name
-
-
-/*-****************************************
-*  Error codes handling
-******************************************/
-#undef ERROR   /* already defined on Visual Studio */
-#define ERROR(name) ZSTD_ERROR(name)
-#define ZSTD_ERROR(name) ((size_t)-PREFIX(name))
-
-ERR_STATIC unsigned ERR_isError(size_t code) { return (code > ERROR(maxCode)); }
-
-ERR_STATIC ERR_enum ERR_getErrorCode(size_t code) { if (!ERR_isError(code)) return (ERR_enum)0; return (ERR_enum) (0-code); }
-
-/* check and forward error code */
-#define CHECK_V_F(e, f)     \
-    size_t const e = f;     \
-    do {                    \
-        if (ERR_isError(e)) \
-            return e;       \
-    } while (0)
-#define CHECK_F(f)   do { CHECK_V_F(_var_err__, f); } while (0)
-
-
-/*-****************************************
-*  Error Strings
-******************************************/
-
-const char* ERR_getErrorString(ERR_enum code);   /* error_private.c */
-
-ERR_STATIC const char* ERR_getErrorName(size_t code)
-{
-    return ERR_getErrorString(ERR_getErrorCode(code));
-}
-
-/**
- * Ignore: this is an internal helper.
- *
- * This is a helper function to help force C99-correctness during compilation.
- * Under strict compilation modes, variadic macro arguments can't be empty.
- * However, variadic function arguments can be. Using a function therefore lets
- * us statically check that at least one (string) argument was passed,
- * independent of the compilation flags.
- */
-static INLINE_KEYWORD UNUSED_ATTR
-void _force_has_format_string(const char *format, ...) {
-  (void)format;
-}
-
-/**
- * Ignore: this is an internal helper.
- *
- * We want to force this function invocation to be syntactically correct, but
- * we don't want to force runtime evaluation of its arguments.
- */
-#define _FORCE_HAS_FORMAT_STRING(...)              \
-    do {                                           \
-        if (0) {                                   \
-            _force_has_format_string(__VA_ARGS__); \
-        }                                          \
-    } while (0)
-
-#define ERR_QUOTE(str) #str
-
-/**
- * Return the specified error if the condition evaluates to true.
- *
- * In debug modes, prints additional information.
- * In order to do that (particularly, printing the conditional that failed),
- * this can't just wrap RETURN_ERROR().
- */
-#define RETURN_ERROR_IF(cond, err, ...)                                        \
-    do {                                                                       \
-        if (cond) {                                                            \
-            RAWLOG(3, "%s:%d: ERROR!: check %s failed, returning %s",          \
-                  __FILE__, __LINE__, ERR_QUOTE(cond), ERR_QUOTE(ERROR(err))); \
-            _FORCE_HAS_FORMAT_STRING(__VA_ARGS__);                             \
-            RAWLOG(3, ": " __VA_ARGS__);                                       \
-            RAWLOG(3, "\n");                                                   \
-            return ERROR(err);                                                 \
-        }                                                                      \
-    } while (0)
-
-/**
- * Unconditionally return the specified error.
- *
- * In debug modes, prints additional information.
- */
-#define RETURN_ERROR(err, ...)                                               \
-    do {                                                                     \
-        RAWLOG(3, "%s:%d: ERROR!: unconditional check failed, returning %s", \
-              __FILE__, __LINE__, ERR_QUOTE(ERROR(err)));                    \
-        _FORCE_HAS_FORMAT_STRING(__VA_ARGS__);                               \
-        RAWLOG(3, ": " __VA_ARGS__);                                         \
-        RAWLOG(3, "\n");                                                     \
-        return ERROR(err);                                                   \
-    } while(0)
-
-/**
- * If the provided expression evaluates to an error code, returns that error code.
- *
- * In debug modes, prints additional information.
- */
-#define FORWARD_IF_ERROR(err, ...)                                                 \
-    do {                                                                           \
-        size_t const err_code = (err);                                             \
-        if (ERR_isError(err_code)) {                                               \
-            RAWLOG(3, "%s:%d: ERROR!: forwarding error in %s: %s",                 \
-                  __FILE__, __LINE__, ERR_QUOTE(err), ERR_getErrorName(err_code)); \
-            _FORCE_HAS_FORMAT_STRING(__VA_ARGS__);                                 \
-            RAWLOG(3, ": " __VA_ARGS__);                                           \
-            RAWLOG(3, "\n");                                                       \
-            return err_code;                                                       \
-        }                                                                          \
-    } while(0)
-
-#endif /* ERROR_H_MODULE */
-
-#endif /* USE_EXTERNAL_ZSTD */

vendor/github.com/DataDog/zstd/errors.go

@@ -1,34 +0,0 @@
-package zstd
-
-/*
-#include "zstd.h"
-*/
-import "C"
-
-// ErrorCode is an error returned by the zstd library.
-type ErrorCode int
-
-// Error returns the error string given by zstd
-func (e ErrorCode) Error() string {
-	return C.GoString(C.ZSTD_getErrorName(C.size_t(e)))
-}
-
-func cIsError(code int) bool {
-	return int(C.ZSTD_isError(C.size_t(code))) != 0
-}
-
-// getError returns an error for the return code, or nil if it's not an error
-func getError(code int) error {
-	if code < 0 && cIsError(code) {
-		return ErrorCode(code)
-	}
-	return nil
-}
-
-// IsDstSizeTooSmallError returns whether the error correspond to zstd standard sDstSizeTooSmall error
-func IsDstSizeTooSmallError(e error) bool {
-	if e != nil && e.Error() == "Destination buffer is too small" {
-		return true
-	}
-	return false
-}

vendor/github.com/DataDog/zstd/external_zstd.go

@@ -1,14 +0,0 @@
-//go:build external_libzstd
-// +build external_libzstd
-
-package zstd
-
-// #cgo CFLAGS: -DUSE_EXTERNAL_ZSTD
-// #cgo pkg-config: libzstd
-/*
-#include<zstd.h>
-#if ZSTD_VERSION_NUMBER < 10400
-#error "ZSTD version >= 1.4 is required"
-#endif
-*/
-import "C"

vendor/github.com/DataDog/zstd/fastcover.c

@@ -1,769 +0,0 @@
-#ifndef USE_EXTERNAL_ZSTD
-/*
- * Copyright (c) Meta Platforms, Inc. and affiliates.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- * You may select, at your option, one of the above-listed licenses.
- */
-
-/*-*************************************
-*  Dependencies
-***************************************/
-#include <stdio.h>  /* fprintf */
-#include <stdlib.h> /* malloc, free, qsort */
-#include <string.h> /* memset */
-#include <time.h>   /* clock */
-
-#ifndef ZDICT_STATIC_LINKING_ONLY
-#  define ZDICT_STATIC_LINKING_ONLY
-#endif
-
-#include "mem.h" /* read */
-#include "pool.h"
-#include "threading.h"
-#include "zstd_internal.h" /* includes zstd.h */
-#include "zstd_compress_internal.h" /* ZSTD_hash*() */
-#include "zdict.h"
-#include "cover.h"
-
-
-/*-*************************************
-*  Constants
-***************************************/
-/**
-* There are 32bit indexes used to ref samples, so limit samples size to 4GB
-* on 64bit builds.
-* For 32bit builds we choose 1 GB.
-* Most 32bit platforms have 2GB user-mode addressable space and we allocate a large
-* contiguous buffer, so 1GB is already a high limit.
-*/
-#define FASTCOVER_MAX_SAMPLES_SIZE (sizeof(size_t) == 8 ? ((unsigned)-1) : ((unsigned)1 GB))
-#define FASTCOVER_MAX_F 31
-#define FASTCOVER_MAX_ACCEL 10
-#define FASTCOVER_DEFAULT_SPLITPOINT 0.75
-#define DEFAULT_F 20
-#define DEFAULT_ACCEL 1
-
-
-/*-*************************************
-*  Console display
-***************************************/
-#ifndef LOCALDISPLAYLEVEL
-static int g_displayLevel = 0;
-#endif
-#undef  DISPLAY
-#define DISPLAY(...)                                                           \
-  {                                                                            \
-    fprintf(stderr, __VA_ARGS__);                                              \
-    fflush(stderr);                                                            \
-  }
-#undef  LOCALDISPLAYLEVEL
-#define LOCALDISPLAYLEVEL(displayLevel, l, ...)                                \
-  if (displayLevel >= l) {                                                     \
-    DISPLAY(__VA_ARGS__);                                                      \
-  } /* 0 : no display;   1: errors;   2: default;  3: details;  4: debug */
-#undef  DISPLAYLEVEL
-#define DISPLAYLEVEL(l, ...) LOCALDISPLAYLEVEL(g_displayLevel, l, __VA_ARGS__)
-
-#ifndef LOCALDISPLAYUPDATE
-static const clock_t g_refreshRate = CLOCKS_PER_SEC * 15 / 100;
-static clock_t g_time = 0;
-#endif
-#undef  LOCALDISPLAYUPDATE
-#define LOCALDISPLAYUPDATE(displayLevel, l, ...)                               \
-  if (displayLevel >= l) {                                                     \
-    if ((clock() - g_time > g_refreshRate) || (displayLevel >= 4)) {             \
-      g_time = clock();                                                        \
-      DISPLAY(__VA_ARGS__);                                                    \
-    }                                                                          \
-  }
-#undef  DISPLAYUPDATE
-#define DISPLAYUPDATE(l, ...) LOCALDISPLAYUPDATE(g_displayLevel, l, __VA_ARGS__)
-
-
-/*-*************************************
-* Hash Functions
-***************************************/
-/**
- * Hash the d-byte value pointed to by p and mod 2^f into the frequency vector
- */
-static size_t FASTCOVER_hashPtrToIndex(const void* p, U32 f, unsigned d) {
-  if (d == 6) {
-    return ZSTD_hash6Ptr(p, f);
-  }
-  return ZSTD_hash8Ptr(p, f);
-}
-
-
-/*-*************************************
-* Acceleration
-***************************************/
-typedef struct {
-  unsigned finalize;    /* Percentage of training samples used for ZDICT_finalizeDictionary */
-  unsigned skip;        /* Number of dmer skipped between each dmer counted in computeFrequency */
-} FASTCOVER_accel_t;
-
-
-static const FASTCOVER_accel_t FASTCOVER_defaultAccelParameters[FASTCOVER_MAX_ACCEL+1] = {
-  { 100, 0 },   /* accel = 0, should not happen because accel = 0 defaults to accel = 1 */
-  { 100, 0 },   /* accel = 1 */
-  { 50, 1 },   /* accel = 2 */
-  { 34, 2 },   /* accel = 3 */
-  { 25, 3 },   /* accel = 4 */
-  { 20, 4 },   /* accel = 5 */
-  { 17, 5 },   /* accel = 6 */
-  { 14, 6 },   /* accel = 7 */
-  { 13, 7 },   /* accel = 8 */
-  { 11, 8 },   /* accel = 9 */
-  { 10, 9 },   /* accel = 10 */
-};
-
-
-/*-*************************************
-* Context
-***************************************/
-typedef struct {
-  const BYTE *samples;
-  size_t *offsets;
-  const size_t *samplesSizes;
-  size_t nbSamples;
-  size_t nbTrainSamples;
-  size_t nbTestSamples;
-  size_t nbDmers;
-  U32 *freqs;
-  unsigned d;
-  unsigned f;
-  FASTCOVER_accel_t accelParams;
-} FASTCOVER_ctx_t;
-
-
-/*-*************************************
-*  Helper functions
-***************************************/
-/**
- * Selects the best segment in an epoch.
- * Segments of are scored according to the function:
- *
- * Let F(d) be the frequency of all dmers with hash value d.
- * Let S_i be hash value of the dmer at position i of segment S which has length k.
- *
- *     Score(S) = F(S_1) + F(S_2) + ... + F(S_{k-d+1})
- *
- * Once the dmer with hash value d is in the dictionary we set F(d) = 0.
- */
-static COVER_segment_t FASTCOVER_selectSegment(const FASTCOVER_ctx_t *ctx,
-                                              U32 *freqs, U32 begin, U32 end,
-                                              ZDICT_cover_params_t parameters,
-                                              U16* segmentFreqs) {
-  /* Constants */
-  const U32 k = parameters.k;
-  const U32 d = parameters.d;
-  const U32 f = ctx->f;
-  const U32 dmersInK = k - d + 1;
-
-  /* Try each segment (activeSegment) and save the best (bestSegment) */
-  COVER_segment_t bestSegment = {0, 0, 0};
-  COVER_segment_t activeSegment;
-
-  /* Reset the activeDmers in the segment */
-  /* The activeSegment starts at the beginning of the epoch. */
-  activeSegment.begin = begin;
-  activeSegment.end = begin;
-  activeSegment.score = 0;
-
-  /* Slide the activeSegment through the whole epoch.
-   * Save the best segment in bestSegment.
-   */
-  while (activeSegment.end < end) {
-    /* Get hash value of current dmer */
-    const size_t idx = FASTCOVER_hashPtrToIndex(ctx->samples + activeSegment.end, f, d);
-
-    /* Add frequency of this index to score if this is the first occurrence of index in active segment */
-    if (segmentFreqs[idx] == 0) {
-      activeSegment.score += freqs[idx];
-    }
-    /* Increment end of segment and segmentFreqs*/
-    activeSegment.end += 1;
-    segmentFreqs[idx] += 1;
-    /* If the window is now too large, drop the first position */
-    if (activeSegment.end - activeSegment.begin == dmersInK + 1) {
-      /* Get hash value of the dmer to be eliminated from active segment */
-      const size_t delIndex = FASTCOVER_hashPtrToIndex(ctx->samples + activeSegment.begin, f, d);
-      segmentFreqs[delIndex] -= 1;
-      /* Subtract frequency of this index from score if this is the last occurrence of this index in active segment */
-      if (segmentFreqs[delIndex] == 0) {
-        activeSegment.score -= freqs[delIndex];
-      }
-      /* Increment start of segment */
-      activeSegment.begin += 1;
-    }
-
-    /* If this segment is the best so far save it */
-    if (activeSegment.score > bestSegment.score) {
-      bestSegment = activeSegment;
-    }
-  }
-
-  /* Zero out rest of segmentFreqs array */
-  while (activeSegment.begin < end) {
-    const size_t delIndex = FASTCOVER_hashPtrToIndex(ctx->samples + activeSegment.begin, f, d);
-    segmentFreqs[delIndex] -= 1;
-    activeSegment.begin += 1;
-  }
-
-  {
-    /*  Zero the frequency of hash value of each dmer covered by the chosen segment. */
-    U32 pos;
-    for (pos = bestSegment.begin; pos != bestSegment.end; ++pos) {
-      const size_t i = FASTCOVER_hashPtrToIndex(ctx->samples + pos, f, d);
-      freqs[i] = 0;
-    }
-  }
-
-  return bestSegment;
-}
-
-
-static int FASTCOVER_checkParameters(ZDICT_cover_params_t parameters,
-                                     size_t maxDictSize, unsigned f,
-                                     unsigned accel) {
-  /* k, d, and f are required parameters */
-  if (parameters.d == 0 || parameters.k == 0) {
-    return 0;
-  }
-  /* d has to be 6 or 8 */
-  if (parameters.d != 6 && parameters.d != 8) {
-    return 0;
-  }
-  /* k <= maxDictSize */
-  if (parameters.k > maxDictSize) {
-    return 0;
-  }
-  /* d <= k */
-  if (parameters.d > parameters.k) {
-    return 0;
-  }
-  /* 0 < f <= FASTCOVER_MAX_F*/
-  if (f > FASTCOVER_MAX_F || f == 0) {
-    return 0;
-  }
-  /* 0 < splitPoint <= 1 */
-  if (parameters.splitPoint <= 0 || parameters.splitPoint > 1) {
-    return 0;
-  }
-  /* 0 < accel <= 10 */
-  if (accel > 10 || accel == 0) {
-    return 0;
-  }
-  return 1;
-}
-
-
-/**
- * Clean up a context initialized with `FASTCOVER_ctx_init()`.
- */
-static void
-FASTCOVER_ctx_destroy(FASTCOVER_ctx_t* ctx)
-{
-    if (!ctx) return;
-
-    free(ctx->freqs);
-    ctx->freqs = NULL;
-
-    free(ctx->offsets);
-    ctx->offsets = NULL;
-}
-
-
-/**
- * Calculate for frequency of hash value of each dmer in ctx->samples
- */
-static void
-FASTCOVER_computeFrequency(U32* freqs, const FASTCOVER_ctx_t* ctx)
-{
-    const unsigned f = ctx->f;
-    const unsigned d = ctx->d;
-    const unsigned skip = ctx->accelParams.skip;
-    const unsigned readLength = MAX(d, 8);
-    size_t i;
-    assert(ctx->nbTrainSamples >= 5);
-    assert(ctx->nbTrainSamples <= ctx->nbSamples);
-    for (i = 0; i < ctx->nbTrainSamples; i++) {
-        size_t start = ctx->offsets[i];  /* start of current dmer */
-        size_t const currSampleEnd = ctx->offsets[i+1];
-        while (start + readLength <= currSampleEnd) {
-            const size_t dmerIndex = FASTCOVER_hashPtrToIndex(ctx->samples + start, f, d);
-            freqs[dmerIndex]++;
-            start = start + skip + 1;
-        }
-    }
-}
-
-
-/**
- * Prepare a context for dictionary building.
- * The context is only dependent on the parameter `d` and can be used multiple
- * times.
- * Returns 0 on success or error code on error.
- * The context must be destroyed with `FASTCOVER_ctx_destroy()`.
- */
-static size_t
-FASTCOVER_ctx_init(FASTCOVER_ctx_t* ctx,
-                   const void* samplesBuffer,
-                   const size_t* samplesSizes, unsigned nbSamples,
-                   unsigned d, double splitPoint, unsigned f,
-                   FASTCOVER_accel_t accelParams)
-{
-    const BYTE* const samples = (const BYTE*)samplesBuffer;
-    const size_t totalSamplesSize = COVER_sum(samplesSizes, nbSamples);
-    /* Split samples into testing and training sets */
-    const unsigned nbTrainSamples = splitPoint < 1.0 ? (unsigned)((double)nbSamples * splitPoint) : nbSamples;
-    const unsigned nbTestSamples = splitPoint < 1.0 ? nbSamples - nbTrainSamples : nbSamples;
-    const size_t trainingSamplesSize = splitPoint < 1.0 ? COVER_sum(samplesSizes, nbTrainSamples) : totalSamplesSize;
-    const size_t testSamplesSize = splitPoint < 1.0 ? COVER_sum(samplesSizes + nbTrainSamples, nbTestSamples) : totalSamplesSize;
-
-    /* Checks */
-    if (totalSamplesSize < MAX(d, sizeof(U64)) ||
-        totalSamplesSize >= (size_t)FASTCOVER_MAX_SAMPLES_SIZE) {
-        DISPLAYLEVEL(1, "Total samples size is too large (%u MB), maximum size is %u MB\n",
-                    (unsigned)(totalSamplesSize >> 20), (FASTCOVER_MAX_SAMPLES_SIZE >> 20));
-        return ERROR(srcSize_wrong);
-    }
-
-    /* Check if there are at least 5 training samples */
-    if (nbTrainSamples < 5) {
-        DISPLAYLEVEL(1, "Total number of training samples is %u and is invalid\n", nbTrainSamples);
-        return ERROR(srcSize_wrong);
-    }
-
-    /* Check if there's testing sample */
-    if (nbTestSamples < 1) {
-        DISPLAYLEVEL(1, "Total number of testing samples is %u and is invalid.\n", nbTestSamples);
-        return ERROR(srcSize_wrong);
-    }
-
-    /* Zero the context */
-    memset(ctx, 0, sizeof(*ctx));
-    DISPLAYLEVEL(2, "Training on %u samples of total size %u\n", nbTrainSamples,
-                    (unsigned)trainingSamplesSize);
-    DISPLAYLEVEL(2, "Testing on %u samples of total size %u\n", nbTestSamples,
-                    (unsigned)testSamplesSize);
-
-    ctx->samples = samples;
-    ctx->samplesSizes = samplesSizes;
-    ctx->nbSamples = nbSamples;
-    ctx->nbTrainSamples = nbTrainSamples;
-    ctx->nbTestSamples = nbTestSamples;
-    ctx->nbDmers = trainingSamplesSize - MAX(d, sizeof(U64)) + 1;
-    ctx->d = d;
-    ctx->f = f;
-    ctx->accelParams = accelParams;
-
-    /* The offsets of each file */
-    ctx->offsets = (size_t*)calloc((nbSamples + 1), sizeof(size_t));
-    if (ctx->offsets == NULL) {
-        DISPLAYLEVEL(1, "Failed to allocate scratch buffers \n");
-        FASTCOVER_ctx_destroy(ctx);
-        return ERROR(memory_allocation);
-    }
-
-    /* Fill offsets from the samplesSizes */
-    {   U32 i;
-        ctx->offsets[0] = 0;
-        assert(nbSamples >= 5);
-        for (i = 1; i <= nbSamples; ++i) {
-            ctx->offsets[i] = ctx->offsets[i - 1] + samplesSizes[i - 1];
-        }
-    }
-
-    /* Initialize frequency array of size 2^f */
-    ctx->freqs = (U32*)calloc(((U64)1 << f), sizeof(U32));
-    if (ctx->freqs == NULL) {
-        DISPLAYLEVEL(1, "Failed to allocate frequency table \n");
-        FASTCOVER_ctx_destroy(ctx);
-        return ERROR(memory_allocation);
-    }
-
-    DISPLAYLEVEL(2, "Computing frequencies\n");
-    FASTCOVER_computeFrequency(ctx->freqs, ctx);
-
-    return 0;
-}
-
-
-/**
- * Given the prepared context build the dictionary.
- */
-static size_t
-FASTCOVER_buildDictionary(const FASTCOVER_ctx_t* ctx,
-                          U32* freqs,
-                          void* dictBuffer, size_t dictBufferCapacity,
-                          ZDICT_cover_params_t parameters,
-                          U16* segmentFreqs)
-{
-  BYTE *const dict = (BYTE *)dictBuffer;
-  size_t tail = dictBufferCapacity;
-  /* Divide the data into epochs. We will select one segment from each epoch. */
-  const COVER_epoch_info_t epochs = COVER_computeEpochs(
-      (U32)dictBufferCapacity, (U32)ctx->nbDmers, parameters.k, 1);
-  const size_t maxZeroScoreRun = 10;
-  size_t zeroScoreRun = 0;
-  size_t epoch;
-  DISPLAYLEVEL(2, "Breaking content into %u epochs of size %u\n",
-                (U32)epochs.num, (U32)epochs.size);
-  /* Loop through the epochs until there are no more segments or the dictionary
-   * is full.
-   */
-  for (epoch = 0; tail > 0; epoch = (epoch + 1) % epochs.num) {
-    const U32 epochBegin = (U32)(epoch * epochs.size);
-    const U32 epochEnd = epochBegin + epochs.size;
-    size_t segmentSize;
-    /* Select a segment */
-    COVER_segment_t segment = FASTCOVER_selectSegment(
-        ctx, freqs, epochBegin, epochEnd, parameters, segmentFreqs);
-
-    /* If the segment covers no dmers, then we are out of content.
-     * There may be new content in other epochs, for continue for some time.
-     */
-    if (segment.score == 0) {
-      if (++zeroScoreRun >= maxZeroScoreRun) {
-          break;
-      }
-      continue;
-    }
-    zeroScoreRun = 0;
-
-    /* Trim the segment if necessary and if it is too small then we are done */
-    segmentSize = MIN(segment.end - segment.begin + parameters.d - 1, tail);
-    if (segmentSize < parameters.d) {
-      break;
-    }
-
-    /* We fill the dictionary from the back to allow the best segments to be
-     * referenced with the smallest offsets.
-     */
-    tail -= segmentSize;
-    memcpy(dict + tail, ctx->samples + segment.begin, segmentSize);
-    DISPLAYUPDATE(
-        2, "\r%u%%       ",
-        (unsigned)(((dictBufferCapacity - tail) * 100) / dictBufferCapacity));
-  }
-  DISPLAYLEVEL(2, "\r%79s\r", "");
-  return tail;
-}
-
-/**
- * Parameters for FASTCOVER_tryParameters().
- */
-typedef struct FASTCOVER_tryParameters_data_s {
-    const FASTCOVER_ctx_t* ctx;
-    COVER_best_t* best;
-    size_t dictBufferCapacity;
-    ZDICT_cover_params_t parameters;
-} FASTCOVER_tryParameters_data_t;
-
-
-/**
- * Tries a set of parameters and updates the COVER_best_t with the results.
- * This function is thread safe if zstd is compiled with multithreaded support.
- * It takes its parameters as an *OWNING* opaque pointer to support threading.
- */
-static void FASTCOVER_tryParameters(void* opaque)
-{
-  /* Save parameters as local variables */
-  FASTCOVER_tryParameters_data_t *const data = (FASTCOVER_tryParameters_data_t*)opaque;
-  const FASTCOVER_ctx_t *const ctx = data->ctx;
-  const ZDICT_cover_params_t parameters = data->parameters;
-  size_t dictBufferCapacity = data->dictBufferCapacity;
-  size_t totalCompressedSize = ERROR(GENERIC);
-  /* Initialize array to keep track of frequency of dmer within activeSegment */
-  U16* segmentFreqs = (U16*)calloc(((U64)1 << ctx->f), sizeof(U16));
-  /* Allocate space for hash table, dict, and freqs */
-  BYTE *const dict = (BYTE*)malloc(dictBufferCapacity);
-  COVER_dictSelection_t selection = COVER_dictSelectionError(ERROR(GENERIC));
-  U32* freqs = (U32*) malloc(((U64)1 << ctx->f) * sizeof(U32));
-  if (!segmentFreqs || !dict || !freqs) {
-    DISPLAYLEVEL(1, "Failed to allocate buffers: out of memory\n");
-    goto _cleanup;
-  }
-  /* Copy the frequencies because we need to modify them */
-  memcpy(freqs, ctx->freqs, ((U64)1 << ctx->f) * sizeof(U32));
-  /* Build the dictionary */
-  { const size_t tail = FASTCOVER_buildDictionary(ctx, freqs, dict, dictBufferCapacity,
-                                                    parameters, segmentFreqs);
-
-    const unsigned nbFinalizeSamples = (unsigned)(ctx->nbTrainSamples * ctx->accelParams.finalize / 100);
-    selection = COVER_selectDict(dict + tail, dictBufferCapacity, dictBufferCapacity - tail,
-         ctx->samples, ctx->samplesSizes, nbFinalizeSamples, ctx->nbTrainSamples, ctx->nbSamples, parameters, ctx->offsets,
-         totalCompressedSize);
-
-    if (COVER_dictSelectionIsError(selection)) {
-      DISPLAYLEVEL(1, "Failed to select dictionary\n");
-      goto _cleanup;
-    }
-  }
-_cleanup:
-  free(dict);
-  COVER_best_finish(data->best, parameters, selection);
-  free(data);
-  free(segmentFreqs);
-  COVER_dictSelectionFree(selection);
-  free(freqs);
-}
-
-
-static void
-FASTCOVER_convertToCoverParams(ZDICT_fastCover_params_t fastCoverParams,
-                               ZDICT_cover_params_t* coverParams)
-{
-    coverParams->k = fastCoverParams.k;
-    coverParams->d = fastCoverParams.d;
-    coverParams->steps = fastCoverParams.steps;
-    coverParams->nbThreads = fastCoverParams.nbThreads;
-    coverParams->splitPoint = fastCoverParams.splitPoint;
-    coverParams->zParams = fastCoverParams.zParams;
-    coverParams->shrinkDict = fastCoverParams.shrinkDict;
-}
-
-
-static void
-FASTCOVER_convertToFastCoverParams(ZDICT_cover_params_t coverParams,
-                                   ZDICT_fastCover_params_t* fastCoverParams,
-                                   unsigned f, unsigned accel)
-{
-    fastCoverParams->k = coverParams.k;
-    fastCoverParams->d = coverParams.d;
-    fastCoverParams->steps = coverParams.steps;
-    fastCoverParams->nbThreads = coverParams.nbThreads;
-    fastCoverParams->splitPoint = coverParams.splitPoint;
-    fastCoverParams->f = f;
-    fastCoverParams->accel = accel;
-    fastCoverParams->zParams = coverParams.zParams;
-    fastCoverParams->shrinkDict = coverParams.shrinkDict;
-}
-
-
-ZDICTLIB_STATIC_API size_t
-ZDICT_trainFromBuffer_fastCover(void* dictBuffer, size_t dictBufferCapacity,
-                                const void* samplesBuffer,
-                                const size_t* samplesSizes, unsigned nbSamples,
-                                ZDICT_fastCover_params_t parameters)
-{
-    BYTE* const dict = (BYTE*)dictBuffer;
-    FASTCOVER_ctx_t ctx;
-    ZDICT_cover_params_t coverParams;
-    FASTCOVER_accel_t accelParams;
-    /* Initialize global data */
-    g_displayLevel = (int)parameters.zParams.notificationLevel;
-    /* Assign splitPoint and f if not provided */
-    parameters.splitPoint = 1.0;
-    parameters.f = parameters.f == 0 ? DEFAULT_F : parameters.f;
-    parameters.accel = parameters.accel == 0 ? DEFAULT_ACCEL : parameters.accel;
-    /* Convert to cover parameter */
-    memset(&coverParams, 0 , sizeof(coverParams));
-    FASTCOVER_convertToCoverParams(parameters, &coverParams);
-    /* Checks */
-    if (!FASTCOVER_checkParameters(coverParams, dictBufferCapacity, parameters.f,
-                                   parameters.accel)) {
-      DISPLAYLEVEL(1, "FASTCOVER parameters incorrect\n");
-      return ERROR(parameter_outOfBound);
-    }
-    if (nbSamples == 0) {
-      DISPLAYLEVEL(1, "FASTCOVER must have at least one input file\n");
-      return ERROR(srcSize_wrong);
-    }
-    if (dictBufferCapacity < ZDICT_DICTSIZE_MIN) {
-      DISPLAYLEVEL(1, "dictBufferCapacity must be at least %u\n",
-                   ZDICT_DICTSIZE_MIN);
-      return ERROR(dstSize_tooSmall);
-    }
-    /* Assign corresponding FASTCOVER_accel_t to accelParams*/
-    accelParams = FASTCOVER_defaultAccelParameters[parameters.accel];
-    /* Initialize context */
-    {
-      size_t const initVal = FASTCOVER_ctx_init(&ctx, samplesBuffer, samplesSizes, nbSamples,
-                            coverParams.d, parameters.splitPoint, parameters.f,
-                            accelParams);
-      if (ZSTD_isError(initVal)) {
-        DISPLAYLEVEL(1, "Failed to initialize context\n");
-        return initVal;
-      }
-    }
-    COVER_warnOnSmallCorpus(dictBufferCapacity, ctx.nbDmers, g_displayLevel);
-    /* Build the dictionary */
-    DISPLAYLEVEL(2, "Building dictionary\n");
-    {
-      /* Initialize array to keep track of frequency of dmer within activeSegment */
-      U16* segmentFreqs = (U16 *)calloc(((U64)1 << parameters.f), sizeof(U16));
-      const size_t tail = FASTCOVER_buildDictionary(&ctx, ctx.freqs, dictBuffer,
-                                                dictBufferCapacity, coverParams, segmentFreqs);
-      const unsigned nbFinalizeSamples = (unsigned)(ctx.nbTrainSamples * ctx.accelParams.finalize / 100);
-      const size_t dictionarySize = ZDICT_finalizeDictionary(
-          dict, dictBufferCapacity, dict + tail, dictBufferCapacity - tail,
-          samplesBuffer, samplesSizes, nbFinalizeSamples, coverParams.zParams);
-      if (!ZSTD_isError(dictionarySize)) {
-          DISPLAYLEVEL(2, "Constructed dictionary of size %u\n",
-                      (unsigned)dictionarySize);
-      }
-      FASTCOVER_ctx_destroy(&ctx);
-      free(segmentFreqs);
-      return dictionarySize;
-    }
-}
-
-
-ZDICTLIB_STATIC_API size_t
-ZDICT_optimizeTrainFromBuffer_fastCover(
-                    void* dictBuffer, size_t dictBufferCapacity,
-                    const void* samplesBuffer,
-                    const size_t* samplesSizes, unsigned nbSamples,
-                    ZDICT_fastCover_params_t* parameters)
-{
-    ZDICT_cover_params_t coverParams;
-    FASTCOVER_accel_t accelParams;
-    /* constants */
-    const unsigned nbThreads = parameters->nbThreads;
-    const double splitPoint =
-        parameters->splitPoint <= 0.0 ? FASTCOVER_DEFAULT_SPLITPOINT : parameters->splitPoint;
-    const unsigned kMinD = parameters->d == 0 ? 6 : parameters->d;
-    const unsigned kMaxD = parameters->d == 0 ? 8 : parameters->d;
-    const unsigned kMinK = parameters->k == 0 ? 50 : parameters->k;
-    const unsigned kMaxK = parameters->k == 0 ? 2000 : parameters->k;
-    const unsigned kSteps = parameters->steps == 0 ? 40 : parameters->steps;
-    const unsigned kStepSize = MAX((kMaxK - kMinK) / kSteps, 1);
-    const unsigned kIterations =
-        (1 + (kMaxD - kMinD) / 2) * (1 + (kMaxK - kMinK) / kStepSize);
-    const unsigned f = parameters->f == 0 ? DEFAULT_F : parameters->f;
-    const unsigned accel = parameters->accel == 0 ? DEFAULT_ACCEL : parameters->accel;
-    const unsigned shrinkDict = 0;
-    /* Local variables */
-    const int displayLevel = (int)parameters->zParams.notificationLevel;
-    unsigned iteration = 1;
-    unsigned d;
-    unsigned k;
-    COVER_best_t best;
-    POOL_ctx *pool = NULL;
-    int warned = 0;
-    /* Checks */
-    if (splitPoint <= 0 || splitPoint > 1) {
-      LOCALDISPLAYLEVEL(displayLevel, 1, "Incorrect splitPoint\n");
-      return ERROR(parameter_outOfBound);
-    }
-    if (accel == 0 || accel > FASTCOVER_MAX_ACCEL) {
-      LOCALDISPLAYLEVEL(displayLevel, 1, "Incorrect accel\n");
-      return ERROR(parameter_outOfBound);
-    }
-    if (kMinK < kMaxD || kMaxK < kMinK) {
-      LOCALDISPLAYLEVEL(displayLevel, 1, "Incorrect k\n");
-      return ERROR(parameter_outOfBound);
-    }
-    if (nbSamples == 0) {
-      LOCALDISPLAYLEVEL(displayLevel, 1, "FASTCOVER must have at least one input file\n");
-      return ERROR(srcSize_wrong);
-    }
-    if (dictBufferCapacity < ZDICT_DICTSIZE_MIN) {
-      LOCALDISPLAYLEVEL(displayLevel, 1, "dictBufferCapacity must be at least %u\n",
-                   ZDICT_DICTSIZE_MIN);
-      return ERROR(dstSize_tooSmall);
-    }
-    if (nbThreads > 1) {
-      pool = POOL_create(nbThreads, 1);
-      if (!pool) {
-        return ERROR(memory_allocation);
-      }
-    }
-    /* Initialization */
-    COVER_best_init(&best);
-    memset(&coverParams, 0 , sizeof(coverParams));
-    FASTCOVER_convertToCoverParams(*parameters, &coverParams);
-    accelParams = FASTCOVER_defaultAccelParameters[accel];
-    /* Turn down global display level to clean up display at level 2 and below */
-    g_displayLevel = displayLevel == 0 ? 0 : displayLevel - 1;
-    /* Loop through d first because each new value needs a new context */
-    LOCALDISPLAYLEVEL(displayLevel, 2, "Trying %u different sets of parameters\n",
-                      kIterations);
-    for (d = kMinD; d <= kMaxD; d += 2) {
-      /* Initialize the context for this value of d */
-      FASTCOVER_ctx_t ctx;
-      LOCALDISPLAYLEVEL(displayLevel, 3, "d=%u\n", d);
-      {
-        size_t const initVal = FASTCOVER_ctx_init(&ctx, samplesBuffer, samplesSizes, nbSamples, d, splitPoint, f, accelParams);
-        if (ZSTD_isError(initVal)) {
-          LOCALDISPLAYLEVEL(displayLevel, 1, "Failed to initialize context\n");
-          COVER_best_destroy(&best);
-          POOL_free(pool);
-          return initVal;
-        }
-      }
-      if (!warned) {
-        COVER_warnOnSmallCorpus(dictBufferCapacity, ctx.nbDmers, displayLevel);
-        warned = 1;
-      }
-      /* Loop through k reusing the same context */
-      for (k = kMinK; k <= kMaxK; k += kStepSize) {
-        /* Prepare the arguments */
-        FASTCOVER_tryParameters_data_t *data = (FASTCOVER_tryParameters_data_t *)malloc(
-            sizeof(FASTCOVER_tryParameters_data_t));
-        LOCALDISPLAYLEVEL(displayLevel, 3, "k=%u\n", k);
-        if (!data) {
-          LOCALDISPLAYLEVEL(displayLevel, 1, "Failed to allocate parameters\n");
-          COVER_best_destroy(&best);
-          FASTCOVER_ctx_destroy(&ctx);
-          POOL_free(pool);
-          return ERROR(memory_allocation);
-        }
-        data->ctx = &ctx;
-        data->best = &best;
-        data->dictBufferCapacity = dictBufferCapacity;
-        data->parameters = coverParams;
-        data->parameters.k = k;
-        data->parameters.d = d;
-        data->parameters.splitPoint = splitPoint;
-        data->parameters.steps = kSteps;
-        data->parameters.shrinkDict = shrinkDict;
-        data->parameters.zParams.notificationLevel = (unsigned)g_displayLevel;
-        /* Check the parameters */
-        if (!FASTCOVER_checkParameters(data->parameters, dictBufferCapacity,
-                                       data->ctx->f, accel)) {
-          DISPLAYLEVEL(1, "FASTCOVER parameters incorrect\n");
-          free(data);
-          continue;
-        }
-        /* Call the function and pass ownership of data to it */
-        COVER_best_start(&best);
-        if (pool) {
-          POOL_add(pool, &FASTCOVER_tryParameters, data);
-        } else {
-          FASTCOVER_tryParameters(data);
-        }
-        /* Print status */
-        LOCALDISPLAYUPDATE(displayLevel, 2, "\r%u%%       ",
-                           (unsigned)((iteration * 100) / kIterations));
-        ++iteration;
-      }
-      COVER_best_wait(&best);
-      FASTCOVER_ctx_destroy(&ctx);
-    }
-    LOCALDISPLAYLEVEL(displayLevel, 2, "\r%79s\r", "");
-    /* Fill the output buffer and parameters with output of the best parameters */
-    {
-      const size_t dictSize = best.dictSize;
-      if (ZSTD_isError(best.compressedSize)) {
-        const size_t compressedSize = best.compressedSize;
-        COVER_best_destroy(&best);
-        POOL_free(pool);
-        return compressedSize;
-      }
-      FASTCOVER_convertToFastCoverParams(best.parameters, parameters, f, accel);
-      memcpy(dictBuffer, best.dict, dictSize);
-      COVER_best_destroy(&best);
-      POOL_free(pool);
-      return dictSize;
-    }
-
-}
-
-#endif /* USE_EXTERNAL_ZSTD */

vendor/github.com/DataDog/zstd/fse.h

@@ -1,628 +0,0 @@
-#ifndef USE_EXTERNAL_ZSTD
-/* ******************************************************************
- * FSE : Finite State Entropy codec
- * Public Prototypes declaration
- * Copyright (c) Meta Platforms, Inc. and affiliates.
- *
- * You can contact the author at :
- * - Source repository : https://github.com/Cyan4973/FiniteStateEntropy
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- * You may select, at your option, one of the above-listed licenses.
-****************************************************************** */
-#ifndef FSE_H
-#define FSE_H
-
-
-/*-*****************************************
-*  Dependencies
-******************************************/
-#include "zstd_deps.h"    /* size_t, ptrdiff_t */
-
-/*-*****************************************
-*  FSE_PUBLIC_API : control library symbols visibility
-******************************************/
-#if defined(FSE_DLL_EXPORT) && (FSE_DLL_EXPORT==1) && defined(__GNUC__) && (__GNUC__ >= 4)
-#  define FSE_PUBLIC_API __attribute__ ((visibility ("default")))
-#elif defined(FSE_DLL_EXPORT) && (FSE_DLL_EXPORT==1)   /* Visual expected */
-#  define FSE_PUBLIC_API __declspec(dllexport)
-#elif defined(FSE_DLL_IMPORT) && (FSE_DLL_IMPORT==1)
-#  define FSE_PUBLIC_API __declspec(dllimport) /* It isn't required but allows to generate better code, saving a function pointer load from the IAT and an indirect jump.*/
-#else
-#  define FSE_PUBLIC_API
-#endif
-
-/*------   Version   ------*/
-#define FSE_VERSION_MAJOR    0
-#define FSE_VERSION_MINOR    9
-#define FSE_VERSION_RELEASE  0
-
-#define FSE_LIB_VERSION FSE_VERSION_MAJOR.FSE_VERSION_MINOR.FSE_VERSION_RELEASE
-#define FSE_QUOTE(str) #str
-#define FSE_EXPAND_AND_QUOTE(str) FSE_QUOTE(str)
-#define FSE_VERSION_STRING FSE_EXPAND_AND_QUOTE(FSE_LIB_VERSION)
-
-#define FSE_VERSION_NUMBER  (FSE_VERSION_MAJOR *100*100 + FSE_VERSION_MINOR *100 + FSE_VERSION_RELEASE)
-FSE_PUBLIC_API unsigned FSE_versionNumber(void);   /**< library version number; to be used when checking dll version */
-
-
-/*-*****************************************
-*  Tool functions
-******************************************/
-FSE_PUBLIC_API size_t FSE_compressBound(size_t size);       /* maximum compressed size */
-
-/* Error Management */
-FSE_PUBLIC_API unsigned    FSE_isError(size_t code);        /* tells if a return value is an error code */
-FSE_PUBLIC_API const char* FSE_getErrorName(size_t code);   /* provides error code string (useful for debugging) */
-
-
-/*-*****************************************
-*  FSE detailed API
-******************************************/
-/*!
-FSE_compress() does the following:
-1. count symbol occurrence from source[] into table count[] (see hist.h)
-2. normalize counters so that sum(count[]) == Power_of_2 (2^tableLog)
-3. save normalized counters to memory buffer using writeNCount()
-4. build encoding table 'CTable' from normalized counters
-5. encode the data stream using encoding table 'CTable'
-
-FSE_decompress() does the following:
-1. read normalized counters with readNCount()
-2. build decoding table 'DTable' from normalized counters
-3. decode the data stream using decoding table 'DTable'
-
-The following API allows targeting specific sub-functions for advanced tasks.
-For example, it's possible to compress several blocks using the same 'CTable',
-or to save and provide normalized distribution using external method.
-*/
-
-/* *** COMPRESSION *** */
-
-/*! FSE_optimalTableLog():
-    dynamically downsize 'tableLog' when conditions are met.
-    It saves CPU time, by using smaller tables, while preserving or even improving compression ratio.
-    @return : recommended tableLog (necessarily <= 'maxTableLog') */
-FSE_PUBLIC_API unsigned FSE_optimalTableLog(unsigned maxTableLog, size_t srcSize, unsigned maxSymbolValue);
-
-/*! FSE_normalizeCount():
-    normalize counts so that sum(count[]) == Power_of_2 (2^tableLog)
-    'normalizedCounter' is a table of short, of minimum size (maxSymbolValue+1).
-    useLowProbCount is a boolean parameter which trades off compressed size for
-    faster header decoding. When it is set to 1, the compressed data will be slightly
-    smaller. And when it is set to 0, FSE_readNCount() and FSE_buildDTable() will be
-    faster. If you are compressing a small amount of data (< 2 KB) then useLowProbCount=0
-    is a good default, since header deserialization makes a big speed difference.
-    Otherwise, useLowProbCount=1 is a good default, since the speed difference is small.
-    @return : tableLog,
-              or an errorCode, which can be tested using FSE_isError() */
-FSE_PUBLIC_API size_t FSE_normalizeCount(short* normalizedCounter, unsigned tableLog,
-                    const unsigned* count, size_t srcSize, unsigned maxSymbolValue, unsigned useLowProbCount);
-
-/*! FSE_NCountWriteBound():
-    Provides the maximum possible size of an FSE normalized table, given 'maxSymbolValue' and 'tableLog'.
-    Typically useful for allocation purpose. */
-FSE_PUBLIC_API size_t FSE_NCountWriteBound(unsigned maxSymbolValue, unsigned tableLog);
-
-/*! FSE_writeNCount():
-    Compactly save 'normalizedCounter' into 'buffer'.
-    @return : size of the compressed table,
-              or an errorCode, which can be tested using FSE_isError(). */
-FSE_PUBLIC_API size_t FSE_writeNCount (void* buffer, size_t bufferSize,
-                                 const short* normalizedCounter,
-                                 unsigned maxSymbolValue, unsigned tableLog);
-
-/*! Constructor and Destructor of FSE_CTable.
-    Note that FSE_CTable size depends on 'tableLog' and 'maxSymbolValue' */
-typedef unsigned FSE_CTable;   /* don't allocate that. It's only meant to be more restrictive than void* */
-
-/*! FSE_buildCTable():
-    Builds `ct`, which must be already allocated, using FSE_createCTable().
-    @return : 0, or an errorCode, which can be tested using FSE_isError() */
-FSE_PUBLIC_API size_t FSE_buildCTable(FSE_CTable* ct, const short* normalizedCounter, unsigned maxSymbolValue, unsigned tableLog);
-
-/*! FSE_compress_usingCTable():
-    Compress `src` using `ct` into `dst` which must be already allocated.
-    @return : size of compressed data (<= `dstCapacity`),
-              or 0 if compressed data could not fit into `dst`,
-              or an errorCode, which can be tested using FSE_isError() */
-FSE_PUBLIC_API size_t FSE_compress_usingCTable (void* dst, size_t dstCapacity, const void* src, size_t srcSize, const FSE_CTable* ct);
-
-/*!
-Tutorial :
-----------
-The first step is to count all symbols. FSE_count() does this job very fast.
-Result will be saved into 'count', a table of unsigned int, which must be already allocated, and have 'maxSymbolValuePtr[0]+1' cells.
-'src' is a table of bytes of size 'srcSize'. All values within 'src' MUST be <= maxSymbolValuePtr[0]
-maxSymbolValuePtr[0] will be updated, with its real value (necessarily <= original value)
-FSE_count() will return the number of occurrence of the most frequent symbol.
-This can be used to know if there is a single symbol within 'src', and to quickly evaluate its compressibility.
-If there is an error, the function will return an ErrorCode (which can be tested using FSE_isError()).
-
-The next step is to normalize the frequencies.
-FSE_normalizeCount() will ensure that sum of frequencies is == 2 ^'tableLog'.
-It also guarantees a minimum of 1 to any Symbol with frequency >= 1.
-You can use 'tableLog'==0 to mean "use default tableLog value".
-If you are unsure of which tableLog value to use, you can ask FSE_optimalTableLog(),
-which will provide the optimal valid tableLog given sourceSize, maxSymbolValue, and a user-defined maximum (0 means "default").
-
-The result of FSE_normalizeCount() will be saved into a table,
-called 'normalizedCounter', which is a table of signed short.
-'normalizedCounter' must be already allocated, and have at least 'maxSymbolValue+1' cells.
-The return value is tableLog if everything proceeded as expected.
-It is 0 if there is a single symbol within distribution.
-If there is an error (ex: invalid tableLog value), the function will return an ErrorCode (which can be tested using FSE_isError()).
-
-'normalizedCounter' can be saved in a compact manner to a memory area using FSE_writeNCount().
-'buffer' must be already allocated.
-For guaranteed success, buffer size must be at least FSE_headerBound().
-The result of the function is the number of bytes written into 'buffer'.
-If there is an error, the function will return an ErrorCode (which can be tested using FSE_isError(); ex : buffer size too small).
-
-'normalizedCounter' can then be used to create the compression table 'CTable'.
-The space required by 'CTable' must be already allocated, using FSE_createCTable().
-You can then use FSE_buildCTable() to fill 'CTable'.
-If there is an error, both functions will return an ErrorCode (which can be tested using FSE_isError()).
-
-'CTable' can then be used to compress 'src', with FSE_compress_usingCTable().
-Similar to FSE_count(), the convention is that 'src' is assumed to be a table of char of size 'srcSize'
-The function returns the size of compressed data (without header), necessarily <= `dstCapacity`.
-If it returns '0', compressed data could not fit into 'dst'.
-If there is an error, the function will return an ErrorCode (which can be tested using FSE_isError()).
-*/
-
-
-/* *** DECOMPRESSION *** */
-
-/*! FSE_readNCount():
-    Read compactly saved 'normalizedCounter' from 'rBuffer'.
-    @return : size read from 'rBuffer',
-              or an errorCode, which can be tested using FSE_isError().
-              maxSymbolValuePtr[0] and tableLogPtr[0] will also be updated with their respective values */
-FSE_PUBLIC_API size_t FSE_readNCount (short* normalizedCounter,
-                           unsigned* maxSymbolValuePtr, unsigned* tableLogPtr,
-                           const void* rBuffer, size_t rBuffSize);
-
-/*! FSE_readNCount_bmi2():
- * Same as FSE_readNCount() but pass bmi2=1 when your CPU supports BMI2 and 0 otherwise.
- */
-FSE_PUBLIC_API size_t FSE_readNCount_bmi2(short* normalizedCounter,
-                           unsigned* maxSymbolValuePtr, unsigned* tableLogPtr,
-                           const void* rBuffer, size_t rBuffSize, int bmi2);
-
-typedef unsigned FSE_DTable;   /* don't allocate that. It's just a way to be more restrictive than void* */
-
-/*!
-Tutorial :
-----------
-(Note : these functions only decompress FSE-compressed blocks.
- If block is uncompressed, use memcpy() instead
- If block is a single repeated byte, use memset() instead )
-
-The first step is to obtain the normalized frequencies of symbols.
-This can be performed by FSE_readNCount() if it was saved using FSE_writeNCount().
-'normalizedCounter' must be already allocated, and have at least 'maxSymbolValuePtr[0]+1' cells of signed short.
-In practice, that means it's necessary to know 'maxSymbolValue' beforehand,
-or size the table to handle worst case situations (typically 256).
-FSE_readNCount() will provide 'tableLog' and 'maxSymbolValue'.
-The result of FSE_readNCount() is the number of bytes read from 'rBuffer'.
-Note that 'rBufferSize' must be at least 4 bytes, even if useful information is less than that.
-If there is an error, the function will return an error code, which can be tested using FSE_isError().
-
-The next step is to build the decompression tables 'FSE_DTable' from 'normalizedCounter'.
-This is performed by the function FSE_buildDTable().
-The space required by 'FSE_DTable' must be already allocated using FSE_createDTable().
-If there is an error, the function will return an error code, which can be tested using FSE_isError().
-
-`FSE_DTable` can then be used to decompress `cSrc`, with FSE_decompress_usingDTable().
-`cSrcSize` must be strictly correct, otherwise decompression will fail.
-FSE_decompress_usingDTable() result will tell how many bytes were regenerated (<=`dstCapacity`).
-If there is an error, the function will return an error code, which can be tested using FSE_isError(). (ex: dst buffer too small)
-*/
-
-#endif  /* FSE_H */
-
-
-#if defined(FSE_STATIC_LINKING_ONLY) && !defined(FSE_H_FSE_STATIC_LINKING_ONLY)
-#define FSE_H_FSE_STATIC_LINKING_ONLY
-#include "bitstream.h"
-
-/* *****************************************
-*  Static allocation
-*******************************************/
-/* FSE buffer bounds */
-#define FSE_NCOUNTBOUND 512
-#define FSE_BLOCKBOUND(size) ((size) + ((size)>>7) + 4 /* fse states */ + sizeof(size_t) /* bitContainer */)
-#define FSE_COMPRESSBOUND(size) (FSE_NCOUNTBOUND + FSE_BLOCKBOUND(size))   /* Macro version, useful for static allocation */
-
-/* It is possible to statically allocate FSE CTable/DTable as a table of FSE_CTable/FSE_DTable using below macros */
-#define FSE_CTABLE_SIZE_U32(maxTableLog, maxSymbolValue)   (1 + (1<<((maxTableLog)-1)) + (((maxSymbolValue)+1)*2))
-#define FSE_DTABLE_SIZE_U32(maxTableLog)                   (1 + (1<<(maxTableLog)))
-
-/* or use the size to malloc() space directly. Pay attention to alignment restrictions though */
-#define FSE_CTABLE_SIZE(maxTableLog, maxSymbolValue)   (FSE_CTABLE_SIZE_U32(maxTableLog, maxSymbolValue) * sizeof(FSE_CTable))
-#define FSE_DTABLE_SIZE(maxTableLog)                   (FSE_DTABLE_SIZE_U32(maxTableLog) * sizeof(FSE_DTable))
-
-
-/* *****************************************
- *  FSE advanced API
- ***************************************** */
-
-unsigned FSE_optimalTableLog_internal(unsigned maxTableLog, size_t srcSize, unsigned maxSymbolValue, unsigned minus);
-/**< same as FSE_optimalTableLog(), which used `minus==2` */
-
-size_t FSE_buildCTable_rle (FSE_CTable* ct, unsigned char symbolValue);
-/**< build a fake FSE_CTable, designed to compress always the same symbolValue */
-
-/* FSE_buildCTable_wksp() :
- * Same as FSE_buildCTable(), but using an externally allocated scratch buffer (`workSpace`).
- * `wkspSize` must be >= `FSE_BUILD_CTABLE_WORKSPACE_SIZE_U32(maxSymbolValue, tableLog)` of `unsigned`.
- * See FSE_buildCTable_wksp() for breakdown of workspace usage.
- */
-#define FSE_BUILD_CTABLE_WORKSPACE_SIZE_U32(maxSymbolValue, tableLog) (((maxSymbolValue + 2) + (1ull << (tableLog)))/2 + sizeof(U64)/sizeof(U32) /* additional 8 bytes for potential table overwrite */)
-#define FSE_BUILD_CTABLE_WORKSPACE_SIZE(maxSymbolValue, tableLog) (sizeof(unsigned) * FSE_BUILD_CTABLE_WORKSPACE_SIZE_U32(maxSymbolValue, tableLog))
-size_t FSE_buildCTable_wksp(FSE_CTable* ct, const short* normalizedCounter, unsigned maxSymbolValue, unsigned tableLog, void* workSpace, size_t wkspSize);
-
-#define FSE_BUILD_DTABLE_WKSP_SIZE(maxTableLog, maxSymbolValue) (sizeof(short) * (maxSymbolValue + 1) + (1ULL << maxTableLog) + 8)
-#define FSE_BUILD_DTABLE_WKSP_SIZE_U32(maxTableLog, maxSymbolValue) ((FSE_BUILD_DTABLE_WKSP_SIZE(maxTableLog, maxSymbolValue) + sizeof(unsigned) - 1) / sizeof(unsigned))
-FSE_PUBLIC_API size_t FSE_buildDTable_wksp(FSE_DTable* dt, const short* normalizedCounter, unsigned maxSymbolValue, unsigned tableLog, void* workSpace, size_t wkspSize);
-/**< Same as FSE_buildDTable(), using an externally allocated `workspace` produced with `FSE_BUILD_DTABLE_WKSP_SIZE_U32(maxSymbolValue)` */
-
-#define FSE_DECOMPRESS_WKSP_SIZE_U32(maxTableLog, maxSymbolValue) (FSE_DTABLE_SIZE_U32(maxTableLog) + 1 + FSE_BUILD_DTABLE_WKSP_SIZE_U32(maxTableLog, maxSymbolValue) + (FSE_MAX_SYMBOL_VALUE + 1) / 2 + 1)
-#define FSE_DECOMPRESS_WKSP_SIZE(maxTableLog, maxSymbolValue) (FSE_DECOMPRESS_WKSP_SIZE_U32(maxTableLog, maxSymbolValue) * sizeof(unsigned))
-size_t FSE_decompress_wksp_bmi2(void* dst, size_t dstCapacity, const void* cSrc, size_t cSrcSize, unsigned maxLog, void* workSpace, size_t wkspSize, int bmi2);
-/**< same as FSE_decompress(), using an externally allocated `workSpace` produced with `FSE_DECOMPRESS_WKSP_SIZE_U32(maxLog, maxSymbolValue)`.
- * Set bmi2 to 1 if your CPU supports BMI2 or 0 if it doesn't */
-
-typedef enum {
-   FSE_repeat_none,  /**< Cannot use the previous table */
-   FSE_repeat_check, /**< Can use the previous table but it must be checked */
-   FSE_repeat_valid  /**< Can use the previous table and it is assumed to be valid */
- } FSE_repeat;
-
-/* *****************************************
-*  FSE symbol compression API
-*******************************************/
-/*!
-   This API consists of small unitary functions, which highly benefit from being inlined.
-   Hence their body are included in next section.
-*/
-typedef struct {
-    ptrdiff_t   value;
-    const void* stateTable;
-    const void* symbolTT;
-    unsigned    stateLog;
-} FSE_CState_t;
-
-static void FSE_initCState(FSE_CState_t* CStatePtr, const FSE_CTable* ct);
-
-static void FSE_encodeSymbol(BIT_CStream_t* bitC, FSE_CState_t* CStatePtr, unsigned symbol);
-
-static void FSE_flushCState(BIT_CStream_t* bitC, const FSE_CState_t* CStatePtr);
-
-/**<
-These functions are inner components of FSE_compress_usingCTable().
-They allow the creation of custom streams, mixing multiple tables and bit sources.
-
-A key property to keep in mind is that encoding and decoding are done **in reverse direction**.
-So the first symbol you will encode is the last you will decode, like a LIFO stack.
-
-You will need a few variables to track your CStream. They are :
-
-FSE_CTable    ct;         // Provided by FSE_buildCTable()
-BIT_CStream_t bitStream;  // bitStream tracking structure
-FSE_CState_t  state;      // State tracking structure (can have several)
-
-
-The first thing to do is to init bitStream and state.
-    size_t errorCode = BIT_initCStream(&bitStream, dstBuffer, maxDstSize);
-    FSE_initCState(&state, ct);
-
-Note that BIT_initCStream() can produce an error code, so its result should be tested, using FSE_isError();
-You can then encode your input data, byte after byte.
-FSE_encodeSymbol() outputs a maximum of 'tableLog' bits at a time.
-Remember decoding will be done in reverse direction.
-    FSE_encodeByte(&bitStream, &state, symbol);
-
-At any time, you can also add any bit sequence.
-Note : maximum allowed nbBits is 25, for compatibility with 32-bits decoders
-    BIT_addBits(&bitStream, bitField, nbBits);
-
-The above methods don't commit data to memory, they just store it into local register, for speed.
-Local register size is 64-bits on 64-bits systems, 32-bits on 32-bits systems (size_t).
-Writing data to memory is a manual operation, performed by the flushBits function.
-    BIT_flushBits(&bitStream);
-
-Your last FSE encoding operation shall be to flush your last state value(s).
-    FSE_flushState(&bitStream, &state);
-
-Finally, you must close the bitStream.
-The function returns the size of CStream in bytes.
-If data couldn't fit into dstBuffer, it will return a 0 ( == not compressible)
-If there is an error, it returns an errorCode (which can be tested using FSE_isError()).
-    size_t size = BIT_closeCStream(&bitStream);
-*/
-
-
-/* *****************************************
-*  FSE symbol decompression API
-*******************************************/
-typedef struct {
-    size_t      state;
-    const void* table;   /* precise table may vary, depending on U16 */
-} FSE_DState_t;
-
-
-static void     FSE_initDState(FSE_DState_t* DStatePtr, BIT_DStream_t* bitD, const FSE_DTable* dt);
-
-static unsigned char FSE_decodeSymbol(FSE_DState_t* DStatePtr, BIT_DStream_t* bitD);
-
-static unsigned FSE_endOfDState(const FSE_DState_t* DStatePtr);
-
-/**<
-Let's now decompose FSE_decompress_usingDTable() into its unitary components.
-You will decode FSE-encoded symbols from the bitStream,
-and also any other bitFields you put in, **in reverse order**.
-
-You will need a few variables to track your bitStream. They are :
-
-BIT_DStream_t DStream;    // Stream context
-FSE_DState_t  DState;     // State context. Multiple ones are possible
-FSE_DTable*   DTablePtr;  // Decoding table, provided by FSE_buildDTable()
-
-The first thing to do is to init the bitStream.
-    errorCode = BIT_initDStream(&DStream, srcBuffer, srcSize);
-
-You should then retrieve your initial state(s)
-(in reverse flushing order if you have several ones) :
-    errorCode = FSE_initDState(&DState, &DStream, DTablePtr);
-
-You can then decode your data, symbol after symbol.
-For information the maximum number of bits read by FSE_decodeSymbol() is 'tableLog'.
-Keep in mind that symbols are decoded in reverse order, like a LIFO stack (last in, first out).
-    unsigned char symbol = FSE_decodeSymbol(&DState, &DStream);
-
-You can retrieve any bitfield you eventually stored into the bitStream (in reverse order)
-Note : maximum allowed nbBits is 25, for 32-bits compatibility
-    size_t bitField = BIT_readBits(&DStream, nbBits);
-
-All above operations only read from local register (which size depends on size_t).
-Refueling the register from memory is manually performed by the reload method.
-    endSignal = FSE_reloadDStream(&DStream);
-
-BIT_reloadDStream() result tells if there is still some more data to read from DStream.
-BIT_DStream_unfinished : there is still some data left into the DStream.
-BIT_DStream_endOfBuffer : Dstream reached end of buffer. Its container may no longer be completely filled.
-BIT_DStream_completed : Dstream reached its exact end, corresponding in general to decompression completed.
-BIT_DStream_tooFar : Dstream went too far. Decompression result is corrupted.
-
-When reaching end of buffer (BIT_DStream_endOfBuffer), progress slowly, notably if you decode multiple symbols per loop,
-to properly detect the exact end of stream.
-After each decoded symbol, check if DStream is fully consumed using this simple test :
-    BIT_reloadDStream(&DStream) >= BIT_DStream_completed
-
-When it's done, verify decompression is fully completed, by checking both DStream and the relevant states.
-Checking if DStream has reached its end is performed by :
-    BIT_endOfDStream(&DStream);
-Check also the states. There might be some symbols left there, if some high probability ones (>50%) are possible.
-    FSE_endOfDState(&DState);
-*/
-
-
-/* *****************************************
-*  FSE unsafe API
-*******************************************/
-static unsigned char FSE_decodeSymbolFast(FSE_DState_t* DStatePtr, BIT_DStream_t* bitD);
-/* faster, but works only if nbBits is always >= 1 (otherwise, result will be corrupted) */
-
-
-/* *****************************************
-*  Implementation of inlined functions
-*******************************************/
-typedef struct {
-    int deltaFindState;
-    U32 deltaNbBits;
-} FSE_symbolCompressionTransform; /* total 8 bytes */
-
-MEM_STATIC void FSE_initCState(FSE_CState_t* statePtr, const FSE_CTable* ct)
-{
-    const void* ptr = ct;
-    const U16* u16ptr = (const U16*) ptr;
-    const U32 tableLog = MEM_read16(ptr);
-    statePtr->value = (ptrdiff_t)1<<tableLog;
-    statePtr->stateTable = u16ptr+2;
-    statePtr->symbolTT = ct + 1 + (tableLog ? (1<<(tableLog-1)) : 1);
-    statePtr->stateLog = tableLog;
-}
-
-
-/*! FSE_initCState2() :
-*   Same as FSE_initCState(), but the first symbol to include (which will be the last to be read)
-*   uses the smallest state value possible, saving the cost of this symbol */
-MEM_STATIC void FSE_initCState2(FSE_CState_t* statePtr, const FSE_CTable* ct, U32 symbol)
-{
-    FSE_initCState(statePtr, ct);
-    {   const FSE_symbolCompressionTransform symbolTT = ((const FSE_symbolCompressionTransform*)(statePtr->symbolTT))[symbol];
-        const U16* stateTable = (const U16*)(statePtr->stateTable);
-        U32 nbBitsOut  = (U32)((symbolTT.deltaNbBits + (1<<15)) >> 16);
-        statePtr->value = (nbBitsOut << 16) - symbolTT.deltaNbBits;
-        statePtr->value = stateTable[(statePtr->value >> nbBitsOut) + symbolTT.deltaFindState];
-    }
-}
-
-MEM_STATIC void FSE_encodeSymbol(BIT_CStream_t* bitC, FSE_CState_t* statePtr, unsigned symbol)
-{
-    FSE_symbolCompressionTransform const symbolTT = ((const FSE_symbolCompressionTransform*)(statePtr->symbolTT))[symbol];
-    const U16* const stateTable = (const U16*)(statePtr->stateTable);
-    U32 const nbBitsOut  = (U32)((statePtr->value + symbolTT.deltaNbBits) >> 16);
-    BIT_addBits(bitC, (BitContainerType)statePtr->value, nbBitsOut);
-    statePtr->value = stateTable[ (statePtr->value >> nbBitsOut) + symbolTT.deltaFindState];
-}
-
-MEM_STATIC void FSE_flushCState(BIT_CStream_t* bitC, const FSE_CState_t* statePtr)
-{
-    BIT_addBits(bitC, (BitContainerType)statePtr->value, statePtr->stateLog);
-    BIT_flushBits(bitC);
-}
-
-
-/* FSE_getMaxNbBits() :
- * Approximate maximum cost of a symbol, in bits.
- * Fractional get rounded up (i.e. a symbol with a normalized frequency of 3 gives the same result as a frequency of 2)
- * note 1 : assume symbolValue is valid (<= maxSymbolValue)
- * note 2 : if freq[symbolValue]==0, @return a fake cost of tableLog+1 bits */
-MEM_STATIC U32 FSE_getMaxNbBits(const void* symbolTTPtr, U32 symbolValue)
-{
-    const FSE_symbolCompressionTransform* symbolTT = (const FSE_symbolCompressionTransform*) symbolTTPtr;
-    return (symbolTT[symbolValue].deltaNbBits + ((1<<16)-1)) >> 16;
-}
-
-/* FSE_bitCost() :
- * Approximate symbol cost, as fractional value, using fixed-point format (accuracyLog fractional bits)
- * note 1 : assume symbolValue is valid (<= maxSymbolValue)
- * note 2 : if freq[symbolValue]==0, @return a fake cost of tableLog+1 bits */
-MEM_STATIC U32 FSE_bitCost(const void* symbolTTPtr, U32 tableLog, U32 symbolValue, U32 accuracyLog)
-{
-    const FSE_symbolCompressionTransform* symbolTT = (const FSE_symbolCompressionTransform*) symbolTTPtr;
-    U32 const minNbBits = symbolTT[symbolValue].deltaNbBits >> 16;
-    U32 const threshold = (minNbBits+1) << 16;
-    assert(tableLog < 16);
-    assert(accuracyLog < 31-tableLog);  /* ensure enough room for renormalization double shift */
-    {   U32 const tableSize = 1 << tableLog;
-        U32 const deltaFromThreshold = threshold - (symbolTT[symbolValue].deltaNbBits + tableSize);
-        U32 const normalizedDeltaFromThreshold = (deltaFromThreshold << accuracyLog) >> tableLog;   /* linear interpolation (very approximate) */
-        U32 const bitMultiplier = 1 << accuracyLog;
-        assert(symbolTT[symbolValue].deltaNbBits + tableSize <= threshold);
-        assert(normalizedDeltaFromThreshold <= bitMultiplier);
-        return (minNbBits+1)*bitMultiplier - normalizedDeltaFromThreshold;
-    }
-}
-
-
-/* ======    Decompression    ====== */
-
-typedef struct {
-    U16 tableLog;
-    U16 fastMode;
-} FSE_DTableHeader;   /* sizeof U32 */
-
-typedef struct
-{
-    unsigned short newState;
-    unsigned char  symbol;
-    unsigned char  nbBits;
-} FSE_decode_t;   /* size == U32 */
-
-MEM_STATIC void FSE_initDState(FSE_DState_t* DStatePtr, BIT_DStream_t* bitD, const FSE_DTable* dt)
-{
-    const void* ptr = dt;
-    const FSE_DTableHeader* const DTableH = (const FSE_DTableHeader*)ptr;
-    DStatePtr->state = BIT_readBits(bitD, DTableH->tableLog);
-    BIT_reloadDStream(bitD);
-    DStatePtr->table = dt + 1;
-}
-
-MEM_STATIC BYTE FSE_peekSymbol(const FSE_DState_t* DStatePtr)
-{
-    FSE_decode_t const DInfo = ((const FSE_decode_t*)(DStatePtr->table))[DStatePtr->state];
-    return DInfo.symbol;
-}
-
-MEM_STATIC void FSE_updateState(FSE_DState_t* DStatePtr, BIT_DStream_t* bitD)
-{
-    FSE_decode_t const DInfo = ((const FSE_decode_t*)(DStatePtr->table))[DStatePtr->state];
-    U32 const nbBits = DInfo.nbBits;
-    size_t const lowBits = BIT_readBits(bitD, nbBits);
-    DStatePtr->state = DInfo.newState + lowBits;
-}
-
-MEM_STATIC BYTE FSE_decodeSymbol(FSE_DState_t* DStatePtr, BIT_DStream_t* bitD)
-{
-    FSE_decode_t const DInfo = ((const FSE_decode_t*)(DStatePtr->table))[DStatePtr->state];
-    U32 const nbBits = DInfo.nbBits;
-    BYTE const symbol = DInfo.symbol;
-    size_t const lowBits = BIT_readBits(bitD, nbBits);
-
-    DStatePtr->state = DInfo.newState + lowBits;
-    return symbol;
-}
-
-/*! FSE_decodeSymbolFast() :
-    unsafe, only works if no symbol has a probability > 50% */
-MEM_STATIC BYTE FSE_decodeSymbolFast(FSE_DState_t* DStatePtr, BIT_DStream_t* bitD)
-{
-    FSE_decode_t const DInfo = ((const FSE_decode_t*)(DStatePtr->table))[DStatePtr->state];
-    U32 const nbBits = DInfo.nbBits;
-    BYTE const symbol = DInfo.symbol;
-    size_t const lowBits = BIT_readBitsFast(bitD, nbBits);
-
-    DStatePtr->state = DInfo.newState + lowBits;
-    return symbol;
-}
-
-MEM_STATIC unsigned FSE_endOfDState(const FSE_DState_t* DStatePtr)
-{
-    return DStatePtr->state == 0;
-}
-
-
-
-#ifndef FSE_COMMONDEFS_ONLY
-
-/* **************************************************************
-*  Tuning parameters
-****************************************************************/
-/*!MEMORY_USAGE :
-*  Memory usage formula : N->2^N Bytes (examples : 10 -> 1KB; 12 -> 4KB ; 16 -> 64KB; 20 -> 1MB; etc.)
-*  Increasing memory usage improves compression ratio
-*  Reduced memory usage can improve speed, due to cache effect
-*  Recommended max value is 14, for 16KB, which nicely fits into Intel x86 L1 cache */
-#ifndef FSE_MAX_MEMORY_USAGE
-#  define FSE_MAX_MEMORY_USAGE 14
-#endif
-#ifndef FSE_DEFAULT_MEMORY_USAGE
-#  define FSE_DEFAULT_MEMORY_USAGE 13
-#endif
-#if (FSE_DEFAULT_MEMORY_USAGE > FSE_MAX_MEMORY_USAGE)
-#  error "FSE_DEFAULT_MEMORY_USAGE must be <= FSE_MAX_MEMORY_USAGE"
-#endif
-
-/*!FSE_MAX_SYMBOL_VALUE :
-*  Maximum symbol value authorized.
-*  Required for proper stack allocation */
-#ifndef FSE_MAX_SYMBOL_VALUE
-#  define FSE_MAX_SYMBOL_VALUE 255
-#endif
-
-/* **************************************************************
-*  template functions type & suffix
-****************************************************************/
-#define FSE_FUNCTION_TYPE BYTE
-#define FSE_FUNCTION_EXTENSION
-#define FSE_DECODE_TYPE FSE_decode_t
-
-
-#endif   /* !FSE_COMMONDEFS_ONLY */
-
-
-/* ***************************************************************
-*  Constants
-*****************************************************************/
-#define FSE_MAX_TABLELOG  (FSE_MAX_MEMORY_USAGE-2)
-#define FSE_MAX_TABLESIZE (1U<<FSE_MAX_TABLELOG)
-#define FSE_MAXTABLESIZE_MASK (FSE_MAX_TABLESIZE-1)
-#define FSE_DEFAULT_TABLELOG (FSE_DEFAULT_MEMORY_USAGE-2)
-#define FSE_MIN_TABLELOG 5
-
-#define FSE_TABLELOG_ABSOLUTE_MAX 15
-#if FSE_MAX_TABLELOG > FSE_TABLELOG_ABSOLUTE_MAX
-#  error "FSE_MAX_TABLELOG > FSE_TABLELOG_ABSOLUTE_MAX is not supported"
-#endif
-
-#define FSE_TABLESTEP(tableSize) (((tableSize)>>1) + ((tableSize)>>3) + 3)
-
-#endif /* FSE_STATIC_LINKING_ONLY */
-
-#endif /* USE_EXTERNAL_ZSTD */

vendor/github.com/DataDog/zstd/fse_compress.c

@@ -1,628 +0,0 @@
-#ifndef USE_EXTERNAL_ZSTD
-/* ******************************************************************
- * FSE : Finite State Entropy encoder
- * Copyright (c) Meta Platforms, Inc. and affiliates.
- *
- *  You can contact the author at :
- *  - FSE source repository : https://github.com/Cyan4973/FiniteStateEntropy
- *  - Public forum : https://groups.google.com/forum/#!forum/lz4c
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- * You may select, at your option, one of the above-listed licenses.
-****************************************************************** */
-
-/* **************************************************************
-*  Includes
-****************************************************************/
-#include "compiler.h"
-#include "mem.h"        /* U32, U16, etc. */
-#include "debug.h"      /* assert, DEBUGLOG */
-#include "hist.h"       /* HIST_count_wksp */
-#include "bitstream.h"
-#define FSE_STATIC_LINKING_ONLY
-#include "fse.h"
-#include "error_private.h"
-#define ZSTD_DEPS_NEED_MALLOC
-#define ZSTD_DEPS_NEED_MATH64
-#include "zstd_deps.h"  /* ZSTD_memset */
-#include "bits.h" /* ZSTD_highbit32 */
-
-
-/* **************************************************************
-*  Error Management
-****************************************************************/
-#define FSE_isError ERR_isError
-
-
-/* **************************************************************
-*  Templates
-****************************************************************/
-/*
-  designed to be included
-  for type-specific functions (template emulation in C)
-  Objective is to write these functions only once, for improved maintenance
-*/
-
-/* safety checks */
-#ifndef FSE_FUNCTION_EXTENSION
-#  error "FSE_FUNCTION_EXTENSION must be defined"
-#endif
-#ifndef FSE_FUNCTION_TYPE
-#  error "FSE_FUNCTION_TYPE must be defined"
-#endif
-
-/* Function names */
-#define FSE_CAT(X,Y) X##Y
-#define FSE_FUNCTION_NAME(X,Y) FSE_CAT(X,Y)
-#define FSE_TYPE_NAME(X,Y) FSE_CAT(X,Y)
-
-
-/* Function templates */
-
-/* FSE_buildCTable_wksp() :
- * Same as FSE_buildCTable(), but using an externally allocated scratch buffer (`workSpace`).
- * wkspSize should be sized to handle worst case situation, which is `1<<max_tableLog * sizeof(FSE_FUNCTION_TYPE)`
- * workSpace must also be properly aligned with FSE_FUNCTION_TYPE requirements
- */
-size_t FSE_buildCTable_wksp(FSE_CTable* ct,
-                      const short* normalizedCounter, unsigned maxSymbolValue, unsigned tableLog,
-                            void* workSpace, size_t wkspSize)
-{
-    U32 const tableSize = 1 << tableLog;
-    U32 const tableMask = tableSize - 1;
-    void* const ptr = ct;
-    U16* const tableU16 = ( (U16*) ptr) + 2;
-    void* const FSCT = ((U32*)ptr) + 1 /* header */ + (tableLog ? tableSize>>1 : 1) ;
-    FSE_symbolCompressionTransform* const symbolTT = (FSE_symbolCompressionTransform*) (FSCT);
-    U32 const step = FSE_TABLESTEP(tableSize);
-    U32 const maxSV1 = maxSymbolValue+1;
-
-    U16* cumul = (U16*)workSpace;   /* size = maxSV1 */
-    FSE_FUNCTION_TYPE* const tableSymbol = (FSE_FUNCTION_TYPE*)(cumul + (maxSV1+1));  /* size = tableSize */
-
-    U32 highThreshold = tableSize-1;
-
-    assert(((size_t)workSpace & 1) == 0);  /* Must be 2 bytes-aligned */
-    if (FSE_BUILD_CTABLE_WORKSPACE_SIZE(maxSymbolValue, tableLog) > wkspSize) return ERROR(tableLog_tooLarge);
-    /* CTable header */
-    tableU16[-2] = (U16) tableLog;
-    tableU16[-1] = (U16) maxSymbolValue;
-    assert(tableLog < 16);   /* required for threshold strategy to work */
-
-    /* For explanations on how to distribute symbol values over the table :
-     * https://fastcompression.blogspot.fr/2014/02/fse-distributing-symbol-values.html */
-
-     #ifdef __clang_analyzer__
-     ZSTD_memset(tableSymbol, 0, sizeof(*tableSymbol) * tableSize);   /* useless initialization, just to keep scan-build happy */
-     #endif
-
-    /* symbol start positions */
-    {   U32 u;
-        cumul[0] = 0;
-        for (u=1; u <= maxSV1; u++) {
-            if (normalizedCounter[u-1]==-1) {  /* Low proba symbol */
-                cumul[u] = cumul[u-1] + 1;
-                tableSymbol[highThreshold--] = (FSE_FUNCTION_TYPE)(u-1);
-            } else {
-                assert(normalizedCounter[u-1] >= 0);
-                cumul[u] = cumul[u-1] + (U16)normalizedCounter[u-1];
-                assert(cumul[u] >= cumul[u-1]);  /* no overflow */
-        }   }
-        cumul[maxSV1] = (U16)(tableSize+1);
-    }
-
-    /* Spread symbols */
-    if (highThreshold == tableSize - 1) {
-        /* Case for no low prob count symbols. Lay down 8 bytes at a time
-         * to reduce branch misses since we are operating on a small block
-         */
-        BYTE* const spread = tableSymbol + tableSize; /* size = tableSize + 8 (may write beyond tableSize) */
-        {   U64 const add = 0x0101010101010101ull;
-            size_t pos = 0;
-            U64 sv = 0;
-            U32 s;
-            for (s=0; s<maxSV1; ++s, sv += add) {
-                int i;
-                int const n = normalizedCounter[s];
-                MEM_write64(spread + pos, sv);
-                for (i = 8; i < n; i += 8) {
-                    MEM_write64(spread + pos + i, sv);
-                }
-                assert(n>=0);
-                pos += (size_t)n;
-            }
-        }
-        /* Spread symbols across the table. Lack of lowprob symbols means that
-         * we don't need variable sized inner loop, so we can unroll the loop and
-         * reduce branch misses.
-         */
-        {   size_t position = 0;
-            size_t s;
-            size_t const unroll = 2; /* Experimentally determined optimal unroll */
-            assert(tableSize % unroll == 0); /* FSE_MIN_TABLELOG is 5 */
-            for (s = 0; s < (size_t)tableSize; s += unroll) {
-                size_t u;
-                for (u = 0; u < unroll; ++u) {
-                    size_t const uPosition = (position + (u * step)) & tableMask;
-                    tableSymbol[uPosition] = spread[s + u];
-                }
-                position = (position + (unroll * step)) & tableMask;
-            }
-            assert(position == 0);   /* Must have initialized all positions */
-        }
-    } else {
-        U32 position = 0;
-        U32 symbol;
-        for (symbol=0; symbol<maxSV1; symbol++) {
-            int nbOccurrences;
-            int const freq = normalizedCounter[symbol];
-            for (nbOccurrences=0; nbOccurrences<freq; nbOccurrences++) {
-                tableSymbol[position] = (FSE_FUNCTION_TYPE)symbol;
-                position = (position + step) & tableMask;
-                while (position > highThreshold)
-                    position = (position + step) & tableMask;   /* Low proba area */
-        }   }
-        assert(position==0);  /* Must have initialized all positions */
-    }
-
-    /* Build table */
-    {   U32 u; for (u=0; u<tableSize; u++) {
-        FSE_FUNCTION_TYPE s = tableSymbol[u];   /* note : static analyzer may not understand tableSymbol is properly initialized */
-        tableU16[cumul[s]++] = (U16) (tableSize+u);   /* TableU16 : sorted by symbol order; gives next state value */
-    }   }
-
-    /* Build Symbol Transformation Table */
-    {   unsigned total = 0;
-        unsigned s;
-        for (s=0; s<=maxSymbolValue; s++) {
-            switch (normalizedCounter[s])
-            {
-            case  0:
-                /* filling nonetheless, for compatibility with FSE_getMaxNbBits() */
-                symbolTT[s].deltaNbBits = ((tableLog+1) << 16) - (1<<tableLog);
-                break;
-
-            case -1:
-            case  1:
-                symbolTT[s].deltaNbBits = (tableLog << 16) - (1<<tableLog);
-                assert(total <= INT_MAX);
-                symbolTT[s].deltaFindState = (int)(total - 1);
-                total ++;
-                break;
-            default :
-                assert(normalizedCounter[s] > 1);
-                {   U32 const maxBitsOut = tableLog - ZSTD_highbit32 ((U32)normalizedCounter[s]-1);
-                    U32 const minStatePlus = (U32)normalizedCounter[s] << maxBitsOut;
-                    symbolTT[s].deltaNbBits = (maxBitsOut << 16) - minStatePlus;
-                    symbolTT[s].deltaFindState = (int)(total - (unsigned)normalizedCounter[s]);
-                    total +=  (unsigned)normalizedCounter[s];
-    }   }   }   }
-
-#if 0  /* debug : symbol costs */
-    DEBUGLOG(5, "\n --- table statistics : ");
-    {   U32 symbol;
-        for (symbol=0; symbol<=maxSymbolValue; symbol++) {
-            DEBUGLOG(5, "%3u: w=%3i,   maxBits=%u, fracBits=%.2f",
-                symbol, normalizedCounter[symbol],
-                FSE_getMaxNbBits(symbolTT, symbol),
-                (double)FSE_bitCost(symbolTT, tableLog, symbol, 8) / 256);
-    }   }
-#endif
-
-    return 0;
-}
-
-
-
-#ifndef FSE_COMMONDEFS_ONLY
-
-/*-**************************************************************
-*  FSE NCount encoding
-****************************************************************/
-size_t FSE_NCountWriteBound(unsigned maxSymbolValue, unsigned tableLog)
-{
-    size_t const maxHeaderSize = (((maxSymbolValue+1) * tableLog
-                                   + 4 /* bitCount initialized at 4 */
-                                   + 2 /* first two symbols may use one additional bit each */) / 8)
-                                   + 1 /* round up to whole nb bytes */
-                                   + 2 /* additional two bytes for bitstream flush */;
-    return maxSymbolValue ? maxHeaderSize : FSE_NCOUNTBOUND;  /* maxSymbolValue==0 ? use default */
-}
-
-static size_t
-FSE_writeNCount_generic (void* header, size_t headerBufferSize,
-                   const short* normalizedCounter, unsigned maxSymbolValue, unsigned tableLog,
-                         unsigned writeIsSafe)
-{
-    BYTE* const ostart = (BYTE*) header;
-    BYTE* out = ostart;
-    BYTE* const oend = ostart + headerBufferSize;
-    int nbBits;
-    const int tableSize = 1 << tableLog;
-    int remaining;
-    int threshold;
-    U32 bitStream = 0;
-    int bitCount = 0;
-    unsigned symbol = 0;
-    unsigned const alphabetSize = maxSymbolValue + 1;
-    int previousIs0 = 0;
-
-    /* Table Size */
-    bitStream += (tableLog-FSE_MIN_TABLELOG) << bitCount;
-    bitCount  += 4;
-
-    /* Init */
-    remaining = tableSize+1;   /* +1 for extra accuracy */
-    threshold = tableSize;
-    nbBits = (int)tableLog+1;
-
-    while ((symbol < alphabetSize) && (remaining>1)) {  /* stops at 1 */
-        if (previousIs0) {
-            unsigned start = symbol;
-            while ((symbol < alphabetSize) && !normalizedCounter[symbol]) symbol++;
-            if (symbol == alphabetSize) break;   /* incorrect distribution */
-            while (symbol >= start+24) {
-                start+=24;
-                bitStream += 0xFFFFU << bitCount;
-                if ((!writeIsSafe) && (out > oend-2))
-                    return ERROR(dstSize_tooSmall);   /* Buffer overflow */
-                out[0] = (BYTE) bitStream;
-                out[1] = (BYTE)(bitStream>>8);
-                out+=2;
-                bitStream>>=16;
-            }
-            while (symbol >= start+3) {
-                start+=3;
-                bitStream += 3U << bitCount;
-                bitCount += 2;
-            }
-            bitStream += (symbol-start) << bitCount;
-            bitCount += 2;
-            if (bitCount>16) {
-                if ((!writeIsSafe) && (out > oend - 2))
-                    return ERROR(dstSize_tooSmall);   /* Buffer overflow */
-                out[0] = (BYTE)bitStream;
-                out[1] = (BYTE)(bitStream>>8);
-                out += 2;
-                bitStream >>= 16;
-                bitCount -= 16;
-        }   }
-        {   int count = normalizedCounter[symbol++];
-            int const max = (2*threshold-1) - remaining;
-            remaining -= count < 0 ? -count : count;
-            count++;   /* +1 for extra accuracy */
-            if (count>=threshold)
-                count += max;   /* [0..max[ [max..threshold[ (...) [threshold+max 2*threshold[ */
-            bitStream += (U32)count << bitCount;
-            bitCount  += nbBits;
-            bitCount  -= (count<max);
-            previousIs0  = (count==1);
-            if (remaining<1) return ERROR(GENERIC);
-            while (remaining<threshold) { nbBits--; threshold>>=1; }
-        }
-        if (bitCount>16) {
-            if ((!writeIsSafe) && (out > oend - 2))
-                return ERROR(dstSize_tooSmall);   /* Buffer overflow */
-            out[0] = (BYTE)bitStream;
-            out[1] = (BYTE)(bitStream>>8);
-            out += 2;
-            bitStream >>= 16;
-            bitCount -= 16;
-    }   }
-
-    if (remaining != 1)
-        return ERROR(GENERIC);  /* incorrect normalized distribution */
-    assert(symbol <= alphabetSize);
-
-    /* flush remaining bitStream */
-    if ((!writeIsSafe) && (out > oend - 2))
-        return ERROR(dstSize_tooSmall);   /* Buffer overflow */
-    out[0] = (BYTE)bitStream;
-    out[1] = (BYTE)(bitStream>>8);
-    out+= (bitCount+7) /8;
-
-    assert(out >= ostart);
-    return (size_t)(out-ostart);
-}
-
-
-size_t FSE_writeNCount (void* buffer, size_t bufferSize,
-                  const short* normalizedCounter, unsigned maxSymbolValue, unsigned tableLog)
-{
-    if (tableLog > FSE_MAX_TABLELOG) return ERROR(tableLog_tooLarge);   /* Unsupported */
-    if (tableLog < FSE_MIN_TABLELOG) return ERROR(GENERIC);   /* Unsupported */
-
-    if (bufferSize < FSE_NCountWriteBound(maxSymbolValue, tableLog))
-        return FSE_writeNCount_generic(buffer, bufferSize, normalizedCounter, maxSymbolValue, tableLog, 0);
-
-    return FSE_writeNCount_generic(buffer, bufferSize, normalizedCounter, maxSymbolValue, tableLog, 1 /* write in buffer is safe */);
-}
-
-
-/*-**************************************************************
-*  FSE Compression Code
-****************************************************************/
-
-/* provides the minimum logSize to safely represent a distribution */
-static unsigned FSE_minTableLog(size_t srcSize, unsigned maxSymbolValue)
-{
-    U32 minBitsSrc = ZSTD_highbit32((U32)(srcSize)) + 1;
-    U32 minBitsSymbols = ZSTD_highbit32(maxSymbolValue) + 2;
-    U32 minBits = minBitsSrc < minBitsSymbols ? minBitsSrc : minBitsSymbols;
-    assert(srcSize > 1); /* Not supported, RLE should be used instead */
-    return minBits;
-}
-
-unsigned FSE_optimalTableLog_internal(unsigned maxTableLog, size_t srcSize, unsigned maxSymbolValue, unsigned minus)
-{
-    U32 maxBitsSrc = ZSTD_highbit32((U32)(srcSize - 1)) - minus;
-    U32 tableLog = maxTableLog;
-    U32 minBits = FSE_minTableLog(srcSize, maxSymbolValue);
-    assert(srcSize > 1); /* Not supported, RLE should be used instead */
-    if (tableLog==0) tableLog = FSE_DEFAULT_TABLELOG;
-    if (maxBitsSrc < tableLog) tableLog = maxBitsSrc;   /* Accuracy can be reduced */
-    if (minBits > tableLog) tableLog = minBits;   /* Need a minimum to safely represent all symbol values */
-    if (tableLog < FSE_MIN_TABLELOG) tableLog = FSE_MIN_TABLELOG;
-    if (tableLog > FSE_MAX_TABLELOG) tableLog = FSE_MAX_TABLELOG;
-    return tableLog;
-}
-
-unsigned FSE_optimalTableLog(unsigned maxTableLog, size_t srcSize, unsigned maxSymbolValue)
-{
-    return FSE_optimalTableLog_internal(maxTableLog, srcSize, maxSymbolValue, 2);
-}
-
-/* Secondary normalization method.
-   To be used when primary method fails. */
-
-static size_t FSE_normalizeM2(short* norm, U32 tableLog, const unsigned* count, size_t total, U32 maxSymbolValue, short lowProbCount)
-{
-    short const NOT_YET_ASSIGNED = -2;
-    U32 s;
-    U32 distributed = 0;
-    U32 ToDistribute;
-
-    /* Init */
-    U32 const lowThreshold = (U32)(total >> tableLog);
-    U32 lowOne = (U32)((total * 3) >> (tableLog + 1));
-
-    for (s=0; s<=maxSymbolValue; s++) {
-        if (count[s] == 0) {
-            norm[s]=0;
-            continue;
-        }
-        if (count[s] <= lowThreshold) {
-            norm[s] = lowProbCount;
-            distributed++;
-            total -= count[s];
-            continue;
-        }
-        if (count[s] <= lowOne) {
-            norm[s] = 1;
-            distributed++;
-            total -= count[s];
-            continue;
-        }
-
-        norm[s]=NOT_YET_ASSIGNED;
-    }
-    ToDistribute = (1 << tableLog) - distributed;
-
-    if (ToDistribute == 0)
-        return 0;
-
-    if ((total / ToDistribute) > lowOne) {
-        /* risk of rounding to zero */
-        lowOne = (U32)((total * 3) / (ToDistribute * 2));
-        for (s=0; s<=maxSymbolValue; s++) {
-            if ((norm[s] == NOT_YET_ASSIGNED) && (count[s] <= lowOne)) {
-                norm[s] = 1;
-                distributed++;
-                total -= count[s];
-                continue;
-        }   }
-        ToDistribute = (1 << tableLog) - distributed;
-    }
-
-    if (distributed == maxSymbolValue+1) {
-        /* all values are pretty poor;
-           probably incompressible data (should have already been detected);
-           find max, then give all remaining points to max */
-        U32 maxV = 0, maxC = 0;
-        for (s=0; s<=maxSymbolValue; s++)
-            if (count[s] > maxC) { maxV=s; maxC=count[s]; }
-        norm[maxV] += (short)ToDistribute;
-        return 0;
-    }
-
-    if (total == 0) {
-        /* all of the symbols were low enough for the lowOne or lowThreshold */
-        for (s=0; ToDistribute > 0; s = (s+1)%(maxSymbolValue+1))
-            if (norm[s] > 0) { ToDistribute--; norm[s]++; }
-        return 0;
-    }
-
-    {   U64 const vStepLog = 62 - tableLog;
-        U64 const mid = (1ULL << (vStepLog-1)) - 1;
-        U64 const rStep = ZSTD_div64((((U64)1<<vStepLog) * ToDistribute) + mid, (U32)total);   /* scale on remaining */
-        U64 tmpTotal = mid;
-        for (s=0; s<=maxSymbolValue; s++) {
-            if (norm[s]==NOT_YET_ASSIGNED) {
-                U64 const end = tmpTotal + (count[s] * rStep);
-                U32 const sStart = (U32)(tmpTotal >> vStepLog);
-                U32 const sEnd = (U32)(end >> vStepLog);
-                U32 const weight = sEnd - sStart;
-                if (weight < 1)
-                    return ERROR(GENERIC);
-                norm[s] = (short)weight;
-                tmpTotal = end;
-    }   }   }
-
-    return 0;
-}
-
-size_t FSE_normalizeCount (short* normalizedCounter, unsigned tableLog,
-                           const unsigned* count, size_t total,
-                           unsigned maxSymbolValue, unsigned useLowProbCount)
-{
-    /* Sanity checks */
-    if (tableLog==0) tableLog = FSE_DEFAULT_TABLELOG;
-    if (tableLog < FSE_MIN_TABLELOG) return ERROR(GENERIC);   /* Unsupported size */
-    if (tableLog > FSE_MAX_TABLELOG) return ERROR(tableLog_tooLarge);   /* Unsupported size */
-    if (tableLog < FSE_minTableLog(total, maxSymbolValue)) return ERROR(GENERIC);   /* Too small tableLog, compression potentially impossible */
-
-    {   static U32 const rtbTable[] = {     0, 473195, 504333, 520860, 550000, 700000, 750000, 830000 };
-        short const lowProbCount = useLowProbCount ? -1 : 1;
-        U64 const scale = 62 - tableLog;
-        U64 const step = ZSTD_div64((U64)1<<62, (U32)total);   /* <== here, one division ! */
-        U64 const vStep = 1ULL<<(scale-20);
-        int stillToDistribute = 1<<tableLog;
-        unsigned s;
-        unsigned largest=0;
-        short largestP=0;
-        U32 lowThreshold = (U32)(total >> tableLog);
-
-        for (s=0; s<=maxSymbolValue; s++) {
-            if (count[s] == total) return 0;   /* rle special case */
-            if (count[s] == 0) { normalizedCounter[s]=0; continue; }
-            if (count[s] <= lowThreshold) {
-                normalizedCounter[s] = lowProbCount;
-                stillToDistribute--;
-            } else {
-                short proba = (short)((count[s]*step) >> scale);
-                if (proba<8) {
-                    U64 restToBeat = vStep * rtbTable[proba];
-                    proba += (count[s]*step) - ((U64)proba<<scale) > restToBeat;
-                }
-                if (proba > largestP) { largestP=proba; largest=s; }
-                normalizedCounter[s] = proba;
-                stillToDistribute -= proba;
-        }   }
-        if (-stillToDistribute >= (normalizedCounter[largest] >> 1)) {
-            /* corner case, need another normalization method */
-            size_t const errorCode = FSE_normalizeM2(normalizedCounter, tableLog, count, total, maxSymbolValue, lowProbCount);
-            if (FSE_isError(errorCode)) return errorCode;
-        }
-        else normalizedCounter[largest] += (short)stillToDistribute;
-    }
-
-#if 0
-    {   /* Print Table (debug) */
-        U32 s;
-        U32 nTotal = 0;
-        for (s=0; s<=maxSymbolValue; s++)
-            RAWLOG(2, "%3i: %4i \n", s, normalizedCounter[s]);
-        for (s=0; s<=maxSymbolValue; s++)
-            nTotal += abs(normalizedCounter[s]);
-        if (nTotal != (1U<<tableLog))
-            RAWLOG(2, "Warning !!! Total == %u != %u !!!", nTotal, 1U<<tableLog);
-        getchar();
-    }
-#endif
-
-    return tableLog;
-}
-
-/* fake FSE_CTable, for rle input (always same symbol) */
-size_t FSE_buildCTable_rle (FSE_CTable* ct, BYTE symbolValue)
-{
-    void* ptr = ct;
-    U16* tableU16 = ( (U16*) ptr) + 2;
-    void* FSCTptr = (U32*)ptr + 2;
-    FSE_symbolCompressionTransform* symbolTT = (FSE_symbolCompressionTransform*) FSCTptr;
-
-    /* header */
-    tableU16[-2] = (U16) 0;
-    tableU16[-1] = (U16) symbolValue;
-
-    /* Build table */
-    tableU16[0] = 0;
-    tableU16[1] = 0;   /* just in case */
-
-    /* Build Symbol Transformation Table */
-    symbolTT[symbolValue].deltaNbBits = 0;
-    symbolTT[symbolValue].deltaFindState = 0;
-
-    return 0;
-}
-
-
-static size_t FSE_compress_usingCTable_generic (void* dst, size_t dstSize,
-                           const void* src, size_t srcSize,
-                           const FSE_CTable* ct, const unsigned fast)
-{
-    const BYTE* const istart = (const BYTE*) src;
-    const BYTE* const iend = istart + srcSize;
-    const BYTE* ip=iend;
-
-    BIT_CStream_t bitC;
-    FSE_CState_t CState1, CState2;
-
-    /* init */
-    if (srcSize <= 2) return 0;
-    { size_t const initError = BIT_initCStream(&bitC, dst, dstSize);
-      if (FSE_isError(initError)) return 0; /* not enough space available to write a bitstream */ }
-
-#define FSE_FLUSHBITS(s)  (fast ? BIT_flushBitsFast(s) : BIT_flushBits(s))
-
-    if (srcSize & 1) {
-        FSE_initCState2(&CState1, ct, *--ip);
-        FSE_initCState2(&CState2, ct, *--ip);
-        FSE_encodeSymbol(&bitC, &CState1, *--ip);
-        FSE_FLUSHBITS(&bitC);
-    } else {
-        FSE_initCState2(&CState2, ct, *--ip);
-        FSE_initCState2(&CState1, ct, *--ip);
-    }
-
-    /* join to mod 4 */
-    srcSize -= 2;
-    if ((sizeof(bitC.bitContainer)*8 > FSE_MAX_TABLELOG*4+7 ) && (srcSize & 2)) {  /* test bit 2 */
-        FSE_encodeSymbol(&bitC, &CState2, *--ip);
-        FSE_encodeSymbol(&bitC, &CState1, *--ip);
-        FSE_FLUSHBITS(&bitC);
-    }
-
-    /* 2 or 4 encoding per loop */
-    while ( ip>istart ) {
-
-        FSE_encodeSymbol(&bitC, &CState2, *--ip);
-
-        if (sizeof(bitC.bitContainer)*8 < FSE_MAX_TABLELOG*2+7 )   /* this test must be static */
-            FSE_FLUSHBITS(&bitC);
-
-        FSE_encodeSymbol(&bitC, &CState1, *--ip);
-
-        if (sizeof(bitC.bitContainer)*8 > FSE_MAX_TABLELOG*4+7 ) {  /* this test must be static */
-            FSE_encodeSymbol(&bitC, &CState2, *--ip);
-            FSE_encodeSymbol(&bitC, &CState1, *--ip);
-        }
-
-        FSE_FLUSHBITS(&bitC);
-    }
-
-    FSE_flushCState(&bitC, &CState2);
-    FSE_flushCState(&bitC, &CState1);
-    return BIT_closeCStream(&bitC);
-}
-
-size_t FSE_compress_usingCTable (void* dst, size_t dstSize,
-                           const void* src, size_t srcSize,
-                           const FSE_CTable* ct)
-{
-    unsigned const fast = (dstSize >= FSE_BLOCKBOUND(srcSize));
-
-    if (fast)
-        return FSE_compress_usingCTable_generic(dst, dstSize, src, srcSize, ct, 1);
-    else
-        return FSE_compress_usingCTable_generic(dst, dstSize, src, srcSize, ct, 0);
-}
-
-
-size_t FSE_compressBound(size_t size) { return FSE_COMPRESSBOUND(size); }
-
-#endif   /* FSE_COMMONDEFS_ONLY */
-
-#endif /* USE_EXTERNAL_ZSTD */

vendor/github.com/DataDog/zstd/fse_decompress.c

@@ -1,318 +0,0 @@
-#ifndef USE_EXTERNAL_ZSTD
-/* ******************************************************************
- * FSE : Finite State Entropy decoder
- * Copyright (c) Meta Platforms, Inc. and affiliates.
- *
- *  You can contact the author at :
- *  - FSE source repository : https://github.com/Cyan4973/FiniteStateEntropy
- *  - Public forum : https://groups.google.com/forum/#!forum/lz4c
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- * You may select, at your option, one of the above-listed licenses.
-****************************************************************** */
-
-
-/* **************************************************************
-*  Includes
-****************************************************************/
-#include "debug.h"      /* assert */
-#include "bitstream.h"
-#include "compiler.h"
-#define FSE_STATIC_LINKING_ONLY
-#include "fse.h"
-#include "error_private.h"
-#include "zstd_deps.h"  /* ZSTD_memcpy */
-#include "bits.h"       /* ZSTD_highbit32 */
-
-
-/* **************************************************************
-*  Error Management
-****************************************************************/
-#define FSE_isError ERR_isError
-#define FSE_STATIC_ASSERT(c) DEBUG_STATIC_ASSERT(c)   /* use only *after* variable declarations */
-
-
-/* **************************************************************
-*  Templates
-****************************************************************/
-/*
-  designed to be included
-  for type-specific functions (template emulation in C)
-  Objective is to write these functions only once, for improved maintenance
-*/
-
-/* safety checks */
-#ifndef FSE_FUNCTION_EXTENSION
-#  error "FSE_FUNCTION_EXTENSION must be defined"
-#endif
-#ifndef FSE_FUNCTION_TYPE
-#  error "FSE_FUNCTION_TYPE must be defined"
-#endif
-
-/* Function names */
-#define FSE_CAT(X,Y) X##Y
-#define FSE_FUNCTION_NAME(X,Y) FSE_CAT(X,Y)
-#define FSE_TYPE_NAME(X,Y) FSE_CAT(X,Y)
-
-static size_t FSE_buildDTable_internal(FSE_DTable* dt, const short* normalizedCounter, unsigned maxSymbolValue, unsigned tableLog, void* workSpace, size_t wkspSize)
-{
-    void* const tdPtr = dt+1;   /* because *dt is unsigned, 32-bits aligned on 32-bits */
-    FSE_DECODE_TYPE* const tableDecode = (FSE_DECODE_TYPE*) (tdPtr);
-    U16* symbolNext = (U16*)workSpace;
-    BYTE* spread = (BYTE*)(symbolNext + maxSymbolValue + 1);
-
-    U32 const maxSV1 = maxSymbolValue + 1;
-    U32 const tableSize = 1 << tableLog;
-    U32 highThreshold = tableSize-1;
-
-    /* Sanity Checks */
-    if (FSE_BUILD_DTABLE_WKSP_SIZE(tableLog, maxSymbolValue) > wkspSize) return ERROR(maxSymbolValue_tooLarge);
-    if (maxSymbolValue > FSE_MAX_SYMBOL_VALUE) return ERROR(maxSymbolValue_tooLarge);
-    if (tableLog > FSE_MAX_TABLELOG) return ERROR(tableLog_tooLarge);
-
-    /* Init, lay down lowprob symbols */
-    {   FSE_DTableHeader DTableH;
-        DTableH.tableLog = (U16)tableLog;
-        DTableH.fastMode = 1;
-        {   S16 const largeLimit= (S16)(1 << (tableLog-1));
-            U32 s;
-            for (s=0; s<maxSV1; s++) {
-                if (normalizedCounter[s]==-1) {
-                    tableDecode[highThreshold--].symbol = (FSE_FUNCTION_TYPE)s;
-                    symbolNext[s] = 1;
-                } else {
-                    if (normalizedCounter[s] >= largeLimit) DTableH.fastMode=0;
-                    symbolNext[s] = (U16)normalizedCounter[s];
-        }   }   }
-        ZSTD_memcpy(dt, &DTableH, sizeof(DTableH));
-    }
-
-    /* Spread symbols */
-    if (highThreshold == tableSize - 1) {
-        size_t const tableMask = tableSize-1;
-        size_t const step = FSE_TABLESTEP(tableSize);
-        /* First lay down the symbols in order.
-         * We use a uint64_t to lay down 8 bytes at a time. This reduces branch
-         * misses since small blocks generally have small table logs, so nearly
-         * all symbols have counts <= 8. We ensure we have 8 bytes at the end of
-         * our buffer to handle the over-write.
-         */
-        {   U64 const add = 0x0101010101010101ull;
-            size_t pos = 0;
-            U64 sv = 0;
-            U32 s;
-            for (s=0; s<maxSV1; ++s, sv += add) {
-                int i;
-                int const n = normalizedCounter[s];
-                MEM_write64(spread + pos, sv);
-                for (i = 8; i < n; i += 8) {
-                    MEM_write64(spread + pos + i, sv);
-                }
-                pos += (size_t)n;
-        }   }
-        /* Now we spread those positions across the table.
-         * The benefit of doing it in two stages is that we avoid the
-         * variable size inner loop, which caused lots of branch misses.
-         * Now we can run through all the positions without any branch misses.
-         * We unroll the loop twice, since that is what empirically worked best.
-         */
-        {
-            size_t position = 0;
-            size_t s;
-            size_t const unroll = 2;
-            assert(tableSize % unroll == 0); /* FSE_MIN_TABLELOG is 5 */
-            for (s = 0; s < (size_t)tableSize; s += unroll) {
-                size_t u;
-                for (u = 0; u < unroll; ++u) {
-                    size_t const uPosition = (position + (u * step)) & tableMask;
-                    tableDecode[uPosition].symbol = spread[s + u];
-                }
-                position = (position + (unroll * step)) & tableMask;
-            }
-            assert(position == 0);
-        }
-    } else {
-        U32 const tableMask = tableSize-1;
-        U32 const step = FSE_TABLESTEP(tableSize);
-        U32 s, position = 0;
-        for (s=0; s<maxSV1; s++) {
-            int i;
-            for (i=0; i<normalizedCounter[s]; i++) {
-                tableDecode[position].symbol = (FSE_FUNCTION_TYPE)s;
-                position = (position + step) & tableMask;
-                while (position > highThreshold) position = (position + step) & tableMask;   /* lowprob area */
-        }   }
-        if (position!=0) return ERROR(GENERIC);   /* position must reach all cells once, otherwise normalizedCounter is incorrect */
-    }
-
-    /* Build Decoding table */
-    {   U32 u;
-        for (u=0; u<tableSize; u++) {
-            FSE_FUNCTION_TYPE const symbol = (FSE_FUNCTION_TYPE)(tableDecode[u].symbol);
-            U32 const nextState = symbolNext[symbol]++;
-            tableDecode[u].nbBits = (BYTE) (tableLog - ZSTD_highbit32(nextState) );
-            tableDecode[u].newState = (U16) ( (nextState << tableDecode[u].nbBits) - tableSize);
-    }   }
-
-    return 0;
-}
-
-size_t FSE_buildDTable_wksp(FSE_DTable* dt, const short* normalizedCounter, unsigned maxSymbolValue, unsigned tableLog, void* workSpace, size_t wkspSize)
-{
-    return FSE_buildDTable_internal(dt, normalizedCounter, maxSymbolValue, tableLog, workSpace, wkspSize);
-}
-
-
-#ifndef FSE_COMMONDEFS_ONLY
-
-/*-*******************************************************
-*  Decompression (Byte symbols)
-*********************************************************/
-
-FORCE_INLINE_TEMPLATE size_t FSE_decompress_usingDTable_generic(
-          void* dst, size_t maxDstSize,
-    const void* cSrc, size_t cSrcSize,
-    const FSE_DTable* dt, const unsigned fast)
-{
-    BYTE* const ostart = (BYTE*) dst;
-    BYTE* op = ostart;
-    BYTE* const omax = op + maxDstSize;
-    BYTE* const olimit = omax-3;
-
-    BIT_DStream_t bitD;
-    FSE_DState_t state1;
-    FSE_DState_t state2;
-
-    /* Init */
-    CHECK_F(BIT_initDStream(&bitD, cSrc, cSrcSize));
-
-    FSE_initDState(&state1, &bitD, dt);
-    FSE_initDState(&state2, &bitD, dt);
-
-    RETURN_ERROR_IF(BIT_reloadDStream(&bitD)==BIT_DStream_overflow, corruption_detected, "");
-
-#define FSE_GETSYMBOL(statePtr) fast ? FSE_decodeSymbolFast(statePtr, &bitD) : FSE_decodeSymbol(statePtr, &bitD)
-
-    /* 4 symbols per loop */
-    for ( ; (BIT_reloadDStream(&bitD)==BIT_DStream_unfinished) & (op<olimit) ; op+=4) {
-        op[0] = FSE_GETSYMBOL(&state1);
-
-        if (FSE_MAX_TABLELOG*2+7 > sizeof(bitD.bitContainer)*8)    /* This test must be static */
-            BIT_reloadDStream(&bitD);
-
-        op[1] = FSE_GETSYMBOL(&state2);
-
-        if (FSE_MAX_TABLELOG*4+7 > sizeof(bitD.bitContainer)*8)    /* This test must be static */
-            { if (BIT_reloadDStream(&bitD) > BIT_DStream_unfinished) { op+=2; break; } }
-
-        op[2] = FSE_GETSYMBOL(&state1);
-
-        if (FSE_MAX_TABLELOG*2+7 > sizeof(bitD.bitContainer)*8)    /* This test must be static */
-            BIT_reloadDStream(&bitD);
-
-        op[3] = FSE_GETSYMBOL(&state2);
-    }
-
-    /* tail */
-    /* note : BIT_reloadDStream(&bitD) >= FSE_DStream_partiallyFilled; Ends at exactly BIT_DStream_completed */
-    while (1) {
-        if (op>(omax-2)) return ERROR(dstSize_tooSmall);
-        *op++ = FSE_GETSYMBOL(&state1);
-        if (BIT_reloadDStream(&bitD)==BIT_DStream_overflow) {
-            *op++ = FSE_GETSYMBOL(&state2);
-            break;
-        }
-
-        if (op>(omax-2)) return ERROR(dstSize_tooSmall);
-        *op++ = FSE_GETSYMBOL(&state2);
-        if (BIT_reloadDStream(&bitD)==BIT_DStream_overflow) {
-            *op++ = FSE_GETSYMBOL(&state1);
-            break;
-    }   }
-
-    assert(op >= ostart);
-    return (size_t)(op-ostart);
-}
-
-typedef struct {
-    short ncount[FSE_MAX_SYMBOL_VALUE + 1];
-} FSE_DecompressWksp;
-
-
-FORCE_INLINE_TEMPLATE size_t FSE_decompress_wksp_body(
-        void* dst, size_t dstCapacity,
-        const void* cSrc, size_t cSrcSize,
-        unsigned maxLog, void* workSpace, size_t wkspSize,
-        int bmi2)
-{
-    const BYTE* const istart = (const BYTE*)cSrc;
-    const BYTE* ip = istart;
-    unsigned tableLog;
-    unsigned maxSymbolValue = FSE_MAX_SYMBOL_VALUE;
-    FSE_DecompressWksp* const wksp = (FSE_DecompressWksp*)workSpace;
-    size_t const dtablePos = sizeof(FSE_DecompressWksp) / sizeof(FSE_DTable);
-    FSE_DTable* const dtable = (FSE_DTable*)workSpace + dtablePos;
-
-    FSE_STATIC_ASSERT((FSE_MAX_SYMBOL_VALUE + 1) % 2 == 0);
-    if (wkspSize < sizeof(*wksp)) return ERROR(GENERIC);
-
-    /* correct offset to dtable depends on this property */
-    FSE_STATIC_ASSERT(sizeof(FSE_DecompressWksp) % sizeof(FSE_DTable) == 0);
-
-    /* normal FSE decoding mode */
-    {   size_t const NCountLength =
-            FSE_readNCount_bmi2(wksp->ncount, &maxSymbolValue, &tableLog, istart, cSrcSize, bmi2);
-        if (FSE_isError(NCountLength)) return NCountLength;
-        if (tableLog > maxLog) return ERROR(tableLog_tooLarge);
-        assert(NCountLength <= cSrcSize);
-        ip += NCountLength;
-        cSrcSize -= NCountLength;
-    }
-
-    if (FSE_DECOMPRESS_WKSP_SIZE(tableLog, maxSymbolValue) > wkspSize) return ERROR(tableLog_tooLarge);
-    assert(sizeof(*wksp) + FSE_DTABLE_SIZE(tableLog) <= wkspSize);
-    workSpace = (BYTE*)workSpace + sizeof(*wksp) + FSE_DTABLE_SIZE(tableLog);
-    wkspSize -= sizeof(*wksp) + FSE_DTABLE_SIZE(tableLog);
-
-    CHECK_F( FSE_buildDTable_internal(dtable, wksp->ncount, maxSymbolValue, tableLog, workSpace, wkspSize) );
-
-    {
-        const void* ptr = dtable;
-        const FSE_DTableHeader* DTableH = (const FSE_DTableHeader*)ptr;
-        const U32 fastMode = DTableH->fastMode;
-
-        /* select fast mode (static) */
-        if (fastMode) return FSE_decompress_usingDTable_generic(dst, dstCapacity, ip, cSrcSize, dtable, 1);
-        return FSE_decompress_usingDTable_generic(dst, dstCapacity, ip, cSrcSize, dtable, 0);
-    }
-}
-
-/* Avoids the FORCE_INLINE of the _body() function. */
-static size_t FSE_decompress_wksp_body_default(void* dst, size_t dstCapacity, const void* cSrc, size_t cSrcSize, unsigned maxLog, void* workSpace, size_t wkspSize)
-{
-    return FSE_decompress_wksp_body(dst, dstCapacity, cSrc, cSrcSize, maxLog, workSpace, wkspSize, 0);
-}
-
-#if DYNAMIC_BMI2
-BMI2_TARGET_ATTRIBUTE static size_t FSE_decompress_wksp_body_bmi2(void* dst, size_t dstCapacity, const void* cSrc, size_t cSrcSize, unsigned maxLog, void* workSpace, size_t wkspSize)
-{
-    return FSE_decompress_wksp_body(dst, dstCapacity, cSrc, cSrcSize, maxLog, workSpace, wkspSize, 1);
-}
-#endif
-
-size_t FSE_decompress_wksp_bmi2(void* dst, size_t dstCapacity, const void* cSrc, size_t cSrcSize, unsigned maxLog, void* workSpace, size_t wkspSize, int bmi2)
-{
-#if DYNAMIC_BMI2
-    if (bmi2) {
-        return FSE_decompress_wksp_body_bmi2(dst, dstCapacity, cSrc, cSrcSize, maxLog, workSpace, wkspSize);
-    }
-#endif
-    (void)bmi2;
-    return FSE_decompress_wksp_body_default(dst, dstCapacity, cSrc, cSrcSize, maxLog, workSpace, wkspSize);
-}
-
-#endif   /* FSE_COMMONDEFS_ONLY */
-
-#endif /* USE_EXTERNAL_ZSTD */

vendor/github.com/DataDog/zstd/hist.c

@@ -1,194 +0,0 @@
-#ifndef USE_EXTERNAL_ZSTD
-/* ******************************************************************
- * hist : Histogram functions
- * part of Finite State Entropy project
- * Copyright (c) Meta Platforms, Inc. and affiliates.
- *
- *  You can contact the author at :
- *  - FSE source repository : https://github.com/Cyan4973/FiniteStateEntropy
- *  - Public forum : https://groups.google.com/forum/#!forum/lz4c
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- * You may select, at your option, one of the above-listed licenses.
-****************************************************************** */
-
-/* --- dependencies --- */
-#include "mem.h"             /* U32, BYTE, etc. */
-#include "debug.h"           /* assert, DEBUGLOG */
-#include "error_private.h"   /* ERROR */
-#include "hist.h"
-
-
-/* --- Error management --- */
-unsigned HIST_isError(size_t code) { return ERR_isError(code); }
-
-/*-**************************************************************
- *  Histogram functions
- ****************************************************************/
-void HIST_add(unsigned* count, const void* src, size_t srcSize)
-{
-    const BYTE* ip = (const BYTE*)src;
-    const BYTE* const end = ip + srcSize;
-
-    while (ip<end) {
-        count[*ip++]++;
-    }
-}
-
-unsigned HIST_count_simple(unsigned* count, unsigned* maxSymbolValuePtr,
-                           const void* src, size_t srcSize)
-{
-    const BYTE* ip = (const BYTE*)src;
-    const BYTE* const end = ip + srcSize;
-    unsigned maxSymbolValue = *maxSymbolValuePtr;
-    unsigned largestCount=0;
-
-    ZSTD_memset(count, 0, (maxSymbolValue+1) * sizeof(*count));
-    if (srcSize==0) { *maxSymbolValuePtr = 0; return 0; }
-
-    while (ip<end) {
-        assert(*ip <= maxSymbolValue);
-        count[*ip++]++;
-    }
-
-    while (!count[maxSymbolValue]) maxSymbolValue--;
-    *maxSymbolValuePtr = maxSymbolValue;
-
-    {   U32 s;
-        for (s=0; s<=maxSymbolValue; s++)
-            if (count[s] > largestCount) largestCount = count[s];
-    }
-
-    return largestCount;
-}
-
-typedef enum { trustInput, checkMaxSymbolValue } HIST_checkInput_e;
-
-/* HIST_count_parallel_wksp() :
- * store histogram into 4 intermediate tables, recombined at the end.
- * this design makes better use of OoO cpus,
- * and is noticeably faster when some values are heavily repeated.
- * But it needs some additional workspace for intermediate tables.
- * `workSpace` must be a U32 table of size >= HIST_WKSP_SIZE_U32.
- * @return : largest histogram frequency,
- *           or an error code (notably when histogram's alphabet is larger than *maxSymbolValuePtr) */
-static size_t HIST_count_parallel_wksp(
-                                unsigned* count, unsigned* maxSymbolValuePtr,
-                                const void* source, size_t sourceSize,
-                                HIST_checkInput_e check,
-                                U32* const workSpace)
-{
-    const BYTE* ip = (const BYTE*)source;
-    const BYTE* const iend = ip+sourceSize;
-    size_t const countSize = (*maxSymbolValuePtr + 1) * sizeof(*count);
-    unsigned max=0;
-    U32* const Counting1 = workSpace;
-    U32* const Counting2 = Counting1 + 256;
-    U32* const Counting3 = Counting2 + 256;
-    U32* const Counting4 = Counting3 + 256;
-
-    /* safety checks */
-    assert(*maxSymbolValuePtr <= 255);
-    if (!sourceSize) {
-        ZSTD_memset(count, 0, countSize);
-        *maxSymbolValuePtr = 0;
-        return 0;
-    }
-    ZSTD_memset(workSpace, 0, 4*256*sizeof(unsigned));
-
-    /* by stripes of 16 bytes */
-    {   U32 cached = MEM_read32(ip); ip += 4;
-        while (ip < iend-15) {
-            U32 c = cached; cached = MEM_read32(ip); ip += 4;
-            Counting1[(BYTE) c     ]++;
-            Counting2[(BYTE)(c>>8) ]++;
-            Counting3[(BYTE)(c>>16)]++;
-            Counting4[       c>>24 ]++;
-            c = cached; cached = MEM_read32(ip); ip += 4;
-            Counting1[(BYTE) c     ]++;
-            Counting2[(BYTE)(c>>8) ]++;
-            Counting3[(BYTE)(c>>16)]++;
-            Counting4[       c>>24 ]++;
-            c = cached; cached = MEM_read32(ip); ip += 4;
-            Counting1[(BYTE) c     ]++;
-            Counting2[(BYTE)(c>>8) ]++;
-            Counting3[(BYTE)(c>>16)]++;
-            Counting4[       c>>24 ]++;
-            c = cached; cached = MEM_read32(ip); ip += 4;
-            Counting1[(BYTE) c     ]++;
-            Counting2[(BYTE)(c>>8) ]++;
-            Counting3[(BYTE)(c>>16)]++;
-            Counting4[       c>>24 ]++;
-        }
-        ip-=4;
-    }
-
-    /* finish last symbols */
-    while (ip<iend) Counting1[*ip++]++;
-
-    {   U32 s;
-        for (s=0; s<256; s++) {
-            Counting1[s] += Counting2[s] + Counting3[s] + Counting4[s];
-            if (Counting1[s] > max) max = Counting1[s];
-    }   }
-
-    {   unsigned maxSymbolValue = 255;
-        while (!Counting1[maxSymbolValue]) maxSymbolValue--;
-        if (check && maxSymbolValue > *maxSymbolValuePtr) return ERROR(maxSymbolValue_tooSmall);
-        *maxSymbolValuePtr = maxSymbolValue;
-        ZSTD_memmove(count, Counting1, countSize);   /* in case count & Counting1 are overlapping */
-    }
-    return (size_t)max;
-}
-
-/* HIST_countFast_wksp() :
- * Same as HIST_countFast(), but using an externally provided scratch buffer.
- * `workSpace` is a writable buffer which must be 4-bytes aligned,
- * `workSpaceSize` must be >= HIST_WKSP_SIZE
- */
-size_t HIST_countFast_wksp(unsigned* count, unsigned* maxSymbolValuePtr,
-                          const void* source, size_t sourceSize,
-                          void* workSpace, size_t workSpaceSize)
-{
-    if (sourceSize < 1500) /* heuristic threshold */
-        return HIST_count_simple(count, maxSymbolValuePtr, source, sourceSize);
-    if ((size_t)workSpace & 3) return ERROR(GENERIC);  /* must be aligned on 4-bytes boundaries */
-    if (workSpaceSize < HIST_WKSP_SIZE) return ERROR(workSpace_tooSmall);
-    return HIST_count_parallel_wksp(count, maxSymbolValuePtr, source, sourceSize, trustInput, (U32*)workSpace);
-}
-
-/* HIST_count_wksp() :
- * Same as HIST_count(), but using an externally provided scratch buffer.
- * `workSpace` size must be table of >= HIST_WKSP_SIZE_U32 unsigned */
-size_t HIST_count_wksp(unsigned* count, unsigned* maxSymbolValuePtr,
-                       const void* source, size_t sourceSize,
-                       void* workSpace, size_t workSpaceSize)
-{
-    if ((size_t)workSpace & 3) return ERROR(GENERIC);  /* must be aligned on 4-bytes boundaries */
-    if (workSpaceSize < HIST_WKSP_SIZE) return ERROR(workSpace_tooSmall);
-    if (*maxSymbolValuePtr < 255)
-        return HIST_count_parallel_wksp(count, maxSymbolValuePtr, source, sourceSize, checkMaxSymbolValue, (U32*)workSpace);
-    *maxSymbolValuePtr = 255;
-    return HIST_countFast_wksp(count, maxSymbolValuePtr, source, sourceSize, workSpace, workSpaceSize);
-}
-
-#ifndef ZSTD_NO_UNUSED_FUNCTIONS
-/* fast variant (unsafe : won't check if src contains values beyond count[] limit) */
-size_t HIST_countFast(unsigned* count, unsigned* maxSymbolValuePtr,
-                     const void* source, size_t sourceSize)
-{
-    unsigned tmpCounters[HIST_WKSP_SIZE_U32];
-    return HIST_countFast_wksp(count, maxSymbolValuePtr, source, sourceSize, tmpCounters, sizeof(tmpCounters));
-}
-
-size_t HIST_count(unsigned* count, unsigned* maxSymbolValuePtr,
-                 const void* src, size_t srcSize)
-{
-    unsigned tmpCounters[HIST_WKSP_SIZE_U32];
-    return HIST_count_wksp(count, maxSymbolValuePtr, src, srcSize, tmpCounters, sizeof(tmpCounters));
-}
-#endif
-
-#endif /* USE_EXTERNAL_ZSTD */

vendor/github.com/DataDog/zstd/hist.h

@@ -1,85 +0,0 @@
-#ifndef USE_EXTERNAL_ZSTD
-/* ******************************************************************
- * hist : Histogram functions
- * part of Finite State Entropy project
- * Copyright (c) Meta Platforms, Inc. and affiliates.
- *
- *  You can contact the author at :
- *  - FSE source repository : https://github.com/Cyan4973/FiniteStateEntropy
- *  - Public forum : https://groups.google.com/forum/#!forum/lz4c
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- * You may select, at your option, one of the above-listed licenses.
-****************************************************************** */
-
-/* --- dependencies --- */
-#include "zstd_deps.h"   /* size_t */
-
-
-/* --- simple histogram functions --- */
-
-/*! HIST_count():
- *  Provides the precise count of each byte within a table 'count'.
- * 'count' is a table of unsigned int, of minimum size (*maxSymbolValuePtr+1).
- *  Updates *maxSymbolValuePtr with actual largest symbol value detected.
- * @return : count of the most frequent symbol (which isn't identified).
- *           or an error code, which can be tested using HIST_isError().
- *           note : if return == srcSize, there is only one symbol.
- */
-size_t HIST_count(unsigned* count, unsigned* maxSymbolValuePtr,
-                  const void* src, size_t srcSize);
-
-unsigned HIST_isError(size_t code);  /**< tells if a return value is an error code */
-
-
-/* --- advanced histogram functions --- */
-
-#define HIST_WKSP_SIZE_U32 1024
-#define HIST_WKSP_SIZE    (HIST_WKSP_SIZE_U32 * sizeof(unsigned))
-/** HIST_count_wksp() :
- *  Same as HIST_count(), but using an externally provided scratch buffer.
- *  Benefit is this function will use very little stack space.
- * `workSpace` is a writable buffer which must be 4-bytes aligned,
- * `workSpaceSize` must be >= HIST_WKSP_SIZE
- */
-size_t HIST_count_wksp(unsigned* count, unsigned* maxSymbolValuePtr,
-                       const void* src, size_t srcSize,
-                       void* workSpace, size_t workSpaceSize);
-
-/** HIST_countFast() :
- *  same as HIST_count(), but blindly trusts that all byte values within src are <= *maxSymbolValuePtr.
- *  This function is unsafe, and will segfault if any value within `src` is `> *maxSymbolValuePtr`
- */
-size_t HIST_countFast(unsigned* count, unsigned* maxSymbolValuePtr,
-                      const void* src, size_t srcSize);
-
-/** HIST_countFast_wksp() :
- *  Same as HIST_countFast(), but using an externally provided scratch buffer.
- * `workSpace` is a writable buffer which must be 4-bytes aligned,
- * `workSpaceSize` must be >= HIST_WKSP_SIZE
- */
-size_t HIST_countFast_wksp(unsigned* count, unsigned* maxSymbolValuePtr,
-                           const void* src, size_t srcSize,
-                           void* workSpace, size_t workSpaceSize);
-
-/*! HIST_count_simple() :
- *  Same as HIST_countFast(), this function is unsafe,
- *  and will segfault if any value within `src` is `> *maxSymbolValuePtr`.
- *  It is also a bit slower for large inputs.
- *  However, it does not need any additional memory (not even on stack).
- * @return : count of the most frequent symbol.
- *  Note this function doesn't produce any error (i.e. it must succeed).
- */
-unsigned HIST_count_simple(unsigned* count, unsigned* maxSymbolValuePtr,
-                           const void* src, size_t srcSize);
-
-/*! HIST_add() :
- *  Lowest level: just add nb of occurrences of characters from @src into @count.
- *  @count is not reset. @count array is presumed large enough (i.e. 1 KB).
- @  This function does not need any additional stack memory.
- */
-void HIST_add(unsigned* count, const void* src, size_t srcSize);
-
-#endif /* USE_EXTERNAL_ZSTD */

vendor/github.com/DataDog/zstd/huf.h

@@ -1,280 +0,0 @@
-#ifndef USE_EXTERNAL_ZSTD
-/* ******************************************************************
- * huff0 huffman codec,
- * part of Finite State Entropy library
- * Copyright (c) Meta Platforms, Inc. and affiliates.
- *
- * You can contact the author at :
- * - Source repository : https://github.com/Cyan4973/FiniteStateEntropy
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- * You may select, at your option, one of the above-listed licenses.
-****************************************************************** */
-
-#ifndef HUF_H_298734234
-#define HUF_H_298734234
-
-/* *** Dependencies *** */
-#include "zstd_deps.h"    /* size_t */
-#include "mem.h"          /* U32 */
-#define FSE_STATIC_LINKING_ONLY
-#include "fse.h"
-
-/* ***   Tool functions *** */
-#define HUF_BLOCKSIZE_MAX (128 * 1024)   /**< maximum input size for a single block compressed with HUF_compress */
-size_t HUF_compressBound(size_t size);   /**< maximum compressed size (worst case) */
-
-/* Error Management */
-unsigned    HUF_isError(size_t code);       /**< tells if a return value is an error code */
-const char* HUF_getErrorName(size_t code);  /**< provides error code string (useful for debugging) */
-
-
-#define HUF_WORKSPACE_SIZE ((8 << 10) + 512 /* sorting scratch space */)
-#define HUF_WORKSPACE_SIZE_U64 (HUF_WORKSPACE_SIZE / sizeof(U64))
-
-/* *** Constants *** */
-#define HUF_TABLELOG_MAX      12      /* max runtime value of tableLog (due to static allocation); can be modified up to HUF_TABLELOG_ABSOLUTEMAX */
-#define HUF_TABLELOG_DEFAULT  11      /* default tableLog value when none specified */
-#define HUF_SYMBOLVALUE_MAX  255
-
-#define HUF_TABLELOG_ABSOLUTEMAX  12  /* absolute limit of HUF_MAX_TABLELOG. Beyond that value, code does not work */
-#if (HUF_TABLELOG_MAX > HUF_TABLELOG_ABSOLUTEMAX)
-#  error "HUF_TABLELOG_MAX is too large !"
-#endif
-
-
-/* ****************************************
-*  Static allocation
-******************************************/
-/* HUF buffer bounds */
-#define HUF_CTABLEBOUND 129
-#define HUF_BLOCKBOUND(size) (size + (size>>8) + 8)   /* only true when incompressible is pre-filtered with fast heuristic */
-#define HUF_COMPRESSBOUND(size) (HUF_CTABLEBOUND + HUF_BLOCKBOUND(size))   /* Macro version, useful for static allocation */
-
-/* static allocation of HUF's Compression Table */
-/* this is a private definition, just exposed for allocation and strict aliasing purpose. never EVER access its members directly */
-typedef size_t HUF_CElt;   /* consider it an incomplete type */
-#define HUF_CTABLE_SIZE_ST(maxSymbolValue)   ((maxSymbolValue)+2)   /* Use tables of size_t, for proper alignment */
-#define HUF_CTABLE_SIZE(maxSymbolValue)       (HUF_CTABLE_SIZE_ST(maxSymbolValue) * sizeof(size_t))
-#define HUF_CREATE_STATIC_CTABLE(name, maxSymbolValue) \
-    HUF_CElt name[HUF_CTABLE_SIZE_ST(maxSymbolValue)] /* no final ; */
-
-/* static allocation of HUF's DTable */
-typedef U32 HUF_DTable;
-#define HUF_DTABLE_SIZE(maxTableLog)   (1 + (1<<(maxTableLog)))
-#define HUF_CREATE_STATIC_DTABLEX1(DTable, maxTableLog) \
-        HUF_DTable DTable[HUF_DTABLE_SIZE((maxTableLog)-1)] = { ((U32)((maxTableLog)-1) * 0x01000001) }
-#define HUF_CREATE_STATIC_DTABLEX2(DTable, maxTableLog) \
-        HUF_DTable DTable[HUF_DTABLE_SIZE(maxTableLog)] = { ((U32)(maxTableLog) * 0x01000001) }
-
-
-/* ****************************************
-*  Advanced decompression functions
-******************************************/
-
-/**
- * Huffman flags bitset.
- * For all flags, 0 is the default value.
- */
-typedef enum {
-    /**
-     * If compiled with DYNAMIC_BMI2: Set flag only if the CPU supports BMI2 at runtime.
-     * Otherwise: Ignored.
-     */
-    HUF_flags_bmi2 = (1 << 0),
-    /**
-     * If set: Test possible table depths to find the one that produces the smallest header + encoded size.
-     * If unset: Use heuristic to find the table depth.
-     */
-    HUF_flags_optimalDepth = (1 << 1),
-    /**
-     * If set: If the previous table can encode the input, always reuse the previous table.
-     * If unset: If the previous table can encode the input, reuse the previous table if it results in a smaller output.
-     */
-    HUF_flags_preferRepeat = (1 << 2),
-    /**
-     * If set: Sample the input and check if the sample is uncompressible, if it is then don't attempt to compress.
-     * If unset: Always histogram the entire input.
-     */
-    HUF_flags_suspectUncompressible = (1 << 3),
-    /**
-     * If set: Don't use assembly implementations
-     * If unset: Allow using assembly implementations
-     */
-    HUF_flags_disableAsm = (1 << 4),
-    /**
-     * If set: Don't use the fast decoding loop, always use the fallback decoding loop.
-     * If unset: Use the fast decoding loop when possible.
-     */
-    HUF_flags_disableFast = (1 << 5)
-} HUF_flags_e;
-
-
-/* ****************************************
- *  HUF detailed API
- * ****************************************/
-#define HUF_OPTIMAL_DEPTH_THRESHOLD ZSTD_btultra
-
-/*! HUF_compress() does the following:
- *  1. count symbol occurrence from source[] into table count[] using FSE_count() (exposed within "fse.h")
- *  2. (optional) refine tableLog using HUF_optimalTableLog()
- *  3. build Huffman table from count using HUF_buildCTable()
- *  4. save Huffman table to memory buffer using HUF_writeCTable()
- *  5. encode the data stream using HUF_compress4X_usingCTable()
- *
- *  The following API allows targeting specific sub-functions for advanced tasks.
- *  For example, it's possible to compress several blocks using the same 'CTable',
- *  or to save and regenerate 'CTable' using external methods.
- */
-unsigned HUF_minTableLog(unsigned symbolCardinality);
-unsigned HUF_cardinality(const unsigned* count, unsigned maxSymbolValue);
-unsigned HUF_optimalTableLog(unsigned maxTableLog, size_t srcSize, unsigned maxSymbolValue, void* workSpace,
- size_t wkspSize, HUF_CElt* table, const unsigned* count, int flags); /* table is used as scratch space for building and testing tables, not a return value */
-size_t HUF_writeCTable_wksp(void* dst, size_t maxDstSize, const HUF_CElt* CTable, unsigned maxSymbolValue, unsigned huffLog, void* workspace, size_t workspaceSize);
-size_t HUF_compress4X_usingCTable(void* dst, size_t dstSize, const void* src, size_t srcSize, const HUF_CElt* CTable, int flags);
-size_t HUF_estimateCompressedSize(const HUF_CElt* CTable, const unsigned* count, unsigned maxSymbolValue);
-int HUF_validateCTable(const HUF_CElt* CTable, const unsigned* count, unsigned maxSymbolValue);
-
-typedef enum {
-   HUF_repeat_none,  /**< Cannot use the previous table */
-   HUF_repeat_check, /**< Can use the previous table but it must be checked. Note : The previous table must have been constructed by HUF_compress{1, 4}X_repeat */
-   HUF_repeat_valid  /**< Can use the previous table and it is assumed to be valid */
- } HUF_repeat;
-
-/** HUF_compress4X_repeat() :
- *  Same as HUF_compress4X_wksp(), but considers using hufTable if *repeat != HUF_repeat_none.
- *  If it uses hufTable it does not modify hufTable or repeat.
- *  If it doesn't, it sets *repeat = HUF_repeat_none, and it sets hufTable to the table used.
- *  If preferRepeat then the old table will always be used if valid.
- *  If suspectUncompressible then some sampling checks will be run to potentially skip huffman coding */
-size_t HUF_compress4X_repeat(void* dst, size_t dstSize,
-                       const void* src, size_t srcSize,
-                       unsigned maxSymbolValue, unsigned tableLog,
-                       void* workSpace, size_t wkspSize,    /**< `workSpace` must be aligned on 4-bytes boundaries, `wkspSize` must be >= HUF_WORKSPACE_SIZE */
-                       HUF_CElt* hufTable, HUF_repeat* repeat, int flags);
-
-/** HUF_buildCTable_wksp() :
- *  Same as HUF_buildCTable(), but using externally allocated scratch buffer.
- * `workSpace` must be aligned on 4-bytes boundaries, and its size must be >= HUF_CTABLE_WORKSPACE_SIZE.
- */
-#define HUF_CTABLE_WORKSPACE_SIZE_U32 ((4 * (HUF_SYMBOLVALUE_MAX + 1)) + 192)
-#define HUF_CTABLE_WORKSPACE_SIZE (HUF_CTABLE_WORKSPACE_SIZE_U32 * sizeof(unsigned))
-size_t HUF_buildCTable_wksp (HUF_CElt* tree,
-                       const unsigned* count, U32 maxSymbolValue, U32 maxNbBits,
-                             void* workSpace, size_t wkspSize);
-
-/*! HUF_readStats() :
- *  Read compact Huffman tree, saved by HUF_writeCTable().
- * `huffWeight` is destination buffer.
- * @return : size read from `src` , or an error Code .
- *  Note : Needed by HUF_readCTable() and HUF_readDTableXn() . */
-size_t HUF_readStats(BYTE* huffWeight, size_t hwSize,
-                     U32* rankStats, U32* nbSymbolsPtr, U32* tableLogPtr,
-                     const void* src, size_t srcSize);
-
-/*! HUF_readStats_wksp() :
- * Same as HUF_readStats() but takes an external workspace which must be
- * 4-byte aligned and its size must be >= HUF_READ_STATS_WORKSPACE_SIZE.
- * If the CPU has BMI2 support, pass bmi2=1, otherwise pass bmi2=0.
- */
-#define HUF_READ_STATS_WORKSPACE_SIZE_U32 FSE_DECOMPRESS_WKSP_SIZE_U32(6, HUF_TABLELOG_MAX-1)
-#define HUF_READ_STATS_WORKSPACE_SIZE (HUF_READ_STATS_WORKSPACE_SIZE_U32 * sizeof(unsigned))
-size_t HUF_readStats_wksp(BYTE* huffWeight, size_t hwSize,
-                          U32* rankStats, U32* nbSymbolsPtr, U32* tableLogPtr,
-                          const void* src, size_t srcSize,
-                          void* workspace, size_t wkspSize,
-                          int flags);
-
-/** HUF_readCTable() :
- *  Loading a CTable saved with HUF_writeCTable() */
-size_t HUF_readCTable (HUF_CElt* CTable, unsigned* maxSymbolValuePtr, const void* src, size_t srcSize, unsigned *hasZeroWeights);
-
-/** HUF_getNbBitsFromCTable() :
- *  Read nbBits from CTable symbolTable, for symbol `symbolValue` presumed <= HUF_SYMBOLVALUE_MAX
- *  Note 1 : If symbolValue > HUF_readCTableHeader(symbolTable).maxSymbolValue, returns 0
- *  Note 2 : is not inlined, as HUF_CElt definition is private
- */
-U32 HUF_getNbBitsFromCTable(const HUF_CElt* symbolTable, U32 symbolValue);
-
-typedef struct {
-    BYTE tableLog;
-    BYTE maxSymbolValue;
-    BYTE unused[sizeof(size_t) - 2];
-} HUF_CTableHeader;
-
-/** HUF_readCTableHeader() :
- *  @returns The header from the CTable specifying the tableLog and the maxSymbolValue.
- */
-HUF_CTableHeader HUF_readCTableHeader(HUF_CElt const* ctable);
-
-/*
- * HUF_decompress() does the following:
- * 1. select the decompression algorithm (X1, X2) based on pre-computed heuristics
- * 2. build Huffman table from save, using HUF_readDTableX?()
- * 3. decode 1 or 4 segments in parallel using HUF_decompress?X?_usingDTable()
- */
-
-/** HUF_selectDecoder() :
- *  Tells which decoder is likely to decode faster,
- *  based on a set of pre-computed metrics.
- * @return : 0==HUF_decompress4X1, 1==HUF_decompress4X2 .
- *  Assumption : 0 < dstSize <= 128 KB */
-U32 HUF_selectDecoder (size_t dstSize, size_t cSrcSize);
-
-/**
- *  The minimum workspace size for the `workSpace` used in
- *  HUF_readDTableX1_wksp() and HUF_readDTableX2_wksp().
- *
- *  The space used depends on HUF_TABLELOG_MAX, ranging from ~1500 bytes when
- *  HUF_TABLE_LOG_MAX=12 to ~1850 bytes when HUF_TABLE_LOG_MAX=15.
- *  Buffer overflow errors may potentially occur if code modifications result in
- *  a required workspace size greater than that specified in the following
- *  macro.
- */
-#define HUF_DECOMPRESS_WORKSPACE_SIZE ((2 << 10) + (1 << 9))
-#define HUF_DECOMPRESS_WORKSPACE_SIZE_U32 (HUF_DECOMPRESS_WORKSPACE_SIZE / sizeof(U32))
-
-
-/* ====================== */
-/* single stream variants */
-/* ====================== */
-
-size_t HUF_compress1X_usingCTable(void* dst, size_t dstSize, const void* src, size_t srcSize, const HUF_CElt* CTable, int flags);
-/** HUF_compress1X_repeat() :
- *  Same as HUF_compress1X_wksp(), but considers using hufTable if *repeat != HUF_repeat_none.
- *  If it uses hufTable it does not modify hufTable or repeat.
- *  If it doesn't, it sets *repeat = HUF_repeat_none, and it sets hufTable to the table used.
- *  If preferRepeat then the old table will always be used if valid.
- *  If suspectUncompressible then some sampling checks will be run to potentially skip huffman coding */
-size_t HUF_compress1X_repeat(void* dst, size_t dstSize,
-                       const void* src, size_t srcSize,
-                       unsigned maxSymbolValue, unsigned tableLog,
-                       void* workSpace, size_t wkspSize,   /**< `workSpace` must be aligned on 4-bytes boundaries, `wkspSize` must be >= HUF_WORKSPACE_SIZE */
-                       HUF_CElt* hufTable, HUF_repeat* repeat, int flags);
-
-size_t HUF_decompress1X_DCtx_wksp(HUF_DTable* dctx, void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize, void* workSpace, size_t wkspSize, int flags);
-#ifndef HUF_FORCE_DECOMPRESS_X1
-size_t HUF_decompress1X2_DCtx_wksp(HUF_DTable* dctx, void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize, void* workSpace, size_t wkspSize, int flags);   /**< double-symbols decoder */
-#endif
-
-/* BMI2 variants.
- * If the CPU has BMI2 support, pass bmi2=1, otherwise pass bmi2=0.
- */
-size_t HUF_decompress1X_usingDTable(void* dst, size_t maxDstSize, const void* cSrc, size_t cSrcSize, const HUF_DTable* DTable, int flags);
-#ifndef HUF_FORCE_DECOMPRESS_X2
-size_t HUF_decompress1X1_DCtx_wksp(HUF_DTable* dctx, void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize, void* workSpace, size_t wkspSize, int flags);
-#endif
-size_t HUF_decompress4X_usingDTable(void* dst, size_t maxDstSize, const void* cSrc, size_t cSrcSize, const HUF_DTable* DTable, int flags);
-size_t HUF_decompress4X_hufOnly_wksp(HUF_DTable* dctx, void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize, void* workSpace, size_t wkspSize, int flags);
-#ifndef HUF_FORCE_DECOMPRESS_X2
-size_t HUF_readDTableX1_wksp(HUF_DTable* DTable, const void* src, size_t srcSize, void* workSpace, size_t wkspSize, int flags);
-#endif
-#ifndef HUF_FORCE_DECOMPRESS_X1
-size_t HUF_readDTableX2_wksp(HUF_DTable* DTable, const void* src, size_t srcSize, void* workSpace, size_t wkspSize, int flags);
-#endif
-
-#endif   /* HUF_H_298734234 */
-
-#endif /* USE_EXTERNAL_ZSTD */

vendor/github.com/DataDog/zstd/huf_decompress_amd64.S

@@ -1,602 +0,0 @@
-/*
- * Copyright (c) Meta Platforms, Inc. and affiliates.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- * You may select, at your option, one of the above-listed licenses.
- */
-
-#include "portability_macros.h"
-
-#if defined(__ELF__) && defined(__GNUC__)
-/* Stack marking
- * ref: https://wiki.gentoo.org/wiki/Hardened/GNU_stack_quickstart
- */
-.section .note.GNU-stack,"",%progbits
-
-#if defined(__aarch64__)
-/* Mark that this assembly supports BTI & PAC, because it is empty for aarch64.
- * See: https://github.com/facebook/zstd/issues/3841
- * See: https://gcc.godbolt.org/z/sqr5T4ffK
- * See: https://lore.kernel.org/linux-arm-kernel/20200429211641.9279-8-broonie@kernel.org/
- * See: https://reviews.llvm.org/D62609
- */
-.pushsection .note.gnu.property, "a"
-.p2align 3
-.long 4                 /* size of the name - "GNU\0" */
-.long 0x10              /* size of descriptor */
-.long 0x5               /* NT_GNU_PROPERTY_TYPE_0 */
-.asciz "GNU"
-.long 0xc0000000        /* pr_type - GNU_PROPERTY_AARCH64_FEATURE_1_AND */
-.long 4                 /* pr_datasz - 4 bytes */
-.long 3                 /* pr_data - GNU_PROPERTY_AARCH64_FEATURE_1_BTI | GNU_PROPERTY_AARCH64_FEATURE_1_PAC */
-.p2align 3              /* pr_padding - bring everything to 8 byte alignment */
-.popsection
-#endif
-
-#endif
-
-#if ZSTD_ENABLE_ASM_X86_64_BMI2
-
-/* Calling convention:
- *
- * %rdi (or %rcx on Windows) contains the first argument: HUF_DecompressAsmArgs*.
- * %rbp isn't maintained (no frame pointer).
- * %rsp contains the stack pointer that grows down.
- *      No red-zone is assumed, only addresses >= %rsp are used.
- * All register contents are preserved.
- */
-
-ZSTD_HIDE_ASM_FUNCTION(HUF_decompress4X1_usingDTable_internal_fast_asm_loop)
-ZSTD_HIDE_ASM_FUNCTION(HUF_decompress4X2_usingDTable_internal_fast_asm_loop)
-ZSTD_HIDE_ASM_FUNCTION(_HUF_decompress4X2_usingDTable_internal_fast_asm_loop)
-ZSTD_HIDE_ASM_FUNCTION(_HUF_decompress4X1_usingDTable_internal_fast_asm_loop)
-.global HUF_decompress4X1_usingDTable_internal_fast_asm_loop
-.global HUF_decompress4X2_usingDTable_internal_fast_asm_loop
-.global _HUF_decompress4X1_usingDTable_internal_fast_asm_loop
-.global _HUF_decompress4X2_usingDTable_internal_fast_asm_loop
-.text
-
-/* Sets up register mappings for clarity.
- * op[], bits[], dtable & ip[0] each get their own register.
- * ip[1,2,3] & olimit alias var[].
- * %rax is a scratch register.
- */
-
-#define op0    rsi
-#define op1    rbx
-#define op2    rcx
-#define op3    rdi
-
-#define ip0    r8
-#define ip1    r9
-#define ip2    r10
-#define ip3    r11
-
-#define bits0  rbp
-#define bits1  rdx
-#define bits2  r12
-#define bits3  r13
-#define dtable r14
-#define olimit r15
-
-/* var[] aliases ip[1,2,3] & olimit
- * ip[1,2,3] are saved every iteration.
- * olimit is only used in compute_olimit.
- */
-#define var0   r15
-#define var1   r9
-#define var2   r10
-#define var3   r11
-
-/* 32-bit var registers */
-#define vard0  r15d
-#define vard1  r9d
-#define vard2  r10d
-#define vard3  r11d
-
-/* Calls X(N) for each stream 0, 1, 2, 3. */
-#define FOR_EACH_STREAM(X) \
-    X(0);                  \
-    X(1);                  \
-    X(2);                  \
-    X(3)
-
-/* Calls X(N, idx) for each stream 0, 1, 2, 3. */
-#define FOR_EACH_STREAM_WITH_INDEX(X, idx) \
-    X(0, idx);                             \
-    X(1, idx);                             \
-    X(2, idx);                             \
-    X(3, idx)
-
-/* Define both _HUF_* & HUF_* symbols because MacOS
- * C symbols are prefixed with '_' & Linux symbols aren't.
- */
-_HUF_decompress4X1_usingDTable_internal_fast_asm_loop:
-HUF_decompress4X1_usingDTable_internal_fast_asm_loop:
-    ZSTD_CET_ENDBRANCH
-    /* Save all registers - even if they are callee saved for simplicity. */
-    push %rax
-    push %rbx
-    push %rcx
-    push %rdx
-    push %rbp
-    push %rsi
-    push %rdi
-    push %r8
-    push %r9
-    push %r10
-    push %r11
-    push %r12
-    push %r13
-    push %r14
-    push %r15
-
-    /* Read HUF_DecompressAsmArgs* args from %rax */
-#if defined(_WIN32)
-    movq %rcx, %rax
-#else
-    movq %rdi, %rax
-#endif
-    movq  0(%rax), %ip0
-    movq  8(%rax), %ip1
-    movq 16(%rax), %ip2
-    movq 24(%rax), %ip3
-    movq 32(%rax), %op0
-    movq 40(%rax), %op1
-    movq 48(%rax), %op2
-    movq 56(%rax), %op3
-    movq 64(%rax), %bits0
-    movq 72(%rax), %bits1
-    movq 80(%rax), %bits2
-    movq 88(%rax), %bits3
-    movq 96(%rax), %dtable
-    push %rax      /* argument */
-    push 104(%rax) /* ilowest */
-    push 112(%rax) /* oend */
-    push %olimit   /* olimit space */
-
-    subq $24, %rsp
-
-.L_4X1_compute_olimit:
-    /* Computes how many iterations we can do safely
-     * %r15, %rax may be clobbered
-     * rbx, rdx must be saved
-     * op3 & ip0 mustn't be clobbered
-     */
-    movq %rbx, 0(%rsp)
-    movq %rdx, 8(%rsp)
-
-    movq 32(%rsp), %rax /* rax = oend */
-    subq %op3,    %rax  /* rax = oend - op3 */
-
-    /* r15 = (oend - op3) / 5 */
-    movabsq $-3689348814741910323, %rdx
-    mulq %rdx
-    movq %rdx, %r15
-    shrq $2, %r15
-
-    movq %ip0,     %rax /* rax = ip0 */
-    movq 40(%rsp), %rdx /* rdx = ilowest */
-    subq %rdx,     %rax /* rax = ip0 - ilowest */
-    movq %rax,     %rbx /* rbx = ip0 - ilowest */
-
-    /* rdx = (ip0 - ilowest) / 7 */
-    movabsq $2635249153387078803, %rdx
-    mulq %rdx
-    subq %rdx, %rbx
-    shrq %rbx
-    addq %rbx, %rdx
-    shrq $2, %rdx
-
-    /* r15 = min(%rdx, %r15) */
-    cmpq %rdx, %r15
-    cmova %rdx, %r15
-
-    /* r15 = r15 * 5 */
-    leaq (%r15, %r15, 4), %r15
-
-    /* olimit = op3 + r15 */
-    addq %op3, %olimit
-
-    movq 8(%rsp), %rdx
-    movq 0(%rsp), %rbx
-
-    /* If (op3 + 20 > olimit) */
-    movq %op3, %rax    /* rax = op3 */
-    cmpq %rax, %olimit /* op3 == olimit */
-    je .L_4X1_exit
-
-    /* If (ip1 < ip0) go to exit */
-    cmpq %ip0, %ip1
-    jb .L_4X1_exit
-
-    /* If (ip2 < ip1) go to exit */
-    cmpq %ip1, %ip2
-    jb .L_4X1_exit
-
-    /* If (ip3 < ip2) go to exit */
-    cmpq %ip2, %ip3
-    jb .L_4X1_exit
-
-/* Reads top 11 bits from bits[n]
- * Loads dt[bits[n]] into var[n]
- */
-#define GET_NEXT_DELT(n)                \
-    movq $53, %var##n;                  \
-    shrxq %var##n, %bits##n, %var##n;   \
-    movzwl (%dtable,%var##n,2),%vard##n
-
-/* var[n] must contain the DTable entry computed with GET_NEXT_DELT
- * Moves var[n] to %rax
- * bits[n] <<= var[n] & 63
- * op[n][idx] = %rax >> 8
- * %ah is a way to access bits [8, 16) of %rax
- */
-#define DECODE_FROM_DELT(n, idx)       \
-    movq %var##n, %rax;                \
-    shlxq %var##n, %bits##n, %bits##n; \
-    movb %ah, idx(%op##n)
-
-/* Assumes GET_NEXT_DELT has been called.
- * Calls DECODE_FROM_DELT then GET_NEXT_DELT
- */
-#define DECODE_AND_GET_NEXT(n, idx) \
-    DECODE_FROM_DELT(n, idx);       \
-    GET_NEXT_DELT(n)                \
-
-/* // ctz & nbBytes is stored in bits[n]
- * // nbBits is stored in %rax
- * ctz  = CTZ[bits[n]]
- * nbBits  = ctz & 7
- * nbBytes = ctz >> 3
- * op[n]  += 5
- * ip[n]  -= nbBytes
- * // Note: x86-64 is little-endian ==> no bswap
- * bits[n] = MEM_readST(ip[n]) | 1
- * bits[n] <<= nbBits
- */
-#define RELOAD_BITS(n)             \
-    bsfq %bits##n, %bits##n;       \
-    movq %bits##n, %rax;           \
-    andq $7, %rax;                 \
-    shrq $3, %bits##n;             \
-    leaq 5(%op##n), %op##n;        \
-    subq %bits##n, %ip##n;         \
-    movq (%ip##n), %bits##n;       \
-    orq $1, %bits##n;              \
-    shlx %rax, %bits##n, %bits##n
-
-    /* Store clobbered variables on the stack */
-    movq %olimit, 24(%rsp)
-    movq %ip1, 0(%rsp)
-    movq %ip2, 8(%rsp)
-    movq %ip3, 16(%rsp)
-
-    /* Call GET_NEXT_DELT for each stream */
-    FOR_EACH_STREAM(GET_NEXT_DELT)
-
-    .p2align 6
-
-.L_4X1_loop_body:
-    /* Decode 5 symbols in each of the 4 streams (20 total)
-     * Must have called GET_NEXT_DELT for each stream
-     */
-    FOR_EACH_STREAM_WITH_INDEX(DECODE_AND_GET_NEXT, 0)
-    FOR_EACH_STREAM_WITH_INDEX(DECODE_AND_GET_NEXT, 1)
-    FOR_EACH_STREAM_WITH_INDEX(DECODE_AND_GET_NEXT, 2)
-    FOR_EACH_STREAM_WITH_INDEX(DECODE_AND_GET_NEXT, 3)
-    FOR_EACH_STREAM_WITH_INDEX(DECODE_FROM_DELT, 4)
-
-    /* Load ip[1,2,3] from stack (var[] aliases them)
-     * ip[] is needed for RELOAD_BITS
-     * Each will be stored back to the stack after RELOAD
-     */
-    movq 0(%rsp), %ip1
-    movq 8(%rsp), %ip2
-    movq 16(%rsp), %ip3
-
-    /* Reload each stream & fetch the next table entry
-     * to prepare for the next iteration
-     */
-    RELOAD_BITS(0)
-    GET_NEXT_DELT(0)
-
-    RELOAD_BITS(1)
-    movq %ip1, 0(%rsp)
-    GET_NEXT_DELT(1)
-
-    RELOAD_BITS(2)
-    movq %ip2, 8(%rsp)
-    GET_NEXT_DELT(2)
-
-    RELOAD_BITS(3)
-    movq %ip3, 16(%rsp)
-    GET_NEXT_DELT(3)
-
-    /* If op3 < olimit: continue the loop */
-    cmp %op3, 24(%rsp)
-    ja .L_4X1_loop_body
-
-    /* Reload ip[1,2,3] from stack */
-    movq 0(%rsp), %ip1
-    movq 8(%rsp), %ip2
-    movq 16(%rsp), %ip3
-
-    /* Re-compute olimit */
-    jmp .L_4X1_compute_olimit
-
-#undef GET_NEXT_DELT
-#undef DECODE_FROM_DELT
-#undef DECODE
-#undef RELOAD_BITS
-.L_4X1_exit:
-    addq $24, %rsp
-
-    /* Restore stack (oend & olimit) */
-    pop %rax /* olimit */
-    pop %rax /* oend */
-    pop %rax /* ilowest */
-    pop %rax /* arg */
-
-    /* Save ip / op / bits */
-    movq %ip0,  0(%rax)
-    movq %ip1,  8(%rax)
-    movq %ip2, 16(%rax)
-    movq %ip3, 24(%rax)
-    movq %op0, 32(%rax)
-    movq %op1, 40(%rax)
-    movq %op2, 48(%rax)
-    movq %op3, 56(%rax)
-    movq %bits0, 64(%rax)
-    movq %bits1, 72(%rax)
-    movq %bits2, 80(%rax)
-    movq %bits3, 88(%rax)
-
-    /* Restore registers */
-    pop %r15
-    pop %r14
-    pop %r13
-    pop %r12
-    pop %r11
-    pop %r10
-    pop %r9
-    pop %r8
-    pop %rdi
-    pop %rsi
-    pop %rbp
-    pop %rdx
-    pop %rcx
-    pop %rbx
-    pop %rax
-    ret
-
-_HUF_decompress4X2_usingDTable_internal_fast_asm_loop:
-HUF_decompress4X2_usingDTable_internal_fast_asm_loop:
-    ZSTD_CET_ENDBRANCH
-    /* Save all registers - even if they are callee saved for simplicity. */
-    push %rax
-    push %rbx
-    push %rcx
-    push %rdx
-    push %rbp
-    push %rsi
-    push %rdi
-    push %r8
-    push %r9
-    push %r10
-    push %r11
-    push %r12
-    push %r13
-    push %r14
-    push %r15
-
-    /* Read HUF_DecompressAsmArgs* args from %rax */
-#if defined(_WIN32)
-    movq %rcx, %rax
-#else
-    movq %rdi, %rax
-#endif
-    movq  0(%rax), %ip0
-    movq  8(%rax), %ip1
-    movq 16(%rax), %ip2
-    movq 24(%rax), %ip3
-    movq 32(%rax), %op0
-    movq 40(%rax), %op1
-    movq 48(%rax), %op2
-    movq 56(%rax), %op3
-    movq 64(%rax), %bits0
-    movq 72(%rax), %bits1
-    movq 80(%rax), %bits2
-    movq 88(%rax), %bits3
-    movq 96(%rax), %dtable
-    push %rax      /* argument */
-    push %rax      /* olimit */
-    push 104(%rax) /* ilowest */
-
-    movq 112(%rax), %rax
-    push %rax /* oend3 */
-
-    movq %op3, %rax
-    push %rax /* oend2 */
-
-    movq %op2, %rax
-    push %rax /* oend1 */
-
-    movq %op1, %rax
-    push %rax /* oend0 */
-
-    /* Scratch space */
-    subq $8, %rsp
-
-.L_4X2_compute_olimit:
-    /* Computes how many iterations we can do safely
-     * %r15, %rax may be clobbered
-     * rdx must be saved
-     * op[1,2,3,4] & ip0 mustn't be clobbered
-     */
-    movq %rdx, 0(%rsp)
-
-    /* We can consume up to 7 input bytes each iteration. */
-    movq %ip0,     %rax  /* rax = ip0 */
-    movq 40(%rsp), %rdx  /* rdx = ilowest */
-    subq %rdx,     %rax  /* rax = ip0 - ilowest */
-    movq %rax,    %r15   /* r15 = ip0 - ilowest */
-
-    /* rdx = rax / 7 */
-    movabsq $2635249153387078803, %rdx
-    mulq %rdx
-    subq %rdx, %r15
-    shrq %r15
-    addq %r15, %rdx
-    shrq $2, %rdx
-
-    /* r15 = (ip0 - ilowest) / 7 */
-    movq %rdx, %r15
-
-    /* r15 = min(r15, min(oend0 - op0, oend1 - op1, oend2 - op2, oend3 - op3) / 10) */
-    movq 8(%rsp),  %rax /* rax = oend0 */
-    subq %op0,     %rax /* rax = oend0 - op0 */
-    movq 16(%rsp), %rdx /* rdx = oend1 */
-    subq %op1,     %rdx /* rdx = oend1 - op1 */
-
-    cmpq  %rax,    %rdx
-    cmova %rax,    %rdx /* rdx = min(%rdx, %rax) */
-
-    movq 24(%rsp), %rax /* rax = oend2 */
-    subq %op2,     %rax /* rax = oend2 - op2 */
-
-    cmpq  %rax,    %rdx
-    cmova %rax,    %rdx /* rdx = min(%rdx, %rax) */
-
-    movq 32(%rsp), %rax /* rax = oend3 */
-    subq %op3,     %rax /* rax = oend3 - op3 */
-
-    cmpq  %rax,    %rdx
-    cmova %rax,    %rdx /* rdx = min(%rdx, %rax) */
-
-    movabsq $-3689348814741910323, %rax
-    mulq %rdx
-    shrq $3,       %rdx /* rdx = rdx / 10 */
-
-    /* r15 = min(%rdx, %r15) */
-    cmpq  %rdx, %r15
-    cmova %rdx, %r15
-
-    /* olimit = op3 + 5 * r15 */
-    movq %r15, %rax
-    leaq (%op3, %rax, 4), %olimit
-    addq %rax, %olimit
-
-    movq 0(%rsp), %rdx
-
-    /* If (op3 + 10 > olimit) */
-    movq %op3, %rax    /* rax = op3 */
-    cmpq %rax, %olimit /* op3 == olimit */
-    je .L_4X2_exit
-
-    /* If (ip1 < ip0) go to exit */
-    cmpq %ip0, %ip1
-    jb .L_4X2_exit
-
-    /* If (ip2 < ip1) go to exit */
-    cmpq %ip1, %ip2
-    jb .L_4X2_exit
-
-    /* If (ip3 < ip2) go to exit */
-    cmpq %ip2, %ip3
-    jb .L_4X2_exit
-
-#define DECODE(n, idx)              \
-    movq %bits##n, %rax;            \
-    shrq $53, %rax;                 \
-    movzwl 0(%dtable,%rax,4),%r8d;  \
-    movzbl 2(%dtable,%rax,4),%r15d; \
-    movzbl 3(%dtable,%rax,4),%eax;  \
-    movw %r8w, (%op##n);            \
-    shlxq %r15, %bits##n, %bits##n; \
-    addq %rax, %op##n
-
-#define RELOAD_BITS(n)              \
-    bsfq %bits##n, %bits##n;        \
-    movq %bits##n, %rax;            \
-    shrq $3, %bits##n;              \
-    andq $7, %rax;                  \
-    subq %bits##n, %ip##n;          \
-    movq (%ip##n), %bits##n;        \
-    orq $1, %bits##n;               \
-    shlxq %rax, %bits##n, %bits##n
-
-
-    movq %olimit, 48(%rsp)
-
-    .p2align 6
-
-.L_4X2_loop_body:
-    /* We clobber r8, so store it on the stack */
-    movq %r8, 0(%rsp)
-
-    /* Decode 5 symbols from each of the 4 streams (20 symbols total). */
-    FOR_EACH_STREAM_WITH_INDEX(DECODE, 0)
-    FOR_EACH_STREAM_WITH_INDEX(DECODE, 1)
-    FOR_EACH_STREAM_WITH_INDEX(DECODE, 2)
-    FOR_EACH_STREAM_WITH_INDEX(DECODE, 3)
-    FOR_EACH_STREAM_WITH_INDEX(DECODE, 4)
-
-    /* Reload r8 */
-    movq 0(%rsp), %r8
-
-    FOR_EACH_STREAM(RELOAD_BITS)
-
-    cmp %op3, 48(%rsp)
-    ja .L_4X2_loop_body
-    jmp .L_4X2_compute_olimit
-
-#undef DECODE
-#undef RELOAD_BITS
-.L_4X2_exit:
-    addq $8, %rsp
-    /* Restore stack (oend & olimit) */
-    pop %rax /* oend0 */
-    pop %rax /* oend1 */
-    pop %rax /* oend2 */
-    pop %rax /* oend3 */
-    pop %rax /* ilowest */
-    pop %rax /* olimit */
-    pop %rax /* arg */
-
-    /* Save ip / op / bits */
-    movq %ip0,  0(%rax)
-    movq %ip1,  8(%rax)
-    movq %ip2, 16(%rax)
-    movq %ip3, 24(%rax)
-    movq %op0, 32(%rax)
-    movq %op1, 40(%rax)
-    movq %op2, 48(%rax)
-    movq %op3, 56(%rax)
-    movq %bits0, 64(%rax)
-    movq %bits1, 72(%rax)
-    movq %bits2, 80(%rax)
-    movq %bits3, 88(%rax)
-
-    /* Restore registers */
-    pop %r15
-    pop %r14
-    pop %r13
-    pop %r12
-    pop %r11
-    pop %r10
-    pop %r9
-    pop %r8
-    pop %rdi
-    pop %rsi
-    pop %rbp
-    pop %rdx
-    pop %rcx
-    pop %rbx
-    pop %rax
-    ret
-
-#endif

vendor/github.com/DataDog/zstd/mem.h

@@ -1,425 +0,0 @@
-#ifndef USE_EXTERNAL_ZSTD
-/*
- * Copyright (c) Meta Platforms, Inc. and affiliates.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- * You may select, at your option, one of the above-listed licenses.
- */
-
-#ifndef MEM_H_MODULE
-#define MEM_H_MODULE
-
-/*-****************************************
-*  Dependencies
-******************************************/
-#include <stddef.h>  /* size_t, ptrdiff_t */
-#include "compiler.h"  /* __has_builtin */
-#include "debug.h"  /* DEBUG_STATIC_ASSERT */
-#include "zstd_deps.h"  /* ZSTD_memcpy */
-
-
-/*-****************************************
-*  Compiler specifics
-******************************************/
-#if defined(_MSC_VER)   /* Visual Studio */
-#   include <stdlib.h>  /* _byteswap_ulong */
-#   include <intrin.h>  /* _byteswap_* */
-#elif defined(__ICCARM__)
-#   include <intrinsics.h>
-#endif
-
-/*-**************************************************************
-*  Basic Types
-*****************************************************************/
-#if  !defined (__VMS) && (defined (__cplusplus) || (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */) )
-#  if defined(_AIX)
-#    include <inttypes.h>
-#  else
-#    include <stdint.h> /* intptr_t */
-#  endif
-  typedef   uint8_t BYTE;
-  typedef   uint8_t U8;
-  typedef    int8_t S8;
-  typedef  uint16_t U16;
-  typedef   int16_t S16;
-  typedef  uint32_t U32;
-  typedef   int32_t S32;
-  typedef  uint64_t U64;
-  typedef   int64_t S64;
-#else
-# include <limits.h>
-#if CHAR_BIT != 8
-#  error "this implementation requires char to be exactly 8-bit type"
-#endif
-  typedef unsigned char      BYTE;
-  typedef unsigned char      U8;
-  typedef   signed char      S8;
-#if USHRT_MAX != 65535
-#  error "this implementation requires short to be exactly 16-bit type"
-#endif
-  typedef unsigned short      U16;
-  typedef   signed short      S16;
-#if UINT_MAX != 4294967295
-#  error "this implementation requires int to be exactly 32-bit type"
-#endif
-  typedef unsigned int        U32;
-  typedef   signed int        S32;
-/* note : there are no limits defined for long long type in C90.
- * limits exist in C99, however, in such case, <stdint.h> is preferred */
-  typedef unsigned long long  U64;
-  typedef   signed long long  S64;
-#endif
-
-/*-**************************************************************
-*  Memory I/O API
-*****************************************************************/
-/*=== Static platform detection ===*/
-MEM_STATIC unsigned MEM_32bits(void);
-MEM_STATIC unsigned MEM_64bits(void);
-MEM_STATIC unsigned MEM_isLittleEndian(void);
-
-/*=== Native unaligned read/write ===*/
-MEM_STATIC U16 MEM_read16(const void* memPtr);
-MEM_STATIC U32 MEM_read32(const void* memPtr);
-MEM_STATIC U64 MEM_read64(const void* memPtr);
-MEM_STATIC size_t MEM_readST(const void* memPtr);
-
-MEM_STATIC void MEM_write16(void* memPtr, U16 value);
-MEM_STATIC void MEM_write32(void* memPtr, U32 value);
-MEM_STATIC void MEM_write64(void* memPtr, U64 value);
-
-/*=== Little endian unaligned read/write ===*/
-MEM_STATIC U16 MEM_readLE16(const void* memPtr);
-MEM_STATIC U32 MEM_readLE24(const void* memPtr);
-MEM_STATIC U32 MEM_readLE32(const void* memPtr);
-MEM_STATIC U64 MEM_readLE64(const void* memPtr);
-MEM_STATIC size_t MEM_readLEST(const void* memPtr);
-
-MEM_STATIC void MEM_writeLE16(void* memPtr, U16 val);
-MEM_STATIC void MEM_writeLE24(void* memPtr, U32 val);
-MEM_STATIC void MEM_writeLE32(void* memPtr, U32 val32);
-MEM_STATIC void MEM_writeLE64(void* memPtr, U64 val64);
-MEM_STATIC void MEM_writeLEST(void* memPtr, size_t val);
-
-/*=== Big endian unaligned read/write ===*/
-MEM_STATIC U32 MEM_readBE32(const void* memPtr);
-MEM_STATIC U64 MEM_readBE64(const void* memPtr);
-MEM_STATIC size_t MEM_readBEST(const void* memPtr);
-
-MEM_STATIC void MEM_writeBE32(void* memPtr, U32 val32);
-MEM_STATIC void MEM_writeBE64(void* memPtr, U64 val64);
-MEM_STATIC void MEM_writeBEST(void* memPtr, size_t val);
-
-/*=== Byteswap ===*/
-MEM_STATIC U32 MEM_swap32(U32 in);
-MEM_STATIC U64 MEM_swap64(U64 in);
-MEM_STATIC size_t MEM_swapST(size_t in);
-
-
-/*-**************************************************************
-*  Memory I/O Implementation
-*****************************************************************/
-/* MEM_FORCE_MEMORY_ACCESS : For accessing unaligned memory:
- * Method 0 : always use `memcpy()`. Safe and portable.
- * Method 1 : Use compiler extension to set unaligned access.
- * Method 2 : direct access. This method is portable but violate C standard.
- *            It can generate buggy code on targets depending on alignment.
- * Default  : method 1 if supported, else method 0
- */
-#ifndef MEM_FORCE_MEMORY_ACCESS   /* can be defined externally, on command line for example */
-#  ifdef __GNUC__
-#    define MEM_FORCE_MEMORY_ACCESS 1
-#  endif
-#endif
-
-MEM_STATIC unsigned MEM_32bits(void) { return sizeof(size_t)==4; }
-MEM_STATIC unsigned MEM_64bits(void) { return sizeof(size_t)==8; }
-
-MEM_STATIC unsigned MEM_isLittleEndian(void)
-{
-#if defined(__BYTE_ORDER__) && defined(__ORDER_LITTLE_ENDIAN__) && (__BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__)
-    return 1;
-#elif defined(__BYTE_ORDER__) && defined(__ORDER_BIG_ENDIAN__) && (__BYTE_ORDER__ == __ORDER_BIG_ENDIAN__)
-    return 0;
-#elif defined(__clang__) && __LITTLE_ENDIAN__
-    return 1;
-#elif defined(__clang__) && __BIG_ENDIAN__
-    return 0;
-#elif defined(_MSC_VER) && (_M_X64 || _M_IX86)
-    return 1;
-#elif defined(__DMC__) && defined(_M_IX86)
-    return 1;
-#elif defined(__IAR_SYSTEMS_ICC__) && __LITTLE_ENDIAN__
-    return 1;
-#else
-    const union { U32 u; BYTE c[4]; } one = { 1 };   /* don't use static : performance detrimental  */
-    return one.c[0];
-#endif
-}
-
-#if defined(MEM_FORCE_MEMORY_ACCESS) && (MEM_FORCE_MEMORY_ACCESS==2)
-
-/* violates C standard, by lying on structure alignment.
-Only use if no other choice to achieve best performance on target platform */
-MEM_STATIC U16 MEM_read16(const void* memPtr) { return *(const U16*) memPtr; }
-MEM_STATIC U32 MEM_read32(const void* memPtr) { return *(const U32*) memPtr; }
-MEM_STATIC U64 MEM_read64(const void* memPtr) { return *(const U64*) memPtr; }
-MEM_STATIC size_t MEM_readST(const void* memPtr) { return *(const size_t*) memPtr; }
-
-MEM_STATIC void MEM_write16(void* memPtr, U16 value) { *(U16*)memPtr = value; }
-MEM_STATIC void MEM_write32(void* memPtr, U32 value) { *(U32*)memPtr = value; }
-MEM_STATIC void MEM_write64(void* memPtr, U64 value) { *(U64*)memPtr = value; }
-
-#elif defined(MEM_FORCE_MEMORY_ACCESS) && (MEM_FORCE_MEMORY_ACCESS==1)
-
-typedef __attribute__((aligned(1))) U16 unalign16;
-typedef __attribute__((aligned(1))) U32 unalign32;
-typedef __attribute__((aligned(1))) U64 unalign64;
-typedef __attribute__((aligned(1))) size_t unalignArch;
-
-MEM_STATIC U16 MEM_read16(const void* ptr) { return *(const unalign16*)ptr; }
-MEM_STATIC U32 MEM_read32(const void* ptr) { return *(const unalign32*)ptr; }
-MEM_STATIC U64 MEM_read64(const void* ptr) { return *(const unalign64*)ptr; }
-MEM_STATIC size_t MEM_readST(const void* ptr) { return *(const unalignArch*)ptr; }
-
-MEM_STATIC void MEM_write16(void* memPtr, U16 value) { *(unalign16*)memPtr = value; }
-MEM_STATIC void MEM_write32(void* memPtr, U32 value) { *(unalign32*)memPtr = value; }
-MEM_STATIC void MEM_write64(void* memPtr, U64 value) { *(unalign64*)memPtr = value; }
-
-#else
-
-/* default method, safe and standard.
-   can sometimes prove slower */
-
-MEM_STATIC U16 MEM_read16(const void* memPtr)
-{
-    U16 val; ZSTD_memcpy(&val, memPtr, sizeof(val)); return val;
-}
-
-MEM_STATIC U32 MEM_read32(const void* memPtr)
-{
-    U32 val; ZSTD_memcpy(&val, memPtr, sizeof(val)); return val;
-}
-
-MEM_STATIC U64 MEM_read64(const void* memPtr)
-{
-    U64 val; ZSTD_memcpy(&val, memPtr, sizeof(val)); return val;
-}
-
-MEM_STATIC size_t MEM_readST(const void* memPtr)
-{
-    size_t val; ZSTD_memcpy(&val, memPtr, sizeof(val)); return val;
-}
-
-MEM_STATIC void MEM_write16(void* memPtr, U16 value)
-{
-    ZSTD_memcpy(memPtr, &value, sizeof(value));
-}
-
-MEM_STATIC void MEM_write32(void* memPtr, U32 value)
-{
-    ZSTD_memcpy(memPtr, &value, sizeof(value));
-}
-
-MEM_STATIC void MEM_write64(void* memPtr, U64 value)
-{
-    ZSTD_memcpy(memPtr, &value, sizeof(value));
-}
-
-#endif /* MEM_FORCE_MEMORY_ACCESS */
-
-MEM_STATIC U32 MEM_swap32_fallback(U32 in)
-{
-    return  ((in << 24) & 0xff000000 ) |
-            ((in <<  8) & 0x00ff0000 ) |
-            ((in >>  8) & 0x0000ff00 ) |
-            ((in >> 24) & 0x000000ff );
-}
-
-MEM_STATIC U32 MEM_swap32(U32 in)
-{
-#if defined(_MSC_VER)     /* Visual Studio */
-    return _byteswap_ulong(in);
-#elif (defined (__GNUC__) && (__GNUC__ * 100 + __GNUC_MINOR__ >= 403)) \
-  || (defined(__clang__) && __has_builtin(__builtin_bswap32))
-    return __builtin_bswap32(in);
-#elif defined(__ICCARM__)
-    return __REV(in);
-#else
-    return MEM_swap32_fallback(in);
-#endif
-}
-
-MEM_STATIC U64 MEM_swap64_fallback(U64 in)
-{
-     return  ((in << 56) & 0xff00000000000000ULL) |
-            ((in << 40) & 0x00ff000000000000ULL) |
-            ((in << 24) & 0x0000ff0000000000ULL) |
-            ((in << 8)  & 0x000000ff00000000ULL) |
-            ((in >> 8)  & 0x00000000ff000000ULL) |
-            ((in >> 24) & 0x0000000000ff0000ULL) |
-            ((in >> 40) & 0x000000000000ff00ULL) |
-            ((in >> 56) & 0x00000000000000ffULL);
-}
-
-MEM_STATIC U64 MEM_swap64(U64 in)
-{
-#if defined(_MSC_VER)     /* Visual Studio */
-    return _byteswap_uint64(in);
-#elif (defined (__GNUC__) && (__GNUC__ * 100 + __GNUC_MINOR__ >= 403)) \
-  || (defined(__clang__) && __has_builtin(__builtin_bswap64))
-    return __builtin_bswap64(in);
-#else
-    return MEM_swap64_fallback(in);
-#endif
-}
-
-MEM_STATIC size_t MEM_swapST(size_t in)
-{
-    if (MEM_32bits())
-        return (size_t)MEM_swap32((U32)in);
-    else
-        return (size_t)MEM_swap64((U64)in);
-}
-
-/*=== Little endian r/w ===*/
-
-MEM_STATIC U16 MEM_readLE16(const void* memPtr)
-{
-    if (MEM_isLittleEndian())
-        return MEM_read16(memPtr);
-    else {
-        const BYTE* p = (const BYTE*)memPtr;
-        return (U16)(p[0] + (p[1]<<8));
-    }
-}
-
-MEM_STATIC void MEM_writeLE16(void* memPtr, U16 val)
-{
-    if (MEM_isLittleEndian()) {
-        MEM_write16(memPtr, val);
-    } else {
-        BYTE* p = (BYTE*)memPtr;
-        p[0] = (BYTE)val;
-        p[1] = (BYTE)(val>>8);
-    }
-}
-
-MEM_STATIC U32 MEM_readLE24(const void* memPtr)
-{
-    return (U32)MEM_readLE16(memPtr) + ((U32)(((const BYTE*)memPtr)[2]) << 16);
-}
-
-MEM_STATIC void MEM_writeLE24(void* memPtr, U32 val)
-{
-    MEM_writeLE16(memPtr, (U16)val);
-    ((BYTE*)memPtr)[2] = (BYTE)(val>>16);
-}
-
-MEM_STATIC U32 MEM_readLE32(const void* memPtr)
-{
-    if (MEM_isLittleEndian())
-        return MEM_read32(memPtr);
-    else
-        return MEM_swap32(MEM_read32(memPtr));
-}
-
-MEM_STATIC void MEM_writeLE32(void* memPtr, U32 val32)
-{
-    if (MEM_isLittleEndian())
-        MEM_write32(memPtr, val32);
-    else
-        MEM_write32(memPtr, MEM_swap32(val32));
-}
-
-MEM_STATIC U64 MEM_readLE64(const void* memPtr)
-{
-    if (MEM_isLittleEndian())
-        return MEM_read64(memPtr);
-    else
-        return MEM_swap64(MEM_read64(memPtr));
-}
-
-MEM_STATIC void MEM_writeLE64(void* memPtr, U64 val64)
-{
-    if (MEM_isLittleEndian())
-        MEM_write64(memPtr, val64);
-    else
-        MEM_write64(memPtr, MEM_swap64(val64));
-}
-
-MEM_STATIC size_t MEM_readLEST(const void* memPtr)
-{
-    if (MEM_32bits())
-        return (size_t)MEM_readLE32(memPtr);
-    else
-        return (size_t)MEM_readLE64(memPtr);
-}
-
-MEM_STATIC void MEM_writeLEST(void* memPtr, size_t val)
-{
-    if (MEM_32bits())
-        MEM_writeLE32(memPtr, (U32)val);
-    else
-        MEM_writeLE64(memPtr, (U64)val);
-}
-
-/*=== Big endian r/w ===*/
-
-MEM_STATIC U32 MEM_readBE32(const void* memPtr)
-{
-    if (MEM_isLittleEndian())
-        return MEM_swap32(MEM_read32(memPtr));
-    else
-        return MEM_read32(memPtr);
-}
-
-MEM_STATIC void MEM_writeBE32(void* memPtr, U32 val32)
-{
-    if (MEM_isLittleEndian())
-        MEM_write32(memPtr, MEM_swap32(val32));
-    else
-        MEM_write32(memPtr, val32);
-}
-
-MEM_STATIC U64 MEM_readBE64(const void* memPtr)
-{
-    if (MEM_isLittleEndian())
-        return MEM_swap64(MEM_read64(memPtr));
-    else
-        return MEM_read64(memPtr);
-}
-
-MEM_STATIC void MEM_writeBE64(void* memPtr, U64 val64)
-{
-    if (MEM_isLittleEndian())
-        MEM_write64(memPtr, MEM_swap64(val64));
-    else
-        MEM_write64(memPtr, val64);
-}
-
-MEM_STATIC size_t MEM_readBEST(const void* memPtr)
-{
-    if (MEM_32bits())
-        return (size_t)MEM_readBE32(memPtr);
-    else
-        return (size_t)MEM_readBE64(memPtr);
-}
-
-MEM_STATIC void MEM_writeBEST(void* memPtr, size_t val)
-{
-    if (MEM_32bits())
-        MEM_writeBE32(memPtr, (U32)val);
-    else
-        MEM_writeBE64(memPtr, (U64)val);
-}
-
-/* code only tested on 32 and 64 bits systems */
-MEM_STATIC void MEM_check(void) { DEBUG_STATIC_ASSERT((sizeof(size_t)==4) || (sizeof(size_t)==8)); }
-
-#endif /* MEM_H_MODULE */
-
-#endif /* USE_EXTERNAL_ZSTD */

vendor/github.com/DataDog/zstd/pool.c

@@ -1,374 +0,0 @@
-#ifndef USE_EXTERNAL_ZSTD
-/*
- * Copyright (c) Meta Platforms, Inc. and affiliates.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- * You may select, at your option, one of the above-listed licenses.
- */
-
-
-/* ======   Dependencies   ======= */
-#include "allocations.h"  /* ZSTD_customCalloc, ZSTD_customFree */
-#include "zstd_deps.h" /* size_t */
-#include "debug.h"     /* assert */
-#include "pool.h"
-
-/* ======   Compiler specifics   ====== */
-#if defined(_MSC_VER)
-#  pragma warning(disable : 4204)        /* disable: C4204: non-constant aggregate initializer */
-#endif
-
-
-#ifdef ZSTD_MULTITHREAD
-
-#include "threading.h"   /* pthread adaptation */
-
-/* A job is a function and an opaque argument */
-typedef struct POOL_job_s {
-    POOL_function function;
-    void *opaque;
-} POOL_job;
-
-struct POOL_ctx_s {
-    ZSTD_customMem customMem;
-    /* Keep track of the threads */
-    ZSTD_pthread_t* threads;
-    size_t threadCapacity;
-    size_t threadLimit;
-
-    /* The queue is a circular buffer */
-    POOL_job *queue;
-    size_t queueHead;
-    size_t queueTail;
-    size_t queueSize;
-
-    /* The number of threads working on jobs */
-    size_t numThreadsBusy;
-    /* Indicates if the queue is empty */
-    int queueEmpty;
-
-    /* The mutex protects the queue */
-    ZSTD_pthread_mutex_t queueMutex;
-    /* Condition variable for pushers to wait on when the queue is full */
-    ZSTD_pthread_cond_t queuePushCond;
-    /* Condition variables for poppers to wait on when the queue is empty */
-    ZSTD_pthread_cond_t queuePopCond;
-    /* Indicates if the queue is shutting down */
-    int shutdown;
-};
-
-/* POOL_thread() :
- * Work thread for the thread pool.
- * Waits for jobs and executes them.
- * @returns : NULL on failure else non-null.
- */
-static void* POOL_thread(void* opaque) {
-    POOL_ctx* const ctx = (POOL_ctx*)opaque;
-    if (!ctx) { return NULL; }
-    for (;;) {
-        /* Lock the mutex and wait for a non-empty queue or until shutdown */
-        ZSTD_pthread_mutex_lock(&ctx->queueMutex);
-
-        while ( ctx->queueEmpty
-            || (ctx->numThreadsBusy >= ctx->threadLimit) ) {
-            if (ctx->shutdown) {
-                /* even if !queueEmpty, (possible if numThreadsBusy >= threadLimit),
-                 * a few threads will be shutdown while !queueEmpty,
-                 * but enough threads will remain active to finish the queue */
-                ZSTD_pthread_mutex_unlock(&ctx->queueMutex);
-                return opaque;
-            }
-            ZSTD_pthread_cond_wait(&ctx->queuePopCond, &ctx->queueMutex);
-        }
-        /* Pop a job off the queue */
-        {   POOL_job const job = ctx->queue[ctx->queueHead];
-            ctx->queueHead = (ctx->queueHead + 1) % ctx->queueSize;
-            ctx->numThreadsBusy++;
-            ctx->queueEmpty = (ctx->queueHead == ctx->queueTail);
-            /* Unlock the mutex, signal a pusher, and run the job */
-            ZSTD_pthread_cond_signal(&ctx->queuePushCond);
-            ZSTD_pthread_mutex_unlock(&ctx->queueMutex);
-
-            job.function(job.opaque);
-
-            /* If the intended queue size was 0, signal after finishing job */
-            ZSTD_pthread_mutex_lock(&ctx->queueMutex);
-            ctx->numThreadsBusy--;
-            ZSTD_pthread_cond_signal(&ctx->queuePushCond);
-            ZSTD_pthread_mutex_unlock(&ctx->queueMutex);
-        }
-    }  /* for (;;) */
-    assert(0);  /* Unreachable */
-}
-
-/* ZSTD_createThreadPool() : public access point */
-POOL_ctx* ZSTD_createThreadPool(size_t numThreads) {
-    return POOL_create (numThreads, 0);
-}
-
-POOL_ctx* POOL_create(size_t numThreads, size_t queueSize) {
-    return POOL_create_advanced(numThreads, queueSize, ZSTD_defaultCMem);
-}
-
-POOL_ctx* POOL_create_advanced(size_t numThreads, size_t queueSize,
-                               ZSTD_customMem customMem)
-{
-    POOL_ctx* ctx;
-    /* Check parameters */
-    if (!numThreads) { return NULL; }
-    /* Allocate the context and zero initialize */
-    ctx = (POOL_ctx*)ZSTD_customCalloc(sizeof(POOL_ctx), customMem);
-    if (!ctx) { return NULL; }
-    /* Initialize the job queue.
-     * It needs one extra space since one space is wasted to differentiate
-     * empty and full queues.
-     */
-    ctx->queueSize = queueSize + 1;
-    ctx->queue = (POOL_job*)ZSTD_customCalloc(ctx->queueSize * sizeof(POOL_job), customMem);
-    ctx->queueHead = 0;
-    ctx->queueTail = 0;
-    ctx->numThreadsBusy = 0;
-    ctx->queueEmpty = 1;
-    {
-        int error = 0;
-        error |= ZSTD_pthread_mutex_init(&ctx->queueMutex, NULL);
-        error |= ZSTD_pthread_cond_init(&ctx->queuePushCond, NULL);
-        error |= ZSTD_pthread_cond_init(&ctx->queuePopCond, NULL);
-        if (error) { POOL_free(ctx); return NULL; }
-    }
-    ctx->shutdown = 0;
-    /* Allocate space for the thread handles */
-    ctx->threads = (ZSTD_pthread_t*)ZSTD_customCalloc(numThreads * sizeof(ZSTD_pthread_t), customMem);
-    ctx->threadCapacity = 0;
-    ctx->customMem = customMem;
-    /* Check for errors */
-    if (!ctx->threads || !ctx->queue) { POOL_free(ctx); return NULL; }
-    /* Initialize the threads */
-    {   size_t i;
-        for (i = 0; i < numThreads; ++i) {
-            if (ZSTD_pthread_create(&ctx->threads[i], NULL, &POOL_thread, ctx)) {
-                ctx->threadCapacity = i;
-                POOL_free(ctx);
-                return NULL;
-        }   }
-        ctx->threadCapacity = numThreads;
-        ctx->threadLimit = numThreads;
-    }
-    return ctx;
-}
-
-/*! POOL_join() :
-    Shutdown the queue, wake any sleeping threads, and join all of the threads.
-*/
-static void POOL_join(POOL_ctx* ctx) {
-    /* Shut down the queue */
-    ZSTD_pthread_mutex_lock(&ctx->queueMutex);
-    ctx->shutdown = 1;
-    ZSTD_pthread_mutex_unlock(&ctx->queueMutex);
-    /* Wake up sleeping threads */
-    ZSTD_pthread_cond_broadcast(&ctx->queuePushCond);
-    ZSTD_pthread_cond_broadcast(&ctx->queuePopCond);
-    /* Join all of the threads */
-    {   size_t i;
-        for (i = 0; i < ctx->threadCapacity; ++i) {
-            ZSTD_pthread_join(ctx->threads[i]);  /* note : could fail */
-    }   }
-}
-
-void POOL_free(POOL_ctx *ctx) {
-    if (!ctx) { return; }
-    POOL_join(ctx);
-    ZSTD_pthread_mutex_destroy(&ctx->queueMutex);
-    ZSTD_pthread_cond_destroy(&ctx->queuePushCond);
-    ZSTD_pthread_cond_destroy(&ctx->queuePopCond);
-    ZSTD_customFree(ctx->queue, ctx->customMem);
-    ZSTD_customFree(ctx->threads, ctx->customMem);
-    ZSTD_customFree(ctx, ctx->customMem);
-}
-
-/*! POOL_joinJobs() :
- *  Waits for all queued jobs to finish executing.
- */
-void POOL_joinJobs(POOL_ctx* ctx) {
-    ZSTD_pthread_mutex_lock(&ctx->queueMutex);
-    while(!ctx->queueEmpty || ctx->numThreadsBusy > 0) {
-        ZSTD_pthread_cond_wait(&ctx->queuePushCond, &ctx->queueMutex);
-    }
-    ZSTD_pthread_mutex_unlock(&ctx->queueMutex);
-}
-
-void ZSTD_freeThreadPool (ZSTD_threadPool* pool) {
-  POOL_free (pool);
-}
-
-size_t POOL_sizeof(const POOL_ctx* ctx) {
-    if (ctx==NULL) return 0;  /* supports sizeof NULL */
-    return sizeof(*ctx)
-        + ctx->queueSize * sizeof(POOL_job)
-        + ctx->threadCapacity * sizeof(ZSTD_pthread_t);
-}
-
-
-/* @return : 0 on success, 1 on error */
-static int POOL_resize_internal(POOL_ctx* ctx, size_t numThreads)
-{
-    if (numThreads <= ctx->threadCapacity) {
-        if (!numThreads) return 1;
-        ctx->threadLimit = numThreads;
-        return 0;
-    }
-    /* numThreads > threadCapacity */
-    {   ZSTD_pthread_t* const threadPool = (ZSTD_pthread_t*)ZSTD_customCalloc(numThreads * sizeof(ZSTD_pthread_t), ctx->customMem);
-        if (!threadPool) return 1;
-        /* replace existing thread pool */
-        ZSTD_memcpy(threadPool, ctx->threads, ctx->threadCapacity * sizeof(ZSTD_pthread_t));
-        ZSTD_customFree(ctx->threads, ctx->customMem);
-        ctx->threads = threadPool;
-        /* Initialize additional threads */
-        {   size_t threadId;
-            for (threadId = ctx->threadCapacity; threadId < numThreads; ++threadId) {
-                if (ZSTD_pthread_create(&threadPool[threadId], NULL, &POOL_thread, ctx)) {
-                    ctx->threadCapacity = threadId;
-                    return 1;
-            }   }
-    }   }
-    /* successfully expanded */
-    ctx->threadCapacity = numThreads;
-    ctx->threadLimit = numThreads;
-    return 0;
-}
-
-/* @return : 0 on success, 1 on error */
-int POOL_resize(POOL_ctx* ctx, size_t numThreads)
-{
-    int result;
-    if (ctx==NULL) return 1;
-    ZSTD_pthread_mutex_lock(&ctx->queueMutex);
-    result = POOL_resize_internal(ctx, numThreads);
-    ZSTD_pthread_cond_broadcast(&ctx->queuePopCond);
-    ZSTD_pthread_mutex_unlock(&ctx->queueMutex);
-    return result;
-}
-
-/**
- * Returns 1 if the queue is full and 0 otherwise.
- *
- * When queueSize is 1 (pool was created with an intended queueSize of 0),
- * then a queue is empty if there is a thread free _and_ no job is waiting.
- */
-static int isQueueFull(POOL_ctx const* ctx) {
-    if (ctx->queueSize > 1) {
-        return ctx->queueHead == ((ctx->queueTail + 1) % ctx->queueSize);
-    } else {
-        return (ctx->numThreadsBusy == ctx->threadLimit) ||
-               !ctx->queueEmpty;
-    }
-}
-
-
-static void
-POOL_add_internal(POOL_ctx* ctx, POOL_function function, void *opaque)
-{
-    POOL_job job;
-    job.function = function;
-    job.opaque = opaque;
-    assert(ctx != NULL);
-    if (ctx->shutdown) return;
-
-    ctx->queueEmpty = 0;
-    ctx->queue[ctx->queueTail] = job;
-    ctx->queueTail = (ctx->queueTail + 1) % ctx->queueSize;
-    ZSTD_pthread_cond_signal(&ctx->queuePopCond);
-}
-
-void POOL_add(POOL_ctx* ctx, POOL_function function, void* opaque)
-{
-    assert(ctx != NULL);
-    ZSTD_pthread_mutex_lock(&ctx->queueMutex);
-    /* Wait until there is space in the queue for the new job */
-    while (isQueueFull(ctx) && (!ctx->shutdown)) {
-        ZSTD_pthread_cond_wait(&ctx->queuePushCond, &ctx->queueMutex);
-    }
-    POOL_add_internal(ctx, function, opaque);
-    ZSTD_pthread_mutex_unlock(&ctx->queueMutex);
-}
-
-
-int POOL_tryAdd(POOL_ctx* ctx, POOL_function function, void* opaque)
-{
-    assert(ctx != NULL);
-    ZSTD_pthread_mutex_lock(&ctx->queueMutex);
-    if (isQueueFull(ctx)) {
-        ZSTD_pthread_mutex_unlock(&ctx->queueMutex);
-        return 0;
-    }
-    POOL_add_internal(ctx, function, opaque);
-    ZSTD_pthread_mutex_unlock(&ctx->queueMutex);
-    return 1;
-}
-
-
-#else  /* ZSTD_MULTITHREAD  not defined */
-
-/* ========================== */
-/* No multi-threading support */
-/* ========================== */
-
-
-/* We don't need any data, but if it is empty, malloc() might return NULL. */
-struct POOL_ctx_s {
-    int dummy;
-};
-static POOL_ctx g_poolCtx;
-
-POOL_ctx* POOL_create(size_t numThreads, size_t queueSize) {
-    return POOL_create_advanced(numThreads, queueSize, ZSTD_defaultCMem);
-}
-
-POOL_ctx*
-POOL_create_advanced(size_t numThreads, size_t queueSize, ZSTD_customMem customMem)
-{
-    (void)numThreads;
-    (void)queueSize;
-    (void)customMem;
-    return &g_poolCtx;
-}
-
-void POOL_free(POOL_ctx* ctx) {
-    assert(!ctx || ctx == &g_poolCtx);
-    (void)ctx;
-}
-
-void POOL_joinJobs(POOL_ctx* ctx){
-    assert(!ctx || ctx == &g_poolCtx);
-    (void)ctx;
-}
-
-int POOL_resize(POOL_ctx* ctx, size_t numThreads) {
-    (void)ctx; (void)numThreads;
-    return 0;
-}
-
-void POOL_add(POOL_ctx* ctx, POOL_function function, void* opaque) {
-    (void)ctx;
-    function(opaque);
-}
-
-int POOL_tryAdd(POOL_ctx* ctx, POOL_function function, void* opaque) {
-    (void)ctx;
-    function(opaque);
-    return 1;
-}
-
-size_t POOL_sizeof(const POOL_ctx* ctx) {
-    if (ctx==NULL) return 0;  /* supports sizeof NULL */
-    assert(ctx == &g_poolCtx);
-    return sizeof(*ctx);
-}
-
-#endif  /* ZSTD_MULTITHREAD */
-
-#endif /* USE_EXTERNAL_ZSTD */

vendor/github.com/DataDog/zstd/pool.h

@@ -1,84 +0,0 @@
-#ifndef USE_EXTERNAL_ZSTD
-/*
- * Copyright (c) Meta Platforms, Inc. and affiliates.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- * You may select, at your option, one of the above-listed licenses.
- */
-
-#ifndef POOL_H
-#define POOL_H
-
-
-#include "zstd_deps.h"
-#define ZSTD_STATIC_LINKING_ONLY   /* ZSTD_customMem */
-#include "zstd.h"
-
-typedef struct POOL_ctx_s POOL_ctx;
-
-/*! POOL_create() :
- *  Create a thread pool with at most `numThreads` threads.
- * `numThreads` must be at least 1.
- *  The maximum number of queued jobs before blocking is `queueSize`.
- * @return : POOL_ctx pointer on success, else NULL.
-*/
-POOL_ctx* POOL_create(size_t numThreads, size_t queueSize);
-
-POOL_ctx* POOL_create_advanced(size_t numThreads, size_t queueSize,
-                               ZSTD_customMem customMem);
-
-/*! POOL_free() :
- *  Free a thread pool returned by POOL_create().
- */
-void POOL_free(POOL_ctx* ctx);
-
-
-/*! POOL_joinJobs() :
- *  Waits for all queued jobs to finish executing.
- */
-void POOL_joinJobs(POOL_ctx* ctx);
-
-/*! POOL_resize() :
- *  Expands or shrinks pool's number of threads.
- *  This is more efficient than releasing + creating a new context,
- *  since it tries to preserve and reuse existing threads.
- * `numThreads` must be at least 1.
- * @return : 0 when resize was successful,
- *           !0 (typically 1) if there is an error.
- *    note : only numThreads can be resized, queueSize remains unchanged.
- */
-int POOL_resize(POOL_ctx* ctx, size_t numThreads);
-
-/*! POOL_sizeof() :
- * @return threadpool memory usage
- *  note : compatible with NULL (returns 0 in this case)
- */
-size_t POOL_sizeof(const POOL_ctx* ctx);
-
-/*! POOL_function :
- *  The function type that can be added to a thread pool.
- */
-typedef void (*POOL_function)(void*);
-
-/*! POOL_add() :
- *  Add the job `function(opaque)` to the thread pool. `ctx` must be valid.
- *  Possibly blocks until there is room in the queue.
- *  Note : The function may be executed asynchronously,
- *         therefore, `opaque` must live until function has been completed.
- */
-void POOL_add(POOL_ctx* ctx, POOL_function function, void* opaque);
-
-
-/*! POOL_tryAdd() :
- *  Add the job `function(opaque)` to thread pool _if_ a queue slot is available.
- *  Returns immediately even if not (does not block).
- * @return : 1 if successful, 0 if not.
- */
-int POOL_tryAdd(POOL_ctx* ctx, POOL_function function, void* opaque);
-
-#endif
-
-#endif /* USE_EXTERNAL_ZSTD */

vendor/github.com/DataDog/zstd/portability_macros.h

@@ -1,174 +0,0 @@
-#ifndef USE_EXTERNAL_ZSTD
-/*
- * Copyright (c) Meta Platforms, Inc. and affiliates.
- * All rights reserved.
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- * You may select, at your option, one of the above-listed licenses.
- */
-
-#ifndef ZSTD_PORTABILITY_MACROS_H
-#define ZSTD_PORTABILITY_MACROS_H
-
-/**
- * This header file contains macro definitions to support portability.
- * This header is shared between C and ASM code, so it MUST only
- * contain macro definitions. It MUST not contain any C code.
- *
- * This header ONLY defines macros to detect platforms/feature support.
- *
- */
-
-
-/* compat. with non-clang compilers */
-#ifndef __has_attribute
-  #define __has_attribute(x) 0
-#endif
-
-/* compat. with non-clang compilers */
-#ifndef __has_builtin
-#  define __has_builtin(x) 0
-#endif
-
-/* compat. with non-clang compilers */
-#ifndef __has_feature
-#  define __has_feature(x) 0
-#endif
-
-/* detects whether we are being compiled under msan */
-#ifndef ZSTD_MEMORY_SANITIZER
-#  if __has_feature(memory_sanitizer)
-#    define ZSTD_MEMORY_SANITIZER 1
-#  else
-#    define ZSTD_MEMORY_SANITIZER 0
-#  endif
-#endif
-
-/* detects whether we are being compiled under asan */
-#ifndef ZSTD_ADDRESS_SANITIZER
-#  if __has_feature(address_sanitizer)
-#    define ZSTD_ADDRESS_SANITIZER 1
-#  elif defined(__SANITIZE_ADDRESS__)
-#    define ZSTD_ADDRESS_SANITIZER 1
-#  else
-#    define ZSTD_ADDRESS_SANITIZER 0
-#  endif
-#endif
-
-/* detects whether we are being compiled under dfsan */
-#ifndef ZSTD_DATAFLOW_SANITIZER
-# if __has_feature(dataflow_sanitizer)
-#  define ZSTD_DATAFLOW_SANITIZER 1
-# else
-#  define ZSTD_DATAFLOW_SANITIZER 0
-# endif
-#endif
-
-/* Mark the internal assembly functions as hidden  */
-#ifdef __ELF__
-# define ZSTD_HIDE_ASM_FUNCTION(func) .hidden func
-#elif defined(__APPLE__)
-# define ZSTD_HIDE_ASM_FUNCTION(func) .private_extern func
-#else
-# define ZSTD_HIDE_ASM_FUNCTION(func)
-#endif
-
-/* Compile time determination of BMI2 support */
-#ifndef STATIC_BMI2
-#  if defined(__BMI2__)
-#    define STATIC_BMI2 1
-#  elif defined(_MSC_VER) && defined(__AVX2__)
-#    define STATIC_BMI2 1 /* MSVC does not have a BMI2 specific flag, but every CPU that supports AVX2 also supports BMI2 */
-#  endif
-#endif
-
-#ifndef STATIC_BMI2
-#  define STATIC_BMI2 0
-#endif
-
-/* Enable runtime BMI2 dispatch based on the CPU.
- * Enabled for clang & gcc >=4.8 on x86 when BMI2 isn't enabled by default.
- */
-#ifndef DYNAMIC_BMI2
-#  if ((defined(__clang__) && __has_attribute(__target__)) \
-      || (defined(__GNUC__) \
-          && (__GNUC__ >= 5 || (__GNUC__ == 4 && __GNUC_MINOR__ >= 8)))) \
-      && (defined(__i386__) || defined(__x86_64__) || defined(_M_IX86) || defined(_M_X64)) \
-      && !defined(__BMI2__)
-#    define DYNAMIC_BMI2 1
-#  else
-#    define DYNAMIC_BMI2 0
-#  endif
-#endif
-
-/**
- * Only enable assembly for GNU C compatible compilers,
- * because other platforms may not support GAS assembly syntax.
- *
- * Only enable assembly for Linux / MacOS / Win32, other platforms may
- * work, but they haven't been tested. This could likely be
- * extended to BSD systems.
- *
- * Disable assembly when MSAN is enabled, because MSAN requires
- * 100% of code to be instrumented to work.
- */
-#if defined(__GNUC__)
-#  if defined(__linux__) || defined(__linux) || defined(__APPLE__) || defined(_WIN32)
-#    if ZSTD_MEMORY_SANITIZER
-#      define ZSTD_ASM_SUPPORTED 0
-#    elif ZSTD_DATAFLOW_SANITIZER
-#      define ZSTD_ASM_SUPPORTED 0
-#    else
-#      define ZSTD_ASM_SUPPORTED 1
-#    endif
-#  else
-#    define ZSTD_ASM_SUPPORTED 0
-#  endif
-#else
-#  define ZSTD_ASM_SUPPORTED 0
-#endif
-
-/**
- * Determines whether we should enable assembly for x86-64
- * with BMI2.
- *
- * Enable if all of the following conditions hold:
- * - ASM hasn't been explicitly disabled by defining ZSTD_DISABLE_ASM
- * - Assembly is supported
- * - We are compiling for x86-64 and either:
- *   - DYNAMIC_BMI2 is enabled
- *   - BMI2 is supported at compile time
- */
-#if !defined(ZSTD_DISABLE_ASM) &&                                 \
-    ZSTD_ASM_SUPPORTED &&                                         \
-    defined(__x86_64__) &&                                        \
-    (DYNAMIC_BMI2 || defined(__BMI2__))
-# define ZSTD_ENABLE_ASM_X86_64_BMI2 1
-#else
-# define ZSTD_ENABLE_ASM_X86_64_BMI2 0
-#endif
-
-/*
- * For x86 ELF targets, add .note.gnu.property section for Intel CET in
- * assembly sources when CET is enabled.
- *
- * Additionally, any function that may be called indirectly must begin
- * with ZSTD_CET_ENDBRANCH.
- */
-#if defined(__ELF__) && (defined(__x86_64__) || defined(__i386__)) \
-    && defined(__has_include)
-# if __has_include(<cet.h>)
-#  include <cet.h>
-#  define ZSTD_CET_ENDBRANCH _CET_ENDBR
-# endif
-#endif
-
-#ifndef ZSTD_CET_ENDBRANCH
-# define ZSTD_CET_ENDBRANCH
-#endif
-
-#endif /* ZSTD_PORTABILITY_MACROS_H */
-
-#endif /* USE_EXTERNAL_ZSTD */

vendor/github.com/DataDog/zstd/threading.c

@@ -1,185 +0,0 @@
-#ifndef USE_EXTERNAL_ZSTD
-/**
- * Copyright (c) 2016 Tino Reichardt
- * All rights reserved.
- *
- * You can contact the author at:
- * - zstdmt source repository: https://github.com/mcmilk/zstdmt
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- * You may select, at your option, one of the above-listed licenses.
- */
-
-/**
- * This file will hold wrapper for systems, which do not support pthreads
- */
-
-#include "threading.h"
-
-/* create fake symbol to avoid empty translation unit warning */
-int g_ZSTD_threading_useless_symbol;
-
-#if defined(ZSTD_MULTITHREAD) && defined(_WIN32)
-
-/**
- * Windows minimalist Pthread Wrapper
- */
-
-
-/* ===  Dependencies  === */
-#include <process.h>
-#include <errno.h>
-
-
-/* ===  Implementation  === */
-
-typedef struct {
-    void* (*start_routine)(void*);
-    void* arg;
-    int initialized;
-    ZSTD_pthread_cond_t initialized_cond;
-    ZSTD_pthread_mutex_t initialized_mutex;
-} ZSTD_thread_params_t;
-
-static unsigned __stdcall worker(void *arg)
-{
-    void* (*start_routine)(void*);
-    void* thread_arg;
-
-    /* Initialized thread_arg and start_routine and signal main thread that we don't need it
-     * to wait any longer.
-     */
-    {
-        ZSTD_thread_params_t*  thread_param = (ZSTD_thread_params_t*)arg;
-        thread_arg = thread_param->arg;
-        start_routine = thread_param->start_routine;
-
-        /* Signal main thread that we are running and do not depend on its memory anymore */
-        ZSTD_pthread_mutex_lock(&thread_param->initialized_mutex);
-        thread_param->initialized = 1;
-        ZSTD_pthread_cond_signal(&thread_param->initialized_cond);
-        ZSTD_pthread_mutex_unlock(&thread_param->initialized_mutex);
-    }
-
-    start_routine(thread_arg);
-
-    return 0;
-}
-
-int ZSTD_pthread_create(ZSTD_pthread_t* thread, const void* unused,
-            void* (*start_routine) (void*), void* arg)
-{
-    ZSTD_thread_params_t thread_param;
-    (void)unused;
-
-    if (thread==NULL) return -1;
-    *thread = NULL;
-
-    thread_param.start_routine = start_routine;
-    thread_param.arg = arg;
-    thread_param.initialized = 0;
-
-    /* Setup thread initialization synchronization */
-    if(ZSTD_pthread_cond_init(&thread_param.initialized_cond, NULL)) {
-        /* Should never happen on Windows */
-        return -1;
-    }
-    if(ZSTD_pthread_mutex_init(&thread_param.initialized_mutex, NULL)) {
-        /* Should never happen on Windows */
-        ZSTD_pthread_cond_destroy(&thread_param.initialized_cond);
-        return -1;
-    }
-
-    /* Spawn thread */
-    *thread = (HANDLE)_beginthreadex(NULL, 0, worker, &thread_param, 0, NULL);
-    if (*thread==NULL) {
-        ZSTD_pthread_mutex_destroy(&thread_param.initialized_mutex);
-        ZSTD_pthread_cond_destroy(&thread_param.initialized_cond);
-        return errno;
-    }
-
-    /* Wait for thread to be initialized */
-    ZSTD_pthread_mutex_lock(&thread_param.initialized_mutex);
-    while(!thread_param.initialized) {
-        ZSTD_pthread_cond_wait(&thread_param.initialized_cond, &thread_param.initialized_mutex);
-    }
-    ZSTD_pthread_mutex_unlock(&thread_param.initialized_mutex);
-    ZSTD_pthread_mutex_destroy(&thread_param.initialized_mutex);
-    ZSTD_pthread_cond_destroy(&thread_param.initialized_cond);
-
-    return 0;
-}
-
-int ZSTD_pthread_join(ZSTD_pthread_t thread)
-{
-    DWORD result;
-
-    if (!thread) return 0;
-
-    result = WaitForSingleObject(thread, INFINITE);
-    CloseHandle(thread);
-
-    switch (result) {
-    case WAIT_OBJECT_0:
-        return 0;
-    case WAIT_ABANDONED:
-        return EINVAL;
-    default:
-        return GetLastError();
-    }
-}
-
-#endif   /* ZSTD_MULTITHREAD */
-
-#if defined(ZSTD_MULTITHREAD) && DEBUGLEVEL >= 1 && !defined(_WIN32)
-
-#define ZSTD_DEPS_NEED_MALLOC
-#include "zstd_deps.h"
-
-int ZSTD_pthread_mutex_init(ZSTD_pthread_mutex_t* mutex, pthread_mutexattr_t const* attr)
-{
-    assert(mutex != NULL);
-    *mutex = (pthread_mutex_t*)ZSTD_malloc(sizeof(pthread_mutex_t));
-    if (!*mutex)
-        return 1;
-    return pthread_mutex_init(*mutex, attr);
-}
-
-int ZSTD_pthread_mutex_destroy(ZSTD_pthread_mutex_t* mutex)
-{
-    assert(mutex != NULL);
-    if (!*mutex)
-        return 0;
-    {
-        int const ret = pthread_mutex_destroy(*mutex);
-        ZSTD_free(*mutex);
-        return ret;
-    }
-}
-
-int ZSTD_pthread_cond_init(ZSTD_pthread_cond_t* cond, pthread_condattr_t const* attr)
-{
-    assert(cond != NULL);
-    *cond = (pthread_cond_t*)ZSTD_malloc(sizeof(pthread_cond_t));
-    if (!*cond)
-        return 1;
-    return pthread_cond_init(*cond, attr);
-}
-
-int ZSTD_pthread_cond_destroy(ZSTD_pthread_cond_t* cond)
-{
-    assert(cond != NULL);
-    if (!*cond)
-        return 0;
-    {
-        int const ret = pthread_cond_destroy(*cond);
-        ZSTD_free(*cond);
-        return ret;
-    }
-}
-
-#endif
-
-#endif /* USE_EXTERNAL_ZSTD */

vendor/github.com/DataDog/zstd/threading.h

@@ -1,145 +0,0 @@
-#ifndef USE_EXTERNAL_ZSTD
-/**
- * Copyright (c) 2016 Tino Reichardt
- * All rights reserved.
- *
- * You can contact the author at:
- * - zstdmt source repository: https://github.com/mcmilk/zstdmt
- *
- * This source code is licensed under both the BSD-style license (found in the
- * LICENSE file in the root directory of this source tree) and the GPLv2 (found
- * in the COPYING file in the root directory of this source tree).
- * You may select, at your option, one of the above-listed licenses.
- */
-
-#ifndef THREADING_H_938743
-#define THREADING_H_938743
-
-#include "debug.h"
-
-#if defined(ZSTD_MULTITHREAD) && defined(_WIN32)
-
-/**
- * Windows minimalist Pthread Wrapper
- */
-#ifdef WINVER
-#  undef WINVER
-#endif
-#define WINVER       0x0600
-
-#ifdef _WIN32_WINNT
-#  undef _WIN32_WINNT
-#endif
-#define _WIN32_WINNT 0x0600
-
-#ifndef WIN32_LEAN_AND_MEAN
-#  define WIN32_LEAN_AND_MEAN
-#endif
-
-#undef ERROR   /* reported already defined on VS 2015 (Rich Geldreich) */
-#include <windows.h>
-#undef ERROR
-#define ERROR(name) ZSTD_ERROR(name)
-
-
-/* mutex */
-#define ZSTD_pthread_mutex_t           CRITICAL_SECTION
-#define ZSTD_pthread_mutex_init(a, b)  ((void)(b), InitializeCriticalSection((a)), 0)
-#define ZSTD_pthread_mutex_destroy(a)  DeleteCriticalSection((a))
-#define ZSTD_pthread_mutex_lock(a)     EnterCriticalSection((a))
-#define ZSTD_pthread_mutex_unlock(a)   LeaveCriticalSection((a))
-
-/* condition variable */
-#define ZSTD_pthread_cond_t             CONDITION_VARIABLE
-#define ZSTD_pthread_cond_init(a, b)    ((void)(b), InitializeConditionVariable((a)), 0)
-#define ZSTD_pthread_cond_destroy(a)    ((void)(a))
-#define ZSTD_pthread_cond_wait(a, b)    SleepConditionVariableCS((a), (b), INFINITE)
-#define ZSTD_pthread_cond_signal(a)     WakeConditionVariable((a))
-#define ZSTD_pthread_cond_broadcast(a)  WakeAllConditionVariable((a))
-
-/* ZSTD_pthread_create() and ZSTD_pthread_join() */
-typedef HANDLE ZSTD_pthread_t;
-
-int ZSTD_pthread_create(ZSTD_pthread_t* thread, const void* unused,
-                   void* (*start_routine) (void*), void* arg);
-
-int ZSTD_pthread_join(ZSTD_pthread_t thread);
-
-/**
- * add here more wrappers as required
- */
-
-#elif defined(ZSTD_MULTITHREAD)    /* posix assumed ; need a better detection method */
-/* ===   POSIX Systems   === */
-#  include <pthread.h>
-
-#if DEBUGLEVEL < 1
-
-#define ZSTD_pthread_mutex_t            pthread_mutex_t
-#define ZSTD_pthread_mutex_init(a, b)   pthread_mutex_init((a), (b))
-#define ZSTD_pthread_mutex_destroy(a)   pthread_mutex_destroy((a))
-#define ZSTD_pthread_mutex_lock(a)      pthread_mutex_lock((a))
-#define ZSTD_pthread_mutex_unlock(a)    pthread_mutex_unlock((a))
-
-#define ZSTD_pthread_cond_t             pthread_cond_t
-#define ZSTD_pthread_cond_init(a, b)    pthread_cond_init((a), (b))
-#define ZSTD_pthread_cond_destroy(a)    pthread_cond_destroy((a))
-#define ZSTD_pthread_cond_wait(a, b)    pthread_cond_wait((a), (b))
-#define ZSTD_pthread_cond_signal(a)     pthread_cond_signal((a))
-#define ZSTD_pthread_cond_broadcast(a)  pthread_cond_broadcast((a))
-
-#define ZSTD_pthread_t                  pthread_t
-#define ZSTD_pthread_create(a, b, c, d) pthread_create((a), (b), (c), (d))
-#define ZSTD_pthread_join(a)         pthread_join((a),NULL)
-
-#else /* DEBUGLEVEL >= 1 */
-
-/* Debug implementation of threading.
- * In this implementation we use pointers for mutexes and condition variables.
- * This way, if we forget to init/destroy them the program will crash or ASAN
- * will report leaks.
- */
-
-#define ZSTD_pthread_mutex_t            pthread_mutex_t*
-int ZSTD_pthread_mutex_init(ZSTD_pthread_mutex_t* mutex, pthread_mutexattr_t const* attr);
-int ZSTD_pthread_mutex_destroy(ZSTD_pthread_mutex_t* mutex);
-#define ZSTD_pthread_mutex_lock(a)      pthread_mutex_lock(*(a))
-#define ZSTD_pthread_mutex_unlock(a)    pthread_mutex_unlock(*(a))
-
-#define ZSTD_pthread_cond_t             pthread_cond_t*
-int ZSTD_pthread_cond_init(ZSTD_pthread_cond_t* cond, pthread_condattr_t const* attr);
-int ZSTD_pthread_cond_destroy(ZSTD_pthread_cond_t* cond);
-#define ZSTD_pthread_cond_wait(a, b)    pthread_cond_wait(*(a), *(b))
-#define ZSTD_pthread_cond_signal(a)     pthread_cond_signal(*(a))
-#define ZSTD_pthread_cond_broadcast(a)  pthread_cond_broadcast(*(a))
-
-#define ZSTD_pthread_t                  pthread_t
-#define ZSTD_pthread_create(a, b, c, d) pthread_create((a), (b), (c), (d))
-#define ZSTD_pthread_join(a)         pthread_join((a),NULL)
-
-#endif
-
-#else  /* ZSTD_MULTITHREAD not defined */
-/* No multithreading support */
-
-typedef int ZSTD_pthread_mutex_t;
-#define ZSTD_pthread_mutex_init(a, b)   ((void)(a), (void)(b), 0)
-#define ZSTD_pthread_mutex_destroy(a)   ((void)(a))
-#define ZSTD_pthread_mutex_lock(a)      ((void)(a))
-#define ZSTD_pthread_mutex_unlock(a)    ((void)(a))
-
-typedef int ZSTD_pthread_cond_t;
-#define ZSTD_pthread_cond_init(a, b)    ((void)(a), (void)(b), 0)
-#define ZSTD_pthread_cond_destroy(a)    ((void)(a))
-#define ZSTD_pthread_cond_wait(a, b)    ((void)(a), (void)(b))
-#define ZSTD_pthread_cond_signal(a)     ((void)(a))
-#define ZSTD_pthread_cond_broadcast(a)  ((void)(a))
-
-/* do not use ZSTD_pthread_t */
-
-#endif /* ZSTD_MULTITHREAD */
-
-
-#endif /* THREADING_H_938743 */
-
-#endif /* USE_EXTERNAL_ZSTD */

vendor/github.com/DataDog/zstd/travis_test_32.sh

@@ -1,19 +0,0 @@
-#!/bin/bash
-# Get utilities
-#yum -y -q -e 0 install wget tar unzip gcc
-apt-get update
-apt-get -y install wget tar unzip gcc
-
-# Get Go
-wget -q https://dl.google.com/go/go1.13.linux-386.tar.gz
-tar -C /usr/local -xzf go1.13.linux-386.tar.gz
-export PATH=$PATH:/usr/local/go/bin
-
-# Get payload
-wget -q https://github.com/DataDog/zstd/files/2246767/mr.zip
-unzip mr.zip
-
-# Build and run tests
-go build
-DISABLE_BIG_TESTS=1 PAYLOAD=$(pwd)/mr go test -v
-DISABLE_BIG_TESTS=1 PAYLOAD=$(pwd)/mr go test -bench .

vendor/github.com/DataDog/zstd/update.txt

@@ -1,56 +0,0 @@
-./lib/common/bitstream.h
-./lib/common/compiler.h
-./lib/compress/zstd_compress_internal.h
-./lib/compress/zstd_fast.h
-./lib/compress/zstd_double_fast.h
-./lib/compress/zstd_lazy.h
-./lib/compress/zstd_ldm.h
-./lib/dictBuilder/cover.c
-./lib/dictBuilder/divsufsort.c
-./lib/dictBuilder/divsufsort.h
-./lib/common/entropy_common.c
-./lib/common/error_private.c
-./lib/common/error_private.h
-./lib/compress/fse_compress.c
-./lib/common/fse_decompress.c
-./lib/common/fse.h
-./lib/compress/huf_compress.c
-./lib/decompress/huf_decompress.c
-./lib/common/huf.h
-./lib/common/mem.h
-./lib/common/pool.c
-./lib/common/pool.h
-./lib/common/threading.c
-./lib/common/threading.h
-./lib/common/xxhash.c
-./lib/common/xxhash.h
-./lib/deprecated/zbuff_common.c
-./lib/deprecated/zbuff_compress.c
-./lib/deprecated/zbuff_decompress.c
-./lib/deprecated/zbuff.h
-./lib/dictBuilder/zdict.c
-./lib/dictBuilder/zdict.h
-./lib/common/zstd_common.c
-./lib/compress/zstd_compress.c
-./lib/decompress/zstd_decompress.c
-./lib/common/zstd_errors.h
-./lib/zstd.h
-./lib/common/zstd_internal.h
-./lib/legacy/zstd_legacy.h
-./lib/compress/zstd_opt.c
-./lib/compress/zstd_opt.h
-./lib/legacy/zstd_v01.c
-./lib/legacy/zstd_v01.h
-./lib/legacy/zstd_v02.c
-./lib/legacy/zstd_v02.h
-./lib/legacy/zstd_v03.c
-./lib/legacy/zstd_v03.h
-./lib/legacy/zstd_v04.c
-./lib/legacy/zstd_v04.h
-./lib/legacy/zstd_v05.c
-./lib/legacy/zstd_v05.h
-./lib/legacy/zstd_v06.c
-./lib/legacy/zstd_v06.h
-./lib/legacy/zstd_v07.c
-./lib/legacy/zstd_v07.h
-