Initial commit

2026-04-02 23:56:28 +03:00
commit 9e30243533
34 changed files with 8413 additions and 0 deletions
--- a/TODO.md
+++ b/TODO.md
@@ -0,0 +1,340 @@
+# TODO
+
+## 1. Multiport server
+
+One pekoe instance serves on multiple ip:port pairs simultaneously. Each listener has its own `max_clients` limit (default 10000 per pair). All listeners share the same pubsub hub — a message published on any listener reaches subscribers on any listener.
+
+**Motivation:** A single process may need to listen on multiple interfaces (e.g., internal `10.0.0.1:6379` and external `0.0.0.0:6380`) without running separate instances. Simplifies deployment and reduces resource overhead.
+
+**Config change:**
+Replace `bind` + `port` with a `listeners` array. Keep old fields for backward compatibility — if `listeners` is absent, fall back to `bind`+`port` as a single listener.
+
+```json
+{
+  "listeners": [
+    { "bind": "0.0.0.0", "port": 6379, "max_clients": 10000 },
+    { "bind": "10.0.0.1", "port": 6380, "max_clients": 5000 }
+  ]
+}
+```
+
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `bind` | string | `"0.0.0.0"` | IP address to bind |
+| `port` | int | `6379` | Port number |
+| `max_clients` | int | `10000` | Max connections for this listener |
+
+**Design:**
+- `internal/server/server.go` — Server holds a slice of `net.Listener`, one per config entry
+- `Run()` starts an accept loop goroutine per listener, all sharing the same hub
+- Shutdown closes all listeners and waits for all connections across all listeners
+- `CLIENT LIST` and `CLIENT INFO` work across all listeners (same hub)
+- Logging includes `listener=addr:port` field to distinguish sources
+
+**Implementation outline:**
+1. Add `Listener` struct to `internal/config/config.go`, add `Listeners []Listener` field to Config
+2. Update `SetDefaults()` / `Validate()` — if `Listeners` empty, build from legacy `Bind`+`Port`; validate each listener; cap `max_clients` per listener at 10000
+3. Refactor `internal/server/server.go` — `Run()` iterates listeners, starts one accept loop each; shutdown closes all listeners
+4. Add `listener` field to log messages in accept/connected events
+5. Create `internal/server/server_test.go` tests for multi-listener startup, shutdown, per-listener client limits
+6. Update config file example, README.md, AGENTS.md
+
+**Edge cases:**
+- Duplicate `bind:port` pairs — reject at config validation
+- One listener fails to bind — fail startup, don't partially start
+- Global `max_clients` from legacy config maps to the single listener's limit
+- `listeners` present alongside legacy `bind`/`port` — `listeners` takes precedence
+
+---
+
+## 2. Cloud-friendly configuration
+
+Make `pekoe` cloud-friendly: allow config values to be set via command-line flags or environment variables, not only config file.
+
+**Motivation:** In container/cloud environments (Docker, Kubernetes, ECS) it's easier to pass configuration via environment variables or flags than to mount config files.
+
+**Design:**
+- Each config option gets a CLI flag (`--port`, `--bind`, `--log-level`, etc.)
+- Each config option gets an env var (`PEKOE_PORT`, `PEKOE_BIND`, `PEKOE_LOG_LEVEL`, etc.)
+- Precedence: CLI flag > env var > config file > default
+- Config file remains the primary source, flags/env override individual values
+- Keep `--config` for specifying the config file path
+
+**Config options to expose:**
+
+| Config key | Flag | Env var |
+|------------|------|---------|
+| `bind` | `--bind` | `PEKOE_BIND` |
+| `port` | `--port` | `PEKOE_PORT` |
+| `max_clients` | `--max-clients` | `PEKOE_MAX_CLIENTS` |
+| `read_timeout` | `--read-timeout` | `PEKOE_READ_TIMEOUT` |
+| `write_timeout` | `--write-timeout` | `PEKOE_WRITE_TIMEOUT` |
+| `max_message_bytes` | `--max-message-bytes` | `PEKOE_MAX_MESSAGE_BYTES` |
+| `log_level` | `--log-level` | `PEKOE_LOG_LEVEL` |
+| `log_file` | `--log-file` | `PEKOE_LOG_FILE` |
+| `pid_file` | `--pid-file` | `PEKOE_PID_FILE` |
+
+**Implementation outline:**
+1. Add `FromEnv()` method to Config that reads PEKOE_* env vars
+2. Add `FromFlags()` method or integrate flag parsing into config loading
+3. Update `config.Load()` to apply overrides in order: defaults → file → env → flags
+4. Update `cmd/pekoe/main.go` to register all flags and pass them to config
+5. Add tests for precedence and edge cases (invalid env values, conflicting flags)
+6. Update README.md and AGENTS.md
+
+**Example usage:**
+```bash
+# Override port via flag
+./pekoe --config config.json --port 8080
+
+# Override via env var
+PEKOE_PORT=8080 PEKOE_LOG_LEVEL=debug ./pekoe
+
+# Docker
+docker run -e PEKOE_PORT=6379 -e PEKOE_LOG_LEVEL=info pekoe
+
+# Kubernetes
+env:
+  - name: PEKOE_PORT
+    value: "6379"
+  - name: PEKOE_BIND
+    value: "0.0.0.0"
+```
+
+**Note:** Depends on task #1 (multiport). If `listeners` config is used, env/flag overrides apply to the first listener only. For multi-listener setups, config file is required.
+
+---
+
+## 3. In-memory message buffer with TTL for pubsub
+
+Buffer messages published to channels with no subscribers. When a subscriber later joins, it receives all buffered messages as if they were just published. Fully transparent to clients — no new commands, no changed behavior.
+
+**Motivation:** In ephemeral pub/sub setups subscribers may connect after publishers have already sent messages. Without buffering those messages are lost. A per-channel buffer with TTL lets late subscribers receive recent messages without needing persistence or external storage.
+
+**Use case flow:**
+1. Publisher sends `PUBLISH news "hello"` — no subscribers on `news`
+2. Message is stored in per-channel buffer with configured TTL
+3. Later, client subscribes to `news` via `SUBSCRIBE news`
+4. Buffered message is delivered immediately as a normal `["message", "news", "hello"]` response
+5. Message is deleted from buffer after delivery
+6. If TTL expires before any subscriber joins, message is silently discarded
+
+**Design:**
+- New package `internal/buffer/` — per-channel message buffer with TTL
+- Each channel holds a slice of buffered messages, each with a creation timestamp and TTL
+- On `PUBLISH` to a channel with zero subscribers, store the message instead of discarding
+- On `SUBSCRIBE`/`PSUBSCRIBE`, flush matching buffered messages to the new subscriber before processing future live messages
+- On delivery, remove the message from the buffer immediately
+- Background goroutine periodically evicts expired messages
+- Pattern subscriptions (`PSUBSCRIBE`) also receive buffered messages from channels matching the pattern
+
+**What does NOT change:**
+- No new commands or subcommands
+- No changes to existing command syntax or responses
+- Client sees normal `["message", channel, payload]` frames for buffered messages
+- `PUBLISH` still returns the count of live subscribers (buffered messages do not count)
+
+**Config options:**
+
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `message_ttl` | int | `0` | Default TTL for buffered messages in seconds (0 = buffering disabled) |
+| `buffer_cleanup_interval` | int | `1` | How often to scan for expired messages (seconds) |
+
+**Implementation outline:**
+1. Create `internal/buffer/buffer.go` — Message struct, Buffer struct (per-channel map of message slices, TTL, locking), Add/Deliver/Expire methods
+2. Create `internal/buffer/buffer_test.go` — tests for add, deliver (FIFO order), TTL expiry, concurrent access, pattern matching
+3. Integrate into `internal/pubsub/hub.go` — on Publish if channel has no subscribers, call buffer.Add; on Subscribe after registering, call buffer.Deliver
+4. Integrate pattern matching — on PatternSubscribe, iterate channels matching the pattern and deliver buffered messages
+5. Add `message_ttl` and `buffer_cleanup_interval` to `internal/config/config.go`
+6. Start background expiry goroutine in server startup, stop on shutdown
+7. Update README.md and AGENTS.md
+
+**Edge cases:**
+- Subscriber joins, receives buffer, then unsubscribes, then re-joins — buffer was already flushed, no re-delivery
+- Multiple messages buffered — delivered in FIFO order
+- Pattern subscriber joins — receives buffered messages from all matching channels
+- TTL expires mid-delivery — expired messages skipped during flush
+- `message_ttl=0` — buffering disabled, behavior unchanged from current
+
+---
+
+## 4. Redis-compatible authentication and encryption
+
+Implement Redis-compatible AUTH command and TLS encryption support.
+
+**Based on Redis analysis:**
+- Redis uses `requirepass` config for simple password auth (ACL is more advanced)
+- HELLO command is the modern way to authenticate during handshake (Redis 6+)
+- TLS uses separate port (`tls-port`) or can coexist with plaintext on different ports
+- Redis uses OpenSSL for TLS via `SSL_CTX` and `SSL` objects
+- HELLO returns map with server info: `server`, `version`, `proto`, `id`, `mode`, `role`, `modules`
+- AUTH is allowed both before and after SUBSCRIBE (Redis 6+)
+- Redis uses `CMD_NO_AUTH` command flag to mark commands that don't require auth
+
+### Authentication (AUTH)
+
+**Redis `requirepass` behavior:**
+- Single password for the "default" user
+- AUTH with single argument uses "default" username
+- AUTH with two arguments: `AUTH username password` (ACL style)
+- If no password configured, AUTH returns error: "AUTH called without any password configured"
+- Wrong password returns: "invalid username and password pair"
+
+**Redis HELLO command behavior:**
+- `HELLO [protover] [AUTH username password] [SETNAME name]`
+- If protocol version not specified, defaults to RESP2 (2)
+- Protocol version must be 2 or 3 (RESP2 or RESP3)
+- AUTH in HELLO authenticates during handshake
+- Returns map with: `server`, `version`, `proto`, `id`, `mode`, `role`, `modules`
+- If not authenticated after HELLO, returns: "HELLO must be called with the client already authenticated"
+
+**Config options:**
+
+| Option | Type | Default | Redis equivalent | Description |
+|--------|------|---------|-------------------|-------------|
+| `require_pass` | string | `""` | `requirepass` | Server password (empty = no auth) |
+| `user` | string | `"default"` | (ACL default) | Default username |
+
+**New commands:**
+
+| Command | Description |
+|---------|-------------|
+| `AUTH [username] <password>` | Authenticate with password |
+| `HELLO [protover] [AUTH username password] [SETNAME name]` | Extended handshake with optional auth |
+
+### TLS Encryption
+
+**Redis TLS behavior:**
+- `tls-port` — separate TLS port, 0 = disabled
+- `port` — plain TCP port, can be 0 to disable plaintext
+- If both `tls-port` and `port` set, server listens on both
+- Certificate files: `tls-cert-file`, `tls-key-file`, `tls-client-cert-file`, `tls-client-key-file`
+- CA: `tls-ca-cert-file`, `tls-ca-cert-dir`
+- Client auth: `tls-auth-clients` — no/yes/optional
+- Protocol versions: `tls-protocols` — "TLSv1.2 TLSv1.3"
+- Ciphers: `tls-ciphers`, `tls-ciphersuites`
+- Session caching: `tls-session-caching`, `tls-session-cache-size`, `tls-session-cache-timeout`
+- Replication TLS: `tls-replication`
+- Cluster TLS: `tls-cluster`
+
+**Config options:**
+
+| Option | Type | Default | Redis equivalent | Description |
+|--------|------|---------|-------------------|-------------|
+| `tls_enabled` | bool | `false` | (derived from tls-port) | Enable TLS |
+| `tls_port` | int | `0` | `tls-port` | TLS port (0 = same as port) |
+| `port` | int | `6379` | `port` | Plain TCP port (0 = TLS only) |
+| `tls_cert_file` | string | `""` | `tls-cert-file` | Path to TLS certificate |
+| `tls_key_file` | string | `""` | `tls-key-file` | Path to TLS private key |
+| `tls_key_pass` | string | `""` | `tls-key-file-pass` | Private key password |
+| `tls_ca_cert_file` | string | `""` | `tls-ca-cert-file` | Path to CA certificate |
+| `tls_ca_cert_dir` | string | `""` | `tls-ca-cert-dir` | Path to CA cert directory |
+| `tls_auth_clients` | string | `"yes"` | `tls-auth-clients` | Client cert auth: no/yes/optional |
+| `tls_client_cert_file` | string | `""` | `tls-client-cert-file` | Client certificate |
+| `tls_client_key_file` | string | `""` | `tls-client-key-file` | Client private key |
+| `tls_protocols` | string | `"TLSv1.2 TLSv1.3"` | `tls-protocols` | TLS versions |
+| `tls_ciphers` | string | `""` | `tls-ciphers` | Server ciphers |
+| `tls_ciphersuites` | string | `""` | `tls-ciphersuites` | TLS 1.3 ciphers |
+| `tls_session_caching` | bool | `true` | `tls-session-caching` | Enable session cache |
+| `tls_replication` | bool | `false` | `tls-replication` | TLS for replica connections |
+| `tls_cluster` | bool | `false` | `tls-cluster` | TLS for cluster bus |
+
+**Implementation outline:**
+1. Add config fields to `internal/config/config.go`
+2. Add validation — if TLS enabled, require cert+key files
+3. Add `internal/auth/auth.go` — password verification with constant-time compare
+4. Add `AUTH` command handler
+5. Add `HELLO` command handler with RESP2/RESP3 support
+6. Add TLS listener support in server — use Go's `crypto/tls` package
+7. Detect TLS connections via `tls.ConnectionState`
+8. Update connection state to track authenticated status
+9. Add `CMD_NO_AUTH` equivalent — allow AUTH/HELLO without auth
+10. Update CLIENT INFO to show TLS/auth status
+11. Update README.md and AGENTS.md
+
+**Edge cases:**
+- AUTH on server without password configured → error
+- HELLO without AUTH when password configured → auth required
+- Plaintext connection to TLS-only server → reject
+- TLS client cert verification → verify against CA if configured
+- Both plaintext and TLS ports configured → support both
+- AUTH in HELLO followed by regular AUTH → both should work
+
+**Security notes:**
+- Use `subtle.ConstantTimeCompare()` for password comparison
+- Log failed AUTH attempts with client address
+- HELLO response should include `authenticated: true/false`
+- Redis 6+ allows AUTH before SUBSCRIBE and in HELLO
+
+---
+
+## 5. Structured JSON logging
+
+In addition to plaintext logging, implement structured JSON logging with config option to switch between them.
+
+**Motivation:** JSON logging enables easy parsing by log aggregation systems (ELK, Loki, CloudWatch, etc.). Plaintext remains for human readability in development.
+
+**Design:**
+
+### Current format (plaintext)
+```
+2024 Apr  2 14:30:15 INFO pekoe[12345]: event=startup addr=0.0.0.0:6379
+```
+
+### JSON format
+```json
+{"timestamp":"2024-04-02T14:30:15Z","level":"INFO","pid":12345,"event":"startup","addr":"0.0.0.0:6379"}
+```
+
+**Config options:**
+
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `log_format` | string | `"text"` | Log format: `text` or `json` |
+| `log_timestamp_format` | string | `"rfc3339"` | Timestamp format for JSON (rfc3339, iso8601, unix) |
+
+**Log levels:**
+- DEBUG, INFO, WARN, ERROR, CRITICAL
+
+**JSON fields:**
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `timestamp` | string | RFC3339 formatted timestamp |
+| `level` | string | Log level (DEBUG, INFO, WARN, ERROR, CRITICAL) |
+| `pid` | int | Process ID |
+| `message` | string | Main log message |
+| `*` | any | Key-value pairs from log message |
+
+**Implementation outline:**
+1. Refactor `internal/logger/logger.go` — separate formatting from logging
+2. Add `formatText()` — current plaintext formatter
+3. Add `formatJSON()` — new JSON formatter
+4. Add config field `LogFormat` in `internal/config/config.go`
+5. Update logger initialization to read format from config
+6. Add tests for both formatters
+7. Handle special characters in JSON output (escaping)
+8. Ensure log file output uses same format as stdout
+
+**Example output:**
+
+**text:**
+```
+2024 Apr  2 14:30:15 INFO pekoe[12345]: event=startup addr=0.0.0.0:6379 max_clients=10000
+```
+
+**json:**
+```json
+{"timestamp":"2024-04-02T14:30:15Z","level":"INFO","pid":12345,"event":"startup","addr":"0.0.0.0:6379","max_clients":10000}
+```
+
+**Edge cases:**
+- Empty message → emit JSON with message: ""
+- Nested key=value → flatten with underscores
+- Binary data in message → base64 encode or skip
+- High-frequency logging → consider JSON marshaling optimization
+
+**Use cases:**
+- Text: development, debugging, simple deployments
+- JSON: Kubernetes, Docker, log aggregation systems, SIEM