pekoe/TODO.md

# TODO

## 1. Multiport server

One pekoe instance serves on multiple ip:port pairs simultaneously. Each listener has its own `max_clients` limit (default 10000 per pair). All listeners share the same pubsub hub — a message published on any listener reaches subscribers on any listener.

**Motivation:** A single process may need to listen on multiple interfaces (e.g., internal `10.0.0.1:6379` and external `0.0.0.0:6380`) without running separate instances. Simplifies deployment and reduces resource overhead.

**Config change:**
Replace `bind` + `port` with a `listeners` array. Keep old fields for backward compatibility — if `listeners` is absent, fall back to `bind`+`port` as a single listener.

```json
{
  "listeners": [
    { "bind": "0.0.0.0", "port": 6379, "max_clients": 10000 },
    { "bind": "10.0.0.1", "port": 6380, "max_clients": 5000 }
  ]
}
```

| Option | Type | Default | Description |
|--------|------|---------|-------------|
| `bind` | string | `"0.0.0.0"` | IP address to bind |
| `port` | int | `6379` | Port number |
| `max_clients` | int | `10000` | Max connections for this listener |

**Design:**
- `internal/server/server.go` — Server holds a slice of `net.Listener`, one per config entry
- `Run()` starts an accept loop goroutine per listener, all sharing the same hub
- Shutdown closes all listeners and waits for all connections across all listeners
- `CLIENT LIST` and `CLIENT INFO` work across all listeners (same hub)
- Logging includes `listener=addr:port` field to distinguish sources

**Implementation outline:**
1. Add `Listener` struct to `internal/config/config.go`, add `Listeners []Listener` field to Config
2. Update `SetDefaults()` / `Validate()` — if `Listeners` empty, build from legacy `Bind`+`Port`; validate each listener; cap `max_clients` per listener at 10000
3. Refactor `internal/server/server.go` — `Run()` iterates listeners, starts one accept loop each; shutdown closes all listeners
4. Add `listener` field to log messages in accept/connected events
5. Create `internal/server/server_test.go` tests for multi-listener startup, shutdown, per-listener client limits
6. Update config file example, README.md, AGENTS.md

**Edge cases:**
- Duplicate `bind:port` pairs — reject at config validation
- One listener fails to bind — fail startup, don't partially start
- Global `max_clients` from legacy config maps to the single listener's limit
- `listeners` present alongside legacy `bind`/`port` — `listeners` takes precedence

---

## 2. Cloud-friendly configuration

Make `pekoe` cloud-friendly: allow config values to be set via command-line flags or environment variables, not only config file.

**Motivation:** In container/cloud environments (Docker, Kubernetes, ECS) it's easier to pass configuration via environment variables or flags than to mount config files.

**Design:**
- Each config option gets a CLI flag (`--port`, `--bind`, `--log-level`, etc.)
- Each config option gets an env var (`PEKOE_PORT`, `PEKOE_BIND`, `PEKOE_LOG_LEVEL`, etc.)
- Precedence: CLI flag > env var > config file > default
- Config file remains the primary source, flags/env override individual values
- Keep `--config` for specifying the config file path

**Config options to expose:**

| Config key | Flag | Env var |
|------------|------|---------|
| `bind` | `--bind` | `PEKOE_BIND` |
| `port` | `--port` | `PEKOE_PORT` |
| `max_clients` | `--max-clients` | `PEKOE_MAX_CLIENTS` |
| `read_timeout` | `--read-timeout` | `PEKOE_READ_TIMEOUT` |
| `write_timeout` | `--write-timeout` | `PEKOE_WRITE_TIMEOUT` |
| `max_message_bytes` | `--max-message-bytes` | `PEKOE_MAX_MESSAGE_BYTES` |
| `log_level` | `--log-level` | `PEKOE_LOG_LEVEL` |
| `log_file` | `--log-file` | `PEKOE_LOG_FILE` |
| `pid_file` | `--pid-file` | `PEKOE_PID_FILE` |

**Implementation outline:**
1. Add `FromEnv()` method to Config that reads PEKOE_* env vars
2. Add `FromFlags()` method or integrate flag parsing into config loading
3. Update `config.Load()` to apply overrides in order: defaults → file → env → flags
4. Update `cmd/pekoe/main.go` to register all flags and pass them to config
5. Add tests for precedence and edge cases (invalid env values, conflicting flags)
6. Update README.md and AGENTS.md

**Example usage:**
```bash
# Override port via flag
./pekoe --config config.json --port 8080

# Override via env var
PEKOE_PORT=8080 PEKOE_LOG_LEVEL=debug ./pekoe

# Docker
docker run -e PEKOE_PORT=6379 -e PEKOE_LOG_LEVEL=info pekoe

# Kubernetes
env:
  - name: PEKOE_PORT
    value: "6379"
  - name: PEKOE_BIND
    value: "0.0.0.0"
```

**Note:** Depends on task #1 (multiport). If `listeners` config is used, env/flag overrides apply to the first listener only. For multi-listener setups, config file is required.

---

## 3. In-memory message buffer with TTL for pubsub

Buffer messages published to channels with no subscribers. When a subscriber later joins, it receives all buffered messages as if they were just published. Fully transparent to clients — no new commands, no changed behavior.

**Motivation:** In ephemeral pub/sub setups subscribers may connect after publishers have already sent messages. Without buffering those messages are lost. A per-channel buffer with TTL lets late subscribers receive recent messages without needing persistence or external storage.

**Use case flow:**
1. Publisher sends `PUBLISH news "hello"` — no subscribers on `news`
2. Message is stored in per-channel buffer with configured TTL
3. Later, client subscribes to `news` via `SUBSCRIBE news`
4. Buffered message is delivered immediately as a normal `["message", "news", "hello"]` response
5. Message is deleted from buffer after delivery
6. If TTL expires before any subscriber joins, message is silently discarded

**Design:**
- New package `internal/buffer/` — per-channel message buffer with TTL
- Each channel holds a slice of buffered messages, each with a creation timestamp and TTL
- On `PUBLISH` to a channel with zero subscribers, store the message instead of discarding
- On `SUBSCRIBE`/`PSUBSCRIBE`, flush matching buffered messages to the new subscriber before processing future live messages
- On delivery, remove the message from the buffer immediately
- Background goroutine periodically evicts expired messages
- Pattern subscriptions (`PSUBSCRIBE`) also receive buffered messages from channels matching the pattern

**What does NOT change:**
- No new commands or subcommands
- No changes to existing command syntax or responses
- Client sees normal `["message", channel, payload]` frames for buffered messages
- `PUBLISH` still returns the count of live subscribers (buffered messages do not count)

**Config options:**

| Option | Type | Default | Description |
|--------|------|---------|-------------|
| `message_ttl` | int | `0` | Default TTL for buffered messages in seconds (0 = buffering disabled) |
| `buffer_cleanup_interval` | int | `1` | How often to scan for expired messages (seconds) |

**Implementation outline:**
1. Create `internal/buffer/buffer.go` — Message struct, Buffer struct (per-channel map of message slices, TTL, locking), Add/Deliver/Expire methods
2. Create `internal/buffer/buffer_test.go` — tests for add, deliver (FIFO order), TTL expiry, concurrent access, pattern matching
3. Integrate into `internal/pubsub/hub.go` — on Publish if channel has no subscribers, call buffer.Add; on Subscribe after registering, call buffer.Deliver
4. Integrate pattern matching — on PatternSubscribe, iterate channels matching the pattern and deliver buffered messages
5. Add `message_ttl` and `buffer_cleanup_interval` to `internal/config/config.go`
6. Start background expiry goroutine in server startup, stop on shutdown
7. Update README.md and AGENTS.md

**Edge cases:**
- Subscriber joins, receives buffer, then unsubscribes, then re-joins — buffer was already flushed, no re-delivery
- Multiple messages buffered — delivered in FIFO order
- Pattern subscriber joins — receives buffered messages from all matching channels
- TTL expires mid-delivery — expired messages skipped during flush
- `message_ttl=0` — buffering disabled, behavior unchanged from current

---

## 4. Redis-compatible authentication and encryption

Implement Redis-compatible AUTH command and TLS encryption support.

**Based on Redis analysis:**
- Redis uses `requirepass` config for simple password auth (ACL is more advanced)
- HELLO command is the modern way to authenticate during handshake (Redis 6+)
- TLS uses separate port (`tls-port`) or can coexist with plaintext on different ports
- Redis uses OpenSSL for TLS via `SSL_CTX` and `SSL` objects
- HELLO returns map with server info: `server`, `version`, `proto`, `id`, `mode`, `role`, `modules`
- AUTH is allowed both before and after SUBSCRIBE (Redis 6+)
- Redis uses `CMD_NO_AUTH` command flag to mark commands that don't require auth

### Authentication (AUTH)

**Redis `requirepass` behavior:**
- Single password for the "default" user
- AUTH with single argument uses "default" username
- AUTH with two arguments: `AUTH username password` (ACL style)
- If no password configured, AUTH returns error: "AUTH called without any password configured"
- Wrong password returns: "invalid username and password pair"

**Redis HELLO command behavior:**
- `HELLO [protover] [AUTH username password] [SETNAME name]`
- If protocol version not specified, defaults to RESP2 (2)
- Protocol version must be 2 or 3 (RESP2 or RESP3)
- AUTH in HELLO authenticates during handshake
- Returns map with: `server`, `version`, `proto`, `id`, `mode`, `role`, `modules`
- If not authenticated after HELLO, returns: "HELLO must be called with the client already authenticated"

**Config options:**

| Option | Type | Default | Redis equivalent | Description |
|--------|------|---------|-------------------|-------------|
| `require_pass` | string | `""` | `requirepass` | Server password (empty = no auth) |
| `user` | string | `"default"` | (ACL default) | Default username |

**New commands:**

| Command | Description |
|---------|-------------|
| `AUTH [username] <password>` | Authenticate with password |
| `HELLO [protover] [AUTH username password] [SETNAME name]` | Extended handshake with optional auth |

### TLS Encryption

**Redis TLS behavior:**
- `tls-port` — separate TLS port, 0 = disabled
- `port` — plain TCP port, can be 0 to disable plaintext
- If both `tls-port` and `port` set, server listens on both
- Certificate files: `tls-cert-file`, `tls-key-file`, `tls-client-cert-file`, `tls-client-key-file`
- CA: `tls-ca-cert-file`, `tls-ca-cert-dir`
- Client auth: `tls-auth-clients` — no/yes/optional
- Protocol versions: `tls-protocols` — "TLSv1.2 TLSv1.3"
- Ciphers: `tls-ciphers`, `tls-ciphersuites`
- Session caching: `tls-session-caching`, `tls-session-cache-size`, `tls-session-cache-timeout`
- Replication TLS: `tls-replication`
- Cluster TLS: `tls-cluster`

**Config options:**

| Option | Type | Default | Redis equivalent | Description |
|--------|------|---------|-------------------|-------------|
| `tls_enabled` | bool | `false` | (derived from tls-port) | Enable TLS |
| `tls_port` | int | `0` | `tls-port` | TLS port (0 = same as port) |
| `port` | int | `6379` | `port` | Plain TCP port (0 = TLS only) |
| `tls_cert_file` | string | `""` | `tls-cert-file` | Path to TLS certificate |
| `tls_key_file` | string | `""` | `tls-key-file` | Path to TLS private key |
| `tls_key_pass` | string | `""` | `tls-key-file-pass` | Private key password |
| `tls_ca_cert_file` | string | `""` | `tls-ca-cert-file` | Path to CA certificate |
| `tls_ca_cert_dir` | string | `""` | `tls-ca-cert-dir` | Path to CA cert directory |
| `tls_auth_clients` | string | `"yes"` | `tls-auth-clients` | Client cert auth: no/yes/optional |
| `tls_client_cert_file` | string | `""` | `tls-client-cert-file` | Client certificate |
| `tls_client_key_file` | string | `""` | `tls-client-key-file` | Client private key |
| `tls_protocols` | string | `"TLSv1.2 TLSv1.3"` | `tls-protocols` | TLS versions |
| `tls_ciphers` | string | `""` | `tls-ciphers` | Server ciphers |
| `tls_ciphersuites` | string | `""` | `tls-ciphersuites` | TLS 1.3 ciphers |
| `tls_session_caching` | bool | `true` | `tls-session-caching` | Enable session cache |
| `tls_replication` | bool | `false` | `tls-replication` | TLS for replica connections |
| `tls_cluster` | bool | `false` | `tls-cluster` | TLS for cluster bus |

**Implementation outline:**
1. Add config fields to `internal/config/config.go`
2. Add validation — if TLS enabled, require cert+key files
3. Add `internal/auth/auth.go` — password verification with constant-time compare
4. Add `AUTH` command handler
5. Add `HELLO` command handler with RESP2/RESP3 support
6. Add TLS listener support in server — use Go's `crypto/tls` package
7. Detect TLS connections via `tls.ConnectionState`
8. Update connection state to track authenticated status
9. Add `CMD_NO_AUTH` equivalent — allow AUTH/HELLO without auth
10. Update CLIENT INFO to show TLS/auth status
11. Update README.md and AGENTS.md

**Edge cases:**
- AUTH on server without password configured → error
- HELLO without AUTH when password configured → auth required
- Plaintext connection to TLS-only server → reject
- TLS client cert verification → verify against CA if configured
- Both plaintext and TLS ports configured → support both
- AUTH in HELLO followed by regular AUTH → both should work

**Security notes:**
- Use `subtle.ConstantTimeCompare()` for password comparison
- Log failed AUTH attempts with client address
- HELLO response should include `authenticated: true/false`
- Redis 6+ allows AUTH before SUBSCRIBE and in HELLO

---

## 5. Structured JSON logging

In addition to plaintext logging, implement structured JSON logging with config option to switch between them.

**Motivation:** JSON logging enables easy parsing by log aggregation systems (ELK, Loki, CloudWatch, etc.). Plaintext remains for human readability in development.

**Design:**

### Current format (plaintext)
```
2024 Apr  2 14:30:15 INFO pekoe[12345]: event=startup addr=0.0.0.0:6379
```

### JSON format
```json
{"timestamp":"2024-04-02T14:30:15Z","level":"INFO","pid":12345,"event":"startup","addr":"0.0.0.0:6379"}
```

**Config options:**

| Option | Type | Default | Description |
|--------|------|---------|-------------|
| `log_format` | string | `"text"` | Log format: `text` or `json` |
| `log_timestamp_format` | string | `"rfc3339"` | Timestamp format for JSON (rfc3339, iso8601, unix) |

**Log levels:**
- DEBUG, INFO, WARN, ERROR, CRITICAL

**JSON fields:**

| Field | Type | Description |
|-------|------|-------------|
| `timestamp` | string | RFC3339 formatted timestamp |
| `level` | string | Log level (DEBUG, INFO, WARN, ERROR, CRITICAL) |
| `pid` | int | Process ID |
| `message` | string | Main log message |
| `*` | any | Key-value pairs from log message |

**Implementation outline:**
1. Refactor `internal/logger/logger.go` — separate formatting from logging
2. Add `formatText()` — current plaintext formatter
3. Add `formatJSON()` — new JSON formatter
4. Add config field `LogFormat` in `internal/config/config.go`
5. Update logger initialization to read format from config
6. Add tests for both formatters
7. Handle special characters in JSON output (escaping)
8. Ensure log file output uses same format as stdout

**Example output:**

**text:**
```
2024 Apr  2 14:30:15 INFO pekoe[12345]: event=startup addr=0.0.0.0:6379 max_clients=10000
```

**json:**
```json
{"timestamp":"2024-04-02T14:30:15Z","level":"INFO","pid":12345,"event":"startup","addr":"0.0.0.0:6379","max_clients":10000}
```

**Edge cases:**
- Empty message → emit JSON with message: ""
- Nested key=value → flatten with underscores
- Binary data in message → base64 encode or skip
- High-frequency logging → consider JSON marshaling optimization

**Use cases:**
- Text: development, debugging, simple deployments
- JSON: Kubernetes, Docker, log aggregation systems, SIEM