pekoe/TODO.md

TODO

1. Multiport server

One pekoe instance serves on multiple ip:port pairs simultaneously. Each listener has its own max_clients limit (default 10000 per pair). All listeners share the same pubsub hub — a message published on any listener reaches subscribers on any listener.

Motivation: A single process may need to listen on multiple interfaces (e.g., internal 10.0.0.1:6379 and external 0.0.0.0:6380) without running separate instances. Simplifies deployment and reduces resource overhead.

Config change: Replace bind + port with a listeners array. Keep old fields for backward compatibility — if listeners is absent, fall back to bind+port as a single listener.

{
  "listeners": [
    { "bind": "0.0.0.0", "port": 6379, "max_clients": 10000 },
    { "bind": "10.0.0.1", "port": 6380, "max_clients": 5000 }
  ]
}

Option	Type	Default	Description
bind	string	"0.0.0.0"	IP address to bind
port	int	6379	Port number
max_clients	int	10000	Max connections for this listener

Design:

• internal/server/server.go — Server holds a slice of net.Listener, one per config entry
• Run() starts an accept loop goroutine per listener, all sharing the same hub
• Shutdown closes all listeners and waits for all connections across all listeners
• CLIENT LIST and CLIENT INFO work across all listeners (same hub)
• Logging includes listener=addr:port field to distinguish sources

Implementation outline:

1. Add Listener struct to internal/config/config.go, add Listeners []Listener field to Config
2. Update SetDefaults() / Validate() — if Listeners empty, build from legacy Bind+Port; validate each listener; cap max_clients per listener at 10000
3. Refactor internal/server/server.go — Run() iterates listeners, starts one accept loop each; shutdown closes all listeners
4. Add listener field to log messages in accept/connected events
5. Create internal/server/server_test.go tests for multi-listener startup, shutdown, per-listener client limits
6. Update config file example, README.md, AGENTS.md

Edge cases:

• Duplicate bind:port pairs — reject at config validation
• One listener fails to bind — fail startup, don't partially start
• Global max_clients from legacy config maps to the single listener's limit
• listeners present alongside legacy bind/port — listeners takes precedence

---

2. Cloud-friendly configuration

Make pekoe cloud-friendly: allow config values to be set via command-line flags or environment variables, not only config file.

Motivation: In container/cloud environments (Docker, Kubernetes, ECS) it's easier to pass configuration via environment variables or flags than to mount config files.

Design:

• Each config option gets a CLI flag (--port, --bind, --log-level, etc.)
• Each config option gets an env var (PEKOE_PORT, PEKOE_BIND, PEKOE_LOG_LEVEL, etc.)
• Precedence: CLI flag > env var > config file > default
• Config file remains the primary source, flags/env override individual values
• Keep --config for specifying the config file path

Config options to expose:

Config key	Flag	Env var
bind	--bind	PEKOE_BIND
port	--port	PEKOE_PORT
max_clients	--max-clients	PEKOE_MAX_CLIENTS
read_timeout	--read-timeout	PEKOE_READ_TIMEOUT
write_timeout	--write-timeout	PEKOE_WRITE_TIMEOUT
max_message_bytes	--max-message-bytes	PEKOE_MAX_MESSAGE_BYTES
log_level	--log-level	PEKOE_LOG_LEVEL
log_file	--log-file	PEKOE_LOG_FILE
pid_file	--pid-file	PEKOE_PID_FILE

Implementation outline:

1. Add FromEnv() method to Config that reads PEKOE_* env vars
2. Add FromFlags() method or integrate flag parsing into config loading
3. Update config.Load() to apply overrides in order: defaults → file → env → flags
4. Update cmd/pekoe/main.go to register all flags and pass them to config
5. Add tests for precedence and edge cases (invalid env values, conflicting flags)
6. Update README.md and AGENTS.md

Example usage:

# Override port via flag
./pekoe --config config.json --port 8080

# Override via env var
PEKOE_PORT=8080 PEKOE_LOG_LEVEL=debug ./pekoe

# Docker
docker run -e PEKOE_PORT=6379 -e PEKOE_LOG_LEVEL=info pekoe

# Kubernetes
env:
  - name: PEKOE_PORT
    value: "6379"
  - name: PEKOE_BIND
    value: "0.0.0.0"

Note: Depends on task #1 (multiport). If listeners config is used, env/flag overrides apply to the first listener only. For multi-listener setups, config file is required.

---

3. In-memory message buffer with TTL for pubsub

Buffer messages published to channels with no subscribers. When a subscriber later joins, it receives all buffered messages as if they were just published. Fully transparent to clients — no new commands, no changed behavior.

Motivation: In ephemeral pub/sub setups subscribers may connect after publishers have already sent messages. Without buffering those messages are lost. A per-channel buffer with TTL lets late subscribers receive recent messages without needing persistence or external storage.

Use case flow:

1. Publisher sends PUBLISH news "hello" — no subscribers on news
2. Message is stored in per-channel buffer with configured TTL
3. Later, client subscribes to news via SUBSCRIBE news
4. Buffered message is delivered immediately as a normal ["message", "news", "hello"] response
5. Message is deleted from buffer after delivery
6. If TTL expires before any subscriber joins, message is silently discarded

Design:

• New package internal/buffer/ — per-channel message buffer with TTL
• Each channel holds a slice of buffered messages, each with a creation timestamp and TTL
• On PUBLISH to a channel with zero subscribers, store the message instead of discarding
• On SUBSCRIBE/PSUBSCRIBE, flush matching buffered messages to the new subscriber before processing future live messages
• On delivery, remove the message from the buffer immediately
• Background goroutine periodically evicts expired messages
• Pattern subscriptions (PSUBSCRIBE) also receive buffered messages from channels matching the pattern

What does NOT change:

• No new commands or subcommands
• No changes to existing command syntax or responses
• Client sees normal ["message", channel, payload] frames for buffered messages
• PUBLISH still returns the count of live subscribers (buffered messages do not count)

Config options:

Option	Type	Default	Description
message_ttl	int	0	Default TTL for buffered messages in seconds (0 = buffering disabled)
buffer_cleanup_interval	int	1	How often to scan for expired messages (seconds)

Implementation outline:

1. Create internal/buffer/buffer.go — Message struct, Buffer struct (per-channel map of message slices, TTL, locking), Add/Deliver/Expire methods
2. Create internal/buffer/buffer_test.go — tests for add, deliver (FIFO order), TTL expiry, concurrent access, pattern matching
3. Integrate into internal/pubsub/hub.go — on Publish if channel has no subscribers, call buffer.Add; on Subscribe after registering, call buffer.Deliver
4. Integrate pattern matching — on PatternSubscribe, iterate channels matching the pattern and deliver buffered messages
5. Add message_ttl and buffer_cleanup_interval to internal/config/config.go
6. Start background expiry goroutine in server startup, stop on shutdown
7. Update README.md and AGENTS.md

Edge cases:

• Subscriber joins, receives buffer, then unsubscribes, then re-joins — buffer was already flushed, no re-delivery
• Multiple messages buffered — delivered in FIFO order
• Pattern subscriber joins — receives buffered messages from all matching channels
• TTL expires mid-delivery — expired messages skipped during flush
• message_ttl=0 — buffering disabled, behavior unchanged from current

---

4. Redis-compatible authentication and encryption

Implement Redis-compatible AUTH command and TLS encryption support.

Based on Redis analysis:

• Redis uses requirepass config for simple password auth (ACL is more advanced)
• HELLO command is the modern way to authenticate during handshake (Redis 6+)
• TLS uses separate port (tls-port) or can coexist with plaintext on different ports
• Redis uses OpenSSL for TLS via SSL_CTX and SSL objects
• HELLO returns map with server info: server, version, proto, id, mode, role, modules
• AUTH is allowed both before and after SUBSCRIBE (Redis 6+)
• Redis uses CMD_NO_AUTH command flag to mark commands that don't require auth

Authentication (AUTH)

Redis requirepass behavior:

• Single password for the "default" user
• AUTH with single argument uses "default" username
• AUTH with two arguments: AUTH username password (ACL style)
• If no password configured, AUTH returns error: "AUTH called without any password configured"
• Wrong password returns: "invalid username and password pair"

Redis HELLO command behavior:

• HELLO [protover] [AUTH username password] [SETNAME name]
• If protocol version not specified, defaults to RESP2 (2)
• Protocol version must be 2 or 3 (RESP2 or RESP3)
• AUTH in HELLO authenticates during handshake
• Returns map with: server, version, proto, id, mode, role, modules
• If not authenticated after HELLO, returns: "HELLO must be called with the client already authenticated"

Config options:

Option	Type	Default	Redis equivalent	Description
require_pass	string	""	requirepass	Server password (empty = no auth)
user	string	"default"	(ACL default)	Default username

New commands:

Command	Description
AUTH [username] <password>	Authenticate with password
HELLO [protover] [AUTH username password] [SETNAME name]	Extended handshake with optional auth

TLS Encryption

Redis TLS behavior:

• tls-port — separate TLS port, 0 = disabled
• port — plain TCP port, can be 0 to disable plaintext
• If both tls-port and port set, server listens on both
• Certificate files: tls-cert-file, tls-key-file, tls-client-cert-file, tls-client-key-file
• CA: tls-ca-cert-file, tls-ca-cert-dir
• Client auth: tls-auth-clients — no/yes/optional
• Protocol versions: tls-protocols — "TLSv1.2 TLSv1.3"
• Ciphers: tls-ciphers, tls-ciphersuites
• Session caching: tls-session-caching, tls-session-cache-size, tls-session-cache-timeout
• Replication TLS: tls-replication
• Cluster TLS: tls-cluster

Config options:

Option	Type	Default	Redis equivalent	Description
tls_enabled	bool	false	(derived from tls-port)	Enable TLS
tls_port	int	0	tls-port	TLS port (0 = same as port)
port	int	6379	port	Plain TCP port (0 = TLS only)
tls_cert_file	string	""	tls-cert-file	Path to TLS certificate
tls_key_file	string	""	tls-key-file	Path to TLS private key
tls_key_pass	string	""	tls-key-file-pass	Private key password
tls_ca_cert_file	string	""	tls-ca-cert-file	Path to CA certificate
tls_ca_cert_dir	string	""	tls-ca-cert-dir	Path to CA cert directory
tls_auth_clients	string	"yes"	tls-auth-clients	Client cert auth: no/yes/optional
tls_client_cert_file	string	""	tls-client-cert-file	Client certificate
tls_client_key_file	string	""	tls-client-key-file	Client private key
tls_protocols	string	"TLSv1.2 TLSv1.3"	tls-protocols	TLS versions
tls_ciphers	string	""	tls-ciphers	Server ciphers
tls_ciphersuites	string	""	tls-ciphersuites	TLS 1.3 ciphers
tls_session_caching	bool	true	tls-session-caching	Enable session cache
tls_replication	bool	false	tls-replication	TLS for replica connections
tls_cluster	bool	false	tls-cluster	TLS for cluster bus

Implementation outline:

1. Add config fields to internal/config/config.go
2. Add validation — if TLS enabled, require cert+key files
3. Add internal/auth/auth.go — password verification with constant-time compare
4. Add AUTH command handler
5. Add HELLO command handler with RESP2/RESP3 support
6. Add TLS listener support in server — use Go's crypto/tls package
7. Detect TLS connections via tls.ConnectionState
8. Update connection state to track authenticated status
9. Add CMD_NO_AUTH equivalent — allow AUTH/HELLO without auth
10. Update CLIENT INFO to show TLS/auth status
11. Update README.md and AGENTS.md

Edge cases:

• AUTH on server without password configured → error
• HELLO without AUTH when password configured → auth required
• Plaintext connection to TLS-only server → reject
• TLS client cert verification → verify against CA if configured
• Both plaintext and TLS ports configured → support both
• AUTH in HELLO followed by regular AUTH → both should work

Security notes:

• Use subtle.ConstantTimeCompare() for password comparison
• Log failed AUTH attempts with client address
• HELLO response should include authenticated: true/false
• Redis 6+ allows AUTH before SUBSCRIBE and in HELLO

---

5. Structured JSON logging

In addition to plaintext logging, implement structured JSON logging with config option to switch between them.

Motivation: JSON logging enables easy parsing by log aggregation systems (ELK, Loki, CloudWatch, etc.). Plaintext remains for human readability in development.

Design:

Current format (plaintext)

2024 Apr  2 14:30:15 INFO pekoe[12345]: event=startup addr=0.0.0.0:6379

JSON format

{"timestamp":"2024-04-02T14:30:15Z","level":"INFO","pid":12345,"event":"startup","addr":"0.0.0.0:6379"}

Config options:

Option	Type	Default	Description
log_format	string	"text"	Log format: text or json
log_timestamp_format	string	"rfc3339"	Timestamp format for JSON (rfc3339, iso8601, unix)

Log levels:

• DEBUG, INFO, WARN, ERROR, CRITICAL

JSON fields:

Field	Type	Description
timestamp	string	RFC3339 formatted timestamp
level	string	Log level (DEBUG, INFO, WARN, ERROR, CRITICAL)
pid	int	Process ID
message	string	Main log message
*	any	Key-value pairs from log message

Implementation outline:

1. Refactor internal/logger/logger.go — separate formatting from logging
2. Add formatText() — current plaintext formatter
3. Add formatJSON() — new JSON formatter
4. Add config field LogFormat in internal/config/config.go
5. Update logger initialization to read format from config
6. Add tests for both formatters
7. Handle special characters in JSON output (escaping)
8. Ensure log file output uses same format as stdout

Example output:

text:

2024 Apr  2 14:30:15 INFO pekoe[12345]: event=startup addr=0.0.0.0:6379 max_clients=10000

json:

{"timestamp":"2024-04-02T14:30:15Z","level":"INFO","pid":12345,"event":"startup","addr":"0.0.0.0:6379","max_clients":10000}

Edge cases:

• Empty message → emit JSON with message: ""
• Nested key=value → flatten with underscores
• Binary data in message → base64 encode or skip
• High-frequency logging → consider JSON marshaling optimization

Use cases:

• Text: development, debugging, simple deployments
• JSON: Kubernetes, Docker, log aggregation systems, SIEM