Market Data Service

The Market Data Service ingests raw trade and book events from the matching engine, derives canonical market data (order books, candles, 24h tickers, mark / index / funding prices), persists it, and distributes real-time feeds to clients over a multiplexed WebSocket.

It is an actor / event-sourced, exactly-once stream processor. Every symbol is owned by exactly one single-threaded actor goroutine, assigned by Kafka partition ownership (cooperative-sticky rebalance), so the service scales horizontally. Each actor consumes engine.event.v1 and derives everything deterministically on event time, then publishes the canonical md.* topics inside a Kafka transaction that atomically commits outputs + consumer offsets. Serving is decoupled: a per-replica fan-out consumer re-reads the md.* topics into an in-memory read cache that backs REST / gRPC / WebSocket, so any replica can serve any symbol’s reads.

Architecture

• Compute consumer (read_committed, cooperative-sticky) consumes engine.event.v1, engine.bookhash.v1, md.spot.raw.v1, and md.instrument.*. A rebalance bridge maps owned partitions → symbols and spawns one actor per symbol.
• Per-symbol actor derives book, trades, OHLCV, 24h ticker, index, mark, and funding on event time, emits the md.* outputs transactionally, and snapshots its state (protobuf, SHA-256 verified) to Postgres for warm restart.
• Fan-out consumer (per-replica group) re-reads all md.* topics into an in-memory read cache backing REST / gRPC / WebSocket.
• Spot ingestor (leader-elected) publishes md.spot.raw.v1 from external exchanges; actors blend non-stale sources into the index price.
• Archiver (cmd/archiver) pages Timescale history to ClickHouse.

Kafka topics

Consumes: engine.event.v1, engine.bookhash.v1, engine.snapshot.v1 (cold start), md.spot.raw.v1, md.instrument.{created,updated,halt,resume}.v1, config.index_price.{created,updated,deleted}.v1.

Produces (canonical, symbol-keyed): md.trades.v1, md.orderbook.delta.v1, md.orderbook.snap.v1, md.ohlcv.v1, md.ticker24h.v1, md.mark.v1, md.index.v1, md.funding.v1, md.spot.raw.v1 (from the ingestor).

REST API

All read endpoints are path-param and served from the in-memory read cache; history endpoints read Timescale. Times are nanoseconds (from_ns / to_ns).

Endpoint	Description
GET /v1/symbols	Instrument list (proxied from metadata-service)
GET /v1/mark/:symbol	Latest mark price
GET /v1/index/:symbol	Latest index price (+ components)
GET /v1/orderbook/:symbol?limit=	Order book snapshot
GET /v1/trades/:symbol?limit=	Recent trades
GET /v1/candles/:symbol?interval=&from_ns=&to_ns=&limit=	OHLCV candles
GET /v1/ticker/:symbol	24h ticker
GET /v1/funding/:symbol	Latest funding rate
GET /v1/{mark,index,funding,trades}/:symbol/history?from_ns=&to_ns=&limit=	Timescale history
GET /healthz, GET /metrics, GET /ws	Health, Prometheus, WebSocket

WebSocket

Multiplexed feed at /ws. See the WebSocket Integration guide and the AsyncAPI spec for the full protocol. In brief: subscribe per (channel, symbol) with { "op": "subscribe", "channel": "...", "symbol": "..." }; set book rounding with { "op": "config", "prec": 0.5 }; server frames are { channel, symbol, type, data, stale } where type is snapshot | delta | pong | ack | error. Channels: book, ticker, trades, mark, index, funding, candle:<interval> (e.g. candle:1m).

gRPC

market.v1 MarketDataService: GetMarkPrice and GetOrderBook (served from the read cache; GetMarkPrice falls back to index-only on cold start), and ReplayStream (streams trades, book, candles:<interval>, mark, index, funding from Timescale history; start_time / end_time are epoch nanoseconds).

Persistence

Timescale (ns-resolution, source_seq in PKs for determinism): md_trade, md_ohlcv, md_mark_history, md_index_history, md_funding_history, md_book_snapshot, plus mds_snapshot (event-sourcing state) and md_instrument. md_funding_history is retained indefinitely (audit-critical). ClickHouse holds *_archive tables populated by the archiver.

Observability

Prometheus mds_* metrics: mds_events_processed_total, mds_output_published_total, mds_book_hash_mismatch_total, mds_late_trade_dropped_total, mds_gap_detected_total, and gauges mds_consumer_lag, mds_actor_mailbox_depth, mds_snapshot_age_event_time_seconds, mds_ws_connections, mds_partitions_owned, mds_spot_ingestor_leader.