Skip to content
TradeX Documentation

Metadata Service

The Metadata Service acts as the canonical reference data and configuration store for the TradeX exchange. It manages trading instruments, fees, risk parameters, funding parameters, and other exchange metadata.

Overview

Section titled “Overview”

The Metadata Service provides:

  • Instrument Management: CRUD operations for trading instruments with maker/checker workflow
  • Versioning: Optimistic locking and version tracking for all configuration changes
  • Event Publishing: Publishes change events to Kafka for downstream consumers
  • Audit Trail: Immutable audit log for all configuration changes
  • Idempotency: Support for idempotent operations via idempotency keys
  • High Performance: gRPC API for internal services with Redis caching
  • Reliable Events: Transactional outbox pattern for guaranteed event delivery

Architecture

Section titled “Architecture”
graph TB Admin[Admin Client] --> REST[REST API] Services[Internal Services] --> GRPC[gRPC API] REST --> Service[Metadata Service] GRPC --> Service Service --> PG[(PostgreSQL)] Service --> Redis[(Redis Cache)] Service --> Outbox[Transactional Outbox] Outbox --> Worker[Outbox Worker] Worker --> Kafka[Kafka Events]

Core Components

Section titled “Core Components”

Data Models

Section titled “Data Models”
  • instruments: Trading instrument configuration (symbol, type, status, etc.)
  • risk_configs: Risk parameters per instrument (leverage limits, position limits, etc.)
  • fees: Fee schedules (maker/taker fees, funding fees, etc.)
  • audit_log: Immutable audit trail for all configuration changes
  • config_snapshots: S3 references for configuration snapshots
  • outbox: Transactional outbox for reliable event publishing
  • idempotency_keys: Idempotency key tracking

REST API (Admin)

Section titled “REST API (Admin)”

Admin endpoints for managing instruments and configuration:

  • POST /v1/instruments - Create instrument (maker mode)
  • POST /v1/instruments/:symbol/approve - Approve instrument (checker mode)
  • POST /v1/instruments/:symbol/halt - Halt trading for an instrument
  • POST /v1/instruments/:symbol/resume - Resume trading for an instrument
  • GET /v1/instruments - List all instruments
  • GET /v1/instruments/:symbol - Get instrument by symbol

gRPC API (Internal)

Section titled “gRPC API (Internal)”

High-performance internal API for reading configuration:

  • GetInstrument(symbol) - Get a single instrument
  • GetAllInstruments() - Stream all instruments
  • WatchConfigUpdates() - Watch for configuration updates (server streaming)
  • GetConfigVersion(version) - Get configuration by version

Maker/Checker Workflow

Section titled “Maker/Checker Workflow”

The service implements a maker/checker workflow for instrument changes:

  1. Maker: Creates or updates an instrument (status: pending)
  2. Checker: Reviews and approves the change (status: active)
  3. Event Publishing: Upon approval, events are published to Kafka
  4. Audit Trail: All changes are logged in the audit log

This ensures that critical configuration changes are reviewed before being applied to the live system.

Event Publishing

Section titled “Event Publishing”

The service publishes the following Kafka events:

  • md.instrument.created.v1 - Instrument created and approved
  • md.instrument.updated.v1 - Instrument updated
  • md.instrument.halt.v1 - Instrument halted
  • md.instrument.resume.v1 - Instrument resumed
  • config.risk.updated.v1 - Risk configuration updated
  • config.funding.updated.v1 - Funding configuration updated
  • config.fees.updated.v1 - Fee configuration updated

Transactional Outbox Pattern

Section titled “Transactional Outbox Pattern”

Events are published using the transactional outbox pattern:

  1. Configuration change is saved to PostgreSQL in a transaction
  2. Event is written to the outbox table in the same transaction
  3. Outbox worker periodically processes outbox events
  4. Events are published to Kafka and marked as processed

This ensures that events are only published after the database transaction commits, providing guaranteed delivery.

Configuration

Section titled “Configuration”

Key Environment Variables

Section titled “Key Environment Variables”
  • KAFKA_BROKERS: Kafka broker addresses
  • SCHEMA_REGISTRY_URL: Schema Registry URL
  • REDIS_URL: Redis connection URL
  • POSTGRES_URL: PostgreSQL connection URL
  • HTTP_PORT: HTTP server port (default: 8080)
  • GRPC_PORT: gRPC server port (default: 50051)
  • REQUIRE_MAKER_CHECKER: Require maker/checker workflow (default: true)
  • CACHE_TTL_SECONDS: Cache time-to-live in seconds (default: 300)
  • OUTBOX_BATCH_SIZE: Number of outbox events to process per batch (default: 100)
  • OUTBOX_POLL_INTERVAL: Interval between outbox polling cycles (default: 1s)

Validation Bounds

Section titled “Validation Bounds”
  • GLOBAL_MAX_LEVERAGE: Global maximum leverage allowed (default: 100)
  • MAX_FUNDING_INTEREST_RATE: Maximum funding interest rate (default: 0.01)
  • ALLOWED_FUNDING_INTERVALS: Comma-separated list of allowed funding intervals in hours (default: 1,4,8,24)

Caching Strategy

Section titled “Caching Strategy”

The service uses Redis for caching frequently accessed data:

  • Instrument Data: Cached with configurable TTL
  • Risk Configs: Cached for fast access
  • Fee Schedules: Cached for quick lookups

Cache invalidation occurs automatically when configuration changes are approved.

Versioning

Section titled “Versioning”

All configuration changes are versioned:

  • Optimistic Locking: Version numbers prevent concurrent modification conflicts
  • Version History: Previous versions can be retrieved via GetConfigVersion
  • Snapshot Support: Configuration snapshots can be stored in S3

Idempotency

Section titled “Idempotency”

The service supports idempotent operations via idempotency keys:

  • Clients can include an Idempotency-Key header in requests
  • Requests with the same idempotency key return the same result
  • Idempotency keys are tracked in the database
  • Prevents duplicate operations from retries

Performance Characteristics

Section titled “Performance Characteristics”
  • gRPC API: High-throughput internal API for configuration reads
  • Redis Caching: Sub-millisecond response times for cached data
  • Batch Processing: Outbox events processed in batches for efficiency
  • Connection Pooling: Optimized database connection management

Observability

Section titled “Observability”

Metrics

Section titled “Metrics”
  • Configuration change operations
  • Cache hit/miss rates
  • Outbox processing latency
  • gRPC request latency

Tracing

Section titled “Tracing”

OpenTelemetry distributed tracing for request flows.

Logging

Section titled “Logging”

Structured logging with:

  • Configuration change events
  • Approval workflow steps
  • Event publishing status
  • Error details

Integration with Other Services

Section titled “Integration with Other Services”

The Metadata Service is consumed by:

  • Market Data Service: Reads instrument configuration for symbol validation
  • Order Service: Validates orders against instrument rules
  • Matching Engine: Reads risk parameters and trading rules
  • Position Service: Reads leverage limits and position constraints
Section titled “Related Documentation”