GCStocks Stream — Level-2 Deep Dive

GCStocks Stream is the constellation's real-time market-data backend. It fans in live and on-demand market data from several independent upstream vendors (crypto and US equities), normalizes everything to a single canonical wire format, and re-emits it to internal consumers as streams (Server-Sent Events) and pull endpoints. It is the data foundation for capital-markets features across the venture — the first being intraday/daily valuation of open positions for the ERP.

Internal architecture

A Maven multi-module monorepo, deliberately split three ways:

contracts — the canonical wire models (tick, quote, OHLCV candle, fundamentals, news). A framework-free module with no serialization-framework coupling (a hand-rolled tick codec), so every producer and consumer shares one definition of the data. That decoupling let the stack absorb a Spring Boot 4 / Jackson 3 migration without touching the wire shapes.
simulator — a first-class mock market source that emits the same canonical contracts, deterministically and configurable per symbol. The whole reactive pipeline is built and tested against it with zero vendor dependency, rate limit, or cost; live vendors are simply additional sources.
aggregator — the product: a reactive (Spring WebFlux / Project Reactor) service that consumes N sources over WebSocket, normalizes, and re-emits.

Inside the aggregator:

MarketSource abstraction — every upstream (the simulator, a keyless crypto feed, a keyed equities feed) implements one interface; enabling a source is configuration, not code. Fan-in is resilient: exponential-backoff reconnect, and a clean upstream close is treated as a reconnect trigger rather than an end-of-stream.
Subscription control plane — a demand-driven manager keeps each (source, symbol) active while it has at least one hold: an explicit pin (control API) or a demand refcount (an SSE consumer subscribed to that symbol, auto-released on disconnect). The upstream subscribe / unsubscribe fires only on the 0↔active transitions — keeping the aggregator automatically inside each vendor's scarce symbol cap while still allowing deliberate always-on symbols.
Capability model — beyond streaming ticks, four capabilities are each mirrored on the simulator and the aggregator: last-price quote, OHLCV candles, fundamentals, and news (list + live stream). Each is a small provider interface built from a config list, so a capability can be served by the simulator, a real vendor, or both.
Fan-out — normalized data leaves as Server-Sent Events (streaming: ticks, news) and REST (pull: quote, candles, fundamentals, news).
Read-model + gated catalog data-mining — a relational read-model persists last-price, price history, and per-symbol statistics. A rate-paced, jittered catalog sweep populates and refreshes it, prioritizing symbols by observed volatility (a Garman-Klass estimate over stored OHLC) and admitting only symbols that clear a volume-first gate (a liquidity floor plus a price band). This turns a scarce vendor budget into a focused working universe — roughly a third of a ~31k-symbol catalog — instead of trying to watch everything. Blocking database work is isolated on a bounded-elastic scheduler so it never stalls the reactive event loop.

Why this shape. The hard problems are (1) fanning many long-lived upstream streams out to many downstream consumers without blocking — hence reactive, not servlet; (2) staying inside metered, capped vendor budgets — hence the demand-driven subscription plane and the volume-gated miner; and (3) keeping one stable data definition across many producers and a framework migration — hence the framework-free contracts module. The rejected alternative throughout was coupling directly to a live vendor; the simulator plus the MarketSource abstraction keep the vendor at arm's length and make the licensed feed a drop-in.

Interfaces

Provides — canonical market data to internal consumers as SSE streams (ticks, news) and REST pull (quote, candles, fundamentals, news); plus a control API to discover the subscribable catalog and pin/unpin symbols at runtime.
Consumes — upstream vendor feeds (a keyless crypto WebSocket; a keyed equities WebSocket + REST), normalized to the canonical contract on ingest. Its read-model lives on the platform's shared data plane.
Planned edge — GCStocks → ERP: provide intraday/daily marks so the ERP can value open positions. GCStocks supplies the market data; the ERP consumes it.

Tech stack

Java 21 · Spring Boot (WebFlux, reactive) · Project Reactor / Reactor Netty · Maven multi-module monorepo (contracts / simulator / aggregator) · WebSocket (upstream) → Server-Sent Events + REST (downstream) · relational read-model with typed, versioned SQL migrations · Docker (multi-stage → JRE runtime) · deployed behind the platform's nginx/TLS and shared data plane.

Status & near roadmap

Pre-production — commercial data licensing underway. Built and running against live vendors in development: the reactive multi-source aggregation, the subscription control plane, all four pull/stream capabilities, the simulator, and the read-model + volume-gated catalog data-mining. A provider survey established that shipping market data inside a commercial product requires a licensed feed; vendor evaluation is underway, and because every source sits behind the MarketSource abstraction, the licensed feed drops in as one more source with no pipeline change.

Near roadmap: a live-session sampler — prioritized WebSocket sampling of the highest-volatility symbols within the vendor symbol cap, promoting names as they heat up — is in design; then the ERP valuation edge.