← All projects Independent project
Microstructure Discovery Factory
I built a system that collects live cryptocurrency market data from multiple exchanges, stores it for analysis and checks it for gaps and errors.
Independent Researcher | Market Data & Systems · August 2026 - Present
Why I built it
I wanted to study how markets behave during busy events. That requires a detailed record of trades and orders, along with a way to tell whether anything was missed. A price chart alone cannot answer those questions.
What I implemented
- Collectors that receive exchange updates and trades over WebSocket connections.
- A storage pipeline that saves the data for later analysis and checks what reached disk.
- Validation and replay tools to investigate missing updates, recover from interruptions and repeat research.
- Monitoring and lifecycle work for a dedicated Alienware Linux collector, separating collection from research workloads.
What the system does
The collector records trades and changes to the order book, the list of displayed buy and sell orders. It processes those updates, stores them and checks the resulting dataset before research uses it. Replay lets me revisit an event with the information that was available at the time.
Documented capture evidence
During one documented Federal Reserve policy event (FOMC), the collector recorded the data below. I used it to study how much displayed Bitcoin and Ether buy and sell interest fell around the announcement, and how it recovered.
One documented FOMC event capture
- capture lanes completed
- 12/12
- order-book (L2) updates
- 653,722
- trades recorded
- 4,339,527
- displayed BTC/ETH depth contraction
- ~68%
Captured around one U.S. Federal Reserve policy announcement. Figures apply to that capture.
TypeScript handles live collection. Python, Parquet and DuckDB support storage checks and analysis. Linux provides a dedicated environment for running the collector.
PythonTypeScriptWebSocketParquetDuckDBLinux
Engineering details
Open a topic for the implementation decisions and checks behind the overview.
Keeping storage and recovery in sync
The pipeline separates receiving an update from confirming that it has reached durable storage. A spool retains pending data; landing and reconciliation check the stored artifacts before research consumes them.
Source identity and exchange, receipt and availability timestamps travel with the data. Research readers use those time boundaries to avoid including information before it was available.
Handling disconnects and busy periods
A reopened connection does not repair a missing order-book interval. The continuity path invalidates book trust on disconnect and starts a fresh snapshot/recovery cycle. Replay retains the interruption.
A separate burst investigation reproduced event-loop starvation in a queue that repeatedly removed the first array element. I replaced that behavior with ring storage and bounded cooperative draining while preserving FIFO order and persistence behavior. Focused regression checks covered the reproduced queue failure.
Running collection on dedicated Linux hardware
The Alienware Linux collector is a practical step from a research workstation toward dedicated collection hardware. The lifecycle work gives acquisition, storage and validation workers a common service owner.
Shutdown stops new inputs before storage workers are torn down. A forced stop retains failure evidence and pending state for explicit recovery. The engineering note explains the implementation checks and the separate physical-host acceptance requirements.
Current scope
The figures describe one event capture, not every collection campaign or a general uptime result. The repository and raw market data remain private. Linux lifecycle software checks and physical-host acceptance are separate pieces of evidence.