Architecture — detailed record by module (session continuity document)

Architecture — detailed record by module (session continuity document)

A new session (human or AI) restores the full context by reading SESSION-HANDOFF.md → strategy-landscape.md → this document → decisions.md → experiment-log.md in that order. When modifying code, update the corresponding section of this document too. (strategy-landscape.md = the analysis document on capital tiers, competitive structure and out-of-scope models, D-034)

0. Collaboration workflow (important — environment constraints)

• The AI sandbox blocks all external network access (Binance/GitHub/PyPI all). Therefore:
• Code writing and synthetic-data validation = AI / real-data runs, pytest and git push = the user's local machine
• Real-data results are delivered by the user pasting terminal output
• The OneDrive file truncation issue is real (D-003): the AI-side mount reads files in a truncated state in every session. The Windows originals are mostly intact. The AI validates with /tmp copies, and final trust rests on the user's local pytest.
• Windows encoding: every text I/O must state encoding="utf-8" explicitly (history of cp949 incidents).
• parquet engine: pyarrow locally; none in the AI sandbox (tested by monkey-patching with pickle).

1. Data layer — src/data/

binance_client.py

• BinanceClient(base_url, kline_path) — futures by default. Spot is BinanceClient("https://api.binance.com", kline_path="/api/v3/klines").
• Built-in retries, backoff and rate limiting (0.3s sleep per request). Only endpoints that need no authentication.
• Pitfall: the batch limit differs by market (futures 1500 / spot 1000). Do not use the batch size to decide when collection ends (bug history before E-09).

fetch.py

• fetch_klines_incremental(client, data_dir, symbol, interval, start_date, market) — Incremental collection. Only after the last candle of the existing parquet. The last candle may be incomplete, so it is dropped and re-collected.
• fetch_funding_incremental / fetch_exchange_spec.
• Path conventions: data/klines/{sym}_{iv}.parquet, data/klines_spot/, data/funding/{sym}_funding.parquet
• Funding intervals are not fixed at 8h (4h for some symbols and periods — SOL/TIA/WIF/ENA). Annualize based on timestamp intervals (see analyze_funding.py).

validate.py

• validate_klines(df, symbol, interval, ...) → (df with flag columns added, QualityReport)
• Checks for missing data (continuity), outliers (±30%, flagged rather than removed), OHLC consistency and duplicates.
• Known legitimate gap: 1 in spot at 2023-03-24 13:00 (exchange halt — no interpolation).

liquidations.py (2026-07-16, session 5)

• Collects and cleans the COIN-M liquidationSnapshot archive (USD-M has been removed from the archive).
• Handling a known defect: every row duplicated 2x → full-field key deduplication. Thanks to the 1-second throttle, this method is provably safe (minimum gap of 1.020s confirmed on the real sample).
• fetch_liquidation_archive (daily ZIPs, idempotent — success, 404 and errors recorded in the manifest), parse_liquidation_csv (normalization + format-change guard), validate_liquidations (throttle consistency check), find_clusters (gap-chained cascade candidates).
• Side convention: side=SELL → long liquidation. Test fixture: tests/fixtures/liq_sample_raw.csv (100 rows of real data, for type checks).

split.py / partitions.py

• seal_split — 1h 60/20/20 chronological split + sha256 seal. Refuses double sealing.
• load_partition(data_dir, name, i_know_what_i_am_doing=False) — test cannot be loaded without the flag; hashes verified.
• partitions.load_sliced(data_dir, df, partition) — a slicer that applies the same date boundaries to unsealed resolutions (1m etc.) and other symbols. Blocks anything after the test boundary (2025-09-16 19:00 UTC).
• Status: the test partition has been opened and consumed (D-009). It can be loaded but must not be used for design.

2. Indicator and signal layers — src/indicators/, src/signals/

indicators/core.py

• Pure functions: sma, ema, rsi(Wilder), bbands, atr. Stateless. Premised on sharing between backtester and live.

signals/ — builder conventions

• build_signal(df, params) → target Series (+1/0/-1) or a (target, stop_dist) tuple
• No shift: signals are based on the candle close. Filling at the next candle open is the engine's responsibility.
• registry.build(name, df, params) → always normalized to (target, stop_dist|None).
• Registered: rsi_meanrev (discarded), vol_breakout (failed), cascade_proxy (rejected), intraday_shock_reversion (rejected, D-020) — kept for record and reproduction.

3. Execution layer — src/engine/

backtest.py — single-symbol loop engine (D-001, 300-line cap)

• Execution model: signal at i close → fill at i+1 open, slippage in the adverse direction, taker fees, funding (at matching open_time; longs pay positive rates), entries below the minimum order skipped.
• Stops (D-004): the stop price is fixed at entry with stop_dist[i-1], intrabar touches fill, gap-throughs fill at the open. After a stop, no re-entry until the signal resets to 0.
• Sizing (D-006): Sizing(size_fraction, leverage, vol_target, vol_lookback, dd_threshold, dd_factor) — vol targeting only reduces (never increases); drawdown reduction is based on the previous candle's equity.
• Bankruptcy: equity ≤ 0 → everything frozen at 0.0 thereafter (leaving negatives contaminates statistics — E-11 bug history).
• Final sizing: option F = Sizing(0.25, 2.0, vol_target=0.40) (D-007).
• [gpt 5.6 sol] cost audit extension (D-021): optional entry/exit fee and slippage fields in CostModel. If None, the existing taker defaults. Only for maker counterfactuals on the same fills; it does not mean real limit fills.

stats.py

• compute_stats(result, bars_per_hour=1.0) — must be 60.0 for 1m data (annualization).
• CAGR for periods shorter than 1 day is treated as n/a.

funding_arb.py — a dedicated funding carry simulator (two legs: spot long + futures short)

• Separate from the single-position engine. Reports a breakdown of funding received, fees and basis.
• Status: on hold (D-011/D-012) — reusable as is for a retrial.

4. Research layer — src/research/

grid_search.py

• grid_search(...) → (df of all combination results, meta{n_combos, ...}). The number of combinations is always recorded.
• Combinations short of the trade count are eligible=False (a Sharpe from 5 trades is noise).
• Watch the bars_per_hour parameter (1m data).

walkforward.py

• walk_forward(...) — rolling train_bars→test_bars, grid re-search per window, continuous capital.
• Passing a sizing_variants dict runs sizing A/B on the same signals (plan 4.2).
• Indicators are recomputed within the window slice → no positions in the warm-up period (requires train_bars ≫ warm-up).

montecarlo.py

• block_bootstrap_mdd(returns, ...) — block bootstrap MDD distribution, P(MDD≤-15%) and so on.
• Limit: it reshuffles observations, so unobserved crashes are not reflected — the estimate is an optimistic lower bound.

5. Portfolio layer — src/portfolio/

rotation.py — strategy 6 (validation in progress)

• build_rotation_targets(closes, params) — mom_n return ranking, trend_n SMA filter, positive-momentum condition, top_k selection, rebalancing every reb_bars.
• run_rotation(dfs, fundings, ...) — sums independent per-symbol engines (capital/K).
• Documented approximations: (a) no compounding transfer between symbols, (b) qty_step 1e-6 (optimistic relative to BTC's coarser step — real exchangeInfo specs must be revalidated if it survives), (c) if approx_broken is True, statistics are invalid.
• Known constraint: at K=2, per-leg capital (365) × sizing approaches the minimum order (100) → frequent skips.

5.5 Harness layer — src/harness/ (Phase 6, implemented 2026-07-14)

dsr.py

• Deflated Sharpe (Bailey & López de Prado 2014). Uses stdlib NormalDist without scipy.
• Unit convention: all non-annualized per-bar Sharpe — never mix with the annualized values of stats.py.
• sharpe_ratio / probabilistic_sharpe / expected_max_sharpe / deflated_sharpe.
• Degeneracy guard: prevents Sharpe blowing up to 1e15 from constant series with std ~1e-18 (a measured bug).

grammar.py

• Block grammar of entry 7 × exit 6 × filter 4 × direction 3 → 4,752 combinations (v1).
• build_signal(df, spec, funding=None) → (target, stop_dist|None) — same registry convention, so the engine and existing validation tools digest it unchanged. spec_id() = 12 characters of the sorted-JSON sha256.
• Engine integration pitfall: the engine forbids re-entry after a stop until the signal resets to 0 → for combinations like atr_stop where the stop is the only exit, a close-based stop mirror at the signal level is mandatory (otherwise no trades ever again after the first stop).
• The funding_sign filter raises ValueError if there are no funding data (no silent pass).

runner.py

• run_harness(train, val, specs, ...) — appends to the ledger immediately per combination (reports/harness_ledger.csv = DSR denominator, never delete) + a per-run CSV. Rerunning the same run_id = skip recorded combo_ids (resume). Failures are recorded in the ledger too.
• Funnel: Stage 0 trade count → Stage 1 DSR (full-ledger N) → Stage 2 val comparison. The pass lines are preregistered under D-017 — do not modify runner.GATES.
• Rerunning the same combination under a new run_id over-counts the denominator — allowed because it errs conservative.

5.6 Pairs layer — src/pairs/ (strategy 4, implemented 2026-07-14)

cointegration.py

• Engle-Granger 2-step, implemented in numpy (no statsmodels dependency). OLS hedge + residual ADF (no constant, lags=1). Critical values from the MacKinnon approximation (5% -3.34).
• rolling_engle_granger(la, lb, window, step) — retest every step, each bar gets the latest fit (the fit uses only data up to that bar — no look-ahead).
• Synthetic measurement: cointegrated pairs detected 100%, independent pairs falsely detected 6% (nominal 5%).

simulator.py

• Two-leg futures simulator (lineage of funding_arb.py). Returns an engine.BacktestResult → compute_stats used as is. Signal close → next open fill, per-leg costs and funding.
• Rules: enter only in EG-passing periods; enter at |z|≥z_in / exit at |z|≤z_out / stop at |z|≥z_stop (after a stop, no re-entry until |z|<z_in returns); forced exit when cointegration breaks.
• Approximations (documented): refits while holding update only the decision z (P&L fixed at entry β), qty step not applied.

6. Scripts — scripts/ (all for local runs)

limit_backtest.py — pending limit + OCO engine (D-022)

• From the bar after the signal, TTL 4 bars, cancel if unfilled, touch and trade-through conditions separated.
• Entry/TP maker, SL/time taker. TP on the entry bar is not recognized, SL is; simultaneous touches favour SL.
• Orders, fills, cancels, fill rate and entry regime are recorded in result meta/trades.

6.5 Event execution and research lineage layer — master architecture v1.0

src/research/registry.py

• SQLite persistent ledger. Experiments are preregistered as RUNNING and then end exactly once as COMPLETED/FAILED

. Rewriting the results of completed rows is refused.

• Stores family/parent, code commit+dirty, data hash/period, cost, funding, fill and latency, all parameters, split, signal/order/fill, metrics, verdict and artifacts.

src/execution/event_exchange.py

• A strategy-common OrderIntent and a deterministic event replay adapter.
• Processing order: book update → fills of existing resting/cancel-pending orders → due ACK/cancel → new intents.
• States: CREATED/SUBMITTED/ACKNOWLEDGED/RESTING/PARTIALLY_FILLED/CANCEL_PENDING and FILLED/REJECTED/CANCELED/EXPIRED. F0/F1/F2 and queue-ahead are explicit.
• Queues are not estimated from OHLCV. F1/F2 measurements are unverified until AggTrade/BBO/L2 are collected.

src/execution/shadow_ledger.py

• An append-only shadow observation store separating exchange time from local receive time.
• submit/ACK/rest/partial/fill/cancel/reject and 1s/5s/30s/60s markouts can accumulate under the same order_id. The real Binance transport and credentials are not yet connected.

src/signals/e17_trend_breakout.py, src/research/event_study.py

• Only completed HTF bars are aligned as-of onto the LTF. E-17A evaluates only directionality with next-bar market orders and per-signal returns, without pullback/maker parameters.
• v1 wiped out under D-025. E-17B onward is not run.

src/signals/macro_trend_families.py

• 2d/20d/60d/90EMA regimes from completed daily bars are aligned with a lag onto 5m.
• 6 entry families are preserved permanently as independent function branches: channel, volume channel, EMA reclaim, EMA cross, RSI reclaim, volatility release. Code is not deleted because it failed.
• apply_frequency_policy applies only the common cooldown and daily signal cap across families.

scripts/run_macro_trend_families.py, run_macro_swing_families.py

• E-20 runs 30m/1h/2h and E-21 runs 6h/12h/24h horizons as separate lineages.
• The long-hold event study finds real funding timestamps and reflects longs paying / shorts receiving.
• Under D-028, RSI reclaim 24h is the closest reference group but not a promotion candidate.

src/execution/aggtrade_replay.py, scripts/run_macro_rsi_maker_replay.py

• Downloads official Binance USD-M daily AggTrade ZIPs and .CHECKSUM, verifies SHA-256, then keeps only preselected order windows as parquet. --config/--run-tag/--cache-prefix separates the train pilot from the independent val, and resuming the same run reuses the date cache.
• Replays F0 touch, F1 opposite-aggressor trade-through and F2 explicit queue-ahead. Records 1/5/30/60-second markouts and PnL per signal including unfilled = 0.
• AggTrade has no BBO/L2, so post-only acceptance and real queues cannot be proven. In E-24 the independent val was negative, so the directional family was rejected even before this constraint was resolved.

7. Open items (checklist for the next session)

1. Check whether 3 months of real liquidation data have accumulated, and the status of the recorder/VPS.
2. For funding carry, only periodically watch the D-011 retrial condition (30-day rolling annualized 8%+).
3. New limit-order searches are preregistered not as numeric variants of D-023 but only as mechanisms with a different order of information, such as a re-pullback after reclaim confirmation.
4. The staged capital × minimum order conflict (D-007) is resolved when a deployment candidate appears.