Roadmap
Direction, not deadlines.
Strategic principle
Section titled “Strategic principle”Deepen extraction intelligence. Do not broaden into transformation, orchestration, or quality.
Adjacent problems get a companion tool (iPoxy, ixora), not feature creep.
Shipped ✅
Section titled “Shipped ✅”| Phase | What was built |
|---|---|
| 0 Simulation | Architecture v4.0 frozen. 28 simulation scenarios validated. |
| 1 Vertical MVP | PostgreSQL connector, profiler, planner, engine, Parquet writer, CLI, state store. |
| 2A Visibility | diagnose, history, metrics commands. Statistical window-based controller. Direction-aware deviation. |
| 2B Extraction Intelligence | Benchmarker. Context-weighted planning (5-dimension similarity). Adaptive intra-run rules. |
| 2C Connectors | MySQL connector. CSV writer. Work-stealing / LPT scheduling. Extraction manifests. |
| 3A RuntimeContext | 11 user-declared hints. 5-stage worker resolution. Advisory system. Verdict system. |
| 3B Intelligence Foundation | EWMA + context blend. Time-decay (λ=0.05, half-life ~14 days). Python API layer. Exception hierarchy. |
| 3C Adoption Unlock | S3 writer (multipart). GCS writer (resumable). SQL Server connector. Rotating writer. |
| 4A Cost Model + Anomaly Detection | Linear cost formula. Cost comparison with dominance filter. 2σ anomaly detection. Broad baseline. |
| 4B Deterministic Replay | Canonical JSON. Plan fingerprint (SHA-256). execute_plan(). replay(). CLI ixtract replay. |
518 tests passing across 14 simulation suites. 5-run real-world validation on local PostgreSQL and Azure SQL Server.
Building 🔧
Section titled “Building 🔧”These items are in progress or scheduled for the next patch:
- Baseline scoping — scope anomaly detection baseline by
(source_type, host, database, object)to prevent cloud vs. local comparisons - Benchmark CLI wiring —
ixtract benchmarkmodule exists; entry point registration pending - State-aware diagnosis — don’t suggest “consider work_stealing” when work_stealing is already active
- RuntimeContext note in diagnosis — flag constrained runs as excluded from controller learning in
diagnoseoutput
Exploring 💡
Section titled “Exploring 💡”These are directions under consideration, not commitments:
Density-aware chunking
Section titled “Density-aware chunking”Split chunks by estimated row count, not by PK range. Solves the skew problem directly: CV > 1.0 today triggers work-stealing as a mitigation; density-aware chunking would prevent unequal chunks from forming in the first place.
Controller exploration phase
Section titled “Controller exploration phase”The current controller can only reduce workers when it detects degradation. It cannot discover that fewer workers would be better from a stable baseline. An exploration phase would periodically probe ±1 worker even when converged.
Predictive scheduling
Section titled “Predictive scheduling”Time-of-day latency profiles and optimal extraction windows. Advisory only — ixtract will never auto-schedule. Requires real history to be useful.
CDC intelligence
Section titled “CDC intelligence”Classify change patterns: append-only, update-heavy, bulk-replace. Recommend extraction frequency based on change velocity. Classification and advisory only — no automatic behavior changes.
Plugin system
Section titled “Plugin system”Connector, writer, and diagnosis plugins. Let users extend ixtract without forking.
Product family
Section titled “Product family”| Product | Status | Purpose |
|---|---|---|
| ixtract | Available — MIT | Extraction runtime |
| iPoxy | Coming soon — MIT | Pipeline reinforcement: Pre (gate), Watch (monitor), Gate (CI/CD) |
| ixora | Coming soon — Commercial | Fleet intelligence: SLA tracking, cost dashboards, multi-team |
Adoption path: Single engineer (ixtract) → Reliability (iPoxy:Pre) → Team scale (iPoxy:Watch + Gate) → Platform (ixora)