A Unified Runtime for Batch–Stream Co-Optimization with Declarative Backpressure and Operator Fusion

Abstract

Modern data-intensive applications increasingly require unified handling of historical (batch) data and continuous (streaming) data under shared correctness and performance objectives. While many systems offer hybrid APIs, their runtimes often preserve separate execution paths, leaving cross-modal optimization and coordinated resource control under-specified. This work describes a unified runtime that co-optimizes batch and stream pipelines through a single temporal-relational intermediate representation, a declarative backpressure specification that compiles into enforceable control policies, and an operator fusion framework that reduces materialization and improves locality without sacrificing determinism requirements. The design treats backpressure not as an emergent property of queues but as a user- and system-declared contract over latency, memory, and energy budgets, enabling predictable behavior under bursty inputs and skew. Operator fusion is formulated as a semantics-preserving transformation over dataflow graphs with explicit state and time, with a cost model that is differentiable for gradient-based tuning and compatible with constraint-based scheduling. The runtime integrates storage-aware execution, approximate query primitives with error accounting, and distributed mechanisms for partitioning, replay, and recovery. Evaluation methodology is discussed in terms of reproducibility, controllable workload synthesis, and measurable trade-offs among throughput, tail latency, and resource utilization. The overall result is a coherent blueprint for batch–stream co-execution where control-plane decisions, physical layout, and code generation are optimized jointly rather than independently.