Pinterest has rolled out an event time-based partition finalization mechanism for its next-generation database ingestion framework, enabling jobs that consume data to know when streaming information is complete enough to use. The company announced the approach in a recent blog post, explaining that it uses Apache Flink to track event time progress and Apache Iceberg metadata to publish a finalization watermark that never moves backward. The system addresses a persistent problem in continuous Change Data Capture ingestion, where data can show up after the wall clock time for a partition has already passed.
Pinterest's ingestion framework relies on CDC, Kafka, Flink, Spark, and Iceberg to transfer database modifications from MySQL, TiDB, and KVStore into analytical tables. The platform handles petabyte-scale data and thousands of pipelines, with online modifications typically becoming available within 15 minutes to an hour. The new mechanism operates inside the Flink to Iceberg sink, where the system pulls out a configured event time for each CDC record. During Flink checkpoints, it gathers minimum, maximum, and percentile event time statistics, storing them in the Iceberg snapshot summary with a t-digest that represents the distribution in just a few hundred bytes. After an Iceberg commit finishes, the system calculates a partition finalization watermark from recent checkpoint statistics, with the default policy using the minimum event time across a configurable number of recent commits and rounding it down to the relevant hour. The watermark is monotonic, which prevents late-arriving records from pushing an already published value backward, though percentile-based policies can sacrifice some completeness for fresher data.
Nitesh Singhal, Architect at Google, called out the design in a LinkedIn post, writing, "The non-regressing safe-to-read watermark is the elegant bit here." Pinterest describes the change by noting the watermark is stored as an Iceberg table property, which lets downstream sensors read it without needing a separate coordination service. Spark also carries the metadata forward to the base table during its CDC upsert commit. The company emphasizes that finalization is no longer a byproduct of a job finishing, a shift from the previous batch-based approach where job completion indicated that a partition was ready to use.
The design builds on existing Flink and Iceberg capabilities, according to Pinterest. Flink watermarks track event time progress in out-of-order streams, while Iceberg snapshots provide persistent table metadata. Pinterest combined these capabilities to derive a table-specific completeness signal and store it alongside the data itself. Unlike batch processing, where a finished job signals readiness, streaming ingestion continuously commits changes without an equivalent stopping point—making it unclear when downstream consumers should start processing. The watermark solves this by publishing a reliable threshold that advances as event time progresses but never retreats, even when late data arrives. Tharun Kumar Reddy Polu, Engineer at Salesforce, identified the core tension in a LinkedIn comment, writing, "The streaming vs batch tradeoff framing is what makes this land. Getting freshness without losing the guarantees batch gave us is the core challenge."
Pinterest implemented the mechanism using extensibility points in its Flink-to-Iceberg sink, with accumulators collecting record statistics and commit processors executing finalization logic around Iceberg commits. Separately, an Apache Iceberg issue discusses extending IcebergSink for downstream integrations, including custom committable metadata such as watermark information. The approach lets Pinterest maintain the speed advantages of streaming ingestion while giving downstream jobs the confidence that batch systems traditionally provided. For organizations managing large-scale data pipelines, the tension between low latency and data completeness often forces a choice between stale-but-complete batch outputs and fresh-but-uncertain streaming feeds. Pinterest's solution suggests that metadata-driven coordination can thread that needle, though success depends on tuning policies to match each pipeline's tolerance for incompleteness.