Why LinkedIn made Pinot's serving layer rebuildable

I got stuck on one sentence in the Pinot documentation for about a week.

It says that when a server dies you bring up another one and it downloads what it needs. My first reaction was that this is just replication with extra steps. Every database can do that. I moved on.

I came back to it later because something kept nagging. And the thing I had skimmed past was the phrase what it needs — because in Pinot, what a server needs turns out to be a very different object from what a Postgres replica needs. That distinction is the whole architecture.

This post is me working that out.

I should be straight about where I'm standing: I have not run Pinot in production. Everything below comes from LinkedIn's 2018 SIGMOD paper, the Apache Pinot docs, and LinkedIn's engineering write-ups. Where I'm inferring rather than citing, I've tried to flag it.

Start with the disk#

Here's the question I used to test my understanding.

A server loses its local disk. What do you ask first?

If you're running a conventional database, you ask how to restore the data that lived on that machine. The data was there. Some of it may have existed nowhere else, which is why you have replicas and backups and a runbook you hope is current.

Ask the same question about a Pinot server and you get:

Which immutable segments should this replacement download?

Read those two questions next to each other and the difference is easy to miss, because they look like the same question phrased differently. They aren't. The first assumes the machine held something irreplaceable. The second assumes it held a copy.

Which is why I'd push back on the summary you sometimes hear, that "Pinot servers are disposable." The servers are. The architecture around them is doing real work to earn that.

The problem that justified all this#

Big data alone doesn't explain Pinot. LinkedIn had enormous datasets long before, and batch systems handled them fine.

The hard part was serving analytical queries over fresh data, inside a product page, while someone waits. The SIGMOD paper describes the shape: continuous ingestion, high query concurrency, low latency, freshness in seconds, arbitrary filtering, graceful degradation under failure.

Compare two queries. This one is trivial:

SELECT *
FROM profile
WHERE member_id = 42;

This one is not:

SELECT viewer_industry, COUNT(*)
FROM profile_views
WHERE profile_owner_id = 42
  AND viewed_at >= CURRENT_DATE - INTERVAL '30' DAY
GROUP BY viewer_industry
ORDER BY COUNT(*) DESC
LIMIT 10;

The second scans a lot of rows, touches a few columns, filters on time, aggregates, sorts, and hands back ten lines. Any relational database will run it. That was never the question.

The question is whether it runs thousands of times a second while the same box is accepting profile updates, messages, and connection requests. Those two workloads want opposite things from the storage engine, and they're now fighting over the same buffer pool.

LinkedIn's InFlow system makes it concrete. Roughly 50,000 network flows per second, enriched through Kafka and Samza, tens of terabytes retained for 30 days, queried interactively while an incident is in progress. Nobody wants to wait on a batch job during an outage.

So the shape that justifies Pinot looks like this: events arrive continuously, writes are append-heavy, queries aggregate across many rows, users expect an answer now, history can be rebuilt from a durable pipeline — and, critically, the analytical system is not where the original fact was born.

Hold onto that last one. It does more work than the rest combined.

The boring answer, which is usually correct#

Before the interesting architecture, the unglamorous one.

Product services -> primary relational database -> read replica
Analytics API    -> read replica
Dashboard/UI     -> Analytics API

For most products this is the right answer and you should stay here as long as you can. It's simple, consistent, cheap, and debuggable at 3am by someone who didn't build it.

It breaks when the two workload shapes genuinely conflict. Transactional writes want small indexes and short locks. Analytical reads want columnar scans and wide time windows. A read replica moves the load to another machine, but it carries the same execution model with it, so you've bought time rather than a solution.

A warehouse protects the primary, but batch-loaded data is often too stale and per-query latency too high for a page someone is looking at. A cache helps when requests repeat — and user-facing analytics is precisely where they don't. Different users, different dimensions, different filters, different windows. The space of possible results grows faster than your hit rate.

That's the corner LinkedIn was in.

Three responsibilities, pulled apart#

The move was to stop treating "the database" as one thing:

1. Durable event and batch pipelines
2. Durable analytical segments and metadata
3. Replaceable query-serving nodes

Drawn out, roughly:

Product services
  -> Kafka event streams
    -> real-time Pinot ingestion
    -> ETL/offline pipeline
 
Offline data and completed real-time segments
  -> deep store
    -> Pinot servers load local segment copies
 
Pinot controller + Helix + ZooKeeper
  -> table config, segment metadata, assignment, cluster state
 
Analytics API
  -> Pinot broker
    -> scatter query to servers
    -> merge partial results

The first time I saw this diagram I counted the boxes and concluded it was over-engineered. That was the wrong thing to look at. The boxes are a consequence. What matters is the line the design draws around ownership — and you can only see that by asking, of each box, "if I delete this, what is gone forever?"

Walk it component by component with that question.

Kafka#

LinkedIn published business events to Kafka; Pinot's real-time servers consumed them so recent data was queryable without waiting for a batch run.

Delete a Pinot server and Kafka still has the events. That makes Kafka a recovery input — but only within its retention window. "We can always replay Kafka" is true right up until the offsets you need have aged out, at which point it becomes a sentence people say in incident reviews.

Segments#

Pinot slices tables into segments: column-oriented, replicated, immutable once written.

This is the load-bearing piece. A server doesn't hold unique mutable rows that only it knows about. It holds committed segment files that it has unpacked and indexed. That's the difference I'd skimmed past. A Postgres replica applies a stream of changes to state it maintains; a Pinot server mounts a file that somebody else already finished writing.

Immutable means two servers can't drift. There's no in-place writer to coordinate with, no version of the truth that exists only in one process's memory.

Deep store#

Completed segments get persisted to what the docs call deep store — the permanent home for segment files. New server needs data, it pulls from there. Local file corrupted, it pulls another copy.

Controllers, Helix, and ZooKeeper#

Controllers own table and segment metadata and decide assignments. Helix tracks desired versus current state. ZooKeeper persists it.

Assign a segment to a server and it downloads the file, loads it, and reports itself online. That's the entire lifecycle.

Brokers and servers#

Servers hold local segment copies and do the scanning. Brokers take a query, work out which servers hold the relevant segments, fan out, merge what comes back, and reply.

If a server times out, a user-facing product may prefer a partial answer over a dead page. That's an availability call, and it's worth being honest that it's a step away from the guarantees people assume the word "database" carries.

One event, end to end#

Abstract descriptions never stuck for me, so here's a single event making the trip.

{
  "profileOwnerId": 42,
  "viewerIndustry": "Software",
  "viewedAt": "2026-07-20T08:15:00Z"
}

The product service writes its transaction and publishes to Kafka. It doesn't wait for Pinot. That's the first decoupling, and it means analytics load can never slow down a profile view.

A real-time server picks the event up and appends it to a consuming segment — one still being filled. It can already be queried, which is where the seconds-level freshness comes from.

At some row, size, or time boundary, consumption stops. Pinot decides which replica commits the finished segment, writes it to durable storage, and brings the other replicas onto identical contents. The segment stops being a thing in flight and becomes a file.

Helix hands replicas out. Each server downloads, maps its indexes, reports online.

Later an offline job may reprocess that whole window and build a better-packed historical version. For a hybrid table the broker splits queries at a time boundary:

Historical range -> offline segments
Recent range     -> real-time segments
Final response   -> merged by the broker

One logical table to the caller. No double counting across the seam. That seam, incidentally, is where a lot of the operational pain lives.

The idea underneath#

This is the part that took me longest, and it's the reason I ended up writing any of this down.

The interesting thing isn't streaming ingestion. Plenty of systems stream. It's that a bounded range of ordered input, plus a fixed schema and index config, produces exactly one analytical artifact:

ordered input events + schema/transforms/index config
  -> deterministic segment builder
    -> immutable analytical segment

Concretely:

Kafka partition: profile-views-7
Offsets: 10000..19999
Schema version: 4
Index config: timestamp + ownerId inverted index
 
Produced segment: profile_views__7__10000__19999

Two servers can't disagree about what that segment contains. Either they loaded the same committed file, or they consumed the same offset range and converged on the same result. Determinism is what makes "just download it again" a safe sentence instead of a hopeful one.

Once that clicked, three layers fell out that I'd previously been mashing together:

Losing the third costs you capacity and latency. It should never cost you the first. Most of the confusion I had — including that week of misreading — came from treating those three as one thing because they're all "the database."

Rebuildable is not the same as cheap#

Worth doing the arithmetic before this sounds like magic.

Say a dead node held 4 TB of segments and its replacement pulls and loads at a sustained 400 MB/s. Ignoring decompression, indexing, contention, and metadata churn:

4 TB / 400 MB/s ~= 2.9 hours

Nearly three hours. Replication probably keeps you serving throughout, but you're running that whole time with less redundancy and less headroom than you designed for. If a second node goes during the window, you find out how much slack you actually had.

The architecture changes what failure feels like. It doesn't make it free.

A small model of the idea#

I wrote this to check my own understanding, so it's deliberately tiny. It models four things: events land in a durable log, bounded ranges become immutable segments, serving nodes hold local copies, and local state can be restored from deep storage — or regenerated from the log, as long as the offsets are still there.

import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.TreeMap;
 
public final class RebuildableAnalyticsDemo {
    record Event(long offset, String tenantId, String action) {}
    record Segment(String id, long startOffset, long endOffset, Map<String, Long> counts) {}
 
    static final class DurableLog {
        private final TreeMap<Long, Event> events = new TreeMap<>();
        private long nextOffset;
 
        long append(String tenantId, String action) {
            long offset = nextOffset++;
            events.put(offset, new Event(offset, tenantId, action));
            return offset;
        }
 
        List<Event> readInclusive(long start, long end) {
            var slice = events.subMap(start, true, end, true).values().stream().toList();
            if (slice.size() != end - start + 1) {
                throw new IllegalStateException("required offsets are no longer retained");
            }
            return slice;
        }
 
        void truncateBefore(long firstOffsetToKeep) {
            events.headMap(firstOffsetToKeep, false).clear();
        }
    }
 
    static final class DeepStore {
        private final Map<String, Segment> segments = new HashMap<>();
 
        void put(Segment segment) {
            segments.put(segment.id(), segment);
        }
 
        Segment get(String id) {
            Segment segment = segments.get(id);
            if (segment == null) throw new IllegalStateException("segment missing from deep store");
            return segment;
        }
 
        void delete(String id) {
            segments.remove(id);
        }
    }
 
    static final class ServingNode {
        private final Map<String, Segment> local = new HashMap<>();
 
        void load(Segment segment) {
            local.put(segment.id(), segment);
        }
 
        long count(String segmentId, String tenantId, String action) {
            Segment segment = local.get(segmentId);
            if (segment == null) throw new IllegalStateException("local segment missing");
            return segment.counts().getOrDefault(tenantId + ":" + action, 0L);
        }
 
        void loseLocalDisk() {
            local.clear();
        }
    }
 
    static Segment buildSegment(String id, List<Event> events) {
        Map<String, Long> counts = new HashMap<>();
        for (Event event : events) {
            counts.merge(event.tenantId() + ":" + event.action(), 1L, Long::sum);
        }
        return new Segment(id, events.getFirst().offset(), events.getLast().offset(), counts);
    }
 
    public static void main(String[] args) {
        DurableLog log = new DurableLog();
        DeepStore deepStore = new DeepStore();
        ServingNode node = new ServingNode();
 
        log.append("tenant-42", "PROFILE_VIEW");
        log.append("tenant-42", "PROFILE_VIEW");
        log.append("tenant-17", "PROFILE_VIEW");
        log.append("tenant-42", "PROFILE_VIEW");
 
        Segment segment = buildSegment("profile-views-0-3", log.readInclusive(0, 3));
        deepStore.put(segment);
        node.load(segment);
 
        System.out.println("Before failure: " + node.count(segment.id(), "tenant-42", "PROFILE_VIEW"));
 
        node.loseLocalDisk();
        node.load(deepStore.get(segment.id()));
        System.out.println("After local recovery: " + node.count(segment.id(), "tenant-42", "PROFILE_VIEW"));
 
        deepStore.delete(segment.id());
        Segment rebuilt = buildSegment(segment.id(), log.readInclusive(segment.startOffset(), segment.endOffset()));
        deepStore.put(rebuilt);
        node.load(rebuilt);
        System.out.println("After log replay: " + node.count(segment.id(), "tenant-42", "PROFILE_VIEW"));
 
        log.truncateBefore(4);
        deepStore.delete(segment.id());
 
        try {
            buildSegment(segment.id(), log.readInclusive(segment.startOffset(), segment.endOffset()));
        } catch (IllegalStateException exception) {
            System.out.println("Expected unrecoverable failure: " + exception.getMessage());
        }
    }
}

Output:

Before failure: 3
After local recovery: 3
After log replay: 3
Expected unrecoverable failure: required offsets are no longer retained

That last line is the one I care about. Every recovery path works right up until the inputs are gone, and then none of them do.

Real Pinot has partition assignment, consuming versus completed segments, commit consensus, replicas, checksums, controllers, ZooKeeper metadata, broker routing, query planning, partial results, retention, tiering, and workload isolation. None of that is here. The model only draws the ownership boundary, because that's the bit I'd been getting wrong.

Break it deliberately#

The fastest way I found to confirm any of this was to imagine tearing pieces out and asking what survives.

Start healthy: segments in deep storage, metadata fine, Kafka holding recent ranges, dashboards live.

Kill every query server and wipe their disks. During the reload you should expect fewer replicas, some query errors or partial results, higher broker latency, a lot of deep-store egress, and CPU and disk pressure while indexes rebuild. The controller notices the servers are gone, replacements get assignments, they download, they load, they come back. The product services never republish a thing. That's the design working.

Now delete the segment from deep storage too. If Kafka still retains the offset range, Pinot can re-ingest and rebuild it. If those offsets have expired and no archive holds the original events, that segment is gone. Not degraded. Gone.

What else you could have done#

Precomputation deserves more credit than it usually gets. If the product only ever shows daily views per profile, one counter per profile beats every architecture in this post.

It falls apart when the query space opens up. Add company, industry, seniority, country, connection degree, device, and arbitrary time ranges, and the combinations explode faster than you can materialise them. Pinot's answer is to keep the detail and the indexes, and let the combination be chosen at query time. That's the trade: storage and operational weight in exchange for not having to predict the questions.

What it costs to run#

The hybrid setup runs streaming and offline pipelines side by side. Two places where parsing, enrichment, and business rules can quietly drift apart, and a class of bug where the same day looks different depending on which side of the time boundary you land.

Immutability makes reads and replication easy and pushes the difficulty into corrections. Fixing one bad row means reading the segment, rebuilding it, regenerating indexes, uploading a replacement, updating metadata, and retiring the old version safely.

A real deployment is brokers, servers, controllers, ZooKeeper, Helix state, Kafka, deep storage, batch builders, minions, schemas, monitoring, and rebalancing automation. That's a platform, and it needs people who know it.

And recovery generates load. A replacement node has to pull its data from somewhere. In a correlated failure or a large rebalance, deep storage and the network become the bottleneck exactly when you least want another one.

If you're not LinkedIn#

Stay on the transactional database until something you can measure says otherwise.

Add indexes. Add a replica. Materialise the reports you actually run. Cache the handful of results that repeat. Push expensive summaries to a background job. Do not stand up Kafka, Pinot, ZooKeeper, deep storage, and batch pipelines because one dashboard takes 300ms instead of 100ms.

The signals that you've genuinely outgrown it: analytical queries eating capacity operational traffic needs, replica lag past what the product tolerates, users wanting data fresher than the warehouse can deliver, cache hit rates that stay bad no matter what you do, indexes hurting write throughput, dashboard latency missing its SLO repeatedly rather than occasionally.

A Pinot-shaped system earns its keep when you need high concurrency, fresh ingestion, multidimensional filtering, a large retained dataset, predictable interactive latency, and fault tolerance — all at once. Any three of those, and something simpler will do.

The trigger was never "we have big data."

It's that nothing simpler could hold freshness, latency, concurrency, and isolation at the same time.

What I think of the decision#

For the workload LinkedIn described, treating serving state as derived looks right to me.

Profile-view analytics, feed signals, dashboards, network-flow exploration — none of these author the underlying business fact. They present it. Once you accept that, optimising those nodes for query latency, ingestion speed, and replaceability follows naturally, and the cost of losing one drops to time and money rather than data.

The price is real: eventual consistency, two ingestion paths, segment rewrites, metadata you must not neglect, partial-result semantics, and a platform to keep alive.

What makes it work is that LinkedIn kept two questions apart that most teams answer with a single system:

1. Where is the original fact durably kept?
2. Where is that fact arranged for fast analytical access?

Pinot answers the second one well. It should only be asked to answer the first if you have deliberately designed the deep store, metadata, backups, and retention to carry that weight — and if you have, you should be able to say so out loud, in detail, without checking.

That's the sentence I'd been misreading. Not "servers are disposable." Serving state is derived, and something else had better own the original.

What I took away#

Derived data can still be critical. Pinot's contents may be reconstructible, and a Pinot outage will still break a feature your users care about. Reconstructible is not the same as unimportant.

Durability and query placement are separable. The machine answering the query has no need to hold the only durable copy. Most database designs conflate these, and once you've seen them apart it's hard to unsee.

Immutability relocates complexity. It buys you easy reads, caching, replication, and recovery. It charges you on updates, compaction, corrections, and deletes.

Your recovery story ends where retention ends. "We can replay Kafka" is a statement about your retention config, and it's worth knowing the number.

Build for workload shape, not headcount. LinkedIn needed this because fresh, multidimensional, high-concurrency analytics fit badly into transactional databases, key-value stores, warehouses, and fixed aggregates. That's a shape argument. It has nothing to do with how big the company is.

Sources#

1. Pinot: Realtime OLAP for 530 Million Users, LinkedIn and ACM SIGMOD, 2018.
2. Pinot Storage Model, Apache Pinot documentation.
3. Deep Store, Apache Pinot documentation.
4. Real-time analytics on network flow data with Apache Pinot, LinkedIn Engineering, 2022.
5. Pauseless Consumption, Apache Pinot documentation.
6. Enhancing OLAP Resilience at LinkedIn, LinkedIn engineers, 2026.