Why Netflix put its CDN inside internet providers

I understood why Netflix needed a CDN. I could not explain why it went through the trouble of putting physical servers inside internet providers.

A normal CDN already brings content closer to users. Netflix could place servers at major internet exchanges, peer directly with ISPs, and avoid much of the public internet. Why ship custom appliances into somebody else's data centre, then deal with rack space, power, routing, failed drives, replacements, monitoring, and years of coordination?

The answer I initially settled on was latency.

It turns out that latency is only part of it.

The cache was close, but not close enough#

Consider a simplified path for watching a Netflix episode:

Netflix CDN → ISP → ISP backbone → home router → television

The CDN server may be geographically close. It may even sit in the same city.

But it can still be outside the ISP's network.

That boundary matters because the video has to cross the connection between Netflix and the ISP, then travel through the ISP's internal network to reach the subscriber. If one million customers of the same ISP watch the same popular title, those bytes repeatedly enter through that boundary.

Netflix's alternative is more aggressive:

Netflix → pre-fill once → appliance inside ISP → subscribers

The company provides qualifying ISP partners with Open Connect Appliances, or OCAs. The ISP supplies rack space, power, connectivity, and BGP sessions. Netflix supplies the hardware, monitors it, manages the content, and replaces it when necessary. Netflix's Open Connect overview says these embedded deployments are intended to move a substantial amount of traffic away from peering or transport circuits.

The obvious benefit is a shorter path.

The more interesting benefit is that Netflix crosses the expensive part of that path once.

Suppose an encoded video file is 1 GB and 20,000 households connected to one ISP watch it during the evening. This is only an illustrative calculation.

The 20 TB of playback traffic still exists, of course. Most of it now stays inside the ISP's own network.

That is a very different traffic problem.

Open Connect schedules traffic before viewers create it#

This was the part I had missed.

Netflix does not wait for every regional cache to discover popularity through live misses. It proactively fills much of its content during configured windows, generally when demand is lower.

Netflix described this process in detail in Netflix and Fill. Once a title has gone through processing, encoding, quality checks, and packaging, the resulting files are placed in Amazon S3. Open Connect then distributes the required files to groups of appliances according to content region, expected popularity, and the number of copies the cluster should hold.

That article is from 2016, so I would not use it to claim that the current fill pipeline is identical. The underlying reason for proactive placement is still visible in Netflix's 2025 cache-miss analysis: Open Connect continues to pre-position files and measures when traffic could not be served from the best local site.

At first, I thought proactive caching was simply an unusually good cache-warming strategy.

It is closer to traffic scheduling.

Netflix knows that a newly released episode will exist before people press Play. It has historical viewing data. Demand is imperfect, but it is predictable enough at a regional level to make useful decisions about which files should be placed where. Netflix has also explained that regional aggregation makes prediction easier than forecasting each subscriber, although placement becomes granular because one episode can produce many files for different devices, bitrates, and encodings.

So Netflix can choose:

• which bytes cross the wider network;
• which route they take;
• where they stop;
• how many copies to keep;
• and, crucially, when the transfer happens.

A popular title can be written to appliances during quieter hours. During prime time, those appliances mostly read files and push them onto network sockets.

The 2016 explanation mentions another benefit: separating fill activity from peak playback reduces read/write contention on appliance disks. The same hardware spends the quiet window accepting content and the busy window serving it.

That is why Open Connect fits Netflix unusually well. Its workload has several properties working together:

• The objects are large.
• Playback repeatedly reads them.
• Many people in one region request the same content.
• Much of the content exists before its demand spike.
• A playable copy on an appliance can be recreated from an upstream source.
• If a prediction is wrong, Netflix can use another Open Connect location.

Change those properties and the design quickly becomes less attractive.

Putting a banking ledger inside ISP networks would be absurd. The state changes constantly, every write matters, and geographic caching would create a consistency problem instead of solving a bandwidth problem.

Netflix is moving replaceable copies of prepared media.

That distinction carries most of the architecture.

What actually happens after Play#

The public architecture has more moving parts than “connect to the nearest cache,” but the request path can still be reduced to one diagram.

The embedded appliance is deliberately narrow. Netflix's current appliance documentation describes OCAs as single-purpose, single-tenant systems optimized for reading from storage, writing to network sockets, serving HTTP, collecting BGP routing information, and reporting health. Current storage-appliance specifications list up to 120 TB of raw storage and roughly 200 Gbps of operational throughput, although Netflix also says the hardware continues to evolve.

The account logic lives elsewhere.

When a member presses Play, Netflix's playback services determine which files are required. Meanwhile, every OCA periodically reports information such as its health, available capacity, learned BGP routes, and current file inventory to Open Connect's control services.

A steering service uses that information to identify suitable appliances and provide the client with ranked URLs. The client then fetches the media directly from an OCA over HTTP.

“Closest” here has a network meaning.

Netflix's 2025 explanation says proximity ranking is based on IP ranges advertised by ISP partners through BGP. A site advertising the longest matching prefix for a client is considered the most proximal site. That is more useful than measuring physical distance between two points on a map.

Now follow one file.

1. A new episode is encoded into multiple representations.
2. Open Connect decides that two embedded sites in a region should carry some of those files.
3. During the fill window, the appliances obtain them from an upstream source.
4. An appliance reports that the files are present and that it is healthy.
5. A subscriber presses Play; playback services select the files needed by that device.
6. Steering maps the subscriber's IP range to an embedded ISP site that has those files.
7. The client receives that appliance near the top of its candidate list and downloads the video.

The appliance serves bytes. It does not become the owner of the title.

If it disappears, Netflix still has other copies.

That sounds safe, but the failure is more interesting than the diagram suggests.

A fallback arrow needs capacity#

I built a small model because I wanted to see one thing clearly: what happens to upstream traffic when an embedded site fails during the busy window?

The model has three files:

• a major launch that the placement algorithm predicts correctly;
• a consistently popular comedy;
• a surprise hit that the prediction misses.

Two ISP sites each have enough storage for the first two files. Traffic that cannot be served locally falls back to an appliance at an internet exchange. The numbers are invented.

import java.util.List;
import java.util.Set;
 
public final class OpenConnectFailureModel {
 
    record Video(String name, int sizeMb, int plays) {
        Video {
            if (name == null || name.isBlank()) {
                throw new IllegalArgumentException("Video name is required");
            }
            if (sizeMb <= 0 || plays < 0) {
                throw new IllegalArgumentException("Invalid video values");
            }
        }
    }
 
    static final class Site {
        private final Set<String> files;
        private final long peakBudgetMb;
        private long servedMb;
        private boolean healthy = true;
 
        Site(Set<String> files, long peakBudgetMb) {
            this.files = Set.copyOf(files);
            this.peakBudgetMb = peakBudgetMb;
        }
 
        boolean tryServe(Video video) {
            if (!healthy || !files.contains(video.name())) return false;
            if (servedMb + video.sizeMb() > peakBudgetMb) return false;
 
            servedMb += video.sizeMb();
            return true;
        }
 
        void fail() {
            healthy = false;
        }
 
        void reset() {
            servedMb = 0;
        }
    }
 
    record Result(long localMb, long fallbackMb, long droppedMb) {}
 
    static Result simulate(
            List<Video> videos,
            List<Site> localSites,
            long fallbackBudgetMb) {
 
        localSites.forEach(Site::reset);
 
        long local = 0;
        long fallback = 0;
        long dropped = 0;
        int nextSite = 0;
 
        for (Video video : videos) {
            for (int play = 0; play < video.plays(); play++) {
                boolean served = false;
 
                for (int i = 0; i < localSites.size(); i++) {
                    Site site = localSites.get(
                            (nextSite + i) % localSites.size());
 
                    if (site.tryServe(video)) {
                        local += video.sizeMb();
                        served = true;
                        break;
                    }
                }
 
                if (!served) {
                    if (fallback + video.sizeMb() <= fallbackBudgetMb) {
                        fallback += video.sizeMb();
                    } else {
                        dropped += video.sizeMb();
                    }
                }
 
                nextSite++;
            }
        }
 
        return new Result(local, fallback, dropped);
    }
 
    public static void main(String[] args) {
        List<Video> videos = List.of(
                new Video("launch", 4_000, 1_200),
                new Video("comedy", 2_500, 800),
                new Video("surprise", 1_500, 600)
        );
 
        Set<String> predictedFiles = Set.of("launch", "comedy");
        Site siteA = new Site(predictedFiles, 4_000_000);
        Site siteB = new Site(predictedFiles, 4_000_000);
 
        System.out.println(simulate(
                videos, List.of(siteA, siteB), 2_500_000));
 
        siteA.fail();
 
        System.out.println(simulate(
                videos, List.of(siteA, siteB), 2_500_000));
    }
}

The first run produces:

Result[localMb=6800000, fallbackMb=900000, droppedMb=0]

Most traffic stays inside the ISP. The surprise hit goes through the fallback path because it was never placed locally.

Then site A fails:

Result[localMb=4000000, fallbackMb=2500000, droppedMb=1200000]

Site B absorbs as much traffic as its artificial peak budget permits. The internet-exchange appliance then reaches its own limit. Some demand remains unserved.

The exact numbers are unimportant. The shape of the failure is the point.

Before the failure, local placement removed a large amount of upstream traffic. After the failure, those same requests arrived suddenly at the next delivery tier.

The backup existed. It was still insufficient.

Netflix's actual system uses appliance health and load when making routing decisions, and traffic can move to another embedded deployment or to Netflix appliances reached through settlement-free interconnection. Its 2025 engineering post also describes analysing whether lower-ranked sites would be overloaded under hypothetical failures.

The difficult part probably is not the server#

Netflix's appliance design is impressive. FreeBSD, NGINX, BIRD, dense storage, large network interfaces, and careful disk-to-socket performance are all interesting engineering topics.

I do not think they are the hardest part of Open Connect.

The harder part is coordinating a distributed serving layer that sits inside networks Netflix does not own.

Netflix controls the appliance and global delivery logic. The ISP controls the surrounding routers, links, power, rack space, route advertisements, and internal topology.

When playback quality drops, the underlying cause could be:

• a missing file;
• an inaccurate popularity forecast;
• an unhealthy appliance;
• an overloaded site;
• a routing change;
• insufficient fallback capacity;
• or a problem elsewhere in the ISP path.

From the viewer's perspective, several of these failures look similar. The video starts slowly, begins at a lower quality, or buffers.

Netflix's cache-miss work is revealing for this reason. The company does more than record whether a requested file existed somewhere in the CDN. It asks whether bytes came from the best local site and, when they did not, tries to classify why. A content miss requires a different fix from a health or capacity miss.

My reading is that this observability is part of the price Netflix pays for localization.

Once thousands of physical machines are spread across many networks, a global cache-hit ratio is too vague. It may tell you that efficiency fell. It does not tell you whether to change placement, replace hardware, add storage, shift traffic, or increase fallback capacity.

I am deliberately avoiding adaptive bitrate streaming here. It changes which representation the client requests and makes file placement more granular, but it does not change the main reason Netflix benefits from moving stable, popular bytes into ISP networks.

Live streaming is a more serious complication. Netflix's 2025 post mentions live streams and advertising as latency-sensitive workloads. A live segment cannot be quietly placed on an appliance the night before. Open Connect can still provide network locality, but the large scheduling advantage of on-demand content becomes weaker.

Most companies should stop much earlier#

A startup building a video product should probably use object storage and a commercial CDN.

That is not a lesser architecture. It is the sensible one.

Commercial CDNs already provide global points of presence, cache management, DDoS protection, routing, observability, and relationships with network providers. Rebuilding those capabilities would consume a team that should probably be working on the product.

As traffic grows, the company can pre-warm important releases, add origin shielding, improve cache keys, use multiple CDNs, and negotiate direct peering where it helps.

Placing company-owned hardware inside ISPs should come much later.

I would only take that idea seriously when the same large objects are repeatedly entering the same networks, transit or transport capacity has become a meaningful cost, demand is predictable enough to pre-position content, and the company can support hardware and routing across organisational boundaries.

Even then, appliances at internet exchanges may be sufficient. Netflix itself says embedded OCAs are not warranted for every ISP; traffic levels, data-centre constraints, and other factors affect the choice.

This design becomes excessive well before it becomes technically possible.

Large infrastructure should follow a recurring workload problem. It should not be adopted because the architecture is interesting.

What I had wrong#

I started with a latency explanation.

After following the file path, the more useful explanation became economic and operational.

Netflix has a workload where the same large, prepared objects are read many times by users clustered inside particular networks. Open Connect lets Netflix move those objects across wider network links during quieter periods, store them near the eventual readers, and keep the busiest delivery traffic inside the ISP.

The custom appliance makes that possible. The forecasting system decides what deserves the space. BGP helps identify which subscribers are near which sites. The control plane watches health, routes, capacity, and file inventory. Fallback sites protect playback when local serving fails.

And every one of those benefits creates another obligation.

Predictions can be wrong. Storage is finite. Sites fail. A healthy fallback can still saturate. Partner networks change. Debugging crosses company boundaries.

That is why I no longer see Open Connect as a particularly advanced cache.

It is a system for deciding where large volumes of predictable traffic should cross the internet, and for moving that traffic before millions of viewers ask for it.

For Netflix's on-demand workload, one copy transferred before prime time can replace thousands of long-distance transfers afterward.

That is the reason the CDN ended up inside the ISP.

Sources#

• Netflix Open Connect — embedded appliances, settlement-free interconnection, ISP responsibilities, nightly fills, and deployment criteria.
• Open Connect Appliances — current hardware characteristics, monitoring, FreeBSD, NGINX, BIRD, HTTP serving, and BGP collection.
• Netflix and Fill — the published 2016 architecture for title preparation, proactive caching, fill windows, and disk contention.
• How Data Science Helps Power Worldwide Delivery of Netflix Content — regional popularity prediction, finite appliance storage, and content allocation.
• Driving Content Delivery Efficiency Through Classifying Cache Misses — playback steering, BGP-based proximity, pre-positioning, and cache-miss classification.