How a Single Engineer used Varnish to Build Tesla a Private CDN
At a recent Varnish Summit in San Francisco Vitaliy Kushnerov, principal IT infrastructure architect at Tesla Motors and Rajasekar Jegannathan, a Tesla Web platform architect talked about how they created a private Content Delivery Network (CDN).
In 2013, the year that the Tesla Model S was the best car ever tested by Consumer Reports, the IT team realized their current infrastructure was not scalable. They needed a CDN that provided a caching engine, a robust Geo-DNS, and service discovery all that performed consistently across the globe.
Don’t we all?
The Tesla website is heavy on video content and imaged documents, which both consume space and take time to download.
The company realized it was not using its resources optimally, said Kushnerov. Customers connect to the service center, and they wait. “So they’re sitting there thinking ‘What a beautiful car, but they can’t get their service process right.’”
A small team created to identify the problem and suggest solutions started by asking what was going wrong? How could they better use existing infrastructure and existing services?
Analysis revealed there were multiple places where the same content was transferred over and over again. “The question then became: ‘How can we pass the data once?’,” said Kushnerov.
A CDN was the obvious answer. But at that time, no reliable CDNs extended reliability into China. Global consistency was critical to Tesla. The team decided a private CDN controlled in-house and under the Tesla umbrella was the way to go.
Once this was finalized, Kushnerov and Jegannathan sat together in front of one monitor and built a proof of concept for the Tesla CDN in about 5 hours. So it turns out that the rumors of Tesla being run like a start-up are true.
The proof of concept in hand, Jegannathan went on to completely architect the project and take it to the level it is right now – globally active and used by thousands of people in the company.
The biggest challenge, Jegannathan said, was to deliver the same user experience across the globe, especially in China. He got help from other Tesla developers, who passed on information about traffic patterns for the data across the globe.
From this information, he created requirements for a CDN provider:
- The ability to use existing hardware and network resources, because as a start-up they had no money for additional resources.
- Persistent storage-based cache for files ranging from 650 MB to 1.7 gigs each.
- The ability to pick storage based on the application.
- Visibility of the data to the team.
- And the big one, the ability to control their network.
Enter Varnish Software, who not only checked off all these boxes but their engineers agreed to work with Jegannathan to make his proof of concept work.
Varnish offers an open source HTTP accelerator for large Web-based sites, and a recently released MSE (Massive Storage Engine) module provides a way to both cache large amounts of data, that, should the system crash, can be quickly reloaded.
“Put in front of application server; it is super simple. Therefore, it is also 200 – 1,000 times faster. So every time you move data from caching layer to application server, Varnish will supply the data in 30-40 microseconds as opposed to typical cache which is 10 – 20 milliseconds,” said Per Buer, founder and chief technology officer of Varnish Software, in an interview earlier that day.
In response to Jegannathan’s request, Varnish engineers coded changes to the MSE to accept re-directs. Now, when a request comes into the Tesla network, the process looks at a map and selects the closest location, then re-directs the request to that location. The Varnish software team worked with Jegannathan all throughout the proof of concept to code new functionality to make Varnish work for Tesla.
Best of all, according to Jegannathan, the VCL (Varnish Configuration Language) gave him control over the CDN. “I love VCL,” he enthused.
Now the Tesla CDN keeps only one copy of each video no matter how many people are trying to fetch it. So if six people want to see the same 7.5 gig video, it is only retrieved once. Jegannathan considered that a big win because there was no money for adding hardware.
Varnish uses one CPU, and even that is partitioned, with priority given to services, then video, then other applications that use images. Also, he’s found that the CDN does not use very much memory, which is also a win.
Also, the “pre-fetching” process constantly looks for any new software/videos available. For example, in Europe, there are five high-volume sites, and the system pre-populates popular caches for those high-volume regions.
Regional tiers can also be pre-warmed for critical sites, meaning that the most oft-used content is downloaded before a request is made. For example, China’s region can be pre-warmed which solves a lot of delivery issues. Local sites are not pre-warmed.
Four-Tier System
On the backend, Jegannathan created a process in VCL that monitors the hardware. If any piece goes down, the process simply redirects to a different server with no engineer engagement. With video streaming, engineers have visibility across the globe. If any problems arise, a technician can see that and fix issues.
With all this as a base, Jegannathan built a four-tier CDN system.
High availability was critical to delivering content to a customer, he said, so it is used for Tiers 1 and 2. Tiers 3 and 4 use a console for service delivery.
Traffic to the company’s website is currently only 2,000 – 10,000 downloads a day, but the size of the content is very big – up to 1.7 gigs per file. This information was critical in determining how the system was built. Different decisions would have been made if the traffic load were higher, J pointed out.
The future for Tesla’s CDN? “Cache in a box,” said Jegannathan. Need a new server due to increasing traffic? Ship one box, with the CDN, pre-loaded for a plug-and-play upgrade.
It’s amazing what one engineer with management support and a start-up environment can do.
