Will real-time data processing replace batch processing?
At Confluent's user conference, Kafka co-creator Jay Kreps argued that stream processing would eventually supplant traditional methods of batch processing altogether.
Absolutely: Businesses operate in real-time and are looking to move their IT systems to real-time capabilities.
Eventually: Enterprises will adopt technology slowly, so batch processing will be around for several more years.
No way: Stream processing is a niche, and there will always be cases where batch processing is the only option.
Frontend Development / Networking

Netlify CEO on Why Netlify Edge Functions Was Built on Deno

Apr 26th, 2022 6:00am by
Featued image for: Netlify CEO on Why Netlify Edge Functions Was Built on Deno
Lead image via Pexels.

The web development platform Netlify recently added a new product to its workflow for those who work from the network’s edge. Netlify Edge Functions began development in 2019, according to Netlify’s CEO Matt Biilmann. In an interview with The New Stack, he described how looking to the future inspired the vision for the company’s latest product.

“As we’re building out our edge network and as we start seeing this category mature, with people building more complex projects in a variety of industries, I believed we’d see a new layer emerge where developers could easily write code that would run on the edge. In the beginning, we weren’t sure what that layer would look like or what it would do. We spent a lot of time investigating WebAssembly as the runtime mechanism but ultimately decided against it. In 2020, we moved our efforts from WASM to our own JavaScript-based edge runtime.”

“The standard JS runtimes like Node.js aren’t really built to be run in a totally multitenant environment or unique process isolation, so we had to start building our own.”

Matt Biilmann, Netlify CEO

A year ago, Netlify’s first version of Edge Functions (named Edge Handlers at the time) received a lukewarm response from test developers. The problems that other proprietary edge computing solutions face, like smooth integration, were also an issue.

This led to the decision to partner with Deno, an open source JavaScript and Typescript runtime. Making Edge Functions a collaborative effort gave developers the ability to use the JavaScript frameworks they are already familiar with to build and deploy edge functions.

An Open Standard

Deno (created by the same person who created Node.js, Ryan Dahl) is an open standard in addition to being open source. Netlify as a company, Biilmann says, started from being inspired by the internet itself and how it isn’t owned by anyone or any one business. Therefore, he says, it’s critical for their contributions to the web development community to be shareable and not confined to one proprietor — even if it’s them.

“Supporting open standards is essential to us. We wanted something built from open standards and not vendor-specific proprietor standards. The reason you keep running it on Netlify should be that the developer experience and productivity of your team are so high that you want to stay with us — not that you’re locked in,” Biilmann explains.

Putting Developer Experience at the Forefront

From the start, Edge Functions was built to cater to the developer experience. Biilmann says, “We want to take away all of the friction. It’s important for the whole ecosystem that an open runtime starts emerging, there hasn’t been an easy way for developers to write globally distributed software.”

He continues, “We want to give the Netlify package where everything just lives in one Github repository, you write the code once, open the pull request and we give you a unique URL where you can see everything run together. We give you observability and logging in one central platform. You can send those logs in to partners like DataDog, New Relic, and so on.”

However, there’s more to a good programming environment for developers than speed and accessibility. Edge Functions includes features that make it easier to diagnose and fix problems. “If you make a mistake and ship something that doesn’t work, you can circle back to any known state in milliseconds, which gives teams this confidence in shipping and affords companies the potential to execute hundreds of releases a day,” Biilmann adds.

“It’s not enough for us to be an infrastructure provider; our whole value proposition is truly about making developers work faster and work better. We had to spend a lot of time during this project on the core developer experience.”

Building Websites on the Edge

Network Edge is becoming more recognizable and preferable for some developers and consumers. “Enterprise customers want to write personalization at the edge or authentication from the edge,” says Biilmann.

Operating on the network edge can have a multitude of benefits, including speed and security. “We can distribute your site to servers all over the world,” Biilmann says. “Typically, if you were in Tokyo and accessing a site that lives on a server in New York, just the route trips to get the first content would take a second before anything happens on your browser. We can get that down to a few milliseconds.”

Like most projects that large companies work on, building Edge Functions was a tall order. Biilmann goes into detail about the effort it took for Edge Functions to come to life. “Creating these kinds of solutions is always demanding. We run a global network and serve a lot of traffic for our customers, and we have to take that into account when building anything for Netlify. There were a lot of challenges in making that system robust, efficient, and scaling it enough to deal with the traffic that our system inherently has to deal with.”

“How does it feel writing this? How do we make everything work together locally when you run it on your own machine in exactly the same way it runs on our globally distributed network? How do we stream logs into our system so we can show developers what’s happening in their edge functions? It’s been a heavy engineering lift.”

Though Edge Functions hasn’t been out for long, Biilmann says that Netlify is actively working on advancing it, as well as working on new projects. “Our team has a whole set of ideas of what more can we do to reduce friction for developers. We’re always working on the next pieces we can integrate to make the experience even more seamless. What I’m most excited about is seeing what the community will build with this new primitive.”

Group Created with Sketch.
TNS owner Insight Partners is an investor in: The New Stack, Deno.
THE NEW STACK UPDATE A newsletter digest of the week’s most important stories & analyses.