“If your goal is to make building applications as easy as possible, the effort can’t stop at the water’s edge when you’ve written the app, it has to include the challenge of running the application in a modern cloud environment. That’s quite difficult for the modern development team,” said Matt DeBergalis, Meteor development group co-founder.
In building Galaxy, the company dealt with a range of issues including managing large numbers of containers, providing high availability to customers, scaling and providing responsiveness to development teams.
It required a runtime in the cloud that manages connected clients.
“We had very specific custom requirements around runtime scheduling,” DeBergalis explained. “We used ECS Scheduler APIs to build a smart application container instance scheduler, and swapped out the default ECS scheduler for our custom scheduler. We also optimized Galaxy to solve for the ‘noisy neighbor’ problem of multi-tenant platforms where certain apps consumed a disproportionate amount of computing resources compared to peer apps. This required monitoring application resource utilization, analyzing trends, and then defining container configurations that met the needs of all our customers.”
The challenge of containers goes beyond running a single process inside a container, he said.
“Where it really gets interesting is when you want to manage a large number of them in a production setting. The dirty secret of containerization is that for production use at scale, it’s still very, very early. When you actually get down the brass tacks of what it takes to run thousands of containers in a reliable setting on cloud infrastructure, you run into all these challenges that surround the containers themselves. How do you reliably restart them in the face of system failure? How do you monitor such a large number at the same time in an automated way? How do you collect performance metrics and logging information on containers? That was the big stumbling block among our own developers,” he said.
“Running a container in production at scale — that was a big ask,” DeBergalis said.
Developers want to be building software; they don’t want to be building the devOps around that software so they can run it. “We found the advantage of container orchestration-as-a-service is that it’s a simpler and faster way to solve a lot of the challenges you run into. Some of them are just maddening,” DeBergalis said.
He says the integration points worked out with ECS helped the company get Galaxy to market faster.
Galaxy is focused on running Meteor apps at scale in the cloud.
“There are a few parts to that, but the biggest is you want to securely run inside of individual containers as many copies as necessary and give people an easy way to scale that number up or down, then you want to able to manage that network connection for each active user – each open browser tab or mobile device that’s using the app – down to a particular container,” DeBergalis said.
“So a lot of technology in Galaxy is about how do we select the best back-end container for each incoming user, what’s the right load balance when you have web sockets that represent a session for a user that could be an hour or even a day – it’s very different from a traditional HTTP app. And how do we keep track of all those clients?”
Meteor allows you to push updated code into the user’s device, which requires careful coordination between the server and the client, and Galaxy allows that to happen so that the user doesn’t notice an interruption, he said.
One early user of the framework has been Ikea.
“It’s different from the Java ecosystem or .NET ecosystem where you have those conventions, a standard way of doing things. So with Meteor, when you sit down to write that application on day one, you’re actually writing code rather than deciding which version of which view layer you should attach to which version of which web socket library and on and on and on. That what we mean by ‘full stack’: It addresses all parts of the application,” DeBergalis said.
Launched in 2012, Meteor has raised $31.2 million in three rounds from investors including Andreessen Horowitz and Matrix Partners.
Founders Matt DeBergalis, Geoff Schmidt and Nick Martin originally set out to create a travel recommendation site. As part of the startup incubator Y Combinator, they built the original version of Meteor as its basis and found the other teams struggling to also build that basis for their apps. Then they realized their framework was more valuable than the travel app.
The company released version 1.0 of its eponymous platform in late 2014 and in version 1.1 added support for Windows and MongoDB 3.0. Meteor supports OS X, Windows, and Linux. Version 1.2 made ECMAScript 2015 its official language and added support for the Angular and React view engines alongside traditional Blaze templates, as well as additional support for mobile application development. In December, it announced a 1.3 release for the Angular package.
“Meteor has its problems, but for many applications – even in Meteor’s current state – there is nothing to compare to it. You don’t need to worry about HTTP calls, network or CRUD abstractions, you don’t need to worry about caching data or manually setting intervals. Those are all taken care of for me!” he wrote.
Sacha Grief, co-author of “Discover Meteor” and involved with other Meteor projects, laid out his views of Meteor’s problems, including that although it makes getting started easy, developers soon hit a wall of complexity. He believes the company’s all-in-one approach has been a liability.
He’s urging Meteor to embrace React, citing Meteor’s strength as a framework for both the server and client side.