Mitchell Hashimoto: Containers or No Containers? That is One Question for 2015

Alex: Hi, I’m Alex Williams of The New Stack, here with Mitchell Hashimoto of Hashicorp.

This matrix of complexity that you’re talking about is something that a lot of people are talking about these days. Docker talks about it; they talked about it in their earlier days. Have you started to see any patterns on how these problems are being solved by different parties? I mean, Docker has their own approach. How does your approach differ, for example, from Docker or from others?

Mitchell: The approach we have seen over and over — we saw VMware do it, and I think that’s very similar to how Docker is doing it — is that you get a new technology breakthrough.

Mitchell Hashimoto, Hashicorp: The New Stack Interview, Part Two

Let’s use VMware as an example. VMware’s breakthrough is virtualization and they showed how virtualization is a solution to all these different problems. As a result, everyone wants to switch everything to virtualization and tie all their workflows and all their processes to virtualization. But then there’s another breakthrough, or there are some things that don’t fit into that box and don’t work well with virtualization. It’s one of these things where it almost always works, but then falls short in some cases. Then when something new happens, you’re stuck on that old island, so to speak.

One trend or pattern we noticed was that if you take all the solutions – if you take Docker’s marketing, or VMware’s marketing, or a lot of these other vendors’ marketing – and you just replace all of their solutions with something else, then they’re still completely valid. Replace containers with VM – they’re solving the same problem from physical to virtual. Taking a look at that, we said, “hmmm, why is that?” We had a theory but we wanted to test it, so we started asking around.

We concluded that the end goals of everything are the same, and the problems that people have at a high level are still the same. You need to know what to start in what order; you need to know what’s running; you need to know how to start those things; you need to know what’s in staging and what’s in production; how to build a container or a VM; you need to know how to build an artifact.

These are very common problems that are identical, so we’re saying that these end goals stay the same, and they’re technology-agnostic – they are not dependent on a technology. So, let’s build for the end goal. Let’s build for the workflow that they want: how to go from dev to build to deploy, and outwards. Let’s design for that, and make it so that all the technology floats underneath, and we float on top of that, and we could replace it in-and-out. So that’s how we’ve been designing software.

Alex: …according to those five principles that you’ve mentioned [dev, build, artifact registry, deploy, and monitor & maintain]?

Mitchell: We’ve broken it down into those five.

Alex: So that’s the foundation for the way you are developing your own product and services?

Mitchell: We don’t think that fundamentally it’s exactly five steps, but they’re where we see vendors split off, and in order for integrations to make the most sense, that’s where we split things off as well.

Alex: Looking back, for instance, when I was talking with you at DockerCon, you were talking about Docker Machine. What answers does it solve? You’ve mentioned it doesn’t solve all of them. How would you assess it?

Mitchell: Well, Docker Machine solved a huge problem for Docker, which is, “How do I get a machine to run my container?” That’s an obvious question and it demanded an obvious answer. But the theory, the principle, that we’re working off of is that the number of servers that you yourself are running is actually going down.

Luke (Kanies) from Puppet told me one time, and I agree with this:

There’s no future where there’s less servers.

Flipside, there’s not less servers, but I believe there is a future where you yourself are running less servers because service providers are picking up the functionality for you.

Going back five years people would say things like, “We could never run our database on a service provider because that’s our crown jewels. We own our users’ data; we can’t let that out.” But now you’re seeing a lot of startups who use database providers. There’s RDS, which you perhaps own. There’s Heroku Postgres. In addition there are things that have been service providers a lot longer: there’s DNS, there’s CDN, there’s all these different services that you used to run yourself. You used to run those servers and had to manage those servers, and now you don’t.

We believe that as time goes on, you yourself are going to run less and less servers. Your apps are going to move to service providers. All of the things that they need are going to move to a service provider. Docker Machine solved the problem of “build me a server,” but in a future where there’s less servers, you need something that instead needs to say, “start me a service provider,” and, “configure me a service provider.”

Alex: So that raises some questions about the servers themselves and how they’re managed. If they’re not managed by the owner, they’re managed by the service provider. What are some of the significant issues that come with that, that you’re starting to see from customers?

Mitchell: If you’re a consumer of service providers, the new issues that are cropping up are in terms of security and access control. When everything was in your data center and you ran everything, you could carefully control who gets access to what and when and how, and record the access. Now, one thing I’m hearing is: in a sea of fifteen service providers, your security is only as strong as your weakest access token.

AWS has a fantastic access control system relative to other service providers. Your weak point is the one provider that doesn’t support key rotation, that only gives you one root access key that has access to everything. That’s where things are going to get tricky for you, in terms of running a big organization and orchestrating change safely, with this one root key floating around that is just waiting for someone to mess up. That’s a big problem that we’ve been looking at; I would say that’s the major problem.

Alex: There’s a company called JumpCloud; they have taken a few pivots, but originally they were trying to find ways to protect the key itself, putting that key in a special place so it’s hard to get to.

Mitchell: I actually think that’s the best you can do because there’s no way we could force these service providers to have the best possible key policies. There’s always going to be that weak service provider out there. The best you can do is actually hide the security risk and control that security risk in a controlled environment.

For example, our approach is: those keys and those secrets should be hidden within Atlas, and the access control should sit in the Atlas layer. The end user and developer operator never see that access key because Atlas is the one using it. Atlas is watching to see what they’re trying to do with the key and doing ACLs at that level. It’s not ideal; you would rather just have a stronger key. But like I said, you can’t force everyone to have perfect situations, so that’s the best you can do.

Alex: So what are you trying to do with your Atlas service to mitigate that issue?

Mitchell: What we’re trying to say – and we’re getting a lot of positive feedback on this – is:

No more dashboards; no more API keys from service providers.

Don’t give anybody access to the Heroku dashboard; don’t give anyone access to the AWS dashboard; don’t give anyone an AWS key; don’t give anyone a Heroku key.” You just give them the Atlas access token. You put all the other secrets into Atlas. The ACL layer is done at Atlas, and it hides everything for you, and Atlas records detailed audit logs.

We’re going to be announcing a system sometime next year for how we’re storing these secrets securely and, actually, a way for you to keep the secrets and never give them to Atlas, and we just access it from that, so you could know when we access it. I think that’s the right approach.

It’s also the right approach because it abstracts away these implementation details. If your app switches from AWS to Google Compute, then the OS structure changes and the key is changed. But the developer actually never knows because the developer is viewing this meta key that is accessing other things.

Alex: You’ve had an interest in security over the years. You were researching it at the University of Washington; I don’t know if it was just a result of the projects you were assigned. What philosophy do you have about security now? For example, we’re starting to see some weaknesses in the Docker environment.

Mitchell: I’m not a security researcher. I have to consult other people for security advice. That’s definitely not something I’d try to invent on my own.

Alex: But you have an interest in it?

Mitchell: I have an interest in it because I think you need to be cognizant of it. In a cloud world, everything is at risk, so to speak. You need to be careful about how you’re doing things and my approach to security is: do what everyone else is doing in terms of the research. Don’t invent your own security. A common word is, “don’t invent your own crypto.” Audit everything; record everything.

Atlas’ database, for example, doesn’t allow deletes, ever. We only allow creates and reads, so the data is only ever-increasing. We don’t lose the previous data, and we just archive it away because it’s useful for auditing and history.

The key is to consult people who do really know what they’re doing. I don’t actually know the Docker security risks, but I’m sure they’ll do the same thing.

Alex: So you’re not that familiar with the Docker security issues right now?

Mitchell: Docker’s always said that Docker’s not a total security container; it’s an isolation mechanism, not a secure isolation mechanism. With my background in LXC, I just never looked at it as that.

Alex: Looking out at 2015, what are some of the issues that will be more complex in this distributed infrastructure world for customers – what are some of the top ones you see?

Mitchell: Number one is service proliferation, where your data center just becomes more and more services. Number two is, inherently becoming multi-data-center and highly-distributed at a much earlier stage. With things like Docker, where you can run things in much smaller units, it becomes a lot easier to start running a lot more services. As a result, we have a management problem, an orchestration problem, and distributed system problems in there.

I think number three is, “How do we migrate to these systems?” The big companies are just now completing their transition to virtualization. The forward-thinking ones have been done, but the old ones are just kind of finishing up now. Now the question is, “Do we move to containers, and how do we move to containers, and when do we move to containers, and where do we move to containers?”

I think 2015’s going to be telling for those three things.

Alex: That’s moving from these vertical infrastructures to these more horizontal ones. That’s still a huge problem that these customers are facing. How are you seeing customers try to resolve those issues now, and do you expect that approach to continue?

Mitchell: The value of going horizontal – and I think that’s just closely tied with the idea of a “cloud,” so I’m going to use those interchangeably right now – the value there is really well-shown now in terms of cost savings and agility. The value prospect’s gone, so now I think there’s enough motivation to start converting those things.

The difficulty is that it does require converting, so you have to write your applications a certain way – you have to access data a certain way. Some people are going to do that for a lot of applications, but I think there is always going to be that one service, or the handful of services, that no one wants to touch again, so you’re just going to put it on a really beastly cloud machine. And that’s alright.

Alex: How do you approach building open source communities today, and how has that changed from your perspectives on it five years ago?

Mitchell: It’s your own little sub-government or sub-community within a community. Here’s the population of the world, and then there’s your little population. So you have a lot of societal problems and leadership problems and politics. The technology is actually the easiest part because it’s so logical. “Yes, it works,” or “No, it doesn’t work.” But the idea of “Do we add a feature?” or “Did this person respond nicely to someone?” All those things are human dynamics, so I view open source at a more philosophical level of, “How do people interact with each other?”

My personal stance on that stems from my two years as an Apple store employee. They train you on how to work with people, and I blogged about this. They have this thing, it’s an acronym, “APPLE,” and I just follow that acronym to the end. It’s really all about empathy: approaching someone really nicely, understanding that if they’re angry that 99 percent of the time a person is angry for a reason. They’re not angry because they’re just rude people. So, I’m trying to understand why they’re angry. You’ve just got to be pretty patient in figuring that out. But at the end of the day, it’s people talking to people making computers work. But the people are doing it, so you gotta think about people.

Alex: Do you see the open source community as continuing that ethos? Is that alive and well? Open source management is starting to change – we’re starting to see different forms of it. Is that having an effect?

Mitchell: I’m interested to see how open source management evolves. For example, the “benevolent dictator” model makes a lot of sense when it’s just your own thing that you put online and there’s no real risk associated with the project. But now, especially in the past few years, we’re starting to see a lot more open source companies — companies where their core products are open source. That gets a lot riskier because, is there a chance that the BDFL is just going to close it up? Is there a chance that you’re going to need a fork? What does the fork mean for users? I think that’s the driving change that’s causing these questions. I’m just interested to see how that plays out. I’m not sure yet.

Alex: We’re starting to see that a lot, how open source communities are managed once they get past that benevolent dictator stage. What about your own projects? You are the BD over Vagrant; are you also the BD over Consul?

Mitchell: So far, our company still follows that open source model where ultimately we’re the gatekeepers, but at the same time we have a history of being very benevolent, I guess. Where this becomes an issue is, for example, historically, I think Sun was viewed as fairly benevolent but, as soon as Oracle bought them, things got a little sketchier.

I think the community’s really happy with the way our projects are run right now, and it’s the right model for right now because it gives us the highest ability of agility. That’s why I’m so interested to see how this plays out, because over time that question’s going to come up again and again, and the answer might change over time, probably will.

Alex: For the communities that you’re managing, what are some of the things you’re paying attention to, to think about how you might adapt?

Mitchell: Well right now, every one seems to be really happy. I don’t see any major issues right now in terms of our communities. I’d say we’re lucky. Our products are used by very large companies and they don’t seem concerned in the least about the way we run our projects.

I’m paying close attention to some other projects. Rocket came out and right away someone asked for a governance board. I’m kind of interested to see how they do that. Does it provide value? Does it provide real power to the users? Is it positive or negative? I’m watching these sort of things, just anticipating that sort of thing coming to us.

Alex: That comes back to what you were saying about how open source is changing a lot. When you were first starting in technology development and programming, the open source communities were about learning — they still are, in many ways. But now there’s a high degree of competition in open source communities.

Mitchell: I think that’s a major part of it. It’s inherently riskier for a project to be tied to a commercial entity. We’ve avoided it by — very recently, just last week —making it very clear what our business model is and what we’re pursuing to make money. Whereas, it’s a lot scarier if the open source project is the maker of the money. It’s just uncertainty there.

Alex: So the Docker example, for instance…

Mitchell: Maybe. It’s not clear what Docker’s really doing yet, so I don’t know.

Alex: There are other examples, too, like OpenStack.

Mitchell: OpenStack, yeah. Chef, for example. Chef has been reversing course and they’ve been making a lot more open lately, which is cool. I don’t know the back story of that, but I’m sure there’s some interesting story there.

Alex: There is a lot we could talk about regarding open source management. We just did a podcast with Donnie Berkholz and Michael Coté on the topic and I think we’ll do it again so we’d like to have you participate in that.

Mitchell, thank you very much for taking some time to talk with us. I look forward to seeing what you do and the direction of your own open source projects.

Mitchell: Great! Thank you, thanks for the time. Nice talking to you.