Facebook and the Myth of an Individual Coding Culture
In a recent post on engineering culture, Facebook software engineer Pierre Raynaud-Richard shared insights into how the social media giant’s dev culture fosters collaboration and encourages pride over the code they create.
Crucial to Raynaud-Richard’s treatise is his debunking of the ‘individual code ownership’ mantra. “Many software development companies believe in and practice “individual code ownership.” This may not sound like such a fundamental principle, but it actually goes a long way toward defining how a software organization works,” writes Raynaud-Richard.
While on the face of it, the approach seems to encourage internal experts proudly creating better written, less buggy code, the actual impact for many companies is that it stifles innovation and wilts developers skills growth. Raynaud-Richard says the approach discourages company innovation, as developers end up favoring the status quo: over time, they defend the code base they have built and have an implicit bias towards it, unable to adapt to new opportunities or challenges that require a radical change in direction. For individual developers, the approach stunts their skills growth by shoe-horning their skill-base and narrowing their ability to identify high impact opportunities.
“Individual code-ownership provides some attractive benefits at first glance: better designed, maintained, and supported code, and longer life-span for components and services. Unfortunately, it does it at the cost of introducing rigid definition of roles, which can limit innovation and company and individual growth,” concludes Raynaud-Richard.
In today’s full stack development environments, software teams know that no one developer can know everything. But many are turning to more creative means to encourage a flexible engineering culture than the increased specialization avenue that the individual code ownership model cultivates.
The Startup Dev Culture
Database-as-a-Service startup Orchestrate has a distributed team of 10 engineers working across five time zones, according to CTO and cofounder Ian Plosker.
“We don’t generally have code ownership,” says Plosker. “That being said, area experts do develop naturally, usually starting with the person who developed or reviewed the code.” Plosker says that from time to time, particular engineers may end up specializing in particular technologies or methods, and they will be drawn into projects more often when that particular technology is being used.
“One person who owns the code is dangerous, because the whole team can then get by with not knowing how things work. Luckily, in a smaller organization, it is harder to get away with that. It is really important that knowledge diffuses around the organization.”
One of the key processes used at Orchestrate to avoid individual code ownership are code reviews: “Code reviews are one of the most crucial places for knowledge transfer, it guarantees at least a second person knows the code, has evaluated that the test cases are complete, and knows how the code base works.”
Orchestrate provides a single API and user interface for a collection of multiple databases. While starting with key-values, the product now enables a number of search formats, including full text, time-ordered and graph searches. The latest offering has been a new geospatial search feature, which avoided the individual code ownership approach as it was developed:
“We use GitHub quite a bit, so you will have a developer working on a feature branch, like when we added the geospatial search features. So the engineer will be working in a feature branch, they open up a pull request to merge that in. Hopefully, they add all the unit tests and integration tests, and that’s when the deep technical evaluation happens.
“If the feature is important we get at least two pairs of eyes on things and GitHub lets you comment on particular lines like saying ‘hey, why did you do it like this?’
“So conversation happens there, there is a lot of knowledge embedded in the pull requests.
“Once it is fully reviewed, then we run it and test it.
“We have a couple of steps for testing: internally, we have a tool called Orchestrate Runner that can build a whole Orchestrate stack on your machine (a Java based HTTP server, the web front end, and all of those things are spun up by our tool internally). That’s the first round of testing.
“Then every commit on GitHub is run by Travis CI. That works pretty well for us, and that will run all of our unit tests and integration tests. We even spin up a mini Orchestrate stack within travis CI.
“We also use Hipchat, we even have created a bot in our Hipchat that can tell us a lot about our code deployment. After each successful build master, we get those uploaded into S3 at the end of the Travis build, and based on the build number, we can ask our chatbot to deploy that code! Like, ‘Hey prodbot, deploy build number 46 to USC staging environment.’ Plosker also points to Mumble, a chat client that allows any of the developer team to pose questions to each other or start informal chats on coding patterns and design choices.
To foster collaboration and group discussion around new feature design, Orchestrate encourages any team member to create a proposal outlining the idea:
“Before we start any big feature, it tends to happen that someone becomes a champion for that feature. So we tend to like writing a proposal for the feature: here’s how it would work, here’s why we should do it, the challenges, the opportunities… these tend to be Google Docs and the entire team will work on this. It happens really organic.
“For example, right now we are working on search faceting, so there is a document that everyone is commenting on. At this point already, we have quite a bit of knowledge circulating.”
The Scalable Business
Online menu and food delivery service Just Eat — which now operates in 13 countries and announced a 58% revenue growth in the first half of 2014 — has an engineering team of about 60 developers across its UK offices alone. The company is seeing “pretty substantial, rapid growth” requiring developers to work together at two office locations and remotely, according to Technical Lead, Anton Jefcoate. “The rate of growth has driven us into some pretty good practices. The foundations were there already, but you can’t get away from the engineering culture you promote when you are scaling up your development team.
“We’ve divided our dev teams into a few groups to tackle different areas: consumer applications, business apps, devs who work on APIs.. the API team is the core of the entire platform. So the API teams are then further split down and they take control of particular parts of the platform.
“Any new project will get a couple of people together from the API driven and native applications team: that’s a minimum of two, maximum of 10 people on any project. It depends on the size of the feature obviously, and how far through the platform the feature will spread.”
Jefcoate thinks Raynaud-Richard is right to call out the shortcomings of an individual code ownership cultural model in software companies.
“The drawbacks are things that run true for us,” says Jefcoate. “I think agile is one of the things that broke that culture. I think before agile, devs used to be isolated, but agile encourages the team over the individual. When you are looking at tracking velocity or the greater good of the team to produce, everyone shares the work out evenly. We very much have a culture of everyone chips in. I believe it’s linked to the working practices, and the type of people that use agile flows: here’s some work to do, everyone gets on deck, and that doesn’t lend itself to an individual code ownership scenario.”
Just Eat replicates a number of cultural aspects that are becoming more familiar amongst cloud-based software startups. Like others in their generation (Just-Eat started in 2001), they have a culture of creating open source projects, and often employ a recruitment process that requests applicants to dogfood the Just-Eat APIs to demonstrate what can be made or to point out opportunities that could improve Just Eat’s code base.
Jefcoate sees the open source approach as a different way to cultivate the pride-in-one’s-code advantage that may come from individual code ownership:
“You only have to look down our stack and see all of the open source tools that we use to solve our problems every day, so as an overall developer community, we really do benefit from the greater good. Open source lets problems get solved by many people; there are very few places you can go to where you have free products that have that amount of tried and tested history to them. Open source works, and I guess for us, to contribute back to that makes a bit of a statement about what we are about as a company. The open source stuff we put out (which is relatively in its infancy), are things that are generic that help solve problems to the developer community. We try and get the benefits of individual code ownership (things like cultivating pride and encouraging the personal touch and personal investment in coding projects), by making sure that developers get to help test new techniques on our platform, that our developers solve problems for the entire Just Eat engineering group. If that happens, then it will probably solve stuff outside of our company, so that’s how our open source projects start off. Its usually about scaling and resiliency. We try to get it adopted internally first and then we see if it is something the community can benefit from.”
To further embed a collaborative culture rather than an individual code ownership model, Just Eat holds quarterly internal hackathons to bring developers together in new team configurations: “With group code ownership, you do miss some of the benefits like how code may be better because developers are more engaged or become domain experts. Internal hackathons are away to encourage our developers to work on personal pet projects.”
Just Eat uses JIRA to track work being done and to coordinate pull requests, and then every pull request gets peer reviewed: “A team is semi-independent, but everything is driven by a global product requirement. Once you get past the global requirement, the teams tend to self-manage to get it done. So a pull request opened up is typically within a team conversation. Then peer reviews are a fantastic process because it is learning as well as verification. We have a lot of automation: we really on our CI for yes/no could this be deployed.”
Jefcoate also sees the company’s use of pair programming (the method of having two programmers work together, line for line, on a coding project) as demonstrating the antithesis of individual coding culture that the company values: “We use pair programming for onboarding new developers and as a general development practice. If I am working on something particularly new, I will pair up with someone and make sure someone else knows about it. Those are done remotely, pairing is one of the best tools for getting code quality standardization.”
The Enterprise Dev Culture
Jakub Nesetril is Founder and CEO at Apiary, an API design and documentation coding suite of tools. This week, Apiary released Apiary for Enterprise to help larger companies steer away from individual code ownership when creating API products and services. Increasingly, Nesetril is seeing enterprises dealing with multiple APIs that have each been written in different ways and that now need to have common coding structures and nomenclatures, worked on by multiple team members rather than individual devs assigned to a particular business unit.
Across the thousands of Apiary users, Nesetril typically sees individual hackers or small teams use their open source tools where each developer may have an almost individual code ownership approach (mostly because they are the only coder). As teams grow they start using the Apiary commercial product, and by the time they reach 20 team members, they move on to Apiary for Teams. It is at this stage that the organizational infrastructure tends to bring in a Product Manager to ensure developer coding is uniform across various API projects.
Amongst enterprise customers — once teams reach developer numbers of 40-60 or more — an API Architect tends to get hired. “One of the things they are struggling with is a consistent design for their APIs, and that’s why we built Apiary for Enterprise,” Nesetril explains. At the heart of the product is a style guide tool that helps companies ensure developers conform to organizational conventions such as nomenclature and data formats.
Nesetril is seeing enterprise customers abandon old notions of individual code ownership, especially since many have inherited a legacy of disparate APIs through merger and acquisitions or because individual use cases drove initial API development projects within disconnected business units.
“Historically, there was a strong push for code ownership as that created accountability, but it also created difficulties for collaboration. We see a lot more shared resources amongst developer teams in the enterprises, where individuals may rotate in and out of various parts of projects,” says Nesetril.
“The trick with collaborative approaches is that they need a single version of truth. Not everyone can be experts on everything inside the company, but it is great when they can work on something internally. The biggest trick with a single version of truth culture is arriving at that place without limiting collaboration.” Nesetril believes that Apiary for Enterprise is ideal for enterprises trying to manage multiple APIs without creating a culture that encourages developers to revert to becoming individual code experts: “It gives them this piece of mind that when they are working on an API, it fits into a proactive level of advanced checking and notifications, then it allows you to be more experimental and ambitious if you are not bogged down by fears of are you doing it wrong with, for example, date formats. People can start making meaningful contributions early on.”
Individual Code Ownership Dies A (Semi-Public) Death
What Raynaud-Richard’s blog post may be actually doing is marking the time and death of individual code ownership. Already, businesses at all levels — from startups to enterprise — are embracing more collaborative development approaches. Google tools and Hipchat are commonly used across developer teams at all levels, and new developer products are emerging that help entrench a more collaborative ethos. In today’s dev environment, a middle road between specialization and general knowledge needs to be pursued where the greatest coding skill is asynchronous consensus building: the ability to cooperate and to facilitate knowledge transfer across a distributed team.
Feature image via Flickr Creative Commons.
