Workflows for the New Developer Experience
Datawire Technical Consultant Daniel Bryant kicked off his Cloud Native London talk saying that, as developers, “A lot of our jobs revolve around taking a good idea and turning it into value for our customers.”
For a lot of us, developers are our customers. What does a good developer experience (DX) actually entail? He says it’s about filling in the space between when we load ideas into shipping containers and when it all falls over into production. His talks focused on the questions to ask in order to identify patterns and workflows to improve that experience.
“Developer experience is primarily about minimizing friction from idea to code to delivering observable business value.” — Daniel Bryant
This is why he says how you construct your platform has an essential impact on your developer’s experience — and, really, if you should be building a platform at all. For him, it all comes down to designing the developer workflow.
The Disconnected Modern Developer Experience
In order to discover good DX, we need to understand the modern developer. Nowadays, a developer is involved with the entire lifecycle from design to develop to test to deploy to operate to support. Autonomous teams have led to speed but it can’t be for everyone. E-commerce? Sure, you want to be the fastest to market. But Bryant says slow down if you’re into nuclear reactors.
He says there’s no doubt that there’s a benefit to breaking systems apart via microservices but he warns this leads to fragmentation of the platform.
“Teams became autonomous and went faster around the loop and delivering value but often each team is developing separately. You can choose tools for the problem at hand but then you have to reformat libraries,” he explained.
“High productivity — and fun — comes from intentionally designing experiences of local deployment, packaging apps, continuous integration and delivery. Decide the workflow you want first and then choose your tools.” — Daniel Bryant
Bryant says this has different teams within the same organization going at different speeds, which makes it all the more essential that one org shares a common platform — for prototyping to mission-critical action to the production phase — instead of reinventing the wheel all the time.
In the new developer-centric cloud-based world of work, we need to understand how we self-service what we are building and how we observe our software.
Bryant described this “new normal” :
- Teams are responsible for aspiring to end-to-end delivery.
- Many components and interactions means code changes that aren’t deployed become stale very quickly because microservices and API consumption will have dramatically changed three months later.
- Developers need to understand business goals more. “We need to open up more room for KPIs [key performance indicators] and end-user empathy,” he said.
- Environmental parity including data is essential. Development must be as close to production as possible or even developing and testing in production. “Things like Kubernetes has allowed us to get even more loosely coupled and even more complex systems — so I can almost guarantee that what I test works in production,” Bryant explained.
- The inner development loop ideally needs to run with the real environment with multi-developer clusters and staging-like clusters, if not clusters directly in production. “You have to be careful not to trip over yourself in shared data,” he warned.
- Each developer needs to be able to run the full software development lifecycle, including self-service, without handoffs, deployments, halts, and even YAML.
- “Staging” now delivers a low return on investment. Instead, favor contract tests and observability in production.
Bryant said all of above are reasons the platform is the main driving factor of internal and external developer experience, as it guides all this autonomy like how you use Kubernetes and how you use serverless.
I'm convinced the majority of people managing infrastructure just want a PaaS. The only requirement: it has to be built by them.
— Kelsey Hightower (@kelseyhightower) April 11, 2017
In the end, there’s a good chance you are going to have to build a platform. Bryant says be wary:
“If you are going to invest in Kubernetes and a toolchain around Kubernetes, make sure you realize that’s what you’re doing because your business isn’t probably around building a platform, and the effects of you building a platform are on your whole organization. Try looking at serverless. Building a platform is not for the feint of heart.”
Building Boundaries around Developer Workflow
It’s clear developer experience nowadays is one of independence, but even the most independent teams have to put up a few guide rails.
“Netflix has talked about so many different platform components in their stack that, even with their freedom and [emphasis on developer] responsibility, they had to combine platforms with a paved road,” Bryant said.
This Netflix Paved Road is the company’s formalizing of a set of expectations and frameworks to follow. Teammates can still veer off that road, but then they have to be ready to be on call. The main reason for this was, if you want a well-integrated stack, you can’t have everyone going off in diverging directions.
Netflix paved this consistently autonomous path around:
- Identifying common problems across teams
- Empowering centralized teams to innovate
- Educating developers to avoid information overload
In Bryant’s experience, there are certain questions you want to ask to understand your developer needs.
Develop and Test Services Locally, Within a Cluster, or BVoth?
“A lot of engineers just want to write code locally, like test-driven development with sensible mocks and stubs, which encourages you to not build a heavily coupled system,” Bryant said, sharing that one of his teams started using service virtualization and stubbing to run more locally. They also tried giving each engineer access to a unique Kubernetes cluster.
“A lot of our journey as software engineers is psychology. Tell me about your history. Tell me about your past. Tell me where you want to go.” — Daniel Bryant
Bryant recommended less expensive ways to get feedback quickly in DevOps and lean software development environments:
- Canary testing — rolling out a new service to a small fraction of your traffic, including operations testing, AB testing, and feature testing.
- Shadow traffic — shadowing real live traffic one way into a test service of the service. You are not returning data but comparing results and looking for errors on the new service.
Again, embracing new ways of development all comes down to what you are trying to build.
“Some teams can be nervous about testing in production — this is good for startups, not for banking apps. With startups and SMEs want you to go as fast as you want but you are responsible for it,” Bryant said.
Developer experience isn’t about following certain rules, but rather deciding what suits your specific developer audience. Bryant said these factors include:
- Do you want to implement guide rails for your dev teams?
- Do you want to constrain what people are going to do, outlining a suggested way of developing and deploying?
- How do you want to enable knowledge sharing?
- Where are your developers in starting their journey with your tooling?
Bryant created the following diagram of some logical pathways of the type of experience your developers may come to expect. For example, in mission-critical updates, you want to prioritize pre-production testing, but in CI/CD (continuous integration and delivery) you want to prototype fast to deliver faster.
His team has agreed to codify these best practices. They identified a pattern where they can automate their “highly extensible” inner developer loop workflow, leveraging tools to automate code localizing, packaging and even deploying within containers, always asking: “Where’s the boring stuff and pain points to automate?”
-
Daniel Bryant’s recommendations of tools and processes to automate the developer workflow.
In the end, in this faster-paced world, you should be looking for anyplace to enhance the developers’ user experience. Bryant says API gateways become more and more important. For CI/CD to work, he says you need to automate code scanning, dependency scanning, container scanning, and security code scanning.
He said you have to make it easy for developers to do the right thing with self-service pipeline creations, including automating as much as possible of testing both non-functional requirements and system critical ones, trying to run everything in environments as real-to-life as possible.
This leads to another pattern in modern computing he mentioned which is observability and testing in production. You constantly have to ask your dev team:
- How are we going to monitor?
- How are we going to log?
- What about dashboard design?
- How do you create your alerts?
Bryant continued that “If you are going to create your own platform, think about developer experience and the workflow before you create the platform. You need to curate the local developer loop, the packaging, CI/CD, deployment control, and how we observe stuff through monitoring, logging and tracing,” before you start to build the platform.
