Will real-time data processing replace batch processing?
At Confluent's user conference, Kafka co-creator Jay Kreps argued that stream processing would eventually supplant traditional methods of batch processing altogether.
Absolutely: Businesses operate in real-time and are looking to move their IT systems to real-time capabilities.
Eventually: Enterprises will adopt technology slowly, so batch processing will be around for several more years.
No way: Stream processing is a niche, and there will always be cases where batch processing is the only option.
DevOps / Platform Engineering

Evolving DevOps: Platform Engineering Takes Center Stage

It's a transformative force, fusing the power of developers and Ops teams, defining a new course for efficiency and innovation in software development.
Aug 7th, 2023 9:29am by
Featued image for: Evolving DevOps: Platform Engineering Takes Center Stage
Image via Shutterstock.

This is the first of a 10-part series.

Imagine standing on the cusp of a seismic shift, a shift that promises to redefine the rules of the game. That’s where we are now with platform engineering, poised to redefine the DevOps world.

A lot of buzz around platform engineering dubs it as a phase. But platform engineering isn’t just another phase; it’s a transformative force, amalgamating the power of developers and Ops teams, defining a new course for efficiency and innovation in software development.

Platform Engineering: Finding the New Golden Path

The golden path represents the simple and straightforward journey that developers instinctively prefer. Guardrails established within a platform ensure that as long as developers stay within these boundaries, they can operate independently and unencumbered throughout their software development life cycle (SDLC).

However, complexities arise when they stray from this path. This encapsulates the core philosophy of platform engineering: to facilitate a level of independence, but only within established bounds.

The Value of Developer Autonomy

Back in the early stages of cloud computing, developers had a lot on their plate: They not only wrote code but also managed the delivery aspects. As cloud usage expanded, Ops teams emerged to help out. They acted as a bridge, making it easier for developers to interact with the cloud.

Today, with technology moving so quickly, there’s a demand for constant updates. Users expect their apps to be safe, efficient and regularly refreshed. Ops teams, though streamlined, have the task of meeting these high standards.

Gone are the days of slow, drawn-out updates. In this fast-paced era, developers feel the push to release updates more frequently, challenging them to strike a balance between speed and quality.

The Role and Evolution of Ops Teams

The traditional approach is hitting roadblocks — making quick software changes is hard to achieve.

Ops teams are now changing their role. Instead of just keeping things running, they’re becoming enablers for developers. They’re making tools that let developers work on their own, especially when they’re dealing with complex cloud environments.

This new focus means Ops teams will not just follow instructions, but give developers the tools to carry them out. When this happens, it’s crucial for Ops teams to watch out for any slowdowns and keep making improvements. This ensures that software gets out smoothly and developers have a good workflow.

We’ve seen DevOps change over time, from hands-on setups to automated tools and now to organized automation systems, making developers even more independent.

Enhancing the Developer Experience: From Dependency to Independence

Developers thrive when they can code, integrate and build without constant checkpoints. But the hold-ups begin when they have to depend on Ops teams often, from development to production. What if developers could oversee everything? That’s the vision.

When Ops teams curate platforms with the developer in mind, they break the chain of dependency. This independence leads to two primary benefits.

First, it boosts developer productivity, as developers, operating within certain guardrails, can seamlessly deploy, manage and operate.

Second, it alleviates Ops burnout. Now that they are not in the critical path of rollouts, they can focus on their core work, focusing on infrastructure efficiency and developer productivity.

This isn’t just a dream. It’s here. We are seeing a shift from manual configurations to automated systems and now to productized automations, which are more structured and allow for greater self-sufficiency among developers.

Hurdles in Embracing Platform Engineering

The Mindset

Adopting a platform-centric approach sounds compelling, but what stands in the way? The primary challenge is the shift in mindset.

DevOps, over the past decade, has employed automation to reduce manual tasks. While there’s a sense of pride in being “100% automated,” it’s essential to recognize that not all automations are “productized.” They may not be universally usable, particularly by someone not deeply involved in creating them.

While automation in DevOps has reduced manual work, it hasn’t fully empowered developers to act independently. The move to a productized approach is aimed at giving developers more control while maintaining organizational standards.

Transitioning from a purely automated mindset to an enabler’s perspective is vital. Automation should be seen as a tool in the larger product toolkit, designed for developer consumption rather than mere workload reduction.

Skills and Frameworks

After the mindset shift comes skill elevation. Writing automation scripts is different from productizing these automations. It requires an intimate understanding of developer workloads and their nonfunctional requirements. Productizing requires an innate ability to understand these needs and to craft solutions that cater to them.

However, a significant challenge today is the lack of comprehensive frameworks. While there’s no shortage of tools available, there’s a missing link: a cohesive framework that binds these tools, ensuring seamless operation.

Shift from Tools to Platforms

Once a cohesive platform or framework is in place, the next hurdle is change management. Developers must be trained to adapt to new operational procedures and workflows that these platforms bring. This isn’t merely about teaching them a new tool; it’s about reshaping their operational habits to integrate with a product that will be central to their daily tasks.

Maintenance and Evolution

A platform, once adopted, isn’t a static entity. Like any product used daily, it either evolves to meet growing needs or risks obsolescence. Maintenance isn’t just about fixing bugs; it’s about ensuring the platform remains relevant, updated and in sync with the changing landscape of development needs. This requires continual investment and attention.

The key is to acknowledge these challenges, address them and ensure a smooth transition for both Ops teams and developers.

The New World: A Symphony of Efficiency

Gearing up for the future platform engineering isn’t just the next big thing; it’s the roadmap for a harmonious DevOps future. It’s about setting the stage and letting developers steer with confidence.

The days of ad hoc automation and isolated operations are phasing out. It’s a new era of collaboration and empowerment. Let’s embrace platform engineering and the future it promises.

Group Created with Sketch.
TNS owner Insight Partners is an investor in: Pragma.
THE NEW STACK UPDATE A newsletter digest of the week’s most important stories & analyses.