Will real-time data processing replace batch processing?
At Confluent's user conference, Kafka co-creator Jay Kreps argued that stream processing would eventually supplant traditional methods of batch processing altogether.
Absolutely: Businesses operate in real-time and are looking to move their IT systems to real-time capabilities.
Eventually: Enterprises will adopt technology slowly, so batch processing will be around for several more years.
No way: Stream processing is a niche, and there will always be cases where batch processing is the only option.
Cloud Services / Kubernetes / Serverless

Cloud Custodian and Capital One Ask, What’s in Your Amazon Cloud?

Jul 14th, 2017 11:55am by
Featued image for: Cloud Custodian and Capital One Ask, What’s in Your Amazon Cloud?

Typically when you think of a product from Capital One, you expect to see the lovely Jennifer Garner, or maybe Samuel L. “Enough is Enough, I Have Had It With These” Jackson, quizzing you about the contents of your wallet. And maybe hear about earning unlimited miles through a credit program with no annual fee. Perhaps the last thing you’d expect to see from the very same company is a policy definition engine for enforcing scripted rules in containerized environments running on Amazon AWS.

“We developed Cloud Custodian to give us a sort of unified view of, and enforcement of, all the policies that are happening in an AWS public cloud account,” explained Capital One Senior Distinguished Engineer Kapil Thangavelu, in a conversation with Alex Williams from the recent O’Reilly Open Source conference for The New Stack Makers.

Developers, Thangavelu told us, tend to do silly things — for example, leaving whole portions of their databases public. As Capital One built out its containerization strategy, its engineers found themselves devising scripts for ensuring that these stupid things don’t yield catastrophic results. But these were case-by-case instances, involving YAML and Python, and soon they needed a way to apply more generic scripts to wider sets of criteria, to enforce policies and apply guardrails as developers wandered into unexplored territories.

Cloud Custodian, he said, emerged from this basic need, especially as these thousands of one-off scripts generated program life cycle issues unto themselves. “You didn’t want to get into a place where you have hundreds of random scripts representing different controls, deployed by different people — maybe with code review, maybe with tests… You don’t want to be in that place where your controls have effectively locked out your ability to innovate.”

Thangavelu went into further detail about how Cloud Custodian works, and how you can put it to work as a rules enforcement manager in your own Amazon-based environments. He discussed how some rules may be implemented as Lambda functions using Amazon’s serverless model and can query live data from its CloudWatch monitoring service.

Listen now to Cloud Custodian and Capital One Ask, What’s In Your Amazon Cloud? — the latest edition of The New Stack Makers.

Cloud Custodian And Capital One Ask: ‘What’s In Your Amazon Cloud?’


1:00: What is Cloud Custodian, and why it was developed
4:08: How policies are written in Cloud Custodian
6:43: Why Lambda?
9:34: Exploring the role of Cloud Watch
12:35: Discussing the evolution of the Cloud Custodian framework
13:04: What’s next for Cloud Custodian

Capital One is a sponsor of The New Stack.

Group Created with Sketch.
TNS owner Insight Partners is an investor in: The New Stack.
THE NEW STACK UPDATE A newsletter digest of the week’s most important stories & analyses.