Cocktails and Dreams: A Programming Layer for the Cloud
Modern cloud platforms offer an ever-expanding array of powerful building blocks for developers, managed services for offloading operational burdens from development teams, and automatic scale and elasticity. These capabilities promise to enable developers to imagine, build, and deliver next-generation applications with services such as AWS Lambda, Docker, Google Kubernetes Engine, and Azure Cosmos DB becoming foundational building blocks for almost any new application being developed today.
And yet, taking advantage of these new managed services is still too hard — cloud infrastructure automation is too hard. Using these services inevitably ends up involving point-and-click consoles, low level REST APIs and CLIs, and esoteric YAML and JSON documents.
While this may have worked when cloud programming just meant “put my code in a cloud-hosted VM,” now that the cloud has become a fundamental part of every application’s architecture, configuration can no longer be an afterthought — we desperately need improved development tools.
As developers, we just want to write code. Having a way to truly program the cloud can unlock productivity, reusability, and the true promise of cloud computing. At Pulumi, that’s what we’ve set out to do.
CoLaDa: Containers AND Lambdas AND Data Services AND Infrastructure
The pace of innovation from cloud vendors, combined with the sheer number of available services is continuously creating new opportunities for developers. While the debate may rage on “containers OR serverless,” the actual answer is always “AND” for the majority of teams of any reasonable size.
Cloud has enabled us to shift from server-based to container-based, to function-based, application delivery. And in doing so has shifted the focus from infrastructure-centric operations to application-centric delivery.
Furthermore, there is still a desire for cloud-like capabilities delivered on-premises, and so while all the cloud vendors offer a flavor of Kubernetes, many larger enterprises will choose to deploy their own, gaining — hopefully — cloud-native benefits such as portability.
Better Developer and DevOps Collaboration Through Real Code
The trend of DevOps roles shifting toward SRE roles is catalyzed by the increasing need of operations specialists to code, and to understand development code. DevOps skills are moving “up the stack.”
The reverse trend is true: developers are increasing tying infrastructure and service choices to their application logic to make use of specific managed features and so are required to take responsibility for those choices.
The lack of a lingua franca for this collaboration — developers speak code, and DevOps, begrudgingly, speak DSLs and configuration — leads to failures to communicate and ultimately to brittle systems and risk. Or as Apple engineer Cindy Sridharan pointed out:
Silos and Specializations of Single Clouds
Multicloud development is real, and many organizations are beginning to explore it, but it often appears unattainable based on the need to master a multitude of tools and frameworks for development.
Each cloud has specialized services but — at a high-level at least — many of the services are designed to provide the same capability: for example AWS Lambda/Azure Functions/Google Functions or AWS S3/Azure Blob Storage/GCP Cloud Storage. Over time, we see this trend of consolidation, much like we saw with operating systems over the past two decades which enabled highly productive programming environments such as Java, .NET, Node.js, and Ruby on Rails, and Kubernetes is helping to accelerate this trend.
For a developer, working with these services requires the right level of abstraction to be productive across any of the clouds, and retain options both on portability, and optimization for a specific implementation.
Delivering a Cloud Programming Model with Pulumi
Recognizing these shifts — of architecture, roles, capabilities — we set about building a way to “program the cloud” by providing the framework and tools to express cloud apps and infrastructure as real code.
This means a feature set that can deliver:
- A programming framework exposing the full set of services on every cloud, which exposes the exact, unopinionated granularity offered by each vendor.
- Bindings for multiple languages with an idiomatic feel for a given language. As a start Pulumi’s framework is available in JavaScript/TypeScript, Go, and Python.
- An engine capable of understanding how to deploy applications and infrastructure, and operate in a mode of “immutable infrastructure”: able to manage and change only the minimum necessary parts of the architecture immutably, while retaining perfect auditability and history.
- A CLI and hosted service to provide those capabilities productively and in a way that integrates into familiar processes and tools that work in a team setting: from IDEs through to CI/CD.
- Multiple layers of abstraction over those services that offer higher productivity and multi-cloud opportunities: standard objects such as “functions”, “queues”, and “tables” that are easier to program.
We launched our “Cloud Development Platform” just a month ago and thousands of developers and DevOps practitioners have tried it out. As a new product, we’re keen for your feedback, so please try it out, join the community, and let us know your feedback.
Example App: A Video Thumbnailer on AWS
How then, does this promise of a programming model for the cloud, the mixing of components, and the delivery of all of this in one expressive and intentional form come together?
Consider the following example: we want to build a simple app to create thumbnails from uploaded videos. (You can see the full details of this application here).
This example is easily delivered using a combination of components — a trivial ‘CoLaDa’ app:
- Use a long-running task to process the uploaded video.
- Event triggers for newly uploaded videos, and the output thumbnails.
- Data Services. Storage for uploaded videos and thumbnails.
The app can be assembled using any variety of tools for each of these components. But each has its own configuration requirements and/or DSLs for control. Also, simply gluing together the components becomes non-trivial, or tedious.
Being able to code all of these components in a single place offers significant productivity advantages: real code means code completion, interactive error checking in your favorite IDE, package management and versioning in the usual way, reusable components, among other benefits. It also provides the most intentional, and expressive, way of describing a modern cloud application, which is lost when navigating a morass of configuration scripts and UIs.
And best of all, using code is highly productive. This application is developed in just ~25 lines of code — there is no hidden YAML lurking here, it’s just real code that expresses application logic and infrastructure requirements (the pre-existing container notwithstanding).
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 |
<strong>const</strong> cloud = require("@pulumi/cloud-aws"); <em>// Define a bucket to store videos and thumbnails.</em> <em>// As we're using AWS, this will create an S3 bucket.</em> <strong>const</strong> bucket = <strong>new</strong> cloud.Bucket("bucket"); <strong>const</strong> bucketName = bucket.bucket.id; <em>// A task which runs a containerized FFMPEG job to extract a thumbnail image.</em> <strong>const</strong> ffmpegThumbnailTask = <strong>new</strong> cloud.Task("ffmpegThumbTask", { build: "./", <em>// folder containing the Dockerfile, which is built as part of the deployment</em> memoryReservation: 128, }); <em>// When a new video is uploaded, run the FFMPEG task on the video file</em> <em>// Use the time index specified in the filename (e.g. cat_00-01.mp4 uses timestamp 00:01)</em> bucket.onPut("onNewVideo", <strong>async</strong> (bucketArgs) => { console.log(`*** New video: file ${bucketArgs.key} was uploaded at ${bucketArgs.eventTime}.`); <strong>const</strong> file = bucketArgs.key; <strong>const</strong> thumbnailFile = file.substring(0, file.indexOf('_')) + '.jpg'; <strong>const</strong> framePos = file.substring(file.indexOf('_')+1, file.indexOf('.')).replace('-',':'); <strong>await</strong> ffmpegThumbnailTask.run({ environment: { "S3_BUCKET": bucketName.get(), "INPUT_VIDEO": file, "TIME_OFFSET": framePos, "OUTPUT_FILE": thumbnailFile, }, }); console.log(`Running thumbnailer task.`); }, { keySuffix: ".mp4" }); <em>// When a new thumbnail is created, log a message.</em> bucket.onPut("onNewThumbnail", <strong>async</strong> (bucketArgs) => { console.log(`*** New thumbnail: file ${bucketArgs.key} was saved at ${bucketArgs.eventTime}.`); }, { keySuffix: ".jpg" }); <em>// Export the bucket name.</em> exports.bucketName = bucketName; |
The Future of Development Means Being Able to Program the Cloud
There’s never a dull time for developers, but this era of “true” cloud computing is particularly exciting as any developer is now only scoped by their imagination, as they are unlimited in delivery at scale.
At Pulumi, we want to help get that code to the cloud faster than ever before, and make the development and operation of this new generation of cloud apps expressive, intentional and — dare we say it — joyful. Let us know how we’ve done.
Feature image via Pixabay.