Serve Dynamic Content to Millions of Users at No Cost
Creating scalable content for today’s massively connected world can come packed with many surprises. Everything from an insane Amazon Web Services bill, exhausted infrastructure teams scrambling to handle high demand or an individual developer sleeping soundly serving millions of users without having to enter in a single credit card number. Everyone wishes to be the latter but usually did not know such a solution exists in the rapidly evolving software community.
It doesn’t have to be this complicated. As an example, Nandi Wong, a Senior Backend Engineer at KintoHub, recently combined the right developer tools to build a dynamic website that serves millions of users at no cost. Not to mention, it was built in only two days.
WARS’ initial goal was to aggregate and inform the general public about everything from hospital queues, stores taking advantage of upselling medical supplies and alerts on neighborhoods that are being impacted by the Corona Virus. Data is publicly available across numerous sources on the web in formats from PDF, HTML websites and some public APIs and requires some processing to extract the exact information so it may be redistributed in a simple format. The team wanted to do update this information every few minutes, especially since a lot of the data is time-sensitive.
In order to serve content to millions of users, without incurring any costs, you must get rid of as many infrastructure dependencies as possible. Most software engineers may think that databases and backend APIs are required for serving dynamic content to consumers. Not anymore! Instead of building a stack that combines horizontal scaling services, data scraping cronjobs, Kubernetes, caching with Redis, CDNs, and databases at a heavy cost to develop and scale, WARS used static website generators and a next-generation cloud environment to scale at no cost.
Introducing Static Website 2.0
The first tool chosen was Gatsby, a widely adopted static website generator. The idea behind static websites is being able to quickly generate content for marketing, company, documentation or product information without having to run a web server. Wait?! Serving a website without a server… are you talking about serverless? No!
Cloudflare can host your static website files on hundreds of servers across the internet. The concept has been around for a while and is known as a Content Delivery Network (CDN) which all cloud providers provide their own flavor. Cloudflare offers an almost infinitely free tier that happily serves your files as long with very few limitations. You still will need an origin server that provides the initial content that gets distributed globally, which usually is hosted for free on Github Pages or other website providers.
The Dilemma of Serving Dynamic Content
Since WARS requires the data to be updated every few minutes, this usually discounts static websites as an option. Traditionally when a user loads the page, one or many API calls will be made to a backend server to grab the data and insert it into the page before rendering the result for every unique user! Usually, data resides in a database, be it MongoDb, Mysql, Postgres, etc. The source of the data is to be populated by another service that is legally scraping the latest information from various sources. Combining all of this, it would take some time to develop, test and build on the cloud of your choice.
Using automation combined with the concept of static website generators, you can regenerate the website with the data you want to serve every few minutes. This means a build is triggered and deployed constantly to Cloudflare. This approach only works when all users view a single source of generated content. If users are looking at specific contextual content, such as their account balance, this approach would not work.
Combine and Simplify All Cloud Operations
KintoHub was chosen to help with automation and hosting as an all-in-one platform for websites, cronjobs, databases and backend services. It picks up code from Github, hosts a free static website with Let’s Encrypt SSL with Cloudflare CDN support and makes it easy to build and repeatedly trigger Cron Job operations that force regenerating and deploying WARS. Under the hood is really Kubernetes and Dockerfiles orchestrated by a simple UI/UX dashboard, but the time to release the idea of WARS was sped up by a lot due to not having to configure all the moving parts on Amazon, Google, Azure or Kubernetes itself.
Today the world is riddled with too many options and if you’re not learning hourly, testing new technologies nor evolving with the latest trends; your next project may incur higher costs or slower go to market times. Open source has revolutionized the world in how we collaborate and share features and functionality. As the software industry moves forward, businesses and individuals need to figure out how to easily integrate the power of the widely available options to be more clear and instantaneously useful.
Hopefully, the following links can be helpful in the never-ending exploration of that journey.
The Cloud Native Computing Foundation is a sponsor of The New Stack.
Feature image via Pixabay.