How to Find and Eliminate Blind Spots in the Cloud
Visibility in the cloud is an important but difficult problem to tackle. Poor visibility can lead to all manner of security risks, including data loss, credential abuse and cloud misconfigurations. It is one of the biggest challenges facing security managers today as they look to adopt cloud technologies.
In an April 2020 survey from Enterprise Strategy Group, researchers revealed that 33% of respondents felt that a lack of visibility into the activity of the infrastructure hosting their cloud native applications was the biggest challenge in securing those apps.
Depth of visibility differs among cloud providers, and each one has its own positive and negative aspects. Let’s look at some of the logging and visibility options that Amazon Web Services (AWS) and Google Cloud Platform (GCP) offer, as well as their blind spots and how to eliminate them.
Know and Understand Your Visibility Options
Cloud providers typically offer some sort of default logging or monitoring at no extra cost, but it is never enough to gain sufficient visibility into what’s going on across your organization. They also provide additional paid services that allow you to gain more visibility into your environments for a variety of use cases. Because each cloud provider does things slightly differently, blind spots and the lack of visibility differ across providers.
The All-Important Control Plane
The “control plane” offered by a cloud provider is essentially what handles “cloud operations,” or API calls. The control plane must be accessible from anywhere on the internet and is what allows users to interact with the cloud and the resources in it.
For example, API calls made through the AWS Command Line Interface (CLI) are routed through the AWS control plane, allowing you to take actions such as launching a new Amazon Elastic Compute Cloud (EC2) instance. In GCP, the control plane handles things such as calls made through the “gcloud” CLI.
Visibility into which API calls are being made in your environment is incredibly important, which is why many of the free, default logging services are provided. These include CloudTrail Event History in AWS (a 90-day history of API calls made in your account) or something like activity logging in GCP, which provides a broad overview of activity in a project. These default offerings have shortcomings, however. Without additional configuration, you will miss critical elements.
To monitor the cloud control plane, you need to use built-in services provided by your cloud provider. But don’t stop there. Those logs should then be exported to an external security information and event management (SIEM) solution, such as Splunk, for further analysis and alerting.
These services include things like configuring AWS CloudTrail for every region and for every event type across your organization, or applying GCP audit logs to all supported services at the organization level. Doing this will help give you the additional insight you need to make more informed security decisions.
But What about Network and Hosts?
When it comes to activity within your cloud network or the hosts within that network, there typically is no free default offering to gain visibility. There may be default logging on your hosts, such as bash history, but there is nothing aggregating those logs and providing you access to all of your hosts through a unified interface.
There are many built-in and third-party offerings available to gain visibility into activity across your network and on your hosts. Some cloud workload protection platforms provide host-based monitoring and visibility that allows you to detect and prevent threats in real time.
Other offerings, such as AWS VPC (Virtual Private Cloud) Traffic Mirroring or GCP Packet Mirroring, can help with full packet capture within your cloud network. VPC flow logs are another useful tool for network visibility and offerings, such as AWS GuardDuty which can monitor those flow logs, as well as DNS logs and CloudTrail logs, to detect threats and unusual activity within your environment.
Pain Points and Blind Spots
No matter which monitoring and visibility options are available, even if you use all of your cloud provider’s built-in services (and even some third-party services), there are likely pieces missing from the puzzle.
The Ins and Outs of Data-Level Events
Data-level events differ from control plane or management events because they are actions performed on specific data rather than on a cloud resource. For example, AWS Simple Storage Service (S3) data events log activity for objects in an S3 bucket, providing insight into who’s interacting with which objects. Without additional configuration (and therefore additional cost), these event types are not logged. Some services offer data-level event logging, such as S3 bucket access logs, but others, such as AWS EBS Direct APIs, do not.
To ensure you are not losing insight into data-level events, you should enable them where possible. Where it is not possible, we recommend that you deny all users access to those data-level APIs.
Undocumented APIs
Every cloud provider publishes documentation for their supported APIs, which allows you to identify what is possible in their specific cloud. The problem is that there are often undocumented APIs supported by each cloud provider that may be exposed when they should not be. Often these undocumented APIs are used to enhance the user experience, such as providing a good UX through the web console the provider supplies.
Because these are undocumented and generally not for direct access, they are not logged where other documented APIs are logged. This lack of logging could allow an attacker to perform certain actions in your environment with no way for you to know they did.
To prevent undocumented APIs from being used maliciously in your environment, it is important to grant permissions on a granular level. That is, don’t grant permissions using wildcards (such as using a “*” in AWS), don’t use “managed” permissions sets as they are often overly permissive, and regularly review the permissions you grant your users to ensure they are properly restricted. By allowing listing permissions on a granular level, you can avoid much of the risk that undocumented APIs expose.
Pre-GA Services
Another potential blind spot in the cloud is pre-GA (general availability) services. This could include alpha, beta, gamma, pre-production or similar services. Such services are often blocked from the public and only permit access to listed users, but many times they are made available for public use while still in pre-GA status.
One example of how to access pre-GA services are the commands available through the “beta” and “alpha” groups of the “gcloud” CLI, which can be accessed with “gcloud beta <service>” and “gcloud alpha <service>,” respectively.
Just because they are pre-GA doesn’t mean they are not logged, but this is where you may encounter services that do not yet support logging and have a potential for abuse. You can prevent access to many of these beta and alpha services in GCP by denying users access to enabling new APIs in their projects and only providing them access to a list of trusted services.
The Principle of Least Privilege
Permissions must be set at a granular level so they only grant access to necessary services and APIs. These risks often arise when wildcards or managed permission sets are applied in an environment where you can’t be 100% sure of every service and action you granted to your user. When setting the permissions granularly, you can ensure that nothing unexpected can be accessed by your users.
Without taking advantage of logging and visibility offerings provided by cloud providers, you can quickly lose insight into what’s actually going on in your environment. Visibility gaps can still exist even when the proper services are enabled, as we’ve discussed with data-level events, undocumented APIs and pre-GA services, so it’s necessary to follow the principle of least privilege to avoid granting access to services and APIs that escape your monitoring capabilities.
You Have a Role to Play
It may seem like it’s the cloud provider’s responsibility to not release undocumented APIs or pre-GA services, but until they do, it is your responsibility to delegate permissions and access in a way that these APIs do not expose your environment to undue risk.
