View Part I of the Series Here
Understanding and Implementing Immutable Backups in the Public Cloud
The Achilles heel is when backups and workloads are under the control of the same identity and access management (IAM) solution. In other words: When attackers compromise the IAM solution, they can delete servers and traditional backups (Figure 1, left). The attackers prepare some scripts and run them against your cloud infrastructure – a couple of minutes later, your applications crash or are deleted and gone. It does not help when the backup solution implements its own IAM solution. Attackers simply delete all resources of the backup solution in the cloud portal without logging in to the backup solution.
Immutable Backups - Challenge (left), Immutability Feature Pattern (Middle), and Disconnected-Backup Tenant Pattern (Right)
It is an existential risk that cloud admin roles can delete all resources, including backups, but two potential solutions exist. Option one is that cloud providers implement an immutability feature (Figure 1, middle). When activated, nobody – also no admin – can change or delete backups. It is a simple and efficient solution for companies.
Figure 2 shows the immutability feature in Azure. It has existed for a while for Object Storage data. Azure now rolls it out for all backups. When you read the text, it might still be in a public preview state, limited to some Azure Regions, or potentially already rolled out completely. A single click in the vault properties activates immutability (Figure 2, 1). First, the vault is in a medium state: all restrictions are in place, e.g., regarding the deletion of backups. Engineers can test the interplay with other components and applications before making the immutability setting unchangeable (Figure 2, 2). It is not difficult, but IT and risk managers must ensure the consistent use of the feature for all backups.
Configuring an Azure Vaults for Immutability
AWS provides similar features named AWS Backup Vault Lock, GCP the Cloud Storage Bucket Lock.
Based on my experience, it is essential to check whether immutability is available for all storage and database types of a concrete cloud design – and to validate whether the backup service meets regulatory requirements such as SEC 17a-4 if applicable. It does not help if a cloud provider celebrates the availability of immutability for one service but a company works with three others. Plus, there are two additional aspects cloud architects must consider for their backup concepts:
- Encryption Keys. When backups are encrypted, the keys for decryption have to be available to perform a backup restore. They have to be “immutable” as well.
- Backup Configurations. If an attacker can switch off the backup creation for weeks, there are no backups if he deletes all VMs and databases. Logging and alarming mechanisms – symbolized by the locks in Figure 1 – are crucial to identify unwanted critical changes.
Alternative: Disconnected Cloud Backups
When a cloud provider does not provide an immutability feature (or not for all storage types), a second pattern, the disconnected backup tenant pattern, can be an alternative. The pattern stores backups in a separate backup tenant in the same or different public cloud. The trick is an entirely separate IAM. Companies can set up their own backup tenant, including hardening and backup strategies, or rely on a 3rd party backup service provider to speed up the implementation of such a solution. In the latter case, the backup service provider manages the cloud backup tenant.
To stress the point: a second tenant has no benefit if there is any connection between the IAMs for the workload and the backup tenants. For example, having one Azure Active Directory that manages a cloud tenant with the workload on GCP and the backups on AWS is not a solution. If you rely on a backup service provider, not one of your employees must be able to delete “immutable” backups in the cloud tenant of the service provider.
A well-designed (!) disconnected backup in a hardened and monitored dedicated cloud tenant with separate access control mechanisms can come close to an immutable backup (or an old-fashioned air-gap backup). The exact risk ratings depend on a company’s risk appetite and technical setup. And sure, as discussed above, monitoring the backup configurations and ensuring the availability of decryption keys remain essential.
The Take-Home Message
Destructive attacks due to state-level tensions and ransomware attacks against SMEs are new threats. Based on their risk appetite, companies must assess and decide whether and how they want to mitigate the associated risk. In the cloud, immutable backups can be a straightforward solution– or dedicated, separate backup cloud tenants. Do not forget the homework to prevent such a scenario: firewalls, hardening, malware detection, endpoint detection, or intrusion detection systems, together with security operations centers, should prevent, detect, or at least contain attacks early. Immutable backups are the last hope. The aim should always be to have immutable backups in the coming years but never need them.
