Cloud Storage bucket relocation: An industry first for non-disruptive bucket migrations

As your operational needs change, sometimes you need to move data residing within Google’s Cloud Storage to a new location, to improve resilience, optimize performance, meet compliance needs, or simply to reorganize your infrastructure. Yet moving buckets can be a daunting, complex, risky endeavor that involves manual scripting, painstaking coordination, and the risk of data loss, or worse yet, extended downtime. This can discourage organizations from making the changes they need to their storage environments.

We recently introduced Cloud Storage bucket relocation, a unique feature among leading hyperscalers that makes it easy  to change your bucket’s location. Bucket relocation eliminates the need for complex manual planning and helps prevent extended downtime, for an easy transition with minimal application disruption, and strong data integrity. Your bucket’s name, and all the object metadata within it, remain identical throughout the relocation, so there are no path changes, and your applications experience minimal downtime while the underlying storage is moved. Furthermore, your objects retain their original storage class (e.g., Standard, Nearline, Coldline, Archive) and time-in-class in the new location. This is key for many cost efficiency strategies, helping ensure capabilities such as Autoclass continue to operate intelligently to optimize your storage costs post-migration.

Bucket relocation is a key capability within the Storage Intelligence suite, alongside tools like Storage Insights, which  provides deep visibility into your storage landscape and identifies optimization opportunities. Bucket relocation then lets you act on these insights, and move your data between diverse Cloud Storage locations — regional locations for low latency, dual-regions for high availability and disaster recovery, or multi-regions for global accessibility — to meet your  business, performance, and compliance objectives.

Bucket relocation under the hood

Bucket relocation relies on two critical techniques. 

• Asynchronous data copy: Bucket relocation leverages a unique and optimized asynchronous data transfer mechanism that copies data in the background to  minimize impact to ongoing operations. Existing operations like writing, reading, and updating objects continue while the entire dataset is being copied.

• Metadata preservation: Historically, Google Cloud customers moved data with the Storage Transfer Service, which copied the objects to a new bucket and deleted existing ones. Bucket relocation, on the other hand, automatically and meticulously moves all your bucket’s and objects’ associated metadata, thereby preserving state. This includes information like:

• Storage class: Your objects retain their original storage class (e.g., Standard, Nearline, Coldline, Archive) in the new location.

• Bucket and object names: The naming structure of your buckets and objects remains identical.

• Creation and update timestamps: These markers are preserved, so that features like object lifecycle management (OLM) rules continue to operate.

• Access Control Lists (ACLs) and IAM policies: Bucket- and object-level permissions are transferred to help maintain your security posture.

• Custom metadata: Any user-defined metadata associated with your objects is also migrated.

By handling the complexities of asynchronous data transfer and automatic metadata migration, bucket relocation minimizes the risks and overhead associated with a manual bucket migration. Crucially, because the bucket name is preserved throughout the relocation process, applications accessing the bucket don’t need to be modified.

Relocate your bucket in a few simple steps

With bucket relocation, you can move your Cloud Storage buckets in three simple steps. Here’s a breakdown:

1. Initiate a dry run:

• Before starting the actual relocation, it’s highly recommended to perform a dry run. This simulates the process without moving any data, allowing you to identify potential issues early on, such as incompatible configurations.
• The dry run checks for incompatibilities like customer-managed encryption keys (CMEK), locked retention policies, objects with temporary holds, and bucket tags, without you having to manually validate each of them.
• Make sure to add the --dry-run flag!

Replace BUCKET_NAME with the name of your bucket and LOCATION with the desired destination.

2. Start the relocation process:

• This step initiates the actual data transfer from the source bucket to the destination bucket. During this phase, you can still read, modify, and delete objects in the bucket. However, the bucket metadata (i.e., bucket-level  parameters and configurations) is write-locked to prevent changes that could affect the relocation.
  Note: Removing the --dry-run flag from the dry-run command initiates the relocation.

3. Finalize the relocation process:

• Once the incremental data copy is complete, you’re ready to trigger the final synchronization step (except when moving between multi-region and configurable dual-region). This involves a brief period where writes to the bucket are disabled to help ensure their data integrity; any last-second changes made to the objects within the bucket while the incremental copy was in progress are copied to the destination. After the data’s integrity is verified, the bucket’s location is updated, and all requests are automatically redirected to the new location. During the final synchronization step, attempts to update objects in the bucket will result in an HTTP 412 error.
• Do not initiate the final synchronization process until the relocation process progress reaches ~99%. This helps you minimize downtime because most of the data has already been synchronized in the background. 

Note: If you’re moving between multi-regions and configurable dual-regions within the same multi-region code, you’re all set — bucket relocation handles the transition in the background, no finalization or downtime required!

The OPERATION_ID is provided as output from Step-2. The OPERATION_ID is listed with the keyword name. For instance:

name: projects/_/buckets/my-bucket/operations/AbCJYd8jKT1n-Ciw1LCNXIcubwvij_TdqO-ZFjuF2YntK0r74

And there you have it — In just three steps, you’ven moved your entire bucket, its data, and metadata, to its new location.

Early users of bucket relocation have had great success with the new feature. 

“With Storage Intelligence and bucket relocation, we effortlessly transitioned to dual-region buckets. The seamless process, powered by the bucket relocation, minimized downtime and ensured data integrity. We migrated the buckets with peace of mind and without the manual headaches.” – Adam Steele, Product Manager, Spotify

“We recently utilized the bucket relocation feature of Storage Intelligence to successfully complete a ~300 bucket migration and PBs of data project from multi-region to regional storage, to optimize network data transfer costs. Without bucket relocation, this process would have required extensive automation and scripting, resulting in increased downtime and effort.” – Deepak Mahato, Data Platform Infrastructure Manager, GroupOn

Experience the ease and efficiency of managing your Cloud Storage buckets with bucket relocation in Storage Intelligence. To learn more, visit the bucket relocation documentation and the Storage Intelligence overview.