Introducing Amazon S3 Transfer Manager for Swift (Developer Preview)

We are pleased to announce the Developer Preview release of the Amazon S3 Transfer Manager for Swift —a high-level file and directory transfer utility for Amazon Simple Storage Service (Amazon S3) built with the AWS SDK for Swift.

By using the S3 Transfer Manager API, you can now perform accelerated uploads of local files and directories to Amazon S3 and accelerated downloads of objects and buckets from Amazon S3. The concurrent transfers of a set of small parts from a single object provide enhanced throughput and reliability. The S3 Transfer Manager is built on top of the AWS SDK for Swift and uses Amazon S3 multipart upload and byte-range or part-number fetches for parallel transfers. You can also track the progress of transfers in real-time.

In this post we’ll explain how to use Amazon S3 Transfer Manager for Swift.

Parallel upload using multipart upload

For the upload operation, the Transfer Manager uses the Amazon S3 multipart upload API; it sends multiple UploadPart  requests concurrently behind the scenes to achieve high performance.

Parallel download using byte-ranges or part numbers

For the download operation, the Transfer Manager utilizes byte-range fetches or part number fetches. Byte-range fetches download the object with byte ranges and works on all objects, regardless of whether it was originally uploaded using multipart upload or not. Part number fetches download the object in parts, using the part number assigned to each object part during upload. The transfer manager splits one GetObject request to multiple smaller requests, each of which retrieves a specific portion of the object. Those requests are also executed through concurrent connections to Amazon S3.

Getting started

To get started with Amazon S3 Transfer Manager for Swift, complete the following steps:

Add the dependency to your Xcode project

  1. Open your project in Xcode and choose your .xcodeproj file, located at the top of the file navigator on the left pane.
  2. Choose the project name that appears on the left pane of the .xcodeproj file window.
  3. Choose the Package Dependencies tab, and choose the + button.
  4. In the Search or Enter Package URL search bar, enter git@github.com:aws/aws-sdk-swift-s3-transfer-manager.git.
  5. Wait for package to load, and once it’s loaded, choose the target you want to add the S3TransferManager module to.

Add the dependency to your Swift package

  1. Add the below to your package definition:
dependencies: [
    .package(
        url: "https://github.com/aws/aws-sdk-swift-s3-transfer-manager.git",
        from: "0.1.0"
    )
],
  1. Add an S3TransferManager module dependency to the target that needs it. For example:
targets: [
    .target(
        name: "YourTargetThatUsesS3TM",
        dependencies: [
            .product(
                name: "S3TransferManager",
                package: "aws-sdk-swift-s3-transfer-manager"
            )
        ]
    )
]

Initialize the S3 Transfer Manager

You can initialize an S3 Transfer Manager instance with all-default settings with the following:

// Creates and uses default S3TM config & S3 client.
let s3tm = try await S3TransferManager()

Or you can pass a configuration object to the initializer to customize the S3 Transfer Manager instance, for example:

// Create the custom S3 client config that you want S3TM to use.
let customS3ClientConfig = try S3Client.S3ClientConfiguration(
    region: "us-west-2",
    . . . custom S3 client configurations . . .
)

// Create the custom S3TM config with the S3 client config initialized above.
let s3tmConfig = try await S3TransferManagerConfig(
    s3ClientConfig: customS3ClientConfig,
    targetPartSizeBytes: 10 * 1024 * 1024, // 10MB part size.
    multipartUploadThresholdBytes: 100 * 1024 * 1024, // 100MB threshold.
    multipartDownloadType: .part
)

// Finally, create the S3TM using the custom S3TM config.
let s3tm = S3TransferManager(config: s3tmConfig)

For more information about what each configuration does, please refer to the documentation comments on S3TransferManagerConfig.

Upload an object

To upload a file to an Amazon S3 bucket, you need to provide the input struct UploadObjectInput, which contains a subset of PutObjectInput struct properties and an array of transfer listeners. You must provide the destination bucket, the S3 object key to use, and the object body.

When the object being uploaded is bigger than the threshold configured by multipartUploadThresholdBytes (16MB default), the S3 Transfer Manager breaks them down into parts, each with the part size configured by targetPartSizeBytes (8MB default), and uploads them concurrently using the multipart upload feature from Amazon S3

// Construct UploadObjectInput.
let uploadObjectInput = UploadObjectInput(
    body: ByteStream.stream(
        FileStream(fileHandle: try FileHandle(forReadingFrom: URL(string: "file-to-upload.txt")!))
    ),
    bucket: "destination-bucket",
    key: "some-key"
)

// Call .uploadObject and save the returned task.
let uploadObjectTask = try s3tm.uploadObject(input: uploadObjectInput)
let uploadObjectOutput = try await uploadObjectTask.value

Download an object

To download an object from an Amazon S3 bucket, you need to provide the input struct DownloadObjectInput, which contains the download destination, a subset of GetObjectInput struct properties, and an array of transfer listeners. The download destination is an instance of Swift’s Foundation.OutputStream. You must provide the download destination, the source bucket, and the S3 object key of the object to download.

When the object being downloaded is bigger than the size of a single part configured by targetPartSizeBytes  (8MB default), the S3 Transfer Manager downloads the object in parts concurrently using either part numbers or byte ranges as configured by multipartDownloadType (.part default).

// Construct DownloadObjectInput.
let downloadObjectInput = DownloadObjectInput(
    outputStream: OutputStream(toFileAtPath: "destination-file.txt", append: true)!,
    bucket: "source-bucket",
    key: "s3-object.txt"
)

// Call .downloadObject and save the returned task.
let downloadObjectTask = try s3tm.downloadObject(input: downloadObjectInput)
let downloadObjectOutput = try await downloadObjectTask.value

 Conclusion

To learn more about how to use the Amazon S3 Transfer Manager for Swift including how to upload a directory, download a bucket, and track transfer progress, visit our README.md on GitHub.

Try out the new S3 Transfer Manager today and let us know what you think via the GitHub issues page!

About the authors

Related Posts