Storage

Multipart uploads

Upload large files in parts for reliability and resumability. Required for files over 5 GB, recommended for files over 100 MB.

On this page6 sections

When to use multipart uploads#

Limit	Value
Recommended threshold	100 MB+
Maximum single upload	5 GB
Maximum object size	5 TB

Benefits#

Resumable: if a part fails, only retry that part instead of the entire file.
Parallel uploads: upload multiple parts simultaneously for faster throughput.
Integrity: each part has its own ETag for verification.

Supported operations#

CreateMultipartUpload: start a new multipart upload
UploadPart: upload a part (5 MB minimum, except the last part)
CompleteMultipartUpload: finalise and create the object
AbortMultipartUpload: cancel and clean up parts
ListParts: list uploaded parts
ListMultipartUploads: list in-progress uploads

Uploading a large file#

Each of these tools handles multipart uploads automatically:

AWS CLI: uses multipart for large files without extra configuration.
JavaScript: use the Upload utility from @aws-sdk/lib-storage.
Python: boto3 handles multipart automatically with upload_file.

Terminal

# AWS CLI automatically uses multipart for large files
aws --profile edge --endpoint-url https://storage.edge.network \
    s3 cp large-video.mp4 s3://my-bucket/videos/

# Configure part size (default is 8MB)
aws configure set s3.multipart_chunksize 64MB

JavaScript

import { S3Client } from '@aws-sdk/client-s3'
import { Upload } from '@aws-sdk/lib-storage'
import { createReadStream } from 'fs'

const client = new S3Client({
  endpoint: 'https://storage.edge.network',
  region: 'us-east-1',
  forcePathStyle: true,
  credentials: {
    accessKeyId: process.env.EDGE_ACCESS_KEY,
    secretAccessKey: process.env.EDGE_SECRET_KEY
  }
})

const upload = new Upload({
  client,
  params: {
    Bucket: 'my-bucket',
    Key: 'large-video.mp4',
    Body: createReadStream('./large-video.mp4')
  },
  queueSize: 4,              // Parallel uploads
  partSize: 1024 * 1024 * 64 // 64MB parts
})

// Track progress
upload.on('httpUploadProgress', (progress) => {
  console.log(`${progress.loaded} / ${progress.total}`)
})

await upload.done()

Python

import os
import boto3
from boto3.s3.transfer import TransferConfig

client = boto3.client(
    's3',
    endpoint_url='https://storage.edge.network',
    aws_access_key_id=os.environ['EDGE_ACCESS_KEY'],
    aws_secret_access_key=os.environ['EDGE_SECRET_KEY']
)

# Configure multipart settings
config = TransferConfig(
    multipart_threshold=1024 * 1024 * 100,  # 100MB threshold
    multipart_chunksize=1024 * 1024 * 64,   # 64MB parts
    max_concurrency=4                        # Parallel uploads
)

# Upload with progress callback
def progress(bytes_transferred):
    print(f"Uploaded {bytes_transferred} bytes")

client.upload_file(
    'large-video.mp4',
    'my-bucket',
    'videos/large-video.mp4',
    Config=config,
    Callback=progress
)

Part size recommendations#

File size	Recommended part size	Approximate parts
100 MB – 1 GB	16 MB – 32 MB	6–32 parts
1 GB – 10 GB	64 MB – 128 MB	16–80 parts
10 GB – 100 GB	128 MB – 256 MB	80–400 parts
100 GB+	256 MB – 512 MB	Varies

Cleaning up incomplete uploads#

Incomplete multipart uploads consume storage. Clean them up with:

Terminal

# List incomplete uploads
aws --profile edge --endpoint-url https://storage.edge.network \
    s3api list-multipart-uploads --bucket my-bucket

# Abort a specific upload
aws --profile edge --endpoint-url https://storage.edge.network \
    s3api abort-multipart-upload \
    --bucket my-bucket \
    --key videos/large-video.mp4 \
    --upload-id YOUR_UPLOAD_ID

Something unclear or out of date? Tell us

View as Markdown Get support