Add Paginator to Recursively Delete All Objects Before Delete Bucket

[zliang-akamai]

📝 Description

• Fixed issues:
  • A ClientError happens in the S3 client doesn't have any error message output
  • Versioned buckets containing objects can't be deleted recursively
  • Buckets containing more than 1000 objects can't be deleted recursively

✔️ How to Test

Automated Testing

pytest tests/integration/obj/test_obj_plugin.py -k "test_large_number_of_files_single_bucket"

Manual Testing

You can create these objects and buckets with OpenTofu/Terraform, but they will run for a very long time to create 2010 objects.

tofu apply -parallelism=100

provider "linode" {
  obj_bucket_force_delete = true
}

resource "linode_object_storage_bucket" "nonversioned-bucket" {
  region = "us-mia"
  label  = "test-cli-recursive-rb1"
}

resource "linode_object_storage_bucket" "versioned-bucket" {
  region = "us-mia"
  label  = "test-cli-recursive-rb2"
  versioning = true
}

resource "linode_object_storage_key" "foobar" {
  label   = "my-key"
}

resource "linode_object_storage_object" "objects" {
  count  = 1005
  bucket = linode_object_storage_bucket.nonversioned-bucket.label
  region = linode_object_storage_bucket.nonversioned-bucket.region
  key    = "myobj-${count.index}"

  access_key = linode_object_storage_key.foobar.access_key
  secret_key = linode_object_storage_key.foobar.secret_key

  content          = "haha"
  content_type     = "text/plain"
  content_language = "en"
}

resource "linode_object_storage_object" "versioned-objects" {
  count  = 1005
  bucket = linode_object_storage_bucket.versioned-bucket.label
  region = linode_object_storage_bucket.versioned-bucket.region
  key    = "myobj-${count.index}"

  access_key = linode_object_storage_key.foobar.access_key
  secret_key = linode_object_storage_key.foobar.secret_key

  content          = "haha"
  content_type     = "text/plain"
  content_language = "en"
}

And then remove them via CLI:

linode obj rb test-cli-recursive-rb1 --recursive;
linode obj rb test-cli-recursive-rb2 --recursive;

[ezilber-akamai]

LGTM! Tests passing locally.

[lgarber-akamai]

Integration tests are passing locally using make install && make MODULE=obj testint, great work!