Consistency of deletion of resources related to activations

[hypnotisttom]

Hi,

Regarding deletion of resources and their ties to activations...

For akamai_edgeworker, if the resource is to be deleted by the plan, and if a version is activated on staging/production, could the deletion of the resource trigger the deactivation of the version as well, given a situation where the activated version may not be in the terraform itself?

For akamai_cloudlets_policy, the deactivation of the cloudlet occurs when the policy is requested to be deleted (similar to what I'm asking for above). What's interesting with cloudlets is the plan has to be applied twice: once for the deactivation, and then once about an hour later to delete the resource, otherwise we run into a "pending activations" error message. While this isn't ideal, it does allow us to delete the cloudlet when a cloudlet activation resource may not be in the TF state, something we cannot do with the EdgeWorker if the EdgeWorker's activation isn't in the TF's state.

Even given the double-apply, if edgeworkers worked the same way as cloudlets, that would help a lot, as I don't currently have a solution for the EdgeWorker delete, given that the activation has to be forgotten by TF so that our build process can run without activating every build.

Thanks,

Tom

[robertolopezlopez]

Hi @hypnotisttom ,

Regarding akamai_edgeworker: yes, deletion of a resource will trigger its deactivation in staging and/or production.

Regarding akamai_cloudlets_policy: yes, you are once again right. As you can see in the resource itself, the API returns an error when we try to delete any active policy.

The reason behind this difference between subproviders lies in the backend. Those are two different APIs, designed and implemented by different teams. While this different behavior is not ideal, there is not much we can do. I will assign the "enhancement" tag to this issue, but I cannot assure that your suggestion will be taken into account (EWs subprovider is already in production and this may be an undesired behavior change for other customers).

In any case, thank you once again for your suggestion.

Regards,

Roberto

[hypnotisttom]

Thanks Roberto. Just to make sure we're saying the same thing...

For EdgeWorkers, the deactivation is not even triggered by a deletion of the EdgeWorker resource itself if the activation resource is not included, whereas for the Cloudlets, the deactivation occurs, then the "activations" message appears, but I can then re-apply the exact plan to finish off the deletion a subsequent time (although it took me a moment to understand that the activations error message also applied to deactivations).

While the cloudlet pattern could be enhanced, it isn't as much an issue for me as the edgeworker one.

In cloudlet policy delete, you are calling the version delete, and I see the version delete for edgeworkers during an EW delete as well: https://github.com/akamai/terraform-provider-akamai/blob/master/pkg/providers/edgeworkers/resource_akamai_edgeworker.go#L335

Are you saying that the backend cloudlet API will perform the deactivation on its version delete, whereas the edgeworker backend API does not process the edgeworker version delete with a deactivation first?

If that's the case, can the provider for EWs effectively call activation delete prior to version delete? https://github.com/akamai/terraform-provider-akamai/blob/master/pkg/providers/edgeworkers/resource_akamai_edgeworker.go#L335

Thanks for the discussion!

Tom

[robertolopezlopez]

@hypnotisttom : please, let me check with the respective teams. We have developed this a while ago and I might be wrong.

[robertolopezlopez]

@hypnotisttom :

Out of memory: upon cloudlet policy deletion, the activations are deleted on server side. Anyway SFTP propagation may be slow, which triggers the mentioned error. When we apply for a second time, this propagation may have finished already and this is why it seems that we need to reapply for a second time.

This happens specially in our test environments; I never expected that this problem could occur in production as well.

Back in time when I was we were developing the cloudlet policy activation resource, we were told that this API defect would be fixed in next API versions. I cannot share any dates.

Anyway, as mentioned above, I am contacting the respective teams for better orientation. I will let you know.

[hypnotisttom]

Thanks so much Roberto!!!

[robertolopezlopez]

Hi @hypnotisttom ,

I got an answer from the EW team:

"""
The implementation we have now is to reject the call to delete if there is an in-progress activation or if there is an active version on production/staging.
If you want to delete it would be necessary to deactivate first, and if an activation is in flight it would be necessary to retry after the activation is done.
"""

Apparently I wasn't 100% right so it was good to double check with them :-) I tried to verify this myself in the testing environment, but unfortunately there were environment problems yesterday.

I am not really sure how we might be able reach consistency between those two API behaviors, even more when in theory the cloudlets behavior is temporary. In any case, I will leave this issue for the team to evaluate.

Kind regards,

Roberto

[hypnotisttom]

That's definitely the behavior I'm seeing.

Okay, let's see if the team can evaluate. What I'd like on the EW side is the deactivation prior to version/ew delete, where the deactivation could be triggered from a resource flag like deactivate_if_needed = true, etc.

The challenge I have is I have no way to trigger the deactivation on the EW side because the activation won't be in the state due to our CI/CD design. I will see if I can find a workaround in the meanwhile.

Of course on the Cloudlet side, the backend API is performing the deactivation on delete, and then we just have to run it twice (once to deactivate, once to delete).

I think allowing the provider to specifically call EW deactivate would make it functionally equivalent. I'll leave it for the team to evaluate. While I don't know the exact driver for the EW backend change, I feel if the resource has a flag that says to deactivate if it is needed (it wouldn't be needed if it isn't activated, and we wouldn't want the deactivation to fail), then it's a conscious decision by the TF resource author to perform the action.

Thanks,

Tom

[robertolopezlopez]

We will see what we can do. Let me understand properly: so you handle the edgeworkers with terraform, but not the activations?

If you would handle the activations as well with terraform, I believe that you may be able to achieve what you intend with the depends_on keyword from resource akamai_edgeworker_activation side. Just an idea. But as in your case this is not possible, just ignore this message.

[hypnotisttom]

Basically, in the CI/CD, the CD is set up to only activate on a "dev" branch or "main" branch - the "dev" branch triggering on staging, and the "main" branch triggering on production. So, we do do the activations in Terraform; however, the problem comes when using feature branches. In a feature branch, no activation is to be performed. We achieve this using the count parameter. However, the absence of the activation resource is seen as a deactivation. So, we forget the activation from the state after performing it. That then allows our feature branches to continue to work on the NEXT version without trampling on the last one.

This works just fine as a process with Cloudlets, presumably because the deletion of the cloudlet actually triggers the deactivation. The activation process works for properties too, although I haven't yet checked to see if properties behave like EdgeWorkers or Cloudlets in terms of deactivation.

So, while this suggestion is being evaluated, the workaround I am going to look into is reimporting the EW activation if we want to deactivate, although there might be some "gotchas" with that.

Can you send me the import string example for akamai_edgeworkers_activation?
I don't see it on this page: https://registry.terraform.io/providers/akamai/akamai/latest/docs/resources/edgeworkers_activation

Thanks,

Tom

[robertolopezlopez]

Uh, that's something that unfortunately we have not implemented: akamai_edgeworkers_activation does not really have any import 😞

But on the other side, the create method of that resource works actually like import - if the provided edgeworker_id and version are active on the given network, its details are written to the local schema. 😄 Probably this should be better documented, please let me know if you have any question on this topic.

[hypnotisttom]

Hmmm... In looking at this, it's a bit tricky because there is also neither an "akamai_edgeworker" nor "akamai_edgeworkers_activation" data source that I could use to find out the currently activated version and then use that in the akamai_edgeworkers_activation resource to re-add to the schema.

Do you have any suggestions for how to get the latest activated version so that I can use that?

Also, do you think the team would be receptive to a suggestion that each resource also have a corresponding data source to get the created resource? I know we were submitting them for the components separately, though it may make sense to just fill all those gaps.

Tom

[kaush-]

Hi Tom,

I understand that at the moment we do not have the corresponding data sources for EW activation and resource. Have you also tried the export-edgeworkker command under https://github.com/akamai/cli-terraform. Given the EW ID, it will export the latest version details and will also create an activation HCL which then can be imported directly in TF using the import.sh script.

Regards,
Piyush

[hypnotisttom]

Hmmm interesting thought. I don't really have the CLI available within the CI/CD pipeline.

Can this basically become an enhancement such that all resources in the provider have corresponding data sources so that they can be retrieved? And that all resources also have import support?

Currently, some do and some don't. Others are requested; some haven't been. It would be nice to just fill the gaps such that every resource can be retrieved with a data source and also imported if need be. Plus, the data source would bring back more info than I'd otherwise know, such as the latest version activated, etc. for a variety of resources.

Thanks,

Tom

[kaush-]

Hi @hypnotisttom,

Thank you for your comment. We have decided to add those two data sources within the scope of next release. I will update this issue once it has been published.

Let me know if you have any other concerns.

Regards,
Piyush

[kaush-]

Hi @hypnotisttom,

A small correction, we might not be able to add this fix in the very next release. But rest assured that we are tracking it internally and will update you as soon as it's ready for release.

Regards,
Piyush

[hypnotisttom]

Okay thank you.

I'm hoping the datastream resources in #327 can also be added. Really the problem is not every resource has a corresponding data source getter, and it's causing numerous problems for implementing streamlined code. I'm doing my best to work around these limitations, although the workarounds are not ideal.

[lkowalsk-akamai-com]

Hi @hypnotisttom , new data sources for Edgeworkers were added with todays (2.4.1) release. Please take a look.