Fix memory error in CPU nightly workflow

[pradnyeshjoshi]

Description

Added manual deletion of dataframes in some integration tests to clear VM memory, this resolves the memory error in CPU nightly workflow.

Related Issues

Checklist:

• I have followed the contribution guidelines and code style for this project.
• I have added tests covering my contributions.
• I have updated the documentation accordingly.
• This PR is being made to staging branch and not to main branch.

[miguelgfierro]

Approved.

However, I'm super puzzled. This was a memory problem, however, we have a large VM with enough memory. Why we have ever encountered this problem before in the hosted VM?

[pradnyeshjoshi]

Approved.

However, I'm super puzzled. This was a memory problem, however, we have a large VM with enough memory. Why we have ever encountered this problem before in the hosted VM?

@miguelgfierro I will close this PR for now as after changing the VM from D13v2 (56GB RAM) to D14v2 (112 GB RAM), I don't get memory error anymore.

[miguelgfierro]

hey @pradnyeshjoshi I don't think it is a bad thing to have this PR. It seems we are spending a lot of memory, and if someone with less resources than Microsoft want to execute this, we are making their life easier.

What do you think?

[pradnyeshjoshi]

hey @pradnyeshjoshi I don't think it is a bad thing to have this PR. It seems we are spending a lot of memory, and if someone with less resources than Microsoft want to execute this, we are making their life easier.

What do you think?

@miguelgfierro Sounds good to me, I'll merge into staging once all the checks pass.