feat: OpenTofu support

[meringu]

what

This is a change to get OpenTofu to work with Atlantis:

• Introduces a --tf-distribution setting that can be set to terraform or opentofu.
• OpenTofu is downloaded via tofudl.
• Added an interface to server/core/terraform package for the Terraform distribution. This matches up with the --tf-distribution setting.
• I would like to also support configuring Terraform or OpenTofu per project in the server side config or repo level config. This would support an easier path to migrate between the two.

why

#3741

tests

• Have refactored out a distribution and made changes to the downloaders for both Terraform and Conftest to be more testable
• I've deployed this branch within my org, and have got successful plan and applies working

references

• Closes Support OpenTofu (epic) #3741

[james0209]

I agree with the premise of this, and I like the solution, but I don't agree that it stops the need for hc-install - I think that is worthy of a discussion.

I do not think your changes inherently conflict with the hc-install PR - it is very easy to do this for example

func (c *DefaultClient) DetectVersion(log logging.SimpleLogging, projectDirectory string) *version.Version {
	return c.distribution.DetectVersion(log, c, projectDirectory)
}

type Distribution interface {
	BinName() string
	SourceURL(v *version.Version, downloadURL string) string
	DetectVersion(log logging.SimpleLogging, c *DefaultClient, projectDirectory string) *version.Version
}

func (*DistributionTerraform) DetectVersion(log logging.SimpleLogging, c DefaultClient, projectDirectory string) *version.Version {
// use hc-install
}

func (*DistributionOpenTofu) ListAvailableVersions(log logging.SimpleLogging, downloadBaseURL string, downloadAllowed bool) ([]string, error) {
// use current implementation - only called by the DetectVersion below
}

func (dt *DistributionOpenTofu) DetectVersion(log logging.SimpleLogging, c DefaultClient, projectDirectory string) *version.Version {
// use current implementation
e.g. tfVersions, err := dt.ListAvailableVersions(log, c.downloadBaseURL, c.downloadAllowed)
}

Especially as the ListAvailableVersions func is only called in the current implementation of DetectVersion - there is no other usage.

I think if you are going to have separate interfaces for OpenTofu vs Terraform implementations, then a discussion on whether to use the same logic vs different logic for each Distribution should be had.

• An example being how this PR already introduces 2 different logic flows for ListAvailableVersions (due to tagging, pre-relese etc.) - with hc-install, that would only be needed by OpenTofu, as it is handled by hc-install for Terraform

[meringu]

Thanks @james0209, that makes sense.

I'd be happy to rebase this over the hc-install change. As you point out, that should be fairly straightforward.

[nitrocode]

Due to wip status, I switched this to a draft as a non draft means that it's ready for review. Please set it as ready to review when ready.

Thank you for the contribution

[chenrui333]

I'd be happy to rebase this over the hc-install change. As you point out, that should be fairly straightforward.

we have merged hc-install update, might worth do the rebase and pick up this PR again. Thanks for all the efforts!

[jamengual]

The OpenTofu team is going to release this downloader to make it easier for us to auto download Opentofu.

https://github.com/janosdebugs/downloader
You can pull it in with a replace statement in go.mod for now until is released today or Monday.

@meringu @nitrocode if any of you want to work on this let us know

[meringu]

Awesome, I rebased this from main yesterday and got to the part where we need to download tofu.

I'm keen to keep working on this, but happy to have it taken over if I'm being too slow.

[meringu]

Tofu downloader has been moved: https://github.com/opentofu/tofudl

[meringu]

Thanks for your patience. I've done the following:

• Merged from main to get the hc-install changes.
• Updated to use tofu downloader.
• Refactored out a distribution and downloader.
• Made a separate downloader for conftest to decouple it from the TF downloaders.

I've done some testing on one of my Atlantis servers. I've been able to do successful plan and applies. Have also tested resolving different version from constraints from the required_version setting.

[X-Guardian]

Hi @meringu, it looks there are a lot of spurious changes included in this PR, .gitignore, dockerfile, a bunch of mock files. Can you remove the ones that are not relevant to this PR. Thanks!

[meringu]

Thanks @X-Guardian, I have cherry picked out a new commit, and only run go generate on the packages I modified.

[meringu]

Have updated the PR description to reflect the changes. The PR is labeled as needs-discussion and waiting-on-response, so please let me know how I can help.

[meringu]

Looks like tofudl is failing to compile on 32bit as this constant is too large: https://github.com/opentofu/tofudl/blob/76c6857d6eac665824b29da4e893e06e0b1896b2/branding/branding.go#L36

I'll look at raising a PR.

[abstractionfactory]

Just a few comments from the Tofu side, thank you for your work!

[meringu]

I believe this is ready for re-review (I don't have permissions to update the tags). I've left the OpenTofu mirror discussion open, If possible I would be keen to leave as is, and add extra flags to Atlantis once we have mirrors in use in the wild or someone requests this feature. This should give us a little more flexibility while the ecosystem matures.

My guess is the security vulnerabilities aren't related to my PR, as main is failing too. If a maintainer feels comfortable sharing the findings, I'd be happy to take a look in this PR or another.

[abstractionfactory]

Hey folks, anything we can do from the OpenTofu side to help move this one along?

[lukemassa]

I tested locally and it seems to work fine, and the code looks good to me!

Thanks for the contribution!

[jamengual]

Thanks @meringu and everyone that helped on this.

addressed