Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

helper/schema: Provide Functionality to Reduce Resident Memory Usage from Schemas #1217

Closed
bflad opened this issue Jun 23, 2023 · 1 comment · Fixed by #1218
Closed

helper/schema: Provide Functionality to Reduce Resident Memory Usage from Schemas #1217

bflad opened this issue Jun 23, 2023 · 1 comment · Fixed by #1218
Assignees
Labels
enhancement New feature or request
Milestone

Comments

@bflad
Copy link
Contributor

bflad commented Jun 23, 2023

SDK version

v2.26.1

Use-cases

This Go module caches the entire provider definition within the provider server, including all data and managed resource schema data via the Resource type having a Schema field directly referencing the map[string]*schema.Schema data. At sufficient provider scale, by nature of many resources and/or large schemas within resources, resident memory usage permanently increases without a way for provider developers to avoid the issue.

Attempted Solutions

Migrate resources to terraform-plugin-framework, which schema data is already behind a method call. This ensures the provider server can treat the schema data as eligible for garbage collection immediately after usage (minus any internal caching the framework itself may perform).

Proposal

Introduce a Resource type SchemaFunc field with a func() map[string]*Schema type, which would store a function pointer for the schema data in memory rather than the data itself. To prevent consumers from needing to check both the old and new schema data fields, introduce a new Resource type SchemaMap method which returns map[string]*Schema and handles either calling the SchemaFunc, if defined, or returning the prior Schema` data.

The Resource type Schema field should NOT be marked as deprecated, as this would create a major ecosystem burden with little benefit for most use cases.

The Resource type InternalValidate method should be updated to return an error if both Schema and SchemaFunc are defined.

References

@bflad bflad added the enhancement New feature or request label Jun 23, 2023
@bflad bflad added this to the v2.27.0 milestone Jun 23, 2023
@bflad bflad self-assigned this Jun 23, 2023
bflad added a commit that referenced this issue Jun 23, 2023
Reference: #1217

This change introduces a new `SchemaFunc` field and `SchemaMap` method to the `Resource` type. Currently, the `Schema` field data of all `Resource` is held in memory for the lifecycle of the provider server, which is problematic for providers with many resources and/or larger schemas in resources. The new field enables provider developers to swap pieces of resident memory usage for the slight additional computation time of reconstituting the data when necessary.

Callers directly referencing the exported `Schema` field should switch to referencing the `SchemaMap` method, which returns the result of `SchemaFunc` or `Schema` in that preference order. To ensure internal usage was migrated to the new method, this change was performed by temporarily commenting out the `Schema` field itself with broken references in non-testing code migrated to the method. The `Schema` field is not marked as deprecated via Go documentation comment as this would introduce a major ecosystem burden to migrate with generally little benefit for most use cases.

The `Resource` type `InternalValidate` method has been updated to return an error if both `Schema` and `SchemaFunc` are defined.

Provider developers are encouraged to migrate resources to the terraform-plugin-framework, as it already behaves in a manner similar to `SchemaFunc` by nature of resource schema data being behind a method call, amongst many of the other benefits outlined at https://developer.hashicorp.com/terraform/plugin/framework-benefits.
bflad added a commit that referenced this issue Jun 28, 2023
Reference: #1217

This change introduces a new `SchemaFunc` field and `SchemaMap` method to the `Resource` type. Currently, the `Schema` field data of all `Resource` is held in memory for the lifecycle of the provider server, which is problematic for providers with many resources and/or larger schemas in resources. The new field enables provider developers to swap pieces of resident memory usage for the slight additional computation time of reconstituting the data when necessary.

Callers directly referencing the exported `Schema` field should switch to referencing the `SchemaMap` method, which returns the result of `SchemaFunc` or `Schema` in that preference order. To ensure internal usage was migrated to the new method, this change was performed by temporarily commenting out the `Schema` field itself with broken references in non-testing code migrated to the method. The `Schema` field is not marked as deprecated via Go documentation comment as this would introduce a major ecosystem burden to migrate with generally little benefit for most use cases.

The `Resource` type `InternalValidate` method has been updated to return an error if both `Schema` and `SchemaFunc` are defined.

Provider developers are encouraged to migrate resources to the terraform-plugin-framework, as it already behaves in a manner similar to `SchemaFunc` by nature of resource schema data being behind a method call, amongst many of the other benefits outlined at https://developer.hashicorp.com/terraform/plugin/framework-benefits.
@github-actions
Copy link

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Jul 29, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant