Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(schema): add API descriptors, struct, oneof & list types, and wire encoding spec #21482

Merged
merged 29 commits into from
Oct 1, 2024

Conversation

aaronc
Copy link
Member

@aaronc aaronc commented Aug 30, 2024

Description

This PR is defines additional kinds and types in cosmossdk.io/schema without implementing support yet:

  • UIntN and IntN for larger fixed width integers
  • struct and oneof types
  • API descriptors for eventual use with crosslang

Deterministic binary key and value encoding formats are also specified.

This PR is intended as a code-based RFC/ADR on eventual features of schema.


Author Checklist

All items are required. Please add a note to the item if the item is not applicable and
please add links to any relevant follow up issues.

I have...

  • included the correct type prefix in the PR title, you can find examples of the prefixes below:
  • confirmed ! in the type prefix if API or client breaking change
  • targeted the correct branch (see PR Targeting)
  • provided a link to the relevant issue or specification
  • reviewed "Files changed" and left comments if necessary
  • included the necessary unit and integration tests
  • added a changelog entry to CHANGELOG.md
  • updated the relevant documentation or specification, including comments for documenting Go code
  • confirmed all CI checks have passed

Reviewers Checklist

All items are required. Please add a note if the item is not applicable and please add
your handle next to the items reviewed if you only reviewed selected items.

Please see Pull Request Reviewer section in the contributing guide for more information on how to review a pull request.

I have...

  • confirmed the correct type prefix in the PR title
  • confirmed all author checklist items have been addressed
  • reviewed state machine logic, API design and naming, documentation is accurate, tests and test coverage

Summary by CodeRabbit

  • New Features

    • Introduced a structured API descriptor system for enhanced API management and compatibility.
    • Added support for "oneof" constructs, allowing for flexible data structure definitions.
    • Launched a new representation for struct types, improving management of fields and compatibility rules.
  • Documentation

    • Enhanced documentation across various components, clarifying field behaviors and encoding specifications for better developer understanding.

Copy link
Contributor

coderabbitai bot commented Aug 30, 2024

Warning

Rate limit exceeded

@tac0turtle has exceeded the limit for the number of commits or files that can be reviewed per hour. Please wait 14 minutes and 57 seconds before requesting another review.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

📥 Commits

Files that changed from the base of the PR and between c1a0ccd and bf33506.

📝 Walkthrough
📝 Walkthrough

Walkthrough

The changes introduce a structured API descriptor system, enhancing compatibility management for APIs. New types and fields are defined across several files, including APIDescriptor, MethodDescriptor, and Field, with updated documentation clarifying their usage. The OneOfType and StructType are also introduced, facilitating the implementation of complex data structures. Overall, the modifications enhance the schema's capability to handle various data types and ensure compatibility across different versions.

Changes

Files Change Summary
schema/api.go Introduced APIDescriptor, MethodDescriptor, and Volatility types, detailing API versioning, method characteristics, and compatibility rules.
schema/field.go Enhanced Field struct with new ElementKind and Size fields, and updated documentation on ReferencedType for clarity regarding field kinds.
schema/kind.go Expanded Kind type definitions with detailed encoding specifications for various data types, introducing new kinds like UIntNKind, StructKind, etc.
schema/object_type.go Updated comments in ObjectType struct to clarify constraints on KeyFields and ValueFields, emphasizing compatibility implications of modifications.
schema/oneof.go New definitions for OneOfType, OneOfCase, and OneOfValue to implement "oneof" constructs, ensuring type safety and compatibility in schema design.
schema/struct.go Introduced StructType to represent struct types, including fields for name, fields list, and sealing status, with rules for modifying sealed/unsealed structs.

Sequence Diagram(s)

sequenceDiagram
    participant Client
    participant API
    participant Descriptor

    Client->>API: Request API Information
    API->>Descriptor: Fetch APIDescriptor
    Descriptor-->>API: Return APIDescriptor
    API-->>Client: Provide API Information
Loading
sequenceDiagram
    participant Client
    participant OneOfType
    participant OneOfCase

    Client->>OneOfType: Define OneOf Type
    OneOfType->>OneOfCase: Add Case
    OneOfCase-->>OneOfType: Case Added
    OneOfType-->>Client: OneOf Type Defined
Loading

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

❤️ Share
🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>, please review it.
    • Generate unit testing code for this file.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai generate unit testing code for this file.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read src/utils.ts and generate unit testing code.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
    • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

@aaronc aaronc marked this pull request as ready for review September 4, 2024 17:42
@aaronc aaronc requested review from testinginprod and a team as code owners September 4, 2024 17:42
Copy link
Contributor

github-actions bot commented Sep 4, 2024

@aaronc your pull request is missing a changelog!

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

Review details

Configuration used: .coderabbit.yml
Review profile: CHILL

Commits

Files that changed from the base of the PR and between 5805792 and c1a0ccd.

Files selected for processing (6)
  • schema/api.go (1 hunks)
  • schema/field.go (1 hunks)
  • schema/kind.go (4 hunks)
  • schema/object_type.go (1 hunks)
  • schema/oneof.go (1 hunks)
  • schema/struct.go (1 hunks)
Additional context used
Path-based instructions (6)
schema/struct.go (1)

Pattern **/*.go: Review the Golang code for conformity with the Uber Golang style guide, highlighting any deviations.

schema/oneof.go (1)

Pattern **/*.go: Review the Golang code for conformity with the Uber Golang style guide, highlighting any deviations.

schema/field.go (1)

Pattern **/*.go: Review the Golang code for conformity with the Uber Golang style guide, highlighting any deviations.

schema/object_type.go (1)

Pattern **/*.go: Review the Golang code for conformity with the Uber Golang style guide, highlighting any deviations.

schema/api.go (1)

Pattern **/*.go: Review the Golang code for conformity with the Uber Golang style guide, highlighting any deviations.

schema/kind.go (1)

Pattern **/*.go: Review the Golang code for conformity with the Uber Golang style guide, highlighting any deviations.

Additional comments not posted (24)
schema/struct.go (3)

1-2: LGTM!

The package declaration is correct and no imports are needed at this point.


3-21: LGTM!

The StructType definition and its documentation comments are well-written and informative. The comments clearly mention the current implementation status and provide a good explanation of the type's purpose and the usage of its fields. The doc comments follow the Golang conventions.


1-21: Code conforms to the Uber Golang style guide.

The code follows the Uber Golang style guide for naming conventions, doc comments, and formatting. There are no deviations from the style guide.

schema/oneof.go (3)

37-43: LGTM!

The OneOfValue definition looks good and is straightforward.


5-19: LGTM!

The OneOfType definition looks good. The comments provide clear instructions on compatible and incompatible changes.

Verify the allowed values for DiscriminantKind.

Ensure that the code that sets the DiscriminantKind field only allows the values mentioned in the comment: Uint8Kind, Int8Kind, Uint16Kind, Int16Kind, or Int32Kind.

Run the following script to verify the allowed values:

Verification successful

No issues found with DiscriminantKind usage.

The DiscriminantKind field is defined but not currently used or assigned in the codebase. There are no issues with the allowed values since it is not being utilized. If this field is intended for future use, ensure that only the specified values are allowed when it is implemented.

Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Description: Verify the allowed values for `DiscriminantKind`.

# Test: Search for assignments to `DiscriminantKind`. Expect: Only allowed values.
rg --type go -A 5 $'DiscriminantKind ='

Length of output: 40


Script:

#!/bin/bash
# Description: Search for the usage of `DiscriminantKind` in a broader context.

# Search for `DiscriminantKind` in struct initializations and function parameters.
rg --type go 'DiscriminantKind'

Length of output: 146


22-34: LGTM!

The OneOfCase definition looks good. The comments provide clear instructions on the fields.

Verify the restriction on the Kind field.

Ensure that the code that sets the Kind field does not allow ListKind.

Run the following script to verify the restriction:

schema/field.go (1)

18-18: LGTM!

The updated comment provides more clarity on the usage of the ReferencedType field.

schema/object_type.go (2)

12-19: LGTM!

The updated comments for the KeyFields attribute provide clearer guidance on the constraints and implications of modifying the attribute. The changes are approved.


25-28: LGTM!

The updated comments for the ValueFields attribute provide clearer guidance on the constraints and implications of modifying the attribute. The changes are approved.

schema/api.go (3)

41-50: LGTM!

The APIDescriptor type is well-defined and the comments provide clear guidelines on compatibility and encoding.


53-85: LGTM!

The MethodDescriptor type is well-defined and the comments provide clear guidelines on compatibility and encoding.


88-98: LGTM!

The Volatility type and its constants are well-defined and the comments provide clear explanations.

schema/kind.go (12)

13-31: Documentation changes look great!

The expanded documentation for the Kind type provides crucial encoding specifications for each kind, enhancing clarity and completeness. This will help developers properly encode and decode data in various formats.


41-44: StringKind encoding documentation looks good!

The expanded documentation for StringKind provides clear specifications for key binary and value binary encodings. The distinction between non-terminal and terminal forms in the key binary encoding is important. Reusing the BytesKind value binary encoding promotes consistency.


51-58: BytesKind encoding documentation looks good!

The expanded documentation for BytesKind provides clear specifications for key binary and value binary encodings. The use of length prefixes in the non-terminal key binary encoding allows for proper decoding. The value binary encoding using offset and length integers provides flexibility for referencing the actual bytes.


64-65: Numeric kind encoding documentation looks good!

The expanded documentation for the numeric kinds (Int8Kind, Uint8Kind, Int16Kind, Uint16Kind, Int32Kind, Uint32Kind, Int64Kind, Uint64Kind) provides clear and consistent specifications for key binary and value binary encodings.

The use of big-endian encoding for keys allows for proper sorting. The first bit inversion technique for signed integer keys is clever for maintaining sort order. The little-endian encoding for values is a common choice for numeric data.

Also applies to: 71-72, 78-79, 85-86, 92-93, 99-100, 107-108, 115-116


123-124: IntegerStringKind and DecimalStringKind JSON encoding documentation looks good!

The expanded documentation for IntegerStringKind and DecimalStringKind provides clear specifications for JSON encoding using base10 integer and decimal strings, respectively. The use of IntegerFormat and DecimalFormat regexes ensures format conformance.

The canonical encoding rules, such as no leading zeros for integers and specific exponential notation rules for decimals, promote consistency and ease of parsing.

Also applies to: 133-134


140-141: BoolKind encoding documentation looks good!

The expanded documentation for BoolKind provides a clear and concise specification for key binary and value binary encodings. The use of a single byte to represent boolean values, with 0 for false and 1 for true, is a common and efficient approach.


151-152: TimeKind and DurationKind encoding documentation looks good!

The expanded documentation for TimeKind and DurationKind provides clear specifications for JSON, key binary, and value binary encodings.

For JSON encoding, the use of ISO 8601 timestamp strings for TimeKind ensures interoperability and ease of parsing. The canonical encoding rules promote consistency. The DurationKind JSON encoding using decimal strings with 's' suffix is straightforward.

The binary encoding specifications are consistent with the encoding of other numeric kinds, ensuring proper sorting and efficient representation.

Also applies to: 160-161


167-168: Float32Kind and Float64Kind encoding documentation looks good!

The expanded documentation for Float32Kind and Float64Kind provides clear specifications for key binary and value binary encodings. The use of IEEE-754 encoding, with 4 bytes for Float32Kind and 8 bytes for Float64Kind, is the standard representation for floating-point numbers, ensuring interoperability and precision.

Also applies to: 174-175


185-188: AddressKind encoding documentation looks good!

The expanded documentation for AddressKind provides clear specifications for key binary and value binary encodings. The use of a 1-byte length prefix allows for efficient encoding and decoding of variable-length addresses. The distinction between non-terminal and terminal forms in the key binary encoding is important for proper encoding.

The maximum supported length of 63 bytes for addresses provides a reasonable limit for address sizes.


196-197: EnumKind encoding documentation looks good!

The expanded documentation for EnumKind provides a clear specification for key binary and value binary encodings. By using the same binary encoding as the EnumType's numeric kind, it ensures consistency and compatibility with the underlying enum representation.


203-204: JSONKind encoding documentation looks good!

The expanded documentation for JSONKind provides a clear specification for key binary and value binary encodings. Using string encoding for both key binary and value binary encodings is a straightforward and interoperable approach for representing JSON data.


206-216: UIntNKind addition looks good, but clarification needed on implementation timeline.

The addition of UIntNKind provides flexibility for representing unsigned integers with arbitrary bit widths. The documentation clearly specifies the encoding formats for Go, JSON, and binary representations, which is consistent with the encoding schemes used for other numeric kinds.

However, the documentation mentions that support for UIntNKind is currently unimplemented.

Can you please clarify the timeline for implementing support for UIntNKind? It would be helpful to know when this kind will be usable in practice.

Comment on lines +21 to +23
// ElementKind is the element type when Kind is ListKind.
// Support for this is currently UNIMPLEMENTED, this notice will be removed when it is added.
ElementKind Kind `json:"element_kind,omitempty"`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add implementation for the ElementKind field.

The ElementKind field seems to be a useful addition for representing the element type of a list. However, the implementation is currently missing.

Please add the implementation for the ElementKind field and remove the notice about the missing implementation from the comment. Let me know if you need any help with the implementation.

Comment on lines +25 to +32
// Size specifies the size or max-size of a field.
// Support for this is currently UNIMPLEMENTED, this notice will be removed when it is added.
// Its specific meaning may vary depending on the field kind.
// For IntNKind and UintNKind fields, it specifies the bit width of the field.
// For StringKind, BytesKind, AddressKind, and JSONKind, fields it specifies the maximum length rather than a fixed length.
// If it is 0, such fields have no maximum length.
// It is invalid to have a non-zero Size for other kinds.
Size uint32 `json:"size,omitempty"`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add implementation for the Size field.

The Size field seems to be a useful addition for specifying the size or max-size of a field. The comment provides a clear explanation of the field's usage for different field kinds. However, the implementation is currently missing.

Please add the implementation for the Size field and remove the notice about the missing implementation from the comment. Let me know if you need any help with the implementation.


// OutputFields is the list of output fields for the method.
//
// It is a COMPATIBLE change to add new output fields to a method,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think for now we should amend this until we have strong guarantees that it cannot be consensus breaking.

Copy link
Contributor

@testinginprod testinginprod left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@tac0turtle tac0turtle added this pull request to the merge queue Oct 1, 2024
Merged via the queue into main with commit 73ee336 Oct 1, 2024
71 of 72 checks passed
@tac0turtle tac0turtle deleted the aaronc/schema-api-types branch October 1, 2024 07:46
alpe added a commit that referenced this pull request Oct 1, 2024
* main:
  docs: amend docs for 52 changes  (#21992)
  test: migrate e2e/authz to system tests (#21819)
  refactor(runtime/v2): use StoreBuilder (#21989)
  feat(schema): add API descriptors, struct, oneof & list types, and wire encoding spec (#21482)
  docs: add instructions to change DefaultGenesis (#21680)
  feat(x/staking)!: Add metadata field to validator info (#21315)
  chore(x/authz)!: Remove account keeper dependency (#21632)
alpe added a commit that referenced this pull request Oct 1, 2024
* main:
  docs: amend docs for 52 changes  (#21992)
  test: migrate e2e/authz to system tests (#21819)
  refactor(runtime/v2): use StoreBuilder (#21989)
  feat(schema): add API descriptors, struct, oneof & list types, and wire encoding spec (#21482)
  docs: add instructions to change DefaultGenesis (#21680)
  feat(x/staking)!: Add metadata field to validator info (#21315)
  chore(x/authz)!: Remove account keeper dependency (#21632)
  chore(contributing): delete link (#21990)
  test(gov): Migrate e2e to system test (#21927)
  test: e2e/client to system tests (#21981)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants