[New Feature] Extending build-tool integration with subcommands, central delivered tools and adding to Dependencies.toml

[gayaldassanayake]

Description

Currently, Ballerina provides bal tool, a way for developers to extend the CLI functionalities by introducing new commands. These tools/ commands are shipped as packages in Ballerina Central and are managed by the inbuilt bal tool command.

With update 9, we are also allowing tools to be integrated with the build command. They will run as a pre-build task. (Design doc - https://docs.google.com/document/d/1XsZK3UlqXv4TLEgRKVc0UBRRilLTmLZ1heDTmIRuylE/edit?usp=sharing).

Describe your problem(s)

1. However, the build integrated tools need to be packed in the distribution(bre/lib) to be used. There is no way to externally publish and use the build integrated tools.
2. We need to support integration of subcommand of tools (eg:- health.fhir) with build.
3. For predictable builds, we need to lock tool versions in Dependencies.toml

Describe your solution(s)

Use the existing strategy of delivering bal tools to distribute the tools integrated with build.
Design doc - https://docs.google.com/document/d/1XsZK3UlqXv4TLEgRKVc0UBRRilLTmLZ1heDTmIRuylE/edit#heading=h.3odx7sx4desy

Related area

-> Compilation

Related issue(s) (optional)

#41415

Suggested label(s) (optional)

No response

Suggested assignee(s) (optional)

No response

Related issues

[gayaldassanayake]

Following are the tasks that have to be completed under this feature.

Phase 1

• Picking tools from a URL class loader
• Dependency version resolution
• Automatically pulling tools
• Restrict tools from having a DOT in id
  • In pack
  • In push
  • Central
  • Tool Loading
• Recording in Dependencies.toml
• Supporting subcommands
• Test tool resolution endpoint (Blocked on endpoint implementation)
• Change the getName() return type to String[]
• Introduce offline resolution
• Introduce a print API for the developers
• Introduce @ToolConfig annotation
• Testing
  • Manual testing of all scenarios
  • Lang tests
  • Distribution tests
  • Full build pipeline
  • Peer testing
  • VSCode functionality testing (manually)
• Documentation
  • Update the developer guide
  • Website documentation
  • Release notes

Phase 2

• Minimum compatibility version
• Local repo support

[gayaldassanayake]

The initial PR has been merged. The remaining tasks are,

• Improve the code coverage
• Reuse code from package resolution - [Improvement]: Reuse the package resolution and bal tool utility methods in Build tool logic #42348
• Add distribution tests
• Minimum compatibility version (Phase 2)
• Local repo support (Phase 2)