JavaScript Documentation: Improve the JS API documentation tool.

[youknowriad]

What problem does this address?

In Gutenberg we use a custom made tool called api-docs to parse the exported JS APIs and document them in the README files of each package and end up in a package in the block editor handbook (useSelect Example).

This is not very different from how the WordPress PHP APIs are extracted and documented on the developer docs.

But there are some subtle differences that make a big difference in terms of results:

• For the PHP APIs, each API has a dedicated page while the JS APIs are all in the same page and some packages have a big list of APIs impacting the reading and sharing experience.
• It is possible for users to comment on PHP API docs to share examples and clarify some important but maybe subtle things.
• I believe it WordPress documentation contributors have the possibility to augment the generated docs (from the code) and complete the information.

What is your proposed solution?

• Consider extracting each JS API to its own dedicated markdown in the handbook (We need to check whether to gitignore these or not). Each markdown file can become a dedicated package on the handbook by adding it to the manifest.json file.
• Consider enabling comments on each of these generated pages. (Probably requires a meta ticket). We already have dedicated pages for components in the handbook, so we may be able to explore this even without changing the output of the existing JS docs.
• Consider whether it's possible to provide a way for contributors to "augment" the generated markdown with a manually written markdown file for each JS API.

I also believe that some of our JS Docs are not great but having dedicated pages for each API will incentivize developers and documentation contributors to actually submit PRs and improve these. Today, these are hidden within a huge README file.

cc @oandregal @tomjn

[tomjn]

Given that we already have designs and templates for the PHP equivalent and post types, does it make sense to reinvent that for this? It seems counter intuitive that there be 2 separate code references on the devhub with differing capabilities.

E.g. the existing code reference can tell me a function was introduced in WP 6.5, likewise it'd be useful to see both PHP and JS functions/hooks when debugging:

This code reference is also plumbed into some existing tools so there'd be synergies

[oandregal]

cc @juanmaguitar for awareness, as we talked about this the other day.

Ideally, we should aim to incorporate the JavaScript APIs to The Reference, that only contains PHP for now. The Reference provides a lot more (focused search, list changes by WordPress version, etc.) and is the centralized place for this kind of thing. This track ticket contains references to past efforts https://meta.trac.wordpress.org/ticket/3063

For context, there are two pieces at work here:

• docgen: this tool generates markdown from the JSDoc comments of the exported entities in a given entry point. This is Gutenberg agnostic.
• api-docs scripts: these are specific to our repo. For example, the update-api-docs.js takes the README.md files of our packages and searchs for a given pattern (<!-- START TOKEN( and <!-- END TOKEN(). Anything in between those tags is substituted by the Markdown generated by docgen for that package.

[youknowriad]

For clarity, I'm not really opinionated on how we implement this. The only requirement I have technically is that the code base and the documentation stay together (in the code base), how it's generated and published is implementation detail.

[tomjn]

Would it help to have a minimum scope for what a single page for an item would contain? As well as what we intend it to apply to? E.g. components have been mentioned but does it make sense to include packages given they sort of already have that in the handbook?

[juanmaguitar]

I have been discussing this issue with @oandregal, and we concluded that the following actions could make a good plan of action to move this issue forward:

• Develop a custom --formatter for the docgen tool to represent the parsed info from JSDoc comments in the same JSON format expected by phpdoc-parser (see example)
  -Exploration: DocGen custom formatter to raw json and phpdoc-parser json formats #63202 (comment)
• Create custom taxonomies for these JS functions
  • wp-parser-js-function - to tag them as JS functions
  • wp-parser-npm-package - so we can list all JS functions of a NPM package
• "Inject" the JSON obtained by docgen into the parser process run via phpdoc-parser and import these functions as wp-parser-function Custom Post Type (the same CPT used for PHP functions)
  • assigning to all these parsed JS functions the wp-parser-js-function taxonomy (so they can be listed apart and somehow styled differently)
  • assigning to each JS function the proper wp-parser-npm-package taxonomy value

This work would affect code at the following repos:

• gutenberg - for the custom output of the parsed JSDoc info
• wporg-developer-2023 theme - for the custom taxonomies registration
• phpdoc-parser plugin - for inserting the custom JSON and adapt the code so this extra functions are properly inserted as CPT with their proper taxonomies

This plan would enable the same functionality for JavaScript (JS) functions available for PHP functions at https://developer.wordpress.org/reference/functions. It would also allow for distinguishing between JS functions and PHP functions.

After this first milestone, we could expand the functionality of this process to also include relationships between JS functions in the same way we have them for PHP functions.

[dmsnell]

At WCEU @juanmaguitar and @adamziel and I discussed calling out to the TypeScript language server from the phpdoc-parser process where the Gutenberg source code is symlinked or otherwise available in the WordPress installation.

This would allow the PHP code to generate the docs from the JS files and make a clean integration - the JS process wouldn't have to match formats, the would be no JS side to this at all.

Language server calls are XML-RPC calls over HTTP so shouldn't be a technical obstacle on the PHP side. We can iterate over all the files, get all the symbols, get all the docblocks attached to them, and get the inferred types whether JS or TS.

I think this is a valuable avenue to explore as it collapses a lot of complexity into a project that already understands the Gutenberg codebase enough to provide IDE inline documentation, jumping, and autocomplete - the TypeScript Language Server.

As discussed this has interesting tie-ins to using php itself with ReflectionClass/ReflectionFunction/etc… introspection which could eliminate the need to depend on nikic/php-parser and provide more knowledge than we currently have, such as resolved namespaces. (A remaining problem dependent on the parser would be inferring filter variable placeholders).

Just wanted to leave the notes so this doesn't get lost as I think it's a viable path to explore to improve our docs generation and simplify it at the same time.