[Doc] Document Core Extension Points

[nknize]

Is your feature request related to a problem? Please describe.

The core has a plugin API designed to enable plugins to extend features of the core. This plugin API guarantees a deprecation/removal path for supporting backward compatibility of core extension points across releases. Despite public, protected, private modifiers any core implementation beyond the plugin API (that is, not directly in the namespace or used by code in the namespace - with the exception of the public facing REST API) does not guarantee backward compatibility! That goes for the core aggregations, field mappers, query framework (including built in scoring), and internal engine components (not already available in EngineConfigFactory).

The problem is, other plugins are extending and depending on internal methods of the core implementation that do not guarantee backward compatibility and expecting backward compatibility.

Describe the solution you'd like

Clear documentation of what core components will guarantee backward compatibility, and what core components should not be extended by the core. Anything beyond can be developed by the community as an enhancement to the plugin API.

Describe alternatives you've considered

We are currently working on an extensibility implementation that will help make it clearer on how to extend the core.

Additional context

The scope of this effort is just to clearly document what core components will guarantee backward compatibility across releases. Included with this issue may be a meta issue to clean up core javadocs to include documentation in code for components that will not provide bwc.

[dblock]

So would we be wasting our time documenting extension points that can't be enforced? We'd still need to be super disciplined in guaranteeing backwards-compatibility and we will always find some new plugin that had expectations around it that weren't met in the past.

Today, it would be safer to assume that everything public is a potentially used extension point, needs to be backwards compatible per semver (have proper deprecation). Then we should be concentrating our efforts on an extracted SDK (#2230).

[nknize]

documenting extension points that can't be enforced?

They can and should be enforced even with an extracted "SDK". @setiah started working that when we first created the fork by creating the javadoc enforcement task based on Lucene's javadoc checker. We just haven't followed up and properly added the javadocs. What this will do is provide a mechanism to fail precommit if anything protected+ does not have javadocs. Yes, it's true developers will blindly slap a javadoc just to get the test to pass but it's a next step better than what we have today (nothing) for building muscle memory around being thoughtful re: modifiers.

Today, it would be safer to assume that everything public is a potentially used extension point, needs to be backwards compatible per semver

It's good practice to think this way when developing, yes; and if something shouldn't be extended (e.g., most everything in the Engine) then diligently use private / package private.