Better operator docs

[fhennig]

This ticket concerns the docs in the operator repositories, not actually this repository.

We want to follow the Explanation/Tutorial/Guide/Reference structure for our docs, and the operators currently do not do that.

What we have now

Currently we often have these pages:

• index: Landing page
• installation: contains information on installing via Helm, via Docker and from source
• configuration: contains information in commandline arugments and environment variables
• usage: This is typically the largest page with many sections explaining various sections of the CRD
• concepts: Contains explanation on operator specific stuff
  • discovery
• CRD reference (sometimes): if it is there, it's good. explains all CRD properties

What's good

• We have an install section, and typically we have a small example of a simple cluster in the usage section.
• The usage section outlines all things possible with the CRD, that's good
• We have a configuration section that shows how the operator can be configured with env vars and commandline arguments
• sometimes we have a CRD reference, that's a good thing
• sometimes we have "concepts", i.e. for zookeeper to explain the ZNode and the discovery profiles

What's not so good

• A clear place for reference is often missing. By that I also mean something that is called "reference". The information is already there in part though, in the "Configuration" section and a bit in the "Usage" section. Also in the CRD reference section if there is one.
• A dedicated "getting started" section is missing, which I think would be the first thing I'd look for as a new user. The install section is a good start, but it doesn't tell me what to do once the operator is installed. However we can take some bits from the usage and expand on that.
• I believe a lot of the information in usage is not as detailed as it needs to be to get to a production ready setup. I think we need to expand this. I believe it would be good, to even have seperate documents. For example for Druid:
  • One page on OPA authorization. All the authorization options Druid supports and also how to connect to an already existing OPA.
  • One page on deep storage, including details on how to connect to an existing HDFS cluster and how to set up deep storage in S3 on different providers
  • One page on connecting to ZooKeeper, incuding on how to connect to an already existing instance
  • This should be a guides section
• Explanation of how the operator works and what it does. This would be good to have so people know what is actually going on. It is also common in other operators out there to explain this.

What it would like like in the end

• quickstart: The Quickstart should contain a simple line to install all the operators needed at once (or maybe a full release) and then a command to install a demo stack. And then some instructions for "hello world usage" (a simple example to verify that the deployed service works)
• Getting Started: This should cover in detail how to get started with this operator. It should contain the following:
  • a reference to a page describing how to set up k8s
  • an install page with various install options (and maybe edge cases considered) (done now)
• explanation/concepts
  • general explanation of what the operator does, objects it creates and why etc. Operator overview pages #282
  • the roles and what they mean
  • special things like the DruidConnection in superset or the ZNode in ZooKeeper
  • explicitly not discovery or other platform-wide concepts. Only link to that.
• How-To Guides Guides for all operators #283
  • ~~deploying an XYCluster (KafkaCluster, DruidCluster, ...) and connecting to it
  • But more detailed guides there: I.e. Kafka: How to connect to a ZooKeeper cluster. This should also include information on how to connect to your own ZooKeeper cluster, if there is already one running. There should be a guide for each dependency.
  • a guide for discovery?
  • A guide to configure i.e. deep storage with HDFS for Druid. specific things (this will be expanded from the usage page), "Working with TrinoCatalogs", "Authentication and Authorization"
• Reference: Reference pages for all operators #284
  • The command line parameters the operator accepts (mostly there already)
  • The environment variables the operator honors (mostly there already)
  • For every CR that it defines:
    • Every field should be documented
    • role specifics should be pointed out too
  • the data fields in the discovery configmap
• release notes: User facing changelog like it is done in superset

Example for Trino

Trino Operator

• Getting Started
  • Installation
  • First Steps
• Concepts
  • Overview: Big picture, how everything relates to each other, what we create, what you create
  • TrinoCluster Resource (Roles explained here)
  • TrinoCatalog
• Guides (How do I do XY?)
  • Security
    • Encryption
    • Authentication (LDAP)
    • Authorization (OPA)
  • Working with catalogues
  • Configuration Overrides
• Reference
  • Resources
    • TrinoCluster
    • TrinoCatalogue
  • Discovery ConfigMap
  • Commandline
  • Environment Variables

[fhennig]

We had a discussion on this ticket and this was the outcome:

• We all agreed on having a dedicated Getting started section for the operators. It should include
  • Installation - how to install the operator (we have that already)
  • "My first product instance" - how to set up a basic instance
  • Check if it works - some way to interact with the product or run some "Hello World" task in it to see if it works
• We came to some important conclusions
  • There shouldn't be redundancy
  • CRDs used in Guides and Tutorials should be automatically tested in some way
  • The diataxis framework seemed like a good idea to all of us
• For other things we discussed, more research might be needed
  • CRD Reference: How can we generate this? (Also, this is not the highest priority right now)
  • For more extensive guides: How can we automatically test the Manifest files etc. If they need to be adapted manually, it'll become hard to maintain (Idea about this: Test operator repository examples via unit tests issues#248)
  • Can we use templating in some capacity for the documentation? I've tried it with the installation page, but I'd like to include operator specific information too. I think templating doesn't work very well for documentation pages.

I have created some tickets: #237, #238, #239, #240

[fhennig]

To make snippets in usage testable, there was this idea: stackabletech/issues#248

[fhennig]

Update

Getting Started Guides have been implemented, but a lot of stuff in the initial post is still not up to par. I'd like to revisit this and keep going.

[fhennig]

This is mostly done, I've moved some comments and links here: #505