chore: docs rework (#927)

Total web-docs overhaul:

- puts a helpful intro at the start, with a demo on the next page
- adds an "architecture" page which discusses in-process vs rpc
- adds a deployment page with mentions how to install/run
- adds a "reference" section which includes:
  - specs
  - protos (in the form of autogenerated markdown for easy navigation)
  - autogenerated CLI docs
- adds FAQ

---------

Signed-off-by: Todd Baert <todd.baert@dynatrace.com>
Co-authored-by: Michael Beemer <beeme1mr@users.noreply.github.com>

.github/workflows/build.yaml

@@ -8,14 +8,12 @@ on:
     paths-ignore:
       - "README.md"
       - "docs/**"
-      - "web-docs/**"
   pull_request:
     branches:
       - main
     paths-ignore:
       - "README.md"
       - "docs/**"
-      - "web-docs/**"
 
 env:
   GO_VERSION: '1.20'

.gitignore

@@ -14,4 +14,5 @@ go.work.sum
 bin/
 
 # built documentation
-site
+site
+.cache/

.gitmodules

@@ -4,3 +4,6 @@
 [submodule "spec"]
 	path = spec
 	url = https://github.com/open-feature/spec.git
+[submodule "schemas"]
+	path = schemas
+	url = https://github.com/open-feature/schemas.git

.markdownlint-cli2.yaml

@@ -11,9 +11,14 @@ config:
       - br
   github-admonition: true
   max-one-sentence-per-line: true
+  code-block-style: false         # not compatible with mkdocs "details" panes
 
 ignores:
   - "**/CHANGELOG.md"
   - "docs/specification"
   - "node_modules"
   - "tmp"
+  - "**/protos.md"  # auto-generated
+  - "schemas"       # submodule
+  - "spec"          # submodule
+  - "test-harness"  # submodule

Makefile

@@ -9,6 +9,8 @@ ZD_CLIENT_IMG ?= zd-client:latest
 FLAGD_PROXY_IMG ?= flagd-proxy:latest
 FLAGD_PROXY_IMG_ZD ?= flagd-proxy:zd
 
+DOCS_DIR ?= docs
+
 workspace-init: workspace-clean
 	go work init
 	$(foreach module, $(ALL_GO_MOD_DIRS), go work use $(module);)
@@ -105,4 +107,19 @@ markdownlint:
 	$(MDL_CMD) davidanson/markdownlint-cli2-rules:$(MDL_DOCKER_VERSION) "**/*.md" 
 
 markdownlint-fix:
-	$(MDL_CMD) --entrypoint="markdownlint-cli2-fix" davidanson/markdownlint-cli2-rules:$(MDL_DOCKER_VERSION) "**/*.md" 
+	$(MDL_CMD) --entrypoint="markdownlint-cli2-fix" davidanson/markdownlint-cli2-rules:$(MDL_DOCKER_VERSION) "**/*.md"
+
+.PHONY: pull-schemas-submodule
+pull-schemas-submodule:
+	git submodule update schemas
+
+.PHONY: generate-proto-docs
+generate-proto-docs: pull-schemas-submodule
+	docker run --rm -v ${PWD}/$(DOCS_DIR)/reference/specifications:/out -v ${PWD}/schemas/protobuf:/protos pseudomuto/protoc-gen-doc --doc_opt=markdown,protos-with-toc.md schema/v1/schema.proto sync/v1/sync_service.proto \
+	&& echo '<!-- WARNING: THIS DOC IS AUTO-GENERATED. DO NOT EDIT! -->' > ${PWD}/$(DOCS_DIR)/reference/specifications/protos.md \
+	&& sed '/^## Table of Contents/,/#top/d' ${PWD}/$(DOCS_DIR)/reference/specifications/protos-with-toc.md >> ${PWD}/$(DOCS_DIR)/reference/specifications/protos.md \
+	&& rm -f ${PWD}/$(DOCS_DIR)/reference/specifications/protos-with-toc.md
+
+.PHONY: run-web-docs
+run-web-docs: generate-docs generate-proto-docs
+	docker run --rm -it -p 8000:8000 -v ${PWD}:/docs squidfunk/mkdocs-material

README.md

@@ -21,7 +21,7 @@
 
 ## What's flagd?
 
-Flagd is a feature flag daemon with a Unix philosophy. Think of it as a ready-made, open source, OpenFeature compliant feature flag backend system.
+Flagd is a feature flag daemon with a Unix philosophy. Think of it as a ready-made, open source, OpenFeature-compliant feature flag backend system.
 
 ## Features
 
@@ -87,7 +87,7 @@ Experiment with flagd in your browser using [the Killercoda tutorial](https://ki
       --uri file:./etc/flagd/example_flags.flagd.json
     ```
 
-    `--uri` can be a local file or any remote endpoint. Use `file:` prefix for local files. eg. `--uri file:/path/to/example_flags.flagd.json`. `gRPC` and `http` have their own requirements. More information can be found [here](docs/configuration/configuration.md#uri-patterns).
+    `--uri` can be a local file or any remote endpoint. Use `file:` prefix for local files. eg. `--uri file:/path/to/example_flags.flagd.json`. `gRPC` and `http` have their own requirements. More information can be found [here](./docs/reference/sync-configuration.md#uri-patterns).
 
     Multiple `--uri` parameters can be specified. In other words, flagd can retrieve flags from multiple sources simultaneously.
 
@@ -120,7 +120,7 @@ Experiment with flagd in your browser using [the Killercoda tutorial](https://ki
 
     Updates to the underlying flag store (e.g. JSON file) are reflected by flagd in realtime. No restarts required.
 
-    flagd also supports boolean, integer, float and object flag types. Read more on the [evaluation examples page](docs/usage/evaluation_examples.md)
+    flagd also supports boolean, integer, float and object flag types.
 
 4. Now that flagd is running, it is time to integrate into your application. Do this by using [an OpenFeature provider in a language of your choice](https://github.com/open-feature/flagd/blob/main/docs/usage/flagd_providers.md).
 
@@ -130,7 +130,7 @@ Experiment with flagd in your browser using [the Killercoda tutorial](https://ki
 
 ## 📝 Further Documentation
 
-Further documentation including flagd configuration options, fractional evaluation, targeting rules and flag configuration merging strategies can be found [on this page](docs/README.md).
+Further documentation including flagd configuration options, fractional evaluation, targeting rules and flag configuration merging strategies can be found at [flagd.dev](https://flagd.dev/), or [in this repository](./docs/index.md).
 
 ## 🫶 Contributing
 

docs/architecture.md

@@ -0,0 +1,45 @@
+# Architecture
+
+flagd architectures fall into two broad categories: those where the evaluation engine is deployed in a standalone process to which the client application connects ([RPC](#rpc-evaluation)), and those where the evaluation engine is embedded into the client application ([in-process](#in-process-evaluation)).
+
+## RPC vs In-Process Evaluation
+
+### RPC evaluation
+
+In RPC-based deployments one or more flagd instances deployed and exposed to client applications in your infrastructure.
+flagd RPC providers use HTTP or gRPC to request flag evaluations from flagd.
+The request payload contains the [flag key](https://openfeature.dev/specification/glossary#flag-key) identifying the flag to be evaluated, as well as the relevant [evaluation context](https://openfeature.dev/specification/glossary#evaluation-context).
+The flagd instance is configured to watch one or more [syncs](./concepts/syncs.md), and merges them to build its set of flags (see [here](./concepts/syncs.md#merging) for more details on flag definition merging).
+When sync sources are updated, flagd will send notifications to clients that flags have changed, enabling applications to react to changes by re-evaluating flags.
+
+This architecture is can be leveraged by very simple clients, since no in-process engine is needed; in fact, you can evaluate flags directly from a terminal console using the `cURL` utility.
+One disadvantage of this pattern is the latency involved in the remote request (though flagd typically takes  <10ms for an evaluation, and can evaluate thousands of flags per second).
+
+```mermaid
+---
+title: RPC Evaluation
+---
+erDiagram
+    flagd ||--o{ "sync (file)" : watches
+    flagd ||--o{ "sync (http)" : polls
+    flagd ||--o{ "sync (grpc)" : "sync.proto (gRPC/stream)"
+    flagd ||--o{ "sync (kubernetes)" : watches
+    "client app (+ flagd RPC provider)" ||--|| flagd : "evaluation.proto (gRPC/stream) / HTTP"
+```
+
+### In-Process evaluation
+
+In-process deployments embed the flagd evaluation engine directly into the client application through the use of an [in-process provider](./deployment.md#in-process).
+The in-process provider is connected via the sync protocol to an implementing [gRPC service](./concepts/syncs.md#grpc-sync) that provides the flag definitions.
+This pattern requires an in-process implementation of the flagd evaluation engine, but has the benefit of no I/O overhead, since no inter-process communication is required.
+
+```mermaid
+---
+title: In-Process Evaluation
+---
+erDiagram
+    "client app (+ flagd in-process provider)" ||--|| "sync (grpc)" : "sync.proto (gRPC/stream)"
+```
+
+<!-- TODO: add link to sync protocol reference entry -->
+<!-- TODO: we might want a dedicated Kubernets section here eventually to talk about the specifics of the K8s implementation -->

docs/assets/demo.flagd.json

@@ -0,0 +1,22 @@
+{
+  "flags": {
+    "show-welcome-banner": {
+      "state": "ENABLED",
+      "variants": {
+        "on": true,
+        "off": false
+      },
+      "defaultVariant": "off"
+    },
+    "background-color": {
+      "state": "ENABLED",
+      "variants": {
+        "red": "#FF0000",
+        "blue": "#0000FF",
+        "green": "#00FF00",
+        "yellow": "#FFFF00"
+      },
+      "defaultVariant": "red"
+    }
+  }
+}

docs/concepts/feature-flagging.md

@@ -0,0 +1,24 @@
+# Feature Flagging
+
+Feature flags are a software development technique that allows teams to enable, disable or change the behavior of certain features or code paths in a product or service, without modifying the source code.
+
+## OpenFeature Compliance
+
+[OpenFeature](https://openfeature.dev/) is an open standard that provides a vendor-agnostic, community-driven API for feature flagging.
+The flagd project is fully OpenFeature-compliant.
+In fact, flagd was initially conceived as a reference implementation for an OpenFeature backend, but has become a powerful tool in its own right.
+For this reason, you'll find flagd's concepts and terminology align with that of the OpenFeature project.
+Within the context of an OpenFeature-compliant feature flag solution, flagd artifacts and libraries comprise the [flag management system](https://openfeature.dev/specification/glossary#flag-management-system) and [providers](https://openfeature.dev/specification/glossary#provider).
+These artifacts and libraries alone won't allow you to evaluate flags in your application - you'll also need the [OpenFeature SDK](https://openfeature.dev/specification/glossary#feature-flag-sdk) for your language as well, which provides the evaluation API for application developers to use.
+
+## Supported Feature Flagging Use-Cases
+
+Below is a non-exhaustive table of common feature flag use-cases, and how flagd supports them:
+
+| Use case                                  | flagd Feature                                                                                                                                                                                                                                                                    |
+| ----------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| flag evaluation                           | Returns the value of a particular feature flag, if the flag is enabled. Supports flags of various types including boolean, numeric, string, and JSON.                                                                                                                            |
+| dynamic configuration                     | Flag definitions from any sync source are monitored for changes, with some syncs supporting near real time updates.                                                                                                                                                              |
+| dynamic (context-sensitive) evaluation    | flagd evaluations are context sensitive. Rules can use arbitrary context attributes as inputs for flag evaluation logic.                                                                                                                                                         |
+| fractional evaluation / random assignment | flagd's [fractional](../reference/custom-operations/fractional-operation.md) custom operation supports pseudorandom assignment of flag values.                                                                                                                                   |
+| progressive roll-outs                     | Progressive roll-outs of new features can be accomplished by leveraging the [fractional](../reference/custom-operations/fractional-operation.md) custom operation as well as automation in your build pipeline, SCM, or infrastructure which updates the distribution over time. |