Add documentation for Winlogbeat modules (elastic#12361) (elastic#12403)

Add documentation for the security and sysmon modules that explains
what the modules accomplish and how to configure them.

Relates elastic#11651

(cherry picked from commit 3cf45cb)

winlogbeat/docs/index.asciidoc

@@ -32,6 +32,8 @@ include::./upgrading.asciidoc[]
 
 include::./configuring-howto.asciidoc[]
 
+include::./modules.asciidoc[]
+
 include::./fields.asciidoc[]
 
 include::{libbeat-dir}/docs/monitoring/monitoring-beats.asciidoc[]

winlogbeat/docs/modules.asciidoc

@@ -0,0 +1,57 @@
+[id="{beatname_lc}-modules"]
+[role="xpack"]
+= Modules
+
+[partintro]
+--
+This section contains detailed information about the available Windows event
+log processing modules contained in {beatname_uc}. More details about each
+module can be found in the module's documentation.
+
+Winlogbeat modules are constructed using the
+<<processor-script,script processor>> where all processing happens within
+Winlogbeat before the events are delivered to the output.
+
+The general goal of each module is to transform events by renaming fields to
+comply with the {ecs-ref}/index.html[Elastic Common Schema] (ECS). The modules
+may also apply additional categorization, tagging, and parsing as necessary.
+
+The default configuration file included in packages has each of the modules
+configured. To apply the modules to your own configuration file you must add
+a `script` processor to your configuration file and point it at the included
+script file for the module. The documentation for each module includes an
+example.
+
+[float]
+=== Usage with Forwarded Events
+
+The `ForwardedEvents` channel can contain events from multiple producers so you
+may want to use multiple modules. This can be achieved by applying multiple
+script processors that are guarded by a conditional `when` statement.
+
+[source,yaml]
+----
+winlogbeat.event_logs:
+- name: ForwardedEvents
+  tags: [forwarded]
+  processors:
+  - script:
+      when.equals.winlog.channel: Security
+      lang: javascript
+      id: security
+      file: ${path.home}/module/security/config/winlogbeat-security.js
+  - script:
+      when.equals.winlog.channel: Microsoft-Windows-Sysmon/Operational
+      lang: javascript
+      id: sysmon
+      file: ${path.home}/module/sysmon/config/winlogbeat-sysmon.js
+----
+
+[float]
+=== Modules
+//pass macro block used here to remove Edit links from modules documentation because it is generated
+pass::[<?edit_url?>]
+include::modules_list.asciidoc[]
+
+
+

winlogbeat/docs/modules/security.asciidoc

@@ -0,0 +1,29 @@
+[[winlogbeat-module-security]]
+[role="xpack"]
+== Security Module
+
+beta[]
+
+The security module processes event log records from the Security log.
+
+The module has transformations for the following event IDs:
+
+* 4624 - An account was successfully logged on.
+* 4625 - An account failed to log on.
+* 4648 - A logon was attempted using explicit credentials.
+
+More event IDs will be added.
+
+[float]
+=== Configuration
+
+[source,yaml]
+----
+winlogbeat.event_logs:
+  - name: Security
+    processors:
+      - script:
+          lang: javascript
+          id: security
+          file: ${path.home}/module/security/config/winlogbeat-security.js
+----

winlogbeat/docs/modules/sysmon.asciidoc

@@ -0,0 +1,35 @@
+[[winlogbeat-module-sysmon]]
+[role="xpack"]
+== Sysmon Module
+
+beta[]
+
+The sysmon module processes event log records from the
+https://docs.microsoft.com/en-us/sysinternals/downloads/sysmon[Sysinternals
+System Monitor] (Sysmon) which is a Windows service and device driver that logs
+system activity to the event log. Sysmon is not bundled with Windows or
+Winlogbeat and must be installed independently.
+
+The default configuration file includes configuration for Sysmon. If you do not
+have Sysmon installed Winlogbeat will log a warning that it could not read from
+the `Microsoft-Windows-Sysmon/Operational` channel. It will continue to read
+from the other configured channels. If you do install Sysmon at a later time
+then a restart of Winlogbeat is required to make it begin reading from the
+channel.
+
+This module was built based on Sysmon v9 event manifests. It contains
+transformations for each of the defined event IDs.
+
+[float]
+=== Configuration
+
+[source,yaml]
+----
+winlogbeat.event_logs:
+  - name: Microsoft-Windows-Sysmon/Operational
+    processors:
+      - script:
+          lang: javascript
+          id: sysmon
+          file: ${path.home}/module/sysmon/config/winlogbeat-sysmon.js
+----

winlogbeat/docs/modules_list.asciidoc

@@ -0,0 +1,12 @@
+////
+This file is generated! See scripts/mage/docs.go or run 'mage docs'.
+////
+
+  * <<{beatname_lc}-module-security,Security>>
+  * <<{beatname_lc}-module-sysmon,Sysmon>>
+
+--
+
+
+include::./modules/security.asciidoc[]
+include::./modules/sysmon.asciidoc[]

winlogbeat/scripts/mage/docs.go

@@ -0,0 +1,95 @@
+// Licensed to Elasticsearch B.V. under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Elasticsearch B.V. licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package mage
+
+import (
+	"fmt"
+	"io/ioutil"
+	"os"
+	"path/filepath"
+	"regexp"
+	"strings"
+
+	"github.com/pkg/errors"
+
+	"github.com/elastic/beats/dev-tools/mage"
+)
+
+const moduleDocsGlob = "module/*/_meta/docs.asciidoc"
+
+var moduleNameRegex = regexp.MustCompile(`module\/(.*)\/_meta\/docs.asciidoc`)
+
+var modulesListTmpl = `
+////
+This file is generated! See scripts/mage/docs.go or run 'mage docs'.
+////
+{{range $module := .Modules}}
+  * <<{beatname_lc}-module-{{$module}},{{$module | title}}>>
+{{- end}}
+
+--
+
+{{range $module := .Modules}}
+include::./modules/{{$module}}.asciidoc[]
+{{- end}}
+`[1:]
+
+func moduleDocs() error {
+	searchPath := filepath.Join(mage.XPackBeatDir(moduleDocsGlob))
+
+	// Find module docs.
+	files, err := mage.FindFiles(searchPath)
+	if err != nil {
+		return err
+	}
+	if len(files) == 0 {
+		return errors.Errorf("No modules found matching %v", searchPath)
+	}
+
+	// Clean existing files.
+	if err := os.RemoveAll(mage.OSSBeatDir("docs/modules")); err != nil {
+		return err
+	}
+
+	// Extract module name from path and copy the file.
+	var names []string
+	for _, f := range files {
+		matches := moduleNameRegex.FindStringSubmatch(filepath.ToSlash(f))
+		if len(matches) != 2 {
+			return errors.Errorf("module path %v does not match regexp", f)
+		}
+		name := matches[1]
+		names = append(names, name)
+
+		// Copy to the docs dirs.
+		if err = mage.Copy(f, mage.CreateDir(mage.OSSBeatDir("docs/modules", name+".asciidoc"))); err != nil {
+			return err
+		}
+	}
+
+	// Generate and write the docs/modules_list.asciidoc file.
+	content, err := mage.Expand(modulesListTmpl, map[string]interface{}{
+		"Modules": names,
+	})
+	if err != nil {
+		return err
+	}
+
+	fmt.Printf(">> update:moduleDocs: Collecting module documentation for %v.\n", strings.Join(names, ", "))
+	return ioutil.WriteFile(mage.OSSBeatDir("docs/modules_list.asciidoc"), []byte(content), 0644)
+}

winlogbeat/scripts/mage/update.go

@@ -32,7 +32,7 @@ func init() {
 
 	dashboards.RegisterImportDeps(build.Build, Update.Dashboards)
 
-	docs.RegisterDeps(Update.FieldDocs)
+	docs.RegisterDeps(Update.FieldDocs, Update.ModuleDocs)
 }
 
 var (
@@ -45,7 +45,7 @@ type Update mg.Namespace
 
 // All updates all generated content.
 func (Update) All() {
-	mg.Deps(Update.Fields, Update.Dashboards, Update.Config, Update.FieldDocs)
+	mg.Deps(Update.Fields, Update.Dashboards, Update.Config, Update.FieldDocs, Update.ModuleDocs)
 }
 
 // Config updates the Beat's config files.
@@ -69,3 +69,8 @@ func (Update) FieldDocs() error {
 	mg.Deps(fb.FieldsAllYML)
 	return mage.Docs.FieldDocs(mage.FieldsAllYML)
 }
+
+// ModuleDocs collects and updates the module documentation.
+func (Update) ModuleDocs() error {
+	return moduleDocs()
+}

x-pack/winlogbeat/module/security/_meta/docs.asciidoc

@@ -0,0 +1,29 @@
+[[winlogbeat-module-security]]
+[role="xpack"]
+== Security Module
+
+beta[]
+
+The security module processes event log records from the Security log.
+
+The module has transformations for the following event IDs:
+
+* 4624 - An account was successfully logged on.
+* 4625 - An account failed to log on.
+* 4648 - A logon was attempted using explicit credentials.
+
+More event IDs will be added.
+
+[float]
+=== Configuration
+
+[source,yaml]
+----
+winlogbeat.event_logs:
+  - name: Security
+    processors:
+      - script:
+          lang: javascript
+          id: security
+          file: ${path.home}/module/security/config/winlogbeat-security.js
+----

x-pack/winlogbeat/module/sysmon/_meta/docs.asciidoc

@@ -0,0 +1,35 @@
+[[winlogbeat-module-sysmon]]
+[role="xpack"]
+== Sysmon Module
+
+beta[]
+
+The sysmon module processes event log records from the
+https://docs.microsoft.com/en-us/sysinternals/downloads/sysmon[Sysinternals
+System Monitor] (Sysmon) which is a Windows service and device driver that logs
+system activity to the event log. Sysmon is not bundled with Windows or
+Winlogbeat and must be installed independently.
+
+The default configuration file includes configuration for Sysmon. If you do not
+have Sysmon installed Winlogbeat will log a warning that it could not read from
+the `Microsoft-Windows-Sysmon/Operational` channel. It will continue to read
+from the other configured channels. If you do install Sysmon at a later time
+then a restart of Winlogbeat is required to make it begin reading from the
+channel.
+
+This module was built based on Sysmon v9 event manifests. It contains
+transformations for each of the defined event IDs.
+
+[float]
+=== Configuration
+
+[source,yaml]
+----
+winlogbeat.event_logs:
+  - name: Microsoft-Windows-Sysmon/Operational
+    processors:
+      - script:
+          lang: javascript
+          id: sysmon
+          file: ${path.home}/module/sysmon/config/winlogbeat-sysmon.js
+----