+Figure 1: The 3 primary groups of computers in the LME architecture, their descriptions and the operating systems / software run by each. +
+ +## Table of contents + +### Installation: + - [Prerequisites - Start deployment here](/docs/markdown/prerequisites.md) + - [Chapter 1 - Set up Windows Event Forwarding](/docs/markdown/chapter1/chapter1.md) + - [Chapter 2 – Sysmon Install](/docs/markdown/chapter2.md) + - [Chapter 3 – Database Install](/docs/markdown/chapter3/chapter3.md) + - [Chapter 4 - Post Install Actions ](/docs/markdown/chapter4.md) + +### Logging Guidance + - [Log Retention](/docs/markdown/logging-guidance/retention.md) + - [Additional Log Types](/docs/markdown/logging-guidance/other-logging.md) + +### Reference: + - [FAQ](/docs/markdown/reference/faq.md) + - [Troubleshooting](/docs/markdown/reference/troubleshooting.md) + - [Guide to Organizational Units](/docs/markdown/chapter1/guide_to_ous.md) + +### Maintenance: + - [Backups](/docs/markdown/maintenance/backups.md) + - [Upgrading](/docs/markdown/maintenance/upgrading.md) + - [Certificates](/docs/markdown/maintenance/certificates.md) diff --git a/backups/.gitkeep b/backups/.gitkeep new file mode 100644 index 00000000..e69de29b diff --git a/build/Readme.md b/build/Readme.md new file mode 100644 index 00000000..f87c46ef --- /dev/null +++ b/build/Readme.md @@ -0,0 +1,29 @@ +# Generating the docs: + +This directory uses [pandoc]() a universal document converter to build the markdown files into a pdf. Due to regulatory concerns we cannot release a pdf here directly, but you can utilize the following script to build the markdown docs into a pdf so you can use them offline if desired. + +In our testing we utilized the macos package manager [homebrew](https://brew.sh/) to install our packages. + +## Installing pandoc + +After you have homebrew make sure to install mactex: +```bash +brew install mactex +``` +Its a huge file but makes compiling everything super easy. Theres probably an equivalent on linux, but idk what it is + +Finally install pandoc: [link](https://pandoc.org/installing.html) +```bash +brew install pandoc +``` + +### Installing on other platforms +Other operating systems adn their respecitve latex/pandoc packages have not been tested nor will they be supported by LME. Since not every organization will have access to a MacOS operating system, but might wish to compile the docs anyway, please reachout and the team will attempt to help you compile the docs into a pdf. Any operating system with a latex package and pandoc executable should be able to accomplish the job. There are also many other ways to convert github flavored markdown to pdf if you google for them, and want to compile using a different method than we've provided here. + +## Compiling: +This command below will compile the markdown docs on macos from the homebrew install pandoc/mactex packages: +```bash +$ pandoc --from gfm --pdf-engine=lualatex -H ./build/setup.tex -V geometry:margin=1in --highlight-style pygments -o docs.pdf -V colorlinks=true -V linkcolor=blue --lua-filter=./build/emoji-filter.lua --lua-filter=./build/makerelativepaths.lua --lua-filter=./build/parse_breaks.lua --table-of-contents --number-sections --wrap=preserve --quiet -s $(cat ./build/includes.txt) +``` + +On a successful compilation it will output the `docs.pdf` file, a pdf of all the docs. There is a small bug where the `troubleshooting.md` table does not display as expected, so if you want the notes in the table offline, we suggest you record the information manually, OR submit a pull request that fixes this bug :smile:. diff --git a/build/emoji-filter.lua b/build/emoji-filter.lua new file mode 100644 index 00000000..9d78bed4 --- /dev/null +++ b/build/emoji-filter.lua @@ -0,0 +1,249 @@ +-- latex-emoji.lua +-- +-- @copyright 2020 Takayuki YATO (aka. "ZR") +-- GitHub: https://github.com/zr-tex8r +-- Twitter: @zr_tex8r +-- This program is distributed under the MIT License. +-- +local filter_name = 'latex-emoji' +---------------------------------------- helpers + +--- Show debug log? +local show_log = true +--- The default emoji font +local default_emojifont = 'TwemojiMozilla.ttf' + +--- Use bxcoloremoji package? +local bxcoloremoji = false +--- The emoji font to use +local emojifont, emojifontoptions = nil +--- All used codepoints +local ucs_used = {} +--- The number of emoji text spans. +local text_count = 0 + +local utils = require 'pandoc.utils' +local concat, insert, pack, unpack = + table.concat, table.insert, table.pack, table.unpack + +--- Shows a debug log. +local function log(fmt, ...) + if not show_log then return end + io.stderr:write(filter_name..": "..fmt:format(...).."\n") +end + +--- Aborts with an error message. +local function abort(fmt, ...) + error(filter_name..": "..fmt:format(...)) +end + +--- Returns the Pandoc-or-ordinary type of v. +-- @return A string that says type name. +local function pantype(v) + local t = type(v) + return (t == 'table') and v.t or t +end + +--- Makes a comma-separated value string. +-- @return A string. +local function clist(...) + local t, u = pack(...), {} + for i = 1, t.n do + local v = (t[i] == nil) and '' or tostring(t[i]) + if v ~= '' then insert(u, v) end + end + return concat(u, ',') +end + +--- Makes the sorted sequence of all keys of a given table. +-- @return A sequence of strings. +local function keys(t) + local u = {} + for k in pairs(t) do insert(u, k) end + table.sort(u) + return u +end + +--- Converts a singleton sequence to its element. +-- @return The sole element of v if v is a singleton; +-- v if v is not a table; otherwise an error is issued. +local function tosingle(v, l) + if type(v) ~= 'table' then return v end + if #v == 1 then return tosingle(v[1], l) end + abort("multiple values given: %s", l) +end + +--- Converts a value to a singleton sequence. +-- @return The empty table if v is nil; v if v is a table; +-- otherwise the singleton of v. +local function toseq(v) + if v == nil then return {} + elseif type(v) == 'table' then return v + else return {v} + end +end + +--- Converts MetaInlines values inside a MetaValue to strings. +-- @return The converted value. (v is not modified.) +local function tostring_meta(v, l) + if type(v) ~= 'table' then return v end + if v.t == 'MetaList' or v.t == nil then + local r = {} + for k, e in pairs(v) do r[k] = tostring_meta(e, l) end + return r + elseif v.t == 'MetaInlines' then + return utils.stringify(v) + else abort("cannot stringify: %s", v.t, l) + end +end + +--- Gets the source to go into the header. +-- @return LaTeX source string +local function get_header() + if not bxcoloremoji or not next(ucs_used) then + return nil + end + return ([[ +\usepackage[%s]{bxcoloremoji} +\newcommand*{\panEmoji}{\coloremoji} +]]):format(clist(emojifont, unpack(emojifontoptions))) +end + +--- Gets the source to go into the head of body. +-- @return LaTeX source string +local function get_prologue() + if bxcoloremoji or not next(ucs_used) then + return nil + end + local fname = emojifont or default_emojifont + local fopts = clist('Renderer=HarfBuzz', unpack(emojifontoptions)); + local ucs = keys(ucs_used) + for i = 1, #ucs do + ucs[i] = ('"%X'):format(ucs[i]) + end + local dcrsrc = concat(ucs, ',\n') + return ([[ +\makeatletter +\ifnum0\ifdefined\directlua\directlua{ + if ("\luaescapestring{\luatexbanner}"):match("LuaHBTeX") then tex.write("1") end + }\fi>\z@ %% LuaHBTeX is ok + \setfontface\p@emoji@font{%s}[%s] +\else + \@latex@error{You must install a new TeX system (TeX Live 2020)\MessageBreak + and then use 'lualatex' engine to print emoji} + {The compilation will be aborted.} + \let\p@emoji@font\relax +\fi +\ifdefined\ltjdefcharrange +\ltjdefcharrange{208}{ +%s} +\ltjsetparameter{jacharrange={-208}} +\fi +\newcommand*{\panEmoji}[1]{{\p@emoji@font#1}} +\makeatother +]]):format(fname, fopts, dcrsrc) +end + +--- For debug. +local function inspect(v) + local t = type(v) + if t == 'userdata' or t == 'function' or t == 'nil' then return t + elseif t == 'table' then + local u, tag = {}, (v.t or 'table') + if tag == 'Str' then return tag..'{'..v.text..'}' end + for i = 1, #v do u[i] = inspect(v[i]) end + return tag..'{'..concat(u, ';')..'}' + else return tostring(v) + end +end + +---------------------------------------- phase 'readmeta' + +--- For Meta elements. +local function readmeta_Meta (meta) + -- bxcoloremoji + if meta.bxcoloremoji == nil then + bxcoloremoji = false + elseif type(meta.bxcoloremoji) == 'boolean' then + bxcoloremoji = meta.bxcoloremoji + else + abort("not a boolean value: bxcoloremoji") + end + --log('bxcoloremoji = %s', bxcoloremoji) + -- emojifont + emojifont = tostring_meta(meta.emojifont, "emojifont") + emojifont = tosingle(emojifont, "emojifont") + --log('emojifont = %s', emojifont) + -- emojifontoptions + emojifontoptions = tostring_meta(meta.emojifontoptions, "emojifontoptions") + emojifontoptions = toseq(emojifontoptions) + for i in ipairs(emojifontoptions) do + emojifontoptions[i] = tosingle(emojifontoptions[i], "emojifontoptions element") + --log('emojifontoptions = %s', emojifontoptions[i]) + end +end + +---------------------------------------- phase 'mainproc' + +--- For Span element. +local function mainproc_Span(span) + if span.classes:includes('emoji', 1) then + text_count = text_count + 1 + local str = utils.stringify(span.content) + for p, uc in utf8.codes(str) do + if not ucs_used[uc] and uc >= 0x100 then + --log("emoji character: U+%04X", uc) + ucs_used[uc] = true + end + end + insert(span.content, 1, pandoc.RawInline('latex', [[\panEmoji{]])) + insert(span.content, pandoc.RawInline('latex', [[}]])) + return span.content + end +end + +--- For Meta elements. +local function mainproc_Meta(meta) + local src = get_header() + if src then + local headers = meta['header-includes'] + if headers == nil then + headers = pandoc.MetaList({}) + elseif pantype(headers) == 'MetaList' then + abort("unexpected metavalue type: header-includes") + end + insert(headers, pandoc.MetaBlocks{pandoc.RawBlock('latex', src)}) + meta['header-includes'] = headers + --log("header successfully appended") + return meta + end +end + +--- For the whole document. +local function mainproc_Pandoc(doc) + --log("number of emoji spans: %s", text_count) + local src = get_prologue() + if src then + insert(doc.blocks, 1, pandoc.RawBlock('latex', src)) + --log("prologue successfully inserted") + return doc + end +end + +---------------------------------------- the filter +if FORMAT == 'latex' then + return { + {-- phase 'readmeta' + Meta = readmeta_Meta; + }; + {-- phase 'mainproc' + Span = mainproc_Span; + Meta = mainproc_Meta; + Pandoc = mainproc_Pandoc; + }; + } +else + log("format '%s' in not supported", FORMAT) +end +---------------------------------------- done + diff --git a/build/includes.txt b/build/includes.txt new file mode 100644 index 00000000..0d680868 --- /dev/null +++ b/build/includes.txt @@ -0,0 +1,15 @@ +Readme.md +./docs/markdown/prerequisites.md +./docs/markdown/chapter1/chapter1.md +./docs/markdown/chapter1/guide_to_ous.md +./docs/markdown/chapter3/chapter3.md +./docs/markdown/chapter3/resilience.md +./docs/markdown/chapter4.md +./docs/markdown/logging-guidance/retention.md +./docs/markdown/logging-guidance/other-logging.md +./docs/markdown/maintenance/backups.md +./docs/markdown/maintenance/certificates.md +./docs/markdown/maintenance/upgrading.md +./docs/markdown/reference/troubleshooting.md +./docs/markdown/reference/faq.md + diff --git a/build/makerelativepaths.lua b/build/makerelativepaths.lua new file mode 100644 index 00000000..cb1d78d9 --- /dev/null +++ b/build/makerelativepaths.lua @@ -0,0 +1,14 @@ +function Image (img) + + --remove invalid urls + if string.find(img.src, "shields.io") then + img.src = "" + return img + end + + --makes paths relative so that links resolve on pandoc compile + img.src = pandoc.path.make_relative(img.src, '/') + + return img +end + diff --git a/build/parse_breaks.lua b/build/parse_breaks.lua new file mode 100644 index 00000000..2a2bf6fe --- /dev/null +++ b/build/parse_breaks.lua @@ -0,0 +1,7 @@ +--- Transform a raw HTML element which contains only a `` +-- into a format-indepentent line break. +function RawInline (el) + if el.format:match '^html' and el.text:match '%' then + return pandoc.LineBreak() + end +end diff --git a/build/setup.tex b/build/setup.tex new file mode 100644 index 00000000..473d2e0c --- /dev/null +++ b/build/setup.tex @@ -0,0 +1,19 @@ +% Contents of listings-setup.tex +\usepackage{fvextra} +\usepackage[utf8]{inputenc} + +\usepackage{lscape,longtable} + +%Note: be sure to put extra line between multi-line code, it will look bad otherwise! +\DefineVerbatimEnvironment{Highlighting}{Verbatim}{breaklines, + commandchars=\\\{\}, + breaksymbolleft=, + frame=single, +} + +%if converting emojis, requires lualatex: +\usepackage{emoji} + + +\iffalse Comments below are for myself in future work: +\fi diff --git a/docs/imgs/AdjustForwardedEventsLogSize.png b/docs/imgs/AdjustForwardedEventsLogSize.png new file mode 100644 index 00000000..891ac985 Binary files /dev/null and b/docs/imgs/AdjustForwardedEventsLogSize.png differ diff --git a/docs/imgs/OverviewDiagram.png b/docs/imgs/OverviewDiagram.png new file mode 100644 index 00000000..252d80b5 Binary files /dev/null and b/docs/imgs/OverviewDiagram.png differ diff --git a/docs/imgs/add-exceptions.png b/docs/imgs/add-exceptions.png new file mode 100644 index 00000000..20313ee7 Binary files /dev/null and b/docs/imgs/add-exceptions.png differ diff --git a/docs/imgs/alert-enable-menu.png b/docs/imgs/alert-enable-menu.png new file mode 100644 index 00000000..f9a312d7 Binary files /dev/null and b/docs/imgs/alert-enable-menu.png differ diff --git a/docs/imgs/backup_pics/policy_1.png b/docs/imgs/backup_pics/policy_1.png new file mode 100644 index 00000000..f28fada5 Binary files /dev/null and b/docs/imgs/backup_pics/policy_1.png differ diff --git a/docs/imgs/backup_pics/policy_2.png b/docs/imgs/backup_pics/policy_2.png new file mode 100644 index 00000000..9514aa1c Binary files /dev/null and b/docs/imgs/backup_pics/policy_2.png differ diff --git a/docs/imgs/backup_pics/policy_3.png b/docs/imgs/backup_pics/policy_3.png new file mode 100644 index 00000000..4da2d2c2 Binary files /dev/null and b/docs/imgs/backup_pics/policy_3.png differ diff --git a/docs/imgs/backup_pics/policy_4.png b/docs/imgs/backup_pics/policy_4.png new file mode 100644 index 00000000..b84ca379 Binary files /dev/null and b/docs/imgs/backup_pics/policy_4.png differ diff --git a/docs/imgs/backup_pics/policy_5.png b/docs/imgs/backup_pics/policy_5.png new file mode 100644 index 00000000..f759e284 Binary files /dev/null and b/docs/imgs/backup_pics/policy_5.png differ diff --git a/docs/imgs/backup_pics/policy_6.png b/docs/imgs/backup_pics/policy_6.png new file mode 100644 index 00000000..d46e01b9 Binary files /dev/null and b/docs/imgs/backup_pics/policy_6.png differ diff --git a/docs/imgs/backup_pics/repository_1.png b/docs/imgs/backup_pics/repository_1.png new file mode 100644 index 00000000..d8796d11 Binary files /dev/null and b/docs/imgs/backup_pics/repository_1.png differ diff --git a/docs/imgs/backup_pics/repository_2.png b/docs/imgs/backup_pics/repository_2.png new file mode 100644 index 00000000..2d51d774 Binary files /dev/null and b/docs/imgs/backup_pics/repository_2.png differ diff --git a/docs/imgs/backup_pics/repository_3.png b/docs/imgs/backup_pics/repository_3.png new file mode 100644 index 00000000..1bdb98bc Binary files /dev/null and b/docs/imgs/backup_pics/repository_3.png differ diff --git a/docs/imgs/backup_pics/snapshot_and_restore.png b/docs/imgs/backup_pics/snapshot_and_restore.png new file mode 100644 index 00000000..c5a03087 Binary files /dev/null and b/docs/imgs/backup_pics/snapshot_and_restore.png differ diff --git a/docs/imgs/chapter_overview.jpg b/docs/imgs/chapter_overview.jpg new file mode 100644 index 00000000..34000e20 Binary files /dev/null and b/docs/imgs/chapter_overview.jpg differ diff --git a/docs/imgs/cisa.png b/docs/imgs/cisa.png new file mode 100644 index 00000000..72b65512 Binary files /dev/null and b/docs/imgs/cisa.png differ diff --git a/docs/imgs/close-index.png b/docs/imgs/close-index.png new file mode 100644 index 00000000..cc356440 Binary files /dev/null and b/docs/imgs/close-index.png differ diff --git a/docs/imgs/createindex.jpg b/docs/imgs/createindex.jpg new file mode 100644 index 00000000..527f932f Binary files /dev/null and b/docs/imgs/createindex.jpg differ diff --git a/docs/imgs/dashboard.jpg b/docs/imgs/dashboard.jpg new file mode 100644 index 00000000..5b052798 Binary files /dev/null and b/docs/imgs/dashboard.jpg differ diff --git a/docs/imgs/default-index-pattern.png b/docs/imgs/default-index-pattern.png new file mode 100644 index 00000000..1e188c87 Binary files /dev/null and b/docs/imgs/default-index-pattern.png differ diff --git a/docs/imgs/default-winlogbeat.png b/docs/imgs/default-winlogbeat.png new file mode 100644 index 00000000..13199f93 Binary files /dev/null and b/docs/imgs/default-winlogbeat.png differ diff --git a/docs/imgs/delete-indices.jpg b/docs/imgs/delete-indices.jpg new file mode 100644 index 00000000..3c283c52 Binary files /dev/null and b/docs/imgs/delete-indices.jpg differ diff --git a/docs/imgs/delete-originals.png b/docs/imgs/delete-originals.png new file mode 100644 index 00000000..af968bf3 Binary files /dev/null and b/docs/imgs/delete-originals.png differ diff --git a/docs/imgs/dev-tools.jpg b/docs/imgs/dev-tools.jpg new file mode 100644 index 00000000..0608a6d4 Binary files /dev/null and b/docs/imgs/dev-tools.jpg differ diff --git a/docs/imgs/discover_tab.jpg b/docs/imgs/discover_tab.jpg new file mode 100644 index 00000000..82e06830 Binary files /dev/null and b/docs/imgs/discover_tab.jpg differ diff --git a/docs/imgs/duplicate-indices.jpg b/docs/imgs/duplicate-indices.jpg new file mode 100644 index 00000000..51968046 Binary files /dev/null and b/docs/imgs/duplicate-indices.jpg differ diff --git a/docs/imgs/edit-update-script.png b/docs/imgs/edit-update-script.png new file mode 100644 index 00000000..ff68ed84 Binary files /dev/null and b/docs/imgs/edit-update-script.png differ diff --git a/docs/imgs/elkstack.jpg b/docs/imgs/elkstack.jpg new file mode 100644 index 00000000..6a1fbc7f Binary files /dev/null and b/docs/imgs/elkstack.jpg differ diff --git a/docs/imgs/error.png b/docs/imgs/error.png new file mode 100644 index 00000000..329219b1 Binary files /dev/null and b/docs/imgs/error.png differ diff --git a/docs/imgs/event_viewer_prompt.png b/docs/imgs/event_viewer_prompt.png new file mode 100644 index 00000000..33c1de08 Binary files /dev/null and b/docs/imgs/event_viewer_prompt.png differ diff --git a/docs/imgs/eventforwarding_overview.jpg b/docs/imgs/eventforwarding_overview.jpg new file mode 100644 index 00000000..a4db666a Binary files /dev/null and b/docs/imgs/eventforwarding_overview.jpg differ diff --git a/docs/imgs/eventviewer.jpg b/docs/imgs/eventviewer.jpg new file mode 100644 index 00000000..efcbe669 Binary files /dev/null and b/docs/imgs/eventviewer.jpg differ diff --git a/docs/imgs/example-exception.png b/docs/imgs/example-exception.png new file mode 100644 index 00000000..92f3555c Binary files /dev/null and b/docs/imgs/example-exception.png differ diff --git a/docs/imgs/exceptions.png b/docs/imgs/exceptions.png new file mode 100644 index 00000000..2a47a4e4 Binary files /dev/null and b/docs/imgs/exceptions.png differ diff --git a/docs/imgs/extra_beats_pics/deletion-enable.png b/docs/imgs/extra_beats_pics/deletion-enable.png new file mode 100644 index 00000000..9a44d8ac Binary files /dev/null and b/docs/imgs/extra_beats_pics/deletion-enable.png differ diff --git a/docs/imgs/extra_beats_pics/filebeat-selection.png b/docs/imgs/extra_beats_pics/filebeat-selection.png new file mode 100644 index 00000000..a8414594 Binary files /dev/null and b/docs/imgs/extra_beats_pics/filebeat-selection.png differ diff --git a/docs/imgs/extra_beats_pics/filebeat.png b/docs/imgs/extra_beats_pics/filebeat.png new file mode 100644 index 00000000..3ba3a140 Binary files /dev/null and b/docs/imgs/extra_beats_pics/filebeat.png differ diff --git a/docs/imgs/extra_beats_pics/ilm.png b/docs/imgs/extra_beats_pics/ilm.png new file mode 100644 index 00000000..976861d4 Binary files /dev/null and b/docs/imgs/extra_beats_pics/ilm.png differ diff --git a/docs/imgs/extra_beats_pics/logstash-writer.png b/docs/imgs/extra_beats_pics/logstash-writer.png new file mode 100644 index 00000000..a71063a1 Binary files /dev/null and b/docs/imgs/extra_beats_pics/logstash-writer.png differ diff --git a/docs/imgs/extra_beats_pics/roles.png b/docs/imgs/extra_beats_pics/roles.png new file mode 100644 index 00000000..c43122a6 Binary files /dev/null and b/docs/imgs/extra_beats_pics/roles.png differ diff --git a/docs/imgs/extra_beats_pics/stack-management.png b/docs/imgs/extra_beats_pics/stack-management.png new file mode 100644 index 00000000..71954915 Binary files /dev/null and b/docs/imgs/extra_beats_pics/stack-management.png differ diff --git a/docs/imgs/extra_beats_pics/update-retention.png b/docs/imgs/extra_beats_pics/update-retention.png new file mode 100644 index 00000000..ead0f3f6 Binary files /dev/null and b/docs/imgs/extra_beats_pics/update-retention.png differ diff --git a/docs/imgs/extra_beats_pics/update-role.png b/docs/imgs/extra_beats_pics/update-role.png new file mode 100644 index 00000000..1d2f623f Binary files /dev/null and b/docs/imgs/extra_beats_pics/update-role.png differ diff --git a/docs/imgs/firstload.jpg b/docs/imgs/firstload.jpg new file mode 100644 index 00000000..ebce2101 Binary files /dev/null and b/docs/imgs/firstload.jpg differ diff --git a/docs/imgs/gpo.jpg b/docs/imgs/gpo.jpg new file mode 100644 index 00000000..9f1a48c8 Binary files /dev/null and b/docs/imgs/gpo.jpg differ diff --git a/docs/imgs/gpo_pics/aduc.jpg b/docs/imgs/gpo_pics/aduc.jpg new file mode 100644 index 00000000..a6fcdead Binary files /dev/null and b/docs/imgs/gpo_pics/aduc.jpg differ diff --git a/docs/imgs/gpo_pics/create_new_object.jpg b/docs/imgs/gpo_pics/create_new_object.jpg new file mode 100644 index 00000000..79e4e321 Binary files /dev/null and b/docs/imgs/gpo_pics/create_new_object.jpg differ diff --git a/docs/imgs/gpo_pics/gpmc.jpg b/docs/imgs/gpo_pics/gpmc.jpg new file mode 100644 index 00000000..d08012bc Binary files /dev/null and b/docs/imgs/gpo_pics/gpmc.jpg differ diff --git a/docs/imgs/gpo_pics/import_done.jpg b/docs/imgs/gpo_pics/import_done.jpg new file mode 100644 index 00000000..05350d20 Binary files /dev/null and b/docs/imgs/gpo_pics/import_done.jpg differ diff --git a/docs/imgs/gpo_pics/import_new_object.jpg b/docs/imgs/gpo_pics/import_new_object.jpg new file mode 100644 index 00000000..67c5d334 Binary files /dev/null and b/docs/imgs/gpo_pics/import_new_object.jpg differ diff --git a/docs/imgs/gpo_pics/link_an_ou.jpg b/docs/imgs/gpo_pics/link_an_ou.jpg new file mode 100644 index 00000000..f0bb4594 Binary files /dev/null and b/docs/imgs/gpo_pics/link_an_ou.jpg differ diff --git a/docs/imgs/gpo_pics/name_new_object.jpg b/docs/imgs/gpo_pics/name_new_object.jpg new file mode 100644 index 00000000..d9694e7f Binary files /dev/null and b/docs/imgs/gpo_pics/name_new_object.jpg differ diff --git a/docs/imgs/gpo_pics/new_ou.jpg b/docs/imgs/gpo_pics/new_ou.jpg new file mode 100644 index 00000000..89b2c7e2 Binary files /dev/null and b/docs/imgs/gpo_pics/new_ou.jpg differ diff --git a/docs/imgs/gpo_pics/optional_features.png b/docs/imgs/gpo_pics/optional_features.png new file mode 100644 index 00000000..f9e0f9a7 Binary files /dev/null and b/docs/imgs/gpo_pics/optional_features.png differ diff --git a/docs/imgs/gpo_pics/rsat_gpmc_optional_features.png b/docs/imgs/gpo_pics/rsat_gpmc_optional_features.png new file mode 100644 index 00000000..3ee9ee21 Binary files /dev/null and b/docs/imgs/gpo_pics/rsat_gpmc_optional_features.png differ diff --git a/docs/imgs/gpo_pics/select_backup.jpg b/docs/imgs/gpo_pics/select_backup.jpg new file mode 100644 index 00000000..2dd2786a Binary files /dev/null and b/docs/imgs/gpo_pics/select_backup.jpg differ diff --git a/docs/imgs/gpo_pics/select_gpo_link.jpg b/docs/imgs/gpo_pics/select_gpo_link.jpg new file mode 100644 index 00000000..6c75de57 Binary files /dev/null and b/docs/imgs/gpo_pics/select_gpo_link.jpg differ diff --git a/docs/imgs/gpoedit.jpg b/docs/imgs/gpoedit.jpg new file mode 100644 index 00000000..785bd537 Binary files /dev/null and b/docs/imgs/gpoedit.jpg differ diff --git a/docs/imgs/healthcheckstatus.jpg b/docs/imgs/healthcheckstatus.jpg new file mode 100644 index 00000000..9e5ef793 Binary files /dev/null and b/docs/imgs/healthcheckstatus.jpg differ diff --git a/docs/imgs/import.jpg b/docs/imgs/import.jpg new file mode 100644 index 00000000..57af6514 Binary files /dev/null and b/docs/imgs/import.jpg differ diff --git a/docs/imgs/import.png b/docs/imgs/import.png new file mode 100644 index 00000000..c49d102e Binary files /dev/null and b/docs/imgs/import.png differ diff --git a/docs/imgs/import1.png b/docs/imgs/import1.png new file mode 100644 index 00000000..7315f0f3 Binary files /dev/null and b/docs/imgs/import1.png differ diff --git a/docs/imgs/import2.png b/docs/imgs/import2.png new file mode 100644 index 00000000..8e035c35 Binary files /dev/null and b/docs/imgs/import2.png differ diff --git a/docs/imgs/index-patterns.png b/docs/imgs/index-patterns.png new file mode 100644 index 00000000..5421402f Binary files /dev/null and b/docs/imgs/index-patterns.png differ diff --git a/docs/imgs/index-selection.png b/docs/imgs/index-selection.png new file mode 100644 index 00000000..a3499b4b Binary files /dev/null and b/docs/imgs/index-selection.png differ diff --git a/docs/imgs/logistics.png b/docs/imgs/logistics.png new file mode 100644 index 00000000..5e1f34d4 Binary files /dev/null and b/docs/imgs/logistics.png differ diff --git a/docs/imgs/nav-bar.png b/docs/imgs/nav-bar.png new file mode 100644 index 00000000..5570d35b Binary files /dev/null and b/docs/imgs/nav-bar.png differ diff --git a/docs/imgs/re-index-script.jpg b/docs/imgs/re-index-script.jpg new file mode 100644 index 00000000..d2ce5478 Binary files /dev/null and b/docs/imgs/re-index-script.jpg differ diff --git a/docs/imgs/restore-details.png b/docs/imgs/restore-details.png new file mode 100644 index 00000000..c517a984 Binary files /dev/null and b/docs/imgs/restore-details.png differ diff --git a/docs/imgs/restore.png b/docs/imgs/restore.png new file mode 100644 index 00000000..4193ce40 Binary files /dev/null and b/docs/imgs/restore.png differ diff --git a/docs/imgs/retention_pics/retention_1.png b/docs/imgs/retention_pics/retention_1.png new file mode 100644 index 00000000..e0d548ff Binary files /dev/null and b/docs/imgs/retention_pics/retention_1.png differ diff --git a/docs/imgs/rules_error.png b/docs/imgs/rules_error.png new file mode 100644 index 00000000..9ca51122 Binary files /dev/null and b/docs/imgs/rules_error.png differ diff --git a/docs/imgs/select-rule.png b/docs/imgs/select-rule.png new file mode 100644 index 00000000..edb78d0a Binary files /dev/null and b/docs/imgs/select-rule.png differ diff --git a/docs/imgs/siem.png b/docs/imgs/siem.png new file mode 100644 index 00000000..a06026ca Binary files /dev/null and b/docs/imgs/siem.png differ diff --git a/docs/imgs/siem1.png b/docs/imgs/siem1.png new file mode 100644 index 00000000..c2030d41 Binary files /dev/null and b/docs/imgs/siem1.png differ diff --git a/docs/imgs/siem2.png b/docs/imgs/siem2.png new file mode 100644 index 00000000..c576dd31 Binary files /dev/null and b/docs/imgs/siem2.png differ diff --git a/docs/imgs/siem3.png b/docs/imgs/siem3.png new file mode 100644 index 00000000..2c18662a Binary files /dev/null and b/docs/imgs/siem3.png differ diff --git a/docs/imgs/siem4.png b/docs/imgs/siem4.png new file mode 100644 index 00000000..f355767e Binary files /dev/null and b/docs/imgs/siem4.png differ diff --git a/docs/imgs/siem5.png b/docs/imgs/siem5.png new file mode 100644 index 00000000..50b12999 Binary files /dev/null and b/docs/imgs/siem5.png differ diff --git a/docs/imgs/siem6.png b/docs/imgs/siem6.png new file mode 100644 index 00000000..f3654abe Binary files /dev/null and b/docs/imgs/siem6.png differ diff --git a/docs/imgs/snap-restore.png b/docs/imgs/snap-restore.png new file mode 100644 index 00000000..4a87f781 Binary files /dev/null and b/docs/imgs/snap-restore.png differ diff --git a/docs/imgs/stack-management.jpg b/docs/imgs/stack-management.jpg new file mode 100644 index 00000000..d5057f6c Binary files /dev/null and b/docs/imgs/stack-management.jpg differ diff --git a/docs/imgs/stack-management.png b/docs/imgs/stack-management.png new file mode 100644 index 00000000..db9d0c33 Binary files /dev/null and b/docs/imgs/stack-management.png differ diff --git a/docs/imgs/sysmon-task-properties.png b/docs/imgs/sysmon-task-properties.png new file mode 100644 index 00000000..99555859 Binary files /dev/null and b/docs/imgs/sysmon-task-properties.png differ diff --git a/docs/imgs/sysvol.jpg b/docs/imgs/sysvol.jpg new file mode 100644 index 00000000..f8b45f0d Binary files /dev/null and b/docs/imgs/sysvol.jpg differ diff --git a/docs/imgs/task-complete.png b/docs/imgs/task-complete.png new file mode 100644 index 00000000..0035f279 Binary files /dev/null and b/docs/imgs/task-complete.png differ diff --git a/docs/imgs/task-status.png b/docs/imgs/task-status.png new file mode 100644 index 00000000..e4cff2ae Binary files /dev/null and b/docs/imgs/task-status.png differ diff --git a/docs/imgs/task.png b/docs/imgs/task.png new file mode 100644 index 00000000..e2128193 Binary files /dev/null and b/docs/imgs/task.png differ diff --git a/docs/imgs/timefilter.jpg b/docs/imgs/timefilter.jpg new file mode 100644 index 00000000..cc3fb7bf Binary files /dev/null and b/docs/imgs/timefilter.jpg differ diff --git a/docs/imgs/timerange.jpg b/docs/imgs/timerange.jpg new file mode 100644 index 00000000..7222d98e Binary files /dev/null and b/docs/imgs/timerange.jpg differ diff --git a/docs/imgs/timerange.png b/docs/imgs/timerange.png new file mode 100644 index 00000000..b004cb48 Binary files /dev/null and b/docs/imgs/timerange.png differ diff --git a/docs/imgs/troubleshooting-overview.jpg b/docs/imgs/troubleshooting-overview.jpg new file mode 100644 index 00000000..a60804d1 Binary files /dev/null and b/docs/imgs/troubleshooting-overview.jpg differ diff --git a/docs/imgs/update-rules.png b/docs/imgs/update-rules.png new file mode 100644 index 00000000..3a4d97a0 Binary files /dev/null and b/docs/imgs/update-rules.png differ diff --git a/docs/imgs/usersec.png b/docs/imgs/usersec.png new file mode 100644 index 00000000..d68332e6 Binary files /dev/null and b/docs/imgs/usersec.png differ diff --git a/docs/imgs/verify.png b/docs/imgs/verify.png new file mode 100644 index 00000000..4c642da5 Binary files /dev/null and b/docs/imgs/verify.png differ diff --git a/docs/imgs/winlogbeat-install.png b/docs/imgs/winlogbeat-install.png new file mode 100644 index 00000000..8ad4019d Binary files /dev/null and b/docs/imgs/winlogbeat-install.png differ diff --git a/docs/imgs/winlogbeat-location.png b/docs/imgs/winlogbeat-location.png new file mode 100644 index 00000000..b187b9cd Binary files /dev/null and b/docs/imgs/winlogbeat-location.png differ diff --git a/docs/imgs/winlogbeat-running.png b/docs/imgs/winlogbeat-running.png new file mode 100644 index 00000000..0f1f20ca Binary files /dev/null and b/docs/imgs/winlogbeat-running.png differ diff --git a/docs/imgs/winscp.jpg b/docs/imgs/winscp.jpg new file mode 100644 index 00000000..17f0f203 Binary files /dev/null and b/docs/imgs/winscp.jpg differ diff --git a/docs/markdown/chapter1/chapter1.md b/docs/markdown/chapter1/chapter1.md new file mode 100644 index 00000000..6658774b --- /dev/null +++ b/docs/markdown/chapter1/chapter1.md @@ -0,0 +1,129 @@ +# Chapter 1 – Setting up Windows Event Forwarding + + +
+Figure 1: Finished state of Chapter 1 +
+ +## Chapter Overview +In this chapter you will: +* Add some Group Policy Objects (GPOs) to your Active Directory (AD). +* Configure the Windows Event Collector listener service. +* Configure clients to send logs to this box. + +## 1.1 Introduction +This chapter will cover setting up the built-in Windows functionality for event forwarding. This effectively takes the individual events (such as a file being opened) and sends them to a central machine for processing. This is similar to the setup discussed in this [Microsoft blog](https://docs.microsoft.com/en-us/windows/security/threat-protection/use-windows-event-forwarding-to-assist-in-intrusion-detection). + +Only a selection of events will be sent from the client's ‘Event Viewer’ to a central ‘Event Collector’. The events will then be uploaded to the database and dashboard in Chapter 3. +This chapter will require the clients and event collector to be Active Directory domain joined and the event collector to be either a Windows server or a Windows client operating system. + +## 1.2 Firewall rules and where to host +You will need TCP port 5985 open between the clients and the Windows Event Collector. You also need port 5044 open between the Windows Event Collector and the Linux server. + +We recommend that this traffic does not go directly across the Internet, so you should host the Windows Event Collector on the local network, in a similar place to the Active Directory server. + +## 1.3 Download LME +There are several files within the LME repo that need to be available on a domain controller. These files will be needed for both Chapters 1 and 2. While there are multiple ways to accomplish this, one simple method is to download the latest release package. + +1. While on a domain controller, download [the desired release](https://github.com/cisagov/lme/releases/). +2. Open File Explorer, locate and extract the release file downloaded in step 1, for example, LME-1.0.zip. +3. Move the LME folder somewhere safe. There is no set location where this folder is required to be, but it should be saved somewhere it won't be inadvertently modified or deleted during the installation process. After installation is complete, the folder can be safely deleted. + +## 1.4 Import Group Policy objects +Group policy objects (GPOs) are a convenient way to administer technical policies across an Active Directory domain. LME comes with two GPOs that work together to forward events from the client machines to the Event Collector. + + ++Figure 2: Setting up Group Policy +
+ +#### 1.4.1 Opening GPMC +While on a domain controller, open the Group Policy Management Console by running ```gpmc.msc```. You can run this command by pressing Windows key + R. + + ++Figure 3: Launching GPMC +
+ +:hammer_and_wrench: If you receive the error `Windows cannot find 'gpmc.msc'`, see [Troubleshooting: Installing Group Policy Management Tools](/docs/markdown/reference/troubleshooting.md#installing-group-policy-management-tools). + +#### 1.4.2 Initialize the GPOs +1. Within the Group Policy Management Console, navigate to the "Group Policy Objects" folder. The exact path will vary, depending on your domain's name. In the example used in Figure 3, the path is `Forest: testme.local / Domains / testme.local / Group Policy Objects`). +2. Right click "Group Policy Objects" and select "New." +3. Create two new GPOs, "LME-WEC-Client" and "LME-WEC-Server." Leave "Source Starter GPO:" as "(none)" for both. + + ++Figure 4: Create a new GPO object +
+ +#### 1.4.3 Import the GPO Settings +1. Right-click the newly created "LME-WEC-Client" object. Select "Import Settings..." +2. Hit "Next" until you reach the "Backup Location" page of the Wizard. NOTE: the "Backup Location" page of the wizard deals with _importing_ settings from a backup, not to be confused with the "Backup GPO" page, which deals with creating a new backup with the current settings. +3. When prompted to specify a "Backup Location," specify `LME-1.0/Chapter 1 Files/Group Policy Objects`, where `LME-1.0` refers to the folder downloaded in step 1.3. +4. On the "Source GPO" page, select "LME-WEC-Client." +5. Click "Next" then "Finish." +6. Repeat the above steps for the "LME-WEC-Server" object, selecting "LME-WEC-Server" on step 4. + +#### 1.4.4 Set the Destination for Forwarded Events +1. Right-click the "LME-WEC-Client" object, then select "Edit." +2. Navigate to `Computer Configuration/Policies/Administrative Templates/Windows Components/Event Forwarding/`. +3. Click "Configure Target Subscription Manager." By "SubscriptionManagers," click "Show." +4. Change the FQDN (Fully Qualified Domain Name) to match your Windows Event Collector box name - this option can be seen in Figure 5 below. This domain name needs to be resolvable from each of the clients. +5. After changing the FQDN, click "Apply" then "OK." + + ++Figure 5: Editing Server Name In Group Policy +
+ +#### 1.4.5 Link the GPOs +To "activate" the GPOs that you previously imported, you need to specify which computers they apply to. Here we describe only one technique of doing this, namely linking GPOs to organizational units (OUs). Advanced users may consider using alternate techniques that better fit their needs. See [Planning GPO Deployment](https://learn.microsoft.com/en-us/windows/security/operating-system-security/network-security/windows-firewall/planning-gpo-deployment) for more information. + +1. Create an OU to hold a subset of client computers that you want to be included in the LME Client group for testing before rolling out LME site-wide. See [Guide to Organizational Units](/docs/markdown/chapter1/guide_to_ous.md). We recommend starting with just a subset for testing before rolling out LME site-wide. +2. Within the Group Policy Management Console, right click the OU containing the client machines. +3. Click "Link an Existing GPO..." +4. Select "LME-WEC-Client," then click "OK." +5. Before linking the LME-WEC-Server, ensure that the Event Collector has been placed in its own OU. If needed, use the above guide on creating OUs in Step 1. +6. Within the Group Policy Management Console, right click the OU containing the Event Collector. +7. Click "Link an Existing GPO..." +8. Select "LME-WEC-Server," then click "OK." + +#### 1.4.6 Restricting Windows Remote Management by IP + +Both the LME-WEC-Server and LME-WEC-Client GPOs include a wildcard filter allowing all IP addresses on the host and client to run a Windows Remote Management (WinRM) Listener and to receive inbound connections using this protocol. **We strongly recommend that this is restricted to IP addresses or ranges specific to your network environment.** + +An example of this would be if you hosted a LAN with the subnet 192.168.2.0/24, then you could only allows NICs residing within the range 192.168.2.1-192.168.2.254 to run a WinRM listener via the GPO policy. + +See Microsoft Document for verification and details: [Installation and configuration for Windows Remote Management](https://docs.microsoft.com/en-us/windows/win32/winrm/installation-and-configuration-for-windows-remote-management) + +The filter setting is located at "Computer Configuration/Policies/Administrative Templates/Windows Components/Windows Remote Management (WinRM)/WinRM Service/allow remote server management through WinRM". + +### 1.5 Windows Event Collector Box Steps +1. On the Windows Event Collector, run Event Viewer by either searching under Start->Run->eventvwr.exe, or under 'Windows Administrative Tools' in the start menu. +2. Click "Subscriptions." +3. If prompted, select "Yes" to start the Windows Event Collector Service (see Figure 6). If no such prompt appears, continue to step 4. + + ++Figure 6: Start the Windows Event Collector Service, if needed. +
+ +4. Download the [lme_wec_config.xml](/Chapter%201%20Files/lme_wec_config.xml) file to the Windows Event Collector server. +5. Run a command prompt as an administrator, change to the directory containing the wec_config.xml file you just downloaded. +6. Run the command ```wecutil cs lme_wec_config.xml``` within the elevated command prompt. There is no output displayed after running this command. + +:hammer_and_wrench: If you receive the error "The forwarder is having a problem communicating with subscription manager..." refer to [Events are not forwarded if the collector is running Windows Server](https://support.microsoft.com/en-in/help/4494462/events-not-forwarded-if-the-collector-runs-windows-server-2019-or-2016). If that does not fix the problem or does not apply, verify that TCP port 5985 is open between the clients and the Windows Event Collector. + +## Chapter 1 - Checklist +1. On the Windows Event Collector, Run Event Viewer by either Start->Run->eventvwr.exe, or under ‘Windows Administrative Tools’ in the start menu. +2. Confirm machines are checking in, as per Figure 7. The 'Source Computers' field should contain the number of machines currently connected. + + ++Figure 7: Event Log Subscriptions +
+ +Note that by default, Windows will update group policy settings only every 90 minutes. Because of this, it's possible that the 'Source Computers' field will be 0 the first time you check the subscriptions page. To force an update, logon to one of the client machines, then from an elevated command prompt, run `gpupdate /force.` After doing that, if you return to the event collector, that specific client should show up under the Source Computers tab. + +## Now move onto [Chapter 2 – Sysmon Install](/docs/markdown/chapter2.md) diff --git a/docs/markdown/chapter1/guide_to_ous.md b/docs/markdown/chapter1/guide_to_ous.md new file mode 100644 index 00000000..78ec9158 --- /dev/null +++ b/docs/markdown/chapter1/guide_to_ous.md @@ -0,0 +1,37 @@ + +## Guide to Organizational Units + +What is an Organizational Unit? +An Organizational Unit can in its simplest form be thought of as a folder to contain Users, Computers and groups. +OUs can be used to select a subset of computers that you want to be included in the LME Client group for testing before rolling out LME site wide. + +### 1 - How to make an OU +**1.1** Open the Group Policy Management Console by running ```gpmc.msc```. You can run this command by pressing Windows key + R. + + ++Figure 1: Launching GPMC +
+ +:hammer_and_wrench: If you receive the error `Windows cannot find 'gpmc.msc'`, see [Troubleshooting: Installing Group Policy Management Tools](/docs/markdown/reference/troubleshooting.md#installing-group-policy-management-tools). + +**1.2** Right click on the domain and select "New Organizational Unit" as seen below. + + ++Figure 2: Making a new OU +
+ +### 2 - Adding clients/servers to OU + +To add Client machines, Servers or Security Groups to a specified OU: + +- Open Active Directory Users and Computers (run `dsa.msc` in the "Run" dialogue box). +- Find the machine(s) that you wish to be in the group and drag and drop the machines into the group. + + ++Figure 3: Open Active Directory Users and Computers +
+ +:hammer_and_wrench: If you receive the error `Windows cannot find dsa.msc`, see [Troubleshooting: Installing Active Directory Domain Services](/docs/markdown/reference/troubleshooting.md#installing-active-directory-domain-services) diff --git a/docs/markdown/chapter2.md b/docs/markdown/chapter2.md new file mode 100644 index 00000000..15326292 --- /dev/null +++ b/docs/markdown/chapter2.md @@ -0,0 +1,130 @@ +# Chapter 2 – Installing Sysmon + +## Chapter Overview +In this chapter you will: +* Setup a GPO or SCCM job to deploy Sysmon across your clients. + +## 2.1 Introduction +Sysmon is a Windows service developed by Microsoft to generate rich Windows event logs with much more information than the default events created in Windows. Having comprehensive logs is critical in monitoring your system and keeping it secure. The information contained within Sysmon's logs are based on settings defined in an XML configuration file and can be configured to your liking, though templates will be provided to get you started. + +**By following this guide and using Sysmon, you are agreeing to the following EULA. +Please read this before continuing. +https://docs.microsoft.com/en-us/sysinternals/license-terms** + +LME supports either GPO or SCCM Deployment. It is your choice which of these you use, but you should not use both. GPO configuration is recommended, as the process very closely resembles the steps taken in [Chapter 1](/docs/markdown/chapter1/chapter1.md). + +## 2.2 GPO Deployment + +Group Policy Object (GPO) deployment involves adding a GPO to the LME clients that creates a Windows 'Scheduled Task' to install Sysmon. The 'Scheduled Task' will periodically connect to a network folder location and run an install script called 'update.bat' to install Sysmon or modify an existing installation. + +Using Microsoft Group Policy to deploy LME requires two main things: +- A location to host the configuration and executables. (e.g. SYSVOL) +- A Group Policy Object (GPO) to create a scheduled task. + +If you get stuck while trying to add and configure GPO's, refer back to Chapter 1 for a quick refresher. + +### 2.2.1 - Folder Layout +A centralized network folder accessible by all machines that are going to be running Sysmon is needed. We suggest inside the SYSVOL directory as a suitable place since this is configured by default to have very restricted write permissions. +**It is extremely important that the folder contents cannot be modified by users, hence recommending SYSVOL folder.** + +The SYSVOL directory is located on the Domain Controller at `C:\Windows\SYSVOL\SYSVOL\+Figure 1: Edit the NETDIR Variable in Both Update Scripts +
+ +### 2.2.3 - Scheduled task GPO Policy +This section sets up a scheduled task to run update.bat (stored on a network folder), distributed through Group Policy. + +1. From a domain controller, open the Group Policy Management editor (Windows key + R, "gpmc.msc"). +2. Create a new GPO, "LME-Sysmon-Task." +3. Right-click the newly created "LME-Sysmon-Task" object. Select "Import Settings..." +4. Hit "Next" until you reach the "Backup Location" page of the Wizard. **NOTE:** the "Backup Location" page of the wizard deals with importing settings from a backup, not to be confused with the "Backup GPO" page, which deals with creating a new backup with the current settings. +5. When prompted to specify a "Backup Location," specify `LME-1.0\Chapter 2 Files\GPO Deployment\Group Policy Objects\`, where `LME-1.0` refers to the folder downloaded in [step 1.3](/docs/markdown/chapter1/chapter1.md#13-download-lme). +6. On the "Source GPO" page, select "LME-Sysmon-Task." Click "Next" then "Finish." +7. Right click the same test Organizational Unit (OU) used for the clients in Chapter 1, click "Link an Existing GPO...," then select "LME-Sysmon-Task." Once the GPO is confirmed as working in your environment then you can link the GPO to a larger OU to deploy LME further. +8. Right click the Lme-Sysmon-Task GPO and select "Edit." +9. Navigate to `Computer Configuration\Preferences\Control Panel Settings\Scheduled Tasks\` +10. Double click "LME-Sysmon-Task," then switch to the "Actions" tab. +11. Click "Start a program," then "Edit." +12. Under "Program/Script," click "Browse," then find and select the "update.bat" file, within the SYSVOL folder (see Figure 2). **NOTE:** the SYSVOL path needs to be manually changed to be in the format of a network path. It **cannot** begin with "C:\\Windows". See Figure 2 for clarification. +13. Click "Apply" to apply the changes to the GPO. + + ++Figure 2: Specify the path to the update.bat file as the action for the scheduled test. +
+ +At this point, the GPO should be properly configured, but without additional intervention, it could take up to 24 hours for the scheduled task to activate. Before it does, Sysmon will not show up as a service on the clients. However, further steps can be taken to ensure immediate installation. +- View the "Triggers" tab of the "LME-Sysmon-Task-Properties" page. Click "Daily," then "Edit..." Note the start time specified. Each day, starting at that specific time, the LME-Sysmon-Task will run, repeating every 30 minutes. If that time has already passed on the day you created the GPO, the task won't activate for the first time until the following day. Generally speaking, you'll want to set the time to the beginning of the day for complete coverage, but you may consider adjusting it temporarily for testing purposes so that it will activate while you can observe it. +- By default, Windows will update group policy settings only every 90 minutes. You can manually trigger a group policy update by running `gpupdate /force` in an elevated Command Prompt window on a given client to apply the GPO to that specific client immediately. + + +## 2.3 SCCM Deployment +While SCCM deployment is not usually the first choice for the deployment of Sysmon we have included an example install and uninstall PowerShell along with a detection criteria that works with SCCM. + +Files for this portion of the tutorial can be found [here](/Chapter%202%20Files/SCCM%20Deployment/). + +Install Program: +```powershell.exe -Executionpolicy unrestricted -file Install_Sysmon64.ps1``` + +Uninstall program: +```powershell.exe -Executionpolicy unrestricted -file Uninstall_Sysmon64.ps1``` + +Detection method: `File exists - C:\Windows\sysmon64.exe` + +## Chapter 2 - Checklist +1. Ensure that your files and folders in the network share are nested and named correctly. Remember that in Windows, case in filenames or folders does not matter. + +``` +NETWORK_SHARE (e.g. SYSVOL) +└── LME + ├── Sysmon + ├── Sysmon64.exe + ├── sysmon.xml + └── update.bat + └── sigcheck64.exe +``` + +2. Do you have the Sysmon service running on a sample of the clients? You can verify this by logging in to one of the clients and pressing Windows key + R, running "services.msc," and searching to see if Sysmon is listed as an active service. +3. Is the Sysmon Eventlog showing data? On one of the clients, open Event Viewer and look in Applications and Services Logs/Microsoft/Windows/Sysmon/Operational. +4. Are you seeing Sysmon logs show up on the Event Collector? On the Event Collector, open Event Viewer and look in the Windows Logs/Forwarded Events folder. + +If any problems are found, restart all of your machines and see [Troubleshooting | Chapter 2 - Installing Sysmon](reference/troubleshooting.md#chapter-2---installing-sysmon) for additional tips. + +## Now move onto [Chapter 3 - Installing the ELK Stack and Retrieving Logs](/docs/markdown/chapter3/chapter3.md) diff --git a/docs/markdown/chapter3/chapter3.md b/docs/markdown/chapter3/chapter3.md new file mode 100644 index 00000000..3a74c219 --- /dev/null +++ b/docs/markdown/chapter3/chapter3.md @@ -0,0 +1,272 @@ +# Chapter 3 – Installing the ELK Stack and Retrieving Logs + +## Chapter Overview +In this chapter you will: +* Install a new Linux server for events to be sent to. +* Run a script to: + * install Docker. + * secure the Linux server. + * secure the Elasticsearch server. + * generate certificates. + * deploy the LME Docker stack. +* Configure the Windows Event Collector to send logs to the Linux server. + +## Introduction +This section covers the installation and configuration of the Database and search functionality on a Linux server. We will install the ‘ELK’ Stack from Elasticsearch for this portion. + +What is the ELK Stack? +"ELK" is the acronym for three free and open projects: Elasticsearch, Logstash, and Kibana. Elasticsearch is a search and analytics engine. Logstash is a server‑side data processing pipeline that ingests data from multiple sources simultaneously, transforms it, and then sends it to a "stash" like Elasticsearch. Kibana lets users visualize data with charts and graphs in Elasticsearch. + + ++Figure 1: Elastic Stack components +
+ +Elasticsearch, Logstash, Kibana, and Winlogbeat are developed by [Elastic](https://www.elastic.co/). Before following this guide and running our install script, you should review and ensure that you agree with the license terms associated with these products. Elastic’s license terms can be found on their GitHub page [here](https://github.com/elastic). By running our install script you are agreeing to Elastic’s terms. + +This script also makes use of use of Docker Community Edition (CE). By following this guide and using our install script you are agreeing to the Docker CE license, which can be found [here](https://github.com/docker/docker-ce/blob/master/LICENSE). + +## 3.1 Getting Started +During the installation guide below you will see that the majority of steps are carried out automatically. Commands or file paths are highlighted in grey boxes. + +You will need a Linux box for this portion, **The deploy script is only tested on Ubuntu Long Term Support (LTS) editions that are currently supported by Docker ([see here](https://docs.docker.com/engine/install/ubuntu/)).** In addition, only installation on a single server is supported. Please see [the resilience documentation](/docs/markdown/chapter3/resilience.md) for more details. + +### 3.1.1 Firewall Rules +You will need port 5044 open for the event collector to send data into the database (on the Linux server). To be able to access the web interface you will need to have firewall rules in place to allow access to port 443 (HTTPS) on the Linux server. + +### 3.1.2 Web Proxy Settings +If the ELK stack is being deployed behind a web proxy and Docker isn't configured to use the proxy, the deploy script can hang without completing due to Docker being unable to pull the required images. + +**If your setup does not include a web proxy, skip straight to step 3.2.** + +Otherwise, to configure Docker to use the web proxy in your environment, do the following before running the deployment script: + +1. Determine the IP address and port of the proxy. +2. Create a systemd drop-in directory for the Docker service: +``` +sudo mkdir -p /etc/systemd/system/docker.service.d +``` +3. Create a file named /etc/systemd/system/docker.service.d/http-proxy.conf that adds the HTTP_PROXY and HTTPS_PROXY environment variables (keep/delete as required for your environment, substituting the IP address/port determined in step 1): +``` +[Service] +Environment="HTTP_PROXY=http://[proxy address or IP]:[proxy port]" +Environment="HTTPS_PROXY=https://[proxy address or IP]:[proxy port]" +``` +4. Reload the service daemon: +``` +sudo systemctl daemon-reload +``` + +Check the [official Docker documentation](https://docs.docker.com/config/daemon/systemd/#httphttps-proxy) for this process, including details on how to bypass the proxy if you have internal image registries which need to be reachable from this host. + +## 3.2 Install LME the easy way using our script + +### 3.2.1 Preparing to Run the Script + +At the time of writing, security updates are only supported for Ubuntu, so please install Ubuntu on a new virtual or physical machine. You may have already done this as part of the pre-requisites in the initial readme file. + +You will also need the IP address and domain name of the Linux server to run the install script. + +To find the IP address, run `ip addr` from the Linux server and look for the IP address after the indicator `inet`. The IP address needs to be reachable from the event collector. See [What firewall rules are needed?](/docs/markdown/prerequisites.md#what-firewall-rules-are-needed) for more details. + +The domain name needs to be resolvable from the event collector. If you're unsure what the server's domain name is, in some cases, it may just be the hostname of local machine, which you can find by running `hostname` from the Linux server. To verify if this is resolvable from the event collector, open PowerShell on the event collector and run `Resolve-DnsName MYDOMAINNAME`, where "MYDOMAINNAME" refers to the domain name of the Linux server. If successful, it will return the IP address of the Linux server. If not, an error such as "DNS name does not exist" error will be returned. In this case, you may need to add a DNS record on the domain controller that points to the Linux server. See [Manage DNS resource records](https://learn.microsoft.com/en-us/windows-server/networking/dns/manage-resource-records?tabs=powershell) to learn more about doing this. + +### 3.2.2 Running the Script + +**The script will prompt for the following:** + +1. Confirmation of intrusive actions that will modify your system docker and apt installed files. +2. Asking for input of the IP address of the local machine. It should automatically populate it with the server's correct local IP address on your network. If not, fill in the IP you found in [Section 3.2.1](#321-preparing-to-run-the-script). +3. Asking for input of the Fully-qualified Domain Name (aka `hostname`) of the local machine (the ELK server). Type in the ELK server's domain name you determined in [Section 3.2.1](#321-preparing-to-run-the-script). +4. Presenting the option of automatically generating self-signed TLS certificates or importing pre-generated certificates. By default self-signed certificates will be used, which will have a validity of two years from the date of install, after which they will need to be renewed. +5. Skipping the Docker installation process. This is available for the case that you already have docker installed. +6. An old elastic user password. If you are installing on top of a previous LME installation, you will need to provide your old LME elastic user password, so the install can properly authenticate with your previous systems. + +Now that you have an Ubuntu machine ready to go as well as its local IP address and hostname, SSH into your Linux server and run the following commands to install LME: + +``` +# Install Git client to be able to clone the LME repository +sudo apt update +sudo apt install git -y +# Download a copy of the LME files +sudo git clone https://github.com/cisagov/lme.git /opt/lme/ +# Change to the LME directory containing files for the Linux server +cd /opt/lme/Chapter\ 3\ Files/ +# Execute script with root privileges +sudo ./deploy.sh install +``` + +Running the above commands will: + +1. Enable auto security updates (Ubuntu Only) +2. Update the system + - Note that the script may request a reboot after running initial updates, especially if it's a new system or one that has not been updated for a long time. Reboot the system and run the script again to continue. +3. Generate TLS certificates. (Optional) +4. Install Docker Community Edition. + - Note that this action is destructive and assumes docker is not installed. Either indicate in the prompt you wish to skip installing docker **OR** uninstall docker before proceeding +5. Configure Docker to run ELK. +6. Change Elasticsearch configuration, including retention based upon disk size. +7. update read/write permission recursively on `/opt/lme` so that only the owner can read the files in that directory. This ensures only root can read the files that get created/written during deploy.sh. If you created that directory as root you will have permission errors. Access the directory using a root shell OR change the permissions for the `/opt/lme` directory so that a regular user can read it if you desire. + + +For details on how to regenerate these certificates, or for instructions in generating and importing certificates from an existing root Certificate Authority (CA) please see the full [certificates documentation](/docs/markdown/maintenance/certificates.md). + +After the script finishes running, it will output a number of usernames and passwords for use when accessing the dashboard and for the internal systems. + +The usernames and passwords will be provided in a message similar to below. + +``` +################################################################################## +## Kibana/Elasticsearch Credentials are (these will not be accessible again!!!!) ## +## +## Web Interface login: +## elastic:+Figure 4: WinSCP Login Prompt +
+ + - If you have a keyfile instead of a password (for example, when accessing AWS servers), see [this article](https://docs.aws.amazon.com/transfer/latest/userguide/getting-started-use-the-service.html). + +#### Method 2: Windows Native SCP +SFTP and SCP have been bundled in Windows since 2018 and will suffice if you're comfortable with a command line. To download the files from the ELK server to your desktop, run the following in a powershell window on the Event Collector, filling in `+Figure 3: Winlogbeat Install Location +
+ +Then, move the 'winlogbeat.yml' file located at ```C:\Program Files\lme\winlogbeat.yml``` into the winlogbeat folder ```C:\Program Files\lme\winlogbeat-8.[x].[y]-windows-x86_64```, overwriting the existing file when prompted to do so. + +Now, open PowerShell as an administrator and run the following command from the winlogbeat directory, allowing the script to run if prompted to do so: ```./install-service-winlogbeat.ps1``` + + ++Figure 4: Winlogbeat Install Script + +Then in the same PowerShell window start the winlogbeat service by running: + +``` +Start-Service winlogbeat +``` + +Lastly, open ```services.msc``` as an administrator, and make sure the winlogbeat service is installed, is set to start automatically, and is running: + + +
+Figure 5: Winlogbeat Service Running
+
+
+## Trusting the certs that secure LME's services
+
+Theres a few steps we need to follow to trust the self-signed cert:
+1. Grab the self-signed certificate authority for LME (done in step [3.2.4](docs/markdown/chapter3/chapter3.md#324-download-files-for-windows-event-collector)).
+2. Have our clients trust the certificate authority (see command below).
+
+This will trust the self signed cert and any other certificates it signs. If this certificate is stolen by an attacker, they can use it to trick your browser into trusting any website they setup. Make sure this cert is kept safe and secure.
+
+We've already downloaded the self-signed cert in previous steps in Chapter 3, so now we just need to tell Windows to trust the certificates our self-signed cert has setup for our LME services.
+
+### Commands:
+These commands should be run on every computer that will access the Kibana front end for LME's Elastic deployment. (i.e https://
+
+Figure 2 - The LME NEW - User Security - Overview
+
+
+***Troubleshooting***
+
+Should problems arise during the reinstallation of Winlogbeat, the relevant logs can be found in ```%programdata%/winlogbeat/``` which may help identify any issues.
+
+## Using Your Own Certificates
+It is possible to use certificates signed by an existing root CA as part of the LME install by generating certificates manually with the correct settings and placing these within the required directory inside the LME folder. **NOTE: The default supported method of LME installation is to use the automatically created self-signed certificates, and we will be unable to support any problems that arise from generating the certificates manually incorrectly.**
+
+### Certificate Creation
+
+The exact method for generating and configuring these certificates will vary dependent upon the method you have used to create your root CA and currently manage certificates in your enterprise. However you choose to generate these, you will need the following certificates to successfully deploy LME (further information on the exact requirements can be found by inspecting the certificate generation methods within the [deploy script](/Chapter%203%20Files/deploy.sh) in Chapter 3 if required):
+
+***Elasticsearch***
+
+This certificate must only be created to peform server authentication and not signing. The certificate must have ```elasticsearch``` as the CommonName and the DNS name ```elasticsearch``` and the IP address ```127.0.0.1``` within its SubjectAltName. If there is a requirement to access Elasticsearch directly from an external perspective the certificate may also have an additional SubjectAltName containing the DNS name of the LME host and its IP address.
+
+***Kibana***
+
+This certificate must only be created to peform server authentication and not signing. The certificate should have a CommonName of ```kibana``` and must have the FQDN of the LME server set as the SubjectAltName. If desired, the server's IP address, the IP address ```127.0.0.1``` or the DNS name ```kibana``` can be set in the SubjectAltName.
+
+***Logstash***
+
+This certificate must only be created to peform server authentication and not signing. The certificate's CommonName must have the FQDN of the LME server set. If desired, the server's DNS name and IP address can be set in the SubjectAltName.
+
+***Winlogbeat***
+
+This certificate must only be created to perform client authentication and not signing. The certificate enables authentication between the Winlogbeat client and the Logstash endpoint. It should be set with the CommonName ```wlbclient```, a SubjectAltName is not required.
+
+### Certificate Locations
+
+Once you have successfully created the required certificates they must be placed in the following locations:
+
+***CA Certificate***
+
+```
+/opt/lme/Chapter\ 3\ Files/certs/root-ca.crt
+```
+
+***Logstash Certificate***
+```
+/opt/lme/Chapter\ 3\ Files/certs/logstash.key
+/opt/lme/Chapter\ 3\ Files/certs/logstash.crt
+```
+
+***Elasticsearch Certificate***
+```
+/opt/lme/Chapter\ 3\ Files/certs/elasticsearch.key
+/opt/lme/Chapter\ 3\ Files/certs/elasticsearch.crt
+```
+
+***Kibana Certificate***
+```
+/opt/lme/Chapter\ 3\ Files/certs/kibana.key
+/opt/lme/Chapter\ 3\ Files/certs/kibana.crt
+```
+
+***Winlogbeat Client Certificate***
+
+In order for the Winlogbeat client certificate to be included in the ```files_for_windows.zip``` file generated by the installer, please ensure they are present as below:
+```
+/opt/lme/Chapter\ 3\ Files/certs/wlbclient.key
+/opt/lme/Chapter\ 3\ Files/certs/wlbclient.crt
+```
+Alternatively these files can be transfered to the Windows Event Collector server separately if desired.
+
+### Installation
+
+Once the certificates have been generated as required and copied into the correct location, simply run the installer as instructed in [Chapter 3](/docs/markdown/chapter3/chapter3.md), selecting "No" when prompted to generate self-signed certificates. The installer should then ensure that the files are in the correct location and proceed as normal, making use of the manually created certificates instead.
+
+## Migrating from Self-Signed Certificates
+
+It is possible to migrate from the default self-signed certificates to manually generated certificates at a later date, for example to move to enterprise certificates post-installation after an initial testing period. This can be done by taking advantage of the "renew" functionality within the deploy script to replace the certificates once they are in the correct place.
+
+**NOTE: The default supported method of LME installation is to use the automatically created self-signed certificates, and we will be unable to support any problems that arise from generating the certificates manually incorrectly.**
+
+To begin this process you will need to generate the required certificates that you intend to use as part of the LME installation going forward. The certificates must meet the requirements set out above under [Certificate Creation](#certificate-creation).
+
+Once the required certificates have been created they must be copied into the correct location, as described in the [Certificate Location](#certificate-locations) section above. If you have an existing installation with self-signed certificates then files will already exist in these locations, and will need to be overwritten with the newly created certificate files.
+
+Once the certificate files have been copied into the correct locations calling the deploy script's "renew" function and prompting it **not** to regenerate any of the certificates will cause it to replace the currently in-use certificates with the newly copied files:
+
+```
+cd /opt/lme/Chapter\ 3\ Files/
+sudo ./deploy.sh renew
+```
+
+```bash
+Do you want to regenerate the root Certificate Authority (warning - this will invalidate all current certificates in use) ([y]es/[n]o): n
+Do you want to regenerate the Logstash certificate ([y]es/[n]o): n
+Do you want to regenerate the Elasticsearch certificate ([y]es/[n]o): n
+Do you want to regenerate the Kibana certificate ([y]es/[n]o): n
+Do you want to regenerate the Winlogbeat client certificate (warning - you will need to re-install Winlogbeat with the new certificate on the WEC server if you do this) ([y]es/[n]o): n
+```
+
+Once this is done Winlogbeat will need to be modified to use the newly created client certificate, as detailed in the [Re-configure Winlogbeat](#re-configure-winlogbeat) section above, substituting your manually created client certificate and key for those stored in the ```new_client_certificates.zip``` file.
+
+
diff --git a/docs/markdown/maintenance/painless-reindex.txt b/docs/markdown/maintenance/painless-reindex.txt
new file mode 100644
index 00000000..3092e357
--- /dev/null
+++ b/docs/markdown/maintenance/painless-reindex.txt
@@ -0,0 +1,13 @@
+POST _reindex?wait_for_completion=false
+{
+ "source": {
+ "index": "winlogbeat-*"
+ },
+ "dest": {
+ "index": "winlogbeat"
+ },
+ "script": {
+ "lang": "painless",
+ "source": "ctx._index = 'winlogbeat-' + (ctx._index.substring('winlogbeat-'.length(), ctx._index.length())) + '-1'"
+ }
+}
\ No newline at end of file
diff --git a/docs/markdown/maintenance/upgrading.md b/docs/markdown/maintenance/upgrading.md
new file mode 100644
index 00000000..bdb28433
--- /dev/null
+++ b/docs/markdown/maintenance/upgrading.md
@@ -0,0 +1,111 @@
+# Upgrading
+
+## 1 Upgrade Paths
+LME is currently on v1.0.
+
+Below you can find the upgrade paths that are currently supported and what steps are required for these upgrades. Note that major version upgrades tend to include significant changes, and so will require manual intervention and will not be automatically applied, even if auto-updates are enabled.
+
+Applying these changes is automated for any new installations. But, if you have an existing installation, you need to conduct some extra steps. **Before performing any of these steps it is advised to take a backup of the current installation using the method described [here](/docs/markdown/maintenance/backups.md).**
+
+### 1.1 Upgrade from v0.5
+
+Since LME's transition from the NCSC to CISA, the location of the LME repository has changed from `https://github.com/ukncsc/lme` to `https://github.com/cisagov/lme`. To obtain any further updates to LME on the ELK server, you will need to transition to the new git repository. Because vital configuration files are stored within the same folder as the git repo, it's simpler to copy the old LME folder to a different location, clone the new repo, copy the files and folders unique to your system, and then optionally delete the old folder. You can do this by running the following commands:
+
+
+```
+sudo mv /opt/lme /opt/lme_old
+sudo git clone https://github.com/cisagov/lme.git /opt/lme
+sudo cp -r /opt/lme_old/Chapter\ 3\ Files/certs/ /opt/lme/Chapter\ 3\ Files/
+sudo cp /opt/lme_old/Chapter\ 3\ Files/docker-compose-stack-live.yml /opt/lme/Chapter\ 3\ Files/
+sudo cp /opt/lme_old/Chapter\ 3\ Files/get-docker.sh /opt/lme/Chapter\ 3\ Files/
+sudo cp /opt/lme_old/Chapter\ 3\ Files/logstash.edited.conf /opt/lme/Chapter\ 3\ Files/
+sudo cp /opt/lme_old/files_for_windows.zip /opt/lme/
+sudo cp /opt/lme_old/lme.conf /opt/lme/
+sudo cp /opt/lme_old/lme_update.sh /opt/lme/
+```
+Finally, you'll need to grab your old dashbouard_update password and add it into the new dashboard_update script:
+```
+OLD_Password=[OLD_PASSWORD_HERE]
+sudo cp /opt/lme/Chapter\ 3\ Files/dashboard_update.sh /opt/lme/
+sed -i "s/dashboardupdatepassword/$OLD_Password/g" /opt/lme/dashboard_update.sh
+```
+
+#### 1.1.1 ELK Stack Update
+You can update the ELK stack portion of LME to v1.0 (including dashboards and ELK stack containers) by running the following on the Linux server:
+
+```
+cd /opt/lme/Chapter\ 3\ Files/
+sudo ./deploy.sh update
+```
+**The last step of this script makes all files only readable by their owner in /opt/lme, so that all root owned files with passwords in them are only readable by root. This prevents a local unprivileged user from gaining access to the elastic stack.**
+
+Once the deploy update is finished, next update the dashboards that are provided alongside LME to the latest version. This can be done by running the below script, with more detailed instructions available [here](/docs/markdown/chapter4.md#411-import-initial-dashboards):
+
+\*\**NOTE:*\*\* *You may need to wait several minutes for Kibana to successfully initialize after the update before running this script during the upgrade process. If you encounter a "Failed to connect" error or an "Entity Too Large" error wait for several minutes before trying again.*
+
+##### Optional Substep: Clear out old dashboards
+**Skip this step if you don't want to clear out the old dashboards**
+
+The LME team will not be maintaining any old dashboards from the old NCSC LME version, so if you would like to clean up your LME you can remove the dashboards by navigating to: https://
+Figure 1: High level overview, linking to documentation chapters
+
+Figure 1: Overview of Network rules
+
+Figure 1: Troubleshooting overview diagram
+
+ Figure 2: Add a feature
+
+ Figure 3: Install RSAT: Group Policy Management Tools
+
+Figure 4 - Steps to import objects
+
+