v1.7.2

## v1.7.2 ||| |---|---| |Date|2024-02-08 |Kind| MINOR release |Author|mschiess@akamai.com - **Minor improvements** - Introduced **Secure Internet Access** (formerly ETP) as INPUT specification (as an alias to ETP) - added "ETP NETCON" to the autoresume feature - prevented "EAA DIRHEALTH" to be mistakenly autoresumed - Imrpoved [log overview](LOG_OVERVIEW.md) readability - Added `--debugloglines` to allow control of input loglines being sent to the debug log - Added [Microsoft Sentinel SIA / ETP integration](SIEM/SENTINEL/Readme.md) documentation - Added a FAQ entry [regarding time synchronization](FAQ.md#error-invalid-timestamp-on-api-call) - [docker] bumped python version to "3.12.2" - [docker] bumped gc_logs version to "0.0.3(beta)" - Fixed a doc error (PR by @pizza0rodeo ) - thanks for your contribution - **BUGFIX** - Fixed a bug in the autoresume function that created a problem with timezones in certain circumstances - **Housekeeping** - improved local container testing
akamai · Feb 8, 2024 · 52174bd · 52174bd
1 parent 2efdc34
commit 52174bd
Show file tree

Hide file tree

Showing 26 changed files with 642 additions and 71 deletions.
diff --git a/Dockerfile b/Dockerfile
@@ -1,4 +1,4 @@
-FROM            python:3.12.0-slim-bookworm
+FROM            python:3.12.2-slim-bookworm
 LABEL           MAINTAINER="Mike Schiessl - mike.schiessl@akamai.com"
 LABEL	        APP_LONG="Akamai Universal Log Streamer"
 LABEL           APP_SHORT="ULS"
@@ -13,7 +13,7 @@ ARG             EXT_DIR="$ULS_DIR/ext"
 ARG             ETP_CLI_VERSION="0.4.5"
 ARG             EAA_CLI_VERSION="0.6.3"
 ARG             MFA_CLI_VERSION="0.1.1"
-ARG             GC_CLI_VERSION="v0.0.2(beta)"
+ARG             GC_CLI_VERSION="v0.0.3(beta)"
 ARG             LINODE_CLI_VERSION="dev"
 
 # ENV VARS

diff --git a/README.md b/README.md
@@ -4,7 +4,7 @@
 
 The Unified Log Streamer (ULS) is designed to simplify SIEM integrations for Akamai Secure Enterprise Access Products
 - [Enterprise Application Access (EAA)](https://www.akamai.com/us/en/products/security/enterprise-application-access.jsp) 
-- [Enterprise Threat Protector (ETP)](https://www.akamai.com/us/en/products/security/enterprise-threat-protector.jsp)
+- [Secure Internet Access Enterprise (SIA)](https://www.akamai.com/us/en/products/security/enterprise-threat-protector.jsp)
 - [Akamai MFA (MFA)](https://www.akamai.com/us/en/products/security/akamai-mfa.jsp)
 - [Guardicore Micro Segmentation](https://www.akamai.com/lp/guardicore)
 
@@ -39,7 +39,7 @@ It can be run directly as Python code, as a provided Docker container,  through
       - [CONHEALTH](docs/LOG_OVERVIEW.md#connector-health-conhealth)
       - [DEVINV](docs/LOG_OVERVIEW.md#device-inventory-devinv)
       - [DIRHEALTH](docs/LOG_OVERVIEW.md#directory-health-dirhealth)
-    - [Enterprise Threat Protectors (ETP)](https://www.akamai.com/us/en/products/security/enterprise-threat-protector.jsp)
+    - [Secure Internet Access Enterprise (SIA) (formerly Enterprise Threat Protector)](https://www.akamai.com/products/secure-internet-access-enterprise)
       - [THREAT](docs/LOG_OVERVIEW.md#threat-log-threat)
       - [AUP](docs/LOG_OVERVIEW.md#accceptable-use-policy-logs-aup)
       - [DNS](docs/LOG_OVERVIEW.md#dns)
@@ -53,7 +53,7 @@ It can be run directly as Python code, as a provided Docker container,  through
       - AGENT
       - SYSTEM
     - [Linode](https://www.linode.com/) (experimental)
-      - AUDIT
+      - [AUDIT](docs/LOG_OVERVIEW.md#audit-logs)
 
 
 - Supported data outputs
@@ -93,6 +93,7 @@ Anyway, details for some specific SIEM solutions can be found in [this directory
 - AKAMAI .edgerc file ([see further documentation here](docs/AKAMAI_API_CREDENTIALS.md))
 - Outbound Internet access (Akamai API, Dockerhub OR Python repositories)
 - Compute resource footprint is minimal: 1 vCPU, 1 GB RAM, 500 MB free disk. See also [High availability](docs/HIGH_AVAILABILITY.md) doc.
+- NTP synchronized time on the OS ULS will be executed.
 
 ### Command Line Usage
 ![ULS command line usage](docs/images/uls_cli_help_example.png)  

diff --git a/bin/modules/UlsArgsParser.py b/bin/modules/UlsArgsParser.py
@@ -16,20 +16,28 @@
 import os
 from distutils.util import strtobool
 
-import config.global_config as uls_config
+import uls_config.global_config as uls_config
 
 
 def init():
     # Argument Parsing
     parser = argparse.ArgumentParser(description=f"{uls_config.__tool_name_long__}",
                                      formatter_class=argparse.RawTextHelpFormatter)
-    # Common params
+    # Loglevel
     parser.add_argument('-l', '--loglevel',
                         action='store',
                         type=str.upper,
                         default=(os.environ.get('ULS_LOGLEVEL') or uls_config.log_level_default),
                         choices=uls_config.log_levels_available,
                         help=f'Adjust the loglevel Default: {uls_config.log_level_default}')
+    # put loglines into debug log
+    parser.add_argument('--debugloglines',
+                        action='store',
+                        type=bool,
+                        nargs='?',
+                        default=(os.environ.get('ULS_DEBUGLOGLINES') or uls_config.log_debugloglines_default),
+                        const=True,
+                        help=f"Should the log_lines appear in debug log? (Default: {uls_config.log_debugloglines_default})")
 
     # Version Information
     parser.add_argument('-v', '--version',

diff --git a/bin/modules/UlsInputCli.py b/bin/modules/UlsInputCli.py
@@ -21,7 +21,7 @@
 import queue
 
 # ULS modules
-import config.global_config as uls_config
+import uls_config.global_config as uls_config
 import modules.aka_log as aka_log
 import modules.UlsTools as UlsTools
 
@@ -210,7 +210,7 @@ def proc_create(self):
                                   shlex.split(self.rawcmd)
 
             # ETP config
-            elif self.product == "ETP":
+            elif self.product == "ETP" or self.product == "SIA":
                 product_path = self.root_path + "/" + uls_config.bin_etp_cli
                 product_feeds = uls_config.etp_cli_feeds
 
@@ -229,9 +229,11 @@ def proc_create(self):
                     # Append End and Starttime
                     if self.endtime:
                         # We need to remove "-f" from the end of the cli cmd if we work with endtime
+                        aka_log.log.debug(f"Internally set my starttime to: {self.endtime}")
                         cli_command = cli_command[:-1]
                         cli_command.extend(self._prep_start_endtime('--end', self.endtime))
                     if self.starttime:
+                        aka_log.log.debug(f"Internally set my starttime to: {self.starttime}")
                         cli_command.extend(self._prep_start_endtime('--start', self.starttime))
 
                 else:
@@ -328,6 +330,9 @@ def proc_create(self):
                     cli_command = [self.bin_python, product_path] +\
                                   self._uls_useragent(self.product, "rawcmd") +\
                                   shlex.split(self.rawcmd)
+            # Mocked output
+            elif self.product == "MOCK":
+                print ("Not yet there")
 
 
             # Everything else (undefined)

diff --git a/bin/modules/UlsMonitoring.py b/bin/modules/UlsMonitoring.py
@@ -19,7 +19,7 @@
 import sys
 
 import modules.aka_log as aka_log
-import config.global_config as uls_config
+import uls_config.global_config as uls_config
 
 
 class UlsMonitoring:

diff --git a/bin/modules/UlsOutput.py b/bin/modules/UlsOutput.py
@@ -25,7 +25,7 @@
 import json
 
 # ULS specific modules
-import config.global_config as uls_config
+import uls_config.global_config as uls_config
 import modules.aka_log as aka_log
 
 

diff --git a/bin/modules/UlsTools.py b/bin/modules/UlsTools.py
@@ -23,7 +23,7 @@
 
 # ULS modules
 import modules.aka_log as aka_log
-import config.global_config as uls_config
+import uls_config.global_config as uls_config
 
 
 def uls_check_sys(root_path, uls_input=None):
@@ -48,7 +48,7 @@ def _check_cli_installed(cli_bin):
             aka_log.log.critical(f"Error checking the cli'tools ")
     if uls_input == "EAA":
         _check_cli_installed(root_path + "/" + uls_config.bin_eaa_cli)
-    elif uls_input == "ETP":
+    elif uls_input == "ETP" or uls_input == "SIA":
         _check_cli_installed(root_path + "/" + uls_config.bin_etp_cli)
     elif uls_input == "MFA":
         _check_cli_installed(root_path + "/" + uls_config.bin_mfa_cli)
@@ -97,7 +97,7 @@ def _get_cli_version(cli_bin, edgerc_mock_file):
     print(f"{uls_config.__tool_name_long__} Version information\n"
           f"ULS Version\t\t{uls_config.__version__}\n\n"
           f"EAA Version\t\t{_get_cli_version(root_path + '/' + uls_config.bin_eaa_cli, my_edgerc_mock_file)}\n"
-          f"ETP Version\t\t{_get_cli_version(root_path + '/' + uls_config.bin_etp_cli, my_edgerc_mock_file)}\n"
+          f"SIA/ETP Version\t\t{_get_cli_version(root_path + '/' + uls_config.bin_etp_cli, my_edgerc_mock_file)}\n"
           f"MFA Version\t\t{_get_cli_version(root_path + '/' + uls_config.bin_mfa_cli, my_edgerc_mock_file)}\n"
           f"GC Version\t\t{_get_cli_version(root_path + '/' + uls_config.bin_gc_cli, my_edgerc_mock_file)}\n"
           f"LINODE Version\t\t{_get_cli_version(root_path + '/' + uls_config.bin_linode_cli, my_edgerc_mock_file)}\n\n"
@@ -187,7 +187,10 @@ def root_path():
 
 def check_autoresume(input, feed, checkpoint_dir=uls_config.autoresume_checkpoint_path):
     # Check if we're in a supported stream / feed
-    if input not in uls_config.autoresume_supported_inputs or feed == "CONHEALTH" or feed == "DEVINV" :
+    if (input not in uls_config.autoresume_supported_inputs or
+            feed == "CONHEALTH" or
+            feed == "DEVINV" or
+            feed == "DIRHEALTH"):
         aka_log.log.critical(f"Input {input} or feed {feed} currently not supported by AUTORESUME - Exiting.")
         sys.exit(1)
 
@@ -216,13 +219,15 @@ def check_autoresume(input, feed, checkpoint_dir=uls_config.autoresume_checkpoin
                             aka_log.log.critical(
                                 f"Unhandeled input data in checkpointfile  \'{checkpoint_full}\' --> {input} / {feed} - Exiting.")
                             sys.exit(1)
-                        checkpoint = int(datetime.datetime(year=int(mytime.split("T")[0].split("-")[0]),
-                                            month=int(mytime.split("T")[0].split("-")[1]),
-                                            day=int(mytime.split("T")[0].split("-")[2]),
-                                            hour=int(mytime.split("T")[1].split(":")[0]),
-                                            minute=int(mytime.split("T")[1].split(":")[1]),
-                                            second=int(mytime.split("T")[1].split(":")[2]),
-                                            ).timestamp())
+                        my_timestamp = datetime.datetime(year=int(mytime.split("T")[0].split("-")[0]),
+                                                         month=int(mytime.split("T")[0].split("-")[1]),
+                                                         day=int(mytime.split("T")[0].split("-")[2]),
+                                                         hour=int(mytime.split("T")[1].split(":")[0]),
+                                                         minute=int(mytime.split("T")[1].split(":")[1]),
+                                                         second=int(mytime.split("T")[1].split(":")[2]),
+                                                         )
+                        checkpoint = int(my_timestamp.replace(tzinfo=datetime.timezone.utc).timestamp())
+
                         aka_log.log.debug(f"Checkpoint timestamp {data['checkpoint']} converted to epoch time {checkpoint}")
                     else:
                         aka_log.log.critical(f"Inconsitent data in checkpointfile  \'{checkpoint_full}\' --> {data} - Exiting.")
@@ -245,7 +250,7 @@ def check_autoresume(input, feed, checkpoint_dir=uls_config.autoresume_checkpoin
     return {'filename': checkpoint_full, 'creation_time': creation_time, 'checkpoint': checkpoint}
 
 
-def write_autoresume_ckpt(input, feed, autoresume_file, logline):
+def write_autoresume_ckpt(input, feed, autoresume_file, logline, current_count):
     aka_log.log.info(f"AUTORESUME - IT's time to write a new checkpoint")
 
     # Adopt the field to the stream / feed
@@ -256,14 +261,16 @@ def write_autoresume_ckpt(input, feed, autoresume_file, logline):
         checkpoint_timestamp = json.loads(checkpoint_line)['query']['time']
     elif input == "EAA" and feed == "ACCESS":
         checkpoint_timestamp = json.loads(checkpoint_line)['datetime']
+    elif input == "ETP" and feed == "NETCON":
+        checkpoint_timestamp = json.loads(checkpoint_line)['connStartTime']
     else:
         aka_log.log.critical(
             f"AUTORESUME - Unhandled Input / Feed detected:  '{input} / {feed}' (this should never happen !!)- Exiting")
         sys.exit(1)
 
     # Write out the file
     try:
-        autoresume_data = {'creation_time': str(datetime.datetime.now()), 'checkpoint': str(checkpoint_timestamp), 'input': input, 'feed': feed}
+        autoresume_data = {'creation_time': str(datetime.datetime.now()), 'checkpoint': str(checkpoint_timestamp), 'input': input, 'feed': feed, 'current_count': current_count}
         with open(autoresume_file, "w") as ckpt_fd:
             json.dump(autoresume_data, ckpt_fd)
         aka_log.log.debug(f"AUTORESUME - Wrote a new checkpoint to {autoresume_file}: {autoresume_data}")

diff --git a/bin/modules/UlsTransformation.py b/bin/modules/UlsTransformation.py
@@ -18,7 +18,7 @@
 import time
 
 # ULS modules
-import config.transformation_config as transform_config
+import uls_config.transformation_config as transform_config
 import jmespath
 import modules.aka_log as aka_log
 

diff --git a/bin/uls.py b/bin/uls.py
@@ -30,7 +30,7 @@
 import modules.UlsMonitoring as UlsMonitoring
 import modules.UlsTransformation as UlsTransformation
 import modules.UlsTools as UlsTools
-import config.global_config as uls_config
+import uls_config.global_config as uls_config
 
 stopEvent = threading.Event()
 
@@ -86,7 +86,6 @@ def main():
         autoresume_data = UlsTools.check_autoresume(input=uls_args.input, feed=uls_args.feed, checkpoint_dir=uls_args.autoresumepath)
         uls_args.starttime = autoresume_data['checkpoint']
         autoresume_file =  autoresume_data['filename']
-
     # Check CLI Environment
     UlsTools.uls_check_sys(root_path=root_path, uls_input=uls_args.input)
 
@@ -170,12 +169,19 @@ def main():
     while not stopEvent.is_set():
         try:
             input_data = event_q.get(block=True, timeout=0.05)
-            aka_log.log.debug(f"<IN> {input_data}")
+            if uls_args.debugloglines:
+                aka_log.log.debug(f"<IN> {input_data}")
             for log_line in input_data.splitlines():
 
                 # Write checkpoint to the checkpoint file (if autoresume is enabled) (not after transformation or filter)
+
                 if uls_args.autoresume and int(my_monitor.get_message_count()) >= autoresume_lastwrite + uls_args.autoresumewriteafter:
-                    UlsTools.write_autoresume_ckpt(uls_args.input, uls_args.feed, autoresume_file, log_line)
+                    aka_log.log.info(f"WRITING AUTORESUME CHECKPOINT - curr_message_count={int(my_monitor.get_message_count())} - last_write = {autoresume_lastwrite}")
+                    UlsTools.write_autoresume_ckpt(uls_args.input,
+                                                   uls_args.feed,
+                                                   autoresume_file,
+                                                   log_line,
+                                                   current_count=int(my_monitor.get_message_count()))
                     autoresume_lastwrite = int(my_monitor.get_message_count())
 
                 # Filter Enhancement
@@ -206,7 +212,8 @@ def main():
                     # Send the data
                     resend_status = my_output.send_data(log_line)
                     my_monitor.increase_message_count(len(log_line))
-                    aka_log.log.debug(f"<OUT> {log_line}")
+                    if uls_args.debugloglines:
+                        aka_log.log.debug(f"<OUT> {log_line}")
                     resend_counter = resend_counter + 1
 
                 if resend_counter == uls_config.main_resend_attempts and\

diff --git a/bin/config/global_config.py → bin/uls_config/global_config.py b/bin/config/global_config.py → bin/uls_config/global_config.py
@@ -1,7 +1,7 @@
 #!/usr/bin/env python3
 
 # Common global variables / constants
-__version__ = "1.7.1"
+__version__ = "1.7.2"
 __tool_name_long__ = "Akamai Unified Log Streamer"
 __tool_name_short__ = "ULS"
 
@@ -40,7 +40,7 @@
 linode_cli_feeds = ['AUDIT']                        # Available LINODE CLI feeds
 
     # INPUT Choices
-input_choices = ['EAA', 'ETP', 'MFA', 'GC', 'LINODE']           # Available input types
+input_choices = ['EAA', 'ETP', 'SIA', 'MFA', 'GC', 'LINODE']           # Available input types
 input_format_choices = ['JSON', 'TEXT']         # Available input format choices (need to be supported by cli)
 
     # OUTPUT Choices
@@ -53,6 +53,7 @@
     # LogLevels
 log_levels_available = ['DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL']
 log_level_default = 'WARNING'
+log_debugloglines_default = False
 
 # INPUT Configuration
 input_rerun_retries = 3                         # Number of rerun attempts before giving up

diff --git a/bin/config/transformation_config.py → bin/uls_config/transformation_config.py b/bin/config/transformation_config.py → bin/uls_config/transformation_config.py