Release 1.3.1 merge into main (#154)

* Update retention function to fix retention policy bug (#143)

* Updated troubleshooting guide to account for index management (#134)

* Update upgrading.md to account for 1.3.1 (#151)

* Update upgrading.md

* Update upgrading.md

---------

Co-authored-by: Andrew Arz <149685528+aarz-snl@users.noreply.github.com>

Chapter 3 Files/deploy.sh

@@ -534,35 +534,39 @@ function pipelineupdate() {
 }
 
 function data_retention() {
-  #show ext4 disk
+  # Show ext4 disk
   DF_OUTPUT="$(df -h -l -t ext4 --output=source,size /var/lib/docker)"
 
-  #pull dev name
+  # Pull device name
   DISK_DEV="$(echo "$DF_OUTPUT" | grep -Po '[0-9]+G')"
 
-  #pull dev size
-  DISK_SIZE_ROUND="${DISK_DEV/G/}"
+  # Pull device size
+  DISK_SIZE="${DISK_DEV/G/}"
 
-  #lets do math to get 75% (%80 is low watermark for ES but as curator uses this we want to delete data *before* the disk gets full)
-  DISK_80=$((DISK_SIZE_ROUND * 80 / 100))
+  # Check if DISK_SIZE is empty or not a number
+  if ! [[ "$DISK_SIZE" =~ ^[0-9]+$ ]]; then
+    echo -e "\e[31m[!]\e[0m DISK_SIZE not an integer or is empty - exiting."
+    exit 1
+  fi
 
-  echo -e "\e[32m[X]\e[0m We think your main disk is $DISK_DEV"
+  echo -e "\e[32m[X]\e[0m We think your main disk is $DISK_DEV and its size is $DISK_SIZE gigabytes"
 
-  if [ "$DISK_80" -lt 30 ]; then
-    echo -e "\e[31m[!]\e[0m LME Requires 128GB of space usable for log retention - exiting"
-    exit 1
-  elif [ "$DISK_80" -ge 90 ] && [ "$DISK_80" -le 179 ]; then
+  if [ "$DISK_SIZE" -lt 128 ]; then
+    echo -e "\e[33m[!]\e[0m Warning: Disk size less than 128GB, recommend a larger disk for production environments. Install continuing..."
+    sleep 3
     RETENTION="30"
-  elif [ "$DISK_80" -ge 180 ] && [ "$DISK_80" -le 359 ]; then
+  elif [ "$DISK_SIZE" -ge 128 ] && [ "$DISK_SIZE" -le 179 ]; then
+    RETENTION="45"
+  elif [ "$DISK_SIZE" -ge 180 ] && [ "$DISK_SIZE" -le 359 ]; then
     RETENTION="90"
-  elif [ "$DISK_80" -ge 360 ] && [ "$DISK_80" -le 539 ]; then
+  elif [ "$DISK_SIZE" -ge 360 ] && [ "$DISK_SIZE" -le 539 ]; then
     RETENTION="180"
-  elif [ "$DISK_80" -ge 540 ] && [ "$DISK_80" -le 719 ]; then
+  elif [ "$DISK_SIZE" -ge 540 ] && [ "$DISK_SIZE" -le 719 ]; then
     RETENTION="270"
-  elif [ "$DISK_80" -ge 720 ]; then
+  elif [ "$DISK_SIZE" -ge 720 ]; then
     RETENTION="365"
   else
-    echo -e "\e[31m[!]\e[0m Unable to determine retention policy - exiting"
+    echo -e "\e[31m[!]\e[0m Unable to determine disk size - exiting."
     exit 1
   fi
 

docs/markdown/maintenance/upgrading.md

@@ -159,7 +159,7 @@ The deploy.sh script should have now created new files on the Linux server at lo
 
 Then reboot your Client computers & Windows Event Collector. On Windows Event Collector open services.msc as an administrator and make sure the winlogbeat service is set to start automatically, and is running.
 
+## 7. Upgrade from 1.3.0 to 1.3.1
 
-
-
+This is a hotfix to the install script and some additional troubleshooting steps added to documentation on space management. Unless you're encountering problems with your current installation, or if your logs are running out of space, there's no need to upgrade to version 1.3.1, as it doesn't offer any additional functionality changes.
 

docs/markdown/reference/troubleshooting.md

@@ -327,3 +327,62 @@ sudo curl -X POST "https://127.0.0.1:9200/_security/user/elastic/_password" -H "
 Replace 'currentpassword' with your current password and 'newpassword' with the password you would like to change it to. 
 
 Utilize environment variables in place of currentpassword and newpassword to avoid saving your password to console history. If not we recommend you clear your history after changing the password with ```history -c```
+
+## Index Management
+
+If you are having issues with your hard disk filling up too fast you can use these steps to delete logs earlier than your current settings.
+
+1. **Log in to Elastic**
+   - Access the Elastic platform and log in with your credentials.
+
+2. **Navigate to Management Section**
+   - In the main menu, scroll down to "Management."
+
+3. **Access Stack Management**
+   - Within the Management section, select "Stack Management."
+
+4. **Select Index Lifecycle Policies**
+   - In Stack Management, find and choose "Index Lifecycle Policies."
+
+5. **Choose the Relevant ILM Policy**
+   - From the list, select `lme_ilm_policy` for editing.
+
+6. **Adjust the Hot Phase Settings**
+   - Navigate to the 'Hot Phase' section.
+   - Expand 'Advanced settings'.
+   - Uncheck "Use recommended defaults."
+   - Change the "Maximum age" setting to match your desired delete phase duration.
+
+     > **Note:** Aligning the maximum age in the hot phase with the delete phase ensures consistency in data retention.
+
+7. **Adjust the Delete Phase Settings**
+   - Scroll to the 'Delete Phase' section.
+   - Find and adjust the "Move data into phase when:" setting.
+   - Ensure the delete phase duration matches the maximum age set in the hot phase.
+
+     > **Note:** This setting determines the deletion timing of your logs. Ensure to back up necessary data before changes.
+
+8. **Save Changes**
+   - Save the adjustments you've made.
+
+9. **Verify the Changes**
+   - Review and ensure that the changes are functioning as intended. Indices may not delete immediately - allow time for job to run.
+
+10. **Document the Changes**
+    - Record the modifications for future reference.
+
+You can also manually delete an index from the GUI under Management > Index Managment or by using the following command: 
+
+```
+curl -X DELETE "https://127.0.0.1:9200/your_index_name" -H "Content-Type: application/json" --cacert /opt/lme/Chapter\ 3\ Files/certs/root-ca.crt -u elastic:yourpassword
+```
+> **Note:**    Ensure this is not your current winlogbeat index in use. You should only delete indices that have already rolled over. i.e. if you have index winlogbeat-00001 and winlogbeat-00002 do NOT delete winlogbeat-00002.
+
+If you only have one index you can manually force a rollover with the following command: 
+
+```
+curl -X POST "https://127.0.0.1:9200/winlogbeat-alias/_rollover" -H "Content-Type: application/json" --cacert /opt/lme/Chapter\ 3\ Files/certs/root-ca.crt -u elastic:yourpassword
+```
+
+This will rollover winlogbeat-00001 and create winlogbeat-00002. You can now manually delete 00001. 
+