The developer at Mystique Unicorn are looking monitor their app from the custom logs generated by the app running inside a Azure VM Can you show them how the can get started ?
We will use Azure Monitor Agent1 to collect the application logs from a specified location(for exampe: /var/log/miztiik*.json
) and push them to a Log Analytics Workspace3. To receive the logs in the correct format we need to define a custom table5,6 and a data collection rule. The rule will collect logs from our source(VM App), transform the logs using Kusto Query Language(KQL) to a structure defined in the custom table and store them in the sink(log analytics workspace).
This repository holds the bicep code to deploy the necessary infrastructure for this demo,
- Virtual Machine(RedHat) inside a Vnet with public access.
- Bootstrapped with custom libs using
customData
script.
- Bootstrapped with custom libs using
- Log Analytics Workspace ( Analytics tier)
- Custom Table
- Data Collection Endpoint - With public access
- Data Collection Rule
- Configured with the log location, KQL query and custom table
-
This demo, instructions, scripts and bicep template is designed to be run in
westeurope
. With few or no modifications you can try it out in other regions as well(Not covered here).- π Azure CLI Installed & Configured - Get help here
- π Bicep Installed & Configured - Get help here
- π VS Code & Bicep Extenstions - Get help here
-
-
Get the application code
https://github.com/miztiik/send-vm-logs-to-azure-monitor cd send-vm-logs-to-azure-monitor
-
-
Let check you have Azure Cli working with
# You should have azure cli preinstalled az account show
You should see an output like this,
{ "environmentName": "AzureCloud", "homeTenantId": "16b30820b6d3", "id": "1ac6fdbff37cd9e3", "isDefault": true, "managedByTenants": [], "name": "YOUR-SUBS-NAME", "state": "Enabled", "tenantId": "16b30820b6d3", "user": { "name": "miztiik@", "type": "user" } }
-
Let us walk through each of the stacks,
-
Stack: Main Bicep The params required for the modules are in
params.json
. Do modify them to suit your need.(Especially theadminPassword.secureString
for the VM. You are strongly encouraged to Just-In-Time access or use SSH key instead of password based authentication). The helper deployment scriptdeploy.sh
will deploy themain.bicep
file. This will inturn call the modules for Resource Group(RG), VNet & Virtual Machine, Log analytics workspace, Custom table, Data collection endpoint, Data collection rule(DCR) and finally attach the DCR to the VM.sh deploy.sh
This stack will create almost all of the resoureces. The RG is created as a deployment at the subscription level. The other resources are deployments at resource group level.
-
Virtual Machine - A Redhat vm is bootstrapped using
customData
to install python3, git. We will use a bash scriptgenerate_data.sh
. It will generate a json conformant log in this location/var/log/miztiik-yyyy-MM-dd.json
for example/var/log/miztiik-2023-04-03.json
. All new entries are appended to the file withnew-line
being the delimitter as expected by Azure Custom Logs.{"request_id": "08b6b258-c6d6-44ff-802e-69d52422095a", "event_type": "inventory_event", "store_id": 45, "cust_id": 4416, "category": "Camera", "sku": 17293, "price": 44.61, "qty": 15 , "discount": 73.2, "gift_wrap": 0, "variant": "MystiqueAutomatoon", "priority_shipping": 1, "TimeGenerated": "2023-04-03T10:54:23", "contact_me": "github.com/miztiik" } {"request_id": "60051c39-6817-4f52-bd8f-59cc9a38be0f", "event_type": "inventory_event", "store_id": 64, "cust_id": 4081, "category": "Camera", "sku": 17815, "price": 50.92, "qty": 16 , "discount": 29.2, "gift_wrap": 1, "variant": "MystiqueAutomatoon", "priority_shipping": 0, "TimeGenerated": "2023-04-03T10:54:24", "contact_me": "github.com/miztiik" } {"request_id": "f340f343-fc0f-40a1-b9f5-2abdd14b929d", "event_type": "inventory_event", "store_id": 35, "cust_id": 338, "category": "Camera", "sku": 16158, "price": 53.76, "qty": 13, "discount": 34.3, "gift_wrap": 1, "variant": "MystiqueAutomatoon", "priority_shipping": 0, "TimeGenerated": "2023-04-03T10:54:25", "contact_me": "github.com/miztiik" }
Formatting a single log line entry properly, will look like this,
{ "request_id": "f5570d0f-5389-4bce-98c3-7ed5380eefb2", "event_type": "inventory_event", "store_id": 6, "cust_id": 206, "category": "Camera", "sku": 100657, "price": 2.52, "qty": 18, "discount": 8.2, "gift_wrap": true, "variant": "red", "priority_shipping": true, "TimeGenerated": "2021-04-25T12:52:40.938937", "contact_me": "github.com/miztiik" }
After successfully deploying the stack, Check the
Resource Groups/Deployments
section for the resources. -
-
-
- Connect to the VM
Connect to the VM using Just-In-Time access. This should allow your IP/IP list to securelly connect to your VM using private IP address. Use the password you updated in the
params.json
ssh miztiik@publicIP
- Generate Log Data
For some reason the customData, bootstrap script fails, will fix it later. Until then manually copy the
generate_data.sh
from this repo to the server. _You may need elevated privileges to create log file in/var/log
.sudo su cd /var/log/ # Copy generate_data.sh to the server sh generate_data.sh &
The script by default is configured to write one log entry every
5
seconds and generate a total100000
log lines to/var/log/miztiik*.json
- Query in Azure Log Analytics Workspace
Login to Azure Log Anaytics portal. You should be able to find it under
/Miztiik_Enterprises_Custom_Logs_To_Monitor_xxx/providers/Microsoft.OperationalInsights/workspaces/lumberyard-payGTier-xxx",
. Navigate to logs, select the custom table(lumberyardTablexxx_CL
) &Run
-
Troubleshooting Azure Monitor Agent
-
In case if you are facing issues, log into the VM and check the logs
/var/log/azure/Microsoft.Azure.Monitor.AzureMonitorLinuxAgent/
for potential causes. -
From the portal, you can query to find if the agent is reporting to Azure Monitor
Heartbeat | where Category == "Azure Monitor Agent" and Computer == "<computer-name>" | take 10
Check if the AMA is running,
systemctl status azuremonitoragent
-
-
Here we have demonstrated how to configure a VM to send application logs to Azure monitor. You can extend the solution to setup alerts on these logs or secure the data collection endpoing to use a private link etc.,
If you want to destroy all the resources created by the stack, Execute the below command to delete the stack, or you can delete the stack from console as well
- Resources created during Deploying The Application
- Any other custom resources, you have created for this demo
# Delete from resource group
az group delete --name Miztiik_Enterprises_xxx --yes
# Follow any on-screen prompt
This is not an exhaustive list, please carry out other necessary steps as maybe applicable to your needs.
This repository aims to show how to Bicep to new developers, Solution Architects & Ops Engineers in Azure.
Thank you for your interest in contributing to our project. Whether it is a bug report, new feature, correction, or additional documentation or solutions, we greatly value feedback and contributions from our community. Start here
Buy me a coffee β.
- Azure Docs: Azure Monitor Agent Overview
- Azure Docs: Azure Monitor Agents -vs- Other Agatnes Overview
- Azure Docs: Collect text logs with Azure Monitor Agent
- Azure Docs: Collect text logs with Log Analytics Agent
- Azure Docs: Custom Table
- Azure Docs: Workspace Table Creation
- Azure Docs: Log Data Plan Basic-vs-Analytics Comparison
- Azure Docs: Custom Log Ingestion using python SDK
- Azure Docs: Custom Metrics to Azure Monitor
- Azure Docs: Check Logs in Azure Logs
- Azure Docs: Troubleshoot Azure Monitor Agent
- [Azure Docs: Sample Data Collection Rule][12]
Level: 200