Merge pull request #4 from delphix/develop

Develop to Main Merge - Release

.gitignore

@@ -0,0 +1,2 @@
+venv/
+.idea/

CONTRIBUTING.md

@@ -1,13 +1,27 @@
-## Contributing
- 1. Fork the project.
- 1. Make your bug fix or new feature.
- 1. Add tests for your code.
- 1. Send a pull request.
+#
+# Copyright (c) 2021, 2022 by Delphix. All rights reserved.
+#
 
-Contributions must be signed as `User Name <user@email.com>`. Make sure to [set up Git with user name and email address](https://git-scm.com/book/en/v2/Getting-Started-First-Time-Git-Setup). Bug fixes should branch from the current stable branch. New features should be based on the release branch.
+This project is currently not accepting external contributions.
 
-## Contributor Agreement
-All contributors are required to sign the Delphix Contributor agreement prior to contributing code to an open source repository. This process is handled automatically by [cla-assistant](https://cla-assistant.io/). Simply open a pull request and a bot will automatically check to see if you have signed the latest agreement. If not, you will be prompted to do so as part of the pull request process.
+<div style="display:none">
+--- Hidden Block ----
+
+    ## Contributing
+     1. Fork the project.
+     2. Refer to [README.md](README.md) for the setup, pre-configurations and running the project.
+     3. Once you have run the script successfully using `make run` (Linux/MacOS) or `python src\main.py` (Windows)
+     7. Make your bug fix or new feature.
+     8. Add tests for your code.
+     9. Send a pull request.
+
+    Contributions must be signed as `User Name <user@email.com>`. Make sure to [set up Git with user name and email address](https://git-scm.com/book/en/v2/Getting-Started-First-Time-Git-Setup). Bug fixes should branch from the current stable branch. New features should be based on the release branch.
+
+    ## Contributor Agreement
+    All contributors are required to sign the Delphix Contributor agreement prior to contributing code to an open source repository. This process is handled automatically by [cla-assistant](https://cla-assistant.io/). Simply open a pull request and a bot will automatically check to see if you have signed the latest agreement. If not, you will be prompted to do so as part of the pull request process.
+
+------------------------
+</div>
 
 ## Code of Conduct
 This project operates under the Delphix Code of Conduct. By participating in this project you agree to abide by its terms.

Makefile

@@ -0,0 +1,95 @@
+#
+# Copyright (c) 2022 by Delphix. All rights reserved.
+#
+
+.PHONY: --check_python --check_env_exists run tests --create_virtual_env --install_dependencies env clean_build clean_env clean build
+
+ROOT_DIR := $(shell pwd)
+VENV := venv
+PYTHON := $(VENV)/bin/python
+PIP := $(VENV)/bin/pip
+DLPX_NR := $(ROOT_DIR)/dist/delphix-nr
+DEFAULT_PYTHON3_ALIAS := python3
+
+define show_version
+	@export LC_ALL=en_AU.UTF-8; $(DLPX_NR) --version
+endef
+
+define check_python_exists
+	@if ! command -v $(DEFAULT_PYTHON3_ALIAS) -V >/dev/null 2>&1; then \
+		echo "Python 3 is NOT present on the system, Please install it"; \
+		echo "If it is installed, then change the "DEFAULT_PYTHON3_ALIAS" in Makefile";\
+		exit 1; \
+	fi
+endef
+
+
+--check_python:
+	@# Help: Checks existence of python virtual environment
+	$(call check_python_exists)
+
+--check_env_exists: --check_python
+	@[ -f $(PYTHON) ] && echo $(PYTHON) exists || (echo $(PYTHON) does NOT exist, use \"make env\" to create a virtual env; exit 1)
+
+
+run: --check_env_exists
+	@# Help: Takes care of checking the prequisites like python, virtual env, dependencies and at last shows the Python Version
+	-@echo `$(PYTHON) -V`
+	export PYTHONPATH=$(ROOT_DIR); $(PYTHON) src/main.py
+
+tests:
+	@# Help: Runs the unit tests inside tests folder and create a report
+	$(PYTHON) -m pytest tests -s -v
+
+--create_virtual_env: --check_python
+	@# Help: Creates a virtual environment
+	-@echo 'Creating Virtual environment'
+	@$(DEFAULT_PYTHON3_ALIAS) -m venv venv || echo 'Python env already exists'
+
+--install_dependencies: requirements.txt
+	@# Help: Installs the dependencies from requirements.txt
+	-@echo 'Installing Dependencies...'
+	@$(PYTHON) -m pip install --upgrade pip
+	@$(PIP) install -r requirements.txt
+
+env: --create_virtual_env --install_dependencies
+	@# Help: Creates a virtual environment with python 3 if not already present
+
+clean_env:
+	@# Help: Clean the virtual env that was created
+	-@rm -rf venv
+
+clean_build:
+	@# Help: Cleans the build files
+	-@rm -rf build/
+	-@rm -rf dist/
+	-@rm -rf delphix-nr.spec
+
+clean: clean_build
+	@# Help: Cleans the pycache, coverage and build files
+	-@rm -rf __pycache__
+	-@rm -f .coverage
+	-@rm -rf .pytest_cache
+	-@rm -rf tests/.pytest_cache
+	-@rm -rf tests/plugin_operations/.pytest_cache
+	-@rm -rf tests/CodeCoverage
+	-@rm -rf tests/Report.html
+	-@rm -rf .dvp-gen-output
+
+build: --check_env_exists clean_build
+	@# Help: Makes the delphix-nr build
+	$(VENV)/bin/pyinstaller --onefile src/main.py -n delphix-nr
+	-@echo 'Build is present at $(ROOT_DIR)/dist/delphix-nr'
+	$(call show_version)
+
+
+# A hidden target
+.hidden:
+help:
+	@printf "%-20s %s\n" "Target" "Description"
+	@printf "%-20s %s\n" "------" "-----------"
+	@make -pqR : 2>/dev/null \
+        | awk -v RS= -F: '/^# File/,/^# Finished Make data base/ {if ($$1 !~ "^[#.]") {print $$1}}' \
+        | sort \
+        | egrep -v -e '^[^[:alnum:]]' -e '^$@$$' \
+        | xargs -I _ sh -c 'printf "%-20s " _; make _ -nB | (grep -i "^# Help:" || echo "") | tail -1 | sed "s/^# Help: //g"'

README.md

@@ -1,104 +1,93 @@
-# Delphix Data Control Tower MultiCloud integration with New Relic
+# Delphix Data Control Tower's Integration with New Relic
 
-This project will allow you to send data from [Delphix Data Control Tower Multicloud](https://docs.delphix.com/dctmc) to [New Relic](https://newrelic.com/) as events. DCT Multicloud is a tool that will allow you to connect to all your Delphix engines on premises or in the cloud (AWS, Azure, Google Cloud, OCI and IBM)
+This project will allow you to send data from [Delphix Data Control Tower (DCT)](https://delphix.document360.io/dct/docs) to [New Relic](https://newrelic.com/) through the Events API. This repository is one component of the Delphix Quickstart. You can learn more on the [Delphix Instant Observability page](https://newrelic.com/instant-observability/delphix).
 
 ![Screenshot](images/image2.png)
 
 
 ## Getting Started
 
-These instructions will provide the code you need to extract data from DCT Multicloud and send it to New Relic.
+
+These instructions will provide the information you need to extract data from DCT and send it to New Relic. The Python script can run from any location with access to both DCT and New Relic.
 
 
 ### Prerequisites
 
-It's assumed that you have a New Relic valid account and one or many [Delphix Engines registered in DCT Multicloud](https://docs.delphix.com/dctmc/connecting-a-delphix-engine).
-DCT Multicloud will extract data from the Delphix Engines and we will use [New Relic Telemetry SDK](https://docs.newrelic.com/docs/telemetry-data-platform/ingest-apis/telemetry-sdks-report-custom-telemetry-data/) to send that data to New Relic.
-For this project, we will use the [Python SDK](https://github.com/newrelic/newrelic-telemetry-sdk-python), however you can use any of the available SDKs in different languages.
+* New Relic Account: [Sign Up](https://newrelic.com/signup)
+* Delphix Data Control Tower (DCT) with one or more engines: [Data Control Tower Docs](https://delphix.document360.io/dct/docs)
+* Python 3.7+: [Python Install](https://www.python.org/downloads)
+* This [GitHub repository](https://github.com/delphix/dct-newrelic-integration)
 
+<hr>
+<h4> Supported Python Versions and OS </h4>
+<hr>
 
-### Installing
+- MacOS - Python3.7 and Python3.8
+- Linux - Python3.7+
+- Windows - Python3.7+
 
-To push the data from Delphix DCT Multicloud to New Relic, we use the script ```dlpx_dct_to_nr.py```.
+<hr>
+
+### Installing
 To use this script we have to do some steps first:
 
-* Generate the [keys to connect to DCT Multicloud](https://docs.delphix.com/dctmc/authentication)
+* Generate the [key to connect to DCT](https://docs.delphix.com/dctmc/authentication)
 * Generate the [New Relic access key](https://docs.newrelic.com/docs/apis/intro-apis/new-relic-api-keys/#ingest-license-key)
+* Note the URL of the DCT instance VM
 
-Once we have these keys we need to replace them in the script:
 
-* In req_headers we replace the DCT Multicloud key
-* In NEW_RELIC_INSERT_KEY we replace the New Relic access key
+### Setup
+The ```src/main.py``` script contains the logic to perform the data upload. However, you must do some configuration first. 
 
-This is the script:
+* We need to supply the above gathered information using 3 environment variables
+  * DCT_HOST_URL
+  * DCT_API_KEY
+  * NEW_RELIC_INSERT_KEY
+* Clone this repository
+* Go inside the project directory - `cd dct-newrelic-integration`
 
-```
-import os
-import requests
-import json
-import sys
-import time
-from newrelic_telemetry_sdk import Event, EventClient
-from requests.packages.urllib3.exceptions import InsecureRequestWarning
-
-requests.packages.urllib3.disable_warnings(InsecureRequestWarning)
-
-DLPX_TYPES= ["engines","sources","dsources","vdbs","environments"]
-#
-# Request Headers ...
-#
-req_headers = {
-	'Authorization': 'apk 2.bnQDDx46Z4CDlIShLw2ZHElWLyKtsmZaBjbQPjui8LcQ3nELbkdEbQJSki6vmwLf'
-}
-
-#
-# Python session, also handles the cookies ...
-#
-session = requests.session()
-
-#
-# Login ...
-#
-os.environ['NEW_RELIC_INSERT_KEY'] = "87a453b2efe4nd4df78b167e7ac457e076c7NRAL"
-print ('')
-for i in DLPX_TYPES:
-
-	response = requests.get('https://localhost:443/v1/'+i, headers=req_headers, verify=False)
-	responsej = json.loads(response.text)
-	print("")
-	print("")
-	print("**********************************************************************************************************************************")
-	event_client = EventClient(os.environ["NEW_RELIC_INSERT_KEY"])
-	NEWRELIC_TYPE="Delphix " + str(i)
-	print (NEWRELIC_TYPE)
-	for line in responsej['items']:
-		event = Event(
-			NEWRELIC_TYPE, line
-		)
-		print (event)
-		response = event_client.send(event)
-		response.raise_for_status()
-		print("Event sent successfully!")
-		print("")
-
-print ('')
-sys.exit(0)
-```
+For Mac and Linux:
+* Run command `make env` (This will create the virtual environment)
+* Run command `make run` (This will run the script and push the data)
+
+For Windows:
+* Check that python 3 is installed
+* Create a virtual environment running `python -m venv venv`
+* Activate the virtual environment by running `venv\Scripts\activate`
+* Install the dependencies by running `pip install -r requirements.txt`
+* Set Python path by - `set PYTHONPATH=.`
+* Run the script using `python src\main.py`
 
-On execution, this script will extract data from all the registered Delphix Engines for the following metrics:
+Note: You may modify the Python script directly, but it is best practice to specify sensitive data through environment variables.
+
+<hr>
+
+In production, it is common to use a scheduler, such as a systemd, nohup, or wininit.exe, to ensure the script continually runs. For example, the following nohup command will run the script every N seconds based on the Interval provided in the `dct_nr_config.ini` file:
+```nohup make run &```
+
+
+On each execution, this script will extract the following metrics from all registered Delphix engines:
 
 * Engines - Data extraction date, CPU Count, Storage, Memory, Engine Type, Version, etc.
 * Environments - Data extraction date, Status, Engine ID, Name, etc.
 * Sources - Data extraction date, Database Type, Database Version, Environment ID, JDBC Connection String, Database Name and Size, etc
 * dSources - Data extraction date, dSource Creation Date, dSource Type, Version, Name, Status, Size, etc.
 * VDBs - Data extraction date, Database Type and Version, Creation Date, Group Name, Name, Parent ID, Size, Status, etc.
 
-This script can be added to cron or any scheduler to run in any time interval. Once the data is available in New Relic, it can be used to be queried or to create dashboards.
 
+More over we have `dct_nr_config.ini` file which can be used to configure 
+- Logging Level 
+- Interval (in seconds): This script keeps running and sleeps for `INTERVAL` seconds after sending the data once.
+- Components we need from DCT APIs
+
+Once the data is available in New Relic, it can be used to be queried or to create dashboards.
+
+
+## Data, Dashboards, and Alerts
 
-## Data and Dashboards
+Once the data is available within New Relic, you are free to leverage it as you wish through customized Dashboards and Alerts. Samples can be found as a part of the [Delphix Quickstart](https://newrelic.com/instant-observability/delphix). 
 
-This is how the raw data looks like in the [Query your data](https://docs.newrelic.com/docs/query-your-data/explore-query-data/get-started/introduction-querying-new-relic-data/#browse-data) window for the VDB metric:
+As an example, this is how the raw data looks like in the [Query your data](https://docs.newrelic.com/docs/query-your-data/explore-query-data/get-started/introduction-querying-new-relic-data/#browse-data) window for the VDB metric:
 
 ![Screenshot](images/image1.png)
 
@@ -115,17 +104,17 @@ SELECT MAX(data_storage_capacity)-MIN(data_storage_used) FROM `Delphix engines`
 
 ## Contributing
 
-Please read [CONTRIBUTING.md](https://github.com/delphix/.github/blob/master/CONTRIBUTING.md) for details on our code of conduct, and the process for submitting pull requests to us.
+This project is currently not accepting external contributions.
 
 
 ## Versioning
 
 We use [SemVer](http://semver.org/) for versioning. For the versions available, see the [tags on this repository](https://github.com/your/project/tags).
 
 
-## Reporting Issues
+## Reporting Issues and Questions
 
-Issues should be reported in the GitHub repo's issue tab. Include a link to it.
+Please report all issues and questions in the [GitHub issue tab](https://github.com/delphix/dct-newrelic-integration/issues) or [Delphix Community page](https://community.delphix.com/home). Please include a complete problem description, error logs if appropriate, and directions on how to reproduce.
 
 
 ## Statement of Support
@@ -146,4 +135,3 @@ License
  See the License for the specific language governing permissions and
  limitations under the License.
  ```
-Copyright (c) 2014, 2016 by Delphix. All rights reserved.

SUPPORT.md

@@ -1,2 +1,12 @@
 ## Support
 This software is provided as-is, without warranty of any kind or commercial support through Delphix. See the associated license for additional details. Questions, issues, feature requests, and contributions should be directed to the community as outlined in the [Delphix Community Guidelines](https://delphix.github.io/community-guidelines.html).
+
+<hr>
+<h4> Supported Python Versions and OS </h4>
+<hr>
+
+- MacOS - Python3.7 and Python3.8
+- Linux - Python3.7+
+- Windows - Python3.7+
+
+<hr>

dct_nr_config.ini

@@ -0,0 +1,11 @@
+[COMPONENTS]
+; This list holds the components for which
+; we send the data to new relic
+; refer to DCT API's for getting these components
+monitor = management/engines, sources, dsources, vdbs, environments
+
+[INTERVAL]
+seconds = 30
+
+[LOGGING]
+LEVEL = INFO