Large refactor

- Removes Adapters
- Adds context to records
- Fixes setups and CI
- Updates Dockerfile and compose
- Updates README and rules

.dockerignore

@@ -5,7 +5,6 @@
 !src/**
 src/*.egg-info
 # Rules
-!rules/README.md
 !rules/rules.yml
 !rules/rules-schema.yml
 # Plugins

.github/workflows/github-ci.yml

@@ -31,7 +31,7 @@ jobs:
       - name: Install packages
         run: |
           python -m pip install .[dev]
-          python -m pip install plugins/.[all]
+          python -m pip install ./plugins[dev]
           python -m pip install -r server/requirements.txt
       - name: Invoke Tests
         run: pytest -c pytest-all.ini

Dockerfile

@@ -6,9 +6,7 @@ RUN chown -R root:dnsmule . && chmod -R 770 .
 USER dnsmule
 RUN python -m venv venv \
     && venv/bin/python -m pip install --no-cache-dir --upgrade pip \
-    && venv/bin/python -m pip install --no-cache-dir .[redis] ./plugins[all] \
+    && venv/bin/python -m pip install --no-cache-dir .[full] ./plugins[full] \
     && venv/bin/python -m pip install --no-cache-dir -r server/requirements.txt
-RUN cp rules/rules.yml rules/rules-old.yml  \
-    && sed -e 's/host: 127.0.0.1/host: redis/g' rules/rules-old.yml > rules/rules.yml
 EXPOSE 8080
 ENTRYPOINT ["venv/bin/python", "-m", "uvicorn", "--host", "0.0.0.0", "--port", "8080", "server.server:app"]

README.md

@@ -7,6 +7,20 @@ Package for rule based dependency scanning and service fingerprinting via DNS.
 This package provides utilities for writing and evaluating verbose and easy to read rule definitions in _YAML_-format.
 There are two builtin rule formats with more available as plugins.
 
+## Installation
+
+```shell
+pip install dnsmule[full] dnsmule_plugins[full]
+```
+
+This will install everything available for DNSMule. You can also choose to install components as necessary.
+
+For installing from the repo you can use:
+
+```shell
+pip install -e .
+```
+
 ## Overview
 
 The DNSMule tool is composed in the following way:
@@ -21,60 +35,222 @@ The DNSMule tool is composed in the following way:
 
 Check out the examples in the [examples](examples) folder. They should get you up and running quickly.
 
-#### Backends
+## YAML Configuration
+
+### Summary
+
+The tool configuration is done through one or multiple rule configuration files. The file structure is defined in the
+[schema file](rules/rules-schema.yml). In addition to some builtin rule types, it is possible to create new types by
+registering handlers or rules programmatically.
+
+Rules support registration per DNS record type and priority for controlling invocation order.
+
+```yaml
+version: '0.0.1'
+rules:
+  - name: o365
+    priority: 10
+    type: dns.regex
+    record: txt
+    config:
+      pattern: '^MS=ms'
+      identification: MICROSOFT::O365
+  - name: ses
+    type: dns.regex
+    record: txt
+    config:
+      pattern: '^amazonses:'
+      identification: AMAZON::SES
+```
+
+#### Example
+
+```python
+from dnsmule.definitions import Record, Result
+from dnsmule.rules import Rules, Rule
 
-Any backend can be used and registered in YAML, the only requirement is extending the `dnsmule.backends.Backend` class.
-A backend is responsible for creating `dnsmule.definition.Record` instances for processing with `dnsmule.rules.Rules`.
+rules: Rules
+
+...
+
+
+@rules.add.A[10]
+def my_scan(record: Record) -> Result:
+    from dnsmule.logger import get_logger
+    get_logger().info('Address %s', record)
+    return record.tag('MY::SCAN')
+
+
+@rules.register('my.rule')
+def create_my_rule(**arguments) -> Rule:
+    ...
+```
+
+Here the `add` is used to directly register a new rule into the ruleset with a given priority. The `register` call
+creates a new handler for rules of type `my.rule`. Any future `my.rule` creations from YAML or code would be routed to
+this factory function.
+
+See the examples folder for more examples and how to use rules in code.
+
+### Plugins and Backends
 
 #### Plugins
 
-Plugins provide DNSMule with additional functionality and/or rule(types).
-Plugins can be registered in YAML.
-It is required that all plugins extend the `dnsmule.plugins.Plugin` class.
-
-#### Rules
-
-This class is responsible for processing every`Record` and producing a `dnsmule.definition.Result` for them.
-The actual processing happens in individual `dnsmule.rules.Rule` instances that are ordered and orchestrated by this
-class.
-
-#### Rule
-
-These are user defined rules that process individual `Record` instances and append information to `record.data`
-or `record.tags`. The `record.data` is a storage for result metadata and additional information. The `record.tags` set
-is a collection of identifications for an input `Record`.
-
-#### Current State
-
-States (Complete, Testing, Developing, Refining) for every feature planned at this moment.
-
-- (Complete) Backends
-    - (Complete) DNSPythonBackend
-    - (Complete) YAML definition
-- (Complete) Rules
-    - (Complete) Builtin types
-    - (Complete) Custom rules
-    - (Complete) Custom rule factories
-    - (Complete) YAML definition
-- (Testing) Plugins
-    - (Complete) Registration in program
-    - (Complete) Registration in YAML
-    - (Complete) YAML definition
-    - (Developing) Increasing test coverage
-- (Refining) Result Storage
-    - (Refining) Registration in program
-    - (Refining) Registration in YAML
-    - (Refining) YAML definition
-- (Developing) DNSMule
-    - (Complete) Rules from YAML
-    - (Complete) Backend from YAML
-    - (Testing) Plugins from YAML
-    - (Developing) Processing and gathering results
-    - (Refining) Result storage
-    - (Refining) Result storage from YAML
-    - (Refining) Combined YAML definition
-
-#### Example server
+It is possible to register plugins using the YAML file:
+
+```yaml
+plugins:
+  - name: dnsmule_plugins.CertCheckPlugin
+    config:
+      callback: false
+```
+
+These are required to extend the `dnsmule.plugins.Plugin` class.
+Plugins are evaluated and initialized before rules.
+Any rules requiring a plugin should list their plugin in this block.
+Plugins are only initialized once and if a plugin already exists in the receiving DNSMule instance
+it will be ignored.
+
+#### Backends
+
+It is possible to define a single backend in a YAML file:
+
+```yaml
+backend:
+  name: 'dnspython.DNSPythonBackend'
+  config:
+    timeout: 5.5
+    resolver: 8.8.8.8
+```
+
+The backend must extend the `dnsmule.backends.Backend` class.
+This declaration is ignored if this is not used in `DNSMule.load` or `DNSMule(file=file)`.
+
+## Editor Support
+
+#### Type Hints and JSON Schema (IntelliJ IDEA, PyCharm, etc.)
+
+It is possible to register the schema file as a custom JSON schema in IntelliJ editors.
+This will give access to typehints and schema validation inside rule files and is especially nice for dynamic rule
+definitions as you get full editor features for python inside the snippets.
+
+- settings...
+- Languages & Frameworks > Schemas and DTDs > JSON Schema Mappings
+- Add a new mapping with the schema file and specified file or pattern
+
+This is configured in the schema using the custom intellij language injection tag:
+
+```yml
+x-intellij-language-injection:
+  language: Python
+  prefix: |+0
+    code hints go here
+    check the schema for more info
+  suffix: ''
+```
+
+Currently, this supports `dns.regex` pattern regex language injection and `dns.dynamic` rule code language injection.
+Type hints and quick documentation are available.
+
+## Builtin Rules
+
+#### Regex rules
+
+Regex rules can be defined with either one `pattern` or multiple `patterns`.
+An example is in the following snippet:
+
+```yml
+rules:
+  - name: test
+    type: dns.regex
+    record: txt
+    config:
+      pattern: '^.*\.hello_world\.'
+      identification: HELLO::WORLD
+      flags:
+        - UNICODE
+        - DOTALL
+  - name: generic_verification
+    type: dns.regex
+    record: txt
+    priority: 10
+    description: Generic Site Regex Collection
+    config:
+      patterns:
+        - '^(.+)(?:-(?:site|domain))?-verification='
+        - '^(.+)(?:site|domain)verification'
+        - '^(.+)_verify_'
+        - '^(\w+)-code:'
+      group: 1
+```
+
+The full definition and additional info is available from the schema file, examples, and code.
+
+#### Dynamic Rules
+
+Dynamic rules are defined as code snippets with one or two methods
+
+An init method that is invoked once after creation
+
+```python
+def init() -> None:
+    add_rule(...)
+```
+
+A process function that is invoked once for each record
+
+```python
+def process(record: Record) -> Result:
+    add_rule(...)
+    return record.result()
+```
+
+Both of these functions have access to the following rule creation method:
+
+```python
+def add_rule(
+        record_type: Union[str, int, RRType],
+        rule_type: str,
+        name: str,
+        *,
+        priority: int = 0,
+        **options,
+) -> None:
+    """
+    :param record_type: Valid DNS record type as text, int, or type
+    :param rule_type:   Valid rule type factory e.g. dns.regex
+    :param name:        Name of the created rule
+    :param priority:    Priority for the created rule, default 0
+    :param options:     Any additional options for the rule factory
+    """
+```
+
+The only globals passed to these methods are:
+
+- \_\_builtins\_\_
+- RRType, Record, Result, Domain, Tag, Config
+    - The Config contains the `config` property passed to the rule from YAML
+- add_rule
+- Any additional globals created by the code itself
+
+When the code is exec'd the result is inspected for:
+
+- init function without parameters
+- process function with a single parameter
+
+Some notes:
+
+- The init function is invoked exactly once.
+- The process function is invoked exactly once for every single Record.
+- Any rules created from the init method will be invoked for every suitable record.
+- Any rules created from the process method will be invoked for suitable records found after creation.
+- Creating DynamicRules from init or process is considered undefined behaviour and care should be taken
+    - The user should call init manually and include fail-safes for only calling it once
+    - The add_rule callback might not be available so you need to pass it manually to the rule
+
+## Other
+
+### Example server
 
 The repo has a `Dockerfile` for easily running the tool using an example server in Docker:
 
@@ -83,11 +259,12 @@ $ ./build-image
 $ ./run-server
 ```
 
-#### Notice
+### Notice
 
 This package is under active development.
 
-#### Additional
+### Additional
 
 - RnD Scripts under [scripts](scripts)
-- Example server under [server](server)
+- Example server under [server](server)
+- Examples for coding under [examples](examples)

docker-compose.yml

@@ -15,6 +15,8 @@ services:
       dockerfile: Dockerfile
     image: dnsmule:latest
     tty: true
+    volumes:
+      - './rules/rules.yml:/opt/dnsmule/rules/rules.yml:ro'
     ports:
       - '127.0.0.1:8080:8080'
     depends_on:

plugins/setup.py

@@ -43,10 +43,12 @@
         'dnsmule',
     ],
     extras_require={
-        'certs': [
+        'dev': [
+            'pytest',
+            'pytest-cov',
             'cryptography',
         ],
-        'all': [
+        'full': [
             'cryptography',
         ]
     },

plugins/src/dnsmule_plugins/certcheck/__init__.py

@@ -1,7 +1,6 @@
 from typing import Optional, Callable, Collection
 
 from dnsmule import DNSMule, Domain, Plugin
-from dnsmule.adapter import patch_storage, Adapter
 from .adapter import load_result, save_result
 from .rule import CertChecker
 
@@ -15,7 +14,6 @@ def get_callback(self, mule: DNSMule) -> Optional[Callable[[Collection[Domain]],
             return mule.scan
 
     def register(self, mule: DNSMule):
-        patch_storage(mule.storage, Adapter(loader=load_result, saver=save_result))
         mule.rules.register(CertChecker.id)(CertChecker.creator(self.get_callback(mule)))