Add support for Airos in OpenWISP2 #91

edoput · 2017-07-15T10:36:15Z

This PR is now closed as the work has been included in an external package, netjsonconfig-airos and will be available from the next release (v0.6.3)

this is the airos v8.3 backend as of today

tests are ok for both python2 and python3

I will go on with squashing commits where possible to keep size down

Another important things to do is to implement the backward_conversion for the radio converter as of now the radio configuration values are mostly hardware dependents. This may be superseded by an approach where the user writes a very verbose netjson with lots of additional values for the radio object.

nemesifier

At the moment I limited myself to style issues which are usually the first thing noticed when reviewing.

Please see my inline comments, fix the flake8/isort issues and update the PR.
In the meanwhile I'll test with an ubiquiti PowerBeam AC feed.

Fed

nemesifier · 2017-07-17T12:33:28Z

docs/source/backends/airos.rst

+        ]
+    }
+
+Leaving the `NetJSON Encryption object <http://netjson.org/rfc.html#rfc.section.5.4.2.1>` empty defaults to no encryption at all


could you please add a dot . at the end of the line?

nemesifier · 2017-07-17T12:34:26Z

docs/source/backends/intermediate.rst

+   }
+
+As a son may be a carrier of a value so we store it in a dictionary instead of adding a *leaf*
+with another level of recursion


missing dot at end of the line

nemesifier · 2017-07-17T12:34:44Z

docs/source/backends/intermediate.rst

+As a son may be a carrier of a value so we store it in a dictionary instead of adding a *leaf*
+with another level of recursion
+
+As an example here we present the tree `('spam', [ { 'eggs': 2 }, { 'snakes' : { 'loved' : 'python' }}])`


maybe a colon : at the end of the line?

nemesifier · 2017-07-17T12:34:52Z

docs/source/backends/intermediate.rst

+
+The intermediate representation is the output of the a :ref:`converter`,
+it is backend specific and is built as a tree structure made from python
+builtins values


missing dot at end of the line

nemesifier · 2017-07-17T12:35:11Z

docs/source/backends/intermediate.rst

+A tree is a *acyclic, directional graph* with an element called *root*.
+
+The root of our tree is stored in the first element of a tuple, along with
+the root's direct sons as a list


missing a colon : at the end of the line?

nemesifier · 2017-07-17T13:09:16Z

tests/airos/dummy.py

@@ -0,0 +1,268 @@
+from netjsonconfig import AirOS
+


I don't like the name of the file, can we find a more professional name?

nemesifier · 2017-07-17T13:10:41Z

tests/airos/test_bridge.py

+        })
+
+        o.to_intermediate()
+


extra blank lines everywhere! :-D :-P

nemesifier · 2017-07-17T13:10:53Z

tests/airos/test_bridge.py

+
+        expected = [
+                    {
+                        '1.comment': '',


check spaces

nemesifier · 2017-07-17T13:11:17Z

tests/airos/test_discovery.py

+
+
+class TestDiscoveryConverter(ConverterTest):
+


at least this one you may remove it :-P

nemesifier · 2017-07-17T13:11:31Z

tests/airos/test_wpasupplicant.py

+                    },
+                    {
+                        'device.1.profile': 'AUTO',
+                        'device.1.status': 'disabled',


check spaces

nemesifier · 2017-07-17T14:04:26Z

netjsonconfig/backends/airos/schema.py

-        wpasupplicant_schema,
-        ])
+        override_schema
+        )


why not on a single line? :-)

because I'm mad and I REALLY LOVE my enter key

nemesifier · 2017-07-18T12:58:42Z

netjsonconfig/backends/airos/schema.py

+AirOS specific JSON-Schema definition
+"""
+from ...schema import schema as default_schema
+from ...schema import DEFAULT_FILE_MODE  # noqa - backward compatibility


you can remove this line too, it's not needed here.

coveralls · 2017-07-20T11:16:32Z

Coverage decreased (-3.2%) to 96.737% when pulling 7c8e27b on airos into a37cdea on master.

nemesifier

Great progress! 👍

As a side note among other things, try to get higher test coverage

nemesifier

More inline feedback

nemesifier · 2017-07-21T11:28:06Z

netjsonconfig/backends/airos/converters.py

+                    'tx': {
+                        'status': 'enabled',
+                    },
+                }


what about a more compact version:

base['flowcontrol'] = { 'rx': {'status': 'enabled'}, 'tx': {'status': 'enabled'}, }

nemesifier · 2017-07-21T11:30:10Z

netjsonconfig/backends/airos/converters.py

+                    # handle explicit address policy
+                    if addr['proto'] == 'dhcp':
+                        temp['autoip'] = {}
+                        temp['autoip']['status'] = 'enabled'


wouldn't an explicit declaration be more compact and readable? Eg:

temp['autoip'] = {'status': 'enabled'}

nemesifier · 2017-07-21T11:33:20Z

netjsonconfig/backends/airos/converters.py

+        result = [
+            {
+                'community': 'public',
+                'contact': '',


you can take this from general.maintainer, see http://netjson.org/rfc.html#general1

nemesifier · 2017-07-21T11:34:11Z

netjsonconfig/backends/airos/converters.py

+            {
+                'community': 'public',
+                'contact': '',
+                'location': '',


we may add general.location to NetJSON in the near future, take this value from there.

nemesifier · 2017-07-21T11:35:21Z

netjsonconfig/backends/airos/renderers.py

+    def cleanup(self, output):
+        stripped = [
+                a.strip() for a in output.splitlines() if a.strip()
+                ]


indentation looks wrong here

nemesifier · 2017-07-21T11:50:25Z

docs/source/backends/intermediate.rst

+=============
+
+.. include:: ../_github.rst
+


remove blank line

nemesifier · 2017-07-21T11:50:53Z

docs/source/backends/airos.rst

+
+    {
+        "type": "DeviceConfiguration",
+        ...


remove these dots because they are not being recognized as JSON

nemesifier · 2017-07-21T11:52:28Z

docs/source/backends/airos.rst

+
+.. include:: ../_github.rst
+
+The ``AirOS`` backend allows to generate AirOS v8.3 compatible configurations.


Should be AirOs here and on the rest of the document

hey @edoput I think you forgot this one

nemesifier · 2017-07-21T11:53:25Z

docs/source/backends/intermediate.rst

+                'comment' : 'traffic vlan'
+            }
+        ]
+   }


keep only the python example, having both is redundant and doesn't help the user much

nemesifier · 2017-07-21T11:53:42Z

docs/source/backends/intermediate.rst

+                        '2.devname' : 'eth0',
+                        '2.id' : '2'
+                        '2.status' : 'enabled',
+                        '2.comment' : 'traffic'


it looks like 8 spaces indentation to me, can you check?

nemesifier · 2017-07-21T13:26:00Z

netjsonconfig/backends/airos/converters.py

+            'external': {
+                'reset': 'enabled',
+            },
+            'timezone': 'GMT',


we'll need to implement this soon, it shouldn't have priority right now but we will have to do it in the 3rd phase

nemesifier

Here's some more detailed feedback on the configuration front.

I'm testing a configuration I downloaded from a production device and I'm comparing it with a config
I am generating with netjsonconfig.

The NetJSON I am using is:

{
    "general":{
        "hostname": "PBPomeziaDiegoApriliana",
        "timezone": "Europe/Rome"
    },
    "gui": {
        "language": "en_US"
    },
    "netmode": "bridge",
    "interfaces": [
        {
            "name": "ath0",
            "type": "wireless",
            "wireless": {
                "radio": "ath0",
                "mode": "access_point",
                "ssid": "ninux-diego-pomezia",
                "wds": true,
                "bssid": "04:18:D6:F8:01:F7"
            }
        },
        {
            "type": "ethernet",
            "name": "eth0",
            "mtu": 1500,
            "addresses": [
                {
                    "management": true,
                    "address": "10.8.2.71",
                    "gateway": "10.8.2.1",
                    "mask": 24,
                    "family": "ipv4",
                    "proto": "static"
                }
            ]
        },
        {
            "type": "ethernet",
            "name": "eth0.24",
            "mtu": 1500
        },
        {
            "name": "br0",
            "type": "bridge",
            "bridge_members": ["ath0", "eth0.24"]
        }
    ],
    "radios": [
        {
            "name": "ath0",
            "channel": 1,
            "channel_width": 20,
            "disabled": true,
            "protocol": "802.11n",
            "tx_power": -4
        }
    ],
    "ntp_servers": ["193.204.114.233", "193.204.114.234"],
    "dns_servers": ["193.204.5.4", "8.8.8.8"],
    "user": {
        "name": "root",
        "password": "gpO0LLzr9C8L.LKdsqGEf.",
        "salt": "WdnaHPry"
    }
}

Spanning Tree Protocol for bridges

Seems always enabled, eg:

bridge.1.stp.status=enabled

But may also be disabled:

bridge.1.stp.status=disabled

Use the stp attribute of bridge interfaces to set this.

If the attribute is omitted, it should mean disabled.

Can comment be omitted if empty/not present?

I see bridge.1.comment= in my generated config, but this type of attribute was not present
in my test configuration that I downloaded from a production device.

Therefore I think it may be omitted if no comment is present in the NetJSON. What do you think?

ebtables

My production conf doesn't have this section.

Can this section be omitted if not specified in NetJSON?

igmpproxy

My production conf doesn't have this section.

Can this section be omitted if not specified in NetJSON?

iptables

My production conf doesn't have this section.

Can this section be omitted if not specified in NetJSON?

netconf flowcontrol

I get this:

+netconf.2.flowcontrol.tx.status=enabled
+netconf.2.flowcontrol.rx.status=enabled

Even though I don't define any flowcontrol in the NetJSON.

I think flowcontrol can be omitted if not defined in the NetJSON, what do you think?

netconf autoneg

I get this:

+netconf.2.autoneg=enabled

Even though I don't define any autoneg in the NetJSON.

I think autoneg can be omitted if not explicitly defined, what do you think?

I think we can add a negotiation parameter to the schema of ethernet interfaces for this.

ntp

This works, can you use the same NTP schema that is being used by OpenWrt and Raspbian?

That way we can keep differences to the minimum and we may even be able to add this new section to the official NetJSON spec.

You can ignore the enable_server property.

pwdog

I have this situation:

-pwdog.status=disabled
pwdog.status=enabled

Is this feature enabled by default on AirOS?

resolv

I get this situation:

-resolv.nameserver.status=enabled
+resolv.status=enabled

Are we sure this is ok?

Default route

I have the following:

-route.1.devname=eth0
-route.1.gateway=10.8.2.1
-route.1.ip=0.0.0.0
-route.1.netmask=0
-route.1.status=enabled

This is the default route, you can infer this information
from the gateway attribute of an address of an interface,
probably only the management interface.

sshd

I have this situation:

-QYva4bK2zA4faxwtZbtOyQ7x4IYNt5Z+Vv381cM1g2NyGetS3qs0mPXt9JaQRi4aYre7MtSkOOJymwpJj4pPJz8t/H3pVLyi17FAXXWVh1j42C4Mi7IBgtYn0T4GMn0FGx3+dRv2rDrwDaA0guC9GKH8PzHsbrlH1ibewJ5VLFcUzFy6CY14muXTFNb/XyvK0Z8fCU2UR3VILVDotDJfJCOEhhOs0uKLNOeyoApbeS2N
-sshd.auth.key.1.comment=nemesis@freedom
-sshd.auth.key.1.status=enabled
-sshd.auth.key.1.type=ssh-rsa
+sshd.port=22
+sshd.status=enabled
+sshd.auth.passwd=enabled

The added lines should not be an issue, but SSH keys are removed and this can cause annoyances.

In the OpenWrt backend one can add an SSH key by specifying a custom file
with path /etc/dropbear/authorized_keys but I would have liked to introduce
a more specific and more portable way to specify SSH keys.

Since most evolved networking firmware support some kind of SSH configuration,
I think it's time to define a new NetJSON attribute for this and we could start
doing it in this backend.

A way of doing it in the NetJSON style would be:

{
    "sshd": {
        "port": 22,
        "enabled": True,
        "password_auth": True,
        "keys": [
            {
                "type": "ssh-rsa",
                "key": "QYva4bK2zA4faxwtZbtOyQ7x4IYNt5Z+Vv381cM1g2NyGetS3qs0mPXt9JaQRi4aYre7MtSkOOJymwpJj4pPJz8t/H3pVLyi17FAXXWVh1j42C4Mi7IBgtYn0T4GMn0FGx3+dRv2rDrwDaA0guC9GKH8PzHsbrlH1ibewJ5VLFcUzFy6CY14muXTFNb/XyvK0Z8fCU2UR3VILVDotDJfJCOEhhOs0uKLNOeyoApbeS2N",
                "comment": "nemesis@freedom",
                "enabled": True
            }
        ]
    }
}

Could you create an additional seciton in the airos schema for this and update the Sshd
converter accordingly?

Wireless ap attribute (bssid)

I have:

-wireless.1.ap=04:18:D6:F8:01:F7

But this attribute is needed, you can use the bssid attribute in
NetJSON wireless interfaces.

WPA supplicant

I have this situation:

-wpasupplicant.device.1.status=disabled
+wpasupplicant.status=disabled
+wpasupplicant.profile.1.network.1.key_mgmt.1.name=NONE
+wpasupplicant.profile.1.network.1.priority=100
+wpasupplicant.profile.1.network.2.status=disabled
 wpasupplicant.profile.1.network.1.ssid=ninux-diego-pomezia
+wpasupplicant.profile.1.name=AUTO
+wpasupplicant.device.1.status=disabled
+wpasupplicant.profile.1.network.2.key_mgmt.1.name=NONE
+wpasupplicant.profile.1.network.2.priority=2
+wpasupplicant.device.1.profile=AUTO

Can we avoid adding the necessary lines if not needed?

nemesifier · 2017-07-21T14:10:11Z

tests/airos/test_wpasupplicant.py

+                        "mode": "station",
+                        "ssid": "ap-ssid-example",
+                    },
+                    "encryption": {


as you pointed out, encryption should go in wireless :-D

nemesifier · 2017-07-21T14:10:20Z

tests/airos/test_wpasupplicant.py

+                        "ssid": "ap-ssid-example",
+                        "bssid": "00:11:22:33:44:55",
+                    },
+                    "encryption": {


as you pointed out, encryption should go in wireless :-D

nemesifier · 2017-07-21T14:10:26Z

tests/airos/test_wpasupplicant.py

+                        "mode": "station",
+                        "ssid": "ap-ssid-example",
+                    },
+                    "encryption": {


as you pointed out, encryption should go in wireless :-D

nemesifier · 2017-07-21T14:10:34Z

tests/airos/test_wpasupplicant.py

+                        "mode": "access_point",
+                        "ssid": "ap-ssid-example",
+                    },
+                    "encryption": {


as you pointed out, encryption should go in wireless :-D

nemesifier

Try also to sort the order of keys so the generated output is always identical (as explained via IM).

nemesifier · 2017-07-24T13:33:27Z

netjsonconfig/backends/airos/schema.py

@@ -60,6 +60,16 @@
                    },
                },
            },
+            "interface_settings": {
+                "properties": {
+                    "authoneg": {


add default, title and description please

nemesifier · 2017-07-24T13:33:39Z

netjsonconfig/backends/airos/schema.py

+                    "authoneg": {
+                        "type": "boolean",
+                    },
+                    "flowcontrol": {


add default, title and description please

nemesifier · 2017-07-24T17:04:34Z

netjsonconfig/backends/airos/schema.py

+                                "type": "string",
+                            },
+                            "enabled": {
+                                "type": "boolean",


better if this boolean defaults to True

nemesifier · 2017-07-25T17:11:30Z

netjsonconfig/backends/airos/converters.py

-        else:
+        try:
+            t = self.wireless()[0]['wireless']
+            if t['mode'] == 'access_point'  and t['encryption']['protocol'] == 'wpa2_personal':


there's a double space before the first and

edoput · 2017-07-27T09:14:01Z

I'm attaching a diff between the "default configuration" (reset the antenna and enable advanced network configuration) and the default netjson

default.txt

there are minimal differencies except for the radio section , the one not implemented in netjsonconfig.

I'll go over other configurations (station+wpa2 personal, ap,...) and post the diff later

nemesifier · 2017-08-09T16:36:59Z

netjsonconfig/backends/airos/wpasupplicant.py

@@ -0,0 +1,152 @@
+from interface import encryption, ssid


I think we need a relative import here because I'm getting an exception while testing

nemesifier · 2017-08-09T16:41:02Z

netjsonconfig/backends/airos/interface.py

+    """
+    Return the encryption dict for a wireless interface
+    """
+    return interface['wireless']['encryption']


the encryption key is not mandatory, see: http://netjson.org/rfc.html#rfc.section.5.4.2
if encryption is omitted it means encryption off. I discovered this because I got an exception while testing.

Please, before fixing this issue could you first create a failing test?

coveralls · 2017-08-31T14:46:52Z

Coverage decreased (-1.04%) to 98.887% when pulling 309e712 on airos into a2611ef on master.

nemesifier · 2018-07-04T18:21:59Z

@edoput this code can be used as a good starting point for future work, or we may resume it together as well when I have cleared out some urgent matters and in case you want to dedicate some more time on it.

Can we leave it open or do you have anything against this?

edoput · 2018-07-04T19:15:13Z

Hey, I extracted this work to a different package that can be imported at runtime with my last contribution

This PR is now closed as the work has been included in an external package, netjsonconfig-airos and will be available from the next release (v0.6.3)

Right now is still sitting there as testing on hardware is still too hard but we can manage to include it better in the openwisp controller.

Because the code has been migrated I think that the discussion should too.

I edited the top comment to explain what happened but didn't leave an explanation at the bottom.

nemesifier · 2018-07-04T19:21:21Z

@edoput that's great! What about the docs you wrote? Maybe those could go in your README?

We should also find a way to give visibility to your package somehow, so the chances to find somebody who will help us out will increase.
We could do 2 things: add a new page in the netjsonconfig docs which links your package; show this PR in the contributors board, I just did that.

Related to #154

nemesifier requested changes Jul 17, 2017

View reviewed changes

nemesifier reviewed Jul 17, 2017

View reviewed changes

edoput force-pushed the airos branch from 56bc598 to 10a87d8 Compare July 17, 2017 15:06

nemesifier requested changes Jul 18, 2017

View reviewed changes

edoput force-pushed the airos branch 3 times, most recently from 1065736 to 25cd7f6 Compare July 20, 2017 11:09

openwisp deleted a comment from coveralls Jul 20, 2017

nemesifier reviewed Jul 20, 2017

View reviewed changes

nemesifier requested changes Jul 21, 2017

View reviewed changes

nemesifier reviewed Jul 21, 2017

View reviewed changes

edoput force-pushed the airos branch from 7c8e27b to bd5d522 Compare July 21, 2017 13:26

nemesifier reviewed Jul 21, 2017

View reviewed changes

nemesifier requested changes Jul 21, 2017

View reviewed changes

nemesifier reviewed Jul 21, 2017

View reviewed changes

edoput force-pushed the airos branch from eada34c to fb5284f Compare July 24, 2017 13:47

nemesifier mentioned this pull request Jul 24, 2017

Overriding json-schema from backend does not override definitions #93

Open

nemesifier reviewed Jul 24, 2017

View reviewed changes

edoput force-pushed the airos branch from 539d8ff to 2d6c413 Compare July 25, 2017 09:29

nemesifier requested changes Jul 25, 2017

View reviewed changes

edoput force-pushed the airos branch from a52df9a to d203d8e Compare August 9, 2017 08:11

nemesifier requested changes Aug 9, 2017

View reviewed changes

edoput force-pushed the airos branch 5 times, most recently from 82338b0 to d6d8668 Compare August 14, 2017 13:54

openwisp deleted a comment from coveralls Aug 31, 2017

airos] added tests and mock backends for defaulted converters

309e712

edoput force-pushed the airos branch from 2beeba2 to 309e712 Compare August 31, 2017 14:44

edoput added 5 commits September 6, 2017 18:55

[airos] handled file creation without tar archive

6f1777b

[airos] removed shadowed status function

78fdce1

[airos] set default value for sys keyword when not present

c8e3daf

[cli] print traceback fully if the verbose flag is passed

9a18b3f

[airos] fixed typo to comply with netjson standard

e796427

edoput closed this Jul 4, 2018

nemesifier reopened this Jul 4, 2018

nemesifier added this to To do (general) in OpenWISP Contributor's Board via automation Jul 4, 2018

nemesifier added the enhancement label Jul 4, 2018

nemesifier changed the title ~~Airos~~ [netjsonconfig] Add support for Airos in OpenWISP 2 Jul 4, 2018

nemesifier changed the title ~~[netjsonconfig] Add support for Airos in OpenWISP 2~~ Add support for Airos in OpenWISP 2 Jul 4, 2018

nemesifier changed the title ~~Add support for Airos in OpenWISP 2~~ Add support for Airos in OpenWISP2 Jul 4, 2018

nemesifier moved this from To do (general) to To do (python & django) in OpenWISP Contributor's Board Jul 4, 2018

nemesifier mentioned this pull request Dec 8, 2018

openwisp_controller.connection module (SSH connections) openwisp/openwisp-controller#31

Merged

[docs] Updated github stars and fork buttons #154

8a41b49

Related to #154

nemesifier force-pushed the airos branch from 672ad7d to 8a41b49 Compare May 22, 2020 01:39

nemesifier removed this from To do (Python & Django) in OpenWISP Contributor's Board Jan 23, 2023


		.. include:: ../_github.rst

		The ``AirOS`` backend allows to generate AirOS v8.3 compatible configurations.



		class TestDiscoveryConverter(ConverterTest):

Add support for Airos in OpenWISP2 #91

Are you sure you want to change the base?

Add support for Airos in OpenWISP2 #91

Conversation

edoput commented Jul 15, 2017 • edited Loading

nemesifier left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

coveralls commented Jul 20, 2017

nemesifier left a comment

Choose a reason for hiding this comment

nemesifier left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nemesifier left a comment • edited Loading

Choose a reason for hiding this comment

Spanning Tree Protocol for bridges

Can comment be omitted if empty/not present?

ebtables

igmpproxy

iptables

netconf flowcontrol

netconf autoneg

ntp

pwdog

resolv

Default route

sshd

Wireless ap attribute (bssid)

WPA supplicant

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nemesifier left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

edoput commented Jul 27, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

coveralls commented Aug 31, 2017 • edited Loading

nemesifier commented Jul 4, 2018

edoput commented Jul 4, 2018 • edited Loading

nemesifier commented Jul 4, 2018

edoput commented Jul 15, 2017 •

edited

Loading

nemesifier left a comment •

edited

Loading

nemesifier left a comment •

edited

Loading

coveralls commented Aug 31, 2017 •

edited

Loading

edoput commented Jul 4, 2018 •

edited

Loading