README.nid-ned.org

NSO in Docker: NED standard form skeleton

This is the README file for the NSO in Docker (NID) standard form NED skeleton. If you see this file (README.nid-ned.org) in a repository, it means the repository follows the standard form.

The NED standard form provides a standardized environment, called a testenv, for how to do development and testing. It ships with a CI configuration file for GitLab that enables the test environment to automatically run in CI. All repositories using these skeletons provide a consistent user interface.

Usage of a repo that follows the NED standard form

Run make all to build and test the NED. You will need to set the NSO_VERSION environment variable and likely NSO_IMAGE_PATH, for example:

export NSO_VERSION=5.3
export NSO_IMAGE_PATH=registry.gitlab.com/my-group/nso-docker/
make all

The all make target will first build images using the build target and then run the test suite using the test target.

NSO_VERSION

As the version of NSO is a parameter throughout the entire NSO in Docker ecosystem, you have to supply the NSO version through the environment variable NSO_VERSION.

NSO_IMAGE_PATH

The NSO_IMAGE_PATH specifies the location of the NSO images that are used for building. If you have built NSO in Docker images locally, i.e. you have the images cisco-nso-base and cisco-nso-dev available locally, you do not need to set NSO_IMAGE_PATH. If you want to use images built elsewhere, perhaps from a CI system, you need to specify the path to the Docker registry hosting the images, like registry.gitlab.com/my-group/nso-docker/.

IMAGE_PATH

IMAGE_PATH specifies the base path for the resulting output images.

In CI, IMAGE_PATH is automatically set, if not already defined (through a CI variable), to the project namespace path. For example, for the project gitlab.com/example/foobar, the IMAGE_PATH would be registry.gitlab.com/example/.

For a local build, IMAGE_PATH does not have a default value but can be manually set.

Building and testing

As part of the build make target, three docker images are produced:

• netsim: a Docker image that runs the NED as a netsim device
• testnso: an NSO image that has the NED loaded
  • it runs NSO on start and is thus ready to use the NED to talk to, for example, the netsim device
• ned: a Docker image that only contains the compiled NED
  • this container can’t be run, it’s only a vessel to carry the compiled output to another Docker image build

To produce these images, a multi-stage Dockerfile is used where the first stage compiles packages and later stages produce the mentioned images.

After build, the test suite will start up a test environment (testenv) that consists of:

• netsim container running the netsim image
• nso container running the testnso image

Then the tests, defined in the testenv-test target of the repository specific Makefile, are run. The skeleton contains an example for how to run basic tests on the NED that verifies (together with the build):

• YANG models compiles (happens as part of build)
• compiled NED loads in NSO (or the testnso container would fail to start)
• netsim starts up
• NSO can write configuration to the netsim and read it back
  • this test is accomplished by loading an XML configuration file that needs to be customised for the particular NED
    • do note however that it is the ONLY step that needs customization to achieve (CI) testing of a NED
    • modify the file at test/device-config-hostname.xml

The test environment - testenv

There are a number of make targets related to the control of testenv:

• testenv-start: Start the test environment
  • the standard topology, which consists of one test NSO instance (of the testnso image) and one netsim (of the netsim image) is defined in the standard form NED skeleton
  • a Docker network specific to this testenv is used
    • this makes it possible to have network localized names, like a netsim can be called and accessed via the name dev1 and Docker handles name resolution to the actual IP address
      • running multiple testenv in parallel won’t collide as we have a network (namespace) per testenv
    • the netsim container is called dev1
  • it is possible to start up more containers, like netsims, which should be achieved by adding them to the testenv-start-extra target in Makefile
    • ensure that you have $(DOCKER_ARGS) in the argument list to docker
      • it starts the container in the correct Docker network and sets the correct label, which is a prerequisite for the testenv-stop target to work
• testenv-build: recompile packages and run request packages reload in all NSO containers
  • only packages in packages/ will be rebuilt
  • included packages that are specified through manifests in the includes/ directory, are not rebuilt
    • as they might not ship with their source code, it might not even be possible
• testenv-clean-build: for all running NSO containers, clean out and rebuild all packages in packages/ and test-packages/ from scratch
• testenv-stop: Stop the test environment
  • it removes all containers labeled with $(CNT_PREFIX)
    • make sure any extra containers you start have this label by adding $(DOCKER_ARGS) to the argument list
    • any anonymous volumes associated with the containers will be removed as well
  • removes the Docker network
• testenv-shell: Get an interactive bash shell in the testnso container
• testenv-cli: Get an interactive NSO CLI (ncs_cli) in the testnso container
• testenv-runcmdC / testenv-runcmdJ: Run a command with ncs_cli, provide the command through the environment variable CMD
  • the command is expected in the C-style CLI syntax for testenv-runcmdC or J-style CLI with testenv-runcmdJ
  • the runcmd targets can be called to run a command, from an interactive shell like make testenv-runcmdJ CMD="show ncs-state version"
  • it can also be called from other make targets, for example to run commands from tests
    • $(MAKE) testenv-runcmdJ CMD="show ncs-state version"

To access NSO via one of its northbound interfaces, like NETCONF or RESTCONF, use the credentials admin / NsoDocker1337.

Docker tags and prefixes

Built images are tagged with the NSO version and “PNS” (“Pipeline NameSpace”, when in a CI context, or “Pseudo NameSpace”, when running locally, outside of CI), like $(NSO_VERSION)-$(PNS). For local builds, PNS is set to your username (modulo some mangling as some characters are forbidden in Docker image tags), e.g. 5.3-kll (for username kll). In CI, PNS is set to the CI pipeline ID, like 5.3-12345. The PNS part means we don’t immediately overwrite the previously built images with the version tag like 5.3, which might be included by other repositories. We don’t want a development version to overwrite the released one.

Use the tag-release target to set the release tags on the image, e.g. go from 5.3-kll to 5.3. The CI configuration automatically does this for CI jobs run on the master branch. You might have to do it locally in case you wish to retag images so they can be tested with other repositories.

In the testenv, the started containers have a name prefix to avoid collisions with other repositories that make use of the NID skeletons. The prefix is available in the Makefiles under the $(CNT_PREFIX) variable and is set to testenv-$(PROJECT_NAME)-$(NSO_VERSION)-$(PNS). It is also possible to override by manually setting the environment variable CNT_PREFIX.

Repository related make targets

• build: Builds the images
• push: Pushes the ned image
• tag-release: Adds a tag with release version, like 5.3
• push-release: Pushes the release version to the Docker registry
  • this is based on the CI_REGISTRY_IMAGE variable set by GitLab CI

Applying the skeleton / Creating a new repo based on the skeleton

The NED standard form comes as a skeleton that can be applied to a repository by copying over a number of files to your repository. If you are starting from scratch, simple copy the skeleton directory (and init git), like:

cp -av ../nso-docker/skeletons/ned my-ned
cd my-ned
git init
git add .
git commit -a -m "Starting from NID skeleton for NEDs"

Place your NED package in the packages/ folder, despite the plural ‘s’ on packages, you should only use a single NED per repository (other skeletons in the NID ecosystem supports multiple packages). This will automatically include them in the build.

If you are building a new NED, you can start a dev-shell to run ncs-make-package. For this we need access to the cisco-nso-dev image, set NSO_VERSION and NSO_IMAGE_PATH accordingly (see top of this file for more information on that).

export NSO_VERSION=5.3
export NSO_IMAGE_PATH=my-registry.example.com/nso-docker/
make dev-shell

Once in the dev-shell we can use ncs-make-package to make a new package. Our package folder is mounted in /src. Let’s say we want to make a NETCONF NED out of some YANG models in device-yang-models/:

cd /src/packages
ncs-make-package --netconf-ned /src/device-yang-models myned
chown -Rv 1000:1000 myned

Note how when you are working in a Docker container you are root and as such, files you create are owned by root. Change ownership to your own id/gid from within the container. Also note how the container is not aware of your username nor group, so you need to use numeric identifiers.

Now we can build our package and start up a testenv:

make build
make testenv-start

Modify the Makefile to apply the tests you want. For the NED skeleton, an example is included that tests the NED & netsim in combination, see the Makefile for more information.

Including external packages

You can include externally built packages by placing a manifest file in the includes/ folder. It is in fact encouraged to build most packages, such as NEDs and other packages on their own separate git repositories where they can be developed and tested in isolation and later include them.

There should be one manifest file in the includes/ directory per package you want to include. The content of the file should be the URL to the Docker image, including the full registry path. For example, to include bgworker, a Python library for writing background workers in NSO, the manifest file could look like this:

${PKG_PATH}bgworker/package:${NSO_VERSION}

When run in CI, PKG_PATH is set to the Docker registry up and including the namespace of the current project. If our project is hosted at http://gitlab.com/example/my-project and the corresponding Docker registry path is registry.gitlab.com/example/my-project/, then PKG_PATH will be set to registry.gitlab.com/example/. NSO_VERSION naturally contains the value of the NSO version we are currently working with. Evaluating our manifest file, if we are running a CI build for NSO 5.3, we see that it will result in the inclusion of registry.gitlab.com/example/bgworker/package:5.3.

It is recommended that PKG_PATH is always used and that you use continuous mirroring to mirror packages to your own Gitlab instance into the same namespace so that this relative inclusion works.

Included packages are included in the testnso container image but not in the final output in the package image.

NEDs do not typically depend on other packages and so includes might not be useful for the NED package itself, however, the test-packages could very well depend on an external package for which the include manifest functionality can be used.

Include extra files

It is possible to include more files in the Docker image by merely placing them in the directory extra-files/. The entire content of extra-files/ is copied to the root of the resulting testnso image, for example, create extra-files/tmp/foobar to have it placed at /tmp/foobar in the testnso image.

Skeleton content

The NED skeleton contains the following files

• README.nid-ned.org: This README file
• .gitlab-ci.yml: a GitLab CI configuration file that runs the standard testenv targets
• nidcommon.mk: Makefile with definitions common across the NID skeletons
• nidned.mk: Makefile with common targets for the NED skeleton
• Makefile: repository specific Makefile, while it comes with the skeleton, this is meant to be customized for each project
• run-netsim.sh: A Docker entrypoint script for running a netsim container
• test/: Directory containing test related files
• packages/: Standard location for placing the NSO package for the NED itself. Despite plural, only supposed to be one NED package (other skeletons in the NID ecosystem supports multiple packages)
• test-packages/: Standard location for placing NSO packages for testing. These are included in the testnso container that can be used to test the NED but aren’t included in the final output.
• includes/: Standard location for placing manifests for including externally built packages
• extra-files/: Standard location for placing extra files to be included in the testnso. Files are relative to the image file system root, i.e. create extra-files/tmp/foobar to have it placed at /tmp/foobar in the Docker image.

Skeleton source location and updating the skeleton

The authoritative origin for the standard form NED skeleton is the nso-docker repository at https://gitlab.com/nso-developer/nso-docker/, specifically in the directory skeletons/ned. To upgrade to a later version of the skeleton, pull the files from that location and avoid touching the Makefile as it typically contains custom modifications. Be sure to include files starting with a dot (.).

Continuous mirroring

In the NSO in Docker (NID) ecosystem, you are encouraged to mirror repositories that you use. If you found this repository outside of your own git hosting system, you should mirror it to your own git host for it to be built there by your own CI system.

While you can rely on binaries built upstream, including them in your NSO system means a build time risk as broken Internet connectivity or similar could mean you cannot download the packages you depend on. If you need to quickly rebuild your system to integrate a small hot fix, such a risk could mean you cannot deploy a new version. Mirroring the git source repositories of your dependencies not only mean you get to build them locally but also allows you to make minor (or major) modifications to the source. It could be to update the .gitlab-ci.yml file to add a build for a different NSO version or a minor patch to a NED. Mirroring was kept in mind while designing NID ecosystem.

We think it is important to keep a copy of your dependencies locally (in your own Gitlab instance) such that you can build it yourself if necessary. We also think it is important to keep dependencies up to date - in fact, we would like to encourage to “live-at-head”, i.e. follow and include the latest version of a dependency. This is why continuous mirroring of an upstream repository makes sense. However, you should not blindly accept new versions into your main NSO system build as it can break your downstream builds. A gating function is needed and we propose a explicit version pinning workflow to provide for that gating function.

While NSO in Docker isn’t specifically built for Gitlab (the intention is to make it more general than that), it is currently well suited to be hosted in Gitlab since the accompanying CI configuration file is for Gitlab CI. Gitlab features a mirroring functionality that can either push or pull in changes from a remote repository. You can use GitLab mirroring to continuously mirror this repository, however, it comes with a major constraint; only fast-forward merging is possible. This essentially prevents you from making even the most minute changes to the repository as continued mirroring will break. While you are encouraged to upstream any patches or changes you might have for this repository and others in the NID world, there are times when you want to make changes, for example if you need to apply a particular CI runner tag or limit the versions of NSO that you build for. To cater to such scenarios, an alternative mirror mechanism is provided: The CI configuration of this repository and the repo skeletons, are capable of mirroring itself from an upstream through a special CI job.

Enable mirroring from an upstream by scheduling a CI job and setting the CI_MODE variable to mirror. You create a CI schedule by going to CI / CD -> Schedules in Gitlab. In addition, you need to set a number of other variables for the mirroring functionality to work:

• CI_MODE: CI_MODE must be set to mirror which will skip running any of the normal build and test jobs and instead only run the mirror job
• GITLAB_HOSTKEY: the public hostkey(s) of the GitLab server
  • run ssh-keyscan URL-OF-YOUR-GITLAB-SERVER to get suitable output to include in the variable value
• GIT_SSH_PRIV_KEY: a private SSH key to use for cloning of its own repository and pushing the updates
  • create a deploy key that has write privileges
    • generate a key locally ssh-keygen -t ed25519 -f my-nso-docker-mirror
    • in GitLab for your repository, go to Settings -> CI / CD -> Deploy keys
    • create a new key, paste in the public part from what you generated
      • Check Write access allowed
  • enter the private key in the GIT_SSH_PRIV_KEY variable
• MIRROR_REMOTE: the URL of the upstream repository that you wish to mirror
  • for example, to mirror the authoritative repo for nso-docker, you would use https://gitlab.com/nso-developer/nso-docker.git
• MIRROR_PULL_MODE: can be set to rebase to do git pull --rebase instead of a normal git pull

Set CI_MODE=mirror in the CI schedule (since this should only apply for that job and not the normal CI jobs). Use the repo wide CI variable section to set at least GITLAB_HOSTKEY and GIT_SSH_PRIV_KEY, possibly MIRROR_REMOTE too (or set from CI schedule). These are multi-line values and it appears some GitLab versions cannot correctly set multi-line values in the CI schedule, instead using repo wide CI variables effectively works around this issue.

The mirroring functionality is quite simple. It will run git clone to get a copy of its own repository (which is why it needs SSH host keys and deploy keys), then add the upstream repository as a HTTP mirror (presuming it is a public repository and does not require any credentials). It will then pull in changes, allowing merge conflicts, and finally push the result to its own repository, thus functionally achieving a mirror. It uses the user name and email of the user who initiated the CI build as the git commit author (for merge commits).