Build PAI Containers from Source

This document describes how to build PAI containers from source and deploy the newly-built images. You can change the source code, and apply the code change to your cluster by following these steps.

Prepare the Config in a Dev-box Container

To build the PAI containers from source, please first follow our installation guide to deploy an OpenPAI cluster, and follow this document to launch a dev-box container. Please use sudo docker exec -it <name-of-your-dev-box-container> bash to go into the dev-box container.

In the dev-box container, clone your source code, and use the following config to pull the cluster config to a config folder.

git clone -b <your-branch> <your-code-repo> <source-code-folder>
cd <source-code-folder>
./paictl.py config pull -o <config-folder>

You can choose a suitable config folder by your own.

Modify the Cluster Config

Use an editor to open <config-folder>/services-configuration.yaml. In services-configuration.yaml, the following config lines decide the docker images that OpenPAI will use in cluster:

• cluster.docker-registry.domain: The registry which stores the images. Default value is the official docker registry docker.io.
• cluster.docker-registry.username: The username of the registry.
• cluster.docker-registry.password: The password of the username.
• cluster.docker-registry.tag: The tag of the images. All images will share the same tag.
• cluster.docker-registry.namespace: The common prefix string used in all images. Default value is openpai.

By default, OpenPAI will use <namespace>/<docker-image-name-of-each-service>:<tag> as the image name. For example, if you have deployed OpenPAI v1.3.0, it will use images like openpai/webportal:v1.3.0, openpai/rest-server:v1.3.0, ..., etc.

To build and use your own images, you should first modify the config. Then build and push your images. Finally restart the cluster. Your images must be stored in one registry so that OpenPAI will be able to use them. If you also want to use the official docker.io to store your images, please modify services-configuration.yaml as follows:

• cluster.docker-registry.domain: Remains the same.
• cluster.docker-registry.username: Uncomment the corresponding config line, and change it to a valid docker.io username.
• cluster.docker-registry.password: Uncomment the corresponding config line, and change it to the password of the username.
• cluster.docker-registry.tag: Change it to a casual string. All images will share one tag.
• cluster.docker-registry.namespace: Your can change it to the docker.io username. Or you can create an organization in DockerHub, and change it to the organization name.

If your network to docker.io is slow, you can also set up a local registry. You can run sudo docker run -d -p 5000:5000 --name registry registry:2 on a local machine, then use <registry-machine-ip>:5000 as the registry. Please notice the registry you set up doesn't support https by default, thus Docker will refuse to pull images from it. Please edit docker's daemon.json to add <registry-machine-ip>:5000 as a known insecure registry in all machines and restart docker service.

To use a local registry, change services-configuration.yaml as follows:

• cluster.docker-registry.domain: Please change it to <registry-machine-ip>:5000.
• cluster.docker-registry.username: No need to change.
• cluster.docker-registry.password: No need to change.
• cluster.docker-registry.tag: Change it to a casual string. All images will share the same tag.
• cluster.docker-registry.namespace: No need to change.

Build and Push Images

After the modification of services-configuration.yaml, you can build and push the image. Make sure you are using the code you have modified or have changed to your branch:

./build/pai_build.py build -c <config-folder>
./build/pai_build.py push -c <config-folder>

Restart PAI Service

First, stop all PAI services:

./paictl.py service stop

Then, push the modifed config to the cluster. Only pushed config will take effect in cluster deployment:

./paictl.py config push -p <config-folder> -m service

Finally, start all PAI services:

./paictl.py service start

Update One Service

Sometimes, only one PAI service has been modified, thus updating containers of this service is enough. But we recommend you to build all images and restart all services at least once in the beginning. The reason is that: PAI services share one docker registry, one image namespace and one tag, and all images should be present. For example, in a newly-installed v1.3.0 cluster, the images used are openpai/webportal:v1.3.0, openpai/rest-server:v1.3.0, ..., etc. If Alice changes the cluster config to use her docker registry and tag, like alice/<image-name>:alice-test, and only build image alice/webportal:alice-test, other images like alice/rest-server:alice-test will be missing, which will cause the cluster to behave abnormally.

To conclude, if you are going to update one service, please first follow the instructions above to build all images and restart all services. In the following time, you can use the same tag as before and only build the containers of that service. Please refer to the following commands:

./build/pai_build.py build -c  <config-folder> -s <service-name>
./build/pai_build.py push -c <config-folder> -i <container-name-1> <container-name-2> ...
./paictl.py service stop -n <service-name>
./paictl.py service start -n <service-name>

One service may contain multiple containers, you can read the log produced by ./build/pai_build.py build to figure out the container names.

For example, if you are modifying webportal, you can only build / push webportal container, and restart webportal service:

./build/pai_build.py build -c  <config-folder> -s webportal
./build/pai_build.py push -c <config-folder> -i webportal
./paictl.py service stop -n webportal
./paictl.py service start -n webportal