Skip to content

Latest commit

 

History

History
336 lines (217 loc) · 11.4 KB

installation.md

File metadata and controls

336 lines (217 loc) · 11.4 KB
Raw

[ Back to index ]

Click here to see the table of contents.

Last revision of this document: September 29, 2024

CM installation

MLCommons Collective Mind framework requires minimal dependencies to run on any platform: python 3+, pip, git, git-lfs, wget. However, most CM automation recipes shared by the community and MLCommons require Python 3.7+ .

By default, CM will pull Git repositories and cache installations and downloaded files in your $HOME/CM directory (Linux/MacOS). You can change it to any another directory using the CM_REPOS environment variable, for example export CM_REPOS=/scratch/CM.

We suggest you not to install cm4mlops package via PIP since you can't control installation of the CM framework and repositories and it doesn't handle CM errors properly - use a newer version of cm init after installing cmind package as described below.

Feel free to use the online installation GUI.

Here are typical installation procedures across different operating systems:

You can find some Docker containers for CM here.

You can customize CM installation using environment variables described here.

You can reuse misc CM utils listed here.

Ubuntu, Debian

We have successfully tested CM with the following system dependencies on Ubuntu 18.x, 20.x, 22.x , 23.x:

sudo apt update && sudo apt upgrade

sudo apt install python3 python3-pip python3-venv git git-lfs wget curl
sudo apt install libgl1-mesa-dev

Note that you must set up virtual env on Ubuntu 23+ before using any Python project:

python3 -m venv cm
source cm/bin/activate

You can now install CM via PIP:

python3 -m pip install cmind

Note that you may need to restart your shell to update PATH to the "cm" binary. Alternatively you can run

source $HOME/.profile

You can check that CM is available and print internal status as follows:

gfursin@mlcommons-ck-cm-dev:~$ cm test core

CM version: 2.3.0

Python executable used by CK: C:\!Progs\Python310\python.exe

Path to CM package:           C:\!Progs\Python310\lib\site-packages\cmind
Path to CM core module:       C:\!Progs\Python310\lib\site-packages\cmind\core.py
Path to CM internal repo:     C:\!Progs\Python310\lib\site-packages\cmind\repo

Path to CM repositories:      D:\Work1\CM

GitHub for CM developments:        https://github.com/mlcommons/ck/tree/master/cm
GitHub for CM automation scripts:  https://github.com/mlcommons/cm4mlops
Reporting issues and ideas:        https://github.com/mlcommons/ck/issues

You are ready to use CM automation meta-framework.

Red Hat

We have successfully tested CM on Red Hat 9 and CentOS 8

sudo dnf update

sudo dnf install python3 python-pip git git-lfs wget curl

python3 -m pip install cmind --user

MacOS

Note that CM currently does not work with Python installed from the Apple Store. Please install Python via brew as described below.

If brew package manager is not installed, please install it as follows (see details here):

/bin/bash -c "$(curl -fsSL https://github.com/raw/Homebrew/install/HEAD/install.sh)"

Don't forget to add brew to PATH environment as described in the end.

Then install python, pip, git and wget:

brew install python3 git git-lfs wget curl

python3 -m pip install cmind

Sometimes python does not add cm and cmr binaries to the PATH environment variable. You may need to find these files and add their path to PATH variable. We plan to simplify this installation in the future.

Windows

  • Configure Windows 10+ to support long paths from command line as admin: 

    reg add "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem" /v LongPathsEnabled /t REG_DWORD /d 1 /f

  • Download and install Git from git-for-windows.github.io.

    • Configure Git to accept long file names: git config --system core.longpaths true

  • Download and install Python 3+ from www.python.org/downloads/windows.

    • Don't forget to select option to add Python binaries to PATH environment!
    • Configure Windows to accept long fie names during Python installation!

  • Install CM via PIP:

python -m pip install cmind

Note that we have reports that CM does not work when Python was first installed from the Microsoft Store. If CM fails to run, you can find a fix here.

We plan to provide a self-sustained package in the future to simplify CM installation on Windows.

CM CLI testing

If the installation is successful, you can run the CM CLI as follows:

gfursin@cmind:~$ cm

cm {action} {automation} {artifact(s)} {flags} @input.yaml @input.json

Note that you may need to relogin to your shell to update the PATH to the CM CLI!

You can also quickly test the installation and check the version as follows:

gfursin@mlcommons-ck-cm-dev:~$ cm test core

CM version: 1.5.0

Python executable used by CK: /usr/bin/python3

Path to CM package:         /home/user/.local/lib/python3.9/site-packages/cmind
Path to CM core module:     /home/user/.local/lib/python3.9/site-packages/cmind/core.py
Path to CM internal repo:   /home/user/.local/lib/python3.9/site-packages/cmind/repo

Path to CM repositories:    /home/user/CM

GitHub for CM developments:        https://github.com/mlcommons/ck/tree/master/cm
GitHub for CM automation scripts:  https://github.com/mlcommons/ck/tree/master/cm-mlops
Reporting issues and ideas:        https://github.com/mlcommons/ck/issues
Joining the open MLPerf workgroup: https://cKnowledge.org/mlcommons-taskforce

CM init

Use the following command to test CM system dependencies (git, wget, curl, etc):

cm init

Note that it will also install stable cm4mlops repository with the automation recipes for MLOps and MLPerf.

You can skip installation of this repository and use the standard CM command to pull this repo as follows:

cm init --min
cm pull repo mlcommons@cm4mlops

CUDA installation

If you plan to use CUDA for your experiments, please follow this guide to detect or install it and other related dependencies (cuDNN, TensorRT) using CM.

CM customization

You can use the following environment variables to customize CM installation:

  • 'CM_REPOS' - change path to the CM repositories and repos.json file.

    By default, CM will keep CM repositories in:

    • $HOME/CM directory on Linux and MacOS
    • %USERPROFILE%\CM directory on Windows

  • 'CM_CONFIG' - provide full path to a JSON or YAML file with the CM configuration. The content of this file will be merged with the "cfg" dictionary from the config.py.

  • 'CM_DEBUG' - if set to 'yes', turn on internal CM debugging and raise errors in CM automations instead of returning a dictionary with an error {'return':ERROR CODE, 'error':'ERROR note'}

  • 'CM_HOME' - change path to the CM python package with the default 'repo' directory. Useful to improve the default automations inside the CM package.

  • 'CM_INDEX' (CM v1.3.0+) - set to {off|no|false} to turn off CM indexing of all artifacts (when on, it speeds up artifact searching and execution of CM scripts by 10..50x)

CM automation scripts

Please go back to index to continue learning about CM interface and scripts.

However, if you are already familiar with the CM/CK concepts, you can use these CM automation scripts for portable MLOps and DevOps from MLCommons directly by installing the following repository:

cm pull repo mlcommons@cm4mlops

You can switch to a development branch of this or any other CM repository as follows:

cm checkout repo mlcommons@cm4mlops --branch=dev

You can switch back to master branch as follows:

cm checkout repo mlcommons@cm4mlops --branch=master

If you plan to participate in our reproducibility and optimization challenges, we suggest you to create a fork of github.com/mlcommons/ck and use it. In such case, you will be able to create PRs with your updates to the main repository. If you already installed above repo, you will need delete it and install your fork as follows:

cm rm repo mlcommons@cm4mlops --all
cm pull repo --url={URL of the fork of github.com/mlcommons/ck}

If you want to use stable CM snapshots of reusable automation recipes (CM scripts), you can download a stable repository from Zenodo (~5MB):

cm rm repo mlcommons@cm4mlops --all
cm pull repo --url=https://zenodo.org/records/10787459/files/cm-mlops-repo-20240306.zip

You can pull repository and checkout a specific branch as follows:

cm rm repo mlcommons@cm4mlops --checkout=dev
cm pull repo --url=https://zenodo.org/records/10787459/files/cm-mlops-repo-20240306.zip

If you use CM scripts with Python outside containers, we suggest you to set up CM Python virtual environment as described here.

Feel free to check these CM tutorials to learn how to use CM to facilitate reproducible research, run MLPerf out-of-the-box and accelerate technology transfer across rapidly evolving software, hardware, models and data.

Running CM scripts via Docker

CM language allows users to run various automation workflows and applications in the same way either inside automatically generated container snapshots or the latest software/hardware stacks (that may fail and then collaboratively improved by the community).

If you have Docker installed, you can run any CM script using Docker and stay in the container to continue running CM commands as follows:

cm docker script --tags=detect,os -j

You can see more examples of using CM with Docker containers in this folder.

You can browse and reuse shared CM containers from cKnowledge and cTuning via Docker hub.

Problems

If you experience problems with CM installation, please report here or reach the community via Discord server to help improve CM and the overall user experience!