Adding conda packaging for libraft and pyraft

README.md

@@ -1,31 +1,36 @@
 # <div align="left"><img src="https://rapids.ai/assets/images/rapids_logo.png" width="90px"/>&nbsp;RAFT: RAPIDS Analytics Framework Toolkit</div>
 
-RAFT is a library containing building-blocks for rapid composition of RAPIDS Analytics. These building-blocks include shared representations, mathematical computational primitives, and utilities that accelerate building analytics and data science algorithms in the RAPIDS ecosystem. Both the C++ and Python components can be included in consuming libraries, providing operations for both dense and sparse matrix formats in the following general categories:
+RAFT is a [Scipy-like](https://scipy.org/) library for scientific computing, containing CUDA-accelerated building-blocks for rapidly composing analytics in the [RAPIDS](https://rapids.ai) ecosystem. These building-blocks include infrastructure as well as mathematical computational primitives, which accelerate the development of algorithms for data science applications. 
+
+By taking a primitives-based approach to algorithm development, RAFT
+1. accelerates algorithm construction time
+2. reduces the maintenance burden by maximizing reuse across projects, and
+3. centralizes the core computations, allowing future optimizations to benefit all algorithms that use them.
+
+RAFT provides a header-only C++ API (with optional shared libraries to accelerate build time) that cover the following general categories:
 
 #####
 | Category | Description / Examples |
 | --- | --- |
-| **Data Formats** | tensor representations and conversions for both sparse and dense formats |
-| **Data Generation** | graph, spatial, and machine learning dataset generation |
-| **Dense Operations** | linear algebra, statistics |
-| **Spatial** | pairwise distances, nearest neighbors, neighborhood / proximity graph construction |
-| **Sparse/Graph Operations** | linear algebra, statistics, slicing, msf, spectral embedding/clustering, slhc, vertex degree |
-| **Solvers** | eigenvalue decomposition, least squares, lanczos |
-| **Tools** | multi-node multi-gpu communicator, utilities |
-
-By taking a primitives-based approach to algorithm development, RAFT accelerates algorithm construction time and reduces
-the maintenance burden by maximizing reuse across projects. RAFT relies on the [RAPIDS memory manager (RMM)](https://github.com/rapidsai/rmm) which, 
-like other projects in the RAPIDS ecosystem, eases the burden of configuring different allocation strategies globally 
-across the libraries that use it. RMM also provides RAII wrappers around device arrays that handle the allocation and cleanup.
-
-RAFT's primary goals are to be fast, simple, reusable, composable, and comprehensive.
-
-## Build/Install RAFT
-
-Refer to the [Build](BUILD.md) instructions for details on building and including the RAFT library in downstream projects.
+| **Data Formats** | sparse & dense, conversions, and data generations |
+| **Data Generation** | sparse, spatial, machine learning datasets |
+| **Dense Linear Algebra** | matrix arithmetic, norms, factorization |
+| **Spatial** | pairwise distances, nearest neighbors, neighborhood graph construction |
+| **Sparse Operations** | linear algebra, slicing, symmetrization, norms, spectral embedding, msf |
+| **Basic Clustering** | spectral clustering, hierarchical clustering, k-means |
+| **Optimizers** | eigenvalue decomposition, least squares, and lanczos |
+| **Statistics** | sampling, moments, metrics |
+| **Distributed Tools** | multi-node multi-gpu infrastructure |
+
+RAFT also provides a Python API that enables the building of multi-node multi-GPU algorithms in the [Dask](https://dask.org/) ecosystem. We are continuing to improve the coverage of the Python API to expose the building-blocks from the categories above.
 
 ## Getting started
 
+### Rapids Memory Manager (RMM)
+RAFT relies heavily on [RMM](https://github.com/rapidsai/rmm) which, 
+like other projects in the RAPIDS ecosystem, eases the burden of configuring different allocation strategies globally 
+across the libraries that use it. RMM also provides [RAII](https://en.wikipedia.org/wiki/Resource_acquisition_is_initialization)) wrappers around device arrays that handle the allocation and cleanup.
+
 ### C++ Example
 
 Most of the primitives in RAFT accept a `raft::handle_t` object for the management of resources which are expensive to create, such CUDA streams, stream pools, and handles to other CUDA libraries like `cublas` and `cusolver`.
@@ -55,17 +60,22 @@ raft::distance::pairwise_distance(handle, input.data(), input.data(),
                                   workspace.data(), metric);
 ```
 
+## Build/Install RAFT
+
+Refer to the [Build](BUILD.md) instructions for details on building and including the RAFT library in downstream projects.
 
 ## Folder Structure and Contents
 
 The folder structure mirrors other RAPIDS repos (cuDF, cuML, cuGraph...), with the following folders:
 
 - `ci`: Scripts for running CI in PRs
-- `conda`: conda recipes and development conda environments
-- `cpp`: Source code for all C++ code. Headers are in the `include` folder and compiled template specializations for the shared libraries are in `src`.
+- `conda`: Conda recipes and development conda environments
+- `cpp`: Source code for all C++ code. 
+  - `include`: The C++ API is fully-contained here 
+  - `src`: Compiled template specializations for the shared libraries
 - `docs`: Source code and scripts for building library documentation
 - `python`: Source code for all Python source code.
 
 ## Contributing
 
-If you are interested in contributing to the RAFT project, please read our [Contributing guidelines](CONTRIBUTING.md). Refer to the [Developer Guide](DEVELOPER_GUIDE.md) for details on the developer guidelines, workflows, and principals. 
+If you are interested in contributing to the RAFT project, please read our [Contributing guidelines](CONTRIBUTING.md). Refer to the [Developer Guide](DEVELOPER_GUIDE.md) for details on the developer guidelines, workflows, and principals. 

build.sh

@@ -18,7 +18,7 @@ ARGS=$*
 # script, and that this script resides in the repo dir!
 REPODIR=$(cd $(dirname $0); pwd)
 
-VALIDARGS="clean libraft pyraft docs -v -g --compilelibs --allgpuarch --nvtx --show_depr_warn -h --nogtest --buildfaiss"
+VALIDARGS="clean libraft pyraft docs -v -g --compile-libs --compile-nn --compile-dist --allgpuarch --nvtx --show_depr_warn -h --nogtest --buildfaiss"
 HELP="$0 [<target> ...] [<flag> ...]
  where <target> is:
    clean            - remove all existing build artifacts and configuration (start over)
@@ -30,7 +30,9 @@ HELP="$0 [<target> ...] [<flag> ...]
  and <flag> is:
    -v               - verbose build mode
    -g               - build for debug
-   --compilelibs    - compile shared libraries
+   --compile-libs    - compile shared libraries for all components
+   --compile-nn     - compile shared library for nn component
+   --compile-dist   - compile shared library for distance component
    --allgpuarch     - build for all supported GPU architectures
    --buildfaiss     - build faiss statically into raft
    --nogtest        - do not build google tests for libraft
@@ -107,10 +109,17 @@ if hasArg --nogtest; then
     COMPILE_LIBRARIES=OFF
     ENABLE_NN_DEPENDENCIES=OFF
 fi
-if hasArg --compilelibs; then
-    COMPILE_LIBRARIES=ON
+
+if hasArg --compile-nn || hasArg --compile-libs; then
     ENABLE_NN_DEPENDENCIES=ON
+    COMPILE_LIBRARIES=ON
+    CMAKE_TARGET="raft_nn_lib;${CMAKE_TARGET}"
+fi
+if hasArg --compile-dist || hasArg --compile-libs; then
+    COMPILE_LIBRARIES=ON
+    CMAKE_TARGET="raft_distance_lib;${CMAKE_TARGET}"
 fi
+
 if hasArg --buildfaiss; then
     BUILD_STATIC_FAISS=ON
 fi
@@ -171,8 +180,18 @@ if (( ${NUMARGS} == 0 )) || hasArg libraft || hasArg docs; then
 
   if (( ${NUMARGS} == 0 )) || hasArg libraft; then
       # Run all c++ targets at once
-      cmake --build  ${CPP_RAFT_BUILD_DIR} -j${PARALLEL_LEVEL} ${VERBOSE_FLAG} ${CMAKE_TARGET}
+      if hasArg --compile-nn || hasArg --compile-dist || hasArg --compile-libs; then
+        if ! hasArg --nogtest; then
+          CMAKE_TARGET="test_raft;${CMAKE_TARGET}"
+        fi
+
+        echo "-- Compiling targets: ${CMAKE_TARGET}"
+        cmake --build  ${CPP_RAFT_BUILD_DIR} -j${PARALLEL_LEVEL} ${VERBOSE_FLAG} --target ${CMAKE_TARGET}
+      else
+        cmake --build  ${CPP_RAFT_BUILD_DIR} -j${PARALLEL_LEVEL} ${VERBOSE_FLAG}
+      fi
   fi
+
 fi
 
 # Build and (optionally) install the cuml Python package

ci/checks/black_lists.sh

@@ -1,7 +1,7 @@
 #!/bin/bash
-# Copyright (c) 2020, NVIDIA CORPORATION.
+# Copyright (c) 2020-2022, NVIDIA CORPORATION.
 ##########################################
-# cuML black listed function call Tester #
+# RAFT black listed function call Tester #
 ##########################################
 
 # PR_TARGET_BRANCH is set by the CI enviroment

ci/checks/copyright.py

@@ -1,4 +1,4 @@
-# Copyright (c) 2020, NVIDIA CORPORATION.
+# Copyright (c) 2020-2022, NVIDIA CORPORATION.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

ci/gpu/build.sh

@@ -1,7 +1,7 @@
 #!/bin/bash
 # Copyright (c) 2020-2022, NVIDIA CORPORATION.
 #########################################
-# cuML GPU build and test script for CI #
+# RAFT GPU build and test script for CI #
 #########################################
 
 set -e
@@ -26,9 +26,6 @@ cd "$WORKSPACE"
 export GIT_DESCRIBE_TAG=`git describe --tags`
 export MINOR_VERSION=`echo $GIT_DESCRIBE_TAG | grep -o -E '([0-9]+\.[0-9]+)'`
 
-# Read options for cloning/running downstream repo tests
-source "$WORKSPACE/ci/prtest.config"
-
 # ucx-py version
 export UCX_PY_VERSION='0.25.*'
 
@@ -89,7 +86,11 @@ export LD_LIBRARY_PATH_CACHED=$LD_LIBRARY_PATH
 export LD_LIBRARY_PATH=$CONDA_PREFIX/lib:$LD_LIBRARY_PATH
 
 gpuci_logger "Build C++ and Python targets"
-"$WORKSPACE/build.sh" libraft pyraft -v
+if hasArg --skip-tests; then
+  "$WORKSPACE/build.sh" libraft pyraft libraft -v --compile-libs --nogtest
+else
+  "$WORKSPACE/build.sh" libraft pyraft libraft -v --compile-libs
+fi
 
 gpuci_logger "Building docs"
 "$WORKSPACE/build.sh" docs -v
@@ -99,7 +100,6 @@ gpuci_logger "Resetting LD_LIBRARY_PATH"
 export LD_LIBRARY_PATH=$LD_LIBRARY_PATH_CACHED
 export LD_LIBRARY_PATH_CACHED=""
 
-
 ################################################################################
 # TEST - Run GoogleTest and py.tests for RAFT
 ################################################################################
@@ -116,7 +116,7 @@ gpuci_logger "GoogleTest for raft"
 cd "$WORKSPACE/cpp/build"
 GTEST_OUTPUT="xml:$WORKSPACE/test-results/raft_cpp/" ./test_raft
 
-gpuci_logger "Python pytest for cuml"
+gpuci_logger "Python pytest for raft"
 cd "$WORKSPACE/python"
 
-python -m pytest --cache-clear --junitxml="$WORKSPACE/junit-cuml.xml" -v -s
+python -m pytest --cache-clear --junitxml="$WORKSPACE/junit-raft.xml" -v -s

ci/local/README.md

@@ -23,13 +23,13 @@ where:
 ```
 
 Example Usage:
-`bash build.sh -r ~/rapids/cuml -i gpuci/rapidsai-base:cuda9.2-ubuntu16.04-gcc5-py3.6`
+`bash build.sh -r ~/rapids/raft -i gpuci/rapidsai-base:cuda9.2-ubuntu16.04-gcc5-py3.6`
 
 For a full list of available gpuCI docker images, visit our [DockerHub](https://hub.docker.com/r/gpuci/rapidsai-base/tags) page.
 
 Style Check:
 ```bash
-$ bash ci/local/build.sh -r ~/rapids/cuml -s
+$ bash ci/local/build.sh -r ~/rapids/raft -s
 $ . /opt/conda/etc/profile.d/conda.sh
 $ conda activate rapids    #Activate gpuCI conda environment
 $ cd rapids
@@ -43,7 +43,7 @@ There are some caveats to be aware of when using this script, especially if you
 
 ### Docker Image Build Repository
 
-The docker image will generate build artifacts in a folder on your machine located in the `root` directory of the repository you passed to the script. For the above example, the directory is named `~/rapids/cuml/build_rapidsai-base_cuda9.2-ubuntu16.04-gcc5-py3.6/`. Feel free to remove this directory after the script is finished.
+The docker image will generate build artifacts in a folder on your machine located in the `root` directory of the repository you passed to the script. For the above example, the directory is named `~/rapids/raft/build_rapidsai-base_cuda9.2-ubuntu16.04-gcc5-py3.6/`. Feel free to remove this directory after the script is finished.
 
 *Note*: The script *will not* override your local build repository. Your local environment stays in tact.
 

ci/local/build.sh

@@ -6,7 +6,7 @@ MINOR_VERSION=`echo $GIT_DESCRIBE_TAG | grep -o -E '([0-9]+\.[0-9]+)'`
 DOCKER_IMAGE="gpuci/rapidsai:${MINOR_VERSION}-cuda10.1-devel-ubuntu16.04-py3.7"
 REPO_PATH=${PWD}
 RAPIDS_DIR_IN_CONTAINER="/rapids"
-CPP_BUILD_DIR="cuML/build"
+CPP_BUILD_DIR="raft/build"
 PYTHON_BUILD_DIR="python/build"
 CONTAINER_SHELL_ONLY=0