This guide is intended for contributors to the google-cloud-cpp
libraries.
This will walk you through the steps necessary to set up your development
workstation to compile the code, run the unit and integration tests, and send
contributions to the project.
- Packaging maintainers or developers who prefer to install the library in a
fixed directory (such as
/usr/local
or/opt
) should consult the packaging guide. - Developers who prefer using a package manager such as vcpkg, or Conda, should follow the instructions for their package manager.
- Developers wanting to use the libraries as part of a larger CMake or Bazel project should consult the quickstart guides for the library or libraries they want to use.
- Developers wanting to compile the library just to run some examples or tests should consult the building and installing section of the top-level README file.
- Contributors and developers to
google-cloud-cpp
, this is the right document.
- Linux
- Windows
- macOS
- Appendix: Linux VM on Google Compute Engine
- Appendix: Windows VM on Google Compute Engine
Install the dependencies needed for your distribution. The top-level
README file lists the minimal development tools necessary to
compile google-cloud-cpp
. But for active development you may want to install
additional tools to run the unit and integration tests.
These instructions will describe how to install these tools for Ubuntu 18.04
(Bionic Beaver). For other distributions you may consult the Dockerfiles in
ci/cloudbuild/dockerfiles/
If you use a different distribution, you will need to use the corresponding
package manager (dnf
, zypper
, apk
, etc.) and find the corresponding
package names.
Install the basic development tools:
sudo curl -o /usr/bin/bazelisk -sSL "https://github.com/bazelbuild/bazelisk/releases/download/v1.18.0/bazelisk-linux-amd64"
sudo chmod +x /usr/bin/bazelisk
sudo ln -s /usr/bin/bazelisk /usr/bin/bazel
sudo apt-get update
sudo apt-get --no-install-recommends install -y apt-transport-https apt-utils \
automake build-essential cmake ca-certificates curl git \
gcc g++ libc-ares-dev libc-ares2 libcurl4-openssl-dev libre2-dev \
libssl-dev m4 make pkg-config tar wget zlib1g-dev
sudo apt install -y ninja-build
sudo apt-get install curl zip unzip tar
Install docker and its buildx
component:
sudo apt install -y docker.io docker-buildx
Configure your user to launch docker without needing sudo
:
sudo usermod -aG docker $USER
Update all packages and reboot:
sudo apt upgrade
sudo shutdown -r now
Verify docker is working:
docker run hello-world
Clone the repository and run a test build:
git clone git@github.com:${GITHUB_USER}/google-cloud-cpp.git
time env -C google-cloud-cpp bazel build //google/cloud/storage/...
#
# real 6m36.547s
# user 0m0.825s
# sys 0m0.630s
You need to install the Google Cloud SDK. These instructions work for a GCE VM, but you may need to adapt them for your environment. Check the instructions on the Google Cloud SDK website for alternatives.
type gcloud || sudo google-cloud-cpp/ci/install-cloud-sdk.sh
By default, gcloud
will use the GCE instance credentials. That may be enough
in most cases, but you may need to use your own credentials:
/usr/local/google-cloud-sdk/bin/gcloud auth login
Don't forget the second breakfast login:
/usr/local/google-cloud-sdk/bin/gcloud auth application-default login
Test checkers:
time env -C google-cloud-cpp ci/cloudbuild/build.sh -t checkers-pr
# The first time is slow, as it needs to create the docker image. The second
# time it should run
# real 0m30.997s
# user 0m0.282s
# sys 0m0.232s
(
echo ENABLED_FEATURES=storage,experimental-storage_grpc,opentelemetry
echo RUN_CLANG_TIDY_FIX=NO
) >$HOME/.cloudcxxrc
time env -C google-cloud-cpp ci/cloudbuild/build.sh --distro fedora-latest-cmake --build development
# The second run should get times such as
# real 1m48.347s
# user 0m0.259s
# sys 0m0.247s
Most of the time we use these tools in docker images, so there is no need to install them in your development environment.
Install the buildifier tool, which we use to format BUILD.bazel
files:
sudo curl -fsSL -o /usr/bin/buildifier https://github.com/bazelbuild/buildtools/releases/download/6.0.0/buildifier-linux-amd64
sudo chmod 755 /usr/bin/buildifier
Install shfmt tool, which we use to format our shell scripts:
sudo curl -L -o /usr/local/bin/shfmt \
"https://github.com/mvdan/sh/releases/download/v3.4.3/shfmt_v3.4.3_linux_amd64"
sudo chmod 755 /usr/local/bin/shfmt
Install cmake_format
to automatically format the CMake list files. We pin this
tool to a specific version because the formatting changes when the "latest"
version is updated, and we do not want the builds to break just because some
third party changed something.
sudo apt install -y python3 python3-pip
pip3 install --upgrade pip
pip3 install cmake_format==0.6.13
Install `black``, which we use to format our Python scripts:
pip3 install black==22.3.0
Install mdformat
, which we use to format our .md
documents:
pip3 install mdformat-gfm==0.3.5 \
mdformat-frontmatter==0.4.1 \
mdformat-footnote==0.1.1
Install typos-cli
for spell and typo checking:
sudo apt install cargo
cargo install typos-cli --version 1.16.1
Install the Python modules used in the integration tests:
pip3 install setuptools wheel
pip3 install git+https://github.com/googleapis/storage-testbench
Add the pip directory to your PATH:
export PATH=$PATH:$HOME/.local/bin
Many IDEs use clang for smart code completion, refactoring, etc. To do this,
clang needs to know how to compile each file. This is commonly done by creating
a compile_commands.json
file at the top of the repo, which CMake knows how to
create for us.
For the purposes of clang-tooling, it's best to build all the code and
dependencies from source so that clangd can find all the symbols and jump you to
the right spot in the source files. We prefer using vcpkg
for this, so clone
the vcpkg repo if you don't already have it checked out somewhere.
VCPKG_ROOT=~/vcpkg # We'll use this later
git clone https://github.com/microsoft/vcpkg "${VCPKG_ROOT}"
Next, use CMake with the
CMAKE_EXPORT_COMPILE_COMMANDS
option to compile the google-cloud-cpp
code, and tell it to use vcpkg
to
build all the dependencies.
cmake -S . -B .build -G Ninja \
--toolchain "${VCPKG_ROOT}/scripts/buildsystems/vcpkg.cmake" \
-DGOOGLE_CLOUD_CPP_ENABLE_CCACHE=ON \
-DGOOGLE_CLOUD_CPP_ENABLE_WERROR=ON \
-DCMAKE_EXPORT_COMPILE_COMMANDS=ON
cmake --build .build
Finally, symlink the .build/compile_commands.json
file into the top of the
repo.
ln -s .build/compile_commands.json .
Note: It's also possible to create compile_commands.json
using bazel
, but
it's not quite as easy. If you want to do that, take a look at
https://github.com/grailbio/bazel-compilation-database.
You may need to clone and compile the code as described here.
Run the tests using:
env -C cmake-out/home ctest --output-on-failure -LE integration-test
Run the Google Cloud Storage integration tests:
env -C cmake-out/home \
$PWD/google/cloud/storage/ci/run_integration_tests_emulator_cmake.sh .
Run the Google Cloud Bigtable integration tests:
env -C cmake-out/home \
$PWD/google/cloud/bigtable/ci/run_integration_tests_emulator_cmake.sh .
You may want to install Docker. This will allow you to use the build scripts to test on multiple platforms.
Once Docker is installed, to avoid needing to prepend sudo
to Docker
invocations, add yourself to the Docker group:
sudo usermod -aG docker $USER
If you mainly use Windows as your development environment, you need to install a
number of tools. We use Chocolatey to drive the
installation, so you would need to install it first. This needs to be executed
in a cmd.exe
shell, running as the Administrator
:
@"%SystemRoot%\System32\WindowsPowerShell\v1.0\powershell.exe" -NoProfile -ExecutionPolicy Bypass -Command "iex (
(New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))" && SET "PATH=%PATH%;%ALLUSERSPROFILE%\chocolatey\bin"
Then you can install the dependencies in the same shell:
choco install -y visualstudio2019community visualstudio2019-workload-nativedesktop visualstudio2019buildtools
choco install -y git cmake ninja bazelisk
You may need to create a new key pair to connect to GitHub. Search the web for how to do this. Then you can clone the code:
> cd \Users\%USERNAME%
> git clone git@github.com:<GITHUB-USERNAME_HERE>/google-cloud-cpp.git
> cd google-cloud-cpp
You can use the CI driver scripts to compile the code. You need to load the MSVC environment variables:
> set MSVC_VERSION=2022
> call "c:\Program Files (x86)\Microsoft Visual Studio\%MSVC_VERSION%\Community\VC\Auxiliary\Build\vcvars64.bat"
Or to setup for 32-bit builds:
> set MSVC_VERSION=2022
> call "c:\Program Files (x86)\Microsoft Visual Studio\%MSVC_VERSION%\Community\VC\Auxiliary\Build\vcvars32.bat"
Then run the CI scripts, for example, to compile and run the tests with CMake in debug mode:
> cd google-cloud-cpp
> powershell -exec bypass ci/kokoro/windows/build.ps1 cmake-debug
While to compile and run the tests with Bazel in debug mode you would use:
> cd google-cloud-cpp
> powershell -exec bypass ci/kokoro/windows/build.ps1 bazel-debug
We used to have instructions to setup manual builds with CMake and Bazel on Windows, but they quickly get out of date.
⚠️ This is a work in progress. These instructions might be incomplete because we don't know how to create a "fresh" macOS install to verify we did not miss a step.
To build on macOS you will need the command-line development tools, and some packages from homebrew. Start by installing the command-line development tools:
sudo xcode-select --install
Verify that worked by checking your compiler:
c++ --version
# Expected output: Apple clang .*
Install homebrew, their home page should have up to date instructions. Once
you have installed homebrew
, use it to install some development tools:
brew update
brew install cmake bazel openssl git ninja
You may also want to install ccache
to improve the rebuild times, and
google-cloud-sdk
if you are planning to run the integration tests.
Then clone the repository:
cd $HOME
git clone git@github.com:<GITHUB-USERNAME_HERE>/google-cloud-cpp.git
cd google-cloud-cpp
The guide generally
works for macOS too. We recommend using Ninja
and vcpkg
for development.
When using vcpkg
you should disable the OpenSSL checks:
git clone -C $HOME https://github.com/microsoft/vcpkg.git
env VCPKG_ROOT=$HOME/vcpkg $HOME/vcpkg/bootstrap-vcpkg.sh
cmake -GNinja -S. -Bcmake-out/ \
-DCMAKE_TOOLCHAIN_FILE=$HOME/vcpkg/scripts/buildsystems/vcpkg.cmake \
-DGOOGLE_CLOUD_CPP_ENABLE_MACOS_OPENSSL_CHECK=OFF
cmake --build cmake-out
The CI scripts follow a similar pattern to the scripts for Linux and Windows:
./ci/kokoro/macos/build.sh bazel # <-- Run the `bazel` CI build
./ci/kokoro/macos/build.sh cmake-vcpkg # <-- Build with CMake
From time to time you may want to setup a Linux VM in Google Compute Engine. This might be useful to run performance tests in isolation, but "close" to the service you are doing development for. The following instructions assume you have already created a project:
PROJECT_ID=$(gcloud config get-value project)
# Or manually set it if you have not configured your default project:
Select a zone to run your VM:
gcloud compute zones list
$ ZONE=... # Pick a zone.
Select the name of the VM:
VM=... # e.g. VM=my-windows-devbox
Then create the virtual machine using:
IMAGE_PROJECT=ubuntu-os-cloud
IMAGE=$(gcloud compute images list \
--project=${IMAGE_PROJECT} \
--filter="family ~ ubuntu-2204" \
--sort-by=~name \
--format="value(name)" \
--limit=1)
PROJECT_NUMBER=$(gcloud projects list \
--filter="project_id=${PROJECT_ID}" \
--format="value(project_number)" \
--limit=1)
gcloud compute instances create ${VM} \
--project=${PROJECT_ID} \
--service-account=${PROJECT_NUMBER}-compute@developer.gserviceaccount.com \
--create-disk=auto-delete=yes,boot=yes,device-name=${VM},image=projects/${IMAGE_PROJECT}/global/images/${IMAGE},mode=rw,size=2048,type=projects/${PROJECT_ID}/zones/${ZONE}/diskTypes/pd-balanced \
--zone=${ZONE} \
--machine-type=n2d-standard-128 \
--network-interface=network-tier=PREMIUM,stack-type=IPV4_ONLY,subnet=default \
--maintenance-policy=MIGRATE \
--provisioning-model=STANDARD \
--scopes=https://www.googleapis.com/auth/cloud-platform \
--no-shielded-secure-boot \
--shielded-vtpm \
--shielded-integrity-monitoring \
--labels=goog-ec-src=vm_add-gcloud \
--reservation-affinity=any
Update the ssh
configuration to include this new VM:
# You need to do this every time you add GCE instances.
gcloud compute config-ssh
# Googlers should do this once. It configures SSH to use a tunnel and to forward
# the authentication agent on all VMs in your project.
cat >>.ssh/config <__EOF__
Host *.${PROJECT_ID}
ProxyCommand=corp-ssh-helper %h %p
ForwardAgent yes
__EOF__
Then login using:
ssh -A ${VM}.${ZONE}.${PROJECT_ID}
If you do not have a Windows workstation, but need a Windows development environment to troubleshoot a test or build problem, it might be convenient to create a Windows VM. The following commands assume you have already created a project:
$ PROJECT_ID=... # Set to your project id
Select a zone to run your VM:
$ gcloud compute zones list
$ ZONE=... # Pick a zone close to where you are...
Select the name of the VM:
$ VM=... # e.g. VM=my-windows-devbox
Then create the virtual machine using:
$ IMAGE=$(gcloud compute images list \
--sort-by=~name \
--format="value(name)" \
--limit=1)
$ PROJECT_NUMBER=$(gcloud projects list \
--filter="project_id=${PROJECT_ID}" \
--format="value(project_number)" \
--limit=1)
$ gcloud compute --project "${PROJECT_ID}" instances create "${VM}" \
--zone "${ZONE}" \
--image "${IMAGE}" --image-project "windows-cloud" \
--boot-disk-size "1024" --boot-disk-type "pd-standard" \
--boot-disk-device-name "${VM}" \
--service-account "${PROJECT_NUMBER}-compute@developer.gserviceaccount.com" \
--machine-type "n1-standard-8" \
--subnet "default" \
--maintenance-policy "MIGRATE" \
--scopes "https://www.googleapis.com/auth/bigtable.admin","https://www.googleapis.com/auth/bigtable.data","https://www.googleapis.com/auth/cloud-platform"
Reset the password for your account:
$ gcloud compute --project "${PROJECT_ID}" reset-windows-password --zone "${ZONE}" "${VM}"
Save that password in some kind of password manager. Then connect to the VM using your favorite RDP client. The Google Cloud Compute Engine documentation suggests some third-party clients that may be useful.