Refer to new Local Mode by SageMaker service for recommended alternative
Helper application to automate setting up local mode
and docker
for SageMaker Studio. You can also install SageMaker Studio Docker UI extension to get a UI interface that can interact with this extension seamlessly.
It provisions an EC2 instance that is used as a remote docker host to running docker daemon. sdocker
does the following:
- Setup networking and security groups between the instance and SageMaker Studio Apps and EFS
- For EFS, a security group called
EFSDockerHost
is created which only allows connections to port 2049 inbound and outbound - For EC2 instance, you can either supply your own security groups or it will be created for you with open outbound rules and inbounds allowing all ports from SageMaker Studio. The minimum required rules are:
- Inbound:
- port 1111 (or DockerHostPort if not default), port 8080 and destination is SageMaker Studio secutiry group. Other ports might be required depending on container usage.
- OutBound:
- Port 443 to pull images from docker registeries.
- Inbound:
- For EFS, a security group called
- Provision EC2 instance
- Mount SageMaker Studio EFS on EC2 instance
- Run a
docker:dind
image as Host docker daemon and map port 1111 (or custom port) to allow access to docker daemon. - Create docker context on the client to connect to docker host
- SageMaker Studio setup in
VPCOnly
mode (PublicInternetOnly
mode is not supported. - VPC has
DNS hostnames
andDNS resolution
options enabled. - Execution role for Studio with the below permissions:
sagemaker:DescribeDomain sagemaker:DescribeUserProfile sagemaker:ListTags elasticfilesystem:DescribeMountTargets elasticfilesystem:DescribeMountTargetSecurityGroups elasticfilesystem:ModifyMountTargetSecurityGroups ec2:RunInstances ec2:TerminateInstances ec2:DescribeInstances ec2:DescribeInstanceTypes ec2:DescribeImages ec2:DescribeSecurityGroups ec2:DescribeNetworkInterfaces ec2:DescribeNetworkInterfaceAttribute ec2:ModifyNetworkInterfaceAttribute ec2:CreateSecurityGroup ec2:AuthorizeSecurityGroupIngress ec2:RevokeSecurityGroupEgress ec2:AuthorizeSecurityGroupEgress ec2:CreateTags
- Docker
- Docker compose (required for
local mode
) - Python 3
- Boto3
You can use this CloudFormation Template to setup minimal environment.
Setup is staightforward, you clone this repo and then run ./setup.sh
:
$ git clone https://github.com/aws-samples/sagemaker-studio-docker-cli-extension.git
$ cd sagemaker-studio-docker-cli-extension
$ ./setup.sh
When running setup.sh
from terminal of KernalGateway
, make sure to activate your conda environment first.
setup.sh
will do the following:
- Create
~/.sagemaker_studio_docker_cli
directory - Create sample
~/.sagemaker_studio_docker_cli/pre-bootstrap.sh
and~/.sagemaker_studio_docker_cli/post-bootstrap.sh
scripts - Setup softlink for
sdocker
to make it possible to run it from anywhere from command line - Install
docker
anddocker-compose
(requireswget
to be installed on system) - Create
~/temp
directory used inlocal mode
- Create
config.yaml
to change temporay directory to~/temp
- Install SageMaker Python SDK v2.80.0 or higher which introduces Remote Docker Host capability (see PR 2864).
1- Create Studio LifeCycle script
#!/bin/bash
set -eux
STATUS=$(python3 -c "import sagemaker_dataprep";echo $?)
if [ "$STATUS" -eq 0 ]; then
echo 'Instance is of Type Data Wrangler'
else
echo 'Instance is not of Type Data Wrangler'
cd ~
if cd sagemaker-studio-docker-cli-extension
then
git reset --hard
git pull
else
git clone https://github.com/aws-samples/sagemaker-studio-docker-cli-extension.git
cd sagemaker-studio-docker-cli-extension
fi
nohup ./setup.sh > docker_setup.out 2>&1 &
fi
2- Encode script content to base64
encoding using below command:
$ LCC_CONTENT=`openssl base64 -A -in <LifeCycle script file>`
3- Create Studio LifeCycle config from environment variable LCC_CONTENT
$ aws sagemaker create-studio-lifecycle-config --studio-lifecycle-config-name sdocker --studio-lifecycle-config-content $LCC_CONTENT --studio-lifecycle-config-app-type KernelGateway
4- Update Studio domain to add LCC to default user settings (optional)
$ aws sagemaker update-domain --domain-id <domain-id> --default-user-settings '{"KernelGatewayAppSettings": {"DefaultResourceSpec": {"InstanceType": "<default instance type>", "LifecycleConfigArn": "arn:aws:sagemaker:<region>:<AWS account ID>:studio-lifecycle-config/sdocker"}}}'
5- Update user profile settings
$ aws sagemaker update-user-profile --domain-id <domain-id> --user-profile-name <user profile> --user-settings '{"KernelGatewayAppSettings": {"DefaultResourceSpec": {"InstanceType": "ml.t3.medium", "LifecycleConfigArn": "arn:aws:sagemaker:<region>:<AWS account ID>:studio-lifecycle-config/sdocker"}, "LifecycleConfigArns": ["arn:aws:sagemaker:<region>:<AWS account ID>:studio-lifecycle-config/sdocker"]}}'
6- Delete JupyterServer app and create a new one for the above to take effect
sdocker
can be configured to do the following (all the below properties are optional):
- Choose a different AMI. Use
ImageId
property to supply required AMI. - Include EC2 key pair. Use
Key
property to supply public ssh key. - Use custom port to connect to Docker Daemon on host. Use
Port
property to supply custom port. By default, port value is 1111. - Cuustomize root EBS volume size. Use
EBSVolumeSize
property to supply required EBS volume size. - Supply instance profile to the Docker Host to be able to perform tasks like logging into ECR service. Use
InstanceProfileArn
property to supply instance profile ARN. - Use custom security groups for Docker Host. Use
HostSGs
property to supply a list of security group ids that will be attached to the Docker Host. If an empty list is provided, CLI extension will automatically create one for you. - Use custom docker images for CPU or GPU instances. By default, CLI extension uses
docker:dind
image for CPU andbrandsight/dind:nvidia-docker
. UseDockerImageURI
andDockerImageNvidiaURI
properties to supply CPU or GPU images respectively. - You can choose to open additional ports by supplying a list of ports (as a string) under
AdditionalPorts
property.
Configuration file location is ~/.sagemaker_studio_docker_cli/sdocker.conf
.
Make sure your AMI has docker daemon installed and running by default. It is only tested on Amazon linux 2
instances. We recommend using AWS Deep Learning Base AMI (Amazon Linux 2).. You can use below ASW CLI command to find latest AWS Deep learning AMI ID:
$ aws ec2 describe-images --region <region> --owners amazon --filters "Name=name,Values=AWS Deep Learning Base AMI (Amazon Linux 2) Version ????"
For more information on how to create an EC2 key pair check this link
An example of a valid configuration ~/.sagemaker_studio_docker_cli/sdocker.conf
file is shown below:
{
"ImageId": "ami-052783664d99ae241",
"Key": "docker-key",
"Port": 1111,
"EBSVolumeSize": 500,
"InstanceProfileArn": "arn:aws:iam::012345678910:instance-profile/some-profile-name",
"HostSGs": ["sg-00000001", "sg-00000002"],
"DockerImageURI": "docker:dind",
"DockerImageNvidiaURI": "brandsight/dind:nvidia-docker",
"AdditionalPorts": [ "8000", "443" ]
}
The InstanceProfileArn
will be assigned to the EC2 Docker Host. This is useful in case you need to use Systems Manager Session Manager.
The DockerImageURI
and DockerImageNvidiaURI
fields is useful if you need to access these docker images from a private registry.
$ sdocker [COMMANDS][OPTIONS]
Where [COMMANDS]
can be:
-
create-host
: Create security groupsDockerHost
andEFSDockerHost
, then provision EC2 Docker Host. Takes the below[OPTIONS]
:--instance-type
[REQUIRED]--subnet-id
-
terminate-current-host
: Terminates current host, this will only work if creation was successful. Takes no[OPTIONS]
Below example creates a docker host using c5.xlarge
instance type:
$ sdocker create-host --instance-type c5.xlarge
Once the host is provisioned and Healthy
it should show below message:
Successfully launched DockerHost on instance i-xxxxxxxxxxxxxxxxx with private DNS ip-xxx-xxx-xxx-xxx.ec2.internal
Waiting on docker host to be ready
Docker host is ready!
ip-xxx-xxx-xxx-xxx.ec2.internal
Successfully created context "ip-xxx-xxx-xxx-xxx.ec2.internal"
ip-xxx-xxx-xxx-xxx.ec2.internall
Current context is now "ip-xxx-xxx-xxx-xxx.ec2.internal"
Then you can use normal docker commands or use SageMaker Python SDK 'local mode'
Only when the Host was successfully created and turned Healthy
, you can use below command to terminate the EC2 instance:
$ sdocker terminate-current-host
Otherwise, you will need to terminate the instance manually.
- Consult
~/.sdocker/sdocker.log
forsdocker
logs. - To troubleshoot issues related to host instance (eg.
Unhealthy
host), check logs in/home/sagemaker-user/.sagemaker_studio_docker_cli/<intance-type_instance-id>/dockerd-logs
folder.
sdocker
does not terminate or stop EC2 instance after it created, always make sure you have terminated unused instances when you are done. You can useterminate-current-host
command to terminate the current host.- Networking is setup between Docker Host, SageMaker Studio and EFS using two Security Groups (listed below), it is recommended to deleted these when you create new SageMaker Studio Domain so
sdocker
can create new ones that are setup correctly:DockerHost
(You can optionally supply your own security groups if you supply a list of security group ids usingHostSGs
property)EFSDockerHost
If you need to deleteEFSDockerHost
without deleting EFS or Studio domain, you can use the below AWS CLI to update mount target with new list of security groups:
$ aws efs modify-mount-target-security-groups --mount-target-id <mount target id> --security-groups <list of security groups>
Then you can go ahead and delete EFSDockerHost
.
- Currenlty,
sdocker
is setup EC2 with400GB
root EBS volume by default which will be mainly used to store docker images. - Docker uses TLS to connect to Docker Host
See CONTRIBUTING for more information.
This library is licensed under the MIT-0 License. See the LICENSE file.