sagemaker-data-prep-train.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "import boto3\n",
    "from sagemaker import get_execution_role\n",
    "from sagemaker.amazon.amazon_estimator import get_image_uri\n",
    "\n",
    "role = get_execution_role()\n",
    "\n",
    "bucket='s3-bucket' # customize to your bucket\n",
    "training_image = get_image_uri(boto3.Session().region_name, 'image-classification')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import urllib.request\n",
    "\n",
    "def download(url):\n",
    "    filename = url.split(\"/\")[-1]\n",
    "    if not os.path.exists(filename):\n",
    "        urllib.request.urlretrieve(url, filename)\n",
    "\n",
    "download('https://github.com/raw/apache/incubator-mxnet/master/tools/im2rec.py')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "mkdir local_data/sunburst\n",
    "mkdir local_data/solid\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "s3sunburst = 's3://{}/cropped-images/Sunburst'.format(bucket)\n",
    "s3solid = 's3://{}/cropped-images/Solid'.format(bucket)\n",
    "!aws s3 cp $s3sunburst local_data/sunburst --recursive --quiet\n",
    "!aws s3 cp $s3solid local_data/solid --recursive --quiet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "python im2rec.py --list --recursive --train-ratio 0.90 woodspread local_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "python im2rec.py --resize 512 --quality 100 --num-thread 4 woodspread local_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Four channels: train, validation, train_lst, and validation_lst\n",
    "s3train = 's3://{}/mxnet_set/train/'.format(bucket)\n",
    "s3validation = 's3://{}/mxnet_set/validation/'.format(bucket)\n",
    "s3train_lst = 's3://{}/mxnet_set/train_lst/'.format(bucket)\n",
    "s3validation_lst = 's3://{}/mxnet_setvalidation_lst/'.format(bucket)\n",
    "\n",
    "# upload the image files to train and validation channels\n",
    "!aws s3 cp woodspread_train.rec $s3train\n",
    "!aws s3 cp woodspread_val.rec $s3validation\n",
    "\n",
    "# upload the lst files to train_lst and validation_lst channels\n",
    "!aws s3 cp woodspread_train.lst $s3train_lst\n",
    "!aws s3 cp woodspread_val.lst $s3validation_lst"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fine-tuning the Image Classification Model\n",
    "\n",
    "### Training parameters\n",
    "There are two kinds of parameters that need to be set for training. The first one are the parameters for the training job. These include:\n",
    "\n",
    "* **Input specification**: These are the training and validation channels that specify the path where training data is present. These are specified in the \"InputDataConfig\" section. The main parameters that need to be set is the \"ContentType\" which can be set to \"application/x-recordio\" or \"application/x-image\" based on the input data format and the S3Uri which specifies the bucket and the folder where the data is present. \n",
    "* **Output specification**: This is specified in the \"OutputDataConfig\" section. We just need to specify the path where the output can be stored after training\n",
    "* **Resource config**: This section specifies the type of instance on which to run the training and the number of hosts used for training. If \"InstanceCount\" is more than 1, then training can be run in a distributed manner. \n",
    "\n",
    "Apart from the above set of parameters, there are hyperparameters that are specific to the algorithm. These are:\n",
    "\n",
    "* **num_layers**: The number of layers (depth) for the network. We use 18 in this samples but other values such as 50, 152 can be used.\n",
    "* **image_shape**: The input image dimensions,'num_channels, height, width', for the network. It should be no larger than the actual image size. The number of channels should be same as the actual image.\n",
    "* **num_training_samples**: This is the total number of training samples. It is set to 15240 for caltech dataset with the current split.\n",
    "* **num_classes**: This is the number of output classes for the new dataset. Imagenet was trained with 1000 output classes but the number of output classes can be changed for fine-tuning. For caltech, we use 257 because it has 256 object categories + 1 clutter class.\n",
    "* **mini_batch_size**: The number of training samples used for each mini batch. In distributed training, the number of training samples used per batch will be N * mini_batch_size where N is the number of hosts on which training is run.\n",
    "* **epochs**: Number of training epochs.\n",
    "* **learning_rate**: Learning rate for training.\n",
    "* **top_k**: Report the top-k accuracy during training.\n",
    "* **resize**: Resize the image before using it for training. The images are resized so that the shortest side is of this parameter. If the parameter is not set, then the training data is used as such without resizing.\n",
    "* **checkpoint_frequency**: Period to store model parameters (in number of epochs).\n",
    "* **use_pretrained_model**: Set to 1 to use pretrained model for transfer learning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "training_image = get_image_uri(boto3.Session().region_name, 'image-classification')\n",
    "\n",
    "# The algorithm supports multiple network depth (number of layers). They are 18, 34, 50, 101, 152 and 200\n",
    "# For this training, we will use 18 layers\n",
    "num_layers = 50\n",
    "# we need to specify the input image shape for the training data\n",
    "image_shape = \"3,512,512\"\n",
    "# we also need to specify the number of training samples in the training set\n",
    "num_training_samples = 13056\n",
    "# specify the number of output classes\n",
    "num_classes = 2\n",
    "# batch size for training\n",
    "mini_batch_size = 10\n",
    "# number of epochs\n",
    "epochs = 50\n",
    "# learning rate\n",
    "learning_rate = 0.00001\n",
    "# period to store model parameters (in number of epochs), in this case, we will save parameters from epoch 2, 4, and 6\n",
    "checkpoint_frequency = 1\n",
    "use_pretrained_model = 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "import time\n",
    "import boto3\n",
    "from time import gmtime, strftime\n",
    "\n",
    "s3 = boto3.client('s3')\n",
    "# create unique job name \n",
    "job_name_prefix = 'sagemaker-woodspread'\n",
    "timestamp = time.strftime('-%Y-%m-%d-%H-%M-%S', time.gmtime())\n",
    "job_name = job_name_prefix + timestamp\n",
    "training_params = \\\n",
    "{\n",
    "    # specify the training docker image\n",
    "    \"AlgorithmSpecification\": {\n",
    "        \"TrainingImage\": training_image,\n",
    "        \"TrainingInputMode\": \"File\"\n",
    "    },\n",
    "    \"RoleArn\": role,\n",
    "    \"OutputDataConfig\": {\n",
    "        \"S3OutputPath\": 's3://{}/{}/output'.format(bucket, job_name_prefix)\n",
    "    },\n",
    "    \"ResourceConfig\": {\n",
    "        \"InstanceCount\": 1,\n",
    "        \"InstanceType\": \"ml.p3.8xlarge\",\n",
    "        \"VolumeSizeInGB\": 30\n",
    "    },\n",
    "    \"TrainingJobName\": job_name,\n",
    "    \"HyperParameters\": {\n",
    "        \"image_shape\": image_shape,\n",
    "        \"num_layers\": str(num_layers),\n",
    "        \"num_training_samples\": str(num_training_samples),\n",
    "        \"num_classes\": str(num_classes),\n",
    "        \"mini_batch_size\": str(mini_batch_size),\n",
    "        \"epochs\": str(epochs),\n",
    "        \"learning_rate\": str(learning_rate),\n",
    "        \"checkpoint_frequency\": str(checkpoint_frequency),\n",
    "        \"use_pretrained_model\": str(use_pretrained_model)\n",
    "    },\n",
    "    \"StoppingCondition\": {\n",
    "        \"MaxRuntimeInSeconds\": 360000\n",
    "    },\n",
    "#Training data should be inside a subdirectory called \"train\"\n",
    "#Validation data should be inside a subdirectory called \"validation\"\n",
    "#The algorithm currently only supports fullyreplicated model (where data is copied onto each machine)\n",
    "    \"InputDataConfig\": [\n",
    "        {\n",
    "            \"ChannelName\": \"train\",\n",
    "            \"DataSource\": {\n",
    "                \"S3DataSource\": {\n",
    "                    \"S3DataType\": \"S3Prefix\",\n",
    "                    \"S3Uri\": s3train,\n",
    "                    \"S3DataDistributionType\": \"FullyReplicated\"\n",
    "                }\n",
    "            },\n",
    "            \"ContentType\": \"application/x-recordio\",\n",
    "            \"CompressionType\": \"None\"\n",
    "        },\n",
    "        {\n",
    "            \"ChannelName\": \"validation\",\n",
    "            \"DataSource\": {\n",
    "                \"S3DataSource\": {\n",
    "                    \"S3DataType\": \"S3Prefix\",\n",
    "                    \"S3Uri\": s3validation,\n",
    "                    \"S3DataDistributionType\": \"FullyReplicated\"\n",
    "                }\n",
    "            },\n",
    "            \"ContentType\": \"application/x-recordio\",\n",
    "            \"CompressionType\": \"None\"\n",
    "        }\n",
    "    ]\n",
    "}\n",
    "print('Training job name: {}'.format(job_name))\n",
    "print('\\nInput Data Location: {}'.format(training_params['InputDataConfig'][0]['DataSource']['S3DataSource']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create the Amazon SageMaker training job\n",
    "sagemaker = boto3.client(service_name='sagemaker')\n",
    "sagemaker.create_training_job(**training_params)\n",
    "\n",
    "# confirm that the training job has started\n",
    "status = sagemaker.describe_training_job(TrainingJobName=job_name)['TrainingJobStatus']\n",
    "print('Training job current status: {}'.format(status))\n",
    "\n",
    "try:\n",
    "    # wait for the job to finish and report the ending status\n",
    "    sagemaker.get_waiter('training_job_completed_or_stopped').wait(TrainingJobName=job_name)\n",
    "    training_info = sagemaker.describe_training_job(TrainingJobName=job_name)\n",
    "    status = training_info['TrainingJobStatus']\n",
    "    print(\"Training job ended with status: \" + status)\n",
    "except:\n",
    "    print('Training failed to start')\n",
    "     # if exception is raised, that means it has failed\n",
    "    message = sagemaker.describe_training_job(TrainingJobName=job_name)['FailureReason']\n",
    "    print('Training failed with the following error: {}'.format(message))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "training_info = sagemaker.describe_training_job(TrainingJobName=job_name)\n",
    "status = training_info['TrainingJobStatus']\n",
    "print(\"Training job ended with status: \" + status)\n",
    "print (training_info)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}