Add adaptive example of a Monte Carlo estimate #19
Conversation
willirath
changed the title
|
Counting pods from the list of pods is not what we want. I'll switch to |
There was a problem hiding this comment.
So you're showing two things here, how to compute a pi approximation using Dask, and how it scales easily on Pangeo cloud platform, you should explain both.
This is really interesting, thanks for that! I don't know if this is completely relevant as a Pangeo example (FWIW I think it is), however I strongly feel that the pi computation and scaling at a lower lever could be of interest for dask-examples cc @mrocklin.
Or is there any public API call for: "How not is the cluster atm?"
len(cluster.pods()) looks fine. You could maybe use cluster.scheduler.workers or other scheduler methods, but not sure it would be much better.
| " alt=\"PI monte-carlo estimate\">\n", | ||
| " \n", | ||
| "Using [Dask's adaptivity](http://docs.dask.org/en/latest/setup/adaptive.html), we'll show that it is possible to scale the available resources to meet almost identical wall times irrespective of the acutal work load:\n", | ||
| "\n", |
There was a problem hiding this comment.
typo in actual.
You should maybe introduce also the Montecarlo estimate of py mechanism?
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "# check Adaptive? for help on adapt's kwargs.\n", |
There was a problem hiding this comment.
Could you prepare the cell with the Adaptive? call?
| "import numpy as np\n", | ||
| "from time import time\n", | ||
| "\n", | ||
| "def calc_pi_mc(size):\n", |
There was a problem hiding this comment.
It would be good to first describe step by step the pi estimation, that would be very interesting.
Then at the end put all inside a function to perform scalability analysis.
|
Thanks @guillaumeeb for this review! I just committed a more verbose version that now (briefly) outlines the method and has more details on why it matters that we can scale the cluster to meet a desired target duration over a wide range of problem sizes. |
willirath
changed the title
| "\n", | ||
| "## Actual timings\n", | ||
| "\n", | ||
| "Aming for a duration of 20 seconds per calculation, this is what we actually get:\n", |
There was a problem hiding this comment.
Typo in Aiming.
Maybe put the results at the end? To give the user an idea of what can be achieved, even if he probably won't scale all the way up.
| "source": [ | ||
| "## Tuning adaptivity\n", | ||
| "\n", | ||
| "The following tunes a Dask cluster to use anywhere between 1 and 400 workers and to scale its size so that any computation is finished within 20 seconds. On Pangeo, time scales for starting / stopping workers are of the order of a few seconds, so we set a startup cost to 5 seconds (instead of the default value of 1 second) and increase possible scale-down times by setting the relevant interval to 2 seconds and the number of times a worker needs to be considered expendable before it is actually killed to `10`. We also reduce the default factor that is applied to adapt the cluster to a more modest `1.2`.\n", |
There was a problem hiding this comment.
time scales for starting / stopping workers are of the order of a few seconds
Not sure this is true for Cloud deployments, is it?
Why only 1.2 scale factor? I'm not sure this is used in most cases.
| "source": [ | ||
| "## The actual calculations\n", | ||
| "\n", | ||
| "We loop over volumes of 50 GB, 100 GB, 200 GB, ..., 3200 GB of double-precision random numbers and estimate $\\pi$ as described above." |
There was a problem hiding this comment.
Maybe run it first we a small size, just so the user can validate and try the method.
And then loop, but maybe not up to 3200 GB?
This adds a notebook with a Monte-Carlo estimate of the number pi. It also shows how to tune
Adaptiveto smoothly auto-scale the cluster to approximately match a given execution time of 20 seconds over a huge range of precisions.