-
Notifications
You must be signed in to change notification settings - Fork 651
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
DOCS-#6949: Create Modin on Dask cluster tutorial #6950
DOCS-#6949: Create Modin on Dask cluster tutorial #6950
Conversation
Signed-off-by: Kirill Suvorov <kirill.suvorov@intel.com>
examples/tutorial/jupyter/execution/pandas_on_dask/cluster/exercise_5.ipynb
Outdated
Show resolved
Hide resolved
"cell_type": "markdown", | ||
"metadata": {}, | ||
"source": [ | ||
"The next step is to setup your AWS credentials. One can set ``AWS_ACCESS_KEY_ID``, ``AWS_SECRET_ACCESS_KEY``\n", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What about aws configure
? Is there such an option?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think no, because aws configure
requires interactive mode, but this is not possible for Jupyter Notebook.
examples/tutorial/jupyter/execution/pandas_on_dask/cluster/exercise_5.ipynb
Outdated
Show resolved
Hide resolved
examples/tutorial/jupyter/execution/pandas_on_dask/cluster/exercise_5.ipynb
Outdated
Show resolved
Hide resolved
examples/tutorial/jupyter/execution/pandas_on_dask/cluster/exercise_5.ipynb
Outdated
Show resolved
Hide resolved
examples/tutorial/jupyter/execution/pandas_on_dask/cluster/exercise_5.ipynb
Outdated
Show resolved
Hide resolved
examples/tutorial/jupyter/execution/pandas_on_dask/cluster/exercise_5.ipynb
Outdated
Show resolved
Hide resolved
examples/tutorial/jupyter/execution/pandas_on_dask/cluster/exercise_5.ipynb
Outdated
Show resolved
Hide resolved
Co-authored-by: Iaroslav Igoshev <Poolliver868@mail.ru>
examples/tutorial/jupyter/execution/pandas_on_dask/cluster/exercise_5.ipynb
Outdated
Show resolved
Hide resolved
"os.environ[\"AWS_ACCESS_KEY_ID\"] = \"\"\n", | ||
"os.environ[\"AWS_SECRET_ACCESS_KEY\"] = \"\"\n", | ||
"os.environ[\"AWS_SESSION_TOKEN\"] = \"\"" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should these be empty strings? If not, Let's specify "<aws_access_key_id>"
, "<aws_secret_access_key>"
, "<aws_session_token>"
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If these strings are empty, connection to the AWS will be failed.
Ok, let's specify this as you suggest.
examples/tutorial/jupyter/execution/pandas_on_dask/cluster/exercise_5.ipynb
Outdated
Show resolved
Hide resolved
examples/tutorial/jupyter/execution/pandas_on_dask/cluster/exercise_5.ipynb
Outdated
Show resolved
Hide resolved
examples/tutorial/jupyter/execution/pandas_on_dask/cluster/exercise_5.ipynb
Outdated
Show resolved
Hide resolved
examples/tutorial/jupyter/execution/pandas_on_dask/cluster/exercise_5.ipynb
Outdated
Show resolved
Hide resolved
" try:\n", | ||
" dir_name = os.path.dirname(file_path)\n", | ||
" if not os.path.exists(dir_name):\n", | ||
" os.makedirs(dir_name)\n", | ||
" if os.path.exists(file_path): # os.path.isfile(file_path):\n", | ||
" return \"File has already existed.\"\n", | ||
" else:\n", | ||
" urllib.request.urlretrieve(file_url, file_path)\n", | ||
" return \"OK\"\n", | ||
" except Exception as ex:\n", | ||
" return str(ex)" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
" try:\n", | |
" dir_name = os.path.dirname(file_path)\n", | |
" if not os.path.exists(dir_name):\n", | |
" os.makedirs(dir_name)\n", | |
" if os.path.exists(file_path): # os.path.isfile(file_path):\n", | |
" return \"File has already existed.\"\n", | |
" else:\n", | |
" urllib.request.urlretrieve(file_url, file_path)\n", | |
" return \"OK\"\n", | |
" except Exception as ex:\n", | |
" return str(ex)" | |
" os.makedirs(filepath, exist_ok=True) | |
" urllib.request.urlretrieve(file_url, file_path) |
Can we use something like this?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We can, but in this case we will have some problems:
- if any exception is raised we won't receive any messages (Dask worker doesn't send any outputs so it must be managed by user)
- Since this function may be called more than once. we mast guarantee the correct and predictable result.
In my opinion my suggestion is better.
examples/tutorial/jupyter/execution/pandas_on_dask/cluster/exercise_5.ipynb
Outdated
Show resolved
Hide resolved
examples/tutorial/jupyter/execution/pandas_on_dask/cluster/exercise_5.ipynb
Outdated
Show resolved
Hide resolved
examples/tutorial/jupyter/execution/pandas_on_dask/cluster/exercise_5.ipynb
Outdated
Show resolved
Hide resolved
Co-authored-by: Iaroslav Igoshev <Poolliver868@mail.ru>
examples/tutorial/jupyter/execution/pandas_on_dask/cluster/exercise_5.ipynb
Outdated
Show resolved
Hide resolved
examples/tutorial/jupyter/execution/pandas_on_dask/cluster/exercise_5.ipynb
Outdated
Show resolved
Hide resolved
examples/tutorial/jupyter/execution/pandas_on_dask/cluster/exercise_5.ipynb
Outdated
Show resolved
Hide resolved
examples/tutorial/jupyter/execution/pandas_on_dask/cluster/exercise_5.ipynb
Outdated
Show resolved
Hide resolved
examples/tutorial/jupyter/execution/pandas_on_dask/cluster/exercise_5.ipynb
Outdated
Show resolved
Hide resolved
Co-authored-by: Iaroslav Igoshev <Poolliver868@mail.ru>
@anmyachev, any comments? |
examples/tutorial/jupyter/execution/pandas_on_dask/cluster/exercise_5.ipynb
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM!
Co-authored-by: Anatoly Myachev <anatoliimyachev@mail.com>
9914b3b
What do these changes do?
flake8 modin/ asv_bench/benchmarks scripts/doc_checker.py
black --check modin/ asv_bench/benchmarks scripts/doc_checker.py
git commit -s
docs/development/architecture.rst
is up-to-date