Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DOCS-#7144: Add information about logging from user defined function #7155

Merged
merged 3 commits into from
Apr 10, 2024

Conversation

arunjose696
Copy link
Collaborator

@arunjose696 arunjose696 commented Apr 8, 2024

What do these changes do?

  • first commit message and PR title follow format outlined here

    NOTE: If you edit the PR title to match this format, you need to add another commit (even if it's empty) or amend your last commit for the CI job that checks the PR title to pick up the new PR title.

  • passes flake8 modin/ asv_bench/benchmarks scripts/doc_checker.py
  • passes black --check modin/ asv_bench/benchmarks scripts/doc_checker.py
  • signed commit with git commit -s
  • Resolves Add information about logging from user defined or remote functions on modin docs #7144
  • tests added and passing
  • module layout described at docs/development/architecture.rst is up-to-date

…ned functions

Signed-off-by: arunjose696 <arunjose696@gmail.com>
docs/usage_guide/advanced_usage/modin_logging.rst Outdated Show resolved Hide resolved
docs/usage_guide/advanced_usage/modin_logging.rst Outdated Show resolved Hide resolved
docs/usage_guide/advanced_usage/modin_logging.rst Outdated Show resolved Hide resolved
Comment on lines 64 to 68
def Udf(df, partition_shape, *dfs):
from modin.config import LogMode, LogMemoryInterval, LogFileSize
LogMode.enable()
# User code goes here
return
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's declare a udf function and pass it in map or apply method like the user interacts with pandas API. The udf will be executed in a remote function.

def udf(x):
    logger stuff
    return x + 1

modin_df.map(udf)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.


.. code-block:: python

@ray.remote
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's declare a udf function and pass it in map or apply method like the user interacts with pandas API. The udf will be executed in a remote function.

def udf(x):
    logger stuff
    return x + 1

modin_df.map(udf)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

Co-authored-by: Iaroslav Igoshev <Poolliver868@mail.ru>
Signed-off-by: arunjose696 <arunjose696@gmail.com>


.. warning::
When attempting to use Modin logging in user defined functions that execute in workers for logging lower-level operators
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Check indents here. There should be 4 spaces like we have in modin/docs/flow/modin/core/dataframe/algebra.rst:L62.

docs/usage_guide/advanced_usage/modin_logging.rst Outdated Show resolved Hide resolved
docs/usage_guide/advanced_usage/modin_logging.rst Outdated Show resolved Hide resolved
docs/usage_guide/advanced_usage/modin_logging.rst Outdated Show resolved Hide resolved
Co-authored-by: Iaroslav Igoshev <Poolliver868@mail.ru>
Signed-off-by: arunjose696 <arunjose696@gmail.com>
@YarShev YarShev merged commit 717a328 into modin-project:master Apr 10, 2024
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add information about logging from user defined or remote functions on modin docs
2 participants