unidist 0.7.0

This release makes wheels for PyPI valid, as well as
upgrade minimal supported python version to 3.9.

Key Features and Updates

• Stability and Bugfixes
  • FIX-#416: Modify setup.py to enable python and platforms specific
    builds (#417)
  • FIX-#416: Add missing files into sdist (#418)
  • FIX-#416: Up minimal python version to 3.9 (#420)
  • FIX-#416: Build cython extension with c++11 support (#421)
  • FIX-#416: Upgrade versioneer to get correct version on Windows
    (#422)
  • FIX-#416: Build bdist and sdist in CI and upload to PyPI (#423)

Contributors

@YarShev

unidist 0.6.0

This release contains a few important improvements for MPI backend.

Key Features and Updates

• Stability and Bugfixes
  • FIX-#402: Fix MPICH version check (#403)
  • FIX-#404: Add psutil as a required dependency for unidist[mpi] (#406)
  • FIX-#407: Make data put into MPI object store immutable (#409)
• Performance enhancements
  • PERF-#360: Re - Implement asyncronous get request with a fix (#405)
• Refactor Codebase
  • REFACTOR-#000: Refactor remote function class for MPI backend (#410)

Contributors

@YarShev
@Retribution98
@arunjose696

unidist 0.5.1

This release contains an important bugfix for MPI backend and
adds psutil as a required dependency for unidist[mpi] package.

Key Features and Updates

• Stability and Bugfixes
  • FIX-#402: Fix MPICH version check (#403)
  • FIX-#404: Add psutil as a required dependency for unidist[mpi] (#406)

Contributors

@YarShev
@Retribution98

unidist 0.5.0

This release adds support for a shared object store for MPI backend,
which can be enabled with UNIDIST_MPI_SHARED_OBJECT_STORE environment variable,
contains a lot of performance enhancements, includes several features,
important documentation updates and a bunch of bugfixes.

Key Features and Updates

• Stability and Bugfixes
  • FIX-#329: Allow remaining communication requests to complete (#330)
  • FIX-#343: Get rid of the hacks in the MPI backend (#344)
  • FIX-#345: Test both modes of MPI execution on Linux only (#346)
  • FIX-#349: Remove the warning regarding MpiPickleThreshold (#350)
  • FIX-#354: Fix unidist.wait for MPI backend (#355)
  • FIX-#363: Fix DataID usage between different MPI processes (#364)
  • FIX-#371: Fix check for out of bounds reserved memory in shared storage (#372)
  • FIX-#375: Fix running an MPI cluster without using shared memory (#377)
  • FIX-#340: Fix hangs at low UNIDIST_MPI_SHARED_OBJECT_STORE_THRESHOLD (#380)
  • FIX-#376: Partition buffers of size larger than the _bigmpi.blocksize into blocks while sending and receiving with MPI (#383)
  • FIX-#337: Fix MpiHosts parameter usage (#378)
  • FIX-#000: Revert PR #387 (#388)
  • FEAT-#392: Raise a warning if shared object storage is enabled but the MPI library does not support it (#393)
  • FIX-#389: Revert async get changes made in #359 (#390)
  • FIX-#394: Fix CI running on Windows (#395)
  • FIX-#396: Fix Split_type call for Unidist on MSMPI (#397)
• Performance enhancements
  • PERF-#365: Cache info about sent metadata of shared data (#366)
  • PERF-#360: Implement asyncronous get request (#359)
  • PERF-#367: Use std::fill to fill service shared buffer with a given value (#373)
• New Features
  • FEAT-#285: Implement shared object storage for MPI backend (#286)
  • FEAT-#341: Add the ability to specify environment variables for MPI workers (#352)
  • FEAT-#347: Get rid of mpi4py-mpich dependency (#358)
  • FEAT-#386: Make MPI shared object store default option (#387)
• Refactor Codebase
  • REFACTOR-#000: Remove unnecessary build step in CI (#339)
  • REFACTOR-#000: Introduce task_info metadata package (#348)
  • REFACTOR-#000: Remove unncessary else branch in mpi init (#351)
  • REFACTOR-#000: Change mpi_state.comm to mpi_state.global_comm (#379)
  • REFACTOR-#000: Change IsMpiSpawnWorkers to MpiSpawn to make it more concise (#382)
• Documentation improvements
  • DOCS-#000: Update the workflow status badge (#332)
  • DOCS-#000: Add the downloads badge in readme (#333)
  • DOCS-#384: Fix warnings when building documentation (#385)
  • DOCS-#336: Add documentation about shared object storage (#391)

Contributors

@YarShev
@Retribution98
@arunjose696

unidist 0.4.1

This release contains an important bugfix for MPI backend.

Key Features and Updates

• Stability and Bugfixes
  • FIX-#329: Allow remaining communication requests to complete (#330)

Contributors

@YarShev

unidist 0.4.0

This release makes MPI the default backend, includes new features and
important documentation updates for it, as well as contains a bunch of bugfixes.

Key Features and Updates

• Stability and Bugfixes
  • FIX-#275: Pass python flags into workers (#276)
  • FIX-#281: Do not shutdown MPI backend multiple times (#282)
  • FIX-#283: Exit monitor and worker processes in SPMD mode (#284)
  • FIX-#294: Fix MPI backend launch for Open MPI in a cluster (#295)
  • FIX-#242: Fix behavior of unidist.wait for MPI backend (#291)
  • FIX-#000: Add licence header to request_store.py (#307)
  • FIX-#296: Make workers exit the loop correctly (#312)
  • FIX-#318: Do not initialize MPI in serialization.py (#319)
  • FIX-#320: Improve CI (#321)
  • FIX-#322: Disable actor tests for Ray (#323)
  • FIX-#324: Pin pydantic<2 to fix CI (#325)
  • FIX-#326: Set I_MPI_SPAWN for Intel MPI when using dynamic spawn (#327)
• Performance enhancements
  • PERF-#287: Optimize python list appends (#290)
• New Features
  • FEAT-#288: Make MPI as default backend (#293)
  • FEAT-#305: Introduce MpiBackoff configuration value (#306)
  • FEAT-#316: Introduce MpiLog configuration value (#317)
• Refactor Codebase
  • REFACTOR-#000: Refactor request store to make it clearer (#311)
• Documentation improvements
  • DOCS-#000: Clarify the phrases regarding pyseq backend (#272)
  • DOCS-#277: Add notes regarding installation of MPI implementations (#278)
  • DOCS-#279: Advise original MPI binaries rather than conda-forge packages (#280)
  • DOCS-#299: Extend troubleshooting page as to MPI errors (#300)
  • DOCS-#301: Add Optimization Notes section to the docs (#302)
  • DOCS-#303: Fix a typo in unidist_on_mpi.rst (#304)
  • DOCS-#314: Add troubleshooting block for 'OpenSSL version mismatch' error (#315)

Contributors

@YarShev
@Retribution98
@arunjose696
@luweizheng

unidist 0.3.0

This release contains a few performance enhancements for MPI backend,
the renaming of backends "multiprocessing" to "pymp" and "python" to "pyseq",
includes a couple of important updates for documentation and multiple bugfixes.

Key Features and Updates

• Stability and Bugfixes
  • FIX-#170: Fix serialization/deserialization for Ray actors (#171)
  • FIX-#172: Fix serialization/deserialization for Dask actors (#173)
  • FIX-#174: Fix serialization/deserialization for MPI actors (#177)
  • FIX-#179: Handle None return values correctly on MPI backend (#180)
  • FIX-#175; FIX-#178: Fix nested remote calls of remote tasks and actor methods (#176)
  • FIX-#187: Make Data_ID unique for different workers on MPI backend (#188)
  • FIX-#183: Fix serialization of dict_keys for MPI backend (#184)
  • FIX-#189: Fix Actor scheduling (#190)
  • FIX-#191: Fix pendinq response after blocking GET request (#192)
  • FIX-#193: Fix actor async execution (#194)
  • FIX-#197: Initialize a backend when doing unidist.put (#198)
  • FIX-#201: Fix warnings to be aligned with latest flake8 (#202)
  • FIX-#211: Fix dask[complete] syntax in conda environment file (#212)
  • FIX-#213: Fix NamedTuple serialization (#215)
  • FIX-#209: Fix set up of conda in CI to make it work properly (#210)
  • FIX-#216: Fix development environment for ray==2.1.0 (#217)
  • FIX-#218: Install Ray from PyPI to make CI work (#220)
  • FIX-#224: Fix recursion in processing remote task arguments and serialization of dictionary that has a tuple key (#225)
  • FIX-#228: Add missing configs to workers initialization (#229)
  • FIX-#233: Fix passing MpiHosts config to workers initialization (#234)
  • FIX-#232: Fix sending out-of-band data over 1 GB (#231)
  • FIX-#235: Adding handler only if logger does not have any handlers (#236)
  • FIX-#246: Slight changes to file formatting to match new version of black (#247)
  • FIX-#253: Append worker rank to logger name (#254)
  • FIX-#256: Fix sending a cleanup command from a worker to itself (#257)
  • FIX-#263: Update the year in the licence header (#264)
  • FIX-#259: Fix cleanup data in the workers (#260)
  • FIX-#269: Update min supported Ray version (#270)
• Performance enhancements
  • PERF-#226: Do not use pickle5 for small object serialization (#227)
  • PERF-#243: Using non-blocking send (#244)
  • PERF-#195: Fix serialization for numpy objects (#200)
  • PERF-#195: Fix serialization for pandas and numpy objects (#196)
• New Features
  • FEAT-#185: Add communication logging (#186)
  • FEAT-#204: Allow PERF prefix in the commit messages (#205)
  • FIX-#248: Rename backends "multiprocessing" to "pymp" and "python" to "pyseq" (#265)
• Refactor Codebase
  • REFACTOR-#237: Replace len_buffers with buffer_count (#238)
• Documentation improvements
  • DOCS-#207: Add a doc section regarding installation of Intel MPI (#208)
  • DOCS-#266: Fix documentation build (#267)
  • DOCS-#221: Add mentioning about Modin and unidist integration in to docs (#222)
• Update testing suite
  • TEST-#181: Added async actor tests (#182)
  • TEST-#89: Set up testing for unidist on push (#206)
  • FIX-#250: Require format for PR descriptions instead of commit descriptions (#251)
  • FIX-#250: Replace lint-commit to check-pr-title in ci.yml (#252)

Contributors

@YarShev
@Retribution98
@arunjose696

unidist 0.2.2

This release contains a few important bugfixes for MPI backend.

Key Features and Updates

• Stability and Bugfixes
  • FIX-#201: Fix warnings to be aligned with latest flake8 (#202)
  • FIX-#195: Fix serialization for numpy objects (#200)
  • FIX-#211: Fix dask[complete] syntax in conda environment file (#212)
  • FIX-#213: Fix NamedTuple serialization (#215)
  • FIX-#224: Fix recursion in processing remote task arguments and serialization of dictionary that has a tuple key (#225)
  • FIX-#228: Add missing configs to workers initialization (#229)

Contributors

@YarShev
@Retribution98

unidist 0.2.1

This release contains multiple important bugfixes for MPI backend,
which makes it be more functional and stable.

Key Features and Updates

• Stability and Bugfixes
  • FIX-#170: Fix serialization/deserialization for Ray actors (#171)
  • FIX-#172: Fix serialization/deserialization for Dask actors (#173)
  • FIX-#174: Fix serialization/deserialization for MPI actors (#177)
  • FIX-#179: Handle None return values correctly on MPI backend (#180)
  • FIX-#175: FIX-#178: Fix nested remote calls of remote tasks and actor methods (#176)
  • FIX-#187: Make Data_ID unique for different workers on MPI backend (#188)
  • FIX-#183: Fix serialization of dict_keys for MPI backend (#184)
  • FEAT-#185: Add communication logging (#186)
  • FIX-#189: Fix Actor scheduling (#190)
  • FIX-#191: Fix pendinq response after blocking GET request (#192)
  • FIX-#195: Fix serialization for pandas and numpy objects (#196)
  • FIX-#193: Fix actor async execution (#194)
  • FIX-#197: Initialize a backend when doing unidist.put (#198)
• Update testing suite
  • TEST-#181: Added async actor tests (#182)

Contributors

@YarShev
@Retribution98

unidist 0.2.0

This release adds support for the dynamic spawning of MPI worker
processes, contains significant refactoring of MPI backend, cut off
unidist CLI, includes important updates for documentation and a bunch of
bugfixes.

Key Features and Updates

• Stability and Bugfixes
  • FIX-#130: Make unidist be pure Python (#131)
  • FIX-#133: Add support of running mpi backend via CLI on Windows (#134)
  • FIX-#69: Pin min msgpack version (#135)
  • FIX-#124: Set max_retries=0 for Ray backend (#136)
  • FIX-#142: Fix flake8 CI fail (#143)
  • FIX-#145: Synchronize Ray with latest changes (#146)
  • FIX-#137: Set threads_per_worker=1 for local Dask cluster (#151)
  • FIX-#152: Clean up MPI backend (#153)
  • FIX-#161: Propagate Backend.put('mpi') to MPI workers (#162)
  • FIX-#163: Fix unwrapping object refs and data IDs (#164)
  • FIX-#167: Set lowest bound for Ray versions (#168)
• New Features
  • FEAT-#3: Add cluster_resources in API (#35)
  • FEAT-#139; REFACTOR-#100: Implement the spawning of MPI processes dynamically (#144)
  • FEAT-#159: Add support for Async Actors on MPI backend (#160)
  • FEAT-#165: Add is_initialized feature to unidist API (#166)
• Refactor Codebase
  • REFACTOR-#149: Cut off unidist CLI (#150)
• Documentation improvements
  • DOCS-#128: Update installation section in docs (#132)
  • DOCS-#85: Add docs on running unidist with different backends (#140)
  • DOCS-#147: Update docs to be in line with the code base (#148)
  • DOCS-#154: Make documentation fail on warning (#155)
  • DOCS-#157: Set fail_on_warning: false (#158)

Contributors

@YarShev, @prutskov, @no-ponomarev