Yank state machine out of Worker class

[crusaderky]

This issue is a tracker for the worker state machine refactor, as described in the wider epic #5736.

In scope for this issue is to yank out all state machine logic from Worker and create two new classes, WorkerState and BaseWorker.

Writing additional unit tests, beyond tweaking the already existing ones, is out of scope.

Contributing PRs:

• Deprecate WorkerState accessors #6579
  Blocked by:
  • Refactor wait_for_state() #6581
  • Yank state machine out of Worker class #6566
    Blocked by:
    • Migrate ensure_communicating transitions to new WorkerState event mechanism #5896
    • Refactor all event handlers #6410
    • Encapsulate Worker.batched_stream.send() #6475
      Blocked by:
      • @fail_hard can kill the whole test suite; hide errors #6474
    • Don't invoke log_event from state machine #6512
    • Make worker state machine methods private #6564

[fjetter]

apparently blocks also acts as a "please close ticket" keyword 🤔

[zklaus]

Just wanted to leave some words of gratitude and encouragement. It's great to see this move forward; memory leaks and deadlocks due to worker transitions have become a real problem in our production code lately, so your work on this is greatly appreciated!

[crusaderky]

I'll have to move _notify_plugins from Worker to WorkerState and I'll have to share the Worker.plugins dict by reference with WorkerState. This is because of

distributed/distributed/worker.py

Line 2783 in 9b8172b

self._notify_plugins("transition", ts.key, start, finish, **kwargs)

@fjetter were you expecting this? Do you have an alternative design in mind?

[crusaderky]

I'm going to apply the following changes to methods and attributes of Worker. This will potentially break third party code that directly interact with (what we believe is) internal API.
Please review and speak up if you see any issues.

• These attributes will be shared by reference between Worker and WorkerState, like it already happens for WorkerMemoryManager:

  • data
  • plugins
  • tasks

• These methods will move from Worker to Worker.state and will become private, with no deprecation cycle:

  • transitions()
  • transition_<begin>_<finish>()
  • ensure_task_exists()
  • validate_task_<state>()

• These attributes and methods will move from Worker to Worker.state, with an accessor in Worker which will raise a DeprecationWarning:

  • tasks
  • log
  • stimulus_log
  • story()
  • stimulus_story()
  • total_out_connections
  • comm_threshold_bytes
  • data_needed
  • data_needed_per_worker
  • ready
  • constrained
  • executing_count
  • executed_count
  • long_running
  • has_what
  • in_flight_tasks
  • in_flight_workers
  • busy_workers
  • waiting_for_data_count
  • generation
  • target_message_size
  • transition_counter
  • transition_counter_max
  • available_resources
  • _missing_dep_flight
  • _executing
  • executing_count property
  • _in_flight_tasks
  • in_flight_tasks property
  • validate_task()

[fjetter]

This list looks good to me.