Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor(framework) Replace asyncio.Event with threading.Event #3471

Merged
merged 67 commits into from
Jul 11, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
67 commits
Select commit Hold shift + click to select a range
57032f2
wip
jafermarq Apr 22, 2024
b885f22
w/ previous
jafermarq Apr 22, 2024
33ada3e
default `create_run`
jafermarq Apr 22, 2024
54539d9
w/ previous
jafermarq Apr 22, 2024
ea946b7
Merge branch 'main' into in-memory-driver
jafermarq Apr 23, 2024
83ff757
Merge branch 'main' into in-memory-driver
jafermarq Apr 24, 2024
13b0e7b
init
jafermarq Apr 25, 2024
242883e
clientdriverproxytests good
jafermarq Apr 25, 2024
6138968
unfusing
jafermarq Apr 25, 2024
0f3ab98
Merge branch 'main' into in-memory-driver
jafermarq Apr 25, 2024
a8ac6ff
Merge branch 'fuse-grpc-driver-and-helper' into in-memory-driver
jafermarq Apr 25, 2024
49ea5df
flower-simulation uses `InMemoryDriver`
jafermarq Apr 25, 2024
8e808f8
Merge branch 'main' into fuse-grpc-driver-and-helper
jafermarq Apr 25, 2024
c2b2f1e
Merge branch 'fuse-grpc-driver-and-helper' into in-memory-driver
jafermarq Apr 25, 2024
46d0268
Merge branch 'main' into in-memory-driver
jafermarq Apr 29, 2024
91443ce
revert and merge w/ main
jafermarq Apr 29, 2024
91a7b31
use InMemoryDriver
jafermarq Apr 29, 2024
e63fd1c
Merge branch 'main' into in-memory-driver
jafermarq Apr 29, 2024
6455809
Merge branch 'main' into in-memory-driver
jafermarq Apr 30, 2024
b2dcc86
updated /w fab
jafermarq Apr 30, 2024
966b55a
wip
jafermarq Apr 30, 2024
2302dc3
completed unittest
jafermarq Apr 30, 2024
944d8f6
Merge branch 'main' into in-memory-driver
jafermarq Apr 30, 2024
a150844
Merge branch 'in-memory-driver' into use-inmemory-driver-in-simulations
jafermarq May 1, 2024
68e77c6
Merge branch 'main' into in-memory-driver
jafermarq May 1, 2024
7e211c6
Merge branch 'main' into in-memory-driver
panh99 May 1, 2024
ece1774
ensure tasks are deleted after pull
jafermarq May 1, 2024
7eb41c5
w/ previous
jafermarq May 1, 2024
9e6fc95
fix
jafermarq May 1, 2024
da9ed0b
Merge branch 'in-memory-driver' into use-inmemory-driver-in-simulations
jafermarq May 1, 2024
c49df89
Merge branch 'main' into in-memory-driver
jafermarq May 6, 2024
e94874e
Merge branch 'in-memory-driver' into use-inmemory-driver-in-simulations
jafermarq May 6, 2024
f8963d3
state-agnostic driver definition
jafermarq May 6, 2024
f4b244c
minor tweak
jafermarq May 6, 2024
5768b08
Merge branch 'main' into in-memory-driver
jafermarq May 6, 2024
96b4c6a
Merge branch 'in-memory-driver' into use-inmemory-driver-in-simulations
jafermarq May 6, 2024
b3e4407
Merge branch 'main' into in-memory-driver
jafermarq May 8, 2024
f1a59c7
Merge branch 'in-memory-driver' into use-inmemory-driver-in-simulations
jafermarq May 8, 2024
76eab14
Merge branch 'main' into in-memory-driver
jafermarq May 8, 2024
9b1599d
Merge branch 'in-memory-driver' into use-inmemory-driver-in-simulations
jafermarq May 8, 2024
a2c2239
Merge branch 'main' into in-memory-driver
jafermarq May 16, 2024
202e2f4
Merge branch 'main' into in-memory-driver
jafermarq May 18, 2024
69349df
Merge branch 'in-memory-driver' into use-inmemory-driver-in-simulations
jafermarq May 18, 2024
b08ba2c
Merge branch 'main' into use-inmemory-driver-in-simulations
danieljanes May 19, 2024
12be6fe
Merge branch 'main' into use-inmemory-driver-in-simulations
jafermarq May 19, 2024
5f803e6
init
jafermarq May 19, 2024
f5d7e87
init
jafermarq May 19, 2024
d6b8cc4
init
jafermarq May 19, 2024
aed45e8
init
jafermarq May 19, 2024
1427958
Merge branch 'main' into simplify-sim-double-queue
jafermarq Jun 10, 2024
2a04fe7
Merge branch 'simplify-sim-double-queue' into simplify-sim-no-asyncio…
jafermarq Jun 10, 2024
5548ce2
Merge branch 'main' into simplify-sim-double-queue
jafermarq Jun 11, 2024
438154c
Merge branch 'simplify-sim-double-queue' into simplify-sim-no-asyncio…
jafermarq Jun 11, 2024
b0a47bd
Merge branch 'main' into simplify-sim-double-queue
jafermarq Jun 14, 2024
aafb61b
Merge branch 'main' into simplify-sim-double-queue
jafermarq Jul 2, 2024
5c0ed99
merge
jafermarq Jul 2, 2024
39f7ba1
fix
jafermarq Jul 2, 2024
723318a
Merge branch 'simplify-sim-double-queue' into simplify-sim-no-asyncio…
jafermarq Jul 2, 2024
cb6c647
remove asyncio cleanup in tests
jafermarq Jul 2, 2024
02a11b9
Merge branch 'main' into simplify-sim-no-asyncio-backends
danieljanes Jul 10, 2024
02c5d1d
refactor(framework) Remove `asyncio` from core Simulation Engine (#3470)
jafermarq Jul 10, 2024
84727f5
Merge branch 'main' into simplify-sim-no-asyncio-backends
danieljanes Jul 10, 2024
cc655d8
Merge branch 'main' into simplify-sim-no-asyncio-backends
jafermarq Jul 10, 2024
f5d2119
py3.8 compatible
jafermarq Jul 10, 2024
d73437f
merge
jafermarq Jul 10, 2024
2404aba
w/ previous
jafermarq Jul 10, 2024
e75f057
merge
jafermarq Jul 11, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 4 additions & 5 deletions src/py/flwr/server/superlink/fleet/vce/vce_api.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,6 @@
"""Fleet Simulation Engine API."""


import asyncio
import json
import sys
import threading
Expand Down Expand Up @@ -63,7 +62,7 @@ def worker(
taskres_queue: "Queue[TaskRes]",
node_states: Dict[int, NodeState],
backend: Backend,
f_stop: asyncio.Event,
f_stop: threading.Event,
) -> None:
"""Get TaskIns from queue and pass it to an actor in the pool to execute it."""
while not f_stop.is_set():
Expand Down Expand Up @@ -123,7 +122,7 @@ def add_taskins_to_queue(
state: State,
queue: "Queue[TaskIns]",
nodes_mapping: NodeToPartitionMapping,
f_stop: asyncio.Event,
f_stop: threading.Event,
) -> None:
"""Put TaskIns in a queue from State."""
while not f_stop.is_set():
Expand Down Expand Up @@ -153,7 +152,7 @@ def run(
nodes_mapping: NodeToPartitionMapping,
state_factory: StateFactory,
node_states: Dict[int, NodeState],
f_stop: asyncio.Event,
f_stop: threading.Event,
) -> None:
"""Run the VCE."""
taskins_queue: "Queue[TaskIns]" = Queue()
Expand Down Expand Up @@ -232,7 +231,7 @@ def start_vce(
backend_name: str,
backend_config_json_stream: str,
app_dir: str,
f_stop: asyncio.Event,
f_stop: threading.Event,
client_app: Optional[ClientApp] = None,
client_app_attr: Optional[str] = None,
num_supernodes: Optional[int] = None,
Expand Down
9 changes: 4 additions & 5 deletions src/py/flwr/server/superlink/fleet/vce/vce_api_test.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,6 @@
"""Test Fleet Simulation Engine API."""


import asyncio
import threading
import time
from itertools import cycle
Expand Down Expand Up @@ -46,7 +45,7 @@
from flwr.server.superlink.state import InMemoryState, StateFactory


def terminate_simulation(f_stop: asyncio.Event, sleep_duration: int) -> None:
def terminate_simulation(f_stop: threading.Event, sleep_duration: int) -> None:
"""Set event to terminate Simulation Engine after `sleep_duration` seconds."""
sleep(sleep_duration)
f_stop.set()
Expand Down Expand Up @@ -148,15 +147,15 @@ def start_and_shutdown(
) -> None:
"""Start Simulation Engine and terminate after specified number of seconds.

Some tests need to be terminated by triggering externally an asyncio.Event. This
Some tests need to be terminated by triggering externally an threading.Event. This
is enabled when passing `duration`>0.
"""
f_stop = asyncio.Event()
f_stop = threading.Event()

if duration:

# Setup thread that will set the f_stop event, triggering the termination of all
# asyncio logic in the Simulation Engine. It will also terminate the Backend.
# logic in the Simulation Engine. It will also terminate the Backend.
termination_th = threading.Thread(
target=terminate_simulation, args=(f_stop, duration)
)
Expand Down
6 changes: 3 additions & 3 deletions src/py/flwr/simulation/run_simulation.py
Original file line number Diff line number Diff line change
Expand Up @@ -129,7 +129,7 @@ def run_serverapp_th(
server_app_run_config: Dict[str, str],
driver: Driver,
app_dir: str,
f_stop: asyncio.Event,
f_stop: threading.Event,
has_exception: threading.Event,
enable_tf_gpu_growth: bool,
delay_launch: int = 3,
Expand All @@ -138,7 +138,7 @@ def run_serverapp_th(

def server_th_with_start_checks(
tf_gpu_growth: bool,
stop_event: asyncio.Event,
stop_event: threading.Event,
exception_event: threading.Event,
_driver: Driver,
_server_app_dir: str,
Expand Down Expand Up @@ -225,7 +225,7 @@ def _main_loop(
# Initialize StateFactory
state_factory = StateFactory(":flwr-in-memory-state:")

f_stop = asyncio.Event()
f_stop = threading.Event()
# A Threading event to indicate if an exception was raised in the ServerApp thread
server_app_thread_has_exception = threading.Event()
serverapp_th = None
Expand Down