Add FL+XGBoost Baseline

[Aml-Hassan-Abd-El-hamid]

Closes Issue #2220

Checklist

(I'll be adding more details later as I go)

• Centralized model:

  • Prepare Datasets
  • Config the model
  • Evaluate on:
    • a9a
    • cod-rna (different from the paper results by around +3% Accuracy)
    • ijcnn1
    • abalone (different from the paper results by around +3.5 in MSE)
    • cpusmall
    • space_ga

• FedXGBllr:

  • Dividing dataset between clients
  • Individual clients performance of their local dataset for comparison purposes
  • Code FedXGBllr strategy following the Algorithm 1 inst. from the paper.
  • Evaluate on:

    • 2 clients:

      • a9a
      • cod-rna
      • ijcnn1
      • abalone
      • cpusmall
      • space_ga

    • 5 clients:

      • a9a
      • cod-rna
      • ijcnn1
      • abalone
      • cpusmall
      • space_ga

    • 10 clients:

      • a9a
      • cod-rna
      • ijcnn1
      • abalone
      • cpusmall
      • space_ga

• Testing

• Documentation

[Aml-Hassan-Abd-El-hamid]

Hi @jafermarq

That's the first working version of the centralized baseline, those are the results that I got for now:

Results for a9a , Task: BINARY , Train: 0.8558051923234419 , Test: 0.8548849398083695
Results for cod-rna , Task: BINARY , Train: 0.9796843727856992 , Test: 0.96986704954409
Results for ijcnn1 , Task: BINARY , Train: 0.9964114706656143 , Test: 0.9611693395630762

Results for abalone , Task: REG , Train: 1.1268618022311907 , Test: 4.806583543721308
Results for cpusmall , Task: REG , Train: 0.7895664538911239 , Test: 7.022483517505609
Results for space_ga , Task: REG , Train: 0.024788769022399276 , Test: 0.02961448763369315

Results for the centralized model after shuffling the dataset instead of choosing the first 75% rows of the dataset:

Results for a9a , Task: BINARY , Train: 0.8539488411454779 , Test: 0.8491524035705511
Results for cod-rna , Task: BINARY , Train: 0.9767785885658168 , Test: 0.9733166756792034
Results for ijcnn1 , Task: BINARY , Train: 0.990865561694291 , Test: 0.9874352822326051

Results for abalone , Task: REG , Train: 2.12119987844593 , Test: 4.6848017634240415
Results for cpusmall , Task: REG , Train: 1.7059859732195364 , Test: 9.064894281750984
Results for space_ga , Task: REG , Train: 0.024452750881655515 , Test: 0.03206095305593539

Results for YearPredictionMSD , Task: REG , Train: 40.99316293430138 , Test: 76.41236208673367

#shuffled dataset, not the original paper hyper params
Results for cpusmall , Task: REG , Train: 3.4195770468748443 , Test: 7.261266237603397

I use the same metrics as the ones in the paper -Accuracy for binary classification, and mean_squared_error for regression-, I tried different numbers of estimators: (100,500), The ones that I'm showing are for n_estimators = 500 because they're better -but not with a big difference- than the one with n_estimators set to 100.

The results are kinda different from the ones shared on the paper:

        a9a , Test: 0.849
	cod-rna, Test: 0.939
	ijcnn1 , Test: 0.963

	abalone , Test: 1.3
	cpusmall  , Test: 6.7
	space_ga , Test: 0.024

        YearPredictionMSD, Test:80.5

As you can see the main differences are in cod-rna and abalone datasets.

If you can give me any tips that can help me with that, please do.

If you have any notes on the implementation style so far, please let me know so I can modify them.

I'll go through the paper and the notebook you provided again to see if I missed any details and also, and I'll also try and reach out to one of the authors to see if he can help with that.

[jafermarq]

Hi @Aml-Hassan-Abd-El-hamid,

I made some changes to the code:

• removed top-level .gitignore changes and moved them to a .gitignore local to your baseline.
• minor change to __init__.py
• moved print in main.py after load_single_dataset() into that function. (<-- could you double check if this looks good to you?)
• Specified types in missing places (e.g. in client.py)

just a couple of small request:

• could you describe either in the README.md or at the top of your server.py why you need a custom server class? (I know why, but mentioning this would be useful for others out there). Keeping it brief ~100word is sufficient.
• Would it make sense to move your sweep.yaml inside conf/?

[Aml-Hassan-Abd-El-hamid]

Hi @Aml-Hassan-Abd-El-hamid,

I made some changes to the code:

* removed top-level `.gitignore` changes and moved them to a `.gitignore` local to your baseline.

* minor change to `__init__.py`

* moved `print` in `main.py` after `load_single_dataset()` into that function. (**<-- could you double check if this looks good to you?**)

* Specified types in missing places (e.g. in `client.py`)

just a couple of small request:

* could you describe either in the `README.md` or at the top of your `server.py` why you need a custom server class? (I know why, but mentioning this would be useful for others out there). Keeping it brief ~100word is sufficient.

* Would it make sense to move your `sweep.yaml` inside `conf/`?

Hi @jafermarq,
Thank you very much for your help.
All the changes are good with me.
Moving the print statements from main.py to the load_single_dataset() in the dataset.py is totally fine, it's basically the same thing -both are called once at the start of the program-.

• for the server, I'll work on adding that ASAP
• I tried to move sweep.yaml inside conf/ before and ended up with an error as sweep.yaml didn't find main.py.

[jafermarq]

@Aml-Hassan-Abd-El-hamid, this looks good! We'll merge it very soon. 💯