Fix GUI resume training

[roomrys]

Description

After attempting a few rounds of training via the GUI, we were a bit surprised to find that the results when resuming training did not perform better than the results for training from scratch. When digging into this, we noticed that the base_checkpoint parameter in the training_config.json file was always null even when directly specifying a file path to the base_checkpoint. It turns out that the CLI arg --base_checkpoint was always overwriting the parameter set in the training config (making the GUI resume training useless).

This PR ensure that the base_checkpoint is only overwritten if a non-default value is specified for --base_checkpoint.

Related: #1130

Types of changes

• Bugfix
• New feature
• Refactor / Code style update (no logical changes)
• Build / CI changes
• Documentation Update
• Other (explain)

Does this address any currently open issues?

[list open issues here]

Outside contributors checklist

• Review the guidelines for contributing to this repository
• Read and sign the CLA and add yourself to the authors list
• Make sure you are making a pull request against the develop branch (not main). Also you should start your branch off develop
• Add tests that prove your fix is effective or that your feature works
• Add necessary documentation (if appropriate)

Thank you for contributing to SLEAP!

❤️

[codecov]

Codecov Report

Merging #1314 (3ed2701) into develop (1e9026d) will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@           Coverage Diff            @@
##           develop    #1314   +/-   ##
========================================
  Coverage    72.38%   72.38%           
========================================
  Files          133      133           
  Lines        23663    23665    +2     
========================================
+ Hits         17128    17130    +2     
  Misses        6535     6535           

Impacted Files	Coverage Δ
sleap/gui/learning/runners.py	32.02% <100.00%> (+0.19%)	⬆️
sleap/nn/training.py	75.42% <100.00%> (+0.02%)	⬆️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[talmo]

I think we need to add one more check here on this fix to address item 4 here: #1291 (reply in thread)

4. Iin the centered_instance.json, I notice that the "training_inds" list shows only the prior trained model's list of frame numbers and does not include any newly annotated frame numbers. # Is this what I should be seeing in the json file when retraining with new training data?

Thanks for pointing that out! I'm not sure we had noticed that previously. The new centered_instance.json probably shouldn't have the list of frame numbers at all and that may indeed be leading to the trainer not using the new labels! We'll look into it as part of the bug fix investigation mentioned above.

Namely, I'm concerned that this part of the DataReaders is kicking in:

sleap/sleap/nn/training.py

Lines 191 to 199 in d05e2bf

	if (
	labels_config.training_inds is not None
	and len(labels_config.training_inds) > 0
	):
	logger.info(
	"Creating training split from explicit indices "
	f"(n = {len(labels_config.training_inds)})."
	)
	training = training.extract(labels_config.training_inds, copy=False)

which results in any new labeled frames to be not used in training when resuming from a previous checkpoint.

This would also lead to issues when resuming from a checkpoint used in another labels project entirely which may have different numbers of labeled frames.

I know the old config isn't used in the Trainer (only the best_model.h5 during model initialization), but it will carry over when doing this from the GUI training config editor.

[roomrys]

The above would only apply if split_by_inds == True.

sleap/sleap/nn/training.py

Lines 173 to 178 in d05e2bf

	if labels_config is not None and labels_config.split_by_inds:
	# First try to split by indices if specified in config.
	if (
	labels_config.validation_inds is not None
	and len(labels_config.validation_inds) > 0
	):

Thus, I created another PR (#1324) which allows users the option to "Split by Indices" if they would like to rerun the exact same experiment in the GUI (to measure any noise, or exited early previously). However, this checkbox is likely most useful for de-selecting "Split by Indices" if selected in the original config (while being transparent).