DPO cleanup

[winglian]

Description

This PR cleans up some hardcoding, improves the integration with trl's DPOTrainer and adds support for dpo prompt_strategies.

[plaguss]

Awesome PR! I left a comment in case you see fit. Also, maybe it could be tackled in a different PR, but the preprocess command could also be updated to allow checking rl datasets:

+    if parsed_cfg.rl:
+        _ = load_rl_datasets(cfg=parsed_cfg, cli_args=parsed_cli_args)
+    else:
+        _ = load_datasets(cfg=parsed_cfg, cli_args=parsed_cli_args)
-    _ = load_datasets(cfg=parsed_cfg, cli_args=parsed_cli_args)

[filippo82]

This is most likely not correct. The strategy includes underscores, not ., such as intel_apply_chatml.

[filippo82]

def load(strategy, cfg):
    try:
        load_fn = strategy.split("_")[-1]
        #strategy = ".".join(strategy.split("_")[:-1])
        LOG.info(load_fn)
        LOG.info(strategy)
        mod = importlib.import_module(f".{load_fn}", "axolotl.prompt_strategies.dpo")
        func = getattr(mod, strategy)
        load_kwargs = {}
        return func(cfg, **load_kwargs)
    except Exception as e:  # pylint: disable=broad-exception-caught
        LOG.warning(e)
        return None

[filippo82]

This works for me.

[winglian]

the intention is the setting is something like

type: chatml.argilla

in which case it will load the argilla function from the axolotl.prompt_strategies.dpo.chatml module.

[filippo82]

Hi @winglian 👋🏻 thanks. That makes sense. I will test it later today 👍🏻