paired kto support

[winglian]

No description provided.

[hamelsmu]

Want to add something on the README about this?

[kashif]

so as mentioned, this is going to unlock the "paired"-kto loss with respect to the DPO dataset, meaning that the batch will have a prompt twice, once for the preferred and once for the unpreferred generation. This means that the KL term in the KTO will have a biased approximation, so kindly keep that in mind

[kashif]

you will also potentially need to pin the TRL version to after this was added >=0.7.5

[winglian]

you will also potentially need to pin the TRL version to after this was added

Seems like 0.7.9 was recently released 4 hours ago, so this should be sufficient?

[kashif]

yes

[teknium1]

What does a dataset for KTO look like

[kashif]

@teknium1 in the DPOTrainer its the same as the DPO one... however for the proper KTO trainer in TRL we went with the convention:

{
       'prompt': List[str],
        'completion': List[str],
        'label': List[bool],
}