Use LogitsProcessor for transformers integration

[lapp0]

Fixes #806

Related: #789

Requesting comment on whether we should unify the logits processor in this PR as described in the last section of this message. @brandonwillard @rlouf

Problem

transformers has two issues in its generate integration:

• if input_ids is empty, it fails
• it has an inconsistent implementation with other models, it is the only model which doesn't use logits processors

Solution

Implement logits processors for transformers.

Proposed Additional Work

We have four implementations of logits processors which do the same thing. Their implementations have drifted and now there are growing differences in their implementation details. We should simplify this and use a single logits processor for all models, ensuring a bug fix or enhancement for one logits processor applies to all models.

• Additionally, this minimizes the changes necessary for mlx library integration (via mlx-lm) #918 minimal
• Furthermore, this issue only would need to be fixed in one module with a singular implementation: Logits Processors Guide integration will be buggy when len(tokens) > 1 in a Write instruction #855

This WIP PR shows generalized logits_processors.py which aims to be compatible with all inference engines, along with a modified generate/regex.py incorporating the new singular logits processor implementation.

[lapp0]

closed in favor of #966