Perplexity Eval for Text Generation Models

[dbogunowicz]

Feature description

A pathway to compute a perplexity of a text generation pipeline given a dataset

Example:

python eval_downstream.py path/to/model.onnx --dataset openai_humaneval --engine onnxruntime --max-samples 16

> openai_humaneval eval results: {'mean_perplexity': 12.780163526535034, 'perplexities': [6.606388092041016, 9.294905662536621, 17.560449600219727, ...

vs

dataset_name="openai_humaneval"
dataset = load_dataset(dataset_name)["test"]

perplexity = evaluate.load("perplexity", module_type="metric")
input_texts = []
for idx, sample in enumerate(dataset):
    input_texts.append(sample["prompt"] + sample["canonical_solution"])
    if idx == 16:
        break
results = perplexity.compute(model_id="facebook/opt-350m",
                             add_start_token=True,
                             predictions=input_texts)
print(results)

> {'perplexities': [6.606388092041016, 9.294907569885254, 17.560461044311523, 13.867135047912598, 17.79571533203125, 12.038525581359863], 'mean_perplexity': 12.780163526535034}

Notable Features

• Implemented the deepsparse port of the HF perplexity metric (https://huggingface.co/spaces/evaluate-metric/perplexity)
• Perplexity evaluation of models WITH kv cache and WITHOUT kv cache support (depends on what type of model model.onnx is)
• Perplexity in both scenarios (cache/no cache for both OPT and CodeGen models) agree exactly with the benchmark, that is the huggingface perplexity calculation
• Since we are directly reusing the pipeline's tokenizer for computing the perplexity, my port removes the need for add_start_token argument - this is detected by the pipeline
• Validated the batched inference of the text generation pipeline

Note: For now, because of the limitation of the kv cache injection (being unable to run "multi-token autoregressive inference"), when kv cache is enabled, the tokens run exclusively through the single-token engine. Once we overcome this hurdle, I'll revisit the perplexity evaluation to ensure that the pipeline's final design is properly supported.

Additional testing

• Make sure that a standard, generative inference with cache and short prompts (single-token engine only) still runs fine
• Make sure that a standard, generative inference with cache and long prompts (single-token and multi-token engine) still runs fine

[dbogunowicz]

We need all the logits that are predicted from sequences: {}, {x1}, {x1, x2}, ... {x1, x2, ... x_n}

[bfineran]

as discussed offline - running right padded for eval will likely not work for the engine (single token prefill) as internally they will build the KV cache assuming left padded and pop from the left side of cache as its built up. in right padded scenario I believe this will delete the actual non-padded values from cache too early.