Add script for doing bulk generation against an endpoint

[aspfohl]

As titled. Example usage:

Current limitations:

• Could hit oom if too much data loaded
• Assumes data is pre-loaded locally supports remote data too!
• Only supports text completion for now (TODO: use SDK?)

Here's an example mcli yaml that works:

name: mpt7b-generate
image: mosaicml/pytorch:latest
# compute:
#   gpus: 1
#   cluster: TODO
command: |-
  for i in {1..20}; do curl http://0.0.0.0:8080/v2/ping && echo 'endpoint is up' && break || echo 'sleeping' && sleep 10; done
  python llm-foundry/scripts/inference/endpoint_generate.py --prompts "The lazy dog jumped over" "The best banana bread recipe is"

integrations:
- integration_type: git_repo
  git_repo: mosaicml/llm-foundry
  pip_install: .
- integration_type: pip_packages
  packages:
    - aiohttp
    - ratelimit

dependent_deployment:
  model:
    download_parameters:
      hf_path: mosaicml/mpt-7b-instruct
  env_variables:
    - key: MODEL_BACKEND_OVERRIDE
      value: "vllm"

env_variables:
  - key: ENDPOINT_URL
    value: "http://0.0.0.0:8080/v2/completions"

[linden-li]

LGTM, thanks for adding this. This should hopefully be less painful once we add batched inference support.