Deploying Mistral 7B-Instruct

Mistral 7B was recently released under 2.0 license. Sounded like a good time to check the current SOTA deployment tooling for models, in general. They recommend, https://github.com/skypilot-org/skypilot .

It's actually a quick and easy way to deploy their model. If you correct the `'run:' |` to include the correct github container repository, ghcr.io/mistralai/mistral-src/vllm:latest , it will deploy the 7b model and give you a PUBLIC endpoint to utilize.

After you test that for a bit, you may try to deploy the 7b-instruct. You may run into a quota issue, navigate to the link and request 64 as the number of vCPUs to increase your limit to. https://us-east-1.console.aws.amazon.com/servicequotas/home/services/ec2/quotas/L-DB2E81BA

Yaml file attached.

12 comments