Blazing Fast Language Inference

What is dedicated inference?

Dedicated inference offers exclusive access to a specific model, ensuring that you are the sole user of the underlying compute resources. Unlike serverless options, you won't have to share these resources with other users, guaranteeing consistent performance.

This makes it particularly suitable for applications that require low latency, have a heavy workload, or involve batch processing tasks.

Is my data stored or used for training?

No, none of your data is stored or used for training.

Where are my models hosted?

Your models are hosted on ISO 27001 certified infrastructure, utilizing cutting-edge GPUs. The infrastructure is located within Europe, adhering to GDPR standards.

How is the quality assessed?

We utilize Eleuther's Evaluation Harness to assess the quality of models across a wide range of tasks. This allows you to compare models against each other, providing a comprehensive understanding of their performance.

However, for practical reasons, we may subsample larger evaluation datasets to 1,000 entries, which may lead to slight variations when compared to other public evaluations.

What does ⚡ mean?

It indicates that the model is quantized using GPTQ.

GPTQ is a quantization methodology that balances a slight reduction in accuracy with notable reductions in computing demand and energy consumption. This approach offers a diverse range of variants, spanning from compact and economical options to larger, near-lossless alternatives, thereby accommodating various use cases and preferences. GPTQ offers a compelling solution for those seeking to trim inference expenses. Leveraging quantized models enhances cost efficiency while maintaining high standards.