ExLlamaV2

Category: AI Tools To Run LOCALLY

Blazing-fast inference engine for quantized LLMs on consumer GPUs. EXL2 format pushes 4-bit and lower with state-of-the-art tokens/sec on a single RTX card.

Open ExLlamaV2 →

Tags

Open Source · Local AI · LLM Runtime · Quantization · GPU · Self-Hostable · 18+ · ExLlamaV2 · Free · free tools · free ai tools · no cost

Similar AI tools in AI Tools To Run LOCALLY

• MLC LLM
• WasmEdge
• LMDeploy
• LocalRecall
• Neural Compressor
• Optimum
• AutoAWQ
• GPTQModel
• llm-awq
• BitsAndBytes
• Liger Kernel
• OpenVINO