1 / 2
【Overview】 oMLX is a native macOS inference server built on MLX that runs local large language models on Apple Silicon. It exposes OpenAI- and Anthropic-compatible API endpoints, so existing OpenAI-compatible clients and tools can use local models as a drop-in backend. The problem it addresses is the cost of running agents against local models. Agent sessions repeatedly resend the same long prompts, and without caching the model recomputes the whole prefix every time. oMLX persists KV cache blocks to disk in safetensors format and keeps hot blocks in RAM and cold blocks on SSD, so a previously seen prefix is restored from disk in milliseconds instead of being recomputed. The stated result is that agent time-to-first-token drops from a 30-90s range to under 5s. It is aimed at developers running coding agents on a Mac — tools such as Claude Code, OpenClaw and Cursor — and at Apple Silicon owners with 64GB or more of RAM who want to serve models locally rather than through a hosted API. 【Key features】 • Paged SSD KV caching: KV cache blocks are written to disk in safetensors format, with hot blocks held in RAM and cold blocks on SSD, so known prefixes are restored quickly rather than recomputed. • Continuous batching: Concurrent requests are handled through mlx-lm's BatchGenerator, delivering up to 4.14× generation speedup at 8× concurrency. • Native macOS menu bar app: Start, stop and monitor the server from the menu bar, with a web dashboard for model management, chat and real-time metrics; the app is signed, notarized and supports in-app auto-update. • Multi-model serving: LLM, VLM, embedding and reranker models load simultaneously, with LRU eviction when memory runs low, and models can be browsed and downloaded from the admin dashboard. • OpenAI + Anthropic compatible API: Serves OpenAI-compatible /v1/chat/completions and native Anthropic /v1/messages endpoints, making it a drop-in backend for Claude Code, OpenClaw, Cursor and other OpenAI-compatible clients. • Tool calling + MCP: Supports JSON, Qwen, Gemma, GLM and MiniMax tool calling formats, MCP tool integration, and trimming of oversized tool results, configurable per model. 【Use cases】 1. Backing Claude Code, OpenClaw and Cursor with local models. 2. Serving LLM, VLM, embedding and reranker models locally. 3. Running local LLM inference with a persistent KV cache for agent sessions. 【Pros and cons】 Pros: - The persistent KV cache is the standout feature: it targets the exact bottleneck of long agent sessions, where repeated prefixes would otherwise be recomputed on every turn, and shortens time-to-first-token noticeably. - Compatibility with both the OpenAI and Anthropic API shapes means existing coding agents can be pointed at oMLX with little or no client-side change. - The menu bar app plus web dashboard keep model management, chat and metrics in one place, which suits developers who want a local server without maintaining a separate service stack. Cons: - Paged SSD KV caching is bounded by available SSD storage and provides no eviction. - The native menu bar app requires macOS 15 or later. - Tool calling and MCP behavior is configured per model, so support depends on the model selected. 【FAQ】 Q: What makes oMLX faster for agent workloads than recomputing from scratch? A: Previously seen KV prefixes are persisted as safetensors blocks on SSD and restored from disk in milliseconds, so agent sessions avoid recomputation. oMLX states this drops agent time-to-first-token from 30-90s to under 5s. Q: Can I point Claude Code or Cursor at oMLX without changing my client? A: Yes. oMLX exposes OpenAI-compatible /v1/chat/completions and native Anthropic /v1/messages endpoints, so it works as a drop-in backend for Claude Code, OpenClaw, Cursor and other OpenAI-compatible clients. Q: What happens when I load more models than fit in memory? A: oMLX loads LLM, VLM, embedding and reranker models simultaneously and applies LRU eviction when memory runs low.
free
WPS AI is an AI assistant built into WPS Office that helps users write documents, generate presentation outlines and…
Baidu Comate is an AI coding assistant built by Baidu on its ERNIE model. It goes beyond simple autocomplete: it…
UniConverter is an AI media conversion and enhancement tool for video, audio, and images. It combines everyday…
Added
10/21/2026
Views
667
Clicks
2
6 similar tools to consider instead of oMLX