Recipes

Install MLX LM, pull a small instruct model, and chat in the terminal. Nothing leaves the Mac.

1. 01Install MLX LM

   Use a virtual environment so the install stays contained. MLX only runs on Apple silicon.

   python3 -m venv ~/.mlx && source ~/.mlx/bin/activate && pip install mlx-lm

2. 02Chat

   The first run downloads the weights from Hugging Face into ~/.cache/huggingface. Later runs start in seconds.

   mlx_lm.chat --model mlx-community/Qwen3-4B-Instruct-2507-4bit

3. 03Check what it cost you

   Watch memory in Activity Monitor while it generates. If you see swap activity, drop to a smaller model or a shorter context.

Run an OpenAI-compatible endpoint on localhost so editors, scripts and chat clients can use your local model.

1. 01Start the server

   mlx_lm.server speaks the OpenAI chat completions API on port 8080.

   mlx_lm.server --model mlx-community/Qwen3-8B-4bit --port 8080

2. 02Point a client at it

   Any OpenAI SDK works: set the base URL to http://localhost:8080/v1 and use any string as the API key.

   curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{"messages":[{"role":"user","content":"hello"}]}'

3. 03Keep it running

   Leave it in a terminal tab, or wrap it in a launchd agent if you want it back after a reboot. Do not expose the port beyond localhost — there is no authentication.

Build a small local retrieval setup: embed a folder of files, then answer questions against them.

1. 01Install the pieces

   An embedding model for search and a chat model for answers. Both run under MLX.

   pip install mlx-lm mlx-embeddings

2. 02Embed the folder

   Chunk each document to roughly 500 tokens with a little overlap, embed each chunk, and store the vectors alongside the source text.

3. 03Answer with citations

   Retrieve the top handful of chunks, paste them into the prompt above the question, and ask the model to quote the source. Keep retrieved context small — it is the part that inflates your KV cache.

Teach a small model your tone, format or domain vocabulary using low-rank adapters, on the Mac you already own.

1. 01Prepare the data

   A JSONL file per split (train.jsonl, valid.jsonl) with one {"text": ...} or chat-format record per line. A few hundred good examples beat tens of thousands of noisy ones.

2. 02Train

   LoRA only trains a small set of extra matrices, so memory stays close to inference levels.

   mlx_lm.lora --model mlx-community/Qwen3-4B-Instruct-2507-4bit --train --data ./data --iters 600 --batch-size 1

3. 03Test, then fuse

   Run with the adapter first. When you are happy, fuse it into a standalone model you can serve.

   mlx_lm.fuse --model mlx-community/Qwen3-4B-Instruct-2507-4bit --adapter-path ./adapters

When nobody has published the quantization you need, convert the original weights locally.

1. 01Check you have the room

   Conversion loads the full-precision weights, so you need roughly 2 bytes per parameter free before the quantized copy is written.

2. 02Convert

   Group size 64 with 4 bits is the common default; 6 or 8 bits costs more memory and loses less.

   mlx_lm.convert --hf-path Qwen/Qwen3-8B -q --q-bits 4 --q-group-size 64 --mlx-path ./qwen3-8b-4bit

3. 03Sanity-check the result

   Ask it something you know the answer to, and something long. Broken quantizations usually reveal themselves as repetition or nonsense within a paragraph.

Run Whisper or Parakeet under MLX to turn recordings into text without uploading anything.

1. 01Install

   mlx-whisper wraps the Whisper family; Parakeet models are faster for English.

   pip install mlx-whisper

2. 02Transcribe

   Point it at any file ffmpeg can read.

   mlx_whisper ./interview.m4a --model mlx-community/whisper-large-v3-turbo

3. 03Batch a folder

   Loop over files in a shell script. Transcription is compute-bound rather than bandwidth-bound, so a Max or Ultra chip helps more here than it does for chat.