Many recent phones can answer prompts from a small local model, but the experience depends on a few concepts that are easy to skip over. This guide explains how to run an LLM on Android, what the file formats and compression terms mean, which parts keep working with no internet, and how to start the on-device server in Ollama Local AI so that other tools on your network can use the same model.
What it means to run an LLM on Android
A large language model is a big set of numeric weights plus code that turns your prompt into text, one token at a time. Running it on Android means the weights are stored on the phone and the arithmetic happens on the phone's own processor, instead of in a data center. A prompt can be answered without leaving the device.
The trade-off is hardware. A phone has much less memory and sustained compute than a workstation or a cloud GPU, so on-device models tend to be smaller and compressed. Most of the choices below are about working within that limit.
GGUF and llama.cpp in plain terms
GGUF is a single-file container for model weights. Besides the numbers, it holds the tokenizer and metadata, so a runtime can load one file and know how to use it. Models published for local use are very often distributed as GGUF.
llama.cpp is an open-source inference engine written in C and C++. It reads GGUF files and runs them efficiently on ordinary hardware, including phone processors. Ollama Local AI embeds llama.cpp, which is how it loads a quantized GGUF model and runs it directly on the phone CPU or GPU with no desktop software involved.
Quantization: smaller files in exchange for some precision
Model weights are usually trained as 16-bit or 32-bit floating-point numbers. Quantization re-encodes them with fewer bits per weight. The file gets smaller, less memory is needed to load it, and inference is often quicker. The price is a small loss of accuracy that grows as you compress harder.
GGUF file names usually include a label such as Q4_K_M or Q8_0. As a rule of thumb, the number after the Q is roughly the bits per weight. A Q8 file keeps more precision and takes more space than a Q4 file, and very low labels such as Q2 shrink the file further but tend to hurt quality noticeably. The trailing letters describe variants of the scheme.
- Start with a mid-range quantization, commonly a Q4 variant, and move up only if answers are poor and the phone has memory to spare.
- If a model fails to load or the app is closed by Android while loading, the model is probably too large for the memory available. Choose a smaller model or a lower quantization.
- A smaller model at a gentle quantization and a larger model at a harsh one are both reasonable. Try each on your own prompts instead of assuming.
- Close other heavy apps before loading a model, since they compete for the same memory.
Choosing a model family and size
The app listing names Llama 3, Mistral, Phi, Gemma and Qwen as supported model families. Each family publishes several parameter sizes, and the small end of that range is the part that suits a phone. Read each model's card for its license, intended use and context length before you download it.
| Goal | What to look for |
|---|---|
| General chat | An instruction-tuned ("instruct" or "chat") variant rather than a base model. |
| Code help from an editor | A code-oriented model or a capable general model, plus a context window long enough for your files. |
| Low memory | Fewer parameters and a lower quantization level. |
| Long inputs | A larger context window, which also increases memory use while the model runs. |
Start the on-device server
The app can do more than chat. It exposes the loaded model as an OpenAI-compatible API on your local Wi-Fi, so editors and scripts can call the phone like any other endpoint. You can install it from Google Play; it requires Android 8.0 or newer.
- Open the app and pick a local GGUF model. Check the app for the models available and how to add one.
- Tap Start Server.
- Note the endpoint the app shows. It looks like
http://192.168.1.50:8080/v1, but that address is only an example. Yours will differ. - Make sure the computer or tool you want to connect is on the same Wi-Fi network as the phone.
The server exposes /v1/chat/completions, /v1/models and /health to devices on the same network.
Test the endpoint from a computer
Before configuring any editor, confirm the phone answers. Replace the placeholders with the address and port shown in the app. First check that the server is up, then ask it which models it offers. The model id in that response is the value you send as model.
curl http://<PHONE_IP>:<PORT>/health
curl http://<PHONE_IP>:<PORT>/v1/models
curl http://<PHONE_IP>:<PORT>/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "<MODEL_ID>",
"messages": [{"role": "user", "content": "Reply with one short sentence."}]
}'Use http, not https, on a local network. Some clients insist on a non-empty API key field even when the endpoint does not need one; a placeholder value such as local-lan is fine for that purpose. Step-by-step editor setups are in the Cursor guide and the VS Code guide.
What works offline and what does not
On-device models work without internet. That is separate from the local network: another device still has to reach the phone, so both need to share a Wi-Fi network or hotspot.
| Task | Needs internet? |
|---|---|
| Chatting with a local GGUF model on the phone | No |
| Serving a local model to a laptop on the same network | No, but a shared local network is required |
| Getting a model file onto the phone for the first time | Usually yes, unless you copy the file over yourself |
| Routing to a cloud provider such as OpenAI, Claude, NVIDIA NIM, Hugging Face, OpenRouter or Gemini | Yes |
| Using a self-hosted Ollama or llama.cpp server on your LAN | No internet, but that machine must be reachable |
When a cloud provider is the better choice
A phone-sized model is a good fit for short questions, drafting, summaries and quick code explanations. It is a weaker fit when you need a model larger than the phone can hold, a very long context window, or high throughput on an older device. In those cases a hosted model is more practical.
Ollama Local AI is a gateway, so you do not have to choose one side permanently. In the app, Add Provider has a Cloud tab (NVIDIA, OpenAI, OpenRouter, Gemini, Claude, Ollama and Custom) and a Local tab. Your tools keep pointing at the same phone endpoint while you switch what sits behind it. Failover between providers and the Auto Route option, which can pick a free model, are available too. Check the app for how each is configured.
Privacy: on-device versus cloud
- On-device model: the prompt is processed on the phone and travels only across your local network to the tool that asked for it.
- Cloud provider: the request goes directly from the phone to that provider and is handled under that provider's own terms. There is no relay server run by the app developer in between.
- Stored keys: API keys and provider settings are kept on the device in AES-256 encrypted storage (EncryptedSharedPreferences). The Google Play data safety section lists no data collected, no third-party sharing, and data encrypted in transit.
Keep the endpoint on your private network. Do not forward its port on your router or otherwise expose it to the public internet. Ollama Local AI is an independent app by FreeRouter Team and is not affiliated with Ollama, OpenAI, Anthropic or any other provider mentioned here.
Where to go next
With the server running, you can point an editor at it. The Cursor guide covers that tool step by step, the VS Code guide covers Continue, Cline, Roo Code and Windsurf, and the OpenAI-compatible API guide explains the endpoint itself. You can also return to the home page for an overview of the app.
