Setting Up Local AI Inference on a Windows Laptop
I wanted to experiment with running large language models locally without relying on paid APIs or cloud GPUs.
I had two machines:
- A Windows laptop with only 8GB usable RAM
- My Mac, which I use as my primary development machine
The idea was simple: use the Windows machine purely for inference, and treat everything else as separate layers.
This article is a step-by-step log of how I set it up.
Step 1: Pulling and testing local models on Windows
I started by installing Ollama on the Windows laptop and testing which models were realistically usable with the available memory.
Given the RAM constraint, larger models were unstable, so I focused on smaller variants that could load and unload reliably.
The models I ended up working with:
- DeepSeek-R1 (1.5B)
- Llama 3.2 (3B)
- Qwen 2.5 (1.5B)
I pulled these models locally and verified that each of them could run end-to-end without freezing the system.
Step 2: Comparing model outputs directly in the terminal
Before adding any APIs or networking, I wanted to see how each model behaved in isolation.
I ran prompts directly in the terminal for each model to:
- Verify output quality
- Observe latency differences
- Check memory usage during inference
This helped set expectations early and made it clear which models were better suited for different types of prompts.
Step 3: Wrapping Ollama with a small inference API
Instead of calling Ollama directly from other machines, I added a small inference service on the Windows laptop.
This service:
- Exposes a /models endpoint
- Accepts inference requests
- Forwards them to Ollama
- Returns back the list of models available
At this point, the Windows laptop effectively became a dedicated model worker.
Step 4: Connecting the Windows and Mac machines securely
The next step was making the Windows inference service reachable from my Mac.
I set up private networking via Tailscale so both machines could communicate as if they were on the same local network, without exposing anything publicly.
Once this was configured, I could call the Windows inference API directly from the Mac.
Step 5: Adding a backend control layer on the Mac
On the Mac, I introduced a backend that acts as a control layer between the frontend and the Windows worker.
Its responsibilities are intentionally minimal:
- Receive requests from the frontend
- Decide which model to use
- Forward the request to the Windows inference API
- Return the response
This backend also enforces a single active inference request at a time, which is important given the memory limits on the Windows machine.
Step 6: Building a minimal frontend to interact with the system
Finally, I added a small frontend to interact with the setup more easily.
The UI allows me to:
- Enter prompts
- Select a model or task
- Send requests
- View responses and latency
There’s no focus on polish here. It’s meant to be a lightweight playground for testing and observing system behavior.Current state of the setup
At this point:
- All inference runs locally on the Windows laptop
- No paid APIs are used
- Only one model is loaded at a time
- The Mac handles routing and orchestration
- The system remains stable within the given memory limits
That was the intended outcome.
Current state of the setup
At this point:
- All inference runs locally on the Windows laptop
- No paid APIs are used
- Only one model is loaded at a time
- The Mac handles routing and orchestration
- The system remains stable within the given memory limits
That was the intended outcome. Yes, I know it isn't much of a big deal for the ones who are deep in the AI world and have already played around with models, this is just me trying something for the first time when I was bored.
This setup is now something I can iterate on slowly, test different ideas against, or extend in small ways, probably even do my own benchmarks on models.
Sharing this here mostly as a build log, in case the process is useful to others experimenting with local inference.