The Cloud Tax No One Talks About
Renting inference from the big providers feels convenient until the invoices start landing. Every token you generate costs money and the meter never stops running. You also give up visibility into exactly which model version is answering and whether your prompts are being used to train future versions. That dependency creates real risk when you are building something that needs to stay online and under your control. Rate limits appear the moment you try to scale beyond a prototype and price changes can break your margins overnight. I watched my own costs climb every time I added another agent or longer context windows. The lack of ownership also means you can lose access if the provider decides your use case violates some new policy.
Running Models Locally with MLX
I use Apple's MLX framework because it is optimized for the silicon I already own and it supports fast quantization. I download open weights from Hugging Face, run the conversion script to quantize them to 4-bit or 8-bit, and then load them with a simple Python server using mlx_lm. The CyberPower GPU box handles the larger models that need more memory while the Mac handles lighter real-time work. The whole process stays on hardware I physically control and I can swap models or add LoRAs without asking anyone for permission. Monitoring is just basic logging and a small dashboard I wrote myself. When a new model drops I test it the same day instead of waiting for an API provider to roll it out. Quantization keeps the speed high even on consumer hardware so I do not need a data center rack to get useful performance.
Zero Marginal Cost with Cloudflare Tunnels
Cloudflare tunnels let me publish the local inference endpoint to the public internet without opening inbound ports or dealing with dynamic DNS. The tunnel binary runs as a lightweight daemon and handles authentication through my Cloudflare account. I get TLS termination, DDoS protection, and a clean domain name for free. Once the tunnel is up the only ongoing cost is electricity and the occasional hardware upgrade. Inference volume can double or triple and my bill stays the same because there is no per-token charge from any third party. I own the uptime too because no one else can shut down my instance for policy reasons. The setup has survived multiple model swaps and even a full OS reinstall without losing the public endpoint. This is the difference between renting a service and owning the actual machine that does the work.
FAQ
What hardware do I need to start?
A recent Mac with Apple silicon works for many models or a small dedicated GPU box if you want to run larger ones without swapping.
How hard is quantization?
MLX and the supporting tools make it a one-command process after you pick the model size that fits your VRAM.
Does this replace cloud APIs completely?
For most internal tools and agents yes. For massive parallel workloads you might still burst to cloud but the baseline stays local and free.