Running Local AI on a VPS: llama3.2, Ollama, and the Cloud Fallback Strategy
Running AI tools on a personal portal usually means one of two things: paying for expensive cloud API calls on every request, or buying a beefy GPU server. TechnoPKG takes a third route — a local LLM on a modest VPS, with a cloud fallback only when needed.
The Setup
The portal runs on a Hostinger KVM2 VPS (4 vCPUs, 8GB RAM, Ubuntu 24.04). Ollama serves a quantized Llama 3.2 1B model locally, handling the majority of AI requests without any external API call. This covers: document summaries, SQL explanations, ERP planning notes, job match analysis, and most of the AI tools in the portal.
For heavier tasks — long documents, complex reasoning, multi-step agentic workflows — the portal falls back to a cloud provider. The fallback is transparent to the user; the interface shows which provider handled the response.
Why Llama 3.2 1B?
The 1B parameter model fits comfortably in 2–3GB of RAM, leaving headroom for the web server, database, and Node.js resume generator running alongside it. On an 8GB VPS, this balance matters. The model handles everyday language tasks accurately enough for a learning portal. For production enterprise use, a larger model would be appropriate — but for learning and demonstration, it performs well.
Keeping the Model Warm
One practical challenge with local models is cold start time. When Ollama unloads a model from memory (typically after 5 minutes of inactivity), the next request can wait 10–30 seconds while the model reloads. TechnoPKG solves this two ways:
- A cron job pings the model every 5 minutes with a lightweight prompt to keep it loaded
- The
keep_aliveparameter is set to 600 seconds on every API call, explicitly telling Ollama to hold the model in memory
Document Size Limits
Large documents (50,000+ words) present a real challenge for a 1B model with a limited context window. TechnoPKG handles this by applying a smart truncation strategy: the first 60% of the document content (typically introduction, key sections) plus the last 40% (conclusions, recommendations). This captures the most important content from most professional documents while staying within the model's effective context.
The Cloud Fallback
The portal is configured to use a cloud provider as a fallback when:
- The local model times out after 3 minutes
- The user explicitly selects a cloud provider
- The task requires a larger context window than the local model supports
Cost Comparison
Running Ollama locally on a VPS that you already pay for brings the marginal cost of AI requests to essentially zero. The VPS costs approximately $10–15/month. A comparable volume of cloud API calls for a learning portal with moderate traffic would cost $20–50/month or more. For a non-commercial personal portal, the local-first approach makes strong economic sense.
Lessons Learned
The biggest surprise was how capable a 1B quantized model is for structured tasks. SQL explanation, document key-point extraction, and cover-letter drafting all work well. Open-ended creative writing and complex multi-step reasoning are where the limitations show — those tasks benefit most from the cloud fallback.
The second lesson: keep-alive configuration matters as much as the model choice. A model that responds in 2 seconds feels completely different from one that takes 25 seconds to warm up, even if the output quality is identical.
This portal is for learning and experimentation only. AI outputs should always be reviewed before use. Not professional advice.
Comments (0)