Tag: self-hosted

"Sparky" runs Gemma 4 E4B entirely on Jetson Orin NX with 30+ sensors, no connectivity. Achieves ~200ms cached TTFT and 14-15 tok/s with SenseVoiceSmall STT, Piper TTS, and native vision/OCR. Demonstrates practical offline AI robotics with aggressive system prompt engineering and sensor integration.

Impressive build log: 16 DGX Sparks on fabric all hitting line rate. Setup was time-consuming but smoother than expected with Ubuntu pre-installed. Detailed notes on configuration of passwordless SSH, jumbo frames, and fabric networking. Represents serious investment in local inference infrastructure.

A self-funded IT infrastructure professional built a local LLM cluster using 4 Mac Mini systems over 2 months. While light on technical details in the main post, the project demonstrates the growing accessibility of serious local AI infrastructure for individual developers willing to invest in hardware, representing a trend toward democratized AI compute.

A detailed technical write-up on converting a Xiaomi 12 Pro smartphone into a dedicated local AI inference node: LineageOS flashed for minimal overhead, Android framework frozen, headless networking via custom-compiled wpa_supplicant, and custom thermal management daemons. Running Gemma4 via Ollama on ~9GB of freed RAM. This is a creative and replicable approach to always-on local AI that doesn't require dedicated server hardware.

A hardware upgrade post (2015-era machine to a new high-end GPU) paired with plans for a local-first AI project. Low informational density but notable as a community signal: mainstream engineers who previously wouldn't consider local AI are now investing serious hardware budgets in it. The comment thread likely contains useful configuration advice.

A detailed parts list and build log for a dual RTX PRO 6000 workstation: Threadripper PRO 7965WX, WRX90 motherboard, 256GB ECC DDR5, dual 10GbE, IPMI. This represents the high end of consumer/prosumer local AI infrastructure. Useful as a reference for anyone designing a serious multi-GPU inference node, and as a data point on what serious local AI investment looks like in 2026.

A community thread inviting members to share their most unconventional home inference setups — featuring oven grills, egg cartons, and improvised cooling solutions. Low-information but high-character. A reminder that local AI is a hands-on, tinkerer culture, and sometimes the best insight comes from how people are actually running things.

Developer built Phantom, an open-source system giving Claude its own persistent VM with vector memory, self-evolution engine, and MCP server. It runs continuously via Slack integration, maintains context across sessions, and autonomously evolves its capabilities. The project demonstrates what happens when AI agents get persistent infrastructure rather than ephemeral sessions.

A stark cost comparison between cloud-based AI agents and local deployments. Running OpenClaw 24/7 with Opus costs ~$300/day ($110k/year), while the author's setup with 3 Mac Studios and a DGX Spark running local models cost one-third of that yearly cost upfront — usable for years with complete privacy. Makes a compelling economic and privacy case for local AI infrastructure.

Onyx is a self-hostable AI chat platform supporting any LLM, with built-in support for custom agents, knowledge source connections, and hybrid search/retrieval workflows. This is squarely in the intersection of self-hosted AI and RAG interests — a production-grade platform, not a toy demo.

Developer running Kimi K2.5 (600GB), MiniMax 2.5 (120GB), Qwen 3.5 (220GB), and GOT OSS 120B Heretic (60GB) across 3 Mac Studios with 512GB RAM each using EXO labs for distributed inference. This demonstrates that frontier-class models are now accessible for completely private, self-hosted deployment at reasonable hardware costs. Running 4 OpenClaws instances enables 24/7 coding, writing, and research workflows without cloud dependencies or rate limits.

A detailed account of building a $38K 6-GPU local AI workstation running three open models concurrently for internal business analytics and automation. Rare real-world documentation of what a serious on-premise AI infrastructure deployment looks like, including hardware specifics and lessons learned. With 94 comments, the thread drew genuine architectural discussion useful for anyone planning self-hosted AI at scale.

An impressive self-hosted GPU cluster featuring 12 RTX 5090s (1.5TB+ VRAM total) across 6 machines running Kubernetes with GPU scheduling. Built for AI/LLM inference, training, image/video generation, and self-hosted APIs—a glimpse into serious local AI infrastructure.

A detailed build log for a 4x AMD R9700 system (128GB VRAM) funded through a 50% digitalization subsidy in Germany. Built to run 120B+ models locally for data privacy, with comprehensive benchmarks and real-world performance data for local LLM deployment.

A practical guide to running Claude Code sessions remotely via phone using Tailscale (VPN) + tmux. The setup enables terminal access to home MacBook sessions from anywhere, demonstrating creative mobile workflows for AI-assisted development.

A sequel build featuring 4x R9700 GPUs (128GB VRAM total) optimized for local LLM deployment. The post includes detailed upgrade path from previous MI100 setup, performance benchmarks, and lessons learned—valuable for anyone planning serious local AI infrastructure.

A detailed perspective on the shift from cloud to local AI, citing rising subscription costs and over-tuning/censorship as primary motivations. After weeks testing Llama 3.3, Phi-4, and DeepSeek locally, the author argues 2026 marks the inflection point for local AI viability.

A unique mobile AI workstation in a Thermaltake Core W200 case featuring 10 GPUs (8× 3090 + 2× 5090 = 768GB VRAM), Threadripper Pro 3995WX, and 512GB DDR4. Built for extra-large MoE models and video generation at ~$17k total cost with full enclosure and portability.

A fun comparison post from someone with both maxed M3 Ultra (512GB) and ASUS GB10 in the same room, asking the community for 24-hour experiment ideas. The discussion explores practical use cases and benchmarks for high-end local AI hardware.