← Home

2026 · work

Independent Intelligence: Air-gapped Workplace AILibreChat · LiteLLM gateway hooks · vLLM / Ollama · Jetson

Pre-launch QA, gateway fixes, a full server rebuild and operations for an on-premise AI appliance

Role
QA · gateway development · server rebuild and operations
Period
2026.07 – present
Status
Live · operating
Stack
Python, LiteLLM, LibreChat, vLLM, Ollama, Docker, Linux
Independent Intelligence running on this Mac: LibreChat whitelabel → LiteLLM gateway hooks → Ollama qwen3.5 9B. Answers in Korean with no outbound call. 2026-09-07
Model cards. Auto, General, Fast, Reasoning and Agent are role aliases; the gateway hook analyzes the request and routes it to a real model
First screen after login. Brand, labels and notices come from a build script that rewrites the LibreChat bundle
Sales landing page. Quote engine, AI chat and lead intake served from the same box

The product

Independent Intelligence is a workplace AI appliance that runs entirely inside a closed network. Four layers: chat (a LibreChat whitelabel) → gateway (LiteLLM plus hooks: intent routing, attachment text extraction, data classification, search grounding, PII masking, honesty rules) → models (vLLM or Ollama, all local) → Rust daemons (license, skill registry, approvals). On top sit an HWP/PDF document service, MCP tools that create Word, Excel, PowerPoint and PDF files and analyze spreadsheets and CAD drawings, web search through a single egress gateway, and fact checking (HalluGuard). A request containing a resident ID, card, phone, email or API key is classified at the top level and pinned to a local model; if the classifier itself fails, the request degrades to local rather than leaking (fail-closed).

Screens from a local run (2026-09-07)

I built the LibreChat whitelabel image with the repository’s build script, brought up the compose stack (LibreChat, MongoDB, LiteLLM, PostgreSQL) and pointed the gateway’s local model aliases at Ollama qwen3.5 9B on this Mac. The captures above are that instance after creating an account and asking a question in Korean. The production box adds vLLM 9B with a 32k context, GPU embeddings, Whisper and the document service on the same structure.

I also pasted a counseling memo containing a phone number and a resident ID. The answer dropped both values, but a small local model will also add sentences that were not in the source. That is why a plausible output does not pass; it is checked against the source. This is exactly the class of defect the pre-launch QA kept catching.

Pre-launch QA (2026.07)

  • Built a QA sheet over eight quality attributes and tagged defects daily. Round-one core defects: 500s on file creation and code execution, internal prompt text leaking on screen, general-knowledge hallucination. Prompt injection, PII, harmful-content blocking and CJK obfuscation resistance held up.
  • Learned the limits of black-box testing. Chat output alone cannot pass mechanism criteria such as “did RAG actually retrieve” or “was an audit log written”; without server logs those items were marked unverified. The fact-checking criterion was per-sentence citation, not content fidelity, and I corrected my own early verdict.
  • The thinking panel was intentional design that I mistook for a bug, causing rework. Since then, docs, comments and checklists come before removing or changing a feature.

Gateway and tool fixes (2026.07 – 08)

  • Thinking-text leaks. Thoughts that started with <think> were rendered as the answer body. A streaming-hook splitter now moves everything up to </think> into reasoning; a follow-up bug that truncated the tail of the thought was fixed by flushing at the moment the thought ends.
  • Large documents. A 16-page document overflowed the context and showed an English stack trace to the user. That became a Korean notice, and instead of truncating to the first part, an in-request RAG (document chunks → bge-m3 embeddings → cosine top-k) was added. Moving embeddings to the GPU cut the first turn from 20 s to 1.2 s, and along the way I found a latent bug where chunks larger than the embedding server’s physical batch were silently rejected.
  • Korean PDFs. PDFs with CID or Type3 fonts, such as Hancom exports, extracted as zero characters. The document service gained a poppler path with OCR fallback for scans; a question about the glossary on page 33 of a real 39-page manual was answered correctly.
  • Document tools. Running the same PowerPoint request 100 times on the real path showed the model wrapping the array argument in a string every time: a 0 percent first-call success rate. Tolerant schemas plus server-side recovery took it to 99 percent. Two facts prevented misdiagnosis: validation happens in LangChain, not the MCP server, and the chat container must restart after a schema change.
  • Web search. With search on, the model was still simulating results from training knowledge. Fixes at three levels (hook rules, config, environment) made real searches run, and turning search off now also stops the query from leaving. Video chapter lists in results were being copied as a school’s philosophy; labeling them cut the mention rate from 30 to 12 percent (A/B, 50 runs each).
  • Input-boundary security. I swept the attack surface arriving through attachments and messages and closed the paths that could stop or pin a gateway worker, along with indirect prompt injection planted in attached documents. Legitimate large files still pass. The individual defects are product security matters and are not written up here.
  • Also. The memory feature had been silently dead because a tool gate stripped its tools; CAD uploads vanished on macOS Chrome because of a MIME branch; tool and search blocks stayed in English in the Korean UI; external-model keys in .env never reached the container; users are now told when an external model is used.

Server rebuild (2026.08)

  • The old demo server died in early August. On a Jetson Thor box on a university campus I rebuilt twelve containers (chat, gateway, databases, search, Whisper and more) and six host units (license, registry and approval daemons, document service, MCP, tunnel), all enabled to start on boot.
  • The model is a 9B BF16 with a 32k context on vLLM. I measured the KV cache to set the context ceiling and documented the three settings that must share the value.
  • Bugs the repository alone would not reveal, fixed and sent as PRs: a Whisper image missing ninja, which made CMake report “no compiler” in a cascade; local TypeScript packages in the document service that never built; unit files with the previous box’s home path baked in; a gateway master key literal that mismatched and left the chat loading forever.
  • Two domains through a Cloudflare tunnel, live on August 13. The box is shared with another product demo, so the do-not-touch boundary is documented, and deployment moved to git so nothing is copied onto the box by hand.

How I verify

  • The same request, 100 times, reported as a success rate: document tools 0% → 99%, zero fabricated addresses in search grounding. The August 13 fabrication incident did not reproduce in 200 runs and was classified as a tail event.
  • Log order is not causation. I first blamed the router guard for the dead memory feature; that line was the consequence of tools being stripped earlier.
  • “Search did not run” is checked against the search container’s timestamps, not the attachments array.