What it is
A self-hosted inference node on a single machine: a small model with a LoRA adapter behind an API, with rate limiting, a reverse proxy and a metrics dashboard.
How it works
Single-machine inference node with rate limiting and metrics
google/flan-t5-base is loaded in 8-bit with a LoRA adapter and served by FastAPI, with PostgreSQL 16 over asyncpg for persistence. Each client is rate-limited per IP through Redis, using MULTI/EXEC transactions. Traefik sits in front, a cloudflared tunnel exposes it, and Prometheus scrapes metrics into a provisioned Grafana dashboard. The whole stack runs under Docker Compose.