High-performance AI inference platform with intelligent model routing — classifies query complexity via embeddings and heuristics, routing simple queries to fast/cheap models and complex ones to GPT-4 class models.
Multi-level caching: L1 in-memory (Moka) + L2 distributed (Redis) with semantic similarity matching via cosine similarity on embeddings.
Full observability stack with Prometheus metrics and pre-configured Grafana dashboards for inference latency, cache hit rates, and model health.
Fully async Rust architecture with horizontal scaling via Docker and Kubernetes.