thushan/olla: High-performance lightweight proxy and load balancer for LLM infrastructure. Intelligent routing, automatic failover and unified model discovery across local and remote inference backend

thushan/olla: High-performance lightweight proxy and load balancer for LLM infrastructure. Intelligent routing, automatic failover and unified model discovery across local and remote inference backend might be the better choice following the LiteLLM debacle (hacked supply chain with data extractor in the package). For me, it's definitely interesting because I simply want to run two models and make them available under a single endpoint, and all the other packages are significantly overkill for that.