INFRASTRUCTURE PLATFORM ENGINEER IV · UF IT
I’m Sai Sri Harsha, an Infrastructure Platform Engineer IV at UF IT. I maintain a LiteLLM gateway on a shared 9B+ tokens/day platform, and build FastAPI services, Redis/Celery streaming workflows, and retrieval systems.
Backend engineering. AI infrastructure.
Thoughtful interfaces.
Shared AI platform behind the LiteLLM gateway I maintain and extend
Real-time token delivery across 12+ generative AI workflows
Production services deployed and operated on Kubernetes
ENGINEERING / CONTROL PLANE TO CLIENT
IDENTITY + ROUTING
API-key authentication, RBAC and team isolation define access. Provider routing and failover, team budgets, and audit logging keep shared usage accountable.
9B+ tokens/day on the shared platformWORKERS + EVENT DELIVERY
Background execution and event delivery are separate from the browser-facing stream. My work redesigned this path for concurrent token delivery across generative AI workflows.
2,000+ concurrent streams · 12+ workflowsEVIDENCE + REGRESSION CHECKS
Document extraction and metadata-filtered retrieval feed answers with source attribution. Evaluation checks relevance, groundedness, tool calls, schema conformance, latency, and provider failures.
Excel/PDF analysis · PostgreSQL/pgvector01 / SELECTED WORK
Three implementations to inspect: layered command-risk analysis, local browser Q&A, and a persistent document-retrieval pipeline.
Command-line risk analysis
A Python command-review CLI combining shell rules, classifier risk, BM25-ranked command history, and LiteLLM into allow, warn, or block decisions.
Local browser Q&A
A TypeScript extension that extracts DOM text and tables, runs Tesseract.js OCR, caches page context in IndexedDB, and answers with extractive sentence scoring.
Document retrieval & reasoning
A Streamlit assistant with 1,000-character document chunks, MiniLM embeddings, persistent Chroma cosine search, selected-document retrieval, and optional web-search augmentation.
02 / A LITTLE MORE HUMAN
My work sits at the intersection of backend architecture, AI infrastructure, and the people who use them.
I’m based in Gainesville, Florida, with an M.S. in Computer Science from the University of Florida. Before UF IT, I built enterprise services at TCS in Hyderabad.
At UF IT, I maintain and extend the LiteLLM gateway on a shared platform processing 9B+ tokens a day. My work also spans NaviGator Prompt Builder, document retrieval, model evaluation, and a departmental Data Science Agent.
03 / HOW THE WORK CONNECTS
Three layers.
One complete experience.
Execution & delivery
FastAPI request handling, Celery workers, Redis Pub/Sub events, and SSE clients. Separating background execution from token delivery.
FastAPI / Celery / Redis / SSEEvidence & evaluation
OCR, chunking, metadata filters, hybrid search, and reranking. Regression checks for groundedness, tool calls, and output schemas.
BM25 / embeddings / pgvector / evaluationProduct & operations
Next.js and TypeScript products backed by services deployed with Kubernetes, Terraform, F5 load balancing, and GitLab CI/CD.
Next.js / Kubernetes / Terraform / GitLab04 / NOTES & THINKING
Projects show what I build. The blog is where I share the ideas, discoveries, and lessons along the way.
Read the journalA FEW THINGS I BELIEVE IN
Reliability over cleverness. Simple systems scale better. Infrastructure as code, and documentation as a first-class feature. Principles that keep the work grounded when the technology gets complicated.
harsha