harshaAI / Multimodal prototype
Vision AI
A FastAPI and Next.js document-chat prototype with layered PDF extraction, OCR fallback, local embeddings, NumPy similarity search, and persisted source metadata.
View repository01 / COMPONENTS & DATA FLOW
Inside the system.
- 01Upload + extraction
- 02Text + OCR fallback
- 03MiniLM + NumPy index
- 04Context → chat
Extraction fallbacks
The backend tries multiple PDF extraction paths and can use pdf2image with pytesseract for scanned pages. The same service accepts document uploads and exposes audio transcription.
Local similarity search
The preferred embedding model is all-MiniLM-L6-v2. Document vectors live in NumPy arrays; retrieval computes cosine similarity and selects up to five matching chunks.
File-based persistence
Chunks and metadata are saved as JSON, while vector arrays are saved as compressed NPZ files. Source records associate retrieved chunks with document IDs and filenames.
02 / IMPLEMENTATION DECISIONS
Why it works this way.
A transparent index
File-based NumPy storage makes the prototype easy to inspect without a vector database. Search scans the stored vectors rather than using an approximate-nearest-neighbor service.
Fallback quality must be visible
The current code returns random vectors if the local embedding model is unavailable or embedding generation fails. That keeps the path executable but does not preserve meaningful semantic retrieval.
03 / SCOPE & TRADEOFFS
The implementation boundary.
A multimodal prototype with a clear improvement boundary: embedding failures should be surfaced explicitly before the retrieval result is trusted.
04 / READ THE IMPLEMENTATION