harshaData / Course project
Incident PDF Ingestor
A Python CLI that downloads public incident PDFs, reconstructs wrapped records with regular expressions, inserts structured rows into SQLite, and reports grouped incident counts.
View repository01 / COMPONENTS & DATA FLOW
Inside the system.
- 01Incident PDF
- 02PyPDF2 + row parser
- 03SQLite records
- 04GROUP BY nature
PDF-to-text boundary
The download function returns PDF bytes through BytesIO. PyPDF2 extracts page text, while the parser skips report headers and identifies new incident lines by date.
Wrapped-record reconstruction
A regular expression matches time, incident number, location, incident nature, and agency code. Continuation lines are joined to the preceding record for another parse attempt.
Relational output
The incidents table has five text columns. Parameterized executemany inserts parsed rows; a GROUP BY query prints occurrence counts by incident nature.
02 / IMPLEMENTATION DECISIONS
Why it works this way.
A batch pipeline with clear stages
The repository implements download, extraction, storage, and reporting. It does not implement a live event-stream processor despite the repository’s historical name.
Rebuild semantics
Database setup drops and recreates the incidents table. A run replaces the table rather than providing an incremental ingestion or deduplication protocol.
03 / SCOPE & TRADEOFFS
The implementation boundary.
A format-specific incident ingestion project. Its parser relies on the report’s known incident and agency labels and should be reevaluated if the PDF layout changes.
04 / READ THE IMPLEMENTATION