harshaData / Data CLI
FBI Data Pipeline
A small, explicit ETL command that reads FBI Wanted API pages or local JSON and normalizes records into a thorn-delimited text format.
View repository01 / COMPONENTS & DATA FLOW
Inside the system.
- 01API page / JSON file
- 02Record normalization
- 03Multi-value fields
- 04Delimited output
Two input adapters
The command accepts either --page for the FBI Wanted API or --file for a UTF-8 JSON file. HTTP errors propagate through requests.raise_for_status.
A normalization function
process_data accepts an items envelope or a list of records. It normalizes subjects and field_offices into lists, handles missing values, and joins non-empty values.
Portable text output
Each output line contains title, subjects, and field offices separated by the thorn character þ. The CLI writes the transformed records to standard output.
02 / IMPLEMENTATION DECISIONS
Why it works this way.
Keep transformation independent
API access and file reading both feed the same normalization function. That makes processing behavior testable without a live HTTP request.
Be precise about the dataset
The source is the FBI Wanted list API. This project is not a general FBI crime-statistics warehouse or a scheduled multi-source pipeline.
03 / SCOPE & TRADEOFFS
The implementation boundary.
A focused data-conversion project with tests for fetching and transformation. Pagination orchestration, retries, scheduling, and warehouse loading are outside the implemented CLI.
04 / READ THE IMPLEMENTATION