harshaSystems / Analytics prototype
AppTrack
A multi-language event collection system: HTTP and UDP ingestion in Go, Kafka transport, and a ClickHouse consumer with column-oriented event storage.
View repository01 / COMPONENTS & DATA FLOW
Inside the system.
- 01JS / Go / Python SDK
- 02Go collector
- 03Kafka topic
- 04ClickHouse events
Two ingestion transports
The collector accepts JSON at POST /api/v1/event and also listens for UDP datagrams. Missing event IDs and timestamps are filled with UUIDs and Unix milliseconds.
Kafka transport
Events are serialized into the uamt-events topic through kafka-go. The consumer reads that topic with the clickhouse-consumer group and maps records into an events insert.
Analytical storage
The ClickHouse events table uses MergeTree ordered by timestamp and event_type. A separate users table uses ReplacingMergeTree with updated_at as its version column.
02 / IMPLEMENTATION DECISIONS
Why it works this way.
Fast acknowledgment is not durable acknowledgment
The HTTP collector responds 202 Accepted after launching a goroutine to publish the event. A later Kafka write failure is logged, so the acknowledgment does not prove durable delivery.
Storage is explicit and inspectable
The schema separates event identity, session, event type, JSON metadata, and system attributes. The current consumer executes one insert per event rather than batching writes.
03 / SCOPE & TRADEOFFS
The implementation boundary.
The repositories establish an ingestion architecture. Durable acknowledgments, insertion batching, and replay/deduplication guarantees would require further work before claiming production-grade delivery.
04 / READ THE IMPLEMENTATION