Data Flow
This document traces how data moves through the entire system, from time tracking in Jira to final payment.
End-to-End Flow
Data Sources
Jira + Tempo → ClickHouse
n8n workflows run on a schedule (~15 min) and sync:
| Source | Destination | Data |
|---|---|---|
| Tempo API | tempo.worklogs_flat | Time entries with duration, user, date, project |
| Jira API | jira.jira_issues | Issue metadata (type, priority, summary) |
| CDM pipeline | cdm.jira_applications | Client deals with amounts, rates, limits |
| CDM pipeline | cdm.jira_customers | Company details, TIN, SWIFT, signers |
| CDM pipeline | cdm.jira_contacts | Client contact persons |
ClickHouse → Prefill → PostgreSQL
The prefill pipeline reads from ClickHouse and produces structured data:
ClickHouse (worklogs + issues + application + customer)
↓ categorize, apply min billable, calculate financials
Prefill Result (preview, not saved)
↓ user confirms
PostgreSQL (invoice + line_items + events)
PostgreSQL → PDF → External Systems
After finalization:
PostgreSQL (invoice + line_items)
↓ render HTML template
Gotenberg (HTML → PDF)
↓ save PDF
PostgreSQL (pdf_data column)
↓ on approval
PlanFact (accounting sync)
Cloudflare R2 (PDF storage)
Didox.uz / Email (delivery)
Data Ownership
| Data | Owner | Storage | Mutability |
|---|---|---|---|
| Worklogs | Tempo | ClickHouse | Read-only (synced by n8n) |
| Jira issues | Jira | ClickHouse | Read-only (synced by n8n) |
| Client info | CDM | ClickHouse | Read-only (synced by CDM pipeline) |
| Invoices | Invoice app | PostgreSQL | Read-write |
| Line items | Invoice app | PostgreSQL | Read-write |
| Events/history | Invoice app | PostgreSQL | Append-only |
| Rate rules | Invoice app | PostgreSQL | Read-write |
| PDFs | Invoice app | PostgreSQL (Phase 1), R2 (Phase 3) | Immutable after generation |
| App settings | Invoice app | PostgreSQL | Read-write |
Timing Considerations
| Operation | Typical Duration | Timeout |
|---|---|---|
| ClickHouse query | 100-500ms | 15s |
| PostgreSQL query | 5-50ms | 5s |
| PDF generation | 1-5s | 30s |
| PlanFact API call | 200-1000ms | 10s |
| n8n data sync | Every ~15 min | N/A |
The main latency concern is data freshness: worklogs logged in Tempo may take up to 15 minutes to appear in ClickHouse. The prefill pipeline always reads the latest available data.