06How it is built
Method and glossary
No dashboard renders a number it computed itself. Everything traces back to one offline pipeline and one pinned artifact set. This page explains that pipeline and defines the terms.
build dfcea33c3556compiled 2026-09-12model 1956349
01
The pipeline
Six stages turn a flat event log into the artifacts this site reads.
- 01
Ingest
The raw event log is read once through DuckDB and written to a column-trimmed Parquet. Nothing loads the whole table into memory.
- 02
Star schema
build_all() derives dim_users, dim_products, fact_sessions, fact_daily_kpis, plus the RFM, affinity, and retention tables.
- 03
Score
Every user with pre-November activity is scored by the frozen LightGBM model. The bulk pass is prediction-identical to per-user scoring.
- 04
Mine affinities
Product pairs bought in the same session are counted, filtered by support and lift, and keyed for lookup in both directions.
- 05
Precompute the grid
The A/B engine runs across a segment by lift by confidence grid so the experiment page can check its live math against fixed points.
- 06
Emit artifacts
Each output is written as byte-deterministic JSON, keyed by commit. The frontend build pins one set and refuses to build without it.
02
The star schema
Table sizes for this build. Every figure elsewhere on the site aggregates one of these.
Row counts
1,645,912 source events
- fact_sessions
- 523,404
- dim_users
- 135,767
- dim_products
- 102,995
- user_rfm_segments
- 13,523
- fact_daily_kpis
- 61
- weekly_retention
- 45
- predictions_product_affinity
- 11
03
Technology
Each choice earns its place by removing a moving part, not adding one.
| Layer | Choice | Why |
|---|---|---|
| Query engine | DuckDB | In-process column store. Runs the pipeline offline and the explorer in the browser from the same SQL. |
| Pipeline | Python, DuckDB SQL, LightGBM | SQL for the joins and aggregations, LightGBM for the purchase model, one script to produce every artifact. |
| Artifacts | Versioned JSON and Parquet | Byte-deterministic output keyed by commit. The frontend build pins one set and fails if it is missing. |
| Frontend | Next.js App Router, static export | Prerendered HTML on a CDN. No server on the request path, so no cold start. |
| Charts | Recharts and hand-built SVG | Recharts for standard series, custom SVG for the cohort grid and funnel where a library would fight the design. |
| In-browser SQL | DuckDB-WASM | Ad-hoc queries over the trimmed event log without shipping the data through a backend. |
04
Glossary
Terms used across the site, with what each one means for this dataset.
Data engineering
DuckDBDuckDB analytics database+
An embedded column-store built for analytical queries. It runs in-process with no server, the way SQLite does for transactional work.
On this data: The build pipeline uses it to turn the raw event log into a star schema. The data explorer runs the same engine compiled to WebAssembly, directly in your browser.
Star schemaDimensional model+
A warehouse layout that splits data into fact tables (measurements, one row per event or per day) and dimension tables (attributes of a user or product). Queries join a fact to a few dimensions.
On this data: dim_users and dim_products hold attributes; fact_sessions and fact_daily_kpis hold the measurements. Row counts are on the method page.
SessionizationSession construction+
Grouping a user's sequential events into visits. Here a visit is one server-assigned session id.
On this data: The event log collapses into fact_sessions, one row per visit, carrying duration, event count, and whether the visit ended in a purchase.
Segmentation
RFMRecency, frequency, monetary+
A segmentation that scores each buyer on how recently they purchased, how often, and how much they have spent, then names groups from the score pattern.
On this data: Each metric is bucketed into fifths with NTILE(5). Recency is inverted so a recent buyer scores high. Segment names come from the recency and frequency scores; spend is reported per segment but does not drive the name.
NTILEN-tile window function+
A SQL function that splits ordered rows into N equal-sized buckets. NTILE(5) produces quintiles.
On this data: Used to turn raw recency, frequency, and spend into 1 to 5 scores for RFM.
Cohort retentionCohort retention analysis+
Group users by the week they first appeared, then track what share is still active in each later week. It separates a discovery problem from a lifecycle problem.
On this data: The retention grid on the user intelligence page is built from weekly_retention. Week 0 is always 100 percent by construction.
Machine learning
Propensity modelPurchase propensity+
A model that estimates the probability a user buys in the next period from their past behavior.
On this data: A LightGBM classifier trained on a temporal split: October behavior predicts a November purchase. Its card and drivers are on the propensity page.
LightGBMLight gradient boosting machine+
A gradient-boosted decision tree library. It grows trees leaf-wise and handles class imbalance well, which suits a low base-rate target like purchase.
On this data: The frozen model is a raw Booster loaded from LightGBM's native text format. Predictions are identical whether it loads from that or the legacy pickle.
AUC-ROCArea under the ROC curve+
The probability that the model ranks a random buyer above a random non-buyer. 0.5 is a coin flip, 1.0 is perfect ranking.
On this data: Reported on held-out users on the propensity page, straight from metrics.json.
Lift (targeting)Top-percentile lift+
The conversion rate of a top slice of the ranked population divided by the overall rate. A lift of 3 means that slice converts three times as often as average.
On this data: Measured on the top five percent of users by predicted propensity.
Experimentation
Market basket liftAssociation rule lift+
For a pair of products, the chance they are bought together divided by what you would expect if the two were independent. Above 1 is a real affinity.
On this data: Product pairs are mined from co-purchases in the same session. The affinity table keeps pairs with at least three co-occurrences and lift above 1.2.
Welch's t-testUnequal-variance t-test+
A two-sample test of whether two means differ that does not assume the groups have the same variance. Standard for A/B conversion comparisons.
On this data: The experiment calculator runs it on the two arms' conversion outcomes, matching the Python engine within rounding.
Statistical powerPower (1 minus type II error)+
The chance a test detects a real effect of a given size. Teams usually aim for 0.8 before running an experiment.
On this data: Computed post-hoc for each calculator run and precomputed across a segment by lift by confidence grid.
05
What this is not
scope The site describes the 2019-10-01 to 2019-11-30 sample: 1,645,912 events and 135,767 users. It is not the full multi-month source, and the aggregate figures scale with that.
model The propensity model is a frozen artifact. Its card on the propensity page reports 0.75 AUC and 4.6x top-5% lift as trained, and those values do not change when the sample does.
freshness There is no live data. A new pipeline run produces a new artifact set under a new commit, and a redeploy picks it up.