Sales Funnel and A/A/B Test

The question

A startup selling food products wanted to understand what users do inside its app, and both questions come out of the same event log. The first is the sales funnel: which screens people pass through on the way to a purchase, how many get there, and where they get stuck. The second is an A/A/B test. The designers want to change the font across the whole app, and the managers think the new look will put users off.

Two control groups kept the old font (246 and 247) and one test group got the new one (248). Having two controls is not redundancy, it is quality control: if two arms that received exactly the same treatment already differ from each other, the split is broken and no difference against the test group proves anything.

The data

A log of 244,126 events with four columns: event name, user ID, a timestamp in seconds, and experiment group. There are five events: main screen, offers, cart, successful payment, and Tutorial.

The file runs from 25 July to 7 August 2019, but collection is only complete from 1 August. In the seven days before, volume climbs from 11 to 2,030 events a day, then jumps to 36,141 on 1 August. I wrote the rule before cutting: a day counts as complete when it reaches 50% of the median of the days after it and stays above that until the end. I reran the same rule from 10% to 90%, and on the volume measure the date never moved off 1 August. The cut cost 2,826 events (1.16%) but only 17 users (0.23%), because almost everyone who appeared earlier showed up again later. What remains is 240,887 events and 7,534 users.

Method, and why this method

In an event log, "duplicate" needs a definition before it becomes a count. Someone who goes back to the main screen three times produces three legitimate rows. So I only treated a row as suspect when all four columns matched, timestamp included, and before seeing the number I set 1% of the base as the limit for removing without further investigation. There were 413 such rows (0.17%), concentrated in Tutorial and payment and never more than four rows apart in the file, which looks like the collector resending data.

To fix the funnel order I went to the clock instead of trusting the screen names. For each user I took the first timestamp of every event and compared events in pairs against a threshold set in advance: an order exists only if A comes before B in at least 90% of cases. The main screen clears it easily, coming first in 93.73% to 94.97% of pairs. After that the sequence falls apart. Offers versus cart is 54.91% against 44.14%, close to a coin toss, and cart and payment fire in the same second for 49.77% of users, which a timestamp in seconds cannot separate. That is why the funnel is strict: each step counts only users who have every step before it, and conversion between the middle steps should be read as membership in those sets, with no guarantee of the path taken. Tutorial stays out. It reaches 840 users (11.15%) and is the very first recorded event for 93.45% of them, so it is an entry screen.

For the experiment I wrote the z-test for proportions by hand. The denominator is every user in the group rather than those who reached a given screen, because filtering on an event would condition the sample on something the font itself might have changed. The A/A test between the controls ran first, then 248 against each control separately and against both combined. That makes 20 tests, and at a 5% alpha the chance of at least one false positive is around 64%, so I applied a Bonferroni correction down to 0.0025.

Findings

Of the 7,419 users who open the main screen, 4,482 reach offers (60.41%), 3,580 reach the cart (79.88%) and 3,429 complete payment (95.78%). The funnel loses 3,990 people, and 2,937 of them drop at the first step. That step is the worst on both measures at once, rate and headcount, which did not have to be true: a step can have the worst rate and still cost few people when the step before it is already small. At the other end, people who reach the cart almost always pay, with only 151 lost at the last step. From first event to payment, 45.51% of the base makes it.

The A/A test passed. None of the five events differed between 246 and 247, the smallest p-value was 0.1146, and that clears the way for the rest of the experiment.

The new font changed nothing measurable. None of the 20 tests came out significant, either at 5% or after correction, and the smallest p-value was 0.0784, for the cart against group 246, a gap of 2.48 percentage points. Against 247 the same event shrinks to 0.78 points (p = 0.5786), and against the combined controls the p-value rises to 0.1818 instead of falling. A real font effect would show up at a similar size against both controls and get sharper with twice the sample. The opposite happened, so that 0.0784 was a quirk of how group 246 was drawn.

The decision it enables

Do not ship the new font, and the reason is cost, not statistics. The test found no effect in either direction: the gain that would justify the switch does not appear, and neither does the harm the managers feared. When the data does not decide, the asymmetry does, and changing the font across the whole app has a certain cost and an unmeasured benefit.

Put the effort into the step from the main screen to offers, where 2,937 of the 3,990 lost users drop, and spend nothing on payment, which already converts at 95.78%.

Three limits come with this decision. The analysis covers seven days, not fourteen. The arms ended up with a 1.09% deviation, above the 1% threshold I had written down, although a chi-square goodness-of-fit test finds no evidence of an uneven split (p = 0.7554). That slack costs some power, so what holds is the absence of evidence for a difference, not proof that the groups are equal. And the log has no field that says why anyone gave up. The hypothesis for the first step, that users open the app and cannot find their way to the catalogue, can only be tested by logging navigation events and time on screen, which the app does not record today.

Links

Repository