The median PR triggers 22 minutes of CI runs before merging

However, the underlying distribution is heavily right skewed:

• The p75 PR generates 1.5 hours of CI before merging
• The average PR generates 1.9 hours of CI before merging, even after dropping the top 2.5% of values

We define CI time as the sum of individual check runs and check statuses a PR triggers after it is submitted to remote. This includes summing concurrent CI runs, so does not equate to (and most likely overstates) the time an engineer might spend waiting for CI.

Two factors that most impact CI time include:

• The number of submits before a PR is merged
• The size of the org an engineer belongs to

The more submits, the more CI runs

To state the obvious, the more times a user submits a PR, the more times CI is triggered. Note: the last bar (6+ submits, p90) has been truncated.

This means factors that “drive submit volume” (namely PR size and # of reviews a PR receives) will also influence CI run times:

• Larger PRs trigger more submits (and more review cycles), thus also triggering more CI runs
• Larger orgs tend are also more likely to require reviews before merge, which also triggers more CI runs

Larger orgs spend more time on CI runs

Interestingly, org size also influences CI runs even when we control for submit volume. The larger the org, the longer CI runs per PR; this is especially true along the right tail of the CI run time distribution.

• The p75 CI run time is 1 hour 16 minutes for orgs up to 50 engineers
• The p75 CI run time jumps to 2 hours 10 minutes for orgs above 50 engineers

After we exceed 50 engineers, the CI run time per PR stabilizes.

As mentioned above, the correlation between larger orgs and longer CI runs holds true even when we standardize for the number of PR submits before merge.