At Keboola, we test. A lot. And when one CI cycle takes over two hours, you start wondering where all that time goes.
We have 2,589 E2E tests and 5,127 unit tests. Unit tests don't trouble us much: they take a few minutes in CI. We run them everywhere. On every commit in a PR and on master. But those E2E tests. There are fewer of them, but they exercise the whole system. From API calls through queues to backend databases such as Snowflake and BigQuery.
Add data providers to the E2E test count and we end up with 3,626 individual test executions in CI. And this took a LONG time. A full cycle took over two hours, with at least an hour and a half spent on E2E tests alone. The rest was setup and other checks. Sometimes a run took even longer because some operations, particularly around BigQuery, needed retries due to eventual consistency.
BTW, about 40% of our unit tests are mock-heavy, and we're increasingly coming to think that those kinds of unit tests are pointless in agentic engineering. AI always writes them “green.” It finds a way to get there, and you pay in tokens to generate them, more tokens to update them when something changes, and most of the time nobody will read them closely anyway. They just eat into the precious time available for each turn. But that's a topic for another day.
Getting tests down to around an hour
First, we tried the usual things: add resources, add another machine for PR E2E tests so runs on master can finish sooner. None of that made a substantial difference.
I do have to admit that the support I added to GitHub Actions for running a single suite, part of a suite, or even one test made it much faster to test a PR before it reached master. That helped the whole team a lot.
Where does the time go?
We started out blind. CI didn't give us detailed timings showing exactly what was consuming the time. So my first step was to add a few metrics that would guide the rest of the optimization.
I added API request timings for each phase of a test. How much time went into preparing the environment, running the test itself, waiting for asynchronous jobs, and cleaning up afterward?
Interesting things started to emerge. Helper methods kept asking the API for the same information over and over. Sometimes the first API call returned everything needed for test setup, only for the suite to immediately request the resource details containing that same information. Individually, these were small, quick, easy-to-miss operations. Together, they consumed a substantial part of the run.
One such check made over five thousand requests during a suite run and took more than twelve minutes. Yet the information was available from a single listing. Another helper added about four seconds to every test. That doesn't sound terrible on its own. Except that in one suite, it added up to almost 27 minutes.
Clean up after yourself!
Looking at the timings for setUp, the test itself, and tearDown brought another surprise. Our tests spent an enormous amount of time in setUp. They checked whether the objects they needed existed, looked for leftovers from previous runs — a badly behaved test that didn't clean up, or a crashed test that never got the chance — and recreated the resources they needed.
Many tests also used non-unique resource names, and their execution and cleanup affected far more than they should. With concurrent runs, one test could delete something another test still needed.
Those are always fun to track down. A test passes once, fails the next time, and looks perfectly healthy when run on its own because nobody else is touching its work :-)
Next, I fixed resource naming so every test created its own unique objects and only managed those. I fixed cleanup so it wouldn't interfere with other suites, and simplified the whole thing.
This could only be properly tested in CI, where plenty of tests can run side by side. The CI server I mentioned earlier, with support for running individual suites, was a big help here. My testing cycle went from several hours to a few dozen minutes.
Test isolation also laid the groundwork for the next step: increasing parallelism and adding more processes to run the tests.
Everyone's finished except one
Then came the division of work. Initially, our runner split tests into parallel chunks based on their count and a simple ordering. But two equally sized piles of tests can take very different amounts of time.
A slow class would end up in one chunk, keeping its process busy for almost an hour while the others had already finished. Things were running in parallel, but the whole suite was still held up by one straggler.
So I started saving individual test durations as a GitHub Actions artifact. I took the timings from the JUnit report. On the next run, the runner would load the saved timings and distribute tests into chunks expected to take roughly the same amount of time.
If timings were missing for an entire suite, it kept the previous split. If a single test had no timing, it used the median test duration for that suite. The next run would have an actual measurement for it.
There was one catch: runs on master needed their own timing history. A PR artifact didn't automatically replace it. So although I'd tested the change in a PR, the first run on master was still slow. It had to save timings for subsequent runs first. Then came the expected improvement: tens of minutes shaved off the run.
The other end has to keep up too
The tests don't put that much load on the GitHub Actions runner itself. They call real APIs, create workspaces, and wait for data from Keboola. But the servers processing all of that have quite a bit to do when so many tests run in parallel.
Another piece of the puzzle was measuring how long tests waited for asynchronous background processing and watching our worker utilization in Datadog.
I found that we didn't have enough worker capacity. Increasing resources for the main application, as I'd described at the beginning, wasn't enough. Another system component was a bottleneck too. More parallel tests would simply have created a longer queue in front of it.
After these changes, the E2E tests themselves came down to roughly an hour. The median successful run from the evening of October 14 onward is 66 minutes. The full cycle, including preparation and other checks, takes about an hour and a half.
Looking at longer periods, the comparison is:
| Successful E2E runs | April 14–May 12 | September 30–October 28 |
|---|---|---|
| Number of runs | 22 | 90 |
| Average duration | 102 min | 69 min |
| Median duration | 101 min | 67 min |
| Median runner-minutes | 1,075 | 901 |
That's a 32% reduction in the average wait for a successful E2E run. Meanwhile, the number of suites increased from 28 to 31. Median runner-minutes, the sum of the durations of individual jobs, fell by 16%. So this isn't simply a case of consuming more machine time at once.
Here, I'm measuring from the start of the first E2E suite to the end of the last. Queue time and environment deployment aren't included. And I'm still comparing successful runs: failing runs with retries can remain long. Across all runs, p90 stayed at 152 minutes. The older sample of successful runs is also smaller, so I wouldn't read too much into individual minutes.
We've also observed a change in the team: before the speedup, we could get two or three batches of PRs into production each day. Now it's six to eight, occasionally ten. CI timings alone obviously don't prove that faster tests caused the entire difference. But less waiting gives us more room to deliver further changes.
A few extras from the reports
During this work, I repeatedly saw individual tests run several times. A test might fail twice and pass on the third attempt. We have plenty of retries in the tests, mainly because of BigQuery, but also for other operations.
I also saw a test fail consistently while retries stretched the suite to two and a half hours. Then the GitHub Actions runner cut it off.
It wasn't just test-level retries, either. Test clients had their own retries with exponential backoff. A single deterministic error could hold a test up for about 34 minutes before it even got a chance to fail. If the suite then retried, we waited all over again. We therefore limited those clients to a single attempt.
I added a report to the artifacts showing which tests needed retries and how much time they spent on them.
Our workflow now also posts updates on each PR: whether tests have been started, whether they're waiting or running, and how they finished.
Because #context matters!
These details help both us developers and our agents. They don't have to look up the information through more GitHub API calls.
Hackathon
Most of this work happened during a three-day team hackathon. We shut ourselves away, code around the same table from morning till night, drink beer, and eat cows and bulls off the grill. The finishing touches came during the week after the hackathon.
I can wholeheartedly recommend doing something like this. I always feel that those three days of absolute focus let us deliver as much work as an entire month.
