Build an XCTest Duration Regression Gate with xcresult

An XCTest that originally took just 4 seconds can gradually grow to 11 seconds while the pipeline remains entirely green. Teams often do not notice until the job queue becomes visibly congested. Total build time is a poor signal for pinpointing this kind of change because dependency resolution, compilation, simulator startup, and result writing are all mixed together. A more reliable approach is to retain the .xcresult from every test job on a NUMACS cloud Mac, extract per-test durations, and compare them against a controlled baseline.

Define What the Gate Measures

The gate should answer, “Is the same test consistently getting slower?” rather than, “Did this job take a few seconds longer than the previous one?” Start by separating the metrics into three levels:

Do not block a merge based on a single result. Short tests are vulnerable to scheduling jitter, while long tests can be amplified by network access, animations, or asynchronous waits. A practical rule uses two thresholds: a candidate enters retesting only when it exceeds both the baseline’s relative increase and its absolute increase. For example, if the baseline is 2 seconds, a 20% increase that adds only 0.1 seconds should not fail. If the baseline is 40 seconds, an increase of 8 seconds warrants investigation.

The goal of a performance gate is not to produce the same number on every run, but to identify repeatable, attributable slowdowns as early as possible.

Fix the Execution Environment and Generate a Result Bundle

First, pin the Xcode path, Scheme, destination, and parallelization policy. The simulator model and OS version must be explicit rather than relying on the “currently booted device.” Do not change the concurrency level dynamically within the same job.

set -euo pipefail

RESULT_DIR="$PWD/artifacts"
RESULT_BUNDLE="$RESULT_DIR/RegressionTests.xcresult"

mkdir -p "$RESULT_DIR"
rm -rf "$RESULT_BUNDLE"

xcodebuild test \
  -workspace Example.xcworkspace \
  -scheme ExampleTests \
  -destination 'platform=iOS Simulator,name=iPhone 16,OS=18.0' \
  -resultBundlePath "$RESULT_BUNDLE" \
  -parallel-testing-enabled NO

The versions in this example are only part of the execution environment. A real project should pin an approved Xcode version and simulator runtime. Run one unmeasured warm-up job first so the simulator can finish booting and required test resources can be written to disk. Run at least three measured samples. If the test itself has high variance, increase the sample count instead of loosening the thresholds until they become meaningless.

Extract Per-Test Durations from xcresult

Recent Xcode versions can use xcresulttool to output the test tree. Because the command interface evolves with Xcode, the parser must be pinned alongside the Xcode version used in CI. Before upgrading, test parser compatibility against saved result bundles.

xcrun xcresulttool get test-results tests \
  --path artifacts/RegressionTests.xcresult \
  --format json > artifacts/tests.json

jq -r '
  .. | objects
  | select(.nodeType? == "Test Case" and .duration? != null)
  | [.name, .duration] | @tsv
' artifacts/tests.json > artifacts/test-durations.tsv

Before comparison, verify that the TSV is not empty and that the test count matches expectations. An empty file must not be accepted as “no regressions.” It usually means the command interface changed, the tests did not run, or the parser criteria no longer match. Test identifiers should include the module, class, and method. For parameterized tests, retain the parameter name as well so distinct samples are not merged incorrectly.

Preserve the Original Evidence During Normalization

Convert every duration to seconds and attach the commit, Xcode version, destination, and job number to each record. Aggregated files simplify comparison, but the original .xcresult should still be retained as a job artifact. When an anomaly occurs, it also provides failure details, activity logs, and attachment context.

Use a Robust Baseline Instead of the Previous Result

The baseline should not be the most recent run on the main branch. One slow startup can distort later decisions, while one unusually fast run can generate numerous false positives. A more robust approach is to collect several recent successful samples from the main branch, calculate the median for each test, and record either a high percentile or the median absolute deviation.

Store fields like these for each test:

{
  "ExampleTests.testParsing": {
    "median_seconds": 3.84,
    "absolute_limit_seconds": 1.5,
    "relative_limit": 0.25,
    "sample_count": 9
  }
}

After a candidate branch first crosses the limits, rerun only the affected test cases or their containing suites. Mark the job as failed only if the retest median still exceeds both median_seconds + absolute_limit_seconds and median_seconds × (1 + relative_limit). New tests should first enter an observation period. Until they have enough samples, report their results without allowing them to block a merge.

Eliminate the Most Common False Regressions

A cold simulator start is the primary source of noise. Test data initialization, first-time font loading, and database table creation can also make the first run slower. Isolate these effects through warm-up runs or an explicit setUp. Next, check whether tests access the network, wait for real time to pass, share user defaults, or reuse files left behind by a previous test case.

Parallel testing can change contention for CPU, memory, and disk. If the goal is to establish a per-test baseline, disable parallel execution. If the goal is to evaluate real pipeline throughput, fix the worker count and include that count as a baseline dimension. Do not combine data from different Xcode versions, system runtimes, or hardware configurations.

When a sudden slowdown appears, check the following in order:

1. Whether the test count or execution destination changed;
2. Whether compilation, installation, or simulator startup introduced a wait;
3. Whether a timeout comes from polling or fixed sleeps;
4. Whether test fixtures grew or were not cleaned up afterward;
5. Whether multiple jobs are competing for the same working directory.

Make Baseline Changes Reviewable

Baseline files should be version-controlled, but ordinary test jobs must only read them and must never overwrite them automatically. When a slowdown is justified, a separate job should generate a diff from stable main-branch samples. Reviewers need to see the old value, new value, sample count, and reason for the change.

Group the final report into “new regressions,” “recovered,” and “under observation,” and include the absolute increase, relative increase, and retest result. This allows the gate to stop hidden performance degradation without letting one simulator fluctuation hold up development. The result is not a fragile stopwatch, but a reproducible, explainable, and reviewable XCTest duration baseline.

Frequently asked questions