Go Wiki: PerformanceMonitoring
The Go project monitors the performance characteristics of the Go implementation as well as that of subrepositories like golang.org/x/tools.
Benchmarks
golang.org/x/benchmarks/cmd/bench
is the entrypoint for our performance tests.
For Go implementations, this runs both the
Sweet (end-to-end benchmarks)
and bent (microbenchmarks)
benchmarking suites.
For the golang.org/x/tools
project, it runs the repository’s benchmarks.
These benchmarks can all be invoked manually, as can cmd/bench
, but using both
Sweet and bent directly will likely offer a better user experience.
See their documentation for more details.
Performance testing principles
Change with the times
Our set of benchmarks is curated. It is allowed to change over time. Sticking to a single benchmark set over a long period of time can easily land us in a situation where we’re optimizing for the wrong thing.
Always perform a comparison
We never report performance numbers in isolation, and only relative to some baseline. This strategy comes from the fact that comparing performance data taken far apart in time, even on the same hardware, can result in a lot of noise that goes unaccounted for. The state of a machine or VM on one day is likely to be very different than the state of a machine or VM on the next day.
We refer to the tested version of source code as the “experiment” and the baseline version of source code as the “baseline.”
Presubmit
Do you have a Gerrit change that you want to run against our benchmarks?
Select a builder containing the word perf
in the “Choose Tryjobs” dialog that
appears when selecting a SlowBot.
There are two kinds of presubmit builders for performance testing:
perf_vs_parent
, which measures the performance delta of a change in isolation.perf_vs_tip
, which measures the performance delta versus the current tip-of-tree for whichever repository the change is for. (Remember to rebase your change(s) before using this one!)
There’s a third special presubmit builder for the tools repository as well which
contains the string perf_vs_gopls_0_11
.
This measures the performance delta versus the release-branch-gopls.0.11
branch
of the tools repository.
Postsubmit
The performance dashboard provides continuous monitoring of benchmark performance for every commit that is made to the main Go repository and other subrepositories. The dashboard, more specifically, displays graphs showing the change in certain performance metrics (also called “units”) over time for different benchmarks. Use the navigation interface at the top of the page to explore further.
The regressions page displays all benchmarks in order of biggest regression to biggest improvement, followed by all benchmarks for which there is no statistically clear answer.
On the graphs, red means regression, blue means improvement.
Baselines
In post-submit, the baseline version for Go repository performance tests is
automatically determined.
For performance tests against changes on release branches, the baseline is always
the latest release for that branch (for example, the latest minor release for
Go 1.21 on release-branch.go1.21
).
For performance tests against tip-of-tree, the baseline is always the latest
overall release of Go.
This is indicated by the name of the builder that produces these benchmark
esults, which contains the string perf_vs_release
.
What this means is that on every minor release of Go, the baseline shifts.
These baseline shifts can be observed in the per-metric view.
Performance tests on subrepositories typically operate against some known
long-term fixed baseline.
For the tools repository, it’s the tip of the release-branch-gopls.0.11
branch.
Per-metric view
Click on any graph’s performance metric name to view a more detailed timeline of the performance deltas for that metric.
This view is particularly useful for identifying the culprit behind a regression and for pinpointing the source of an improvement.
Sometimes, a performance change happens because a benchmark has changed or
because the baseline version being used as changed.
This view also displays information about the baseline versions and the version
of golang.org/x/benchmarks
that was used to produce the results to help identify
when this happens.
This content is part of the Go Wiki.