I really like how simple this API is.

Time advances when every goroutine is idle.

How does time work when goroutines aren't idle? Does it stand still, or does it advance at the usual rate? If it stands still, it seems like that could break software that assumes time will advance during computation (that maybe that's rare in practice). If it advances at the usual rate, it seems like that reintroduces a source of flakiness. E.g., in your example, the 1 second sleep will advance time by 1 second, but then on a slow system the checking thread may still not execute for a long time.

What are the bounds of the fake time implementation? Presumably if you're making direct system calls that interact with times or durations, we're not going to do anything about that. Are we going to make any attempt at faking time in the file system?

If every goroutine is idle and there are no timers scheduled, Run panics.

What if a goroutine is blocked on a channel that goes outside the group? This came to mind in the context of whether this could be used to coordinate a multi-process client/server test, though I think it would also come up if there's any sort of interaction with a background worker goroutine or pool.

or is the goroutine calling Wait.

What happens if multiple goroutines in a group call Wait? I think the options are to panic or to consider all of them idle, in which case they would all wake up when every other goroutine in the group is idle.

What happens if you have nested groups, say group A contains group B, and a goroutine in B is blocked in Wait, and then a goroutine in A calls Wait? I think your options are to panic (though that feels wrong), wake up both if all of the goroutines in group A are idle, or wake up just B if all of the goroutines in B are idle (but this block waking up A until nothing is calling Wait in group B).

How does time work when goroutines aren't idle?

Time stands still, except when all goroutines in a group are idle. (Same as the playground behaves, I believe.) This would break software that assumes time will advance. You'd need to use something else to test that case.

What are the bounds of the fake time implementation?

The time package: Now, Since, Sleep, Timer, Ticker, etc.

Faking time in the filesystem seems complicated and highly specialized, so I don't think we should try. Code which cares about file timestamps will need to use a test fs.FS or some such.

What if a goroutine is blocked on a channel that goes outside the group?

As proposed, this would count as an idle goroutine. If you fail to isolate the system under test this will probably cause problems, so don't do that.

What happens if multiple goroutines in a group call Wait?

As proposed, none of them ever wake up and your test times out, or possibly panics if we can detect that all goroutines are blocked in that case. Having them all wake at the same time would also be reasonable.

What happens if you have nested groups

Oh, I didn't think of that. Nested groups are too complicated, Run should panic if called from within a group.

This is a very interesting proposal!

I feel worried that the synctest.Run characteristic of establishing a "goroutine group" and blocking until it completes might make it an attractive nuisance for folks who see it as simpler than arranging for the orderly completion of many goroutines using other synchronization primitives. That is: people may be tempted to use it in non-test code.

Assuming that's a valid concern (if it isn't then I'll retract this entire comment!), I could imagine mitigating it in two different ways:

• Offer "goroutine groups" as a standalone synchronization primitive that synctest.Run is implemented in terms of, offering the "wait for completion of this and any other related goroutines" mechanism as a feature separate from synthetic time. Those who want to use it in non-test code can therefore use the lower-level function directly, instead of using synctest.Run.
• Change the synctest.Run design in some way that makes it harder to misuse. One possible idea: make synctest.Run take a testing.TB as an additional argument, and then in every case where the proposal currently calls for a panic use t.FailNow() instead. It's inconvenient (though of course not impossible) to obtain a testing.TB implementation outside of a test case or benchmark, which could be sufficient inconvenience for someone to reconsider what they were attempting.

(I apologize in advance if I misunderstood any part of the proposal or if I am missing something existing that's already similarly convenient to synctest.Run.)

The fact that synctest goroutine groups always use a fake clock will hopefully act as discouragement to using them in non-test code. Defining goroutines blocked on I/O as not being idle also discourages use outside of tests; any goroutine reading from a network connection defeats synctest.Wait entirely.

I think using idle-wait synchronization outside of tests is always going to be a mistake. It's fragile and fiddly, and you're better served by explicit synchronization. (This prompts the question: Isn't this fragile and fiddly inside tests as well? It is, but using a fake clock removes much of the sources of fragility, and tests often have requirements that make the fiddliness a more worthwhile tradeoff. In the expiring cache example, for example, non-test code will never need to guarantee that a cache entry expires precisely at the nanosecond defined.)

So while perhaps we could offer a standalone synchroniziation primitive outside of synctest, I think we would need a very good understanding of when it would be appropriate to use it.

As for passing a testing.TB to synctest.Run, I don't think this would do much to prevent misuse, since the caller could just pass a &testing.T{}, or just nil. I don't think it would be wise to use synctest outside of tests, but if someone disagrees, then I don't think it's worth trying to stop them.

Interesting proposal. I like that it allows for waiting for a group of goroutines, as opposed to all goroutines in my proposal (#65336), though I do have some concerns:

• Complexity of implementation: Having to modify every time-related function may increase complexity for non-test code. Would it make more sense to outsource the mock time implementation to a third party library? The Wait() function should be sufficient for the third party library to function deterministically, and goroutines started by Run() would behave like normal goroutines in all aspects.

• Timeouts: In my proposal, WaitIdle() returns a <-chan struct{} since it allows for a test harness to abort the test if it takes too long (e.g. 30 seconds in case the test gets stuck in an infinite loop). Would it make sense for the Wait() function here to return a chan too to allow for timeouts?

One of the goals of this proposal is to minimize the amount of unnatural code required to make a system testable. Mock time implementations require replacing calls to idiomatic time package functions with a testable interface. Putting fake time in the standard library would let us just write the idiomatic code without compromising testability.

For timeouts, the -timeout test flag allows aborting too-slow tests. Putting an explicit timeout in test code is usually a bad idea, because how long a test is expected to run is a property of the local system. (I've seen a lot of tests inside Google which set an explicit timeout of 5 or 10 seconds, and then experience flakiness when run with -tsan and on CI systems that execute at a low batch priority.)

Also, it would be pointless for Wait to return a <-chan struct{}, because Wait must be called from within a synctest group and therefore the caller doesn't have access to a real clock.

I wanted to evaluate practical usage of the proposed API.

I wrote a version of Run and Wait based on parsing the output of runtime.Stack. Wait calls runtime.Gosched in a loop until all goroutines in the current group are idle.

I also wrote a fake time implementation.

Combined, these form a reasonable facsimile of the proposed synctest package, with some limitations: The code under test needs to be instrumented to call the fake time functions, and to call a marking function after creating new goroutines. Also, you need to call a synctest.Sleep function in tests to advance the fake clock.

I then added this instrumentation to net/http.

The synctest package does not work with real network connections, so I added an in-memory net.Conn implementation to the net/http tests.

I also added an additional helper to net/http's tests, which simplifies some of the experimentation below:

var errStillRunning = errors.New("async op still running")

// asyncResult is the result of an asynchronous operation.
type asyncResult[T any] struct {}

// runAsync runs f in a new goroutine,
// and returns an asyncResult which is populated with the result of f when it finishes.
// runAsync calls synctest.Wait after running f.
func runAsync[T any](f func() (T, error)) *asyncResult[T]

// done reports whether the asynchronous operation has finished.
func (r *asyncResult[T]) done() bool

// result returns the result of the asynchronous operation.
// It returns errStillRunning if the operation is still running.
func (r *asyncResult[T]) result() (T, error)

One of the longest-running tests in the net/http package is TestServerShutdownStateNew (https://go.googlesource.com/go/+/refs/tags/go1.22.3/src/net/http/serve_test.go#5611). This test creates a server, opens a connection to it, and calls Server.Shutdown. It asserts that the server, which is expected to wait 5 seconds for the idle connection to close, shuts down in no less than 2.5 seconds and no more than 7.5 seconds. This test generally takes about 5-6 seconds to run in both HTTP/1 and HTTP/2 modes.

The portion of this test which performs the shutdown is:

shutdownRes := make(chan error, 1)
go func() {
	shutdownRes <- ts.Config.Shutdown(context.Background())
}()
readRes := make(chan error, 1)
go func() {
	_, err := c.Read([]byte{0})
	readRes <- err
}()

// TODO(#59037): This timeout is hard-coded in closeIdleConnections.
// It is undocumented, and some users may find it surprising.
// Either document it, or switch to a less surprising behavior.
const expectTimeout = 5 * time.Second

t0 := time.Now()
select {
case got := <-shutdownRes:
	d := time.Since(t0)
	if got != nil {
		t.Fatalf("shutdown error after %v: %v", d, err)
	}
	if d < expectTimeout/2 {
		t.Errorf("shutdown too soon after %v", d)
	}
case <-time.After(expectTimeout * 3 / 2):
	t.Fatalf("timeout waiting for shutdown")
}

// Wait for c.Read to unblock; should be already done at this point,
// or within a few milliseconds.
if err := <-readRes; err == nil {
	t.Error("expected error from Read")
}

I wrapped the test in a synctest.Run call and changed it to use the in-memory connection. I then rewrote this section of the test:

shutdownRes := runAsync(func() (struct{}, error) {
	return struct{}{}, ts.Config.Shutdown(context.Background())
})
readRes := runAsync(func() (int, error) {
	return c.Read([]byte{0})
})

// TODO(#59037): This timeout is hard-coded in closeIdleConnections.
// It is undocumented, and some users may find it surprising.
// Either document it, or switch to a less surprising behavior.
const expectTimeout = 5 * time.Second

synctest.Sleep(expectTimeout - 1)
if shutdownRes.done() {
	t.Fatal("shutdown too soon")
}

synctest.Sleep(2 * time.Second)
if _, err := shutdownRes.result(); err != nil {
	t.Fatalf("Shutdown() = %v, want complete", err)
}
if n, err := readRes.result(); err == nil || err == errStillRunning {
	t.Fatalf("Read() = %v, %v; want error", n, err)
}

The test exercises the same behavior it did before, but it now runs instantaneously. (0.01 seconds on my laptop.)

I made an interesting discovery after converting the test: The server does not actually shut down in 5 seconds. In the initial version of this test, I checked for shutdown exactly 5 seconds after calling Shutdown. The test failed, reporting that the Shutdown call had not completed.

Examining the Shutdown function revealed that the server polls for closed connections during shutdown, with a maximum poll interval of 500ms, and therefore shutdown can be delayed slightly past the point where connections have shut down.

I changed the test to check for shutdown after 6 seconds. But once again, the test failed.

Further investigation revealed this code (https://go.googlesource.com/go/+/refs/tags/go1.22.3/src/net/http/server.go#3041):

st, unixSec := c.getState()
// Issue 22682: treat StateNew connections as if
// they're idle if we haven't read the first request's
// header in over 5 seconds.
if st == StateNew && unixSec < time.Now().Unix()-5 {
	st = StateIdle
}

The comment states that new connections are considered idle for 5 seconds, but thanks to the low granularity of Unix timestamps the test can consider one idle for as little as 4 or as much as 6 seconds. Combined with the 500ms poll interval (and ignoring any added scheduler delay), Shutdown may take up to 6.5 seconds to complete, not 5.

Using a fake clock rather than a real one not only speeds up this test dramatically, but it also allows us to more precisely test the behavior of the system under test.

Another slow test is TestTransportExpect100Continue (https://go.googlesource.com/go/+/refs/tags/go1.22.3/src/net/http/transport_test.go#1188). This test sends an HTTP request containing an "Expect: 100-continue" header, which indicates that the client is waiting for the server to indicate that it wants the request body before it sends it. In one variation, the server does not send a response; after a 2 second timeout, the client gives up waiting and sends the request.

This test takes 2 seconds to execute, thanks to this timeout. In addition, the test does not validate the timing of the client sending the request body; in particular, tests pass even if the client waits

The portion of the test which sends the request is:

resp, err := c.Do(req)

I changed this to:

rt := runAsync(func() (*Response, error) {
	return c.Do(req)
})
if v.timeout {
	synctest.Sleep(expectContinueTimeout-1)
	if rt.done() {
		t.Fatalf("RoundTrip finished too soon")
	}
	synctest.Sleep(1)
}
resp, err := rt.result()
if err != nil {
	t.Fatal(err)
}

This test now executes instantaneously. It also verifies that the client does or does not wait for the ExpectContinueTimeout as expected.

I made one discovery while converting this test. The synctest.Run function blocks until all goroutines in the group have exited. (In the proposed synctest package, Run will panic if all goroutines become blocked (deadlock), but I have not implemented that feature in the test version of the package.) The test was hanging in Run, due to leaking a goroutine. I tracked this down to a missing net.Conn.Close call, which was leaving an HTTP client reading indefinitely from an idle and abandoned server connection.

In this case, Run's behavior caused me some confusion, but ultimately led to the discovery of a real (if fairly minor) bug in the test. (I'd probably have experienced less confusion, but I initially assumed this was a bug in the implementation of Run.)

At one point during this exercise, I accidentally called testing.T.Run from within a synctest.Run group. This results in, at the very best, quite confusing behavior. I think we would want to make it possible to detect when running within a group, and have testing.T.Run panic in this case.

My experimental implementation of the synctest package includes a synctest.Sleep function by necessity: It was much easier to implement with an explicit call to advance the fake clock. However, I found in writing these tests that I often want to sleep and then wait for any timers to finish executing before continuing.

I think, therefore, that we should have one additional convenience function:

package synctest

// Sleep pauses the current goroutine for the duration d,
// and then blocks until every goroutine in the current group is idle.
// It is identical to calling time.Sleep(d) followed by Wait.
//
// The caller of Sleep must be in a goroutine created by Run,
// or a goroutine transitively started by Run.
// If it is not, Sleep panics.
func Sleep(d time.Duration) {
	time.Sleep(d)
	Wait()
}

The net/http package was not designed to support testing with a fake clock. This has served as an obstacle to improving the state of the package's tests, many of which are slow, flaky, or both.

Converting net/http to be testable with my experimental version of synctest required a small number of minor changes. A runtime-supported synctest would have required no changes at all to net/http itself.

Converting net/http tests to use synctest required adding an in-memory net.Conn. (I didn't attempt to use net.Pipe, because its fully-synchronous behavior tends to cause problems in tests.) Aside from this, the changes required were very small.

My experiment is in https://go.dev/cl/587657.

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

Commenting here due to @rsc's request:

Relative to my proposal #65336, I have the following concerns:

• Goroutine grouping: the only precedent for goroutine having a user-visible identity is runtime.LockOSThread(), and even then, it is set-only: a goroutine can not know whether it is locked to a thread or not without parsing runtime.Stack() output. Having these special "test mode" goroutines feels like a violation of goroutines being interchangeable anonymous workers, insofar as the Go runtime hides the goroutine ID from user code. Having a global wait is acceptable in the case of tests since it is unlikely for background goroutines to be present to interfere with the wait (and possibly actually desirable to catch those too).
• Overriding standard library behavior: again, there is no precedent for standard library functions to behave differently based on what goroutine they are called from. The standard idiomatic way to do this is to define an interface (e.g. fs.FS) and direct all calls through the interface, and the implementation of the interface can be mocked at test time. If it is desirable to keep the current Run()/Wait() API, I would still strongly advocate for not changing the behavior of the standard time package, and instead incorporate a mock clock implementation in another package (likely under testing).

Regarding overriding the time package vs. providing a testing implementation:

The time package provides a number of commonly-used, exported functions, where code that makes direct use of these functions cannot be properly tested. I think this makes it unique in the standard library. For example, code which directly calls time.Sleep cannot be tested properly, because inserting a real delay inherently makes a test slow, and because there is no non-flaky way for a test to ensure that a certain amount of time has elapsed.

In contrast, we can test code which calls os.Open by providing it with the name of a file in a test directory. We can test code which calls net.Listen by listening on a loopback interface. The io/fs.FS interface may be used to create a testable seam in a system, but it isn't required.

Time is fundamentally different in that there is no way to use real time in a test without making the test flaky and slow.

Time is also different from an fs.File or a net.Conn in that there is only one plausible production implementation of time. A fs.FS might be the local filesystem, or an embedded set of static files, or a remote filesystem of some kind. A net.Conn might be a TCP or TLS connection. But it is difficult to come up with occasions outside of tests when time.Sleep should do anything other than sleep for the defined amount of time.

Since we can't use real time in tests, we can insert a testable wrapper around the time package as you propose. This requires that we avoid the idiomatic and easy-to-use time package functions. We essentially put an asterisk next to every existing function in the time package that deals with the system clock saying, "don't actually use this, or at least not in code you intend to test".

In addition, if we define a standard testable wrapper around the clock, we are essentially declaring that all public packages which deal with time should provide a way to plumb in a clock. (Some packages do this already, of course; crypto/tls.Config.Time is an example in std).

That's an option, of course. But it would be a very large change to the Go ecosystem as a whole.

the only precedent for goroutine having a user-visible identity is runtime.LockOSThread()

The pprof.SetGoroutineLabels disagrees.

insofar as the Go runtime hides the goroutine ID from user code

It doesn't try to hide it, more like tries to restrict people from relying on numbers.

Having a global wait is acceptable in the case of tests since it is unlikely for background goroutines to be present to interfere with the wait (and possibly actually desirable to catch those too).

If I understood proposal correctly, it will wait for any goroutine (and recursively) that was started using go statement from the func passed to Run. It will not catch anything started before or sidewise. Which brings the good question: @neild will it also wait for time.AfterFunc(...) goroutines if time.AfterFunc(...) was called in the chain leading to synctest.Run?

@neild will it also wait for time.AfterFunc(...) goroutines if time.AfterFunc(...) was called in the chain leading to synctest.Run?

Yes, if you call AfterFunc from within a synctest group then the goroutine started by AfterFunc is also in the group.

Given that there's more precedent for goroutine identity than I had previously thought, and seeing how pprof.Do() works, I am onboard with the idea of goroutine groups.

However, I'm still a little ambivalent about goroutine groups affecting time package / standard library behavior, and theoretically a test running in synctest mode may want to know the real world time for logging purposes (I guess that could be solved by adding a time.RealNow() or something similar). The Wait() primitive seems to provide what is necessary for a third-party package to provide the same functionality without additional runtime support, so it could be worth exploring this option a bit more.

That being said, I agree that plumbing a time/clock interface through existing code is indeed tedious, and having time modified to conditionally use a mock timer may be the lesser evil. But it still feels a little icky to me for some reason.

Thanks for doing the experiment. I find the results pretty compelling.

I think, therefore, that we should have one additional convenience function: [synctest.Sleep]

I don't quite understand this function. Given the fake time implementation, if you sleep even a nanosecond past timer expiry, aren't you already guaranteed that those timers will have run because the fake time won't advance to your sleep deadline until everything is blocked again?

Nested groups are too complicated, Run should panic if called from within a group.

Partly I was wondering about nested groups because I've been scheming other things that the concept of a goroutine group could be used for. Though it's true that, even if we have groups for other purposes, it may make sense to say that synctest groups cannot be nested, even if in general groups can be nested.

Given the fake time implementation, if you sleep even a nanosecond past timer expiry, aren't you already guaranteed that those timers will have run because the fake time won't advance to your sleep deadline until everything is blocked again?

You're right that sleeping past the deadline of a timer is sufficient. The synctest.Wait function isn't strictly necessary at all; you could use time.Sleep(1) to skip ahead a nanosecond and ensure all currently running goroutines have parked.

It's fairly natural to sleep to the exact instant of a timer, however. If a cache entry expires in some amount of time, it's easy to sleep for that exact amount of time, possibly using the same constant that the cache timeout was initialized with, rather than adding a nanosecond.

Adding nanoseconds also adds a small but real amount of confusion to a test in various small ways: The time of logged events drifts off the integer second, rate calculations don't come out as cleanly, and so on.

Plus, if you forget to add the necessary adjustment or otherwise accidentally sleep directly onto the instant of a timer's expiry, you get a race condition.

Cleaner, I think, for the test code to always resynchronize after poking the system under test. This doesn't have to be a function in the synctest package, of course; synctest.Sleep is a trivial two-liner using exported APIs. But I suspect most users of the package would use it, or at least the ones that make use of the synthetic clock.

I've been scheming other things that the concept of a goroutine group could be used for.

I'm very intrigued! I've just about convinced myself that there's a useful general purpose synchronization API hiding in here, but I'm not sure what it is or what it's useful for.

For what it's worth, I think it's a good thing that virtual time is included in this, because it makes sure that this package isn't used in production settings. It makes it only suitable for tests (and very suitable).

It sounds like the API is still:

// Package synctest provides support for testing concurrent code.
package synctest

// Run executes f in a new goroutine.
//
// The new goroutine and any goroutines transitively started by it form a group.
// Run waits for all goroutines in the group to exit before returning.
//
// Goroutines in the group use a synthetic time implementation.
// The initial time is midnight UTC 2000-01-01.
// Time advances when every goroutine is idle.
// If every goroutine is idle and there are no timers scheduled,
// Run panics.
func Run(f func())

// Wait blocks until every goroutine within the current group is idle.
//
// A goroutine is idle if it is blocked on a channel operation,
// mutex operation,
// time.Sleep,
// a select with no cases,
// or is the goroutine calling Wait.
//
// A goroutine blocked on an I/O operation, such as a read from a network connection,
// is not idle. Tests which operate on a net.Conn or similar type should use an
// in-memory implementation rather than a real network connection.
//
// The caller of Wait must be in a goroutine created by Run,
// or a goroutine transitively started by Run.
// If it is not, Wait panics.
func Wait()

Damien suggested adding also:

// Sleep pauses the current goroutine for the duration d,
// and then blocks until every goroutine in the current group is idle.
// It is identical to calling time.Sleep(d) followed by Wait.
//
// The caller of Sleep must be in a goroutine created by Run,
// or a goroutine transitively started by Run.
// If it is not, Sleep panics.
func Sleep(d time.Duration) {
	time.Sleep(d)
	Wait()
}

The difference between time.Sleep and synctest.Sleep seems subtle enough that it seems like you should have to spell out the Wait at the call sites where you need it. The only time you really need Wait is if you know someone else is waking up at that very moment. But then if they've both done the Sleep+Wait form then you still have a problem. You really only want some of the call sites (maybe just one) to use the Sleep+Wait form. I suppose that the production code will use time.Sleep since it's not importing synctest, so maybe it's clear that the test harness is the only one that will call Sleep+Wait. On the other hand, fixing a test failure by changing s/time.Sleep/synctest.Sleep/ will be a strange-looking bug fix. Better to have to add synctest.Wait instead. If we really need this, it could be synctest.SleepAndWait but that's what statements are for. Probably too subtle and should just limit the proposal to Run and Wait.

Some additional suggestions for the description of the Wait() function:

// A goroutine is idle if it is blocked on a channel operation,
// mutex operation (...),
// time.Sleep,
// a select operation with or without cases,
// or is the goroutine calling Wait.
//
// A goroutine blocked on an I/O operation, such as a read from a network connection,
// is not idle. Tests which operate on a net.Conn or similar type should use an
// in-memory implementation rather than a real network connection.
//
// A goroutine blocked on a direct syscall (via the syscall package) is also not idle,
// even if the syscall itself sleeps.

Additionally, for "mutex operation", let's list out the the exact operations considered for implementation/testing completeness:

• sync.Cond.Wait()
• sync.Mutex.Lock()
• sync.RWMutex.Lock()
• sync.RWMutex.RLock()
• sync.WaitGroup.Wait()

The API looks simple and that is excellent.

What I am worried about is the unexpected failure modes, leading to undetected regressions, which might need tight support in the testing package to detect.

Imagine you unit test your code but are unable to mock out a dependency. Maybe due to lack of experience or bad design of existing code I have to work with.

That dependency that suddenly starts calling a syscall (e.g. to lazily try to tune the library using a sync.Once instead of on init time and having a timeout).

Without support in testing you will never detect that now and only your tests will suddenly time out after an innocent minor dependency update.

May I ortgogonally to the previous comment suggest to limit this package to standard library only to gather more experience with that approach before ?

That would also allow to sketch out integration with the testing package in addition to finding more pitfalls.

What I am worried about is the unexpected failure modes, leading to undetected regressions, which might need tight support in the testing package to detect.

Can you expand more on what you mean by undetected regressions?

If the code under test (either directly, or through a dependency) unexpectedly calls a blocking syscall, Wait will wait for that syscall to complete before proceeding. If the syscall completes normally (the code is using os/exec to execute a subprocess, for example), then everything should operate as expected--the operation completes and the test proceeds. If the syscall is waiting on some event (reading from a network socket, perhaps), then the test will hang, which is a detectable event. You can look at goroutine stacks from the timed-out test to analyze the reason for the hang.

Without support in testing

What kind of support are you thinking of?

What does this do?

func TestWait(t *testing.T) {
    synctest.Run(func() {
        synctest.Wait()
    })
}

Does it succeed or panic? It's not clear to me from the API docs because:

If every goroutine is idle and there are no timers scheduled, Run panics.

A goroutine is idle if it [...] is the goroutine calling Wait.

This is obviously a degenerate case, but I think it also applies if a test wanted to get the fake time features when testing otherwise non-concurrent code.

What does this do?

func TestWait(t *testing.T) {
    synctest.Run(func() {
        synctest.Wait()
    })
}

In this case, the goroutine calling synctest.Wait() should never enter idle because there's nothing to wait for, and hence a panic should not occur.

b.Loop handles scaling b.N internally, which involves a time.Since on b.start, which is time.Now before calling the benchmark function. Since synctest uses a completely different clock, time.Since on a time from outside the bubble won't work properly.

With a normal b.N loop, scaling is handled by calling the benchmark function multiple times and timing the entire thing, which means all timing occurs outside the bubble.

@prattmic Thank you for the insights. I had a hunch it was something along those lines.

I'm wondering if there is a way to detect the use of b.Loop in a bubble and show some helpful output to explain that timings are unable to be accurate and therefore are not displayed; or, if it is incorrect to mix synctest a b.Loop, refuse to build the benchmark?

We could also have B.Loop use the real clock, even when in a bubble. Not saying that's the right choice, but it's an option.

If we don't have B.Loop use the real clock, it should probably panic when invoked in a bubble.

Current proposal status

The testing/synctest package will be available in Go 1.24 as an experimental package. The package is not available by default. To use it, build your program with the environment variable GOEXPERIMENT=synctest.

The experimental package API is as follows:

// Run executes f in a new goroutine.
//
// The new goroutine and any goroutines transitively started by it form
// an isolated "bubble".
// Run waits for all goroutines in the bubble to exit before returning.
//
// Goroutines in the bubble use a synthetic time implementation.
// The initial time is midnight UTC 2000-01-01.
//
// Time advances when every goroutine in the bubble is blocked.
// For example, a call to time.Sleep will block until all other
// goroutines are blocked and return after the bubble's clock has
// advanced. See [Wait] for the specific definition of blocked.
//
// If every goroutine is blocked and there are no timers scheduled,
// Run panics.
//
// Channels, time.Timers, and time.Tickers created within the bubble
// are associated with it. Operating on a bubbled channel, timer, or ticker
// from outside the bubble panics.
func Run(f func())

// Wait blocks until every goroutine within the current bubble,
// other than the current goroutine, is durably blocked.
// It panics if called from a non-bubbled goroutine,
// or if two goroutines in the same bubble call Wait at the same time.
//
// A goroutine is durably blocked if can only be unblocked by another
// goroutine in its bubble. The following operations durably block
// a goroutine:
//   - a send or receive on a channel from within the bubble
//   - a select statement where every case is a channel within the bubble
//   - sync.Cond.Wait
//   - time.Sleep
//
// A goroutine executing a system call or waiting for an external event
// such as a network operation is not durably blocked.
// For example, a goroutine blocked reading from an network connection
// is not durably blocked even if no data is currently available on the
// connection, because it may be unblocked by data written from outside
// the bubble or may be in the process of receiving data from a kernel
// network buffer.
//
// A goroutine is not durably blocked when blocked on a send or receive
// on a channel that was not created within its bubble, because it may
// be unblocked by a channel receive or send from outside its bubble.
func Wait()

As this package is experimental, it is not subject to the Go 1 compatibility promise. Depending on experience and feedback, we may promote testing/synctest to a fully-supported package, possibly with incompatible changes from the current version; continue the experiment in future versions, again possibly with amendments; or drop the experiment entirely and remove the package.

Your feedback is essential! If you try out testing/synctest, please report your experiences, both positive and negative, in this issue.

Known issues

Time advances forever after main test goroutine exits

This call to Run never returns:

synctest.Run(func() {
  time.NewTicker(1 * time.Second)
})

The problem is that Run does not return so long as advancing the fake clock causes some event to happen. The time.Ticker provides a never-ending source of events. A similar problem happens with any background goroutine that performs a periodic action:

synctest.Run(func() {
  go func() {
    for {
      time.Sleep(1 * time.Second)
      // do something
    }
  }()
}

A potential fix for this issue is to say that the fake clock stops advancing once the goroutine started by Run returns. With this change the first example with a time.Ticker would return, and the second example with a sleeping background goroutine would panic.

Confusing error when go.mod Go version is less than 1.23

Go 1.23 added a new implementation of channel-based timers (https://go.dev/wiki/Go123Timer), with slightly different semantics. The new implementation is used in programs where package main is in a module with a go.mod declaring go1.23 or later. The GODEBUG setting asynctimerchan can force the old or new semantics.

The current testing/synctest implementation is not compatible with the old implementation. If you try to use testing/synctest in a program where package main's go.mod declares a pre-1.23 version of Go, you get a confusing error about "synctest.Run not supported with asynctimerchan!=0".

We either need to support asynctimerchan=1 or improve the error message.

(testing.TB).Cleanup does not work well in synctest bubbles

In the following code, the cleanup function is executed outside the bubble, and panics due to operating on a bubbled channel from outside the bubble:

func Test(t *testing.T) {
  synctest.Run(func() {
    ch := make(chan struct{})
    t.Cleanup(func() {
      close(ch)
    })
  })
}

In general, T.Cleanup doesn't work well when used inside a synctest bubble. This is unfortunate, because T.Cleanup is tremendously useful.

(*testing.B).Loop does not work in synctest bubbles

When used inside a synctest bubble, B.Loop uses the fake clock and gets confused.

Either B.Loop should use the real clock, or it should panic when used inside a bubble.

Deadlock errors don't print useful stacks

When every goroutine in a bubble is durably blocked and advancing the current time won't unblock any of them, Run panics: "deadlock: all goroutines in bubble are blocked".

In a test, this panic results in the stack for the goroutine that called Run being printed, but not the stacks of any other goroutines. In this situation, the user almost certainly wants to see the stacks of all the goroutines in the bubble.

Change https://go.dev/cl/641176 mentions this issue: testing/synctest: add some examples

Change https://go.dev/cl/642422 mentions this issue: main.star: set GOEXPERIMENT=synctest for builders testing >=go1.24

Change https://go.dev/cl/642675 mentions this issue: main.star: set GOEXPERIMENT=synctest for x/net builders testing >=go1.24

Original Post Summary

The original post proposes a new package, testing/synctest, for testing concurrent Go code. (@neild, issue) The package offers two primary features: simulated time control and synchronization for asynchronous operations. (@neild, issue)

The core API consists of synctest.Run, which executes a function in a separate goroutine with synthetic time, and synctest.Wait, which blocks until all goroutines within the Run-created group are idle. (@neild, issue) Synthetic time starts at "midnight UTC 2000-01-01" and only advances when all goroutines within a group are idle. (@neild, issue) The Run function waits for all goroutines in the group to exit before returning. (@neild, issue) A goroutine is considered idle if blocked on channel operations, mutexes, time.Sleep, selects without cases, or the Wait function itself. Goroutines blocked on I/O are specifically not idle. (@neild, issue)

The post includes an example of testing a concurrent cache, demonstrating how synctest eliminates flakiness and slowness compared to traditional time.Sleep based tests. (@neild, issue) The post also mentions experience with golang.org/x/net/http2 tests as motivation and acknowledges limitations related to I/O operations, suggesting in-memory replacements like net.Pipe for network connections. (@neild, issue) A later edit suggests a synctest.Sleep function that combines time.Sleep and Wait for convenience. (@neild, issue)

Discussion Themes

• API Design and Placement: Discussion arose regarding the API's simplicity and potential misuse outside of testing. (@aclements, issue comment; @apparentlymart, issue comment) Suggestions included providing standalone synchronization primitives, requiring a testing.TB argument for Run, or limiting the package to the standard library. One commenter supported the inclusion of virtual time to explicitly prevent production use. (@rsc, issue comment) There were also discussions about making the API a method on testing.T for better integration with existing testing infrastructure. (@narqo, issue comment; @neild, issue comment) Some later comments suggested alternative APIs like synctest.Enable or synctest.Start for easier use in testing frameworks, but @neild expressed reservations about these approaches. (@adamluzsi, issue comment; @neild, issue comment)

• Synchronization and Idle Goroutines: Commenters explored edge cases involving idle goroutine detection, nested groups, interactions with external goroutines, and the behavior of multiple simultaneous Wait calls. (@aclements, issue comment; @neild, issue comment) The implications of time freezing during non-idle periods and the scope of the fake time implementation were also discussed. The author clarified that nested groups are not supported and that Run should panic if called within a group. (@neild, issue comment) Later, further complications were discovered relating to goroutines blocked on global mutexes like those in crypto/rand and reflect. (@neild, issue comment)

• Bubbling Behavior and Interaction with Standard Library: Debate focused on whether or not to bubble channels and the handling of timeouts. Concerns were raised about the complexity of modifying time-related functions and the potential for interference with background goroutines. (@gh2o, issue comment) Suggestions included using a third-party mock time implementation and implementing timeouts for Wait. Later, discussion centered on the behavior of channels crossing bubble boundaries, and whether or not to panic in such cases, particularly with respect to timer channels versus all channels. (@cherrymui, issue comment; @rsc, issue comment) A concern about lazy initialization of global channels within a bubble was raised. (@prattmic, issue comment) The interaction with the testing.T lifecycle, specifically t.Cleanup, presented a challenge, as cleanup functions running outside the bubble could panic when interacting with bubbled channels. (@neild, issue comment)

• Real World Examples and Testing: @neild provided detailed reports and code examples of using a prototype implementation of synctest with the net/http package and the etcd project. (@neild, issue comment; @neild, issue comment) These experiments highlighted both successes and challenges in applying the proposed API, including issues with leaked goroutines, the need for synctest.Wait after time.Sleep, and complexities in handling network connections within tests. A subsequent attempt to apply synctest to a project using the clockwork fake time library showed potential for simplifying tests and the system under test. (@neild, issue comment) Another commenter shared a successful application of synctest with network mocking using a third-party library. (@paskozdilar, issue comment)

Next Steps

The testing/synctest package will be included in Go 1.24 as an experimental package behind the GOEXPERIMENT=synctest build tag. (issue comment) The proposal remains open for feedback and further iteration based on real-world usage during the experimental phase. Several known issues are documented, including the indefinite advancement of time after the main test goroutine exits, compatibility issues with older Go versions, interactions with t.Cleanup and B.Loop, and unhelpful stack traces in deadlock situations. These issues, along with community feedback, will inform the future direction of the package, potentially leading to its promotion, further amendments, or removal.

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

Change https://go.dev/cl/645675 mentions this issue: testing/synctest: add an example of testing networked code

Change https://go.dev/cl/645719 mentions this issue: runtime: print stack traces for bubbled goroutines on synctest deadlock

@neild
Thank you for this great proposal! I was really excited while reading through it, and while going over the example code, I happened to notice a very minor issue that’s unrelated to the main content of the proposal.

In the TestCacheEntryExpires function, the callback function is defined to return an int, but it uses fmt.Sprintf to return a string. This causes a small type mismatch.

c := NewCache(2 * time.Second, func(key string) int {
	count++
	return fmt.Sprintf("%v:%v", key, count)
})

Suggested correction:

c := NewCache(2 * time.Second, func(key string) string {
	count++
	return fmt.Sprintf("%v:%v", key, count)
})

and

func TestCacheEntryExpires(t *testing.T) {
        synctest.Run(func() {
                count := 0
                        c := NewCache(2 * time.Second, func(key string) int {
                        count++
                        return fmt.Sprintf("%v:%v", key, count)
                })

Suggested correction:

func TestCacheEntryExpires(t *testing.T) {
        synctest.Run(func() {
                count := 0
                        c := NewCache(2 * time.Second, func(key string) string {
                        count++
                        return fmt.Sprintf("%v:%v", key, count)
                })

This is just a minor detail and doesn’t affect the overall proposal, but I thought I’d point it out. Thanks again for the proposal—I’m really looking forward to seeing how this develops!

Great feature! Really exciting! I've tried this on my project, and it works very well. And I've also learned a lot from reading your discussions.

You're right that sleeping past the deadline of a timer is sufficient. The synctest.Wait function isn't strictly necessary at all; you could use time.Sleep(1) to skip ahead a nanosecond and ensure all currently running goroutines have parked.

I found a case where synctest.Wait() maybe necessary:

var a int

func TestPrintA(t *testing.T) {
	synctest.Run(func() {
		go func() { println(a) }()
		synctest.Wait()
		a = 2
	})
}

Running this case with go test -race will pass. However, if I replace synctest.Wait() with time.Sleep(1), I will get race detected during execution of test. I think this should be the expected behavior. The one that uses sleep should behave like it were executed out of bubble. But we can still use synctest.Wait to access internals without triggering data race detector. This is very convenient. But note that this can only sequence test code after the system-under-test, not before, e.g., this will not work and still reports data race:

var a int

func printA() {
	time.Sleep(1 * time.Second)
	println(a)
}

func TestPrintA(t *testing.T) {
	synctest.Run(func() {
		go printA()
		a = 2
		synctest.Wait()
	})
}

But I can just move a = 2 before creating the goroutine.

However, while race detector is aware of synctest.Wait(), it seems not aware of that synctest.Run() also waits for all goroutines to finish. e.g. this will report an unexpected data race:

var a int

func TestUpdate(t *testing.T) {
	synctest.Run(func() {
		println(a)
	})
	synctest.Run(func() {
		a = 2
	})
}

==================
WARNING: DATA RACE
Write at 0x0001029e08d0 by goroutine 9:
  huww98.cn/playground/a.TestUpdate.func2()
      /Users/huww98/source/playground/a/synctest_test.go:15 +0x28

Previous read at 0x0001029e08d0 by goroutine 8:
  huww98.cn/playground/a.TestUpdate.func1()
      /Users/huww98/source/playground/a/synctest_test.go:12 +0x28

Goroutine 9 (running) created at:
  internal/synctest.Run()
      /Users/huww98/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24rc3.darwin-arm64/src/runtime/synctest.go:178 +0x104
  testing.tRunner()
      /Users/huww98/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24rc3.darwin-arm64/src/testing/testing.go:1792 +0x180
  testing.(*T).Run.gowrap1()
      /Users/huww98/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24rc3.darwin-arm64/src/testing/testing.go:1851 +0x40

Goroutine 8 (finished) created at:
  internal/synctest.Run()
      /Users/huww98/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24rc3.darwin-arm64/src/runtime/synctest.go:178 +0x104
  testing.tRunner()
      /Users/huww98/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24rc3.darwin-arm64/src/testing/testing.go:1792 +0x180
  testing.(*T).Run.gowrap1()
      /Users/huww98/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24rc3.darwin-arm64/src/testing/testing.go:1851 +0x40
==================
--- FAIL: TestUpdate (0.00s)
    testing.go:1490: race detected during execution of test

This can also cause failure when invoke the test with go test -race -count 2:

var a int

func TestUpdate(t *testing.T) {
	synctest.Run(func() {
		a = 2
		println(a)
	})
}

However, while race detector is aware of synctest.Wait(), it seems not aware of that synctest.Run() also waits for all goroutines to finish. e.g. this will report an unexpected data race:

Thanks, that's an interesting point I hadn't thought of. I think it should be straightforward to tell the race detector that there's a happens-before relationship between a bubble's goroutines exiting and Run returning.

Change https://go.dev/cl/647755 mentions this issue: runtime: establish happens-before between goroutine and bubble exit

@pierrre That's a known issue - see #71488.

Change https://go.dev/cl/651555 mentions this issue: runtime: print stack traces for bubbled goroutines on synctest deadlock

👋🏼 Hi there, not sure if this was mentioned but time.NewTicker that doesn't call Stop does not end up panicking. The test instead just hangs forever. Here's a repro:

func TestTicker(t *testing.T) {
	synctest.Run(func() {
		time.NewTicker(time.Second)
	})
}

The above test hangs forever. If I instead call Stop, then it will pass correctly.

I expect the above should panic similar to when a goroutine is left hanging when func ends like so:

func TestDanglingGoroutine(t *testing.T) {
	synctest.Run(func() {
		go func() {
			select {}
		}()
	})
}

More general feedback:

1. Overall, I really like how my concurrent tests are looking code-wise and the fact that tests run fast due to the fake clock is even better.
2. Onboarding was a little difficult because even though the synctest.Run(func () { ... }) API is simple, there are a lot of implicit rules such as: all goroutines must exit by the end of the of func. That said, the rule itself was great because it forced me to write more correct code that handled cleaning up goroutines and also gave me confidence that I'm not leaking any goroutines either.

Thanks for the report!

time.NewTicker that doesn't call Stop does not end up panicking. The test instead just hangs forever.

This is on my known issues list. (#67434 (comment))

I think that the fix here is going to be to stop advancing time once the goroutine started by Run exits. Any leftover Tickers or Timers will not fire. If there are any goroutines blocked reading from a ticker, Run will panic with a deadlock error.

there are a lot of implicit rules such as: all goroutines must exit by the end of the of func. That said, the rule itself was great because it forced me to write more correct code that handled cleaning up goroutines and also gave me confidence that I'm not leaking any goroutines either.

Thanks especially for the experience report on this behavior. It's a bit unusual to enforce goroutine cleanup (there's a lot of code out there that leaks goroutines, especially in tests), so I'm glad to hear that it seems to have worked out as intended for you.