Source file src/runtime/mgc.go
1 // Copyright 2009 The Go Authors. All rights reserved. 2 // Use of this source code is governed by a BSD-style 3 // license that can be found in the LICENSE file. 4 5 // Garbage collector (GC). 6 // 7 // The GC runs concurrently with mutator threads, is type accurate (aka precise), allows multiple 8 // GC thread to run in parallel. It is a concurrent mark and sweep that uses a write barrier. It is 9 // non-generational and non-compacting. Allocation is done using size segregated per P allocation 10 // areas to minimize fragmentation while eliminating locks in the common case. 11 // 12 // The algorithm decomposes into several steps. 13 // This is a high level description of the algorithm being used. For an overview of GC a good 14 // place to start is Richard Jones' gchandbook.org. 15 // 16 // The algorithm's intellectual heritage includes Dijkstra's on-the-fly algorithm, see 17 // Edsger W. Dijkstra, Leslie Lamport, A. J. Martin, C. S. Scholten, and E. F. M. Steffens. 1978. 18 // On-the-fly garbage collection: an exercise in cooperation. Commun. ACM 21, 11 (November 1978), 19 // 966-975. 20 // For journal quality proofs that these steps are complete, correct, and terminate see 21 // Hudson, R., and Moss, J.E.B. Copying Garbage Collection without stopping the world. 22 // Concurrency and Computation: Practice and Experience 15(3-5), 2003. 23 // 24 // 1. GC performs sweep termination. 25 // 26 // a. Stop the world. This causes all Ps to reach a GC safe-point. 27 // 28 // b. Sweep any unswept spans. There will only be unswept spans if 29 // this GC cycle was forced before the expected time. 30 // 31 // 2. GC performs the mark phase. 32 // 33 // a. Prepare for the mark phase by setting gcphase to _GCmark 34 // (from _GCoff), enabling the write barrier, enabling mutator 35 // assists, and enqueueing root mark jobs. No objects may be 36 // scanned until all Ps have enabled the write barrier, which is 37 // accomplished using STW. 38 // 39 // b. Start the world. From this point, GC work is done by mark 40 // workers started by the scheduler and by assists performed as 41 // part of allocation. The write barrier shades both the 42 // overwritten pointer and the new pointer value for any pointer 43 // writes (see mbarrier.go for details). Newly allocated objects 44 // are immediately marked black. 45 // 46 // c. GC performs root marking jobs. This includes scanning all 47 // stacks, shading all globals, and shading any heap pointers in 48 // off-heap runtime data structures. Scanning a stack stops a 49 // goroutine, shades any pointers found on its stack, and then 50 // resumes the goroutine. 51 // 52 // d. GC drains the work queue of grey objects, scanning each grey 53 // object to black and shading all pointers found in the object 54 // (which in turn may add those pointers to the work queue). 55 // 56 // e. Because GC work is spread across local caches, GC uses a 57 // distributed termination algorithm to detect when there are no 58 // more root marking jobs or grey objects (see gcMarkDone). At this 59 // point, GC transitions to mark termination. 60 // 61 // 3. GC performs mark termination. 62 // 63 // a. Stop the world. 64 // 65 // b. Set gcphase to _GCmarktermination, and disable workers and 66 // assists. 67 // 68 // c. Perform housekeeping like flushing mcaches. 69 // 70 // 4. GC performs the sweep phase. 71 // 72 // a. Prepare for the sweep phase by setting gcphase to _GCoff, 73 // setting up sweep state and disabling the write barrier. 74 // 75 // b. Start the world. From this point on, newly allocated objects 76 // are white, and allocating sweeps spans before use if necessary. 77 // 78 // c. GC does concurrent sweeping in the background and in response 79 // to allocation. See description below. 80 // 81 // 5. When sufficient allocation has taken place, replay the sequence 82 // starting with 1 above. See discussion of GC rate below. 83 84 // Concurrent sweep. 85 // 86 // The sweep phase proceeds concurrently with normal program execution. 87 // The heap is swept span-by-span both lazily (when a goroutine needs another span) 88 // and concurrently in a background goroutine (this helps programs that are not CPU bound). 89 // At the end of STW mark termination all spans are marked as "needs sweeping". 90 // 91 // The background sweeper goroutine simply sweeps spans one-by-one. 92 // 93 // To avoid requesting more OS memory while there are unswept spans, when a 94 // goroutine needs another span, it first attempts to reclaim that much memory 95 // by sweeping. When a goroutine needs to allocate a new small-object span, it 96 // sweeps small-object spans for the same object size until it frees at least 97 // one object. When a goroutine needs to allocate large-object span from heap, 98 // it sweeps spans until it frees at least that many pages into heap. There is 99 // one case where this may not suffice: if a goroutine sweeps and frees two 100 // nonadjacent one-page spans to the heap, it will allocate a new two-page 101 // span, but there can still be other one-page unswept spans which could be 102 // combined into a two-page span. 103 // 104 // It's critical to ensure that no operations proceed on unswept spans (that would corrupt 105 // mark bits in GC bitmap). During GC all mcaches are flushed into the central cache, 106 // so they are empty. When a goroutine grabs a new span into mcache, it sweeps it. 107 // When a goroutine explicitly frees an object or sets a finalizer, it ensures that 108 // the span is swept (either by sweeping it, or by waiting for the concurrent sweep to finish). 109 // The finalizer goroutine is kicked off only when all spans are swept. 110 // When the next GC starts, it sweeps all not-yet-swept spans (if any). 111 112 // GC rate. 113 // Next GC is after we've allocated an extra amount of memory proportional to 114 // the amount already in use. The proportion is controlled by GOGC environment variable 115 // (100 by default). If GOGC=100 and we're using 4M, we'll GC again when we get to 8M 116 // (this mark is computed by the gcController.heapGoal method). This keeps the GC cost in 117 // linear proportion to the allocation cost. Adjusting GOGC just changes the linear constant 118 // (and also the amount of extra memory used). 119 120 // Oblets 121 // 122 // In order to prevent long pauses while scanning large objects and to 123 // improve parallelism, the garbage collector breaks up scan jobs for 124 // objects larger than maxObletBytes into "oblets" of at most 125 // maxObletBytes. When scanning encounters the beginning of a large 126 // object, it scans only the first oblet and enqueues the remaining 127 // oblets as new scan jobs. 128 129 package runtime 130 131 import ( 132 "internal/cpu" 133 "internal/runtime/atomic" 134 "unsafe" 135 ) 136 137 const ( 138 _DebugGC = 0 139 _FinBlockSize = 4 * 1024 140 141 // concurrentSweep is a debug flag. Disabling this flag 142 // ensures all spans are swept while the world is stopped. 143 concurrentSweep = true 144 145 // debugScanConservative enables debug logging for stack 146 // frames that are scanned conservatively. 147 debugScanConservative = false 148 149 // sweepMinHeapDistance is a lower bound on the heap distance 150 // (in bytes) reserved for concurrent sweeping between GC 151 // cycles. 152 sweepMinHeapDistance = 1024 * 1024 153 ) 154 155 // heapObjectsCanMove always returns false in the current garbage collector. 156 // It exists for go4.org/unsafe/assume-no-moving-gc, which is an 157 // unfortunate idea that had an even more unfortunate implementation. 158 // Every time a new Go release happened, the package stopped building, 159 // and the authors had to add a new file with a new //go:build line, and 160 // then the entire ecosystem of packages with that as a dependency had to 161 // explicitly update to the new version. Many packages depend on 162 // assume-no-moving-gc transitively, through paths like 163 // inet.af/netaddr -> go4.org/intern -> assume-no-moving-gc. 164 // This was causing a significant amount of friction around each new 165 // release, so we added this bool for the package to //go:linkname 166 // instead. The bool is still unfortunate, but it's not as bad as 167 // breaking the ecosystem on every new release. 168 // 169 // If the Go garbage collector ever does move heap objects, we can set 170 // this to true to break all the programs using assume-no-moving-gc. 171 // 172 //go:linkname heapObjectsCanMove 173 func heapObjectsCanMove() bool { 174 return false 175 } 176 177 func gcinit() { 178 if unsafe.Sizeof(workbuf{}) != _WorkbufSize { 179 throw("size of Workbuf is suboptimal") 180 } 181 // No sweep on the first cycle. 182 sweep.active.state.Store(sweepDrainedMask) 183 184 // Initialize GC pacer state. 185 // Use the environment variable GOGC for the initial gcPercent value. 186 // Use the environment variable GOMEMLIMIT for the initial memoryLimit value. 187 gcController.init(readGOGC(), readGOMEMLIMIT()) 188 189 work.startSema = 1 190 work.markDoneSema = 1 191 lockInit(&work.sweepWaiters.lock, lockRankSweepWaiters) 192 lockInit(&work.assistQueue.lock, lockRankAssistQueue) 193 lockInit(&work.strongFromWeak.lock, lockRankStrongFromWeakQueue) 194 lockInit(&work.wbufSpans.lock, lockRankWbufSpans) 195 } 196 197 // gcenable is called after the bulk of the runtime initialization, 198 // just before we're about to start letting user code run. 199 // It kicks off the background sweeper goroutine, the background 200 // scavenger goroutine, and enables GC. 201 func gcenable() { 202 // Kick off sweeping and scavenging. 203 c := make(chan int, 2) 204 go bgsweep(c) 205 go bgscavenge(c) 206 <-c 207 <-c 208 memstats.enablegc = true // now that runtime is initialized, GC is okay 209 } 210 211 // Garbage collector phase. 212 // Indicates to write barrier and synchronization task to perform. 213 var gcphase uint32 214 215 // The compiler knows about this variable. 216 // If you change it, you must change builtin/runtime.go, too. 217 // If you change the first four bytes, you must also change the write 218 // barrier insertion code. 219 // 220 // writeBarrier should be an internal detail, 221 // but widely used packages access it using linkname. 222 // Notable members of the hall of shame include: 223 // - github.com/bytedance/sonic 224 // 225 // Do not remove or change the type signature. 226 // See go.dev/issue/67401. 227 // 228 //go:linkname writeBarrier 229 var writeBarrier struct { 230 enabled bool // compiler emits a check of this before calling write barrier 231 pad [3]byte // compiler uses 32-bit load for "enabled" field 232 alignme uint64 // guarantee alignment so that compiler can use a 32 or 64-bit load 233 } 234 235 // gcBlackenEnabled is 1 if mutator assists and background mark 236 // workers are allowed to blacken objects. This must only be set when 237 // gcphase == _GCmark. 238 var gcBlackenEnabled uint32 239 240 const ( 241 _GCoff = iota // GC not running; sweeping in background, write barrier disabled 242 _GCmark // GC marking roots and workbufs: allocate black, write barrier ENABLED 243 _GCmarktermination // GC mark termination: allocate black, P's help GC, write barrier ENABLED 244 ) 245 246 //go:nosplit 247 func setGCPhase(x uint32) { 248 atomic.Store(&gcphase, x) 249 writeBarrier.enabled = gcphase == _GCmark || gcphase == _GCmarktermination 250 } 251 252 // gcMarkWorkerMode represents the mode that a concurrent mark worker 253 // should operate in. 254 // 255 // Concurrent marking happens through four different mechanisms. One 256 // is mutator assists, which happen in response to allocations and are 257 // not scheduled. The other three are variations in the per-P mark 258 // workers and are distinguished by gcMarkWorkerMode. 259 type gcMarkWorkerMode int 260 261 const ( 262 // gcMarkWorkerNotWorker indicates that the next scheduled G is not 263 // starting work and the mode should be ignored. 264 gcMarkWorkerNotWorker gcMarkWorkerMode = iota 265 266 // gcMarkWorkerDedicatedMode indicates that the P of a mark 267 // worker is dedicated to running that mark worker. The mark 268 // worker should run without preemption. 269 gcMarkWorkerDedicatedMode 270 271 // gcMarkWorkerFractionalMode indicates that a P is currently 272 // running the "fractional" mark worker. The fractional worker 273 // is necessary when GOMAXPROCS*gcBackgroundUtilization is not 274 // an integer and using only dedicated workers would result in 275 // utilization too far from the target of gcBackgroundUtilization. 276 // The fractional worker should run until it is preempted and 277 // will be scheduled to pick up the fractional part of 278 // GOMAXPROCS*gcBackgroundUtilization. 279 gcMarkWorkerFractionalMode 280 281 // gcMarkWorkerIdleMode indicates that a P is running the mark 282 // worker because it has nothing else to do. The idle worker 283 // should run until it is preempted and account its time 284 // against gcController.idleMarkTime. 285 gcMarkWorkerIdleMode 286 ) 287 288 // gcMarkWorkerModeStrings are the strings labels of gcMarkWorkerModes 289 // to use in execution traces. 290 var gcMarkWorkerModeStrings = [...]string{ 291 "Not worker", 292 "GC (dedicated)", 293 "GC (fractional)", 294 "GC (idle)", 295 } 296 297 // pollFractionalWorkerExit reports whether a fractional mark worker 298 // should self-preempt. It assumes it is called from the fractional 299 // worker. 300 func pollFractionalWorkerExit() bool { 301 // This should be kept in sync with the fractional worker 302 // scheduler logic in findRunnableGCWorker. 303 now := nanotime() 304 delta := now - gcController.markStartTime 305 if delta <= 0 { 306 return true 307 } 308 p := getg().m.p.ptr() 309 selfTime := p.gcFractionalMarkTime + (now - p.gcMarkWorkerStartTime) 310 // Add some slack to the utilization goal so that the 311 // fractional worker isn't behind again the instant it exits. 312 return float64(selfTime)/float64(delta) > 1.2*gcController.fractionalUtilizationGoal 313 } 314 315 var work workType 316 317 type workType struct { 318 full lfstack // lock-free list of full blocks workbuf 319 _ cpu.CacheLinePad // prevents false-sharing between full and empty 320 empty lfstack // lock-free list of empty blocks workbuf 321 _ cpu.CacheLinePad // prevents false-sharing between empty and nproc/nwait 322 323 wbufSpans struct { 324 lock mutex 325 // free is a list of spans dedicated to workbufs, but 326 // that don't currently contain any workbufs. 327 free mSpanList 328 // busy is a list of all spans containing workbufs on 329 // one of the workbuf lists. 330 busy mSpanList 331 } 332 333 // Restore 64-bit alignment on 32-bit. 334 _ uint32 335 336 // bytesMarked is the number of bytes marked this cycle. This 337 // includes bytes blackened in scanned objects, noscan objects 338 // that go straight to black, objects allocated as black during 339 // the cycle, and permagrey objects scanned by markroot during 340 // the concurrent scan phase. 341 // 342 // This is updated atomically during the cycle. Updates may be batched 343 // arbitrarily, since the value is only read at the end of the cycle. 344 // 345 // Because of benign races during marking, this number may not 346 // be the exact number of marked bytes, but it should be very 347 // close. 348 // 349 // Put this field here because it needs 64-bit atomic access 350 // (and thus 8-byte alignment even on 32-bit architectures). 351 bytesMarked uint64 352 353 markrootNext uint32 // next markroot job 354 markrootJobs uint32 // number of markroot jobs 355 356 nproc uint32 357 tstart int64 358 nwait uint32 359 360 // Number of roots of various root types. Set by gcMarkRootPrepare. 361 // 362 // nStackRoots == len(stackRoots), but we have nStackRoots for 363 // consistency. 364 nDataRoots, nBSSRoots, nSpanRoots, nStackRoots int 365 366 // Base indexes of each root type. Set by gcMarkRootPrepare. 367 baseData, baseBSS, baseSpans, baseStacks, baseEnd uint32 368 369 // stackRoots is a snapshot of all of the Gs that existed 370 // before the beginning of concurrent marking. The backing 371 // store of this must not be modified because it might be 372 // shared with allgs. 373 stackRoots []*g 374 375 // Each type of GC state transition is protected by a lock. 376 // Since multiple threads can simultaneously detect the state 377 // transition condition, any thread that detects a transition 378 // condition must acquire the appropriate transition lock, 379 // re-check the transition condition and return if it no 380 // longer holds or perform the transition if it does. 381 // Likewise, any transition must invalidate the transition 382 // condition before releasing the lock. This ensures that each 383 // transition is performed by exactly one thread and threads 384 // that need the transition to happen block until it has 385 // happened. 386 // 387 // startSema protects the transition from "off" to mark or 388 // mark termination. 389 startSema uint32 390 // markDoneSema protects transitions from mark to mark termination. 391 markDoneSema uint32 392 393 bgMarkDone uint32 // cas to 1 when at a background mark completion point 394 // Background mark completion signaling 395 396 // mode is the concurrency mode of the current GC cycle. 397 mode gcMode 398 399 // userForced indicates the current GC cycle was forced by an 400 // explicit user call. 401 userForced bool 402 403 // initialHeapLive is the value of gcController.heapLive at the 404 // beginning of this GC cycle. 405 initialHeapLive uint64 406 407 // assistQueue is a queue of assists that are blocked because 408 // there was neither enough credit to steal or enough work to 409 // do. 410 assistQueue struct { 411 lock mutex 412 q gQueue 413 } 414 415 // sweepWaiters is a list of blocked goroutines to wake when 416 // we transition from mark termination to sweep. 417 sweepWaiters struct { 418 lock mutex 419 list gList 420 } 421 422 // strongFromWeak controls how the GC interacts with weak->strong 423 // pointer conversions. 424 strongFromWeak struct { 425 // block is a flag set during mark termination that prevents 426 // new weak->strong conversions from executing by blocking the 427 // goroutine and enqueuing it onto q. 428 // 429 // Mutated only by one goroutine at a time in gcMarkDone, 430 // with globally-synchronizing events like forEachP and 431 // stopTheWorld. 432 block bool 433 434 // q is a queue of goroutines that attempted to perform a 435 // weak->strong conversion during mark termination. 436 // 437 // Protected by lock. 438 lock mutex 439 q gQueue 440 } 441 442 // cycles is the number of completed GC cycles, where a GC 443 // cycle is sweep termination, mark, mark termination, and 444 // sweep. This differs from memstats.numgc, which is 445 // incremented at mark termination. 446 cycles atomic.Uint32 447 448 // Timing/utilization stats for this cycle. 449 stwprocs, maxprocs int32 450 tSweepTerm, tMark, tMarkTerm, tEnd int64 // nanotime() of phase start 451 452 // pauseNS is the total STW time this cycle, measured as the time between 453 // when stopping began (just before trying to stop Ps) and just after the 454 // world started again. 455 pauseNS int64 456 457 // debug.gctrace heap sizes for this cycle. 458 heap0, heap1, heap2 uint64 459 460 // Cumulative estimated CPU usage. 461 cpuStats 462 } 463 464 // GC runs a garbage collection and blocks the caller until the 465 // garbage collection is complete. It may also block the entire 466 // program. 467 func GC() { 468 // We consider a cycle to be: sweep termination, mark, mark 469 // termination, and sweep. This function shouldn't return 470 // until a full cycle has been completed, from beginning to 471 // end. Hence, we always want to finish up the current cycle 472 // and start a new one. That means: 473 // 474 // 1. In sweep termination, mark, or mark termination of cycle 475 // N, wait until mark termination N completes and transitions 476 // to sweep N. 477 // 478 // 2. In sweep N, help with sweep N. 479 // 480 // At this point we can begin a full cycle N+1. 481 // 482 // 3. Trigger cycle N+1 by starting sweep termination N+1. 483 // 484 // 4. Wait for mark termination N+1 to complete. 485 // 486 // 5. Help with sweep N+1 until it's done. 487 // 488 // This all has to be written to deal with the fact that the 489 // GC may move ahead on its own. For example, when we block 490 // until mark termination N, we may wake up in cycle N+2. 491 492 // Wait until the current sweep termination, mark, and mark 493 // termination complete. 494 n := work.cycles.Load() 495 gcWaitOnMark(n) 496 497 // We're now in sweep N or later. Trigger GC cycle N+1, which 498 // will first finish sweep N if necessary and then enter sweep 499 // termination N+1. 500 gcStart(gcTrigger{kind: gcTriggerCycle, n: n + 1}) 501 502 // Wait for mark termination N+1 to complete. 503 gcWaitOnMark(n + 1) 504 505 // Finish sweep N+1 before returning. We do this both to 506 // complete the cycle and because runtime.GC() is often used 507 // as part of tests and benchmarks to get the system into a 508 // relatively stable and isolated state. 509 for work.cycles.Load() == n+1 && sweepone() != ^uintptr(0) { 510 Gosched() 511 } 512 513 // Callers may assume that the heap profile reflects the 514 // just-completed cycle when this returns (historically this 515 // happened because this was a STW GC), but right now the 516 // profile still reflects mark termination N, not N+1. 517 // 518 // As soon as all of the sweep frees from cycle N+1 are done, 519 // we can go ahead and publish the heap profile. 520 // 521 // First, wait for sweeping to finish. (We know there are no 522 // more spans on the sweep queue, but we may be concurrently 523 // sweeping spans, so we have to wait.) 524 for work.cycles.Load() == n+1 && !isSweepDone() { 525 Gosched() 526 } 527 528 // Now we're really done with sweeping, so we can publish the 529 // stable heap profile. Only do this if we haven't already hit 530 // another mark termination. 531 mp := acquirem() 532 cycle := work.cycles.Load() 533 if cycle == n+1 || (gcphase == _GCmark && cycle == n+2) { 534 mProf_PostSweep() 535 } 536 releasem(mp) 537 } 538 539 // gcWaitOnMark blocks until GC finishes the Nth mark phase. If GC has 540 // already completed this mark phase, it returns immediately. 541 func gcWaitOnMark(n uint32) { 542 for { 543 // Disable phase transitions. 544 lock(&work.sweepWaiters.lock) 545 nMarks := work.cycles.Load() 546 if gcphase != _GCmark { 547 // We've already completed this cycle's mark. 548 nMarks++ 549 } 550 if nMarks > n { 551 // We're done. 552 unlock(&work.sweepWaiters.lock) 553 return 554 } 555 556 // Wait until sweep termination, mark, and mark 557 // termination of cycle N complete. 558 work.sweepWaiters.list.push(getg()) 559 goparkunlock(&work.sweepWaiters.lock, waitReasonWaitForGCCycle, traceBlockUntilGCEnds, 1) 560 } 561 } 562 563 // gcMode indicates how concurrent a GC cycle should be. 564 type gcMode int 565 566 const ( 567 gcBackgroundMode gcMode = iota // concurrent GC and sweep 568 gcForceMode // stop-the-world GC now, concurrent sweep 569 gcForceBlockMode // stop-the-world GC now and STW sweep (forced by user) 570 ) 571 572 // A gcTrigger is a predicate for starting a GC cycle. Specifically, 573 // it is an exit condition for the _GCoff phase. 574 type gcTrigger struct { 575 kind gcTriggerKind 576 now int64 // gcTriggerTime: current time 577 n uint32 // gcTriggerCycle: cycle number to start 578 } 579 580 type gcTriggerKind int 581 582 const ( 583 // gcTriggerHeap indicates that a cycle should be started when 584 // the heap size reaches the trigger heap size computed by the 585 // controller. 586 gcTriggerHeap gcTriggerKind = iota 587 588 // gcTriggerTime indicates that a cycle should be started when 589 // it's been more than forcegcperiod nanoseconds since the 590 // previous GC cycle. 591 gcTriggerTime 592 593 // gcTriggerCycle indicates that a cycle should be started if 594 // we have not yet started cycle number gcTrigger.n (relative 595 // to work.cycles). 596 gcTriggerCycle 597 ) 598 599 // test reports whether the trigger condition is satisfied, meaning 600 // that the exit condition for the _GCoff phase has been met. The exit 601 // condition should be tested when allocating. 602 func (t gcTrigger) test() bool { 603 if !memstats.enablegc || panicking.Load() != 0 || gcphase != _GCoff { 604 return false 605 } 606 switch t.kind { 607 case gcTriggerHeap: 608 trigger, _ := gcController.trigger() 609 return gcController.heapLive.Load() >= trigger 610 case gcTriggerTime: 611 if gcController.gcPercent.Load() < 0 { 612 return false 613 } 614 lastgc := int64(atomic.Load64(&memstats.last_gc_nanotime)) 615 return lastgc != 0 && t.now-lastgc > forcegcperiod 616 case gcTriggerCycle: 617 // t.n > work.cycles, but accounting for wraparound. 618 return int32(t.n-work.cycles.Load()) > 0 619 } 620 return true 621 } 622 623 // gcStart starts the GC. It transitions from _GCoff to _GCmark (if 624 // debug.gcstoptheworld == 0) or performs all of GC (if 625 // debug.gcstoptheworld != 0). 626 // 627 // This may return without performing this transition in some cases, 628 // such as when called on a system stack or with locks held. 629 func gcStart(trigger gcTrigger) { 630 // Since this is called from malloc and malloc is called in 631 // the guts of a number of libraries that might be holding 632 // locks, don't attempt to start GC in non-preemptible or 633 // potentially unstable situations. 634 mp := acquirem() 635 if gp := getg(); gp == mp.g0 || mp.locks > 1 || mp.preemptoff != "" { 636 releasem(mp) 637 return 638 } 639 releasem(mp) 640 mp = nil 641 642 if gp := getg(); gp.syncGroup != nil { 643 // Disassociate the G from its synctest bubble while allocating. 644 // This is less elegant than incrementing the group's active count, 645 // but avoids any contamination between GC and synctest. 646 sg := gp.syncGroup 647 gp.syncGroup = nil 648 defer func() { 649 gp.syncGroup = sg 650 }() 651 } 652 653 // Pick up the remaining unswept/not being swept spans concurrently 654 // 655 // This shouldn't happen if we're being invoked in background 656 // mode since proportional sweep should have just finished 657 // sweeping everything, but rounding errors, etc, may leave a 658 // few spans unswept. In forced mode, this is necessary since 659 // GC can be forced at any point in the sweeping cycle. 660 // 661 // We check the transition condition continuously here in case 662 // this G gets delayed in to the next GC cycle. 663 for trigger.test() && sweepone() != ^uintptr(0) { 664 } 665 666 // Perform GC initialization and the sweep termination 667 // transition. 668 semacquire(&work.startSema) 669 // Re-check transition condition under transition lock. 670 if !trigger.test() { 671 semrelease(&work.startSema) 672 return 673 } 674 675 // In gcstoptheworld debug mode, upgrade the mode accordingly. 676 // We do this after re-checking the transition condition so 677 // that multiple goroutines that detect the heap trigger don't 678 // start multiple STW GCs. 679 mode := gcBackgroundMode 680 if debug.gcstoptheworld == 1 { 681 mode = gcForceMode 682 } else if debug.gcstoptheworld == 2 { 683 mode = gcForceBlockMode 684 } 685 686 // Ok, we're doing it! Stop everybody else 687 semacquire(&gcsema) 688 semacquire(&worldsema) 689 690 // For stats, check if this GC was forced by the user. 691 // Update it under gcsema to avoid gctrace getting wrong values. 692 work.userForced = trigger.kind == gcTriggerCycle 693 694 trace := traceAcquire() 695 if trace.ok() { 696 trace.GCStart() 697 traceRelease(trace) 698 } 699 700 // Check that all Ps have finished deferred mcache flushes. 701 for _, p := range allp { 702 if fg := p.mcache.flushGen.Load(); fg != mheap_.sweepgen { 703 println("runtime: p", p.id, "flushGen", fg, "!= sweepgen", mheap_.sweepgen) 704 throw("p mcache not flushed") 705 } 706 } 707 708 gcBgMarkStartWorkers() 709 710 systemstack(gcResetMarkState) 711 712 work.stwprocs, work.maxprocs = gomaxprocs, gomaxprocs 713 if work.stwprocs > ncpu { 714 // This is used to compute CPU time of the STW phases, 715 // so it can't be more than ncpu, even if GOMAXPROCS is. 716 work.stwprocs = ncpu 717 } 718 work.heap0 = gcController.heapLive.Load() 719 work.pauseNS = 0 720 work.mode = mode 721 722 now := nanotime() 723 work.tSweepTerm = now 724 var stw worldStop 725 systemstack(func() { 726 stw = stopTheWorldWithSema(stwGCSweepTerm) 727 }) 728 729 // Accumulate fine-grained stopping time. 730 work.cpuStats.accumulateGCPauseTime(stw.stoppingCPUTime, 1) 731 732 // Finish sweep before we start concurrent scan. 733 systemstack(func() { 734 finishsweep_m() 735 }) 736 737 // clearpools before we start the GC. If we wait the memory will not be 738 // reclaimed until the next GC cycle. 739 clearpools() 740 741 work.cycles.Add(1) 742 743 // Assists and workers can start the moment we start 744 // the world. 745 gcController.startCycle(now, int(gomaxprocs), trigger) 746 747 // Notify the CPU limiter that assists may begin. 748 gcCPULimiter.startGCTransition(true, now) 749 750 // In STW mode, disable scheduling of user Gs. This may also 751 // disable scheduling of this goroutine, so it may block as 752 // soon as we start the world again. 753 if mode != gcBackgroundMode { 754 schedEnableUser(false) 755 } 756 757 // Enter concurrent mark phase and enable 758 // write barriers. 759 // 760 // Because the world is stopped, all Ps will 761 // observe that write barriers are enabled by 762 // the time we start the world and begin 763 // scanning. 764 // 765 // Write barriers must be enabled before assists are 766 // enabled because they must be enabled before 767 // any non-leaf heap objects are marked. Since 768 // allocations are blocked until assists can 769 // happen, we want to enable assists as early as 770 // possible. 771 setGCPhase(_GCmark) 772 773 gcBgMarkPrepare() // Must happen before assists are enabled. 774 gcMarkRootPrepare() 775 776 // Mark all active tinyalloc blocks. Since we're 777 // allocating from these, they need to be black like 778 // other allocations. The alternative is to blacken 779 // the tiny block on every allocation from it, which 780 // would slow down the tiny allocator. 781 gcMarkTinyAllocs() 782 783 // At this point all Ps have enabled the write 784 // barrier, thus maintaining the no white to 785 // black invariant. Enable mutator assists to 786 // put back-pressure on fast allocating 787 // mutators. 788 atomic.Store(&gcBlackenEnabled, 1) 789 790 // In STW mode, we could block the instant systemstack 791 // returns, so make sure we're not preemptible. 792 mp = acquirem() 793 794 // Update the CPU stats pause time. 795 // 796 // Use maxprocs instead of stwprocs here because the total time 797 // computed in the CPU stats is based on maxprocs, and we want them 798 // to be comparable. 799 work.cpuStats.accumulateGCPauseTime(nanotime()-stw.finishedStopping, work.maxprocs) 800 801 // Concurrent mark. 802 systemstack(func() { 803 now = startTheWorldWithSema(0, stw) 804 work.pauseNS += now - stw.startedStopping 805 work.tMark = now 806 807 // Release the CPU limiter. 808 gcCPULimiter.finishGCTransition(now) 809 }) 810 811 // Release the world sema before Gosched() in STW mode 812 // because we will need to reacquire it later but before 813 // this goroutine becomes runnable again, and we could 814 // self-deadlock otherwise. 815 semrelease(&worldsema) 816 releasem(mp) 817 818 // Make sure we block instead of returning to user code 819 // in STW mode. 820 if mode != gcBackgroundMode { 821 Gosched() 822 } 823 824 semrelease(&work.startSema) 825 } 826 827 // gcMarkDoneFlushed counts the number of P's with flushed work. 828 // 829 // Ideally this would be a captured local in gcMarkDone, but forEachP 830 // escapes its callback closure, so it can't capture anything. 831 // 832 // This is protected by markDoneSema. 833 var gcMarkDoneFlushed uint32 834 835 // gcDebugMarkDone contains fields used to debug/test mark termination. 836 var gcDebugMarkDone struct { 837 // spinAfterRaggedBarrier forces gcMarkDone to spin after it executes 838 // the ragged barrier. 839 spinAfterRaggedBarrier atomic.Bool 840 841 // restartedDueTo27993 indicates that we restarted mark termination 842 // due to the bug described in issue #27993. 843 // 844 // Protected by worldsema. 845 restartedDueTo27993 bool 846 } 847 848 // gcMarkDone transitions the GC from mark to mark termination if all 849 // reachable objects have been marked (that is, there are no grey 850 // objects and can be no more in the future). Otherwise, it flushes 851 // all local work to the global queues where it can be discovered by 852 // other workers. 853 // 854 // This should be called when all local mark work has been drained and 855 // there are no remaining workers. Specifically, when 856 // 857 // work.nwait == work.nproc && !gcMarkWorkAvailable(p) 858 // 859 // The calling context must be preemptible. 860 // 861 // Flushing local work is important because idle Ps may have local 862 // work queued. This is the only way to make that work visible and 863 // drive GC to completion. 864 // 865 // It is explicitly okay to have write barriers in this function. If 866 // it does transition to mark termination, then all reachable objects 867 // have been marked, so the write barrier cannot shade any more 868 // objects. 869 func gcMarkDone() { 870 // Ensure only one thread is running the ragged barrier at a 871 // time. 872 semacquire(&work.markDoneSema) 873 874 top: 875 // Re-check transition condition under transition lock. 876 // 877 // It's critical that this checks the global work queues are 878 // empty before performing the ragged barrier. Otherwise, 879 // there could be global work that a P could take after the P 880 // has passed the ragged barrier. 881 if !(gcphase == _GCmark && work.nwait == work.nproc && !gcMarkWorkAvailable(nil)) { 882 semrelease(&work.markDoneSema) 883 return 884 } 885 886 // forEachP needs worldsema to execute, and we'll need it to 887 // stop the world later, so acquire worldsema now. 888 semacquire(&worldsema) 889 890 // Prevent weak->strong conversions from generating additional 891 // GC work. forEachP will guarantee that it is observed globally. 892 work.strongFromWeak.block = true 893 894 // Flush all local buffers and collect flushedWork flags. 895 gcMarkDoneFlushed = 0 896 forEachP(waitReasonGCMarkTermination, func(pp *p) { 897 // Flush the write barrier buffer, since this may add 898 // work to the gcWork. 899 wbBufFlush1(pp) 900 901 // Flush the gcWork, since this may create global work 902 // and set the flushedWork flag. 903 // 904 // TODO(austin): Break up these workbufs to 905 // better distribute work. 906 pp.gcw.dispose() 907 // Collect the flushedWork flag. 908 if pp.gcw.flushedWork { 909 atomic.Xadd(&gcMarkDoneFlushed, 1) 910 pp.gcw.flushedWork = false 911 } 912 }) 913 914 if gcMarkDoneFlushed != 0 { 915 // More grey objects were discovered since the 916 // previous termination check, so there may be more 917 // work to do. Keep going. It's possible the 918 // transition condition became true again during the 919 // ragged barrier, so re-check it. 920 semrelease(&worldsema) 921 goto top 922 } 923 924 // For debugging/testing. 925 for gcDebugMarkDone.spinAfterRaggedBarrier.Load() { 926 } 927 928 // There was no global work, no local work, and no Ps 929 // communicated work since we took markDoneSema. Therefore 930 // there are no grey objects and no more objects can be 931 // shaded. Transition to mark termination. 932 now := nanotime() 933 work.tMarkTerm = now 934 getg().m.preemptoff = "gcing" 935 var stw worldStop 936 systemstack(func() { 937 stw = stopTheWorldWithSema(stwGCMarkTerm) 938 }) 939 // The gcphase is _GCmark, it will transition to _GCmarktermination 940 // below. The important thing is that the wb remains active until 941 // all marking is complete. This includes writes made by the GC. 942 943 // Accumulate fine-grained stopping time. 944 work.cpuStats.accumulateGCPauseTime(stw.stoppingCPUTime, 1) 945 946 // There is sometimes work left over when we enter mark termination due 947 // to write barriers performed after the completion barrier above. 948 // Detect this and resume concurrent mark. This is obviously 949 // unfortunate. 950 // 951 // See issue #27993 for details. 952 // 953 // Switch to the system stack to call wbBufFlush1, though in this case 954 // it doesn't matter because we're non-preemptible anyway. 955 restart := false 956 systemstack(func() { 957 for _, p := range allp { 958 wbBufFlush1(p) 959 if !p.gcw.empty() { 960 restart = true 961 break 962 } 963 } 964 }) 965 if restart { 966 gcDebugMarkDone.restartedDueTo27993 = true 967 968 getg().m.preemptoff = "" 969 systemstack(func() { 970 // Accumulate the time we were stopped before we had to start again. 971 work.cpuStats.accumulateGCPauseTime(nanotime()-stw.finishedStopping, work.maxprocs) 972 973 // Start the world again. 974 now := startTheWorldWithSema(0, stw) 975 work.pauseNS += now - stw.startedStopping 976 }) 977 semrelease(&worldsema) 978 goto top 979 } 980 981 gcComputeStartingStackSize() 982 983 // Disable assists and background workers. We must do 984 // this before waking blocked assists. 985 atomic.Store(&gcBlackenEnabled, 0) 986 987 // Notify the CPU limiter that GC assists will now cease. 988 gcCPULimiter.startGCTransition(false, now) 989 990 // Wake all blocked assists. These will run when we 991 // start the world again. 992 gcWakeAllAssists() 993 994 // Wake all blocked weak->strong conversions. These will run 995 // when we start the world again. 996 work.strongFromWeak.block = false 997 gcWakeAllStrongFromWeak() 998 999 // Likewise, release the transition lock. Blocked 1000 // workers and assists will run when we start the 1001 // world again. 1002 semrelease(&work.markDoneSema) 1003 1004 // In STW mode, re-enable user goroutines. These will be 1005 // queued to run after we start the world. 1006 schedEnableUser(true) 1007 1008 // endCycle depends on all gcWork cache stats being flushed. 1009 // The termination algorithm above ensured that up to 1010 // allocations since the ragged barrier. 1011 gcController.endCycle(now, int(gomaxprocs), work.userForced) 1012 1013 // Perform mark termination. This will restart the world. 1014 gcMarkTermination(stw) 1015 } 1016 1017 // World must be stopped and mark assists and background workers must be 1018 // disabled. 1019 func gcMarkTermination(stw worldStop) { 1020 // Start marktermination (write barrier remains enabled for now). 1021 setGCPhase(_GCmarktermination) 1022 1023 work.heap1 = gcController.heapLive.Load() 1024 startTime := nanotime() 1025 1026 mp := acquirem() 1027 mp.preemptoff = "gcing" 1028 mp.traceback = 2 1029 curgp := mp.curg 1030 // N.B. The execution tracer is not aware of this status 1031 // transition and handles it specially based on the 1032 // wait reason. 1033 casGToWaitingForSuspendG(curgp, _Grunning, waitReasonGarbageCollection) 1034 1035 // Run gc on the g0 stack. We do this so that the g stack 1036 // we're currently running on will no longer change. Cuts 1037 // the root set down a bit (g0 stacks are not scanned, and 1038 // we don't need to scan gc's internal state). We also 1039 // need to switch to g0 so we can shrink the stack. 1040 systemstack(func() { 1041 gcMark(startTime) 1042 // Must return immediately. 1043 // The outer function's stack may have moved 1044 // during gcMark (it shrinks stacks, including the 1045 // outer function's stack), so we must not refer 1046 // to any of its variables. Return back to the 1047 // non-system stack to pick up the new addresses 1048 // before continuing. 1049 }) 1050 1051 var stwSwept bool 1052 systemstack(func() { 1053 work.heap2 = work.bytesMarked 1054 if debug.gccheckmark > 0 { 1055 // Run a full non-parallel, stop-the-world 1056 // mark using checkmark bits, to check that we 1057 // didn't forget to mark anything during the 1058 // concurrent mark process. 1059 startCheckmarks() 1060 gcResetMarkState() 1061 gcw := &getg().m.p.ptr().gcw 1062 gcDrain(gcw, 0) 1063 wbBufFlush1(getg().m.p.ptr()) 1064 gcw.dispose() 1065 endCheckmarks() 1066 } 1067 1068 // marking is complete so we can turn the write barrier off 1069 setGCPhase(_GCoff) 1070 stwSwept = gcSweep(work.mode) 1071 }) 1072 1073 mp.traceback = 0 1074 casgstatus(curgp, _Gwaiting, _Grunning) 1075 1076 trace := traceAcquire() 1077 if trace.ok() { 1078 trace.GCDone() 1079 traceRelease(trace) 1080 } 1081 1082 // all done 1083 mp.preemptoff = "" 1084 1085 if gcphase != _GCoff { 1086 throw("gc done but gcphase != _GCoff") 1087 } 1088 1089 // Record heapInUse for scavenger. 1090 memstats.lastHeapInUse = gcController.heapInUse.load() 1091 1092 // Update GC trigger and pacing, as well as downstream consumers 1093 // of this pacing information, for the next cycle. 1094 systemstack(gcControllerCommit) 1095 1096 // Update timing memstats 1097 now := nanotime() 1098 sec, nsec, _ := time_now() 1099 unixNow := sec*1e9 + int64(nsec) 1100 work.pauseNS += now - stw.startedStopping 1101 work.tEnd = now 1102 atomic.Store64(&memstats.last_gc_unix, uint64(unixNow)) // must be Unix time to make sense to user 1103 atomic.Store64(&memstats.last_gc_nanotime, uint64(now)) // monotonic time for us 1104 memstats.pause_ns[memstats.numgc%uint32(len(memstats.pause_ns))] = uint64(work.pauseNS) 1105 memstats.pause_end[memstats.numgc%uint32(len(memstats.pause_end))] = uint64(unixNow) 1106 memstats.pause_total_ns += uint64(work.pauseNS) 1107 1108 // Accumulate CPU stats. 1109 // 1110 // Use maxprocs instead of stwprocs for GC pause time because the total time 1111 // computed in the CPU stats is based on maxprocs, and we want them to be 1112 // comparable. 1113 // 1114 // Pass gcMarkPhase=true to accumulate so we can get all the latest GC CPU stats 1115 // in there too. 1116 work.cpuStats.accumulateGCPauseTime(now-stw.finishedStopping, work.maxprocs) 1117 work.cpuStats.accumulate(now, true) 1118 1119 // Compute overall GC CPU utilization. 1120 // Omit idle marking time from the overall utilization here since it's "free". 1121 memstats.gc_cpu_fraction = float64(work.cpuStats.GCTotalTime-work.cpuStats.GCIdleTime) / float64(work.cpuStats.TotalTime) 1122 1123 // Reset assist time and background time stats. 1124 // 1125 // Do this now, instead of at the start of the next GC cycle, because 1126 // these two may keep accumulating even if the GC is not active. 1127 scavenge.assistTime.Store(0) 1128 scavenge.backgroundTime.Store(0) 1129 1130 // Reset idle time stat. 1131 sched.idleTime.Store(0) 1132 1133 if work.userForced { 1134 memstats.numforcedgc++ 1135 } 1136 1137 // Bump GC cycle count and wake goroutines waiting on sweep. 1138 lock(&work.sweepWaiters.lock) 1139 memstats.numgc++ 1140 injectglist(&work.sweepWaiters.list) 1141 unlock(&work.sweepWaiters.lock) 1142 1143 // Increment the scavenge generation now. 1144 // 1145 // This moment represents peak heap in use because we're 1146 // about to start sweeping. 1147 mheap_.pages.scav.index.nextGen() 1148 1149 // Release the CPU limiter. 1150 gcCPULimiter.finishGCTransition(now) 1151 1152 // Finish the current heap profiling cycle and start a new 1153 // heap profiling cycle. We do this before starting the world 1154 // so events don't leak into the wrong cycle. 1155 mProf_NextCycle() 1156 1157 // There may be stale spans in mcaches that need to be swept. 1158 // Those aren't tracked in any sweep lists, so we need to 1159 // count them against sweep completion until we ensure all 1160 // those spans have been forced out. 1161 // 1162 // If gcSweep fully swept the heap (for example if the sweep 1163 // is not concurrent due to a GODEBUG setting), then we expect 1164 // the sweepLocker to be invalid, since sweeping is done. 1165 // 1166 // N.B. Below we might duplicate some work from gcSweep; this is 1167 // fine as all that work is idempotent within a GC cycle, and 1168 // we're still holding worldsema so a new cycle can't start. 1169 sl := sweep.active.begin() 1170 if !stwSwept && !sl.valid { 1171 throw("failed to set sweep barrier") 1172 } else if stwSwept && sl.valid { 1173 throw("non-concurrent sweep failed to drain all sweep queues") 1174 } 1175 1176 systemstack(func() { 1177 // The memstats updated above must be updated with the world 1178 // stopped to ensure consistency of some values, such as 1179 // sched.idleTime and sched.totaltime. memstats also include 1180 // the pause time (work,pauseNS), forcing computation of the 1181 // total pause time before the pause actually ends. 1182 // 1183 // Here we reuse the same now for start the world so that the 1184 // time added to /sched/pauses/total/gc:seconds will be 1185 // consistent with the value in memstats. 1186 startTheWorldWithSema(now, stw) 1187 }) 1188 1189 // Flush the heap profile so we can start a new cycle next GC. 1190 // This is relatively expensive, so we don't do it with the 1191 // world stopped. 1192 mProf_Flush() 1193 1194 // Prepare workbufs for freeing by the sweeper. We do this 1195 // asynchronously because it can take non-trivial time. 1196 prepareFreeWorkbufs() 1197 1198 // Free stack spans. This must be done between GC cycles. 1199 systemstack(freeStackSpans) 1200 1201 // Ensure all mcaches are flushed. Each P will flush its own 1202 // mcache before allocating, but idle Ps may not. Since this 1203 // is necessary to sweep all spans, we need to ensure all 1204 // mcaches are flushed before we start the next GC cycle. 1205 // 1206 // While we're here, flush the page cache for idle Ps to avoid 1207 // having pages get stuck on them. These pages are hidden from 1208 // the scavenger, so in small idle heaps a significant amount 1209 // of additional memory might be held onto. 1210 // 1211 // Also, flush the pinner cache, to avoid leaking that memory 1212 // indefinitely. 1213 forEachP(waitReasonFlushProcCaches, func(pp *p) { 1214 pp.mcache.prepareForSweep() 1215 if pp.status == _Pidle { 1216 systemstack(func() { 1217 lock(&mheap_.lock) 1218 pp.pcache.flush(&mheap_.pages) 1219 unlock(&mheap_.lock) 1220 }) 1221 } 1222 pp.pinnerCache = nil 1223 }) 1224 if sl.valid { 1225 // Now that we've swept stale spans in mcaches, they don't 1226 // count against unswept spans. 1227 // 1228 // Note: this sweepLocker may not be valid if sweeping had 1229 // already completed during the STW. See the corresponding 1230 // begin() call that produced sl. 1231 sweep.active.end(sl) 1232 } 1233 1234 // Print gctrace before dropping worldsema. As soon as we drop 1235 // worldsema another cycle could start and smash the stats 1236 // we're trying to print. 1237 if debug.gctrace > 0 { 1238 util := int(memstats.gc_cpu_fraction * 100) 1239 1240 var sbuf [24]byte 1241 printlock() 1242 print("gc ", memstats.numgc, 1243 " @", string(itoaDiv(sbuf[:], uint64(work.tSweepTerm-runtimeInitTime)/1e6, 3)), "s ", 1244 util, "%: ") 1245 prev := work.tSweepTerm 1246 for i, ns := range []int64{work.tMark, work.tMarkTerm, work.tEnd} { 1247 if i != 0 { 1248 print("+") 1249 } 1250 print(string(fmtNSAsMS(sbuf[:], uint64(ns-prev)))) 1251 prev = ns 1252 } 1253 print(" ms clock, ") 1254 for i, ns := range []int64{ 1255 int64(work.stwprocs) * (work.tMark - work.tSweepTerm), 1256 gcController.assistTime.Load(), 1257 gcController.dedicatedMarkTime.Load() + gcController.fractionalMarkTime.Load(), 1258 gcController.idleMarkTime.Load(), 1259 int64(work.stwprocs) * (work.tEnd - work.tMarkTerm), 1260 } { 1261 if i == 2 || i == 3 { 1262 // Separate mark time components with /. 1263 print("/") 1264 } else if i != 0 { 1265 print("+") 1266 } 1267 print(string(fmtNSAsMS(sbuf[:], uint64(ns)))) 1268 } 1269 print(" ms cpu, ", 1270 work.heap0>>20, "->", work.heap1>>20, "->", work.heap2>>20, " MB, ", 1271 gcController.lastHeapGoal>>20, " MB goal, ", 1272 gcController.lastStackScan.Load()>>20, " MB stacks, ", 1273 gcController.globalsScan.Load()>>20, " MB globals, ", 1274 work.maxprocs, " P") 1275 if work.userForced { 1276 print(" (forced)") 1277 } 1278 print("\n") 1279 printunlock() 1280 } 1281 1282 // Set any arena chunks that were deferred to fault. 1283 lock(&userArenaState.lock) 1284 faultList := userArenaState.fault 1285 userArenaState.fault = nil 1286 unlock(&userArenaState.lock) 1287 for _, lc := range faultList { 1288 lc.mspan.setUserArenaChunkToFault() 1289 } 1290 1291 // Enable huge pages on some metadata if we cross a heap threshold. 1292 if gcController.heapGoal() > minHeapForMetadataHugePages { 1293 systemstack(func() { 1294 mheap_.enableMetadataHugePages() 1295 }) 1296 } 1297 1298 semrelease(&worldsema) 1299 semrelease(&gcsema) 1300 // Careful: another GC cycle may start now. 1301 1302 releasem(mp) 1303 mp = nil 1304 1305 // now that gc is done, kick off finalizer thread if needed 1306 if !concurrentSweep { 1307 // give the queued finalizers, if any, a chance to run 1308 Gosched() 1309 } 1310 } 1311 1312 // gcBgMarkStartWorkers prepares background mark worker goroutines. These 1313 // goroutines will not run until the mark phase, but they must be started while 1314 // the work is not stopped and from a regular G stack. The caller must hold 1315 // worldsema. 1316 func gcBgMarkStartWorkers() { 1317 // Background marking is performed by per-P G's. Ensure that each P has 1318 // a background GC G. 1319 // 1320 // Worker Gs don't exit if gomaxprocs is reduced. If it is raised 1321 // again, we can reuse the old workers; no need to create new workers. 1322 if gcBgMarkWorkerCount >= gomaxprocs { 1323 return 1324 } 1325 1326 // Increment mp.locks when allocating. We are called within gcStart, 1327 // and thus must not trigger another gcStart via an allocation. gcStart 1328 // bails when allocating with locks held, so simulate that for these 1329 // allocations. 1330 // 1331 // TODO(prattmic): cleanup gcStart to use a more explicit "in gcStart" 1332 // check for bailing. 1333 mp := acquirem() 1334 ready := make(chan struct{}, 1) 1335 releasem(mp) 1336 1337 for gcBgMarkWorkerCount < gomaxprocs { 1338 mp := acquirem() // See above, we allocate a closure here. 1339 go gcBgMarkWorker(ready) 1340 releasem(mp) 1341 1342 // N.B. we intentionally wait on each goroutine individually 1343 // rather than starting all in a batch and then waiting once 1344 // afterwards. By running one goroutine at a time, we can take 1345 // advantage of runnext to bounce back and forth between 1346 // workers and this goroutine. In an overloaded application, 1347 // this can reduce GC start latency by prioritizing these 1348 // goroutines rather than waiting on the end of the run queue. 1349 <-ready 1350 // The worker is now guaranteed to be added to the pool before 1351 // its P's next findRunnableGCWorker. 1352 1353 gcBgMarkWorkerCount++ 1354 } 1355 } 1356 1357 // gcBgMarkPrepare sets up state for background marking. 1358 // Mutator assists must not yet be enabled. 1359 func gcBgMarkPrepare() { 1360 // Background marking will stop when the work queues are empty 1361 // and there are no more workers (note that, since this is 1362 // concurrent, this may be a transient state, but mark 1363 // termination will clean it up). Between background workers 1364 // and assists, we don't really know how many workers there 1365 // will be, so we pretend to have an arbitrarily large number 1366 // of workers, almost all of which are "waiting". While a 1367 // worker is working it decrements nwait. If nproc == nwait, 1368 // there are no workers. 1369 work.nproc = ^uint32(0) 1370 work.nwait = ^uint32(0) 1371 } 1372 1373 // gcBgMarkWorkerNode is an entry in the gcBgMarkWorkerPool. It points to a single 1374 // gcBgMarkWorker goroutine. 1375 type gcBgMarkWorkerNode struct { 1376 // Unused workers are managed in a lock-free stack. This field must be first. 1377 node lfnode 1378 1379 // The g of this worker. 1380 gp guintptr 1381 1382 // Release this m on park. This is used to communicate with the unlock 1383 // function, which cannot access the G's stack. It is unused outside of 1384 // gcBgMarkWorker(). 1385 m muintptr 1386 } 1387 1388 func gcBgMarkWorker(ready chan struct{}) { 1389 gp := getg() 1390 1391 // We pass node to a gopark unlock function, so it can't be on 1392 // the stack (see gopark). Prevent deadlock from recursively 1393 // starting GC by disabling preemption. 1394 gp.m.preemptoff = "GC worker init" 1395 node := new(gcBgMarkWorkerNode) 1396 gp.m.preemptoff = "" 1397 1398 node.gp.set(gp) 1399 1400 node.m.set(acquirem()) 1401 1402 ready <- struct{}{} 1403 // After this point, the background mark worker is generally scheduled 1404 // cooperatively by gcController.findRunnableGCWorker. While performing 1405 // work on the P, preemption is disabled because we are working on 1406 // P-local work buffers. When the preempt flag is set, this puts itself 1407 // into _Gwaiting to be woken up by gcController.findRunnableGCWorker 1408 // at the appropriate time. 1409 // 1410 // When preemption is enabled (e.g., while in gcMarkDone), this worker 1411 // may be preempted and schedule as a _Grunnable G from a runq. That is 1412 // fine; it will eventually gopark again for further scheduling via 1413 // findRunnableGCWorker. 1414 // 1415 // Since we disable preemption before notifying ready, we guarantee that 1416 // this G will be in the worker pool for the next findRunnableGCWorker. 1417 // This isn't strictly necessary, but it reduces latency between 1418 // _GCmark starting and the workers starting. 1419 1420 for { 1421 // Go to sleep until woken by 1422 // gcController.findRunnableGCWorker. 1423 gopark(func(g *g, nodep unsafe.Pointer) bool { 1424 node := (*gcBgMarkWorkerNode)(nodep) 1425 1426 if mp := node.m.ptr(); mp != nil { 1427 // The worker G is no longer running; release 1428 // the M. 1429 // 1430 // N.B. it is _safe_ to release the M as soon 1431 // as we are no longer performing P-local mark 1432 // work. 1433 // 1434 // However, since we cooperatively stop work 1435 // when gp.preempt is set, if we releasem in 1436 // the loop then the following call to gopark 1437 // would immediately preempt the G. This is 1438 // also safe, but inefficient: the G must 1439 // schedule again only to enter gopark and park 1440 // again. Thus, we defer the release until 1441 // after parking the G. 1442 releasem(mp) 1443 } 1444 1445 // Release this G to the pool. 1446 gcBgMarkWorkerPool.push(&node.node) 1447 // Note that at this point, the G may immediately be 1448 // rescheduled and may be running. 1449 return true 1450 }, unsafe.Pointer(node), waitReasonGCWorkerIdle, traceBlockSystemGoroutine, 0) 1451 1452 // Preemption must not occur here, or another G might see 1453 // p.gcMarkWorkerMode. 1454 1455 // Disable preemption so we can use the gcw. If the 1456 // scheduler wants to preempt us, we'll stop draining, 1457 // dispose the gcw, and then preempt. 1458 node.m.set(acquirem()) 1459 pp := gp.m.p.ptr() // P can't change with preemption disabled. 1460 1461 if gcBlackenEnabled == 0 { 1462 println("worker mode", pp.gcMarkWorkerMode) 1463 throw("gcBgMarkWorker: blackening not enabled") 1464 } 1465 1466 if pp.gcMarkWorkerMode == gcMarkWorkerNotWorker { 1467 throw("gcBgMarkWorker: mode not set") 1468 } 1469 1470 startTime := nanotime() 1471 pp.gcMarkWorkerStartTime = startTime 1472 var trackLimiterEvent bool 1473 if pp.gcMarkWorkerMode == gcMarkWorkerIdleMode { 1474 trackLimiterEvent = pp.limiterEvent.start(limiterEventIdleMarkWork, startTime) 1475 } 1476 1477 decnwait := atomic.Xadd(&work.nwait, -1) 1478 if decnwait == work.nproc { 1479 println("runtime: work.nwait=", decnwait, "work.nproc=", work.nproc) 1480 throw("work.nwait was > work.nproc") 1481 } 1482 1483 systemstack(func() { 1484 // Mark our goroutine preemptible so its stack 1485 // can be scanned or observed by the execution 1486 // tracer. This, for example, lets two mark workers 1487 // scan each other (otherwise, they would 1488 // deadlock). We must not modify anything on 1489 // the G stack. However, stack shrinking is 1490 // disabled for mark workers, so it is safe to 1491 // read from the G stack. 1492 // 1493 // N.B. The execution tracer is not aware of this status 1494 // transition and handles it specially based on the 1495 // wait reason. 1496 casGToWaitingForSuspendG(gp, _Grunning, waitReasonGCWorkerActive) 1497 switch pp.gcMarkWorkerMode { 1498 default: 1499 throw("gcBgMarkWorker: unexpected gcMarkWorkerMode") 1500 case gcMarkWorkerDedicatedMode: 1501 gcDrainMarkWorkerDedicated(&pp.gcw, true) 1502 if gp.preempt { 1503 // We were preempted. This is 1504 // a useful signal to kick 1505 // everything out of the run 1506 // queue so it can run 1507 // somewhere else. 1508 if drainQ, n := runqdrain(pp); n > 0 { 1509 lock(&sched.lock) 1510 globrunqputbatch(&drainQ, int32(n)) 1511 unlock(&sched.lock) 1512 } 1513 } 1514 // Go back to draining, this time 1515 // without preemption. 1516 gcDrainMarkWorkerDedicated(&pp.gcw, false) 1517 case gcMarkWorkerFractionalMode: 1518 gcDrainMarkWorkerFractional(&pp.gcw) 1519 case gcMarkWorkerIdleMode: 1520 gcDrainMarkWorkerIdle(&pp.gcw) 1521 } 1522 casgstatus(gp, _Gwaiting, _Grunning) 1523 }) 1524 1525 // Account for time and mark us as stopped. 1526 now := nanotime() 1527 duration := now - startTime 1528 gcController.markWorkerStop(pp.gcMarkWorkerMode, duration) 1529 if trackLimiterEvent { 1530 pp.limiterEvent.stop(limiterEventIdleMarkWork, now) 1531 } 1532 if pp.gcMarkWorkerMode == gcMarkWorkerFractionalMode { 1533 atomic.Xaddint64(&pp.gcFractionalMarkTime, duration) 1534 } 1535 1536 // Was this the last worker and did we run out 1537 // of work? 1538 incnwait := atomic.Xadd(&work.nwait, +1) 1539 if incnwait > work.nproc { 1540 println("runtime: p.gcMarkWorkerMode=", pp.gcMarkWorkerMode, 1541 "work.nwait=", incnwait, "work.nproc=", work.nproc) 1542 throw("work.nwait > work.nproc") 1543 } 1544 1545 // We'll releasem after this point and thus this P may run 1546 // something else. We must clear the worker mode to avoid 1547 // attributing the mode to a different (non-worker) G in 1548 // traceGoStart. 1549 pp.gcMarkWorkerMode = gcMarkWorkerNotWorker 1550 1551 // If this worker reached a background mark completion 1552 // point, signal the main GC goroutine. 1553 if incnwait == work.nproc && !gcMarkWorkAvailable(nil) { 1554 // We don't need the P-local buffers here, allow 1555 // preemption because we may schedule like a regular 1556 // goroutine in gcMarkDone (block on locks, etc). 1557 releasem(node.m.ptr()) 1558 node.m.set(nil) 1559 1560 gcMarkDone() 1561 } 1562 } 1563 } 1564 1565 // gcMarkWorkAvailable reports whether executing a mark worker 1566 // on p is potentially useful. p may be nil, in which case it only 1567 // checks the global sources of work. 1568 func gcMarkWorkAvailable(p *p) bool { 1569 if p != nil && !p.gcw.empty() { 1570 return true 1571 } 1572 if !work.full.empty() { 1573 return true // global work available 1574 } 1575 if work.markrootNext < work.markrootJobs { 1576 return true // root scan work available 1577 } 1578 return false 1579 } 1580 1581 // gcMark runs the mark (or, for concurrent GC, mark termination) 1582 // All gcWork caches must be empty. 1583 // STW is in effect at this point. 1584 func gcMark(startTime int64) { 1585 if gcphase != _GCmarktermination { 1586 throw("in gcMark expecting to see gcphase as _GCmarktermination") 1587 } 1588 work.tstart = startTime 1589 1590 // Check that there's no marking work remaining. 1591 if work.full != 0 || work.markrootNext < work.markrootJobs { 1592 print("runtime: full=", hex(work.full), " next=", work.markrootNext, " jobs=", work.markrootJobs, " nDataRoots=", work.nDataRoots, " nBSSRoots=", work.nBSSRoots, " nSpanRoots=", work.nSpanRoots, " nStackRoots=", work.nStackRoots, "\n") 1593 panic("non-empty mark queue after concurrent mark") 1594 } 1595 1596 if debug.gccheckmark > 0 { 1597 // This is expensive when there's a large number of 1598 // Gs, so only do it if checkmark is also enabled. 1599 gcMarkRootCheck() 1600 } 1601 1602 // Drop allg snapshot. allgs may have grown, in which case 1603 // this is the only reference to the old backing store and 1604 // there's no need to keep it around. 1605 work.stackRoots = nil 1606 1607 // Clear out buffers and double-check that all gcWork caches 1608 // are empty. This should be ensured by gcMarkDone before we 1609 // enter mark termination. 1610 // 1611 // TODO: We could clear out buffers just before mark if this 1612 // has a non-negligible impact on STW time. 1613 for _, p := range allp { 1614 // The write barrier may have buffered pointers since 1615 // the gcMarkDone barrier. However, since the barrier 1616 // ensured all reachable objects were marked, all of 1617 // these must be pointers to black objects. Hence we 1618 // can just discard the write barrier buffer. 1619 if debug.gccheckmark > 0 { 1620 // For debugging, flush the buffer and make 1621 // sure it really was all marked. 1622 wbBufFlush1(p) 1623 } else { 1624 p.wbBuf.reset() 1625 } 1626 1627 gcw := &p.gcw 1628 if !gcw.empty() { 1629 printlock() 1630 print("runtime: P ", p.id, " flushedWork ", gcw.flushedWork) 1631 if gcw.wbuf1 == nil { 1632 print(" wbuf1=<nil>") 1633 } else { 1634 print(" wbuf1.n=", gcw.wbuf1.nobj) 1635 } 1636 if gcw.wbuf2 == nil { 1637 print(" wbuf2=<nil>") 1638 } else { 1639 print(" wbuf2.n=", gcw.wbuf2.nobj) 1640 } 1641 print("\n") 1642 throw("P has cached GC work at end of mark termination") 1643 } 1644 // There may still be cached empty buffers, which we 1645 // need to flush since we're going to free them. Also, 1646 // there may be non-zero stats because we allocated 1647 // black after the gcMarkDone barrier. 1648 gcw.dispose() 1649 } 1650 1651 // Flush scanAlloc from each mcache since we're about to modify 1652 // heapScan directly. If we were to flush this later, then scanAlloc 1653 // might have incorrect information. 1654 // 1655 // Note that it's not important to retain this information; we know 1656 // exactly what heapScan is at this point via scanWork. 1657 for _, p := range allp { 1658 c := p.mcache 1659 if c == nil { 1660 continue 1661 } 1662 c.scanAlloc = 0 1663 } 1664 1665 // Reset controller state. 1666 gcController.resetLive(work.bytesMarked) 1667 } 1668 1669 // gcSweep must be called on the system stack because it acquires the heap 1670 // lock. See mheap for details. 1671 // 1672 // Returns true if the heap was fully swept by this function. 1673 // 1674 // The world must be stopped. 1675 // 1676 //go:systemstack 1677 func gcSweep(mode gcMode) bool { 1678 assertWorldStopped() 1679 1680 if gcphase != _GCoff { 1681 throw("gcSweep being done but phase is not GCoff") 1682 } 1683 1684 lock(&mheap_.lock) 1685 mheap_.sweepgen += 2 1686 sweep.active.reset() 1687 mheap_.pagesSwept.Store(0) 1688 mheap_.sweepArenas = mheap_.allArenas 1689 mheap_.reclaimIndex.Store(0) 1690 mheap_.reclaimCredit.Store(0) 1691 unlock(&mheap_.lock) 1692 1693 sweep.centralIndex.clear() 1694 1695 if !concurrentSweep || mode == gcForceBlockMode { 1696 // Special case synchronous sweep. 1697 // Record that no proportional sweeping has to happen. 1698 lock(&mheap_.lock) 1699 mheap_.sweepPagesPerByte = 0 1700 unlock(&mheap_.lock) 1701 // Flush all mcaches. 1702 for _, pp := range allp { 1703 pp.mcache.prepareForSweep() 1704 } 1705 // Sweep all spans eagerly. 1706 for sweepone() != ^uintptr(0) { 1707 } 1708 // Free workbufs eagerly. 1709 prepareFreeWorkbufs() 1710 for freeSomeWbufs(false) { 1711 } 1712 // All "free" events for this mark/sweep cycle have 1713 // now happened, so we can make this profile cycle 1714 // available immediately. 1715 mProf_NextCycle() 1716 mProf_Flush() 1717 return true 1718 } 1719 1720 // Background sweep. 1721 lock(&sweep.lock) 1722 if sweep.parked { 1723 sweep.parked = false 1724 ready(sweep.g, 0, true) 1725 } 1726 unlock(&sweep.lock) 1727 return false 1728 } 1729 1730 // gcResetMarkState resets global state prior to marking (concurrent 1731 // or STW) and resets the stack scan state of all Gs. 1732 // 1733 // This is safe to do without the world stopped because any Gs created 1734 // during or after this will start out in the reset state. 1735 // 1736 // gcResetMarkState must be called on the system stack because it acquires 1737 // the heap lock. See mheap for details. 1738 // 1739 //go:systemstack 1740 func gcResetMarkState() { 1741 // This may be called during a concurrent phase, so lock to make sure 1742 // allgs doesn't change. 1743 forEachG(func(gp *g) { 1744 gp.gcscandone = false // set to true in gcphasework 1745 gp.gcAssistBytes = 0 1746 }) 1747 1748 // Clear page marks. This is just 1MB per 64GB of heap, so the 1749 // time here is pretty trivial. 1750 lock(&mheap_.lock) 1751 arenas := mheap_.allArenas 1752 unlock(&mheap_.lock) 1753 for _, ai := range arenas { 1754 ha := mheap_.arenas[ai.l1()][ai.l2()] 1755 clear(ha.pageMarks[:]) 1756 } 1757 1758 work.bytesMarked = 0 1759 work.initialHeapLive = gcController.heapLive.Load() 1760 } 1761 1762 // Hooks for other packages 1763 1764 var poolcleanup func() 1765 var boringCaches []unsafe.Pointer // for crypto/internal/boring 1766 var uniqueMapCleanup chan struct{} // for unique 1767 1768 // sync_runtime_registerPoolCleanup should be an internal detail, 1769 // but widely used packages access it using linkname. 1770 // Notable members of the hall of shame include: 1771 // - github.com/bytedance/gopkg 1772 // - github.com/songzhibin97/gkit 1773 // 1774 // Do not remove or change the type signature. 1775 // See go.dev/issue/67401. 1776 // 1777 //go:linkname sync_runtime_registerPoolCleanup sync.runtime_registerPoolCleanup 1778 func sync_runtime_registerPoolCleanup(f func()) { 1779 poolcleanup = f 1780 } 1781 1782 //go:linkname boring_registerCache crypto/internal/boring/bcache.registerCache 1783 func boring_registerCache(p unsafe.Pointer) { 1784 boringCaches = append(boringCaches, p) 1785 } 1786 1787 //go:linkname unique_runtime_registerUniqueMapCleanup unique.runtime_registerUniqueMapCleanup 1788 func unique_runtime_registerUniqueMapCleanup(f func()) { 1789 // Create the channel on the system stack so it doesn't inherit the current G's 1790 // synctest bubble (if any). 1791 systemstack(func() { 1792 uniqueMapCleanup = make(chan struct{}, 1) 1793 }) 1794 // Start the goroutine in the runtime so it's counted as a system goroutine. 1795 go func(cleanup func()) { 1796 for { 1797 <-uniqueMapCleanup 1798 cleanup() 1799 } 1800 }(f) 1801 } 1802 1803 func clearpools() { 1804 // clear sync.Pools 1805 if poolcleanup != nil { 1806 poolcleanup() 1807 } 1808 1809 // clear boringcrypto caches 1810 for _, p := range boringCaches { 1811 atomicstorep(p, nil) 1812 } 1813 1814 // clear unique maps 1815 if uniqueMapCleanup != nil { 1816 select { 1817 case uniqueMapCleanup <- struct{}{}: 1818 default: 1819 } 1820 } 1821 1822 // Clear central sudog cache. 1823 // Leave per-P caches alone, they have strictly bounded size. 1824 // Disconnect cached list before dropping it on the floor, 1825 // so that a dangling ref to one entry does not pin all of them. 1826 lock(&sched.sudoglock) 1827 var sg, sgnext *sudog 1828 for sg = sched.sudogcache; sg != nil; sg = sgnext { 1829 sgnext = sg.next 1830 sg.next = nil 1831 } 1832 sched.sudogcache = nil 1833 unlock(&sched.sudoglock) 1834 1835 // Clear central defer pool. 1836 // Leave per-P pools alone, they have strictly bounded size. 1837 lock(&sched.deferlock) 1838 // disconnect cached list before dropping it on the floor, 1839 // so that a dangling ref to one entry does not pin all of them. 1840 var d, dlink *_defer 1841 for d = sched.deferpool; d != nil; d = dlink { 1842 dlink = d.link 1843 d.link = nil 1844 } 1845 sched.deferpool = nil 1846 unlock(&sched.deferlock) 1847 } 1848 1849 // Timing 1850 1851 // itoaDiv formats val/(10**dec) into buf. 1852 func itoaDiv(buf []byte, val uint64, dec int) []byte { 1853 i := len(buf) - 1 1854 idec := i - dec 1855 for val >= 10 || i >= idec { 1856 buf[i] = byte(val%10 + '0') 1857 i-- 1858 if i == idec { 1859 buf[i] = '.' 1860 i-- 1861 } 1862 val /= 10 1863 } 1864 buf[i] = byte(val + '0') 1865 return buf[i:] 1866 } 1867 1868 // fmtNSAsMS nicely formats ns nanoseconds as milliseconds. 1869 func fmtNSAsMS(buf []byte, ns uint64) []byte { 1870 if ns >= 10e6 { 1871 // Format as whole milliseconds. 1872 return itoaDiv(buf, ns/1e6, 0) 1873 } 1874 // Format two digits of precision, with at most three decimal places. 1875 x := ns / 1e3 1876 if x == 0 { 1877 buf[0] = '0' 1878 return buf[:1] 1879 } 1880 dec := 3 1881 for x >= 100 { 1882 x /= 10 1883 dec-- 1884 } 1885 return itoaDiv(buf, x, dec) 1886 } 1887 1888 // Helpers for testing GC. 1889 1890 // gcTestMoveStackOnNextCall causes the stack to be moved on a call 1891 // immediately following the call to this. It may not work correctly 1892 // if any other work appears after this call (such as returning). 1893 // Typically the following call should be marked go:noinline so it 1894 // performs a stack check. 1895 // 1896 // In rare cases this may not cause the stack to move, specifically if 1897 // there's a preemption between this call and the next. 1898 func gcTestMoveStackOnNextCall() { 1899 gp := getg() 1900 gp.stackguard0 = stackForceMove 1901 } 1902 1903 // gcTestIsReachable performs a GC and returns a bit set where bit i 1904 // is set if ptrs[i] is reachable. 1905 func gcTestIsReachable(ptrs ...unsafe.Pointer) (mask uint64) { 1906 // This takes the pointers as unsafe.Pointers in order to keep 1907 // them live long enough for us to attach specials. After 1908 // that, we drop our references to them. 1909 1910 if len(ptrs) > 64 { 1911 panic("too many pointers for uint64 mask") 1912 } 1913 1914 // Block GC while we attach specials and drop our references 1915 // to ptrs. Otherwise, if a GC is in progress, it could mark 1916 // them reachable via this function before we have a chance to 1917 // drop them. 1918 semacquire(&gcsema) 1919 1920 // Create reachability specials for ptrs. 1921 specials := make([]*specialReachable, len(ptrs)) 1922 for i, p := range ptrs { 1923 lock(&mheap_.speciallock) 1924 s := (*specialReachable)(mheap_.specialReachableAlloc.alloc()) 1925 unlock(&mheap_.speciallock) 1926 s.special.kind = _KindSpecialReachable 1927 if !addspecial(p, &s.special, false) { 1928 throw("already have a reachable special (duplicate pointer?)") 1929 } 1930 specials[i] = s 1931 // Make sure we don't retain ptrs. 1932 ptrs[i] = nil 1933 } 1934 1935 semrelease(&gcsema) 1936 1937 // Force a full GC and sweep. 1938 GC() 1939 1940 // Process specials. 1941 for i, s := range specials { 1942 if !s.done { 1943 printlock() 1944 println("runtime: object", i, "was not swept") 1945 throw("IsReachable failed") 1946 } 1947 if s.reachable { 1948 mask |= 1 << i 1949 } 1950 lock(&mheap_.speciallock) 1951 mheap_.specialReachableAlloc.free(unsafe.Pointer(s)) 1952 unlock(&mheap_.speciallock) 1953 } 1954 1955 return mask 1956 } 1957 1958 // gcTestPointerClass returns the category of what p points to, one of: 1959 // "heap", "stack", "data", "bss", "other". This is useful for checking 1960 // that a test is doing what it's intended to do. 1961 // 1962 // This is nosplit simply to avoid extra pointer shuffling that may 1963 // complicate a test. 1964 // 1965 //go:nosplit 1966 func gcTestPointerClass(p unsafe.Pointer) string { 1967 p2 := uintptr(noescape(p)) 1968 gp := getg() 1969 if gp.stack.lo <= p2 && p2 < gp.stack.hi { 1970 return "stack" 1971 } 1972 if base, _, _ := findObject(p2, 0, 0); base != 0 { 1973 return "heap" 1974 } 1975 for _, datap := range activeModules() { 1976 if datap.data <= p2 && p2 < datap.edata || datap.noptrdata <= p2 && p2 < datap.enoptrdata { 1977 return "data" 1978 } 1979 if datap.bss <= p2 && p2 < datap.ebss || datap.noptrbss <= p2 && p2 <= datap.enoptrbss { 1980 return "bss" 1981 } 1982 } 1983 KeepAlive(p) 1984 return "other" 1985 } 1986