Source file src/runtime/mgc.go
1 // Copyright 2009 The Go Authors. All rights reserved. 2 // Use of this source code is governed by a BSD-style 3 // license that can be found in the LICENSE file. 4 5 // Garbage collector (GC). 6 // 7 // The GC runs concurrently with mutator threads, is type accurate (aka precise), allows multiple 8 // GC thread to run in parallel. It is a concurrent mark and sweep that uses a write barrier. It is 9 // non-generational and non-compacting. Allocation is done using size segregated per P allocation 10 // areas to minimize fragmentation while eliminating locks in the common case. 11 // 12 // The algorithm decomposes into several steps. 13 // This is a high level description of the algorithm being used. For an overview of GC a good 14 // place to start is Richard Jones' gchandbook.org. 15 // 16 // The algorithm's intellectual heritage includes Dijkstra's on-the-fly algorithm, see 17 // Edsger W. Dijkstra, Leslie Lamport, A. J. Martin, C. S. Scholten, and E. F. M. Steffens. 1978. 18 // On-the-fly garbage collection: an exercise in cooperation. Commun. ACM 21, 11 (November 1978), 19 // 966-975. 20 // For journal quality proofs that these steps are complete, correct, and terminate see 21 // Hudson, R., and Moss, J.E.B. Copying Garbage Collection without stopping the world. 22 // Concurrency and Computation: Practice and Experience 15(3-5), 2003. 23 // 24 // 1. GC performs sweep termination. 25 // 26 // a. Stop the world. This causes all Ps to reach a GC safe-point. 27 // 28 // b. Sweep any unswept spans. There will only be unswept spans if 29 // this GC cycle was forced before the expected time. 30 // 31 // 2. GC performs the mark phase. 32 // 33 // a. Prepare for the mark phase by setting gcphase to _GCmark 34 // (from _GCoff), enabling the write barrier, enabling mutator 35 // assists, and enqueueing root mark jobs. No objects may be 36 // scanned until all Ps have enabled the write barrier, which is 37 // accomplished using STW. 38 // 39 // b. Start the world. From this point, GC work is done by mark 40 // workers started by the scheduler and by assists performed as 41 // part of allocation. The write barrier shades both the 42 // overwritten pointer and the new pointer value for any pointer 43 // writes (see mbarrier.go for details). Newly allocated objects 44 // are immediately marked black. 45 // 46 // c. GC performs root marking jobs. This includes scanning all 47 // stacks, shading all globals, and shading any heap pointers in 48 // off-heap runtime data structures. Scanning a stack stops a 49 // goroutine, shades any pointers found on its stack, and then 50 // resumes the goroutine. 51 // 52 // d. GC drains the work queue of grey objects, scanning each grey 53 // object to black and shading all pointers found in the object 54 // (which in turn may add those pointers to the work queue). 55 // 56 // e. Because GC work is spread across local caches, GC uses a 57 // distributed termination algorithm to detect when there are no 58 // more root marking jobs or grey objects (see gcMarkDone). At this 59 // point, GC transitions to mark termination. 60 // 61 // 3. GC performs mark termination. 62 // 63 // a. Stop the world. 64 // 65 // b. Set gcphase to _GCmarktermination, and disable workers and 66 // assists. 67 // 68 // c. Perform housekeeping like flushing mcaches. 69 // 70 // 4. GC performs the sweep phase. 71 // 72 // a. Prepare for the sweep phase by setting gcphase to _GCoff, 73 // setting up sweep state and disabling the write barrier. 74 // 75 // b. Start the world. From this point on, newly allocated objects 76 // are white, and allocating sweeps spans before use if necessary. 77 // 78 // c. GC does concurrent sweeping in the background and in response 79 // to allocation. See description below. 80 // 81 // 5. When sufficient allocation has taken place, replay the sequence 82 // starting with 1 above. See discussion of GC rate below. 83 84 // Concurrent sweep. 85 // 86 // The sweep phase proceeds concurrently with normal program execution. 87 // The heap is swept span-by-span both lazily (when a goroutine needs another span) 88 // and concurrently in a background goroutine (this helps programs that are not CPU bound). 89 // At the end of STW mark termination all spans are marked as "needs sweeping". 90 // 91 // The background sweeper goroutine simply sweeps spans one-by-one. 92 // 93 // To avoid requesting more OS memory while there are unswept spans, when a 94 // goroutine needs another span, it first attempts to reclaim that much memory 95 // by sweeping. When a goroutine needs to allocate a new small-object span, it 96 // sweeps small-object spans for the same object size until it frees at least 97 // one object. When a goroutine needs to allocate large-object span from heap, 98 // it sweeps spans until it frees at least that many pages into heap. There is 99 // one case where this may not suffice: if a goroutine sweeps and frees two 100 // nonadjacent one-page spans to the heap, it will allocate a new two-page 101 // span, but there can still be other one-page unswept spans which could be 102 // combined into a two-page span. 103 // 104 // It's critical to ensure that no operations proceed on unswept spans (that would corrupt 105 // mark bits in GC bitmap). During GC all mcaches are flushed into the central cache, 106 // so they are empty. When a goroutine grabs a new span into mcache, it sweeps it. 107 // When a goroutine explicitly frees an object or sets a finalizer, it ensures that 108 // the span is swept (either by sweeping it, or by waiting for the concurrent sweep to finish). 109 // The finalizer goroutine is kicked off only when all spans are swept. 110 // When the next GC starts, it sweeps all not-yet-swept spans (if any). 111 112 // GC rate. 113 // Next GC is after we've allocated an extra amount of memory proportional to 114 // the amount already in use. The proportion is controlled by GOGC environment variable 115 // (100 by default). If GOGC=100 and we're using 4M, we'll GC again when we get to 8M 116 // (this mark is computed by the gcController.heapGoal method). This keeps the GC cost in 117 // linear proportion to the allocation cost. Adjusting GOGC just changes the linear constant 118 // (and also the amount of extra memory used). 119 120 // Oblets 121 // 122 // In order to prevent long pauses while scanning large objects and to 123 // improve parallelism, the garbage collector breaks up scan jobs for 124 // objects larger than maxObletBytes into "oblets" of at most 125 // maxObletBytes. When scanning encounters the beginning of a large 126 // object, it scans only the first oblet and enqueues the remaining 127 // oblets as new scan jobs. 128 129 package runtime 130 131 import ( 132 "internal/cpu" 133 "internal/runtime/atomic" 134 "unsafe" 135 ) 136 137 const ( 138 _DebugGC = 0 139 _FinBlockSize = 4 * 1024 140 141 // concurrentSweep is a debug flag. Disabling this flag 142 // ensures all spans are swept while the world is stopped. 143 concurrentSweep = true 144 145 // debugScanConservative enables debug logging for stack 146 // frames that are scanned conservatively. 147 debugScanConservative = false 148 149 // sweepMinHeapDistance is a lower bound on the heap distance 150 // (in bytes) reserved for concurrent sweeping between GC 151 // cycles. 152 sweepMinHeapDistance = 1024 * 1024 153 ) 154 155 // heapObjectsCanMove always returns false in the current garbage collector. 156 // It exists for go4.org/unsafe/assume-no-moving-gc, which is an 157 // unfortunate idea that had an even more unfortunate implementation. 158 // Every time a new Go release happened, the package stopped building, 159 // and the authors had to add a new file with a new //go:build line, and 160 // then the entire ecosystem of packages with that as a dependency had to 161 // explicitly update to the new version. Many packages depend on 162 // assume-no-moving-gc transitively, through paths like 163 // inet.af/netaddr -> go4.org/intern -> assume-no-moving-gc. 164 // This was causing a significant amount of friction around each new 165 // release, so we added this bool for the package to //go:linkname 166 // instead. The bool is still unfortunate, but it's not as bad as 167 // breaking the ecosystem on every new release. 168 // 169 // If the Go garbage collector ever does move heap objects, we can set 170 // this to true to break all the programs using assume-no-moving-gc. 171 // 172 //go:linkname heapObjectsCanMove 173 func heapObjectsCanMove() bool { 174 return false 175 } 176 177 func gcinit() { 178 if unsafe.Sizeof(workbuf{}) != _WorkbufSize { 179 throw("size of Workbuf is suboptimal") 180 } 181 // No sweep on the first cycle. 182 sweep.active.state.Store(sweepDrainedMask) 183 184 // Initialize GC pacer state. 185 // Use the environment variable GOGC for the initial gcPercent value. 186 // Use the environment variable GOMEMLIMIT for the initial memoryLimit value. 187 gcController.init(readGOGC(), readGOMEMLIMIT()) 188 189 work.startSema = 1 190 work.markDoneSema = 1 191 lockInit(&work.sweepWaiters.lock, lockRankSweepWaiters) 192 lockInit(&work.assistQueue.lock, lockRankAssistQueue) 193 lockInit(&work.strongFromWeak.lock, lockRankStrongFromWeakQueue) 194 lockInit(&work.wbufSpans.lock, lockRankWbufSpans) 195 } 196 197 // gcenable is called after the bulk of the runtime initialization, 198 // just before we're about to start letting user code run. 199 // It kicks off the background sweeper goroutine, the background 200 // scavenger goroutine, and enables GC. 201 func gcenable() { 202 // Kick off sweeping and scavenging. 203 c := make(chan int, 2) 204 go bgsweep(c) 205 go bgscavenge(c) 206 <-c 207 <-c 208 memstats.enablegc = true // now that runtime is initialized, GC is okay 209 } 210 211 // Garbage collector phase. 212 // Indicates to write barrier and synchronization task to perform. 213 var gcphase uint32 214 215 // The compiler knows about this variable. 216 // If you change it, you must change builtin/runtime.go, too. 217 // If you change the first four bytes, you must also change the write 218 // barrier insertion code. 219 // 220 // writeBarrier should be an internal detail, 221 // but widely used packages access it using linkname. 222 // Notable members of the hall of shame include: 223 // - github.com/bytedance/sonic 224 // 225 // Do not remove or change the type signature. 226 // See go.dev/issue/67401. 227 // 228 //go:linkname writeBarrier 229 var writeBarrier struct { 230 enabled bool // compiler emits a check of this before calling write barrier 231 pad [3]byte // compiler uses 32-bit load for "enabled" field 232 alignme uint64 // guarantee alignment so that compiler can use a 32 or 64-bit load 233 } 234 235 // gcBlackenEnabled is 1 if mutator assists and background mark 236 // workers are allowed to blacken objects. This must only be set when 237 // gcphase == _GCmark. 238 var gcBlackenEnabled uint32 239 240 const ( 241 _GCoff = iota // GC not running; sweeping in background, write barrier disabled 242 _GCmark // GC marking roots and workbufs: allocate black, write barrier ENABLED 243 _GCmarktermination // GC mark termination: allocate black, P's help GC, write barrier ENABLED 244 ) 245 246 //go:nosplit 247 func setGCPhase(x uint32) { 248 atomic.Store(&gcphase, x) 249 writeBarrier.enabled = gcphase == _GCmark || gcphase == _GCmarktermination 250 } 251 252 // gcMarkWorkerMode represents the mode that a concurrent mark worker 253 // should operate in. 254 // 255 // Concurrent marking happens through four different mechanisms. One 256 // is mutator assists, which happen in response to allocations and are 257 // not scheduled. The other three are variations in the per-P mark 258 // workers and are distinguished by gcMarkWorkerMode. 259 type gcMarkWorkerMode int 260 261 const ( 262 // gcMarkWorkerNotWorker indicates that the next scheduled G is not 263 // starting work and the mode should be ignored. 264 gcMarkWorkerNotWorker gcMarkWorkerMode = iota 265 266 // gcMarkWorkerDedicatedMode indicates that the P of a mark 267 // worker is dedicated to running that mark worker. The mark 268 // worker should run without preemption. 269 gcMarkWorkerDedicatedMode 270 271 // gcMarkWorkerFractionalMode indicates that a P is currently 272 // running the "fractional" mark worker. The fractional worker 273 // is necessary when GOMAXPROCS*gcBackgroundUtilization is not 274 // an integer and using only dedicated workers would result in 275 // utilization too far from the target of gcBackgroundUtilization. 276 // The fractional worker should run until it is preempted and 277 // will be scheduled to pick up the fractional part of 278 // GOMAXPROCS*gcBackgroundUtilization. 279 gcMarkWorkerFractionalMode 280 281 // gcMarkWorkerIdleMode indicates that a P is running the mark 282 // worker because it has nothing else to do. The idle worker 283 // should run until it is preempted and account its time 284 // against gcController.idleMarkTime. 285 gcMarkWorkerIdleMode 286 ) 287 288 // gcMarkWorkerModeStrings are the strings labels of gcMarkWorkerModes 289 // to use in execution traces. 290 var gcMarkWorkerModeStrings = [...]string{ 291 "Not worker", 292 "GC (dedicated)", 293 "GC (fractional)", 294 "GC (idle)", 295 } 296 297 // pollFractionalWorkerExit reports whether a fractional mark worker 298 // should self-preempt. It assumes it is called from the fractional 299 // worker. 300 func pollFractionalWorkerExit() bool { 301 // This should be kept in sync with the fractional worker 302 // scheduler logic in findRunnableGCWorker. 303 now := nanotime() 304 delta := now - gcController.markStartTime 305 if delta <= 0 { 306 return true 307 } 308 p := getg().m.p.ptr() 309 selfTime := p.gcFractionalMarkTime + (now - p.gcMarkWorkerStartTime) 310 // Add some slack to the utilization goal so that the 311 // fractional worker isn't behind again the instant it exits. 312 return float64(selfTime)/float64(delta) > 1.2*gcController.fractionalUtilizationGoal 313 } 314 315 var work workType 316 317 type workType struct { 318 full lfstack // lock-free list of full blocks workbuf 319 _ cpu.CacheLinePad // prevents false-sharing between full and empty 320 empty lfstack // lock-free list of empty blocks workbuf 321 _ cpu.CacheLinePad // prevents false-sharing between empty and nproc/nwait 322 323 wbufSpans struct { 324 lock mutex 325 // free is a list of spans dedicated to workbufs, but 326 // that don't currently contain any workbufs. 327 free mSpanList 328 // busy is a list of all spans containing workbufs on 329 // one of the workbuf lists. 330 busy mSpanList 331 } 332 333 // Restore 64-bit alignment on 32-bit. 334 _ uint32 335 336 // bytesMarked is the number of bytes marked this cycle. This 337 // includes bytes blackened in scanned objects, noscan objects 338 // that go straight to black, objects allocated as black during 339 // the cycle, and permagrey objects scanned by markroot during 340 // the concurrent scan phase. 341 // 342 // This is updated atomically during the cycle. Updates may be batched 343 // arbitrarily, since the value is only read at the end of the cycle. 344 // 345 // Because of benign races during marking, this number may not 346 // be the exact number of marked bytes, but it should be very 347 // close. 348 // 349 // Put this field here because it needs 64-bit atomic access 350 // (and thus 8-byte alignment even on 32-bit architectures). 351 bytesMarked uint64 352 353 markrootNext uint32 // next markroot job 354 markrootJobs uint32 // number of markroot jobs 355 356 nproc uint32 357 tstart int64 358 nwait uint32 359 360 // Number of roots of various root types. Set by gcMarkRootPrepare. 361 // 362 // nStackRoots == len(stackRoots), but we have nStackRoots for 363 // consistency. 364 nDataRoots, nBSSRoots, nSpanRoots, nStackRoots int 365 366 // Base indexes of each root type. Set by gcMarkRootPrepare. 367 baseData, baseBSS, baseSpans, baseStacks, baseEnd uint32 368 369 // stackRoots is a snapshot of all of the Gs that existed 370 // before the beginning of concurrent marking. The backing 371 // store of this must not be modified because it might be 372 // shared with allgs. 373 stackRoots []*g 374 375 // Each type of GC state transition is protected by a lock. 376 // Since multiple threads can simultaneously detect the state 377 // transition condition, any thread that detects a transition 378 // condition must acquire the appropriate transition lock, 379 // re-check the transition condition and return if it no 380 // longer holds or perform the transition if it does. 381 // Likewise, any transition must invalidate the transition 382 // condition before releasing the lock. This ensures that each 383 // transition is performed by exactly one thread and threads 384 // that need the transition to happen block until it has 385 // happened. 386 // 387 // startSema protects the transition from "off" to mark or 388 // mark termination. 389 startSema uint32 390 // markDoneSema protects transitions from mark to mark termination. 391 markDoneSema uint32 392 393 bgMarkDone uint32 // cas to 1 when at a background mark completion point 394 // Background mark completion signaling 395 396 // mode is the concurrency mode of the current GC cycle. 397 mode gcMode 398 399 // userForced indicates the current GC cycle was forced by an 400 // explicit user call. 401 userForced bool 402 403 // initialHeapLive is the value of gcController.heapLive at the 404 // beginning of this GC cycle. 405 initialHeapLive uint64 406 407 // assistQueue is a queue of assists that are blocked because 408 // there was neither enough credit to steal or enough work to 409 // do. 410 assistQueue struct { 411 lock mutex 412 q gQueue 413 } 414 415 // sweepWaiters is a list of blocked goroutines to wake when 416 // we transition from mark termination to sweep. 417 sweepWaiters struct { 418 lock mutex 419 list gList 420 } 421 422 // strongFromWeak controls how the GC interacts with weak->strong 423 // pointer conversions. 424 strongFromWeak struct { 425 // block is a flag set during mark termination that prevents 426 // new weak->strong conversions from executing by blocking the 427 // goroutine and enqueuing it onto q. 428 // 429 // Mutated only by one goroutine at a time in gcMarkDone, 430 // with globally-synchronizing events like forEachP and 431 // stopTheWorld. 432 block bool 433 434 // q is a queue of goroutines that attempted to perform a 435 // weak->strong conversion during mark termination. 436 // 437 // Protected by lock. 438 lock mutex 439 q gQueue 440 } 441 442 // cycles is the number of completed GC cycles, where a GC 443 // cycle is sweep termination, mark, mark termination, and 444 // sweep. This differs from memstats.numgc, which is 445 // incremented at mark termination. 446 cycles atomic.Uint32 447 448 // Timing/utilization stats for this cycle. 449 stwprocs, maxprocs int32 450 tSweepTerm, tMark, tMarkTerm, tEnd int64 // nanotime() of phase start 451 452 // pauseNS is the total STW time this cycle, measured as the time between 453 // when stopping began (just before trying to stop Ps) and just after the 454 // world started again. 455 pauseNS int64 456 457 // debug.gctrace heap sizes for this cycle. 458 heap0, heap1, heap2 uint64 459 460 // Cumulative estimated CPU usage. 461 cpuStats 462 } 463 464 // GC runs a garbage collection and blocks the caller until the 465 // garbage collection is complete. It may also block the entire 466 // program. 467 func GC() { 468 // We consider a cycle to be: sweep termination, mark, mark 469 // termination, and sweep. This function shouldn't return 470 // until a full cycle has been completed, from beginning to 471 // end. Hence, we always want to finish up the current cycle 472 // and start a new one. That means: 473 // 474 // 1. In sweep termination, mark, or mark termination of cycle 475 // N, wait until mark termination N completes and transitions 476 // to sweep N. 477 // 478 // 2. In sweep N, help with sweep N. 479 // 480 // At this point we can begin a full cycle N+1. 481 // 482 // 3. Trigger cycle N+1 by starting sweep termination N+1. 483 // 484 // 4. Wait for mark termination N+1 to complete. 485 // 486 // 5. Help with sweep N+1 until it's done. 487 // 488 // This all has to be written to deal with the fact that the 489 // GC may move ahead on its own. For example, when we block 490 // until mark termination N, we may wake up in cycle N+2. 491 492 // Wait until the current sweep termination, mark, and mark 493 // termination complete. 494 n := work.cycles.Load() 495 gcWaitOnMark(n) 496 497 // We're now in sweep N or later. Trigger GC cycle N+1, which 498 // will first finish sweep N if necessary and then enter sweep 499 // termination N+1. 500 gcStart(gcTrigger{kind: gcTriggerCycle, n: n + 1}) 501 502 // Wait for mark termination N+1 to complete. 503 gcWaitOnMark(n + 1) 504 505 // Finish sweep N+1 before returning. We do this both to 506 // complete the cycle and because runtime.GC() is often used 507 // as part of tests and benchmarks to get the system into a 508 // relatively stable and isolated state. 509 for work.cycles.Load() == n+1 && sweepone() != ^uintptr(0) { 510 Gosched() 511 } 512 513 // Callers may assume that the heap profile reflects the 514 // just-completed cycle when this returns (historically this 515 // happened because this was a STW GC), but right now the 516 // profile still reflects mark termination N, not N+1. 517 // 518 // As soon as all of the sweep frees from cycle N+1 are done, 519 // we can go ahead and publish the heap profile. 520 // 521 // First, wait for sweeping to finish. (We know there are no 522 // more spans on the sweep queue, but we may be concurrently 523 // sweeping spans, so we have to wait.) 524 for work.cycles.Load() == n+1 && !isSweepDone() { 525 Gosched() 526 } 527 528 // Now we're really done with sweeping, so we can publish the 529 // stable heap profile. Only do this if we haven't already hit 530 // another mark termination. 531 mp := acquirem() 532 cycle := work.cycles.Load() 533 if cycle == n+1 || (gcphase == _GCmark && cycle == n+2) { 534 mProf_PostSweep() 535 } 536 releasem(mp) 537 } 538 539 // gcWaitOnMark blocks until GC finishes the Nth mark phase. If GC has 540 // already completed this mark phase, it returns immediately. 541 func gcWaitOnMark(n uint32) { 542 for { 543 // Disable phase transitions. 544 lock(&work.sweepWaiters.lock) 545 nMarks := work.cycles.Load() 546 if gcphase != _GCmark { 547 // We've already completed this cycle's mark. 548 nMarks++ 549 } 550 if nMarks > n { 551 // We're done. 552 unlock(&work.sweepWaiters.lock) 553 return 554 } 555 556 // Wait until sweep termination, mark, and mark 557 // termination of cycle N complete. 558 work.sweepWaiters.list.push(getg()) 559 goparkunlock(&work.sweepWaiters.lock, waitReasonWaitForGCCycle, traceBlockUntilGCEnds, 1) 560 } 561 } 562 563 // gcMode indicates how concurrent a GC cycle should be. 564 type gcMode int 565 566 const ( 567 gcBackgroundMode gcMode = iota // concurrent GC and sweep 568 gcForceMode // stop-the-world GC now, concurrent sweep 569 gcForceBlockMode // stop-the-world GC now and STW sweep (forced by user) 570 ) 571 572 // A gcTrigger is a predicate for starting a GC cycle. Specifically, 573 // it is an exit condition for the _GCoff phase. 574 type gcTrigger struct { 575 kind gcTriggerKind 576 now int64 // gcTriggerTime: current time 577 n uint32 // gcTriggerCycle: cycle number to start 578 } 579 580 type gcTriggerKind int 581 582 const ( 583 // gcTriggerHeap indicates that a cycle should be started when 584 // the heap size reaches the trigger heap size computed by the 585 // controller. 586 gcTriggerHeap gcTriggerKind = iota 587 588 // gcTriggerTime indicates that a cycle should be started when 589 // it's been more than forcegcperiod nanoseconds since the 590 // previous GC cycle. 591 gcTriggerTime 592 593 // gcTriggerCycle indicates that a cycle should be started if 594 // we have not yet started cycle number gcTrigger.n (relative 595 // to work.cycles). 596 gcTriggerCycle 597 ) 598 599 // test reports whether the trigger condition is satisfied, meaning 600 // that the exit condition for the _GCoff phase has been met. The exit 601 // condition should be tested when allocating. 602 func (t gcTrigger) test() bool { 603 if !memstats.enablegc || panicking.Load() != 0 || gcphase != _GCoff { 604 return false 605 } 606 switch t.kind { 607 case gcTriggerHeap: 608 trigger, _ := gcController.trigger() 609 return gcController.heapLive.Load() >= trigger 610 case gcTriggerTime: 611 if gcController.gcPercent.Load() < 0 { 612 return false 613 } 614 lastgc := int64(atomic.Load64(&memstats.last_gc_nanotime)) 615 return lastgc != 0 && t.now-lastgc > forcegcperiod 616 case gcTriggerCycle: 617 // t.n > work.cycles, but accounting for wraparound. 618 return int32(t.n-work.cycles.Load()) > 0 619 } 620 return true 621 } 622 623 // gcStart starts the GC. It transitions from _GCoff to _GCmark (if 624 // debug.gcstoptheworld == 0) or performs all of GC (if 625 // debug.gcstoptheworld != 0). 626 // 627 // This may return without performing this transition in some cases, 628 // such as when called on a system stack or with locks held. 629 func gcStart(trigger gcTrigger) { 630 // Since this is called from malloc and malloc is called in 631 // the guts of a number of libraries that might be holding 632 // locks, don't attempt to start GC in non-preemptible or 633 // potentially unstable situations. 634 mp := acquirem() 635 if gp := getg(); gp == mp.g0 || mp.locks > 1 || mp.preemptoff != "" { 636 releasem(mp) 637 return 638 } 639 releasem(mp) 640 mp = nil 641 642 if gp := getg(); gp.syncGroup != nil { 643 // Disassociate the G from its synctest bubble while allocating. 644 // This is less elegant than incrementing the group's active count, 645 // but avoids any contamination between GC and synctest. 646 sg := gp.syncGroup 647 gp.syncGroup = nil 648 defer func() { 649 gp.syncGroup = sg 650 }() 651 } 652 653 // Pick up the remaining unswept/not being swept spans concurrently 654 // 655 // This shouldn't happen if we're being invoked in background 656 // mode since proportional sweep should have just finished 657 // sweeping everything, but rounding errors, etc, may leave a 658 // few spans unswept. In forced mode, this is necessary since 659 // GC can be forced at any point in the sweeping cycle. 660 // 661 // We check the transition condition continuously here in case 662 // this G gets delayed in to the next GC cycle. 663 for trigger.test() && sweepone() != ^uintptr(0) { 664 } 665 666 // Perform GC initialization and the sweep termination 667 // transition. 668 semacquire(&work.startSema) 669 // Re-check transition condition under transition lock. 670 if !trigger.test() { 671 semrelease(&work.startSema) 672 return 673 } 674 675 // In gcstoptheworld debug mode, upgrade the mode accordingly. 676 // We do this after re-checking the transition condition so 677 // that multiple goroutines that detect the heap trigger don't 678 // start multiple STW GCs. 679 mode := gcBackgroundMode 680 if debug.gcstoptheworld == 1 { 681 mode = gcForceMode 682 } else if debug.gcstoptheworld == 2 { 683 mode = gcForceBlockMode 684 } 685 686 // Ok, we're doing it! Stop everybody else 687 semacquire(&gcsema) 688 semacquire(&worldsema) 689 690 // For stats, check if this GC was forced by the user. 691 // Update it under gcsema to avoid gctrace getting wrong values. 692 work.userForced = trigger.kind == gcTriggerCycle 693 694 trace := traceAcquire() 695 if trace.ok() { 696 trace.GCStart() 697 traceRelease(trace) 698 } 699 700 // Check that all Ps have finished deferred mcache flushes. 701 for _, p := range allp { 702 if fg := p.mcache.flushGen.Load(); fg != mheap_.sweepgen { 703 println("runtime: p", p.id, "flushGen", fg, "!= sweepgen", mheap_.sweepgen) 704 throw("p mcache not flushed") 705 } 706 } 707 708 gcBgMarkStartWorkers() 709 710 systemstack(gcResetMarkState) 711 712 work.stwprocs, work.maxprocs = gomaxprocs, gomaxprocs 713 if work.stwprocs > ncpu { 714 // This is used to compute CPU time of the STW phases, 715 // so it can't be more than ncpu, even if GOMAXPROCS is. 716 work.stwprocs = ncpu 717 } 718 work.heap0 = gcController.heapLive.Load() 719 work.pauseNS = 0 720 work.mode = mode 721 722 now := nanotime() 723 work.tSweepTerm = now 724 var stw worldStop 725 systemstack(func() { 726 stw = stopTheWorldWithSema(stwGCSweepTerm) 727 }) 728 729 // Accumulate fine-grained stopping time. 730 work.cpuStats.accumulateGCPauseTime(stw.stoppingCPUTime, 1) 731 732 // Finish sweep before we start concurrent scan. 733 systemstack(func() { 734 finishsweep_m() 735 }) 736 737 // clearpools before we start the GC. If we wait the memory will not be 738 // reclaimed until the next GC cycle. 739 clearpools() 740 741 work.cycles.Add(1) 742 743 // Assists and workers can start the moment we start 744 // the world. 745 gcController.startCycle(now, int(gomaxprocs), trigger) 746 747 // Notify the CPU limiter that assists may begin. 748 gcCPULimiter.startGCTransition(true, now) 749 750 // In STW mode, disable scheduling of user Gs. This may also 751 // disable scheduling of this goroutine, so it may block as 752 // soon as we start the world again. 753 if mode != gcBackgroundMode { 754 schedEnableUser(false) 755 } 756 757 // Enter concurrent mark phase and enable 758 // write barriers. 759 // 760 // Because the world is stopped, all Ps will 761 // observe that write barriers are enabled by 762 // the time we start the world and begin 763 // scanning. 764 // 765 // Write barriers must be enabled before assists are 766 // enabled because they must be enabled before 767 // any non-leaf heap objects are marked. Since 768 // allocations are blocked until assists can 769 // happen, we want to enable assists as early as 770 // possible. 771 setGCPhase(_GCmark) 772 773 gcBgMarkPrepare() // Must happen before assists are enabled. 774 gcMarkRootPrepare() 775 776 // Mark all active tinyalloc blocks. Since we're 777 // allocating from these, they need to be black like 778 // other allocations. The alternative is to blacken 779 // the tiny block on every allocation from it, which 780 // would slow down the tiny allocator. 781 gcMarkTinyAllocs() 782 783 // At this point all Ps have enabled the write 784 // barrier, thus maintaining the no white to 785 // black invariant. Enable mutator assists to 786 // put back-pressure on fast allocating 787 // mutators. 788 atomic.Store(&gcBlackenEnabled, 1) 789 790 // In STW mode, we could block the instant systemstack 791 // returns, so make sure we're not preemptible. 792 mp = acquirem() 793 794 // Update the CPU stats pause time. 795 // 796 // Use maxprocs instead of stwprocs here because the total time 797 // computed in the CPU stats is based on maxprocs, and we want them 798 // to be comparable. 799 work.cpuStats.accumulateGCPauseTime(nanotime()-stw.finishedStopping, work.maxprocs) 800 801 // Concurrent mark. 802 systemstack(func() { 803 now = startTheWorldWithSema(0, stw) 804 work.pauseNS += now - stw.startedStopping 805 work.tMark = now 806 807 // Release the CPU limiter. 808 gcCPULimiter.finishGCTransition(now) 809 }) 810 811 // Release the world sema before Gosched() in STW mode 812 // because we will need to reacquire it later but before 813 // this goroutine becomes runnable again, and we could 814 // self-deadlock otherwise. 815 semrelease(&worldsema) 816 releasem(mp) 817 818 // Make sure we block instead of returning to user code 819 // in STW mode. 820 if mode != gcBackgroundMode { 821 Gosched() 822 } 823 824 semrelease(&work.startSema) 825 } 826 827 // gcMarkDoneFlushed counts the number of P's with flushed work. 828 // 829 // Ideally this would be a captured local in gcMarkDone, but forEachP 830 // escapes its callback closure, so it can't capture anything. 831 // 832 // This is protected by markDoneSema. 833 var gcMarkDoneFlushed uint32 834 835 // gcDebugMarkDone contains fields used to debug/test mark termination. 836 var gcDebugMarkDone struct { 837 // spinAfterRaggedBarrier forces gcMarkDone to spin after it executes 838 // the ragged barrier. 839 spinAfterRaggedBarrier atomic.Bool 840 841 // restartedDueTo27993 indicates that we restarted mark termination 842 // due to the bug described in issue #27993. 843 // 844 // Protected by worldsema. 845 restartedDueTo27993 bool 846 } 847 848 // gcMarkDone transitions the GC from mark to mark termination if all 849 // reachable objects have been marked (that is, there are no grey 850 // objects and can be no more in the future). Otherwise, it flushes 851 // all local work to the global queues where it can be discovered by 852 // other workers. 853 // 854 // This should be called when all local mark work has been drained and 855 // there are no remaining workers. Specifically, when 856 // 857 // work.nwait == work.nproc && !gcMarkWorkAvailable(p) 858 // 859 // The calling context must be preemptible. 860 // 861 // Flushing local work is important because idle Ps may have local 862 // work queued. This is the only way to make that work visible and 863 // drive GC to completion. 864 // 865 // It is explicitly okay to have write barriers in this function. If 866 // it does transition to mark termination, then all reachable objects 867 // have been marked, so the write barrier cannot shade any more 868 // objects. 869 func gcMarkDone() { 870 // Ensure only one thread is running the ragged barrier at a 871 // time. 872 semacquire(&work.markDoneSema) 873 874 top: 875 // Re-check transition condition under transition lock. 876 // 877 // It's critical that this checks the global work queues are 878 // empty before performing the ragged barrier. Otherwise, 879 // there could be global work that a P could take after the P 880 // has passed the ragged barrier. 881 if !(gcphase == _GCmark && work.nwait == work.nproc && !gcMarkWorkAvailable(nil)) { 882 semrelease(&work.markDoneSema) 883 return 884 } 885 886 // forEachP needs worldsema to execute, and we'll need it to 887 // stop the world later, so acquire worldsema now. 888 semacquire(&worldsema) 889 890 // Prevent weak->strong conversions from generating additional 891 // GC work. forEachP will guarantee that it is observed globally. 892 work.strongFromWeak.block = true 893 894 // Flush all local buffers and collect flushedWork flags. 895 gcMarkDoneFlushed = 0 896 forEachP(waitReasonGCMarkTermination, func(pp *p) { 897 // Flush the write barrier buffer, since this may add 898 // work to the gcWork. 899 wbBufFlush1(pp) 900 901 // Flush the gcWork, since this may create global work 902 // and set the flushedWork flag. 903 // 904 // TODO(austin): Break up these workbufs to 905 // better distribute work. 906 pp.gcw.dispose() 907 // Collect the flushedWork flag. 908 if pp.gcw.flushedWork { 909 atomic.Xadd(&gcMarkDoneFlushed, 1) 910 pp.gcw.flushedWork = false 911 } 912 }) 913 914 if gcMarkDoneFlushed != 0 { 915 // More grey objects were discovered since the 916 // previous termination check, so there may be more 917 // work to do. Keep going. It's possible the 918 // transition condition became true again during the 919 // ragged barrier, so re-check it. 920 semrelease(&worldsema) 921 goto top 922 } 923 924 // For debugging/testing. 925 for gcDebugMarkDone.spinAfterRaggedBarrier.Load() { 926 } 927 928 // There was no global work, no local work, and no Ps 929 // communicated work since we took markDoneSema. Therefore 930 // there are no grey objects and no more objects can be 931 // shaded. Transition to mark termination. 932 now := nanotime() 933 work.tMarkTerm = now 934 getg().m.preemptoff = "gcing" 935 var stw worldStop 936 systemstack(func() { 937 stw = stopTheWorldWithSema(stwGCMarkTerm) 938 }) 939 // The gcphase is _GCmark, it will transition to _GCmarktermination 940 // below. The important thing is that the wb remains active until 941 // all marking is complete. This includes writes made by the GC. 942 943 // Accumulate fine-grained stopping time. 944 work.cpuStats.accumulateGCPauseTime(stw.stoppingCPUTime, 1) 945 946 // There is sometimes work left over when we enter mark termination due 947 // to write barriers performed after the completion barrier above. 948 // Detect this and resume concurrent mark. This is obviously 949 // unfortunate. 950 // 951 // See issue #27993 for details. 952 // 953 // Switch to the system stack to call wbBufFlush1, though in this case 954 // it doesn't matter because we're non-preemptible anyway. 955 restart := false 956 systemstack(func() { 957 for _, p := range allp { 958 wbBufFlush1(p) 959 if !p.gcw.empty() { 960 restart = true 961 break 962 } 963 } 964 }) 965 if restart { 966 gcDebugMarkDone.restartedDueTo27993 = true 967 968 getg().m.preemptoff = "" 969 systemstack(func() { 970 // Accumulate the time we were stopped before we had to start again. 971 work.cpuStats.accumulateGCPauseTime(nanotime()-stw.finishedStopping, work.maxprocs) 972 973 // Start the world again. 974 now := startTheWorldWithSema(0, stw) 975 work.pauseNS += now - stw.startedStopping 976 }) 977 semrelease(&worldsema) 978 goto top 979 } 980 981 gcComputeStartingStackSize() 982 983 // Disable assists and background workers. We must do 984 // this before waking blocked assists. 985 atomic.Store(&gcBlackenEnabled, 0) 986 987 // Notify the CPU limiter that GC assists will now cease. 988 gcCPULimiter.startGCTransition(false, now) 989 990 // Wake all blocked assists. These will run when we 991 // start the world again. 992 gcWakeAllAssists() 993 994 // Wake all blocked weak->strong conversions. These will run 995 // when we start the world again. 996 work.strongFromWeak.block = false 997 gcWakeAllStrongFromWeak() 998 999 // Likewise, release the transition lock. Blocked 1000 // workers and assists will run when we start the 1001 // world again. 1002 semrelease(&work.markDoneSema) 1003 1004 // In STW mode, re-enable user goroutines. These will be 1005 // queued to run after we start the world. 1006 schedEnableUser(true) 1007 1008 // endCycle depends on all gcWork cache stats being flushed. 1009 // The termination algorithm above ensured that up to 1010 // allocations since the ragged barrier. 1011 gcController.endCycle(now, int(gomaxprocs), work.userForced) 1012 1013 // Perform mark termination. This will restart the world. 1014 gcMarkTermination(stw) 1015 } 1016 1017 // World must be stopped and mark assists and background workers must be 1018 // disabled. 1019 func gcMarkTermination(stw worldStop) { 1020 // Start marktermination (write barrier remains enabled for now). 1021 setGCPhase(_GCmarktermination) 1022 1023 work.heap1 = gcController.heapLive.Load() 1024 startTime := nanotime() 1025 1026 mp := acquirem() 1027 mp.preemptoff = "gcing" 1028 mp.traceback = 2 1029 curgp := mp.curg 1030 // N.B. The execution tracer is not aware of this status 1031 // transition and handles it specially based on the 1032 // wait reason. 1033 casGToWaitingForGC(curgp, _Grunning, waitReasonGarbageCollection) 1034 1035 // Run gc on the g0 stack. We do this so that the g stack 1036 // we're currently running on will no longer change. Cuts 1037 // the root set down a bit (g0 stacks are not scanned, and 1038 // we don't need to scan gc's internal state). We also 1039 // need to switch to g0 so we can shrink the stack. 1040 systemstack(func() { 1041 gcMark(startTime) 1042 // Must return immediately. 1043 // The outer function's stack may have moved 1044 // during gcMark (it shrinks stacks, including the 1045 // outer function's stack), so we must not refer 1046 // to any of its variables. Return back to the 1047 // non-system stack to pick up the new addresses 1048 // before continuing. 1049 }) 1050 1051 var stwSwept bool 1052 systemstack(func() { 1053 work.heap2 = work.bytesMarked 1054 if debug.gccheckmark > 0 { 1055 // Run a full non-parallel, stop-the-world 1056 // mark using checkmark bits, to check that we 1057 // didn't forget to mark anything during the 1058 // concurrent mark process. 1059 startCheckmarks() 1060 gcResetMarkState() 1061 gcw := &getg().m.p.ptr().gcw 1062 gcDrain(gcw, 0) 1063 wbBufFlush1(getg().m.p.ptr()) 1064 gcw.dispose() 1065 endCheckmarks() 1066 } 1067 1068 // marking is complete so we can turn the write barrier off 1069 setGCPhase(_GCoff) 1070 stwSwept = gcSweep(work.mode) 1071 }) 1072 1073 mp.traceback = 0 1074 casgstatus(curgp, _Gwaiting, _Grunning) 1075 1076 trace := traceAcquire() 1077 if trace.ok() { 1078 trace.GCDone() 1079 traceRelease(trace) 1080 } 1081 1082 // all done 1083 mp.preemptoff = "" 1084 1085 if gcphase != _GCoff { 1086 throw("gc done but gcphase != _GCoff") 1087 } 1088 1089 // Record heapInUse for scavenger. 1090 memstats.lastHeapInUse = gcController.heapInUse.load() 1091 1092 // Update GC trigger and pacing, as well as downstream consumers 1093 // of this pacing information, for the next cycle. 1094 systemstack(gcControllerCommit) 1095 1096 // Update timing memstats 1097 now := nanotime() 1098 sec, nsec, _ := time_now() 1099 unixNow := sec*1e9 + int64(nsec) 1100 work.pauseNS += now - stw.startedStopping 1101 work.tEnd = now 1102 atomic.Store64(&memstats.last_gc_unix, uint64(unixNow)) // must be Unix time to make sense to user 1103 atomic.Store64(&memstats.last_gc_nanotime, uint64(now)) // monotonic time for us 1104 memstats.pause_ns[memstats.numgc%uint32(len(memstats.pause_ns))] = uint64(work.pauseNS) 1105 memstats.pause_end[memstats.numgc%uint32(len(memstats.pause_end))] = uint64(unixNow) 1106 memstats.pause_total_ns += uint64(work.pauseNS) 1107 1108 // Accumulate CPU stats. 1109 // 1110 // Use maxprocs instead of stwprocs for GC pause time because the total time 1111 // computed in the CPU stats is based on maxprocs, and we want them to be 1112 // comparable. 1113 // 1114 // Pass gcMarkPhase=true to accumulate so we can get all the latest GC CPU stats 1115 // in there too. 1116 work.cpuStats.accumulateGCPauseTime(now-stw.finishedStopping, work.maxprocs) 1117 work.cpuStats.accumulate(now, true) 1118 1119 // Compute overall GC CPU utilization. 1120 // Omit idle marking time from the overall utilization here since it's "free". 1121 memstats.gc_cpu_fraction = float64(work.cpuStats.GCTotalTime-work.cpuStats.GCIdleTime) / float64(work.cpuStats.TotalTime) 1122 1123 // Reset assist time and background time stats. 1124 // 1125 // Do this now, instead of at the start of the next GC cycle, because 1126 // these two may keep accumulating even if the GC is not active. 1127 scavenge.assistTime.Store(0) 1128 scavenge.backgroundTime.Store(0) 1129 1130 // Reset idle time stat. 1131 sched.idleTime.Store(0) 1132 1133 if work.userForced { 1134 memstats.numforcedgc++ 1135 } 1136 1137 // Bump GC cycle count and wake goroutines waiting on sweep. 1138 lock(&work.sweepWaiters.lock) 1139 memstats.numgc++ 1140 injectglist(&work.sweepWaiters.list) 1141 unlock(&work.sweepWaiters.lock) 1142 1143 // Increment the scavenge generation now. 1144 // 1145 // This moment represents peak heap in use because we're 1146 // about to start sweeping. 1147 mheap_.pages.scav.index.nextGen() 1148 1149 // Release the CPU limiter. 1150 gcCPULimiter.finishGCTransition(now) 1151 1152 // Finish the current heap profiling cycle and start a new 1153 // heap profiling cycle. We do this before starting the world 1154 // so events don't leak into the wrong cycle. 1155 mProf_NextCycle() 1156 1157 // There may be stale spans in mcaches that need to be swept. 1158 // Those aren't tracked in any sweep lists, so we need to 1159 // count them against sweep completion until we ensure all 1160 // those spans have been forced out. 1161 // 1162 // If gcSweep fully swept the heap (for example if the sweep 1163 // is not concurrent due to a GODEBUG setting), then we expect 1164 // the sweepLocker to be invalid, since sweeping is done. 1165 // 1166 // N.B. Below we might duplicate some work from gcSweep; this is 1167 // fine as all that work is idempotent within a GC cycle, and 1168 // we're still holding worldsema so a new cycle can't start. 1169 sl := sweep.active.begin() 1170 if !stwSwept && !sl.valid { 1171 throw("failed to set sweep barrier") 1172 } else if stwSwept && sl.valid { 1173 throw("non-concurrent sweep failed to drain all sweep queues") 1174 } 1175 1176 systemstack(func() { 1177 // The memstats updated above must be updated with the world 1178 // stopped to ensure consistency of some values, such as 1179 // sched.idleTime and sched.totaltime. memstats also include 1180 // the pause time (work,pauseNS), forcing computation of the 1181 // total pause time before the pause actually ends. 1182 // 1183 // Here we reuse the same now for start the world so that the 1184 // time added to /sched/pauses/total/gc:seconds will be 1185 // consistent with the value in memstats. 1186 startTheWorldWithSema(now, stw) 1187 }) 1188 1189 // Flush the heap profile so we can start a new cycle next GC. 1190 // This is relatively expensive, so we don't do it with the 1191 // world stopped. 1192 mProf_Flush() 1193 1194 // Prepare workbufs for freeing by the sweeper. We do this 1195 // asynchronously because it can take non-trivial time. 1196 prepareFreeWorkbufs() 1197 1198 // Free stack spans. This must be done between GC cycles. 1199 systemstack(freeStackSpans) 1200 1201 // Ensure all mcaches are flushed. Each P will flush its own 1202 // mcache before allocating, but idle Ps may not. Since this 1203 // is necessary to sweep all spans, we need to ensure all 1204 // mcaches are flushed before we start the next GC cycle. 1205 // 1206 // While we're here, flush the page cache for idle Ps to avoid 1207 // having pages get stuck on them. These pages are hidden from 1208 // the scavenger, so in small idle heaps a significant amount 1209 // of additional memory might be held onto. 1210 // 1211 // Also, flush the pinner cache, to avoid leaking that memory 1212 // indefinitely. 1213 forEachP(waitReasonFlushProcCaches, func(pp *p) { 1214 pp.mcache.prepareForSweep() 1215 if pp.status == _Pidle { 1216 systemstack(func() { 1217 lock(&mheap_.lock) 1218 pp.pcache.flush(&mheap_.pages) 1219 unlock(&mheap_.lock) 1220 }) 1221 } 1222 pp.pinnerCache = nil 1223 }) 1224 if sl.valid { 1225 // Now that we've swept stale spans in mcaches, they don't 1226 // count against unswept spans. 1227 // 1228 // Note: this sweepLocker may not be valid if sweeping had 1229 // already completed during the STW. See the corresponding 1230 // begin() call that produced sl. 1231 sweep.active.end(sl) 1232 } 1233 1234 // Print gctrace before dropping worldsema. As soon as we drop 1235 // worldsema another cycle could start and smash the stats 1236 // we're trying to print. 1237 if debug.gctrace > 0 { 1238 util := int(memstats.gc_cpu_fraction * 100) 1239 1240 var sbuf [24]byte 1241 printlock() 1242 print("gc ", memstats.numgc, 1243 " @", string(itoaDiv(sbuf[:], uint64(work.tSweepTerm-runtimeInitTime)/1e6, 3)), "s ", 1244 util, "%: ") 1245 prev := work.tSweepTerm 1246 for i, ns := range []int64{work.tMark, work.tMarkTerm, work.tEnd} { 1247 if i != 0 { 1248 print("+") 1249 } 1250 print(string(fmtNSAsMS(sbuf[:], uint64(ns-prev)))) 1251 prev = ns 1252 } 1253 print(" ms clock, ") 1254 for i, ns := range []int64{ 1255 int64(work.stwprocs) * (work.tMark - work.tSweepTerm), 1256 gcController.assistTime.Load(), 1257 gcController.dedicatedMarkTime.Load() + gcController.fractionalMarkTime.Load(), 1258 gcController.idleMarkTime.Load(), 1259 int64(work.stwprocs) * (work.tEnd - work.tMarkTerm), 1260 } { 1261 if i == 2 || i == 3 { 1262 // Separate mark time components with /. 1263 print("/") 1264 } else if i != 0 { 1265 print("+") 1266 } 1267 print(string(fmtNSAsMS(sbuf[:], uint64(ns)))) 1268 } 1269 print(" ms cpu, ", 1270 work.heap0>>20, "->", work.heap1>>20, "->", work.heap2>>20, " MB, ", 1271 gcController.lastHeapGoal>>20, " MB goal, ", 1272 gcController.lastStackScan.Load()>>20, " MB stacks, ", 1273 gcController.globalsScan.Load()>>20, " MB globals, ", 1274 work.maxprocs, " P") 1275 if work.userForced { 1276 print(" (forced)") 1277 } 1278 print("\n") 1279 printunlock() 1280 } 1281 1282 // Set any arena chunks that were deferred to fault. 1283 lock(&userArenaState.lock) 1284 faultList := userArenaState.fault 1285 userArenaState.fault = nil 1286 unlock(&userArenaState.lock) 1287 for _, lc := range faultList { 1288 lc.mspan.setUserArenaChunkToFault() 1289 } 1290 1291 // Enable huge pages on some metadata if we cross a heap threshold. 1292 if gcController.heapGoal() > minHeapForMetadataHugePages { 1293 systemstack(func() { 1294 mheap_.enableMetadataHugePages() 1295 }) 1296 } 1297 1298 semrelease(&worldsema) 1299 semrelease(&gcsema) 1300 // Careful: another GC cycle may start now. 1301 1302 releasem(mp) 1303 mp = nil 1304 1305 // now that gc is done, kick off finalizer thread if needed 1306 if !concurrentSweep { 1307 // give the queued finalizers, if any, a chance to run 1308 Gosched() 1309 } 1310 } 1311 1312 // gcBgMarkStartWorkers prepares background mark worker goroutines. These 1313 // goroutines will not run until the mark phase, but they must be started while 1314 // the work is not stopped and from a regular G stack. The caller must hold 1315 // worldsema. 1316 func gcBgMarkStartWorkers() { 1317 // Background marking is performed by per-P G's. Ensure that each P has 1318 // a background GC G. 1319 // 1320 // Worker Gs don't exit if gomaxprocs is reduced. If it is raised 1321 // again, we can reuse the old workers; no need to create new workers. 1322 if gcBgMarkWorkerCount >= gomaxprocs { 1323 return 1324 } 1325 1326 // Increment mp.locks when allocating. We are called within gcStart, 1327 // and thus must not trigger another gcStart via an allocation. gcStart 1328 // bails when allocating with locks held, so simulate that for these 1329 // allocations. 1330 // 1331 // TODO(prattmic): cleanup gcStart to use a more explicit "in gcStart" 1332 // check for bailing. 1333 mp := acquirem() 1334 ready := make(chan struct{}, 1) 1335 releasem(mp) 1336 1337 for gcBgMarkWorkerCount < gomaxprocs { 1338 mp := acquirem() // See above, we allocate a closure here. 1339 go gcBgMarkWorker(ready) 1340 releasem(mp) 1341 1342 // N.B. we intentionally wait on each goroutine individually 1343 // rather than starting all in a batch and then waiting once 1344 // afterwards. By running one goroutine at a time, we can take 1345 // advantage of runnext to bounce back and forth between 1346 // workers and this goroutine. In an overloaded application, 1347 // this can reduce GC start latency by prioritizing these 1348 // goroutines rather than waiting on the end of the run queue. 1349 <-ready 1350 // The worker is now guaranteed to be added to the pool before 1351 // its P's next findRunnableGCWorker. 1352 1353 gcBgMarkWorkerCount++ 1354 } 1355 } 1356 1357 // gcBgMarkPrepare sets up state for background marking. 1358 // Mutator assists must not yet be enabled. 1359 func gcBgMarkPrepare() { 1360 // Background marking will stop when the work queues are empty 1361 // and there are no more workers (note that, since this is 1362 // concurrent, this may be a transient state, but mark 1363 // termination will clean it up). Between background workers 1364 // and assists, we don't really know how many workers there 1365 // will be, so we pretend to have an arbitrarily large number 1366 // of workers, almost all of which are "waiting". While a 1367 // worker is working it decrements nwait. If nproc == nwait, 1368 // there are no workers. 1369 work.nproc = ^uint32(0) 1370 work.nwait = ^uint32(0) 1371 } 1372 1373 // gcBgMarkWorkerNode is an entry in the gcBgMarkWorkerPool. It points to a single 1374 // gcBgMarkWorker goroutine. 1375 type gcBgMarkWorkerNode struct { 1376 // Unused workers are managed in a lock-free stack. This field must be first. 1377 node lfnode 1378 1379 // The g of this worker. 1380 gp guintptr 1381 1382 // Release this m on park. This is used to communicate with the unlock 1383 // function, which cannot access the G's stack. It is unused outside of 1384 // gcBgMarkWorker(). 1385 m muintptr 1386 } 1387 1388 func gcBgMarkWorker(ready chan struct{}) { 1389 gp := getg() 1390 1391 // We pass node to a gopark unlock function, so it can't be on 1392 // the stack (see gopark). Prevent deadlock from recursively 1393 // starting GC by disabling preemption. 1394 gp.m.preemptoff = "GC worker init" 1395 node := new(gcBgMarkWorkerNode) 1396 gp.m.preemptoff = "" 1397 1398 node.gp.set(gp) 1399 1400 node.m.set(acquirem()) 1401 1402 ready <- struct{}{} 1403 // After this point, the background mark worker is generally scheduled 1404 // cooperatively by gcController.findRunnableGCWorker. While performing 1405 // work on the P, preemption is disabled because we are working on 1406 // P-local work buffers. When the preempt flag is set, this puts itself 1407 // into _Gwaiting to be woken up by gcController.findRunnableGCWorker 1408 // at the appropriate time. 1409 // 1410 // When preemption is enabled (e.g., while in gcMarkDone), this worker 1411 // may be preempted and schedule as a _Grunnable G from a runq. That is 1412 // fine; it will eventually gopark again for further scheduling via 1413 // findRunnableGCWorker. 1414 // 1415 // Since we disable preemption before notifying ready, we guarantee that 1416 // this G will be in the worker pool for the next findRunnableGCWorker. 1417 // This isn't strictly necessary, but it reduces latency between 1418 // _GCmark starting and the workers starting. 1419 1420 for { 1421 // Go to sleep until woken by 1422 // gcController.findRunnableGCWorker. 1423 gopark(func(g *g, nodep unsafe.Pointer) bool { 1424 node := (*gcBgMarkWorkerNode)(nodep) 1425 1426 if mp := node.m.ptr(); mp != nil { 1427 // The worker G is no longer running; release 1428 // the M. 1429 // 1430 // N.B. it is _safe_ to release the M as soon 1431 // as we are no longer performing P-local mark 1432 // work. 1433 // 1434 // However, since we cooperatively stop work 1435 // when gp.preempt is set, if we releasem in 1436 // the loop then the following call to gopark 1437 // would immediately preempt the G. This is 1438 // also safe, but inefficient: the G must 1439 // schedule again only to enter gopark and park 1440 // again. Thus, we defer the release until 1441 // after parking the G. 1442 releasem(mp) 1443 } 1444 1445 // Release this G to the pool. 1446 gcBgMarkWorkerPool.push(&node.node) 1447 // Note that at this point, the G may immediately be 1448 // rescheduled and may be running. 1449 return true 1450 }, unsafe.Pointer(node), waitReasonGCWorkerIdle, traceBlockSystemGoroutine, 0) 1451 1452 // Preemption must not occur here, or another G might see 1453 // p.gcMarkWorkerMode. 1454 1455 // Disable preemption so we can use the gcw. If the 1456 // scheduler wants to preempt us, we'll stop draining, 1457 // dispose the gcw, and then preempt. 1458 node.m.set(acquirem()) 1459 pp := gp.m.p.ptr() // P can't change with preemption disabled. 1460 1461 if gcBlackenEnabled == 0 { 1462 println("worker mode", pp.gcMarkWorkerMode) 1463 throw("gcBgMarkWorker: blackening not enabled") 1464 } 1465 1466 if pp.gcMarkWorkerMode == gcMarkWorkerNotWorker { 1467 throw("gcBgMarkWorker: mode not set") 1468 } 1469 1470 startTime := nanotime() 1471 pp.gcMarkWorkerStartTime = startTime 1472 var trackLimiterEvent bool 1473 if pp.gcMarkWorkerMode == gcMarkWorkerIdleMode { 1474 trackLimiterEvent = pp.limiterEvent.start(limiterEventIdleMarkWork, startTime) 1475 } 1476 1477 decnwait := atomic.Xadd(&work.nwait, -1) 1478 if decnwait == work.nproc { 1479 println("runtime: work.nwait=", decnwait, "work.nproc=", work.nproc) 1480 throw("work.nwait was > work.nproc") 1481 } 1482 1483 systemstack(func() { 1484 // Mark our goroutine preemptible so its stack 1485 // can be scanned. This lets two mark workers 1486 // scan each other (otherwise, they would 1487 // deadlock). We must not modify anything on 1488 // the G stack. However, stack shrinking is 1489 // disabled for mark workers, so it is safe to 1490 // read from the G stack. 1491 // 1492 // N.B. The execution tracer is not aware of this status 1493 // transition and handles it specially based on the 1494 // wait reason. 1495 casGToWaitingForGC(gp, _Grunning, waitReasonGCWorkerActive) 1496 switch pp.gcMarkWorkerMode { 1497 default: 1498 throw("gcBgMarkWorker: unexpected gcMarkWorkerMode") 1499 case gcMarkWorkerDedicatedMode: 1500 gcDrainMarkWorkerDedicated(&pp.gcw, true) 1501 if gp.preempt { 1502 // We were preempted. This is 1503 // a useful signal to kick 1504 // everything out of the run 1505 // queue so it can run 1506 // somewhere else. 1507 if drainQ, n := runqdrain(pp); n > 0 { 1508 lock(&sched.lock) 1509 globrunqputbatch(&drainQ, int32(n)) 1510 unlock(&sched.lock) 1511 } 1512 } 1513 // Go back to draining, this time 1514 // without preemption. 1515 gcDrainMarkWorkerDedicated(&pp.gcw, false) 1516 case gcMarkWorkerFractionalMode: 1517 gcDrainMarkWorkerFractional(&pp.gcw) 1518 case gcMarkWorkerIdleMode: 1519 gcDrainMarkWorkerIdle(&pp.gcw) 1520 } 1521 casgstatus(gp, _Gwaiting, _Grunning) 1522 }) 1523 1524 // Account for time and mark us as stopped. 1525 now := nanotime() 1526 duration := now - startTime 1527 gcController.markWorkerStop(pp.gcMarkWorkerMode, duration) 1528 if trackLimiterEvent { 1529 pp.limiterEvent.stop(limiterEventIdleMarkWork, now) 1530 } 1531 if pp.gcMarkWorkerMode == gcMarkWorkerFractionalMode { 1532 atomic.Xaddint64(&pp.gcFractionalMarkTime, duration) 1533 } 1534 1535 // Was this the last worker and did we run out 1536 // of work? 1537 incnwait := atomic.Xadd(&work.nwait, +1) 1538 if incnwait > work.nproc { 1539 println("runtime: p.gcMarkWorkerMode=", pp.gcMarkWorkerMode, 1540 "work.nwait=", incnwait, "work.nproc=", work.nproc) 1541 throw("work.nwait > work.nproc") 1542 } 1543 1544 // We'll releasem after this point and thus this P may run 1545 // something else. We must clear the worker mode to avoid 1546 // attributing the mode to a different (non-worker) G in 1547 // traceGoStart. 1548 pp.gcMarkWorkerMode = gcMarkWorkerNotWorker 1549 1550 // If this worker reached a background mark completion 1551 // point, signal the main GC goroutine. 1552 if incnwait == work.nproc && !gcMarkWorkAvailable(nil) { 1553 // We don't need the P-local buffers here, allow 1554 // preemption because we may schedule like a regular 1555 // goroutine in gcMarkDone (block on locks, etc). 1556 releasem(node.m.ptr()) 1557 node.m.set(nil) 1558 1559 gcMarkDone() 1560 } 1561 } 1562 } 1563 1564 // gcMarkWorkAvailable reports whether executing a mark worker 1565 // on p is potentially useful. p may be nil, in which case it only 1566 // checks the global sources of work. 1567 func gcMarkWorkAvailable(p *p) bool { 1568 if p != nil && !p.gcw.empty() { 1569 return true 1570 } 1571 if !work.full.empty() { 1572 return true // global work available 1573 } 1574 if work.markrootNext < work.markrootJobs { 1575 return true // root scan work available 1576 } 1577 return false 1578 } 1579 1580 // gcMark runs the mark (or, for concurrent GC, mark termination) 1581 // All gcWork caches must be empty. 1582 // STW is in effect at this point. 1583 func gcMark(startTime int64) { 1584 if gcphase != _GCmarktermination { 1585 throw("in gcMark expecting to see gcphase as _GCmarktermination") 1586 } 1587 work.tstart = startTime 1588 1589 // Check that there's no marking work remaining. 1590 if work.full != 0 || work.markrootNext < work.markrootJobs { 1591 print("runtime: full=", hex(work.full), " next=", work.markrootNext, " jobs=", work.markrootJobs, " nDataRoots=", work.nDataRoots, " nBSSRoots=", work.nBSSRoots, " nSpanRoots=", work.nSpanRoots, " nStackRoots=", work.nStackRoots, "\n") 1592 panic("non-empty mark queue after concurrent mark") 1593 } 1594 1595 if debug.gccheckmark > 0 { 1596 // This is expensive when there's a large number of 1597 // Gs, so only do it if checkmark is also enabled. 1598 gcMarkRootCheck() 1599 } 1600 1601 // Drop allg snapshot. allgs may have grown, in which case 1602 // this is the only reference to the old backing store and 1603 // there's no need to keep it around. 1604 work.stackRoots = nil 1605 1606 // Clear out buffers and double-check that all gcWork caches 1607 // are empty. This should be ensured by gcMarkDone before we 1608 // enter mark termination. 1609 // 1610 // TODO: We could clear out buffers just before mark if this 1611 // has a non-negligible impact on STW time. 1612 for _, p := range allp { 1613 // The write barrier may have buffered pointers since 1614 // the gcMarkDone barrier. However, since the barrier 1615 // ensured all reachable objects were marked, all of 1616 // these must be pointers to black objects. Hence we 1617 // can just discard the write barrier buffer. 1618 if debug.gccheckmark > 0 { 1619 // For debugging, flush the buffer and make 1620 // sure it really was all marked. 1621 wbBufFlush1(p) 1622 } else { 1623 p.wbBuf.reset() 1624 } 1625 1626 gcw := &p.gcw 1627 if !gcw.empty() { 1628 printlock() 1629 print("runtime: P ", p.id, " flushedWork ", gcw.flushedWork) 1630 if gcw.wbuf1 == nil { 1631 print(" wbuf1=<nil>") 1632 } else { 1633 print(" wbuf1.n=", gcw.wbuf1.nobj) 1634 } 1635 if gcw.wbuf2 == nil { 1636 print(" wbuf2=<nil>") 1637 } else { 1638 print(" wbuf2.n=", gcw.wbuf2.nobj) 1639 } 1640 print("\n") 1641 throw("P has cached GC work at end of mark termination") 1642 } 1643 // There may still be cached empty buffers, which we 1644 // need to flush since we're going to free them. Also, 1645 // there may be non-zero stats because we allocated 1646 // black after the gcMarkDone barrier. 1647 gcw.dispose() 1648 } 1649 1650 // Flush scanAlloc from each mcache since we're about to modify 1651 // heapScan directly. If we were to flush this later, then scanAlloc 1652 // might have incorrect information. 1653 // 1654 // Note that it's not important to retain this information; we know 1655 // exactly what heapScan is at this point via scanWork. 1656 for _, p := range allp { 1657 c := p.mcache 1658 if c == nil { 1659 continue 1660 } 1661 c.scanAlloc = 0 1662 } 1663 1664 // Reset controller state. 1665 gcController.resetLive(work.bytesMarked) 1666 } 1667 1668 // gcSweep must be called on the system stack because it acquires the heap 1669 // lock. See mheap for details. 1670 // 1671 // Returns true if the heap was fully swept by this function. 1672 // 1673 // The world must be stopped. 1674 // 1675 //go:systemstack 1676 func gcSweep(mode gcMode) bool { 1677 assertWorldStopped() 1678 1679 if gcphase != _GCoff { 1680 throw("gcSweep being done but phase is not GCoff") 1681 } 1682 1683 lock(&mheap_.lock) 1684 mheap_.sweepgen += 2 1685 sweep.active.reset() 1686 mheap_.pagesSwept.Store(0) 1687 mheap_.sweepArenas = mheap_.allArenas 1688 mheap_.reclaimIndex.Store(0) 1689 mheap_.reclaimCredit.Store(0) 1690 unlock(&mheap_.lock) 1691 1692 sweep.centralIndex.clear() 1693 1694 if !concurrentSweep || mode == gcForceBlockMode { 1695 // Special case synchronous sweep. 1696 // Record that no proportional sweeping has to happen. 1697 lock(&mheap_.lock) 1698 mheap_.sweepPagesPerByte = 0 1699 unlock(&mheap_.lock) 1700 // Flush all mcaches. 1701 for _, pp := range allp { 1702 pp.mcache.prepareForSweep() 1703 } 1704 // Sweep all spans eagerly. 1705 for sweepone() != ^uintptr(0) { 1706 } 1707 // Free workbufs eagerly. 1708 prepareFreeWorkbufs() 1709 for freeSomeWbufs(false) { 1710 } 1711 // All "free" events for this mark/sweep cycle have 1712 // now happened, so we can make this profile cycle 1713 // available immediately. 1714 mProf_NextCycle() 1715 mProf_Flush() 1716 return true 1717 } 1718 1719 // Background sweep. 1720 lock(&sweep.lock) 1721 if sweep.parked { 1722 sweep.parked = false 1723 ready(sweep.g, 0, true) 1724 } 1725 unlock(&sweep.lock) 1726 return false 1727 } 1728 1729 // gcResetMarkState resets global state prior to marking (concurrent 1730 // or STW) and resets the stack scan state of all Gs. 1731 // 1732 // This is safe to do without the world stopped because any Gs created 1733 // during or after this will start out in the reset state. 1734 // 1735 // gcResetMarkState must be called on the system stack because it acquires 1736 // the heap lock. See mheap for details. 1737 // 1738 //go:systemstack 1739 func gcResetMarkState() { 1740 // This may be called during a concurrent phase, so lock to make sure 1741 // allgs doesn't change. 1742 forEachG(func(gp *g) { 1743 gp.gcscandone = false // set to true in gcphasework 1744 gp.gcAssistBytes = 0 1745 }) 1746 1747 // Clear page marks. This is just 1MB per 64GB of heap, so the 1748 // time here is pretty trivial. 1749 lock(&mheap_.lock) 1750 arenas := mheap_.allArenas 1751 unlock(&mheap_.lock) 1752 for _, ai := range arenas { 1753 ha := mheap_.arenas[ai.l1()][ai.l2()] 1754 clear(ha.pageMarks[:]) 1755 } 1756 1757 work.bytesMarked = 0 1758 work.initialHeapLive = gcController.heapLive.Load() 1759 } 1760 1761 // Hooks for other packages 1762 1763 var poolcleanup func() 1764 var boringCaches []unsafe.Pointer // for crypto/internal/boring 1765 var uniqueMapCleanup chan struct{} // for unique 1766 1767 // sync_runtime_registerPoolCleanup should be an internal detail, 1768 // but widely used packages access it using linkname. 1769 // Notable members of the hall of shame include: 1770 // - github.com/bytedance/gopkg 1771 // - github.com/songzhibin97/gkit 1772 // 1773 // Do not remove or change the type signature. 1774 // See go.dev/issue/67401. 1775 // 1776 //go:linkname sync_runtime_registerPoolCleanup sync.runtime_registerPoolCleanup 1777 func sync_runtime_registerPoolCleanup(f func()) { 1778 poolcleanup = f 1779 } 1780 1781 //go:linkname boring_registerCache crypto/internal/boring/bcache.registerCache 1782 func boring_registerCache(p unsafe.Pointer) { 1783 boringCaches = append(boringCaches, p) 1784 } 1785 1786 //go:linkname unique_runtime_registerUniqueMapCleanup unique.runtime_registerUniqueMapCleanup 1787 func unique_runtime_registerUniqueMapCleanup(f func()) { 1788 // Create the channel on the system stack so it doesn't inherit the current G's 1789 // synctest bubble (if any). 1790 systemstack(func() { 1791 uniqueMapCleanup = make(chan struct{}, 1) 1792 }) 1793 // Start the goroutine in the runtime so it's counted as a system goroutine. 1794 go func(cleanup func()) { 1795 for { 1796 <-uniqueMapCleanup 1797 cleanup() 1798 } 1799 }(f) 1800 } 1801 1802 func clearpools() { 1803 // clear sync.Pools 1804 if poolcleanup != nil { 1805 poolcleanup() 1806 } 1807 1808 // clear boringcrypto caches 1809 for _, p := range boringCaches { 1810 atomicstorep(p, nil) 1811 } 1812 1813 // clear unique maps 1814 if uniqueMapCleanup != nil { 1815 select { 1816 case uniqueMapCleanup <- struct{}{}: 1817 default: 1818 } 1819 } 1820 1821 // Clear central sudog cache. 1822 // Leave per-P caches alone, they have strictly bounded size. 1823 // Disconnect cached list before dropping it on the floor, 1824 // so that a dangling ref to one entry does not pin all of them. 1825 lock(&sched.sudoglock) 1826 var sg, sgnext *sudog 1827 for sg = sched.sudogcache; sg != nil; sg = sgnext { 1828 sgnext = sg.next 1829 sg.next = nil 1830 } 1831 sched.sudogcache = nil 1832 unlock(&sched.sudoglock) 1833 1834 // Clear central defer pool. 1835 // Leave per-P pools alone, they have strictly bounded size. 1836 lock(&sched.deferlock) 1837 // disconnect cached list before dropping it on the floor, 1838 // so that a dangling ref to one entry does not pin all of them. 1839 var d, dlink *_defer 1840 for d = sched.deferpool; d != nil; d = dlink { 1841 dlink = d.link 1842 d.link = nil 1843 } 1844 sched.deferpool = nil 1845 unlock(&sched.deferlock) 1846 } 1847 1848 // Timing 1849 1850 // itoaDiv formats val/(10**dec) into buf. 1851 func itoaDiv(buf []byte, val uint64, dec int) []byte { 1852 i := len(buf) - 1 1853 idec := i - dec 1854 for val >= 10 || i >= idec { 1855 buf[i] = byte(val%10 + '0') 1856 i-- 1857 if i == idec { 1858 buf[i] = '.' 1859 i-- 1860 } 1861 val /= 10 1862 } 1863 buf[i] = byte(val + '0') 1864 return buf[i:] 1865 } 1866 1867 // fmtNSAsMS nicely formats ns nanoseconds as milliseconds. 1868 func fmtNSAsMS(buf []byte, ns uint64) []byte { 1869 if ns >= 10e6 { 1870 // Format as whole milliseconds. 1871 return itoaDiv(buf, ns/1e6, 0) 1872 } 1873 // Format two digits of precision, with at most three decimal places. 1874 x := ns / 1e3 1875 if x == 0 { 1876 buf[0] = '0' 1877 return buf[:1] 1878 } 1879 dec := 3 1880 for x >= 100 { 1881 x /= 10 1882 dec-- 1883 } 1884 return itoaDiv(buf, x, dec) 1885 } 1886 1887 // Helpers for testing GC. 1888 1889 // gcTestMoveStackOnNextCall causes the stack to be moved on a call 1890 // immediately following the call to this. It may not work correctly 1891 // if any other work appears after this call (such as returning). 1892 // Typically the following call should be marked go:noinline so it 1893 // performs a stack check. 1894 // 1895 // In rare cases this may not cause the stack to move, specifically if 1896 // there's a preemption between this call and the next. 1897 func gcTestMoveStackOnNextCall() { 1898 gp := getg() 1899 gp.stackguard0 = stackForceMove 1900 } 1901 1902 // gcTestIsReachable performs a GC and returns a bit set where bit i 1903 // is set if ptrs[i] is reachable. 1904 func gcTestIsReachable(ptrs ...unsafe.Pointer) (mask uint64) { 1905 // This takes the pointers as unsafe.Pointers in order to keep 1906 // them live long enough for us to attach specials. After 1907 // that, we drop our references to them. 1908 1909 if len(ptrs) > 64 { 1910 panic("too many pointers for uint64 mask") 1911 } 1912 1913 // Block GC while we attach specials and drop our references 1914 // to ptrs. Otherwise, if a GC is in progress, it could mark 1915 // them reachable via this function before we have a chance to 1916 // drop them. 1917 semacquire(&gcsema) 1918 1919 // Create reachability specials for ptrs. 1920 specials := make([]*specialReachable, len(ptrs)) 1921 for i, p := range ptrs { 1922 lock(&mheap_.speciallock) 1923 s := (*specialReachable)(mheap_.specialReachableAlloc.alloc()) 1924 unlock(&mheap_.speciallock) 1925 s.special.kind = _KindSpecialReachable 1926 if !addspecial(p, &s.special, false) { 1927 throw("already have a reachable special (duplicate pointer?)") 1928 } 1929 specials[i] = s 1930 // Make sure we don't retain ptrs. 1931 ptrs[i] = nil 1932 } 1933 1934 semrelease(&gcsema) 1935 1936 // Force a full GC and sweep. 1937 GC() 1938 1939 // Process specials. 1940 for i, s := range specials { 1941 if !s.done { 1942 printlock() 1943 println("runtime: object", i, "was not swept") 1944 throw("IsReachable failed") 1945 } 1946 if s.reachable { 1947 mask |= 1 << i 1948 } 1949 lock(&mheap_.speciallock) 1950 mheap_.specialReachableAlloc.free(unsafe.Pointer(s)) 1951 unlock(&mheap_.speciallock) 1952 } 1953 1954 return mask 1955 } 1956 1957 // gcTestPointerClass returns the category of what p points to, one of: 1958 // "heap", "stack", "data", "bss", "other". This is useful for checking 1959 // that a test is doing what it's intended to do. 1960 // 1961 // This is nosplit simply to avoid extra pointer shuffling that may 1962 // complicate a test. 1963 // 1964 //go:nosplit 1965 func gcTestPointerClass(p unsafe.Pointer) string { 1966 p2 := uintptr(noescape(p)) 1967 gp := getg() 1968 if gp.stack.lo <= p2 && p2 < gp.stack.hi { 1969 return "stack" 1970 } 1971 if base, _, _ := findObject(p2, 0, 0); base != 0 { 1972 return "heap" 1973 } 1974 for _, datap := range activeModules() { 1975 if datap.data <= p2 && p2 < datap.edata || datap.noptrdata <= p2 && p2 < datap.enoptrdata { 1976 return "data" 1977 } 1978 if datap.bss <= p2 && p2 < datap.ebss || datap.noptrbss <= p2 && p2 <= datap.enoptrbss { 1979 return "bss" 1980 } 1981 } 1982 KeepAlive(p) 1983 return "other" 1984 } 1985