Concurrency — `conc` patterns

Field	Value
Type	Agent Reference
Source	~/.copilot/agents/_refs/go-backend-engineer/concurrency.md
Description	Not specified

Source Content

Concurrency — conc patterns

Default to Sourcegraph’s conc over raw goroutines + sync.WaitGroup + channels. conc gives panic recovery, structured cancellation, and result aggregation with type safety.

When to reach for which primitive

Need	Use
Fan-out N tasks, collect results	pool.NewWithResults[T]().WithContext().WithMaxGoroutines(n)
Fan-out N tasks, just wait	pool.New().WithContext()
Run a few unrelated tasks concurrently	conc.NewWaitGroup()
Stream items through a pipeline	stream.New()
Iterate over a slice with bounded parallelism	iter.ForEach, iter.Map

Bounded fan-out with results

import "github.com/sourcegraph/conc/pool"

func (s *userService) BatchGet(ctx context.Context, ids []string) ([]*domain.User, error) {
    logger := telemetry.FromContext(ctx)
    p := pool.NewWithResults[*domain.User]().
        WithContext(ctx).
        WithMaxGoroutines(10) // bound the parallelism
    for _, id := range ids {
        id := id
        p.Go(func(ctx context.Context) (*domain.User, error) {
            return s.userRepo.GetByID(ctx, id)
        })
    }
    users, err := p.Wait()
    if err != nil {
        logger.Error("batch_get_failed", zap.Error(err))
        return nil, fmt.Errorf("batch get users: %w", err)
    }
    return users, nil
}

p.Wait() returns the first error and cancels in-flight tasks via the shared context. No goroutine leaks.

Iterator-style parallel map

import "github.com/sourcegraph/conc/iter"

results, err := iter.MapErr(ids, func(id *string) (*domain.User, error) {
    return s.userRepo.GetByID(ctx, *id)
})

Use this when you’d reach for lo.Map but need bounded goroutines + error propagation.

Pipeline / stream

import "github.com/sourcegraph/conc/stream"

s := stream.New().WithMaxGoroutines(8)
for _, id := range ids {
    id := id
    s.Go(func() stream.Callback {
        u, err := s.userRepo.GetByID(ctx, id)
        return func() { handle(u, err) }
    })
}
s.Wait()

Preserves input order while running stages in parallel.

Object pooling — sync.Pool

For zero-allocation hot paths (request DTOs, byte buffers, hash builders):

var bufPool = sync.Pool{New: func() any { return &bytes.Buffer{} }}

func render(...) string {
    buf := bufPool.Get().(*bytes.Buffer)
    buf.Reset()
    defer bufPool.Put(buf)
    // ... use buf ...
    return buf.String()
}

Profile first (go test -bench -memprofile); only pool things that show up in heap profiles. sync.Pool is wrong for stateful objects or anything with finalisers.

Hard rules

• Always propagate context.Context into goroutines — never spawn a detached goroutine inside a request handler.
• Always capture loop variables (id := id) before passing into closures (Go ≤1.21; 1.22+ is safe but be explicit anyway).
• Never swallow panics. conc recovers them as errors; raw goroutines need defer recover() + log + re-raise on the calling goroutine.
• Bound parallelism. WithMaxGoroutines should be set based on downstream capacity (DB pool size, external rate limit), not “feels right”.
• Test concurrent code with -race. Required in CI.