Better context timeout error messages

[brandur]

This one's aimed at producing somewhat better ergonomics when it comes
to working with context timeouts throughout the project. A sizable
problem that we have right now is that in case of timeout, all that
comes back is the generic error from context.DeadlineExceeded,
"context deadline exceeded". You don't know where it came from exactly
as it might've bubble up through many layers of the stack. If multiple
timeouts were in use simultaneously, either one might've caused the
timeout.

Here, introduce a timeout helper that aims to let us easily trace the
origin of a timeout. Traditionally, use of a timeout looked like this:

func (p *StandardPilot) JobGetAvailable(ctx context.Context, exec riverdriver.Executor, state ProducerState, params *riverdriver.JobGetAvailableParams) ([]*rivertype.JobRow, error) {
    ctx, cancel := context.WithTimeout(ctx, rivercommon.HotOperationTimeout)
    defer cancel()

    return exec.JobGetAvailable(ctx, params)

Here, we add WithTimeout/WithTimeoutV which are used like this:

func (p *StandardPilot) JobGetAvailable(ctx context.Context, exec riverdriver.Executor, state ProducerState, params *riverdriver.JobGetAvailableParams) ([]*rivertype.JobRow, error) {
    return timeoututil.WithTimeoutV(ctx, rivercommon.HotOperationTimeout, "StandardPilot.JobGetAvailable", func(ctx context.Context) ([]*rivertype.JobRow, error) {
        return exec.JobGetAvailable(ctx, params)
    })

The helpers apply a timeout with context.WithTimeout the same way the
first block works, but when a return timeout error is detected, they
wrap it in a more contextual form of the error. So previously, you'd get
this:

context deadline exceeded

With the helpers, you get this, containing the name of the operation and
the timeout that'd been applied:

StandardPilot.JobGetAvailable timed out after 1ns: context deadline exceeded

Unfortunately it has to be a helper function with inner callback because
even using context.WithTimeoutCause, ctx.Err still returns only a
naked context.DeadlineExceeded. You need to call context.Cause(ctx)
to get the error that was set as cause.

That said, everything else works. A context.DeadlineExceeded that was
wrapped on the way out is still recognized and maintains its wrapper.
Multiple levels of WithTimeout can be nested and the right error
message still gets returned, depending on which level timed out.

[brandur]

@bgentry Thoughts on this? I changed only a single instance of context timeout to show what usage would roughly look like at first, but if we went through with it, it shouldn't be too difficult to have an LLM canvas the whole codebase to update everything. Or we could do it over time.