Make error_on_callbacks a per-instance / producer config option (#2366)

[wgu9]

Summary

Closes #2366.

Previously the only way to change error_on_callbacks (whether exceptions
raised inside Future callbacks/errbacks are re-raised vs. only logged) was to
override the Future.error_on_callbacks class attribute, which has global
effect — not viable in a large app with multiple producers that need different
behavior.

This makes it a per-instance / per-producer option while preserving the
existing global-default mechanism:

• Future(error_on_callbacks=...) now stores a per-instance value. A small
  read-only error_on_callbacks property returns the instance value when set,
  otherwise the class-level default Future._default_error_on_callbacks
  (still False). Passing None (the default) means "inherit the class
  default", so existing global overrides keep working.
• KafkaProducer gains an error_on_callbacks config (default None) that is
  threaded through RecordAccumulator → ProducerBatch → each per-record
  FutureRecordMetadata, so users can opt in without global side effects.

Default behavior is unchanged.

Notes

• test/__init__.py sets the testing global default via the new
  Future._default_error_on_callbacks = True (was Future.error_on_callbacks = True),
  keeping "always fail during testing" intact.
• The producer hot path is unaffected: this only swaps a class-attribute read
  for an instance/property read; no new locking.

Testing

pytest test/test_future.py test/producer/   # 315 passed

New tests:

• test/test_future.py::TestFutureErrorOnCallbacks — explicit value overrides
  the class default in both directions; callback/errback raising; already-done path.
• test/producer/test_producer_batch.py — ProducerBatch propagates
  error_on_callbacks to record futures (default inherits, explicit overrides).

[dpkp]

This seems like an automated AI bot PR. Are you a human? What is your intention here?

[wgu9]

This seems like an automated AI bot PR. Are you a human? What is your intention here?

real person here. My intention was to make a genuine fix for #2366: exposing an option to enable error_on_callbacks in KafkaProducer.send. I reviewed the issue, followed the existing producer/future patterns, and ran the local producer/future tests, which passed.

I did use AI assistance while preparing the PR, but I’m personally responsible for the change and happy to address review feedback myself. If this kind of AI-assisted PR is not welcome here, I’m happy to close it and I apologize for the noise.

[dpkp]

AI-assisted code is absolutely welcome (I've been using it extensively). But my question was more about "why" you chose to focus on this issue. Have you been experiencing the error_on_callbacks problem in your kafka-python application? Because I'm not actually sure whether this is an issue that needs a fix. I've left it open because I also haven't decided it doesn't need a fix. And so if you've just stumbled on the project and found an open issue and thought that with AI help you could probably write a patch then it's less helpful -- because of course I could point my AI coder at this issue for a fix just as easily.

So what I'm looking for is more human insight into the why we should change this. What is the actual pain or problem you are feeling as a user of kafka-python? Can you expand on that?

[wgu9]

AI-assisted code is absolutely welcome (I've been using it extensively). But my question was more about "why" you chose to focus on this issue. Have you been experiencing the error_on_callbacks problem in your kafka-python application? Because I'm not actually sure whether this is an issue that needs a fix. I've left it open because I also haven't decided it doesn't need a fix. And so if you've just stumbled on the project and found an open issue and thought that with AI help you could probably write a patch then it's less helpful -- because of course I could point my AI coder at this issue for a fix just as easily.

So what I'm looking for is more human insight into the why we should change this. What is the actual pain or problem you are feeling as a user of kafka-python? Can you expand on that?

I don't have a production pain point for this one. I am a Kafka user for years and I came across the open issue and thought it looked like a clean fix, not from hitting it myself in an app. Appreciate the candid and thoughtful reply, it's a good reminder. I'll close this.

[dpkp]

Right on, well hey I totally appreciate the help. Once a new feature / arg gets merged I need to maintain it and for that I want to have strong user-empathy for what's underlying it so I know how to manage it in the future.

In this case I've mostly dropped the callback/errback approach from kafka-python internals and I think probably in the next release there will be a fully async interface so that the preferred approach for catching errors would become:

try:
    result = await producer.send(...)  # possibly with some timeout option added to send?
except ExceptionFoo:
    handle_foo_error()

and in that world I think that there would no longer be a need for something like error_on_callbacks per-instance. But I'm not 100% sure and so hoping someone might provide more insight.