Config errors sometimes result in exit code 0

[wyattscarpenter]

Several tests in cmdline.pyproject.test result in configuration errors, yet assert == Return code: 0. This is probably bad; they should probably give a return code of 2, for use error. See, for more information, #6003 (because apparently it isn't documented yet). See also #16764, which may be related.

Originally noted by @sterliakov in #19494 (comment), who also had this to say:

I really think this should be fixed, 2 sounds reasonable and consistent with other tools, most Unix tools do that (grep --wtf and ls --not-an-option do exit with 2 on my system, for example, and bad config file is semantically close to bad flags). Note that mypy --wtf also exits with 2, so we have a precedent, and ruff exits with 2 if it finds an unknown key in its pyproject.toml section.

[sterliakov]

Some current exit codes:

• 0 on success (OK)
• 0 when only notes were produced (reasonable)
• 1 when some non-blocking errors were produced (OK)
• 2 when some blocking (e.g. invalid syntax) errors are encountered (makes sense as another code, but not 2?)
• 2 when mypy should be re-ran after --install-types (?)
• 2 when CLI flags are wrong (and exit without checking) (OK)
• 0 when config file is invalid (and check everything nevertheless - why?) (wrong)

Frankly, this is a mess. Code 2 should not be used for --install-types or blocking errors IMO - 2 is de-facto standard for "please do not invoke me like this". "Run again" and "invalid syntax" have so little in common that I struggle to understand the reason to combine them under one code.

[wyattscarpenter]

I agree with your outlook here.

[wyattscarpenter]

I suppose "when mypy should be re-run after --install-types" (which I think is probably this) is sort of like a crash in mypy, from the perspective that it doesn't let mypy actually check the types of your program(?)

[wyattscarpenter]

Even so, I don't feel very strongly about that interpretation of --install-types, and the wonderful thing about integers is that they are free. Maybe --install-types could exit with 3!

[sterliakov]

They aren't free, we are limited to 8 bits, and probably should stay away from using 127, as well as 130 and other signal-caused codes:) Should be more than enough for us, though.

I'm all for introducing new codes as necessary. Yes, this will break someone's automation, but that's a good moment to punish for not locking all dependencies... Seriously, however, we should really think twice here and try to reduce the impact to the minimum.

[wyattscarpenter]

The current title of this issue, "Config errors sometimes result in exit code 0" is indeed a duplicate of that other issue. But there are a couple things noted in your comment above that don't fall under that category. So I think we should either reopen this and change its title, or create a new issue(s) about how syntax errors in the Python source files give 2 when they should give 1, and how install-types should give 3 instead of 2. (Of course, I could open such issues, and may do so later.)

[sterliakov]

Nice, let's really have them as separate issues - it will be easier to reason about every choice as a separate issue&PR IMO. Thanks!