Fix BNGsim bridge silent fallbacks: strict routing error (#109) + Perl-faithful list-arg grammar (#110)#111
Open
wshlavacek wants to merge 7 commits into
Open
Conversation
…spect actions
Under simulator='bngsim', the routing bridge silently fell back to the
legacy BNG2.pl subprocess engine whenever the Python parser
(modelapi.bngmodel) could not inspect a BNGL's actions — e.g. a typo'd or
unknown simulate argument (atoll, abstol, par_scan_vals) that bngmodel
rejects but BNG2.pl tolerates. This violated the simulator='bngsim'
contract ("use BNGsim, or error"): the caller got legacy output with no
exception and no warning. A parity sweep over 895 models hit this on 9
models.
_classify_bngl_actions_for_bngsim() returned ROUTE_SUBPROCESS for the
un-inspectable case unconditionally, with no reference to simulator.
Fix, at the classify_bngsim_route call site where simulator is in scope:
- simulator='bngsim' + un-inspectable actions -> ROUTE_ERROR, propagating
the underlying parse reason so the caller learns why it couldn't route
to BNGsim. Both consumers (main.py, runner.run) already raise
BNGSimError(route.reason) on ROUTE_ERROR.
- simulator='auto' keeps the subprocess fallback but emits a one-time
WARNING per file (deduped across the ~4 routing queries per run).
- simulator='subprocess' is unchanged (returns before action inspection).
To surface the reason, the routing parse now returns (actions, reason):
_parse_bngl_actions_for_routing and the memoized loader (renamed
_load_bngl_routing_actions; _load_bngl_actions_for_routing kept as a thin
list-only wrapper for its other callers) carry the parse message. The
unavailable-BNGsim case already returned ROUTE_ERROR and is untouched.
Adds tests covering the bngsim error, the auto one-time-warning fallback,
the subprocess no-inspect path, reason propagation through the cache, and
an end-to-end repro of the atoll-typo model from the issue.
…n args ActionList.define_parser's list-valued argument grammar was stricter than BNG2.pl/Perl: arg_type_list matched elements with `pp.Word(pp.nums + ".")` (digits and '.' only) inside a plain `pp.delimitedList`, so it rejected - scientific notation, e.g. par_scan_vals=>[2.3e-10,5.1e-10] - a trailing comma, e.g. par_scan_vals=>[1,2,3,] Both are valid Perl that BNG2.pl parses and runs. When a model used either (commonly par_scan_vals on parameter_scan), modelapi.bngmodel raised BNGParseError, so under simulator='bngsim' the bridge couldn't inspect the actions and silently fell back to the legacy subprocess (the RuleWorld#109 class) — the model never ran on bngsim. Real models hit this: RuleHub's Mitra2019/15-igf1r fits and Salazar-Cavazos2019 CHO_EGFR best-fit. Broaden arg_type_list to use arg_type_expr (which already spans e/E and +/-), allow an empty list, and tolerate one optional trailing comma: arg_type_list = "[" + Optional(delimitedList(quote_word ^ arg_type_expr)) + Optional(",") + "]" Still rejects genuinely-malformed lists (double commas `[1,,2]`, unclosed `[1,2,`). Adds parametrized accept/reject tests over the issue's matrix and the affected real-model forms.
…legacy Follow-up to the RuleWorld#109 contract. classify_bngsim_route returned the _classify_bngl_actions_for_bngsim result verbatim, so under an explicit simulator='bngsim' request a BNGL with an action the router doesn't recognize still silently ran on the legacy BNG2.pl stack (ROUTE_SUBPROCESS was never promoted to ROUTE_ERROR). RuleWorld#109 only closed the parse-failure (actions==None) hole; this closes the unrecognized-action hole. BNGsim can run such a model via the BNG2.pl backend hook — the router just can't confirm an unknown action is safe to delegate — so strict mode now surfaces ROUTE_ERROR with the reason instead of degrading silently. PLA and BNGsim-unsupported methods are genuine BNGsim incapabilities, so they stay on the subprocess route in every mode (auto and bngsim alike); auto keeps the legacy fallback for unrecognized actions too. Threads `simulator` into _classify_bngl_actions_for_bngsim (default 'auto', back-compatible) and promotes only the unrecognized-action branch. Adds tests: bngsim unknown-action -> error, auto unknown-action -> subprocess, and bngsim PLA / unsupported-method -> subprocess.
The router lumped "method BNGsim can't run" together as ROUTE_SUBPROCESS,
so a malformed/unknown method (a typo like 'oed', or 'quadratic') silently
ran on the legacy stack in auto AND bngsim modes. BNG2.pl's simulate()
validates the method against its $METHODS hash (cvode/ssa/pla/psa/nf, plus
the 'ode' alias) and itself rejects anything else with "Simulation method
'...' is not a valid option" — so routing an unknown method to legacy just
defers to a less clear BNG2.pl error.
Split the unsupported-method handling (_unsupported_method_route):
- A method in the valid BNG universe that this BNGsim can't run — pla, or nf
without an NFsim build — stays on the subprocess route (genuine/build
incapability), in both modes.
- A method outside the valid set is malformed and now errors in both auto
and bngsim, surfacing it instead of silently degrading.
_KNOWN_BNGL_METHODS = {ode, cvode, ssa, psa, pla, nf, rm} is taken verbatim
from BNG2.pl's $METHODS keys (+ ode alias, + rm RuleMonkey extension), so
valid methods like the cvode alias keep their existing legacy routing rather
than newly erroring.
Unrecognized *actions* are deliberately NOT treated this way: BNG2.pl
supports many actions outside the router's curated list (setVolume,
setOption, readSBML, quit, ...), so auto keeps the legacy fallback for them
(only bngsim errors, per the prior commit). Methods are a closed,
BNG2.pl-validated set; actions are open.
Adds parametrized tests: unknown method -> error in both modes; cvode alias
and nf-without-nfsim -> subprocess (regression guards, no spurious errors).
method=>"rm" routes to BNGsim's RuleMonkey (via the rm->nf rewrite + backend hook) only when RuleMonkey is installed. But the router marked rm supported UNCONDITIONALLY — _method_supported_by_bngsim_for_routing never checked RuleMonkey, unlike nf/NFsim — and classify_bngsim_route took no rulemonkey arg. So an rm model on a BNGsim without RuleMonkey still classified as ROUTE_BNGL_BNGSIM, then run_bngl_with_bngsim (which hardcodes simulator="auto") silently fell back to legacy — invisibly even under simulator='bngsim'. Same silent-downgrade class as RuleWorld#109. Make the classifier RuleMonkey-aware (thread bngsim_has_rulemonkey through classify_bngsim_route -> _classify_bngl_actions_for_bngsim -> _method_supported_by_bngsim_for_routing) so the decision is made up front where strict mode is enforced, not in the simulator-blind runner. Treat nf-without-NFsim and rm-without-RuleMonkey uniformly as *build* incapabilities (_BNGSIM_OPTIONAL_COMPONENT_METHODS): BNGsim could run them with the component installed, so simulator='bngsim' errors (surfacing "install NFsim/RuleMonkey") while auto falls back to legacy. This matches the existing direct BNG-XML nf behavior (classify already errored there under bngsim). NOTE: this revises the BNGL nf-without-NFsim case shipped in the previous commit (was subprocess in both modes) so nf and rm are consistent and the strict-bngsim silent gap is closed for both. pla stays subprocess in both modes (a categorical incapability, not a missing component); the cvode alias keeps its legacy route. Updates tests accordingly and adds rm coverage.
test_atomic_supported_bngl_methods_use_bngsim forced bngsim_available=True but let bngsim_has_nfsim/bngsim_has_rulemonkey default to the ambient module values, which are False in any environment without a feature-complete BNGsim (all CI runners). After "rm" routing was made RuleMonkey-aware, the rm case correctly routes to subprocess there, so the test's hard-coded bngl-bngsim expectation failed on all 15 matrix jobs. Force bngsim_has_nfsim=True and bngsim_has_rulemonkey=True in the test so the "supported methods route to BNGsim" assertion is environment-independent, matching how it already forces bngsim_available. Verified by rerunning the routing + method-defaults suites with both component flags monkeypatched to False (the CI condition): 93 passed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two related fixes to the optional BNGsim bridge, both addressing cases where a model silently ran on the legacy BNG2.pl subprocess when the caller expected BNGsim. One hardens the routing contract; the other removes the most common trigger.
Closes #109
Closes #110
#109 —
simulator='bngsim'must error, not silently fall backUnder
simulator='bngsim', the routing bridge silently routed to the legacy BNG2.pl subprocess whenever the Python parser (modelapi.bngmodel) couldn't inspect a model's actions — e.g. an unrecognizedsimulateargument thatbngmodelrejects but BNG2.pl tolerates. That violates thesimulator='bngsim'contract ("use BNGsim, or error"): the caller got legacy output with no exception and no warning. A parity sweep over 895 models hit this on 9 models._classify_bngl_actions_for_bngsim()returnedROUTE_SUBPROCESSfor the un-inspectable case unconditionally, with no reference tosimulator.Fix (
bionetgen/core/tools/bngsim_bridge.py), at theclassify_bngsim_routecall site wheresimulatoris in scope:simulator='bngsim'+ un-inspectable actions →ROUTE_ERROR, propagating the underlying parse reason so the caller learns why it couldn't route to BNGsim. Both consumers (main.py,runner.run) already raiseBNGSimError(route.reason)onROUTE_ERROR.simulator='auto'keeps the subprocess fallback but emits a one-time WARNING per file.simulator='subprocess'is unchanged (returns before action inspection).To surface the reason, the routing parse now returns
(actions, reason)through the memoized loader (_load_bngl_routing_actions;_load_bngl_actions_for_routingkept as a thin list-only wrapper for its other callers). The unavailable-BNGsim case already returnedROUTE_ERRORand is untouched.#110 — action grammar: accept e-notation and trailing commas in list args
ActionList.define_parser(bionetgen/core/utils/utils.py) was stricter than BNG2.pl/Perl for list-valued arguments.arg_type_listmatched elements withpp.Word(pp.nums + ".")(digits and.only) inside a plainpp.delimitedList, so it rejected:par_scan_vals=>[2.3e-10,5.1e-10]par_scan_vals=>[1,2,3,]Both are valid Perl that BNG2.pl parses and runs. When a model used either (commonly
par_scan_valsonparameter_scan),bngmodelraisedBNGParseError— which is exactly what triggered the #109 silent-fallback class. Real models hit this: RuleHub'sMitra2019/15-igf1rfits (e-notation and trailing commas) andSalazar-Cavazos2019CHO_EGFR best-fit (e-notation).Fix: broaden
arg_type_listto usearg_type_expr(which already spanse/Eand+/-), allow an empty list, and tolerate one optional trailing comma:Still rejects genuinely-malformed lists — double commas (
[1,,2]) and unclosed lists ([1,2,).#109 follow-up — strict mode also errors on unrecognized actions
#109's first fix closed the parse-failure hole (actions can't be inspected →None). Butclassify_bngsim_routereturned the_classify_bngl_actions_for_bngsimresult verbatim, so under an explicitsimulator='bngsim'request a BNGL with an action the router doesn't recognize still silently ran on the legacy stack (itsROUTE_SUBPROCESSwas never promoted toROUTE_ERROR).BNGsim can run such a model via the BNG2.pl backend hook — the router just can't confirm an unknown action is safe to delegate — so strict mode now surfaces
ROUTE_ERRORwith the reason instead of degrading silently. PLA and BNGsim-unsupported methods are genuine BNGsim incapabilities, so they stay on the subprocess route in every mode;autokeeps the legacy fallback for unrecognized actions too.simulatoris threaded into_classify_bngl_actions_for_bngsim(default'auto', back-compatible).#109 follow-up — unknown simulation methods error instead of silent legacy
The router treated every BNGsim-unsupported method the same (
ROUTE_SUBPROCESS), so a malformed/unknown method — a typo likeoed, orquadratic— silently ran on the legacy stack in bothautoandbngsim. BNG2.pl's ownsimulate()validates the method against its$METHODShash (cvode/ssa/pla/psa/nf, plus theodealias) and rejects anything else ("Simulation method '...' is not a valid option"), so this is faithful, not stricter than BNG2.pl.Now split (
_unsupported_method_route):pla, ornfwithout an NFsim build — stays on the subprocess route (genuine/build incapability), in both modes._KNOWN_BNGL_METHODS = {ode, cvode, ssa, psa, pla, nf, rm}is taken verbatim from BNG2.pl's$METHODSkeys (+odealias, +rmRuleMonkey extension), so valid methods like thecvodealias keep their existing legacy routing rather than newly erroring.Note the deliberate asymmetry with actions: BNG2.pl supports many actions outside the router's curated list (
setVolume,setOption,readSBML,quit, …), soautokeeps the legacy fallback for unrecognized actions (onlybngsimerrors). Methods are a closed, BNG2.pl-validated set; actions are open.#109 follow-up —
rm/RuleMonkey awareness; build incapabilities error under strict modemethod=>"rm"routes to BNGsim's RuleMonkey (via therm→nfrewrite + backend hook) only when RuleMonkey is installed — but the router markedrmsupported unconditionally (_method_supported_by_bngsim_for_routingnever checked RuleMonkey, unlikenf/NFsim), andclassify_bngsim_routetook no RuleMonkey arg. So anrmmodel on a BNGsim without RuleMonkey still classified asROUTE_BNGL_BNGSIM, thenrun_bngl_with_bngsim(which hardcodessimulator="auto") silently fell back to legacy — invisibly even undersimulator='bngsim'.Now the classifier is RuleMonkey-aware (
bngsim_has_rulemonkeythreaded through), so the decision is made up front where strict mode is enforced, not in the simulator-blind runner.nf-without-NFsim andrm-without-RuleMonkey are treated uniformly as build incapabilities (_BNGSIM_OPTIONAL_COMPONENT_METHODS):simulator='bngsim'errors (surfacing "install NFsim/RuleMonkey"),autofalls back. This matches the existing direct BNG-XMLnfbehavior.plastays subprocess in both modes (a categorical incapability, not a missing component); thecvodealias keeps its legacy route.Why together
#110 makes these models parseable; #109 ensures that when something still isn't inspectable,
bngsimerrors instead of silently degrading. Shipping them together closes the silent-fallback gap end to end.Tests
tests/test_bngsim_routing_classifier.py: bngsim error path, auto one-time-warning fallback, subprocess no-inspect path, reason propagation through the cache, and an end-to-end repro of the unrecognized-argument model.tests/test_bng_parsing.py: parametrized accept/reject grammar tests over the issue Action grammar rejects scientific-notation and trailing commas in list args (par_scan_vals) — stricter than BNG2.pl, causes silent legacy fallback under simulator='bngsim' #110 matrix and the affected real-model forms.