Bound the recursion in exception group split and subgroup - #8584

• This PR follows our AI policy

Summary

BaseExceptionGroup.split and .subgroup recurse into nested groups through vm.call_method, which pushes no Python frame. sys.setrecursionlimit therefore never sees the nesting, and a group nested deeply enough runs the native stack out:

from test.support import exceeds_recursion_limit

e = TypeError(1)
for _ in range(exceeds_recursion_limit()):   # 150000
    e = ExceptionGroup("eg", [e])
e.split(TypeError)

CPython answers RecursionError. RustPython dumps core, and so does the test module that covers it:

0:00:00 load avg: 2.86 [1/1] test_exception_group
timeout: the monitored command dumped core

Lib/test/test_exception_group.py says as much in two skip reasons, TODO: RUSTPYTHON; Segfault, on test_deep_split and test_deep_subgroup. Both are unskipped here.

Fix

The two recursive vm.call_method calls now sit inside vm.with_recursion, which is the Py_EnterRecursiveCall counterpart already used for comparison, repr, __length_hint__ and __subclasscheck__. It measures the native stack rather than counting frames, which is the right budget for recursion that never reaches the evaluator.

Measurements

Deep case, one process each:

Shallow behaviour is untouched, checked case by case against CPython: split and subgroup on a flat group, on a nested one, with a predicate instead of a type, with no match at all, and at depth 100 where neither implementation is anywhere near the limit. Every one agrees.

test_exception_group ran three times after unskipping, 52 tests, SUCCESS each time, and test_except_star, test_traceback, test_asyncio.test_taskgroups, test_baseexception and test_exceptions are green too. Also cargo test --workspace --exclude rustpython_wasm --exclude rustpython-venvlauncher --exclude rustpython-capi (41 binaries, no failures), cargo test in crates/capi, cargo fmt --check, the clippy invocation from CI, and scripts/check_redundant_patches.py on the test file.

Summary by CodeRabbit

• Bug Fixes
  • Added recursion protection when processing nested exception groups.
  • Prevented excessive recursion from causing native stack overflows while preserving existing subgroup and split behavior.