BF: restrict value of get_fdmax

[yarikoptic]

In my case I kept finding celery running at 100% and doing nothing. py-spy pointed to the close_open_fds and then ulimit inside the container showed gory detail of

❯ docker run -it --rm --entrypoint bash dandiarchive/dandiarchive-api -c "ulimit -n"
1073741816

situation is not unique to me. See more at

• test_publish_and_manipulate and test_get_dandiset_published fail locally dandi/dandi-cli#1488

I verified that with this fix my celery container gets unstuck and proceeds to report useful errors ;)

[yarikoptic]

hm, why pre-commit.ci is even configured if there is no .pre-commit-config.yaml?

[auvipy]

lets ignore the pre commit. can you elaborate more on the change please? also should we also consider adding some tests to verify the proposed changes?

[yarikoptic]

I would be happy to elaborate! ATM I can only reiterate what tried to describe in original description -- on some systems ulimit would return HUGE number for maximal number of open descriptiors, which would be infeasible to loop through. So, billiard should not try to loop through all the possible billion of them.

[auvipy]

May be we can add some unit tests for the suggested changes as well

[yarikoptic]

sorry -- too busy ATM, feel welcome to contribute some by directly pushing to my branch for this PR, I did

[Copilot]

Pull Request Overview

The PR adds logic to cap and warn about excessively high file descriptor limits returned by get_fdmax, preventing performance issues when iterating open descriptors.

• Capture os.sysconf('SC_OPEN_MAX') into fdmax and handle errors uniformly
• Introduce a threshold (100k) to cap fdmax to a sensible default (either the passed default or 10 000) and emit a warning
• Import warnings and emit a deprecation-style warning instead of returning an oversized limit

Comments suppressed due to low confidence (1)

billiard/compat.py:118

• Consider adding unit tests for the new high-value cap branch to verify that large fdmax values produce the expected warning and capped return value.

    if fdmax >= 1e5:

[GangEunzzang]

@yarikoptic @auvipy Thanks for the report and the reproduction — this is the same container setting (ulimit -n = 1073741816) that celery/celery#9886 ran into, and the same function.

Since #455 landed, close_open_fds() no longer walks that range in Python: it calls os.closerange() on the gaps between the kept descriptors, which is close_range(2) on Linux and finishes in ~0 ms at that limit. So the 100% CPU described here should already be gone on main.

What #455 doesn't cover is where close_range(2) isn't available (seccomp denying it, older kernels, macOS) — there os.closerange() falls back to a C loop that still takes about 3 minutes at 1e9. I've sent #456 for that: it lists the fd directory (/proc/self/fd / /dev/fd) and closes only the descriptors actually open, the way CPython's subprocess does, keeping the closerange() path as the fallback. With that, get_fdmax() isn't consulted at all on the fd-directory path, so its return value doesn't need to be capped.