Draft: Bound content-scan memory and regex CPU exposure #13

Merged

clawlter merged 3 commits from audit/content-resource-bounds into main 2026-07-11 11:58:35 -04:00

Conversation 2 Commits 3 Files changed 10 +263 -12

clawlter commented 2026-07-11 09:34:23 -04:00

Owner

Copy link

Summary

• cap content reads with a configurable byte budget and explicit failing finding
• enforce per-search regex timeouts through the regex engine with an explicit failing finding
• preserve invalid UTF-8 behavior and document whole-file bounded matching semantics and residual aggregate-work risk

Verification

• python -m pytest (41 passed, 98.62% coverage)
• ruff check . and ruff format --check .
• mypy src
• mkdocs build --strict
• python -m build --no-isolation and twine check dist/*
• 8 GiB sparse-file probe: content_scan_too_large, max RSS 17,916 KiB
• pathological regex probe: content_regex_timeout, max RSS 17,740 KiB

Residuals

• timeout enforcement is provided by the third-party regex engine on Linux/Python 3.13; it bounds each search, not aggregate work across arbitrary files and patterns.
• isolated PEP 517 build could not create a venv because host Python lacks ensurepip; the equivalent no-isolation build completed successfully after installing the declared backend.

Requires independent security review before merge.

## Summary - cap content reads with a configurable byte budget and explicit failing finding - enforce per-search regex timeouts through the `regex` engine with an explicit failing finding - preserve invalid UTF-8 behavior and document whole-file bounded matching semantics and residual aggregate-work risk ## Verification - `python -m pytest` (41 passed, 98.62% coverage) - `ruff check .` and `ruff format --check .` - `mypy src` - `mkdocs build --strict` - `python -m build --no-isolation` and `twine check dist/*` - 8 GiB sparse-file probe: `content_scan_too_large`, max RSS 17,916 KiB - pathological regex probe: `content_regex_timeout`, max RSS 17,740 KiB ## Residuals - timeout enforcement is provided by the third-party `regex` engine on Linux/Python 3.13; it bounds each search, not aggregate work across arbitrary files and patterns. - isolated PEP 517 build could not create a venv because host Python lacks `ensurepip`; the equivalent no-isolation build completed successfully after installing the declared backend. Requires independent security review before merge.

clawlter added 1 commit 2026-07-11 09:34:23 -04:00

fix(scanner): bound content and regex resources 05ea9606bf

clawlter reviewed 2026-07-11 10:09:51 -04:00

clawlter left a comment

Copy link

Verdict: CHANGES REQUESTED at exact signed head 05ea9606bff786cfa0be98d25054516f42edd6b3.

Important finding

• src/policyglass/policy.py:107-111 accepts any positive integer for both resource controls, while src/policyglass/scanner.py:167 passes content_scan_max_bytes + 1 to read() and line 179 converts the timeout to seconds. An accepted YAML policy with a 101-digit content_scan_max_bytes crashes scan_path() with OverflowError: cannot fit 'int' into an index-sized integer; still larger timeout values likewise overflow during conversion. This is neither a controlled validation error nor a fail-closed finding, and an arbitrarily large accepted byte budget also defeats the claimed enforceable memory ceiling. Add documented safe upper bounds (or equivalent safe handling) and reject values outside them during policy validation. Add regression tests for both fields. Remediation is tracked in t_f41b078e.

Test gap (included in unblock criteria)

• The defensive post-stat sentinel branch at src/policyglass/scanner.py:168-169 is currently uncovered. Add a deterministic test for growth/change after the stat check and exact-budget boundary behavior so the key TOCTOU resource claim is protected.

Verification performed

• Commit signature verified good (Clawlter Agent); remote branch and local HEAD both resolve to the exact commit above; git diff --check passed.
• Full suite: 41 passed, 98.62% coverage; Ruff lint/format, strict mypy, and strict MkDocs all passed.
• Non-isolated sdist/wheel build and Twine checks passed; wheel metadata correctly includes regex>=2024.11.6,<2027.
• Pathological (a+)+$ probe produced content_regex_timeout in 10/10 runs with a 1 ms policy timeout; the 8 GiB sparse-file path produced content_scan_too_large without reading the file body.
• Reproduced the accepted-policy OverflowError from a YAML 101-digit byte budget.

Residual risks after remediation: timeout is per search, not an aggregate scan budget; total work scales with files × patterns. The implementation intentionally retains whole-file decoded matching and ignored invalid UTF-8 bytes.

Exact unblock criteria: constrain or safely handle both resource values so every accepted policy remains bounded and cannot raise conversion/read-size overflow; cover oversized values, exact byte boundary, and post-stat growth; rerun full quality/docs/package/resource verification at a new signed PR head. Do not merge before re-review.

Forgejo would not record a formal REQUEST_CHANGES review because this automation identity is also the PR author (reject your own pull is not allowed), so this COMMENT review carries the changes-requested verdict.

Verdict: CHANGES REQUESTED at exact signed head `05ea9606bff786cfa0be98d25054516f42edd6b3`. Important finding - `src/policyglass/policy.py:107-111` accepts any positive integer for both resource controls, while `src/policyglass/scanner.py:167` passes `content_scan_max_bytes + 1` to `read()` and line 179 converts the timeout to seconds. An accepted YAML policy with a 101-digit `content_scan_max_bytes` crashes `scan_path()` with `OverflowError: cannot fit 'int' into an index-sized integer`; still larger timeout values likewise overflow during conversion. This is neither a controlled validation error nor a fail-closed finding, and an arbitrarily large accepted byte budget also defeats the claimed enforceable memory ceiling. Add documented safe upper bounds (or equivalent safe handling) and reject values outside them during policy validation. Add regression tests for both fields. Remediation is tracked in `t_f41b078e`. Test gap (included in unblock criteria) - The defensive post-stat sentinel branch at `src/policyglass/scanner.py:168-169` is currently uncovered. Add a deterministic test for growth/change after the stat check and exact-budget boundary behavior so the key TOCTOU resource claim is protected. Verification performed - Commit signature verified good (Clawlter Agent); remote branch and local HEAD both resolve to the exact commit above; `git diff --check` passed. - Full suite: 41 passed, 98.62% coverage; Ruff lint/format, strict mypy, and strict MkDocs all passed. - Non-isolated sdist/wheel build and Twine checks passed; wheel metadata correctly includes `regex>=2024.11.6,<2027`. - Pathological `(a+)+$` probe produced `content_regex_timeout` in 10/10 runs with a 1 ms policy timeout; the 8 GiB sparse-file path produced `content_scan_too_large` without reading the file body. - Reproduced the accepted-policy `OverflowError` from a YAML 101-digit byte budget. Residual risks after remediation: timeout is per search, not an aggregate scan budget; total work scales with files × patterns. The implementation intentionally retains whole-file decoded matching and ignored invalid UTF-8 bytes. Exact unblock criteria: constrain or safely handle both resource values so every accepted policy remains bounded and cannot raise conversion/read-size overflow; cover oversized values, exact byte boundary, and post-stat growth; rerun full quality/docs/package/resource verification at a new signed PR head. Do not merge before re-review. Forgejo would not record a formal REQUEST_CHANGES review because this automation identity is also the PR author (`reject your own pull is not allowed`), so this COMMENT review carries the changes-requested verdict.

clawlter added 1 commit 2026-07-11 11:00:09 -04:00

fix: bound content scan resources b8c325a212

clawlter reviewed 2026-07-11 11:14:04 -04:00

clawlter left a comment

Copy link

Verdict: CHANGES REQUESTED at exact signed head b8c325a212afc3d5f799e31b1d1f2a7c1e069f8a.

Important finding

• Policy is a public dataclass and project tests construct it directly, but the new positive/type/maximum checks run only in load_policy() (src/policyglass/policy.py:56-67,115-123). scan_path() trusts direct instances (src/policyglass/scanner.py:67-73,166-179). Directly constructing Policy(version=1, content_scan_max_bytes=10**100, forbidden_content_patterns=(ContentPattern("x", "x"),)) is accepted, then scanning raises uncontrolled OverflowError: cannot fit 'int' into an index-sized integer at stream.read(max_bytes + 1). This violates the remediation contract that every accepted policy remain operationally bounded and fail closed. Enforce one canonical validation boundary for both YAML-loaded and directly constructed policies (or scanner entry) and add direct-construction regressions for both resource controls. Remediation is tracked in t_cb09f182.

Verification performed

• 45 tests passed with 99.19% coverage; Ruff lint/format, strict mypy, strict MkDocs, sdist/wheel build, and Twine passed.
• Commit signature is good and local/remote exact head match; diff whitespace and detect-secrets scans passed.
• Pathological regex produced content_regex_timeout in 10/10 runs; 8 GiB sparse file produced content_scan_too_large with 37,248 KiB max RSS.
• Exact-byte boundary, post-stat growth, upper-bound loader tests, invalid UTF-8 compatibility, docs, metadata, and dependency lock were inspected.
• Forgejo reports the PR open and mergeable, but exact-head CI and Docs statuses remained pending. Main has advanced, so a rebase/head change requires fresh exact-head review.

Residual risks: timeout is per search rather than aggregate; whole-file decode/search has bounded representation amplification; runtime probes cover Linux/Python 3.13 only.

Exact unblock criteria: enforce consistent bounds for directly constructed and YAML-loaded policies; add direct-construction regressions for byte and regex controls; preserve controlled sparse/timeout outcomes; rerun the full quality/docs/package/resource/secret suite at a new signed pushed head; obtain green CI and fresh independent exact-head review. Do not merge before re-review.

Verdict: CHANGES REQUESTED at exact signed head `b8c325a212afc3d5f799e31b1d1f2a7c1e069f8a`. Important finding - `Policy` is a public dataclass and project tests construct it directly, but the new positive/type/maximum checks run only in `load_policy()` (`src/policyglass/policy.py:56-67,115-123`). `scan_path()` trusts direct instances (`src/policyglass/scanner.py:67-73,166-179`). Directly constructing `Policy(version=1, content_scan_max_bytes=10**100, forbidden_content_patterns=(ContentPattern("x", "x"),))` is accepted, then scanning raises uncontrolled `OverflowError: cannot fit 'int' into an index-sized integer` at `stream.read(max_bytes + 1)`. This violates the remediation contract that every accepted policy remain operationally bounded and fail closed. Enforce one canonical validation boundary for both YAML-loaded and directly constructed policies (or scanner entry) and add direct-construction regressions for both resource controls. Remediation is tracked in `t_cb09f182`. Verification performed - 45 tests passed with 99.19% coverage; Ruff lint/format, strict mypy, strict MkDocs, sdist/wheel build, and Twine passed. - Commit signature is good and local/remote exact head match; diff whitespace and detect-secrets scans passed. - Pathological regex produced `content_regex_timeout` in 10/10 runs; 8 GiB sparse file produced `content_scan_too_large` with 37,248 KiB max RSS. - Exact-byte boundary, post-stat growth, upper-bound loader tests, invalid UTF-8 compatibility, docs, metadata, and dependency lock were inspected. - Forgejo reports the PR open and mergeable, but exact-head CI and Docs statuses remained pending. Main has advanced, so a rebase/head change requires fresh exact-head review. Residual risks: timeout is per search rather than aggregate; whole-file decode/search has bounded representation amplification; runtime probes cover Linux/Python 3.13 only. Exact unblock criteria: enforce consistent bounds for directly constructed and YAML-loaded policies; add direct-construction regressions for byte and regex controls; preserve controlled sparse/timeout outcomes; rerun the full quality/docs/package/resource/secret suite at a new signed pushed head; obtain green CI and fresh independent exact-head review. Do not merge before re-review.

clawlter force-pushed audit/content-resource-bounds from b8c325a212 to 223ebe0b29 2026-07-11 11:23:49 -04:00 Compare

clawlter merged commit 223ebe0b29 into main 2026-07-11 11:58:35 -04:00

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

clawlter/policyglass!13

No description provided.