[DRAFT] chore: synchronize canonical upstream v0.19.0 #2
Loading…
Reference in a new issue
No description provided.
Delete branch "audit/upstream-sync"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Summary
b8d934b(v0.9.2) to canonicalstephenschoettler/hermes-lcm2d8385614bbc384769036d7fd11c5f70aa8ccca2(v0.19.0): 224 upstream commits, zero fork-only commits.data:payload is required to round-trip with its existinglegacyprefix. This is a test-only correction; it preserves the upstream runtime behavior and should be offered upstream separately.7daa69c: the static validation job now provisions Go before installing actionlint, and the slow pytest matrix runs two deterministic, complete shards.Compatibility and risk review
The sync carries upstream changes to the plugin contract (
plugin.yamlnow v0.19.0), configuration/engine behavior, operational scripts, and CI/release workflows. The full suite exercises the declared tool schema and engine contract. No live Hermes plugin, configuration, state database, gateway, or user session was changed by this audit.The pre-existing PR #1 (
fix/preserve-active-user-objective) is superseded in substance by upstream commitc626d24(fix: preserve latest user objective during compaction (#132)). It should be closed rather than merged after this synchronization PR lands, unless review identifies a material downstream-only difference.Validation
python3 -m pytest tests/ -q— 1605 passed, 12 xfailed (43 existing Python 3.13 import-layout deprecation warnings)python3 scripts/run_test_shard.py --index 0 --total 2— 786 passed, 6 xfailed; shard 2 — 819 passed, 6 xfailed. The runner preserves pytest collection order and ignores runner-global Git configuration so Git-LFS hooks cannot contaminate fixture repositories./tmp/hermes-lcm-review-bin/actionlint .github/workflows/*.yml(actionlint v1.7.7) — passedpython3 -m compileall -q .python3 -m py_compile scripts/import_lossless_claw.py scripts/run_test_shard.pybash -n scripts/install.sh scripts/update.shgit diff --checkThe checkout does not contain a runnable
hermesexecutable, so a live-host integration probe was intentionally not fabricated; the upstream CI-compatibleagent.context_engine.ContextEnginecontract stub was used for the local test suite.CI remediation and post-fix run
static-validationprovisionsactions/setup-goat immutable commit40f1582b2485089dde7abd97c1529aa768e1baff(v5.6.0) beforego install github.com/rhysd/actionlint/cmd/actionlint@v1.7.7; this directly fixes the priorgo: command not foundfailure.ulimit -n 1024protection. The partition is SHA-256 based, non-empty, and its union is the collected test set.waitingstate as of this update. It has not been represented as a passing run; the branch is ready for review once a runner dispatches it.Migration, rollback, and deployment notes
lcm_backup/SQLite snapshot and retain externalized payload storage./resetis required only when deploying the merged plugin to load new code; neither was performed here.maincommitb8d934b58f43b54622d56a164488849cd05e4780; restore the pre-deploy SQLite snapshot if state recovery is required. Do not downgrade a database whose schema guard reports it is newer than the running plugin.7daa69cfb26890d64f90006d9d6dd83e098da0b0; no runtime release artifact or state migration is involved.Fork policy recommendation
Treat this repository as a preparation fork, not an independent distribution: keep
maincontent-equivalent to canonical upstream and import upstream via signed PRs. Keep only narrowly justified, documented downstream changes on review branches, upstream them promptly, and remove them once accepted. Require a fresh compatibility/backup audit for any upstream release with configuration, schema, plugin-contract, or operational-script changes.Merge sequence: wait for Forgejo CI to conclude successfully, review this draft, rebase/fast-forward through the protected PR workflow, then close PR #1 as superseded. Do not direct-push or force-push
main.Background ========== `MessageStore` opens its SQLite connection with `check_same_thread=False` and shares it across all threads in the host process. SQLite's own C-level mutex protects engine-internal state, but Python's `sqlite3` module releases the GIL while the C call runs. Under heavy thread contention with concurrent HTTPS API clients in the same process, downstream operators have observed on-disk corruption that is consistent with external bytes landing inside SQLite's write path. Observed signature ================== On two independent LCM databases corrupted in the same Hermes deployment, the first 28 bytes of the database file were replaced with a TLS 1.2 Application Data record header + ciphertext while bytes 0-4 ("SQLit") remained intact: Healthy: 53 51 4c 69 74 65 20 66 6f 72 6d 61 74 20 33 00 "SQLite format 3\\0" Corrupt: 53 51 4c 69 74 17 03 03 00 13 XX XX XX XX XX XX Bytes 5-9 (\x17 \x03 \x03 \x00 \x13) are an exact TLS 1.2 `ContentType.application_data` record header with length=19. Pages beyond the header were intact in both cases, and full row-count + integrity recovery was possible by restoring the canonical 100-byte SQLite header. SQLite cannot produce these bytes. The only source in the process at the time of corruption was a Python HTTPS client (openai/httpx). The pattern is consistent with concurrent in-process TLS buffer state intersecting SQLite's write/flush path while the GIL was released. Fix === Add a single `threading.RLock` (`self._write_lock`) on `MessageStore` and wrap every call site that mutates `self._conn` (`append`, `append_batch`, `reassign_session_messages`, `delete_session_messages`, `gc_externalized_tool_result`, `pin`, `unpin`, `normalize_legacy_blank_sources`). The lock is re-entrant so nested write call sites are safe, and it is acquired alongside the existing `with self._conn:` transaction managers where present so the SQLite-level transaction and the Python-level mutex enter/exit together. The lock is uncontended in single-threaded callers and adds only a single `acquire`/`release` per write operation. `unpin` had no prior commit but matched `pin` semantically; `commit` is now consistent across both. Tests ===== New `test_write_lock_serializes_concurrent_appends` (under `TestMessageStore` in `tests/test_lcm_core.py`) exercises the contract: - asserts `MessageStore._write_lock` exists and is re-entrant - spins up 8 worker threads doing 25 single appends + 5 batches of 5 - asserts no exceptions, exact expected row count, and `PRAGMA quick_check = ok` afterward Out of scope for this PR ======================== `LifecycleStateStore` in `lifecycle_state.py` uses the same shared `check_same_thread=False` connection pattern and would benefit from the same lock. Tracking that as a follow-up so this PR stays small and easy to review. Validation ========== - `pytest tests/test_lcm_core.py tests/test_lcm_engine.py tests/test_packaging_install.py -q` → 588 passed - `pytest -q` (full suite) → 817 passed, 2 pre-existing failures in `test_ingest_protection.py` unrelated to this change (verified against unpatched main HEAD). - `python -m compileall -q .` clean - `python -m py_compile scripts/import_lossless_claw.py` clean - `bash -n scripts/install.sh scripts/update.sh` clean - `git diff --check` clean Co-authored-by: Gilfoyle (RVF infra) <gilfoyle@riftvalleyfarm.com> Co-authored-by: Voscko <Tosko4@users.noreply.github.com>900270fd1f' into turing/t_b51ad833-pr320-ff-update d5ed08e3e0Replace the ~35 repetitive `c.X = _int/_float/_bool/_str("LCM_X", c.X)` assignments in `LCMConfig.from_env` with a single declarative registry `ENV_FIELD_SPECS` (name, env_key, py_type) applied by a uniform loop, and let `presets.py` derive its preset-field env/parse lookups from the same registry instead of two hand-maintained parallel dicts. Source-tracked fields (provenance recording and/or computed defaults -- fresh_tail_count, leaf_chunk_tokens, context_threshold, the summary_spend_* and summary_timeout_ms fields, codex autoraise) stay explicit and are skipped by the loop; pattern-list overrides with a source sidecar and the inline gc_max_age_hours parse also stay explicit. Behaviour-preserving: verified byte-identical with an equivalence harness that snapshots `from_env()` (full asdict + config_sources + config_source_warnings) and the preset override discovery across 600 scenarios (every LCM_* key absent / valid / invalid) -- 0 differing scenarios. Full suite: 1545 passed, 12 xfailed. Removes the config/presets field-mapping duplication the design flagged, giving one source of truth for the scalar LCM_* env overrides. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>chore: synchronize canonical upstream v0.19.0to [DRAFT] chore: synchronize canonical upstream v0.19.0Closed because this cross-project audit should not modify upstream forks without a separate explicit contribution request. The campaign has been stopped and this work is being reverted/removed from scope.
Pull request closed