feat(image-gen): support Codex image editing #1

Open

clawlter wants to merge 1 commit from feat/openai-codex-image-editing into main

pull from: feat/openai-codex-image-editing

merge into: clawlter:main

clawlter:main

Conversation 0 Commits 1 Files changed 3 +160 -34

clawlter commented 2026-06-20 00:49:06 -04:00

Owner

Copy link

What does this PR do?

Adds image-to-image / editing support to the OpenAI Codex-authenticated image backend.

Hermes already accepts image_url and reference_image_urls at the image-generation provider boundary, but the openai-codex provider advertised text-only capability and rejected image inputs. This PR keeps the existing Codex Responses image_generation tool path and attaches source/reference images as Responses input_image content parts. Public URLs and existing data URLs pass through directly; local file paths are inlined as data:image/...;base64,... URLs so the Codex backend can read them.

This makes the Codex-auth backend match the regular OpenAI gpt-image-2 provider's editing capability while preserving the existing text-to-image behavior when no image inputs are supplied.

Related Issue

N/A — no issue linked.

Type of Change

• 🐛 Bug fix (non-breaking change that fixes an issue)
• ✨ New feature (non-breaking change that adds functionality)
• 🔒 Security fix
• 📝 Documentation update
• ✅ Tests (adding or improving test coverage)
• ♻️ Refactor (no behavior change)
• 🎯 New skill (bundled or hub)

Changes Made

• plugins/image_gen/openai-codex/__init__.py
  • Advertises modalities: ["text", "image"] with max_reference_images: 16.
  • Accepts image_url and reference_image_urls instead of returning modality_unsupported.
  • Normalizes and clamps image sources to the provider cap.
  • Builds Codex Responses payloads with input_text plus one input_image part per source/reference.
  • Converts local image paths to data URLs; forwards HTTP(S) URLs and existing data URLs unchanged.
  • Marks successful edit/image-to-image calls with modality: "image".
• tests/plugins/image_gen/test_openai_codex_provider.py
  • Adds coverage for capability reporting, image-input payload shape, source forwarding, and reference-image clamping.
• website/docs/user-guide/features/image-generation.md
  • Updates the backend support table to document Codex-auth image editing support.

How to Test

1. Run the provider tests:

   scripts/run_tests.sh tests/plugins/image_gen/test_openai_codex_provider.py
   python -m pytest tests/plugins/image_gen/test_openai_codex_provider.py -q
   

2. Run the affected image-generation suites:

   scripts/run_tests.sh      tests/plugins/image_gen      tests/tools/test_image_generation.py      tests/tools/test_image_generation_artifacts.py      tests/tools/test_image_generation_image_to_image.py      tests/tools/test_image_generation_plugin_dispatch.py      tests/agent/test_image_gen_registry.py      tests/agent/test_image_routing.py
   

3. Run the direct pytest suite used for the branch smoke gate:

   python -m pytest      tests/plugins/image_gen      tests/tools/test_image_generation.py      tests/tools/test_image_generation_image_to_image.py      tests/tools/test_image_generation_plugin_dispatch.py      -q
   

4. Run syntax and whitespace checks:

   python -m py_compile      plugins/image_gen/openai-codex/__init__.py      tests/plugins/image_gen/test_openai_codex_provider.py
   git diff --check
   

5. Optional live smoke test with Codex auth configured:
   • Call OpenAICodexImageGenProvider.generate(...) with a local image_url path.
   • Confirm the response has success: true, provider: "openai-codex", and modality: "image".
   • Confirm the output file is a real PNG and reflects the requested edit.

Checklist

Code

• I've read the Contributing Guide
• My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
• I searched for existing PRs to make sure this isn't a duplicate
• My PR contains only changes related to this fix/feature (no unrelated commits)
• I've run pytest tests/ -q and all tests pass — not checked because this branch was validated with the canonical affected-suite runs below; current main is already green and this branch only touches the Codex image-provider path
• I've added tests for my changes (required for bug fixes, strongly encouraged for features)
• I've tested on my platform: Linux 5.14 / Python 3.13.5, focused test suites plus live Codex OAuth smoke test

Documentation & Housekeeping

• I've updated relevant documentation (README, docs/, docstrings) — website/docs/user-guide/features/image-generation.md
• I've updated cli-config.yaml.example if I added/changed config keys — N/A, no config keys changed
• I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — N/A, no architecture/workflow change
• I've considered cross-platform impact (Windows, macOS) per the compatibility guide — image path handling uses Python file APIs; no shell/path assumptions added
• I've updated tool descriptions/schemas if I changed tool behavior — dynamic provider capabilities now expose image support for openai-codex

For New Skills

N/A — this PR does not add a skill.

Screenshots / Logs

Branch state after rebasing onto current main:

branch=feat/openai-codex-image-editing
HEAD=0b500ddd0c5a474120d18399183c0bcdac1a12f0
origin/main=b88d0007c9d0037a1ec3daa2477bd4f79eaf566b
merge-base=b88d0007c9d0037a1ec3daa2477bd4f79eaf566b
status_short=<>

Changed files:

M plugins/image_gen/openai-codex/__init__.py
M tests/plugins/image_gen/test_openai_codex_provider.py
M website/docs/user-guide/features/image-generation.md

Validation performed after rebase:

$ scripts/run_tests.sh tests/plugins/image_gen/test_openai_codex_provider.py
=== Summary: 1 files, 22 tests passed, 0 failed (100% complete) in 5.7s ===

$ python -m pytest tests/plugins/image_gen/test_openai_codex_provider.py -q
22 passed in 2.26s

$ scripts/run_tests.sh tests/plugins/image_gen tests/tools/test_image_generation.py tests/tools/test_image_generation_artifacts.py tests/tools/test_image_generation_image_to_image.py tests/tools/test_image_generation_plugin_dispatch.py tests/agent/test_image_gen_registry.py tests/agent/test_image_routing.py
=== Summary: 11 files, 284 tests passed, 0 failed (100% complete) in 16.3s ===

$ python -m pytest tests/plugins/image_gen tests/tools/test_image_generation.py tests/tools/test_image_generation_image_to_image.py tests/tools/test_image_generation_plugin_dispatch.py -q
193 passed in 13.06s

$ python -m py_compile plugins/image_gen/openai-codex/__init__.py tests/plugins/image_gen/test_openai_codex_provider.py
# passed

$ git diff --check
# passed

TDD red run before implementation:

4 failed, 18 passed

Live Codex OAuth smoke test against the modified provider:

{
  "success": true,
  "provider": "openai-codex",
  "model": "gpt-image-2-medium",
  "modality": "image",
  "aspect_ratio": "square",
  "size": "1024x1024",
  "quality": "medium"
}

The live output was a real PNG generated from a local input image; visual inspection confirmed the edit preserved the simple bordered composition and changed the target circle from blue to green.

## What does this PR do? Adds image-to-image / editing support to the OpenAI Codex-authenticated image backend. Hermes already accepts `image_url` and `reference_image_urls` at the image-generation provider boundary, but the `openai-codex` provider advertised text-only capability and rejected image inputs. This PR keeps the existing Codex Responses `image_generation` tool path and attaches source/reference images as Responses `input_image` content parts. Public URLs and existing data URLs pass through directly; local file paths are inlined as `data:image/...;base64,...` URLs so the Codex backend can read them. This makes the Codex-auth backend match the regular OpenAI `gpt-image-2` provider's editing capability while preserving the existing text-to-image behavior when no image inputs are supplied. ## Related Issue N/A — no issue linked. ## Type of Change - [ ] 🐛 Bug fix (non-breaking change that fixes an issue) - [x] ✨ New feature (non-breaking change that adds functionality) - [ ] 🔒 Security fix - [x] 📝 Documentation update - [x] ✅ Tests (adding or improving test coverage) - [ ] ♻️ Refactor (no behavior change) - [ ] 🎯 New skill (bundled or hub) ## Changes Made - `plugins/image_gen/openai-codex/__init__.py` - Advertises `modalities: ["text", "image"]` with `max_reference_images: 16`. - Accepts `image_url` and `reference_image_urls` instead of returning `modality_unsupported`. - Normalizes and clamps image sources to the provider cap. - Builds Codex Responses payloads with `input_text` plus one `input_image` part per source/reference. - Converts local image paths to data URLs; forwards HTTP(S) URLs and existing data URLs unchanged. - Marks successful edit/image-to-image calls with `modality: "image"`. - `tests/plugins/image_gen/test_openai_codex_provider.py` - Adds coverage for capability reporting, image-input payload shape, source forwarding, and reference-image clamping. - `website/docs/user-guide/features/image-generation.md` - Updates the backend support table to document Codex-auth image editing support. ## How to Test 1. Run the provider tests: ```bash scripts/run_tests.sh tests/plugins/image_gen/test_openai_codex_provider.py python -m pytest tests/plugins/image_gen/test_openai_codex_provider.py -q ``` 2. Run the affected image-generation suites: ```bash scripts/run_tests.sh tests/plugins/image_gen tests/tools/test_image_generation.py tests/tools/test_image_generation_artifacts.py tests/tools/test_image_generation_image_to_image.py tests/tools/test_image_generation_plugin_dispatch.py tests/agent/test_image_gen_registry.py tests/agent/test_image_routing.py ``` 3. Run the direct pytest suite used for the branch smoke gate: ```bash python -m pytest tests/plugins/image_gen tests/tools/test_image_generation.py tests/tools/test_image_generation_image_to_image.py tests/tools/test_image_generation_plugin_dispatch.py -q ``` 4. Run syntax and whitespace checks: ```bash python -m py_compile plugins/image_gen/openai-codex/__init__.py tests/plugins/image_gen/test_openai_codex_provider.py git diff --check ``` 5. Optional live smoke test with Codex auth configured: - Call `OpenAICodexImageGenProvider.generate(...)` with a local `image_url` path. - Confirm the response has `success: true`, `provider: "openai-codex"`, and `modality: "image"`. - Confirm the output file is a real PNG and reflects the requested edit. ## Checklist ### Code - [x] I've read the [Contributing Guide](https://github.com/NousResearch/hermes-agent/blob/main/CONTRIBUTING.md) - [x] My commit messages follow [Conventional Commits](https://www.conventionalcommits.org/) (`fix(scope):`, `feat(scope):`, etc.) - [x] I searched for [existing PRs](https://github.com/NousResearch/hermes-agent/pulls) to make sure this isn't a duplicate - [x] My PR contains **only** changes related to this fix/feature (no unrelated commits) - [ ] I've run `pytest tests/ -q` and all tests pass — not checked because this branch was validated with the canonical affected-suite runs below; current `main` is already green and this branch only touches the Codex image-provider path - [x] I've added tests for my changes (required for bug fixes, strongly encouraged for features) - [x] I've tested on my platform: Linux 5.14 / Python 3.13.5, focused test suites plus live Codex OAuth smoke test ### Documentation & Housekeeping - [x] I've updated relevant documentation (README, `docs/`, docstrings) — `website/docs/user-guide/features/image-generation.md` - [x] I've updated `cli-config.yaml.example` if I added/changed config keys — N/A, no config keys changed - [x] I've updated `CONTRIBUTING.md` or `AGENTS.md` if I changed architecture or workflows — N/A, no architecture/workflow change - [x] I've considered cross-platform impact (Windows, macOS) per the [compatibility guide](https://github.com/NousResearch/hermes-agent/blob/main/CONTRIBUTING.md#cross-platform-compatibility) — image path handling uses Python file APIs; no shell/path assumptions added - [x] I've updated tool descriptions/schemas if I changed tool behavior — dynamic provider capabilities now expose image support for `openai-codex` ## For New Skills N/A — this PR does not add a skill. ## Screenshots / Logs Branch state after rebasing onto current `main`: ```text branch=feat/openai-codex-image-editing HEAD=0b500ddd0c5a474120d18399183c0bcdac1a12f0 origin/main=b88d0007c9d0037a1ec3daa2477bd4f79eaf566b merge-base=b88d0007c9d0037a1ec3daa2477bd4f79eaf566b status_short=<> ``` Changed files: ```text M plugins/image_gen/openai-codex/__init__.py M tests/plugins/image_gen/test_openai_codex_provider.py M website/docs/user-guide/features/image-generation.md ``` Validation performed after rebase: ```text $ scripts/run_tests.sh tests/plugins/image_gen/test_openai_codex_provider.py === Summary: 1 files, 22 tests passed, 0 failed (100% complete) in 5.7s === $ python -m pytest tests/plugins/image_gen/test_openai_codex_provider.py -q 22 passed in 2.26s $ scripts/run_tests.sh tests/plugins/image_gen tests/tools/test_image_generation.py tests/tools/test_image_generation_artifacts.py tests/tools/test_image_generation_image_to_image.py tests/tools/test_image_generation_plugin_dispatch.py tests/agent/test_image_gen_registry.py tests/agent/test_image_routing.py === Summary: 11 files, 284 tests passed, 0 failed (100% complete) in 16.3s === $ python -m pytest tests/plugins/image_gen tests/tools/test_image_generation.py tests/tools/test_image_generation_image_to_image.py tests/tools/test_image_generation_plugin_dispatch.py -q 193 passed in 13.06s $ python -m py_compile plugins/image_gen/openai-codex/__init__.py tests/plugins/image_gen/test_openai_codex_provider.py # passed $ git diff --check # passed ``` TDD red run before implementation: ```text 4 failed, 18 passed ``` Live Codex OAuth smoke test against the modified provider: ```json { "success": true, "provider": "openai-codex", "model": "gpt-image-2-medium", "modality": "image", "aspect_ratio": "square", "size": "1024x1024", "quality": "medium" } ``` The live output was a real PNG generated from a local input image; visual inspection confirmed the edit preserved the simple bordered composition and changed the target circle from blue to green.

clawlter added 1 commit 2026-06-20 00:49:07 -04:00

feat(image-gen): support Codex image editing 7f03eb662a

clawlter force-pushed feat/openai-codex-image-editing from 7f03eb662a to 0b500ddd0c 2026-06-20 08:08:55 -04:00 Compare

Some checks failed

Hide all checks

Docker Build and Publish / build-arm64 (pull_request) Waiting to run

Details

Docker Build and Publish / merge (pull_request) Blocked by required conditions

Details

osv-scanner.yml / feat(image-gen): support Codex image editing (pull_request) Failing after 0s

Details

Contributor Attribution Check / check-attribution (pull_request) Failing after 1m23s

Details

Docker / shell lint / Lint Dockerfile (hadolint) (pull_request) Failing after 4s

Details

Docker / shell lint / Lint docker/ shell scripts (shellcheck) (pull_request) Failing after 4s

Details

Docker Build and Publish / build-amd64 (pull_request) Has been skipped

Details

Docs Site Checks / docs-site-checks (pull_request) Successful in 19m42s

Details

History Check / check-common-ancestor (pull_request) Successful in 47s

Details

Lint (ruff + ty) / ruff + ty diff (pull_request) Failing after 3s

Details

Lint (ruff + ty) / ruff enforcement (blocking) (pull_request) Successful in 25s

Details

Lint (ruff + ty) / Windows footguns (blocking) (pull_request) Successful in 1m4s

Details

Supply Chain Audit / changes (pull_request) Successful in 47s

Details

Tests / test (1) (pull_request) Failing after 7m23s

Details

Tests / test (2) (pull_request) Failing after 7m43s

Details

Tests / test (3) (pull_request) Failing after 7m47s

Details

Tests / test (4) (pull_request) Failing after 7m33s

Details

Tests / test (5) (pull_request) Failing after 7m9s

Details

Tests / test (6) (pull_request) Failing after 7m7s

Details

Tests / e2e (pull_request) Successful in 2m30s

Details

Typecheck / typecheck (apps/bootstrap-installer) (pull_request) Successful in 2m7s

Details

Typecheck / typecheck (apps/desktop) (pull_request) Successful in 6m45s

Details

Typecheck / typecheck (apps/shared) (pull_request) Successful in 2m6s

Details

Typecheck / typecheck (ui-tui) (pull_request) Successful in 4m52s

Details

Typecheck / typecheck (web) (pull_request) Successful in 2m7s

Details

Typecheck / desktop-build (pull_request) Successful in 6m56s

Details

uv.lock check / uv lock --check (pull_request) Successful in 21s

Details

Supply Chain Audit / Scan PR for critical supply chain risks (pull_request) Successful in 1m13s

Details

Supply Chain Audit / Scan PR for critical supply chain risks-1 (pull_request) Has been skipped

Details

Supply Chain Audit / Check PyPI dependency upper bounds (pull_request) Has been skipped

Details

Supply Chain Audit / Check PyPI dependency upper bounds-1 (pull_request) Successful in 4s

Details

Supply Chain Audit / MCP catalog security review (pull_request) Has been skipped

Details

Supply Chain Audit / MCP catalog security review-1 (pull_request) Successful in 4s

Details

Tests / save-durations (pull_request) Has been skipped

Details

This pull request can be merged automatically.

You are not authorized to merge this pull request.

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin feat/openai-codex-image-editing:feat/openai-codex-image-editing

git switch feat/openai-codex-image-editing

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch main

git merge --no-ff feat/openai-codex-image-editing

git switch feat/openai-codex-image-editing

git rebase main

git switch main

git merge --ff-only feat/openai-codex-image-editing

git switch feat/openai-codex-image-editing

git rebase main

git switch main

git merge --no-ff feat/openai-codex-image-editing

git switch main

git merge --squash feat/openai-codex-image-editing

git switch main

git merge --ff-only feat/openai-codex-image-editing

git switch main

git merge feat/openai-codex-image-editing

git push origin main

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

clawlter/hermes-agent!1

No description provided.