[Image Generation & Edit] Trust Your Critic: Robust Reward Modeling and Reinforcement Learning for Faithful Image Editing and Generation
[Video Generation] FIRM-Video: Check Before You Score for Reliable Text-to-Video Reward Modeling

Why This Project

• Critics are the bottleneck. The FIRM series is built around a simple belief: RL for visual generation only works when the reward model is faithful, stable, and hard to hack.
• Three task-specific data pipelines. FIRM-Edit uses a difference-first scoring pipeline, FIRM-Gen uses a plan-then-score pipeline to reduce MLLM hallucinations, and FIRM-Video uses a check-before-score pipeline to ground video scores in verified evidence.
• Benchmarks for critic reliability. FIRM-Bench provides human-annotated evaluation for editing and generation critics, while FIRM-Video-Bench extends this evaluation to video reward models.
• Reward shaping that actually holds up. CME and QMA are designed to prevent the shortcut behavior that appears when rewards are naively combined.

FIRM At A Glance

Track	Component
Image Editing	FIRM-Edit-370K, FIRM-Edit-8B, FIRM-Qwen-Edit
Image Generation	FIRM-Gen-293K, FIRM-Gen-8B, FIRM-SD-3.5
Video Generation	FIRM-Video-90K, FIRM-Video-8B, FIRM-Video-Bench
Benchmarking	FIRM-Bench, FIRM-Video-Bench

Repository Layout

TrustYourCritic/
├── generation/   # Image Generation RL training and reward serving
└── editing/      # Image Edit RL training, reward serving, reproduction scripts
└── video_generarion/   # Reserved for our planned Video Generation RL code release — stay tuned

Important Notes

• To avoid Python package conflicts, install and run GenRL/EditRL in separate environments.

Quick Start

1) Image Generation RL

cd generation
conda create -n FIRM-Gen python=3.10 -y
conda activate FIRM-Gen
pip install -e .

i ) Launch Reward Server First

python generation/flow_grpo/reward_model_server.py

ii ) Change the Training Configuration

• generation/config/nft_flux2_klein.py
• generation/config/nft_qwen_image.py
• generation/config/nft_zimage_turbo.py
• generation/config/nft.py

iii ) Start Training

bash generation/scripts/train_sd35_sharegpt_qwenvl.sh

2) Image Edit RL

cd editing
conda create -n FIRM-Edit python=3.10 -y
conda activate FIRM-Edit
pip install -e .

i ) Launch Reward Server First

## Change the default ip and port to your perference
python editing/reward_server/reward_server_qwen3_vl_8b_sft.py

ii ) Change the Training Configuration

• editing/config/kontext_nft_qwen3vl_8b_sft.py
• editing/config/kontext_nft_qwen3vl_8b.py
• editing/config/kontext_nft_qwen25vl_32b_non_logits.py

iii ) Start Training

bash editing/examples/train_qwen_image_edit.sh

3) Video Generation RL

Coming soon.

Data Perparation

Image Generation RL

Expected JSON file like:

[
  {"input_prompt": "A cinematic portrait of a fox in snow."}
]

Image Edit RL

Expected dataset layout:

dataset-root/
├── images/
├── train_metadata.jsonl
└── test_metadata.jsonl

Each JSONL line:

{"prompt": "make the sky sunset orange", "image": "images/example.jpg", "requirement": "preserve identity"}

Video Generation RL

Coming soon.

Evaluation

The code and data for FIRM-Bench&FIRM-Bench-Video are hosted on Hugging Face.

We provide inference and evaluation scripts for FIRM-Bench&FIRM-Bench-Video. We recommend deploying the model with vLLM for inference.

FIRM-Bench-Gen

Inference

python FIRM-Bench-Gen/vllm_infer.py \
  --input FIRM-Bench-Gen/bench_v1.jsonl \
  --output FIRM-Bench-Gen/result/xxx.jsonl \
  --image-root FIRM-Bench-Gen/ \
  --api-url xxxxx

MAE Calculation

python FIRM-Bench-Gen/gen_mae.py \
  --gt FIRM-Bench-Gen/result/human_bench_v1.jsonl \
  --pred FIRM-Bench-Gen/result/xxx.jsonl

FIRM-Bench-Edit

Inference

python FIRM-Bench-Edit/vllm_infer.py \
  --input FIRM-Bench-Edit/bench_v1.jsonl \
  --output FIRM-Bench-Edit/result/xxx.jsonl \
  --image-root FIRM-Bench-Edit/ \
  --api-url xxxxx

MAE Calculation

python FIRM-Bench-Edit/edit_mae.py \
  --gt FIRM-Bench-Edit/result/human_bench_v1.jsonl \
  --pred FIRM-Bench-Edit/result/xxx.jsonl

FIRM-Bench-Video

Inference

bash scripts/infer.sh all 32 qwen3vl8b \
  data/firm-video-bench.json \
  http://127.0.0.1:8000/v1 \
  Qwen3-VL-8B-Instruct

Metric Calculation

python scripts/metric.py \
  --gt_file data/firm-video-bench.json \
  --results_dir results

Acknowledgements

This repository was shaped by several open-source projects that pushed RL for image generation and image editing forward:

• flow-grpo
• DiffusionNFT
• Edit-R1