feat: 重构OpenResty模块部分以支持动态编译#13291
Conversation
|
人工简单过了一下 , 想法很棒，希望多打磨，多自己思考一下。最好gpt回了啥就不要直接粘贴到pr里了， 多说说自己的想法和怎么测的。 以下是几个人工看到的问题，具体逻辑还没看完，如果抽空看到问题我会丢一下review i18n没做完，ai偷懒了，只做了zh en nginx_module.go 里有大量openai gpt家模型特别喜欢干的一件事，做一堆碎函数，然后整个业务代码看着跳来跳去。测试函数和某些部分也有gpt最爱的过度防御代码. 以及有一对临时乱丢的全局变量等，代码很不干净 如果可以 很多字符串建议不要内嵌代码，而是用常量 regxp正则建议请统一管理，而不是定义一个全局，否则这种正则表达式找起来很麻烦 最好让大模型自己画一个状态机、泳道图，看看他到底想根据你的目标做什么，并且pr一定要有自己的测试过程和想法。如果可以，希望多打磨打磨思考思考（别把我的回复丢给ai让他继续罐slop）。 |
|
|
我的建议是 尽量收敛逻辑 把静态编译和动态编译分开 减少可能带来的麻烦 |
- remove auto-to-static fallback; dynamic build failure now reports the error and hints switching to static build manually - gate dynamic builds on module support files (Dockerfile.modules + module.catalog.json) instead of version numbers, expose dynamicSupported in the modules API - collect repeated path/status/operate strings into constants - move nginx module regex patterns into utils/re with semantic helpers - reorganize nginx_module.go around the main build flows and inline single-use thin helpers
- build mode radio offers only dynamic and static (auto maps to dynamic for legacy data) - disable the dynamic option with a hint when the installed OpenResty version lacks dynamic build support
Fill in the new nginx module keys for all eleven language files (translations other than zh/en are draft machine translations).
- probe each non-static module's configure params when loading the module list and report dynamicSupport=supported/unsupported up front - normalize the legacy auto build mode to dynamic
- build drawer lists dynamic modules (tagged, hot-reload) and static modules (tagged, full rebuild + container restart) separately - disable the dynamic option per module when its params do not support dynamic build, distinct from the version gate hint - drop the auto build mode wording everywhere and sync all eleven language files
- add a purpose hint explaining dynamic (hot reload) vs static (full rebuild + container restart) - drop the per-module mode tags now that section headers carry the semantics - allow submitting with zero dynamic modules selected when static modules are present, so static-only users can trigger a build
The mirror selected in the build dialog (or CONTAINER_PACKAGE_URL in the app env as fallback) is now forwarded as a build arg so the module builder uses the same apt source as the static build path. test-builder gains a --mirror option.
38935ec to
1acfea2
Compare
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1acfea2871
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| enable: true, | ||
| params: '', | ||
| packages: '', | ||
| buildMode: 'dynamic', |
There was a problem hiding this comment.
When the installed OpenResty version does not support dynamic builds, the dynamic radio button is disabled, but new modules still start with buildMode: 'dynamic'. In that context a user can fill the rest of the form and submit the disabled-but-selected value, creating an enabled dynamic module that the backend later keeps inactive because there is no Dockerfile.modules; the default should switch to static when dynamicSupported is false.
Useful? React with 👍 / 👎.
| const rules = computed(() => ({ | ||
| mirror: [Rules.requiredSelect], | ||
| }; | ||
| modules: staticModules.value.length === 0 ? [Rules.requiredSelect] : [], |
There was a problem hiding this comment.
After a user disables or deletes the last static module, the backend still needs one more build to clear the saved RESTY_CONFIG_OPTIONS_MORE path (staticNginxBuildRequired handles that), but this rule makes the hidden modules field required whenever there are no currently enabled static modules. If there are also no enabled dynamic modules to select, form.validate() fails before /openresty/build is called, so the stale static module remains baked into the image.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Pull request overview
This PR refactors 1Panel’s OpenResty module management to support building and loading dynamic Nginx/OpenResty modules (.so) where possible, reducing the need for full static recompiles and enabling hot-reload behavior. It also adds local Linux scripts for builder/ABI verification and installation diagnostics, plus frontend/UI + i18n updates to expose the new module/build state.
Changes:
- Add a new agent-side dynamic-module build pipeline (target resolution, per-module Docker builds, artifact validation, managed
load_moduleconfig generation, and reload orchestration), while preserving static builds as a fallback path. - Update the OpenResty module UI to surface build mode/status/compatibility and to support targeted dynamic module builds from the build drawer.
- Add Linux test/diagnostic scripts to validate module builder templates, combined load order, hot reload behavior, and to collect on-host install state for triage.
Reviewed changes
Copilot reviewed 27 out of 27 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| scripts/openresty-modules/test-builder.sh | New integration test runner for building and validating dynamic modules against appstore templates and runtime images |
| scripts/openresty-modules/README.md | Documentation for builder tests, diagnostics, and manual test matrix |
| scripts/openresty-modules/diagnose-install.sh | New diagnostic collector for installed OpenResty module state, mounts, artifacts, and container checks |
| frontend/src/views/website/website/nginx/module/operate/index.vue | Module create/edit drawer: add build-mode controls, load order, and last error display |
| frontend/src/views/website/website/nginx/module/index.vue | Module list: show build mode/status/compatibility; switch uses @change; refresh/revert behavior on failure |
| frontend/src/views/website/website/nginx/module/build/index.vue | Build drawer: add dynamic module selection, force rebuild toggle, and hints for dynamic vs static behavior |
| frontend/src/lang/modules/zh.ts | Add new i18n keys for build mode/status/compatibility and dynamic module messaging (zh) |
| frontend/src/lang/modules/zh-Hant.ts | Add new i18n keys (zh-Hant) |
| frontend/src/lang/modules/tr.ts | Add new i18n keys (tr) |
| frontend/src/lang/modules/ru.ts | Add new i18n keys (ru) |
| frontend/src/lang/modules/pt-br.ts | Add new i18n keys (pt-BR) |
| frontend/src/lang/modules/ms.ts | Add new i18n keys (ms) |
| frontend/src/lang/modules/lo.ts | Add new i18n keys (lo) |
| frontend/src/lang/modules/ko.ts | Add new i18n keys (ko) |
| frontend/src/lang/modules/ja.ts | Add new i18n keys (ja) |
| frontend/src/lang/modules/fa.ts | Add new i18n keys (fa) |
| frontend/src/lang/modules/es-es.ts | Add new i18n keys (es-ES) |
| frontend/src/lang/modules/en.ts | Add new i18n keys (en) |
| frontend/src/api/interface/nginx.ts | Extend Nginx module/build request and module model types for buildMode/status/compatibility/artifacts |
| agent/utils/re/re.go | Add regex helpers for validating module package names, artifact paths, and checksums |
| agent/app/service/nginx.go | Wire OpenResty build/update/get-modules flows to the new dynamic module service logic and response shape |
| agent/app/service/nginx_module.go | New core implementation for dynamic module builds, artifact validation, managed config reconciliation, and static-build integration |
| agent/app/service/nginx_module_test.go | Add unit tests for normalization, param sanitization, build selection logic, and compose mount merging |
| agent/app/service/app_utils.go | Preserve module mounts during OpenResty upgrades; prebuild dynamic modules during upgrade; integrate new build flow |
| agent/app/dto/response/nginx.go | Extend Nginx module/build config response payload with build mode/status/compatibility/artifacts/dynamic support |
| agent/app/dto/request/nginx.go | Extend build and module update requests to include module selection, force rebuild, and module build metadata |
| agent/app/dto/nginx.go | Extend persisted module state schema with builds/targets/artifacts and associated metadata |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| die() { | ||
| log "ERROR: $*" | ||
| return 1 | ||
| } |
| <el-form-item v-if="module.buildMode !== 'static'" :label="$t('nginx.loadOrder')" prop="loadOrder"> | ||
| <el-input-number v-model="module.loadOrder" :min="0" :max="9999" /> | ||
| </el-form-item> |
| Params string `json:"params"` | ||
| BuildMode string `json:"buildMode" validate:"omitempty,oneof=auto dynamic static"` | ||
| Provider string `json:"provider" validate:"omitempty,oneof=local prebuilt"` | ||
| LoadOrder int `json:"loadOrder" validate:"omitempty,min=0,max=9999"` | ||
| } |
What this PR does / why we need it?
当前的模块添加/删除采用静态编译，每次升级或者修改模块都需要执行完整的Nginx编译操作，耗时通常长达30分钟，同时在低配置环境下经常出现编译失败情况。
Summary of your change
将原先的静态编译重构为自动识别，在模块支持的情况下默认采用动态编译方式进行引入，只编译对应模块，通过Nginx配置文件方式引入对应模块的加载。
Please indicate you've done the following:
关联 1Panel-dev/appstore#8644 ，关联 #10653
由于涉及到的内容较多，虽然已经进行本地测试通过，但是可能仍然需要团队进一步的测试以确保可靠性。
以下是完整细节：
OpenResty 动态模块支持（.so 本地构建 + load_module 热加载 + 升级自动重编译）
关联 PR：1Panel-dev/appstore（openresty 模板配套改动）。建议 appstore 先合并，面板侧功能依赖模板里的
Dockerfile.modules与模块目录。面向的问题
现在 1Panel 里给 OpenResty 加一个模块（比如 rtmp、web_dav），唯一的路是静态编译：把模块参数拼进
RESTY_CONFIG_OPTIONS_MORE，然后整个镜像从源码重新编译一遍，再重建容器。实测在 4 核 OVH VPS 机器上一次要 27~31 分钟，期间面板任务被占住，最后还要重启容器造成秒级闪断，性能和时间消耗过大。Nginx 从 1.9.11 起支持
load_module动态加载.so，OpenResty 编译时带--with-compat就能做到二进制兼容。只需要用和运行时镜像一致的基线把模块单独编成.so，挂载进容器，写一行load_module，reload 一下就生效了——不重新编译整个 OpenResty，也不重启容器。可以大幅度降低修改模块后的编译等待时间和性能占用。这个 PR 实现的内容：
.so，校验通过后热加载，容器不重启（实测构建前后容器 StartedAt 不变）实现方式
模块配置储存
appstore 侧每个 openresty 版本目录下新增两份文件：
build/module.catalog.json：模块目录，描述每个模块怎么构建——源码准备脚本（script）、编译期 apt 依赖（packages）、configure 参数（params）、加载顺序（loadOrder）。目前收录自带的 5 个：ngx_brotli、rtmp、web_dav、geoip2、http_substitutions_filterbuild/module.json：安装时落到应用目录的 per-install 配置，初始内容与 catalog 一致，之后用户在面板上的改动（启用/构建方式/参数）都写在这里面板后端所有模块操作都围绕应用目录下的
module.json进行，catalog 只作为初始模板。仓库根新增scripts/check-openresty-modules.sh校验两者逐字节一致（这两个文件现在是人工同步的，容易改一个忘一个），并配了一个独立的 GitHub workflow 在 openresty 相关变更时自动跑。动态构建链路
核心是
agent/app/service/nginx_module.go。一次动态构建的流程：docker image inspect拿，失败回退宿主机架构并记 WARNING）、镜像 Id、Dockerfile.modules的 SHA-256，算出 target key。镜像或者 builder 变了，key 就变，旧构建自然标记为 stale，触发重编docker build跑Dockerfile.modules，它就是FROM 1panel/openresty:<版本>（ABI 基线和运行时完全一致）加一层编译环境，configure 用--with-compat，模块参数里的--add-module=会被规范化成--add-dynamic-module=。产物从容器里拷出来，带 SHA-256 清单nginx -t，再按 loadOrder 把所有启用模块的load_module拼起来做一次联合nginx -t——防止模块之间符号冲突conf/modules-enabled/1panel-module-<loadOrder>-<模块名>.conf（写前快照，失败可还原），容器内nginx -t通过后nginx -s reload。全程容器不重启回滚语义：构建失败时旧的 ready 构建和旧配置原样保留，模块行只多一个失败标记和原因；managed conf 写入/校验失败时还原到写入前的快照。（见测试一节）。
签名兼容性潜在问题
最初
Dockerfile.modules的 configure 参数漏了--with-http_realip_module --with-http_dav_module，结果编出来的.so在运行时load_module全部被 nginx 拒绝（is not binary compatible）——nginx 动态模块校验 configure 签名，第 29/31 位对不上就拒载。已进行修正，三个版本的 builder 与运行时镜像签名一致，测试脚本里有针对这个错误的专项检查（not binary compatible出现即 FAIL）。另一个安全相关的点：模块 params 用 go-shellwords 解析，但这个库遇到未加引号的
;&|<>会静默截断而不是报错，原来写在解析后的逐参数检查根本摸不到这些字符。现在在解析前对原始串做元字符预检直接拒绝（parseDynamicModuleParams），对应测试TestNormalizeDynamicModuleParams。auto 模式与回退语义
构建方式里
auto的含义是"优先动态，动态走不通回退静态兜底"：auto会回退。用户显式选dynamic而构建失败时，只报错不切换static，lastError里保留动态失败原因（UI 上能直接看到，用户可以随时改回 auto）升级链路的踩的三个坑（实测发现，已修）
拿官方仓库真实版本做升级测试时撞到三个问题，已经进行修复：
buildDynamicNginxModules，而它第一步就要读build/Dockerfile.modules——老安装没这个文件，哪怕一个模块都没启用过，升级也直接炸。现在没有动态构建任务时整个 target 解析直接跳过build/tmp对不上（tarball 版本不匹配），构建报错把升级拖死。现在降级为 WARNING：升级继续，动态模块保持下线，面板里能看到原因。显式在面板点"构建"仍然是明确报错，不会静默handleUpgradeCompose只从旧 compose 保留deploy/restart，官方新 compose 没有./modules、./conf/modules-enabled两个挂载，升级后容器里根本看不到.so——而且因为 include 是 glob，nginx -t照样通过，属于毫无征兆的静默失效。现在升级时会把旧 compose 里这两条挂载合并到新 compose第 2 点有个实测时发现的细节值得记录：只要安装目录里的
Dockerfile.modules还在，且其RESTY_VERSION与新版本上游 OpenResty 版本一致（比如 1.31.1.1-0 → 1.31.1.1-2-1，上游都是 1.31.1.1），升级时动态模块其实能正常按新镜像重编并加载——已经在真实官方包上验证通过。前端
网站 → OpenResty 设置 → 模块页改为三列：构建方式 / 构建状态（失败带原因 tooltip）/ 兼容性（compatible / stale / static）。构建抽屉支持勾选多个模块一起构建、选 apt 镜像源。编辑抽屉里 auto 的文案写的是"自动（动态构建）"，避免误解成"自动检测"。
测试
环境：AlmaLinux 10（4C8G），Docker CE 29.6.2，面板官方 v2.2.3 安装后替换为 PR 编译的二进制，OpenResty 用应用商店正常安装。
单元测试：
agent下go test ./app/service -run 'Module'12 个全过，go vet零错误。覆盖参数规范化、元字符预检、构建记录查找、失败回滚保留旧构建、target 缺 builder 哨兵、compose 挂载合并、回退判定与翻转。构建器集成测试（
scripts/openresty-modules/test-builder.sh，不依赖面板，直接从 appstore 模板构建）：三个版本（1.27.1.2-5-1-focal / 1.29.2.5-0-noble / 1.31.1.1-0-noble）× 5 个模块全部 PASS，含单模块加载、loadOrder 联合加载、起容器热 reload、负例回滚，每个.so有 SHA-256 记录，全程无not binary compatible。面板端到端：
modules/下有.so+ manifest，modules-enabled/生成 load_module 配置，容器内nginx -t通过，构建前后容器 StartedAt 不变modules-enabled/文件名按 loadOrder 数字前缀排序exit 1强制重新构建，任务失败，旧.so与旧配置完整保留，nginx -t照常通过nginx -t正常升级端到端（共四轮）：
nginx -T可见 4 条load_module），compose 挂载保留--with-debug的模块（动态必然拒绝，静态可编），构建 → 日志 WARNING 指名模块 → 落盘 static → 27 分钟编译期旧容器持续在线 → 成功后切换，nginx -V含--with-debug；升级路径同场景同样验证通过诊断：
scripts/openresty-modules/diagnose-install.sh在回滚后和升级后各采集一次，8 项检查全过，failed_checks=0。它会核对产物校验和、managed conf、compose 配置、只读挂载、镜像一致性、容器状态与nginx -t。复测方式：装一台 Linux + Docker，按
scripts/openresty-modules/README.md跑./test-builder.sh --appstore <appstore目录>即可复现构建器测试矩阵；面板侧按 HANDOVER 文档的阶段 5-7 操作（面板 API 与 UI 操作等价）。需要评审注意的点
resolveNginxModuleTarget的 target key 组成：版本 + 架构 + 镜像 Id + builder digest，任何一项变化都会触发全量重编，这是有意为之（宁可多编不可错载）reconcileDynamicNginxModuleConfig的快照/还原时序：先校验再写 conf、写失败还原、nginx -t失败也还原app_utils.go）里降级与回退的状态迁移：缺 builder → 降级继续；动态失败 → auto 翻转 static 交给后续 buildNginxRESTY_CONFIG_OPTIONS_MORE进全量编译libbrotli-dev（focal 镜像无头文件）尚未实现的部分
1.21.4.3-3-3-focal这个老版本没加模块文件。升级到/从它升级会走降级路径（升级本身不受影响，但动态模块会下线）。回填模板或在文档里直接声明buildModeAuto等新增 key（回退显示英文），需要进行翻译补充回滚
面板二进制与
/opt/1panel/db有备份可直接还原；OpenResty 侧用面板"重建"即回官方模板。