[3.0][Testing] Run the integration tests in CI, with or without Docker by albertlast · Pull Request #9659 · SimpleMachines/SMF

albertlast · 2026-09-07T06:00:10Z

Description

The suites from #9345 and #9347 have never run in CI. The PHPUnit job is a runner
with no database and no Settings.php, so IntegrationTestCase skips all of them:

PR	job	result
#9345	Unit tests (ubuntu, 8.4)	327 tests, 20 skipped — exactly the integration suite
#9347	Unit tests (ubuntu, 8.4)	350 tests, 43 skipped — exactly the integration suite

A green check on either proves the unit suite passed and nothing else.

Underneath that is a second problem. Everything that can produce a forum
(install-forum.sh, reset.sh, test.sh) drove docker compose exec, so the
integration suite was unreachable for anyone not running the stack. The unit job
matrixes windows-latest deliberately, because people develop SMF on Windows;
the integration suite excluded exactly those people, and CI could not run it
either.

So this does both: the tooling learns to work without Docker, and CI uses that
same path.

Local is the default

SMF_RUNNER picks where the work happens and defaults to local — the PHP on
this machine and a database it can already reach. --docker, or
SMF_RUNNER=docker in .env, asks for the stack, and nothing about that path
changes.

.dev/install-forum.sh --engine mysql            # this machine
.dev/install-forum.sh --docker --engine mysql   # the stack

Local is the default because it is the one that needs nothing installed beyond
what SMF itself requires, and the only one available on a machine without Docker.
This does change what an existing command means, so three things guard it:
require_local_deps names --docker in its failure message, the README leads
with it, and the failure when there is no local database is loud rather than
silent.

Why `.dev/` and not `tools/`

The tooling moves out of .docker/, since neither runner is the second-class one
now. The leading dot is deliberate rather than cosmetic:
check-smf-index.php — which php.yml runs on every PR — requires an exact
index.php stub in every directory except the dot ones, ./other and
./vendor. The repository root is the forum's webroot, so a plain tools/ would
be served by a real install and would need a stub in every subdirectory. It is
the same reason .docker/ and .github/ are spelled the way they are.

.docker/ keeps what builds and configures containers: php/, mysql/,
postgres/, env.example. compose.yaml is untouched.

`.dev/db.php`

The two runners reach a database differently. Under Docker the mysql and psql
clients are already inside the container; locally they would be a dependency
nothing else here needs, while mysqli and pgsql are ones SMF has anyway. So
the local runner goes through PHP, and any machine that can serve the forum can
run these scripts with no database client installed.

The web server

Locally the forum is served by PHP's own built-in server. That is enough for
these tests: SMF routes on the query string and on PATH_INFO
(Sources/QueryString.php:153), there is no .htaccess at the root, and
Sapi::isSoftware() returns false for an unrecognised SERVER_SOFTWARE rather
than misbehaving. It is not Apache, though, so the compose stack stays the closer
thing to production and the README says so.

`--fail-on-skipped`, and why `IntegrationTestCase` changes

The skip is a convenience locally and a lie in CI: a job wired to the wrong
database would report a green run having tested nothing. PHPUnit has
--fail-on-skipped for exactly this.

It does not reach a skip raised in setUpBeforeClass(). That marks the class
as skipped rather than its tests, and only skipped tests count towards the exit
code — so with the check where #9345 put it, the flag was inert in the one case
that matters most. Measured, on a tree with no Settings.php:

skip inside a test method:   exit=1
skip in setUp():             exit=1
skip in setUpBeforeClass():  exit=0     <-- where the check was

So the check moves to setUp(). Installation memoises its answer, so asking
once per test costs a function call. tearDown() gains an isset(Db::$db) guard
to go with it: PHPUnit runs it even for a test setUp() skipped, and without it
a run that should read as "43 skipped, no forum" reads as 18 errors instead.

The workflow

A second job installs a forum on the runner and runs the suite, on MySQL and on
PostgreSQL. Both, for the reason #9345 gives: the counter regression in
ModSettingsTest passes on MySQL with the bug still in place and fails only on
PostgreSQL, so one engine would have proved nothing. It uses service containers
and the same .dev scripts a person would, rather than a second install path
that could drift from the one people actually run.

It is ubuntu-latest only — windows-latest runners cannot run Linux
containers. Windows coverage stays with the unit job, and the local runner is
what lets a Windows developer run these by hand.

The unit job narrows to composer test-unit, so it reports what it ran instead
of counting 20-43 skips as part of a passing run.

Verified

Every command was run. Two of the three defects below are ones the verification
found in this branch's own code, which seemed worth saying rather than quietly
fixing:

db_empty was a silent no-op on MySQL. GROUP_CONCAT truncates at
group_concat_max_len, 1024 bytes by default, which SMF's 72 tables pass
comfortably. The reset dropped nothing and reported success. The names are now
collected first and the DROP built from them.
db.php swallowed errors after the first statement. Folding
next_result() into a loop condition means a batch that failed halfway exits
0 — which is how the bug above stayed hidden.
--fail-on-skipped was inert, as above.

	result
Local runner, MySQL	43 tests, 0 skipped, green
Local runner, PostgreSQL	43 tests, 0 skipped, green
Docker runner, both engines, whole suite	182 tests, green
No `Settings.php`, with `--fail-on-skipped`	43 skipped, exit 1
No `Settings.php`, without it	43 skipped, exit 0 — still usable
Wrong `SMF_ADMIN_PASS`, with `--fail-on-skipped`	4 skipped, exit 1
`composer test-unit`	139 tests, 0 skipped
`composer test`	182 tests, green

Also: a full local-mode install from scratch on both engines (reset.sh →
db_empty → stage_installer → both installer passes), the same through
--docker, user.sh check and use-engine.sh both ways, and .dev/ci.sh
green — including check-signed-off.php, check-smf-license.php,
check-smf-languages.php, check-smf-index.php and check-version.php, which
is what confirms the .dev/ naming above. shellcheck is clean on all seven
scripts, at the same level the originals were.

And the workflow itself, on this pull request:

Integration tests (mysql)        pass  1m39s   OK (43 tests, 180 assertions)
Integration tests (postgresql)   pass    57s   OK (43 tests, 179 assertions)

0 skipped on both, which is the number that matters — the whole point is that
a green run here can no longer mean an empty one. The logs show each job
installing a forum, serving it and running the suite against it.

Not verified on Windows yet

The local runner exists so that a developer on Windows can run this suite, and it
has been exercised on Linux only. Nothing in it is platform-specific by design —
the database goes through mysqli/pgsql rather than a CLI client precisely so
that Windows needs nothing extra, and serve_start probes through PHP rather
than curl for the same reason — but that is an argument, not a test. Happy to
hold this until someone has run .dev/test.sh --engine mysql on Windows.

Merge order

Merge #9345 and #9347 before this one, and the PRs they name before that.
This branch contains the whole chain, so the diff shown here is mostly theirs;
once they land and this is rebased on release-3.0, what is left is .dev/,
.github/workflows/phpunit.yml and the IntegrationTestCase change.

The upgrade tooling that landed while this was being written
(compare-upgrade.sh, interrupt-upgrade.sh, rerun-upgrade.sh,
upgrade-readings.sh, schema-tool.php) moves to .dev/ with the rest, in its
own commit: it sources lib.sh and uses $DEV_DIR, so leaving it behind would
break it. Those five pin SMF_RUNNER=docker for themselves and pass --docker
on to the scripts they invoke, since orchestrating containers is all they can
mean and a shell variable does not cross into a child process.

One thing still to do: #9330 and #9657 also touch .docker/, so both want a
rebase on top of this. Only .docker/README.md genuinely collides — #9330's
.docker/baseline/ subtree has its own self-contained lib.sh and does not
source the top-level one.

Issues References (Fixes|Related|Closes)

Depends on [3.0][Testing] Add HTTP smoke tests that drive a running forum #9347 — runs the HTTP tests it adds, and changes IntegrationTestCase.
Depends on [3.0][Testing] Add an integration test suite that runs against a real forum #9345 — the integration suite and its skip behaviour.
Related to [3.0][Testing] Add a Docker development environment for MySQL and PostgreSQL #9317, [3.0][Testing] Add a PHPUnit suite for the parts that need no database #9326, [3.0][Testing] Install the forum from the command line #9344, [3.0][Testing] Add an integration test suite that runs against a real forum #9345.

🤖 Generated with Claude Code

The integration suite reaches the database, but not a page. Everything between a request arriving and HTML coming back - the session, the cookies, the theme, the templates, the permission checks - had no automated coverage at all, and that is where the failures people actually report live. Requests have to be real ones: obExit(), redirectexit() and fatal*() all end in exit, and Db::$db, ActionTrait::$obj and Theme::$loaded cannot be reset, so a test process can carry out one request in itself and no more. tests/Support/HttpClient.php is a small browser built on the curl extension the forum already requires, so this costs no new dependency. Three files to start: a sweep of the pages a guest can reach, the login journey, and starting a topic and replying to it. Every one of them ends in assertNoErrorsLogged(), which is the point - SMF records most of what goes wrong in log_errors rather than showing it, so a page can return a flawless 200 while logging an undefined index on every hit. Four things about SMF made these harder to write than expected, and each is commented where it bites rather than worked around silently: - The first request of a new session regenerates it, so a security token minted on the very first page a visitor sees can never be validated. It looks like a broken token, not a replaced session. - Only the button that was clicked gets submitted. The posting form offers "preview" and "post"; sending both means preview wins and the post is never made, with an ordinary 200 to show for it. - Security::spamProtection() allows one login or post every two seconds per IP, and tests are much faster than people, so submitForm() waits it out once instead of failing at random. - curl only writes cookies with an expiry to its jar file, so a handle opened per request loses the session every time. HTTP tests cannot be wrapped in a transaction - the request runs in the web server's process on its own connection, and on MySQL's REPEATABLE READ an open transaction here would never see what it wrote, quietly making assertNoErrorsLogged() incapable of failing. IntegrationTestCase gains usesTransaction() so they can opt out, and PostingTest removes what it creates through Topic::remove(). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: albertlast <mathiaspapealbert@hotmail.com>

The suite ships the machinery but not the instructions for using it, and three things are not discoverable from reading it: which of the three suites a new test belongs in, who the request is actually made as, and where the endpoint and field names come from. The second is the one that misleads. HttpTestCase inherits actingAs() from IntegrationTestCase, where it repoints User::$me in the PHPUnit process - but the request is handled by Apache in another process, which knows only the cookie. Calling it in an HTTP test changes nothing and leaves the assertions describing a guest, confidently. The identity of a request here is the cookie jar and nothing else. Field names are the opposite problem: they look like something to look up, and are not. submit() scrapes the form the way a browser does, which is what carries the session check and the security token - both named differently for every session, so a hand built POST body gets a 403 it cannot fix. The worked example prints them to make that concrete. Also notes that install.php left in the board root puts an errorbox on every page an administrator sees, which fails assertLooksLikeAForumPage() and crowds out whatever the test was looking at. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: albertlast <mathiaspapealbert@hotmail.com>

The tests skip rather than fail when they cannot sign in, which says what is wrong but not what to do about it. user.sh answers both halves: check says whether the password the suite is using is the right one, and reset puts a forum installed some other way back on the credentials it expects. Signed-off-by: albertlast <mathiaspapealbert@hotmail.com>

The SMF/section_comments fixer puts the Public methods banner at the top of the group, and the DataProvider attribute belongs to the method under it, not above the banner. That one file was all the style check was failing on. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: albertlast <mathiaspapealbert@hotmail.com>

The scripts that build a forum and run the integration suite against it all drove `docker compose exec`, so the suite was out of reach for anyone not running the stack. The unit job matrixes windows-latest deliberately, because people develop SMF on Windows; the integration suite excluded exactly those people, and CI could not run it either. SMF_RUNNER now picks where the work happens, and defaults to local: the php on this machine and a database it can already reach. --docker, or SMF_RUNNER=docker in .env, asks for the stack instead, and nothing about that path changes. The tooling moves to .dev/, since neither runner is the second-class one now. The name keeps its leading dot on purpose: check-smf-index.php wants an index.php stub in every directory but the dot ones, other/ and vendor/, and the repository root is the forum's webroot, so a plain tools/ would be served by a real install. .dev/db.php is how the local runner reaches a database. Under Docker the mysql and psql clients are already in the container; locally they would be a dependency nothing else here needs, while mysqli and pgsql are ones SMF has anyway. So any machine that can serve the forum can run these scripts. Locally the forum is served by PHP's own server, which is enough for the tests: SMF routes on the query string and on PATH_INFO, and there is no .htaccess at the root, so nothing here wants mod_rewrite. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: albertlast <mathiaspapealbert@hotmail.com>

The suite skips itself when there is no forum to talk to, so that `composer test` stays useful on a machine with nothing installed. In CI that same kindness is a lie: a job wired to the wrong database reports a green run having tested nothing. PHPUnit has --fail-on-skipped for exactly this, but it does not reach a skip raised in setUpBeforeClass(): that marks the class as skipped rather than its tests, and only skipped tests count towards the exit code. The check moves to setUp(), where it does count. Installation memoises its answer, so asking once per test costs a function call. tearDown() now checks that there is a connection before rolling back. PHPUnit runs it even for a test setUp() skipped, and without the guard a run that should read as "43 skipped, no forum" reads as 18 errors instead. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: albertlast <mathiaspapealbert@hotmail.com>

The 43 tests in tests/Integration/ have never run in CI. The PHPUnit job is a runner with no database and no Settings.php, so every one of them skipped: 20 of them on the pull request that added the suite, 43 on the one that added the HTTP tests. A green check there proved the unit suite passed and nothing else. A second job installs a forum on the runner and runs them, on MySQL and on PostgreSQL. Both, because the two disagree: the counter regression in ModSettingsTest passes on MySQL with the bug still in place and fails only on PostgreSQL, so one engine would have proved nothing. It uses service containers and the same .dev scripts a person would, rather than a second install path that could drift from the one people actually run. --fail-on-skipped is the point of the job. Both the missing-forum skip and the wrong-password one exist so the suite stays usable on a machine with no forum; in CI they would turn a job that tested nothing into a green tick. The unit job narrows to the unit suite, so it reports what it ran instead of counting those skips as part of a passing run. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: albertlast <mathiaspapealbert@hotmail.com>

It sources lib.sh and refers to the directory the scripts live in, both of which the move took with it, so leaving it behind breaks it. Unlike the rest, these four scripts and schema-tool.php cannot mean anything but the compose stack: they drive upgrade.php inside the web container, kill it part way through, and read databases the stack owns. So they set SMF_RUNNER=docker for themselves and call require_docker, and nobody running them has to remember a flag. They pass --docker on to the reset.sh and install-forum.sh they invoke, since a shell variable does not cross into a child process. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: albertlast <mathiaspapealbert@hotmail.com>

albertlast and others added 4 commits September 6, 2026 10:54

github-actions Bot added github_actions Pull requests that update GitHub Actions code Unit Testing labels Sep 7, 2026

albertlast and others added 4 commits September 7, 2026 08:06

albertlast force-pushed the tests/ci-integration branch from 5b7bc9a to 948a2a2 Compare September 7, 2026 06:29

Navigation Menu

Uh oh!

[3.0][Testing] Run the integration tests in CI, with or without Docker - #9659

[3.0][Testing] Run the integration tests in CI, with or without Docker#9659
albertlast wants to merge 8 commits into
SimpleMachines:release-3.0from
albertlast:tests/ci-integration

albertlast commented Sep 7, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

albertlast commented Sep 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Local is the default

Why .dev/ and not tools/

.dev/db.php

The web server

--fail-on-skipped, and why IntegrationTestCase changes

The workflow

Verified

Not verified on Windows yet

Merge order

Issues References (Fixes|Related|Closes)

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

albertlast commented Sep 7, 2026 •

edited

Loading

Why `.dev/` and not `tools/`

`.dev/db.php`

`--fail-on-skipped`, and why `IntegrationTestCase` changes