lazy-loading config with GabbitFixture not working in gabbit>=4.0

[tobias-urdin]

With the move to use httpx and running the WSGI application in a external process called with subprocess it's no longer possible to lazy-load the config with a global variable and a config fixture that updates the global variable.

In Gnocchi the setup_app() [1] used for intercept is no longer lazy loaded but called directly when httpx.WSGITransport is initialized so the ConfigFixture that updates the global variable needed for the application is None, it's unclear if it's really because the WSGI server is run in an external process now or because setup_app() is called directly before any fixtures has been started.

Do you have any solution or ideas on how we can solve this so that we can unpin and use gabbi>=4.0 instead of using the older versions not using httpx?

I'm thinking this might also impact OpenStack Placement and potentially other OpenStack projects using the same solution with a config fixture but a newer version of gabbi might not be used there yet.

[1] https://github.com/gnocchixyz/gnocchi/blob/master/gnocchi/tests/functional/fixtures.py#L56

[cdent]

Thanks for reporting this. I used to have the placement functional tests running in gabbi's CI, and that probably would have caught this but at some point they got more complicated to run than they had been in the past and I decided gabbi didn't need to be responsible for that anymore. Oh well.

I'm a bit confused though, because as far as I can tell an external process never gets involved when using WSGITransport. It works much the same way as wsgi-intercept does: it uses a local stream instead of a socket. So that may be a read herring.

More likely is that the binding of the wsgi app, and its configuration, are somehow happening differently. I'll need to do a bit of digging to see what's going on there.

[cdent]

@tobias-urdin Have you got a traceback from one of the gnocchi test failures? My local box is missing too much of the required stuff to quickly run the functional tests so if there's one handy to look at that would be useful. If not I'll need to take some time (when I'm not working) to set up a working test env.

[tobias-urdin]

@tobias-urdin Have you got a traceback from one of the gnocchi test failures? My local box is missing too much of the required stuff to quickly run the functional tests so if there's one handy to look at that would be useful. If not I'll need to take some time (when I'm not working) to set up a working test env.

You can look at this run that gives a traceback: https://github.com/gnocchixyz/gnocchi/actions/runs/15904107359/job/44854232022

[cdent]

This probably won't fix it, but just to be sure can you try return a callable from setup_app, instead of calling load_app, something like:

diff --git a/gnocchi/tests/functional/fixtures.py b/gnocchi/tests/functional/fixtures.py
index 457175e2..be55f97c 100644
--- a/gnocchi/tests/functional/fixtures.py
+++ b/gnocchi/tests/functional/fixtures.py
@@ -52,8 +52,10 @@ LOAD_APP_KWARGS = None


 def setup_app():
-    global LOAD_APP_KWARGS
-    return app.load_app(**LOAD_APP_KWARGS)
+    def x():
+        global LOAD_APP_KWARGS
+        return app.load_app(**LOAD_APP_KWARGS)
+    return x

The core issue is that load_app is getting called long before LOAD_APP_KWARGS is getting defined. Doing my suggestion above probably won't change this, but we'll see. More than likely it will tell you something about incorrect args.

I'm struggling to think of a good way to fix this. Things we know:

• the gnocchi fixture is trying to do is load in a new configuration for each test file
• the failure that is happening is that long before we encounter the fixture, the wsgi app is failing to load because it has no configuration

So maybe what needs to happen here is for initial version of LOAD_APP_KWARGS to be a valid (but meaningless configuration) rather than None?

That will at least get you past the currently presenting error, and reveal whatever the next one is (if any).

[tobias-urdin]

Interesting, testing that locally actually starts the test suite. Let me ask if we can add that to gnocchixyz/gnocchi#1458 directly and see if it passes, thanks!

[cdent]

testing that

Which that? The diff, or the valid LOAD_APP_KWARGS

[tobias-urdin]

testing that

Which that? The diff, or the valid LOAD_APP_KWARGS

Sorry, the patch diff that you provided made the test suite run without that error, but we need other fixes since the Gnocchi CI is also broken to know for sure, we will add it to gnocchixyz/gnocchi#1458 and see. Thanks!

[cdent]

Please follow up here if there's anything further to do on the gabbi side. Thanks again.

[Callum027]

Hi there, in my testing in gnocchixyz/gnocchi#1458 the suggested patch appears to cause the following test failures:

• https://github.com/gnocchixyz/gnocchi/actions/runs/15936973181/job/44958768930?pr=1458
• https://github.com/gnocchixyz/gnocchi/actions/runs/15936973181/job/44958768983?pr=1458

gabbi.suitemaker.test_gabbi_resource-aggregation_aggregate_metric_with_groupby_on_project_id_with_filter.test_request
---------------------------------------------------------------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):

      File "/github/workspace/.tox/py39-postgresql-file-sqlalchemy14/lib/python3.9/site-packages/gabbi/suitemaker.py", line 108, in do_test
    return test_method(*args, **kwargs)

      File "/github/workspace/.tox/py39-postgresql-file-sqlalchemy14/lib/python3.9/site-packages/gabbi/case.py", line 98, in wrapper
    func(self)

      File "/github/workspace/.tox/py39-postgresql-file-sqlalchemy14/lib/python3.9/site-packages/gabbi/case.py", line 165, in test_request
    self._run_test()

      File "/github/workspace/.tox/py39-postgresql-file-sqlalchemy14/lib/python3.9/site-packages/gabbi/case.py", line 619, in _run_test
    self._run_request(

      File "/github/workspace/.tox/py39-postgresql-file-sqlalchemy14/lib/python3.9/site-packages/gabbi/case.py", line 511, in _run_request
    response, content = self.http.request(

      File "/github/workspace/.tox/py39-postgresql-file-sqlalchemy14/lib/python3.9/site-packages/gabbi/httpclient.py", line 49, in request
    response = self.client.request(

      File "/github/workspace/.tox/py39-postgresql-file-sqlalchemy14/lib/python3.9/site-packages/httpx/_client.py", line 825, in request
    return self.send(request, auth=auth, follow_redirects=follow_redirects)

      File "/github/workspace/.tox/py39-postgresql-file-sqlalchemy14/lib/python3.9/site-packages/httpx/_client.py", line 914, in send
    response = self._send_handling_auth(

      File "/github/workspace/.tox/py39-postgresql-file-sqlalchemy14/lib/python3.9/site-packages/httpx/_client.py", line 942, in _send_handling_auth
    response = self._send_handling_redirects(

      File "/github/workspace/.tox/py39-postgresql-file-sqlalchemy14/lib/python3.9/site-packages/httpx/_client.py", line 979, in _send_handling_redirects
    response = self._send_single_request(request)

      File "/github/workspace/.tox/py39-postgresql-file-sqlalchemy14/lib/python3.9/site-packages/httpx/_client.py", line 1014, in _send_single_request
    response = transport.handle_request(request)

      File "/github/workspace/.tox/py39-postgresql-file-sqlalchemy14/lib/python3.9/site-packages/httpx/_transports/wsgi.py", line 134, in handle_request
    result = self.app(environ, start_response)

    TypeError: x() takes 0 positional arguments but 2 were given

[cdent]

Thanks for trying @Callum027

TypeError: x() takes 0 positional arguments but 2 were given

Yeah, this is what I thought would probably happen. That's why I said above "This probably won't fix it". It was an experiment to understand how things are fitting together.

So perhaps try the next option I mentioned above:

So maybe what needs to happen here is for initial version of LOAD_APP_KWARGS to be a valid (but meaningless configuration) rather than None?

That will get past the initial error and expose whether the config can change as the fixture is called.

If that doesn't work the other options I can think of are:

1. Change the gnocchi functional tests to do one of:
   1. not need/want the ConfigFixture (this is presumably a non-starter but listing it for completeness) or somehow change the details of the ConfigFixture so that it doesn't rely on setup_app to adjust the config (move the config changing deeper into the app).
   2. Move the config adjustjustments outside of the fixture, into the test harness. This essentially means having a load_tests call for each yaml file, each with a different intercept value, where each intercept is a differently configured wsgi app.
   3. Choose to not do intercepting at all: Run the wsgi-app as a subprocess. When gabbi was made the idea was to avoid subprocess for the sake of speed so wsgi-intercept was used, and within Placement no additional processes were necessary so it was possible to have many tests without external processes. This is not the case with gnocchi: pifpaf is spinning up plenty of other stuff as far as I can tell.
2. Change gabbi so it created the intercept as part of the fixture stack, rather than the suite maker stack. In effect this would be a bit like making it run time instead of compile time. This is basically how it worked when wsgi-intercept was being used, but that only worked because wsgi-intercept is a monkey patch on in-memory classes, clobbering the http libraries. The "transport" style that httpx uses isn't like that, so achieving the same positioning would mean completely changing how the tests get their http client member. Likely a huge and risky change. It might be worth trying, but not for the faint of heart.

[kajinamit]

I wonder if something like #344 could be a potential fix. In theory it should delay build of wsgi app, which currently triggers config load.

It seems we need a few more additional adjustment to adopt the tests we maintain in gnocchi, though, which we still have to look into...