refactor: replace libsodium with cryptography by Hoeze · Pull Request #59 · EGA-archive/crypt4gh

Hoeze · 2026-09-15T22:50:16Z

tl;dr crypt4gh.sodium becomes a small Python module on top of cryptography, with the same functions and byte-identical results. crypt4gh is then pure Python: one wheel for all platforms, no C compiler, no libsodium submodule. Speed stays about the same. OpenSSH keys are now parsed by cryptography too, which replaces about 150 lines of our own parser.

Why

cryptography is already a dependency. crypt4gh/keys/c4gh.py uses its ChaCha20Poly1305 for key files.
Since cryptography 47, ChaCha20Poly1305.encrypt_into and decrypt_into write into the caller's buffer. Avoiding PyNaCl's allocations was the reason for the own bindings in 1.8.
Without a C extension there are no per-platform wheels, no compiler step for sdist installs, and no -march=native or linker flags to get right.
Speed stays about the same. For a 512 MiB file, crypt4gh encrypt ran at 260 MB/s and decrypt at 383 MB/s, against 259 and 339 MB/s with part 1's libsodium fix.

What changes

crypt4gh/sodium.py replaces crypt4gh/sodium.c. It has the same seven functions, the same exception types and the same results:
- ChaCha20-Poly1305 uses encrypt_into/decrypt_into. A bad tag still raises ValueError.
- The key exchange is X25519 plus BLAKE2b-512, as in libsodium's crypto_kx.
- Ed25519 to X25519, for OpenSSH keys: secret keys go through SHA-512 and clamping. Public keys use the map u = (1+y)/(1-y) in plain Python, with libsodium's checks: on the curve, not of small order, and in the prime-order subgroup. That costs about 3 ms per key.
The module keeps the name crypt4gh.sodium, so the callers and import crypt4gh.sodium keep working.
The dependency becomes cryptography>=47.
The release workflow builds the sdist and one py3-none-any wheel with python -m build, instead of the cibuildwheel matrix. The test workflow drops the system/bundled matrix.
The libsodium submodule, build_libsodium.py and the libsodium install docs are removed.
crypt4gh/keys/ssh.py loads OpenSSH keys with load_ssh_private_key and load_ssh_public_key, instead of its own cipher table, bcrypt call and blob parser. The passphrase is still only requested for encrypted keys, and errors still end in "Invalid Key or Passphrase". The debug log no longer prints private key bytes.
Keys encrypted with aes256-ctr (OpenSSH's default) or aes256-cbc work as before, and aes256-gcm@openssh.com keys now work too. Keys encrypted with aes128 or aes192 (CTR or CBC) or 3des-cbc no longer load; ssh-keygen -p -Z aes256-ctr -f <key> re-encrypts them. cryptography already warns that its TripleDES is going away.

The first commit is the switch. The second removes what is left of libsodium. The third moves the OpenSSH key parsing to cryptography.

Part 1 added a test that decrypts files written by 1.8.6, so CI checks that this PR still reads existing files.

Stack

This PR is part 3 of 3. A PR from a fork can only target master, so this PR also contains parts 1 and 2. Please review only the last commits. Once parts 1 and 2 are merged with merge commits, the diff shrinks to those three.

Stacked on #58.

Since 1.8 replaced PyNaCl, segment encryption and decryption hold the GIL, so threads run one at a time. Release it around the libsodium calls. The Py_buffer views keep the buffers alive and unresizable meanwhile. Also reject keys that are not 32 bytes, and return NULL when the key exchange fails or kx_client gets bad arguments. Before, these paths returned a value with an exception set, which Python reported as SystemError.

The testsuite only ran on pushes, so pull requests from forks got no CI. Run it on pull requests and on demand too, with a read-only token and without persisting credentials in the checkout. A newer push cancels the older runs of the same branch or pull request.

The testsuite encrypts and decrypts with the same code. A bug that both directions share, like a wrong hash in the key exchange, passes every test but breaks all existing files. Decrypt two files that the released 1.8.6 wrote, one sent with a crypt4gh key and one with an ssh key. Checking the sender also covers the conversion of ssh public keys.

--enable-opt adds -march=native -mtune=native. The release wheels are built on GitHub runners, so their generic code paths (BLAKE2b, SHA-512, the curve25519 field arithmetic) require the runner's CPU features, e.g. BMI2 and AVX2 in the 1.8.6 manylinux x86_64 wheel. libsodium still selects its SIMD implementations at runtime.

The extension declared libsodium only for the bundled build. System builds relied on -lsodium in LDFLAGS, which setuptools puts before the object files. Linkers with --as-needed (the Ubuntu default, and part of conda's LDFLAGS) then drop libsodium, and the import fails with undefined symbols. The -Wl,--no-as-needed workaround in the docs and in CI is no longer needed.

configure lists test/Makefile and test/default/Makefile in AC_CONFIG_FILES, so building a wheel from the sdist failed with "cannot find input file: 'test/default/Makefile.in'".

Metadata moves to [project], with an SPDX license expression. The extension is declared in [tool.setuptools.ext-modules]. The bundled libsodium build moves to build_libsodium.py, which setuptools loads through [tool.setuptools.cmdclass]. setuptools becomes a build requirement instead of an entry in requirements.txt, and the release workflow builds the sdist with python -m build. The clean command goes away with setup.py.

pip install . builds from the checkout, so a broken sdist went unnoticed. Build the sdist and install that, which is what pip does wherever no wheel matches.

The C extension and the bundled libsodium give way to a Python module with the same name and functions, and byte-identical results: - ChaCha20-Poly1305 uses cryptography's encrypt_into/decrypt_into, so it still writes into the caller's buffers. They need cryptography 47. - The key exchange is X25519 plus BLAKE2b-512, as in libsodium's crypto_kx. - The Ed25519 to X25519 conversion for OpenSSH keys is plain Python, with the same checks as libsodium. crypt4gh is now a pure Python package, so the release builds one wheel with python -m build instead of a cibuildwheel matrix.

Nothing builds against it anymore. Drop the submodule, its sdist rules, the system/bundled test matrix and the libsodium install instructions.

cryptography's load_ssh_private_key and load_ssh_public_key replace the hand-written OpenSSH parser: the cipher table, the bcrypt KDF call, the blob decoding and the parsing of public key lines. Only the conversion to X25519 keys stays. The passphrase is still only requested for encrypted keys, errors still end in "Invalid Key or Passphrase", and the debug log no longer prints private key bytes. Keys encrypted with aes256-ctr (OpenSSH's default) or aes256-cbc work as before, and aes256-gcm@openssh.com keys now work too. Keys encrypted with aes128 or aes192 (CTR or CBC) or 3des-cbc no longer load; `ssh-keygen -p -Z aes256-ctr -f <key>` re-encrypts them.

Cito · 2026-09-16T09:32:27Z

+# See https://packaging.python.org/discussions/install-requires-vs-requirements/
+dependencies = [ # include version when needed
+    "docopt-ng",
+    "cryptography>=47", # for ChaCha20Poly1305.encrypt_into and decrypt_into


shouldn't that be >=50?

I tried python 3.9 with cryptography==47 and that works 🤷
Is there any reason to raise it to v50?

pyproject.toml needs cryptography 47 for encrypt_into and decrypt_into, while requirements.txt still asked for any version.

ssh-keygen only produces valid keys, so no test reached the checks in the conversion to curve25519. The three crafted keys are the identity point, a y that is not on the curve, and the base point plus a point of order 2.

requires-python has been >=3.9 for a while, the README still said 3.6. An encrypted OpenSSH key loads if it uses aes256-ctr, aes256-cbc or aes256-gcm@openssh.com.

autodoc imports crypt4gh, so without the package it documents nothing and the API page lists no functions. Sphinx exits 0 either way, which is why this went unnoticed. The install needs no compiler now.

The module holds no libsodium bindings any more, so the name no longer fits. crypt4gh.sodium stays as an alias that warns on import, and version 2 can drop it.

ssh.parse_private_key is public API. In crypt4gh 1.8 it takes the stream after the magic word, and get_private_key passes it that way. The switch to cryptography changed it to take the whole PEM file. That broke callers that parse keys in memory, and get_private_key read the key file twice. It takes the stream again and wraps the key back into PEM for cryptography.

Hoeze added 11 commits September 15, 2026 15:15

fix(build): keep the libsodium test Makefiles in the sdist

570f67e

configure lists test/Makefile and test/default/Makefile in AC_CONFIG_FILES, so building a wheel from the sdist failed with "cannot find input file: 'test/default/Makefile.in'".

ci: test the package installed from the sdist

fde2c26

pip install . builds from the checkout, so a broken sdist went unnoticed. Build the sdist and install that, which is what pip does wherever no wheel matches.

build: remove the bundled libsodium

05c1a48

Nothing builds against it anymore. Drop the submodule, its sdist rules, the system/bundled test matrix and the libsodium install instructions.

This was referenced Sep 15, 2026

build: portable wheels, buildable sdist, pyproject.toml instead of setup.py #58

Open

fix(sodium): release the GIL during encryption, validate key sizes #57

Open

Hoeze marked this pull request as ready for review September 15, 2026 23:21

Cito reviewed Sep 16, 2026

View reviewed changes

Comment thread crypt4gh/sodium.py

Cito reviewed Sep 16, 2026

View reviewed changes

Hoeze added 6 commits September 17, 2026 11:52

build: require cryptography>=47 in requirements.txt

a9a854b

pyproject.toml needs cryptography 47 for encrypt_into and decrypt_into, while requirements.txt still asked for any version.

test: reject invalid ed25519 public keys

2f9719a

ssh-keygen only produces valid keys, so no test reached the checks in the conversion to curve25519. The three crafted keys are the identity point, a y that is not on the curve, and the base point plus a point of order 2.

docs: correct the python version and name the OpenSSH ciphers

5ccbb36

requires-python has been >=3.9 for a while, the README still said 3.6. An encrypted OpenSSH key loads if it uses aes256-ctr, aes256-cbc or aes256-gcm@openssh.com.

docs: install the package when building on Read the Docs

7434a7b

autodoc imports crypt4gh, so without the package it documents nothing and the API page lists no functions. Sphinx exits 0 either way, which is why this went unnoticed. The install needs no compiler now.

refactor: rename crypt4gh.sodium to crypt4gh.crypto

f596e87

The module holds no libsodium bindings any more, so the name no longer fits. crypt4gh.sodium stays as an alias that warns on import, and version 2 can drop it.

silverdaz mentioned this pull request Oct 12, 2026

Refactor with Cryptography #62

Open

Navigation Menu

Repository navigation

refactor: replace libsodium with cryptography - #59

refactor: replace libsodium with cryptography#59
Hoeze wants to merge 17 commits into
EGA-archive:masterfrom
ghga-de:refactor/cryptography

Hoeze commented Sep 15, 2026 •

edited

Loading

Uh oh!

Uh oh!

Cito Sep 16, 2026

Uh oh!

Hoeze Sep 17, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Hoeze commented Sep 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why

What changes

Stack

Uh oh!

Uh oh!

Cito Sep 16, 2026

Choose a reason for hiding this comment

Uh oh!

Hoeze Sep 17, 2026

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Hoeze commented Sep 15, 2026 •

edited

Loading