Weekly open source: a pure-Rust crypto/networking stack goes public
The big story this week is a stack. Four pieces went public and interlock cleanly: purecrypto (the crypto floor), compcol (the compression floor), puressh (SSH on top of purecrypto), and rsurl (a curl that speaks 14 protocols, built on both). No OpenSSL, no C — the whole tower is Rust. Separately, btmalloc landed: a research allocator organized around one unusual idea.
The pure-Rust stack
The shape: purecrypto provides every cryptographic primitive;
compcol provides every compression codec; puressh builds SSH on
purecrypto; rsurl builds a curl-shaped client on purecrypto +
compcol, and reaches back into puressh for sftp:// and scp://.
Each layer depends only on the ones below it, and none of them
link C. And fstool — last week's disk-image toolkit — became
the first heavy downstream consumer, ripping out five C-backed
compression crates for compcol and moving its encrypted-DMG path
onto purecrypto.
purecrypto
purecrypto — "a
pure-Rust, no_std cryptography toolkit" — reached v0.5.0. The
surface is wide: RSA, ECDSA, Ed25519, the NIST post-quantum trio
(ML-KEM, ML-DSA, SLH-DSA), X.509 + CA + CRL, TLS 1.2/1.3, DTLS
1.2/1.3, and QUIC, with a CLI on top. This week alone it added
RFC 9180 HPKE, Encrypted Client Hello end-to-end, RFC 8879
certificate compression (via compcol), RFC 7250 raw public keys,
OCSP stapling, P-384 and P-256 ECDHE, a native secp256k1
pseudo-Mersenne field backend, ristretto255 (RFC 9496), and
hazmat low-level APIs for threshold work.
The other half of the week was hardening: a parallel security
audit drove fixes across DTLS/QUIC/TLS/RSA/X.509/FFI — bounding
QUIC ACK-range processing against CPU exhaustion, rejecting
un-offered cipher suites in ServerHello, validating p·q == n on
private-key import, enforcing RFC 5280 nameConstraints across
the chain, and zeroizing key material on drop throughout.
compcol
compcol — "a
collection of compression algorithms in pure Rust", zero runtime
dependencies, unsafe_code = "forbid" crate-wide — also reached
v0.5.0. It puts every codec behind one streaming trait: the
modern set (deflate / zlib / gzip / LZMA / xz / Zstandard / Brotli
/ LZ4 / Snappy / LZW / LZO), and a long tail of legacy formats
that mostly don't exist in pure Rust anywhere else — RAR 1/2/3/5,
LZX, Amiga LZX, Quantum, PPMd, LZFSE, ADC, bzip2, Microsoft Xpress
/ Xpress Huffman, LZNT1, Deflate64, ZIP Shrink/Reduce/Implode,
LZS, LZSS, and three StuffIt methods (LZAH, method-13, Arsenic),
each validated against real archives. A by-name factory makes any
of them selectable from a config string or the compcol CLI
filter. This is the codec floor under both rsurl (HTTP
Content-Encoding) and purecrypto (TLS cert compression).
puressh
puressh is new this
week — a pure-Rust SSH library "in the spirit of libssh", every
primitive coming from purecrypto, no unsafe, no FFI. It went
from initial scaffold to a working stack fast: client, server
(sshd with privilege drop, PAM, fork-per-connection), SFTP v3,
SCP, ssh-keygen, known_hosts with TOFU, and the full
forwarding set — agent (-A), X11 (-X/-Y), local (-L),
remote (-R), and direct/tcpip-forward. There's a C ABI and an
interactive PTY shell with SIGWINCH and exit-status.
Most of the week's commits, though, are a single sustained
security pass — dozens of them: strict-KEX for CVE-2023-48795,
capping userauth field lengths, gating SHA-1 ssh-rsa and the
none method behind explicit opt-in, rejecting RSA host keys
under 2048 bits, O_NOFOLLOW on SCP receive, jail-aware symlink
rejection in SFTP, redacting passwords in Debug, and zeroizing
passphrases across the FFI boundary. Status is honest in the
README: scaffolding filling in, but the security model is being
built in from the start rather than bolted on.
rsurl
rsurl — "a pure-Rust
implementation of curl", TLS via purecrypto, no OpenSSL — is also
new, and the protocol list is the headline: HTTP/HTTPS, HTTP/2,
HTTP/3, WS/WSS, FTP/FTPS, IMAP, POP3, LDAP, MQTT, DICT, GOPHER,
RTSP, TFTP, and FILE. It ships as library, C FFI, and CLI, with
curl-parity flags (-d/--data-*, -F multipart, -T upload,
-x proxy, cookie jar in Netscape format).
This week filled in the hard protocols: HTTP/2 flow control,
stream multiplexing, HPACK Huffman + dynamic table, and a
process-wide connection pool; HTTP/3 QPACK dynamic-table decoding
wired into dispatch; WebSocket bidirectional frames with
permessage-deflate; IMAP STARTTLS + SASL; LDAP substring /
presence / extensibleMatch filters; FTP upload + resume; RTSP
session flow; and sftp:// + scp:// routed through puressh. It
swapped flate2 for compcol so Content-Encoding (gzip, deflate,
br, zstd, even compress/LZW) decodes through the in-house stack.
Like the others, it took a dedicated security pass — CRLF
injection across protocols, decompression-bomb bounds, a panic
unwind barrier across the C ABI.
fstool
fstool — the pure-Rust
disk-image / filesystem toolkit that landed last week — spent W22
(v0.4.1 → v0.4.9) becoming the stack's proving ground. The
compression migration is the through-line: flate2, lzma-rs,
lz4_flex, minilzo-rs, and bzip2-rs all came out, replaced by
compcol as the single codec backend (zip/DMG/HFS+ zlib+deflate,
LZMA, lz4+lzo, DMG bzip2+LZFSE), and the encrypted-DMG path moved
onto purecrypto.
That migration unlocked a parade of read-only archive formats, all
via compcol: Microsoft CAB (multi-block MSZIP via preset
dictionary), RAR5 (store + compressed, including solid archives),
Amiga LZX, 7-Zip (Copy/LZMA/BZip2/Deflate), StuffIt classic
SIT!, SEA ARC, and LHA/LZH. On the disk side it added a classic
HFS reader (DiskCopy 4.2 floppies, System ≤ 8), the DiskCopy 4.2
container backend, Apple Partition Map, HFS resource-fork
read/inventory/extract, --path-style {unix|native}, and ls -R
+ readline editing in the shell. It also took a security pass of
its own: bounding untrusted allocations and validating geometry
across every reader (ext/xfs/ntfs/hfs+/apfs/f2fs/exfat/fat/iso9660/
squashfs/tar/grf) and the GPT/DMG/qcow2 header parsers against
malicious images.
btmalloc: the call site as primary key
btmalloc is a
research malloc/free/realloc replacement (in C) built around
one idea the production allocators leave on the table: the call
site is the primary key. Every allocation is grouped into a
partition by hash(__builtin_return_address(0)), and placement,
lifetime cohorting, and reclamation all derive from it. It went
from an mmap-per-alloc baseline through phases A–G in the week:
per-thread region caches, lifetime cohorting, empty-slab decommit,
io_uring-backed async madvise, memfd-backed Mesh-style compaction,
freelist safe-linking, and a call-site heap profiler.
The README reports a head-to-head against seven other allocators (glibc, jemalloc, mimalloc, tcmalloc, snmalloc, ffmalloc, hardened_malloc): cross-thread free statistically tied with snmalloc for fastest, and the tightest memory footprint of all eight under fragmenting churn (2.4× live bytes, where the mainstream performance allocators sit at 16–18× and never give it back).
But the call-site key — the whole premise — is also the catch. It buys those two wins (a freed slot returns to its home partition no matter which thread frees it, and cohorting by origin keeps the footprint tight), but hashing the return address and resolving the owning partition on every call is overhead the mainstream allocators simply don't carry, and it drags on the common path. Much of the week's perf work was chasing that owner-resolve cost (pluggable resolution engines, profile-guided hot-path cuts) and it doesn't fully come out. That's the verdict: a nice idea whose central mechanism is too expensive to make a general-purpose drop-in. btmalloc stays a research project — the value is what the experiment shows, not the allocator it produces.
univdreams: a real QuickTime runtime
univdreams spent
the week getting an actual QuickTime engine running under its
Win32 emulator. The chain: unpack MSI cab streams so real file
bytes hit the VFS, walk InstallExecuteSequence and run
CustomActions (DLL queue + dispatch), pre-load qtmlclient.dll +
quicktime.qts, then InitializeQTML and EnterMovies both
return noErr — a full QuickTime runtime, up. From there the VfW
encode/decode harness drives QT's built-in codec components.
Supporting work: a large Win9x-era user32/gdi32 surface for old
installers, RegOpenKeyEx/RegQueryValueEx over a virtual
registry, a synthetic environment block, and dynamic
LoadLibraryA driven by VFS lookup of PE imports.
goro: more native lowering
The VM kept absorbing PHP. This week's lowering moved compound and
inc/dec assignment off AST-delegation: $arr[k] OP= rhs and
$arr[k]++ on local arrays, $obj->prop OP=/++, dynamic-name
property and static-property compound/inc-dec
($obj->$x OP=, Cls::${$x}++), variable-variable assignment
($$name = v), unset/isset/empty on object properties, and
the nullsafe $obj->prop ?? default. Plus a bug44660 inc/dec
error-verb fix and an OP_ARRAY_PRE_CHECK_LOCAL fix for
array-compound regressions.
origami: the MD engine moves to the GPU
origami — the
molecular-dynamics engine — grew a GPU backend. The whole Langevin
integrator now runs on the GPU via WGPU: LJ + Coulomb + GB
Born-radius terms in fused kernels, a Verlet-list nonbonded kernel
that kills the last O(N²) inner loop, bond/angle/dihedral/improper
fused into one kernel, BAOAB integration with an xoshiro128++ RNG,
and SHAKE constraints unlocking dt = 2 fs. This week the GPU path
picked up Shrake-Rupley SASA — dot-density area and energy first,
then smooth-coverage forces — wired into the integrator, and
reached end-to-end RNA dynamics on the GPU (with a CHARMM27 5-term
dihedral fix). On the RNA side: an A-form helix builder (an
anti-glycosidic fix, canonical twist found via a 4D
backbone-torsion grid search), hydrogenation of X-ray RNA
structures, and native-PDB validation against UUCG and GNRA
tetraloops.
Still the personal-curiosity track. Proteins and nucleic acids are the output side of the machine I eventually want to be able to read.
OxideAV: Blu-ray remux + container depth
The framework ran its weekly multi-agent sweeps (rounds 177 → 193,
each a fan-out across the codec/container/3D crates). The
substantive landings: a bluray:// remux fan-out via
MultiTitleSource (%s.<ext> per-title output), and a small
consolidation — oxideav-aiff
was created and then folded straight into
oxideav-iff::aiff
(which also grew ILBM 24-bit true-colour and ANIM vertical-delta
decode in the same week).
Also this week
- chiefstaker — follow-up on last week's audit pass: CI tightening and a required-metadata test update.
- azusa-opensource-recipes — build-recipe touch-ups in the AzusaOS package set.
Next week
The stack keeps settling: purecrypto and compcol toward stable
APIs, puressh filling in its state machines past scaffolding, rsurl
closing the remaining HTTP/3 gaps, fstool toward its v0.5 stable
surface. univdreams drives real QuickTime
codecs through the now-live runtime. origami pushes the GPU
backend at larger systems.