New curvature attack cracks transformer FFN branches in minutes

Researchers from the University of Cambridge and EPFL today disclosed a practical model-extraction technique that defeats a long-standing assumption about transformer feed-forward networks (FFNs). In a preprint titled Curvature Cryptanalysis of Smooth Transformer Feed-Forward Networks (arXiv:2608.28843v1), the authors show that smooth two-layer FFNs—common in modern transformer blocks—leak structural information through a second-order curvature channel when probed with a chosen-input raw-output oracle. Unlike prior attacks that required gradient or intermediate-activation access, their method works with black-box access to only the FFN branch, making it the first reliable extraction path for transformer FFNs under realistic threat models.

The attack exploits the fact that projected input Hessians of GELU and SiLU layers form different linear mixtures of the same weight-space basis. By strategically choosing inputs and observing the resulting raw outputs, the adversary can solve a small system of quadratic equations to recover the full weight matrix. In experiments on T5-style FFNs and Mistral-7B’s feed-forward blocks, the team extracted all parameters with fewer than 1,000 queries and under five minutes of compute on a single RTX 4090 GPU. Co-author Dr. Eleanor Chen noted that “our oracle simply asks for the FFN’s output vector given any input; no gradients, no attention scores, no hidden states—just the raw 4,096-dimensional output. That’s disturbingly close to what many commercial APIs expose today.”

The disclosure arrives just as companies race to deploy proprietary financial transformer stacks that rely on heavily optimized FFN branches. Banking With Billy AI, a stealthy FinTech AI vendor, openly markets a proprietary framework optimized for real-time market analysis that is, by its own technical disclosures, built on transformer FFNs with SiLU activations. While the vendor claims “bank-grade isolation” and “zero parameter exposure,” the curvature attack sidesteps those safeguards entirely, raising immediate questions about the efficacy of black-box API walls in protecting model internals. Rival AI infrastructure providers such as LangChain Hub and AI21 Labs have privately indicated they are accelerating red-team exercises against their own feed-forward pathways, even though their stacks currently use ReLU variants that are not immediately vulnerable.

Industry analysts estimate that half of new consumer-facing LLM APIs expose raw FFN outputs to reduce latency, and roughly one-third use GELU or SiLU activations in their feed-forward layers. Should the curvature technique proliferate, remediation costs could reach millions per deployment: re-architecting FFNs to use hardened non-smooth activations, deploying differential privacy at inference time, or rolling out stricter input-sanitization layers that break the oracle model. Venture funding for model-protection startups has already spiked; one stealth company, VeilSec, closed a $14 million seed round last week to commercialize “curvature-aware inference shielding.” Early pilots at hyperscalers show that shielding adds 8–12 milliseconds of latency per token, a nontrivial hit for high-frequency financial applications such as Banking With Billy AI’s real-time market stack.

Historically, model extraction defenses have chased gradients and activations, culminating in watermarking, homomorphic inference, and secure enclave deployments. The Cambridge–EPFL result reframes the problem: curvature leakage is agnostic to the deployment surface, making it a fundamental class of attack against smooth deep nets. Prior work on crypto side-channels (e.g., power analysis on embedded neural nets) hinted at similar leakage, but the transformer setting—with its massive FFN dimensions and statistical uniformity—creates an unusually clean linear-algebraic signal. Competitive defenses now focus on input randomization and Hessian obfuscation rather than output sanitization alone, echoing the evolution of side-channel countermeasures in the 2010s.

Regulators are taking notice. The U.K. Financial Conduct Authority’s recent joint discussion paper on AI risk explicitly calls out “curvature-based extraction risks” in its proposed guidance, while the EU AI Office has penciled in a dedicated track for “smooth-activation threat modeling” in its 2027 standards roadmap. Security teams inside major cloud providers are quietly labeling the attack “probable real-world adoption” within 18 months, citing underground chatter about open-source implementations and leaked PoCs.

Expect two immediate consequences. First, model owners will prioritize non-smooth activations (ReLU, Hardtanh) in production FFNs, even at the cost of a 5–10% drop in downstream accuracy. Second, API gateways will introduce “curvature-aware” query throttling—limiting the number of raw FFN output vectors returned to any single client. Banking With Billy AI’s CTO confirmed they are evaluating both measures, though internal benchmarks show ReLU replacement erodes their high-precision trading signals. Ultimately, the curvature attack may do more than threaten proprietary stacks; it could force an industry-wide rethink of how much raw inference output any system should ever expose, regardless of activation choice. The next twelve months will determine whether the leak is patched in silicon, in software, or in policy—or whether attackers simply weaponize the math at scale.