Gram-Space Manifold
Muon

Geometric Constraints

The Stiefel manifold provides a natural constraint space for weight matrices, preserving orthogonality while allowing for rich geometric structure. By constraining our weights to this manifold, we observe emergent properties that are not apparent in unconstrained optimization.

This geometric perspective reveals connections between the structure of learned representations and the underlying mathematical constraints imposed during training.

The manifold structure provides a principled way to understand how neural networks organize information, revealing patterns that emerge from the interaction between optimization dynamics and geometric constraints.