While infinity-norm robustness of NNs can be certified and directly optimized by methods such as interval bound propagation, these approximations do not work for L2-norm robustness. We investigate the feasibility of a solution based on ellipsoid propagation (see
slides). By replacing ReLU with halfspace-projection, ellipsoid propagation can be approximated using a set of close-formed, low-rank updates (outlined
here). The approach works well for MNIST but scaling further seems challenging.