Ezra Winston

contact

I am a PhD student in the CMU ML department, advised by Zico Kolter. I'm interested in advancing our fundamental understanding of deep learning, both through theory and experimentation. My current focus is on transformers, starting from basic questions like representation capacity and the role of dynamic sparse attention. I've also worked on novel, mathematically-principled approaches to deep learning in areas like equilibrium models, provable robustness, data poisoning, and domain adaptation.

ML publications

• Local Signal Adaptivity: Provable Feature Learning in Neural Networks Beyond Kernels. Stefani Karp, Ezra Winston, Yuanzhi Li & Aarti Singh
  NeurIPS 2021. code talk
  

  We offer a theoretical explanation for the performance gap between NNs and NTKs, based on finding a sparse signal in the presence of noise. On a simple data distribution with a sparse signal amidst high-variance noise, a simple CNN trained with SGD learns to threshold out the noise and find the signal. On the other hand, the corresponding NTK, with a fixed set of predetermined features, is unable to adapt to the signal. The theory is borne out in experiments on sythetic datasets such as CIFAR-10 images placed on ImageNet backgrounds.
• Estimating Lipschitz constants of monotone deep equilibrium models. Chirag Pabbaraju*, Ezra Winston* & J. Zico Kolter
  ICLR 2021. code
• Monotone operator equilibrium networks. Ezra Winston & J. Zico Kolter
  NeurIPS 2020. code spotlight talk
  

  Deep equilibrium models suffer from unstable convergence and lack guarantees that a solution exists. We develop a new class of implicit-depth model based on monotone-operator theory. There is a close connection between finding the equilibrium point of an implicit network and solving a form of monotone operator splitting problem, which admits efficient solvers with guaranteed, stable convergence. We show how to parameterize models such as CNNs to guarantee monotonicity, and how to implement the resulting iterative solvers.
• Certified Robustness to Label-Flipping Attacks via Randomized Smoothing. Elan Rosenfeld, Ezra Winston, Pradeep Ravikumar & J. Zico Kolter
  ICML 2020. talk
• Domain Adaptation with Asymmetrically-Relaxed Distribution Alignment. Yifan Wu, Ezra Winston, Divyansh Kaushik & Zachary Lipton
  ICML 2019.
  

  Adversarial approaches to domain adaptation consist of aligning source and target encodings, often motivated as minimizing two (of three) terms in a theoretical bound on target error. Unfortunately, this minimization can cause arbitrary increases in the third term, a problem guaranteed to arise under shifting label distributions. We propose asymmetrically-relaxed distribution alignment, a new approach that overcomes some limitations of standard domain-adversarial algorithms.
• Answering Cloze-style Software Questions Using Stack Overflow. Ezra Winston, Bhuwan Dhingra, Kathryn Mazaitis, Graham Neubig & William W. Cohen
  Machine Learning for Programming Workshop, FLOC 2018. talk

ML preprints and projects

• Monotone deep Boltzmann machines. Zhili Feng, Ezra Winston & J. Zico Kolter.
  preprint.
  

  We develop the monotone DBM, which, unlike the RBM, allows for arbitrary self-connection in each layer, but which restricts the weights in a manner that guarantees the existence and global uniqueness of a mean-field fixed point. To do this, we leverage the monotone Deep Equilibrium model, and show that a particular choice of activation results in a fixed-point iteration that gives a variational mean-field solution. This approach allows for efficient approximate inference in fully-general weight structures for DBMs.
• Provable adversarial L2 robustness by propagating ellipsoids. Ezra Winston, Eric Wong, & J. Zico Kolter
  
  

  While infinity-norm robustness of NNs can be certified and directly optimized by methods such as interval bound propagation, these approximations do not work for L2-norm robustness. We investigate the feasibility of a solution based on ellipsoid propagation (see slides). By replacing ReLU with halfspace-projection, ellipsoid propagation can be approximated using a set of close-formed, low-rank updates (outlined here). The approach works well for MNIST but scaling further seems challenging.
• Counterfactual-based Incrementality Measurement in a Digital Ad-Buying Platform. Prasad Chalasani, Ari Buchalter, Jaynth Thiagarajan & Ezra Winston
  arXiv preprint, 2017.
• Cloze Question Answering Using Weak Knowledge and a Language Model. Ezra Winston, Bhuwan Dhingra, Kathryn Mazaitis, Graham Neubig & William W. Cohen
  preprint.

Other publications

• A framework for co-optimization algorithm performance and its application to worst-case optimization. Elena Popovici & Ezra Winston
  Theoretical Computer Science, vol. 567, 2015.
• On the practicality of optimal output mechanisms for co-optimization algorithms. Elena Popovici, Ezra Winston & Anthony Bucci
  FOGA, 2012.
• Deformations of associahedra and visibility graphs. Satyan L. Devadoss, Rahul Shah, Xuancheng Shao & Ezra Winston.
  Contributions to Discrete Mathematics, vol. 7(1), 2012.
• Linear inequalities for enumerating chains in partially ordered sets. Samuel K. Hsiao, Lauren Rose, Rachel Stahl and Ezra M. Winston
  Transactions of the American Mathematical Society, vol. 363, 2011.

Patents

• System and method of a monotone operator neural network. US patent no. US11468276B2
• Method and system to classify sensor data with improved training robustness. US patent app. no. US20210089842A1
• Methods, systems, and devices for counterfactual-based incrementality measurement in digital ad-bidding platform. US patent no. US10977697B2