We show that any memory-constrained, first-order algorithm which minimizes d-dimensional, 1-Lipschitz convex functions over the unit ball to 1/poly(d) accuracy using at most d^{1.25 - δ} bits of memory must make at least \(\(\tilde{\Omega }(d^{1 + (4/3)\delta ...\)

Minimizing a convex function with access to a first order oracle--that returns the function evaluation and (sub)gradient at a query point--is a canonical optimization problem and a fundamental primitive in machine learning. Gradient-based methods are the ...