@hjkl yeah, i'm aware of Lipschitz and the like, but most of my experience is honestly just tweaking numbers, trying ideas, and implementing any paper that interests me. i personally find it easier to try things than theorize them.

in my mind, deep learning is more like a highly unstable system trying to settle than anything like convex optimization. a lot of modern techniques seem to be based on simple intuition instead of pages of proof — compare resnets to SELU. just some random thoughts.