Check out Andrew Gordon Wilson's excellent paper "Deep Learning is Not so Mysterious or Different" for a discussion of the ways in which existing learning theory does and doesn't work neural nets.

https://arxiv.org/pdf/2503.02113