What I’d like to know is.. If a good model can be trained with much fewer GPUs using a breakthrough technique, can the breakthrough technique be used by OpenAI, MSFT et al who has loads of GPUs to train a model that is orders of magnitude better than their state of the art today?

We’ve been getting the impression that the limiting factor was the number of GPUs right? If so, this reduces that bottleneck and frees them up to do even better right?