Mercury: Ultra-fast language models based on diffusion

Actually in most deep learning schemes for science adding in the "laws of nature" as constraints makes things much worse. For example, all the best weather prediction models utilize basically zero fluid dynamics. Even though a) global weather can be in principle predicted by using the Navier-Stokes equations and b) deep learning models can be used to approximately evaluate the Navier-Stokes equations, we now know that incorporating physics into these models is mostly a mistake.

The intuitive reason might be that unconstrained optimization is easier than constrained optimization, particularly in high dimensions, but no one really knows the real reason. It may be that we are not yet at the end of the "bigger is better" regime, and at the true frontier we must add the laws of natures to eke out the last remaining bits of performance possible.