This article was written in 1960 by the physicist Eugene Wigner.
At first, he tells two stories about how statistics is useful for teaching us things about the world and how it seems absurd that those ideas show up in the world (one character says there’s no way that pi could be related to a gaussian distribution), and another character is incredulous that you can sample a relatively small sample from a large population and draw meaningful conclusions about the entire population with a carefully designed experiment.
He goes on to say that mathematics has an uncanny usefulness in so many natural sciences that it must have some larger tie in to science.
Wigner discusses some extra uses in the natural sciences where mathematics was useful, such as the law of falling bodies.
He also notes that if mathematics could ever establish a theory for the phenomena of consciousness, it could prove frustrating, since they could not prove their own consistency. (Much like godel’s incompleteness theorem).
Some reasons why math has snuck its way into the sciences could be because mathematics is just a toolset for solving complicated problems. When scalars proved insufficient, vectors and later on tensors were invented, along with methods to simplify calculations on them. Another example is approximations and stochastic methods being picked up by mathematicians to solve hard problems in biology, the weather, and physics.