My background was also math and physics, before going into econ. However, I was already pretty skeptical of using the same kind of mathematics in econ that gets used in physics. I think the Austrians get that mostly right.

I do think there's a huge role for set theory and abstract algebra, though; a la Kenneth Arrow's famous work.

I mostly do econometrics in my own research. While I don't think that's the right way to understand economics, I do think it's invaluable for understanding economic history.