Correct, if you just see two different addresses of the same wallet, there is no way for you to tell that they belong to the same wallet. You would need additional information in order to be able to tell.