They cannot access your funds, but they can see what addresses and transactions your wallet is querying for. So it's more of a privacy concern if you're using their servers.