Yeah it's really visible when you go there. I was there a couple weeks this summer, and I remember in particular the first little city I got to. In walking around in the evening, I saw and heard only foreigners - i.e. people not speaking German - all through the main part of town, for the first 45 minutes, and then I finally heard someone speaking German.
It wasn't an isolated suburb or anything, this was the main part of a small town, and it was filled with people - younger people too, teenagers - who were not speaking German. That's a good indication that they're not integrating - they're not speaking the local language.
I wonder if it affects the personality/mood of the country as well. It seemed kind of dour. I know Germans have that reputation, but when I crossed the border to the Netherlands (which seems to have MANY MANY fewer immigrants) everybody seemed happier. I wrote a post on it: Big difference in atmosphere crossing a border - Germany to Netherlands