Water could be considered a pollutant if your home was flooded with it. If the Atmosphere was flooded with CO2 maybe that would have serious consequences for life on earth. Does that sound scary?
The levels of CO2 in the atmosphere is 0.04%. Its basically a trace gas. And plants, grass, algae, consume it. The CO2 that humans release already existed. Stored underground in the form of natural gas and oil, among other things. When we use it we are releasing the carbon back into the atmosphere. And it will be consumed by plants and what not again. And these plants among other things will eventually become oil again, its a cycle and therefore you can argue Earth is naturally carbon neutral.