Sadly, money is the reason for most. My wife and I both of us would like to have mere kids unfortunately we are on the edge, financially speaking.
The most "hardest" kid is the first, after that, having more babies becomes a desire. Okay, it's not the same for everyone but I'd say for most.
Money is the driving issue with most of the other father's I've talked to.