Not crazy about the idea to downvote. If it were implemented, I think it should be much more expensive than upvoting. Like someone already stated, blocking is a good alternative.