The problem is, of course, that they will always impede traffic to some degree. If they can't, then they might as well just be outlawed, which is obviously unacceptable...
Well, I'm really referencing those who take the 'right to be on the road' to the extreme - where two or more riders ride side-by-sid taking up an entire lane or one rider rides in the middle of the road. Surely even if they ride on the edge of the road they'll slow responsible drivers down, but when you have a group of Tour de France wannabes taking up an entire lane of a two-lane road, it's begging for trouble.