Since the spacers are so small in diameter, they have a very low moment of inertia. Their effect on handling is much less than if the wheels themselves were heavier by the weight of the spacer. It is still unsprung mass though.