The main issue is not that it’s intrinsically tied to monetary value, but rather the question of who controls the votes. If a person is given votes without tokens, then they are still subject to the rules of whoever controls the actual tokens and can redelegate them away.