RFP for improving Ranked Choice Voting support in Snapshot

[AvsA]

Preamble

The ENS DAO has never shied away from experimenting with governance and learning from it. Steward Elections use approval voting, the choice for Endownment Manager was a ranked choice voting, we even had a novel knapsack algorithm to select Service Providers. The crypto space has been lucky to be able to experiment in practice what are often academic debates on governance that could take decades in politics. What we learned from these past experiences:

1. Ranked Choice Voting is a better choice for expressing user opinions. While Approval is better than a simple FPTP vote, it leads to some undesired results: voting on too many candidates is the same as not voting at all, and the binary choice makes it easier to pressure delegates (“please also approve me” is an easier ask than “please rank me as you number one”)
2. Instant Run-Off Voting has fallen short of our expectations. The result is a black box where it’s quite hard to understand the full context of the vote (without some complex sankey charts) and it can be quite unpredictable but most importantly, it fails to deliver it’s main allure: avoid polarization. In an election dominated by two very opposite candidates, a more moderate third candidate will be eliminated on the elimination round and the election will fall back to the two main adversaries.

IRV is often used interchangeably with RCV as if they’re the same but they’re not. The first is about how you vote, the second, how you count the votes. There are uncountable other algorithms that, given ballots with ranks, can be used to pick a winner. There’s a whole category just for Condorcet systems, which are algorithms that will attempt to put run all possible one to one matches and pick the candidate who would win all of them (if such candidate exists). An issue with many of these is that, like IRV, explaining how these systems work can be hard and it feels like a black box where the result is inscrutable. I’ve written more about that challenge.

One crucial factor for us is that our governance happens all in the open, over a long period of 5-7 days, and the ongoing result can be seen on a small column in snapshot. Delegates can change their mind at any point during the period, often based on the ongoing result or social pressure. For that reason, it’s important that at any point someone should be able to easily follow the current stage of the vote, and how it can be influenced, in the simplest form. Another important factor for us is that often we run elections with more than one possible winner, so the result must be able to rank all candidates, no only pick one winner. For that reason, we have picked the Copeland method as the desired method for governance in the future.

Explanation of Round-Robin / Copeland

This RFP is for someone to implement Roundrobin style ranked vote system, AKA Copeland Method. One of the main reasons we are preferring this is it leads to a simpler UI which will be familiar to anyone who ever watched a sports tournament, since it’s the logic behind sports league tournament. While technically Round-Robin and Copeland have slight differences when dealing with ties (which is rare for a highly divisible token) we will be calling the method Round-Robin since it’s a more self explanatory name.

If you have followed any league sports competition, like UEFA, Conmebol, NFL, MLB, NBA, etc, then you already know how Round-robin voting works. Only difference is that all individual matches are replaced by a ranked ballot. That familiarity is an important part of its appeal.

Specification of the algorithm:

1. The ballot should allow the voter to rank any amount of candidates. A clear line (labeled “None of the Below”) should differentiate candidates who were ranked or not in the ballot. A vote with no ranked candidates is a valid vote (although, a good UI should try to confirm if it was not an error).
2. All candidates are compared head to head, and their absolute number of votes will be displayed. In a match between a ranked and unranked candidates, the unranked automatically loses. In a match between two unranked candidates, the ballot will not count to either’s total vote in that match.
3. All matches are calculated and displayed sorted by absolute number of votes. Each victory awards 1 point, ties or losses award 0 points
4. The Average Support for each candidate will also be calculated: it’s the sum of the absolute votes each got in each match (on either side), divided by the amount of matches it was featured in. Average Support is used as a tiebreaker. If all a candidate received was 100k votes ranking them in 1st place, their average support should be 100k. Some elections might want to set a minimum “average support” as an exclusion criteria or quorum.
5. While None of the below should neither score or rank, it’s average support should also be calculated and displayed on the results table. Depending on the rules for that particular election, it can be used as an exclusion criteria or minimum quorum. removed this section as unnecessary.

An example of how it all could be displayed in a Snapshot UI is below. We believe that althought the system can be complex under the hood, its output is quite clear. It’s easy for anyone to follow along what is happening now and what could happen in the future. In the example below, Charlie is ahead, Alice’s best strategy is probably to compare herself favourably with Charlie, but Bob’s best strategy is to compare favourably against Alice.

Request for Proposals

We therefore request developers who would be willing to improve ENS governance by implementing this option. Requirements:

• must be compatible with Snapshot. Either it’s a direct PR on the Snapshot side (acceptance by snapshot is a responsibility of the developer) or it’s compatible with Snapshot (for example an alternative UI for Snapshot’s existing Ranked Choice Voting)
• must implement an UI in which the user will somehow express their opinions of all candidates (with the option of NOT expressing opinions about some of them). An acceptable default would be a simple drag to rank, but we would consider other alternatives.
• must implement an UI showing the resulting calculation in real time, similar to the example above.

Payment would be split in milestones, with the last one being delivered when the option is live on snapshot (or another site compatible with it). The selection of the developer for this will take in consideration their past record of contributions as well as their asking price and proposed timeframe.

If you’re interested in participating, please leave a comment or contact us via DMs so we can arrange to analyze proposals.

This RFP is a suggestion for an improvement for the future of governance. It’s not a suggestion on how the next governance decisions should be made.

[dennison]

Hello @AvsA, this is a pretty cool request! I had a couple quick questions as we think through a submission.

If the resulting output was not hosted on Snapshot, what would the requirements be for compatibility for other tools? For example, if we (Tally.xyz) were to be awarded this RFP, what are our requirements for making the resulting UI compatible (or usable) for other governance providers like Agora?

Is the output intended to be standalone? Like an encapsulated React component?

Are other providers considered competitive if Snapshot elects to build the feature themselves natively?

Thank you!

[clowes.eth]

Hi @AvsA,

This is really interesting - appreciate the time taken to put it together. Alongside the supplementary blog posts my knowledge of voting mechanisms has increased significantly.

A query…

You use the football (soccer) league example. In a situation whereby 20 people are applying for a position (steward for example) am I correct to think that in principle if you are an open book in terms of who you’ll consider (rank) then you’re going to have to select winners in 190 different ‘games’? Assuming so, I’m not sure how one would make that accessible from a UI will be a tough ask and similarly it risks delegate burnout in that a delegate may in principle want to consider everyone, but in practice won’t because of the effort involved…

[NameSys]

@clowes.eth

then you’re going to have to select winners in 190 different ‘games’

No. Each voter will rank N candidates once. You’ll then construct pairwise matches implicitly, and decide the winner between each pair by counting who has been ranked higher more often than the other from M ballots. The post should be more thorough and explain this. The sports analogy is limited and does not tell you how to construct pairwise matches when they don’t exist explicitly.

[AvsA]

@clowes.eth

you’re going to have to select winners in 190 different

@NameSys

The post should be more thorough and explain this.

Yes, NameSys is correct and I apologize if it wasn’t clear. The sports is a mere analogy: in sports all teams play against each other and then from that you calculate their standings. In a Roundrobin/Copeland/Condorcet election you rank the candidates, then we calculate their matches, and from that we calculate their standings. From a UI perspective you just need to drag people around.

I’ll try to make tools to explain this better

[clowes.eth]

Thanks for the clarification @NameSys / @AvsA

[AvsA]

I made a google sheets simulating RoundRobin vote.

RoundRobin Simulation

Make a copy of the sheet, then change the candidates names and the ballots to see the result automatically calculated.

[NameSys]

NameSys is interested in this RFP and we are formally declaring our intent to float a proposal when the applications open.

[AvsA]

Applications are open. Please DM me for any questions if you want.

[nick.eth]

@AvsA

For that reason, we have picked the Copeland method as the desired method for governance in the future.

Who is ‘we’ and how was it picked?

@AvsA

All candidates are compared head to head, and their absolute number of votes will be displayed. In a match between a ranked and unranked candidates, the unranked automatically loses. In a match between two unranked candidates, the ballot will not count to either’s total vote in that match.

This needs a better explanation. How are the candidates ranked head to head? I assume this involves examining every ballot, and for each one where candidate A wins, allocating them that ballot’s votes, but it’s not explicit.

@AvsA

The Average Support for each candidate will also be calculated: it’s the sum of the absolute votes each got in each match (on either side), divided by the amount of matches it was featured in.

What does “on either side” mean here? Wouldn’t every candidate be “featured in” the same number of matches, as long as there’s a single ballot that ranks all candidates?

The example UI does make things much clearer, and I can see how the results would be immediately comprehensible. It’s less clear to me that this voting system is fair and satisfies important voting system properties; are there any proofs of this?

[AvsA]

@nick.eth

Who is ‘we’ and how was it picked?

We the Metagov stewards, but in practice they deferred that to me.

@nick.eth

How are the candidates ranked head to head?

The Round Robin simulation has an actual implementation. The algorithm works as such:

• For all candidates, compile a list of all possible matches. If N is the number of candidates, then the possible matches will be N*(N-1).
  • For each of these matches, count the ballots for that match.
  • If both candidates are ranked on the ballot, then the total weight of that ballot goes to whichever candidate is ranked higher.
  • If only one candidate is ranked on a ballot, then count the total weight of that value to that candidate (we interpret not ranking a candidate meaning that you’d rather have any of the ranked ones)
  • If neither candidate is ranked on the ballot then it doesn’t count on that match
  • When we tally up each match, we count one “Victory Point” for the winning candidate. We then tally all candidates by how many victories they won.

@nick.eth

What does “on either side” mean here? Wouldn’t every candidate be “featured in” the same number of matches, as long as there’s a single ballot that ranks all candidates?

To calculate the “average support”, we go back to each match and average out how many votes each candidate got in each match. This is used as a tiebreaker if candidates are tied on amount of victories. To use a soccer analogy this would be equivalent to the average goals scored by each team in a match: the team who scored the most goals in the championship is not necessarily the winner, but it’s a good indication of overall performance. I’m open to suggestions for other tiebreakers.

Yes, this is the same as saying “we sum all votes gained in each match and divide by N*(N-1)”

@nick.eth

The example UI does make things much clearer, and I can see how the results would be immediately comprehensible. It’s less clear to me that this voting system is fair and satisfies important voting system properties; are there any proofs of this?

Yes, the Arrow theorem says in theory there’s no perfect voting system. And yes this is a strongly opinionated RFP. It’s my opinion that Ranking is better than simple approval, but it’s also my opinion that IRV is a really bad algorithm for choosing a candidate from ranking. So bad, in fact that it might turn people off from ranking altogether.

A Yee Diagram represents candidates in a 2 dimensional political spectrum. Each pixel is colored as who the winner would be if that pixel represented the center of political opinion, the average voter in a bell distributed curve. You can see IRV creates some chaotic results. Plurality makes it impossible for some candidates other than the two extremes to win. Approval is not bad and creates similar results to Condorcet, but I’d like to have the option for a ranked choice which doesn’t suck.

Copeland is a Condorcet compliant election, meaning that:

• If there’s a candidate that would beat any other candidate in all possible head to head elections, then that candidate is guaranteed to win.
• If there’s a candidate that would lose to all candidates in an all possible head to head elections, then that candidate is guaranteed to lose.
• If one candidate is ranked first by the majority of voters then that candidate wins

Copeland doesn’t have some other strong assurances like Clone resistance and monotonicity, which means that there are situations in which adding irrelevant candidates might alter the result in unexpected ways. But it’s also one of the most, in my opinion, intuitive of the methods, due to its similarity with sports championships which have a wide public appeal.

This isn’t also a proposal that all elections MUST use copeland, but rather an RFP to make that option possible in Snapshot, as another tool at the disposal of all DAOs.

[nick.eth]

@AvsA

For all candidates, compile a list of all possible matches. If N is the number of candidates, then the possible matches will be N*(N-1).

Shouldn’t this be n^2 / 2? Otherwise, you will have matches for both A vs B and B vs A.

@AvsA

• For each of these matches, count the ballots for that match.

Aren’t all ballots for all matches?

@AvsA

• If both candidates are ranked on the ballot, then the total weight of that ballot goes to whichever candidate is ranked higher.
• If only one candidate is ranked on a ballot, then count the total weight of that value to that candidate (we interpret not ranking a candidate meaning that you’d rather have any of the ranked ones)
• If neither candidate is ranked on the ballot then it doesn’t count on that match
• When we tally up each match, we count one “Victory Point” for the winning candidate. We then tally all candidates by how many victories they won.

This is clear, thank you.

@AvsA

Approval is not bad and creates similar results to Condorcet, but I’d like to have the option for a ranked choice which doesn’t suck.

Approval is almost indistinguishable from Condorcet here. Isn’t that a strong argument for favoring the simpler of the two?

@AvsA

Copeland is a Condorcet compliant election, meaning that:

• If there’s a candidate that would beat any other candidate in all possible head to head elections, then that candidate is guaranteed to win.
• If there’s a candidate that would lose to all candidates in an all possible head to head elections, then that candidate is guaranteed to lose.
• If one candidate is ranked first by the majority of voters then that candidate wins

Thank you, those are important properties to have, and it’s good to know this satisfies them.

@AvsA

This isn’t also a proposal that all elections MUST use copeland, but rather an RFP to make that option possible in Snapshot, as another tool at the disposal of all DAOs.

[AvsA]

@nick.eth

Shouldn’t this be n^2 / 2? Otherwise, you will have matches for both A vs B and B vs A.

It’s actually n*(n-1)/2 otherwise you’d also be counting all matches of A x A. If there are 5 candidates there are 20 matches.

@nick.eth

Aren’t all ballots for all matches?

Now you’re just nitpicking, you got what I am saying: for each match, check the rankings for each ballot for these two particular candidates.

@nick.eth

Approval is almost indistinguishable from Condorcet here. Isn’t that a strong argument for favoring the simpler of the two?

Yee diagrams are a great way to visually show things like the spoiler effect or the chaotic nature of IRV votes, but they don’t represent all the subtleties of each voting. I do think that, once the issues with IRV are removed, there are some advantages for RCV over simply approval. Being able to say: “I want a chocolate ice cream, but I’d also take a strawberry, otherwise I don’t want any, thanks” is more rich in information than saying “I only eat chocolate or strawberry ice cream”. Score voting (giving 0 to 5 stars to each candidate) also has that subtlety but there’s a good argument that score voting, when used strategically, becomes just approval voting.

We’ve seem some practical examples in ENS voting. For example, in the last steward election, there were lots of good candidates for some of the WG. However if you vote on 5 of the six candidates, you’re in practice just voting down the sixth one and you won’t make much of a difference in the election. In this case, if you do have a preference for one candidate over the other, it’s better not to approve all of candidates so to influence the election towards the ones you truly prefer.

Another difference is how it changes social pressures over votes: you, like me, probably had people in your DMs during the service provider vote asking to approve one or another project. I find much easier to defend your own ballot by saying “I like you, in fact more than Alice or Bob, but you’re not as good as Charlie” than just to say “I like you but did not approve you due to strategic decisions on my ballot” .

[nick.eth]

@AvsA

@nick.eth

Aren’t all ballots for all matches?

Now you’re just nitpicking, you got what I am saying: for each match, check the rankings for each ballot for these two particular candidates.

Not nitpicking; I’m just checking that the denominator is the same in each case? You don’t need to count the ballots afresh for each match.

[snowdot]

@AvsA

must implement an UI showing the resulting calculation in real time, similar to the example above.

For alternative UIs (e.g Dhive) this would be a difficult task as Snapshot’s API doesn’t provide a way to track modified votes unless the indexer performs constant reindexing of all the votes to identify these changes (not a scalable workaround).

It’s something we’ve been actively interested in solving and it’s also the main reason why we can’t display rn charts related to active off-chain proposals. We’ve created an ongoing feature request on Snapshot that will help simplify the process of fetching modified votes: Snapshot feature request.

[slobo.eth]

@snowdot

@AvsA

must implement an UI showing the resulting calculation in real time, similar to the example above.

For alternative UIs (e.g Dhive) this would be a difficult task as Snapshot’s API doesn’t provide a way to track modified votes unless the indexer performs constant reindexing of all the votes to identify these changes (not a scalable workaround).

Maybe off topic, but why is reindexing votes every 5 minutes not a scalable solution? Or is your point that would not be considered “real time”.

[snowdot]

@slobo.eth

Maybe off topic, but why is reindexing votes every 5 minutes not a scalable solution? Or is your point that would not be considered “real time”.

For a project like ours, which aggregates governance data across multiple DAOs, the number of active proposals increases rapidly. As we integrate more DAOs, the computational demands increase, leading to significant database strain, potential latency issues, and added pressure on Snapshot’s API.

To give an example, consider that 100 active proposals at once (e.g. Arbitrum had 97 proposals created in a single day). With thousands of voters, this could result in over 600 API calls per voters indexing cycle (e.g. when refreshing voters data). At this rate, we would exceed Snapshot’s API quota of 2 million calls per month in less than two weeks. And this doesn’t even account for additional API calls needed for governance data or other operations for individual DAOs.

[NameSys]

I think ENS DAO should consider its own governance frontend if it intends to use highly customised voting strategies with varying computational/indexing requirements. At some point, something will break Snapshot’s one-shoe-fits-all UI and data management infrastructure. That’s the least a large DAO can do and something that at least one of the service providers should have built (instead of several teams handing out subnames at $50/piece)

[slobo.eth]

@snowdot

To give an example, consider that 100 active proposals at once (e.g. Arbitrum had 97 proposals created in a single day). With thousands of voters, this could result in over 600 API calls per voters indexing cycle (e.g. when refreshing voters data)

That makes a lot of sense. Thanks for that clear explanation.

[AvsA]

---

Update on this RFP

I’ve had a conversation with Snapshot and they committed to implementing the new strategy without cost. They gave no specific timeframe but believed they could ship in a couple of months.

That’s just the internal calculation of the result, without much modifications to the visualization on the UI. For that, I would still recommend Metagov to separate some funds for alternative visualizations on Dhive, Agora, or other third party websites.