Research Proposal: Real-time Community Health Analytics

Proposal Summary

RnDAO is requesting funding for a research project on Real-time Community Health Analytics for DAOs.

The project aims to provide insights for the ENS DAO that improves its ability to collaborate effectively and operate a thriving community.

We’ll review the academic literature, previous web3 work, and work with community leaders to identify the most powerful health analytics for Web3 communities and with special focus on ENS.

For this, we’ll research different applications of Organisational Network Analysis, pulse-surveys, and other techniques; build the necessary tooling (ONA and pulse survey bot); and analyse the data to provide insights to ENS.

Proposal Rationale

ENS DAO holds the largest active membership in the DAO Ecosystem, as measured by DeepDAO (86k active members) and Snapshot (53k members).

Like many DAOs, ENS DAO depends on the health of its community and its vibez—yet understanding and measuring Community Health is challenging.;

Today, DAOs are left to rely on the limited reporting provided by Discord or Discourse and a patchwork of “homemade” surveys to fill in the gaps. These solutions are time-consuming for community managers and contributors to use, and the results, hampered by poor indicators and/or poor sampling, are unreliable.

Perhaps even direr, the lack of real-time analytics leaves community leaders without established baselines to measure against to understand the impact of community-focused initiatives, monitor shocks to the system, or rapidly gauge the impact of system-wide changes (such as migrating to mainnet).

To address these limitations, RnDAO is launching a research project to explore techniques employed both inside and outside the DAO ecosystem to surface those requiring minimal administrative effort from DAO community managers and minimal disruption to DAO members. We believe this approach ensures more representative insights and paves the way for broad adoption of real-time analytics and benchmarking across DAOs.

Beyond Traditional Surveys

This research proposal focuses on the use of two critical techniques as a starting point:

Organizational Network Analysis (ONA) is a structured way to visualize how communications, information, and value creation take place through an organization based on interaction graphs. ONA has been shown to provide a wide range of insights to improve contributor retention, avoid member burnout, predict team performance and community resilience, identify key contributors, enable decentralization, and improve coordination. Although relatively new, ONA is gaining in popularity over traditional survey tools.

Pulse Surveys are frequent and automated micro-surveys that provide both qualitative and quantitative insights. Pulse Surveys have been shown, in traditional organizations, to increase employee response rate and employee engagement with related initiatives. They’re also used as a tool for culture design and implementing culture change. Lastly, Pulse Surveys significantly reduce admin work for community managers and related roles.

Although our initial focus is Community Health metrics, this sets the foundation for further applications. The combination of ONA and pulse surveys offer unprecedented levels of actionable insights in real-time. Some of the potential applications and insights for the ENS DAO are:

  • Generate baseline metrics for Community Health / vibez to quantify and better understand the impact that a specific event is having on a community and/or sub-groups within the community
  • Predict which contributors are likely to leave the DAO and take preventive action (without breaching privacy)
  • Build funnels to track member onboarding and identify areas for improvement
  • Identify measurements of decentralization to serve as KPIs or Insights metrics
  • Monitor specific topics like contributor wellbeing, alignment, community experience, etc. in near real-time
  • Attract talent and investment with objective Community Health metrics instead of proxy metrics like member count or proposal count, or financial metrics such as TVL
  • Help new contributors find context-rich mentors outside of the existing pool of well-known but time-poor candidates

In addition to the initial research on Community Health, the potential applications mentioned above (and others to be found) can enable more effective and targeted efforts to build healthier DAO communities.

This research also helps reduce the tooling gap in DAOs compared to the employee and stakeholder experience at traditional corporations.

Project Description

  1. Interview DAO community managers to identify key pain points in maintaining Community Health, defining Community Health indicators, and, subsequently, managing them
  2. Host community discussions with other researchers and practitioners and triangulate insights across a literature review, which includes:
    • Review of recent work on DAO Health surveys (e.g., Talent DAO’s work on survey questions for DAO health) and Blockchain Ecosystem Health (e.g., The Governauts and Token Engineering Academy’s work on modeling Near, Ocean Protocol, and others on ecosystem health)
    • Review of recent scientific research on Organisational Network Analysis
    • Review of recent research on belonging, employee retention, burnout, sense of community, and social media’s impact on wellbeing.
  3. Select a set of data points for collection and a hypothesis for analysis
  4. Build a data collection tool that combines Organisational Network Analysis with Pulse Survey functionality
  5. Deploy data collection tool in the discord of participating DAO(s)
  6. Analyze the insight on network structure and dynamics and provide a real-time indicator of Community Health
  7. Agree on follow-up initiatives and experiments to improve the sustainability, inclusivity, and scalability of the community

Deliverables

  • Synthesis of literature review and a conceptual framework on Organisational Network Analysis applied to DAO community Health and Vibez to deliver insights on the sustainability, responsibility, and generativity of a community
  • Data analysis and insights on DAO Community Health, using the ONA and Pulse Survey data collection tool, to identify potential risks and challenges and create baselines to benchmark the evolution of the community
  • Workshop with DAO members to brainstorm initiatives and interventions to improve the resilience and health of the community

Risk Considerations

All raw data collected must be anonymized. All charts must be reviewed prior to publication to minimize exposure to social engineering by malicious actors.

Strong access control must be enforced and managed by a person with legal obligation to act in good faith. Additionally, we will comply with GDPR wherever necessary.

ONA has already proven its value in traditional organizations and social network analysis but, to the best of our knowledge, it has yet to be applied to DAOs. As a consequence, as a research project in a new field (Community Health with the use of Organisational Network Analysis as applied to DAOs), there is a natural level of uncertainty. To mitigate this risk, we’ve decided to split the cost of the research among multiple DAOs.

The findings, especially early results, must not be taken for granted. Small errors in the model can lead to interpretations that have outsized impact on the actual health of a community. Where relevant, we will employ a combination of industry best practice and existing solutions to better model complex systems and validate correctness.

Team

This research project is led by RnDAO, an innovation center focused on empowering humane collaboration through DAO research and tooling.

Team Leads

Danielo

Previously, Head of Governance at Aragon, 8 years experience in Organization Design consulting (clients include Google, BCG, Daymler, The UN, and multiple startups), Visiting lecturer at Oxford University.

Twitter: https://twitter.com/_Daniel_Ospina

LinkedIn: https://www.linkedin.com/in/conductal/

Katerinabc

Ph.D. in Team Dynamics using Social Network Analysis, Teaching Collaboration, and Organizational Performance at Northwestern University (since 2016).

Co-organized Learning in Networks sessions at the International Conference of Social Network Analysis (2018 - 2020), and previously advised a people analytics company on social network metrics.

Twitter: twitter.com/katerinabohlec

Linkedin: linkedin.com/in/katerinab

Github: https://github.com/katerinabc/

Hegadget.eth

Software Engineer at Aragon. Previously, Product Manager at Neolyze (Business Intelligence Dashboard for Instagram).

Github: https:/github.com/gadget2git/

Twitter: https://twitter.com/mr_gadget22

Team Advisor

Sam

Previously, Head of Technical Research at Aragon. Previously, Lead Developer of the official JavaScript API for the Ethereum blockchain.

Github: https://github.com/nivida

Twitter: https://twitter.com/furter_samuel

Note: Other RnDAO members will participate throughout the process in user research, literature review, tool development, and workshop facilitation.

Budget & Timeline

Budget

$30,000 in USDC (or equivalent)*

  • Academic Literature Review and User Interviews: $8,000
  • Data Collection Tool Development: $17,000
  • Data Analysis, Insights Report, and Community Workshop: $5,000

* the requested budget forms part of a total $90,000 target. Total project costs will be split across multiple, participating DAOs. For large communities (2,000+ members), project costs are bespoke.

Estimated Timeline

Week 1-6

  • Literature Review
  • Data Collection Tool Development

Week 6-8

  • Data Collection Tool Implementation

Week 8-10

  • Data Analysis and Report Creation
  • Workshop with Grant Sponsor team to discuss findings and recommendations
Should the ENS DAO fund Real-time Community Health Analytics by RnDAO?
  • Yes
  • No

0 voters

3 Likes

This seems like it could be worthwhile to fund. Tagging @inplco from the recently formed analytics subgroup.

5 Likes

Hello RnDAO,

Nice to see Daniel and the gang here; I met Daniel and couple of other RnDAOists at the satellite FAOist event on Day 1 in Amsterdam. Good times :slight_smile:

I have recently committed to a strictly quantitative semi-annual survey of the DAO, which this subWG will use in a very elaborate quantitative study modelling the several attributes of the DAO, its perception, fair compensation in the ecosystem and governance strategies to be implemented within. In that respect, your proposal is very much aligned with my intent. Having said that, we need at least one such survey in around 6 weeks from today since the steward elections are due to take place around June 30. Can your proposal include fast-tracking of at least one quantitative survey before, say June 27? I am more than happy to help your team in person to get this done faster.

Further, ENS operates slightly differently than most major DAOs and very differently from the smaller DAOs. The reason for this is that ENS ecosystem is highly diverse since ENS Protocol is as much an infrastructure as it is a product, encompassing concepts ranging from identity to web3 to authentication. This results in a system where larger contributions to ENS generally (not exclusively) come from small organisations, which is complemented by a very diverse range of individual human contributions. Your proposal is one example of a small organisation collaborating with ENS. My point is that ENS is as much a B2B system as it is P2B. Your current proposal targets P2B systems where individual contributors play the majority role; you might have to rethink on this a little bit to align yourself with the additional B2B nature of ENS. For instance, I can gather the following organisations (which you will need to survey) from the top of my head who are plugged into ENS right now are:

True Names Limited (@nick)
SpruceID (@rocco)
Esteroids | ETH.LIMO (@neiman)
Flipside Crypto (@fig)
Orca Protocol (@julz)
Show Karma (@mmurthy)
Interplanetary Company
Nifty Chat (@slobo.eth)
Protocol Guild (not yet finalised)
Nomic Foundation (rejected once)

along with several contributors running ENS Leaderboard (@gregskril), ENS Tools, Dune Dashboard, Support Mods (@serenae), plus many more key individual contributors who have added in non-technical ways to the ENS ecosystem and community. Then there are satellite communities like the 999, 10K (@aox.eth), 100K, 24h etc clubs that have recently sprung up with thousands of members. These communities are of utmost importance to ENS since they are the real “consumers” of ENS and drive the adoption of .eth. It will be a very challenging task to be able to survey these communities although it does come with a huge upside of being able to gather massive amounts of raw data on “consumer” behaviours.

Overall, I mostly support this proposal and we should fund this if RnDAO believes they can survey and sample this new ecosystem meaningfully.

3 Likes

I don’t feel that this research would be valuable to the ENS DAO at this stage.

While there are a large number of $ENS tokenholders, the DAO is still very much in its infancy. We are in the process of setting up appropriate structures to encourage more active participation in the DAO.

I would like to see research like this in 6-12 months when there is more meaningful feedback and data to collect. An analysis of community health of the DAO at this stage seems to be very premature.

5 Likes

Can we get a TL;DR of the proposal? The summary is basically a title.

What kind of bespoke tools do you have in mind building? There are already many tools and services are available so I suggest you research existing tools first and suggest what date is missing that needs building.

1 Like

Do you have an example in mind of such a tool?

I agree with the timing argument. 6 months from now would be ideal.

As @noturhandle mentioned, DeepDAO already gives great insight.
Also if the data is on-chain , you can create lots of dashboard using Dune unless you are solely relying the data based from surveys.

I think the tools referred to in the post are not analytics tools (e.g. DeepDAO) but rather the survey tools. In other words, this ‘analytics’ is trying to sample the social spectrum, not the number spectrum. It is impossible to avoid overlaps though. I also think Flipside Crypto has the likelihood of being more useful than DeepDAO since we can custom request all we want from their CEA workstream.

Thanks for the responses, everyone. I’ll answer each in turn below:

Can your proposal include fast-tracking of at least one quantitative survey before, say June 27?

Hey @inplco, thanks for the detailed response. I missed the DAOist event, so I may have missed you, unfortunately. Daniel says hey :wave:

Ideally, we’d spend the initial 6 weeks evaluating options and implementations. Fast-tracking development of a single solution to meet a June 27 deadline will likely distort the value of the initial research. We’re open to collaborating with you to draft the survey and evaluate any existing tools in those initial weeks.

Your current proposal targets P2B systems where individual contributors play the majority role; you might have to rethink on this a little bit to align yourself with the additional B2B nature of ENS.

Great point. We’ve given this some thought and agree that, at least in this initial phase, we’d be unable to extend the research to include the entire ENS ecosystem, including ENS’ partner communities and satellite communities. That being said, our goal is to ensure our research informs future development of community health analytics tools and we’d aim to capture the problem in our work. We’d suggest basing the research on the elements of ENS that are P2B, and taking a sample-based approach with a few communities across the ecosystem. Once, we’re confident we have an approach that works, we can expand the scope to include B2B relationships.

An analysis of community health of the DAO at this stage seems to be very premature.

Hey @alisha.eth, thanks for responding so quickly and honestly.
While the DAO is undeniably in its infancy, this is an opportune time to begin monitoring health. If the research was complete today, I’m confident it would prove itself a useful aid when evaluating the structures most appropriate for increasing active participation in ENS DAO.

Notably, the project’s purpose is to contribute to DAO collective knowledge around the problem of community health through joint research. Therefore, while you may argue that the value to ENS of a single round of analytics around community health offers limited immediate utility to ENS DAO, funding the research and contributing ENS’s unique problem-set today informs the roadmaps of a number of future products, improves the experience for current and future contributors and members, and lowers the administrative overhead for community managers across the ecosystem. Additionally, any findings will also directly serve to inform ENS’s evolution as a DAO.

Can we get a TL;DR of the proposal? The summary is basically a title.

@neiman You’re right :man_facepalming: I’ll update the summary now. Let me know if it’s an improvement.

What kind of bespoke tools do you have in mind building? There are already many tools and services are available so I suggest you research existing tools first and suggest what date is missing that needs building.

Hey @matoken.eth. I agree, there are lots of tools available for analysis of on-chain data. @inplco puts it well—we’re interested in the health of social connectivity within the community.

As an example, you can imagine a survey bot connected to a dashboard that a community manager or steward uses to issue Pulse Surveys to members. They can then cross-reference the results with Organizational Network Analysis.

4 Likes

You should add a poll to the post with the title

Should the ENS DAO fund Real-time Community Health Analytics by RnDAO?

and options

  • Yes
  • Not now, in 6 months
  • No

You can add the poll through the settings icon in the draft window. This will act as an informal pulse check.

I am convinced that we should fund this proposal now. I will add my comments and reasons on the poll.

3 Likes

From what I understand of your initial post you’re proposing to do only pulse surveys apart from ONA? That’s something we never do at work, we’d use pulse surveys in conjunction with other surveys only to avoid the common issue of there being too many long surveys.

Each survey takes quite a long time to draft as well, as it’s tailored for each client. The idea of a semi-generalized pulse survey bot seems like a minor nightmare to me.

You’d need to do several different types of surveys in order to survey well, and I think that designing those surveys well would take a longer time than simply asking people at this point. We’re simply not that many people regularly active in the DAO yet, certainly not enough to warrant the use of ONA or even to produce meaningful results using it.

I think that a good start would rather be more basic and easily attainable metrics while the DAO is young, and then review and adopt more advanced (and costly) analytics methods as the DAO grows and the need for specific data increases as more complex problems present themselves.

1 Like

Hi @inplco,

I’ll add a poll now but I suggest we remove the “in 6 months” option as participation in the research project won’t be an option.

Are you happy for me to proceed on that basis?

1 Like

Yes, please go ahead with ‘Yes’ and ‘No’ options.

1 Like

Hi @Cthulu.eth, thanks for responding and congrats on the ENS name.

The research will leverage pulse surveys, ONA, and other techniques to provide insights to ENS about its community health. We’re not constrained to the techniques mentioned in the proposal.

Each survey takes quite a long time to draft as well, as it’s tailored for each client. The idea of a semi-generalized pulse survey bot seems like a minor nightmare to me.

You’d need to do several different types of surveys in order to survey well, and I think that designing those surveys well would take a longer time than simply asking people at this point. We’re simply not that many people regularly active in the DAO yet, certainly not enough to warrant the use of ONA or even to produce meaningful results using it.

This is the problem statement we’re tackling. It’s currently not clear which approach will work for DAOs and, ideally, we can identify which of the available solutions makes an acceptable trade off between effort and reward on behalf of DAOs and their community managers.

However, the goal here is to determine the best technique to analyze community health in real-time, not simply the output of the initial surveys conducted during our research.

2 Likes

Poll added

1 Like

Thanks for the clarification, that sounds fair enough. I was afraid at first that we’d just be flooded with pulse surveys :slight_smile:

2 Likes

I have voted ‘YES’ on the proposal after discussions in this thread.

  • Considering that the research work itself will take 10 weeks (theoretically :wink:; reality is always longer) and it will take another 4 weeks for the proposal to pass and such, I believe the results of this proposal will be available right in time for elections for Q1/2 2023. It is time ENS DAO funded a proposal with conviction.

  • We also need immediate help in designing very good surveys for upcoming elections in 15 days, if possible. I will personally push for it to make it happen despite time being very short.

  • Independent of the research led by RnDAO, A&R subWG will be working on ungameable models for delegate and steward reputation and impact in Q3/4. RnDAO’s work will add to that workstream through their surveys. We need this starting next term to have a consistent report of the entire term by Dec 2022. This cannot wait until end of year.

2 Likes

Have you spoke to Coordinape guys? They do have some sort of network mapping. Maybe you can use their tool for cross valuation and focus on analysis so that you don’t have to build the whole toolset. IMHO, the network graph looks fancy but doesn’t really give much insight to what you can do with the data. Do you have any example case you are going to use as an aspiration?

3 Likes