Notary Leaderboard

[panges2]

Introducing the Notary leaderboard! 🚀

With over 100 applications in the 4th Notary election, we see a growing interest in joining the Notary community. As it grows we want to foster accountability, encourage “good” behavior and discourage “bad” behavior.

we'll experiment with many social accountability methods starting with the Notary Leaderboard. The community has embraced this idea as seen with the alternate version created by @AlexxNica

This is a crucial step in the development of the notary community, so with the help of the DMOB team, the governance team has developed its own internal version of the leaderboard and is now releasing it:

LINK TO NOTARY LEADERBOARD V1 HERE

This website aims to:

1. Increase transparency in the notary community by publicly displaying general information about the notaries.
2. Flag what we believe to be “bad” notary behavior
3. Be a place to collect information for future notary risk scores, rewards, and penalties.

Future of Notary Leaderboard 💡

The current iteration of the website serves as a performance dashboard for notaries, presenting relevant information in a report card format. At present, it does not yet have the capability to flag "bad" notary behavior, as data points alone are insufficient for such identification. However, future development plans include using a combination of data points to identify problematic behavior heuristically. In the same time, we also plan to eventually aggregate these metrics into a "Notary Risk Score", which will be used as a data point in the determination of rewards and penalties for notaries.

Data for the website is sourced exclusively from the DMOB API, which currently limits cross-checking statsitcs. The hope is that other teams will eventually collect data and contribute to the cross-checking process.

The aim is for the Notary community to eventually become self-moderating, with tools like this website providing accountability mechanisms. If you have any questions or feedback regarding the website, please reach out to @philippe Pangestu in the Filecoin Foundation Slack or comment here!

Issues ⚒️

While we are making continuous improvements, there are several limitations with the Notary Leaderboard that we are currently aware of:

• Although the statistics change when we filter by DA/EFIL+, notaries that did not do those types of allocations still show up. eg. If we filter by just EFIL+ it still shows non-EFIL+ notaries, but their statistics are just zeroed.
• Notary Score and Calls Attended statistics are missing from the report card.
• Some average TTDs are calculated with little datapoints. Could be important to display confidence metric.
• Audit trail in client allocation pop-up should link to comment where datacap is issued.
• Improved tooltip needed to explain "show inactive notaries" toggle.
• Definition of Average TTD requires clearer explanation in a tooltip.
• Clients on profile page need to be sorted by latest allocation.
• All issues (both open and closed) should be displayed when clicking "issues created".

This list is not final as this is a iterative process and we are likely to face issues in the future.

Tips for usage 📚

Landing Page

On the landing page, you can use the following toggles to refine the data display:

• Activity Types: Choose from subsets of LDN (Large Dataset notary), DA (Direct allocation), and EFIL+. For example, if both LDN and DA are toggled, average TTD (time to datacap) will be calculated with all notary activity for LDN and DA
• Regions: Filter the notaries based on their geographical locations.
• TTD Sorting: Sort the notaries based on their TTD in either ascending or descending order.
• Active Notaries: Include or exclude inactive notaries from the display. Inactivity is defined as a lack of signatures in the last two weeks.

The notary report cards provide a quick overview of the notary's information:

• A profile picture, generated by the signing address and the notary's GitHub profile, is displayed along with the notary's name and organization (e.g. Charles Cao, Filswan).
• The signing address and GitHub profile of the notary are also included.
• The card displays various statistics about the notary, with tooltips available for more information by hovering over the icons next to each statistic. Note that the statistics for calls attended and notary score are still in development, as they lack an automated data pipeline.
• The "View Profile" button leads to a more detailed profile page, showcasing information about each datacap allocation.

Profile Page

On the top of the Notary Profile page, there are several functionalities available:

• Filter by any subset of notary activity, similar to the Landing Page.
• Access the Notary's Application, providing in-depth information about their proposed method of due diligence and datacap allocation.
• View a list of issues created by the Notary on the right side.

On the left side of the profile page, an expanded version of the notary report card is displayed, featuring additional statistics, such as the number of previous approvals and proposals. For a more in-depth understanding of each statistic, hover over the accompanying tooltip icon for more information.

The center of the profile page presents information on each client who has received datacap from the notary. Note that each unique client address is treated as a unique client, and some individuals may have multiple client addresses. The information per client address includes:

• Organization name and address of the client
• Issued datacap amount from the notary to the client
• Available datacap amount in the client address
• Type of allocation (LDN, DA, or EFIL+)
• Timestamp of the last allocation made by the notary to the client address
• By clicking the client name, you can access a more in-depth view of the allocations to that specific client.

Client Allocation Pop-Up

On the client allocation popup, you can view more in-depth information about the client and the tranches of datacap made to that client.

[liyunzhi-666]

Right now it seems to be ranked just by Average TTD as a reference, would consider adding other parameters (like Unique clients, DataCap allocated, etc.) for weight ranking?

[panges2]

Noted, we could implement a sort feature based on other parameters. Currently, we are not using this leaderboard to score or rank notaries in any way.

As I mentioned before, however, we may have a notary reputation/risk score in the future. If we do, metrics such as Average TTD alone would not be enough. We will likely aggregate many metrics and figure out a formula to calculate the score.

As a next step, we plan to add more metrics: notary engagement, notary score, TTD confidence.

[sxxfuture-official]

Should it reflect the number of notary governance meetings？

[panges2]

It should,s and it is one of the issues we are trying to fix. Right now, its N/A

[Aaron01230]

Will the number of meetings attended in history be displayed here?

[panges2]

As of now, just the meetings going forward