Submission Claims and Payouts in V2
Published on May 30, 2023,
Last updated on May 30, 2023,
Metrics App V2 release estimated: In the coming weeks
At MetricsDAO, we have always emphasized that at the center of what we do is our analyst community. When it comes to challenge submissions and analytical outputs, clarity in thought and purpose is one of the main criteria we have. Given these, it is important for us to do right by our analysts by giving more clarity and certainty to a core functionality in the Metrics App: challenge payments. To address this, we are introducing new submissions and payment systems for challenges in the upcoming V2 upgrade to the Metrics App.
Initial System
In Metrics App V1, submissions to challenges are unlimited and the payment values are uncapped up to the value of the total rewards pool. The rewards are distributed based on the overall submission score; the higher the score, the higher the reward. This has the benefit of maximizing the distribution of rewards across all analysts as long as the submission is deemed valid (i.e. has a score). Analysts are also incentivized to give the best possible output in order to receive the highest reward amongst other analysts.
However, the negative consequence of this is the dilution of rewards as more valid submissions come in since the whole rewards pool has to be distributed across all submissions. There is also a dilution of quality as any submission with a non-zero score will get rewarded. At the other end of the spectrum, this also allows a single analyst to win the whole reward pool if they are the only one that submits an output regardless of their score as long as it is above zero. Both scenarios can potentially lead to bad outcomes for our partners/challenge sponsors as they will see their funds either get sent to a multitude of subpar submissions or a single subpar submission.
New Rewards System
With Metrics App V2, we are closely linking the quality of submissions with their corresponding rewards by introducing submission limits and rewards pool bucketing.
Submission Limits
Challenge sponsors will now be able to set a cap on the number of submissions per challenge. With this cap, the maximum reward amount per submission is also set. For example, a challenge that has a reward pool of USD 10,000 and has a submission limit of 10 will have a maximum reward per submission of USD 1,000.
Max Reward = Reward Pool / Submission Limit
| Reward Pool | Submission Limit | Max Reward |
|---|---|---|
USD 10,000 | 10 | USD 1,000 |
Analysts can claim the right to submit for these challenges on a first come, first served basis. It goes without saying that after the set submission limit has been reached, no further submissions will be accepted.
Rewards Pool Bucketing
The second part of the change introduces the modulation of maximum rewards by setting rewards buckets. Challenges will now have tiers according to submission grade ranges. These grade ranges will represent a proportion of the maximum reward that a submission can get if its score falls within that tier. To ensure certainty, the ranges used for specific challenges will always be visible to analysts before they claim the right to submit.
Continuing the above example, we can set five different grade ranges with five different reward proportions.
Rewards Earned = Reward Proportion based on Grade Range * Max Reward
| Grade Range | Reward Proportion | Max Reward | Rewards Earned |
|---|---|---|---|
90 and above | 100% | USD 1,000 | USD 1,000 |
75 to 89 | 75% | USD 1,000 | USD 750 |
50 to 74 | 50% | USD 1,000 | USD 500 |
25 to 49 | 25% | USD 1,000 | USD 250 |
0 to 24 | 0% | USD 1,000 | USD 0 |
The grade ranges and reward proportions can also be set to simulate a “pass or fail” environment.
| Grade Range | Reward Proportion | Max Reward | Rewards Earned |
|---|---|---|---|
70 and above | 100% | USD 1,000 | USD 1,000 |
0 to 69 | 0% | USD 1,000 | USD 0 |
Benefits and Problems Addressed
With these changes to the submission and rewards systems, analysts can now have certainty on how much rewards they can get based on their submission grade. These payments are not dependent anymore on the number of submissions that get submitted by other analysts.
With the submission limits, our partners can also have more control over the use of their budget in order to maximize the value they can get from the analytics submissions. Any unused portion of the rewards pool can be returned to the challenge sponsor which they can use to fund future challenges, allowing them to further maximize their utility.
We can simulate a scenario where we can observe these two benefits.
Rewards and Submission Limit Settings
| Reward Pool | Submission Limit | Max Reward |
|---|---|---|
USD 10,000 | 10 | USD 1,000 |
Grade Range and Reward Proportion Settings
| Grade Range | Reward Proportion | Max Reward | Rewards Earned |
|---|---|---|---|
90 and above | 100% | USD 1,000 | USD 1,000 |
75 to 89 | 75% | USD 1,000 | USD 750 |
50 to 74 | 50% | USD 1,000 | USD 500 |
25 to 49 | 25% | USD 1,000 | USD 250 |
0 to 24 | 0% | USD 1,000 | USD 0 |
Actual Submissions and Grades
| Submission | Score | Rewards Earned |
|---|---|---|
1 | 93 | USD 1000 |
2 | 91.25 | USD 1000 |
3 | 92.4 | USD 1000 |
4 | 90 | USD 1000 |
5 | 90 | USD 1000 |
6 | 71.25 | USD 500 |
7 | 70.50 | USD 500 |
8 | 31.75 | USD 250 |
9 | 40 | USD 250 |
10 | 0 | USD 0 |
| Rewards Pool | Total Rewards Disbursed | Remainder (Reclaimable) |
|---|---|---|
USD 10,000 | USD 6,500 | USD 3,500 |
As you can see, analysts now have a better idea of what their payouts are regardless of the number of submissions as soon as they receive their scores. Our partners also get the benefit of paying only for the quality they expect and receive back any surplus from the rewards pool.
The set limits on submissions remove the incentive for subpar outputs as each submission spot becomes more valuable. In order to maximize the rewards, analysts are incentivized to give their best for the highest score possible.
With the removal of the “winner take all” aspect of the rewards from V1, this design also has the potential to foster a more collaborative environment amongst analysts since their submission outputs do not affect another analyst’s payout anymore. This is thanks to the certainty in payment outputs introduced by the grade ranges and their corresponding rewards proportions.
Summary of Changes
| Payments in Metrics App V1 | Payments in Metrics App V2 | |
|---|---|---|
Submissions | Unlimited | Limited |
Rewards Pool | Maximized and distributed to all submissions | Proportioned according to grade range |
Benefits | - All submissions rewarded - Encourages submission volume | - Encourages submission quality - Makes payout more certain - Increases efficiency in rewards usage - Sponsors can re-use the surplus from rewards pools across multiple challenges - Allows for a more collaborative environment - Certainty in use of budget |
Problems | - Can produce a high number of subpar submissions - Allows for “winner take all” situations regardless of submission quality - Unpredictable use of budget | - Rewards pool not fully maximized for analysts payouts in single challenges
|
Regarding timelines, version 2 of the protocol has already been written and tested. Our application and implementation team is exposing the functionalities of the protocol and we can expect some changes in the coming weeks, including cross-chain payments and more specific permissioning around challenge participation.
Our goal with these changes is to introduce more certainty in the reward payment system for our analysts. We also want to help our partners maximize the value they get from their resources. All of these go towards ensuring all participants get the most out of the Metrics App’s labor marketplace.
If you want to dive deeper into the technical aspects of these changes, you can watch the community call here. If you have further questions or suggestions around things like the implementation of grade ranges and their respective proportions, feel free to reach out in Discord in the # 🛠│app-general channel.
Last updated
