Dear Challenger,
Thank you all for the interest and engagement we’ve seen over the past three months. We have an important update to share. We've identified an issue in one of our scoring metrics that will affect the current leaderboard. We want to be fully transparent about what happened and how we're addressing it. This one is on us and we wholeheartedly apologize for the disruption caused. It was important to us that we rectify the issue promptly and well in advance of the final test set / submission deadline. We’re hosting a Discord Q&A on Friday to support anyone with questions.
What’s the Issue?
The Discrimination Score (PDS) code in cell-eval and used by the leaderboard does not precisely match our described methodology nor intent. Specifically, we inadvertently added an absolute value operation when comparing perturbation to control expression prior to the L1-distance calculation This means the PDS metric was only scoring on the magnitude of perturbations, and not including the direction as we intended.
How Did The Issue Get There?
As we considered metrics for the Challenge, we had initially focused solely on magnitude of effect to compare the similarity of predictions to the true effect. When we decided on the final formulation, we accidentally carried over an absolute value function in the scoring algorithm. As a result our scoring code did not match our described methodology or the methods section in our STATE model paper. It also made the metric less biologically meaningful.
What’s the Fix?
We're resolving the issue in cell-eval today and you can find the updated code here. We have also updated our blog post to reflect this change. In other places, such as on the website and the STATE model paper, the scoring is described accurately and stays the same.
How is the Challenge Changing?
Leaderboard Freeze and Reset Tomorrow
We are freezing the leaderboard at 1 pm PT (8 pm UTC) today. Submitting results to the Challenge will be disabled for 24 hours while we make updates to the scoring system. If you are in the middle of a submission when we implement this freeze, it will show up on the leaderboard after we re-enable submissions.
Tomorrow, Thursday, October 2 at 1 pm PT (8 pm UTC), we will update the leaderboard with recalculated scores for all current entries. Note that the updated PDS calculation is more challenging since it includes directional change and we expect there to be significant shifts in the current leaderboard rankings. Once the leaderboard is updated, you will be able to begin submitting results again and see your entries scored using the corrected code.
Later Test Set; Later Final Entries
Given this change, we want to give you all additional time to optimize your models. We will release the final test set two weeks later than originally scheduled, on November 10 rather than October 27. Final entries will now be due November 17 rather than November 3. Winners will still be announced in early December. Our website timeline and rules have been updated to reflect these new dates.
Our Thinking, and an Apology
We have carefully considered the consequences of this correction to you, the participants in the Challenge. We understand this causes disruption. We know many entrants were tuning their models to the performance measured by cell-eval. We concluded, though, that it is critical to have the scoring code match what we describe it to be doing, and match what we believe to be a more biologically meaningful metric. We hope the timing extensions will give everyone a chance to adjust to this correction. But we know we have created a bumpy experience in this first version of the Challenge, and we sincerely apologize for that.
Discord Q&A with the Challenge Organizers on October 3
For questions about the update above or anything else related to the Challenge, our organizers will be available on Discord this Friday, October 3 between 11 am and 12 pm PT (6 pm to 7 pm UTC), to provide quick answers. You can start populating your questions on a dedicated Thread in the #general channel in case you won’t be available during the session. We’ll be sure to add some of the exchanges to the FAQ page on our website, where we’ve been capturing many questions and responses throughout the competition.
We are very grateful for your participation in this first year of the Challenge and we look forward to your continued involvement. Community input is an important part of the Challenge and we want to continue to hear from you all about the best ways to craft this event to advance the field of virtual cell modeling in future years.
We’ll share an update in late October on what to expect during the final weeks of the competition. Please stay tuned and continue to connect with us via help@virtualcellchallenge.org.
- The Virtual Cell Challenge Team
