Sign In

Communications of the ACM

ACM News

Furious AI Researcher Creates a List of Non-reproducible Machine Learning Papers


View as: Print Mobile App Share: Send by email Share on reddit Share on StumbleUpon Share on Hacker News Share on Tweeter Share on Facebook
A frustrated researcher.

A researcher's frustration with reproducing the results of a machine learning research paper was the basis for creating "Papers Without Code," a website that aims to create a centralized list of machine learning papers that are not implementable.

Credit: The Next Web

On February 14, a researcher who was frustrated with reproducing the results of a machine learning research paper opened up a Reddit account under the username ContributionSecure14 and posted the r/MachineLearning subreddit: "I just spent a week implementing a paper as a baseline and failed to reproduce the results. I realized today after googling for a bit that a few others were also unable to reproduce the results. Is there a list of such papers? It will save people a lot of time and effort."

The post struck a nerve with other users on r/MachineLearning, which is the largest Reddit community for machine learning.

"Easier to compile a list of reproducible ones…," one user responded.

"Probably 50%-75% of all papers are unreproducible. It's sad, but it's true," another user wrote. "Think about it, most papers are 'optimized' to get into a conference. More often than not the authors know that a paper they're trying to get into a conference isn't very good! So they don't have to worry about reproducibility because nobody will try to reproduce them."

A few other users posted links to machine learning papers they had failed to implement and voiced their frustration with code implementation not being a requirement in ML conferences.

The next day, ContributionSecure14 created "Papers Without Code," a website that aims to create a centralized list of machine learning papers that are not implementable.

"I'm not sure if this is the best or worst idea ever but I figured it would be useful to collect a list of papers which people have tried to reproduce and failed," ContributionSecure14 wrote on r/MachineLearning. "This will give the authors a chance to either release their code, provide pointers or rescind the paper. My hope is that this incentivizes a healthier ML research culture around not publishing unreproducible work."

 

From The Next Web
View Full Article

 


 

No entries found