Of more than 300 papers on learning from demonstration published in 2024, about 20% offered highly notable contributions. The rest were incremental improvements on existing methods or applications of them in new domains. That is the finding of a year-long reading effort by five senior robotics researchers: Aude Billard, Renaud Detry, Nadia Figueroa, Dongheui Lee and Kunpeng Yao. Maximilian Foriest supplied scripts for the language-model analysis.
The scale of the problem
Learning from demonstration, or LfD, covers methods in which human experts teach robots. The authors chose it as a single subarea because reading the whole robotics literature is not feasible. In 2024 alone, IEEE published 46,968 papers on robotics or automation, and the authors note that IEEE accounts for only a fraction of what is available online.
Submissions to robotics journals have grown each year over the past decade. Growth reached 26% in 2023 and 31% in 2024. The authors attribute this to several factors, including rising public and private interest and the availability of AI tools that help write papers and code. Published papers have followed the same curve, even though editorial boards have lowered acceptance rates to contain it.
Conferences show the same pattern. ICRA doubled the number of papers it published over ten years and reached approximately 1,800 in 2024. The time between submission and publication has fallen by 50%, which the authors link to community pressure to publish quickly. They say IJRR, RSS and CoRL follow the same trend.
What the reading showed
Five senior authors read the papers themselves and carried out both qualitative and quantitative analysis. They found that notable contributions did not necessarily correlate with higher download or citation counts. A paper's visibility, in other words, was not a reliable signal of its value.
The authors argue that locating the strongest work matters for two reasons. It reduces the risk that novel results go unnoticed, and it reduces duplicated effort.
How well automated tools performed
The team also tested scripts and large language models on the same task. These tools worked fairly faithfully for general quantitative assessment, such as collecting precise numbers from a paper. They also summarised individual works well.
They failed at judging importance. The authors report that the tools cannot recognise a paper that revisits a problem already solved. They also miss cases where an abstract or a stated claim overstates the contribution actually reported in the body of the paper.
Recommendations and analysis
The authors make three recommendations. First, build a research engine that restores the weight of editorial and peer-review work, ranking papers by evaluation scores and peer-reviewed status. Google Scholar and IEEE Xplore, in their account, put all publications on equal footing regardless of review or venue reputation. Second, consider a blind publication model and topic-based social media posting, so that author names and institutions become secondary to content. Third, use language models for what they do well, namely summarising and extracting quantitative data, while recognising that a tool able to judge novelty could one day erode the expertise it imitates.
The study matters because it measures a problem most researchers describe only anecdotally. If roughly one paper in five carries the main contribution, then a reader who samples by citations or downloads is working with a filter that, by the authors' own evidence, does not track quality. That is a concrete argument for ranking by peer-review signals, and it fits the finding that current search tools ignore them.
Several limits should be kept in view. The data cover one subarea and one year, so the 20% figure cannot be assumed to hold for other fields of robotics or for other years. The judgement of what counts as highly notable came from the five senior authors, who designed the study and did the reading. It is expert opinion, not an independent benchmark, and the source does not describe how disagreements were handled. The comparison between humans and language models was also run on a single set of tools, so it does not show that no future model could assess novelty.
The proposals for a peer-review-weighted search engine and for blind publication are untested. The source offers no evidence yet that either would improve how researchers find strong work. What the study does establish is a baseline: in one well-defined area, careful human reading found a small share of standout papers, and automated tools could not reliably find them.