Abstract
Following the fourth edition of the workshop on Reproducible Research in Pattern Recognition (RRPR) at the International Conference on Pattern Recognition (ICPR), this paper reports the main discussions that were held during and after the workshop. In particular, the integration of reproducible research inside an international conference was the first main axis of reflection. Further discussions addressed the ways of initiating or imposing reproducible research, as well as the problem of performance comparisons of published research papers that emerges due to the fact that the reported results are often based on different implementations and datasets.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
- 2.
Source: https://github.com/lixin4ever/Conference-Acceptance-Rate (accessed on 2 April 2023).
- 3.
However, for GPT-4 [19], OpenAI published the evaluation code, which makes comparison with their claimed results easy.
References
Colom, M., Kerautret, B., Krähenbühl, A.: An Overview of Platforms for Reproducible Research and Augmented Publications. In: Kerautret, B., Colom, M., Lopresti, D., Monasse, P., Talbot, H. (eds.) RRPR 2018. LNCS, vol. 11455, pp. 25–39. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-23987-9_2
Artifact review and badging, 2020. Revised August 24. https://www.acm.org/publications/policies/artifact-review-and-badging-current. Accessed October 14
Raff, E.: A step toward quantifying independently reproducible machine learning research. In: Advances in Neural Information Processing Systems. Curran Associates Inc, (2019)
NeurIPS 2022 Paper Checklist Guidelines. https://neurips.cc/Conferences/2022/PaperInformation/PaperChecklist (accessed in 26 February 2023)
Pineau, J.: Improving Reproducibility in Machine Learning Research (a Report from the NeurIPS 2019 Reproducibility Program). J. Mach. Learn. Res., 22(1) (2022). Publisher: JMLR.org
The Machine Learning Reproducibility Checklist. https://www.cs.mcgill.ca/ jpineau/ReproducibilityChecklist.pdf. Accessed 13 Mar 2023
Strubell, E., Ganesh, A., McCallum, A.: Energy and policy considerations for deep learning in NLP. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 3645–3650 (2019)
ML Reproducibility Challenge 2022, 2022. https://paperswithcode.com/rc2022,. Accessed 4 Mar 2023
ICMR reproducibility, 2023. https://icmr-reproducibility.github.io/website/cfp2023/, Accessed 4 Mar 2023
Arévalo, M., Escobar, C., Monasse, P., Monzón, N., Colom, M.: The IPOL Demo System: A Scalable Architecture of Microservices for Reproducible Research. In: Kerautret, B., Colom, M., Monasse, P. (eds.) RRPR 2016. LNCS, vol. 10214, pp. 3–16. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-56414-2_1
IPOL demo system development. https://github.com/ipol-journal/ipolDevel. Accessed 26 Feb 2023
Call for demonstrations of the IJCAI international conference. https://github.com/ipol-journal/ipolDevelhttps://ijcai-23.org/call-for-demos. Accessed 1 Apr 2023
Rougier, N.P., Hinsen, K.: ReScience C: A Journal for Reproducible Replications in Computational Science. In: Kerautret, B., Colom, M., Lopresti, D., Monasse, P., Talbot, H. (eds.) RRPR 2018. LNCS, vol. 11455, pp. 150–156. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-23987-9_14
Colom, M., Kerautret, B., Limare, N., Monasse, P., and Jean-Michel Morel. IPOL: a new journal for fully reproducible research; analysis of four years development. In: Badra, M., Boukerche, A.,mUrien, P., eds 7th International Conference on New Technologies, Mobility and Security, NTMS 2015, Paris, France, July 27–29, 2015, pp. 1–5. IEEE (2015)
Johnson, A., Bulgarelli, L.: Tom Pollard. Leo Anthony Celi, and Roger Mark. MIMIC-IV, Steven Horng (2021)
Bommasani, R., et al.: On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258 (2021)
Fredrikson, M., Jha, S., Ristenpart, T.: Model inversion attacks that exploit confidence information and basic countermeasures. In: Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security, CCS ’15, pp. 1322–1333, New York, NY, USA (2015). Association for Computing Machinery
Brown, T., et al.: Language models are few-shot learners. Adv. Neural Inf. Process. Syst. 33, 1877–1901 (2020)
OpenAI. GPT-4 Technical Report. arXiv preprint arXiv:2303.08774 (2023)
van Dis, E.A.M., Bollen, J., Zuidema, W., van Rooij, R., Bockting, C.L.: ChatGPT: five priorities for research. Nature 614(7947), 224–226 (2023)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Kerautret, B., Kirchheim, K., Lopresti, D., Ngo, P., Tomaszewska, P. (2023). Promoting Reproducibility of Research Results in International Events (Report from the \(4^{th}\) RRPR). In: Kerautret, B., Colom, M., Krähenbühl, A., Lopresti, D., Monasse, P., Perret, B. (eds) Reproducible Research in Pattern Recognition. RRPR 2022. Lecture Notes in Computer Science, vol 14068. Springer, Cham. https://doi.org/10.1007/978-3-031-40773-4_9
Download citation
DOI: https://doi.org/10.1007/978-3-031-40773-4_9
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-40772-7
Online ISBN: 978-3-031-40773-4
eBook Packages: Computer ScienceComputer Science (R0)