publications

2024

1. ReproHum #0712-01: Human Evaluation Reproduction Report for “Hierarchical Sketch Induction for Paraphrase Generation”

   Mohammad Arvan, and Natalie Parde

   In Proceedings of the Fourth Workshop on Human Evaluation of NLP Systems (HumEval) @ LREC-COLING 2024, May 2024

   Abs

   Human evaluations are indispensable in the development of NLP systems because they provide direct insights into how effectively these systems meet real-world needs and expectations. Ensuring the reproducibility of these evaluations is vital for maintaining credibility in natural language processing research. This paper presents our reproduction of the human evaluation experiments conducted by Hosking et al. (2022) for their paraphrase generation approach. Through careful replication we found that our results closely align with those in the original study, indicating a high degree of reproducibility.

2. Machine Learning and Open Science: On Risks and Challenges

   Mohammad Arvan

   In , Jul 2024

   PDF

2023

1. Missing Information, Unresponsive Authors, Experimental Flaws: The Impossibility of Assessing the Reproducibility of Previous Human Evaluations in NLP

   Anya Belz, Craig Thomson, Ehud Reiter, and 36 more authors

   In The Fourth Workshop on Insights from Negative Results in NLP, May 2023
2. Investigating Reproducibility at Interspeech Conferences: A Longitudinal and Comparative Perspective

   Mohammad Arvan, A. Seza Doğruöz, and Natalie Parde

   In Proc. INTERSPEECH 2023, May 2023
3. Human Evaluation Reproduction Report for Data-to-text Generation with Macro Planning

   Mohammad Arvan, and Natalie Parde

   In Proceedings of the 3rd Workshop on Human Evaluation of NLP Systems, May 2023
4. Exploring Variation of Results from Different Experimental Conditions

   Maja Popovic, Mohammad Arvan, Natalie Parde, and 1 more author

   In Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023, May 2023
5. Linguistic Cognitive Load Analysis on Dialogues with an Intelligent Virtual Assistant

   Mohammad Arvan, Mina Valizadeh, Parian Haghighat, and 3 more authors

   In Proceedings of the Annual Meeting of the Cognitive Science Society, May 2023
6. Effects of an intelligent virtual assistant on office task performance and workload in a noisy environment

   Parian Haghighat, Toan Nguyen, Mina Valizadeh, and 4 more authors

   Applied ergonomics, May 2023

2022

1. Reproducibility of Exploring Neural Text Simplification Models: A Review

   Mohammad Arvan, Luı́s Pina, and Natalie Parde

   In Proceedings of the 15th International Conference on Natural Language Generation: Generation Challenges, Jul 2022

   Abs PDF

   The reproducibility of NLP research has drawn increased attention over the last few years. Several tools, guidelines, and metrics have been introduced to address concerns in regard to this problem; however, much work still remains to ensure widespread adoption of effective reproducibility standards. In this work, we review the reproducibility of Exploring Neural Text Simplification Models by Nisioi et al. (2017), evaluating it from three main aspects: data, software artifacts, and automatic evaluations. We discuss the challenges and issues we faced during this process. Furthermore, we explore the adequacy of current reproducibility standards. Our code, trained models, and a docker container of the environment used for training and evaluation are made publicly available.

2. Reproducibility in Computational Linguistics: Is Source Code Enough?

   Mohammad Arvan, Luı́s Pina, and Natalie Parde

   In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, Jul 2022
3. Human Interaction with Intelligent Virtual Assistant in a Noisy Environment

   Parian Haghighat, Toan Nguyen, Mina Valizadeh, and 4 more authors

   In Proceedings of the Human Factors and Ergonomics Society Annual Meeting, Jul 2022

2016

1. RoboCup rescue 2016 team description paper MRL

   Farshid Najafi, Mehdi Dadvar, Soheil Habibian, and 5 more authors

   Robocup Rescue 2016 TDP Collection, Jul 2016