Graduate Research Study: NLP Reproducibility Project

Overview

Graduate students will conduct a research reproducibility and evaluation project focused on replicating results from an existing NLP research paper (ACL/NAACL 2025).
The project can be done individually or in pairs. Deliverables include a reproducibility report, source code, and a presentation.

Project Guidelines

• Research Paper Selection
  • Choose an ACL/NAACL 2025 NLP paper.
  • Submit details in the provided Excel sheet (first-come, first-served).
  • Must follow the NLP Reproducibility Checklist (Dodge et al., 2019).
• Research Study
  • Analyze the reproducibility of the selected paper.
  • Work individually or in pairs (pairs must submit a signed contribution statement).
  • Address methodology, implementation, and challenges.
  • Consider computational constraints and document adaptations.
  • Deliver the presentation, source code, and report by their respective deadlines.
• Recommendations for Limited Compute
  • Use smaller or open-source alternatives (e.g., Llama 2/3, Flan-T5 base).
  • Parameter-efficient fine-tuning encouraged.
  • Free compute resources (Google Colab, Kaggle) may be leveraged.
  • Report must describe compromises and impacts on reproducibility.

Deadlines & Milestones

• Paper Selection: September 22, 2025 (11:59 PM CST)
• Research Presentation:
  • Presentations: TBD
• Report Submission: TBD
• Source Code Submission: TBD