A Tale about PRO and Monsters
Preslav Nakov, Francisco Guzman and Stephan Vogel
The 51st Annual Meeting of the Association for Computational Linguistics - Short Papers (ACL Short Papers 2013)
Sofia, Bulgaria, August 4-9, 2013
While exploring tuning on long sentences, we made an unexpected discovery: that PRO can fall victim to monsters – overly long negative examples with very low BLEU+1 scores, which are unsuitable for learning; as a result, testing BLEU drops by 3 points absolute. We propose several efficient ways to address the problem, using length- and BLEU+1-based cut-offs, outlier filters, stochastic sampling, and random acceptance. The best of these fixes not only slay and protect against monsters, but also yield higher stability for PRO as well as improved test-time BLEU scores. Thus, we recommend them to anybody using PRO, monster-believer or not.
Conference Manager (V2.61.0 - Rev. 2792M)