Advertisement
WP5 invites to a MediaFutures lunch seminar with Assistant Professor Jeremy Barnes, IXA Taldea / HiTZ Zentroa, University of the Basque Country (UPV/EHU).Tittel: Challenges for automatic summarization evaluation in low-resource languages
Abstract:
In this talk, I will describe a transferable methodology for determining how effective available metrics are at identifying quality differences in automatic news summary quality in Basque and Spanish. After collecting a metevaluation dataset (BASSE) to evaluate five criteria (coherence, consistency, fluency, relevance, and 5W1H), we benchmark how well automatic evaluation metrics and LLM judges are able to capture differences in these criteria. Our results suggest that proprietary LLM judge models currently give the most consistent results across criteria, while a careful combination of automatic metrics can give similar results.
Jeremy Barnes will come physically to our office. If there is interest, we will stream the seminar online.
Advertisement
Event Venue & Nearby Stays
Media City Bergen, Lars Hilles gate 30, Bergen, Norway
