Publications

Lessons from the Trenches on Reproducible Evaluation of Language Models featured image
Natural Language Processing Large Language Models Deep Learning Evaluation Reproducibility

Lessons from the Trenches on Reproducible Evaluation of Language Models

Effective evaluation of language models remains an open challenge in NLP. Researchers and engineers face methodological issues such as the sensitivity of models to evaluation …

Stella Biderman
,
Hailey Schoelkopf
,
Lintang Sutawika
,
Leo Gao
,
Jonathan Tow
,
Baber Abbasi
,
Alham Fikri Aji
,
Pawan Sasanka Ammanamanchi
,
Sidney Black
,
Jordan Clive
,
Anthony DiPofi
,
Julen Etxaniz
,
Benjamin Fattori
,
Jessica Zosa Forde
,
Charles Foster
,
Jeffrey Hsu
,
Mimansa Jaiswal
,
Wilson Y. Lee
,
Haonan Li
,
Charles Lovering
,
Niklas Muennighoff
,
Ellie Pavlick
,
Jason Phang
,
Aviya Skowron
,
Samson Tan
,
Xiangru Tang
,
Kevin A. Wang
,
Genta Indra Winata
,
François Yvon
,
Andy Zou
•
Irakurri gehiago
IKER-GAITU: research on language technology for Basque and other low-resource languages featured image
Natural Language Processing Large Language Models Deep Learning Multilinguality Basque

IKER-GAITU: research on language technology for Basque and other low-resource languages

The general objective of the IKER-GAITU project is to research on language technology to increase the presence of Basque in the digital environment. It will be carried out between …

Eneko Agirre
,
Itziar Aldabe
,
Xabier Arregi
,
Mikel Artetxe
,
Unai Atutxa
,
Ekhi Azurmendi
,
Iker De la Iglesia
,
Julen Etxaniz
,
Victor García-Romillo
,
Inma Hernaez-Rioja
,
others
•
PDF
Irakurri gehiago
XNLIeu: a dataset for cross-lingual NLI in Basque featured image
Natural Language Processing Large Language Models Deep Learning Multilinguality Basque

XNLIeu: a dataset for cross-lingual NLI in Basque

XNLI is a popular Natural Language Inference (NLI) benchmark widely used to evaluate cross-lingual Natural Language Understanding (NLU) capabilities across languages. In this …

Maite Heredia
,
Julen Etxaniz
,
Muitze Zulaika
,
Xabier Saralegi
,
Jeremy Barnes
,
Aitor Soroa
•
Irakurri gehiago
Latxa: An Open Language Model and Evaluation Suite for Basque featured image
Natural Language Processing Large Language Models Deep Learning Multilinguality Basque

Latxa: An Open Language Model and Evaluation Suite for Basque

We introduce Latxa, a family of large language models for Basque ranging from 7 to 70 billion parameters. Latxa is based on Llama 2, which we continue pretraining on a new Basque …

Julen Etxaniz
,
Oscar Sainz
,
Naiara Perez
,
Itziar Aldabe
,
German Rigau
,
Eneko Agirre
,
Aitor Ormazabal
,
Mikel Artetxe
,
Aitor Soroa
•
Irakurri gehiago
NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark featured image
Natural Language Processing Large Language Models Evaluation Data Contamination Deep Learning

NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark

In this position paper, we argue that the classical evaluation on Natural Language Processing (NLP) tasks using annotated benchmarks is in trouble. The worst kind of data …

Oscar Sainz
,
Jon Ander Campos
,
Iker García-Ferrero
,
Julen Etxaniz
,
Oier Lopez de Lacalle
,
Eneko Agirre
•
Irakurri gehiago
Do Multilingual Language Models Think Better in English? featured image
Natural Language Processing Large Language Models Deep Learning Multilinguality

Do Multilingual Language Models Think Better in English?

Translate-test is a popular technique to improve the performance of multilingual language models. This approach works by translating the input into English using an external …

Julen Etxaniz
,
Gorka Azkune
,
Aitor Soroa
,
Oier Lopez de Lacalle
,
Mikel Artetxe
•
Irakurri gehiago
Grounding Language Models for Compositional and Spatial Reasoning featured image
Artificial Intelligence Deep Learning Natural Language Processing Computer Vision Grounding Visual Reasoning Compositional Reasoning Spatial Reasoning

Grounding Language Models for Compositional and Spatial Reasoning

Humans can learn to understand and process the distribution of space, and one of the initial tasks of Artificial Intelligence has been to show machines the relationships between …

Julen Etxaniz
,
Oier Lopez de Lacalle
,
Aitor Soroa
•
Irakurri gehiago
ProMeta: softwarearen garapenerako prozesuen definizio eta ezarpenerako sistema metaereduetan oinarrituta featured image
Software Engineering Web Development

ProMeta: softwarearen garapenerako prozesuen definizio eta ezarpenerako sistema metaereduetan oinarrituta

The objective of the project is to build a system for the definition and implementation of software development processes based on metamodels. In fact, there are several …

Julen Etxaniz
•
Irakurri gehiago