OpenFact at the CLEF 2024 conference
This was the 25th edition of the CLEF (Conference and Labs of the Evaluation Forum), a key event in the field of information and language technologies, focusing on the evaluation of information systems. Thanks to a variety of test tasks, participants can compare and improve their algorithms, which promotes progress in information science. Conference website: clef2024.imag.fr.
The works presented by our scientists concerned various methods that were developed in competitions in the area of information credibility, text verifiability assessment, multi-author writing style analysis and early risk prediction on the Internet. These methods can contribute to the development of tools for automatic information quality assessment and fake news detection.
In the competition on credibility of information, our scientists’ method took first place. The aim of the competition was to verify the robustness of popular text classification approaches used for credibility assessment problems. The task was to develop a method that would most effectively modify texts in such a way that different classification algorithms would change their decision to the opposite. These texts were related to 5 problem areas: assessing news bias, detecting propaganda, fact-checking, detecting rumors and COVID-19-related disinformation. Among all the teams that reported their results, our scientists’ method obtained the highest score, which took into account measures related to the level of effectiveness of changes in texts, semantic similarity and Levenshtein’s edit distance. The largest number of points allowed it to take first place in the ranking, ahead of methods developed by, among others, the University of Zurich (UZH). The winning method was developed by a team consisting of: Dr. Włodzimierz Lewoniewski, Dr. Piotr Stolarski, Dr. Milena Stróżyna, Dr. Elżbieta Lewańska, Aleksandra Wojewoda, Ewelina Księżniak, Marcin Sawiński. Details about the developed method can be found in the article entitled “OpenFact at CheckThat! 2024: Combining multiple attack methods for effective adversarial text generation”.
In a competition for assessing the verifiability of texts in different languages, our scientists took second place in the rankings – for English and Arabic. The aim of the competition was to detect check-worthy texts. The use of language models based on transformers and cross-linguistic transfer learning techniques allowed us to win two second places in the rankings. The competition was attended by 26 teams from centers around the world, including Queen Mary University of London, Sorbonne Université, University of Montréal, Airbus Defence and Space. Our team took second place in terms of the F1 value in text analysis. The F1 measure is a commonly used measure for evaluating machine learning models, especially in the case of class imbalance. The method was developed by the OpenFact team consisting of: Marcin Sawiński, Prof. Krzysztof Węcel and Ewelina Księżniak. Details about the developed method can be found in the article entitled “OpenFact at CheckThat! 2024: Cross-Lingual Transfer Learning for Check-Worthiness Detection“. It is worth mentioning that in 2023, the OpenFact project team took first place in the “CLEF-2023 CheckThat! Lab” competition – the best method for detecting sentences in English that require verification due to the possibility of misleading.
The OpenFact team also took part in a competition for the analysis of multi-author writing style. The competition task was to detect places where the author changes in a text written by several authors. The key issue was to check whether it is possible to recognize differences in writing style. The work resulted in an article entitled “Team OpenFact at PAN 2024: Fine-Tuning BERT Models with Stylometric Enhancements”, describing the approach used. Authors of the publication: Ewelina Księżniak, Prof. Krzysztof Węcel, Marcin Sawiński.
The Department of Information Systems is currently implementing the OpenFact research project, headed by Prof. Witold Abramowicz. As part of this project, tools for automatic detection of fake news in Polish are being developed. In July 2024, the results of the OpenFact project were assessed by National Center for Research and Development as the best in Poland for the second year in a row. Winning prestigious competitions confirms that the achievements of our team are significant on a global scale and that the methods developed by the OpenFact team achieve equally high effectiveness in other languages.
The OpenFact project is financed by the National Center for Research and Development under the INFOSTRATEG I program “Advanced information, telecommunications and mechatronic technologies”.
Outside the OpenFact project, scientists from the Department of Information Systems also achieved the best result in the competition in the field of early risk prediction on the Internet. The aim of the competition was to develop a system for early recognition of people with anorexia based on their activity on social media platforms. Among the numerous teams that submitted their solutions, the method developed by our scientists achieved the best result according to the ERDE (Early Risk Detection Error) metric. This metric takes into account both the precision of the results and the time needed to alert about the possibility that a given person may suffer from anorexia. This aspect was a key element of the competition and has a real impact when using early warning systems. Authors of the method: Oskar Riewe-Perła and Prof. Agata Filipowska. The author’s solution is described in the article entitled “Combining Recommender Systems and Language Models in Early Detection of Signs of Anorexia”.