Reducing the Influence of Content on Syllogistic Reasoning in Large Language Models

The research was conducted as part of SemEval-2026 Task 11, which focused on disentangling the influence of semantic content from the formal logical validity of arguments in language model reasoning. The study addressed a fundamental question in contemporary artificial intelligence: can language models truly reason logically, or are their judgments influenced by their knowledge of the world and the semantic plausibility of the arguments presented to them?

The method proposed by Ewelina Księżniak involved progressively replacing content-bearing words with abstract symbols and artificially generated pseudo-words while preserving the logical structure of the arguments. Experiments showed that this approach can reduce the influence of semantic content on model predictions and improve reasoning accuracy. An additional analysis of the model’s internal representations provided insights into how information about the semantic plausibility of arguments is encoded across different layers of the model.

The submitted system ranked 12th on the official competition leaderboard, with the top 11 systems achieving identical scores. The findings highlight both the potential for reducing content-driven biases in language models and the challenges associated with the stability and reliability of their evaluation. The full paper is available through ACL Anthology. DOI: 10.18653/v1/2026.semeval-1.272

ACL 2026 (the Annual Meeting of the Association for Computational Linguistics) took place on July 2–7, 2026, in San Diego, USA. It is one of the world’s leading international scientific conferences in natural language processing and computational linguistics, bringing together researchers from around the globe.