AI models face challenges in understanding cultural nuances

A new study by Cohere Labs highlights the difficulties AI chatbots face in grasping cultural context. Researchers tested 28 AI models with cultural riddles, revealing significant gaps in their understanding. One example was the question, 'What’s the German 9/11?' Only one model correctly identified it as November 9, 1989, the day the Berlin Wall fell, while others misinterpreted the date format. Julia Kreutzer, a senior research scientist at Cohere Labs, emphasized the need for AI to grasp nuanced cultural concepts rather than just memorizing facts. The team developed the 'CulturalRiddles' benchmark, which evaluates models based on their ability to connect cultural knowledge. The benchmark involved volunteers from 45 countries, contributing riddles in 51 languages. The study tested 28 AI models, with Google’s Gemini 3.1 Pro Preview achieving the highest score at 79.4%, followed by OpenAI’s GPT-5.5 at 76.3% and Anthropic’s Claude Opus 4.7 Thinking at 66%. The researchers used volunteer evaluations to refine the AI models and assess answer quality.