This benchmark evaluates large language models (LLMs) using 759 NYT Connections puzzles, with additional words included to increase difficulty.

As of Feb 4, 2025, there is a new version of the benchmark. The standard NYT Connections benchmark is nearing saturation, with o1 scoring 90.7 and o3, along with other reasoning models, expected this year. The current rules require knowing only three categories, letting the fourth fall into place. To increase difficulty, Extended Connections adds up to four extra trick words to each puzzle. We double-check that none of the added words fit into any category used in the corresponding puzzle. New puzzles have expanded the total from 436 to 651 and then to 759 on July 14, 2025. Rankings changed little, but the benchmark is now ready for o3.

To counteract the possibility of an LLM's training data including the solutions, we have also tested only the 100 latest puzzles. Note that lower scores do not necessarily indicate that NYT Connections solutions are in the training data, as the difficulty of the first puzzles was lower.

To explore how top language models (LLMs) compare to humans in the New York Times Connections puzzle, we used official NYT performance data from December 2024 to February 2025, as analyzed by u/Bryschien1996, alongside a simulated gameplay setup that mirrors the human experience. This setup involves a multi-step process where solvers iteratively propose groups, receive feedback ("correct," "one away," "incorrect"), and are allowed up to four mistakes before failing. According to NYT data, the average human player solved approximately 71% of puzzles over the three-month period from December 2024 to February 2025, with solve rates ranging from 39% on the toughest days (e.g., February 2, 2025) to 98% on the easiest (e.g., February 26, 2025). It's worth noting that NYT Connections players are self-selected and likely perform better than the general population. We collected data from nine LLMs spanning a range of scores in the Extended Connections benchmark.

The results reveal that top reasoning LLMs from OpenAI consistently outperform the average human player. DeepSeek R1 performs closest to the level of an average NYT Connections player.

Elite human players, however, set a higher standard, achieving a 100% win rate during the same period:

o1, with a 98.9% win rate, comes close to this elite level. o1-pro, which has not yet been tested in this gameplay simulation setup, might be able to match these top humans. Thus, directly determining whether AI achieves superhuman performance on NYT Connections could hinge on comparing the number of mistakes made before fully 89E1 solving each puzzle.

This benchmark evaluates large language models (LLMs) using 436 NYT Connections puzzles. Three different prompts, not optimized for LLMs through prompt engineering, are used. Both uppercase and lowercase puzzles are assessed.

• A temperature setting of 0 was used
• Partial credit is awarded if the puzzle isn't completely solved.
• Only one attempt is allowed per puzzle. Humans solving puzzles on the NYT website get four attempts and a notification when they're one step away from the solution.
• Multi-turn ensemble is my unpublished system. It utilizes multiple LLMs, multi-turn dialogues, and other proprietary techniques. It is slower and more costly to run but it does very well. It outperforms non-o1 LLMs on MMLU-Pro and GPQA.
• This benchmark is not affiliated with the New York Times

• PACT - Benchmarking LLM negotiation skill in multi-round buyer-seller bargaining
• BAZAAR - Evaluating LLMs in Economic Decision-Making within a Competitive Simulated Market
• Public Goods Game (PGG) Benchmark: Contribute & Punish
• Elimination Game: Social Reasoning and Deception in Multi-Agent LLMs
• Step Race: Collaboration vs. Misdirection Under Pressure

• LLM Thematic Generalization Benchmark
• LLM Creative Story-Writing Benchmark
• LLM Round‑Trip Translation Benchmark
• Mapping LLM Style and Range in Flash Fiction
• LLM Confabulation/Hallucination Benchmark
• LLM Deceptiveness and Gullibility
• LLM Divergent Thinking Creativity Benchmark

• Oct 15, 2025: Claude Haiku 4.5 added.
• Oct 14, 2025: Claude Sonnet 4.5, Deepseek V3.2 Exp, GLM-4.6 added.
• Sep 19, 2025: Grok 4 Fast, Qwen 3 Next 80B A3B Thinking, LongCat Flash Chat added.
• Sep 6, 2025: Kimi K2-0905 added.
• Sep 5, 2025: Qwen 3 Max Preview, Qwen 3 235B A22B 25-07 Instruct added.
• Aug 23, 2025: GPT-5 high reasoning and Cohere Command A Reasoning (16K) added.
• Aug 22, 2025: DeepSeek 3.1, Qwen 3 30B A3B 25-07, Mistral Medium 3.1, GPT-5 minimal and low reasoning added.
• Aug 7, 2025: GPT-5 added.
• Aug 5, 2025: Claude Opus 4.1, GPT-OSS-120B added.
• July 28, 2025: GLM-4.5, Qwen 3 235B A22B 25-07 Thinking added.
• July 14, 2025: 108 new puzzles added. Kimi K2 added.
• July 10, 2025: Grok 4 added.
• July 3, 2025: Qwen 3 32B, GLM4-32B-0414 added.
• July 2, 2025: Baidu Ernie 4.5 300B A47B, MiniMax-M1, Mistral Small 3.2 added.
• June 10, 2025: o3-pro added.
• June 5, 2025: Gemini 2.5 Pro Preview 06-05 added.
• May 28, 2025: DeepSeek R1 05/28 added.
• May 22, 2025: Claude 4 models added.
• May 7, 2025: Gemini 2.5 Pro Preview 05-06 added. Mistral Medium 3 added.
• Apr 30, 2025: Qwen 3 added.
• Apr 18, 2025: o3, o4-mini, Gemini 2.5 Flash Preview added.
• Apr 15, 2025: GPT-4.1 added.
• Apr 10, 2025: Grok 3 added.
• Apr 5, 2025: Llama 4 Maverick, Llama 4 Scout added.
• Mar 28, 2025: GPT-4o March 2025 added.
• Mar 25, 2025: 50 new questions added. Gemini 2.5 Pro Exp 03-25 and DeepSeek V3-0324 added.
• Mar 23, 2025: Humans vs. LLMs section added.
• Mar 21, 2025: o1-pro added. o3-mini-high added.
• Mar 17, 2025: Cohere Command A and Mistral Small 3.1 added.
• Mar 12, 2025: Gemma 3 27B added.
• Mar 7, 2025: Qwen QwQ added.
• Feb 27, 2025: GPT-4.5 Preview added.
• Feb 24, 2025: Claude 3.7 Sonnet Thinking, Clade 3.7 Sonnet, GPT-4o Feb 2025, Qwen 2.5 Max, GPT-4o 2024-11-20 added.
• Feb 6, 2025: Gemini 2.0 Pro Exp 02-05 added.
• Feb 4, 2025: A new, more challenging version with extra words in each puzzle. Separate scoring for the 100 newest questions. Correlation heatmap.
• Jan 31, 2025: o3-mini (72.4) added.
• Jan 30, 2025: Mistral Small 3 (10.5) added.
• Jan 29, 2025: DeepSeek R1 (54.5) added.
• Jan 28, 2025: Qwen 2.5 Max (18.6) added.
• Jan 22, 2025: Phi-4 (11.6), Nova Pro (12.5), Gemini 2.0 Flash Thinking Exp 01-21 (37.0) added.
• Jan 16, 2025: Gemini 2.0 Flash Thinking Exp, o1, MiniMax-Tex-o1 added. Gemini 2.0 Flash Thinking Exp sometimes hits the output token limit.
• Dec 27, 2024: GPT-4o 2024-11-20, Llama 3.3 70B, Gemini 2.0 Flash Exp, Deepseek-V3 added. Gemini 2.0 Flash Thinking Exp could not be benchmarked because its output gets cut off for some puzzles.
• Claude 3.5 Haiku added. 13.7.
• Claude 3.5 Sonnet (2024-10-22) added. Improves from 25.9 from 24.4.
• Grok Beta added. Improves from 21.3 to 23.7. It's described as "experimental language model with state-of-the-art reasoning capabilities, best for complex and multi-step use cases. It is the successor of Grok 2 with enhanced context length."
• Follow @lechmazur on X (Twitter) for other upcoming benchmarks and more.