CipherBank Benchmark

Benchmark description

CipherBank, a comprehensive benchmark designed to evaluate the reasoning capabilities of LLMs in cryptographic decryption tasks. CipherBank comprises 2,358 meticulously crafted problems, covering 262 unique plaintexts across 5 domains and 14 subdomains, with a focus on privacy-sensitive and real-world scenarios that necessitate encryption. From a cryptographic perspective, CipherBank incorporates 3 major categories of encryption methods, spanning 9 distinct algorithms, ranging from classical ciphers to custom cryptographic techniques.

Model Performance

We evaluate state-of-the-art LLMs on CipherBank, e.g., GPT-4o, DeepSeek-V3, and cutting-edge reasoning-focused models such as o1 and DeepSeek-R1. Our results reveal significant gaps in reasoning abilities not only between general-purpose chat LLMs and reasoning-focused LLMs but also in the performance of current reasoning-focused models when applied to classical cryptographic decryption tasks, highlighting the challenges these models face in understanding and manipulating encrypted data.

Model	CipherBank Score (%)
Qwen2.5-72B-Instruct	0.55
Llama-3.1-70B-Instruct	0.38
DeepSeek-V3	9.86
GPT-4o-mini-2024-07-18	1.00
GPT-4o-2024-08-06	8.82
gemini-1.5-pro	9.54
gemini-2.0-flash-exp	8.65
Claude-Sonnet-3.5-1022	45.14
DeepSeek-R1	25.91
gemini-2.0-flash-thinking	13.49
o1-mini-2024-09-12	20.07
o1-2024-12-17	40.59

Please see paper & website for more information:

Citation

If you find CipherBank useful for your research and applications, please cite using this BibTeX:

@misc{li2025cipherbankexploringboundaryllm,
      title={CipherBank: Exploring the Boundary of LLM Reasoning Capabilities through Cryptography Challenges}, 
      author={Yu Li and Qizhi Pei and Mengyuan Sun and Honglin Lin and Chenlin Ming and Xin Gao and Jiang Wu and Conghui He and Lijun Wu},
      year={2025},
      eprint={2504.19093},
      archivePrefix={arXiv},
      primaryClass={cs.CR},
      url={https://arxiv.org/abs/2504.19093}, 
}

11 个文件

浏览文件

数据集版权信息

本数据集的许可证为 Apache License 2.0。如有违反相关条款，请联系 WEHUB，我们将及时处理。查看许可证

yu0226/CipherBank 已完整同步