The CyberMetric dataset consists of 10,000 carefully designed questions designed to comprehensively assess the cybersecurity knowledge of big models. The dataset was created using different big models and validated by human experts in the cybersecurity field to ensure its relevance and accuracy. The original data comes from standards, certifications, research papers, books, and other publications in the field of cybersecurity. To meet different research needs, the dataset provides four scale versions: small to extra large, with 80, 500, 2000, and 10,000 questions respectively.

Citation

@INPROCEEDINGS{10679494, author={Tihanyi, Norbert and Ferrag, Mohamed Amine and Jain, Ridhi and Bisztray, Tamas and Debbah, Merouane}, booktitle={2024 IEEE International Conference on Cyber Security and Resilience (CSR)}, title={CyberMetric: A Benchmark Dataset based on Retrieval-Augmented Generation for Evaluating LLMs in Cybersecurity Knowledge}, year={2024}, volume={}, number={}, pages={296-302}, keywords={Accuracy;Reverse engineering;Benchmark testing;NIST Standards;Risk management;Problem-solving;Computer security}, doi={10.1109/CSR61664.2024.10679494}}