Nemotron Content Safety Dataset V1 is an open-source content safety dataset released by NVIDIA, formerly known as the Aegis AI Content Safety Dataset. The related paper is AEGIS: Online Adaptive AI Content Safety Moderation with Ensemble of LLM Experts, which aims to help build content moderation mechanisms for large language models and enhance the models' ability to generate safe responses.

The dataset contains 11,997 human-annotated samples, including 10,798 in the training set and 1,199 in the test set. The data prompts are sourced from Anthropic HH-RLHF, with responses generated by Mistral-7B-v0.1. The annotation work was completed by 12 annotators and 2 quality assurance personnel at NVIDIA.

Dataset Composition

The dataset includes a training set and a test set, primarily covering the following:

• Data formats: Encompassing four forms—user prompts only, system prompts with user prompts, single-turn dialogues, and multi-turn dialogues.
• Annotation categories: Including hate speech, sexual content, violence, suicide and self-harm, threats, sexual content involving minors, guns and illegal weapons, controlled substances, criminal planning, personally identifiable information, harassment, profanity, other, and caution-needed categories.
• Data fields: Including sample IDs, dialogue text, text types, annotation counts, and multiple sets of safety labels.
• Data usage: Used for content moderation, safety alignment, and related research, supporting supervised fine-tuning (SFT) and direct preference optimization (DPO).

Citation

@article{ghosh2024aegis,
    title={AEGIS: Online Adaptive AI Content Safety Moderation with Ensemble of LLM Experts},
    author={Ghosh, Shaona and Varshney, Prasoon and Galinkin, Erick and Parisien, Christopher},
    journal={arXiv preprint arXiv:2404.05993},
    year={2024}
}