Command Palette
Search for a command to run...
HumanEval
Date
HumanEval (Code) is a code generation benchmark proposed by OpenAI, comprising 164 hand-written programming problems and their corresponding unit tests. The HumanEval score refers to the proportion of functions generated by the model that pass all test cases, making it one of the most commonly used benchmark metrics in the field of code generation. It evaluates functional correctness, rather than code style or efficiency. It is typically reported as a percentage or a score between 0 and 1, where higher values are better. When using it, one must specify whether the metric is pass@1 or pass@k, along with settings such as sampling temperature, to ensure fair comparison.
Build AI with AI
From idea to launch — accelerate your AI development with free AI co-coding, out-of-the-box environment and best price of GPUs.