bigcode

/

santacoder

like 331

Follow

BigCode 1.11k

Also, what value of temperature did you use? Thank you.

For pass@100, here's the execution command in bigcode evaluation harness (for pass@1 use temperature 0.2)

accelerate launch  main.py \
  --model bigcode/santacoder \
  --max_length_generation 512 \
  --tasks humaneval \
  --n_samples 200 \
  --batch_size 200 \
  --temperature 0.8 \
  --allow_code_execution \
  --trust_remote_code

The reported number is the HumanEval version of MultiPL-E to run it you can replace humaneval with multiple-py

Thank you @loubnabnl