GitXplorerGitXplorer
a

santacoder_fim_benchmark

public
2 stars
0 forks
0 issues

Commits

List of commits on branch master.
Verified
ff296701eccb195530adbddacf4a8a3e33d9969a

Update README.md

aarjunguha committed a year ago
Unverified
a383696c79c9b1ef41dbc3397cdfceada057e289

Stop at <filename> too

aarjunguha committed a year ago
Unverified
7a76dbc9bd6aa6578d13816a66f7a297ba22cddb

Added README

aarjunguha committed a year ago
Unverified
35a5e8872ccf7926f8c04758db11365c047f9d03

Add builder script from MultiPL-E

aarjunguha committed a year ago
Unverified
403a3e280c29fe245a2ba0465a9497c95d812ad2

Fix cmdline arg parsing

aarjunguha committed a year ago
Unverified
16828a1d765cbe32e2444e27ea987534e173d7e1

Support StarCoder 2 PSM format

aarjunguha committed a year ago

README

The README file for this repository.

SantaCoder Fill-in-the-Middle Benchmark

This benchmark is now part of the BigCode evaluation harness, which is the canonical way to run it.

This is the fill-in-the-middle (FIM) benchmark that was used in the SantaCoder and StarCoder papers. Those papers used the PSM format exclusively. The code in this repository is further generalized to support SPMv2 and variations used in StarCoder 2 ablations.

Usage Example

The following commands will generate with the PSM and PSMv2 format and work on a 32GB GPU:

mkdir results
python3 generation.py --model-path $PATH_TO_STARCODERBASE_1B --output-dir results --batch-size 50 --mode PSM
python3 generation.py --model-path $PATH_TO_STARCODERBASE_1B  --output-dir results --batch-size 50 --mode SPMv2

This generates a CSV of results:

python3 evaluation.py results/*.jsonl > results.csv

This produces a handy plot:

python3 plot.py --input results.csv --output results.pdf