You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: ADVANCED_USAGE.md
+5-6Lines changed: 5 additions & 6 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -69,7 +69,6 @@ Below are all the arguments for `bigcodebench.evaluate` for the remote evaluatio
69
69
-`--tokenizer_legacy`: Whether to use the legacy tokenizer, default to `False`
70
70
-`--samples`: The path to the generated samples file, default to `None`
71
71
-`--no_execute`: Whether to not execute the samples, default to `False`
72
-
-`--local_execute`: Whether to execute the samples locally, default to `False`
73
72
-`--remote_execute_api`: The API endpoint for remote execution, default to `https://bigcode-bigcodebench-evaluator.hf.space/`, you can also use your own Gradio API endpoint by cloning the [bigcodebench-evaluator](https://huggingface.co/spaces/bigcode/bigcodebench-evaluator) repo and check `Use via API` at the bottom of the HF space page
74
73
-`--pass_k`: The `k` in `Pass@k`, default to `[1, 5, 10]`, e.g. `--pass_k 1,5,10` will evaluate `Pass@1`, `Pass@5` and `Pass@10`
75
74
-`--calibrated`: Whether to use the calibrated samples, default to `True`
@@ -208,10 +207,10 @@ You are strongly recommended to use a sandbox such as [docker](https://docs.dock
208
207
# If you want to change the RAM data segment limit (in MB, 30 GB by default): `--max-data-limit`
209
208
# If you want to change the RAM stack limit (in MB, 10 MB by default): `--max-stack-limit`
210
209
# If you want to increase the execution time limit (in seconds, 240 seconds by default): `--min-time-limit`
0 commit comments