What‘s the environment of each question in expbench?

[Fu-Dayuan]

And how can we reproduce the result？
Since some "file masking strategy" has been used.

[patrickkon]

Hi @Fu-Dayuan , thanks for your interest in the project! I've added a draft evaluation pipeline within benchmark/exp_bench (will be polishing the code throughout the week). You can refer to the instructions which will instantiate the environment (including masking) and run the eval on different supported LLMs and agents. Please note that this is time consuming and cost intensive though (e.g., weeks and thousands of dollars) to run!