# DiscoveryBench with OpenHands [DiscoveryBench](https://github.com/allenai/discoverybench/) [(Paper)](https://arxiv.org/abs/2407.01725v1) contains 264 tasks collected across 6 diverse domains, such as biology, economics, and sociology. It incorporates discovery workflows from published papers to approximate the real-world challenges faced by researchers.

## Setup Environment and LLM Configuration 1. Please follow instructions mentioned [here](https://github.com/openlocus/OpenHands/blob/discoverybench-openhands-integration/evaluation/README.md#setup) to setup OpenHands development environment and LLMs locally 2. Execute the bash script to start DiscoveryBench Evaluation ``` ./evaluation/benchmarks/discoverybench/scripts/run_infer.sh [YOUR MODEL CONFIG] ``` Replace `[YOUR MODEL CONFIG]` with any model the model that you have set up in `config.toml` ## Run Inference on DiscoveryBench Instances When the `run_infer.sh` script is started, it will automatically pull the latest DiscoveryBench instances & set up the agent environment. The OpenHands agent is invoked to process the task within this environment, producing a hypothesis. We then evaluate it against the “gold” hypothesis provided by DiscoveryBench. The evaluation result, along with the agent chat history is logged to `output.jsonl` under `evaluation_outputs`. ``` ./evaluation/benchmarks/discoverybench/scripts/run_infer.sh [MODEL_CONFIG] [GIT_COMMIT] [AGENT] [EVAL_LIMIT] [NUM_WORKERS] ``` - `MODEL_CONFIG`: Name of the model you want to evaluate with - `GIT_COMMIT`: This should be the git commit hash or release tag for OpenHands, e.g., HEAD or a specific tag like 0.6.2. - `AGENT`: Use CoderActAgent, right now it only supports that. - `EVAL_LIMIT`: Number of samples to evaluate. - `NUM_WORKERS`: Number of workers to parallelize the evaluation process.