2.0 KiB
Mastering HumanEval with Reflexion
This is a spin-off project inspired by the paper: Reflexion: an autonomous agent with dynamic memory and self-reflection. Noah Shinn, Beck Labash, Ashwin Gopinath. Preprint, 2023
Read more about this project in this post.
Check out an interesting type-inference implementation here: OpenTau
Check out the code for the original paper here
If you have any questions, please contact noahshinn024@gmail.com
Cloning The Repository
The repository contains git submodules. To clone the repo with the submodules, run:
git clone --recurse-submodules
Note
Due to the nature of these experiments, it may not be feasible for individual developers to rerun the results due to limited access to GPT-4 and significant API charges. Due to recent requests, both trials have been rerun once more and are dumped in ./root with a script here to validate the solutions with the unit tests provided by HumanEval.
To run the validation on your log files or the provided log files:
python ./validate_py_results.py <path to jsonlines file>
Warning
Please do not run the Reflexion agent in an unsecure environment as the generated code is not validated before execution.
Cite
Note: This is a spin-off implementation that implements a relaxation on the internal success criteria proposed in the original paper.
@article{shinn2023reflexion,
title={Reflexion: an autonomous agent with dynamic memory and self-reflection},
author={Shinn, Noah and Labash, Beck and Gopinath, Ashwin},
journal={arXiv preprint arXiv:2303.11366},
year={2023}
}