Question about the "RAG (Current-File)" setting on the leaderboard

[Espresso-Kp]

Hi, thanks for the great work and for maintaining the leaderboard!

I have a question about the configuration used for models listed as + RAG (Current-File). Could you clarify what this setup exactly means?

• Does “Current-File” mean that the full content of the file being evaluated is provided directly to the model as part of the context (instead of retrieving it)?
• Is the RAG component here using sparse retrieval or dense retrieval?
• Does RAG(Current-File) mean the combination of RAG(sparse/dense) and Current-File as Context?

Understanding this would help me reproduce the evaluation setup more accurately.
Thanks a lot!

[shanchaoL]

Thanks for the question!

Yes, “Current-File” means that the entire file containing the target function (with the function body removed) is provided directly to the model as context. In practice, we provide the prefix and suffix surrounding the target function, with the constraint that their combined length does not exceed the context limits described in the paper.

RAG (sparse-retrieval) refers specifically to sparse retrieval, and RAG (dense-retrieval) refers to dense retrieval.

RAG (Current-File) refers the Current-File setting only. It is not a combination of RAG + Current-File. We will correct the leaderboard wording to make this clearer and avoid confusion.

[Espresso-Kp]

Hi, thanks for the reply. However, I’m a bit confused about the “Current-File” setting — it outperforms many RAG strategies, which suggests that RAG may not even be necessary when the “Current-File” context is available. However, I have doubts about the realism of your setting of the "Current-File", because in real software development, the content after the target function is not available during 0-to-1 implementation. In practice, you wouldn’t see the code that appears below the function before you’ve written the function itself. From my perspective, only including the prefix for the "Current-File" setting is more realistic.

Could you share your insights on this, including why the evaluation includes the suffix (i.e., the content that comes after the target function)?

[shanchaoL]

Thanks for the follow-up. Here are the clarifications.

1. On “Current-File” vs. RAG.
   All of these three approaches could still be missing contexts. It just appears that for certain models, having more complete local information helps generate code that passes the repo’s test cases more reliably, which is why “Current-File” might look stronger in the evaluation.

2. Why we include the suffix
   Real-world development isn’t strictly linear. Developers naturally scroll, revisit, and move back and forth within a file while implementing or modifying code. Because this non-autoregressive workflow often exposes both prefix and suffix, we think including the suffix reflects a realistic development scenario.

[Espresso-Kp]

Thanks for the clarification. I agree with your reasoning about the non-linear software development scenario.

For the first question, do you have detailed statistics about the three different task levels (self-contained, file-level, and repository-level)? I wonder if the reason the "Current-File" setting outperforms the RAG setting in general is that it solves more 'self-contained' and 'file-level' tasks than RAG, but actually less in 'repo-level' tasks as it doesn't include any content from other files, but RAG does. I think it would align with my expectations if that is the case.

[shanchaoL]

Thanks for the follow up.

You can refer to Table 5 in our paper (for GPT-4o and GPT-4o-mini). The picture is mixed: For GPT-4o-mini, Current-File tends to outperform other context settings. For GPT-4o, RAG (dense) actually does better than Current-File on both self-contained and file-level tasks.

So the gap isn’t simply because Current-File solves more contained tasks; it seems to depend on the model and the context strategy, and we don’t yet have a definitive explanation for that yet.