Lesson 17. Self-Critique and What Reasoning Models Changed

Mission link: To diagnose a failing agent, you need to recognize when to add a reflection stage (a separate loop that evaluates past failures and carries lessons forward) versus when the model's own internal reasoning makes that loop redundant.
Primary source: Paper: "Reflexion: Language Agents with Verbal Reinforcement Learning", Shinn et al., 2023
Prerequisites: Lesson 16

Warm-up

1. ▢ In a plan-then-execute agent, what happens if a step fails and the harness does not support replanning?

2. ▢ What is the advantage of explicit replanning over just running the agent again from the start?

Know this

Reflexion: Adding a reflection loop

The Reflexion paper introduces an explicit reflection stage into the agent loop. After an attempt fails (the agent's answer is wrong, a task does not complete, a test fails), the harness does not immediately retry. Instead, it generates a natural-language reflection on what went wrong. This reflection is stored (often in a file or memory buffer) and included in the context for the next attempt.

The loop looks like this:

1. Run the agent on a task.
2. Check whether the task succeeded (a test passes, a human evaluates the answer as correct).
3. If it failed, generate a reflection: "I tried X, but X failed because Y."
4. Store the reflection in a persistent memory.
5. Run the agent again, this time including the stored reflection in the context.
6. Go back to step 2.

This is different from a simple retry. A simple retry runs the agent again with the same context. Reflexion runs the agent again with the reflection in context, so the second attempt can learn from the first attempt's failure.

Why reflection is not free

Reflexion costs extra model calls. If the first attempt fails, you now need a call to generate the reflection, then another call to run the second attempt. For tasks with low failure rates, this overhead is not worth paying. For tasks where many attempts are needed, the cumulative cost is high.

Reflection also consumes context budget. Each reflection takes tokens in the context window. If you build up many reflections across attempts, you eventually have to prune or summarize them. This is a different version of the context-management problem from earlier lessons.

Reflection makes sense only if the reflection itself is grounded in real information: a test that failed, an error message, a human evaluation. If the task has no clear failure signal, reflection has nothing concrete to react to and may simply rationalize the previous wrong answer.

Models with extended internal reasoning

In recent years, some frontier models have gained the ability to do extended internal reasoning before producing an answer. This reasoning is not visible in the transcript (it happens inside a single model turn, before the output is returned) but it allows the model to self-correct, double-check its work, and catch obvious mistakes before they appear in the output.

This changes, but does not eliminate, the case for explicit reflection loops. Some of what a reflection loop used to provide (catching an obviously wrong intermediate step) now happens inside the model's own reasoning, so adding a reflection loop on top provides less marginal benefit than it used to.

However, explicit reflection still has an advantage that internal reasoning does not: it can incorporate information that only exists after a tool call was made. If a tool call returns "Permission denied" or a test fails, the model's internal reasoning cannot have seen that failure signal during its original reasoning phase, because the tool had not run yet. An explicit reflection loop can observe the tool's actual result and incorporate it into the next attempt.

So the question is not "reflection or internal reasoning," but rather "when does the extra cost of external reflection earn its keep despite internal reasoning?"

When explicit self-critique earns its cost

Use external reflection when:

• The task has a clear, checkable failure signal (a test suite passes or fails, an API returns an error, a human rates the answer as wrong or right).
• A second or third attempt, informed by the failure, is likely to succeed where the first attempt failed.
• The task is worth retrying: high enough value that the extra model calls are justified.

Do not add reflection when:

• The task has no clear failure signal (e.g., "write a summary" with no test or rubric to check correctness).
• The failure is not the model's fault (e.g., a required API is down, not something the model can fix).
• The first attempt is already very likely to succeed and failures are rare.

Designing reflection signals

Reflection is only as good as the signal it receives. If you tell the harness "this task failed because the user said so," the reflection might generate "I tried to write a summary but the user did not like it." That is too vague for the next attempt to act on. A better failure signal would be "the summary is missing the customer's stated requirements" or "the summary is longer than the 100-word limit."

When building a reflecting agent, design the failure check carefully. What does success look like? What concrete evidence proves failure? That evidence is what the reflection will react to.

Practice

1. ▢ An agent writes code to solve a LeetCode problem. The test suite runs the code and reports either "All tests pass" or prints a specific failing test case. After a failure, the harness generates a reflection and runs the agent again with the reflection in context. Why is this a good fit for Reflexion?

2. ▢ An agent is asked to write product copy for an e-commerce site. The agent produces copy and submits it. There is no test suite, no rubric, and no automatic check. A human marketing manager reads it and says "This is not right, try again." Would Reflexion help here?

3. ▢ A model with extended internal reasoning can now self-correct during its own reasoning phase. Which of the following is still an advantage of an explicit external reflection loop in this context?

   • a) External reflection is always faster than internal reasoning.
   • b) External reflection can incorporate information (like a tool error or test failure) that only exists after the model's first attempt.
   • c) External reflection is cheaper in context tokens than internal reasoning.
   • d) External reflection works with all models, even those without internal reasoning capability.

4. ▢ You are building an agent that processes job applications. The harness has a clear signal: approved or rejected by a hiring manager. You want to use Reflexion to improve the agent's decisions over multiple attempts. What should each reflection capture to be useful for the next attempt?

Real-world reps

• [ ] Take a task that involves iteration and has a clear pass/fail signal (e.g., solving a problem, passing a test, meeting a rubric). Implement a reflecting agent loop: attempt, check failure, generate reflection, retry with reflection in context. Run it five times and count how many attempts the agent needs with reflection versus without. Record the difference.
• [ ] Find a task in your work that has no clear failure signal (e.g., writing, design, creative work). Describe how you would add one (e.g., "compare against a rubric," "have a peer review," "check specific criteria"). Would adding a failure signal make a reflecting agent useful, or is the task better handled with a human loop?
• [ ] Tomorrow: Read a Reflexion trajectory (from a paper, a blog post, or something you built). Identify the reflection text. Would the reflection have been useful if the model had internal reasoning? Why or why not?

Going further

• Paper: "Reflexion: Language Agents with Verbal Reinforcement Learning", Shinn et al., 2023
• Resources

Not landing? Reread the primary source at the top, since this lesson compresses it and compression is where understanding leaks. Check the glossary for any term that felt slippery.

If the lesson itself is unclear rather than the material, that is a defect: open an issue.