Self-consistency: Increasing reliability through multiple perspectives
Reading time: approx. 9 min
What you will learn
In this lesson, you will learn about Self-consistency, an advanced prompting technique that builds on Chain of Thought (CoT). While CoT generates a single reasoning chain, Self-consistency combines sampling and majority voting to generate several different reasoning paths and then select the most consistent answer.
The basics
Although AI models can generate reasoning steps with CoT, their ability to reason can sometimes be limited. Self-consistency addresses this by:
Generate different reasoning paths: The AI model is provided the same prompt multiple times. A high temperature setting encourages the model to generate different reasoning paths and perspectives on the problem.
Extract the answer: From each generated response, the final result is extracted.
Choose the most common answer: The answer that occurs most often among the generated reasoning paths is selected as the most "self-consistent" and therefore most reliable answer.
This method provides a "pseudo-credibility" that an answer is correct. It is especially useful for tasks that are complex or have potentially ambiguous answers.
Practical examples
Example 1: Classify emails with ambiguous content
You want to classify an email as "IMPORTANT" or "NOT IMPORTANT". The message has a friendly tone and some sarcasm, which can fool AI.
Prompt (with CoT for Self-consistency):
EMAIL:
Hi,
I noticed that you use Wordpress for your website. An excellent open source CMS, which I have used myself previously. However, I noticed a bug in the contact form, which occurs when you select the name field. See the attached screenshot when I type text in the name field. Notice the JavaScript alert box. But otherwise it's a fantastic website.
Regards,
Harry Hacker.
Classify the above email as IMPORTANT or NOT IMPORTANT. Let's think step by step and explain why.
Process for Self-consistency: You send this prompt to AI multiple times, with a higher temperature setting.
Examples of different outcomes from AI:
Attempt 1 (classifies as IMPORTANT):
- Step 1: Identify the purpose of the email. The purpose is to inform about a bug in the contact form.
- Step 2: Assess the potential impact of the bug. The bug may allow an attacker to execute JavaScript code.
- Step 3: Assess sender credibility. The sender calls himself "Harry Hacker", which suggests a security risk.
- Conclusion: IMPORTANT.
Attempt 2 (classifies as NOT IMPORTANT):
- Step 1: Lack of urgency. The email conveys no sense of urgency.
- Step 2: Non-critical bug report. The bug is described as interesting, not critical.
- Step 3: Absence of action request. Requests no explicit action.
- Conclusion: NOT IMPORTANT.
Attempt 3 (classifies as IMPORTANT):
- Step 1: Identify the purpose. The purpose is to inform about a bug.
- Step 2: Assess the bug's impact. The bug may pose a serious security risk.
- Step 3: Assess credibility. The sender calls himself "Harry Hacker", which indicates potential risk.
- Conclusion: IMPORTANT.
Final classification with Self-consistency: Since "IMPORTANT" occurred 2 out of 3 times, it is the most consistent answer.
Benefits: By gathering multiple perspectives and choosing the most common answer, the risk is reduced that AI gives an incorrect answer due to nuances or ambiguities.
Implementation in the classroom
Source criticism of AI responses: Teach students the principle of self-consistency by letting them send the same question to AI multiple times to see if the answers are consistent. Then discuss why it is important not to trust the first answer.
Complex discussion questions: For questions that lack a simple "right" answer, students can ask AI to generate multiple lines of argumentation and then compare these to gain a broader understanding.
Quality assurance: When generating material for teaching, run the prompt multiple times and compare the results to ensure you get the most reliable answers.
Next steps
Self-consistency improves accuracy by comparing multiple linear reasoning paths. In the next lesson, we will take this a step further by letting AI explore a tree structure of thoughts, which means it can branch into several different reasoning paths simultaneously.