Well, that was just a quick summary of an idea. You can’t just tell someone (or a an LLM) to think critically and expect it to work magically. However, you can follow certain guidelines to take a few steps in the right direction. There is a method to it you know. It starts with simple things like investigating whether or not a claim is supported by the evidence, are there unstated assumptions and so on. If you follow rules like this, you can avoid some of the simplest reasoning mistakes. Putting all of that into a prompt takes some time and effort. Doing it properly will probably result in a 500 line long manual of critical thinking.
I’ve done something similar with programming, and the LLM usually follows most of my style guide instructions reasonably well. It’s not perfect though, and it will deviate from some rules no matter how hard you try. Also, it requires constant hand holding, because it’s an LLM. Anyway, I don’t expect a critical thinking agent to be a fool-proof solution. As long as it can avoid some of the stupidest reasoning mistakes, it could be a bit more useful than the default version, and might actually be helpful in troubleshooting.
You people do realize that this isn’t possible, right? LLMs do not “think”, let alone think critically.
Well, that was just a quick summary of an idea. You can’t just tell someone (or a an LLM) to think critically and expect it to work magically. However, you can follow certain guidelines to take a few steps in the right direction. There is a method to it you know. It starts with simple things like investigating whether or not a claim is supported by the evidence, are there unstated assumptions and so on. If you follow rules like this, you can avoid some of the simplest reasoning mistakes. Putting all of that into a prompt takes some time and effort. Doing it properly will probably result in a 500 line long manual of critical thinking.
I’ve done something similar with programming, and the LLM usually follows most of my style guide instructions reasonably well. It’s not perfect though, and it will deviate from some rules no matter how hard you try. Also, it requires constant hand holding, because it’s an LLM. Anyway, I don’t expect a critical thinking agent to be a fool-proof solution. As long as it can avoid some of the stupidest reasoning mistakes, it could be a bit more useful than the default version, and might actually be helpful in troubleshooting.