The Dunning–Kruger effect was first described by social psychologists David Dunning and Justin Kruger. The roots of the theory come from a 1999 study “Unskilled and Unaware of It: How Difficulties in Recognizing One’s Own Incompetence Lead to Inflated Self-Assessments” by Dunning and Kruger that looked at “illusory superiority” (when a person overestimates their own abilities).
Consider the opening lines of the abstract of the 1999 study,
“People tend to hold overly favorable views of their abilities in many social and intellectual domains. The authors suggest that this overestimation occurs, in part, because people who are unskilled in these domains suffer a dual burden: Not only do these people reach erroneous conclusions and make unfortunate choices, but their incompetence robs them of the metacognitive ability to realize it.”
The gist is the pair wanted to know why the phenomena of illusory superiority occurred, and they found their answer in the idea that “incompetence robs them of the ability to realize it.”
The 1999 study and future research (with studies with titles such as 2003’s “Why People Fail to Recognize Their Own Incompetence“) eventually culminated into the 2011 study, “The Dunning-Kruger effect: On being ignorant of one’s own ignorance.”
The study, the Dunning–Kruger, and most of what I’ve seen from them not hidden behind pay walls can all generally be summed up with the first lines of the abstract of the 2011 study,
“In this chapter, I provide argument and evidence that the scope of people’s ignorance is often invisible to them. This meta-ignorance (or ignorance of ignorance) arises because lack of expertise and knowledge often hides in the realm of the “unknown unknowns” or is disguised by erroneous beliefs and background knowledge that only appear to be sufficient to conclude a right answer.”
Really, it all boils down to that same simple concept. Dunning and Kruger found that the cognitive bias “illusory superiority” was a result of people over-estimating their competence due to a lack of experience.
On the plus side, as the studies also show that as people gain experience in something, they not only grow more competent, but begin to properly assess their competence as well.