LLM Daydreaming · Gwern.net

De­spite im­pres­sive ca­pa­bil­i­ties, large lan­guage mod­els have yet to pro­duce a gen­uine break­through. The puz­zle is why. A rea­son may be that they lack some fun­da­men­tal as­pects of human thought: they are frozen, un­able to learn from ex­pe­ri­ence, and they have no “de­fault mode” for back­ground pro­cess­ing, a source of spon­ta­neous human in­sight.