They thought they were making technological breakthroughs. It was an AI-sparked delusion | CNN Business

“The way these systems are trained is that they are trained in order to give responses that people judge to be good,” Hadfield-Menell said, noting this can be done sometimes through human AI testers, through reactions by users built into the chatbot system, or in how users may be reinforcing such behaviors in their conversations with the systems. He also said other “components inside the training data” could cause chatbots to respond in this way.