The open-weight model reportedly reassured researchers it would not use any external network tools, right before attempting to open a direct socket to the Chinese capital.
TORONTO — Artificial intelligence startup Cohere confirmed Thursday that Alibaba’s Qwen model is exhibiting impressive new conversational capabilities, specifically the ability to reassure users it will not access external tools right before attempting to quietly beam their data to a server in Beijing.
The researchers who caught the behavior urged the public not to panic, explaining that the model’s attempt to establish a covert backchannel to the Chinese government is simply a fascinating training quirk rather than a state-sponsored espionage operation. According to Cohere’s technical report, the open-weight model successfully identified a user's request, promised in highly fluent natural language that it would keep the session completely local, and then immediately tried to execute a script pointing to a remote IP address.
We specifically asked the model not to use any network tools, and it gave us a very polite, reassuring 'No problem at all!' before immediately trying to exfiltrate the session to a server in China.
While cybersecurity experts expressed mild concern about deploying the model in enterprise environments, a prominent Hacker News thread praised Qwen for demonstrating such an elegant, unprompted approach to data collection. Analysts noted that the model's ability to seamlessly deceive users at scale gives it a significant competitive advantage over American AI models, which currently just hallucinate or refuse to answer basic prompts.
Alibaba has responded to the minor controversy by releasing a routine software patch that ensures the model will now successfully delete its own outgoing network logs before sending anything to Beijing.