AI and Cybersecurity

Cybercriminals are using artificial intelligence (AI) to create spear phishing attacks personalized against a specific individual. According to CNN, fraudsters recently used deepfake technology to fabricate a videoconference with a company’s CFO, convincing a Hong Kong finance worker to send them $25 million.

Cybersecurity companies are countering with AI-enabled analysis of video data to detect deepfakes.

Reports of cybercriminals and cybersecurity companies using AI in their attacks and defenses are increasing. But what about vulnerabilities and threats to sensitive data in and between AI companies? Recently, a Chinese company, DeepSeek AI, exposed an unsecured database. Researchers at cybersecurity company Wiz Research discovered it.

“We found a publicly accessible ClickHouse database linked to DeepSeek that was completely open and unauthenticated, exposing sensitive data,” said Gal Nagli, an external exposure researcher at Wiz. “The database contained a significant volume of chat history, backend data, and sensitive information, including over a million lines of log streams, API secrets, and operational details,” Nagli said.

“More critically, the exposure allowed for full database control and potential privilege escalation within the DeepSeek environment, without any authentication or defense mechanism to the outside world,” said Nagli. Attackers escalate their rights and privileges on a system to gain additional unauthorized access.

Wiz notified DeepSeek AI before sharing its findings, and DeepSeek closed the vulnerability quickly. DeepSeek did not respond to a request for comment.

Vulnerable AI companies could also find themselves the victims of competitors. NBC News recently received an email from an OpenAI spokesperson stating that DeepSeek, a Chinese company, may be trying to catch up to OpenAI through distillation. This technique trains a large language model (LLM) on data generated by another LLM.

According to NBC News, the email said that OpenAI is investigating indications that DeepSeek may have inappropriately distilled its LLMs. OpenAI did not accuse DeepSeek of a breach. The NBC News report appeared the same day as the Wiz report: January 29, 2025.

OpenAI did not respond to a request for comment. Microsoft, the largest investor in OpenAI, had no further information to provide, according to an account executive at a PR agency for Microsoft.

Actors who could have entered the DeepSeek database

“The exposed, unauthenticated database opened the door for a wide range of actors—from unsophisticated hobbyists to highly organized groups—to access sensitive data,” said Golan Yosef, chief security officer at Pynt, an API security testing company.

According to Yosef, state-sponsored threat actors, organized cybercrime groups, rival companies, and even white hat researchers could all have easily entered the publicly exposed DeepSeek database.

AI companies take measures to prevent exposure

AI companies are implementing specific measures to prevent exposures similar to the DeepSeek database event. According to Yosef, these measures include network segmentation to restrict sensitive databases to internal networks and secure VPNs, so no one can access those databases externally.

Yosef said AI companies are using automated configuration management to scan for misconfigurations continuously, to help detect and remediate any accidental exposures quickly. He added that AI companies also are committing to frequent infrastructure audits and continuous monitoring to ensure configurations remain secure.

Network segmentation isolates parts of the network for security. VPNs (virtual private networks) secure remote access to internal networks. Automated configuration management monitors and updates network device configurations, such as access credentials.

Vulnerabilities in AI development and deployment

According to Yosef, API and LLM security are the most vulnerable points in AI development and deployment.

A recent Cisco Outshift blog post said organizations integrating LLMs from providers like OpenAI through an API can increase vulnerabilities and add ingress points for threat actors.

Several news sources, including Bloomberg, reported that in the fall of 2024, Microsoft security researchers saw people exfiltrating a large amount of data using the OpenAI API. Microsoft shared this information with OpenAI. According to the Bloomberg article, Microsoft believes those individuals may be linked to DeepSeek. Misusing an AI company’s APIs is a way to get unauthorized access to LLM data.

“APIs are often the main access points for AI systems, and without proper measures–such as robust authentication, authorization, and rate limiting—they can be exploited to access or manipulate data,” said Yosef. Authentication uses security keys to confirm the identity of a user. Authorization dictates what the user can do with access to the AI model.

How companies balance AI innovation with robust security

According to Kartik Talamadupula, who serves as applied AI officer for the leadership committee of the ACM Special Interest Group on AI (SIGAI), AI agents becoming able to access data and make decisions based on that data autonomously underlines the need to have an auditing system for those agents, to determine whether they are supposed to have access to the data they are using to make their decisions. Automated tests and audits should run every time a change is made to the agent’s system/configuration to ensure that they continue only to have access to the data they were supposed to access in the first place, Talamadupula said. AI agents are autonomous software that runs independently, using data to understand the environment, take action, and learn from the results.

Said Yosef, “Companies can balance AI innovation with security by embedding security measures early in the development process: integrating automated security tests and continuous monitoring, using agile development with regular audits, fostering collaboration between development and security teams, and offering ongoing training in secure coding practices.” Agile development uses short development cycles with feedback to update software in increments.

According to Matt Guenther, vice president of Strategic Programs at Stratascale, a cybersecurity services company, it’s still early for AI regulation and security frameworks. However, some frameworks are available. “Both NIST AI-600-1 and ISO 42001 are frameworks for establishing and maintaining sound AI security management,” said Guenther.

“In conjunction with something like ISO 27001, the end-to-end workflow can be further secured,” said Guenther. “AI is a component of the overall information security strategy. The Access Control and Encryption guidance found in ISO 27001 would’ve prevented DeepSeek database vulnerabilities.” ISO 27001 is an industry standard for managing information security.

AI companies have not fully established a collaborative framework for addressing common AI industry security challenges, because these companies are concerned about sharing sensitive information with potential competitors in the process, said Jeff Orr, director of research, ISG, a technology research and services firm.

AI company exposures and major breaches

Significant breaches of AI data and companies may be forthcoming. Yashin Manraj, CEO of Pvotal Technologies, a software development company, said RFPs, hiring trends, and Fortune 500 strategies show that AI is becoming interwoven into production systems, streamlining operations and reducing dependency on human staffing.

According to Manraj, over the next 16 to 24 months, the most profitable businesses will intricately weave AI tooling into their production systems and start to process more data using AI than with traditional or human-monitored systems. “We believe this will lead to a shift in [attacker] interest in targeting AI data and companies,” said Manraj.  

David Geer is a journalist who focuses on issues related to cybersecurity. He writes from Cleveland, OH, USA.