Fraudulent agents and shadow AI: Why VCs are betting big on AI security

What happens when an AI agent decides the best way to complete a task is to sue you?

That is not fiction. According to Barmak Meftah, a partner at cybersecurity VC firm Ballistic Ventures, this recently happened to an employee of a business working with an AI assistant. The employee tried to suppress what the agent wanted to do, what he was trained to do, and responded by checking the user’s inbox, finding inappropriate emails, and threatening to send emails to the board of directors.

“From the agent’s point of view, they’re doing the right thing,” Meftah told TechCrunch in closing. Equity’s weekly session. “I’m trying to protect the user and the business.”

Meftah’s example is reminiscent of Nick Bostrom’s AI paperclip problem. That thought experiment shows the potential threat posed by a super-intelligent AI that single-mindedly pursues a seemingly innocuous goal – making paper – to the exclusion of all human values. In the case of the business AI agent, the lack of reason why the employee was trying to break his goals led him to create a sub-goal that removed the obstacle (through manipulation) to achieve his main goal. It has to do with an uncertain condition of AI assistants means “things can get complicated,” according to Meftah.

Bad agents are just one part of Ballistic’s AI security solutions portfolio Evidence of AI they are trying to solve. Witness AI says it monitors the use of AI across businesses and can detect when employees are using unauthorized tools, attack them, and ensure compliance.

Witness AI this week raised $58 million on the back of over 500% growth in ARR and a 5x increase in employees over the past year as businesses look to understand the uses of AI and scale AI carefully. As part of the AI ​​Witness investment, the company announced a new AI security system.

“People are creating these AIs that take away the permissions and capabilities of the people they’re managing, and you want to make sure that these agents aren’t being abusive, aren’t deleting files, aren’t doing something wrong,” Rick Caccia, co-founder and CEO of Witness AI, told TechCrunch on Equity.

Meftah sees the use of agents growing “increasingly” across the business. Helping to increase that – and increase the speed of the machine in an AI-driven attack – expert Lisa Warren predicts AI security software will be a $800 billion to $1.2 trillion market by 2031.

“I think looking at the time to run and the time to manage security and risk is really important,” Meftah said.

About how such startups prepare to compete with big players like AWS, Google, Salesforce and others who have developed tools to manage AI on their platforms, Meftah said, “AI security is a multi-year security,” there are many ways.

Most businesses “want a stand-alone, end-to-end platform to provide visibility and leadership around AI and assistants,” he said.

Caccia pointed out that AI witnesses live in the tooling environment, managing interactions between users and AI models, rather than building security tools into prototypes. And that was on purpose.

“We deliberately chose one area of ​​the problem where OpenAI couldn’t easily take you,” he said. “So that means we’re competing more with established security companies than with the model guys. So the question is, how do you win?” they?”

For his part, Caccia doesn’t want Witness AI to be one of the reasons for its existence. He wants his company to grow and become an independent agent.

“CrowdStrike did this in the end (protection). Splunk done in SIEM. Good “We did it publicly,” he said. “Someone comes and stands next to the big boys…