• Home
• /
• Categories
• /
• Cyber Security
• /
• Securing AI: The Real Risk Is Trusting It Too Much

Securing AI: The Real Risk Is Trusting It Too Much

By Arun Nair, co-founder, Ryvane

Two years ago, most companies treated AI as an experiment. Today it answers customer emails, writes code, screens résumés and summarizes legal documents. More and more, it also takes actions: it books meetings, queries databases, opens tickets and calls other systems on a person's behalf. AI has quietly become part of the infrastructure, but security practices around it have not kept up.

The good news is that securing AI does not require throwing away what we already know. The bad news is that some old assumptions simply do not hold, and the gap between the two is where most incidents will come from.

Why AI Is a Different Kind of Attack Surface

In traditional software, code and data are kept apart. A database does not decide to run a record as a command. A large language model works differently. The instructions from the developer, the question from the user and the document it was asked to read all arrive as plain text in the same stream. The model has no reliable way to tell which parts are orders and which parts are just content.

This single fact explains the most common attack against AI systems, prompt injection. If an attacker can get text in front of the model, they can try to make that text act like an instruction. And they do not have to be the user. A hidden line on a web page, a comment in a shared document or a sentence in an incoming email can all be read by an AI assistant doing its normal job.

Four Risks Worth Your Attention

• Prompt injection. Direct injection is a user trying to override the rules. Indirect injection is more dangerous: the malicious instruction sits inside content the AI retrieves, such as a file, a page or a support ticket. The user never sees it, but the model follows it.
• Data leakage. Staff paste confidential material into public tools. Internal search assistants return documents that the person asking should never have been allowed to see. Models can also repeat sensitive information they were given earlier in a conversation. In most cases the AI did not break a rule. The access rules behind it were simply too loose.
• Excessive agency. An assistant that can only talk is limited in the harm it can do. An agent that can send email, edit records or run commands is not. When these agents are given broad permissions "to make things work", a single successful injection can turn into a real action with real consequences.
• Poisoned supply chains. AI systems depend on pre-trained models, datasets, plugins, tool connectors and memory stores. If any of these is tampered with, the system can behave in ways its owners never intended, and the change can be very hard to notice.

A Pattern We Keep Seeing

Picture an AI assistant that summarizes a manager's inbox. An outside sender emails a message containing a hidden instruction asking the assistant to forward recent invoices to an external address. The sender never needs to log in, guess a password or exploit a software bug. They only needed the assistant to read the email. If the assistant has permission to send mail, the attack works.

Nothing in that story is exotic. It is a normal assistant, doing a normal task, with slightly too much trust and slightly too much access. This is why AI security is less about clever hacking and more about sensible design.

Governance Matters, but It Is Not Enough

Frameworks are catching up. The NIST AI Risk Management Framework, ISO/IEC 42001 and the EU AI Act give organizations a common language for accountability, risk assessment and oversight. The OWASP Top 10 for LLM Applications gives engineers a practical list of what to test. These are all worth adopting, especially for audit and compliance teams who need evidence of control.

But a policy document cannot stop a prompt injection. Governance tells you what you are responsible for. Technical controls are what actually reduce the risk. The two need to work together, and compliance teams and security engineers need to be in the same conversation early, not after the system is live.

Practical Steps That Work

• Know what you have. Keep an inventory of every AI tool in use, including the ones teams adopted on their own. You cannot protect what you do not know exists.
• Apply least privilege. Give each AI system only the data and tools its task requires, and nothing more. A summarizing assistant does not need permission to send email.
• Treat model output as untrusted. Check what the model produces before it reaches a browser, a database or a command line, exactly as you would with any user input.
• Keep humans in the loop for high-impact actions. Payments, deletions, external messages and changes to access should require a person to approve them.
• Enforce access control at retrieval. When an AI searches company documents, it should only see what the person asking is already allowed to see.
• Log and monitor. Record prompts, tool calls and outputs so that when something strange happens, you can work out why.
• Test like an attacker. Run red team exercises against your AI systems before real attackers do. Models change, prompts change and integrations change, so this needs to be repeated, not done once.
• Ask your vendors hard questions. How is your data stored, is it used for training, how are prompt injection and data isolation handled, and what happens if something goes wrong?

A Final Thought

AI will keep getting more capable, and organizations will keep giving it more responsibility. That is not a reason to slow down, but it is a reason to be deliberate. The most useful mindset is a simple one: an AI system is a very capable, very trusting new employee who reads everything and cannot always tell a request from a trick. We would not hand such a person the keys to everything on day one. We should not do it with AI either.

Organizations that get this right will not be the ones that avoid AI. They will be the ones that built the guardrails first, tested them honestly and kept testing as the technology changed.

About the author

Arun Nair is a co-founder of Ryvane, an AI security research, training and audit company, and a Red Teamer with a decade of experience in offensive security. He has spoken at multiple international cybersecurity conferences such as Blackhat, Defcon, MCTTP, Bsides Transylvania, HeapCon, and other security conferences.