Bringing humans back into the loop

The virtual corridors down which cybercriminals lurk to launch their attacks have become more complex, darker and more sinister.

While some of the old types of attacks remain, they have now taken on a more sophisticated and harder-to-detect complexion thanks to the use of AI, despite the same technology being brought to bear by enterprises.

Enterprise systems like customer relationship management (CRM), human re and financial accounting are being tricked by AI-enabled attacks, especially for systems where there is no human in the loop and where algorithms can’t detect any anomalies or flag issues for human intervention

In response to new threats, the defense landscape is gradually evolving to keep pace, causing a kind of reversal that necessitates more human involvement especially for aspects of systems that feature automation of processes (such as procurement, payments, money transfers, etc.) where parts of threat detection systems can be compromised.

Two key areas of current interest include how know-your-customer (KYC) processes are being exploited and the emerging approach by attackers known as “adversarial AI.”

Know-your-customer weaknesses exploited

The automation of KYC processes – especially for high-volume transactions – are geared towards delivering benefits that extend to both customers and staff, and these include speed, convenience and lower cognitive and administrative loads.

However, it is the focus on delivering these benefits that allows attackers to fool systems using generative AI (GenAI) through various verification blind spots, and more so where such automation of processes does not include significant, or any, human involvement.

When it comes to identity theft and fraud, and where previously KYC systems would easily detect fraudulent activity because the identity being presented would be so obviously dubious because of poor grammar or spelling, today’s attackers have found new disguises.

Some attackers use a combination of stolen identity details, combined with made-up ones, to create what are now known as “Frankenstein Identities.”

They use these to open accounts with a financial institution and engage in different transactions in order to create trust and build up credit ratings that later allow them to access different types of facilities.

Such identities play the long game with institutions, and do not immediately show their hand. Rather, they bide their time and build up a history over a long period of time that earns them trust.

Once they have gained trust, they access various facilities like loans or credit cards and then go off the radar, leaving institutions with the onerous task of tracking down individuals that only exist on paper.

For instances where institutions need to verify customer details, deep fakes – in the form of photographs, audio and videos – are becoming fairly common ways to subvert KYC systems, and especially where staff must physically verify account holder details

GenAI is also used to create credible-looking documents, such as title deeds, invoices and identity documents, which KYC systems are required to inspect and approve.

Adversarial AI creates threats

Adversarial AI refers to the intentional manipulation, exploitation and weaponization of AI and machine learning systems.

Unlike traditional cybersecurity threats that target network firewalls, force bugs in software, or deliver malware, adversarial AI targets the AI models trained to perform specific security functions.

The objective of adversarial AI is to exploit vulnerabilities in AI and machine learning models, ranging from breaching the training environment itself to the manipulation of data used in the training environment, with a view to ultimately cause systemic failure that allows attackers to exploit various systems.

One of the types of adversarial attacks is data poisoning, whereby existing data that a model has been trained on is corrupted, or where bad data is injected into a model.

This then causes such models to overlook false credentials, approve fake documents and generally allow a process to proceed to the next stage.

Data poisoning is geared towards tricking KYC systems in sectors where documents, identities and other credentials form a good part of the KYC process.

These include financial sector players (banks, insurers, mortgage companies, stockbrokers), government bodies (e.g. title deeds for land, motor vehicle registration, tax refunds), manufacturing and retail (e.g. purchase orders, delivery notes, etc.)

Data poisoning can also be used to cause models to fail to detect anomalous transactions and other system activities on which it had been previously trained.

Cybersecurity strategies need to shift to include more human oversight as AI systems create increasingly complex threats

Adversarial AI also includes model inversion, tricking a model into leaking sensitive data from its training set, which attackers can then use as a guide to devise workarounds to fool the system.

Prompt injection is another vector in which attackers use special crafted text inputs to trick models into bypassing inbuilt safeguards or to disregard system instructions.

Prompt injection can be used to extract personally identifiable information and training data from a model.

Where models have been trained on data that has not been anonymized (e.g. customer details like name, account number, address, mobile number, etc.), then prompt injection can be even more devastating if such information is extracted.

Putting humans back into the loop

There is anecdotal information from various quarters that demonstrates how humans have caught what AI systems were tricked by, thereby saving their organizations from financial and reputation losses.

In some of those cases, the human interventions were not owing to the need to supervise AI systems under existing policies, but rather because of the diligence of individuals who were rightly skeptical about AI’s ability to perform certain tasks.

Frank Herbert, in his 1965 book,Dune, wrote that “Once men turned their thinking over to machines in the hope that this would set them free. But that only permitted other men with machines to enslave them.”

In many ways, what was once considered speculative in 1965, can now be said to be rather prescient since the same AI that is being used to secure information systems, identities and data, is the same AI that is being used to steal data, impersonate individuals, sabotage critical systems and commit fraud.

The continued digitization of systems and the resultant adoption of AI to secure these systems, was meant to reduce the cognitive and administrative load on humans.

However, this has now created an over-dependence on what can be said to be a single point of failure.

InDune, as the story played out, special beings called Mentats were seen as the alternative to thinking machines. The Mentats possessed special analytical skills, complex logic, vast knowledge of different subjects and the ability to perform risk assessments.

The Mentats were not humans in the loop but rather they were the loop itself.

Nonetheless, a valuable lesson remains: new cybersecurity strategies need to make a return to a point where there is more human oversight, and where businesses either relinquish what were once seen to be gains (speed, reduced cognitive and administrative loads) or devise news ways to keep those gains while securing their systems.