In early 2024, a finance worker at the engineering giant Arup joined a video call with what appeared to be the company’s chief financial officer and several other colleagues. The meeting looked ordinary. The faces were familiar. The voices were right. Over the course of the call, the worker was persuaded to transfer HK$200 million—roughly $25.6 million—across 15 transactions. Every other participant on that call, including the CFO, was an AI-generated deepfake.
That was two years ago. The technology has only gotten better, cheaper, and more accessible since then. By 2026, a usable voice clone can be built from three to five seconds of recorded audio. Real-time face-swapping tools can put someone else’s face over yours on a live video call, blinking and turning on cue when asked to prove you’re human. And the fraud economy built around synthetic impersonation has grown into an estimated $40 billion annual market, according to a report published in October 2026.
The question is no longer whether AI can impersonate someone convincingly. It can. The question is whether anything we currently rely on to tell real from fake—our eyes, our ears, our liveness checks, our identity documents—can still be trusted. And if not, what replaces them?
The Collapse of the “Unforgeable” Signal
For years, the defense against impersonation rested on a simple premise: some things are too hard to fake. A live video call, where the person blinks and responds in real time. A voice that matches a known speaker’s biometric profile. A government-issued ID with holograms and microprinting.
Each of those signals is now forgeable.
An anti-financial crime leader interviewed by ACAMS put it bluntly: “Documentary identification is becoming essentially useless.” AI tools can replicate holograms, microprinting, ultraviolet imagery, and scannable barcodes. They can generate facial photographs convincing enough to defeat biometric verification checks. And voice cloning can replicate speech patterns from just a few seconds of audio.
The scale of the problem is staggering. In April 2026, identity verification firm Jumio reported a 700% year-on-year rise in injection attempts—attacks that introduce manipulated images or video directly into verification systems, bypassing the camera entirely. Deepfake incidents in the fintech sector jumped 700% in a single year. And in a survey of senior cybersecurity leaders, 36% reported at least one deepfake incident during a video call in the past 12 months.
The New Arms Race: Detection vs. Generation
The first instinct of the security industry was to build better detectors. And they have. Modern deepfake detection systems analyze facial action units, labial motion, spatial-frequency artifacts, and multimodal inconsistencies. Some claim accuracy rates above 97% on benchmark datasets.
But there’s a fundamental problem with the detection-first approach, and it’s the reason companies like World are pivoting to a different strategy altogether. As World put it, detecting deepfakes “can be especially challenging, lead[ing] to an arms race between deepfake detection and creation“. Every detector is trained on yesterday’s fakes. Today’s fakes are built to evade yesterday’s detectors. And tomorrow’s fakes will be built to evade today’s.
A comprehensive review published in ScienceDirect in October 2026 argued that reliable deepfake detection requires a shift “from benchmark-centered optimization toward robust, multimodal, and deployment-oriented evaluation and design” . In other words: performing well in the lab doesn’t mean performing well in the wild, where media is compressed, reposted, re-encoded, and stripped of metadata.
Detection is necessary. But it cannot be the foundation of trust. Because the asymmetry favors the attacker: they only need to succeed once, while the defender needs to succeed every time.
The Pivot: Proving You’re Human, Not Proving the Fake Is Fake
On October 19, 2026, World—the identity verification company co-founded by OpenAI CEO Sam Altman—launched a suite of tools designed to solve the problem from the other direction. Instead of trying to detect every fake, World is trying to cryptographically prove who is real.
The flagship product is World ID for Zoom. Here’s how it works: a worker visits a physical Orb device, which scans their iris to create a cryptographically secure biometric reference. A selfie through the mobile app then verifies the human participant against that reference during Zoom calls. If the check succeeds, a “Human Badge” appears next to their name.
Meeting hosts can require verification before anyone joins, or request it mid-call. The system compares three images: the original Orb enrollment, a fresh smartphone selfie, and the live video feed. The result is not a judgment about whether the video looks fake—it’s a cryptographic assertion that the person on screen is the same person who enrolled at the Orb.
World has also launched World ID for Agents, which lets an AI agent present proofs on behalf of a verified human, and a World ID app that stores and presents identity credentials. The initial partners include Bain Capital, Pantera Capital, and VanEck Funds. More than 18 million people have already received Proof of Human verification.
The logic is compelling: if you can’t reliably tell what’s fake, build a system where the default is “unverified” and trust is granted only through proof. It’s a inversion of the internet’s original architecture, where anonymity was the default and identity was optional.
The Other Layer: Content Provenance
Parallel to the “proof of human” movement is a quieter but equally important effort to label content at the point of creation.
C2PA Content Credentials are a metadata standard that records how a piece of media was made or edited—whether AI tools were used, and what changes were applied. SynthID, developed by Google DeepMind, is an invisible watermark embedded in AI-generated content that can survive compression and metadata stripping.
In May 2026, Google made both verification systems accessible to ordinary users. You can now right-click or circle an image in Chrome or Search and ask, “Is this made with AI?” The tool checks for both SynthID watermarks and C2PA metadata in a single interface. OpenAI has begun embedding SynthID into images generated by ChatGPT, Codex, and its API. TikTok has labeled over 3 billion pieces of content as AI-generated through a combination of Content Credentials, watermarking, and creator tools.
This is meaningful progress. But it has a glaring limitation: it only works if the label is there. Content without a watermark or credential is not necessarily real—it’s just unlabeled. And the most malicious actors are precisely the ones who will strip metadata, remove watermarks, and re-encode their fakes before distributing them. As The Verge put it, “SynthID and C2PA can only detect watermarks if they’ve been added in the first place”.
Provenance tells you what a file claims to be. It doesn’t tell you what it is.
The Regulatory Backstop (and Its Limits)
Lawmakers are beginning to respond. The EU AI Act’s Article 50 transparency obligations took full effect on August 2, 2026. AI-generated or AI-manipulated imagery, video, audio, and text must be identifiable as artificial, and deepfakes must be disclosed to viewers. Violations carry fines of up to €15 million or 3% of global annual turnover.
In the United States, the TAKE IT DOWN Act was enacted in 2025, targeting non-consensual intimate deepfakes. The FTC Impersonation Rule is being used to pursue AI-enabled fraud. And New York has adopted the first state law requiring disclosure of AI-generated performers in advertising.
But regulation has a structural disadvantage: it moves at the speed of legislation, while the technology moves at the speed of a software update. The EU AI Act’s transparency rules apply to “providers and deployers” of AI systems—but they cannot force a criminal in a jurisdiction beyond reach to label their deepfake. Regulation sets the floor for legitimate actors. It does not stop the illegitimate ones.
The Human Problem: Trust Is Not a Technical Setting
The deepest challenge is not technological. It’s psychological.
A 2025 Deloitte study found that nearly 70% of respondents worried synthetic content would be used to deceive them, nearly 60% struggled to distinguish AI content from human-made material, and 84% believed synthetic content should be clearly labelled. People know the problem exists. They know they can be fooled. And they know that the old signals—seeing, hearing, recognizing—are no longer reliable.
But knowing is not the same as being protected. The scams that succeed are not the ones that fool sophisticated technologists. They’re the ones that fool a 79-year-old man who sells everything he owns and travels 2,000 kilometers to meet an online girlfriend who turns out to be AI-generated. They’re the ones that fool a dentist who hands over 93 million pesos because she saw a video of the president endorsing an investment—a video that was a deepfake. They’re the ones that fool a finance worker on a video call where everyone looks and sounds exactly right.
The ACAMS report ranks AI-powered identity fraud as the second most significant threat facing anti-financial crime professionals globally. Identity theft and synthetic fraud ranks as the second hardest fraud typology to detect, behind only the exploitation of existing business relationships. The criminals are not just making fakes. They’re building synthetic ecosystems—complete fictional identities supported by fabricated businesses, websites, customer reviews, and social media profiles, all maintained by autonomous AI agents. These identities can pass initial due diligence, build a plausible transaction history, and then be activated for fraud at scale, months later.
So How Will We Know Who Is Real?
There is no single answer. But a new architecture of trust is emerging, built on four layers that work together—and none of which is sufficient on its own.
Layer 1: Cryptographic proof of personhood. Systems like World ID that bind a physical human to a digital credential through biometric enrollment. This doesn’t tell you who someone is; it tells you that a unique human is present. For high-stakes interactions—banking, corporate video calls, government services—this may become the default requirement.
Layer 2: Content provenance. C2PA and SynthID that label media at the point of creation. This won’t catch everything, but it raises the cost of undetected forgery and gives platforms and users a signal to work with.
Layer 3: Multimodal, real-time detection. Detection systems layered across voice, video, behavioral biometrics, and device signals. The best tools now flag manipulated media in under three seconds, and the more elaborate the deepfake, the more likely it trips one of the other checks.
Layer 4: Institutional and social verification. Out-of-band confirmation. A phone call to a known number. A code word agreed upon in advance. A policy that no transfer above a certain amount is authorized on the basis of a single video call. The mundane, low-tech practices that security professionals have recommended for years—and that the Arup finance worker, tragically, did not follow.
The World ID approach is not a panacea. It requires physical hardware (an Orb), it raises privacy questions about biometric enrollment, and it creates a new kind of dependency—on a single company, a single standard, a single point of failure. But it represents a philosophical shift that is likely to define the next era of digital trust: stop trying to prove that the fake is fake. Start proving that the real is real.
AI can now pretend
to be almost anyone. It can wear your face, speak in your voice, and sit across from your colleagues on a video call. The old signals—the ones we trusted without thinking—have been compromised. Seeing is no longer believing. Hearing is no longer believing. Even a live conversation is no longer proof that the person on the other end is who they claim to be.
What replaces those signals is not a single technology or a single law. It is a layered system of proofs and checks, some cryptographic, some institutional, some stubbornly low-tech. It is less convenient than the world we had before. It requires more friction, more verification, more moments of “prove it” that feel intrusive and exhausting.
But that friction is the price of trust in an age when anything can be synthesized. The question is not whether we will pay it. The question is whether we will build systems that make the payment worth it—systems that protect the vulnerable, hold the impersonators accountable, and preserve the possibility of genuine human connection in a world where the fake is indistinguishable from the real.
The Arup worker transferred $25 million because the faces on the call looked real. The next time, the face might be verified, the voice might be signed, and the transfer might be blocked by a policy that says: a video call is not enough. That won’t solve the problem. But it’s a start.
