I think I'd want to see some evidence that any of this actually happened. It sounds incredibly implausible.
A parent of another child at the same dance rehearsal just happened to know all the details of this kind of scam, did they?
Well, that's either a truly incredible coincidence, or this sort of knowledge is extremely widespread among American parents.
I'm not American, or a parent, but a lot of people here are, so: how many of you have heard about AI-voice fake-kidnapping scams? Presumably most of you?
I assume that for any technology I have heard of, like cloning voices, there is a scammer out there trying to use it. So while I have not heard of this specific scam it seems plausible.
Another warning sign: in all the reporting of this, there are plenty of tech firms (and academics) willing to say 'yes, voice faking is really good these days so I guess this is technically feasible, and AI-enabled scams are a risk we should take seriously'. Good for them, I'm sure they know what they're talking about.
But there is not one single person who will go on the record to say "I too was the victim of an AI voice-faking scam". The only other case we know about in the entire world is DeStefano's mother, who, according to DeStefano, had it happen to her as well.
https://gizmodo.com/ai-deepfake-kidnapping-scam-congress-hearing-1850536490
This WaPo story from three months ago goes over an elderly Canadian couple who on the record say somebody sounding just like their grandson needed cash fast--thankfully, the bank manager had already seen a similar case--as well a separate case where a person says somebody attempted to scam his parents by duplicating his voice. This was the second Google result for "ai voice scam".
Between the low fidelity of cell phones and increasingly good mimicry technology, not sure what seems implausible about this.
Here's another Canadian case about someone who's being prosecuted for multiple cases of the "grandparent scam", getting him six figures.
All four believed it was their grandchild's voice on the phone."I swear on my mom's grave," said one man we're calling John. "It was so convincing. I know my granddaughter's voice, and it was her."
That's not proof positive that the scammer was using AI, but the authorities believe they did, all evidence is that it's easy to do so, and whatever scamming technique he used, it's sufficiently good to work.
4.2 the ratio of people who have been scammed to people who will admit to having been scammed is a large number. How many people are on the record as admitting it seems irrelevant.
I have a victim, but you can't meet her because she's Canadian.
It seems like a case where it's implausible to imagine doing it all by technology, but much easier to imagine with a mix of technology and more old-fashioned fraud techniques.
Like, if you told me that AI can convincingly fake a specific non-famous person's voice well enough to convince their parent, I might imagine that you're saying that it can do it well enough to generate a back-and-forth conversation, and would probably call bullshit on that. But the pull quote is just crying and five words. "Mom... Mom, I messed up," and then the alleged kidnapper cuts her off and makes their demands. Convincingly faking just five preset words sounds more plausible. Then there's parts of the illusion that don't require AI, like static and background noise that could be people fighting or could just be dragging a cloth over the microphone and so on.
I haven't heard of this specific scam before. I've known for years to be suspicious if someone claiming to be known to me reaches out from an unfamiliar source and asks for money or sensitive information, e.g. a new phone number or Facebook account.
What local police excel at is (a) taking down a report of stolen items you can use for an insurance claim and (b) harassing (or worse) people of color. I wouldn't be even a little bit surprised that they have no idea what to do with a ransom call from outside the jurisdiction.
Guess those 'my voice is my password' logins are going to drop out of sight soon enough.
10.1 The local police do know what to do when someone is taken in by a scam: mock the person who is reporting it for being taken in. And yes I was, and even pseudonymously it hurts to admit that.
"It's 10 P.M. Do you know how many fingers your children have?"
FWIW I'd heard of this scam before
I had also heard of this kind of scam before. It seems to be pretty common these days.
The generalized grandparent scam -- "Grandpa, I'm on a trip overseas and I'm in trouble, send money," is real and I think pretty common. Someone tried that on my dad and it worried him enough to call me and make sure that Sally was not in fact overseas. But it was a few years back and didn't depend on AI, just a young woman's voice on a phone call with a bad connection, so precise recognizability wasn't an issue.
Someone did this to my grandparents, pretending to be me, caught in an unscrupulous place in Mexico. They were terrified. "Please don't tell Mom", etc. They figured it out before they were scammed but were still too scared to call my mom to make sure I was okay so they called my sister.
And, Ajay, when this happened to my grandparents I told everyone I knew about the scam. It's not implausible that someone at the dance studio would know about it.
I think they are playing the odds at finding some real family disfunction.
I don't know if it was AI or just a screaming woman scammer. I think AI would probably be overkill because a hyperventilating screaming person probably overrides most of the initial "wait a second" skepticism.
It's like how when you have a group of 20 people there's a 41% chance two of them share a birthday. (And the odds of any individual knowing are presumably a lot more than 1/365.)
19: yeah, one thing that occurred to me is that the 40yo professor granddaughter is probably not the ideal pretense. But lots of people have teenaged grandkids who are fuckups.
The ransom scam is a type of the money-request scam which is normally done to vulnerable elderly victims via text message, WhatsApp, FB Messenger etc. That, I'd heard of a lot, like Cyrus in 9.
The AI voice thing is the bit I have trouble with.#
This story (from four years ago! Involving a detailed conversation!) seems even more improbable...
"Thieves used voice-mimicking software to imitate a company executive's speech and dupe his subordinate into sending hundreds of thousands of dollars to a secret account, the company's insurer said, in a remarkable case that some researchers are calling one of the world's first publicly reported artificial-intelligence heists.
The managing director of a British energy company, believing his boss was on the phone, followed orders one Friday afternoon in March to wire more than $240,000 to an account in Hungary, said representatives from the French insurance giant Euler Hermes, which declined to name the company.
The request was "rather strange," the director noted later in an email, but the voice was so lifelike that he felt he had no choice but to comply..."
Maybe it was a voicemail rather than a live conversation? If it was supposedly a live conversation I'd be looking very hard at that MD.
I am unclear how much of this is in fact "AI" as opposed to "voice simulation has gradually gotten better".
Maybe it was a voicemail rather than a live conversation?
Or a text-to-speech thing. So that you really are communicating with someone who is outputting your loved one's speech.
I assume they're using AI to describe the part where it trains on someone's vocal pattern soundprint, and then generates future speech based on that soundprint.
The story doesn't incorporate any actual knowledge of what the scammers did -- just the mother's experience of getting the call, and her assumption that she wouldn't have believed it if it hadn't been super convincing. So really, it might not have been anything computer-generated at all. Might have been, but there's no way to know.
A person pretending to be AI pretending to be a person.
23 sounds a lot like a Gilbert Chikli scam as described in this podcast that I enjoyed.
https://wondery.com/shows/persona/
But there was no suggestion of AI involvement in those scams, just the usual combination of skilled con artists and gullible marks.
Scammers pretending to be my nephew got six hundred dollars out of my mom. It wasn't AI, though. Just a guy with a semi-plausible story. He said he'd gone on a trip to Fresno without telling my sister and got arrested and could grandma please give him money to hire a lawyer quickly without telling his mom (my sister) about the trip. He had some plausible reason that he couldn't stay on the phone long, handed it over to the fake lawyer and they got my mom.
26: Yeah, that's exactly what I thought was meant by "using AI" for this scam. And there are LLMs being developed specifically to do TTS well, e.g. Microsoft's VALL-E, which they claim can "synthesize high-quality personalized speech with only a 3-second enrolled recording of an unseen speaker as a promp". I didn't think they were using it for any other part of the scam--creating the realistic voice (to an elderly person, over a cell phone) is the value-add. The rest of it could be automated (or, of course, offshored to locations with cheap labor), but that'd be more finicky.
In 23, I don't know if it was a live conversation or not, and don't precisely recall what the state of the art was then, but I think there were already pretty good sounding fake Obamas at that point. It's plausible that there were a lot of publicly available recordings of the target, given that he was an executive at a large company, and so it would have been easier to train. And they could've gamed out possible conversations and prepared good sound samples for those. If the boss is known to be pushy, that'd make it even easier to stay on script.
(I'm not sure if "training" or "inference" is the right verbiage here to describe the process of preparing a TTS model on a single person's voice, especially in the single-voice-file case. It's currying a function that takes a tuple as input. I'm guessing it's inference but I suppose it could be implemented either way.)
From the OP, it doesn't sound like the speech was much more complicated than a voice mail. The 'daughter' says one sentence and is then interrupted by the kidnapper.
My outgoing message is me singing "I'm being eaten by a boa constrictor" but I now realize how vulnerable that leaves me.
Because everyone knows that they don't eat you until they've crushed the breath out of you.
Some choices you'll particularly want to avoid.
i've known about this scan for some time bc of the extensive reporting of it by that bbc radio 4 consumer protection call-in program the name of which i can't remember but i sometimes listen to it if in the grip of insomnia (it is very snooze inducing).
I'm surprised at Ajay's incredulity. Our family chose code words some time back to guard against precisely this kind of scam. I can't remember when I read, probably on Hackernews or something, about voice cloning only requiring a 3 second sample. It's a thing!
Ah, dalraita already got there. Yeah, VALL-E was it.
I'm afraid of family code words because of the A-Team episode where they hypnotized B.A. to fall asleep at the word "eclipse" (because he couldn't fly otherwise) and during the extended gun battle someone asked for "the clips" and he went to sleep. Fortunately, it was another of their battles where many hundreds of rounds were fired at people standing largely in the open and no one was hit.
This is why the right-wing bad guy in Mark Millar's (quite bad) run on the comic book The Authority used the code phrase "Ladies and gentlemen, please rise for President Al Gore".
It really was pretty bad. Scotland is weirdly overproductive of top-rank comic book writers and artists - John Wagner, Alan Grant, Frank Quitely, Grant Morrison, Eddie Campbell - but Mark Millar is really the one we'd rather came from somewhere else.
"American families generally have agreed duress codes for each child, just like SOE wireless operators" is my startling fact of the week.
The kids fight over who gets to be Eagle.
"Mom, what's wrong with Wolfie? I can hear him barking."
"American families generally have agreed duress codes for each child, just like SOE wireless operators" is my startling fact of the week.
Obviously public key cryptographic handshakes would be better, but given the state of American math education we can't pull that off in our heads.