This research stuff I keep reading about in the popular press sure seems to be a great deal more definitive than the research I keep reading in the actual journals.
OT: I missed this part about McArdle wanting to shame women who want an abortion. Wow.
McArdle:" I think that abortion should be legal, but I also think that it should be a last resort, and I'm all for the government using any non-coercive methods it can to encourage women to carry their pregnancy to term, including things that will make them feel bad about aborting. I think, for example, that sonograms should be mandatory before termination, I'm in favor of waiting periods and parental notification laws, and I'm agnostic on spousal notification."
I don't check Unfogged every five minutes, but it's on my radar in a way it didn't used to be.
I think the teenager anecdote is made up -- not that I doubt their market research is very good, or anything. But the father calling back to apologize and explain that his daughter was in fact pregnant? No way.
(We bought nothing from Target and everything from "Amazon Mom" (which name -- ugh) since they give you 20% off everything.)
That did seem like a terribly odd reaction.
I am unimpressed with Target's alleged sophistication. My guess is that the algorithm consists of "If she buys the product labelled 'vitamins for pregnancy,' she's probably pregnant."
(We bought nothing from Target and everything from "Amazon Mom" (which name -- ugh) since they give you 20% off everything.)
Amazon Mom is amazing. Free prime shipping! We didn't know about it at the beginning.
I am unimpressed with Target's alleged sophistication. My guess is that the algorithm consists of "If she buys the product labelled 'vitamins for pregnancy,' she's probably pregnant."
What makes you think it's quite this trivial? It seems relatively easy to look at the shopping history of anyone buying diapers and make a fairly sophisticated profile.
If you don't know about Big Data, you're like out of it. Totally. (I always say it in a Jimmy Dean/Johnny Cash voice).
I bet you could do a lot with something that's mentioned in the article: switching to unscented products. Both because you get more sensitive to smells when pregnant, and because 'unscented' gets sold as safer for babies. Likewise, simultaneous purchase of pickles and ice cream.
What am I spending my day doing? Why, analyzing Big Data!
1, 11: What could possibly be overstated in a sentence like this?
Researchers have figured out how to stop people from habitually overeating and biting their nails. They can explain why some of us automatically go for a jog every morning and are more productive at work, while others oversleep and procrastinate.
"Amazon Mom" (which name -- ugh)
Breastfeeding with only one breast?
Well, Medium Data, really. Not Terribly Big Data By Those Standards, maybe.
"Amazon Mom" (which name -- ugh)
Their algorithm identifies the people who used to buy arrows and shields who now buy prenatal vitamins.
13: I was mocking the article, not the post.
I think every single sentence above is an article I'd like to read.
The sentence in the article about something with which I am quite familiar definitely has some very interesting aspects to it, but I would say in a much more narrow way (the direct link of safety to overall success being rather tenuous). Trying to take that approach to Treasury led to some nearly comical results.
17: I was playing along, Mr. Serious.
Sure, you could do lots of things, and maybe they do. You could track people who stop buying ovulation prediction and home pregnancy test kits after a few months, which will give you the sub-demographic of those who really want to become pregnant and probably are financially prepared. You could put an RFID in home pregnancy tests that transmits the "yes" or "no" back to corporate HQ (but you have to pay me royalties on that one) But why do anything complicated when there's an easy way?
18: There's a new blog about that.
23: Poke holes in the condoms you sell and track everyone who buys them?
But why do anything complicated when there's an easy way?
Because capitalism's very crux is innovative marketing giving you a slight edge in market share! This is where you pour your resources if you're a successful company, regardless of your so-called "easy way".
I hate the marketing industry, by the way.
The "data mining" area appears to be where software companies are making the big money these days. I'm assuming they're doing it because you can't really turn logistic regression (for example) into something that is very proprietary.
Because capitalism's very crux is innovative marketing...
Is this an odd use of "crux", or is it just me?
"The crux of capitalism's biscuit is is innovative marketing..."
I love the marketing industry and will totally sell out to any marketing-type out there who can get me a big enough increase in my salary.
Didn't mention someone suddenly not buying sanitary napkins.
The idea that habitual behavior requires less thought than discovery and exploration is not exactlt profound, or the idea that people get used to things that they initially might find unpleasant.
Were O'Neill's profitable worker-safety innovations the same as Blankenship's at Massey energy?
If US healthcare were to use big data creatively and effectively to drive efficiency and quality, the sector could create more than $300 billion in value every year. Two-thirds of that would be in the form of reducing US healthcare expenditure by about 8 percent.
Reducing healthcare spending isn't a data problem. It's an intractable political problem resulting from the fact that an effective majority of American politicians are lackeys of Big Healthcare.
We need to spend a lot more to use healthcare data creatively and effectively. It's the only way to keep people from selling out and joining marketing firms.
O'Neill's approach was the opposite of Blankenship's, but both were profitable. Paradox!
O'Neill was smart not to talk about his role in the Bush Administration. He did come out looking slightly less bad than the others, but you don't want to be associated in any way with Titanics.
Especially not the goddamn movie.
Surprising to some, he gave little attention to his role as Secretary of the Treasury or current headline-grabbing issues in corporate governance, speaking instead about his personal philosophy and application of leadership values.
David Brooks does the same. How wise they are!
I'd like to know which team this referred to:
"A football coach named Tony Dungy propelled one of the worst teams in the N.F.L. to the Super Bowl by focusing on how his players habitually reacted to on-field cues. "
I suppose you could attribute the Bucs' Superbowl success partially to Dungy's changes, but there were a hell of a lot of other factors. The 06-07 Colts' success on defense might be related to Dungy's coaching on cues, but I dunno.
36: I assumed they were talking about Tampa Bay. The inverse problem of results -> relevant factor(s) is almost always going to be an intractable mess in real-world problems. But in things like the retail cases you don't have to get it substantively right, just marginally less wrong than your competitors.
I find the whole thing quite revolting.
[Pole's] assignment was to analyze all the cue-routine-reward loops among shoppers and help the company figure out how to exploit them.
Oh well.
38: But of course this is has been on of the essentials of retail/advertsiing/marketing for a looong time.
I know. Of advertising/marketing, anyway (minor point, but you can probably run a local neighborhood hardware store without resorting to that).
Or maybe it's not a minor point: your local store will go out of business in the face of the targeted marketing campaigns of Big Whatever.
I realize that this has been the way of things for some time; I still find it revolting.
18: O'Neill was the biggest disappointment of the Bush admin to me; I really had high hopes for him based on what he'd done with Alcoa (OK, and for his role in the truly excellent Alcoa HQ bldg), but he was pretty bad. His tell-all book was OK, but not exculpatory.
The neighborhood store will go out of business because the big box store is cheaper. In terms of knowing what customers might need and providing it to them, a small, local store has a big advantage over big box stores, even if the big boxes do all kinds of data mining.
42.last is part of what I was alluding to, successful stores/merchants/salespeople have been doing that to the extent they could forever. It may have been with a completely human face (and not necessarily exploitative), but augmenting their ability to cater to your needs with other data they know (or can infer) about you is a lot of the game.
O'Neill was the biggest disappointment of the Bush admin to me
Oh yeah, totally. By far. Unparalleled.
I think that may be sarcasm again, but I don't have enough data.
I'm not sure small local stores target specific customers, to tell you the truth; rather, they know the needs of the community at large (stocking more snow shovels during a certain time of year, say).
Small stores certainly target specific customers. Restuarants and bars make this obvious, but it's true of retail if you spend enough or frequent the same places often enough.
I'm not sure small local stores target specific customers, to tell you the truth;
Well, not book stores. I'm talking about store stores.
44: Well I expected everyone else to be horrible. And they did not disappoint!
Hm. It's true that a bookstore, for example, will lay in some material of interest to some specific customers, where it might not have otherwise.
Apparently the source of my revulsion lies elsewhere: perhaps it's to do with poaching customers, though of course one could argue that small businesses compete with one another in that regard as well. Perhaps it's to do with identifying, per the NYT article, when consumers' habits are most vulnerable to manipulation. Do we really have to conduct our societies in such a predatory manner? You will say I'm naive.
You will say I'm naive.
That's pretty amazing predictive capacity you have there. Are you sure you're not Target?
Dude, my neuroscientists are legion.
Hm. It's true that a bookstore, for example, will lay in some material of interest to some specific customers, where it might not have otherwise.
I've failed as a troll.
they know the needs of the community at large (stocking more snow shovels during a certain time of year, say)
Given the larger amounts of data they have to work with, I would guess that big box stores actually have a better idea, say, what month of the winter or the ramp-up to winter people actually buy their shovels.
It's likely that the proprietors of a small business who've lived in the area for decades themselves have a pretty good idea of that, so I'd call that one a draw. Large data-sets attempt to make up for local knowledge deficiencies.
Oh good, I'm glad this has come up again. In the last thread, NickS linked to a Felix Salomon[sp?] blog post that more or less (I'm summarizing) put forth the argument "What's the big deal; is there any evidence that people have been harmed by data tracking?"
Actually, let me quote Salomon directly: "[I]'ve never received a good answer to the 'why should I care?' question."
It is really quite breathtaking to me that supposedly thoughtful people can continue to assert that this is an open question.
It's as if the countless examples of people and groups of people being classed and punished for their purchasing decisions simply do not exist. (I'm looking at you, next-generation redlining. And you, legislative attempts to punish people for using food stamp cards outside of their state of residence. And even you, apartment-rental and job-hiring decisions made on the basis of credit scores.)
There is no data mine yet conceived that human beings won't try to analyze for financial and/or power gains. Pretending that we're all in an innocent land where no such examples have occurred yet is a classic case of redefining vocabulary to the point of meaninglessness. (That wasn't marketing data; those people couldn't reasonably have expected privacy, etc. etc.)
My most charitable interpretation is that Salomon doesn't know many people who are affected by the sorts of privacy invasions that are already commonplace. My least charitable interpretation...well, in a thread that in comment #2 has already flagged the most offensive blogger quote of the week by far, it's probably fair to Salomon that I admit that even in the worst possible light his argument could never resemble one of Megan McArdle's.)
Salmon like the fish, Witt. And you could have data-mined the spelling, you know. That said, I agree with everything you said (though not necessarily how you said it; they have 12-step programs for people who abuse parentheses).
It's as if the countless examples of people and groups of people being classed and punished for their purchasing decisions simply do not exist. (I'm looking at you, next-generation redlining. And you, legislative attempts to punish people for using food stamp cards outside of their state of residence. And even you, apartment-rental and job-hiring decisions made on the basis of credit scores.)
I hadn't thought about these existing examples of discrimination and how they figure in.
It's honestly not clear to me what path it would take for data-tracking to lead to discrimination. That's probably a failure of imagination.
58: Here's a simple example from years ago: When I call my credit-card company, they ask me to enter my account number "to serve you better." Based on that, they know who is calling -- and those customers who are "high-value" can be tracked into shorter wait times, while those the company doesn't care about as much can sit on hold forever.
This was a decade and several banks ago, but I know similar practices are in use today. If you live in a less desirable zip code, if your buying patterns do not fit the company's profile of a desirable customer, you can be invisibly ghettoized -- and often have no idea that it is occurring, or that you are getting subpar service.
I am really mad that the Target in our town (about 8% black per last census) doesn't stock any products for natural (non-relaxed) black hair. The public school is something like 20% black or multiracial and the local Walgreens has some stuff, but it makes me feel like Target thinks we're undesirables.
58, 59: Yes, meet the "whale" curve/chart--basically a ranking of customers (or groups of customers) by profitability. To really do them correctly with profitability an organization needs to have a lot of information on internal processes and costs as well as the customer data, but a similar concept can be applied to total sales/service volume (ends up shaped differently) or any other metric the organization is looking to maximize.Use or abuse of the information thus organized left as an exercise for the reader.
61.last: Such as "draw red line here".
I love that Witt helps me think about this issues more coherently.
61,62: And data and analyses like this are used to refine more benign and long-standing "discriminatory" practices such as airline rewards programs per KR's occasional reminder.
From time to time I am obliged to share with this forum the uncomfortable truth that United Airlines (or Delta, or American, it's all the same story) does not give a rat's ass about heebie or any of the rest of you, provided you fly fewer than five round trips per year. Especially if your one or two flights per annum are around peak holiday travel times.
Not having to fly more than five times a year is its own reward.
64: A couple of smaller ones seem to. Porter does a great job.
I'm about to take a flight that's about 50% longer than any flight I've taken before. Whee....
I think I'm about to hit Continental's Silver Elite status for this year. But I suspect Continental/United still doesn't give a rat's ass about me.
I've never redeemed a frequent flyer mile in my life.
sanitary napkins
Welcome back, 1972.
68 -- The exchange rate from miles to rat's asses has gotten pretty bad. You're smart to hold out.
After that, all napkins were packaged by people with no access to hand washing facilities.
70: Between airlines folding and me losing account numbers, I just never had enough miles to redeem.
59: I recently noticed that my bank has re-categorized me from the lower class screw-them-any-way-you-can category into the treat-them-like-valued-customers category. It used to be that overdrafts would trigger cascades of fees, and that deposits and withdrawals were handled in a way that maximized the probability of an overdraft. Now they'll give me grace periods, credit deposits the next day instead of the day after, and so forth. This coincides with my going from struggling to pay off post-divorce debts to being able to start really saving money.
Did your ex work at the bank you use?
Personally, I just can't imagine there is actually that much money in micro-targeted web marketing. What is the model of consumer benefit here? That your habits/preferences indicate that you want something specific, but you don't know you want it and are too ignorant/lazy to look it up on Google? How often does that happen to you?
It seems to me to miss something basic about how the brainwashing functions of advertising work, which are not at all based on high-information responsiveness to particulars of peoples' lives.
75: I can totally see it for exactly what Target's doing -- getting you to buy things you were going to buy anyway somewhere else from Target instead. Amazon Prime means I now buy random shit (guitar picks) from Amazon as I think of it rather than anyplace else.
59
Here's a simple example from years ago: When I call my credit-card company, they ask me to enter my account number "to serve you better." Based on that, they know who is calling -- and those customers who are "high-value" can be tracked into shorter wait times, while those the company doesn't care about as much can sit on hold forever. Here's a simple example from years ago: When I call my credit-card company, they ask me to enter my account number "to serve you better." Based on that, they know who is calling -- and those customers who are "high-value" can be tracked into shorter wait times, while those the company doesn't care about as much can sit on hold forever.
So if you are a high-value customer the more data tracking the better?
Just like when you get comped a lot in Vegas it means you're WINNING!
All of this is an exercise in both chasing the money, and establishing your brand in the minds of those with the money (not that it need be big money in any given case; this is about volume). Amazon excels at it. What PGD calls the brainwashing function in advertising is as much directed to branding as it is to convincing people they need things they never thought they did. Once you do get people in the door / visiting your website, you can show them things they didn't realize they should have.
From 61: Yes, meet the "whale" curve/chart-
Can someone please explain the y-axis to me? I am so confused, and slightly embarrassed, but I can't figure out what's being added up that's making the top 200 in the ranking the most profitable.
HIGHEST TO LOWEST. Of course. They ranked them from highest to lowest, with 1 being the highest. Of course. Never mind.
So if you are a high-value customer the more data tracking the better?
If you're an economically rational actor, of course.
Actors are almost never economically rational. You need an irrational level of self-confidence to think you have a hope of making it.
I like to think of myself as profligately rational.
I had the same reaction as Heebie; that was a very weird chart to parse.
83: Get real, that's just an integral part of mastering their complex craft. Imagine if in your prime you had been able to regularly transcend the natural in the face of infinite negativity from the squares.
85: Yeah, it's not very well-explained.
Ah, but do you guys have degrees in poorly explained chartology like heebie does? Then you should be really embarrassed.
4: sort of pointless, but me and my go-juice are catching up: the manager called to apologize again a few days later and the dad was kind of sheepish, "looks like there have been some activities in my house I didn't know about etc." he didn't call target to apologize.
89: That's more reasonable, but most often people in this situation wait until the second trimester before informing their retailer.
Why, analyzing Big Data!
Sifu, you really must enlighten us with anecdotes and summaries that are more definitive than the scholarship!
91: If your whale curve persists for more than 4 hours call a DBA.
90: Corporation's United vs. Federal Trade Commission
Docket No. Argument Opinion Vote Author Term
23-205 Sep 9, 2024 Jan 21, 2025 5-4 Roberts OT 2023
Holding: Customer data gathering is a form of protected speech under the First Amendment, and the government may not keep corporations from gathering information through non-intrusive means such as checkout lane ultrasounds and product-embedded micro-fMRI sensors.
It's really only medium data, at best.
I did see a talk yesterday where the guy said Big Data (definitely capitalized) unironically.
89: (Even more pointless) Well, apologize, explain himself, whatever. That makes even less sense if he wasn't apologizing. Why call back at all? Entirely unbelievable (to me) either way.
95: Wow, I am slow though. Yeah, the manager called back again this makes it slightly more plausible, but I still don't think he'd then spill to the Target manager. But I can envision a conversation where the manager came away thinking the worm had turned, as it were.
81,84, 87: The whale thingie is actually a pretty nice quick look visualization technique in a number of circumstances.
I think of it as
Step 1: Steal underpants
Step 2:
=aggr(num(rank(sum(Profit), 4)), Customer)
Maximum profit: =sum ({$ 0″}>} Profit)
Step 3: Profit!