The transcript from this week’s MIB: Joshua B. Miller on the Hot Hand, is below.
You can stream/download the full conversation, including the podcast extras on iTunes, Bloomberg, Overcast, and Stitcher. Our earlier podcasts can all be found at iTunes, Stitcher, Overcast, and Bloomberg.
~~~
This is Masters in Business with Barry Ritholtz on Bloomberg Radio.
BARRY RITHOLTZ, HOST, MASTERS IN BUSINESS: This week on the podcast, I have a fascinating guest and if you’re at all interested in statistical analysis of sports, behavioral finance, data analysis, understanding streakiness, understanding the Monty Hall (ph) problem and then extrapolating that towards things like the hot hand in basketball, you are going to find this to be absolutely fascinating.
Joshua Benjamin Miller comes from California where he basically racked up all the degrees he could at some of the UC schools before getting his PhD in economics in Minnesota. Josh and his co-author have taken apart some of the more interesting statistical assumptions made in the original hot hand study with Tom Gilovich and Amos Tversky and they found something really unusual by looking at the data from a slightly different perspective.
And I approached their paper with tremendous amount of skepticism, I thought the randomness of the hot hands was a fairly well proven study that Tversky and Gilovich did, but when you look at the data and you looked at how they analyzed it, it’s hard not to reach the conclusion that there is some sort of a hot hand. It’s quite sophisticated mathematics, but Josh does a very nice job reducing its – some very easily understandable probability, we don’t — no math is required, you just have to know the difference between a head or a tail when you’re flipping a coin.
If you’re at all interested in anything probability, sports related, statistical, you are going to find this to be a fascinatingly wonky and tremendously interesting conversation.
So with no further ado, my conversation with the economist and statistician, Josh Miller.
My special guest today is Joshua Benjamin Miller, he is the co-author along with Adam Sanjurjo, of a fascinating paper that puts challenge to the myth of the myth of the hot hands. He comes to us with a BA in economics and an MA in mathematical statistics from UC Santa Barbara, he has his PhD from University of Minnesota, and he is currently a professor in the economics department at the University of Alicante in Spain where he focuses his research on behavioral economics, judgment and decision-making, game theory and statistical and experimental methods, Josh Miller, welcome to Bloomberg.
MILLER: Thanks for having me, Barry.
RITHOLTZ: So a little background, we kind of met after I interviewed Thomas Gilovich who I was mostly interested in due to all of his work on behavioral finance, but he also co-authored a fascinating paper that basically pointed out the hot hands. He co-authored that with Amos Tversky by the way, that the hot hand was really a myth and it was just ,we were all being fooled by randomness.
How did that paper come to your attention and what fascinated you by it?
MILLER: So that paper came to the attention of my co-author Adams, also at the University of Alicante, pretty much everyone who takes a behavioral economics class and even earlier gets exposed to that paper, it’s one of the prime examples of bias because it’s apparently powerful.
RITHOLTZ: It’s part of the canon of look how easily we’re all fooled.
MILLER: Exactly.
And in the beginning of any kind of behavioral economics class you have to show the real world implications first to kind of motivate students and here, it’s this one that professionals fall victim to, and they are so resistant, they were shown that this – and we haven’t defined hot hand yet, but the hot hand is this idea that you’re in a zone that success breeds success and if you look at basketball players and coaches, they all believe in this thing and so when they discovered that there was no pattern there, and they came and revealed the output of their research, the professionals were — it was difficult to convince them.
RITHOLTZ: There is tremendous pushback and there’s a famous quote that’s been referenced, was it Red Auerbach up at the Boston Celtics, I don’t care what this professor says…
MILLER: So they do a study, who cares?
RITHOLTZ: Yes.
MILLER: So I mean the stubbornness that came out of the practitioners was really dramatic, specifically, you can convince someone that is motivated to get things right if you can demonstrate that they will benefit from it, and they just discounted it and so there’s this famous quote from Amos Tversky, after all the stubbornness that they encountered repeatedly, people just not even looking at the evidence they were showing them.
That he said I’ve been in thousand arguments and won them all but convinced no one.
(LAUGHTER)
RITHOLTZ: And he was very famous for being on not only quite brilliant but a little hardheaded and a little aggressive when it came to debating people according at least according …
MILLER: Before my time.
RITHOLTZ: At least according to Michael Lewis’ book “The Undoing Project” between Kahneman and Tversky, they were two very distinct personality types. So let’s get back to you before we — we are going to spend a lot of time on the hot hands. You’re not what I would think of as a traditional economist, what sort of work do you focus on?
MILLER: Both my co-author and I focus on individual decision-making, and so were looking at …
RITHOLTZ: Is it individual decision-making within a group, within an institution?
MILLER: Both …
RITHOLTZ: Or just as a lone wolf ….
MILLER: So the psychological factors like my co-author works on search and attention in things like this, but there are also factors of the institution, the design, like how information is presented to you?
And these things while it may be important individually, they also bubble up in terms of how it affects your decision-making groups and how it affects financial markets and so in the end, it does impact policy and economic outcomes.
RITHOLTZ: It has real-world effects in other words, these aren’t just ivory tower abstract discussions, this is real-world application for how decisions are making and how information is presented.
MILLER: That’s right.
RITHOLTZ: So that is really quite interesting. When you — and you mentioned one of your research areas is behavioral finance, has all the low hanging fruit in the space been picked or is there still lots and lots of things to be discovered?
MILLER: There’s still lots of fruit, whether it’s low hanging, I think you always have to work for it…
RITHOLTZ: Right.
MILLER: So I think the way you get the fruit you have to think a lot about how to measure things, have a theoretical grounding in what you are trying to get at and you can’t just rely on existing data and existing things that have been counted, you have to go out and measure things yourself a bit and do some work to collect that data.
And so what makes a lot of the modern work you’ll see is going beyond just the choices that people make, like when you’re paying them to make decisions and looking at the choices, I mean you can get a lot more about you learn a lot more about what people want and what they believe by looking at other things like reaction times, how they search for things, what they’re paying attention to, and…
RITHOLTZ: In other words, you’re not just bringing in a bunch of undergrads, sticking them in a room giving them 20 bucks for the night and say we’re going to put through a series of things, you are looking at a very different data set that’s measuring very different things.
MILLER: Yes, I mean you can improve even with the undergrads but I think a lot of the innovative work goes and collects unique data from unique subjects, like I have a friend, Alex Ylmas (ph) I just saw him present this very interesting — it’s in the topic of finance, we went and looked at institutional investors, tons of data, these were people with big positions and found that they are actually quite skilled at buying stocks, but they aren’t so skilled at selling them and it seems to be distinct skill.
RITHOLTZ: It’s very distinct skills because it’s easy to buy, that’s the easy part selling is where the money gets made, those are not on equal level of difficulty, things I’m absolutely not surprised to hear the selling demonstrates less skill than buying. Is that basically what …
(Crosstalk)
MILLER: So the finding is that they were unloading extreme winners too quickly before they really exploited the information advantage that they had, so they made a good job choosing it, but they sold it too soon.
RITHOLTZ: A classic mistake.
Let’s talk a little bit about the original 1985 hot hand paper which as we discussed earlier became canon in the world of behavioral finance, when did you first start to get an inkling that the original thesis might not have been all it seems to be?
MILLER: So the original inkling was that people sometimes overreact, it’s that it’s a myth, the thing doesn’t exist.
RITHOLTZ: Why does that generate such a strong intuitive pushback from people. I mean I have my thesis, I am curious as to yours.
MILLER: I think everyone has some experience in their own athletic performance where they have moments where they are particularly locked-in and then they realize that outside of athletic performance, you just have these moments where you’re in the zone, that’s the best word for it, you’re just — you are firing on all cylinders and you would expect that you would see that somehow in basketball data as well.
RITHOLTZ: So my personal experience I used to play hoops as a kid but as I’ve gotten older, I’ve become a tennis player and I know from personal experience, it takes a good 20 minutes for me to calibrate my forehand so that I am consistently hitting the ball more or less towards where I wanted, more or less with the right amounts been more or less with the right height, but it’s not something that I could just grab a racket and swing and oh there it is, it takes a while to – too fast, too much whip, keep — loosen your wrist, bring it around, make sure you’re dropping the hit, like I’m running through a series of steps in my head, hey you’re too close, watch your footwork, one after another and I am now good enough to know I suck, I’m in that saw come in that Dunning Kruger drop where, you know I used to think it was good, now I realize I’m really I’m good enough to know how good I actually am not.
But it takes a while to calibrate that. I imagine a basketball player in the midst of a game has to go through some sort of fine tuning of their shooting, you can warm up all you want when you’re just shooting by yourself before the game but when people are on you and you are running, it has to be a very, very different set of circumstances or am I overstating this?
MILLER: Well I mean that’s the strongest intuition is based on this calibration there’s probably other elements when I get to that later on but if you look at, yes, I mean if you’re sitting on the bench for 10 minutes and then you come off, that’s very different, in the NFL you see field goal kickers warming up before you go…
RITHOLTZ: Right.
MILLER: And you don’t see that so much in the NBA, they don’t like an extra hoop on the side.
RITHOLTZ: That’s right.
MILLER: So I would imagine, yes, that’s an important element there.
RITHOLTZ: What else is so intuitively attractive about the idea of the hot hand? Is it simply just the zone, is it the adrenaline and the endorphins? Why do we think that hey, suddenly I’m on a streak, why do we believe that streak is going to continue? And I’m not talking about blackjack or roulette or games of chance, these are really games of skill played at the highest level.
MILLER: So why do we believe it? I would imagine sometimes we believe it it’s not really there and so there is this feeling, and part of the feeling is feedback, you see that you’re successful, it gives you some confidence it’s not always simply this zone that emerges, sometimes you get a few successes in a row and it gives you more confidence in your training.
RITHOLTZ: Right.
MILLER: You don’t overthink it and you return and trust your training, so you are essentially unconscious when you bring that, whereas if you maybe miss a few in a row, you lose your confidence, you start making adjustments, and if you are making adjustments, you are not going to have much consistency.
RITHOLTZ: So let’s go back to the original research. Tom Gilovich, one of the co-authors set about the work that you and Adam did, this is unlike a lot of stuff that’s come down the pike since 1985, this is truly interesting. How encouraging was that from one of the original authors who ostensibly disprove the hot hand?
MILLER: I mean it’s always nice when somebody appreciates your work especially someone of Tom Gilovich’s stature, at the time that he said that, our paper while it have gone through the public peer review process, it hadn’t gone through the formal one. And so just last week, our paper was finally published, it’s online, not in the print edition yet in “Econometrica” which is you know, a top journal in economics, there’s this top-five, they are kind of all equal.
RITHOLTZ: Right.
MILLER: And so now it’s been kind of formally taken in so I think Tom Gilovich might have a different opinion now that it’s gone through this process.
RITHOLTZ: So the paper is ready to be published or was just published?
MILLER: Yes so the November issue of “Econometrica” it came out and that’s …
RITHOLTZ: That’s got to be very exciting.
MILLER: Very exciting, yes. It’s something we celebrate.
RITHOLTZ: So what’s the takeaway from the original research? What was it that was wrong in the structure of the original “myth of the hot hands” paper?
MILLER: Right, so the original hot hand paper, they were interested in seeing if people do better after recent success than after recent failure, that was the most important measure like is your probability of success increase when you’ve hit in a few in a row versus if you missed a few in a row.
RITHOLTZ: Right.
MILLER: And well, we don’t know if someone’s probably is – it seems like our best guess would be would be just look at the percentage of time they make it, and so they just look at all the events when you had a streak of recent successes and all the events when you have a streak of recent failures and just see what the change in your shooting percentage is between those two conditions.
And that’s very natural and very intuitive to expect that’ll be your best guess, and they do that and they don’t find any difference and so that’s how the problem was set up.
RITHOLTZ: So before we get to your solution, the immediate pushback is hey, after shooter gets on a bit of a streak, the defense collapses on them, they become — they’re forced to either pass the ball more or take more difficult shots, at the time in 1985, there was no way to account for that difference, however in the intervening years every shot gets marked, you described this in one of your publications recently, explain the degree of difficulty that is now tracked on every single basketball shots that is taken.
MILLER: Right, so there is a — it’s a new company now, I don’t remember the company but sport view was the first company that did this where they have optical tracking of the ball ….
(Crosstalk)
RITHOLTZ: The exact place where the person would shoot.
MILLER: I mean the precision isn’t super high but it gets in the general area and so you can control for a lot more factors than you could say in 1985 where they had the 76ers and they’re just looking at the play-by-play out …
(Crosstalk)
RITHOLTZ: Right.
MILLER: And so you even in that data, what they would find is yes, it have this evidence of the defense adjusting to what they believed is the hot hand making it more difficult for the player, but the players also shoot from time to time to keep the defense honest, and so the important thing isn’t so much as the player doing better in the context of the game but does it help the teammates if they are hot because it opens up things for the team mates.
RITHOLTZ: Makes sense.
MILLER: Yes, so the innovations that have happened, I think the first innovation actually was Justin Rao who was a head economist at HomeAway, he was the first one to actually come out in and measure how many defenders are around the player and try to control from these things in a different way by using the videos and show that yes there’s a lot of evidence that there’s this defense factor and if you just control for a few of these things the effects that they have found in the previous study went way.
RITHOLTZ: So in other words, what looks like it’s random is you’re shooting the same percentage but with a whole lot more defensive activity on you therefore it’s a continuation of the streak.
MILLER: Yes so he didn’t necessarily find evidence of the streak there because he controlled for a subset of factors but as you add more and more controls, it looks like there might be some evidence but these are very difficult things to measure in the context of the game. The original study had this critical test and it has been repeated with other teams on where they take them, they pay them to shoot the basketball.
RITHOLTZ: So in other words, you are not playing during live game, you are just doing foul shooting or three point shooting or whatever.
MILLER: Exactly, or you look at NBA three-point shooting contest, and in those studies, you can get rid of the defense and get a little more zero in on your question a little more.
RITHOLTZ: Let’s talk a little bit about the surprising math of coin flips, my best guess in my understanding of statistics has always been if you take a true coin and flip it, the odds of a header hotel is 50-50, this is regardless of what came before it, coins have no memory, but you found something surprising in the data set after you flipped a coin hundred times and you were to pick a specific series, the odds are somewhat different. Explain that.
MILLER: So my co-author, Adam Sanjurjo and I after having watched the NBA three-point shooting contest, we had a particular player Craig Hodges who was obviously hot and we went and used the original analysis on his data in and said that he wasn’t, and that was puzzling and so we had to go and see well we don’t really know how Craig Hodges generated his shots, it’s kind of a black box but let’s create the environment we have the ground truth, we know what’s happening, and so coin flips is a world like this, so you can actually go and flip a coin many, many times or do it on a computer and see what you get if you analyze – like we’re interested is the probability of heads after a few heads different than the probability of heads after a few tails, we know that’s the same ,that’s — we have the ground truth.
RITHOLTZ: Right.
MILLER: But now let’s go out and generate that data and make our best guess from that data, what’s our best guess is the percentage of heads that you get after a few heads in a row the same as the percentage of heads you get after few tails in a row, and analyzing in the way they analyzed it, we found that no, it’s different, the percentage of tails after a few heads in a row is higher.
RITHOLTZ: Which is so counterintuitive because prospectively, so understand before people lose their minds and start sending emails, we’re not talking about is looking forward in a live situation.
MILLER: Right.
RITHOLTZ: No matter what the previous was that was a true coin you could have 1,000 heads in a row, highly improbable but not mathematically impossible, the odds on that next left are still going to be 50-50, that’s not what we’re saying, we’re saying flip a coin hundred times look at the data set and then go back and randomly pick any head in that order or any tail on the order, what are the odds that the next flip is a head or a tail and it turns out that’s not 50 percent.
MILLER: That’s correct.
RITHOLTZ: So explain that because it’s a complete — it blows people’s minds because you’ve been taught over and over again hey coins have no memory but that’s not what this is, this is an existing data set, when we randomly pull any of those flips, what are the probabilities as to the outcome in the next flip after it’s already been done?
MILLER: Right.
RITHOLTZ: So how do you end up with 40 percent instead of 50 percent?
MILLER: So the complete explanation would take some time, but we can kind of get an intuition on if you flip a coin 100 times, there is going to be a certain number of heads and tails there when you’re done.
RITHOLTZ: About 50-50 but no guarantee.
MILLER: No guarantee but it is going to be some number. Now if you just choose any flip, your best guess just to the with your best guess is 50 percent, if I choose flip 42, for example, my best guess is 50 percent or 50 percent heads, that’s my – and so that’s different though then if I choose flip 42 because flip 41 is a heads. So if I choose one of the flips where the previous flip is a heads or just a flip that is head and see what the next flip is, the same way of looking at it.
Now there’s something else because the flip you chose, because the previous flip was a head is using information about the outcomes of adjacent flips and that information kind of gets contained within your flip and that’s this is this gets a little complicated but one way to think about as you’ve taken a heads away from the finite number of heads that you have right and you can’t see it again …
RITHOLTZ: You have reduced that data set and the remaining tails now should be slightly higher than heads.
MILLER: It shifts. And there’s another element that uses the current — the way they are arranged that makes it a bit stronger, but that would take some time.
RITHOLTZ: That’s a longer explanation.
This smells to me slightly like the Monty Hall problem, is there any element — in the Monty Hall problem you go from choosing One in three to one in two, so suddenly what was a 33 percent chance becomes a 50-50 why not make the switch, that is a little counterintuitive but once you see the statistics it you can’t unsee utility, you always should make the change.
If there is a – is there a tiny element of this in that?
MILLER: More than tiny, so my co-author Adam Sanjurjo and I also had another paper connecting this to the Monty Hall problem and explaining it via this principle from bridge which is the principle of restricted choice which is essentially the intuition of base rule.
And so the way to think about it in the Monty Hall problem, you have these three doors, right? So you are on this game show, there’s three doors, well let’s make this our problem exactly the same as Monty Hall.
RITHOLTZ: Except it’s 100 doors not necessarily three doors so it becomes much harder …
(Crosstalk)
MILLER: We can make it three doors.
(Crosstalk)
MILLER: S your game show — you are on a game show and usually they have this car and two goats and you got to find the car, and if there was a car behind one of the doors, you got to guess, well let’s get rid of the goats and cars, now, let’s just flip a coin behind each door.
RITHOLTZ: Right.
MILLER: So behind each one is 50-50 but you’re the contestant, the host knows what the outcome of the flips are, you don’t, you want to guess hey where is that where’s the heads, let’s say you want to find the heads. So you guess door three.
Now if you guessed door three, let’s say the host looks behind the door you didn’t guess, door one and two and he’s going to reveal a heads if you can’t — so let’s say the host open storyline and shows you a head do you want to switch or do you want to stay? If you’re looking for the heads, you want to stay, if you looking for the tails, you want to switch, now the intuition is not to be clear immediately but if you think about now the host looking at door one and two used information about both doors to determine which door to open up to. Now if both doors were heads, the host could’ve opened door two.
RITHOLTZ: It doesn’t matter, right.
MILLER: But if it was heads tales, the host had to open door one because the host is going to show you a heads.
RITHOLTZ: Right. He is not going to show you tail because that is what you’re looking to avoid, that’s the goat to you.
MILLER: So the world, we don’t know which world we’re in where the first is heads and the second is tails and the first is heads and the second is heads.
But the world where it is heads tales is the world where the host is more restricted, the host as to open door one, and so the world.
RITHOLTZ: And you should avoid door two in those circumstances because it’s a higher probability of …
MILLER: If you’re hunting for the heads, you should avoid door two because tails is more likely because in the world of heads tails, the host had to open-door one right and it’s …
RITHOLTZ: Right, so it’s a higher probability so that door three even though it’s a coin that’s flipped independent of the other two when you’re dealing with that data set, you’re better off with three because of the circumstances that led the host to pick one and not pick two.
MILLER: Right.
RITHOLTZ: That makes some rational degree of sense. Once you get the Monty Hall aspect of this, it makes a whole lot more sense, it just seems, it is quite fascinating.
We were discussing on the coin flip issue and the hot hand scenario, let’s circle back to that hot hand and the original research, the original research said that if there’s a streak of three hits in basketball or three misses in basketball, the odds of the next shot going in or not is whatever the shooters historical shooting percentage is which sort of seems that there’s no hot hand, but that presumes after streak that their next shot should be dead center in their percentage, you found out it should be worse than that. Explain.
MILLER: Exactly so that’s a counterintuitive thing, if you go out and watch a player shoot a basketball you look at their shooting percentage after the streak of hits and compare it to the shooting percentage after a streak of misses and you find that it’s the same, then that, the intuitive thing is to say, they have the same rate but actually you would expect them to do worse after hitting…
(Crosstalk)
RITHOLTZ: Explain that because that is the most fascinating part of it. Someone’s on a shooting streak where you take a data set of the whole run of shots, what do you find after the streak and why is that?
MILLER: Yes.
RITHOLTZ: So you said you find their percentage actually goes down after a streak?
MILLER: In the world where there is no hot hand where they are a consistent shooter, the percentage will go down after streak in the data not in reality, the probability is always the same as we don’t observe the probability, we calculate the percentage and that’s where the biases come in.
RITHOLTZ: Right.
MILLER: And so the original authors found that the shooting percentage was around the same, and that’s correct, we go and we check and they were right, they report, they did all the analysis in that sense, the calculations correctly, but that the mistake is understanding the benchmark, you have to go out and say okay now let’s look at the world where we know we can control it.
So on a computer you can say we can generate coin flips, or we can make a player that has no hot hand and then look at how that player does and we and analyze the data and we realize of they should do worse after a few in a row, so once you adjust for that bias, you find out that actually if they’re doing the same, that’s indicative that they’re doing about 10 percentage points or more better after hitting a few in a row than missing a few in a row, and that’s huge, that’s like the difference between the median and the best NBA three point shooter.
RITHOLTZ: Thereby confirming the hot hand.
So I have to challenge the data set because again, everything about this each step along the way is so counterintuitive, so why would we expect the shooter who is on a streak, who is in the zone who has a hot hand, whatever we want to call it, why would he we would expect his shooting percentage to be lower after they hit several shots in a row?
MILLER: Why would we expect it to be lower of a real human or for …
(Crosstalk)
RITHOLTZ: For anybody, for a professional, for a real human, when you look at the data set of here’s all the NBA streak shooters or all the NBA shooters, what does the data show after a streak their shooting percentage actually becomes.
MILLER: So if you’re talking about live-action games, we have those issues that we spoke about, the defense will adjust and so that becomes a little bit more complicated.
RITHOLTZ: So let’s talk about three-point contests…
(Crosstalk)
MILLER: So if the hot hand didn’t exist in a world like that, we would expect players to shoot worse after making a few in a row in the data.
RITHOLTZ: So simply just mean reversion is that what it is?
MILLER: It’s not mean reversion, it’s the same thing we talked about with the coin flips.
RITHOLTZ: Right.
MILLER: And so as a researcher, you’re taking the data after it’s already been generated and you are picking through it you’re looking only at the events that you are interested in, right? You are looking at the probability of success given recent success, you are just picking out those events when they had recent success, let’s say where they just made three in a row.
RITHOLTZ: So you are changing the data set, so now there is three less, so someone shoots three in a row when we’re looking at the data sat, let’s they’ve shot 20 shots and after three in a row, how they do? Well guess what? You pulled three hits out of the said meaning there’s a disproportionate number of misses left.
MILLER: Yes, that’s part of the bias.
RITHOLTZ: Okay.
MILLER: And there’s this other element we didn’t quite get into is that you can — you have this essentially a stopping rule so as you collect the data, the moment they miss it, you’re not interested anymore in looking …
RITHOLTZ: Right.
MILLER: Because you are going to wait for a streak of hits again.
RITHOLTZ: Right.
MILLER: So you are biased towards stopping at a miss.
So you might get a miss right away and everything you collected in those events 100 percent of their shots are misses because you just collect one shot and they are all misses.
RITHOLTZ: Right.
MILLER: And so you are biasing towards collecting misses in a sense.
RITHOLTZ: Got it. That’s quite fascinating. So what other areas like this are you studying because it’s really — it’s really quite fascinating stuff, are there other sport myths that you’re looking at that have a probabilistic element that’s very counterintuitive or is this pretty much the biggest one out there?
MILLER: This is the biggest one that we’re studying.
RITHOLTZ: A lot of what you’re doing is statistical and probability work at a level that the average sports fan is really not familiar with, forget the live game when you explain relative to a three-point shooting contest, it’s really not so much about the streakiness of the shooter but the mathematics of the data set and I think that is really counterintuitive but it doesn’t seem anyone’s been able to disprove what you and your co-author have found.
MILLER: So there have been a lot of challenges to that original study and legitimate challenges, there are issues with what they call statistical power. So we have a friend and colleague Daniel Stone that made this nice point that you have this thing called measurement error, we want to know how do you do after hitting a few in a row, that’s what we actually look at, but what we’re really interested in is how will you do when you’re hot?
So hitting a few in a row, you are not always hot.
RITHOLTZ: Right.
MILLER: And so you can underestimate how hot someone is if you use only the data that you can observe which is zeros and ones…
RITHOLTZ: Right.
MILLER: So the econometrician, the statistician has kind of a weak measure of that. So you know, this kind of evidence is…
RITHOLTZ: Just the mathematical evidence, do you ever do interviews with players? Do you ever say to them “hey were you in the zone” how did you feel?” How do you find that data set?
MILLER: So that data, the original study looked at data like that, they spoke to the 76ers and they asked them kind of qualitative questions, “did you get in the zone?” and …
RITHOLTZ: “Did you feel hot?” or …
MILLER: They all do.
RITHOLTZ: Right.
MILLER: But it’s hard to work with that, it is just looking at whether they believe in it or not.
RITHOLTZ: Right.
MILLER: But then getting a sense of do they believe in it too much or not, that gets a bit harder because you have to be able to somehow measure, they have to decide when are they hot, so you really need a lot more cooperation from say like a coach or player to kind of sit there and maybe watch the games with you or something like that. That would be maybe of a better way of testing their, you know, their beliefs.
RITHOLTZ: So when Tversky and Gilovich’s original study came out, I’m forgetting the third person.
MILLER: Robert Vallone.
RITHOLTZ: Right, the V in GVT.
When that study came out, there was a tremendous amount of pushback from coaches around the league, we mentioned Red Auerbach, your study comes out and you basically say “no, you professional coaches, you were right, there is a hot hand, there is a streak.” What sort of feedback have you gotten from players and coaches about your research?
MILLER: Well, we’re not entirely sure whether players and coaches were ever frazzled by the original study. So you know validating their beliefs for them is a “yeah, we kind of never believed that result to begin with.”
RITHOLTZ: Right.
MILLER: So we haven’t gone and sought the opinion of you know, players and coaches because it’s not so clear how far that original conclusion reached into that world. Well it did, especially you can see announcers mentioning it, but yes.
RITHOLTZ: So when you, what about some fourth quarter the outlier players, if you look at a Michael Jordan or a Steph Curry, guys who literally become just unconscious and what Reggie Miller is another one and the most improbable shots on a consistent basis start to drop, when you look at players like that, do different players seem to have a different set of streakiness, a different hot hand? Can you can you calibrate how much of a hot hands different players have?
MILLER: So using game data that’s a bit more of a challenge, so my co-author Adam and I looked at Spanish semi pro players, we could collect a lot more data, we had more than cooperation and there seemed to be a clear difference with players and the obvious one is that you centers and forwards, people that don’t shoot that often…
RITHOLTZ: Right.
MILLER: It’s hard for them to get on a roll.
RITHOLTZ: Right.
MILLER: Because they had to be consistent when they are kind of not that consistent when they shoot.
RITHOLTZ: They don’t touch the ball all that often and all that long.
MILLER: Right and so those are people you expect maybe, they can’t really sustain a streak and that’s a we find, there are some players that can and some players that seem like they can. If we go to real NBA players, that’s a bit of a challenge, so we have looked at the three-point shooting contest, and we have a paper on that.
The issue with the three-point shooting contest is a lot of players don’t have much more than say 100 shots total in the contest, some have a few more you have a Craig Hodges who has over 500 in our data and we find evidence there, but what we can say is that among all the three-point shooting contest contestants, there were way more that did better after a few in a row than — hitting a few in a row than missing a few in a row than you would expect, but you don’t really know which of them are really hot. You just know there’s more than you would expect but you need more data to be really confident when you pick out an individual.
RITHOLTZ: So it at this point in the state of research on the hot hand, do you have any doubt that the hot exists?
MILLER: I don’t have any doubt that the hot hand exists, what you mean by the hot hand is where the doubts come in because there’s many different mechanisms that can lead to evidence in the data that your success after recent success is higher, than rates higher than after recent failure.
RITHOLTZ: So the confidence factor, the endorphin factor, the further pressure that the other team is placing, all those things add up. You ask a player, they are going to say, yes, of course you get hot. But when you ask a statistician, the data supports it as well.
Quite fascinating.
We have been speaking to Joshua Miller he is an economics professor and researcher at the University of Alicante in Spain. If you enjoyed this conversation, well be sure and come back and check out our podcast extras where we keep the tape rolling and we continue discussing all things statistical, sports, and behavior. You can find that at iTunes, Overcast, Stitcher, Bloomberg.com, wherever your finer podcasts are sold.
We love your comments, feedback, and suggestions. Write to us at MIBPodcast@Bloomberg.net, you can follow me on Twitter @Ritholtz or check out my daily column at Bloomberg.com/opinion.
I’m Barry Ritholtz, you’re listening to Masters in Business on Bloomberg Radio.
Welcome to the podcast. So Josh I have to tell you, I was very much a skeptic, a little background, so first, I’m a fan of Gilovich for a long time, when I — when I started in this business of hundred years ago as a trader, it was before the bad old days of behavioral economics had made its way to Wall Street and I found a book by Gilovich “How We Know What Isn’t So. ”
It was the first mass book or popular book not that it was all that popular, but it was the first book for a popular audience that had an enormous behavioral finance component to it. So I found him absolutely intriguing, he led me down the rabbit hole of behavioral finance and it’s been an enormous influence on my professional career, because very often when I couldn’t figure out what the hell is going on according to what the head trader was saying, behavioral finance gave a much better answer and the same is true when you’re looking at markets with the economy or what people get wrong.
So my bias was to say, Tversky, Gilovich, these are two legends, of course, they’re right.
But I have to tell you, this having gotten through as much of your paper as I could until the formula starts to show up, it’s a compelling argument that we look at the data set, players on a streak from — within that data set should have a lower shooting percentage following three in a row than you would intuitively inspect — expect, and when they don’t shoot worse, in it of is evidence of the hot hand, it’s such an eloquent and unexpected way to do the analysis of the hot hands.
I have to ask how did you guys come upon that? I mean I would never, I’m not a statistician but I would never thought, because so much of it is so intuitive I would not have thought hey let’s look at what the expected shot is because with coins, it should be 50-50, why would you expect it to be anything less following three in a row, how did you sort of work your way towards that research?
MILLER: So you both my co-author, Adams Sanjurjo and I, we didn’t see any problem in that respect with the original paper, so we didn’t say “Oh, they are clearly making a mistake here.” No one did. As soon as we discovered this thing, we’ve gone and we’ve asked statisticians, people that are very good, they look at they said, oh, you know, maybe he’s underpowered or they might have some little quibbles but they don’t have any expectation that you would shoot worse after a few in a row.
In order to do that, you actually have to go out and simulate or go sit down and really calculate and so it doesn’t strike you in anyway. So we discovered it, it was a bit of a stroke of luck, we were looking at the NBA three-point contest data, we had to analyze it very quickly using a method different than the way we used it so we just use the method of the original study which was much quicker to run.
So we ran that and we found this player who we knew was hot, and I mentioned that earlier, Craig Hodges, and he shot no better after making a few in a row and that just didn’t make sense.
RITHOLTZ: Was that a brute force quick down and dirty and so you move to something a little more sophisticated. What’s a better word for this?
MILLER: Yes, so the sophistication came later, so we just took the test using the original study and that measure and it wasn’t showing anything and we — that didn’t agree with our perception of what we saw in those videos, and some of the elementary things he did, like he hit 19 in a row at one point, never missed more than five and he was around a 50 percent shooter which will be you never expect from a coin.
RITHOLTZ: Right…
(Crosstalk)
RITHOLTZ: 19 in a row is astonishing.
MILLER: Yes, it’s incredible, so then we went and we said. Well, what if he were a coin, what if Craig Hodges was a coin? So let’s just generate his shots as if he was a coin, where he is really 50-50…
RITHOLTZ: Right.
MILLER: And repeat this, like imagine we do this many, many times and look what we’d expect from all — you know, if we run this many times and we see, you actually shoot worse after making a few in a row, and that seems very — we were struck, this doesn’t seem like it’s right but this is what the analysis is giving us, we have to understand this.
RITHOLTZ: This is what the data is saying. So two things we discussed, one is after you have a streak of six in a row and you have a finite number of shots, well now there are six less heads in that group so therefore there’s a higher probability of tails after that, that makes perfect sense, right? Because you are just changing the remaining data set by what you are looking at.
MILLER: Given a fixed data set.
RITHOLTZ: Given a fixed number of coins, fixed number shots.
And then of course mean reversion assumes after a long streak of heads you should start to see the streak of tails…
MILLER: That’s a gambler’s fallacy a little bit there…
(Crosstalk)
RITHOLTZ: So let’s go into that, explain that.
MILLER: Yes, so I mean the gambler’s fallacy is this idea that comes out of the casino and it’s been known for hundreds of years.
RITHOLTZ: Right.
MILLER: That if you see say five, six blacks in a row at a Roulette table it feels like that the red must be more likely.
RITHOLTZ: Right.
MILLER: And so people get drawn into this and they start betting more, maybe …
RITHOLTZ: But it is still 49, 49 and the …
MILLER: Yes, it’s like…
RITHOLTZ: It’s almost 50-50 regardless.
MILLER: Right so in reality the probabilities haven’t changed.
RITHOLTZ: But when you look at a fixed data set that you expect to be 50-50, not prospectively at the roulette table in real time, but we know that “Hey, there’s 100 coin flips, we are going to assume half of them are tails and half for heads after you’ve had a wrong — long streak of heads, the assumption is that out of the full data set, there should be more tails coming up. In real-time, it’s truly the gambler’s fallacy, but when you are looking retrospectively with the data set, it is basically just a variation of “hey you have already exhausted a lot of heads therefore there are more tails out there.”
MILLER: Yes, exactly.
And as we mentioned before, there’s a little bit, there is the extra wrinkle on top that determines on how these streaks ordered so when you pick up a shot because the previous three were heads, the shot you pick is either heads or tails but it’s more likely to be tails, one because of the heads that were removed.
RITHOLTZ: Right.
MILLER: Two, because if it were a tails, you’ve interrupted the streak and you can’t begin until — you have to wait until you begin.
RITHOLTZ: So you are pulling a big chunk of the possible selections out so all the streaks come out, they are all heads, so you’re not picking that one and plus the total number of heads that you have used, so what’s left becomes just from a data set group with less to become a 40 percent not 50 percent probability, which is fair. So you guys are doing this research, at what point do you say, holy cow, this is really a fascinating discovery, like it’s not just a tiny –10 percent is a huge number in this sort of data series, when did you guys look at each other and say hey this is something really important?
MILLER: We knew it was a big deal the moment we saw it.
RITHOLTZ: Really?
MILLER: We were on the phone, we were…
RITHOLTZ: You didn’t say to yourself, this has to be wrong, 10 percent, how did nobody pick this up? In 30 years, nobody has seen this?
MILLER: So we had – this is two years after we’ve begun the project, well ,maybe not that long, but almost two years and we had read every paper in the literature, so we knew on one had….
(Crosstalk)
RITHOLTZ: Nobody had seen this before.
MILLER: No one had seen, so we knew it was a big deal for that literature.
So the only question we had is hey, how new – well, we run the – you know, we can trust the computer, right? And of course you have to make sure you didn’t make an error in your code.
RITHOLTZ: Right.
MILLER: So you have to sit down and do the simple example to make sure you didn’t do a calculation error. So once we did that, we said, okay, this is clearly a true thing, now the only question is did anyone noticed about coin flips before, is this a new discovery about coin flips? And yes there are some mathematical things that are somewhat related, but no, it was even new in that dimension so we knew we had something really big.
And that was exciting because you have this moment where you are the only person in the world who knows something and it’s kind of — it’s an exciting moment.
RITHOLTZ: I feel that way every day I wake up and I have that sensation so I can appreciate you probably not as solidly based as yours, at least that’s what my wife would tell me.
(LAUGHTER)
So that’s amazing, you guys come up with this incredible breakthrough, nobody has found this, it’s been decades and it’s been widely accepted, it’s become part of the canon but it’s classic confirmation bias which is so reflexive and meta, there is a study that says people are fooled by randomness and think there are streaks which turns out perhaps to be confirmation bias by behaviorists who are warning people against being fooled by randomness and seeing what they want to see.
It’s got a little bit of Mandelbrot reflectiveness built into it, it’s it is quite amazing.
MILLER: Yes, so in a sense that mistake proves kind of the spirit of the general point about misinterpreting randomness.
RITHOLTZ: Yes.
MILLER: Even the best of us, the best researchers that are out there still make these mistakes due to randomness and while saying others are making the mistake, you’re making the mistake …
RITHOLTZ: Even within — so they accidentally prove their point which is it’s very easy to be fooled by a random dataset into thinking there is a broader conclusion there, until subsequent research discovers that “hey, this isn’t quite as random as you think it is, there’s a 10 percent gap between true randomness and the remaining data set. That’s quite fascinating.
So you guys look at each other and say, hey we’re onto something real, how do they progress from there? What year was this? This was …
MILLER: This is February 2015 …
RITHOLTZ: Okay.
MILLER: When we found this.
And we knew so we knew it was important, so we presented our work and when you see the eyes light up, you realize it’s even bigger than you thought it was, and then you realize hey wait a minute we don’t have the paper yet and now other people know about it.
RITHOLTZ: Who did you present it to originally?
MILLER: So Oxford University, that was the first kind of reveal …
RITHOLTZ: Right.
MILLER: And you see the eyes light up in the room …
RITHOLTZ: Are you genuinely concerned at that moment, uh-oh, someone is going to try and beat us to publication.
MILLER: Yes. And so we put everything aside ..
RITHOLTZ: Right.
MILLER: And we just, we went to the grind, within two months we had the paper and …
RITHOLTZ: No one was going to catch you at that point, you had enough of a head start and you were the original people who found this. So two months later, the preliminary papers come out…
MILLER: We put it online.
RITHOLTZ: You posted online in BER and everywhere else so…
(Crosstalk)
MILLER: Wherever, just get that timestamp.
RITHOLTZ: Wherever finer white papers are sold. And so that’s what – April of 2015?
MILLER: So the paper went online in June of 2015.
RITHOLTZ: Okay.
MILLER: That was our first.
RITHOLTZ: What’s the response to that?
MILLER: The response was big, so a statistician at Columbia University, Andrew Gelman who has this blog …
RITHOLTZ: Everybody has heard of Andrew Gelman, or let me rephrase that, anybody who’s interested in statistics knows who Gelman at Columbia is. Fair statement?
MILLER: He’s at the crossroads of pretty much all the social sciences when it comes to data and statistics …
RITHOLTZ: Right.
MILLER: And so getting attention from Andrew Gelman.
RITHOLTZ: Huge.
MILLER: It’s huge, and that was …
RITHOLTZ: High fives all around.
MILLER: Yes, but it’s also scary when you get attention from Andrew Gelman because if you made a mistake, it’s open peer review season.
RITHOLTZ: Right.
MILLER: They are getting in there in the comments, he’ll get to, you know, they are just having fun, they love talking about data and they are not going to worry about how you feel about it because they are just interested in the main points like what does the statistics say, and you are sitting there sweating bullets hoping you got — you didn’t make a mistake somewhere.
RITHOLTZ: It’s – at that level, this isn’t twitter fights and ad hominem attacks, it’s “Hey, let’s get into the math, let’s see if they did their crunching the numbers correctly, let’s see if we can find an error in their modeling, what did Gelman discover?
MILLER: So Gelman went and did the work himself and he found — what he found agreed with what we found and so he said “hey, guess what, there is a hot hand” that was his post and then it kind of snowballed from there.
RITHOLTZ: That’s it, so then there’s a Wall Street Journal piece on it …
MILLER: Yes.
RITHOLTZ: And then there was an ESPN or Sports Illustrated, one of the…
(Crosstalk)
MILLER: At that point it was….
(Crosstalk)
MILLER: We were like okay, when is the 15 minutes going to end?
RITHOLTZ: Right.
MILLER: But I guess you know, the news world I still so kind of Balkanized by this point that like…
(Crosstalk)
RITHOLTZ: It hops from subject to subject, it just kept rotating and — I saw something and you guys published another — a number of fair I have to say, your published popular stuff I think you undersold the math on this because it’s not that you dumbed it down, it’s that you were so circumspect and so maybe modest is the right word, like I’m a different person than you, I would have written something that said “Dudes, listen up, the whole no hot hand things, let us show you why that’s not true, here’s the math, it’s 10 percent, it’s a giant impact and here’s why.”
Like I thought you guys were very circumspect in your, what was it the conversation or the…
MILLER: Yes, the Australian …
(Crosstalk)
RITHOLTZ: That was like a fairly modest discussion, you know, I would’ve been like “Hey, pay attention to this, we’re changing our understanding of sports streakiness, this is a big deal.”
What other applications are there of the finding of both the flips of coins and the streakiness of shooters, where else can this be applied, are there other uses of this mathematical or I should call it statistical observation?
MILLER: Yes, so that the bias that that we found has an — it can manifest itself in many areas, it’s not just about time, right? So we are looking at like how you did recently, does that affect how you do in the future, how you do next, right? We found some biases there, but it’s not about time, it’s essentially about space because you’re looking at data and we represent time with space because we have, you know, period one, period two, period three, they are all next to each other.
RITHOLTZ: Right.
MILLER: And so you have this kind of one-dimensional spatial thing …
RITHOLTZ: Continuum along the line.
MILLER: Along the line.
But it can go in either direction, it’s not you know, times arrow that’s determining it.
RITHOLTZ: Right.
MILLER: If I hit three in a row, the chance of the previous one that just preceded that streak is a heads is actually lower too.
RITHOLTZ: For the exact same reason.
MILLER: For the exact same reason.
RITHOLTZ: Which means that the actual streakiness of the players and relevance to the prior one even though we would expect to be relevant to the subsequent one it’s all the same statistical data set prior less heads in the remaining pool et cetera.
MILLER: So if you can extend this beyond time and talk about space, right? So if you’re interested — you know, if I’m surrounded by red people, am I more likely to be blue? You might go and “Hey, let’s look at the data set” and see …
RITHOLTZ: This is the ping-pong balls in the vase statistical problem.
MILLER: Yes so you know the people study segregation and clustering…
RITHOLTZ: Right.
MILLER: And where people live and things like this and so you might go into a data set and use this intuitive measure like let’s see if I’m more likely to be blue if I’m surrounded by reds.
RITHOLTZ: Right.
MILLER: You have the same issue here. Now, you know, if I were a blue, I’ve kind of excluded other possibilities of being surrounded by reds wherever that blue is, that actually makes blue more likely for some of the same reasons why we have this bias, you know, when we’re talking about time. And so there are potentially many other areas where biases similar to this could manifest themselves, so we are going to be stumbling in…
RITHOLTZ: So I remember a couple years ago the cancer clusters around power lines which a lot of statisticians came out and said, well, no, this is just the heads and tails problem again, you have all these non-clusters around other power line so if it’s a causal element why is it causing it here but not half mile down the same power line, it’s just a random aggregation of data and you’re seeing something that it of course, you are going to get 10 heads in a row if you flip a coin a million times, that’s all you’re seeing, do you have an application to those sort of cognitive issues?
MILLER: So we haven’t found a specific application, to be honest, we haven’t and scoured that literature. We have found papers that have measures of clustering like how likely am I going to live next to someone of who is like me versus not like me depending on who’s around on and there are some measures that are biased for a similar reason that we have this bias.
Now in the cancer cluster one, that’s a little bit different, and it’s because you what you say is it’s got a blade of grass fallacy, there’s lots of blade of grass, you shoot a speck of water and it hits the blade of grass, the blade of grass, look, I’m so lucky it was coming for me. It had to hit some blade of grass…
RITHOLTZ: Right.
(Crosstalk)
Someone’s got to win the lottery….
(Crosstalk)
MILLER: Someone got to win the – you know, the chances that somebody wins the lottery is super high…
RITHOLTZ: The chances that it’s you, not so much.
MILLER: Yes.
RITHOLTZ: So that’s interesting. So before I get to my favorite questions I ask all my guests, I have to ask you what else are you guys working on? What other research is coming from the minds that brought you proof that the hot hand exists?
(Crosstalk)
MILLER: In our world, it is very tempting to move on to the next one before finishing what you started.
RITHOLTZ: So you have not exhausted everything out of this one piece of data.
MILLER: Right. So we have a lot of your I’s to dot and T’s to cross but you know, a little bit more than that, you want to finish and get the message out but also share the other insights that we have because you know …
RITHOLTZ: They come out of the same …
(Crosstalk)
MILLER: So this is our – you know, so the main – and but there are other very subtle and interesting insights that we have because when you master something you and you come back after working on something for a while, there’s a lot to share.
RITHOLTZ: So tell us about what other insights can be derived from the hot hand paper.
MILLER: So there’s another result in that study, the Gilovich and Tversky study that Gilovich mentions in the book that you talked about earlier which is okay, so they kind of got the people you know that they measured hot hand in a certain way and ate they realize will maybe were not capturing everything that means the hot hand and maybe some players are seeing something that we, the statisticians, the econometrician, you know, aren’t measuring, and if they had people predict and bet on outcomes and they found that their bets don’t really correlate with the outcomes.
And so that’s kind of evidence, what okay what even if we’re not measuring everything look, if the players were seeing something, you think they would bet successfully, and that you could also take that as evidence that now that there is a hot hand there, at least it’s evidence that they’re somehow not using and exploiting it in a profitable way but there is actually a mistake as well in that analysis, which is even if someone were perfect at detecting the hot hand, they knew they, and you could imagine Anne and Bob, Bob is the shooter, Anne is predictor, she’s observing Bob, she knows when Bob’s hot and whenever Bob’s hot, she’s going to predict that Bob’s going to make the shot.
RITHOLTZ: Right.
MILLER: Now, you would expect she’s good then — that good, then her bets or predictions are going to correlate really well with Bob’s outcome.
RITHOLTZ: Right.
MILLER: But actually you wouldn’t expect that, and that’s another counterintuitive thing is that while she’s perfect at detecting his state, the outcome of the state is noisy, you are just getting one draw from Bob’s ….
RITHOLTZ: Right.
MILLER: Even if Bob is moving from a 70 percent to 85 percent probability shooter, if you only take one draw from that urn, are you not getting a very good signal of Bob’s state.
RITHOLTZ: You need millions.
MILLER: You need a lot.
(Crosstalk)
MILLER: Yes, and even if you are getting many predictions from Anne and Bob, you are still only getting one drawn on each one and so the evidence that they have, there was actually enough evidence that is consistent with Anne being very good at detecting it. and actually with you reanalyzed the data, you find that Bob shoots seven — around seven percentage points better when Anne predicts that he is going to make it…
RITHOLTZ: Really.
MILLER: Than Anne predicts that he’d miss it.
So in their data, to the next message so in their data they have real people that are paid, basketball players are betting on each other’s shots and that’s the evidence that we find.
RITHOLTZ: That’s quite interesting. That sounds that betting on the outcome of the shot sounds very much like fund managers selecting stocks for a portfolio. Have you applied any of the hot hands to how do fund managers do when they are on a hot streak or a cold streak? And there’s a ton of mean reversion in that data series.
MILLER: Right. So we haven’t gone and analyzed. The mechanisms for being hot in the financial world are going to be quite different than in the basket world, right?
So you know one way of thinking …
RITHOLTZ: You look for SEC indictments, you look for, no I’m just kidding.
For sure, it becomes so affected by such large macro things it’s hard to give credit or not.
MILLER: Or your model of the world happens to be uniquely fit the current situation and you recognize that.
RITHOLTZ: Right, and that may or may not be temporary.
MILLER: And that’s going to – you know, that will probably expire but that’s a very different mechanism than say how it would emerge in say a basketball game.
RITHOLTZ: So I interrupted you, what else did you do see is an application of this elsewhere?
MILLER: An application of our …
RITHOLTZ: Of your — of what you’ve discovered to the world finance.
MILLER: To the world of finance.
RITHOLTZ: To the world of finance.
MILLER: So the immediate applications are maybe not so much but if you think about people picking stocks and say not so much investing but someone wants to prove that they’re good on at predicting when a stock is going to go up or down. You have to pay attention not to how often they are right but how much money they make when they are right and when they are wrong because it’s very easy to game these things. So if I were to say but it’s 50-50, I want to prove that I’m good at predicting coin flips…
RITHOLTZ: Right.
MILLER: And so every month you know, a coin is flipped each day, the stock goes up or down but I only bet when there’s three heads in a row and the stock goes up three times in a row and I bet that it’s going to go down, right? In any given month, I’m going to be right more often than I’m wrong. And so I can game know if you don’t — if you brackets the month level, I’m going to be more often right in certain months and it is going to look like I’m doing well but the thing you haven’t paid attention to is how often I was right and how often I was wrong in the months that I did poorly.
And so if I’m always betting that it’s going to go down when there’s a few ups in a row, there’s going to be those months where it keeps going down, right? But you know, there’s going to be few of those months, and there’s going to be many months when I did well but I didn’t predict very many times and so you’re not controlling for often I predicted and so it looks like I do really well but if I were to bet, I wouldn’t be making any money because I would be losing a lot of money in the months where I didn’t predict that well and only winning a little bit of money in the months that I predicted well.
RITHOLTZ: The interesting thing is if you talk to active traders who have been successful, they’re not aiming for 50-50, they are aiming for those opportunities where a trade becomes a winner and they don’t sell too early, so it’s not your batting average, it’s how far the ball goes when you actually hit it.
MILLER: That’s right.
RITHOLTZ: Meaning you could have a 20 percent winning trading record but in terms of percentage of winning trades but in terms of dollars won and loss, those 20 more than make up the remaining 80 and I always find a lot of new traders don’t understand that, they think they’re hitting for percentage but they’re not. They are hitting for distance to bring in a different sports metaphor.
Anything else you want to share about the researcher or what you guys might have coming out in the near future you and your co-author?
MILLER: Off the top of my head, no, I think we the one thing that we will say is actually there’s one thing, so it’s not simply that we found that the original analysis was invalid and the original conclusions were invalid. If you go back and you reanalyzed that data, you find that players shoot a lot better but is not simply in that data set, and so we have gone and collected many other data sets that replicated the original, so they got the same conclusion because they had the same bias that the original study had.
And so when you go back and you fix that, you find evidence everywhere and that’s — we have a paper that were finishing that’s showing how robust our conclusions were.
RITHOLTZ: So I know that they’re all sorts of interesting awards for mathematical and statistical research, are you looking at applying for any of these, how does that work? Can you self nominate? Does the institution have to nominate you? How do you say – does that process work, have you guys thought about this at all?
MILLER: It’s not something we’ve thought about.
RITHOLTZ: Well, let me plant that seed and if this is significant enough, you should apply for either a grant or a mathematical award although most of these, you have to be nominated by other people but how hard is that to have your department chair nominate you? That’s easy enough.
I have to ask, I didn’t ask this earlier, you grew up in California and you went to Santa Barbara, how did you end up in Spain?
MILLER: So I think you remember 2008 2009.
RITHOLTZ: A little bit.
MILLER: So that academic job market was an interesting one I was going on the market in late 2008 2009, and so a lot of academic appointments, not appointments, advertisers for positions were disappearing because of the crisis …
RITHOLTZ: You would think academia is with large endowments and what have you somewhat is insulated from the vagaries of the stock market and even the broader economy but apparently not.
MILLER: Yes, and so my advisor came to me and said look, this year everyone’s applying everywhere so you need to apply everywhere even if you weren’t thinking about it. So at that time, I applied everywhere and it was great because it opened my mind to the great opportunities that are there.
So I moved to Italy in 2009, that was my first stop.
RITHOLTZ: Where were you in Italy?
MILLER: Bocconi University in Milan.
RITHOLTZ: There are worse places in the world to ride out a recession.
MILLER: It was good times.
RITHOLTZ: I can imagine.
And the from Milan, how do you end up in Spain?
MILLER: So I wanted to join my co-author and finish our work.
RITHOLTZ: Is that where he was located?
MILLER: Yes, and he still is so we are both at the University of Alicante.
RITHOLTZ: On the lovely Mediterranean Sea, again, parts of that whole Mediterranean coast is just spectacular, isn’t it?
So you don’t miss California too much?
MILLER: I get back a couple of times a year.
RITHOLTZ: Quite interesting. All right, let’s jump to our favorite questions. I can’t believe you guys never thought of saying hey maybe we should apply this for some of these grants and some of these award …
MILLER: We think of applying to grants but the award thing, I don’t know.
RITHOLTZ: All right. I always thought academics had to do stuff like that in order to maintain their academic standing.
MILLER: For grants, you do.
(Crosstalk)
RITHOLTZ: Grants and…
MILLER: I do apply for money, that is for sure, but the recognition is a good idea.
RITHOLTZ: Yes, not a bad idea, let me a — when you give your acceptance speech.
(Crosstalk)
MILLER: Definitely.
(LAUGHTER)
RITHOLTZ: All right, so jump into our favorite questions which I’ll modify slightly because most people I think don’t know you personally so let me ask the question, what’s the most important thing that your friends and family don’t know about you?
MILLER: Friends and family so you’d mention this question earlier and I was going to say the most important thing we’ve actually already revealed, which is it’s what most people didn’t know, we haven’t shared this much is that what this might be in this research, the first joint Eureka moment. Usually someone discover something, even if it’s a simultaneous discovery, someone will discover something at one point in time and somebody in another but no one knew about it…
RITHOLTZ: The electric light bulb is a classical, radio is another classic example.
MILLER: So but my co-author and I, we aren’t in the same room, we’re on the phone at the same time we both had the goal is there and if it happens, it happens.
RITHOLTZ: So who are some of your mentors in your early career?
MILLER: So my early career I would say and my co-author would say the same, you know, my advisor, Aldo Rustichini, he is a professor at University of Minnesota, and he just – he is a neuroscientist, he is a mathematician, he is an economist, he is all about the science.
RITHOLTZ: Right, just like a Renaissance person.
MILLER: He is a Renaissance person, and you know when you see that and you see someone that’s just you zeroed in on that and you see how they work, you kind of – you get, you can absorb what they do through osmosis, and my co-author would say the same say the same thing about his advisor, Vince Crawford at Oxford University, a very deep guy, very brilliant, both very brilliant and those are formative years when you are in grad school.
RITHOLTZ: For sure, for sure.
So what other behaviorists and statisticians influenced your approach to thinking about the mathiness of things like shooting streaks?
MILLER: So that I would say the statisticians that have influenced me out there is one, there is Andrew Gelman, reading his blog has been eye-opening for so many people and he just introduced how to think about data in a way that most people don’t get in the formal training…
RITHOLTZ: Right.
MILLER: Because he is dealing with real practical examples all the time, and so I would say he’s been one of the biggest influences.
RITHOLTZ: Interesting, what about on the behavioral side?
MILLER: On the behavioral side. there are just so many great you know, there was this Vanguard of the people that the folks that came in the 80s that really had to fight through you know that the review process of the all the skepticism towards why is psychology relevant to economics, why are these other social science disciplines, what do they have to say about people really had to fight a lot of skepticism…
RITHOLTZ: Give us some names. I’m putting you on the spot.
MILLER: Okay, so the people that had to fight through that I mean, Amos Tversky and Daniel Kahneman were very influential but they …
RITHOLTZ: They were within psychology.
MILLER: They were within psychology, they were fine, so the people that had to deal you know what this kind of push back, I mean, you say that you know, Richard Baylor you know as much as you he he’s been a bit skeptical of our work but you have to respect, you know, both his, the insights he has into human behavior and also just what he had to fight through to get listened to.
RITHOLTZ: So he was a guest and my favorite quote from him was early on he decided he would never convince his peers, so he thought I’m going to bypass them and just try and convince the grad students and we’ll just wait it out, after enough funerals, we will have one, and it’s really he turned have to be quite true if you — if you are influencing the next generation, that’s far more impactful then what Tversky said about winning all these arguments and convincing nobody. It turned out to be very clever.
Anybody else you want to mention from that group or …
(Crosstalk)
MILLER: No, I wouldn’t want to single out any person. You know, if you look at the people that really kind of did a lot of the fighting that push the ideas through but in terms of the idea level, there’s so many.
RITHOLTZ: Right.
MILLER: So even in – you know, when we were presenting our work, we had somebody that say like Con Cameron (ph) Cameron Caltech, he can …
RITHOLTZ: I actually just met him at a conference and he’s really a fascinating dude.
MILLER: Yes and I you know there’s a lot of similarities that I recognize in him and it’s similar to the advisor, Aldo Rustichini at the University of Minnesota, just – he is all about the science and really, I mean he came to one of our talks and he brought up footnote 72 which is like the weakest point, that point he really wanted …
RITHOLTZ: He found it.
MILLER: He found it.
(LAUGHTER)
MILLER: We had a nice discussion about it, he saw our perspective after we had a talk and we were like, wow, he’s really taking this seriously, and it’s just, it’s nice to see that.
RITHOLTZ: That’s got to be so delightful. I believe we have him teed up for the spring as a guest. Yes, he’s really, I love the work he does with virtual reality and showing people in incredible detail what they are going to look like when they’re older and it affects their decision-making dramatically in terms of planning not just like a computer generated picture that’s been aged but when you have this immersive VR experience of here’s your life when you’re 80, it leads all sorts of amazing changes when you’re 40.
It’s quite astonishing.
(Crosstalk)
RITHOLTZ: So I’m glad you brought that up, so I’m going to put down Gelman as one of those people who influenced your approach to statistics, let’s talk about books, what are some of your favorite books?
So I’ll pick out a book, we could pick out a lot of nonfiction books and a lot of books like that, it hits you at the right time and then if I tell you that book, I might, you know, if I were to look at it now, it might feel trivial, obvious things, like you never know if the book is targeted for the right person, so I will bring up a book that both my co-author and I were lot very much influenced by with its literature.
RITHOLTZ: Right.
MILLER: So there’s this book called the Alexandria Quartet by Lawrence Durrell, we both read in our university days, and it’s kind of at the blind man and the elephant but for human relationships and it has really novel ideas. Four books, the first three books are about three different perspectives on relationships and events that happened in a particular time in Egypt before World War II, and from different peoples perspectives and so that’s the space kind of – so this was inspired a bit by Einstein relativity, so they took three different perspectives in space and then they go forward in time and do the reflection back on those relationships and it gives you this kind of humility, see like how small your perspective is.
RITHOLTZ: Right.
MILLER: How much misinformation you have about what is happening and it’s a nice read, at least it was when I was in my 20s.
RITHOLTZ: That sounds quite fascinating, this question is what people ask me more about than any other question because they want to get a book recommendation from somebody who’s accomplished something, done something interesting, has some experience and when someone says oh and by the way this book is worth reading, it’s the greatest endorsement anybody can ever get.
So I’m going to press you and say give us one or two more books even if you think they may have been very time specific to you.
MILLER: So Duncan Watts has this book called “Everything is Obvious.”
RITHOLTZ: Yes.
MILLER: Beautiful book.
RITHOLTZ: It’s so interesting, it’s all about client-side bias and how you see things after the fact that it’s — you’re the first person who’s brought that book up and I find it — I love the cover with the wheel, I think that’s a triangle instead of the circle, it’s really a very fascinating book.
MILLER: You know you have this curse of knowledge that once you know something, it is obvious to you, you can’t imagine how not obvious it would be to someone else.
RITHOLTZ: Right.
MILLER: And the references in that book, I mean he’s an academic so he’s really given you know the roadmap and like if you want to go beyond that book, all the references are there in that book, it is great.
And then I would say another book that is along the lines only because in the last year, I read it, “Super Forecasting” by Philip Tetlock.
RITHOLTZ: Another prior guest, delightful.
MILLER: It gives you humility to kind of realize how – if you want to start projecting three years five years…
RITHOLTZ: You are wasting your time.
MILLER: You are wasting your time.
But you know, there’s a lot more than that. The disciplined approach, right? So it is not simply, and one mistake you can make is that you mistake I made say around the financial crisis time, I really was convinced Citibank would be bailed out, they wouldn’t let Citibank completely crash.
RITHOLTZ: Well, you were not wrong, they did bail them out.
MILLER: Eventually.
RITHOLTZ: It was $2 at the time, but still. They were bailed out. Had you said the same about Lehman Brothers, that would’ve been a different situation. What’s most fascinating about Tetlock talk about recognizing your own issue, Tetlock’s original book if you go back was on expert political judgment and that nobody is good at forecasting and he then over time, led to what led from him going from “Oh, we’re really bad as a species at forecasting” to “But a handful of people have do a number of things that make them better at it” and therefore that’s how you end up with “Superforecasters” that’s a fascinating arc over I don’t know 20 years separating the two books?
MILLER: Yes, and the thing I really got from the book is not getting fixated on your in your one insight and putting all your cards and that …
RITHOLTZ: 100 percent.
MILLER: So these “Superforecasters” are right in the long run, they’re using the law of large numbers, it’s not that they are saying oh, I have this one idea, I’m going to fix it, Citibank has to be bailed out, no, no, take all your ideas and spread your bets across all your ideas just like the forecasters and you’ll do well eventually but don’t get fixated on that one and I think that’s a nice feature.
RITHOLTZ: Quite fascinating. Any other books before we move on?
MILLER: I think that’s good.
RITHOLTZ: Those three, those are three good ones.
So what are you excited about right now? What are you jazzed about in the world of academic research?
MILLER: Well, I think mastery is addictive…
RITHOLTZ: Mastery.
MILLER: There’s a lot of drive-by research out there and we’re all a little guilty of it because you know there’s a pressure to publish ….
RITHOLTZ: Publish to peers, for sure.
MILLER: And you’ve gone and you’ve really dug into something you really mastered something and just mined as much of the gold as you have but you also have this feeling of mastery, it just feels so great and wanting to do that again, right?
So the exciting thing is to take that understanding of how good it feels, how fun it is to master something and take it to the next subject while the course still finishing what you started, so that’s the exciting – what’s next.
RITHOLTZ: Really, really interesting. There have been all sorts of criticisms of the lack of reproducibility in academic research, what changes are you looking forward to into do you think that increased big data in AI is ever going to help us with this reproducibility problem we’re running into in academic research and in other research, why aren’t we seeing academic research being replicated and even corporate research being replicated?
MILLER: Well, I think there’s a lot of cherry picking that happens so when you go out and you analyze something and you measure 10 different things and you just pick out the things that worked, you’re not acknowledging what didn’t work …
RITHOLTZ: Right.
MILLER: And so you have this kind of winners curse in a sense …
RITHOLTZ: Right.
MILLER: Where ..
RITHOLTZ: That’s Thaler’s in his early books.
MILLER: Yes, we don’t have time to explain that, I realized, but …
RITHOLTZ: So let me re-ask that question, so what are you looking forward to, what changes do you think are going to affect your world?
MILLER: The changes that are going to affect the academic world, so I think that though the important change that is going to make this better, we are going to fix this – fix, maybe not, but make it better is the idea of preregistration.
RITHOLTZ: Meaning what?
MILLER: You preregister what are going to analyze, you register your predictions and so you know your hands are tied, you can’t…
(Crosstalk)
RITHOLTZ: So therefore, you are going to go out and actually research what you’re claiming as opposed to look at this anomaly let’s talk about that.
MILLER: Right.
RITHOLTZ: Even though it could be random or cherry picked or what have you.
MILLER: And still valuing even — whatever your conclusion is don’t take that as truth, we have to make sure it also replicates.
RITHOLTZ: Right.
MILLER: Because even then, you may have — what if you don’t find that you decide not to write it up?
RITHOLTZ: Isn’t that a big issue that people don’t publish on negative findings, because there is value to say, hey we analyzed this, we couldn’t find anything.
MILLER: It’s a huge issue because you then you get this kind of implicit cherry picking in that like I don’t want to spend my time writing up this paper because it’s not a big finding, well then no one is seeing that so the papers we see are the ones that are kind of implicitly selected and so you have the same kind of degrees of freedom that’s happening but it is like socially.
RITHOLTZ: What do you call survivorship bias about things that don’t work out? So I guess it is just straight up survivorship bias, right? In other words…
(Crosstalk)
MILLER: The research ideas that don’t make it to the publication stage have died and so the publication ones — are kind of the ones that are randomly better but not necessarily truly better.
RITHOLTZ: Interesting. Tells about a time you failed and what you learned from the experience?
MILLER: So one failure, it was a success and a failure so a friend of mine Patrick Flanagan from graduate school we set up his GarageBand hedge fund we called it, we are loaning money on the Internet, we thought we had this great idea and it was, it weathered the crisis we didn’t lose money, we got like 5 percent, it wasn’t big-money, maybe a hundred thousand or something, we were grad students.
RITHOLTZ: But this is peer-to-peer lending, that …
(Crosstalk)
MILLER: Peer to peer lending.
But the thing we didn’t anticipate was the legal uncertainty of the enterprise and we weren’t lawyers and we just – we model, we had a real confidence in our model, we had a automated bidding algorithm, it was great, we’re doing well on but the thing we didn’t get is that the SEC could potentially crackdown on this and ruin our business model because they changed the rules completely that made it impractical so we j just left because instead of having a direct connection to the person that you’re loaning, it was now mediated through the company and now you have to somehow price in the risk of the company itself rather than the loan, and so…
RITHOLTZ: Interesting, quite interesting.
What do you do for fun when you’re not crunching numbers?
MILLER: Well, I mean this is pretty fun, right?
(LAUGHTER)
Here I am in New York, you get to travel around, you get to meet with your co-authors finish your papers and nice locations and you know, meet interesting people, I would say just seeing family — because I get to travel so much, I get to see my family and friends in different cities and it’s a great thing.
RITHOLTZ: What sort of advice would you give to millennial or recent college grad who was interested in a career either in behavioral finance or statistics or any of the sort of work that you do? Economics, whatever.
MILLER: Don’t be too impatient to have life figured out, it’s not too late, I’ve seen people in their 20s and 30s go back, change, go back to school maybe they have to start a little lower rank school than they wanted to begin with but you can get funding at those schools and if you work hard, you can transfer, you can apply to another school, you can move up and a lot of people get this kind of false notion that oh if I didn’t, if I wasn’t a serious student in high school or in college that I’m too far behind.
It’s like, no, if you really motivated and you know, you’re capable and there are plenty of people that are, you can catch up, you just have to be patient, take a few years off.
RITHOLTZ: It’s never too late to get serious.
MILLER: Yes, never too late to get serious.
RITHOLTZ: And our final question, what do you know about the world of statistics and data analytics today you wish you knew a decade or so ago?
MILLER: Fake data simulation basically creating you want to — you can’t just go out and analyze data and show that however I analyze it I still get the same result, no, you have to sit down and generate fake data so what if the world look like this, how would my analysis behave? What if the world looked like that? How would my analysis behave? So you have to do this hard work of building models of the world and then seeing what does your analytical approach tell you under those different kind of assumptions about the model the world and to do that, you need fake data and that gives you a lot of insight.
RITHOLTZ: And so when you say fake data, I think of that as a counterfactual or how do you…
MILLER: I guess everything is a counterfactual because models are wrong, right?
RITHOLTZ: But some are useful.
MILLER: Some are useful, and you want to know how under different assumptions if the world you know looked differently than you think it looks, is your analysis still going to say something meaningful or not and you need to actually go out and check that and a lot of people don’t do that and so that’s what happened in this hot hand example, right?
I mean what would this analysis give you if there were no hot hand.
RITHOLTZ: Quite interesting. We have been speaking to Josh Miller of the University of Alicante.
If you enjoyed this conversation, well, be sure look up an inch or down an inch on Apple iTunes, where you can see the past 250 or so previous conversations we’ve had. We love your comments, feedback, and suggestions, write to us at MIBPodcast@Bloomberg.net.
I would be remiss if I did not thank the crack staff that helps put together these conversations each week, Madena Parwana is our producer, Michael Batnick is my head of research, Taylor Riggs is our Booker/producer, Atika Valbrun is our project manager, Tim Haro (ph) is our audio engineer.
I’m Barry Ritholtz, you’ve been listening to Masters in Business on Bloomberg Radio.
END
