00:00What makes someone a prophet? And I mean this in the context of prediction primarily, because a prophecy is just a prediction with more meaning or weight to it. And if we're just gonna say that a prophet is someone who makes predictions, does every prediction have to be right? If 99% of the time a person predicts the future correctly but 1% of the time they're wrong, are they somehow not a prophet anymore? I think at that level of accuracy you kind of have to concede that their understanding of the future far surpasses the average person's. So do we allow for any error before labeling someone a prophet, and if so, what is the cutoff mark? And do they have to communicate it perfectly to the person who receives the prophecy? Classically we read of these prophecies which are tragically misunderstood.

00:46But to me, that kind of seems like they didn't really make an accurate prediction if they couldn't communicate it well. It seems odd that after the fact a predictor can just say no you misunderstood me after the event has already occurred. But certainly some people are better at prediction, so can you somehow quantify this skill? Because it seems to me that everyone predicts the future with some degree of accuracy. It's what you're doing all the time, you're always thinking about what you're going to do next, and mostly your predictions are about you, but sometimes you make predictions about the world at large. And in both the local context concerning your own actions and the global context, some people are just fundamentally better predictors than others. We see this in the real world in terms of investment, which is generally how we quantify who the best predictors are.

01:34It's who is making the most money off their investment decisions. But in that case people are only lending their future based on to a very narrow set of possibilities. It's a thin band of all potential futures that are considered, like a corona of time is obscuring your future sight of what could be. In the context of how we currently think about prediction, it's just about the market, and ultimately it's about whether your own access to money or other assets in the future will increase. And I think this is actually rather limiting. Predictions aren't really that special, they're just thoughts about the future. What am I going to do later? Where am I going to eat? But some of those predictions have great value to other people, and it seems reasonable to me that we should be able to measure and quantify who the best predictors are.

02:22And who makes predictions that best benefit society. And I think that would actually be a very useful thing for civilization, if the people who are the best predictors, who understood the future the best, were given more influence over society. For example, we have all these pundits on TV shows and they're constantly making these ridiculous predictions, but no one really holds them accountable to their word once they've been shown to be wrong simply by the passing of time. And then when something unexpected happens the very next day, the same people who made such inaccurate predictions are the same people we're listening to to give their hot take about why it did happen. And it seems to me that those people don't really know why whatever happened happened. They're just reacting to the event and in retrospect trying to correct for their error. If they really understood the state of the world and how it changes, they would have had a reasonable prediction that the unexpected event was going to occur.

03:09Or take politicians for another example. These are people who really build their career off of promises which are really just predictions, and there's no real scoring criteria for this, and no politician is held accountable to these predictions. So we tend to lean towards whoever makes the grandest promises and not those who can actually enact positive meaningful change. If we actually cared about meaningful effective government, we would promote community organizers who have actually transformed the lives and futures of the most people to the highest seats of power. And it stands to reason that you can measure this, because sometimes those pundits and politicians are right. And there are actually people have attempted to do this in the Good Judgment Project.

03:56Philip Tetlock attempted to quantify the capacity to predict strictly determinable things, and he details his findings in the book Superforecasters: The Art and Science of Prediction. The way they scored this was something called Brier scores. So Brier scores don't treat outcomes as a simple binary choice. It's not about just predicting whether something will or will not happen, it's about predicting the probability of an event to occur. So you could say something is 50% likely to happen because you don't have enough information, or a hundred percent because you know it will occur, or 10% because it's highly unlikely but there's still a small chance of it happening. And the notion and scoring would be: if you group together all the predictions you marked as being 80% probable, for example, if 80% of them did occur then you would be scored well.

04:46But if of all these predictions only 60% occurred or a hundred percent occurred you'd be scored less well. And similarly, if you group together all the predictions you marked as a hundred percent likely and a hundred percent did occur, that would improve your score even more, because your confidence also factors into the score. So for example you might be tempted to mark everything as having a 50% chance of occurring because there's only two choices, either the event does happen or it doesn't. And first, there's no real reason why all predictions being considered are equally likely to happen or not happen. And second, these probabilities are about confidence regarding uncertainty. We make our predictions relative to the quality of information we have. If we have better data we feel more confident, and having greater confidence improves your score if you end up being correct.

05:37The more uncertainty you mark events as having, or the closer to 50%, the less that is going to contribute to a good score. So if a person took all the things you marked as 50% probable and split them accurately into buckets of 0 and 100%, they're going to have a better score than you, even if of all those things 50% of them did occur and 50% didn't, just like you had predicted. And there are people and apps who attempt to score predictors using this method. PredictionBook, Good Judgment Open, and Metaculus all have a foundation in their scoring functions in Brier scores. But I would argue there's actually a lot of things wrong with Brier scores. Things aren't as strictly determinable as the people who are writing these predictions make them out to be. Often there is ambiguity and inconsistency in how the prediction was defined.

06:26And there's often disagreement after the fact about whether the event actually occurred. For example, if you make a prediction that the US will go to war with Iran but the US starts carrying out continuous military action in Iran without actually declaring war, there may be some debate about whether the event was actually predicted accurately, because it's ultimately coming down to our definition of the word, the definition of war. And then you're only allowed to make predictions about things that an administrator has determined to be determinable, so there's a lot of power in simply getting to write the predictions. And because Brier scores require an exact end date for determining whether an event occurred, if you predict something will occur on an exact date and then it happens two seconds after midnight, you would have to mark that wrong.

07:15But that doesn't really intrinsically feel like an error. So I think you can actually do much better than Brier scores, especially in terms of how you're scoring and how language is being used to define the prediction and the determinability of the outcome. The other problem is I think it's actually hard to phrase every thought in terms of a prediction. Certainly predictions are a very powerful way for assessing truth, because the foundations of science and how we move forward in terms of knowledge is whether our models make better predictions. But prescient people aren't always concerned with prediction. Sometimes they're simply ahead of their times in terms of knowledge or belief. For example, Bernie Sanders supported gay rights 30 years before Hillary Clinton, and there's some level of prescience there. But it's not exactly like he said he knows that the rest of society will catch up at a certain date.

08:02So there is some foresight there in terms of being ahead of the times. It's not really a prediction, you might not know exactly when culture is going to turn, but you came to a conclusion far before others, you were ahead of the curve. This is a big problem with Brier scores. You have to phrase a truth to be evaluated as a prediction with a discrete and precise end date to determine whether something has occurred or not. If you go past that date it's marked as wrong, but you can be simply ahead of the curve, the collective belief. And I would suggest you're still prescient even though you haven't made any specific predictions, and rather you just arrived at a conclusion before others. You understood a truth before everyone else did, so you should be able to quantify this as well, and without having to specifically phrase it as a prediction.

08:49Because you know this newly acquired knowledge is correct, you just don't know when the rest of the society will catch up. And forcing someone to predict when a certain percentage of the population will have that belief and by a specific date kind of misses the point. You should be able to phrase that belief just in the form of a statement and be scored as more prescient if while at the moment most people disagree with you, in the future it becomes common knowledge or belief, even if you don't know exactly when that's going to happen, because you're ahead of the curve. So I think there's actually many ways to quantify how well a person predicts the future, and a big part of that is how useful that prediction is to society at large. So part of that assessment comes down to the language being used to define the predictions that are being evaluated.

09:38And part of it comes down to how much the crowd at large cares about those predictions. And I don't think you should have any singular person or entity controlling how you express your belief about the future. I do think that the notion of probability is important, rather than just saying something will or won't happen, but language itself is fuzzy, and how you speak your vision of the future shouldn't be limited by those who write the predictions people are allowed to evaluate. And if you've been watching my last two videos, you might be starting to connect the dots on what I'm suggesting here. And I will eventually spell it out exactly, but that is for another video. So what do you think? How often do you think about the future? Do you think you're good at prediction? Do you think some people or cultures might think about the future more often than others?

10:26Do you think those people may have an unfair advantage if this were the system to drive society? Is there a way to allow people with more past focused thoughts to also participate? Let me know your thoughts in the comments. Feel free to quantify the goodness of this video by liking it, and feel free to quantify the truth of it by sharing with others. If you're interested in this line of thought and feel particularly aligned with my mind, don't forget to press subscribe and the notification bell. [Music]