The Crystal Ball - A Thought Experiment on Determinism

Be warned that this post might sound like rambling to many people, and that is ok. It is meant to be playful and I think it will be fun to people who often ponder the same questions as I do here.

As a data scientist, especially one quite fond of causal inference, I have always wondered about how far can we take causality as a principle. The farthest destination I could always think of is hard determinism - the idea that having enough information on the system, any state of the system is predictable from the previous one. That is the ultimate rule of cause and effect, taken to the extreme.

However, there is one problem. Hard determinism such as this is contradictory to the idea of the free will, which not only feels good to have (and pretty devastating to not have), but is also an important concept in many other aspects - one of them being a key part in our society, since without free will we cannot talk about responsibility, and without that our whole judicial system falls apart, as well as many other social constructs.

This article is about one thought experiment I run in my head to try to make sense of these things, as most philosophical debates (and scientific debates, for that matter) on the question were quite inconclusive to me (and most of them either too simplistic or too incomprehensible to take them seriously). What better can we do than an experiment, even if it runs only in our head!

The Hypothesis

Every experiment needs a hypothesis. The null hypothesis we start with is pretty simple:

H0: The world is fully deterministic.

To be more specific, this means that knowing a state of the world at one point in time implies that you can derive all future states of the world in an unambiguous way.

Since this is true, then we can assume there could be a machine that can predict the next state of the world given the current state. Such a machine can then predict any future state if applied recursively, and provide an answer about the state at any future point in time.

Some may say that this machine is technically impossible, but if this is true, then in my opinion, the question of whether the world is deterministic is not scientific. In any case, this won't stop us from doing our thought experiment. We will also return to this a bit later and see that it may not be necessary for the machine to model the entire universe we live in in order to make predictions about it's parts, so the machine will probably seem more plausible.

The Crystal Ball Experiment

But now let's assume that the world is deterministic and we have a machine that can tell us future states. Let's call this machine the crystal ball. We can imagine it's interface is an actual crystal ball, and when we look at it, it generates a 3D rendering of the world at a certain point in the future.

Now, since the world is fully deterministic, then our actions as part of the world, must be deterministic too. This means that our choice is an illusion, and our actions are fixed, dependent only on the current state of the world, and whatever happens, we cannot change our destiny.

The experiment is then conducted in the following way: we are placed in front of two, regular, gray buttons, nothing to distinguish one from another (i.e. we have done everything we can to eliminate any bias).
Anyway, we have a left button and a right button. We have no idea what each of them is doing, but we are told to look into the crystal ball and it will tell us what we are destined to press. Once the crystal ball says it's output, we are instructed to choose what to do¹.

The experimenter then proceeds to feed the current state of the world to the crystal ball, we look at the crystal ball and see a world in which we are pressing the left button. Armed with this information, is it possible for us to choose to press the right button?

If the null hypothesis is true, we should not be able. Whatever the crystal ball says, it must happen, since it's a machine that can completely describe our deterministic reality.

Playing a Game of Strict Experimentation

The tricky part about the claim of hard determinism is that there is no room for probabilities. Probability from such a perspective is usually considered as a tool we utilize when we don't have all information. But the crystal ball is such a machine that has all information, so it must not output any probabilities, but a hard, deterministic output.

On the other hand, we humans are not the crystal ball, and we are allowed to use probabilities to describe our experiment. And even though this experiment happens in our head, let's play along since it's fun and we like to play science. Doing that, we see that our null basically states that the probability that we observe an outcome $Y = c$ given that the prediction from the crystal ball is $C = c$, must be equal to 1:

If we say this is a Bernoulli random variable, then it's standard deviation is $\sigma^2 = p(1-p)$. Since $p=1$, then $\sigma^2=\sigma=0$. Then, regardless of the sample size we have in our experiment, if we only find a single case of the crystal ball failing to make a correct prediction, the difference will be statistically significant, and we would be able to reject the hypothesis.

Let's assume from now on, that the entire population of homo sapiens has been included in the experiment, as the governments of the world realized this is a fundamental and existential question that can determine the fate of humanity.

We also assume we are the single sample that actually wants to do the opposite of what the crystal ball says, because that's how we are, and no crystal ball is gonna tell us what to do!

If this is the case, and even if we are the only ones acting in spite to the crystal ball prediction, it seems we have no choice but to reject the null hypothesis.

The Critic

For the rest of the article, I'll try to play the critic and attempt to invalidate this whole experiment. Since I might be naturally biased as this experiment has been going around in my head for a while, there may be things I missed. If you think of any, feel free to send me an email, I'd be glad to read it.

Interference

We have mentioned this before. We could argue that the state of the world is different from the one we fed to the crystal ball once we posses the knowledge of it's answer. Thus, the crystal ball's response actually interferes with the outcome, i.e. it is a confounder. However, the crystal ball should know that. If our world is deterministic, then the interference itself is a deterministic phenomenon. So I don't think this argument stands, and it's essentially an argument against hard determinism. The crystal ball seems unable to predict the consequences of it's own actions, thus making those actions nondeterministic.

I also think it's worth thinking why such a thing as an "objective observer" is necessary for a deterministic world. The world we live in is the world that contains the observer as a part of it - if it must eliminate the observer in order to be deterministic, then it's a world that is different from ours. In other words, it's a world that doesn't exist, as far as we know it. And we don't need our observer to be the all-knowing crystal ball, but we are using a crystal ball here in order to take hard determinism all the way to the end of the rabbit hole. So I insist that the experiment must contain the element of an impact that the crystal ball prediction has on the world - and if the world is actually deterministic, it will not matter.

If above sounds like hand-waving, there is a better way to see the fault at the root of this argument against the experiment.

A clear contradiction arises if we imagine that the experimenter secretly has another crystal ball (B) in his office - an exact copy of the one used in the experiment (A). B does not interfere with the experiment, and it can predict correctly the result of the experiment - that being that A predicts a world where we pressed the left button, and after that we press the right button. B can and will predict this will happen. However, this is a problem, since:

• A and B are modeling the same deterministic world from the same initial state.
• A predicts left
• B predicts right

The two crystal balls have different predictions starting from the same state of the world. This cannot happen in a deterministic world, where there is only one possible outcome.

The Halting Problem

Maybe the crystal ball cannot compute the answer? It will know that the next state of the world is that it will have told us to press left, and we would press right. But then the crystal ball would have to say right instead of left. However, this means we would press left. This can go on to infinity, and the crystal ball cannot compute it, so it will halt. This argumentation implies that the crystal ball is a computer. If the crystal ball is a computer, it would indeed get stuck and it would never complete, and this whole situation is reminiscent of the famous Halting Problem.

We won't go deep into it, but let me just quote wikipedia on this:

For any program f that might determine if programs halt, a "pathological" program g, called with some input, can pass its own source and its input to f and then specifically do the opposite of what f predicts g will do. No f can exist that handles this case. A key part of the proof is a mathematical definition of a computer and program, which is known as a Turing machine.

If you carefully look at this, you'll notice that this whole experiment is just one instance of the Halting problem, where we are the "pathological" program g, and the crystal ball is the program f, only instead of predicting whether g will halt, it predicts whether which button g will decide to press, i.e. g presses left = g will halt.

Since we can reduce the thought experiment to the Halting Problem, given that the crystal ball is a computer, and since such a machine cannot exist, we can conclude that either the crystal ball cannot exist, or that it cannot be a computer.

But we are very persistent in our thought experiment, and want to keep going, so we assume the crystal ball is not a computer. It's some advanced machine from the future, where computers as we know them, i.e. Turing machines, are outdated.

However, it is interesting that, by the looks of it, a computer that has to model a world which contains another system that has the ability to process it's output and act upon it, cannot exist, as it will eventually encounter a "pathological" situation which will reduce it to the Halting Problem.

This Machine is Simply not Realistic

I tend to agree with this. We mentioned before that this makes the question Is hard determinism true? nonscientific, as one of the main conditions for something to be scientific is to be verifiable by observation. I think the only way for this to happen when determinism and free will are in question is to know all the deterministic laws by which the world works, make predictions based on them and try to choose to do otherwise. The crystal ball is just a proxy for materializing this knowledge.

However, if we have all the deterministic laws of the world defined, doesn't that make the world deterministic by definition? I don't think so, it's still true that the only way to prove that these laws hold in general is to make observations based on them and see if they match reality. Otherwise, our laws are just words on paper.

So, what we can actually argue is that it's simply not possible to do this. The machine is the part of the world, and it cannot possibly be able to represent as many states as the whole world because the machine is it's subset. Furthermore, information cannot travel faster than light, so representing the state of the whole universe is virtually impossible. I think this is a valid argument, but the thing is, we don't have to model the whole world. We can model only the relevant part of it. This is because most of the "components" of the world are not strongly correlated with each other, especially in short periods of time. Also, the speed of light limit is actually helping us in this case, since we know information that hasn't been able to arrive to the "scene" of the experiment, could not impact the experiment at all. Also, we know we can do this, since we are already doing it! There are elaborate and quite precise simulations of many phenomena. We model the orbit of the Moon around the Earth quite precisely without representing the state of the whole universe or all the people on Earth living their daily lives, right? This is because the Moon's orbit doesn't care about what humans are doing (unless we maybe decide to nuke it very hard for some reason).

We can then constrain the crystal ball to only model the consciousness of the person in the experiment, along with the sensory input from their surroundings. The computational theory of mind that is dominant today states that human consciousness is a result of computation and the organization of information in the brain. If we can mimic this organization, and do so in a faster hardware than the organic tissue of our brain, we can basically transfer our consciousness into a machine and mimic our consciousness with superpowers. So this could be the crystal ball. Then it doesn't have to make the prediction faster than the universe, instead it should only be faster than the brain.

This makes the machine quite more plausible, since this machine is either possible, or we are claiming that nothing can be faster that the brain.

Consciousness

This also means we must change our hypothesis:

H'0: Consciousness is fully deterministic.

However, note that hard determinism is such a strong claim that if it's true, everything, including consciousness, must be deterministic. So if we manage to reject the H'0, we are still rejecting H0 as well.

So, now we imagine the experiment happening again in this context, but the outcome doesn't change. We, the contrarians, choose the exact opposite of what the crystal ball says.

It seems that the mere existence of consciousness creates quite a few problems. In data science, we are all about making predictive models. AI has it as it's dream to create a general intelligence - in some of the definitions of that goal, it means exactly the same type of intelligence as human intelligence, only on steroids. But if we are able to that, then we have a predictive model of how an individual brain will work that gives us predictions before the person has decided, which gives us our crystal ball. Then again, if that is true, we run into the paradoxes described here, i.e. cases where the consciousness that we are modeling acquires knowledge on our prediction and changes so much that it makes our predictive model false.

There seems to be an upper bound on the amount of time in the future the crystal ball can make a prediction about a consciousness that is observing that prediction, and that upper bound is the time it takes to process the output of the crystal ball. This could be the time interval in which our, human, consciousness is deterministic. After the knowledge is acquired, it seems like it resets, and everything dilutes into some kind of a probability wave.

It is clear that the experiment we have in mind here leaves a much larger time window where we can process the information from the crystal ball. So an experiment that is designed to target this small time window could probably be created if we had a crystal ball, and this seems very plausible to succeed in accepting the null hypothesis².

Since I like to take things to the limit, I often think what happens if we transfer our consciousness to a machine that is much faster, and then model this consciousness where the time window decreases? And what if we eventually reach the minimum time window possible (assuming our brain is not already there)? This means both the crystal ball and our consciousness will have the same time window. This basically means that the actions of such a consciousness would become unpredictable, as there is no time to make a prediction before the consciousness becomes aware of it.

Causality

How does causality stand with all that we said here? Because it seems like accepting the laws of cause and effect is pretty much the same as accepting hard determinism. This, as we have discussed, doesn't seem to work.

In my opinion, causality is a much weaker statement than hard determinism, as it doesn't necessarily state that all future states can be predicted unambiguously from previous states. In fact, it might be interpreted in reverse - all past states can be estimated from the current state. I.e. every effect has a clear cause, rather than every cause has a clear effect. So, as long as it stays clear of the deterministic future, i.e. not probabilistic, I think it can survive³.

Conclusion

So, after all of this pondering, in my opinion, the question of hard determinism is quite resolved. The most basic resolution is - if a crystal ball cannot exist, it is an unscientific question. Yet if it can exist, it will fail the experiment. If a theory is plausible only when you cannot test it's claims, it is enough at least for me to not consider it seriously. So as far as I'm considered, I have no reason to reject the idea of a free-will, even if somewhat limited⁴.

More interesting questions arise when thinking about the role that an observer has in the world. It definitely seems that an objective observer is physically impossible, but also thinking about the two crystal balls giving two contradictory predictions, it seems that the world is a different place for each observer. This is, in fact, nothing new, and it's not my goal to pretend it is. But it is something that can also be acknowledged from this thought experiment alone.

Regarding the observer, many people might remember how the observer also has a big role in quantum mechanics. However, this post deliberately avoids all of that. There are two reasons for this. First one is, I am not a quantum physicist. And even from little knowledge I have of quantum mechanics, I know it's very easy to get it wrong. The second reason is that I think quantum mechanics is not needed for contradictions to arise in the idea of determinism, and as the thought experiment proposes, the inherent uncertainty in our reality could be seen in the phenomena on a macro level as well, especially when things like consciousness are the object of observation.

And as a final note, I'd still insist that in order to show how something models a supposedly deterministic consciousness, weather it's a crystal ball, a mathematical equation, or some other model, it must do so in an experiment similar to the one described here, where the key part is that the consciousness is given a chance to receive the information on the prediction that happens after this event. Everything else regarding a question such as this is too big of a simplification, IMHO, and not good enough to reject the existence of a free will. It is as if Newton would make a claim on his third law, but only when we are not ourselves causing the object to exhibit force on another! A deterministic phenomenon is deterministic whether we play a game of an "objective" observer or not.

Hope you found this interesting, and that it didn't waste your time! Cheers!