Removing Uncertainty


Suppose you are in process of

a. Asking a stranger which province he comes from.

b. Writing a search routine using one of the 3rd generation computer languages.

c. Trying to find out what went wrong in a circuit board that consists of a lot of electronic components.

Now you are given the freedom to formulate your question, or to design your process to accomplish these tasks.

What is the best approach?

A guiding principle that we can follow is this

To remove the maximum amount of uncertainty of the subject matter on each step of your process.

In (a) you should not ask a question like "Do you come from Alberta?" You will start a guessing game if the guy says "no". That question can only remove one unknown out of twelve. You can do a lot better with a simple question "Which province do you come from?"

Yes, that is trivial. Let's switch to the programming question of (b). For the sake of discussion we will simplify the question to compare two types of search methods - the binary search and the sequential search.

Every programmer knows that binary search works much faster than a sequential search. The reason is that each step of a binary search will eliminate half (or almost half in case of odd number) of the elements from the list being considered. In my language you are removing almost 50% of your remaining ignorance at any step. And that is a very efficient process.

In (c ) if you know the structure of the circuitry and the probability of failure for each of its components you can design a procedure to minimize the number of steps required to identify the bad component.

Let us consider a very simple example of 6 light bulbs in series and assume one of them is out of order.

---O---O---O---O---O---O---

If you use an ammeter to test each bulb it will take a maximum of 6 steps to locate the bad one. If you follow the following scheme then you only need 3 steps to accomplish the task:

Test the first 3 as a group (Step 1)
   If there is no current, then you know one of these three is bad. 
      Then you test the first two as a group (Step 2).
            If current is passing through, the third one must be the bad one.
            If there is no current, then test the first one (Step 3), 
            and that will tell you whether number one or two is bad.
   If there is current in the first three, then you know that the bad one is within the 
   last 3 bulbs and you can simply apply the previous procedure (the one we use for the
   the first 3) to the last three light bulbs. 

This process is like a binary search in the sense that you remove almost 50% of the remaining uncertainty at each step.

In this example we have assumed that the rate of failure for these light bulbs is the same. We can in fact say each one of them has 1/6 of the chance of making the whole system fail. In this way we can relate the problem to a probability space with six equally likely events (sorry, a little jargon here).

In a "probability space" we can establish a mathematical formula to quantify the "uncertainty" or "ignorance" that we have mentioned so many times. With that formula we will be in a better position to tackle the more complex problems. Such a formula will provide the measure of the uncertainty of the system and is called the Entropy.

The Chinese translation of entropy is "æi"¡C I don't know whether this is a new invention or a word from some ancient Chinese scripts. In physics, however, entropy (which is represented by a similar formula) is interpreted as a measurement of "chaos". And that is closely related to the second law of Thermodynamics.

In our daily life we don't need to know the definition of entropy. However, the general principle that we mentioned above may find some applications in our business activities. In fact, it has already been used in areas like information gatherings and computer program optimizations. And I heard that it has also been used in business negotiations.