The context here is that we have some desired vector that we want to build out of a set of basis vectors through weighted summation. The case where this is easiest is when all of our vectors are orthogonal with respect to each other. Recalling that a dot product of two vectors gives us a measure of their similarity, two vectors are orthogonal if their dot product is 0. A basic example of this is the set , or the rotation of these vectors 45 degrees, .

If we have an orthogonal basis set of vectors, then to generate the weights for each of the basis vectors we simply take the dot product between each and our desired vector . For example, with our basis sets from above, the weights to generate the vector can be found as

where denotes dot (or inner) product, and leads to

And now we have weights such that for each basis set . Written generally, to find the weights we have . The denominator here is the norm of , introduced for generality. In the example set our basis sets were composed of unit vectors (vectors with magnitude = 1), but in general normalization is required.

Now, what if we don’t have an orthogonal basis set? Trouble, that’s what. With a non-orthogonal basis set, such as , when we try our dot product business to find our coefficients looks what happens

and

,

which is not a good reconstruction of our desired vector, . And the more the cross contribution to the same dimensions between different basis vectors, the worse this becomes. Of course, we could use a least squares method to find our basis set coefficients, but that involves matrix multiplications and inverses, and generally becomes more complex than we want.

So, let’s say we have a basis set of two different, but non-orthogonal vectors, and . We instead want two vectors and which describe the same space, but are orthogonal. By describing the same space, I mean that their span is the same. And by span I mean the set of values that can be generated through weighted summation of the two vectors. So we set , and the task is now to find the appropriate . As a conceptual description, we want to be equal to , but only covering the area of space that isn’t covering already. To do this, we can calculate at the overlap between and , then subtract out that area from . The result should then give us the same area of state space covered by and , but in a set of orthogonal vectors and .

Mathematically, we calculate the overlap between and with a dot product, , normalized by the magnitude of , , and then subtract from . All together we have

.

Using our non-orthogonal example above,

Due to roundoff during the calculation of , the dot product between and isn’t exactly zero, but it’s close enough for our purposes.

OK, great. But what about if we have 3 vectors in our basis set and want 3 orthogonal vectors (assuming we’ve moved to a 3-dimensional space) that span the same space? In this case, how do we calculate ? Well, carrying on with our intuitive description, you might assume that the calculation would be the same as for , but now you must subtract out from that is covered by and . And you would be correct:

.

In general, we have

.

And now we know how to take a given set of basis vectors and generate a set of orthogonal vectors that span the same space, which we can then use to easily calculate the weights over our new basis set of vectors to generate the desired vector, .

[…] do this, we’ll use Gram-Schmidt orthogonalization, which I describe in a recent post. To actually follow this orthogonalization process we’ll need to define an inner product and […]