Lay Chapter 6,

Orthogonality and Least Squares

Orthogonal Matrices

In[1]:=

Note that the columns of u are orthogonal, and they are all of length 10.

In[3]:=

Out[4]//MatrixForm=

Representation of Vectors in V == S ⊕

Let's find the projection of a vector y onto S = span(u1,u2,u3).

In[5]:=

Are the u's mutually orthogonal?

In[10]:=

Out[11]//MatrixForm=

In[12]:=

Out[13]//MatrixForm=

Yes, the u's are mutually orthogonal

Find the projection of y onto span(u1,u2,u3).

In[14]:=

Out[15]//MatrixForm=

Then y - yHat is in

In[16]:=

Out[17]//MatrixForm=

... and the required decomposition is

In[18]:=

Let's check that.

Is yHat in span(u1,u2,u3)?

In[19]:=

Out[20]//MatrixForm=

Is y-yHat in ?

In[21]:=

Out[21]=

Is y == yHat + (y - yHat)?

This one should be easy.

In[22]:=

Out[22]=

Projection of a Vector onto the Column Space of an Orthogonal Matrix

In[23]:=

Note that the columns of u are orthogonal, and they are all of length 10.

In[26]:=

Out[27]//MatrixForm=

Therefore, the closest point to y in the column space of u is yHat, where

In[28]:=

Let's formalize that result. Let W be the column space of U.

See Lay 6.3, Theorem 10, p399.

In[29]:=

Out[30]=

Check: Verify that yHat really is in the column space of u ...

In[31]:=

Out[31]=

...and that y-yHat is orthogonal to each of the columns of u.

In[32]:=

Out[32]=

QR Decomposition

In[33]:=

Mathematica has built-in support for QR factorization, but its definition of the matrices involved differs somewhat from that of Lay. Consult Mathematica's Help Browser for the function QRDecomposition, and see that entry's "Further Examples."

In[35]:=

Out[36]=

Note that the matrix Q of Lay is the matrix Conjugate[Transpose[q]] of Mathematica.

In[37]:=

Out[38]//MatrixForm=

Check.

In[39]:=

Out[39]=

QR Decomposition

In[40]:=

Mathematica has built-in support for QR factorization, but its definition of the matrices involved differs somewhat from that of Lay. Consult Mathematica's Help Browser for the function QRDecomposition, and see that entry's "Further Examples."

In[42]:=

Out[43]=

Note that the matrix Q of Lay is the matrix Conjugate[Transpose[q]] of Mathematica.

In[44]:=

Out[45]//MatrixForm=

Check.

In[46]:=

Out[46]=

Least-Squares Solution of an Inconsistent System

Find a least-squares solution to the inconsistent system ax=b, where

In[47]:=

Soln: Use the normal equations.

In[50]:=

Out[51]//MatrixForm=

In[52]:=

Out[53]//MatrixForm=

Now solve the system ax = b.

In[54]:=

Out[55]=

In[56]:=

Out[56]=

Least-Squares Solution of an Inconsistent System

Find a least-squares solution to the inconsistent system ax=b, where

In[57]:=

Soln: Use the normal equations.

In[60]:=

Out[61]//MatrixForm=

In[62]:=

Out[63]//MatrixForm=

Now solve the system ax = b.

In[64]:=

Out[65]=

Here is a clearer view of the solution.

In[66]:=

Out[66]=

and, better still, we have

In[67]:=

Least-Squares Error

(1) Find a least-squares solution to the inconsistent system ax=b, where

In[68]:=

Soln: Use the normal equations.

In[71]:=

Out[72]//MatrixForm=

In[73]:=

Out[74]//MatrixForm=

Now solve the system ax = b.

In[75]:=

Out[76]=

xHat is the least-squares solution. In this case it is unique because a is invertible: xHat = b.

In[77]:=

Out[77]=

(2) Find a least-squares error of this approximation.

In[78]:=

Out[78]=

Least-Squares Solution via QR Decomposition

Find a least-squares solution to the system ax=b, where

In[79]:=

Let's use the QR Decomposition of a to find xHat, as in Lay Theorem 15, page 414.

(1) Calculate the QR Decomposition of a.

In[82]:=

Out[83]=

Note that the matrix Q of Lay is the matrix Conjugate[Transpose[q]] of Mathematica.

In[84]:=

Out[85]//MatrixForm=

Check.

In[86]:=

Out[86]=

(2) Use the QR Decomposition of a to calculate xHat.

In[87]:=

Out[87]=

Note that Lay suggests that, for numerical reasons, it is preferable to solve for xHat directly from the equation

r.xHat==Transpose[Q].b.

In[88]:=

Let's try that.

In[91]:=

Out[92]=

Created by Mathematica (February 25, 2005) |