Singular Value Decomposition

An Example

Calculating a singular value decomposition

Consider the following matrix:

(Lay 7.4, Example 3)

In[1]:=

Find the eigendata for A.

In[3]:=

Out[6]//MatrixForm=

In[7]:=

Out[8]//MatrixForm=

Construct V.

The columns of the matrix V are the orthonormal eigenvectors of A, taken in the proper order.

In[9]:=

Out[13]//MatrixForm=

Check that the columns of V are orthonormal.

In[14]:=

Out[15]//MatrixForm=

Check the diagonalization.

In[16]:=

Out[16]=

Construct Σ .

In[17]:=

In[18]:=

Out[18]=

Out[22]//MatrixForm=

Construct U .

A has rank 2. The two columns of U are the normalized vectors A.v1 and A.v2.

In[23]:=

Out[26]//MatrixForm=

Check the singular value decomposition, A == U.Σ..

In[27]:=

Out[28]=

Compare with Mathematica's singular value decomposition.

Mathematica's numerical result.

In[29]:=

Out[32]=

Our numerical result.

These two results agree up to a sign change on some rows and columns.

In[33]:=

Out[34]=

The Four Fundamental Subspaces

Consider the matrix of the previous example, ...

In[35]:=

... and its singular value decomposition, A == U.Σ..

In[36]:=

Out[37]=

a is an mxn matrix with rank r.

In[38]:=

The singular value decomposition of a displays orthogonal bases for the four fundamental subspaces of a as follows:

(See Lay 7.4, p479)

In[41]:=

The Moore-Penrose Inverse and Least-Squares Solutions

Consider the matrix of the previous example, ...

In[42]:=

... and its singular value decomposition, A == U.Σ..

In[43]:=

Out[44]=

a is an mxn matrix with rank r.

In[45]:=

The Moore-Penrose Inverse, A+, or pseudoinverse of A is computed from its singular value decomposition, A == U.Σ., as follows:

(See Lay 7.4, p479)

In[48]:=

Suppose we wish to solve the least-squares problem A.x == b.

In[53]:=

Let

In[54]:=

Then,

In[55]:=

where bHat is the orthogonal projection of b onto Col A == Col Ur.

Therefore, xHat is a least squares solution of A.x == b.

It can be shown that this xHat, obtained via the Moore-Penrose inverse of A,

is the shortest least-squares solution of A.x == b

(See Lay 7.4, p480)

Another Example

Calculating another singular value decomposition

Consider the following matrix:

(Lay 7.4, Example 4)

In[56]:=

Find the eigendata for A.

In[58]:=

Out[61]//MatrixForm=

In[62]:=

Out[63]//MatrixForm=

Construct V.

The columns of the matrix V are the orthonormal eigenvectors of A, taken in the proper order.

In[64]:=

Out[67]//MatrixForm=

Check that the columns of V are orthonormal.

In[68]:=

Out[69]//MatrixForm=

Check the diagonalization.

In[70]:=

Out[70]=

Construct Σ .

In[71]:=

In[72]:=

Out[72]=

Out[75]//MatrixForm=

Construct U .

Calculate u1

In[76]:=

Out[76]=

Extend {u1} to an orthonormal basis of R^3.

In[77]:=

Out[85]//MatrixForm=

Check that the columns of U are orthonormal.

In[86]:=

Out[87]//MatrixForm=

Check the singular value decomposition, A == U.Σ..

In[88]:=

Out[89]=

Compare with Mathematica's singular value decomposition.

Mathematica's numerical result.

In[90]:=

Out[93]=

Our numerical result.

These agree up to the precision of the computer arithmetic that is involved.

In[94]:=

Out[95]=

The Four Fundamental Subspaces

Consider the matrix of the previous example, ...

In[96]:=

... and its singular value decomposition, A == U.Σ..

In[97]:=

Out[98]=

a is an mxn matrix with rank r.

In[99]:=

The singular value decomposition of a displays orthogonal bases for the four fundamental subspaces of a as follows:

(See Lay 7.4, p479)

In[102]:=

The Moore-Penrose Inverse and Least-Squares Solutions

Consider the matrix of the previous example, ...

In[103]:=

... and its singular value decomposition, A == U.Σ..

In[104]:=

Out[105]=

a is an mxn matrix with rank r.

In[106]:=

The Moore-Penrose Inverse, A+, or pseudoinverse of A is computed from its singular value decomposition, A == U.Σ., as follows:

(See Lay 7.4, p479)

In[109]:=

Suppose we wish to solve the least-squares problem A.x == b.

In[114]:=

Let

In[115]:=

Then,

In[116]:=

where bHat is the orthogonal projection of b onto Col A == Col Ur.

Therefore, xHat is a least squares solution of A.x == b.

It can be shown that this xHat, obtained via the Moore-Penrose inverse of A,

is the shortest least-squares solution of A.x == b

(See Lay 7.4, p480)

Created by Mathematica (April 5, 2005) |