Here is a proof that, for square \(n \times n\) matrices \(A\) and \(B\), if either \(A\) or \(B\) is invertible, then \(A B\) and \(B A\) are similar:
Let \(P\) be the matrix \(P = B^{-1} A\). Then we have
\(A B = B P\)
and
\(B A = P B\)
Since A B and B A are both square \(n \times n\) matrices, they are similar if there exists a non-singular matrix \(S\) such that
\(A B = S B A\)
Substituting the expressions for \(A B\) and \(B A\), we get
\(B P = S P B\)
Since \(B\) is invertible, we can multiply both sides by \(B^{-1}\) to get
\(P = S P\)
Since \(P\) is a non-singular matrix, we must have \(S = I\), the identity matrix. This shows that \(A B\) and \(B A\) are similar.
Alternatively, we can show that \(B A \) and \(A B\) are similar in a similar way, using the matrix \(Q = A^{-1} B\) instead of \(P\).
This proves that, for square \(n \times n\) matrices \(A\) and \(B\), if either \(A\) or \(B\) is invertible, then \(A B\) and \(B A\) are similar.