Try
A change of coordinates must respect every symmetry, not just one chosen matrix. Work over C, in finite-dimensional spaces, unless another field is stated. A representation ρ of a group G assigns an invertible linear map ρ(g) to each g, with ρ(gh)=ρ(g)ρ(h). A representation is irreducible when its space is nonzero and its only invariant subspaces are zero and the whole space. Revisit invariant complements and Maschke’s theorem for these definitions and the finite-group decomposition theorem.
Consider the dihedral group of the square, D₄, of order eight. Let its rotation and reflection act by
R = ((0,−1),(1,0)), S = ((1,0),(0,−1)).
Check R⁴=I, S²=I and SRS=R⁻¹. These relations give a representation. An intertwiner T from ρ on V to σ on W is a linear map V→W satisfying Tρ(g)=σ(g)T for every g. An invertible intertwiner is an equivalence of representations. For an endomorphism, σ=ρ, so the condition says T commutes with every symmetry. It does not merely say that T is a linear map V→V.
Predict
Predict all endomorphisms of this square representation. Does a quarter turn itself qualify? It commutes with R, but SR=−RS, so it fails the reflection condition. Write T=((a,b),(c,d)) before reading the calculation below. Compare TR with RT, then TS with ST. Checking the two generators suffices: intertwining is preserved under products and inverses, so it propagates to every group element.
The representation is irreducible over C. Indeed, a proper nonzero subspace of C² would be a line. If it is R-invariant it must be an eigenline of R. The two eigenlines are spanned by (1,−i) and (1,i), with eigenvalues i and −i; S exchanges them. Neither line is invariant under both generators. This is a direct proof, independent of the scalar-endomorphism result we will establish.
Experiment
Complete the matrix products on paper, then compare the entries in this table.
| Required equality | Left-hand matrix | Right-hand matrix | Consequence |
|---|
| TR=RT | ((b,−a),(d,−c)) | ((−c,−d),(a,b)) | c=−b and d=a |
| TS=ST | ((a,−b),(c,−d)) | ((a,b),(−c,−d)) | b=c=0 |
Together these equations force T=aI. The first equation alone leaves a two-dimensional family; the second removes its extra direction. Record which equation rules out your predicted quarter turn. This calculation is a finite certificate for this representation, not a proof about all irreducibles.
Now change coordinates by P=((1,1),(0,1)), setting σ(g)=Pρ(g)P⁻¹. Predict whether an intertwiner from ρ to σ must still be scalar in the displayed coordinates.
Check the changed-coordinate calculation
The matrices are σ(r)=((1,−2),(1,−1)) and σ(s)=((1,−2),(0,−1)). If Tρ(g)=σ(g)T, multiplication on the left by P⁻¹ shows that P⁻¹T commutes with ρ(g). Thus T=aP, not generally aI. For example P itself satisfies both generator equations and is invertible. “Scalar endomorphism” compares a representation with itself; an equivalence between different coordinate descriptions need not look scalar.
Understand
First prove a statement valid over any field. If T is an intertwiner and v lies in ker T, then Tρ(g)v=σ(g)Tv=0. Thus ker T is invariant. If w=Tv lies in im T, then σ(g)w=Tρ(g)v lies in im T; the image is invariant as well. For irreducible nonzero V and W, a nonzero T therefore has kernel zero and image W. It is an isomorphism. This is the first form of Schur’s lemma. The zero map is always an intertwiner and must not be called an isomorphism.
For a complex irreducible representation, take an intertwining endomorphism T. A complex eigenvalue λ exists because V is finite-dimensional and nonzero. The map T−λI also intertwines and has a nonzero kernel. The preceding result says a nonzero such map would be invertible, impossible here. Hence T−λI=0: every endomorphism is scalar. The eigenvalue step is exactly where the complex-field hypothesis enters. This theorem is different from Schur triangularization of an arbitrary complex matrix; triangularization neither assumes a group action nor says all entries reduce to one scalar.
Let Hom_G(V,W) denote the vector space of intertwiners. For two complex irreducibles it has dimension zero when they are inequivalent: any nonzero member would give an equivalence. If A:V→W is one equivalence, then A⁻¹T is an endomorphism of V for every T in Hom_G(V,W), so T=λA. Therefore its dimension is one. The conclusion concerns irreducibles, not arbitrary representations of the same dimension.
If G is abelian, each ρ(g) commutes with every ρ(h), so Schur’s lemma makes every ρ(g) scalar in an irreducible complex representation. Every line would then be invariant. Nonzero irreducibility forces dimension one. For a finite abelian group, Maschke’s theorem consequently decomposes every complex representation into invariant lines, giving a simultaneous eigenbasis. Separately diagonalizing individual matrices does not produce a common basis: our R and S are each diagonalizable, yet their eigenlines do not coincide. Commutation and the invariant-line argument supply the missing compatibility.
Apply
Change the field, not the equation. On R² let C₃ act by rotation
A = ((−1/2,−√3/2),(√3/2,−1/2)).
Prove this representation is irreducible over R, then find a nonscalar commuting endomorphism. Do not infer reducibility just from a complex eigenvalue.
Check the field boundary
A has characteristic polynomial z²+z+1, with no real root. A real invariant line would supply a real eigenvector, so no such line exists. Nevertheless J=R above commutes with A because A=−I/2+(√3/2)J. J is not a real scalar matrix. More precisely, TJ=JT forces T=((a,b),(−b,a)), a two-dimensional real endomorphism space. Every nonzero member has determinant a²+b²>0, consistent with the first form of Schur’s lemma. The scalar form fails over R because such an endomorphism need not have a real eigenvalue. After complexification the two J-eigenlines are invariant under A, so the representation splits into two complex one-dimensional representations. It is not complex irreducible.
For a second changed case, take the direct sum of two copies of the square representation. Write an endomorphism as four two-by-two blocks. Each block must intertwine a square copy with another, so each is scalar. The resulting commutant has complex dimension four and consists of block matrices ((aI,bI),(cI,dI)). This explains why multiplicities create more endomorphisms even though each individual summand is irreducible.
Continue
Keep three claims separate: nonzero intertwiners between irreducibles are isomorphisms over any field; scalar endomorphisms use an algebraically closed field such as C; complete decomposition for finite groups over C comes from Maschke’s theorem. None of these is inferred from one computed matrix example.
Continue to characters and orthogonality, where averaging an action on Hom spaces turns the dimensions zero and one into character inner products. For background on the different matrix theorem, return to Schur triangularization and normal operators.
