Visible actions on symmetric spaces

Toshiyuki KOBAYASHI Research Institute for Mathematical Sciences Kyoto University

Abstract

A visible action on a complex manifold is a holomorphic action that admits a J-transversal totally real submanifold S. It is said to be strongly visible if there exists an orbit-preserving anti-holomorphic diffeomorphism σ such that σ|_{S} = id.

In this paper, we prove that for any Hermitian symmetric space D = G/K the action of any symmetric subgroup H is strongly visible. The proof is carried out by finding explicitly an orbit-preserving antiholomorphic involution and a totally real submanifold S.

Our geometric results provide a uniform proof of various multiplicityfree theorems of irreducible highest weight modules when restricted to reductive symmetric pairs, for both classical and exceptional cases, for both finite and infinite dimensional cases, and for both discrete and continuous spectra.

Mathematics Subject Classifications (2000): Primary 53C35. Secondary 14M15, 22E46, 32M15, 32M05, 57S20.

Keywords and phrases: visible action, complex manifold, symmetric space, multiplicity-free representation, semisimple Lie group

Contents

1 Introduction and main results 2

2 Preliminary results 4

3 Visible actions on non-compact G/K 12 4 Visible actions on compact G/K 18

5 Visible actions of unipotent subgroups 24

6 Applications to representation theory 25

7 Coisotropic, polar, and visible actions 28

1 Introduction and main results

Suppose a Lie group H acts holomorphically on a connected complex manifold D with complex structure J. We recall from [15, Definition 2.3]:

Definition 1.1. The action is visible if there exist a (non-empty) H-invariant open subset D′ of D and a totally real submanifold S of D′ satisfying the following two conditions:

S meets every H-orbit in D′,  (1.1.1)
J_{x}(T_{x}S) ⊂ T_{x}(H · x) for all x ∈ S (J-transversality).  (1.1.2)

Obviously, a transitive action is visible. Conversely, a visible action requires the existence of an H-orbit whose dimension is at least half the real dimension of D. Further, in [16, Definition 3.3.1], we have introduced:

Definition 1.2. The action is strongly visible if there exist an H-invariant open subset D′ of D, a submanifold S (slice) of D′, and an anti-holomorphic diffeomorphism σ of D′ satisfying the following three conditions:

S meets every H-orbit in D′,  (1.2.1)
σ|_{S} = id,  (1.2.2)
σ preserves each H-orbit in D′.  (1.2.3)

A strongly visible action is visible (see [16, Theorem 4]). The concept of strongly visible actions is used as a crucial assumption on base spaces D for the propagation theorem of multiplicity-free property from fibers to spaces of sections of equivariant holomorphic vector bundles over D (see [17]).

The aim of this article is to give a systematic study of strongly visible actions on symmetric spaces. In a previous paper [16, Theorem 11], we have discussed the case where D is a complex symmetric space G_{ℂ}/K_{ℂ}:

Fact 1.3. Suppose G/K is a Riemannian symmetric space, and G_{ℂ}/K_{ℂ} is its complexification. Then, the G-action on the complex symmetric space G_{ℂ}/K_{ℂ} is strongly visible.

In this article, our focus is on the case where D is a Hermitian symmetric space. A typical example is:

Example 1.4. Let G = KAN be the Iwasawa decomposition of G := SL(2, ℝ). Then, all of the actions of K, A and N on the Hermitian symmetric space D := G/K are strongly visible, as one can see easily from the following figures where D is realized as the Poincaré disk and the dotted lines give slices:

K-orbits  A-orbits  N-orbits

Figure 1.3 (a)  Figure 1.3 (b)  Figure 1.3 (c)

In this case, both (G, K) and (G, A) form symmetric pairs, while N is a maximal unipotent subgroup of G.

These three examples are generalized to the following Theorems 1.5 and 1.10, which are our main results of this paper.

Theorem 1.5. Suppose G is a semisimple Lie group such that D := G/K is a Hermitian symmetric space. Then, for any symmetric pair (G, H), the H-action on D is strongly visible.

Example 1.6 (the Siegel upper half space). Let G/K = Sp(n, ℝ)/U(n). Then, the action of the subgroup H is strongly visible if H = GL(n, ℝ), U(p, q), Sp(p, ℝ) × Sp(q, ℝ) (p + q = n), or Sp(\frac{n}{2}, ℂ) (n : even).

The pair (G × G, diag(G)) is a classic example of symmetric pairs, where diag(G) := {(g, g) : g ∈ G}. Thus, Theorem 1.5 also includes:

Theorem 1.7. Let D_{1}, D_{2} be two Hermitian symmetric spaces of a compact simple Lie group G_{U}. Then the diagonal action of G_{U} on D_{1} ×D_{2} is strongly visible.

Example 1.8. The diagonal action of SU(n) on the direct product of two Grassmann varieties Gr_{p}(ℂ^{n}) × Gr_{k}(ℂ^{n}) (1 ≤ p, k ≤ n) is strongly visible.

Theorem 1.9. Let D be a non-compact irreducible Hermitian symmetric space G/K, and \overline{D} denote the Hermitian symmetric space equipped with the opposite complex structure.

1) The diagonal action of G on D × D is strongly visible.

2) The diagonal action of G on D × \overline{D} is strongly visible.

The third case of Example 1.4 is generalized as follows:

Theorem 1.10. Suppose D = G/K is a Hermitian symmetric space without compact factor. If N is a maximal unipotent subgroup of G, then the Naction on D is strongly visible.

This paper is organized as follows: In Section 2, we translate geometric conditions of (strongly) visible actions into an algebraic language by using the structural theory of semisimple symmetric pairs. The proof of our main result, Theorem 1.5, is given in Section 3 (non-compact case) and in Section 4 (compact case), and that of Theorem 1.10 is given in Section 5.

As Fact 1.3 gives a new proof (see [17]) of the Cartan–Gelfand theorem that the Plancherel formula for a Riemannian symmetric space is multiplicityfree (induction of representations), our geometric results here give a number of multiplicity-free theorems, in particular, in branching problems (restriction of representations) for both finite and infinite dimensional representations and for discrete and continuous spectra. Such applications to representation theory are discussed in Section 6.

Concepts related to visible actions on complex manifolds are polar actions on Riemannian manifolds and coisotropic actions on symplectic manifolds. Since Hermitian symmetric spaces D are Kähler, we can compare these three concepts for D. Some comments on this are given in Section 7.

2 Preliminary results

This section provides sufficient conditions by means of Lie algebras for the geometric conditions (1.1.1) – (1.2.3) for strongly visible actions in the setting where D is a Hermitian symmetric space.

2.1 Semisimple symmetric pairs

Let G be a semisimple Lie group with Lie algebra 𝔤. Suppose that τ is an involutive automorphism of G. We write G_{0}^{τ} for the identity component of G^{τ} := {g ∈ G : τg = g}. The pair (G, H) (or the pair (𝔤, 𝔥) of their Lie algebras) is called a (semisimple) symmetric pair if a subgroup H satisfies G_{0}^{τ} ⊂ H ⊂ G^{τ}. Unless otherwise mentioned, we shall take H to be G_{0}^{τ} because Theorem 1.5 follows readily from this case. We use the same letter τ to denote its differential, and set

𝔤^{±τ} := {Y ∈ 𝔤 : τY = ±Y }.

Then, 𝔤^{τ} is the Lie algebra of H. Since τ^{2} = id, we have a direct sum decomposition 𝔤 = 𝔤^{τ} + 𝔤^{−τ}.

It is known that there exists a Cartan involution θ of G commuting with τ. Take such θ, and we write K := G^{θ} = {g ∈ G : θg = g}. Then we have a Cartan decomposition 𝔤 = 𝔨 + 𝔭 ≡ 𝔤^{θ} + 𝔤^{−θ}. We shall allow G to be non-linear, and therefore K is not necessarily compact. The real rank of 𝔤, denoted by ℝ- rank 𝔤, is defined to be the dimension of a maximal abelian subspace of 𝔤^{−θ}.

As (τθ)^{2} = id, the pair (𝔤, 𝔤^{τθ}) also forms a symmetric pair. The Lie group G^{τθ} = {g ∈ G : (τθ)g = g} is a reductive Lie group with Cartan involution θ|_{G^{τθ}}, and its Lie algebra 𝔤^{τθ} is reductive with Cartan decomposition

𝔤^{τθ} = 𝔤^{θ,τθ} + 𝔤^{−θ,τθ} = 𝔤^{θ,τ} + 𝔤^{−θ,−τ},

where we have used the notation 𝔤^{−θ,−τ} and alike, defined as follows:

𝔤^{−θ,−τ} := {Y ∈ 𝔤 : (−θ)Y = (−τ)Y = Y }.

The real rank of 𝔤^{θτ} is referred to as the split rank of the symmetric space G/H, denoted by ℝ-rank G/H or ℝ-rank 𝔤/𝔤^{τ}. That is,

ℝ- rank 𝔤^{τθ} = ℝ- rank 𝔤/𝔤^{τ}.  (2.1.1)

In particular, ℝ- rank 𝔤 = ℝ- rank 𝔤/𝔨 if we take τ to be the Cartan involution θ.

2.2 Stability of H-orbits

Retain the setting in Subsection 2.1. Suppose furthermore that there is another involutive automorphism σ of G such that σθ = θσ and στ = τσ.

The commutativity of σ and θ implies that the automorphism σ stabilizes K, and therefore induces a diffeomorphism of G/K, for which we shall use the same letter σ. Then, σ sends H-orbits on G/K to H-orbits because στ = τσ. However, σ may permute H-orbits, and may not preserve each H-orbit. In this subsection, we give a sufficient condition in terms of the real rank condition that σ preserves each H-orbit. The conclusion of Lemma 2.2 meets the requirements (1.2.1)–(1.2.3) of strongly visible conditions, and thus Lemma 2.2 will play a key role in the proof of our main theorem.

Lemma 2.2. Let G be a semisimple Lie group, θ a Cartan involution, and D := G/K the corresponding Riemannian symmetric space. Let σ and τ be involutive automorphisms of G. We set H := G_{0}^{τ}. We assume the following two conditions:

σ, τ and θ commute with one another.  (2.2.1)
ℝ- rank 𝔤^{τθ} = ℝ- rank 𝔤^{σ,τθ}.  (2.2.2)

Then the followings hold:

1) For any x ∈ D, there exists g ∈ H such that σ(x) = g · x. In particular, σ preserves each H-orbit on D.

2) Let 𝔞 be a maximal abelian subspace in

𝔤^{−θ,σ,τθ} = 𝔤^{−θ,σ,−τ} := {Y ∈ 𝔤 : (−θ)Y = σY = (−τ)Y = Y }.  (2.2.3)

Then, the submanifold S := (exp 𝔞)K meets every H-orbit in D and σ|_{S} = id.

Proof. 1) First, let us show that if h ∈ H then g := σ(h)h^{−1} ∈ H. In fact, by using στ = τσ and τ(h) = h, we have

τ(g) = τσ(h) τ(h^{−1}) = στ(h) τ(h)^{−1} = σ(h) h^{−1} = g.

Hence, g ∈ G^{τ}. Moreover, since the image of the continuous map

H → G, h ↦ σ(h) h^{−1}

is connected, we have proved g ∈ G_{0}^{τ} = H.

Next, let 𝔞 be as in 2). In light of the Cartan decomposition 𝔤^{σ,τθ} = 𝔤^{θ,σ,τθ} + 𝔤^{−θ,σ,τθ}, we have dim 𝔞 = ℝ- rank 𝔤^{σ,τθ}. Then, the assumption (2.2.2) shows dim 𝔞 = ℝ- rank 𝔤^{τθ}. As 𝔤^{τθ} = 𝔤^{θ,τ} + 𝔤^{−θ,−τ} is a Cartan decomposition of 𝔤^{τθ}, this means that 𝔞 is also a maximal abelian subspace in 𝔤^{−θ,−τ}.

We write A for the analytic subgroup of G with Lie algebra 𝔞. Then we have a generalized Cartan decomposition (see [4, §2])

G = HAK.  (2.2.4)

Fix x ∈ G/K. Then, according to the decomposition (2.2.4), we find h ∈ H and a ∈ A such that

x = ha · o,

where o := eK ∈ G/K.

On the other hand, we have σ(a) = a because 𝔞 ⊂ 𝔤^{−θ,σ,−τ} ⊂ 𝔤^{σ}. Therefore, if we set g := σ(h)h^{−1} ∈ G_{0}^{τ}, then

σ(x) = σ(h) σ(a) · o = σ(h) h^{−1}ha · o = g · x.

In particular, we have σ(H ·x) = H ·σ(x) = Hg ·x = H ·x. Thus, σ preserves each H-orbit on G/K.

2) The submanifold S meets every H-orbit by (2.2.4), and σ|_{S} = id because 𝔞 ⊂ 𝔤^{σ}.

2.3 Involutions on Hermitian Symmetric Space G/K

This subsection gives a brief review on basic results on submanifolds of Hermitian symmetric spaces.

Let G be a non-compact simple Lie group G with Cartan decomposition 𝔤 = 𝔨+𝔭. G is said to be of Hermitian type if the center 𝔠(𝔨) of 𝔨 is non-trivial. Then, it is known that dim 𝔠(𝔨) = 1 and that there exists a characteristic element Z ∈ 𝔠(𝔨) such that

𝔤_{ℂ} := 𝔤 ⊗ ℂ = 𝔨_{ℂ} ⊕ 𝔭_{+} ⊕ 𝔭_{−}  (2.3.1)

is the eigenspace decomposition of ad(Z) with eigenvalues 0, \sqrt{−1} and −\sqrt{−1}, respectively.

Let G_{ℂ} be a connected complex Lie group with Lie algebra 𝔤_{ℂ}, and Q^{−} the parabolic subgroup of G_{ℂ} with Lie algebra 𝔨_{ℂ} + 𝔭_{−}. Then the natural homomorphism G → G_{ℂ} induces an open embedding G/K ↪ G_{ℂ}/Q^{−}, from which the complex structure on G/K is induced. This complex structure on G/K is given by the left G-translation of

J_{o} := Ad(exp(\frac{π}{2} Z)) : T_{o}(G/K) → T_{o}(G/K)  (2.3.2)

at the origin o = eK ∈ G/K.

Suppose τ is an involutive automorphism of G. We may and do assume that τ commutes with θ (by taking a conjugation by an inner automorphism if necessary). Then τ stabilizes the Cartan decomposition 𝔤 = 𝔨 + 𝔭, and particularly the one dimensional subspace 𝔠(𝔨). Since τ^{2} = id, we have either

τZ = Z,  (2.3.3)

or

τZ = −Z.  (2.3.4)

It follows from the definition (2.3.2) of the complex structure on G/K that the condition (2.3.3) has the following geometric meaning:

τ acts holomorphically on the Hermitian symmetric space G/K,

G^{τ}/K^{τ} ↪ G/K defines a complex submanifold.

On the other hand, the condition (2.3.4) implies

τ acts anti-holomorphically on the Hermitian symmetric space G/K,

G^{τ}/K^{τ} ↪ G/K defines a totally real submanifold.

We say the involution τ (or the symmetric pair (𝔤, 𝔤^{τ})) is of holomorphic type (respectively, anti-holomorphic type) if τ satisfies (2.3.3) (respectively, (2.3.4)).

The following Tables 2.3.1 and 2.3.2 give the classification of semisimple symmetric pairs (𝔤, 𝔤^{τ}) for simple Lie algebras 𝔤 such that the pair (𝔤, 𝔤^{τ}) is of holomorphic type and of anti-holomorphic type, respectively. Table 2.3.2 is equivalent to the classification of totally real symmetric spaces G^{τ}/K^{τ} of a Hermitian symmetric space G/K (see [3], [11], [12], [13]). For later purposes, we label these symmetric spaces in the left column of Tables 2.3.1 and 2.3.2.

(𝔤, 𝔤^{τ}) of holomorphic type

𝔤  𝔤^{τ}
1 𝔰𝔲(p, q)  𝔰(𝔲(i, j) + 𝔲(p − i, q − j))
2 𝔰𝔲(n, n)  𝔰𝔬^{∗}(2n)
3 𝔰𝔲(n, n)  𝔰𝔭(n, ℝ)
4 𝔰𝔬^{∗}(2n)  𝔰𝔬^{∗}(2p) + 𝔰𝔬^{∗}(2n − 2p)
5 𝔰𝔬^{∗}(2n)  𝔲(p, n − p)
6 𝔰𝔬(2, n)  𝔰𝔬(2, p) + 𝔰𝔬(n − p)
7 𝔰𝔬(2, 2n)  𝔲(1, n)
8 𝔰𝔭(n, ℝ)  𝔲(p, n − p)
9 𝔰𝔭(n, ℝ)  𝔰𝔭(p, ℝ) + 𝔰𝔭(n − p, ℝ)
10 𝔢_{6(−14)}  𝔰𝔬(10) + 𝔰𝔬(2)
11 𝔢_{6(−14)}  𝔰𝔬^{∗}(10) + 𝔰𝔬(2)
12 𝔢_{6(−14)}  𝔰𝔬(8, 2) + 𝔰𝔬(2)
13 𝔢_{6(−14)}  𝔰𝔲(5, 1) + 𝔰𝔩(2, ℝ)
14 𝔢_{6(−14)}  𝔰𝔲(4, 2) + 𝔰𝔲(2)
15 𝔢_{7(−25)}  𝔢_{6(−78)} + 𝔰𝔬(2)
16 𝔢_{7(−25)}  𝔢_{6(−14)} + 𝔰𝔬(2)
17 𝔢_{7(−25)}  𝔰𝔬(10, 2) + 𝔰𝔩(2, ℝ)
18 𝔢_{7(−25)}  𝔰𝔬^{∗}(12) + 𝔰𝔲(2)
19 𝔢_{7(−25)}  𝔰𝔲(6, 2)

Table 2.3.1 (𝔤, 𝔤^{τ}) of anti-holomorphic type

𝔤  𝔤^{τ}
20 𝔰𝔲(p, q)  𝔰𝔬(p, q)
21 𝔰𝔲(n, n)  𝔰𝔩(n, ℂ) + ℝ
22 𝔰𝔲(2p, 2q)  𝔰𝔭(p, q)
23 𝔰𝔬^{∗}(2n)  𝔰𝔬(n, ℂ)
24 𝔰𝔬^{∗}(4n)  𝔰𝔲^{∗}(2n) + ℝ
25 𝔰𝔬(2, n)  𝔰𝔬(1, p) + 𝔰𝔬(1, n − p)
26 𝔰𝔭(n, ℝ)  𝔤𝔩(n, ℝ)
27 𝔰𝔭(2n, ℝ)  𝔰𝔭(n, ℂ)
28 𝔢_{6(−14)}  𝔣_{4(−20)}
29 𝔢_{6(−14)}  𝔰𝔭(2, 2)
30 𝔢_{7(−25)}  𝔢_{6(−26)} + 𝔰𝔬(1, 1)
31 𝔢_{7(−25)}  𝔰𝔲^{∗}(8)

Table 2.3.2

2.4 Proof of Theorem 1.5 for H = K

This subsection gives a proof of Theorem 1.5 in the case where G is a noncompact simple Lie group and H = K. This is an immediate consequence of Lemma 2.2 with τ = θ (namely, H = K) if we show:

Lemma 2.4. Suppose G is a non-compact, simply connected, simple Lie group such that G/K is a Hermitian symmetric space. Let θ be a Cartan involution corresponding to K. Then there exists an involutive automorphism σ of G satisfying the following three conditions:

σ and θ commute.  (2.4.1)
ℝ- rank 𝔤 = ℝ- rank 𝔤^{σ}.  (2.4.2)
σZ = −Z.  (2.4.3)

Proof. The following table gives a choice of σ ∈ Aut(𝔤) (and hence an automorphism of the simply-connected G) for each non-compact simple Lie group G of Hermitian type:

(𝔤, 𝔤^{σ}) satisfying (2.4.1), (2.4.2) and (2.4.3)

𝔤  𝔤^{σ}  ℝ- rank 𝔤 = ℝ- rank 𝔤^{σ}
𝔰𝔲(p, q)  𝔰𝔬(p, q)  min(p, q)
𝔰𝔬^{∗}(2n)  𝔰𝔬(n, ℂ)  [\frac{1}{2}n]
𝔰𝔭(n, ℝ)  𝔤𝔩(n, ℝ)  n
𝔰𝔬(2, n) 𝔰𝔬(1, n − 1) + 𝔰𝔬(1, 1)  min(n, 2)
𝔢_{6(−14)}  𝔰𝔭(2, 2)  2
𝔢_{7(−25)}  𝔰𝔲^{∗}(8)  3

Table 2.4.1

All of these pairs (𝔤, 𝔤^{σ}) appear in Table 2.3.2, showing that they are of antiholomorphic type. The real rank condition (2.4.2) can be verified directly (see the above Table). Hence, we have proved Lemma.

Remark 2.4.2. The choice of σ is not unique. For example, we may choose 𝔤^{σ} ≃ 𝔢_{6(−26)} + ℝ in place of 𝔤^{σ} ≃ 𝔰𝔲^{∗}(8) if 𝔤 = 𝔢_{7(−25)}.

2.5 Proof of Theorem 1.9

This subsection gives a proof of Theorem 1.9 that concerns with the diagonal action of G on the direct product D × D or D × \overline{D}. We shall see that Lemma 2.4 is again a key ingredient of the proof as in the proof of Theorem 1.5 for H = K.

Let G be a non-compact, simple Lie group such that G/K is a Hermitian symmetric space. We use the letter θ′ in place of the previous θ to denote the corresponding Cartan involution of G. Then θ(g_{1}, g_{2}) := (θ′g_{1}, θ′g_{2}) defines a Cartan involution of the direct product group G × G.

We define an involutive automorphism τ of G × G by τ(g_{1}, g_{2}) := (g_{2}, g_{1}). Then (G × G)^{τ} = diag(G).

Proof of Theorem 1.9. 1) Let σ′ ∈ Aut(G) be the involution given in Lemma 2.4. Now, we set σ(g_{1}, g_{2}) := (σ′g_{1}, σ′g_{2}). Obviously, τ, θ and σ all commute. Further, σ acts on D×D as an anti-holomorphic diffeomorphism because so does σ′ on D by (2.4.3). In light that

(𝔤 ⊕ 𝔤)^{τθ} = {(X, θ′X) : X ∈ 𝔤} ≃ 𝔤,
(𝔤 ⊕ 𝔤)^{σ,τθ} = {(X, θ′X) : X ∈ 𝔤^{σ′}} ≃ 𝔤^{σ′},

we have ℝ- rank(𝔤 ⊕ 𝔤)^{τθ} = ℝ- rank(𝔤 ⊕ 𝔤)^{σ,τθ} from (2.4.2). Let 𝔞′ be a maximal abelian subspace of 𝔤^{σ′,−θ′}. We set

𝔞 := {(X, −X) : X ∈ 𝔞′} = {(X, θ′X) : X ∈ 𝔞′},

and define a submanifold of D × D by

S := exp 𝔞 · (o, o).

Then Lemma 2.2 applied to (G × G, D × D) shows that the diagonal action of G on D × D = (G × G)/(K × K) is strongly visible.

2) We define an involution σ of G × G by σ := τθ, namely, σ(g_{1}, g_{2}) := (θ′g_{2}, θ′g_{1}) for g_{1}, g_{2} ∈ G. (This σ is different from the one used in 1).) Then σ acts anti-holomorphically on D × \overline{D}. Obviously, σ = τθ, τ and θ all commute. Furthermore, the rank condition ℝ- rank(𝔤 ⊕ 𝔤)^{τθ} = ℝ- rank(𝔤 ⊕ 𝔤)^{σ,τθ} automatically follows from σ = τθ. Thus, it follows from Lemma 2.2 that the diagonal action of G on D × \overline{D} is also strongly visible. Hence, Theorem 1.9 has been proved.

3 Visible actions on non-compact G/K

In this section, we give a proof of Theorem 1.5 in the case where G has no compact factor. The compact case will be proved in Section 4.

3.1 Existence of anti-holomorphic involutions

Throughout this section, let G be a non-compact, simply-connected, simple Lie group of Hermitian type. Suppose τ is an involutive automorphism of G such that H = G_{0}^{τ}. Then, owing to Lemma 2.2, the proof of Theorem 1.5 is reduced to the following:

Lemma 3.1. Suppose that 𝔤 = 𝔨+𝔭 is a real simple Lie algebra of Hermitian type and that Z is a generator of the center c(𝔨) of 𝔨. Let τ be an involutive automorphism of 𝔤, commuting with the Cartan involution θ. Then there exists an involutive automorphism σ of 𝔤 satisfying the following three conditions:

σ, τ and θ commute with one another.  (3.1.1)
ℝ- rank 𝔤^{τθ} = ℝ- rank 𝔤^{σ,τθ}.  (3.1.2)
σZ = −Z.  (3.1.3)

The rest of this section will be spent for the proof of Lemma 3.1. We shall divide the proof into the following three cases:

Case I. τZ = −Z  (Table 2.3.2, 20 ∼ 31).
Case II. τZ = Z, 𝔤 is classical  (Table 2.3.1, 1 ∼ 9).
Case III. τZ = Z, 𝔤 is exceptional  (Table 2.3.1, 10 ∼ 19).

We have already proved Lemma 3.1 for τ = θ in Subsection 2.4 (namely, special cases of Cases II and III). In the subsequent subsections, we shall choose σ in the following way:

Case I. Take σ := τθ (Subsection 3.2).
Case II. Take σ as in Table 3.3.1 (Subsection 3.3).
Case III. Take σ such that 𝔤^{σ} ≃ 𝔰𝔭(2, 2) if 𝔤 = 𝔢_{6(−14)}, and 𝔤^{σ} ≃ 𝔰𝔲^{∗}(8)
if 𝔤 = 𝔢_{7(−25)} (Subsections 3.4–3.6).

3.2 Proof of Lemma 3.1 in Case I

Suppose τZ = −Z. We set σ := τθ. Then, the conditions (3.1.1) and (3.1.3) are automatically satisfied. Since 𝔤^{σ,τθ} = 𝔤^{τθ}, the real rank condition (3.1.2) is obvious. Thus, Lemma 3.1 in Case I is proved.

3.3 Proof of Lemma 3.1 in Case II

There are 9 families of semisimple symmetric pairs (𝔤, 𝔤^{τ}) in Case II, namely, the cases 1 ∼ 9 as labeled in Table 2.3.1. Then, we can take an involutive automorphism σ of 𝔤 as in the following Table 3.3.1. The conditions (3.1.1) and (3.1.3) are clear. The real rank condition (3.1.2) is verified directly (see the right column of the table below). Hence, Lemma 3.1 in Case II is proved.

(𝔤, 𝔤^{τ})  𝔤^{σ}  𝔤^{σ,τ} = 𝔤^{σ} ∩ 𝔤^{τ}  ℝ- rank 𝔤^{τθ} = ℝ- rank 𝔤^{σ,τθ}
1  𝔰𝔬(p, q)  𝔰𝔬(i, j) + 𝔰𝔬(p − i, q − j) min(i, q − j) + min(p − i, j)
2  𝔰𝔬(n, n)  𝔰𝔬(n, ℂ)  n
3  𝔰𝔩(n, ℂ) + ℝ  𝔤𝔩(n, ℝ)  [\frac{n}{2}]
4  𝔰𝔬(n, ℂ)  𝔰𝔬(p, ℂ) + 𝔰𝔬(n − p, ℂ)  min(p, n − p)
5  𝔰𝔬(n, ℂ)  𝔰𝔬(p, n − p)  [\frac{p}{2}] + [\frac{n−p}{2}]
6  𝔰𝔬(1, 1) + 𝔰𝔬(1, n − 1) 𝔰𝔬(1, p) + 𝔰𝔬(n − p − 1)  min(2, n − p)
7  𝔰𝔬(1, n) + 𝔰𝔬(1, n)  𝔰𝔬(1, n)  1
8  𝔤𝔩(n, ℝ)  𝔰𝔬(p, n − p)  n
9  𝔤𝔩(n, ℝ)  𝔤𝔩(p, ℝ) + 𝔤𝔩(n − p, ℝ)  min(p, n − p)

Table 3.3.1

3.4 ϵ-family of symmetric pairs

We shall prove Lemma 3.1 in Case III. We shall take σ so that 𝔤^{σ} ≃ 𝔰𝔭(2, 2) for 𝔤 = 𝔢_{6(−14)}, and 𝔤^{σ} ≃ 𝔰𝔲^{∗}(8) for 𝔤 = 𝔢_{7(−25)}. The non-trivial part is to prove that we can take σ such that στ = τσ. (Two involutions do not always commute. See Subsection 4.2 for counterexamples in classical cases.)

We have already proved that Theorem 1.5 holds if τ = θ (see Subsection 2.4) or if τ is of anti-holomorphic type (see Subsection 3.2). Building on these cases, we shall give a proof of the remaining cases, that is, Lemma 3.1 in Case III. For this, we set up to make new pairs from old. First, we recall quickly the notion of ϵ-families of symmetric pairs [27], which enables us to avoid tedious computations for exceptional groups. Our approach below might be of some use for a systematic study of three involutions (σ, τ, θ) of complex simple Lie algebras (cf. [23]).

Let 𝔤 be a semisimple Lie algebra, τ an involutive automorphism of 𝔤, and θ a Cartan involution of 𝔤 commuting with τ. Fix a maximal abelian subspace 𝔞 of 𝔤^{−θ,−τ}. For λ ∈ 𝔞^{∗}, we define 𝔤(𝔞; λ) := {X ∈ 𝔤 : ad(H)X = λ(H)X for H ∈ 𝔞}, and set Σ(𝔞) ≡ Σ(𝔤, 𝔞) := {λ ∈ 𝔞^{∗}\{0} : 𝔤(𝔞; λ) ≠ {0}}. Rossmann proved that Σ(𝔞) satisfies the axiom of root system ([31, Theorem 5]). We say a map

ϵ : Σ(𝔞) ∪ {0} → {1, −1}

is a signature of Σ(𝔞) if ϵ satisfies ϵ(α + β) = ϵ(α)ϵ(β) for any α, β, α + β ∈ Σ(𝔞), ϵ(−α) = ϵ(α) for any α ∈ Σ(𝔞) and ϵ(0) = 1. To a signature ϵ of Σ(𝔞), we associate an involution τ_{ϵ} of 𝔤 defined by

τ_{ϵ}(X) := ϵ(λ)τ(X) for X ∈ 𝔤(𝔞; λ), λ ∈ Σ(𝔞) ∪ {0}.

Then τ_{ϵ} defines another symmetric pair (𝔤, 𝔥_{ϵ}). The set

F ((𝔤, 𝔥)) := {(𝔤, 𝔥_{ϵ}) : ϵ is a signature of Σ(𝔞)}

is said to be an ϵ-family of symmetric pairs ([27, §6]). This set is also referred to as K_{ϵ}-family of symmetric pairs if τ = θ. For example, {(𝔰𝔩(n, ℝ), 𝔰𝔬(p, n − p)) : 0 ≤ p ≤ n} forms a K_{ϵ}-family.

To make new pairs (σ, τ) from old, the following result is useful:

Lemma 3.4. Let τ and σ be involutive automorphisms of 𝔤.

1) If the pair (σ, τ) satisfies (3.1.1) and (3.1.2), then so does (σ, τ_{ϵ}) for any τ_{ϵ} (the choice of 𝔞 is specified in the proof).

2) If the pair (σ, τ) satisfies (3.1.1), then so does (σ, τθ).

Proof. 1) Take a maximal abelian subspace 𝔞 in

𝔤^{−θ,σ,τθ} = 𝔤^{−θ,σ,−τ} = {Y ∈ 𝔤 : (−θ)Y = σY = (−τ)Y = Y }.

Then, 𝔞 is also a maximal abelian subspace of 𝔤^{−θ,−τ} by the rank condition (3.1.2). Let τ_{ϵ} be an involutive automorphism of 𝔤 associated to a signature ϵ of Σ(𝔤, 𝔞).

To see στ_{ϵ} = τ_{ϵ}σ, we take an arbitrary root vector X ∈ 𝔤(𝔞; λ). Then we have

στ_{ϵ}(X) = σ(ϵ(λ)τ(X)) = ϵ(λ)σ(τ(X)).

On the other hand, σX ∈ 𝔤(𝔞; σλ) = 𝔤(𝔞; λ) because σ|_{𝔞} = id. Thus,

τ_{ϵ}σ(X) = ϵ(λ)τ(σ(X)) = ϵ(λ)σ(τ(X)).

This proves στ_{ϵ} = τ_{ϵ}σ on 𝔤(𝔞; λ) for all λ. Hence, the pair (σ, τ_{ϵ}) satisfies (3.1.1).

As ϵ(0) = 1, we have τ_{ϵ}(Y) = τ(Y) = −Y if Y ∈ 𝔞 (⊂ 𝔤(𝔞; 0)). This implies

𝔞 ⊂ 𝔤^{−θ,σ,τ_{ϵ}θ} = 𝔤^{−θ,σ,−τ_{ϵ}} = {Y ∈ 𝔤 : (−θ)Y = σY = (−τ_{ϵ})Y = Y }.

Since ℝ- rank 𝔤^{σ,τ_{ϵ}θ} is by definition the dimension of a maximal abelian subspace contained in 𝔤^{−θ,σ,τ_{ϵ}θ}, we have

ℝ- rank 𝔤^{σ,τ_{ϵ}θ} ≥ dim 𝔞 = ℝ- rank 𝔤^{τθ} = ℝ- rank 𝔤^{τ_{ϵ}θ} ≥ ℝ- rank 𝔤^{σ,τ_{ϵ}θ}.

Therefore, the pair (σ, τ_{ϵ}) satisfies (3.1.2).

2) Obvious from the definition.

3.5 New pairs (σ, τ) from old

We have already proved Lemma 3.1 in the following two cases:

i) τ = θ (equivalent to Lemma 2.4). ii) τ satisfies τZ = −Z (Subsection 3.2). The lemma below gives a coherent understanding of the set of involutions for which Lemma 3.1 holds.

Lemma 3.5. Let τ be an involutive automorphism of 𝔤 commuting with a Cartan involution θ.

1) If Lemma 3.1 holds for τ, then so does it for τ_{ϵ} associated to any signature ϵ of Σ(𝔤, 𝔞). To be more precise, let σ be an automorphism of 𝔤 satisfying (3.1.1), (3.1.2) and (3.1.3), and 𝔞 be a maximal abelian subspace in 𝔤^{−θ,σ,−τ}. Then, τ_{ϵ}θ = θτ_{ϵ} and the same σ satisfies (3.1.1), (3.1.2) and (3.1.3) for τ_{ϵ}.

2) If Lemma 3.1 holds for τ by taking an involution σ, then so does it for τθ, provided ℝ- rank 𝔤^{τ} = ℝ- rank 𝔤^{σ,τ}. Namely, the same σ satisfies (3.1.1), (3.1.2) and (3.1.3) for τθ.

Proof. Readily follows from Lemma 3.4.

The next subsection shows that Lemma 3.1 in the exceptional cases (namely, (10) ∼ (19) in Table 2.3.1) is reduced to (i) or (ii) by an iterating application of Lemma 3.5.

3.6 Proof of Lemma 3.1 in Case III

In terms of the labels of symmetric pairs in Tables 2.3.1 and 2.3.2, the scheme of the proof is described in the following diagrams:

𝔤 = 𝔢_{6(−14)}
10 | 11 | 12
|
13 | 14
𝔤 = 𝔢_{7(−25)}
15 | 16 | 30
|
17 | 18
19 | 31

Here, the symmetric pairs connected by horizontal path mean that they belong to the same ϵ-family. That is, we recall from [27] that the following is a list of an ϵ-family of symmetric pairs: 𝔤 = 𝔢_{6(−14)} : {(10), (11), (12)}, {(13), (14)}. 𝔤 = 𝔢_{7(−25)} : {(15), (16), (30)}, {(17), (18)}, {(19), (31)}.

The circle (i.e. (10) and (15)) means that τ = θ, while the box (i.e. (30) and (31)) means that τ is of anti-holomorphic type.

Since Lemma 3.1 holds for (10), (15) (τ = θ case) and also for (30), (31) (τZ = −Z case), so does it for any member of {(10), (11), (12)}, {(15), (16), (30)}, {(19), (31)} by Lemma 3.5 (1).

A next step is an observation: 𝔢_{6(−14)} : Lemma 3.1 holds for (11) by taking 𝔤^{σ} ≃ 𝔰𝔭(2, 2). If (𝔤, 𝔤^{τ}) ≃ (11),

then (𝔤, 𝔤^{τθ}) ≃ (13) and (𝔤^{τ}, 𝔤^{σ,τ}) ≃ (𝔰𝔬^{∗}(10) + 𝔰𝔬(2), 𝔰𝔭(2, ℂ)). 𝔢_{7(−25)} : Lemma 3.1 holds for (16) by taking 𝔤^{σ} ≃ 𝔰𝔲^{∗}(8). If (𝔤, 𝔤^{τ}) ≃ (16),

then (𝔤, 𝔤^{τθ}) ≃ (17) and (𝔤^{τ}, 𝔤^{σ,τ}) ≃ (𝔢_{6(−14)} + 𝔰𝔬(2), 𝔰𝔭(2, 2)). Here, the proof of the above isomorphisms concerning (𝔤^{τ}, 𝔤^{σ,τ}) is straight-

forward because we know στ = τσ.

Then, Lemma 3.1 holds for (13) and (17) by Lemma 3.5 (2). In turn, Lemma 3.1 holds for any member of {(13), (14)} and {(17), (18)} by using again Lemma 3.5 (1).

This proves Lemma 3.1 for all exceptional cases (10) ∼ (19), namely, in Case III. Thus, we have finished the proof of Lemma 3.1.

4 Visible actions on compact G/K

This section gives a proof of Theorem 1.5 in the case where D is a compact symmetric space. This is reduced to the following two cases:

Case I. D = G_{U}/K.

Case II. D = (G_{U} × G_{U})/(K_{1} × K_{2}), H_{U} = diag(G_{U}). Here, G_{U} is a connected compact simple Lie group, and G_{U}/K, G_{U}/K_{1} and G_{U}/K_{2} are compact irreducible Hermitian symmetric spaces. (Instead of the letters G and H, we shall use G_{U} and H_{U} to emphasize compactness in Section 4).

Theorem 1.5 in Cases I and II will be proved in Subsections 4.4 and 4.5, respectively. Together with the non-compact case proved in Section 3, the proof of Theorem 1.5 will be completed.

4.1 Existence of anti-holomorphic involutions

Without loss of generality, we may and do assume that G_{U} is simply connected. We denote by τ and θ the involutive automorphisms of G_{U} such that (G_{U})^{τ} = H_{U} and (G_{U})^{θ} = K, respectively. Since G_{U} is simply connected, both H_{U} and K are automatically connected.

For g ∈ G_{U}, we define an involution τ^{g} by

τ^{g}(x) := gτ(g^{−1}xg)g^{−1} (x ∈ G_{U}).

Here is a key lemma:

Lemma 4.1. Suppose we are in the above setting. Then, there exists an involutive automorphism σ of G_{U} satisfying the following three conditions (by an abuse of notation, we replace τ^{g} with τ below by taking some g ∈ G_{U} if necessary):

σθ = θσ, στ = τσ.  (4.1.1)
(𝔤_{U})^{σ,−τ,−θ} contains a maximal abelian subspace in (𝔤_{U})^{−τ,−θ}. (4.1.2)
The induced action of σ on D = G_{U}/K is anti-holomorphic.  (4.1.3)

Remark. Lemma 4.1 is a compact case counterpart of Lemma 3.1. In contrast to the condition (3.1.1) in the non-compact case, we have not required τθ = θτ here. In fact, different from the non-compact case, it may happen that τ^{g}θ ≠ θτ^{g} for all g ∈ G_{U} (see Type II in Subsection 4.2). Nevertheless, Lemma 4.1 asserts that one can find σ that commutes with θ and τ simultaneously.

We shall divide the proof into the following cases: Type I: τ^{g}θ = θτ^{g} for some g ∈ G_{U}. Type II: τ^{g}θ ≠ θτ^{g} for any g ∈ G_{U}.

4.2 Proof of Lemma 4.1 in Type I

This subsection gives a proof of Lemma 4.1 in Type I. Type I parallels the corresponding result (see Lemma 3.1) for the non-compact Riemannian symmetric pair (G, K) dual to (G_{U}, K).

Let G_{ℂ} be a complexification of G_{U}. Since G_{U} is simply connected, G_{ℂ} is also simply connected. Therefore, any automorphism of G_{U} extends to a holomorphic automorphism of the complex Lie group G_{ℂ}. For τ, θ, … ∈ Aut(G_{U}), we shall use the same letters τ, θ, … to denote the holomorphic extensions ∈ Aut(G_{ℂ}), and also the differentials ∈ Aut(𝔤_{ℂ}).

Let G be a connected subgroup of G_{ℂ} with Lie algebra

𝔤 := 𝔨 + \sqrt{−1}(𝔤_{U})^{−θ} = (𝔤_{U})^{θ} + \sqrt{−1}(𝔤_{U})^{−θ}.  (4.2.1)

Then G is a non-compact simple Lie group with a maximal compact subgroup K.

In Type I, we shall simply write τ for τ^{g} and may assume τθ = θτ. Then, the decomposition (4.2.1) is invariant by τ, and consequently, the holomorphic extension τ ∈ Aut(G_{ℂ}) stabilizes G. Now, we take σ ∈ Aut(𝔤) as in Lemma 3.1, extend it holomorphically on G_{ℂ}, and restrict it to G_{U} (we use the same letter σ).

Then, σθ = θσ and στ = τσ hold on G_{U} because so do they on G. It follows from σθ = θσ that σ induces a diffeomorphism of D = G_{U}/K. Furthermore, this is anti-holomorphic, because σZ = −Z and the complex structure on G_{U}/K is given by the left G_{U}-translation of Ad(exp(\frac{π}{2}Z)) as is the case of the non-compact Hermitian symmetric space G/K (see Subsection 2.3).

Since the condition (3.1.2) means that 𝔤^{σ,−τ,−θ} contains a maximal abelian subspace of 𝔤^{−τ,−θ}, the condition (4.1.2) follows from

(𝔤_{U})^{−τ,−θ} = \sqrt{−1}𝔤^{−τ,−θ} and (𝔤_{U})^{σ,−τ,−θ} = \sqrt{−1}𝔤^{σ,−τ,−θ}.

Therefore, all the conditions (4.1.1) – (4.1.3) are verified. Thus, we have proved Lemma 4.1 for Type I.

4.3 Proof of Lemma 4.1 in Type II

In contrast to Type I, τ (or any of its conjugation) cannot stabilize a noncompact real form G of G_{ℂ} in Type II. Thus, we cannot reduce Type II to the non-compact results in Section 3. Instead, our strategy here is to find a “large” subalgebra, say 𝔤′_{U}, of 𝔤_{U}, such that τ commutes with θ when restricted to 𝔤′_{U}. The definition of 𝔤′_{U} and a precise formulation of “largeness” will be given in Claim 4.3.

There are two cases up to conjugation for Type II: Type II-1 (𝔤_{U}, 𝔤_{U}^{τ}, 𝔤_{U}^{θ}) = (𝔰𝔲(2n), 𝔰𝔭(n), 𝔰𝔲(2p′ + 1) + 𝔰𝔲(2q′ + 1) + \sqrt{−1}ℝ), Type II-2 (𝔤_{U}, 𝔤_{U}^{τ}, 𝔤_{U}^{θ}) = (𝔰𝔬(2n), 𝔰𝔬(2p′ + 1) + 𝔰𝔬(2q′ + 1), 𝔲(n)), where n = p′ + q′ + 1.

The proof for Type II-2 follows from Type II-1 by switching the role of τ and θ in 𝔰𝔬(2n) = 𝔰𝔲(2n) ∩ 𝔤𝔩(2n, ℝ). Therefore, we shall consider mostly Type II-1, but for the convenience of the reader, we sometimes supply the formula for Type II-2 in addition.

By using matrix realization, we suppose 𝔤_{U}^{θ} is a subalgebra of 𝔤_{U} defined by

θ(W) := g_{0} W g_{0}^{−1},

where

g_{0} := diag(\underbrace{1, …, 1}_{p′}, \underbrace{−1, …, −1}_{q′+1}, \underbrace{1, …, 1}_{p′+1}, \underbrace{−1, …, −1}_{q′}) ∈ GL(2n, ℝ).

Furthermore, by taking conjugation of τ by G_{U} if necessary, we may and do assume that 𝔤_{U}^{τ} is a subalgebra of 𝔤_{U} defined by

X ↦ J_{n}\overline{X}J_{n}^{−1},

where we set

J_{n} := (\begin{matrix}0 & −I_{n} \\ I_{n} & 0 \end{matrix}) ∈ GL(2n, ℝ).

Then, we have

𝔤_{U}^{τ,θ} ≡ 𝔤_{U}^{τ} ∩ 𝔤_{U}^{θ} ≃ \begin{cases}𝔰𝔭(p′) + 𝔰𝔭(q′) + 𝔲(1) & (Type II-1), \\ 𝔲(p′) + 𝔲(q′) & (Type II-2). \end{cases}  (4.3.1)

The non-commutativity τθ ≠ θτ arises from the odd parity of 2p′ + 1 and 2q′ + 1 in the both cases, and is reflected by the fact that neither (𝔤_{U}^{τ}, 𝔤_{U}^{τ,θ}) nor (𝔤_{U}^{θ}, 𝔤_{U}^{τ,θ}) is a symmetric pair. The idea of the following claim is to pull 2p′ and 2q′ out of 2p′ + 1 and 2q′ + 1.

Claim 4.3. We can realize the Lie algebra:

𝔤′_{U} := \begin{cases}𝔰𝔲(2n − 2) & (Type II-1), \\ 𝔰𝔬(2n − 2) & (Type II-2), \end{cases}

as a subalgebra of 𝔤_{U} such that the following three conditions are satisfied:

Both τ and θ stabilize 𝔤′_{U}.  (4.3.2)
τ|_{𝔤′_{U}} and θ|_{𝔤′_{U}} commute.  (4.3.3)
(𝔤_{U})^{−τ,−θ} = (𝔤′_{U})^{−τ,−θ}.  (4.3.4)

Proof. We consider the subspace ℂ^{2n−2} = ℂ^{2(p′+q′)} ⊂ ℂ^{2n} corresponding to the partition

2n = p′ + 1 + q′ + p′ + 1 + q′,

and embed 𝔤′_{U} in 𝔤_{U} accordingly. Then, clearly (4.3.2) and (4.3.3) hold, and the triple (𝔤′_{U}, (𝔤′_{U})^{τ}, (𝔤′_{U})^{θ}) of Lie algebras is given by

(𝔰𝔲(2n − 2), 𝔰𝔭(n − 1), 𝔰𝔲(2p′) + 𝔰𝔲(2q′) + \sqrt{−1}ℝ) (Type II-1),
(𝔰𝔬(2n − 2), 𝔰𝔬(2p′) + 𝔰𝔬(2q′), 𝔲(n − 1))  (Type II-2).

We recall n = p′ + q′ + 1. For A ∈ M(p′, q′; ℂ), we set

X(A) := (\begin{matrix}0 & 0 & A \\ 0 & 0 & 0 \\ −A^{∗} & 0 & 0 \end{matrix}) , Y (A) := (\begin{matrix}0 & 0 & A \\ 0 & 0 & 0 \\ −^{t}A & 0 & 0 \end{matrix}) ∈ M(n, ℂ).

Then, by a simple matrix computation we have

(𝔤_{U})^{−τ,−θ} = \begin{cases}\{(\begin{matrix}X(A) & Y (B) \\ \overline{Y (B)} & −\overline{X(A)} \end{matrix}) & : A, B ∈ M(p′, q′; ℂ)\} (Type II-1), \\ \{(\begin{matrix}X(A) & Y (B) \\ −Y (B) & −X(A) \end{matrix}) & : A, B ∈ M(p′, q′; ℝ)\} (Type II-2). \end{cases}

Now, (4.3.4) is clear. Thus, Claim 4.3 is proved.

Let us return to the proof of Lemma 4.1 in Type II.

We define σ ∈ Aut(𝔤_{U}) by

σ(W) := I_{n,n} \overline{W} I_{n,n}^{−1},

where

I_{n,n} := (\begin{matrix}I_{n} & 0 \\ 0 & −I_{n} \end{matrix}) ∈ GL(2n, ℝ).

Then we have σθ = θσ, στ = τσ and

𝔤_{U}^{σ} ≃ \begin{cases}𝔰𝔬(2n) & (Type II-1), \\ 𝔰𝔬(n) + 𝔰𝔬(n) & (Type II-2), \end{cases}  (4.3.5)

(𝔤_{U})^{σ,−τ,−θ} = \begin{cases}\{(\begin{matrix}X(A) & Y (B) \\ \overline{Y (B)} & −\overline{X(A)} \end{matrix}) & : A, \sqrt{−1}B ∈ M(p′, q′; ℝ)\} (Type II-1), \\ \{(\begin{matrix}X(A) & 0 \\ 0 & −X(A) \end{matrix}) & : A ∈ M(p′, q′; ℝ)\} (Type II-2). \end{cases}

We note that σ acts anti-holomorphically on both G_{U}/K and its complex submanifold G′_{U}/K′ := G′_{U}/G′_{U} ∩ K (see Table 2.3.2).

Now, we consider (𝔤′_{U}, θ|_{𝔤′_{U}}, τ|_{𝔤′_{U}}). This is of Type I by (4.3.3), for which Lemma 4.1 has been already proved in Subsection 4.2. In fact, the restriction σ|_{𝔤′_{U}} of the above choice of σ satisfies the conclusion of Lemma 4.1. In particular, if we take a maximal abelian subspace 𝔱 of (𝔤′_{U})^{σ,−τ,−θ}, then it is also a maximal abelian subspace of (𝔤′_{U})^{−τ,−θ}.

Next, we consider (𝔤, θ, τ) which is of Type II. In view of (4.3.4), 𝔱 is also a maximal abelian subspace of (𝔤_{U})^{−τ,−θ}. Hence, the condition (4.1.2) is satisfied. We have already seen (4.1.1) and (4.1.3). Thus, we have proved Lemma 4.1 in Type II.

4.4 Stability of H_{U}-orbits

We are ready to complete the proof of Theorem 1.5 in Case I, namely, for a compact simple G_{U}. Here is a compact case counterpart of Lemma 2.2:

Lemma 4.4.1. Suppose that three involutive automorphisms, τ, θ and σ of G_{U} satisfy the conditions (4.1.1), (4.1.2) and (4.1.3). Then,

1) For any x ∈ D = G_{U}/K, there exists g ∈ H_{U} such that σ(x) = g · x. In particular, each H_{U}-orbit on D is preserved by σ.

2) Take a maximal abelian subspace 𝔱 in (𝔤_{U})^{σ,−τ,−θ}, and we define a submanifold S of D = G_{U}/K by S := (exp 𝔱)K. Then, S meets every H_{U}-orbit in D, and σ|_{S} = id.

The proof parallels that of Lemma 2.2. For this, all we need now is the following lemma, which is a compact analog of a generalized Cartan decomposition G = HAK (see (2.2.4)).

Lemma 4.4.2. Let G_{U} be a semisimple connected compact Lie group with Lie algebra 𝔤_{U}, and τ and θ two involutive automorphisms of G_{U}. Take a maximal abelian subspace 𝔱 in (𝔤_{U})^{−τ,−θ}, and let T be the analytic subgroup of G_{U} with Lie algebra 𝔱. Then we have

G_{U} = (G_{U})_{0}^{τ} T (G_{U})_{0}^{θ} .

Proof. See Hoogenboom [8, Theorem 6.10] for the case τθ = θτ, and Matsuki [23, Theorem 1] for the general case where τ may not commute with θ.

Now, Lemma 4.4.1 combined with Lemma 4.1 implies Theorem 1.5 for a compact simple G_{U}. Hence, we have shown Theorem 1.5 in Case I.

4.5 Proof of Theorem 1.5 (compact case)

This subsection gives a proof of Theorem 1.5 in Case II. Suppose that both G_{U}/K_{1} and G_{U}/K_{2} are compact Hermitian symmetric spaces. We write θ_{1}, θ_{2} for the corresponding involutive automorphisms of G_{U}. Then, applying Lemma 4.1 to (θ_{1}, θ_{2}) in place of (θ, τ), we find an involution σ′ ∈ Aut(G_{U}) satisfying the following three conditions:

σ′θ_{i} = θ_{i}σ′ (i = 1, 2).  (4.5.1)
The induced action of σ′ on G_{U}/K_{i} (i = 1, 2) is anti-holomorphic. (4.5.2)
(𝔤_{U})^{σ′,−θ_{1},−θ_{2}} contains a maximal abelian subspace of (𝔤_{U})^{−θ_{1},−θ_{2}}. (4.5.3)

We remark that the condition (4.5.2) for i = 2 is not included in Lemma 4.1, but follows automatically by our choice of σ′.

Now, we define three involutive automorphisms τ, θ and σ on G_{U} × G_{U} by τ(g_{1}, g_{2}) := (g_{2}, g_{1}), θ := (θ_{1}, θ_{2}) and σ := (σ′, σ′), respectively. Then (G_{U} × G_{U})^{τ} = diag(G_{U}). By using the identification

(𝔤_{U} ⊕ 𝔤_{U})^{−τ} = {(X, −X) : X ∈ 𝔤_{U}} ⟶^{∼} 𝔤_{U}, (X, −X) ↦ X,

we have

(𝔤_{U} ⊕ 𝔤_{U})^{−τ,−θ} ≃ (𝔤_{U})^{−θ_{1},−θ_{2}},
(𝔤_{U} ⊕ 𝔤_{U})^{σ,−τ,−θ} ≃ (𝔤_{U})^{σ′,−θ_{1},−θ_{2}}.

Then it follows from (4.5.3) that (𝔤_{U} ⊕𝔤_{U})^{σ,−τ,−θ} contains a maximal abelian subspace of (𝔤_{U} ⊕ 𝔤_{U})^{−τ,−θ}. Thus, we can apply Lemma 4.4.1 to τ, θ and σ ∈ Aut(G_{U} × G_{U}), and therefore conclude that the diagonal action of G_{U} on G_{U}/K_{1} × G_{U}/K_{2} is strongly visible. Hence Theorem 1.5 in Case II has been proved. Together with Subsection 4.4, we have proved Theorem 1.5 for compact case.

By Sections 3 and 4, we have now completed the proof of Theorem 1.5.

5 Visible actions of unipotent subgroups

This section gives a proof of the strong visibility of the maximal unipotent group N action on the Hermitian symmetric space G/K (Theorem 1.10). The proof parallels to that for the K-action on G/K.

Proof of Theorem 1.10. Without loss of generality, we may and do assume that G is a non-compact, simply connected, simple Lie group of Hermitian type. We take σ to be the involution of G as in Lemma 2.4.

Let 𝔞 be a maximal abelian subspace in 𝔤^{σ,−θ}, and set A := exp 𝔞 and S := A · o ⊂ G/K. We fix a positive system Σ^{+}(𝔞) of the restricted root system Σ(𝔞) ≡ Σ(𝔤, 𝔞), and define 𝔫_{+} := ∑_{λ∈Σ^{+}} 𝔤(𝔞; λ). Then we have an Iwasawa decomposition G = N_{+}AK where N_{+} = exp(𝔫_{+}) is a maximal unipotent subgroup of G. Since 𝔞 ⊂ 𝔤^{σ}, σ(𝔤(𝔞; λ)) = 𝔤(𝔞; λ) for any λ. In particular, σ stabilizes 𝔫_{+}.

Let N_{+} · x be an N_{+}-orbit through x ∈ G/K. We write x = nak · o, where o = eK. Then N_{+} · x = N_{+}a · o, and σ(N_{+} · x) = σ(N_{+})σ(a) · o = N_{+}a · o = N_{+} · x. Thus, σ preserves each N_{+}-orbit on G/K. Furthermore, σ acts anti-holomorphically on G/K by (2.4.3) and σ|_{S} = id by 𝔞 ⊂ 𝔤^{σ}. Hence, the action of N_{+} on G/K is strongly visible. Since any maximal unipotent subgroup N is conjugate to N_{+}, we have proved Theorem 1.10.

6 Applications to representation theory

In [17], we have given an abstract theorem on propagation of multiplicity-free property of representations from fibers to spaces of sections for equivariant holomorphic vector bundles. Its main assumption is that actions on the base spaces are strongly visible. Accordingly, if we find strongly visible actions on complex manifolds, then we can expect a number of multiplicity-free theorems.

This section gives a brief explanation about how our geometric results (e.g. Theorems 1.5 and 1.10) are applied to such multiplicity-free theorems by confining ourselves to the line bundle cases (representations on fibers are automatically irreducible). Detailed proof of these applications is given in a separate paper [19]. Surprisingly, there was no literature, to the best of our knowledge, before [14], on a systematic study of multiplicity-free theorems for the restriction with respect to general symmetric pairs, although a number of explicit branching laws had been previously known especially for finite dimensional representations in the classical case (see [9, 21, 24] for example). Our applications include both finite and infinite dimensional representations, and both discrete and continuous spectra.

First, suppose G is a connected Lie group of Hermitian type. Retain the setting as in Subsection 2.3. Let (π, H) be an irreducible unitary representation of G, and H_{K} the underlying (𝔤_{ℂ}, K)-module. Then, it is known that the K-module H_{K}^{𝔭_{+}} := {v ∈ H_{K} : dπ(Y)v = 0 for any Y ∈ 𝔭_{+}} is either zero or irreducible. We say (π, H) is an irreducible unitary highest weight representation if H_{K}^{𝔭_{+}} ≠ {0}. Furthermore, π is of scalar type if dim H_{K}^{𝔭_{+}} = 1. Then, Theorem 1.5 leads us to the following multiplicity-free theorem:

Corollary 6.1 ([14], [19, Theorem A]). If (π, H) is an irreducible unitary highest weight representation of G of scalar type, then for any symmetric pair (G, H), the restriction π|_{H} is multiplicity-free.

Remark 6.2. Once we know the branching law is a priori multiplicity-free, it would be interesting and reasonable to try to find its explicit formula. It is noteworthy that “new” irreducible spherical unitary representations of H may occur as discrete summands in the setting of Corollary 6.1 (e.g. [26]). We remark that irreducible spherical unitary representations have not been classified for general reductive Lie groups (see [2]). Remark 6.3. The multiplicity-free theorem for the vector bundle case [17] strengthens Corollary 6.1, namely, one can relax the hypothesis of scalar type to the following condition:

H_{K}^{𝔭_{+}} is multiplicity-free as a Z_{H∩K}(𝔞)-module.  (6.1.1)

Here, Z_{H∩K}(𝔞) is the centralizer of 𝔞 in H ∩ K, and 𝔞 is a maximal abelian subspace of 𝔤^{−θ,σ,τθ} = 𝔤^{−θ,σ,−τ} if H is defined by an involution τ and σ is another involution given by Lemma 3.1.

The condition (6.1.1) is obviously satisfied if H_{K}^{𝔭_{+}} is one dimensional (i.e. π is of scalar type). The smaller the dimension of the totally real submanifold S = (exp 𝔞) · o is, the larger the centralizer Z_{H∩K}(𝔞) becomes and the more likely the condition (6.1.1) is satisfied for the K-type H_{K}^{𝔭_{+}}. This indicates how the slice S plays a crucial role in the multiplicity-free theorem.

Corollary 6.4. Suppose π_{1} and π_{2} are unitary highest weight modules of scalar type. Then, the tensor product representation π_{1} ⊗ π_{2} is multiplicityfree.

The above tensor product is discretely decomposable. On the other hand, the following case corresponding to Theorem 1.9 (2) contains continuous spectra in general:

Corollary 6.5. Retain the setting of Corollary 6.4. Then the tensor product π_{1} ⊗ π_{2}^{∗} is multiplicity-free.

The following is a descendent of Theorem 1.10.

Corollary 6.6 ([16, Theorem 33]). Let G be a non-compact simple Lie group of Hermitian type, and N a maximal unipotent subgroup. If π is an irreducible unitary highest weight representation of scalar type, then the restriction π|_{N} is multiplicity-free.

One of the simplest examples for Corollary 6.6 is the fact that the Hardy space (an irreducible representation of G = SL(2, ℝ) has simple spectra supported on the half line on the Fourier transform side (decomposition into irreducible representations of N ≃ ℝ).

So far, we have discussed a non-compact G. Next, let us consider the compact case. Suppose now G is a connected, compact simple Lie group. We take a Cartan subalgebra 𝔱 of 𝔤, and fix a positive system Δ^{+}(𝔤_{ℂ}, 𝔱_{ℂ}). For a dominant integral weight λ ∈ 𝔱_{ℂ}^{∗}, we write π_{λ} for the irreducible finite dimensional representation of 𝔤_{ℂ} with highest weight λ. We say that π_{λ} is pan type if (𝔤, 𝔤_{λ}) forms a symmetric pair where 𝔤_{λ} is the Lie algebra of the isotropy group G_{λ} = {g ∈ G : Ad^{∗}(g)λ = λ} (here, we regard 𝔱_{ℂ}^{∗} ⊂ 𝔤_{ℂ}^{∗} via the Killing form), that is,

𝔤_{λ} := {Y ∈ 𝔤 : λ([Y, Z]) = 0 for any Z ∈ 𝔤}.

See Richardson–Röhrle–Steinberg [30] or [16, Lemma 6.2.2] for the list of such π_{λ}.

Then, the following two corollaries are obtained again from Theorem 1.5. They may be regarded as finite dimensional versions of Corollary 6.1, and Corollaries 6.4 and 6.5, respectively.

Corollary 6.7 ([16, Theorem 26]). Let π_{λ} be a pan representation of a connected compact Lie group G. Then the restriction π_{λ}|_{H} is multiplicityfree with respect to any symmetric pair (G, H).

Corollary 6.8 ([16, Theorem 25], [22]). The tensor product representation π_{λ} ⊗ π_{μ} of any two pan representations π_{λ} and π_{μ} is multiplicity-free.

Remark 6.9. As in Remark 6.3, we can strengthen Corollaries 6.7 and 6.8 by using the vector bundle case [17]. This strengthened version covers, for instance, Stembridge’s classification [32] of the pairs (π_{1}, π_{2}) of two irreducible finite dimensional representations of GL_{n} such that π_{1}⊗π_{2} is multiplicity-free. It also covers some further multiplicity-free theorem such as the restriction of ‘nearly rectangular shape’ representations to symmetric subgroups ([21]).

Corollary 6.7 could be proved alternatively by a traditional approach, that is, by showing the H_{ℂ}-sphericity (the existence of an open orbit of a Borel subgroup):

Proposition 6.10. Let X be a compact Hermitian symmetric space, and G_{ℂ} the group of biholomorphic transformations of X. Then X is H_{ℂ}-spherical for any H_{ℂ} such that (G_{ℂ}, H_{ℂ}) is a complex symmetric pair.

Conversely, Proposition 6.10 is obtained as a corollary of Theorem 1.5 via Corollary 6.7 and [33] (see [16, Corollary 15]) from our view point. Remark 6.11. Corollary 6.7 holds for any symmetric pair (G, H). For individual pairs (G, H), there are sometimes more families of representations π_{λ} such that the restrictions π_{kλ}|_{H} are multiplicity-free for all k ∈ ℕ even if they are not pan representations, or equivalently, even if (𝔤, 𝔤_{λ}) are not symmetric pairs.

For example, for (G, H) = (U(n), U(p) × U(q)) (p + q = n), this is the case if

𝔤_{λ} = \begin{cases}𝔲(1)^{n} & if min(p, q) = 1, \\ 𝔲(n_{1}) + 𝔲(n_{2}) + 𝔲(n_{3}) & if min(p, q) = 2, \\ 𝔲(n_{1}) + 𝔲(n_{2}) + 𝔲(n_{3}) (min(n_{1}, n_{2}, n_{3}) = 1) & if min(p, q) ≥ 3, \end{cases}

where n = n_{1} + n_{2} + n_{3} (see [15, Theorem 3.3]). See [18] for an explicit construction of a totally real submanifold S that meets every H-orbit on the coadjoint orbit Ad^{∗}(G) · λ (generalized flag variety). We note that unlike the symmetric case treated in this article, S is not a flat submanifold in the non-symmetric (𝔤, 𝔤_{λ}).

7 Coisotropic, polar, and visible actions

We conclude this paper with some comments on the following three related concepts on group actions:

1) (Strongly) visible actions on a complex manifold (Definition 1.2).

2) Coisotropic actions on a symplectic manifold ([5, 10]).

3) Polar actions on a Riemannian manifold ([28, 29]).

Suppose a compact Lie group H acts on a symplectic manifold M by symplectic automorphisms. The action is called coisotropic if one and hence all principal H-orbits H · x are coisotropic with respect to the symplectic form ω, i.e. T_{x}(H · x)^{⊥ω} ⊂ T_{x}(H · x). By [16, Theorems 4 and 7], Theorems 1.5 and 1.10 imply:

Corollary 7.1. 1) In Theorem 1.5, the H-action on D is also coisotropic and visible.

2) In Theorem 1.10, the N-action on D is also coisotropic and visible.

On the other hand, an isometric action of a compact Lie group H on a Riemannian manifold M is called polar if there exists a closed connected submanifold S of M that meets every H-orbit orthogonally. By a classic theorem of R. Hermann [7], the H-action on G/K is polar if G is compact and both (G, H) and (G, K) are symmetric pairs.

Since the polar action on an irreducible compact homogeneous Kähler manifold is coisotropic by [28, Theorem 1.1] and visible by [16, Theorem 6], Corollary 7.1 (1) for compact D follows also from Hermann’s theorem [7]. However, what we needed for strong visibility was not only the construction of a slice but also that of an anti-holomorphic diffeomorphism σ. This was a core of the proof of Theorem 1.5 in Sections 3 and 4. We note that (strongly) visible actions are not always polar in the non-symmetric case in general (see [18]).

References

[1] H. Alikawa, Multiplicity free branching laws for outer automorphisms of simple Lie algebras, to appear in J. Math. Soc. Japan.

[2] Barbasch D, The spherical unitary dual of split and p-adic groups, preprint.

[3] J. Faraut and G. Ólafsson, Causal semisimple symmetric spaces, the geometry and harmonic analysis, Semigroups in Algebra, Geometry and Analysis, Walter de Gruyter, 1995, 3–32.

[4] M. Flensted-Jensen, Discrete series for semisimple symmetric spaces, Ann. of Math., 111 (1980), 253–311.

[5] V. Guillemin and S. Sternberg, Multiplicity-free spaces, J. Differential Geom., 19 (1984), 31–56.

[6] E. Heintze, R. S. Palais, C.-L. Terng, G. Thorbergsson, Hyperpolar actions on symmetric spaces, Geometry, Topology, and Physics for Raoul Bott, International Press, 1994, 214–245.

[7] R. Hermann, Variational completeness for compact symmetric spaces, Proc. Amer. Math. Soc., 11 (1960), 544–546.

[8] R. Hoogenboom, Intertwining functions on compact Lie groups, CWI Tract, 5 (1984), Stichting Mathematisch Centrum, Centrum voor Wiskunde en Informatica, Amsterdam, 1–86.

[9] R. Howe, Perspectives on invariant theory: Schur duality, multiplicityfree actions and beyond, Israel Math. Conf. Proc., 8 (1995), 1–182.

[10] A. T. Huckleberry and T. Wurzbacher, Multiplicity-free complex manifolds, Math. Ann., 286 (1990), 261–280.

[11] H. Jaffee, Real forms of Hermitian symmetric spaces, Bull. Amer. Math. Soc., 81 (1975), 456–458.

[12] , Anti-holomorphic automorphisms of the exceptional symmetric domains, J. Differential Geom., 13 (1978), 79–86.

[13] S. Kobayashi and T. Nagano, On filtered Lie algebras and geometric structures I, J. Math. Mech., 13 (1964), 875–908.

[14] T. Kobayashi, Multiplicity-free theorem in branching problems of unitary highest weight modules, Proceedings of the Symposium on Representation Theory held at Saga, Kyushu 1997 (ed. K. Mimachi), (1997), 9–17.

[15] , Geometry of multiplicity-free representations of GL(n), visible actions on flag varieties, and triunity, Acta Appl. Math., 81 (2004), 129–146.

[16] , Multiplicity-free representations and visible actions on complex manifolds, Publ. RIMS, 41 (2005), 497–549 (a special issue of Publications of RIMS commemorating the fortieth anniversary of the founding of the Research Institute for Mathematical Sciences).

[17] , Propagation of multiplicity-free property for holomorphic vector bundles, preprint.

[18] , A generalized Cartan decomposition for the double coset space (U(n_{1}) × U(n_{2}) × U(n_{3}))\U(n)/(U(p) × U(q)), preprint.

[19] , Multiplicity-free theorems of the restriction of unitary highest weight modules with respect to reductive symmetric pairs, to appear in Progr. Math., Birkhäuser.

[20] A. Korányi and J. A. Wolf, Realization of hermitian symmetric spaces as a generalized half-planes, Ann. of Math., 81 (1965), 265–285.

[21] C. Krattenthaler, Identities for classical group characters of nearly rectangular shape, J. Algebra, 209 (1998), 1–61.

[22] P. Littelmann, On spherical double cones, J. Algebra, 166 (1994), 142– 157.

[23] T. Matsuki, Double coset decompositions of reductive Lie groups arising from two involutions, J. Algebra, 177 (1997), 49–91.

[24] S. Okada, Applications of minor summation formulas to rectangularshaped representations of classical groups, J. Algebra, 205 (1998), 337– 367.

[25] G. Ólafsson and B. Ørsted, Generalizations of the Bargmann transform, Lie Theory and its application in physics (eds. H.-D. Döbner, V. K. D́obrev, and J. Hilgert), World Sci., 1996, 3–14.

[26] B. Ørsted and G. Zhang, Tensor products of analytic continuations of holomorphic discrete series, Canad. J. Math., 49 (1997), 1224–1241.

[27] T. Oshima and J. Sekiguchi, The restricted root system of a semisimple symmetric pair, Adv. Stud. Pure Math., 4 (1984), 433–497.

[28] F. Podestà and G. Thorbergsson, Polar and coisotropic actions on Kähler manifolds, Trans. Amer. Math. Soc., 354 (2002), 1759–1781.

[29] , Coisotropic actions on compact homogeneous Kähler manifolds, Math. Z., 243 (2003), 471–490.

[30] R. Richardson, G. Röhrle and R. Steinberg, Parabolic subgroup with abelian unipotent radical, Invent. Math., 110 (1992), 649–671.

[31] W. Rossmann, The structure of semisimple symmetric spaces, Canad. J. Math., 31 (1979), 156–180.

[32] J. R. Stembridge, Multiplicity-free products of Schur functions, Ann. Comb., 5 (2001), 113–121.

[33] É. B. Vinberg and B. N. Kimelfeld, Homogeneous domains on flag manifolds and spherical subgroups of semisimple Lie groups, Funct. Anal. Appl., 12 (1978), 168–174