MULTIPLICITY-FREE RESTRICTIONS OF UNITARY HIGHEST

WEIGHT MODULES FOR REDUCTIVE SYMMETRIC PAIRS

Toshiyuki KOBAYASHI

University of Tokyo

Abstract. Let π be a unitary highest weight module of a semisimple Lie group G, and (G, G′) a semisimple symmetric pair. The main theorems are:

i) If π has a one dimensional minimal K-type, then the multiplicity is free in the branching law of the restriction π|_{G′}. ii) If the natural inclusion G′/K′ ↪ G/K is holomorphic between Hermitian symmetric spaces, then the multiplicity is uniformly bounded in the branching law of the restriction π|_{G′}.

Our approach also gives analogous multiplicity-one theorems in branching problems for finite dimensional representations of compact groups. Our results present a number of new settings of multiplicity-one branching laws and also give a new and unified proof of some of known multiplicity-one theorems such as the Clebsch-Gordan formula of tensor products, the Plancherel theorem for a Hermitian symmetric space (also for line bundle cases), the Kostant-Schmid K-type formula of holomorphic discrete series, and recent examples of branching laws involving continuous spectra by Ørsted-Zhang and van Dijk-Hille.

Contents

§1. Introduction and statement of main results §2. Reproducing kernel and commutativity of End_{H}(H) §3. Anti-holomorphic involutions on Hermitian symmetric spaces §4. Proof of Theorem C and Theorem D §5. Proof of Theorem A and Theorem B §6. Counter examples §7. Finite dimensional cases §8. Appendix: associated bundles on Hermitian symmetric spaces

§1. Introduction and statement of main results

\underline{1.1.} Our object of study is the restriction of an irreducible unitary highest weight module of a real reductive Lie group with respect to semisimple symmetric pairs with focus on multiplicity in the branching law.

1991 Mathematics Subject Classification. Primary 22E46, Secondary 53C35. Key words and phrases. unitary highest weight module, multiplicity free, branching rule, re-

ductive symmetric pair, Hermitian symmetric space, reproducing kernel, semisimple Lie group.

Typeset by AMS-TEX

\underline{1.2.} First, we recall the notion of the multiplicity-free decomposition (in both continuous and discrete spectra) of a unitary representation.

Let H be a Lie group of type I in the sense of von Neumann algebras. Any reductive Lie group is the case. We denote by Ĥ the unitary dual of H, namely, the set of equivalence classes of irreducible unitary representations of H. Suppose that (π, H) is a unitary representation of H defined on a (second countable) Hilbert space H. Then π is decomposed uniquely into irreducible unitary representations of H by means of the direct integral of Hilbert spaces:

(1.2.1)  π ≃ ∫Ĥ m_{π}(μ)μ dσ(μ),

where dσ(μ) is a Borel measure on Ĥ equipped with the Fell topology, and the multiplicity function m_{π} : Ĥ → ℕ ∪ {∞} is uniquely defined almost everywhere with respect to the measure dσ.

Let End(H) be the ring of continuous operators on H, and End_{H}(H) the commutant of {π(g) : g ∈ H} in End(H). Then End_{H}(H) is a commutative ring if and only if m_{π}(μ) ≤ 1 for almost all μ ∈ Ĥ with respect to the measure dσ by Schur’s lemma for the direct integral of unitary representations. This property leads us to:

Definition 1.2. We say that the unitary representation (π, H) is multiplicity-free if End_{H}(H) is commutative.

The condition in Definition 1.2 especially implies that the multiplicity of a discrete spectrum in (1.2.1), namely, the dimension of the space of continuous Hintertwining operators Hom_{H}(μ, π) is equal to 0 or 1 for any μ ∈ Ĥ.

\underline{1.3.} There are two natural settings for the irreducible decomposition of a unitary representation; one is an induced representation from a smaller group (e.g. harmonic analysis on homogeneous spaces), and the other is the restriction from a larger group (e.g. tensor product representations).

Let G be a Lie group, and H a closed subgroup of G. The G-irreducible decomposition of an induced representation L^{2}-Ind_{H}^{G} τ (τ ∈ Ĥ) is called a Plancherel formula, while the H-irreducible decomposition of the restriction π|_{H} (π ∈ Ĝ) is called a branching law.

Suppose (G, H) is a semisimple symmetric pair. Then the multiplicity in the Plancherel formula L^{2}-Ind_{H}^{G} τ is finite if dim τ < ∞ (see [1]), especially if τ is the trivial representation 1. However, the multiplicity is not always finite if dim τ = ∞.

As a counterpart of induced representations L^{2}-Ind_{H}^{G} τ (τ ∈ Ĥ, dim τ = ∞), the multiplicity of the branching law π|_{H} (π ∈ Ĝ) is not always finite, too. See, for example, [30], Part I, §0 for a wild behavior of branching laws even though (G, H) is a symmetric pair with H non-compact.

Much has been studied on the induced representation L^{2}-Ind_{H}^{G} 1 (harmonic analysis on a reductive symmetric space G/H), while very little has been known on general branching problems π|_{H} for π ∈ Ĝ. One of the difficulties in the branching problems arises from the possibility of infinite multiplicity in the irreducible decomposition of π|_{H} if H is noncompact. In [30], we have given a sufficient condition on the triple (G, H, π) such that the multiplicity in π|_{H} is finite. Among various settings of branching problems, one expects especially a simple and detailed study if π|_{H} is multiplicity-free as a representation of H. So, it would be useful to tell a priori if the branching law is multiplicity free. This paper provides a sufficient condition for multiplicity-free restrictions. The setting will be:

a) π ∈ Ĝ is an irreducible unitary highest weight module (see §1.4),

b) (G, H) is a semisimple symmetric pair (see §1.5).

\underline{1.4.} Let G be a non-compact simple Lie group with finite center, θ a Cartan involution of G, and K := {g ∈ G : θg = g}. We write 𝔤 = 𝔨 + 𝔭 for the Cartan decomposition of the Lie algebra 𝔤 of G, corresponding to the Cartan involution θ. We assume that G is of Hermitian type, that is, the center 𝔠(𝔨) of 𝔨 is non-trivial. Then, it is well-known that dim 𝔠(𝔨) = 1 and that one can find Z ∈ 𝔠(𝔨) so that

𝔤_{ℂ} := 𝔤 ⊗ ℂ = 𝔨_{ℂ} ⊕ 𝔭^{+} ⊕ 𝔭^{−}

is the direct sum decomposition of eigenspaces of ad(Z) with eigenvalues 0, \sqrt{−1} and −\sqrt{−1}, respectively.

Definition 1.4. Let (π, H) be an irreducible unitary representation of G, and H_{K} the underlying (𝔤_{ℂ}, K)-module. The representation (π, H) is called an irreducible unitary highest weight module if H_{K}^{𝔭^{+}} ≠ {0}, where we put

H_{K}^{𝔭^{+}} := {v ∈ H_{K} : dπ(Y)v = 0 for any Y ∈ 𝔭^{+}}.

Then, H_{K}^{𝔭^{+}} is an irreducible finite dimensional representation of K. We say that π is of scalar type (or of scalar minimal K-type) if H_{K}^{𝔭^{+}} is one dimensional. There always exist countably many unitary highest weight modules of scalar type for any noncompact simple Lie group of Hermitian type. We say π is a holomorphic discrete series representation for G, if π is an irreducible unitary highest weight module that can be realized in a closed G-invariant subspace of L^{2}(G). Lowest weight modules and anti-holomorphic discrete series representations are defined similarly with 𝔭^{+} replaced by 𝔭^{−}.

We refer to [8], [7] and [23] for the classification of irreducible unitary highest weight modules.

\underline{1.5.} Let τ be an involutive automorphism of G. We may assume that τ commutes with θ (by using an inner automorphism if necessary). Then τ stabilizes 𝔨 and also 𝔠(𝔨). Because τ^{2} = id and 𝔠(𝔨) = ℝZ, there are two exclusive possibilities:

(1.5.1)  τZ = Z,
(1.5.2)  τZ = −Z.

Let G^{τ} := {g ∈ G : τg = g} and K^{τ} := G^{τ} ∩ K. It is well-known (see [27], [20], [21], [9]) that the condition (1.5.1) has the following geometric meanings:

1-a) τ acts holomorphically on the Hermitian symmetric space G/K, 1-b) G^{τ} /K^{τ} ↪ G/K is a complex submanifold.

On the other hand, (1.5.2) implies:

2-a) τ acts anti-holomorphically on the Hermitian symmetric space G/K, 2-b) G^{τ} /K^{τ} ↪ G/K is a totally real submanifold. We write G_{0}^{τ} for the identity component of G^{τ}. If a subgroup H of G satisfies

G_{0}^{τ} ⊂ H ⊂ G^{τ}, then we say that (G, H) is a symmetric pair corresponding to τ. We note that the Lie algebra 𝔥 of H is equal to 𝔤^{τ} := {X ∈ 𝔤 : τX = X}. An infinitesimal classification of irreducible semisimple symmetric pairs was accomplished by Berger [3]. For the readers’ convenience, an infinitesimal classification of the irreducible semisimple symmetric pairs that satisfy (1.5.1) (resp. (1.5.2)) is given in Table 3.3.1 (resp. Table 3.3.2).

\underline{1.6.} We are ready to state our main results. Let G be a simple non-compact Lie group of Hermitian type, and τ an involutive automorphism commuting with θ. We write (G, H) for the symmetric pair corresponding to τ.

Theorem A. If π ∈ Ĝ is a unitary highest weight module of scalar type, then the restriction π|_{H} is multiplicity free.

The branching law π|_{H} usually involves continuous spectra if τ satisfies (1.5.1) (cf. [30], Part III, Theorem 5.3). On the contrary, it is always discretely decomposable if τ satisfies (1.5.2).

Here is a uniform estimate of multiplicities when we drop the assumption that π is of scalar type.

Theorem B. We assume that τ satisfies (1.5.1). Let π be an irreducible unitary highest weight module of G. Then there is a constant C(π) < ∞ depending only on π with the following properties:

1) The restriction π|_{H} splits into a discrete Hilbert sum of irreducible unitary representations of H:

π|_{H} ≃ _{\begin{matrix}∑ & ⊕ \\ μ∈Ĥ \end{matrix}}m_{π}(μ)μ,

with multiplicity satisfying

m_{π}(μ) ≤ C(π) for all μ ∈ Ĥ.

2) We can take C(π) = 1 if π is of scalar type.

The second statement is a direct consequence of Theorem A and B (1).

Here are multiplicity-one theorems for the decomposition of tensor products, which are parallel to Theorems A and B:

Theorem C. Let π_{1}, π_{2} ∈ Ĝ be unitary highest (or lowest) weight modules of scalar type. Then the tensor product π_{1} ˆ⊗π_{2} is multiplicity free as a representation of G.

Theorem C asserts the multiplicity is one in both discrete and continuous spectra. We note that continuous spectra may arise only if (π_{1}, π_{2}) is the pair (highest, lowest) or (lowest, highest).

On the other hand, if π_{1} and π_{2} are simultaneously highest weight modules, then the tensor product π_{1} ˆ⊗π_{2} decomposes discretely. Dropping the assumption of “scalar type”, we have still a uniform estimate of multiplicities:

Theorem D. Let π_{1} and π_{2} be irreducible unitary highest weight modules of G. Then, there is a constant C(π_{1}, π_{2}) < ∞ with the following properties:

1) The tensor product π_{1} ˆ⊗π_{2} splits into a discrete Hilbert sum of irreducible unitary representations of G:

π_{1} ˆ⊗π_{2} ≃ _{\begin{matrix}∑ & ⊕ \\ μ∈Ĝ \end{matrix}}m_{π_{1},π_{2}}(μ)μ,

with multiplicity satisfying

m_{π_{1},π_{2}}(μ) ≤ C(π_{1}, π_{2}) for all μ ∈ Ĝ.

2) We can take C(π_{1}, π_{2}) = 1 if both π_{1} and π_{2} are of scalar type.

\underline{1.7.} Although our main concern is with infinite dimensional representations, we also prove a finite dimensional version of multiplicity-free theorems (Theorems 7.3 and 7.4). It gives a unified explanation of some of known multiplicity-free results ([19], [32], [36], [38], [43], [48], cf. Remark 7.5.1). It also presents some other branching problems of finite dimensional representations for which there are, to our knowledge, no explicit branching formulae in the literature.

\underline{1.8.} We illustrate the above theorems by several examples.

Example 1.8. 1) We denote by π_{n} the holomorphic discrete series representation of SL(2, ℝ) with minimal K-type χ_{n} (n ≥ 2), where χ_{n} (n ∈ ℤ) stands for a character of SO(2). Likewise π_{−n} denotes the anti-holomorphic discrete series representation of SL(2, ℝ) with minimal K-type χ_{−n} (n ≥ 2). We write π_{\sqrt{−1}ν}^{ε} (ε = ±1, ν ∈ ℝ) for the unitary principal series representations of SL(2, ℝ). We have a unitary equivalence π_{\sqrt{−1}ν}^{ε} ≃ π_{−\sqrt{−1}ν}^{ε}. We write χ_{ζ} (ζ ∈ ℝ) for a unitary character of SO_{0}(1, 1) ≃ ℝ.

Let m ≥ n ≥ 2. Then, the following branching formulae are classically known. They are simplest cases of multiplicity-free results in Theorems A, B-(2), C, and D-(2):

(1.8.1)(a) π_{n}|_{SO_{0}(1,1)} ≃ ∫_{−∞}^{∞} χ_{ζ} dζ,
(1.8.1)(b)  π_{n}|_{SO(2)} ≃ \begin{matrix}∑ & ⊕ \\ k∈ℕ \end{matrix}χ_{n+2k},

(1.8.1)(c)  π_{m} ˆ⊗π_{−n} ≃ ∫_{0}^{∞} π_{\sqrt{−1}ν}^{(−1)^{m−n}}dν ⊕ ∑_{\begin{matrix}k∈ℕ \\ 0≤2k≤m−n−2 \end{matrix}} π_{m−n−2k},

(1.8.1)(d)  π_{m} ˆ⊗π_{n} ≃ \begin{matrix}∑ & ⊕ \\ k∈ℕ \end{matrix}π_{m+n+2k}.

The geometry we need in our proof is, for example,

i) Given any element z in the Poincaré disk D, there exists φ ∈ ℝ such that e^{\sqrt{−1}φ}z = \overline{z} (the geometry for the proof of (1.8.1)(b)). ii) Given any two elements z, w ∈ D, there exists a linear fractional transform T on D such that T (z) = \overline{z} and T (w) = \overline{w} (the geometry for the proof of (1.8.1)(d)).

We note that any holomorphic discrete series representation of SL(2, ℝ) is of scalar type, so that the assumptions in Theorems A and C are automatically satisfied.

2) Here is a “finite dimensional version” of the above example. Let σ_{n} be the irreducible n + 1-dimensional representation of SU(2). Then we have the following branching formulae: For m, n ∈ ℕ,

σ_{m} ⊗ σ_{n} = σ_{n+m} ⊕ σ_{n+m−2} ⊕ ⋯ ⊕ σ_{|n−m|}  (Clebsch-Gordan formula),
σ_{n}|_{SO(2)} = χ_{n} ⊕ χ_{n−2} ⊕ ⋯ ⊕ χ_{−n}.

These multiplicity-free results are explained as simplest cases of Theorems 7.3 and 7.4.

3) Let G be a connected simple non-compact Lie group of Hermitian type, and π ∈ Ĝ a holomorphic discrete series representation of G of scalar type.

3-a) Hua, B. Kostant, W. Schmid and K. Johnson ([46], [25]), obtained an explicit branching formula of the restriction π|_{K} (K-type formula), which is multiplicity free. This branching law is closely connected to invariant theory (e.g. so called GL_{m} ×GL_{n}-duality; [19]). A special case of Theorem B (2) by putting τ = θ gives a new proof of the multiplicity-free property of the above formula.

3-b) Furthermore, Theorem B (2) suggests a generalization of the Kostant-Schmid multiplicity-free K-type formula to the setting of a noncompact subgroup H = G^{τ} where τ is not necessarily a Cartan involution. The author recently obtained an explicit branching law of the restriction π|_{H} for any semisimple symmetric pair (G, H) satisfying (1.5.1) (see [29], Theorem C; Bertram-Hilgert obtained some special cases independently [4]).

3-c) In contrast to (3-b), the branching law π|_{H} for a semisimple symmetric pair (G, H) satisfying (1.5.2) is decomposed into continuous spectra with multiplicity one without any discrete spectrum, if π is a holomorphic discrete series of scalar type. This case is recently studied by G. ’Olafsson-B.Ørsted ([39]).

4) Suppose π_{1} is a highest weight module and π_{2} is a lowest weight module, and both being of scalar type.

4-a) If both π_{1} and π_{2} are furthermore discrete series representations, then the tensor product π_{1} ˆ⊗π_{2} is unitarily equivalent to the regular representation on L^{2}(G/K, χ), the Hilbert space of L^{2}-sections of the G-equivalent line bundle G ×_{K} ℂ_{χ} → G/K associated to a unitary character χ of K (R. Howe, J. Repka; [18], [44]). In particular, the corresponding special case of Theorem C gives a new proof of the multiplicity-free property of the Plancherel formula for L^{2}(G/K, χ), which was obtained by J. Heckman and N. Shimeno ([15], [47]) in the 90’s generalizing a work of Harish-Chandra and Helgason in the 60’s in the case where χ is the trivial representation. We should note that “multiplicity free property” in the Plancherel formula fails for a (non-Riemannian) semisimple symmetric space (see [2], [5] for the description of the multiplicity in terms of a coset of the Weyl group).

4-b) Theorem C applied to non-discrete series representations π_{1} and π_{2} (i.e. tensor products of some singular unitary representations) provides new settings of multiplicity-free branching laws. Some of them might be interesting from the view point of representation theory to construct “small” representations in the irreducible components and also from the view point of harmonic analysis which is relevant to the canonical representation in the sense of Vershik-Gelfand-Graev. Once we know the branching law is a priori multiplicity-free, it would be promising to obtain its explicit formula. Only special cases have been worked in this direction so far, for G = SU(2, 2) by G. Zhang and B. Ørsted [40]; for G = SU(n, 1) by G. van Dijk and S. Hille [6]. Their results show that different family of representations can occur in the branching laws with multiplicity free.

5) The restriction of the Segal-Shale-Weil representation ϖ of Mp(n, ℝ) with respect to U(p, n − p) (precisely, its double covering) decomposes discretely into so called the ladder representations of U(p, n − p) with multiplicity free (e.g. [26], Introduction). This multiplicity-free result can be explained in the context of Howe’s correspondence because (U(p, n − p), U(1)) forms a dual pair in Mp(n, ℝ), and also in the context of our Theorem B (2) because (𝔰𝔭(n, ℝ), 𝔲(p, n − p)) forms a semisimple symmetric pair. Some of multiplicity-free branching laws corresponding to Theorems B (2) and D (2) can be obtained by using the so-called “see-saw rule” in the context of the Howe’s dual pair, which we shall report in a subsequent paper.

\underline{1.9.} This paper is organized as follows: In §2, we shall give the commutativity of the ring of intertwiners of a unitary representation realized in a space of holomorphic sections on a line bundle over a complex manifold D, following a theorem of Faraut-Thomas, whose idea goes back to Gelfand’s proof of the commutativity of the Hecke algebra L^{1}(K\G/K), [12]. In §3, we consider the case where D is a Hermitian symmetric space with a holomorphic involutive automorphism τ. A key step is Lemma 3.2 which gives a sufficient condition for the assumption (2.2.3) of Theorem 2.2, by proving the existence of a “nice” involutive automorphism σ of G satisfying a certain rank condition. Then, the proof of Theorems A and C can be reduced to the existence theorem of such σ, which is proved in §4 and §5. In the same sections, we also complete the proof of Theorems B and D, which follow from Theorems A and C, respectively. In §6, we illustrate by examples that the assumptions in Theorems A and B are sharp in the sense that our conclusion fails if we drop any of the assumptions. In §7, analogous results of Theorems A and C are proved for finite dimensional representations of compact groups. In §8 (Appendix), we present some basic results on homogeneous line bundles for the convenience of the reader, which give a sufficient condition for the assumption (2.2.2) in Theorem 2.2.

The results of this paper were announced in [29].

§2. Reproducing kernel and commutativity of End_{H}(H)

Faraut and Thomas [10], in the case of trivial twisting parameter, gave a sufficient condition for the commutativity of End_{H}(H) by using the theory of reproducing kernels, which we extend to the general, twisted case in this preliminary section. The proof parallels to theirs, except for some necessary change (especially, (2.2.2)), in our formulation for equivariant line bundles.

\underline{2.1.} Let L → D be a holomorphic line bundle over a complex manifold D. We denote by O(L) the space of holomorphic sections of L → D. Then O(L) carries a Fréchet topology by the uniform convergence on compact sets. If a Lie group H acts holomorphically and equivariantly on the holomorphic line bundle L → D, then H defines a (continuous) representation on O(L) by the pull-back of sections.

Let {U_{α}} be a trivializing neighbourhood of D, and g_{αβ} ∈ O^{×}(U_{α} ∩U_{β}) the transition functions of the holomorphic line bundle L → D. Then an anti-holomorphic line bundle \overline{L} → D is a complex line bundle with the transition functions \overline{g_{αβ}}. We denote by \overline{O}(\overline{L}) the space of anti-holomorphic sections for \overline{L} → D.

Suppose σ is an anti-holomorphic diffeomorphism of D. Then the pull-back σ^{∗}L → D is an anti-holomorphic line bundle over D. In turn, \overline{σ^{∗}L} → D is a holomorphic line bundle over D (see Appendix for more details).

\underline{2.2.} Here is a main machinery to prove the multiplicity-free results in branching problems of unitary highest weight modules (Theorem A and Theorem C in Introduction).

Theorem 2.2. Let (π, H) be a unitary representation of a Lie group H. Assume that there exist an H-equivariant holomorphic line bundle L → D and an antiholomorphic involutive diffeomorphism σ of D with the following three conditions: (2.2.1) There is an injective (continuous) H-intertwining map H → O(L). (2.2.2) There exists an isomorphism of H-equivariant holomorphic line bundles

Ψ: L ⟶^{∼} \overline{σ^{∗}L}.

(2.2.3) Given x ∈ D, there exists g ∈ H such that σ(x) = g · x.

Then, the ring End_{H}(H) is a commutative algebra. Consequently, (π, H) is multiplicity-free (see Definition 1.2).

Remark 2.3. 1) The assumption (2.2.2) is automatically satisfied if L → D is a trivial bundle. In this case, Theorem 2.2 was proved in [10].

2) We shall prove a sufficient condition for (2.2.2) in Lemma 8.6 when D is a Hermitian symmetric space.

3) The assumption (2.2.3) is automatically satisfied if H acts transitively on D. But we are interested in a more general setting where each H-orbit has a positive codimension in D. A sufficient condition for (2.2.3) will be found in Lemma 3.2 when D is a symmetric space.

\underline{2.4.} Suppose O(L) contains a Hilbert space H such that the point evaluation map

O(L) ⊃ H → L_{z} ≃ ℂ, f ↦ f(z)

is continuous with respect to the Hilbert topology on H. If {φ_{ν}} is an orthonormal basis of H, then the reproducing kernel of H is defined by

K_{H}(x, y) ≡ K(x, y) := ∑_{ν} φ_{ν}(x)\overline{φ_{ν}(y)} ∈ O(L)ˆ⊗\overline{O}(\overline{L}).

Then K(x, y) is independent of the choice of an orthonormal basis {φ_{ν}}.

Lemma 2.4. 1) For each y ∈ D, K(·, y) ∈ H ⊗ \overline{L_{y}} (≃ H) and (f(·), K(·, y))_{H} = f(y) for any f ∈ H.

2) Let K_{i}(x, y) be the reproducing kernels of Hilbert spaces H_{i} ⊂ O(L) with inner products ( , )_{H_{i}}, respectively, for i = 1, 2. If K_{1} ≡ K_{2}, then H_{1} = H_{2} and ( , )_{H_{1}} = ( , )_{H_{2}}.

3) If K_{1}(x, x) = K_{2}(x, x) for any x ∈ D, then K_{1} ≡ K_{2}.

Proof. (1) and (2) are standard. We review only the way how to recover H with its inner product from a given reproducing kernel. For each y ∈ D, we fix an isomorphism L_{y} ≃ ℂ. Through this isomorphism, we can regard K(·, y) ∈ H ⊗ \overline{L_{y}} as an element of H. The Hilbert space H is the completion of the ℂ-span of {K(·, y) : y ∈ D} with pre-Hilbert structure

(2.4.1)  (K(·, y_{1}), K(·, y_{2}))_{H} := K(y_{2}, y_{1}) ∈ L_{y_{2}} ⊗ \overline{L_{y_{1}}} (≃ ℂ).

This procedure is independent of the choice of the isomorphism L_{y} ≃ ℂ.

3) We denote by \overline{D} the complex manifold that is equipped with the conjugate complex structure on D. Then, \overline{L} → \overline{D} is a holomorphic line bundle, and K(·, ·) ≡ K_{H}(·, ·) is a holomorphic section of the holomorphic line bundle L⊠\overline{L} → D×\overline{D}. As the diagonal embedding ι: D → D×\overline{D}, z ↦ (z, z) is totally real, (K_{1}−K_{2})|_{ι(D)} ≡ 0 implies K_{1} − K_{2} ≡ 0 by the unicity theorem of holomorphic functions. □

\underline{2.5.} Suppose we are in the setting of Theorem 2.2. We define an anti-linear map

J: O(L) → O(L), f ↦ Jf,

by Jf(z) := \overline{f(σ(z))} (z ∈ D), where we used the isomorphism Ψ_{∗}: O(L) ≃ O(\overline{σ^{∗}L}).

Lemma 2.5. In the setting of Theorem 2.2, we identify H with a subspace of O(L). Then, J is an isometry from H onto H.

Proof. We put H̃ := J(H), equipped with the inner product

(2.5.1)  (Jf_{1}, Jf_{2})H̃ := (f_{2}, f_{1})_{H}, for f_{1}, f_{2} ∈ H.

If {φ_{ν}} is an orthonormal basis of H, then H̃ is a Hilbert space with orthonormal basis {Jφ_{ν}}. Hence, the reproducing kernel of H̃ is given by KH̃(x, y) = K_{H}(σ(y), σ(x)) because

(2.5.2) KH̃(x, y) = ∑_{ν} Jφ_{ν}(x)\overline{Jφ_{ν}(y)} = ∑_{ν} \overline{φ_{ν}(σ(x))} \overline{\overline{φ_{ν}(σ(y))}} = K_{H}(σ(y), σ(x)),

where we have used the isomorphism Ψ (see (2.2.2)). We fix x ∈ D and take g ∈ H such that σ(x) = g · x (see (2.2.3)). Putting x = y in (2.5.2), we have

KH̃(x, x) = K_{H}(σ(x), σ(x)) = K_{H}(g · x, g · x) = K_{H}(x, x).

Here, the last equality follows from the assumption that (π, H) is a unitary representation of H. Then, by Lemma 2.4, the Hilbert space H̃ coincides with H and

(2.5.3)  (Jf_{1}, Jf_{2})_{H} = (f_{2}, f_{1})_{H}, for f_{1}, f_{2} ∈ H,

which is what we wanted to prove. □

Lemma 2.6 (see [10]). Suppose A ∈ End_{H}(H). Then the adjoint operator A^{∗} of A satisfies

(2.6.1)  A^{∗} = JAJ^{−1}.

Proof. We divide the proof into three steps. \underline{Step 1} (positive self-adjoint case): Assume A ∈ End_{H}(H) is a positive self-adjoint operator. Let H_{A} be the Hilbert completion of H by the pre-Hilbert structure

(2.6.2)  (f_{1}, f_{2})_{H_{A}} := (Af_{1}, f_{2})_{H}, for f_{1}, f_{2} ∈ H.

If f_{1}, f_{2} ∈ H and g ∈ H, then

(π(g)f_{1}, π(g)f_{2})_{H_{A}} = (Aπ(g)f_{1}, π(g)f_{2})_{H}
= (π(g)Af_{1}, π(g)f_{2})_{H} = (Af_{1}, f_{2})_{H} = (f_{1}, f_{2})_{H_{A}}.

Therefore, (π, H) extends to a unitary representation on H_{A}. Applying (2.5.3) to both H_{A} and H, we have

(Af_{1}, f_{2})_{H} = (f_{1}, f_{2})_{H_{A}} = (Jf_{2}, Jf_{1})_{H_{A}} = (AJf_{2}, Jf_{1})_{H}

= (Jf_{2}, A^{∗}Jf_{1})_{H} = (Jf_{2}, JJ^{−1}A^{∗}Jf_{1})_{H} = (J^{−1}A^{∗}Jf_{1}, f_{2})_{H}.

Hence, A = J^{−1}A^{∗}J, and (2.6.1) follows. \underline{Step 2} (self-adjoint case): Assume A ∈ End_{H}(H) is a self-adjoint operator. Let A = ∫ λdE_{λ} be the spectral decomposition of A. Then every projection operator E_{λ} ∈ End(H) also commutes with π(g) for all g ∈ H, namely, E_{λ} ∈ End_{H}(H). We define

A_{+} := ∫_{λ≥0} λdE_{λ}, A_{−} := ∫_{λ<0} λdE_{λ}.

Then A = A_{+} + A_{−}. Let I be the identity operator on H. As a positive self-adjoint operator A_{+} + I is an element of End_{H}(H), we have (A_{+} + I)^{∗} = J(A_{+} + I)J^{−1} by Step 1, whence A_{+}^{∗} = JA_{+}J^{−1}. Applying Step 1 again to −A_{−}, we have A_{−}^{∗} = JA_{−}J^{−1}. Thus,

A^{∗} = A_{+}^{∗} + A_{−}^{∗} = JA_{+}J^{−1} + JA_{−}J^{−1} = J(A_{+} + A_{−})J^{−1} = JAJ^{−1}.

Step 3 (general case): Suppose A ∈ End_{H}(H). Then A^{∗} also commutes with π(g) (g ∈ H) because π is unitary. We put B := \frac{1}{2}(A + A^{∗}) and C := \frac{\sqrt{−1}}{2}(A^{∗} − A). Then, both B and C are self-adjoint operators commuting with π(g) (g ∈ H). It follows from Step 2 that B^{∗} = JBJ^{−1} and C^{∗} = JCJ^{−1}. As J is an anti-linear map, we have

(\sqrt{−1}C)^{∗} = −\sqrt{−1}C^{∗} = −\sqrt{−1}JCJ^{−1} = J(\sqrt{−1}C)J^{−1}.

Hence, A = B + \sqrt{−1}C also satisfies A^{∗} = JAJ^{−1}. □

\underline{2.7.} Proof of Theorem 2.2. Let A, B ∈ End_{H}(H). By Lemma 2.6, we have

(AB)^{∗} = J(AB)J^{−1} = JAJ^{−1}JBJ^{−1} = A^{∗}B^{∗}.

On the other hand, (AB)^{∗} = B^{∗}A^{∗}. Hence, A^{∗}B^{∗} = B^{∗}A^{∗}, and then AB = BA. Therefore, End_{H}(H) is commutative. □

§3. Anti-holomorphic involutions on Hermitian symmetric spaces

In this section, we prepare a setup to apply Theorem 2.2 for the branching laws of unitary highest weight modules. The main results here are Propositions 3.6 and 3.7 that assure the multiplicity-free branching laws by assuming the existence of certain involutions satisfying the rank conditions (3.2.2) or (3.7.2). The proof of Theorems A and C (see Introduction) will be completed in §4 and §5 when we prove the existence of such involutions.

\underline{3.1.} Let G be a semisimple Lie group with Lie algebra 𝔤. Suppose that τ is an involutive automorphism of G. We also use the same letter τ to denote the differential of τ. We put 𝔤^{±τ} := {Y ∈ 𝔤 : τY = ±Y }. Then, we have a direct sum decomposition 𝔤 = 𝔤^{τ} + 𝔤^{−τ}.

We choose a Cartan involution θ of G commuting with τ, and write K := G^{θ} = {g ∈ G : θg = g}. Then 𝔤 = 𝔨 + 𝔭 ≡ 𝔤^{θ} + 𝔤^{−θ} is called a Cartan decomposition of 𝔤. We shall allow G to be non-linear, in particular, K is not necessarily compact. The real rank of 𝔤, denoted by ℝ- rank 𝔤, is defined to be the dimension of a maximal abelian subspace of 𝔤^{−θ}.

We define ℝ- rank 𝔤/𝔤^{τ} to be the dimension of a maximal abelian subspace of

𝔤^{−θ,−τ} := {Y ∈ 𝔤 : (−θ)Y = (−τ)Y = Y }.

The definition is independent of the choice of a maximal abelian subspace. In view of 𝔤^{θτ} ∩ 𝔤^{−θ} = 𝔤^{−θ,−τ}, we have

(3.1.1)  ℝ- rank 𝔤^{θτ} = ℝ- rank 𝔤/𝔤^{τ} ,

especially, ℝ- rank 𝔤 = ℝ- rank 𝔤/𝔨 by putting τ = θ.

Furthermore, suppose that σ is an involutive automorphism of G such that σ, τ and θ commute with one another. We set

G^{σ,τ} := G^{σ} ∩ G^{τ} = {g ∈ G : σg = τg = g}.

Then (G^{σ}, G^{σ,τ}) forms a reductive symmetric pair, because σ and τ commute. The commutativity of σ and θ implies that the automorphism σ: G → G stabilizes K and induces a diffeomorphism of G/K, for which we use the same notation σ.

\underline{3.2.} The following key lemma is a sufficient condition for (2.2.3) of Theorem 2.2 in the setting where D = G/K is a non-compact Riemannian symmetric space.

Lemma 3.2. Let σ and τ be involutive automorphisms of G. We assume that the pair (σ, τ) satisfies the following two conditions: (3.2.1) σ, τ and θ commute with one another. (3.2.2) ℝ- rank 𝔤/𝔤^{τ} = ℝ- rank 𝔤^{σ}/𝔤^{σ,τ}.

Then for any x ∈ G/K, there exists g ∈ G_{0}^{τ} such that

σ(x) = g · x.

Proof. The assumption (3.2.1) assures that θ|_{G^{σ}} is a Cartan involution of a reductive Lie group G^{σ} and that τ|_{G^{σ}} is an involutive automorphism of G^{σ} commuting with θ|_{G^{σ}}. Take a maximal abelian subspace 𝔞 in

𝔤^{−θ,σ,−τ} := {Y ∈ 𝔤 : (−θ)Y = σY = (−τ)Y = Y }.

We have then dim 𝔞 = ℝ- rank 𝔤^{σ}/𝔤^{σ,τ} by definition. By using the assumption (3.2.2), we have dim 𝔞 = ℝ- rank 𝔤/𝔤^{τ}. This means that 𝔞 is also a maximal abelian subspace in

𝔤^{−θ,−τ} = {Y ∈ 𝔤 : (−θ)Y = (−τ)Y = Y }.

Let A be an analytic subgroup of G with Lie algebra 𝔞. Then we have a generalized Cartan decomposition (see [11], §2)

(3.2.3)  G = G_{0}^{τ}AK.

Let o := eK ∈ G/K. Fix x ∈ G/K. Then, according to the decomposition (3.2.3), we find h ∈ G_{0}^{τ} and a ∈ A such that

x = ha · o.

We put g := σ(h) h^{−1}. We claim g ∈ G_{0}^{τ}. In fact, by using στ = τσ and τh = h, we have

τ(g) = τσ(h) τ(h^{−1}) = στ(h) τ(h)^{−1} = σ(h) h^{−1} = g.

Hence, g ∈ G^{τ}. Moreover, since the image of the continuous map

G_{0}^{τ} → G, h ↦ σ(h) h^{−1}

is connected, we have g ∈ G_{0}^{τ}.

On the other hand, we have σ(a) = a because 𝔞 ⊂ 𝔤^{−θ,σ,−τ} ⊂ 𝔤^{σ}. Therefore we have

σ(x) = σ(h) σ(a) · o = σ(h) h^{−1}ha · o = g · x,

proving the lemma. □

\underline{3.3.} (Hermitian symmetric space G/K)

Throughout the rest of this section, we assume that G is of Hermitian type. We retain the notation of §1.4 and §1.5. Let G_{ℂ} be a connected complex Lie group with Lie algebra 𝔤_{ℂ}, and Q^{−} the parabolic subgroup of G_{ℂ} with Lie algebra 𝔨_{ℂ}+𝔭^{−}. Then we have a Borel embedding G/K ↪ G_{ℂ}/Q^{−}, from which the complex structure on G/K is induced. (We remark that the embedding G/K ↪ G_{ℂ}/Q^{−} is well-defined, even though G is not necessarily a subgroup of G_{ℂ}.)

Suppose τ is an involutive automorphism of G commuting with θ. We recall from §1.5 that there are two exclusive possibilities:

(1.5.1)  τZ = Z,
(1.5.2)  τZ = −Z.

Here is the classification of semisimple symmetric pairs (𝔤, 𝔤^{τ}) with 𝔤 simple such that the pair (𝔤, 𝔤^{τ}) satisfies the condition (1.5.1) (respectively, (1.5.2)). Table 3.3.2 is equivalent to the classification of totally real symmetric spaces G^{τ} /K^{τ} of a Hermitian symmetric space G/K (see [9], [20], [21], [27]).

(𝔤, 𝔤^{τ}) satisfying (1.5.1) τZ = Z

𝔤  𝔤^{τ}
𝔰𝔲(p, q)  𝔰(𝔲(i, j) + 𝔲(p − i, q − j))
𝔰𝔲(n, n)  𝔰𝔬^{∗}(2n)
𝔰𝔲(n, n)  𝔰𝔭(n, ℝ)
𝔰𝔬^{∗}(2n)  𝔰𝔬^{∗}(2p) + 𝔰𝔬^{∗}(2n − 2p)
𝔰𝔬^{∗}(2n)  𝔲(p, n − p)
𝔰𝔬(2, n)  𝔰𝔬(2, p) + 𝔰𝔬(n − p)
𝔰𝔬(2, 2n)  𝔲(1, n)
𝔰𝔭(n, ℝ)  𝔲(p, n − p)
𝔰𝔭(n, ℝ)  𝔰𝔭(p, ℝ) + 𝔰𝔭(n − p, ℝ)
𝔢_{6(−14)}  𝔰𝔬(10) + 𝔰𝔬(2)
𝔢_{6(−14)}  𝔰𝔬^{∗}(10) + 𝔰𝔬(2)
𝔢_{6(−14)}  𝔰𝔬(8, 2) + 𝔰𝔬(2)
𝔢_{6(−14)}  𝔰𝔲(5, 1) + 𝔰𝔩(2, ℝ)
𝔢_{6(−14)}  𝔰𝔲(4, 2) + 𝔰𝔲(2)
𝔢_{7(−25)}  𝔢_{6(−78)} + 𝔰𝔬(2)
𝔢_{7(−25)}  𝔢_{6(−14)} + 𝔰𝔬(2)
𝔢_{7(−25)}  𝔰𝔬(10, 2) + 𝔰𝔩(2, ℝ)
𝔢_{7(−25)}  𝔰𝔬^{∗}(12) + 𝔰𝔲(2)
𝔢_{7(−25)}  𝔰𝔲(6, 2)

Table 3.3.1

(𝔤, 𝔤^{τ}) satisfying (1.5.2) τZ = −Z

𝔤  𝔤^{τ}
𝔰𝔲(p, q)  𝔰𝔬(p, q)
𝔰𝔲(n, n)  𝔰𝔩(n, ℂ) + ℝ
𝔰𝔲(2p, 2q)  𝔰𝔭(p, q)
𝔰𝔬^{∗}(2n)  𝔰𝔬(n, ℂ)
𝔰𝔬^{∗}(4n)  𝔰𝔲^{∗}(2n) + ℝ
𝔰𝔬(2, n)  𝔰𝔬(1, p) + 𝔰𝔬(1, n − p)
𝔰𝔭(n, ℝ)  𝔤𝔩(n, ℝ)
𝔰𝔭(2n, ℝ)  𝔰𝔭(n, ℂ)
𝔢_{6(−14)}  𝔣_{4(−20)}
𝔢_{6(−14)}  𝔰𝔭(2, 2)
𝔢_{7(−25)}  𝔢_{6(−26)} + 𝔰𝔬(1, 1)
𝔢_{7(−25)}  𝔰𝔲^{∗}(8)

Table 3.3.2

\underline{3.4.} We review a discrete decomposition theorem of branching laws in the following special settings:

Fact 3.4.1. 1) If τ satisfies (1.5.1) and if π is an irreducible unitary highest weight module of G, then the restriction π|_{G_{0}^{τ}} splits into a discrete Hilbert sum of irreducible unitary representations of G_{0}^{τ}:

(3.4.1)  π|_{G_{0}^{τ}} ≃ ∑_{μ∈Ĝ_{0}^{τ}}^{⊕}m_{π}(μ)μ  (Hilbert direct sum),

where the multiplicity m_{π}(μ) is finite for every μ.

2) Let π_{1}, π_{2} be irreducible unitary highest weight modules of G. Then the tensor product π_{1} ˆ⊗π_{2} splits into a discrete Hilbert sum of irreducible unitary representations of G, each of finite multiplicity. Furthermore, if at least one of π_{1} or π_{2} is a holomorphic discrete series representation for G, then any irreducible summand is also a holomorphic discrete series representation.

Proof. See [31], Theorem 6.3; also [37]; [35], Theorem 4.2; [24], Corollary 2.3 (some references treat only the case of holomorphic discrete series representations). □

Remark 3.4.2. Some remarks of Fact 3.4.1 are in order.

1) If τ = θ (Cartan involution), then τZ = Z (see (1.5.1)). In this case, Fact 3.4.1 (1) is a special case of Harish-Chandra’s theorem (π is any irreducible unitary representation of G).

2) The point in Fact 3.4.1 (1) and (2) is that there is no continuous spectrum in the irreducible decomposition formula. On the other hand, if τ satisfies (1.5.2) instead of (1.5.1) and if π is a holomorphic discrete series representation for G, then the restriction π|_{G_{0}^{τ}} is not discretely decomposable ([30], Part III, Theorem 5.3). In this case, R. Howe, J. Repka, and G. ’Olafsson-B. Ørsted studied irreducible decompositions of π|_{G_{0}^{τ}} by means of the L^{2}-harmonic analysis on Riemannian symmetric spaces ([18], [44], [39]).

3) Finite multiplicity result in the branching law (3.4.1) still holds in a wider setting where π is a Zuckerman-Vogan’s A_{𝔮}(λ) ([51]) which is not necessarily a highest weight module, provided π is discretely decomposable as a (𝔤_{ℂ}^{τ}, K^{τ})-module (a proof of Wallach’s conjecture in the discretely decomposable case; see [30], Part III, Corollary 4.3). If the restriction is not discretely decomposable, the multiplicity of the branching law can be infinite even though (G, H) is a semisimple symmetric pair.

4) Theorems B and D assert that multiplicities in the above Fact is uniformly bounded with respect to irreducible summands. This is not always the case for non-holomorphic discrete series (see Example 6.3).

\underline{3.5.} It is well-known that an irreducible unitary highest weight module can be realized as a subrepresentation of the space of global holomorphic sections of an equivariant line bundle over a Hermitian symmetric space. We supply a proof here for the convenience of the reader.

Lemma 3.5. Let (π, H) ∈ Ĝ be an irreducible unitary highest weight module of scalar type. Then there is a unitary character χ: K → ℂ^{×} such that we have an injective continuous G-homomorphism H → O(G/K, L). Here, L := G×_{K} (χ, ℂ) → G/K is the G-equivariant holomorphic line bundle associated to χ. In particular, any irreducible unitary highest weight module is realized in a Hilbert space with reproducing kernel.

Proof. As (π, H) is a highest weight module of scalar type, U := H_{K}^{𝔭^{+}} (see Definition 1.4) is a one dimensional representation, denoted by χ, of K. Let ( , )_{H} be a G-invariant inner product on H. Then K acts unitarily on H, especially on U. We write ( , )_{U} for the induced inner product on U ≃ ℂ. We consider the map

G × H × U → ℂ, (g, v, u) ↦ (π(g)^{−1}v, u)_{H} = (v, π(g)u)_{H}.

For each fixed g ∈ G and v ∈ H, the map U → ℂ, u ↦ (π(g)^{−1}v, u)_{H} is an antilinear functional on U. Then there exists a unique element F_{v}(g) ∈ U by the Riesz representation theorem for U ≃ ℂ such that

(F_{v}(g), u)_{U} = (π(g)^{−1}v, u)_{H} for any u ∈ U.

Then it is easy to see that F_{v}(gk) = χ(k)^{−1}F_{v}(g) and F_{π(g′)v}(g) = F_{v}(g′^{−1}g) for any g, g′ ∈ G, k ∈ K and v ∈ H. As u is a smooth vector in H, (F_{v}(g), u)_{U} = (v, π(g)u)_{H} is a C^{∞}-function on G. Then F_{v}(g) is a C^{∞}-function on G with value in U ≃ ℂ for each fixed v ∈ H. Thus, we have a non-zero G-intertwining operator given by

F: H → C^{∞}(G ×_{K} U), v ↦ F_{v}.

As U is annihilated by 𝔭^{+}, F_{v} is a holomorphic section of the holomorphic vector bundle G ×_{K} U → G/K, that is, F_{v} ∈ O(G ×_{K} U). Then, the non-zero map F: H → O(G ×_{K} U) is injective because H is irreducible. Furthermore, F is continuous by the closed graph theorem. Thus, H has a reproducing kernel given by the point evaluation map (see §2.4). Hence the lemma. □

\underline{3.6.} The next Proposition is a stepping-stone of Theorem A and Theorem B(2). We shall prove in Lemma 5.1 that its assumption (i.e. the existence of σ) is always satisfied.

Proposition 3.6. Suppose G is a simple Lie group such that G/K is Hermitian. Let τ be an involutive automorphism of G that stabilizes K, and H a subgroup of G such that G_{0}^{τ} ⊂ H ⊂ G^{τ}. Assume that there exists an involutive automorphism σ of G satisfying (3.2.1), (3.2.2) and (3.6.1) σZ = −Z.

Let π be an irreducible unitary highest weight module of G of scalar type. Then the restriction π|_{H} decomposes into irreducible representations of H with multiplicity free.

Proof. The assumption (3.6.1) implies that the induced involutive diffeomorphism σ: G/K → G/K is anti-holomorphic (see §1.5). By using the assumption (3.2.1) and (3.2.2), for any x ∈ G/K, one can find g ∈ G_{0}^{τ} satisfying σ(x) = g · x, by Lemma 3.2. Hence, the assumption (2.2.3) of Theorem 2.2 is satisfied. By Lemma 3.5, there are a holomorphic G-equivariant line bundle L → G/K and an injective continuous G-intertwining map H → O(L), whence the assumption (2.2.1) is also satisfied. By Lemma 8.6 in the Appendix, we have an isomorphism \overline{σ^{∗}L} ≃ L between G-equivariant holomorphic line bundles over G/K. Therefore, all assumptions of Theorem 2.2 are satisfied and thus the ring End_{H}(H) is commutative. □

\underline{3.7.} The next Proposition is a stepping-stone of Theorem D (2). We shall prove in Lemma 4.1.1 that its assumption (i.e. the existence of σ) is always satisfied.

Proposition 3.7. Suppose G is a simple non-compact Lie group such that G/K is Hermitian. Assume that there exists an involutive automorphism σ of G satisfying the following three conditions: (3.7.1) σ and θ commute. (3.7.2) ℝ- rank 𝔤 = ℝ- rank 𝔤^{σ}. (3.7.3) σZ = −Z.

Let (π_{1}, H_{1}), (π_{2}, H_{2}) ∈ Ĝ be irreducible unitary highest weight modules of scalar type. Then the tensor product π_{1} ˆ⊗π_{2} decomposes discretely into irreducible unitary representations of G with multiplicity free.

The proof of Proposition 3.7 parallels to that of Proposition 3.6.

Proof. It follows from (3.7.3) that we have an anti-holomorphic involutive diffeomorphism:

G/K × G/K → G/K × G/K, (g_{1}K, g_{2}K) ↦ (σ(g_{1})K, σ(g_{2})K).

Let τ be an involutive automorphism of G×G defined by τ(g_{1}, g_{2}) := (g_{2}, g_{1}). Then (G × G)^{τ} = diag(G) := {(g, g) : g ∈ G}. We observe the following equalities:

ℝ- rank(𝔤 ⊕ 𝔤)/(𝔤 ⊕ 𝔤)^{τ}  = ℝ- rank 𝔤,
ℝ- rank(𝔤 ⊕ 𝔤)^{σ×σ}/(𝔤 ⊕ 𝔤)^{τ,σ×σ} = ℝ- rank 𝔤^{σ}.

Thus, the assumption (3.7.2) implies

ℝ- rank(𝔤 ⊕ 𝔤)/(𝔤 ⊕ 𝔤)^{τ} = ℝ- rank(𝔤 ⊕ 𝔤)^{σ×σ}/(𝔤 ⊕ 𝔤)^{τ,σ×σ}.

Therefore, given (x_{1}, x_{2}) ∈ G/K × G/K = (G × G)/(K × K), there exists (g, g) ∈ (G × G)^{τ} satisfying (g · x_{1}, g · x_{2}) = (σ(x_{1}), σ(x_{2})) by Lemma 3.2.

We realize the Hilbert space H_{1} ˆ⊗H_{2} in the space of global holomorphic sections of G × G- equivariant holomorphic line bundle L_{1} ⊠ L_{2} → G/K × G/K, as in Lemma 3.5. Then we have an isomorphism of holomorphic line bundles

\overline{(σ × σ)^{∗}(L_{1} ⊠ L_{2})} ≃ L_{1} ⊠ L_{2}

by Lemma 8.6 in the Appendix. Now, applying Theorem 2.2 with (L → D, H, H, σ) replaced by (L_{1} ⊠ L_{2} → G/K × G/K, H_{1} ˆ⊗H_{2}, diag(G), σ × σ), we conclude that the commutant of {(π_{1}(g), π_{2}(g)) : g ∈ G} in End(H_{1} ˆ⊗H_{2}) is commutative. That is, the tensor product π_{1} ˆ⊗π_{2} is multiplicity free as a G-module (see Definition 1.2).

On the other hand, it follows from Fact 3.4.1 (2) that the tensor product π_{1} ˆ⊗π_{2} decomposes discretely into irreducible representations of diag(G). Hence, we have proved Proposition. □

§4. Proof of Theorem C and Theorem D

In this section we complete the proof of Theorem C and Theorem D. First we shall prove Theorem C and Theorem D (2) (multiplicity-free result) in §4.1 and §4.2, and then deduce Theorem D (1) from Theorem D (2) by tensoring argument (§4.3– 4.5).

Throughout this section, we may and do assume that G is simply connected so that any automorphism of 𝔤 lifts to G.

For the proof of Theorem C, we divide into two cases: Case I) π_{1} and π_{2} are highest (or lowest) weight modules at the same time (§4.1). Case II) π_{1} is a highest weight module and π_{2} is a lowest one (or vice versa) (§4.2).

\underline{4.1.} Thanks to Proposition 3.7, Theorem D (2) (or Theorem C in the case I) is a consequence of the following lemma:

Lemma 4.1.1. Suppose G is a non-compact, simply connected, simple Lie group such that G/K is a Hermitian symmetric space. Let θ be a Cartan involution corresponding to K. Then there exists an involutive automorphism σ of G satisfying (3.7.1), (3.7.2) and (3.7.3). Here we recall: (3.7.1) σ and θ commute. (3.7.2) ℝ- rank 𝔤 = ℝ- rank 𝔤^{σ}. (3.7.3) σZ = −Z.

Proof. We prove the Lemma based on the classification of 𝔤. We first note that for any involutive automorphism σ of G, there exists g ∈ G such that σ^{g}θ = θσ^{g} if we put σ^{g}(x) := gσ(g^{−1}xg)g^{−1} for x ∈ G. Thus, (3.7.1) is always satisfied after replacing σ by some σ^{g}. The remaining conditions (3.7.2) and (3.7.3) (cf. Table 3.3.2) are satisfied if we choose σ ∈ Aut(G) in the following Table 4.1.2 for each simple non-compact Lie group G of Hermitian type:

(𝔤, 𝔤^{σ}) satisfying (3.7.2) and (3.7.3)
𝔤  𝔤^{σ}  ℝ- rank 𝔤 = ℝ- rank 𝔤^{σ}
𝔰𝔲(p, q)  𝔰𝔬(p, q)  min(p, q)
𝔰𝔬^{∗}(2n)  𝔰𝔬(n, ℂ)  [\frac{1}{2}n]
𝔰𝔭(n, ℝ)  𝔤𝔩(n, ℝ)  n
𝔰𝔬(2, n) 𝔰𝔬(1, n − 1) + 𝔰𝔬(1, 1)  2
𝔢_{6(−14)}  𝔰𝔭(2, 2)  2
𝔢_{7(−25)}  𝔰𝔲^{∗}(8)  3

Table 4.1.2.

Here, we have assumed n ≥ 2 for 𝔤 = 𝔰𝔬(2, n). The above table completes the proof of Lemma. □

Remark 4.1.3. The choice of σ satisfying (3.7.1), (3.7.2) and (3.7.3) is not unique. For example, we may choose 𝔤^{σ} ≃ 𝔢_{6(−26)} + ℝ instead of 𝔤^{σ} ≃ 𝔰𝔲^{∗}(8) if 𝔤 = 𝔢_{7(−25)}.

\underline{4.2.} Let us prove Theorem C in the case II. We define two involutions τ and σ on G × G by τ(g_{1}, g_{2}) := (g_{2}, g_{1}) and σ(g_{1}, g_{2}) := (θg_{2}, θg_{1}) for g_{1}, g_{2} ∈ G. Then (G × G)^{τ} = diag(G). Obviously, σ, τ and a Cartan involution θ × θ of G × G all commute. We write M for the Hermitian symmetric space G/K, and \overline{M} for the conjugate complex manifold. Then σ acts anti-holomorphically on M × \overline{M}. We observe the following equations:

ℝ- rank(𝔤 ⊕ 𝔤)/(𝔤 ⊕ 𝔤)^{τ} = ℝ- rank 𝔤,
ℝ- rank(𝔤 ⊕ 𝔤)^{σ}/(𝔤 ⊕ 𝔤)^{σ,τ} = ℝ- rank 𝔤/𝔨 = ℝ- rank 𝔤.

It follows from Lemma 3.2 that for any (x_{1}, x_{2}) ∈ M × \overline{M} there exists (g, g) ∈ (G × G)^{τ} such that σ(x_{1}, x_{2}) = (g, g) · (x_{1}, x_{2}). The outer tensor product π_{1} ⊠ π_{2} is realized in a subspace of holomorphic sections of some holomorphic line bundle over M × \overline{M}, provided π_{1} (resp. π_{2}) is a unitary highest (resp. lowest) weight module of scalar type. Therefore, Theorem C in the case II follows from Theorem 2.2. □

Now, the argument in §4.1 and §4.2 completes the proof of Theorem C and Theorem D (2).

\underline{4.3.} In order to deduce Theorem D (1) from Theorem D (2), we prepare the following two lemmas.

Lemma 4.3. Suppose that (π, H) is an irreducible unitary highest weight module of G. Then there exist an irreducible unitary highest weight module π′ of scalar type and a finite dimensional representation F of G such that π_{K} occurs as a subquotient of the tensor product π′_{K} ⊗ F.

Proof. We denote by C the analytic subgroup of G with Lie algebra 𝔠(𝔨) = ℝZ, the center of 𝔨. As (π, H) is a unitary representation of G, H_{K}^{𝔭_{+}} is an irreducible (finite dimensional) unitary representation of K. We may write a K-module H_{K}^{𝔭_{+}} as σ ⊗ χ_{0}, where σ ∈ K̂ such that σ|_{C} is trivial and χ_{0} is a unitary character of K. Let χ′ be a unitary character of K such that χ′ is trivial on the center Z_{G} of G (namely, χ′ is well-defined as a representation of Ad_{G}(K) ≃ K/Z_{G}). For later purpose, we take χ′ so that −\sqrt{−1}dχ′(Z) ≫ 0. There exists an irreducible finite dimensional representation F of G such that F^{𝔭_{+}} ≃ σ ⊗ χ′ as K-modules because σ ⊗ χ′ is well-defined as an algebraic representation of Ad_{G}(K). We define one dimensional representation χ := χ_{0} ⊗ (χ′)^{∗} of K. Because −\sqrt{−1}dχ(Z) ≪ 0, there exists an irreducible unitary highest weight module (π′, H′) such that

(H′_{K})^{𝔭_{+}} ≃ χ.

As H′_{K} ⊗F is a (𝔤_{ℂ}, K)-module of finite length, all subquotient modules are highest weight modules. Then, H_{K} arises as a subquotient of H′_{K} ⊗ F in view of

H_{K}^{𝔭_{+}} ≃ σ ⊗ χ_{0} ≃ χ ⊗ (σ ⊗ χ′) ≃ (H′_{K})^{𝔭_{+}} ⊗ F^{𝔭_{+}} ⊂ (H′_{K} ⊗ F)^{𝔭_{+}}.

Hence, we have shown Lemma 4.3. □

\underline{4.4.} Let π be a (𝔤_{ℂ}, K)-module of finite length. We denote by m(π) the number of irreducible (𝔤_{ℂ}, K)-modules (counted with multiplicity) occurring as subquotients in π. Here is a uniform estimate of m(π) under the operation of tensor products:

Lemma 4.4. Let F be a finite dimensional representation of a real reductive connected Lie group G. Then there exists a constant C ≡ C(F) such that

m(π ⊗ F) ≤ C,

for any irreducible (𝔤_{ℂ}, K)-module π.

Proof. We divide the proof into three steps: Step 1 (π is a finite dimensional representation): We shall prove

(4.4.1)  m(π ⊗ F) ≤ dim F

for any finite dimensional representation π of G.

Let 𝔟 = 𝔱 + 𝔲 be a Borel subalgebra of 𝔤_{ℂ} with 𝔲 nilradical. We denote by H^{j}(𝔲, V) the j-th cohomology group of the Lie algebra 𝔲 with coefficients in a 𝔲-module V. Since 𝔟 is solvable, we can choose a 𝔟-stable filtration

F = F_{k} ⊃ F_{k−1} ⊃ ⋯ ⊃ F_{0} = {0},

such that dim F_{i}/F_{i−1} = 1. We will show by induction on i that

(4.4.2)  dim H^{0}(𝔲, π ⊗ F_{i}) ≤ i.

This will imply m(π ⊗ F) = dim H^{0}(𝔲, π ⊗ F) ≤ k = dim F. The inequality (4.4.2) is trivial if i = 0. Suppose (4.4.2) is proved for i − 1. The short exact sequence of 𝔟-modules

0 → π ⊗ F_{i−1} → π ⊗ F_{i} → π ⊗ (F_{i}/F_{i−1}) → 0

gives rise to a long exact sequence

0 → H^{0}(𝔲, π ⊗ F_{i−1}) → H^{0}(𝔲, π ⊗ F_{i}) → H^{0}(𝔲, π ⊗ (F_{i}/F_{i−1}))

→ H^{1}(𝔲, π ⊗ F_{i−1}) → …

of 𝔱-modules. In particular, we have

(4.4.3) dim H^{0}(𝔲, π ⊗ F_{i}) ≤ dim H^{0}(𝔲, π ⊗ F_{i−1}) + dim H^{0}(𝔲, π ⊗ (F_{i}/F_{i−1})).

Because F_{i}/F_{i−1} is trivial as a 𝔲-module, we have

(4.4.4)  H^{0}(𝔲, π ⊗ (F_{i}/F_{i−1})) = H^{0}(𝔲, π) ⊗ (F_{i}/F_{i−1}).

In light that H^{0}(𝔲, π) is the space of highest weight vectors, the dimension of the right side of (4.4.4) is one. Now, the inductive assumption combined with (4.4.3) yields dim H^{0}(𝔲, π ⊗ F_{i}) ≤ i, as desired. Step 2 (π is a principal series representation): Here π is allowed to be reducible. Let P = LN be a Levi decomposition of a minimal parabolic subgroup P of G, W an irreducible finite dimensional representation of L, and Ind_{P}^{G}(W) the underlying (𝔤_{ℂ}, K)-module of a principal series representation induced from the representation W ⊠ 1 of P = LN (without ρ-shift). Then we have an isomorphism of (𝔤_{ℂ}, K)- modules

Ind_{P}^{G}(W) ⊗ F ≃ Ind_{P}^{G}(W ⊗ F),

where F is regarded as a P -module on the right side. We take a P -stable filtration

W_{n} := W ⊗ F ⊃ W_{n−1} ⊃ ⋯ ⊃ W_{0} = {0}

such that each W_{i}/W_{i−1} is irreducible as a P -module. We notice that n ≤ dim F by applying Step 1 to the L-module F |_{L}. We note that there is an upper bound, say m(G), of m(Ind_{P}^{G}(W)) depending only on G and not on an irreducible representation W of L. As Ind_{P}^{G}(W ⊗ F) is isomorphic to ⨁_{i=1}^{n} Ind_{P}^{G}(W_{i}/W_{i−1}) in the Grothendieck group of (𝔤_{ℂ}, K)-modules, we have proved that

m(Ind_{P}^{G}(W) ⊗ F) ≤ n m(G) ≤ (dim F) m(G)

for any irreducible finite dimensional representation W of L. Step 3 (general case): By Casselman’s subrepresentation theorem, the general case follows from Step 2. □

\underline{4.5.} Now let us complete the proof of Theorem D. Let π = π_{1}⊠π_{2} be an irreducible unitary highest weight module of G′ := G × G. It follows from Lemma 4.3 that there exist an irreducible unitary highest weight module π′ = π′_{1} ⊠ π′_{2} of scalar type and a finite dimensional representation F of G′ such that π_{K} occurs as a subquotient of π′_{K} ⊗ F. Fix μ ∈ Ĝ. We denote by Hom_{G}[V_{1}, V_{2}] the space of (𝔤_{ℂ}, K)-homomorphisms between the Grothendieck groups of (V_{1})_{K} and (V_{2})_{K} for G-modules V_{1} and V_{2} of finite length. Then we have

dim Hom_{G} [μ, π|_{diag(G)}] ≤ dim Hom_{G} [μ, (π′ ⊗ F)|_{diag(G)}]
= dim Hom_{G} [μ ⊗ (F^{∗}|_{diag(G)}), π′|_{diag(G)}]

≤ m(μ ⊗ (F^{∗}|_{diag(G)})) (4.5.1)

≤ C(F^{∗}).

Here the inequality (4.5.1) follows from the multiplicities free result of π′|_{diag(G)} ≃ π′_{1} ˆ⊗π′_{2} in Theorem D (2). In the last inequalities C(F^{∗}) is the constant in Lemma 4.4. This completes the proof of Theorem D. □

§5. Proof of Theorem A and Theorem B

In this section, we shall complete the proof of Theorems A and B.

First, let us explain Theorem B can be deduced from Theorem A, as we proved Theorem D from Theorem C in §4. Suppose Theorem A is proved. If τ satisfies τZ = Z (see (1.5.1)), then (π, H) is decomposed discretely by Fact 3.4.1 (1). Hence Theorem A implies Theorem B (2). Then, the uniformly bounded estimate of the multiplicity for the general case (i.e. Theorem B (1)) follows from the multiplicityfree result for the scalar type (i.e. Theorem B (2)) by using Lemmas 4.3 and 4.4.

Thus, we shall concentrate on the proof of Theorem A.

\underline{5.1.} In light of Proposition 3.6, the proof of Theorem A will be completed if we show the following lemma.

Lemma 5.1. Suppose G is a simply connected, simple non-compact Lie group of Hermitian type. Let τ be an involutive automorphism of G such that τ commutes with a Cartan involution θ. Then there exists an involutive automorphism σ of G satisfying (3.2.1), (3.2.2) and (3.6.1). Here we recall: (3.2.1) σ, τ and θ commute with one another. (3.2.2) ℝ- rank 𝔤/𝔤^{τ} = ℝ- rank 𝔤^{σ}/𝔤^{σ,τ}. (3.6.1) σZ = −Z.

The rest of this section is devoted to the proof of Lemma 5.1. The proof of Lemma 5.1 is divided into three cases: Case (5.1.a) τZ = −Z (§5.2). Case (5.1.b) τZ = Z, 𝔤 is classical (§5.3). Case (5.1.c) τZ = Z, 𝔤 is exceptional (§5.4–§5.9).

\underline{5.2.} (Proof of Lemma 5.1 in the case (5.1.a)) If τZ = −Z, we put σ := τθ. Then (3.2.1) and (3.6.1) are automatically satisfied. Let us verify (3.2.2). In view of 𝔤^{σ}/𝔤^{σ,τ} = 𝔤^{θτ} /𝔤^{θτ,τ} = 𝔤^{θτ} /𝔤^{θτ,θ}, we have ℝ- rank 𝔤^{σ}/𝔤^{σ,τ} = ℝ- rank 𝔤^{θτ}. On the other hand, ℝ- rank 𝔤/𝔤^{τ} = ℝ- rank 𝔤^{θτ} (see (3.1.1)). Hence (3.2.2) is also satisfied. Thus, Lemma 5.1 in the case (5.1.a) is proved. □

\underline{5.3.} (Proof of Lemma 5.1 in the case (5.1.b)) Let us enrich Table 3.3.1 with a list of 𝔤^{θτ} and ℝ- rank 𝔤/𝔤^{τ} = ℝ- rank 𝔤^{θτ}:

𝔤  𝔤^{τ}  𝔤^{θτ}  ℝ- rank 𝔤/𝔤^{τ}
𝔰𝔲(p, q) 𝔰(𝔲(i, j) + 𝔲(p − i, q − j)) 𝔰(𝔲(i, q − j) + 𝔲(p − i, j)) min(i, q − j) + min(j, p − i)
𝔰𝔲(n, n)  𝔰𝔬^{∗}(2n)  𝔰𝔭(n, ℝ)  n
𝔰𝔲(n, n)  𝔰𝔭(n, ℝ)  𝔰𝔬^{∗}(2n)  [\frac{n}{2}]
𝔰𝔬^{∗}(2n) 𝔰𝔬^{∗}(2p) + 𝔰𝔬^{∗}(2n − 2p)  𝔲(p, n − p)  min(p, n − p)
𝔰𝔬^{∗}(2n)  𝔲(p, n − p)  𝔰𝔬^{∗}(2p) + 𝔰𝔬^{∗}(2n − 2p)  [\frac{p}{2}] + [\frac{n−p}{2}]
𝔰𝔬(2, n) 𝔰𝔬(2, p) + 𝔰𝔬(n − p)  𝔰𝔬(2, n − p) + 𝔰𝔬(p)  min(n − p, 2)
𝔰𝔬(2, 2n)  𝔲(1, n)  𝔲(1, n)  1
𝔰𝔭(n, ℝ)  𝔲(p, n − p)  𝔰𝔭(p, ℝ) + 𝔰𝔭(n − p, ℝ)  n
𝔰𝔭(n, ℝ) 𝔰𝔭(p, ℝ) + 𝔰𝔭(n − p, ℝ)  𝔲(p, n − p)  min(p, n − p)
𝔢_{6(−14)}  𝔰𝔬(10) + 𝔰𝔬(2)  𝔢_{6(−14)}  2
𝔢_{6(−14)} 𝔰𝔬^{∗}(10) + 𝔰𝔬(2)  𝔰𝔲(5, 1) + 𝔰𝔩(2, ℝ)  2
𝔢_{6(−14)} 𝔰𝔬(8, 2) + 𝔰𝔬(2)  𝔰𝔬(8, 2) + 𝔰𝔬(2)  2
𝔢_{6(−14)} 𝔰𝔲(5, 1) + 𝔰𝔩(2, ℝ)  𝔰𝔬^{∗}(10) + 𝔰𝔬(2)  2
𝔢_{6(−14)} 𝔰𝔲(4, 2) + 𝔰𝔲(2)  𝔰𝔲(4, 2) + 𝔰𝔲(2)  2
𝔢_{7(−25)}  𝔢_{6(−78)} + 𝔰𝔬(2)  𝔢_{7(−25)}  3
𝔢_{7(−25)}  𝔢_{6(−14)} + 𝔰𝔬(2)  𝔰𝔬(10, 2) + 𝔰𝔩(2, ℝ)  3
𝔢_{7(−25)} 𝔰𝔬(10, 2) + 𝔰𝔩(2, ℝ)  𝔢_{6(−14)} + 𝔰𝔬(2)  2
𝔢_{7(−25)} 𝔰𝔬^{∗}(12) + 𝔰𝔲(2)  𝔰𝔲(6, 2)  2
𝔢_{7(−25)}  𝔰𝔲(6, 2)  𝔰𝔬^{∗}(12) + 𝔰𝔲(2)  3

Table 5.3.1 If 𝔤 is a classical Lie algebra, then Lemma 5.1 is proved by taking σ as follows:

Choice of σ satisfying (3.2.1), (3.2.2) and (3.6.1) in classical case

𝔤  𝔤^{σ}  𝔤^{σ,τ} = 𝔤^{σ} ∩ 𝔤^{τ}  𝔤^{σ,θτ} = 𝔤^{σ} ∩ 𝔤^{θτ}
𝔰𝔲(p, q)  𝔰𝔬(p, q)  𝔰𝔬(i, j) + 𝔰𝔬(p − i, q − j)  𝔰𝔬(i, q − j) + 𝔰𝔬(p − i, j)
𝔰𝔲(n, n)  𝔰𝔬(n, n)  𝔰𝔬(n, ℂ)  𝔤𝔩(n, ℝ)
𝔰𝔲(n, n)  𝔰𝔬(n, n)  𝔤𝔩(n, ℝ)  𝔰𝔬(n, ℂ)
𝔰𝔬^{∗}(2n)  𝔰𝔬(n, ℂ)  𝔰𝔬(p, ℂ) + 𝔰𝔬(n − p, ℂ)  𝔰𝔬(p, n − p)
𝔰𝔬^{∗}(2n)  𝔰𝔬(n, ℂ)  𝔰𝔬(p, n − p)  𝔰𝔬(p, ℂ) + 𝔰𝔬(n − p, ℂ)
𝔰𝔬(2, n) 𝔰𝔬(1, n − 1) + 𝔰𝔬(1, 1) 𝔰𝔬(1, p) + 𝔰𝔬(n − p − 1) 𝔰𝔬(1, 1) + 𝔰𝔬(1, n − p − 1) + 𝔰𝔬(p)
𝔰𝔬(2, 2n) 𝔰𝔬(1, n) + 𝔰𝔬(1, n)  𝔰𝔬(1, n)  𝔰𝔬(1, n)
𝔰𝔭(n, ℝ)  𝔤𝔩(n, ℝ)  𝔰𝔬(p, n − p)  𝔤𝔩(p, ℝ) + 𝔤𝔩(n − p, ℝ)
𝔰𝔭(n, ℝ)  𝔤𝔩(n, ℝ)  𝔤𝔩(p, ℝ) + 𝔤𝔩(n − p, ℝ)  𝔰𝔬(p, n − p)

Table 5.3.2

It is obvious that we can choose σ in the above table such that σ, τ and θ commute with one another.

The rank condition (3.2.2), or equivalently, ℝ- rank 𝔤^{θτ} = ℝ- rank 𝔤^{σ,θτ}, can be verified by comparing 𝔤^{θτ} in Table 5.3.1 with 𝔤^{σ,θτ} in Table 5.3.2. The condition (3.6.1) (i.e. σZ = −Z) holds because all the pair (𝔤, 𝔤^{σ}) in Table 5.3.2 is included in Table 3.3.2. This shows that Lemma 5.1 holds in the case (5.1.b). □

\underline{5.4.} The rest of this section is devoted to the proof of Lemma 5.1 in the case (5.1.c) (there are 10 exceptional cases in view of Table 5.3.1). We shall show that there exists an involutive automorphism σ of 𝔤 satisfying (3.2.1), (3.2.2) and (3.6.1) such that 𝔤^{σ} ≃ 𝔰𝔭(2, 2) for 𝔤 = 𝔢_{6(−14)} (there are 5 cases), and 𝔤^{σ} ≃ 𝔰𝔲^{∗}(2, 2) for 𝔤 = 𝔢_{7(−25)} (there are also 5 cases), respectively. A main difficulty in the following proof arises from the fact that we cannot tell a priori the commutativity of two involutive automorphisms σ and τ of a semisimple Lie algebra. In fact, there exist two involutive automorphisms σ and τ of a semisimple Lie algebra 𝔤 such that σ never commutes with any conjugation of τ by inner automorphisms (such a phenomenon does not occur if τ is a Cartan involution).

In order to treat exceptional cases as systematically as possible, we shall give a brief review of ϵ-families of semisimple symmetric spaces [42]. Let 𝔤 be a semisimple Lie algebra, τ an involutive automorphism of 𝔤, and θ a Cartan involution of 𝔤 commuting with τ. Fix a maximal abelian subspace 𝔞 of 𝔤^{−θ,−τ}. Then Σ(𝔞) ≡ Σ(𝔤, 𝔞) satisfies the axiom of root system (see [45], Theorem 5).

A map ϵ: Σ(𝔞)∪{0} → {1, −1} is called a signature of Σ(𝔞) if ϵ satisfies ϵ(α+β) = ϵ(α)ϵ(β) for any α, β, α + β ∈ Σ(𝔞), ϵ(−α) = ϵ(α) for any α ∈ Σ(𝔞) and ϵ(0) = 1 (see [42], (1.9.3.1)). To a signature ϵ of Σ(𝔞), we associate an involution τ_{ϵ} of 𝔤 defined by τ_{ϵ}(X) := ϵ(λ)τ(X) if X ∈ 𝔤(𝔞; λ), λ ∈ Σ(𝔞) ∪ {0}. Then τ_{ϵ} defines another symmetric pair (𝔤, 𝔥_{ϵ}). The set

F ((𝔤, 𝔥)) := {(𝔤, 𝔥_{ϵ}) : ϵ is a signature of Σ(𝔞)}

is called an ϵ-family (or H_{ϵ}-family) of symmetric pairs ([42], §6). It is called a K_{ϵ}-family of symmetric pairs if τ = θ (a Cartan involution). For example, each of the followings:

{(𝔰𝔩(n, ℝ), 𝔰𝔬(p, n − p)) : 0 ≤ p ≤ n}, {(𝔢_{6(−14)}, 𝔰𝔬(10) + 𝔰𝔬(2)), (𝔢_{6(−14)}, 𝔰𝔬^{∗}(10) + 𝔰𝔬(2)), (𝔢_{6(−14)}, 𝔰𝔬(8, 2) + 𝔰𝔬(2))}, {(𝔢_{7(−25)}, 𝔢_{6(−78)} + 𝔰𝔬(2)), (𝔢_{7(−25)}, 𝔢_{6(−14)} + 𝔰𝔬(2)), (𝔢_{7(−25)}, 𝔢_{6(−26)} + 𝔰𝔬(1, 1))}

is a K_{ϵ}-family of symmetric pairs [41]. Some other examples of H_{ϵ}-families will be given in §5.9. We shall sometimes refer to Tables in [42] (cf. [28], p. 660 for some corrections) for the computation of 𝔤^{τ_{ϵ}} and 𝔤^{θτ}.

To make new pairs (σ, τ) from old, the following result is useful.

Lemma 5.4. Let τ and σ be involutive automorphisms of 𝔤. If the pair (σ, τ) satisfies (3.2.1) and (3.2.2), so does (σ, τ_{ϵ}).

Proof. Take a maximal abelian subspace 𝔞 in

𝔤^{−θ,σ,−τ} = {Y ∈ 𝔤 : (−θ)Y = σY = (−τ)Y = Y },

which is also a maximal abelian subspace of 𝔤^{−θ,−τ} by the rank condition (3.2.2). Let τ_{ϵ} be an involutive automorphism of 𝔤 associated to a signature ϵ of Σ(𝔤, 𝔞).

Let us show that στ_{ϵ} = τ_{ϵ}σ. Fix X ∈ 𝔤(𝔞; λ). Then we have

στ_{ϵ}(X) = σ(ϵ(λ)τ(X)) = ϵ(λ)σ(τ(X)).

On the other hand, because σ|_{𝔞} = id, we have σX ∈ 𝔤(𝔞; σλ) = 𝔤(𝔞; λ). Thus,

τ_{ϵ}σ(X) = ϵ(λ)τ(σ(X)) = ϵ(λ)σ(τ(X)).

This shows στ_{ϵ} = τ_{ϵ}σ on 𝔤(𝔞; λ) for all λ. Hence, the pair (σ, τ_{ϵ}) satisfies (3.2.1). As ϵ(0) = 1, we have τ_{ϵ}(Y) = τ(Y) = −Y if Y ∈ 𝔞 (⊂ 𝔤(𝔞; 0)). This implies

𝔞 ⊂ 𝔤^{−θ,σ,−τ_{ϵ}} = {Y ∈ 𝔤 : (−θ)Y = σY = (−τ_{ϵ})Y = Y }.

Since ℝ- rank 𝔤^{σ}/𝔤^{σ,τ_{ϵ}} is by definition the dimension of a maximal abelian subspace contained in 𝔤^{−θ,σ,−τ_{ϵ}}, we have

ℝ- rank 𝔤^{σ}/𝔤^{σ,τ_{ϵ}} ≥ dim 𝔞 = ℝ- rank 𝔤/𝔤^{τ} = ℝ- rank 𝔤/𝔤^{τ_{ϵ}} ≥ ℝ- rank 𝔤^{σ}/𝔤^{σ,τ_{ϵ}}.

Therefore, the pair (σ, τ_{ϵ}) satisfies (3.2.2). □

\underline{5.5.} It will be useful to pin down special cases of Lemma 5.4:

Lemma 5.5. 1) Let σ be an involutive automorphism of G commuting with a Cartan involution θ such that ℝ- rank 𝔤 = ℝ- rank 𝔤^{σ}. Then the pair (σ, θ_{ϵ}) satisfies (3.2.1) and (3.2.2) for any θ_{ϵ} in a K_{ϵ}-family.

2) Let μ be an involutive automorphism of G commuting with a Cartan involution θ. Then the pair (μθ, μ_{ϵ}) satisfies (3.2.1) and (3.2.2) for any μ_{ϵ}.

Proof. We apply Lemma 5.4 to the following settings:

1) Put τ := θ. As ℝ- rank 𝔤/𝔤^{θ} = ℝ- rank 𝔤 and ℝ- rank 𝔤^{σ}/𝔤^{σ,θ} = ℝ- rank 𝔤^{σ}, the rank condition ℝ- rank 𝔤/𝔤^{τ} = ℝ- rank 𝔤^{σ}/𝔤^{σ,τ} amounts to our assumption ℝ- rank 𝔤 = ℝ- rank 𝔤^{σ}. Thus, the pair (σ, θ) satisfies (3.2.1) and (3.2.2). It follows from Lemma 5.4 that (σ, θ_{ϵ}) also satisfies (3.2.1) and (3.2.2).

2) Put σ := μθ. It follows from μθ = θμ that σ, μ and θ commute with one another. As ℝ- rank 𝔤^{σ}/𝔤^{σ,μ} is the dimension of a maximal abelian subspace of

𝔤^{−θ,σ,−μ} = 𝔤^{−θ,μθ,−μ} = 𝔤^{−θ,−μ},

the rank condition ℝ- rank 𝔤/𝔤^{μ} = ℝ- rank 𝔤^{σ}/𝔤^{σ,μ} is automatically satisfied. Therefore, the pair (μθ, μ) satisfies (3.2.1) and (3.2.2). Hence, the pair (μθ, μ_{ϵ}) also satisfies (3.2.1) and (3.2.2) by Lemma 5.4. □

\underline{5.6.} The following lemma shows that Lemma 5.1 holds if (𝔤, 𝔤^{τ}) belongs to a K_{ϵ}-family.

Lemma 5.6. Lemma 5.1 holds for τ = θ_{ϵ} for any θ_{ϵ} corresponding to a K_{ϵ}-family.

Proof. We can take an involutive automorphism σ commuting with θ such that σZ = −Z and ℝ- rank 𝔤 = ℝ- rank 𝔤^{σ} by Lemma 4.1.1. Then, Lemma follows from Lemma 5.5 (1). □

In particular, Lemma 5.6 shows that Lemma 5.1 holds in the the following five cases (see a list of K_{ϵ}-families in §5.4):

𝔤 = 𝔢_{6(−14)}; 𝔤^{τ} = 𝔰𝔬(10) + 𝔰𝔬(2), 𝔰𝔬^{∗}(10) + 𝔰𝔬(2), 𝔰𝔬(8, 2) + 𝔰𝔬(2),
𝔤 = 𝔢_{7(−25)}; 𝔤^{τ} = 𝔢_{6(−78)} + 𝔰𝔬(2), 𝔢_{6(−14)} + 𝔰𝔬(2).

\underline{5.7.} In light of Table 5.3.1, we are now left with the five exceptional cases:

𝔤 = 𝔢_{6(−14)}; 𝔤^{τ} = 𝔰𝔲(5, 1) + 𝔰𝔩(2, ℝ), 𝔰𝔲(4, 2) + 𝔰𝔲(2),
𝔤 = 𝔢_{7(−25)}; 𝔤^{τ} = 𝔰𝔬(10, 2) + 𝔰𝔩(2, ℝ), 𝔰𝔬^{∗}(12) + 𝔰𝔲(2), 𝔰𝔲(6, 2),

to complete the proof of Lemma 5.1.

As we have mentioned before, it is not obvious that two involutions σ and τ of 𝔤 commute with each other by using a suitable conjugation. By using Lemma 5.5, we prove the commutativity of σ and τ in the following exceptional cases.

Lemma 5.7. 1) Suppose μ is an involutive automorphism of a real semisimple Lie algebra 𝔤 such that ℝ- rank 𝔤 = ℝ- rank 𝔤^{μ}. Then there exist a Cartan involution θ and a maximal abelian space 𝔞 of 𝔤^{−θ} = {Y ∈ 𝔤 : θY = −Y } such that μθ, θ_{ϵ}θ and θ commute with one another for any θ_{ϵ} defined by a signature ϵ of Σ(𝔤, 𝔞).

2) There exist mutually commuting involutive automorphisms σ and τ of 𝔤 := 𝔢_{7(−25)} such that 𝔤^{σ} ≃ 𝔰𝔲^{∗}(8) and 𝔤^{τ} ≃ 𝔰𝔬(10, 2) + 𝔰𝔩(2, ℝ).

3) There exist mutually commuting involutive automorphisms σ and τ of 𝔤 := 𝔢_{6(−14)} such that 𝔤^{σ} ≃ 𝔰𝔭(2, 2) and 𝔤^{τ} ≃ 𝔰𝔲(5, 1) + 𝔰𝔩(2, ℝ).

Proof. 1) We choose a Cartan involution θ which commutes with μ and take 𝔞 such that 𝔞 ⊂ 𝔤^{−θ,μ}. Then it follows from Lemma 5.5 (1) that μ, θ_{ϵ} and θ commute with one another.

2) If 𝔤 := 𝔢_{7(−25)}, then the K_{ϵ}-family of 𝔤 is given by

{(𝔢_{7(−25)}, 𝔢_{6(−78)} + 𝔰𝔬(2)), (𝔢_{7(−25)}, 𝔢_{6(−14)} + 𝔰𝔬(2)), (𝔢_{7(−25)}, 𝔢_{6(−26)} + 𝔰𝔬(1, 1))}.

We apply (1) to the case where 𝔤^{μ} ≃ 𝔰𝔲^{∗}(8). Then ℝ- rank 𝔤 = ℝ- rank 𝔤^{μ} = 3. We put σ := μθ and τ := θ_{ϵ}θ, where we choose θ_{ϵ} so that 𝔤^{θ_{ϵ}} ≃ 𝔢_{6(−14)} + 𝔰𝔬(2). Then we have 𝔤^{σ} = 𝔤^{μθ} ≃ 𝔰𝔲^{∗}(8) and 𝔤^{τ} = 𝔤^{θ_{ϵ}θ} ≃ 𝔰𝔬(10, 2) + 𝔰𝔩(2, ℝ) (see [42], Table I). Hence, (2) is shown.

3) If 𝔤 := 𝔢_{6(−14)}, then the K_{ϵ}-family of 𝔤 is given by

{(𝔢_{6(−14)}, 𝔰𝔬(10) + 𝔰𝔬(2)), (𝔢_{6(−14)}, 𝔰𝔬^{∗}(10) + 𝔰𝔬(2)), (𝔢_{6(−14)}, 𝔰𝔬(8, 2) + 𝔰𝔬(2))}.

We apply (1) to the case where 𝔤^{μ} ≃ 𝔰𝔭(2, 2). Then ℝ- rank 𝔤 = ℝ- rank 𝔤^{μ} = 2. We put σ := μθ and τ := θ_{ϵ}θ, where we choose θ_{ϵ} so that 𝔤^{θ_{ϵ}} ≃ 𝔰𝔬^{∗}(10) + 𝔰𝔬(2). Then we have 𝔤^{σ} = 𝔤^{μθ} ≃ 𝔰𝔭(2, 2) and 𝔤^{τ} = 𝔤^{θ_{ϵ}θ} ≃ 𝔰𝔲(5, 1) + 𝔰𝔩(2, ℝ) (see [42], Table I). Hence, (3) is shown. □

\underline{5.8.} Let us find 𝔤^{σ,τ} and ℝ- rank 𝔤^{σ}/𝔤^{σ,τ} for the pairs (σ, τ) that are given in Lemma 5.7 (2) and (3).

Lemma 5.8. 1) In the setting of Lemma 5.7 (2), 𝔤^{σ,τ} ≃ 𝔰𝔲^{∗}(4) + 𝔰𝔲^{∗}(4) + ℝ ≃ 𝔰𝔬(5, 1) + 𝔰𝔬(5, 1) + ℝ.

2) In the setting of Lemma 5.7 (3), 𝔤^{σ,τ} ≃ 𝔰𝔲^{∗}(4) + ℝ ≃ 𝔰𝔬(5, 1) + ℝ.

3) In each case, ℝ- rank 𝔤^{σ}/𝔤^{σ,τ} = ℝ- rank 𝔤/𝔤^{τ} (see (3.2.2)).

Proof. We denote by c- rank 𝔤 := rank 𝔨 for a real reductive Lie algebra 𝔤 with a Cartan involution θ.

1) As 𝔤^{σ,τ} = 𝔤^{σ} ∩ 𝔤^{τ} ⊂ 𝔤^{σ}, c- rank 𝔤^{σ,τ} ≤ c- rank 𝔤^{σ} = c- rank 𝔰𝔲^{∗}(8) = 4. On the other hand, (𝔤^{τ} , 𝔤^{σ,τ}) = (𝔰𝔬(10, 2) + 𝔰𝔩(2, ℝ), 𝔤^{σ,τ}) is a symmetric pair, and so the subalgebra 𝔤^{σ,τ} with c- rank 𝔤^{σ,τ} ≤ 4 must be isomorphic to one of

𝔰𝔬(9, 1) + 𝔰𝔬(1, 1) + ℝ, 𝔰𝔬(7, 1) + 𝔰𝔬(3, 1) + ℝ, or 𝔰𝔬(5, 1) + 𝔰𝔬(5, 1) + ℝ

by classification [3]. As (𝔤^{σ}, 𝔤^{σ,τ}) = (𝔰𝔲^{∗}(8), 𝔤^{σ,τ}) is also a symmetric pair, we conclude that 𝔤^{σ,τ} is isomorphic to 𝔰𝔬(5, 1) + 𝔰𝔬(5, 1) + ℝ ≃ 𝔰𝔲^{∗}(4) + 𝔰𝔲^{∗}(4) + ℝ because two others cannot be subalgebras of 𝔰𝔲^{∗}(8). Then

ℝ- rank 𝔤^{σ}/𝔤^{σ,τ} = ℝ- rank 𝔤^{σ,τθ} = ℝ- rank 𝔰𝔭(2, 2)  = 2,
ℝ- rank 𝔤/𝔤^{τ} = ℝ- rank 𝔤^{τθ} = ℝ- rank(𝔢_{6(−14)} + 𝔰𝔬(2)) = 2.

Hence, ℝ- rank 𝔤^{σ}/𝔤^{σ,τ} = ℝ- rank 𝔤/𝔤^{τ}.

2) As 𝔤^{σ,τ} = 𝔤^{σ} ∩ 𝔤^{τ} ⊂ 𝔤^{σ}, c- rank 𝔤^{σ,τ} ≤ c- rank 𝔤^{σ} = c- rank 𝔰𝔭(2, 2) = 4. On the other hand, as (𝔤^{τ} , 𝔤^{σ,τ}) = (𝔰𝔲(5, 1) + 𝔰𝔩(2, ℝ), 𝔤^{σ,τ}) is a symmetric pair, the subalgebra 𝔤^{σ,τ} with c- rank 𝔤^{σ,τ} ≤ 4 must be isomorphic to 𝔰𝔲^{∗}(4)+ℝ ≃ 𝔰𝔬(5, 1)+ℝ by classification. Then

ℝ- rank 𝔤^{σ}/𝔤^{σ,τ} = ℝ- rank 𝔤^{σ,τθ} = ℝ- rank 𝔰𝔭(2, ℂ)  = 2,
ℝ- rank 𝔤/𝔤^{τ} = ℝ- rank 𝔤^{τθ} = ℝ- rank(𝔰𝔬^{∗}(10) + 𝔰𝔬(2)) = 2.

Hence, ℝ- rank 𝔤^{σ}/𝔤^{σ,τ} = ℝ- rank 𝔤/𝔤^{τ}. □

\underline{5.9.} Proof of Lemma 5.1. As we saw at the beginning of §5.7, we are left with five exceptional cases to complete the proof of Lemma 5.1.

First, we recall that the following symmetric pairs form H_{ϵ}-families (see [42], Table V):

{(𝔢_{6(−14)}, 𝔰𝔲(5, 1) + 𝔰𝔩(2, ℝ)), (𝔢_{6(−14)}, 𝔰𝔲(4, 2) + 𝔰𝔲(2))}, (5.9.1) {(𝔢_{7(−25)}, 𝔰𝔬(10, 2) + 𝔰𝔩(2, ℝ)), (𝔢_{7(−25)}, 𝔰𝔬^{∗}(12) + 𝔰𝔲(2))}, (5.9.2) {(𝔢_{7(−25)}, 𝔰𝔲(6, 2)), (𝔢_{7(−25)}, 𝔰𝔲^{∗}(8))}. (5.9.3)

Then, Lemma 5.1 holds for

(𝔤, 𝔤^{τ}) = (𝔢_{7(−25)}, 𝔰𝔬(10, 2) + 𝔰𝔩(2, ℝ)),

by taking σ such that 𝔤^{σ} ≃ 𝔰𝔲^{∗}(8). In fact, (3.2.1) follows from Lemma 5.7 (2), (3.2.2) from Lemma 5.8 (3), and (3.6.1) from Table 3.3.2.

By applying Lemma 5.4 to the above triple (𝔤, σ, τ), the triple (𝔤, σ, τ_{ϵ}) also satisfies (3.2.1), (3.2.2) and (3.6.1). In particular, if we take τ_{ϵ} such as 𝔤^{τ_{ϵ}} ≃ 𝔰𝔬^{∗}(12) + 𝔰𝔲(2) (see (5.9.2)), then we conclude (changing notation from τ_{ϵ} to τ) that Lemma 5.1 also holds for

(𝔤, 𝔤^{τ}) = (𝔢_{7(−25)}, 𝔰𝔬^{∗}(12) + 𝔰𝔲(2)).

Likewise, it follows from Lemmas 5.7 (3) and 5.8 (3) that Lemma 5.1 holds for

(𝔤, 𝔤^{τ}) = (𝔢_{6(−14)}, 𝔰𝔲(5, 1) + 𝔰𝔩(2, ℝ)).

By (5.9.1) and Lemma 5.4, Lemma 5.1 also holds for another H_{ϵ}-family:

(𝔤, 𝔤^{τ}) = (𝔢_{6(−14)}, 𝔰𝔲(4, 2) + 𝔰𝔲(2)).

Finally, let (𝔤, 𝔤^{μ}) = (𝔢_{7(−25)}, 𝔰𝔲^{∗}(8)). We fix a Cartan involution θ of 𝔤 commuting with μ as usual. Then (μθ, μ_{ϵ}) satisfies (3.2.1) and (3.2.2) from Lemma 5.5 (2). We take μ_{ϵ} so that 𝔤^{μ_{ϵ}} ≃ 𝔰𝔲(6, 2) (see (5.9.3)). We put σ := μθ and τ := μ_{ϵ}. Then, σZ = −Z because μZ = −Z (see Table 3.3.2). Therefore, Lemma 5.1 holds for

(𝔤, 𝔤^{τ}) = (𝔢_{7(−25)}, 𝔰𝔲(6, 2)).

In light of the list at the beginning of §5.7, we have proved Lemma 5.1 for all the exceptional cases. Hence, we have finished the proof of Lemma 5.1 and Theorems A and B. □

§6. Counter examples

\underline{6.1.} In this section, we illustrate by counter examples how the conclusions fail if we relax each of our assumptions in Theorems A or B. Let (G, H) be a reductive symmetric pair corresponding to an involutive automorphism τ. We shall see

i) The multiplicity of the restriction π|_{H} can be greater than one if a highest weight module π is not of scalar type (cf. Theorem B (2)). ii) The multiplicity of the restriction π|_{H} is not always uniformly bounded if π is not a highest weight module (cf. Theorem B (1)).

Although our concern in this paper is mainly with a non-compact subgroup H, there already exist counter examples in the case where G′ is compact (i.e. τ = θ). We shall give examples of (i) in §6.2, and (ii) in §6.3.

\underline{6.2.} Let G = Sp(2, ℝ). We take a compact Cartan subalgebra 𝔱. We take a basis \{f_{1}, f_{2}\} of \sqrt{−1}𝔱^{∗} such that Δ(𝔤, 𝔱) = \{±f_{1} ± f_{2}, ±2f_{1}, ±2f_{2}\} and fix a positive system Δ^{+}(𝔨, 𝔱) := {f_{1} − f_{2}}. In what follows, we shall use the notation (λ_{1}, λ_{2}) to denote the character λ_{1}f_{1} + λ_{2}f_{2} of 𝔱. Given (p, q) ∈ ℤ^{2} with p ≥ q, we denote by F (U(2), (p, q)) the irreducible representation of U(2) with highest weight (p, q) = pf_{1} + qf_{2}. Then dim F (U(2), (p, q)) = p − q + 1.

The set of holomorphic discrete series representations of G is parametrized by λ := (λ_{1}, λ_{2}) ∈ ℕ^{2} with λ_{1} > λ_{2} > 0. We denote by V_{λ} ∈ Ĝ the holomorphic discrete series representation of G characterized by

Z(𝔤)-infinitesimal character = λ_{1}f_{1} + λ_{2}f_{2} (Harish-Chandra parameter),
minimal K-type = F (U(2), (λ_{1} + 1, λ_{2} + 2)) (Blattner parameter).

We note that V_{λ} is of scalar type if and only if λ_{1} = λ_{2} + 1.

Example 6.2 (multiplicity of K-type of holomorphic discrete series V_{λ}).

(6.2.1)  sup_{τ∈K̂} dim Hom_{K}(τ, V_{λ}|_{K}) = [\frac{λ_{1} − λ_{2} + 1}{2}] .

In particular, the multiplicity of each K-type τ occurring in V_{λ} is uniformly bounded with respect to τ ∈ K̂ for fixed λ = (λ_{1}, λ_{2}). The right side of (6.2.1) = 1 if and only if either of the following two cases:

(6.2.2)(a) λ_{1} = λ_{2} + 1 (i.e. V_{λ} is of scalar type),
(6.2.2)(b) λ_{1} = λ_{2} + 2 (i.e. V_{λ} is of two dimensional minimal K-type).

Thus, the branching law is not multiplicity free if λ_{1} − λ_{2} > 2.

Sketch of Proof. It follows from the Blattner formula for a holomorphic discrete series representation ([25],[46]) that the K-type formula of V_{λ} is given by

V_{λ}|_{K} ≃ F (U(2), (λ_{1} + 1, λ_{2} + 2)) ⊗ S(ℂ^{3})

= F (U(2), (λ_{1} + 1, λ_{2} + 2)) ⊗ ⨁_{\begin{matrix}a≥b≥0 \\ (a,b)∈ℕ^{2} \end{matrix}} F (U(2), (2a, 2b)),

where U(2) acts on ℂ^{3} ≃ S^{2}(ℂ^{2}) as the symmetric tensor of the natural representation. Then (6.2.1) is an elementary computation of the Clebsch-Gordan formula on the tensor product of irreducible representations of U(2) (Example 1.8 (2)). □

\underline{6.3.} We continue the setting of §6.2. There exist 4-closed orbits of K_{ℂ} ≃ GL(2, ℂ) on the full flag variety of G_{ℂ} = Sp(2, ℂ). Correspondingly, there are 4 series of discrete series representations of G. Among them, two are holomorphic and antiholomorphic discrete series representations, and 2 other series are not. We denote by W_{λ} ∈ Ĝ the discrete series representation of G with parameter λ := (λ_{1}, λ_{2}) ∈ ℕ^{2} (λ_{1} > λ_{2} > 0) characterized by

Z(𝔤)-infinitesimal character = λ_{1}f_{1} + λ_{2}f_{2} (Harish-Chandra parameter),
minimal K-type = F (U(2), (λ_{1} + 1, λ_{2}))  (Blattner parameter).

Then W_{λ} is a discrete series representation of G that is neither holomorphic nor anti-holomorphic. We define a θ-stable Borel subalgebra 𝔮 = 𝔱_{ℂ} + 𝔲 such that

Δ(𝔲 ∩ 𝔭_{ℂ}, 𝔱) := {2f_{1}, f_{1} + f_{2}, −2f_{2}}, Δ(𝔲 ∩ 𝔨_{ℂ}, 𝔱) := {f_{1} − f_{2}}.

Then, the Harish-Chandra module (W_{λ})_{K} is isomorphic to a cohomological parabolic induction R_{𝔮}^{1}(ℂ_{(λ_{1},λ_{2})}) with the notation in [51]. We set μ_{1} := λ_{1} + 1 and μ_{2} := λ_{2}.

Example 6.3 (multiplicity of K-type of non-holomorphic discrete series W_{λ}). We define the multiplicity of the K-type F (U(2), (p, q)) occurring in W_{λ} by

m_{λ}(p, q) := dim Hom_{K}(F (U(2), (p, q)), W_{λ}|_{K}).

Then, m_{λ}(p, q) ≠ 0 only if (p, q) ∈ ℤ^{2} satisfies

(6.3.1)  p ≥ μ_{1}, p − q ≥ μ_{1} − μ_{2} and p − q ∈ 2ℤ + μ_{1} + μ_{2}.

Then,

(6.3.2)  m_{λ}(p, q) = 1 + min([\frac{p − μ_{1}}{2} ] , \frac{p − q − μ_{1} + μ_{2}}{2} ).

In particular, for each fixed λ, the K-multiplicity in W_{λ} is not uniformly bounded, namely,

sup_{τ∈K̂} dim Hom_{K}(τ, W_{λ}|_{K}) = sup_{(p,q) satisfies (6.3.1)} m_{λ}(p, q) = ∞.

Sketch of Proof. The symmetric algebra S(𝔲 ∩ 𝔭_{ℂ}) is decomposed into irreducible representations of 𝔱 as

S(𝔲 ∩ 𝔭_{ℂ}) ≃ ⨁_{a,b,c∈ℕ} S^{a}(F (𝔱, 2f_{1})) ⊗ S^{b}(F (𝔱, f_{1} + f_{2})) ⊗ S^{c}(F (𝔱, −2f_{2}))

≃ ⨁_{a,b,c∈ℕ} F (𝔱, (2a + b, b − 2c)).

We denote by H^{j}(𝔲∩𝔨_{ℂ}, π) the j-th cohomology group of the Lie algebra 𝔲∩𝔨_{ℂ} with coefficients in a 𝔲∩𝔨_{ℂ}-module π. If π is a representation of K, then H^{j}(𝔲∩𝔨_{ℂ}, π) is a representation space of 𝔱. In particular, the Kostant’s version of the Borel-Weil-Bott theorem (e.g. [50], Chapter 3) says that

H^{j}(𝔲 ∩ 𝔨_{ℂ}, F (U(2), (p, q))) = \{ \begin{matrix}F (𝔱, (p, q)) & (j = 0), \\ F (𝔱, (q − 1, p + 1)) & (j = 1), \\ \{0\} & (j ≠ 0, 1). \end{matrix}

By using the Blattner formula due to Hecht-Schmid (e.g. [50], Theorem 6.3.12), the K-type formula of W_{λ} is given by

m_{λ}(p, q) = dim Hom_{K}(F (U(2), (p, q)), W_{λ}|_{K})
= ∑_{j=0}^{1}(−1)^{j} dim Hom_{𝔱}(H^{j}(𝔲 ∩ 𝔨_{ℂ}, F (U(2), (p, q)), S(𝔲 ∩ 𝔭_{ℂ}) ⊗ ℂ_{(μ_{1},μ_{2})})
=♯{(a, b, c) ∈ ℕ^{3} : p = 2a + b + μ_{1}, q = b − 2c + μ_{2}}
− ♯{(a, b, c) ∈ ℕ^{3} : q − 1 = 2a + b + μ_{1}, p + 1 = b − 2c + μ_{2}}
=♯{(a, b, c) ∈ ℕ^{3} : p = 2a + b + μ_{1}, q = b − 2c + μ_{2}}
=1 + min([\frac{p − μ_{1}}{2} ] , \frac{p − q − μ_{1} + μ_{2}}{2} ).

The other statement is an easy consequence of (6.3.2). □

§7. Finite dimensional cases

\underline{7.1.} The method in previous sections also applies to finite dimensional representations, leading us to a multiplicity-free theorem on branching laws of compact Lie groups. Though there exists an algorithm for branching problems of finite dimensional representations based on the Weyl character formula (e.g. [32], [33]), we should remark that actual computations of branching laws are often difficult combinatorial problems. So, we need another idea to prove multiplicity-free property. Some of known multiplicity-free branching laws of compact groups are explained in the context of dual pairs (e.g. [19]), by the complex geometry (e.g. [49]), and by combinatorial methods (e.g. [38], see Remark 7.5.1).

Our method based on Theorem 2.2 gives a different approach of multiplicity-free restrictions of finite dimensional representations. It is illustrated by the following correspondence (see Lemma 7.6):

a non-compact simple group G ↔ a compact simple group G_{U}
a unitary highest weight module ↔ a finite dimensional module
scalar type  ↔ “rectangular-shaped” highest weight [38]
Theorem A, Theorem C  ↔ Theorem 7.3, Theorem 7.4.

\underline{7.2.} Let 𝔤_{ℂ} be a complex simple Lie algebra. We take a Cartan subalgebra 𝔧 of 𝔤_{ℂ}, and fix a positive system Δ^{+}(𝔤_{ℂ}, 𝔧). We denote by {α_{1}, …, α_{n}} the set of simple roots, and {ω_{1}, …, ω_{n}} the set of the fundamental dominant weights. That is, if we define an inner product (·, ·) on the ℝ-span of {α_{1}, …, α_{n}} = the ℝ-span of {ω_{1}, …, ω_{n}}, which is invariant under the Weyl group W (𝔤_{ℂ}, 𝔧), then

(7.2.1)  \frac{2(ω_{i}, α_{j})}{(α_{j}, α_{j})} = δ_{ij}.

We label the simple roots in the Dynkin diagram as follows:

∘_{1}−∘_{2}− ⋯ −∘_{n} (A_{n})

∘_{1}−∘_{2}− ⋯ −∘= >∘_{n} (B_{n})

∘_{1}−∘_{2}− ⋯ −∘< =∘_{n} (C_{n})

∘_{1}−∘_{2}− ⋯ − ∘^{n−1} ; | ; ∘_{n−2} −∘_{n} (D_{n})

∘_{1}−∘_{2}−∘^{6} ; | ; ∘_{3}−∘_{4}−∘_{5} (E_{6})

∘_{1}−∘_{2}−∘_{3}−∘^{7} ; | ; ∘_{4}−∘_{5}−∘_{6} (E_{7})

We denote by F (𝔤_{ℂ}, ∑_{i} m_{i}ω_{i}) the irreducible finite dimensional representation of 𝔤_{ℂ} with highest weight ∑_{i=1}^{n} m_{i}ω_{i} for m_{1}, …, m_{n} ∈ ℕ. We shall also regard F (𝔤_{ℂ}, ∑_{i} m_{i}ω_{i}) as a holomorphic representation of G_{ℂ}, a simply connected complex Lie group with Lie algebra 𝔤_{ℂ}.

\underline{7.3.} Here is a multiplicity-free theorem of the irreducible decomposition of the tensor product of finite dimensional representations.

Theorem 7.3. Let 𝔤_{ℂ} be a complex simple Lie algebra of type A_{n}, B_{n}, C_{n}, D_{n}, E_{6} or E_{7}. Then, the tensor product F (𝔤_{ℂ}, kω_{i}) ⊗ F (𝔤_{ℂ}, lω_{i}) is decomposed into irreducible representations of 𝔤_{ℂ} with multiplicity free, for any k, l ∈ ℕ if (𝔤_{ℂ}, i) is in the following list:

(7.3.1)  i = 1, 2, …, n for A_{n};
(7.3.2)  i = 1  for B_{n};
(7.3.3)  i = n  for C_{n};
(7.3.4)  i = 1, n − 1, n for D_{n};
(7.3.5)  i = 1, 5  for E_{6};
(7.3.6)  i = 1  for E_{7}.

\underline{7.4.} Here is a multiplicity-free theorem of the branching law of a finite dimensional representation.

Theorem 7.4. The restriction F (𝔤_{ℂ}, kω_{i})|_{𝔥_{ℂ}} has a multiplicity-free decomposition into irreducibles of 𝔥_{ℂ} for any k ∈ ℕ, if (𝔤_{ℂ}, 𝔥_{ℂ}) and i are in Table 7.4.1. Here, we put K_{m} := {2, 4, …, 2m − 2} ∪ {m}.

multiplicity-free restriction F (𝔤_{ℂ}, kw_{i})|_{𝔥_{ℂ}} (k ∈ ℕ)
𝔤_{ℂ} 𝔥_{ℂ} i
1) 𝔰𝔩(n + 1, ℂ) 𝔰𝔩(k, ℂ) + 𝔰𝔩(n + 1 − k, ℂ) + ℂ (0 ≤ k ≤ n + 1) 1, 2, …, n
2) 𝔰𝔩(n + 1, ℂ)  𝔰𝔬(n + 1, ℂ)  1, 2, …, n
3) 𝔰𝔩(2m, ℂ)  𝔰𝔭(m, ℂ)  K_{m}
4) 𝔰𝔬(2n + 1, ℂ) 𝔰𝔬(k, ℂ) + 𝔰𝔬(2n + 1 − k, ℂ) (0 ≤ k ≤ 2n + 1)  1
5) 𝔰𝔭(n, ℂ)  𝔰𝔭(k, ℂ) + 𝔰𝔭(n − k, ℂ) (0 ≤ k ≤ n)  n
6) 𝔰𝔭(n, ℂ)  𝔤𝔩(n, ℂ)  n
7) 𝔰𝔬(2n, ℂ)  𝔰𝔬(k, ℂ) + 𝔰𝔬(2n − k, ℂ) (k ∈ K_{n})  n − 1, n
8) 𝔰𝔬(2n, ℂ)  𝔰𝔬(k, ℂ) + 𝔰𝔬(2n − k, ℂ) (0 ≤ k ≤ 2n)  1
9) 𝔰𝔬(2n, ℂ)  𝔤𝔩(n, ℂ)  1, n − 1, n
10)  𝔢_{6}  𝔰𝔬(10, ℂ) + 𝔰𝔬(2, ℂ)  1, 5
11)  𝔢_{6}  𝔰𝔩(6, ℂ) + 𝔰𝔩(2, ℂ)  1, 5
12)  𝔢_{6}  𝔣_{4}  1, 5
13)  𝔢_{6}  𝔰𝔭(4, ℂ)  1, 5
14)  𝔢_{7}  𝔢_{6} + 𝔰𝔬(2, ℂ)  1
15)  𝔢_{7}  𝔰𝔬(12, ℂ) + 𝔰𝔩(2, ℂ)  1
16)  𝔢_{7}  𝔰𝔩(8, ℂ)  1

Table 7.4.1.

\underline{7.5.} Example 1.8 (2) is the simplest example of Theorem 7.3 (i = n = 1 and 𝔤_{ℂ} = A_{1} = 𝔰𝔩(2, ℂ)) and Theorem 7.4 (k = i = n = 1). Apart from infinite dimensional representations, a number of explicit branching laws for Theorem 7.3 and Theorem 7.4 were known, summarized as follows.

Remark 7.5.1. Some of the multiplicity-free results could be proved by using so called the Littlewood-Richardson rule and the algorithm of K. Koike and I. Terada in [33], although the required computations may involve many cancellations. Recently, S. Okada obtained explicit branching laws for some cases in Theorems 7.3 and 7.4 by combinatorial methods. For example, explicit decomposition formulae of the tensor product for (7.3.2), (7.3.3), and (7.3.4) in Theorem 7.3 were proved in [38], Theorem 2.5 (1), (2), and (3), respectively. Explicit branching laws for special cases with i = k in Theorem 7.4 (1); with i = [\frac{n+1}{2}] in Theorem 7.4 (2); with i = m in Theorem 7.4 (3); and with i = n − 1, n in Theorem 7.4 (9) are obtained in [38], Theorem 2.1; Theorem 2.6 (1), (2); and Theorem 2.3 (3), respectively (cf. [32], Proposition 1.5.3; [43]). Also, those for Theorem 7.4 (5) and (7) are given in [38], Theorem 2.2 (2) and (3), respectively. J. Stembridge, [48], Theorem 4.1 obtained a branching law for Theorem 7.4 (6) (see also [38], Theorem 2.3 (2); cf. I. Macdonald, [36], Ex.I.5.16).

However, combinatorial methods for branching laws do not reveal clearly the reason why the multiplicity is one. Theorems 7.3 and 7.4 give a unified explanation of the above mentioned multiplicity-free property by complex geometry (see Theorem 2.2), and also present some other branching problems of both classical and exceptional groups for which there are, to our knowledge, no explicit branching formulae in the literature. For example, Theorem 7.4 (1), (2) and (3) for generic parameter (n, k, i); Theorem 7.3 (5), (6); and Theorem 7.4 (10)–(16) are the cases.

\underline{7.6.} The idea of the proof of Theorems 7.3 and 7.4 is based on the Borel-Weil construction of irreducible finite dimensional representations by using a maximal parabolic subalgebra.

Suppose we are in the setting of §7.2. We fix a simple root α_{i}, and define ad(𝔧)- stable subalgebras 𝔩_{i}_{ℂ} and 𝔫_{i}_{ℂ}^{−} of 𝔤_{ℂ} by

Δ(𝔩_{i}_{ℂ}, 𝔧) = ℤ-span of {α_{1}, …, α_{i}^{∧}, …, α_{n}} ∩ Δ(𝔤_{ℂ}, 𝔧),
Δ(𝔫_{i}_{ℂ}^{−}, 𝔧) = Δ^{−}(𝔤_{ℂ}, 𝔧) \ Δ(𝔩_{i}_{ℂ}, 𝔧).

Then, 𝔩_{i}_{ℂ} + 𝔫_{i}_{ℂ}^{−} is a maximal parabolic subalgebra of 𝔤_{ℂ}. Let G_{ℂ} be a simply connected complex Lie group with Lie algebra 𝔤_{ℂ}. We write P_{i}_{ℂ}^{−} = L_{i}_{ℂ}N_{i}_{ℂ}^{−} for the corresponding maximal parabolic subgroup of G_{ℂ}.

The fundamental weight ω_{i} defines a character of the Lie algebra 𝔩_{iℂ} because ω_{i} is orthogonal to the ℤ-span of {α_{1}, …, α_{i}^{∧}, …, α_{n}} by (7.2.1). Then, kω_{i} lifts to a character of L_{i}_{ℂ} if and only if k ∈ ℤ because G_{ℂ} is simply connected. We extend it trivially to N_{i}_{ℂ}^{−}, and write ℂ_{kω_{i}} for the resulting character of P_{i}_{ℂ}^{−}. Let

L_{kω_{i}} := G_{ℂ} ×_{P_{i}}_{ℂ}^{−} ℂ_{kω_{i}} → G_{ℂ}/P_{i}_{ℂ}^{−}

be the holomorphic line bundle associated to the character ℂ_{kω_{i}}. The space of holomorphic sections O (L_{kω_{i}}) is naturally a holomorphic representation space of G_{ℂ}. Then, the Borel-Weil construction asserts (cf. Lemma 3.5):

Lemma 7.6. We have an isomorphism of representations of G_{ℂ} (also of 𝔤_{ℂ}):

F (𝔤_{ℂ}, kω_{i}) ≃ O (L_{kω_{i}}) .

Representations F (𝔤_{ℂ}, kω_{i}) are called “rectangular-shaped representations” in the terminology of [38] for classical cases.

\underline{7.7.} Retain the setting of §7.6. We observe that (𝔤_{ℂ}, 𝔩_{iℂ}) is a complex semisimple symmetric pair if and only if i satisfies the assumption in Theorem 7.3. Then, there exists a real form 𝔤 of 𝔤_{ℂ} such that the analytic subgroup G with Lie algebra 𝔤 has a maximal compact subgroup K = G ∩ L_{iℂ}. By the classification of simple real Lie algebras, the correspondence between 𝔤 and (𝔤_{ℂ}, i) is:

𝔤 = 𝔰𝔲(i, n + 1 − i) if 𝔤_{ℂ} is of type A_{n} (i = 1, 2, …, n);
𝔤 = 𝔰𝔬(2, 2n − 1)  if 𝔤_{ℂ} is of type B_{n} (i = 1);
𝔤 = 𝔰𝔭(n, ℝ)  if 𝔤_{ℂ} is of type C_{n} (i = n);
𝔤 = 𝔰𝔬(2, 2n − 2)  if 𝔤_{ℂ} is of type D_{n} (i = 1);
𝔤 = 𝔰𝔬^{∗}(2n)  if 𝔤_{ℂ} is of type D_{n} (i = n − 1 or n);
𝔤 = 𝔢_{6(−14)}  if 𝔤_{ℂ} is of type E_{6} (i = 1 or 5);
𝔤 = 𝔢_{7(−25)}  if 𝔤_{ℂ} is of type E_{7} (i = 1).

Table 7.7.1

We note that G/K is an irreducible Hermitian symmetric space of the noncompact type. Let θ be the Cartan involution of G corresponding to K, and 𝔤 = 𝔨 + 𝔭 a Cartan decomposition.

Let G_{U} be a connected subgroup of G_{ℂ} with Lie algebra 𝔨 + \sqrt{−1}𝔭. Then G_{U} is a maximal compact subgroup of G_{ℂ}, and G_{U}/K is a Hermitian symmetric space of the compact type.

Suppose τ is an involutive automorphism of the Lie algebra 𝔤 commuting with θ. The complex linear extention of τ lifts to a holomorphic automorphism of G_{ℂ} because G_{ℂ} is simply connected. Then τ stabilizes both G and G_{U}. We shall use the same notation τ for the complex linear extension, for the lift to G_{ℂ}, and for the restrictions to G and G_{U}. Analogous notation will be used for other involutions θ and σ. Then, a compact version of Lemma 3.2 is formulated in the following:

Lemma 7.7. Assume (3.2.1) and (3.2.2) are satisfied. Then, for any x ∈ G_{U}/K, there exists g ∈ (G_{U})_{0}^{τ} such that σ(x) = g · x.

Proof. We note that the complexified linear maps θ, τ and σ ∈ End(𝔤_{ℂ}) stabilize 𝔤_{U}. Let \sqrt{−1}𝔞 be a maximal abelian subspace of

(𝔤_{U})^{−θ,σ,−τ} = \{Y ∈ 𝔤_{U} : (−θ)Y = σY = (−τ)Y = Y \} = \sqrt{−1}𝔤^{−θ,σ,−τ} .

We note that 𝔞 is a maximal abelian subspace of 𝔤^{−θ,σ,−τ} and also of 𝔤^{−θ,−τ} by the rank condition (3.2.2). Accordingly, \sqrt{−1}𝔞 is also a maximal abelian subspace of (𝔤_{U})^{−θ,−τ}. Then there is a generalized Cartan decomposition (see [17], Theorem 6.10)

(7.7.2)  G_{U} = (G_{U})_{0}^{τ} T K,

where T is a connected compact subgroup of G_{U} with Lie algebra \sqrt{−1}𝔞. Now, by using (7.7.2) instead of (3.2.3), the proof parallels exactly to that of Lemma 3.2 with 𝔤 replaced by 𝔤_{U}. □

\underline{7.8.} Proof of Theorem 7.3. The proof parallels to that of Theorem C, and so we shall give only a necessary modification.

We realize the representation (π_{1}, F (𝔤_{ℂ}, kω_{i})) of G_{ℂ} on O(L_{kω_{i}}), the space of global holomorphic sections of the line bundle L_{kω_{i}} → G_{ℂ}/P_{iℂ}^{−} by Lemma 7.6. Likewise, for the representation (π_{2}, F (𝔤_{ℂ}, lω_{i})) of G_{ℂ} on O(L_{lω_{i}}).

By the assumption on i, there exists a real form G of G_{ℂ} such that K = G ∩ L_{iℂ} is a maximal compact subgroup of G. We form a maximal compact subgroup G_{U} of G_{ℂ} (see §7.7). Then, both G_{U}/K and G/K are Hermitian symmetric spaces with complex structures induced by G_{U}/K ⟶^{∼} G_{ℂ}/P_{i}_{ℂ}^{−} ↩ G/K. It follows from Lemma 4.1.1 that we can take an involutive automorphism σ of 𝔤 satisfying (3.7.1), (3.7.2) and (3.7.3). We extend σ holomorphically to a simply connected group G_{ℂ} and restrict it to G_{U}. Then the induced involutive diffeomorphism

G_{U}/K × G_{U}/K → G_{U}/K × G_{U}/K, (g_{1}K, g_{2}K) ↦ (σ(g_{1})K, σ(g_{2})K)

is anti-holomorphic by (3.7.3). We define a holomorphic involutive automorphism τ on G_{ℂ} × G_{ℂ} by τ(g_{1}, g_{2}) := (g_{2}, g_{1}). Then (G_{U} × G_{U})^{τ} = diag(G_{U}). Therefore, for any (x_{1}, x_{2}) ∈ G_{U}/K × G_{U}/K, there exists (g, g) ∈ (G_{U} × G_{U})^{τ} such that (g · x_{1}, g · x_{2}) = (σ(x_{1}), σ(x_{2})) by Lemma 7.7 applied to G_{U} × G_{U}.

On the other hand, by using the biholomorphic map G_{U}/K ⟶^{∼} G_{ℂ}/P_{i}_{ℂ}^{−}, we write the holomorphic line bundle L_{kω_{i}} as a G_{U}-homogeneous line bundle over G_{U}/K:

L_{kω_{i}} ≃ G_{U} ×_{K} ℂ_{kω_{i}} → G_{U}/K.

Thus, we have an isomorphism of holomorphic line bundles \overline{σ^{∗}L_{kω_{i}}} ≃ L_{kω_{i}} (see Lemma 8.6). Likewise, \overline{σ^{∗}L_{lω_{i}}} ≃ L_{lω_{i}}.

Applying Theorem 2.2, we conclude that the commutant of {(π_{1}(g), π_{2}(g)) : g ∈ G_{U}} in End (O(L_{kω_{i}}) ⊗ O(L_{lω_{i}})) is commutative. Hence, the tensor product π_{1}⊗π_{2} is multiplicity free as G_{U}-modules. By Weyl’s unitary trick, the multiplicityfree result is true also as representations of 𝔤_{ℂ}. □

\underline{7.9.} Proof of Theorem 7.4.

First, we explain how to obtain Table 7.4.1. Suppose that G is a non-compact connected simple Lie group of Hermitian type and retain the notation in §1.4. We assume that G is contained in a simply connected complex simple Lie group G_{ℂ}. Let τ be an involutive automorphism of G commuting with θ. Then we shall associate a triple (𝔤_{ℂ}, 𝔥_{ℂ}, i) to the data (𝔤, θ, τ) with the following two properties:

i) 𝔨_{ℂ} ≃ Z_{𝔤_{ℂ}}(ω_{i}) = 𝔩_{iℂ}, the Levi part of a maximal parabolic subalgebra of 𝔤_{ℂ}, ii) 𝔥_{ℂ} := 𝔤_{ℂ}^{τ}, where τ stands for a complex linear extension of τ.

We note that the outer automorphisms of 𝔤_{ℂ} for type A_{n}, D_{n} and E_{6} allow different choices of i, namely, i = p or n + 1 − p for 𝔤 = 𝔰𝔲(p, n + 1 − p), i = n − 1 or n for 𝔤 = 𝔰𝔬^{∗}(2n), and i = 1 or 5 for 𝔤 = 𝔢_{6(−14)} (see Table 7.7.1).

Here is a table of (𝔤, 𝔤^{τ}), from which the triple (𝔤_{ℂ}, 𝔥_{ℂ}, i) in Table 7.7.1 (1)∼(16) is obtained (we note that such (𝔤, 𝔤^{τ}) is not unique).

How to obtain Table 7.4.1 from Table 3.3.1 and Table 3.3.2

𝔤  𝔥 = 𝔤^{τ}
1) {𝔰𝔲(i, n + 1 − i) \atop 𝔰𝔲(i, n + 1 − i)} {𝔰(𝔲(i, k − i) + 𝔲(n + 1 − k)) \atop 𝔰(𝔲(k) + 𝔲(i − k, n + 1 − i))} {(k ≥ i) \atop (i ≥ k)}
2) 𝔰𝔲(i, n + 1 − i)  𝔰𝔬(i, n + 1 − i)
3) {𝔰𝔲(m, m) \atop 𝔰𝔲(2i, 2m − 2i)}  {𝔰𝔭(m, ℝ) \atop 𝔰𝔭(i, m − i)}
4) 𝔰𝔬(2, 2n − 1)  𝔰𝔬(1, k − 1) + 𝔰𝔬(1, 2n − k)
5)  𝔰𝔭(n, ℝ)  𝔰𝔭(k, ℝ) + 𝔰𝔭(n − k, ℝ)
6)  𝔰𝔭(n, ℝ)  𝔲(n)
7)  {𝔰𝔬^{∗}(2n) \atop 𝔰𝔬^{∗}(2n)}  {𝔰𝔬^{∗}(2k) + 𝔰𝔬^{∗}(2n − 2k) \atop 𝔰𝔬(n, ℂ)}
8) 𝔰𝔬(2, 2n − 2)  𝔰𝔬(1, k − 1) + 𝔰𝔬(1, 2n − k − 1)
9) {𝔰𝔬^{∗}(2n) \atop 𝔰𝔬(2, 2n − 2)}  {𝔲(n) \atop 𝔲(1, n − 1)}
10)  𝔢_{6(−14)}  𝔰𝔬(10) + 𝔰𝔬(2)
11)  𝔢_{6(−14)}  𝔰𝔲(5, 1) + 𝔰𝔩(2, ℝ)
12)  𝔢_{6(−14)}  𝔣_{4(−20)}
13)  𝔢_{6(−14)}  𝔰𝔭(2, 2)
14)  𝔢_{7(−25)}  𝔢_{6(2)} + 𝔰𝔬(2)
15)  𝔢_{7(−25)}  𝔰𝔬(10, 2) + 𝔰𝔩(2, ℝ)
16)  𝔢_{7(−25)}  𝔰𝔲(6, 2)

Table 7.9.1

Now, let us complete the proof of Theorem 7.4. First, we take a real form 𝔤 of 𝔤_{ℂ} and an involutive automorphism τ of 𝔤 corresponding to a given triple (𝔤_{ℂ}, 𝔥_{ℂ}, i) as in Table 7.9.1. Next, we take an involutive automorphism σ of 𝔤 as in Lemma 5.1 so that τ and σ satisfy (3.2.1), (3.2.2) and (3.6.1). Then, it follows from (3.6.1) that the induced diffeomorphism σ acts anti-holomorphically on G_{U}/K. For any x ∈ G_{U}/K, there exists g ∈ (G_{U})_{0}^{τ} such that g ·x = σ(x) by Lemma 7.7. We realize the representation (π, F (𝔤_{ℂ}, kω_{i})) of G_{ℂ} on the representation space O(L_{kω_{i}}) by Lemma 7.6. Then, we have an isomorphism of holomorphic line bundles \overline{σ^{∗}L_{kω_{i}}} ≃ L_{kω_{i}} (see Lemma 8.6). Hence, the commutant of π((G_{U})_{0}^{τ}) in End(F (𝔤_{ℂ}, kω_{i})) is commutative by Theorem 2.2. Thus, the restriction of F (𝔤_{ℂ}, kω_{i}) to 𝔥_{ℂ} = 𝔤_{ℂ}^{τ} is multiplicity free.

§8. Appendix: associated bundles on Hermitian symmetric spaces

\underline{8.1.} In this section, we supply basic operations on homogeneous vector bundles for the convenience of the reader. The main goal is Lemma 8.6.

Let M be a real manifold, and V a (finite dimensional) vector space over ℂ. Suppose that we are given an open covering {U_{α}} of M and transition functions

g_{αβ}: U_{α} ∩ U_{β} → GL_{ℂ}(V)

satisfying the following compatibility conditions:

g_{αβ} g_{βγ} g_{γα} ≡ id on U_{α} ∩ U_{β} ∩ U_{γ}; g_{αα} ≡ id on U_{α}.

A complex vector bundle V over M with typical fiber V is constructed as the equivalence class of ∐_{α}(U_{α} × V), where (x, v) ∈ U_{β} × V and (y, w) ∈ U_{α} × V are defined to be equivalent if y = x and w = g_{αβ}(x)v. Then, the space of sections Γ(M, V) is identified with the collection

(8.1.1)  {(f_{α}) : f_{α} ∈ C^{∞}(U_{α}, V), f_{α}(x) = g_{αβ}(x)f_{β}(x), for x ∈ U_{α} ∩ U_{β}}.

If M is a complex manifold and if every g_{αβ} is holomorphic (or anti-holomorphic), then V → M is a holomorphic (anti-holomorphic, respectively) vector bundle.

\underline{8.2.} Let G be a Lie group, K a closed subgroup of G, and M := G/K a homogeneous manifold. We can take an open covering {U_{α}} of M such that for each α there is a local section φ_{α}: U_{α} → G of the principal bundle G → G/K. Given a representation χ: K → GL_{ℂ}(V), we define the homogeneous vector bundle, V := G ×_{K} (χ, V). Then V is associated with the transition functions:

g_{αβ}: U_{α} ∩ U_{β} → GL_{ℂ}(V), g_{αβ}(x) := χ(φ_{α}(x)^{−1}φ_{β}(x)).

The section Γ(M, V) is identified with the following subspace of C^{∞}(G, V):

(8.2.1)  {f ∈ C^{∞}(G, V) : f(gk) = χ^{−1}(k)f(g), for g ∈ G, k ∈ K}.

\underline{8.3.} Let G′ be a Lie group, K′ a closed subgroup of G′, and M′ := G′/K′ the homogeneous manifold. Suppose that σ: G′ → G is a Lie group homomorphism such that σ(K′) ⊂ K. We use the same letter σ to denote by the induced map M′ → M, g′K′ ↦ σ(g′)K. Then the pull-back of the vector bundle V → M, denoted by σ^{∗}V → M′, is associated to the representation

χ ∘ σ: K′ → GL_{ℂ}(V).

Then we have a commuting diagram of the pull-back of sections (see (8.2.1)):

σ^{∗}: Γ(M, V) → Γ(M′, σ^{∗}V), (f_{α})_{α} ↦ (f_{α} ∘ σ)_{α}
∩  ∩
σ^{∗}: C^{∞}(G, V) → C^{∞}(G′, V), f ↦ f ∘ σ.

\underline{8.4.} Suppose that V and W are complex vector spaces and that ξ: V → W is an anti-linear bijective map. Then, we have an anti-holomorphic group isomorphism

GL_{ℂ}(V) → GL_{ℂ}(W), g ↦ g^{ξ} := ξ ∘ g ∘ ξ^{−1}.

Let V → M be a complex vector bundle with transition functions g_{αβ} : U_{α} ∩ U_{β} → GL_{ℂ}(V). Then, one constructs a complex vector bundle ξ_{∗}V → M with the transition functions g_{αβ}^{ξ}: U_{α} ∩ U_{β} → GL_{ℂ}(W). We have a natural homomorphism

ξ_{∗}: Γ(M, V) → Γ(M, ξ_{∗}V), (f_{α}) ↦ (ξ ∘ f_{α}),

which is well-defined because the compatibility condition in (8.1.1) is satisfied as follows: If x ∈ U_{α} ∩ U_{β} then

g_{αβ}^{ξ}(x)(ξ ∘ f_{β})(x) = (ξ ∘ g_{αβ}(x) ∘ ξ^{−1})(ξ ∘ f_{β})(x) = ξ ∘ g_{αβ}(x)f_{β}(x) = ξ ∘ f_{α}(x).

If V is the homogeneous vector bundle G ×_{K} (χ, V) associated to a representation χ: K → GL_{ℂ}(V), then ξ_{∗}V is isomorphic to G ×_{K} (χ^{ξ}, W) associated to the representation

χ^{ξ}: K → GL_{ℂ}(W), k ↦ χ^{ξ}(k) := ξ ∘ χ(k) ∘ ξ^{−1}.

\underline{8.5.} We are particularly interested in the case where G′ = G, K′ = K in the setting of §8.3, and where V = W = ℂ and ξ(z) := z̄, the complex conjugate of z. By the identification of GL_{ℂ}(ℂ) with ℂ^{×}, we have g^{ξ} = \overline{g} for g ∈ GL_{ℂ}(V) ≃ ℂ^{×}. Thus, χ^{ξ} coincides with the conjugate representation

\overline{χ}: K → GL_{ℂ}(W) ≃ ℂ^{×}, k ↦ \overline{χ(k)},

for χ ∈ Hom(K, ℂ^{×}). Thus, we have an isomorphism

(8.5.1)  ξ_{∗}σ^{∗}V ≃ G ×_{K} (\overline{χ ∘ σ}, ℂ),

and have a formula

ξ_{∗} ∘ σ^{∗}: Γ(M, V) → Γ(M, ξ_{∗}σ^{∗}V), (f_{α}) ↦ (\overline{f_{α} ∘ σ}).

\underline{8.6.} We apply §8.5 to a specific setting where M = G/K is an irreducible Hermitian symmetric space. Suppose σ is an involutive automorphism of G such that σZ = −Z (see (1.5.2)), where Z is a generator of the center 𝔠(𝔨) of 𝔨.

Lemma 8.6. Let χ: K → ℂ^{×} be a unitary character. We denote by V the homogeneous line bundle G ×_{K} (χ, ℂ). Then we have an isomorphism of G-equivariant holomorphic line bundles: ξ_{∗}σ^{∗}V ≃ V.

Proof. In view of (8.5.1), it suffices to show \overline{χ ∘ σ} = χ. As the character χ of K is unitary, we have \overline{χ(k)} = χ(k^{−1}) for any k ∈ K. In view of σZ = −Z, we have

\overline{χ ∘ σ(exp tZ)} = \overline{χ(exp(−tZ))} = χ(exp tZ), (t ∈ ℝ).

On the other hand, if k ∈ [K, K], then \overline{χ ∘ σ(k)} = 1 = χ(k). As K = exp 𝔠(𝔨)·[K, K], we have shown \overline{χ ∘ σ} = χ. Hence Lemma. □

⟨Acknowledgement⟩

I would like to thank Professor J. Faraut who kindly explained his result [10]. I am also grateful to Professor P. Torasso who invited me to the CIMPA school on Invariant Theory in Tunisia, July-August 1996, when the main results of this paper were obtained.

References

[1] E. P. van den Ban, Invariant differential operators on a semisimple symmetric space and finite multiplicities in a Plancherel formula, Ark. Mat. 25 (1987), 175–187.

[2] E. P. van den Ban and H. Schlichtkrull, The most continuous part of the Plancherel decomposition for a reductive symmetric space, Ann. of Math. 145 (1997), 267–364.

[3] M. Berger, Les espaces symétriques non compacts, Ann. Sci. Ecole Norm. Sup. (3) 74 (1957), 85–177.

[4] W. Bertram and J. Hilgert, Hardy spaces and analytic continuation of Bergman spaces, Bull. Soc. Math. France 126 (1998), 435–482.

[5] P. Delorme, Formule de Plancherel pour les espaces symétriques réductifs, Ann. of Math. 147 (1998), 417–452.

[6] G. van Dijk and S. C. Hille, Canonical representations related to hyperbolic spaces, J. Funct. Anal. 147 (1997), 109–139.

[7] T. Enright, R. Howe and N. Wallach, A classification of unitary highest weight modules, in Representation theory of reductive groups, Progress in Math. Birkhäuser 40 (1983), 97-143.

[8] T. Enright and A. Joseph, An intrinsic classification of unitary highest weight modules, Math. Ann. 288 (1990), 571–594.

[9] J. Faraut and G. Ólafsson, Causal semisimple symmetric spaces, the geometry and harmonic analysis, Semigroups in Algebra, Geometry and Analysis (1995), Walter de Gruyter, 3–32.

[10] J. Faraut and E. Thomas, Invariant Hilbert spaces of holomorphic functions, preprint.

[11] M. Flensted-Jensen, Discrete series for semisimple symmetric spaces, Ann. of Math. 111 (1980), 253-311.

[12] I. M. Gelfand, Spherical functions on symmetric spaces, Dokl. Akad. Nauk. SSSR 70 (1950), 5–8.

[13] E. Gutkin, Coefficients of Clebsch-Gordan for the holomorphic discrete series, Letters in Math. Phys. 3 (1979), 185–192.

[14] Harish-Chandra, Representations of semi-simple Lie groups, I, III, Trans. A. M. S. 75, (1953), 185–243; 76, (1954), 234–253; IV, Amer. J. Math. 77, (1955), 743–777.

[15] G. Heckman and H. Schlichtkrull, Harmonic analysis and special functions on symmetric spaces., vol. 16, Academic Press, Inc., San Diego, 1994.

[16] S. Helgason, A duality for symmetric spaces with applications to group representations I, II, Adv. in Math. 5, (1970), 1–154; 22, (1976), 187–219.

[17] R. Hoogenboom, Intertwining functions on compact Lie groups, CWI Tract 5 (1984), Stichting Mathematisch Centrum, Centrum voor Wiskunde en Informatica, Amsterdam, 1–86.

[18] R. Howe, Reciprocity laws in the theory of dual pairs, Representation Theory of Reductive Groups (P. C. Trombi, ed.), vol. 40, Progress in Mathematics, Birkhäuser, Boston, 1983, pp. 159–175.

[19] , Perspectives on invariant theory: Schur duality, multiplicity-free actions and beyond, Israel Math. Conf. Proc. 8 (1995), 1–182.

[20] H. Jaffee, Real forms of Hermitian symmetric spaces, Bull. Amer. Math. Soc. 81 (1975), 456–458.

[21] , Anti-holomorphic automorphisms of the exceptional symmetric domains, J. Differential Geom. 13 (1978), 79–86.

[22] H. P. Jakobsen, Tensor products, reproducing kernels, and power series, J. Funct. Anal. 31 (1979), 293–305.

[23] , Hermitian symmetric spaces and their unitary highest weight modules, J. Funct. Anal. 52 (1983), 385–412.

[24] H. P. Jakobsen and M. Vergne, Restrictions and expansions of holomorphic representations, J. Funct. Anal. 34 (1979), 29–53.

[25] K. Johnson, On a ring of invariant polynomials on a Hermitian symmetric space, J. Algebra 67 (1980), 72–81.

[26] M. Kashiwara and M. Vergne, On the Segal-Shale-Weil representations and harmonic polynomials, Invent. Math. 44 (1978), 1-47.

[27] S. Kobayashi and T. Nagano, On filtered Lie algebras and geometric structures I, J. Math. Mech. 13 (1964), 875–908.

[28] T. Kobayashi, A necessary condition for the existence of compact Clifford-Klein forms of homogeneous spaces of reductive type, Duke Math. J. 67 (1992), 653–664.

[29] , Multiplicity free branching laws for unitary highest weight modules, Proceedings of the Symposium on Representation Theory held at Saga, Kyushu 1997 (K. Mimachi, ed.), 1997, pp. 9–17.

[30] , Discrete decomposability of the restriction of A_{𝔮}(λ) with respect to reductive subgroups and its applications, Part I, Invent. Math. 117, (1994), 181–205; Part II, Ann. of Math. 147, (1998), 1–21; Part III, Invent. Math. 131, (1998), 229–256.

[31] , Discrete series representations for the orbit spaces arising from two involutions of real reductive Lie groups, J. Funct. Anal. 152 (1998), 100–135.

[32] K. Koike and I. Terada, Young diagrammatic methods for the representation theory of the classical groups of type B_{n}, C_{n}, D_{n}, J. Algebra 107 (1987), 466–511.

[33] , Young diagrammatic methods for the restriction of representations of complex classical Lie groups to reductive subgroups of maximal rank, Adv. in Math. 79 (1990), 104–135.

[34] S. Lang, SL_{2}(ℝ), Addison-Wesley, MA, 1975.

[35] R. Lipsman, On the unitary representation of a semisimple Lie group given by the invariant integral on its Lie algebra, Adv. Math. Suppl. 6, (1979) 143–158; II, Canad. J. Math. 29, (1977) 1217–1222.

[36] I. G. Macdonald, Symmetric Functions and Hall Polynomials, Oxford University Press, 1997.

[37] S. Martens, The characters of the holomorphic discrete series, Proc. Nat. Acad. Sci. USA 72 (1975), 3275-3276.

[38] S. Okada, Applications of minor summation formulas to rectangular-shaped representations of classical groups, J. Algebra 205 (1998), 337–367.

[39] G. Ólafsson and B. Ørsted, Generalizations of the Bargmann transform, Lie Theory and its application in physics (Clausthal, 1995) (Dobrev and Döbner, eds.), World Sci. Publishing, 1996, pp. 3–14,.

[40] B. Ørsted and G. Zhang, Tensor products of analytic continuations of holomorphic discrete series, Canad. J. Math. 49 (1997), 1224–1241.

[41] T. Oshima and J. Sekiguchi, Eigenspaces of invariant differential operators on an affine symmetric space, Invent. Math. 57 (1980), 1–81.

[42] , The restricted root system of a semisimple symmetric pair, Advanced Studies in Pure Math. 4 (1984), 433-497.

[43] R. A. Proctor, Shifted plane partitions of trapezoidal shape, Proc. Amer. Math. Soc. 89 (1983), 553–559.

[44] J. Repka, Tensor products of holomorphic discrete series representations, Canad. J. Math. 31 (1979), 836-844.

[45] W. Rossmann, The structure of semisimple symmetric spaces, Canad. J. Math. 31 (1979), 156-180.

[46] W. Schmid, Die Randwerte holomorphe Funktionen auf hermetisch symmetrischen Raumen, Invent. Math. 9 (1969-70), 61–80.

[47] N. Shimeno, The Plancherel formula for spherical functions with a one-dimensional K-type on a simply connected simple Lie group of Hermitian type, J. Funct. Anal. 121 (1994), 330–388.

[48] J. R. Stembridge, Hall-Littlewood symmetric functions and Rogers-Ramanujan identities, Trans. Amer. Math. 319 (1990), 469–498.

[49] É. B. Vinberg and B. N. Kimelfeld, Homogeneous domains on flag manifolds and spherical subgroups of semisimple Lie groups, Funct. Anal. Appl. 12 (1978), 168–174.

[50] D. Vogan, Jr., Representations of real reductive Lie groups, Birkhäuser, 1981.

[51] , Unitary Representations of Reductive Lie Groups, Ann. Math. Stud. 118, Princeton U. P., 1987.