## Branching Problems of Unitary Representations, (invited lecture), International Congress of Mathematicians (ICM 2002), Beijing, China, 20-28 August 2002.

Let π be an irreducible unitary representation of a group G. A branching law is the irreducible decomposition of π with regard to a subgroup G':

π|G' \simeq Ĝ mπ(τ)τdμ(τ)     (a direct integral).

Such a decomposition is unique, for example, if G' is a real reductive group, and the multiplicity mπ: Ĝ \to N ∪ {∞} makes sense as a measurable function on the unitary dual Ĝ.

Special cases of branching problems include (or reduce to) the followings: Clebsch-Gordan coefficients, Littlewood-Richardson rules, decomposition of tensor product representations, character formulas, Blattner formulas, Plancherel theorems for homogeneous spaces, description of breaking symmetries in quantum mechanics, theta-lifting in automorphic forms, etc. The restriction of unitary representations serves also as a method to study discontinuous groups for non-Riemannian homogeneous spaces (e.g. ).

Our interest is in the branching problems for a pair of real reductive groups GG', especially for non-compact G'. In this generality, there is no known algorithm to find branching laws. Even worse, branching laws usually contain both discrete and continuous spectrum with possibly infinite multiplicities (the multiplicity is infinite, for example, in the decomposition of the tensor product of two principal series representations of SL(n, C) for n ≥ 3).

In order to single out a nice class of branching problems, the author put emphasis on the following notion in :

Definition. We say the restriction π|G' is G'-admissible if it decomposes discretely and the multiplicity mπ(τ) is finite for any τ ∈ Ĝ.

a) (Harish-Chandra) G'=K, a maximal compact subgroup of G.
b) (Howe) π is the Weil representation, and (G, G') is a compact dual pair.

In these examples, either the subgroup G' or the representation π is very special, namely, G' is compact or π has a highest weight. Surprisingly, without such assumptions, there is still a fairly rich family of the triple (G, G', π) such that the restriction π|G' is G'-admissible. The following criterion, proved in  by using micro-local analysis and an earlier idea of Kashiwara-Vergne and Howe, asserts that the "balance" of G' and π is crucial to the G'-admissibility.

Theorem. Let GG' be a pair of real reductive groups, and π ∈ Ĝ. If Cone(G') ∩ ASK(π)={0}, then the restriction π|G' is G'-admissible.

Here, ASK(π) is the asymptotic K-support of π (), and Cone(G') is a closed cone determined by G' [8, Definition 4.2].

Example.
1) If G' = K, then the assumption of Theorem is automatically fulfilled because Cone(G') = {0}. This special case corresponds to Harish-Chandra's admissibility theorem.
2) If (G, G') = (SO(4,2), SO(4,1)), then about "67%" of irreducible unitary representations of G with regular integral infinitesimal characters are G'-admissible when restricted to G' ([5,7]).

Harish-Chandra's admissibility theorem for compact G' laid a foundation of algebraic theory (so called (g, K)-modules) of unitary representations. We then ask:

What can we expect from the admissibility theorem for non-compact G'?

In particular, we are interested in the effect of the non-existence of continuous spectrum in geometric realizations of branching problems of unitary representations. The second half of the talk will discuss some of recent progress on the applications of discretely decomposable branching laws:

1. Representation theory.
Understanding of "small" representations.
2. Automorphic forms.
Topology of modular varieties for Clifford-Klein forms.
3. Lp-analysis.
(New) discrete spectrum for (non-symmetric) homogeneous spaces.
References
  B. H. Gross and N. R. Wallach, Progress in Math. 123 (1994), 289-304.  R. Howe, Proc. Symp. Pure Math. 33 (1979), 275-285, AMS.  M. Kashiwara and M. Vergne, LNM 728 (1979), 177-200.  T. Kobayashi, Memoirs of AMS 462 (1992), 106pp.  T. Kobayashi, Invent. Math. 117 (1994), 181-205.  T. Kobayashi, Annals of Math. 147 (1998), 709-729.  T. Kobayashi, Invent. Math. 131 (1998), 229-256.  T. Kobayashi, Advanced Study in Pure Math. 26 (2000), 98-126.  J.-S. Li, Manuscripta Math. 91 (1996), 163-177.  G. Margulis, Bul. Soc. Math. France 125 (1997), 1-10.
[ proceedings | programme | abstract(pdf) | abstract(dvi) | abstract(ps) ]