SINGULAR UNITARY REPRESENTATIONS

AND DISCRETE SERIES FOR

INDEFINITE STIEFEL MANIFOLDS U(p, q; 𝔽)/U(p − m, q; 𝔽)

Toshiyuki Kobayashi Department of Mathematics University of Tokyo Hongo, Tokyo 113, Japan.

Typeset by AMS-TEX

Contents

0. Introduction - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 1 1. Notation - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 13

1. θ-stable parabolic subalgebra - - - - - - - - - - - - - - - - - - - - - - - - - - - 13 2. good range, fair range - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 15 3. cohomological parabolic induction - - - - - - - - - - - - - - - - - - - - - - - - 16 4. results from Zuckerman and Vogan - - - - - - - - - - - - - - - - - - - - - - - 16 5. results from Harish-Chandra and Oshima-Matsuki - - - - - - - - - - - - - - 17

2. Main results - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 19 1. G = Sp(p, q) - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 19 2. main theorem for G = Sp(p, q) - - - - - - - - - - - - - - - - - - - - - - - - - - 20 3. G = U(p, q) - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 22 4. main theorem for G = U(p, q) - - - - - - - - - - - - - - - - - - - - - - - - - - - 23 5. G = SO_{0}(p, q) - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 25 6. main theorem for G = SO_{0}(p, q) - - - - - - - - - - - - - - - - - - - - - - - - - 26 7. list and figures of various conditions on parameters - - - - - - - - - - - - - 28 8. remarks - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 33

3. Further notations and preliminary results - - - - - - - - - - - - - - - - - - 35 1. Jantzen-Zuckerman’s translation functor - - - - - - - - - - - - - - - - - - - - 35 2. induction by stages - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 36 3. definition of A (λ ▷ λ′) - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 36 4. A (λ ▷ λ′) and derived functor modules - - - - - - - - - - - - - - - - - - - - - 37 5. some symbols - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 40

4. Some explicit formulas on K multiplicities - - - - - - - - - - - - - - - - - 42 1. preliminaries - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 42 2. some alternating polynomials - - - - - - - - - - - - - - - - - - - - - - - - - - - 48 3. result in quaternionic case - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 52 4. result in complex case - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 54 5. result in real case - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 56 6. some auxiliary lemmas - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 58 7. proof for quarternionic case - - - - - - - - - - - - - - - - - - - - - - - - - - - - 62 8. proof for complex case - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 65 9. proof for real case - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 69

5. An alternative proof of the sufficiency for R_{𝔮}^{S}(ℂ_{λ}) ≠ 0 - - - - - - - - - - 70 1. theorem: sufficient condition for R_{𝔮}^{S}(ℂ_{λ}) ≠ 0 - - - - - - - - - - - - - - - - - 71 2. key lemmas - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 75 3. proof of the combinatorial part - - - - - - - - - - - - - - - - - - - - - - - - - - 76

6. Proof of irreducibility results - - - - - - - - - - - - - - - - - - - - - - - - - - - 78 1. irreducibility in the fair range - - - - - - - - - - - - - - - - - - - - - - - - - - - 78 2. twisted differential operators - - - - - - - - - - - - - - - - - - - - - - - - - - - 79 3. theorem - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 81 4. irreducibility result - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 81 5. Vogan’s idea on the translation principle for A_{γ}(𝔩 : 𝔤) - - - - - - - - - - - - 82 6. notations about GL(n, ℂ) and Sp(n, ℂ) - - - - - - - - - - - - - - - - - - - - - 83 7. definition of C_{λ} - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 85 8. verification of (6.5.4)(a) - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 87 9. verification of (6.5.4)(b) - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 91 10. verification of (6.5.4)(c) - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 92 11. proof of Corollary(6.4.1) - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 93

7. Proof of vanishing results outside the fair range - - - - - - - - - - - - - 96 1. proof in complex case - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 96 2. vanishing result in quaternionic case - - - - - - - - - - - - - - - - - - - - - - 97 3. maximal parabolic case - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 98 4. general parabolic case - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 99

8. Proof of the inequivalence results - - - - - - - - - - - - - - - - - - - - - - - 101 1. quarternionic case - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 101 2. orthogonal case - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 102

References - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 104

Abstract

This paper treats the relatively singular part of the unitary dual of pseudo-orthogonal groups over 𝔽 = ℝ, ℂ and ℍ. These representations arise from discrete series for indefinite Stiefel manifolds U(p, q; 𝔽)/U(p − m, q; 𝔽) (2m ≤ p). Thanks to the duality theorem between D-module construction and Zuckerman’s derived functor modules (ZDF-modules), these discrete series are naturally described in terms of ZDF-modules with possibly singular parameters. Some techniques including a new K-type formula are offered to find the explicit condition deciding whether the corresponding ZDF-modules R_{𝔮}^{S}(ℂ_{λ}) vanish or not. We also investigate the irreducibility and pairwise inequivalence among these ZDF-modules. Although our concern is limited to the discrete series, our approach is purely algebraic and applicable to a less special setting. It is an interesting phenomenon that our discrete series sometimes give a sharper condition for unitarizability of ZDF-modules than those given by Vogan (1984) algebraically. This phenomenon does not occur in the case of discrete series for group manifolds or semisimple symmetric spaces.

1980 Mathematics Subject Classification (1985 Revision)

primary: 22E46, secondary: 43A85,22E45,22E47.

\underline{keywords and phrases}: unitary representations of semisimple Lie groups, Zuckerman’s derived functor modules, discrete series, symmetric spaces, pseudo-orthogonal groups