CAPELLI IDENTITIES, DEGENERATE SERIES

AND HYPERGEOMETRIC FUNCTIONS^{∗}

Toshio Oshima^{†}

Abstract. The Capelli identity is extended to the case of minors. Related differential operators in the generalized identities give the annihilator of the degenerate principal series for GL(n) and characterizes the image of the Poisson transform of the hyperfunctions on several boundaries of GL(n). Hypergeometric functions are defined through realizations of some special sections of the degenerate principal series and the realizations on boundaries of GL(n) generalize Gelfand’s hypergeometric functions. Related Radon transforms for Grassmannians are examined.

1. Generalized Capelli identities and related operators

The classical Capelli identity can be considered as a quantization of the formula det^{t}AB = det A · det B in the linear algebra. We quantize more general identities

(1.1) det ( ∑_{ν=1}^{n} x_{νi}y_{νj})\begin{matrix}1≤i≤m \\ 1≤j≤m \end{matrix} = ∑_{1≤ν_{1}<···<ν_{m}≤n} det (x_{ν_{i}j})\begin{matrix}1≤i≤m \\ 1≤j≤m \end{matrix} · det (y_{ν_{i}j})\begin{matrix}1≤i≤m \\ 1≤j≤m \end{matrix}

for 2mn commutative variables x_{νi} and y_{νi} with 1 ≤ i ≤ m and 1 ≤ ν ≤ n (cf. [Si, II §5 Theorem 9]), where the left hand side of (1.1) is zero if m > n, and we get

Theorem 1.1. (Generalized Capelli identities) Let I = {i_{k}}_{1≤k≤m} and J = {j_{ℓ}}_{1≤ℓ≤m} be sequences of positive integers. Then

(1.2) det ( ∑_{ν=1}^{n} x_{νi_{k}} \frac{∂}{∂x_{νj_{ℓ}}} + (m − ℓ)δ_{i_{k}j_{ℓ}})\begin{matrix}1≤k≤m \\ 1≤ℓ≤m \end{matrix}

= \{ \begin{matrix}∑_{1≤ν_{1}<···<ν_{m}≤n} det (x_{ν_{p}i_{q}})\begin{matrix}1≤p≤m \\ 1≤q≤m \end{matrix} · det (\frac{∂}{∂x_{ν_{p}j_{q}}} )\begin{matrix}1≤p≤m \\ 1≤q≤m \end{matrix} & if m ≤ n, \\ 0 & if m > n. \end{matrix}

Here δ_{ij} is the Kronecker symbol and for a matrix A = (A_{ij}) = (A_{ij})\begin{matrix}1≤i≤m \\ 1≤j≤m \end{matrix}

with (i, j)-components A_{ij} in an associative algebra, we define

(1.3)  det A = ∑_{σ∈𝔖_{m}} sgn(σ)A_{σ(1)1}A_{σ(2)2} ⋯ A_{σ(m)m},

Typeset by AMS-TEX

∗A talk on a symposium on Representation Theory held at Okinawa, December 12–15, 1995.

†Department of Mathematical Sciences, University of Tokyo, Komaba, Tokyo 153, JAPAN

where 𝔖_{m} is the m-th symmetric group.

Using the Laplace expansion of the determinant with respect to the first column, the theorem is proved by induction because (1.2) is a formula for arbitrary minors.

Remark 1.2. If m = n, Theorem 1.1 is reduced to the Capelli identity

det ( ∑_{ν=1}^{n} x_{νi} \frac{∂}{∂x_{νj}} + (m − j)δ_{ij})\begin{matrix}1≤i≤n \\ 1≤j≤n \end{matrix} = det (x_{ij})\begin{matrix}1≤i≤n \\ 1≤j≤n \end{matrix} · det (\frac{∂}{∂x_{ij}} )\begin{matrix}1≤i≤n \\ 1≤j≤n \end{matrix}.

Put 𝔤_{ℂ} = 𝔤𝔩(n, ℂ) = Lie(GL(n, ℂ)). Let U(𝔤) be the universal enveloping algebra of 𝔤_{ℂ}. Then 𝔤_{ℂ} is naturally identified with the n^{2}-dimensional vector space of n×nmatrices whose components are complex numbers. Let E_{kl} denote the element of 𝔤_{ℂ} whose (i, j)-component equals δ_{ki}δ_{lj}. We put E_{i} = E_{ii} for simplicity. Put 𝔞 = ∑_{i=1}^{n} ℂE_{i}, 𝔫 = ∑_{1≤i<j≤n} ℂE_{ji} and 𝔫̄ = ∑_{1≤i<j≤n} ℂE_{ij}.

Definition 1.3. Let I = {i_{1}, …, i_{m}} and J = {j_{1}, …, j_{m}} be sequences of positive integers with 1 ≤ i_{μ} ≤ n and 1 ≤ j_{ν} ≤ n. Then for λ ∈ ℂ we put

(1.4)  D_{IJ}(λ) = det (E_{i_{μ}j_{ν}} + (λ + m − ν)δ_{i_{μ}j_{ν}} )\begin{matrix}1≤μ≤m \\ 1≤ν≤m \end{matrix}.

We naturally identify E_{ij} with the left invariant vector filed on G = GL(n, ℂ). Then E_{ij} is of the form

(1.5)  E_{ij} = ∑_{ν=1}^{n} x_{νi} \frac{∂}{∂x_{νj}}

under the coordinates (x_{ij})\begin{matrix}1≤i≤n \\ 1≤j≤n \end{matrix} ∈ G and the left hand side of (1.2) is identified

with D_{IJ}(0) and morover it becomes D_{IJ}(λ) under the automorphism of U(𝔤) satisfying E_{ij} ↦ E_{ij} + λδ_{ij}. Hence we call D_{IJ}(λ) generalized Capelli operators.

We quote some fundamental properties of D_{IJ}(λ) from [O4, §2]:

Theorem 1.4. i) D_{σ(I)σ′(J)}(λ) = sgn(σ) sgn(σ′)D_{IJ}(λ) for σ, σ′ ∈ 𝔖_{m}.

ii) For an ordered set of indices I = {i_{1}, …, i_{m}} and for k ∈ I, we define ι_{ℓ}^{k}(I) = {i′_{1}, …, i′_{m}} by

i′_{p} = \{ \begin{matrix}ℓ & if i_{p} = k, \\ i_{p} & if i_{p} ≠ k, \end{matrix}

for p = 1, …, m. Moreover we put ι_{ℓ}^{k}(I) = ∅ if k ∉ I. Then

[E_{kℓ}, D_{IJ}(λ)] = D_{ι_{k}^{ℓ}(I)J}(λ) − D_{Iι_{ℓ}^{k}(J)}(λ).

Here we define D_{IJ}(λ) = 0 if #I ≠ #J.

iii) Suppose I = {i_{1} < ⋯ < i_{m}} and J = {j_{1} < ⋯ < j_{m}}. Then

D_{IJ}(λ) = \{ \begin{matrix}0 mod 𝔫̄U(𝔤) + U(𝔤)𝔫 & if I ≠ J, \\ ∏_{ν=1}^{m}(E_{i_{ν}i_{ν}} + λ + ν − 1) mod U(𝔤)𝔫 & if I = J. \end{matrix}

The following lemma says that the power of matrices of (D_{IJ})_{IJ} also have the same property as in Theorem 1.4 i) and ii).

Lemma 1.5. Fix a positive number m with 1 ≤ m ≤ n. Let F (i)_{IJ} be elements of U(𝔤) which satisfy

(1.6)

F (i)_{σ(I)σ′(J)} = sgn(σ) sgn(σ′)F (i)_{IJ} for σ, σ′ ∈ 𝔖_{m},
[E_{kℓ}, F (i)_{IJ}] = F (i)_{ι_{k}^{ℓ}(I)J} − F (i)_{Iι_{ℓ}^{k}(J)}

for i = 1 and 2. Here I and J are ordered sets of m positive integers being smaller or equal to n and we put F (i)_{IJ} = 0 if #I ≠ #J. Then F (3)_{IJ} = ∑_{V} F (1)_{IV} F (2)_{V J} also satisfies (1.6) with i = 3. Here the sum runs over all sets V = {ν_{1}, ν_{2}, …, ν_{m}} of indices with 1 ≤ ν_{1} < ν_{2} < ⋯ < ν_{m} ≤ n.

Viewing the above lemma, it is interesting to study the power of (D_{IJ}(λ))_{IJ}.

Definition 1.6. We define

\begin{aligned}F_{ij}^{1}(λ) & = E_{ij} + λδ_{ij}, \\ (F_{ij}^{k}(λ_{1}, …, λ_{k}))\begin{matrix}1≤i≤n \\ 1≤j≤n \end{matrix} & = (F_{ij}^{1}(λ_{1}))\begin{matrix}1≤i≤n \\ 1≤j≤n \end{matrix} ⋯ (F_{ij}^{1}(λ_{k}))\begin{matrix}1≤i≤n \\ 1≤j≤n \end{matrix}. \end{aligned}

Theorem 1.7. i) F_{ij}^{k}(λ_{1}, …, λ_{k}) is a symmetric function of λ_{1}, …, λ_{k}. ii) The action of 𝔤_{ℂ}:

[E_{μν}, F_{ij}^{k}] = F_{μj}^{k}δ_{νi} − F_{iν}^{k}δ_{μj}.

iii) A formula related to the Bruhat decomposition:

F_{ij}^{k}(λ_{1}, …, λ_{k}) ≡ 0 mod U(𝔤)𝔫 if i > j,
F_{ij}^{k}(λ_{1}, …, λ_{k}) ≡ 0 mod 𝔫̄U(𝔤) if i < j,
F_{ii}^{k}(λ_{1}, …, λ_{k}) ≡ F_{ii}^{k−1}(λ_{1}, …, λ_{k−1}) · (E_{i} + λ_{k} + i − 1)

− ∑_{ν=1}^{i−1} F_{νν}^{k−1}(λ_{1}, …, λ_{k−1}) mod U(𝔤)𝔫,

F_{ii+1}^{k}(λ_{1}, …, λ_{k}) ≡ F_{ii+1}^{k−1}(λ_{1}, …, λ_{k−1})(E_{i+1} + λ_{k} + i)

+ E_{ii+1}F_{ii}^{k−1}(λ_{1}, …, λ_{k−1}) mod U(𝔤)𝔫.

iv) Let Θ = {0 < n_{1} < n_{2} < ⋯ < n_{L} = n} be a sequence of positive integers. Put n_{0} = 0 and define

J_{μ} = U(𝔤)𝔫 + ∑_{ν=1}^{L} ∑_{j=n_{ν−1}+1}^{n_{ν}} U(𝔤)(E_{j} + λ_{ν} + n_{ν−1})

Then

F_{ii}^{k}(λ_{1}, …, λ_{k})

≡ \{ \begin{matrix}0 & mod J_{μ} if i ≤ n_{k}, \\ ∏_{ν=1}^{k}(λ_{ν} − λ_{k+1} + n_{ν−1} − n_{ν}) & mod J_{μ} if n_{k} < i ≤ n_{k+1}, \end{matrix}

F_{ii+1}^{k}(λ_{1}, …, λ_{k})

≡ ∏_{ν=1}^{ℓ−1}(λ_{ν} − λ_{ℓ} + n_{ν−1} − n_{ν} − n_{ℓ−1} + i) ∏_{ν=ℓ+1}^{k}(λ_{ν} − λ_{ℓ} − n_{ℓ−1} + i)E_{ii+1}

mod J_{μ} if n_{ℓ−1} < i < n_{ℓ} and k ≥ ℓ.

2. Degenerate principal series of GL(n) (global version)

Let G be GL(n, ℝ) or U(p, q) with p ≥ q or GL(m, ℍ) or GL(n, ℂ), where n = p + q and n = 2m. Our argument in the sequel is valid for the direct product of these groups but for simplicity we assume this. We will mainly consider the case when G ≠ GL(n, ℂ) but the results also hold for GL(n, ℂ) with some obvious modifications which we will give if necessary.

Then G is a real form of (a direct product of some copies of) GL(n, ℂ) and its Lie algebra 𝔤 is defined by the following involutions:

(2.1) 𝔤𝔩(n, ℂ) ∋ X ↦ \{ \begin{matrix}X̄ & if 𝔤 = 𝔤𝔩(n, ℝ), \\ −I_{pq}^{t}X̄I_{pq} & if 𝔤 = 𝔲(p, q), \\ −J_{m}X̄J_{m} & if 𝔤 = 𝔤𝔩(m, ℍ), \end{matrix}

𝔤𝔩(n, ℂ) ⊕ 𝔤𝔩(n, ℂ) ∋ (X, Y) ↦ (Ȳ , X̄)  if 𝔤 = 𝔤𝔩(n, ℂ),

where I_{pq} is the permutation matrix representing ( \begin{matrix}1 & ··· & q & q+1 & ··· & p & p+1 & ··· & p+q \\ p+q & ··· & p+1 & q+1 & ··· & p & q & ··· & 1 \end{matrix} )

and J_{m} is the direct product of m-copies of ( \begin{matrix}0 & 1 \\ −1 & 0 \end{matrix} ). Moreover 𝔤𝔩(n, ℂ) is identified

with a subalgebra of 𝔤𝔩(n, ℂ) ⊕ 𝔤𝔩(n, ℂ) by the map 𝔤𝔩(n, ℂ) ∋ X ↦ (X, X̄). Then the element of the quaternion number field ℍ is identified with a 2 × 2 complex matrix by

(2.2)  ℍ ∋ a + bi + cj + dk ↦ ( \begin{matrix}a + bi & c + di \\ −c + di & a − bi \end{matrix} ) for a, b, c, d ∈ ℝ.

We choose maximally split Cartan subalgebras 𝔧 by

(2.3) 𝔧 = \{ \begin{matrix}∑_{i=1}^{n} ℝE_{i} & if G = GL(n, ℝ), \\ ∑_{i=1}^{n} ℝE_{i} + ℝ\sqrt{−1}E_{i} & if G = GL(n, ℂ), \\ ∑_{i=1}^{m} ℝ(E_{2i−1} + E_{2i}) + ℝ\sqrt{−1}(E_{2i−1} − E_{2i}) & if G = GL(m, ℍ), \\ ∑_{i=1}^{q} ℝ(E_{i} − E_{n+1−i}) + ℝ\sqrt{−1}(E_{i} + E_{n+1−i}) \\ + ∑_{i=q+1}^{2n−q} ℝ\sqrt{−1}E_{i} & if G = U(p, q) \end{matrix}

so that they are compatible with our arguments in §1. Then K = G ∩ U(n) is a maximal compact subgroup of G and θ : G ∋ g ↦ ^{t}ḡ^{−1} or 𝔤 ∋ X ↦ −^{t}X̄ is the Cartan involution.

Put 𝔞_{𝔭} = 𝔧 ∩ 𝔤𝔩(n, ℝ) and 𝔱 = 𝔧 ∩ \sqrt{−1}𝔤𝔩(n, ℝ). Since the complexification (𝔤_{ℂ}, 𝔧_{ℂ}) of (𝔤, 𝔧) is (𝔤𝔩(n, ℂ), 𝔞) (or the direct sum of its two copies), we define a fundamental system as in §1 and a compatible fundamental system Φ(𝔞_{𝔭}) for the restricted root system for (𝔤, 𝔞_{𝔭}). Then the restricted root systems are of type A_{n−1}^{1}, A_{n−1}^{2}, A_{m−1}^{4}, BC_{q}^{2(p−q),2,1} and C_{q}^{2,1}, respectively, where the superfices represent the multiplicities of roots in the order according to the length of roots.

Let P_{Ξ} be the compatible parabolic subgroup of G with the Langlands decomposition P_{Ξ} = M_{Ξ}A_{Ξ}N_{Ξ} and the Levi decomposition P_{Ξ} = L_{Ξ}N_{Ξ} with L_{Ξ} = M_{Ξ}A_{Ξ}.

The identity component of M_{Ξ} is denoted by M_{Ξ}^{o} and we put N̄_{Ξ} = θ(N_{Ξ}). The Lie algebras of K, M_{Ξ}, A_{Ξ}, N_{Ξ}, N̄_{Ξ} and L_{Ξ} are denoted by 𝔨, 𝔪_{Ξ}, 𝔞_{Ξ}, 𝔫_{Ξ}, 𝔫̄_{Ξ} and 𝔩_{Ξ}, respectively and then 𝔞_{Ξ} ⊂ 𝔧 ⊂ 𝔞, ℂ ⊗_{ℝ} 𝔧 = 𝔞 and ℂ ⊗_{ℝ} Lie(N_{Ξ}) ⊂ 𝔫. Note that 𝔞 and 𝔫 should be replaced by 𝔞 ⊕ 𝔞 and 𝔫 ⊕ 𝔫, respectively, if G = GL(n, ℂ).

Fix an irreducible finite dimensional representation (δ, V_{δ}) of M_{Ξ}. For any element ξ of the complex dual 𝔞_{Ξ}^{∗} of 𝔞_{Ξ}, we have an irreducible representation (τ_{δ,ξ}, V_{δ}) of P_{Ξ} by P ∋ man ↦ δ(m)a^{ξ} ∈ V_{δ} for m ∈ M_{Ξ}, a ∈ A_{Ξ} and n ∈ N_{Ξ}. Then we define the space of hyperfunction sections of (degenerate) principal series of G:

(2.4) B(G/P_{Ξ}; δ, ξ) = {f ∈ B(G) ⊗ V_{δ}; f(gp) = τ_{δ,ξ}(p^{−1})f(g) for p ∈ P_{Ξ}}.

Here B(G) is the space of hyperfunctions on G and they are left G-modules by (π_{g}f)(x) = f(g^{−1}x) for g ∈ G.

Let d(δ, ξ) denote the lowest weight of the representation τ_{δ,ξ} of L_{Ξ} with respect to the order defined above. Then d(δ, ξ) ∈ 𝔞^{∗} and d(δ, ξ) = dδ + ξ, where dδ is the lowest weight of δ and we identified 𝔞 and 𝔞^{∗} by the Killing form of 𝔤𝔩(n, ℂ). Put

(2.5)  d(δ, ξ) = ∑_{i=1}^{n} d(δ, ξ)_{i}e_{i},

Here the right hand side is replaced by ∑_{i=1}^{n} d(δ, ξ)_{i}(e_{i}, 0) + ∑_{i=1}^{n} d(δ, ξ)′_{i}(0, e_{i}) if G = GL(n, ℂ).

Theorem 2.1. For an irreducible finite dimensional representation δ of M_{Ξ} we define the sequence Θ = {n_{1} < ⋯ < n_{L} = n} of positive integers so that for any i with 1 ≤ i ≤ n

(2.6) there exists k with n_{k−1} < i < n_{k} if and only if d(δ, ξ)_{i} = d(δ, ξ)_{i+1} for any ξ ∈ 𝔞_{Ξ}^{∗},

where n_{0} = 0 as in §1. Then any u ∈ B(G/P_{Ξ}; δ, ξ) satisfies

D_{IJ}(−λ_{k} − n_{k−1})u = 0 (2.7)

for #I = #J = n − n_{k} + n_{k−1} + 1 and k = 1, …, L,

F_{ij}^{L}(−λ_{1} − n_{0}, …, −λ_{L} − n_{L−1})u = 0 for i, j = 1, …, n. (2.8) by putting

(2.9)  λ_{ν} = d(δ, ξ)_{n_{ν}} for ν = 1, …, L.

Remark 2.2. We give examples in the case when P_{Ξ} is maximal and δ is trivial. Then M_{Ξ} is characterized by a unique positive simple root α_{k} whose root space is not contained in 𝔪_{Ξ}. Note that dim a_{Ξ}^{∗} = 2. Then

(2.10)  Θ = \{ \begin{matrix}\{k, n\} & if G = GL(n, ℝ), \\ \{k, n\} & if G = GL(n, ℂ). \\ \{2k, 2m\} & if G = GL(m, ℍ), \\ \{q, 2q\} & if G = U(q, q) and k = q, \\ \{k, p + q − k, p + q\} & if G = U(p, q) and 2k < p + q, \end{matrix}

Moreover λ_{2} = 0 if G = U(p, q) and 2k < p + q. Note that in the case of the maximal parabolic subgroup defining Shilov boundary of the symmetric space G/K with G = U(p, q), the above number k equals q.

We give another realization of B(G/P_{Ξ}; δ, ξ). Put Z(𝔞_{𝔭}) = K ∩exp \sqrt{−1}𝔞_{𝔭} whose elements are, in our case, diagonal matrices with diagonal components ±1. Then the Cartan subgroup J of G which is the centralizer of 𝔧 equals Z(𝔞_{𝔭}) exp 𝔧. Note that J ≃ ℤ_{2}^{ℓ} × exp 𝔧, where

(2.11)  ℓ = \{ \begin{matrix}n & if G = GL(n, ℝ), \\ 0 & if G = GL(n, ℂ) or GL(m, ℍ) or U(p, q). \end{matrix}

Since M_{Ξ} is a linear group, the representation δ = (τ_{δ}, V_{δ}) of M_{Ξ} has a unique highest weight vector v_{δ} ∈ V_{δ} up to constant multiple. Similarly the dual δ^{∗} = (τ_{δ^{∗}}, V_{δ}^{∗}) of δ has a unique highest weight vector v_{δ}^{∗} with ⟨v_{δ}^{∗}, v_{δ}⟩ = 1. Then the map

B(G/P_{Ξ}; δ, ξ) ∋ f ↦ ⟨v_{δ}^{∗}, f⟩ ∈ B(G)

is an injective G-homomorphism whose image equals

(2.12)

B(G/P_{Ξ}; δ, ξ)_{𝔫} = {u ∈ B(G); Xu = 0 for X ∈ 𝔫,
Hu = ⟨dδ + ξ, H⟩ for H ∈ 𝔞, dim U(𝔪_{Ξ})u < ∞,
u(gz) = ϵ_{δ}(z)^{−1}u(g) for z ∈ Z(𝔞_{𝔭})}

and its inverse is given by B(G/P_{Ξ}; δ, ξ)_{𝔫} ∋ ϕ ↦ ϕ ⊗ v_{δ}. Here ϵ_{δ} : Z(𝔞_{𝔭}) → {±1} is defined so that τ_{δ}(z)v_{δ} = ϵ_{δ}(z)v_{δ} and the last condition given by ϵ_{δ} is necessary only when M_{Ξ} is not connected, which means G = GL(n, ℝ).

Theorem 2.4. Fix a function f in B(G/P_{Ξ}; δ, ξ)_{𝔫}. Let λ = (λ_{1}, …, λ_{L}) ∈ ℂ^{L} and let Θ = {n_{1} < ⋯ < n_{L} = n} be a sequence of positive integers. Assume

(2.13)  λ_{k} = d(δ, ξ)_{i} if n_{k−1} < i ≤ n_{k},
(2.14) (λ_{i} + n_{i}) − (λ_{j} + n_{j−1}) ∉ {1, …, max{n_{i} − n_{i−1}, n_{j} − n_{j−1}} − 1}
for 1 ≤ i < j ≤ L.

If f satisfies (2.7) or (2.8), then

(2.15)  E_{ii+1}f = 0 for n_{k−1} < i < n_{k} and k = 1, …, L.

3. Representations on symmetric spaces

In this section we review on representations realized on symmetric spaces of non-compact type and their characterization by differential equations. We consider a general real connected semisimple Lie group G. The notation used here is in general restricted only in this section. The statements in this section are known results or at least a reformulation or an easy consequence of known facts (cf. [H3], [K– ], [Ko], [KR], [O1], [Sn1] etc.) and this section can be read without referring to other sections.

Let K be a maximal compact subgroup of G modulo the center of G, θ be the corresponding Cartan involution and 𝔤 = 𝔨 + 𝔭 be the corresponding Cartan decomposition of the Lie algebra 𝔤 of G. Fix a maximal abelian subspace 𝔞_{𝔭} of 𝔭. Let Σ(𝔞_{𝔭}) be the set of the roots defined by the pair (𝔤, 𝔞_{𝔭}) and fix a positive system Σ(𝔞_{𝔭})^{+}. We denote its Weyl group by W (𝔞_{𝔭}) and the fundamental system by Φ(𝔞_{𝔭}), and the half of the sum of the positive roots counting their multiplicities is denoted by ρ. Let G = KAN be the Iwasawa decomposition of G corresponding to Σ(𝔞_{𝔭})^{+}. Then P = MAN is a minimal parabolic subgroup of G. Here M is the centralizer of 𝔞_{𝔭} in K. We denote by 𝔨, 𝔪 and 𝔫 the Lie algebras of K, M and N, respectively.

Let U(𝔤) be the universal enveloping algebra of the complexification 𝔤_{ℂ} of 𝔤, which we identify with the algebra of left invariant differential operators on G. In general, for a subalgebra 𝔩 of 𝔤, we denote by U(𝔩) the universal enveloping algebra of the complexification 𝔩_{ℂ} of 𝔩. Let S(𝔤) be the symmetric algebra of 𝔤_{ℂ}. Then the map sym of symmetrization of S(𝔤) to U(𝔤) defines a K-linear bijection. By the Killing form on 𝔤_{ℂ} we identify the space O(𝔭_{ℂ}) of polynomial functions on the complexification 𝔭_{ℂ} of 𝔭 with the symmetric algebra of 𝔭_{ℂ}. Let O(𝔭)^{K} be the space of all K-invariant polynomials on 𝔭_{ℂ} and H be the space of all K-harmonic polynomials on 𝔭_{ℂ}. Then we have the following K-linear bijection

(3.1)  \begin{matrix}H ⊗ O(𝔭)^{K} ⊗ U(𝔨) & ˜→ U(𝔤) \\ h ⊗ p ⊗ k & ↦ sym(h) ⊗ sym(p) ⊗ k \end{matrix}

because of the Cartan decomposition 𝔤 = 𝔨 + 𝔭 and the K-linear bijection H ⊗ O(𝔭)^{K} ∋ h ⊗ p ↦ hp ∈ O(𝔭) studied by [KR].

We denote by A(G) and B(G) the space of real analytic functions and that of hyperfunctions on G, respectively. Then they are left G-modules by (π_{g}f)(x) = f(g^{−1}x) for g ∈ G and the functions f in the spaces. We write A(G)_{K} for the space of all the K-finite elements of A(G).

By the decomposition

(3.2)  U(𝔤) = 𝔫U(𝔫 + 𝔞_{𝔭}) ⊕ U(𝔞_{𝔭}) ⊕ U(𝔤)𝔨

coming from the Iwasawa decomposition 𝔤 = 𝔨 + 𝔞_{𝔭} + 𝔫, we define D_{𝔞_{𝔭}} ∈ U(𝔞_{𝔭}) for D ∈ U(𝔤) so that D − D_{𝔞_{𝔭}} ∈ 𝔫U(𝔫 + 𝔞_{𝔭}) + U(𝔤)𝔨 and put γ(D) = e^{−ρ}^{∘}D_{𝔞_{𝔭}}^{∘}e^{ρ}. Here e^{ρ} is the function on A defined by e^{ρ}(a) = a^{ρ}.

Note that the kernel of the restriction of γ to the space of all the K-invariants U(𝔤)^{K} of U(𝔤) equals U(𝔤)^{K} ∩U(𝔤)𝔨 and the restriction defines the Harish-Chandra isomorphism

(3.3)  γ̄ : 𝔻(G/K) ≃ U(𝔤)^{K}/(U(𝔤)^{K} ∩ U(𝔤)𝔨) → U(𝔞_{𝔭})^{W}

onto the space U(𝔞_{𝔭})^{W} of all the W (𝔞_{𝔭})-invariants in U(𝔞_{𝔭}). Here 𝔻(G/K) is the ring of invariant differential operators on G/K. Note also that sym(O(𝔭)^{K}) is isomorphic to 𝔻(G/K) through γ and γ̄ as K-modules.

Identifying U(𝔞_{𝔭}) with the space of polynomial functions on the complex dual 𝔞_{ℂ}^{∗} of 𝔞_{𝔭}, we put

(3.4)  γ_{λ}(D) = γ(D)(λ) ∈ ℂ

for λ ∈ 𝔞_{ℂ}^{∗} and D ∈ U(𝔤). Now we define

(3.5)  J_{λ} = U(𝔤)𝔨 + ∑_{p∈O(𝔭)^{K}} U(𝔤)(sym(p) − γ_{λ}(sym(p)))

and

(3.6)  A(G/K; M_{λ}) = {u ∈ A(G); Du = 0 for D ∈ J_{λ}}

and put A(G/K; M_{λ})_{K} = A(G)_{K} ∩ A(G/K; M_{λ}). Here A(G/K; M_{λ}) is naturally a subspace of the space A(G/K) of real analytic functions on G/K because the function in A(G/K; M_{λ}) is right K-invariant. Now we can state our basic theorem.

Theorem 3.1. Fix λ ∈ 𝔞_{ℂ}^{∗} and define a bilinear form

(3.7)  \begin{matrix}H ⊗ A(G/K; M_{λ}) & → ℂ \\ (h, u) & ↦ ⟨h, u⟩ = (sym(h)u)(e) \end{matrix}

and for a subspace V of A(G/K; M_{λ}), put

(3.8)  H(V) = {h ∈ H; ⟨h, u⟩ = 0 for u ∈ V }.

i) The bilinear form ⟨ , ⟩ is K-invariant and non-degenerate.

ii) If V is a subspace of A(G/K; M_{λ})_{K}, then

V = {u ∈ A(G/K; M_{λ})_{K}; ⟨h, u⟩ = 0 for h ∈ H(V)}.

iii) There are natural bijections between the following sets of modules.

𝔙(λ) = {V ⊂ A(G/K; M_{λ}); V is a close subspace of C^{∞}(G) and G-invariant},
𝔙(λ)_{K} = {V_{K} ⊂ A(G/K; M_{λ})_{K}; V_{K} is a 𝔤-invariant subspace},
𝔍(λ) = {J ⊃ J_{λ}; J is a left ideal of U(𝔤)}.

Here the bijections are given by

(3.9)  𝔙(λ) ∋ V ↦ V ∩ A(G)_{K} ∈ 𝔙(λ)_{K},
(3.10)  𝔙(λ)_{K} ∋ V_{K} ↦ J_{λ} + ∑_{p∈H(V_{K})} U(𝔤) sym(p) ∈ 𝔍(λ),
(3.11)  𝔍(λ) ∋ J ↦ {u ∈ A(G); Du = 0 for D ∈ J} ∈ 𝔙(λ).

The G-module (3.12) B(G/P; L_{λ}) = {f ∈ B(G); f(gman) = a^{λ−ρ}f(g) for (g, m, a, n) ∈ G×M ×A×N}

is the space of hyperfunction sections of spherical principal series of G parametrized by λ ∈ 𝔞_{ℂ}^{∗}. Put A(G/P; L_{λ}) = B(G/P; L_{λ}) ∩ A(G). Define the K-fixed element K ×A×N ∋ (k, a, n) ↦ 1_{λ}(kan) = a^{λ−ρ} of A(G/P; L_{λ}) and put P_{λ}(g) = 1_{−λ}(g^{−1}). By the G-invariant bilinear form

(3.13)  \begin{matrix}B(G/P; L_{λ}) × A(G/P; L_{−λ}) & → ℂ \\ (ϕ, f) & ↦ ⟨ϕ, f⟩_{λ} = ∫_{K} ϕ(k)f(k)dk \end{matrix}

with the normalized Haar measure dk on K, we have the Poisson transform

(3.14)  \begin{aligned}P_{λ} : B(G/P; L_{λ}) & → B(G) \\ ϕ & ↦ P_{λ}ϕ(g) = ⟨π_{g^{−1}}ϕ, 1_{−λ}⟩_{λ} = ∫_{K} ϕ(gk)dk \\ & = ∫_{K} ϕ(k)P_{λ}(k^{−1}g)dk. \end{aligned}

Then it is known that the image of P_{λ} is contained in A(G/K; M_{λ}) because DP_{λ} = γ_{λ}(D)P_{λ} for D ∈ U(𝔤)^{K}.

For λ ∈ 𝔞_{ℂ}^{∗} and w ∈ W (𝔞_{𝔭}), let A(λ, w) denote the intertwining operator of B(G/P; L_{λ}) to B(G/P; L_{wλ}), which meromorphically depends on λ and satisfies (A(λ, w)ϕ)(g) = ∫N̄_{w} ϕ(gw̄n_{w})dn_{w} if ϕ is continuous and moreover if Re⟨λ, α⟩ < 0 for all α ∈ Σ(𝔞_{𝔭})^{+}. Here w̄ ∈ K is a representative of w, N̄_{w} = θ(N) ∩ w̄^{−1}Nw̄ and dn_{w} is the Haar measure on N̄_{w} normalized by ∫N̄_{w} 1_{−2ρ}(n_{w})dn_{w} = 1.

For α ∈ Σ(𝔞_{𝔭}) and w ∈ W (𝔞_{𝔭}), we put

(3.15)

Σ(𝔞_{𝔭})_{o}^{+} = \{α ∈ Σ(𝔞_{𝔭})^{+}; \frac{α}{2} ∉ Σ(𝔞_{𝔭})\},
Σ(w) = {α ∈ Σ(𝔞_{𝔭})_{o}^{+}; −wα ∈ Σ(𝔞_{𝔭})^{+}},
e_{α}(λ) = \{Γ (\frac{λ_{α}}{4} + \frac{m_{α}}{4} + \frac{1}{2}) Γ (\frac{λ_{α}}{4} + \frac{m_{α}}{4} + \frac{m_{2α}}{2} )\}^{−1} ,
e(λ) = ∏_{α∈Σ(𝔞_{𝔭})_{o}^{+}} e_{α}(λ), e_{w}(λ) = ∏_{α∈Σ(w)} e_{α}(λ),

d_{α}(λ) = \frac{Γ(m_{α} + m_{2α})}{Γ(\frac{m_{α}}{2} + \frac{m_{2α}}{2} )}2^{1− \frac{λα}{2} − \frac{mα}{2}} \sqrt{π}Γ (\frac{λ_{α}}{2} ) ,
d(λ) = ∏_{α∈Σ(𝔞_{𝔭})_{o}^{+}} d_{α}(λ), d_{w}(λ) = ∏_{α∈Σ(w)} d_{α}(λ),

where m_{α} is the multiplicity of the root α and λ_{α} = 2\frac{⟨λ,α⟩}{⟨α,α⟩}.

It is known that the regularized intertwining operators

(3.16)  Ã(w, λ) = d_{w}(−λ)A(w, λ)

depend holomorphically on λ ∈ 𝔞_{ℂ}^{∗}. Note that Ã(w, λ) may be zero for some λ. The following theorem is the main result in [K– ].

Theorem 3.2. Fix λ ∈ 𝔞_{ℂ}^{∗}. Let w be an element of W with the minimal length which satisfies

(3.17)  Re⟨wλ, α⟩ ≥ 0 for all α ∈ Σ(𝔞_{𝔭})^{+}.

i) P_{λ} defines a topological G-isomorphism of A(G/P; L_{λ}) onto A(G/K; M_{λ}) if and only if e(λ) ≠ 0.

ii) Im A(λ, w) and Im P_{λ} are G-isomorphic under the Poisson transform

(3.18)  P_{wλ} : B(G/P; L_{wλ}) ˜→ A(G/K; M_{λ}).

Remark 3.3. i) Since the image of the intertwining operator A(λ, w) is always closed (cf. [O3, Lemma 7.3]), Theorem 3.1 says that the image of the Poisson transform P_{λ} of B(G/P; L_{λ}) is always characterized as the space of solutions of a system of differential operators corresponding to suitable elements of U(𝔤).

ii) Suppose e(λ) ≠ 0. Let D′(G) and C^{∞}(G) denote the space of distributions and that of C^{∞}-functions on G, respectively. Then (cf. [OS])

P_{λ}(B(G/P; L_{λ}) ∩ D′(G))
= {u ∈ A(G/K; M_{λ}); there exist C and k with |u(g)| ≤ C exp k|g|},
P_{λ}(B(G/P; L_{λ}) ∩ C^{∞}(G))
= {u ∈ A(G/K; M_{λ}); there exist k such that for any D ∈ U(𝔨)
we can choose C_{D} > 0 with |π_{D}u(g)| ≤ C_{D} exp k|g|}.

Here U(𝔨) is the universal enveloping algebra of the complexification of 𝔨 and |g| = ⟨H, H⟩^{\frac{1}{2}} with the Killing form ⟨ , ⟩ if g ∈ K exp HK with H ∈ 𝔞_{𝔭}. Note that U(𝔨) in the above may be replaced by U(𝔤).

Theorem 3.4. i) The map

(3.19)  H ∋ h ↦ π_{sym(h)}1_{λ} ∈ A(G/P; L_{λ})_{K}

is K-linear. It is bijective if and only if e(−λ) ≠ 0.

ii)

(3.20)  γ_{λ}(D) = π_{D}(1_{−λ})(e) for D ∈ U(𝔤).

iii) Putting

(3.21)  H_{λ} = {h ∈ H; γ_{λ}(sym(Ad(k)h)) = 0 for all k ∈ K}

and

(3.22)  J̄_{λ} = J_{λ} + ∑_{h∈H_{λ}} U(𝔤) sym(h),

we have

(3.23)  Im P_{λ} = {u ∈ A(G); Du = 0 for D ∈ J̄_{λ}}.

For a subset Ξ of Φ(𝔞_{𝔭}), let W_{Ξ} be a subgroup of W (𝔞_{𝔭}) generated by reflections with respect to the elements of Ξ and put P_{Ξ} = P W_{Ξ}P. Let P_{Ξ} = M_{Ξ}A_{Ξ}N_{Ξ} be the Langlands decomposition of P_{Ξ} with A_{Ξ} ⊂ A. For an element μ of the complex dual 𝔞_{Ξ,ℂ}^{∗} of the Lie algebra 𝔞_{Ξ} of A_{Ξ}, the space of hyperfunction sections of spherical degenerate series is defined by

(3.24) B(G/P_{Ξ}; L_{Ξ,μ}) = {f ∈ B(G); f(gman) = a^{ρ−μ}f(g)

for (g, m, a, n) ∈ G × M_{Ξ} × A_{Ξ} × N_{Ξ}}.

Then as in the case of the minimal parabolic subgroup, we can define Poisson transform

(3.25) \begin{aligned}P_{Ξ,μ} : B(G/P_{Ξ}; L_{Ξ,μ}) & → B(G) \\ ϕ & ↦ (P_{Ξ,μ}ϕ)(g) = ⟨π_{g^{−1}}ϕ, 1_{Ξ,−μ}⟩_{Ξ,μ} \\ & = ∫_{K} ϕ(gk)dk = ∫_{K} ϕ(k)P_{Ξ,μ}(k^{−1}g)dk. \end{aligned}

Here ⟨ , ⟩_{Ξ,μ} is the bilinear form of B(G/P_{Ξ}; L_{Ξ,μ}) × A(G/P_{Ξ}; L_{Ξ,−μ}) defined on the integral over K, 1_{Ξ,μ}(kman) = a^{ρ−μ} for (k, m, a, n) ∈ K × M_{Ξ} × A_{Ξ} × N_{Ξ} and P_{Ξ,μ}(g) = 1_{Ξ,μ}(g^{−1}).

Now we remark

Lemma 3.5. We have naturally

(3.26)  B(G/P_{Ξ}; L_{Ξ,μ}) ⊂ B(G/P; L_{μ+ρ(Ξ)}),
(3.27)  Im P_{Ξ,μ} = Im P_{μ−ρ(Ξ)}.

Here we identify 𝔞_{𝔭} with its dual by the Killing form and define ρ_{Ξ} = ρ|_{𝔞_{Ξ}} and ρ(Ξ) = ρ − ρ_{Ξ}.

Corollary 3.6. i) P_{Ξ,μ} is injective if e(μ + ρ(Ξ)) ≠ 0. In particular, the Poisson transform P_{ρ_{Ξ}} : B(G/P_{Ξ}) → A(G/K; M_{ρ}) is injective.

ii) If e(−μ + ρ(Ξ))e(μ + ρ(Ξ)) ≠ 0, then B(G/P_{Ξ}; L_{Ξ,μ}) is irreducible.

Remark 3.7. i) The calculation of H_{λ} in (3.21) is equivalent to the determnination of the kernel of P^{γ}(λ) defined by [Ko, §4].

ii) Under the notation in Lemma 3.5

(3.28)  [H_{μ+ρ(Ξ)} : δ] ≥ [Ind_{M}^{K} 1 : δ] − [Ind_{M_{Ξ}(K)}^{K} 1 : δ] for δ ∈ K̂

and the equality holds if and only if P_{Ξ,μ} is injective.

Many statements in this section can be generalized to line bundles (cf. [Sn1]) or vector bundles over G/K but we will omit here.

4. Poisson transforms^{1}

In this section we study the realization of degenerate principal series on the Riemannian symmetric space G/K. For simplicity we mention a result here when G = GL(n, ℝ) but a similar result holds in other cases (cf. [OSn]) where G = GL(n, ℂ) and GL(n, ℍ) and also in the case of sections of a homogeneous line bundles over U(p, q)/U(p) × U(q).

Given a positive integer L and a sequence of positive integers Θ = {n_{1}, …, n_{L}} with 0 = n_{0} < n_{1} < ⋯ < n_{l} = n, we define a parabolic subgroup

(4.1) P_{Θ} = \{p = ( \begin{matrix}g_{1} & & & 0 \\ ∗ & g_{2} \\ ⋮ & ⋮ & ... \\ ∗ & ∗ & ⋯ & g_{L} \end{matrix} ) ∈ GL(n, ℝ); g_{k} ∈ GL(n_{k} − n_{k−1}, ℝ)\} .

Note that the subgroup P_{{1,... ,n}} = {(x_{ij}) ∈ G; x_{ij} = 0 if i < j} is a minimal parabolic subgroup of G, which will be simply denoted by P.

For μ = (μ_{1}, …, μ_{L}) ∈ ℂ^{L}, we define a G-submodule

(4.2)  B(G/P_{Θ}; μ) = {f ∈ B(G); f(xp) = τ_{Θ,μ}(p^{−1})f(x) for p ∈ P_{Θ}}

belonging to degenerate spherical principal series of G, where

(4.3)  τ_{Θ,μ}(p) = | det g_{1}|^{−μ_{1}} ⋯ | det g_{L}|^{−μ_{L}} for p ∈ P_{Θ}

is the character of P_{Θ} under the expression of p in (4.1).

Now we introduce the Poisson kernel

(4.4)

Φ_{Θ}^{μ}(x) = Φ_{n_{1}}(x)^{ξ_{1}} ⋯ Φ_{n_{L}}(x)^{ξ_{L}},
Φ_{m}(x) = ∑_{1≤ν_{1}<···<ν_{m}≤n} |det (x_{iν_{j}})\begin{matrix}1≤i≤m \\ 1≤j≤m \end{matrix}|^{2} ,
2ξ_{k} = (μ_{k} + n_{k−1}) − (μ_{k+1} + n_{k+1}) for k = 1, …, L − 1,
2ξ_{L} = μ_{L} + n_{L−1}.

It is easy to see that

Φ_{Θ}^{μ}(pxk) = τ_{Θ,−μ−2ρ(Θ)}(p)Φ_{Θ}^{μ}(x) for p ∈ P_{Θ} and k ∈ K

and therefore we have a G-homomorphism

(4.5) \begin{aligned}P_{Θ}^{μ} : B(G/P_{Θ}; μ) ∋ f ↦ (P_{Θ}^{μ}f)(x) & = ∫_{K} f(xk)dk = ∫_{K} f(k)Φ_{Θ}^{μ}(k^{−1}x)dk \\ & ∈ B(G/K). \end{aligned}

Note that B(G/P_{Θ}; μ) ⊂ B(G/P; ζ) with ζ = (ζ_{1}, …, ζ_{n}) defined by ζ_{p} = μ_{q} for n_{q−1} < p ≤ n_{q}. The map P_{{1,... ,n}}^{ζ} was studied in [K– ] (cf. §3) and for a generic ζ it is injective and the image is the totality of the real analytic functions u on G/K which satisfy

(4.6)  D_{k}u = χ_{ζ}(D_{k})u for k = 1, …, n.

Here 𝔻(G/K) = ℂ[D_{1}, …, D_{n}] with ord D_{k} = k.

1Most parts in this section are taken from [O4, §5].

Theorem 4.1. i) Any u ∈ Im P_{Θ}^{μ} is real analytic and satisfies

(4.7) D_{IJ}(−μ_{k} − n_{k−1})u = 0 for #I = #J = n − n_{k} + n_{k−1} + 1 and k = 1, …, L

and

(4.8)  \begin{matrix}F_{ij}^{L}(−μ_{1} − n_{0}, …, −μ_{L} − n_{L−1})u = (D_{k} − χ_{ζ}(D_{k}))u = 0 \\ for i, j = 1, …, n and k = 1, …, L − 1. \end{matrix}

ii) If the condition

(4.9)  (μ_{i} + n_{i}) − (μ_{j} + n_{j−1} + 1) ∉ {0, 1, 2, 3, 4, … } for 1 ≤ i < j ≤ L

holds, P_{Θ}^{μ} is a topological G-isomorphism onto the subspace of C^{∞}(G/K) which is the totality of the solutions of the system of equations (4.7) or equivalently (4.8).

Example 4.2. Suppose G = SL(2, ℝ) and P = \{ ( \begin{matrix}∗ & 0 \\ ∗ & ∗ \end{matrix} ) ∈ G\}. Then the natural

action on ℙ_{ℂ}^{1} = (ℂ^{2} − \{0\})/ℂ^{×} ∋ [ \begin{matrix}z_{1} \\ z_{2} \end{matrix} ] is given by the left multiplication. The fixed point group with respect to i = z_{1}/z_{2} equals SO(2) and the symmetric space G/SO(2) is identified with the upper half plane H_{+} = \{ [ \begin{matrix}x+yi \\ 1 \end{matrix} ] ∈ ℙ_{ℂ}^{1}; y > 0\}. Since

( \begin{matrix}\sqrt{y} & \frac{x}{\sqrt{y}} \\ 0 & \frac{1}{\sqrt{y}} \end{matrix} ) [ \begin{matrix}i \\ 1 \end{matrix} ] = [ \sqrt{y}i + \frac{x}{\sqrt{y}}_{\frac{1}{\sqrt{y}}} ] = [ \begin{matrix}x + yi \\ 1 \end{matrix} ] ∈ H_{+} ⊂ ℙ_{ℂ}^{1},

we have Ψ_{1} = (\sqrt{y})^{2} + (\frac{x}{\sqrt{y}} )^{2} = \frac{x^{2}+y^{2}}{y}. Hence if μ_{1} = μ_{2} = 0, we get the usual Poisson kernel Φ_{\{1,2\}}^{μ} = \frac{y}{x^{2}+y^{2}} for H_{+} and Theorem 4.1 says the isomorphism of the space of harmonic functions on H_{+} onto that of hyperfunctions on the circle G/P ≃ ℝ ∪ {∞} ⊂ ℙ_{ℂ}^{1}.

Example 4.3. Suppose Θ = {n}. Then P_{Θ} = G and B(G/P_{Θ}; μ) = ℂ| det x|^{μ} for generic μ ∈ ℂ and we have the equality (cf. Theorem 1.4 iii)):

D_{{1,... ,n}{1,... ,n}}(λ)| det x|^{μ} = (μ + λ)(μ + λ + 1) ⋯ (μ + λ + n − 1)| det x|^{μ}.

If we put λ = 0 and μ = s + 1, this corresponds to Cayley’s formula

det (\frac{∂}{∂x_{ij}} )\begin{matrix}1≤i≤n \\ 1≤j≤n \end{matrix}(det x)^{s+1} = (s + 1)(s + 2) ⋯ (s + n)(det x)^{s}

in view of the Capelli identity. Note that this equality defines the b-function of det x and hence μ is the meromorphic parameter of | det x|^{μ} and its poles are contained in {−1, −2, −3, … } (cf. [B], [Sm]).

Remark 4.4. i) Theorem 4.1 in the case Θ = {1, n} is given in [O1], where it is conjectured that in general there exists a system of operators D satisfying a suitable condition for γ(D) and characterizing the image of the Poisson transform. When G = GL(n, ℝ), the conjecture corresponds to Theorem 4.1 and Corollary 2.1.

ii) If Re μ_{j} < Re μ_{j+1} + 1 for j = 1, …, L − 1, then (4.9) is valid.

iii) In [J1] and [J2] some differential equations characterizing Im P_{Θ}^{0} are given, which are less explicit than ours.

iv) Let w_{Θ} ∈ 𝔖_{n} with Ad(w_{Θ})𝔫̄ ∩ 𝔫 = Lie(P_{Θ}) ∩ 𝔫. As is mentioned in §3, Im P_{Θ}^{μ} = Im P_{{1,... ,n}}^{ζ′} with ζ′_{j} = ζ_{w_{Θ}(j)} + w_{Θ}(j) − j and hence for η ∈ ℂ^{n}, it is interesting to study the system of differential equations characterizing Im P_{{1,... ,n}}^{η}.

5. Hypergeometric functions^{2}

In general, suppose G is a linear reductive Lie group and moreover suppose that we are given three closed subgroups P_{Θ}, Q_{1} and Q_{2} of G and their finite dimensional representations (τ_{Θ}, V_{Θ}), (τ_{1}, V_{1}) and (τ_{2}, V_{2}), respectively, which satisfy the conditions

(5.1) P_{Θ} is a parabolic subgroup of G, (5.2) the double cosets Q_{j}\G/P_{Θ} have open cosets for j = 1 and 2.

Define the degenerate principal series as in §2:

B(G/P_{Θ}; τ_{Θ}) = {ϕ(g) ∈ B(G) ⊗ V_{Θ}; ϕ(gp) = τ_{Θ}(p^{−1})ϕ(g) for p ∈ P_{Θ}}.

Let P_{Θ} = L_{Θ}N_{Θ} be a Levi decomposition of P_{Θ}. Fix a Cartan involution θ of G with θ(L_{Θ}) = L_{Θ}. Then K = {g ∈ G; θ(g) = g} is a maximal compact subgroup of G. Let V_{Θ}^{∗} be the dual space of V_{Θ} and let τ_{Θ}^{∗} be the representation of P_{Θ} on V_{Θ}^{∗} such that

(5.3) B(G/P_{Θ}; τ_{Θ}) × A(G/P_{Θ}; τ_{Θ}^{∗}) ∋ (ϕ, ψ) ↦ ⟨ϕ, ψ⟩_{Θ} = ∫_{K}⟨ϕ(k), ψ(k)⟩dk ∈ ℂ

defines a G-invariant bilinear form. Note that (τ_{Θ}^{∗}, V_{Θ}^{∗}) is the tensor product of the contragredient representation of τ_{Θ} and the character det(Ad) of P_{Θ} on 𝔫_{Θ} = Lie(N_{Θ}), and that if ⟨ϕ(k), ψ(k)⟩ is an integrable functions on K,

(5.4)  ⟨ϕ, ψ⟩_{Θ} = ∫N̄_{Θ}⟨ϕ(n), ψ(n)⟩dn

with a suitably normalized Haar measure dn on N̄_{Θ} = θ(N_{Θ}).

Definition 5.1. For given functions ϕ ∈ B(G/P_{Θ}; τ_{Θ})⊗V_{1} and ψ ∈ B(G/P_{Θ}; τ_{Θ}^{∗})⊗ V_{2} satisfying

(5.5)  \begin{aligned}ϕ(q_{1}x) & = τ_{1}(q_{1})ϕ(x) for q_{1} ∈ Q_{1}, \\ ψ(q_{2}x) & = τ_{2}(q_{2})ψ(x) for q_{2} ∈ Q_{2}, \end{aligned}

we call a V_{1} ⊗ V_{2}-valued function

(5.6)  Φ_{ϕ,ψ}(x) = ∫_{K}⟨ϕ(xk), ψ(k)⟩dk

on G a hypergeometric function.

By the G-invariance of the bilinear form we have

(5.7)  Φ_{ϕ,ψ}(q_{1}xq_{2}) = τ_{1}(q_{1})τ_{2}(q_{2}^{−1})f(x) for (q_{1}, q_{2}) ∈ Q_{1} × Q_{2}

and for elements D of the universal enveloping algebra U(𝔤) of 𝔤_{ℂ} = ℂ ⊗ Lie(G)

(5.8)  π_{D}(Φ_{ϕ,ψ}) = 0 if π_{D}(ϕ) = 0.

Note that the following map defines a G-homomorphism:

(5.9)  B(G/P_{Θ}; τ_{Θ}) ∋ f ↦ (T_{ψ}f)(x) = ∫_{K}⟨f(xk), ψ(k)⟩dk ∈ B(G/Q_{2}; τ_{2}).

2This section is extracted from [O4, §6].

Example 5.2. Suppose P_{Θ} is a minimal parabolic subgroup. If Q_{1} and Q_{2} are a maximal compact subgroup, the integral representations of the corresponding hypergeometric functions are Eisenstein integrals. In particular, if τ_{1} and τ_{2} are trivial representations, we have the integral representations of zonal spherical functions.

Now we will consider the case when G = GL(n, ℝ) and furthermore P_{Θ} and Q_{2} are parabolic subgroups. In particular, we examine the case when they are maximal, namely, Θ = {ℓ, n} and Q_{2} = P_{{k,n}}. Suppose ℓ < k < n and n = mℓ with a positive integer m satisfying m > 1. To assure the existence of the nontrivial intertwining operator (5.9), we assume τ_{Θ} = τ_{μ,0} with μ = (k, 0) ∈ ℂ^{2}. In this case, the inter-

twining operator A(λ, w) in §3 with w = ( \begin{matrix}1 & 2 & ··· & ℓ & ℓ+1 & ℓ+2 & ··· & k & k+1 & ··· & n \\ k−ℓ+1 & k−ℓ+2 & ··· & k & 1 & 2 & ··· & k−ℓ & k+1 & ··· & n \end{matrix} )

converges for any continuous function f ∈ B(G/P_{Θ}; τ_{Θ}). Note that the corresponding kernel function is a measure whose support is the compact double coset in Q_{2}\G/P_{Θ} and N̄_{w} = {(x_{ij}) ∈ G; x_{ij} = δ_{ij} if i ≥ ℓ or j ≤ ℓ or j > k}.

Lemma 5.3. The intertwining operator A(λ, w) defines a G-homomorphism

(5.10)  T_{w} : B(G/P_{{ℓ,n}}; (k, 0)) → B(G/P_{{k,n}}; (ℓ, 0)).
ϕ ↦ ∫_{O(k)×I_{n−k}} ϕ(gk)dk

Through the anti-automorphism G ∋ g ↦ g^{−1} ∈ G, B(G/P_{{ℓ,n}}; (k, 0)) is canonically identified with the space of hyperfunctions ϕ on the ℓn-dimensional manifold

(5.11)  M(ℓ, n) = {(t_{ij}) = ( . . . . . ._{t_{ℓ1} ··· t_{ℓn}}^{t_{11} ··· t_{1n}} ); t_{ij} ∈ ℝ and rank (t_{ij}) = ℓ}

which satisfy

(5.12)  ϕ(g(t_{ij})) = | det g|^{−k}ϕ((t_{ij})) for g ∈ GL(ℓ, ℝ).

Similarly B(G/P_{{k,n}}; (ℓ, 0)) is canonically identified with the space of hyperfunctions Φ on the kn-dimensional manifold

(5.13)  M(k, n) = {(y_{ij}) = ( . . . . . . ._{y_{k1} ··· y_{kn}}^{y_{11} ··· y_{1n}} ); y_{ij} ∈ ℝ and rank (y_{ij}) = k}

which satisfy

(5.14)  Φ(g(y_{ij})) = | det g|^{−ℓ}Φ((y_{ij})) for g ∈ GL(k, ℝ).

Denoting GL(ℓ, ℝ)_{+} = {g ∈ GL(ℓ, ℝ); det g > 0}, we choose Q_{1} = GL(ℓ, ℝ)_{+} × ⋯ × GL(ℓ, ℝ)_{+} ⊂ GL(n, ℝ) and define τ_{1} by

τ_{1}(g) = (det g_{1})^{−α_{1}} ⋯ (det g_{m})^{−α_{m}} for g = (g_{1}, …, g_{m}) ∈ Q_{1},

where α = (α_{1}, …, α_{m}) ∈ ℂ^{m} with the condition

(5.15)  α_{1} + ⋯ + α_{m} + k = 0.

Let ϵ = (ϵ_{1}, …, ϵ_{m}) ∈ {1, −1}^{m}. Using a function

(5.16)  |s|_{ϵ_{p}}^{α_{p}} = \{ \begin{matrix}|s|^{α_{p}} & if ϵ_{p}s > 0, \\ 0 & if ϵ_{p}s ≤ 0 \end{matrix}

on ℝ, we give a function

(5.17) ϕ(t) = \frac{1}{2}( ∏_{p=1}^{m} | det (t_{ij})\begin{matrix}1≤i≤ℓ \\ (p−1)ℓ<j≤pℓ \end{matrix}|_{ϵ_{p}}^{α_{p}} + ∏_{p=1}^{m} | det (t_{ij})\begin{matrix}1≤i≤ℓ \\ (p−1)ℓ<j≤pℓ \end{matrix}|_{−ϵ_{p}}^{α_{p}})

in Definition 5.1, which belongs to B(G/P_{{ℓ,n}}; (k, 0)).

Put x_{w(i)w(j)} = y_{ij} for i = 1, …, ℓ and j = 1, …, n. Then in GL(n, ℝ)

(5.18) ( \begin{matrix}t_{11} & & & t_{1k} \\ . & . & . & . & . & . & . \\ t_{ℓ1} & & & t_{ℓk} \\ & & 1 \\ . & . & . \\ & & & & 1 \end{matrix} ) w^{−1} ( \begin{matrix}y_{11} & ··· & y_{1n} \\ . & . & . & . & . & . & . & . \\ y_{k1} & ··· & y_{kn} \\ . & . & . & . & . & . & . & . \\ . & . & . & . & . & . & . & . \\ . & . & . & . & . & . & . & . \end{matrix} ) = ( \begin{matrix}∑_{ν=1}^{k} t_{1ν}x_{ν1} & ··· & ∑_{ν=1}^{k} t_{1ν}x_{νn} \\ . & . & . & . & . & . & . & . & . & . & . & . & . & . & . & . & . & . & . & . & . & . & . \\ ∑_{ν=1}^{k} t_{ℓν}x_{ν1} & ··· & ∑_{ν=1}^{k} t_{ℓν}x_{νn} \\ . & . & . & . & . & . & . & . & . & . & . & . & . & . & . & . & . & . & . & . & . & . & . \\ . & . & . & . & . & . & . & . & . & . & . & . & . & . & . & . & . & . & . & . & . & . & . \\ . & . & . & . & . & . & . & . & . & . & . & . & . & . & . & . & . & . & . & . & . & . & . \end{matrix} ) w^{−1}

and we have our hypergeometric function

(5.19) Φ(α, ϵ; x) = ∫_{ℝ^{ℓ(k−ℓ)}} ∏_{p=1}^{m} | det ( ∑_{ν=1}^{k} t_{iν}x_{νj})\begin{matrix}1≤i≤ℓ \\ (p−1)ℓ<j≤pℓ \end{matrix}|_{ϵ_{p}}^{α_{p}} ∏_{1≤i≤ℓ<j≤k} dt_{ij}

with the convention

(5.20)  t_{ij} = δ_{ij} if 1 ≤ j ≤ ℓ.

Theorem 5.4. The hypergeometric function Φ(α, ϵ; x) on M(k, n) = {(x_{ij}); x_{ij} ∈ ℝ, 1 ≤ i ≤ k, 1 ≤ j ≤ n, rank(x_{ij}) = k} satisfies the following equations.

i) A right GL(ℓ, ℝ)_{+} × ⋯ × GL(ℓ, ℝ)_{+}-invariance:

Φ(xg) = ( ∏_{p=1}^{m}(det g_{p})^{α_{p}})Φ(x) for g = g_{1} ⊗ ⋯ ⊗ g_{m} with g_{p} ∈ GL(ℓ, ℝ)_{+}.

ii) A left GL(k, ℝ)-invariance:

Φ(gx) = | det g|^{−ℓ}Φ(x) for g ∈ GL(k, ℝ).

iii) Generalized Capelli operators:

(5.21)  \begin{matrix}det (\frac{∂}{∂x_{i_{μ}j_{ν}}} )\begin{matrix}1≤μ≤ℓ+1 \\ 1≤ν≤ℓ+1 \end{matrix}Φ(x) = 0 \\ for 1 ≤ i_{1} < ⋯ < i_{ℓ+1} ≤ k and 1 ≤ j_{1} < ⋯ < j_{ℓ+1} ≤ n. \end{matrix}

The two invariances in Theorem 5.4 are infinitesimally as follows:

(5.22) ∑_{i=1}^{k} x_{i,μ+ℓp} \frac{∂Φ}{∂x_{i,ν+ℓp}} = α_{p}δ_{μν}Φ for 1 ≤ μ ≤ ℓ, 1 ≤ ν ≤ ℓ, 0 ≤ p < m,

(5.23)  ∑_{ν=1}^{n} x_{iν} \frac{∂Φ}{∂x_{jν}} = −ℓδ_{ij}Φ for 1 ≤ i ≤ k, 1 ≤ j ≤ k.

Remark 5.5. i) Note that GL(n, ℝ)/P_{{ℓ,n}} ≃ O(n)/O(ℓ) × O(n − ℓ). Hence the definition of Φ(α, ϵ; x) is an integration of a function on O(n)/O(ℓ) × O(n − ℓ) over its submanifold O(k)/O(ℓ) × O(k − ℓ), which is a Radon transform on real Grassmannians (cf. [H4, Chap. 1]).

ii) Suppose ℓ = 1. The integration can be rewritten as

(5.24)  Φ(α, ϵ; x) = C ∫_{t_{1}^{2}+···+t_{k}^{2}=1} ∏_{j=1}^{n} | ∑_{p=1}^{k} t_{p}x_{pj}|_{ϵ_{p}}^{α_{p}} ω̂

with a suitable constant C and

(5.25)  ω̂ = ∑_{p=1}^{k}(−1)^{p+1}t_{p}dt_{1} ∧ dt_{2} ∧ ⋯ ∧ dt_{p−1} ∧ dt_{p+1} ∧ ⋯ ∧ dt_{k}.

This integral representation coincides with the one given in [G] and the corresponding equations in Theorem 5.4 with ℓ = 1 are same as in [GG]. This hypergeometric function is also studied by [A].

For the analysis of hypergeometric functions in the case when G = P_{{ℓ,n}} and Q_{2} = P_{{k,n}}, the following theorem is essential.

Theorem 5.6. The intertwining operator in Lemma 5.3 is the topological Gisomorphism onto the solution space V_{k,n}^{ℓ} of the system (5.21) under our realization using (5.13) and (5.14) if

(5.26)  0 < ℓ < k < n and ℓ + k < n.

Corollary 5.7. Let Q be a closed subgroup of GL(n, ℝ) and let (τ, V) be a finite dimensional representation of Q. If (5.26) holds, the integral transformation R : ϕ ↦ ∫_{SO(k)} ϕ(kx)dk is a bijection between S(ℓ, n; τ) and S(k, n; ℓ, τ). Here S(ℓ, n; τ) is the space of V -valued hyperfunctions ϕ on M(ℓ, n) satisfying (5.12) and

(5.27)  ϕ(tg) = τ(g^{−1})ϕ(t) for g ∈ Q.

Moreover S(k, n; ℓ, τ) is the space of of V -valued hyperfunctions Φ(x) on M(k, n) satisfying the equations given in Theorem 5.4 ii) and iii) and

(5.28)  Φ(xg) = τ(g^{−1})Φ(x) for g ∈ Q.

In this corollary ‘hyperfunctions’ can be replaced by ‘Schwartz’s distributions’ or ‘C^{∞}-functions’ or ‘real analytic functions’, which is clear by our way of the proof.

Lastly we give other examples of Q_{1} for the same P_{Θ} and Q_{2}.

Example 5.9. For A = (A_{ij}) ∈ 𝔤𝔩(n, ℝ) with n = mℓ and for μ = 1, …, m and ν = 1, …, m, put

𝔲_{1} = {A ∈ 𝔤𝔩(n, ℝ); M_{μν}(A) = 0 if μ ≠ ν − 1},
𝔲_{2} = {A ∈ 𝔤𝔩(n, ℝ); M_{μν}(A) = 0 if μ > ν,
M_{ν,ν+i}(A) = M_{ν+1,ν+i+1}(A) for i = 1, …, m − 2 and ν = 1, …, m − i − 1},
M_{μν}(A) = (A_{(μ−1)ℓ+i,(ν−1)ℓ+j})\begin{matrix}1≤i≤ℓ \\ 1≤j≤ℓ \end{matrix}.

If Q_{1} equals the nilpotent group exp 𝔲_{1} or exp 𝔲_{2}, condition (5.2) is valid and we can define hypergeometric functions with respect to the character of Q_{1} which has m−1 continuous parameters. When ℓ = 1, the hypergeometric functions correspond to those discussed in [GRS] and [KHT].

References

[A] K. Aomoto, On the structure of integrals of power products of linear functions, Sc. Papers Coll. Gen Education, Univ. of Tokyo 27 (1977), 49–61.

[BV] N. Berline and M. Vergne, Equations du Hua et noyau de Poisson, Lect. Notes in Math. 880 (1981), 1–51, Springer.

[B] I. N. Bernstein, The analytic continuation of generalized functions with respect to a parameter, Funct. Anal. Appl. 6 (1972), 273–285.

[C1] A. Capelli, Üeber die Zurückführung der Cayley’schen Operation Ω auf gewöhnliche Polar-Operationen, Math. Ann. 29 (1887), 331–338.

[C2] , Sur les opérations dans la théorie des formes algébriques, Math. Ann. 37 (1890), 1–37.

[F] P. Funk, Über eine geometrische Anwendung der Abelschen Intergralgleichung, Math. Ann. 77 (1916), 129–135.

[G] I. M. Gelfand, General theory of hypergeometric functions, Soviet Math. Dokl. 33 (1986), 573–577.

[GG] I. M. Gelfand and S. I. Gelfand, Generalized hypergeometric equations, Soviet Math. Dokl 33 (1986), 643–646.

[GGR] I. M. Gelfand, M. I. Graev and R. Rosu, The problem of integral geometry and intertwining operators for a pair of real Grassmannian manifolds, J. Operator Theory 12 (1984), 339– 383.

[GRS] I. M. Gelfand, V. S. Retakh and V. V. Serganova, Generalized airy functions, Schubert cells and Jordan groups, Dokl. Akad. Nauk SSSR 298 (1988), 17–21.

[GK] S. G. Gindikin and F. I. Karpelevič, Plancherel measure of Riemannian symmetric spaces of non-positive curvature, Funct. Anal. Appl. 1 (1967), 28–32.

[Go1] F. Gonzalez, Invariant differential operators and the range of the Radon D-plane transform, Math. Ann. 287 (1990), 627–635.

[Go2] , On the Range of the Radon transform of Grassmann manifolds, preprint, to appear in J. Funct. Anal.

[Gr1] E. Grinberg, On the images of Radon transforms, Duke Math. J. 52 (1985), 939–972.

[Gr2] , Radon transforms on higher rank Grassmanians, J. of Diff. Geom. 24 (1986), 53–68.

[H1] S. Helgason, Differential operators on homogeneous spaces, Acta Math. 102 (1959), 239-299.

[H2] , A duality for symmetric spaces with applications to group representations, Advances in Math. 5 (1970), 1–154.

[H3] , A duality for symmetric spaces with applications to group representations, II, Differential equations and Eigenspace representations, Advances in Math. 22 (1976), 187– 219.

[H4] , Geometric Analysis on Symmetric Spaces, Mathematical Surveys and Monographs, Vol. 39, American Mathematical Society, 1994.

[Jf] F. John, Bestimmung einer Funktion aus ihren Integralen über gewisse Mannigfaltigkeiten, Math. Ann. 109 (1934), 488–520.

[J1] K. D. Johnson, Generalized Hua operators and parabolic subgroups, The case of SL(n, ℂ) and SL(n, ℝ), Trans. A. M. S. 281 (1984), 417–429.

[J2] , Generalized Hua operators and parabolic subgroups, Ann. of Math. 120 (1984), 477–495.

[Ka] T. Kakehi, Range characterization of Radon transforms on S^{n} and P^{n}R, J. Math. Kyoto Univ. 33 (1993), 215–228.

[K–] M. Kashiwara, A. Kowata, K. Minemura, K. Okamoto, T. Oshima and M. Tanaka, Eigenfunctions of invariant differential operators on a symmetric space, Ann. of Math. 107 (1978), 1–39.

[KO] M. Kashiwara and T. Oshima, Systems of differential equations with regular singularities and their boundary value problems, Ann. of Math. 106 (1977), 145–200.

[KHT] H. Kimura, Y. Haraoka and K. Takano, The generalized confluent hypergeometric functions, Proc. Japan Acad. 68A (1992), 290–295.

[Kr] A. Korányi, Hua-type integrals, hypergeometric functions and symmetric polynomials, International symposium in memory of Hua Loo Keng, vol. II, Analysis, Science Press, Beijing and Springer-Verlag, Berlin, 1991, pp.169–180.

[KM] A. Korányi and P. Malliavin, Poisson formula and compound diffusion associated to an overdetermined elliptic system on the Siegel half plane of rank two, Acta Math. 134 (1975), 185–209.

[Ko] B. Kostant, On the existence and irreducibilities of certain series of representations, Bull. Amer. Math. Soc 75 (1969), 627–642.

[KR] B. Kostant and S. Rallis, Orbits and Lie group representations associated to symmetric spaces, Amer. J. Math. 93 (1971), 753-809.

[L] M. Lassale, Les équations de Hua d’un domaine borneé symétrique de type tube, Invent. math. 77 (1984), 129–161.

[O1] T. Oshima, Boundary value problems for various boundaries of symmetric spaces, RIMS KôKyûroku, Kyoto Univ. 281 (1976), 211–226. (Japanese)

[O2] , A realization of Riemannian symmetric spaces, J. Math. Soc. Japan 30 (1978), 117–132.

[O3] , Multiplicities of representations on homogeneous spaces, Abstracts from the conference ‘Harmonic Analysis on Lie Groups held at Sandbjerg Gods’, August 26-30, 1991, edited by N. V. Pedersen, Mathematical Institute, Copenhagen University, 1991.

[O4] , Generalized Capelli identities and boundary value problems for GL(n), preprint, 1995, UTMS 95-44, Dept. of Mathematical Sciences, Univ. of Tokyo (to appear in the Proceeding of symposiums on Algebraic Analysis held at Katata and RIMS, Kyoto Univ., June/July 1995, Word Scientific).

[OS] T. Oshima and J. Sekiguchi, Eigenspaces of invariant differential operators on an affine symmetric space, Invent. Math. 57 (1980), 1–81.

[OSn] T. Oshima and N. Shimeno, in preparation.

[R] J. Radon, Über die Bestimmung von Funktionen durch ihre Integralwerte längs gewisser Mannigfaltigkeiten, Ber. Berh. Sächs. Akad. Wiss. Leipzig. Math.-Nat. kl. 69 (1917), 262– 277.

[Si] I. Satake, Linear Algebra, Marcel Dekker, 1975.

[Sm] M. Sato, The theory of prehomogeneous vector spaces, notes by T. Shintani, Sûgaku no Ayumi 15-1 (1970), 181–227. (Japanese)

[Sh] H. Sekiguchi, The Penrose transform for certain non-compact homogeneous manifolds of U(n, n), PhD thesis presented to Univ. of Tokyo, 1995.

[Sn1] N. Shimeno, Eigenspaces of invariant differential operators on a homogeneous line bundle on a Riemannian symmetric space, J. Fac. Sci. Univ. Tokyo, Sect. 1A 37 (1990), 201–234.

[Sn2] , Boundary value problems for the Shilov boundary of a bounded symmetric domain of tube type, preprint, to appear in J. Funct. Anal..

[We] H. Weyl, The Classical Groups, 2nd ed., Princeton University Press, 1946.