Sampling Reconstruction of Stochastic Signals– the Roots in the Fifties

In this article we are interested in the beginnings and the development of the sampling theory in signal analysis of stochastic signals, locating these in the early fifties. Besides the most important papers by Parzen (1956), Balakrishnan (1957), Belyaev (1959), and Lloyd (1959) we expose and report on few other interesting articles not widely known, giving an overview of the topic. Zusammenfassung: In dieser Arbeit sind wir an den Anfängen und den ersten Entwicklungen auf dem Gebiet der Stichprobentheorie in der Signal Analyse von Stochastischen Prozessen interessiert, wobei wir diesen An-fang den frühen Fünfziger Jahren zuordnen. Neben den wichtigsten Arbeiten berichten wirüber einige weitere interessante Artikel, die relativ unbekannt sind, und geben dadurch einen¨Uberblicküber dieses Thema.


Introduction
A communication system is a system which consists of essentially five parts: an information source (e.g., in telephony, a human voice), a transmitter (a telephone), the channel (a pair of wires or a band of radio frequency), the receiver (a telephone), and the destination (a human ear).
The communication process is apt to errors.For example, the wire might be damaged so the transfer of the signal can be interrupted by some kind of noise.If this happens, the receiver should be able to reconstruct the original, or almost surely original signal.
In the modelling of a communication procedure and the information transmission the role of the sampling theorem in crucial.Let us consider a class of functions which is defined on some common domain.The main question is: can we find a discrete subset Λ of this domain such that every member of the class is determined uniquely by the collection of values that it takes on Λ and, if this is the case, how can we recover such a function completely using these "sampled values" only?This is the problem of sampling (analogue to digital transform) and reconstruction (digital to analogue transform).
For a deterministic band-limited function X(t) (signal in the sequel) on the segment [−π/∆, π/∆], ∆ > 0, (band-limited means that any component of this function which has a frequency above a certain bound should be zero, or at least sufficiently close to zero to allow us to neglect its influence on the resulting reconstruction) representable in the form of a Fourier or Fourier-Stieltjes integral an interpolation formula was obtained by Whittaker (1915).If X is stationary in the wide sense, Shannon's sampling Theorem reads as follows: (still in sense of formal equality).We point out that Whittaker is calling the (1) like series "cardinal series".Independently of the mathematicians' works, the same formula was obtained by Kotel'nikov (1933), who pointed out its great importance for communication theory, and by Shannon (1948), who deals with the problems that can occur in some communication system.For this reason the above mentioned formula in the Russian literature is called Kotel'nikov formula whereas in the Western literature it is known as the Shannon Formula or the Shannon Principle.
We would like to point out the following: At the end of the 1940s all prerequisites for the extension of sampling theorem to stochastic weakly stationary processes were developed-the Nyquist-Shannon or Sampling Principle (the band-limited signal X(t) can be perfectly reconstructed linearly by means of its values sampled at equidistant values, if X(t) is band-limited to less then half of the sampling rate), the spectral representation formula for wide sense stationary stochastic processes, and the Karhunen theorem (Yaglom, 1987, pp. 447-448).So, the stochastic sampling theorem could be expected soon.Therefore, except Belyaev's oversampling paper, all other mentioned results follow the described research direction (Oswald, Parzen, Balakrishnan, Lloyd).

Early Sampling Results
Here and in what follows we are interested in sampling of stochastic signals in its early phase.In case where X(t) is a stationary random process the interpolation problem of the best approximation of the random variable X(t) by the mean square limit was studied by Yaglom (1949).The general mean squared error is expressed in terms of the spectral distribution function F (λ) of the process X(t) is given in there.This expression implies that σ 2 ∆ = 0 for any t if the distance between any two B. Draščić 67 points of the frequency spectrum of X(t) differs from any multiple of 2π/∆.Obviously, the last condition is satisfied, if the entire frequency spectrum Λ of X(t) lies within the segment −π/∆ < λ < π/∆.The same result also implies that the above mentioned equality also holds, if all the points of the frequency spectrum belong to the closed interval −π/∆ ≤ λ ≤ π/∆, but its two endpoints do not belong simultaneously to the discrete spectrum of the process.One of the earliest articles touching the sampling of band-limited stochastic signals is Oswald's Signaux aleatoires a spectre limité, published in 1951.There he considers the band-limited signals and their transformations and takes the sampling principle as true without giving a proof (even Shannon gave only a nice construction of the principle but without exact proof): "... On sait que tout signal X(t) dont le spectre occupe la bande de fréquences (−F/2, F/2) est entièremant caractérisé par les valeurs ... Un tel signal X(t) est aléatoire, lorsque les ordonnées X obéissent à une loi de probabilié quelconque, de densité p(X k ).Dans le cas plus simple, celui d'un processus stationnaire, p(X k ) = p(X) ne dépend pas de l'instant t k considéré..." (Oswald, 1951) The first published proof of the Shannon formula for stationary process is given by Balakrisnan in 1957, and later on, other proofs of the same theorem were proposed (for example Lloyd, 1959, Belyaev, 1959, and Beutler, 1961, who assumed that only one of the end points of the segment −π/∆ ≤ ω ≤ π/∆ may be a discontinuity point of F (λ), since F (λ) is left-continuous).
A quite simple proof of the Shannon principle is given in Parzen (1956) and is nicely presented in the book Rozanov (1967).Parzen's famous article is a Technical Report of the Stanford University and was in fact never published.This paper was submitted approximately in the same time when Balakrishnan submitted his pioneering article, at the end of 1956.He gives "... a method of proof of the sampling theorem, both for the cases where the (support, op.au.) interval I is centered at the origin and where it is not ... and yields several useful generalizations to functions of several variables and random functions..." (Parzen, 1956) This shows that the publishment of the Balakrishnan's proof started an avalanche of the development of the sampling theory, which later led to a great number of published papers.Sampling theory is still of great interest to mathematicians all over the world and there is still a great number of open problems that can be discussed.

The Shannon Principle
Considering the signal reconstruction, Shannon introduced the above mentioned principle and announced the proof to be given in a subsequent article.In this second article (Shannon, 1949) he gave a very nice construction of the sampling series like (1), but still without a formal proof (for details, see (Shannon, 1949)).The formal proof was given eight years later in the famous article of Balakrishnan (1957).
Before we state the Shannon Principle, let us introduce same definitions.Assume that we observe a characteristic X of some physical system at time t of some time interval T .More specifically, let the value of X at time t be not given in advance, but be described by a random variate.Then the collection of all random values X(t) for t ∈ T can be considered as a random variable changing over time, i.e. we can say that X(t), t ∈ T , is a random function of time.In this case we also say that X(t), t ∈ T is a random or stochastic process.Here we are interested in stationary stochastic processes.We say that a stochastic process is stationary in the "wide sense" or second order stationary, if If the autocorrelation function ρ of the process X(t) satisfies then F is called the spectral distribution function and the spectral density function is the density function which corresponds to the distribution function f , whenever a density exists.
The following formulation of the sampling theorem is due to Balakrishnan (1957): Theorem 1 Let {X(t), −∞ < t < +∞} be a real or complex-valued stochastic process, stationary in the wide sense possessing a spectral density which vanishes outside the interval of angular frequency [−2πW, 2πW ], W > 0. Then X(t) has the representation: for every t, where lim stands for the limit in the mean square sense, i.e.
The idea of the proof is that the right hand side of (3) is the best linear estimate in the mean square sense of X(t) in terms of X(n/2W ), such that we have zero estimation error and the sampling principle per se for non-random functions is applied to the covariance function B(t) of the process to yield We say that a complex function is an entire function if it is analytical at all finite points of the complex plane, and the random process X(t) is called analytic in the region D if almost all sample functions of its components can be analytically continued in D. The necessary and sufficient conditions for the analyticity of a process are that if the covariance B(t, s) is an analytic function of two variables in a neighborhood of the point (t 0 , t 0 ), then the random process is analytic in a neighborhood of this point.
It can be also shown that for the Gaussian process this condition is necessary and sufficient (see Belyaev, 1959).
For the covariance of a stationary process to be analytic for |t| ≤ r it is necessary and sufficient that where F (λ) is a spectral distribution function.
For a band-limited random processes it also holds that if the covariance B(t) of a stationary process is an entire function of exponential type with exponent not exceeding σ, then almost all the sample functions are entire functions of exponential type with exponent not exceeding σ.
The proofs of the above stated claims can be found in e.g.(Belyaev, 1959).
Theorem 3 Let {X(t), −∞ < t < +∞} be a zero-mean weakly stationary stochastic process, band-limited to w > 0 whose covariance function has the form Then for almost all sample functions the formula is valid, where w > w is any fixed number.
In the proving procedure Belyaev took the idea of the so-called "oversampling procedure".Namely, if the stochastic signal is band-limited to some bandwidth w, then it is bandlimited to any bandwidth w > w.We define the truncated Kotel'nikov series The mean-square truncation error n (X, t) attains the upper bound and by the Borel-Cantelli lemma the convergence in (4) follows in the almost sure sense.
The next very important person in the development of the sampling reconstruction of stochastic signals is Lloyd (1959).In a very serious and demanding article he gives a necessary and sufficient condition on the random spectral measure for a weakly stationary stochastic process, which allows a process to be exactly restored from its samples.Here we only consider his main theorem.Then the sampling series converges to X(t) with probability 1, i.e.

Theorem 4
Suppose the spectral distribution of process X(t) has an open support Λ whose translates {Λ − nh −1 , −∞ < n < ∞} are mutually disjoint.Suppose further that there exist numbers a, b such that a > 1/2 , b > 0 , a + b/2