What the Coherent DSP Actually Does

Scope

This article walks the coherent receiver's digital signal processing chain in order, explaining what each stage corrects and why it is needed, then shows how the same chain, selectively shortened, produces the reduced-power coherent-lite designs now filling the short-reach tiers. It assumes the optical front end covered in coherent detection and the optical hybrid; the DSP itself has a summary in coherent DSP. Direct-detection PAM4 links, scored by TDECQ, are a different world and out of scope.

What the front end hands over

A coherent receiver mixes the incoming signal with a local-oscillator laser in a 90-degree optical hybrid, on each of two polarizations, and digitizes the four photocurrents. The result is a faithful digital copy of the received optical field: amplitude and phase, both polarizations, at full bandwidth. That is the point of coherent detection, and it is also the problem, because the field as received is corrupted in four independent ways that the DSP must undo before a single bit can be read. Everything below happens in electronics, at the line rate, in a chip that is the coherent module's largest consumer of power.

Chromatic dispersion compensation

Fiber spreads a pulse because different wavelengths travel at different speeds, and over a long link the accumulated chromatic dispersion smears symbols across many of their neighbors. In a coherent system this is trivially reversible, and the reason is worth stating plainly: dispersion applies a known phase to each frequency, quadratic in frequency and linear in distance, and nothing else. Because the receiver has the full field, it can apply the exact inverse phase with a static digital filter and recover the original pulses. This single capability is why coherent links deleted dispersion-compensating fiber from the network: a spool of specialty fiber at every amplifier hut became a fixed filter in a chip. The filter is long for long links, hundreds of taps for hundreds of kilometers, and its length is one of the terms that makes the DSP big.

Adaptive equalization and polarization demultiplexing

The two polarizations of the fiber arrive mixed together, rotated by an angle that drifts as the fiber is disturbed, and further spread by polarization-mode dispersion. A bank of adaptive filters, arranged as a two-by-two matrix and updated continuously by an algorithm such as the constant-modulus or a decision-directed rule, separates the two polarizations and tracks their drift in real time. This stage also mops up residual chromatic dispersion and the link's other slowly varying linear distortions. Unlike the dispersion filter, it is adaptive, because what it corrects changes on millisecond timescales as the fiber moves.

Carrier recovery

The signal laser and the local oscillator are independent, so their frequencies differ by an unknown offset and their phases wander with the linewidth of each. To read the phase of a QAM constellation, the receiver must estimate and remove both. Frequency-offset estimation locks the two lasers to a common reference in software; carrier-phase estimation then tracks the residual phase noise symbol by symbol. Carrier recovery is where laser linewidth turns into a system requirement: a noisier laser forces a faster, more aggressive phase tracker and eventually limits the constellation order the link can carry. It is the stage most sensitive to the quality of the light.

Timing, and then FEC

Timing recovery aligns the sampling clock to the symbol stream, and the forward error correction decoder, typically a soft-decision code, corrects the residual errors down to the delivered bit error ratio. FEC is the safety net under the whole chain, and its coding gain is what lets every earlier stage be designed to a pre-FEC threshold rather than to error-free operation.

What coherent-lite sheds

The chain above is written for a long-haul link, and its cost scales with the distances it is built to survive. Shorten the link and each stage can shrink. A campus link of a few kilometers accumulates little dispersion, so the compensation filter drops from hundreds of taps to a handful; the reach margin that demanded a high-order constellation is not needed, so a lower-order format relaxes the carrier-phase tracker; the equalizer can be shorter. The reduced-DSP or coherent-lite designs now appearing take exactly these cuts, trading reach they do not need for the power the full DSP would spend. This is the same ladder the industry is climbing in reverse from the top: full-reach 400ZR and 800ZR at the metro edge, then leaner grades toward 100ZR for the shortest coherent hops and future 1600ZR at the top of the range. Each grade is a different answer to one question, how much of the DSP chain the reach actually requires, and the answer is set by the physics in the sections above.

The practical reading: when a coherent module's power or cost is the problem, the fix is always found in the DSP, and which stage to cut is decided by which impairment the link is short enough to ignore.

References: Savory, "Digital coherent optical receivers: algorithms and subsystems," IEEE J. Sel. Top. Quantum Electron. 16, 1164 (2010); Kikuchi, "Fundamentals of coherent optical fiber communications," J. Lightwave Technol. 34, 157 (2016); OIF 400ZR and higher-rate coherent implementation agreements. The coherent DSP entry summarizes the chain, and Pluggable, LRO, LPO, CPO covers the direct-detection module ladder alongside which the ZR grades sit.