Modulation format

A modulation format specifies how information is encoded onto an optical carrier signal. Different formats trade off spectral efficiency, OSNR sensitivity, complexity, and bit rate. The choice of modulation format is one of the most consequential decisions in optical communications system design.

Fundamental dimensions of modulation.

Combinations: dual-polarization 16-QAM (DP-16QAM) uses amplitude + phase + polarization for 4 bits per symbol per polarization × 2 polarizations = 8 bits per channel-use.

Spectral efficiency. The bits per Hz a format carries:

Dual-polarization formats double the bit rate without doubling bandwidth.

OSNR requirements. Higher-order formats need a higher OSNR for the same bit error rate. Theoretical values for dual-polarization coherent formats at 32 GBaud, for a pre-FEC BER of $2\times {10}^{-2}$ (a typical soft-decision FEC threshold):

The requirement rises by 3 dB for each doubling of the symbol rate, and practical transceivers add an implementation penalty; the QAM entry gives the relation between SNR and OSNR. No further OSNR is needed for the post-FEC error rate: below the FEC threshold the decoder output is error-free for practical purposes. Direct-detection formats such as NRZ and PAM4 are mostly used in short, unamplified links, where receiver sensitivity and dispersion rather than OSNR set the limit.

Coherent detection recovers both quadratures and both polarizations of the optical field, which is what lets coherent formats carry several bits per symbol: 4 for DP-QPSK, 8 for DP-16QAM and 12 for DP-64QAM.

Direct-detection formats (no local oscillator needed at receiver):

• NRZ-OOK (Non-Return-to-Zero On-Off Keying): the simplest format. Two states: 0 (laser off) and 1 (laser on). Used in 10G/25G Ethernet, basic FTTH, and short-reach datacom.
• PAM4 (Pulse-Amplitude Modulation, 4 levels): four intensity levels carrying 2 bits per symbol. Used in 100G/400G short-reach links (100G-DR, 400G-DR4, 400G-FR4).
• DPSK (Differential Phase Shift Keying): phase modulation, with each bit's phase relative to the previous bit. 3-dB better OSNR than OOK. Used in submarine systems before coherent was common.

Coherent formats (require a local oscillator at receiver):

• BPSK (Binary Phase Shift Keying): two phase states 180° apart. 1 bit per symbol.
• QPSK (Quadrature Phase Shift Keying): four phase states 90° apart. 2 bits per symbol.
• 8-PSK: eight phase states 45° apart. 3 bits per symbol.
• 16-QAM (16-state Quadrature Amplitude Modulation): 4 levels on each of I and Q, in a square constellation. 4 bits per symbol.
• 64-QAM: 8 × 8 constellation. 6 bits per symbol.
• Probabilistic shaping: Higher-density constellation but inner points used more often, giving "fractional" bits per symbol with continuous tradeoff between rate and reach.

Polarization multiplexing. Single-mode fiber supports two orthogonal polarization states; modern coherent systems use both. "DP" prefix (dual-polarization) doubles bits per symbol. Coherent receivers separate the two polarizations via DSP.

Modulation format × bit rate matrix.

Eye diagram appearance.

• NRZ: 2 distinct intensity levels with a wide eye opening; the canonical eye diagram
• PAM4: 4 intensity levels; 3 stacked "eyes" of equal opening
• QPSK on in-phase axis: 2 levels (same in quadrature)
• 16-QAM: 4 levels on in-phase, 4 levels on quadrature; constellation diagram more useful than eye diagram

FEC and coding. Higher-order modulation formats require forward error correction (FEC) to operate at acceptable error rates. Modern FEC overhead is 20 – 25%:

• BCH or Reed-Solomon codes: legacy, low complexity
• LDPC (Low-Density Parity-Check) codes: modern, near-Shannon-limit performance
• Polar codes: emerging in some applications

The "FEC threshold" is the pre-FEC BER below which post-FEC BER falls to operating-acceptable levels (typically ${10}^{-12}$ or ${10}^{-15}$). Modern soft-decision FEC has threshold around $1-2\times {10}^{-2}$.

Format selection in practice.

Symbol rate (baud rate). Distinct from bit rate: the symbol rate is the number of distinct waveform changes per second. For a fixed bit rate, higher-order modulation gives a lower symbol rate, easing the burden on electronic components.

The choice often comes down to: can the electronics drive 50+ GBaud (PAM4) or only 25 GBaud (DP-QPSK)? In 2025, the trend is to higher baud rate (130 – 200 GBaud) combined with probabilistically shaped 64-QAM: about 130 GBaud carries 1.2T per channel, and 1.6T needs 160 to 200 GBaud once the FEC overhead is included.

How a modulation format fits into the path from host bits to light and back is traced in Bits to Light.

References: Agrawal, Fiber-Optic Communication Systems (4th ed., 2010), Ch. 8 (extensive coverage of modulation formats); Winzer & Essiambre, "Advanced optical modulation formats" Proc. IEEE 2006, the foundational comprehensive review of optical modulation; ITU-T G.698.2 / G.709 for OTN modulation standards.