Modulation format

A modulation format specifies how information is encoded onto an optical carrier signal. Different formats trade off spectral efficiency, OSNR sensitivity, complexity, and bit rate. The choice of modulation format is one of the most consequential decisions in optical communications system design.

Fundamental dimensions of modulation.

Dimension	Effect on signal
Amplitude	Multiple intensity levels (NRZ = 2, PAM4 = 4, etc.)
Phase	Multiple phase states (BPSK = 2, QPSK = 4, 8PSK = 8)
Polarization	2 orthogonal polarizations carrying independent data
Frequency	Multiple wavelengths (subcarriers) per channel
Time slot	(not a modulation format per se, but a framing dimension)

Combinations: dual-polarization 16-QAM (DP-16QAM) uses amplitude + phase + polarization for 8 bits per symbol per polarization × 2 polarizations = 8 bits per channel-use.

Spectral efficiency. The bits per Hz a format carries:

Format	Bits/symbol	Spectral efficiency (max, before FEC overhead)
OOK / NRZ	1	1 bit/Hz
PAM4	2	2 bit/Hz
QPSK / DQPSK	2	2 bit/Hz
8-PSK	3	3 bit/Hz
16-QAM	4	4 bit/Hz
32-QAM	5	5 bit/Hz
64-QAM	6	6 bit/Hz
256-QAM	8	8 bit/Hz
DP-QPSK (dual-polarization)	4	4 bit/Hz
DP-16QAM	8	8 bit/Hz
DP-64QAM	12	12 bit/Hz

Dual-polarization formats double the bit rate without doubling bandwidth.

OSNR requirements. Higher-order formats need higher OSNR for the same bit error rate:

Format @ 100 Gb/s line rate	OSNR threshold for BER = 2×10⁻² (pre-FEC)	OSNR threshold for BER = 10⁻¹⁵ (post-FEC)
NRZ-OOK	28 dB/0.1nm	35 dB/0.1nm
PAM4	23 dB/0.1nm	30 dB/0.1nm
DP-QPSK	12 dB/0.1nm	17 dB/0.1nm
DP-16QAM	17 dB/0.1nm	22 dB/0.1nm
DP-64QAM	23 dB/0.1nm	30 dB/0.1nm

Coherent formats (DP-QPSK, DP-16QAM, DP-64QAM) benefit from coherent detection, which provides 3 dB SNR improvement over direct detection of the same symbol rate.

Direct-detection formats (no local oscillator needed at receiver):

• NRZ-OOK (Non-Return-to-Zero On-Off Keying): the simplest format. Two states: 0 (laser off) and 1 (laser on). Used in 10G/25G Ethernet, basic FTTH, and short-reach datacom.
• PAM4 (Pulse-Amplitude Modulation, 4 levels): four intensity levels carrying 2 bits per symbol. Used in 100G/400G short-reach links (100G-DR, 400G-DR4, 400G-FR4).
• DPSK (Differential Phase Shift Keying): phase modulation, with each bit's phase relative to the previous bit. 3-dB better OSNR than OOK. Used in submarine systems before coherent was common.

Coherent formats (require a local oscillator at receiver):

• BPSK (Binary Phase Shift Keying): two phase states 180° apart. 1 bit per symbol.
• QPSK (Quadrature Phase Shift Keying): four phase states 90° apart. 2 bits per symbol.
• 8-PSK: eight phase states 45° apart. 3 bits per symbol.
• 16-QAM (16-state Quadrature Amplitude Modulation): 4 amplitude × 4 phase, in a square constellation. 4 bits per symbol.
• 64-QAM: 8 × 8 constellation. 6 bits per symbol.
• Probabilistic shaping: Higher-density constellation but inner points used more often, giving "fractional" bits per symbol with continuous tradeoff between rate and reach.

Polarization multiplexing. Single-mode fiber supports two orthogonal polarization states; modern coherent systems use both. "DP" prefix (dual-polarization) doubles bits per symbol. Coherent receivers separate the two polarizations via DSP.

Modulation format × bit rate matrix.

Bit rate	Common formats
10 G	NRZ-OOK
25 G	NRZ-OOK, PAM4 (rare)
40 G	NRZ-OOK, DPSK
50 G	PAM4
100 G	NRZ-OOK ($\sim$ 10G × 10 lanes), PAM4 (4 lanes), DP-QPSK (coherent)
200 G	DP-QPSK, DP-16QAM
400 G	PAM4 (8 lanes 50G), DP-16QAM, DP-8QAM, DP-PS-QAM
600 G	DP-16QAM, DP-64QAM
800 G	DP-64QAM, DP-PS-QAM
1.2 T	DP-256QAM (research)

Eye diagram appearance.

• NRZ: 2 distinct intensity levels with a wide eye opening; the canonical eye diagram
• PAM4: 4 intensity levels; 3 stacked "eyes" of equal opening
• QPSK on in-phase axis: 2 levels (same in quadrature)
• 16-QAM: 4 levels on in-phase, 4 levels on quadrature; constellation diagram more useful than eye diagram

FEC and coding. Higher-order modulation formats require forward error correction (FEC) to operate at acceptable error rates. Modern FEC overhead is 20 – 25%:

• BCH or Reed-Solomon codes: legacy, low complexity
• LDPC (Low-Density Parity-Check) codes: modern, near-Shannon-limit performance
• Polar codes: emerging in some applications

The "FEC threshold" is the pre-FEC BER below which post-FEC BER falls to operating-acceptable levels (typically $10^{-12}$ or $10^{-15}$). Modern soft-decision FEC has threshold around $1 - 2 \times 10^{-2}$.

Format selection in practice.

Reach	Bit rate	Recommended format
100 m (datacom)	100 G	PAM4 × 4 lanes (100G-DR4 or similar)
2 km (datacom + reach)	100 G	PAM4 × 4 or LR4 NRZ × 4
10 km	100 G	LR4 NRZ × 4 or coherent DP-QPSK
80 km	100 G	Coherent DP-QPSK
500 km	200 G	Coherent DP-QPSK or DP-16QAM
1500 km	400 G	Coherent DP-16QAM (modern)
Submarine 6500 km	200 G	Coherent DP-QPSK + Raman+EDFA

Symbol rate (baud rate). Distinct from bit rate: the symbol rate is the number of distinct waveform changes per second. For a fixed bit rate, higher-order modulation gives a lower symbol rate, easing the burden on electronic components.

Format	Bit rate	Baud rate
NRZ-OOK	100 G	100 GBaud
PAM4	100 G	50 GBaud
DP-QPSK	100 G	25 GBaud
DP-16QAM	100 G	12.5 GBaud
DP-64QAM	100 G	8.3 GBaud

The choice often comes down to: can the electronics drive 50+ GBaud (PAM4) or only 25 GBaud (DP-QPSK)? In 2025, the trend is to higher baud rate (90 – 130 GBaud) combined with high-order modulation (PCS-shaped 64-QAM or 256-QAM) for 1.6T per channel.

References: Agrawal, Fiber-Optic Communication Systems (4th ed., 2010), Ch. 8 — extensive coverage of modulation formats; Winzer & Essiambre, "Advanced optical modulation formats" Proc. IEEE 2006, the foundational comprehensive review of optical modulation; ITU-T G.698.2 / G.709 for OTN modulation standards.