Modulation instability

Modulation instability (MI) is the exponential growth of small amplitude or phase perturbations on an intense, nearly continuous wave propagating in a medium with anomalous dispersion and an intensity-dependent refractive index from the Kerr effect. Noise near the carrier is amplified into two symmetric sidebands, and the wave breaks up into a periodic train of short pulses. In standard single-mode fiber at 1550 nm, with group-velocity dispersion $\beta_2 \approx -22$ ps²/km and nonlinear parameter $\gamma \approx 1.3$ /(W·km), a 1 W pump has its maximum gain at 54.7 GHz (0.44 nm) on either side of the carrier, with a power gain coefficient of 2.6 /km.

Gain spectrum

The instability follows from a linear stability analysis of the nonlinear Schrödinger equation. A continuous wave of power $P_0$ is perturbed by a small modulation at angular frequency offset $\Omega$; the perturbation grows with the power gain

$$g(\Omega) = |\beta_2\,\Omega|\sqrt{\Omega_c^2 - \Omega^2},$$

where the cutoff frequency is

$$\Omega_c^2 = \frac{4\gamma P_0}{|\beta_2|}.$$

Gain exists only for $|\Omega| < \Omega_c$ and only when $\beta_2 < 0$; in the normal-dispersion regime a scalar continuous wave is stable. The gain is largest at

$$\Omega_\text{max} = \frac{\Omega_c}{\sqrt{2}} = \sqrt{\frac{2\gamma P_0}{|\beta_2|}},$$

where it reaches $g_\text{max} = 2\gamma P_0$. The peak gain is independent of dispersion; the dispersion sets only where in frequency the gain appears.

In physical terms the process is phase-matched four-wave mixing in which two pump photons create a signal and an idler photon. Self-phase modulation adds a nonlinear phase that would otherwise leave the sidebands mismatched, and anomalous dispersion supplies the opposite phase mismatch, so the two cancel at $\Omega_\text{max}$.

Worked example

For $\beta_2 = -22$ ps²/km, $\gamma = 1.3$ /(W·km) and $P_0 = 1$ W, $\Omega_\text{max} = 0.344$ rad/ps, a frequency offset of 54.7 GHz, which is 0.44 nm at 1550 nm. The gain band ends at $\Omega_c/2\pi = 77.4$ GHz. The peak gain of 2.6 /km corresponds to 11.3 dB/km, so over 5 km a seeded sideband grows by $e^{13}$. The modulation period, $2\pi/\Omega_\text{max}$, is 18.3 ps, which sets the spacing of the pulse train that forms. At 10 W of peak power the peak moves to 173 GHz (1.39 nm), the period shortens to 5.8 ps, and the gain rises to 26 /km. Because $\Omega_\text{max}$ scales as $\sqrt{P_0}$, measuring the sideband spacing at a known power gives $\gamma/|\beta_2|$, and repeating it at a second power checks the $\sqrt{P_0}$ scaling.

Observation in the lab

MI is observed by launching a narrow-linewidth, amplified continuous wave or long (nanosecond) pulses into a few kilometers of fiber in its anomalous-dispersion window and recording the output on an optical spectrum analyzer. Spontaneous MI grows from amplified spontaneous emission or vacuum noise and appears as two broad lobes symmetric about the pump, often with cascaded higher-order lobes at multiples of the offset. Induced MI uses a weak seed at a chosen offset, which produces a cleaner, regular pulse train; an autocorrelator then shows the modulation period. Long pulses are preferred to true continuous waves at high power, because stimulated Brillouin scattering would otherwise reflect most of a narrow-linewidth pump before MI develops.

Where it matters

• Supercontinuum generation. With long pump pulses, MI breaks the pulse into many fundamental solitons that then shift and broaden the spectrum. Because the process starts from noise, the resulting supercontinuum varies from shot to shot and has poor coherence.
• Long-haul transmission. In anomalous-dispersion links with in-line amplifiers, MI amplifies amplifier noise near the signal carrier and degrades the signal-to-noise ratio; periodic power variation along the link can also produce sidebands at other offsets.
• Frequency combs and pulse sources. Induced MI in fiber, and its analog in microresonators, generates high-repetition-rate pulse trains and is one route to the onset of Kerr microcombs.

Pitfalls

The formula above neglects fiber loss, higher-order dispersion and the Raman response. Loss reduces $P_0$ along the fiber, so the peak drifts to lower offset with distance and the effective gain is set by the effective length. Near the zero-dispersion wavelength $|\beta_2|$ becomes small, the band moves far from the pump and fourth-order dispersion can open additional gain bands, including in the normal regime. In birefringent or two-mode fibers, cross-phase modulation between polarizations allows vector MI even with normal dispersion.

Common questions

Why does modulation instability need anomalous dispersion?

The Kerr effect raises the phase of the intense carrier relative to weak sidebands. Only anomalous dispersion gives a phase mismatch of the opposite sign that can cancel it, so the four-wave-mixing gain is phase-matched. With normal dispersion the two contributions add and no single-polarization gain band forms.

How is modulation instability related to solitons?

Both arise from the same balance of anomalous dispersion and self-phase modulation. MI is the instability of a continuous wave under that balance, and its nonlinear evolution produces a train of pulses that approach fundamental solitons.

References: G. P. Agrawal, Nonlinear Fiber Optics, 6th ed. (Academic Press, 2019); K. Tai, A. Hasegawa and A. Tomita, "Observation of modulational instability in optical fibers," Phys. Rev. Lett. 56, 135 (1986); J. M. Dudley, G. Genty and S. Coen, "Supercontinuum generation in photonic crystal fiber," Rev. Mod. Phys. 78, 1135 (2006).