Optical PCIe 7.0 connection hits 128 GT/s

So I guess what this makes me wonder is: Why are we using electrical signals to connect the data lanes between components and computers these days, rather than moving everything to optical for data movement (obviously power would stay electrical, but that's already on separate lines)? I assume there's an element of cost, and once the photons get where they're going they have to be turned back into electrical signals to actually be used until such time as we get around to getting pure light based computers working (someday but not yet...), but that must not overwhelm the advantages or we wouldn't be looking at this being developed.

Since thunderbolt is related to PCIe, there's this that goes into copper vs optical there: https://en.wikipedia.org/wiki/Thunderbolt_(interface)#Copper...

So we're moving towards crystals from Stargate? Neat

Intel's product name foe Thunderbolt was initially was "Light Peak".

> I assume there's an element of cost, and once the photons get where they're going they have to be turned back into electrical signals to actually be used until such time as we get around to getting pure light based computers working (someday but not yet...)

You got it. We can't make optical transceivers as good as electrical ones. Not as small or power-efficient.

They require significantly different fabrication processes, and we don't know how to fab them into the same chip as electrical ones. I mean: you can either have photonics, or performant digital (or analog) electronics.

We've gotten really, really good at making small electronics, per the latest tech coming out of Intel & TSMC. We are... not that good at making photonics.

Changing from light to electricity (and vice versa) is relatively slow, expensive, and cumbersome.

Additionally, we don’t have a decent way of transferring significant power over fiber optics.

So since everything has to have copper power fed to it anyway, unless there is some compelling reason (like distance) to make optical/fibers disadvantages worth it, copper only is usually simpler and better.

At least for now.

> Not as small or power-efficient.

I wonder what the latency for switching medium is these days too (for the super small transceivers). To my understanding optical is better for attenuation than electric (less noise, and thus easier to shove more frequencies and higher frequencies on the same pipe), and can be faster (both medium dependent, neither yet approaching the upper bound of c).

I'm imaging the latency incurred by the transceiver is eventually offset from the gains in the signal path (for signal paths relevant to circuit boards and ICs)

Depending how you do the actual modulation, optical modulation does not add any significant latency (there can be no processing involved and the extra transmission length you'd need for the modulation (i.e. Electroopotic conversion, rf amplifiers...) is negligible.

The big issue is really 1. Photonic waveguides are much larger than electronic ones (due to the wavelength) 2. You loose dynamic range and in EO conversion (shot noise is significant at optical frequencies) 3. Co integration of optics and photonics components is nontrivial due to the different materials and processes. 4. Power efficiency of EO conversion is also not that great.

Where photonics shines is for transmission of high frequencies (i.e. a lot of data) over long distances and being immune to EM interference. So there is certainly a tradeoff for at what transmission distances to go optical and as data rates keep going up the tradeoff length has become shorter and shorter. Intel, Nvidia, AMD et al. All do research into optical interconnect.

Photonic wavelengths are shorter than electronic wavelengths. Why are photonics waveguides bigger then?

What’s an electronic wavelength in this context? What’s its size? Photonic ones I assume are in near-IR, on the order of a micrometer.

electrical connections don't require electrons fly all the way across dielectric materials

I believe it was originally supposed to be optical.

I seem to recall information travels slower in fiber (optical) vs wires (electric), resp. ~2c/3 vs ~c, or am I remembering it wrong? Or is it a significantly different optical medium?

If so, does that matter at all here? Dunno if that holds up for such kind of devices and/or at these scales (much shorter distance, but also much higher speed).

Optical fiber communication operates at 1.55 micro meters or 193THz. Electronics operates in the electromagnetic spectrum or at GHz. There might be no fixed size or wavelength in either case, but the shortest wavelength for radio signals that can be transmitted with acceptable loss with today’s technology is mili meters.

Which brings the question: why operating wavelengths are smaller but “waveguides” are bigger in optical fiber communication. In fact, fiber itself is a waveguide and its diameter is tens of micro meters.

IIRC the lower speed in fiber-optic cables has to do with the the refractive index of the glass, and maybe some bouncing introduced by curvature.

I'm not sure what kind of refractive indices are possible in much smaller photonic circuits, particularly if it's not practical to develop and run everything in a permanent vacuum.

The full optical wave is contained in the dielectric conductor. This conductor needs it's minimum cross section such that the wave can propagate. If it is too small then the wave can not propagate. Also there is a maximum cross section if you want single mode operation.

You get to this result if you take the electromagntic wave equation - a partial differential equation - and solve that for your transmission line configuration.

The proper analogy in the realm of electrical waveguides is the hollow waveguide. The hollow waveguide supports TE- and TM-modes but not TEM modes just like a dielectric conductor. The size is also a function of the dielectric constant ε.

What we mostly use are TEM waveguides like microstrips or coaxial cables. The difference between electrical waveguides that supports TEM modes and waveguides that supports TE/TM modes is that the former has two independent potential planes and the latter only one. Also TEM waveguides do not have a lower cutoff frequency. A TEM wave with any frequency can propagate on any microstrip configuration.

This is not true for TE/TM waves.

What's important to understand is that for microstrips/coaxial cables the power isn't transferred in the metal but in the space (dielectric) around the metal - see Poynting vector. So what happens if you have a second conductor in that space? You get crosstalk! So TEM transmission lines do not contain the wave like hollow waveguides or optical fibers (edit: ok coaxial cables do, microstrips don't)

Now the question, how big is the microstrip? Is it just the width of the signal conductor? No, it is not.

Edit: The width of the metal lines in a chip is given by the current it must carry - current density requirement, electro-migration issues. Power lines are wide because they have to supply power to the circuit but logic traces in CMOS technology only carry negligible amount of current. In circuits like RF power amplifiers with bipolar transistors the trace width is much larger because it has to carry a much larger current. But again, microstrip lines do not have a lower cutoff frequency.

> We can't make optical transceivers as good as electrical ones. Not as small or power-efficient.

I was under the impression that for 10Gb and above network transceivers, optical SFPs weren't getting as hot as copper ones. Is that difference related to something else?

There's a distinction between carrying high speed electrical signals across a PCB and carrying them over 30m or 100m (10GbT range.) Those "long reach" electrical transcievers are chock full of both analog and digital wizardry to both push and also decipher what becomes quite a mushy signal, which is why it's so energy intensive.

You can also think about it another way: SFPs are also connected with high bandwidth electrical links; for 10GE that signal is a pure straight 10.3125 Gbaud. Yet the SFPs don't heat up as much. You can also look up 10Gbase-KR, which is "stretching those plain PCB signals as far as we possibly can", as well as DAC cables and their ranges.

State of the art [cf. https://www.xilinx.com/products/technology/high-speed-serial... ] for SERDES blocks (= what makes your short-range PCB electrical link) is ca. ≤ 150Gbaud at PAM4 (2 bits per baud), i.e. ca. 300Gbit/s, but you need error correction at that point. PCIe 7.0 pulls back to a safe (and cheaper to manufacture) 64Gbaud with PAM4 to get its 128GT/s.

> To my understanding optical […] can be faster (both medium dependent) […]

The speed of light in optical fiber — for all types based on glass, ignoring miniscule differences — is 68% that of air/vaccuum. And that's not changing, and no state-of-the-art high speed applications are being developed on plastic fibre or free air.

So, latency wise, on runs with non-negligible length, optical will lose out to electrical, which is generally quite close to the speed of light. (Except of course after some point the electrical signal is just noise, and if you factor in delay caused by amplifiers/repeaters it becomes much harder of a question.)

There's this kinda-famous story of some HFT company pulling a copper cable across some bay, because they'd gain some nanoseconds compared to the fiber they had.

The transceiver latency for "long-range" links (well, call 100m long range for copper…) is actually worse for the copper links, as the whole DSP getup you need for that takes a few symbol times to process. Optical transceivers are just optodiodes and "simple" amplifiers, the latency is much less than a symbol.

(symbol = unit of transmission, roughly 1 bit on "old" fibre [≤ 25G lane rate], ca. 4 bit for 10Gbase-T, will vary more for faster connections.)

There's single-mode hollow-core fibers - which constrain the light via non-refractive / non-dielectric physics magic, i.e. without slowing down, to the hollow core. Not yet commercially available, I think.

Nice! But even after they're commercially available, putting fibers into places has massive inertia, along the lines of ≥10 years… and on short ranges (inter-chip connections up to building/city networks) you really don't care :)

> I assume there's an element of cost,

This assumption is very correct. Optical interconnects are extraordinarily expensive relative to copper. We have the art of manufacturing copper PCBs and connectors mastered. Putting optical interconnects into a system requires that the signal go through transceivers at either end as well as external optical cables, which are not integrated into the PCB. It’s extra components and complexity everywhere.

The reason optical interconnects are being explored here is that next gen PCIe is so extremely fast that the signals cannot travel very far in PCBs without significant losses. PCBs built for these speeds require special, expensive materials on the layers with those signals. They might require retimer chips to amplify the signal past a certain distance. These limitations may not apply to consumer motherboards with a single GPU very near to the CPU, but datacenter motherboards might need to connect many GPUs across a large chassis. The distances involved could require multiple retimer chips along the way and very expensive PCBs. Going to optical interconnects could introduce much more flexibility into where the GPUs or other add in cards are located.

As a side note: while the O in OCuLink does stand for optical that variant is not in use. Every OCuLink connector and cable are just ordinary copper interconnection.

Meanwhile Samtec has PCIe active optical cables, they have had them since 2012, it's a very niche application currently.

> They require significantly different fabrication processes, and we don't know how to fab them into the same chip as electrical ones.

There are actually a few commercial fabs that will monolithically integrate the photonics, analog electronics, and digital electronics, all in the same CMOS process. See for example GF’s process:

https://www.cmc.ca/globalfoundries-fotonix-45spclo/

Integrating good optical sources in silicon remains a challenge, but companies like Intel have mastered hybrid bonding and other packaging techniques. TSMC too has a strong silicon photonics effort.

We're a lot closer to the limits of copper than I realized. Apparently motherboard designers have to make the length of clock/data traces for DDR5 memory as close to equal as possible, otherwise the entire bus just doesn't work right (maybe this isn't news to other folks).

This sounds like something that could help us become really good at photonics though. The major issue when trying to displace the good old silicon transistors is that we have invested so much in the tech and they are just so good that competing technologies with higher potential but starting at the bottom of the S curve are simply not competitive. PCIe is widely used, if it switches to photonics, despite the current shortcomings this is very encouraging for photonics.

The speed of electrical waves in a chip is ~c/2.

The relative permittivity εr of SiO2 is ~4.

c = c0 / sqrt(εr)

c0 = 1 /sqrt(ε0εr × μ0μr) and in vacuum εr=μr=1.

But the frequency needs to be sufficiently high in order to observe wave propagation, let's say >10GHz.

For low frequencies the electric conductor behaves more like a RC chain.

You have to do that for all high-speed multipin interfaces. Generally, once you get past roughly 100 MHZ, it is good practice to match the lengths as well as possible, even though it probably doesn't matter until a couple of gigahertz (it depends on many factors, but generally, signals move at 15 cm per nanosecond on PCBs).

Considering the PCIe context, I was assuming we were talking about off-chip connections, i.e. diff microstrips/striplines in FR4 and air. εr is 2.6-ish there.

But you're right, I might have accidentally mixed in some radio connection bits, with the HFT company anecdote.

I know when silicon photonics were brand new, they had a big limitation because they only had designs that fired out of the plane of the chip, not across it. That limits you to interconnects and not cross-chip signaling. And since for a CPU you need a heat sink on top that means you have to fire down, toward the motherboard.

Also it turns out the speed of light in glass is not that impressive. So encoding and decoding at the ends eats up the speed advantage. That’s my impression as to why a lot of high profile articles on optical logic came out shortly thereafter. What if we just keep it as light for longer?

If memory serves the original plan for Light Peak was power wires and shielding braided around a fiber optic core.

Hence why we ended up with USB-C cables.

Less fragile, similar data rate, simpler BOM, less sensitive to dirt and debris, plenty of power.

What's the minimum bend radius for plastic fiber these days?

In general that's correct, however it's not quite as simple. For latency you need to consider group velocity (not phase velocity), which depends on the waveguide and frequency of the EM wave. I'm actually not quite sure what it is for electric waves, but for most photonic structures it's very similar to the phase velocity. The phase velocity is about c/1.5 for silica, but more like c/3.5-c/2 for most materials used in photonic integration.

I'm not an expert on integrated electronic circuits, but I guess the difference could matter depending on application.

Sorry this is a somewhat handwavy explanation that glosses over many details, but the size of components in electronic circuits (i.e. transistors and connections) fundamentally depends on the de Broglie wavelength of the electrons (some tens/hundreds of pm) why the size of a photonic circuit depends on the wavelength of light (order of 1 mu m). So one can make much more compact electronic circuits than photonic. Note that is somewhat different to transmission lines which are larger for electronics (depending on the RF carrier frequency) than optics.

Around 10 mm for OS2 cables.

Larger than ‘getting kinked in a desk drawer’. Though smaller than when the market made the decision of course.

You still need the copper wires to do power delivery, so either you end up with an even thicker cable, or multi-purpose the copper cables for signaling too.

I'm thankful you posted this and adjacent chain of responses and I enjoyed learning what you shared =)

Viva la HN