Optical PCIe 7.0 connection hits 128 GT/s

Here I am still on PCI-E 3.0...

Most hardware (NVMe drives, GPUs, etc) doesn't run at more than 4.0 speeds anyway. The primary advantage of 5.0 and higher is that it'll allow that hardware to use fewer CPU lanes, eg what requires 4.0 x4 could use 6.0 x1.

It kinda worked that way in the past too. IIRC, newer AGP graphics cards would be keyed for 4x or 8x slots but would barely use more bandwidth than the original 2x slots provided.

It felt like we were on 3 for a long time, and then all of a sudden got 4 through 6 (and soon 7) in quick succession. I'd be curious to know what motivated that - maybe GPGPU taking off?

AI/GPU communication is definitely driving it forward now. It is a speed race for how quickly you can move data around.

Definitely data centre usage of some sort.

We didn't really bottleneck AGP8X until nVidia dropped the 200 series GeForce GPU. Then PCI-E was pretty much a requirement.

> eg what requires 4.0 x4 could use 6.0 x1

FWIW, this is only true for newer hardware. ie if you plugged in a pcie gen3x16 device into a pcie gen4x8 slot, although the bandwidth provided is in the same ballpark, the device will only run at pcie gen3x8.

So we'll need until the devices upgrade themselves to gen4 in this scenario to make use of higher bandwidth.

Really? I hadn't heard of GPU or GPGPU pushing bandwidth recently. Networking certainly does. 400GbE cards exceed PCIe 4.0 x16 bandwidth, 800 is here, and 1.6 apparently in the works. Disk too though, just because a single disk (or even network phy) may not max out a PCI slot does not mean you want to dedicate more lanes than necessary to them because you likely want a bunch of them.

the big use cases are inter-computer communication and nvme ssds. pcie 4x16 gets you 400 gbps Ethernet. 6x16 will be 1.6 tbps. for SSDs, it's the difference between needing 4 and 1 lanes to saturate your bandwidth. a modern 2u server at this point can have an ungodly amount of incredibly fast storage, and can expose all that data to everyone else without a bandwidth bottleneck.

We are at PCIe5 in the Dell XE9680. We add in 8x400G cards and they talk directly to the Network/ 8xGPUs (via rocev2).

800G ethernet is here at the switches (Dell Z9864F-ON is beautiful... 128 ports of 400G), but not yet at the server/NIC level, that comes with PCIe6. We are also limited to 16 chassis/128 GPUs in a single cluster right now.

NVMe is getting faster all the time, but is pretty standard now. We put 122TB into each server, so that enables local caching of data, if needed.

All of this is designed for the highest speed available today that we can get on the various bus where data is transferred.

Nvlink 4.0 used to connect H100 GPUs today is almost as fast as PCIe-7.0 (12.5GBs vs 16GBs). By the time PCIe-7.0 is available I’m sure NVlink will be much faster. So, yeah, GPUs are currently the most bandwidth hungry devices on the market.

I'm not saying the device itself would negotiate a higher PCIe version. I'm saying that the 4.0 x4 M.2 NVMe slot on your mobo would map to only one 6.0 CPU lane.

I can we can stack them? If we had 8x NVMe drives bundled I assume we'd need PCI 7.

I wonder if any of this trickles down into cloud providers reducing costs again. After all if we have zounds of fast storage, surely slower storage becomes cheaper?

It could still help connecting the chipset.

Not all pci-e lanes on your motherboard are created equal: Some are directly attached to the CPU others are connected to the chipset, which in turn is connected to the CPU.

It's possible to convert a single 5.0 x16 connection coming from the CPU to 2 4.0 x16 connections.

We do not directly compete with them as we are more of a niche based solution for businesses that want their own private cloud and do not want to undertake the many millions in capopex to build and deploy their own super computer clusters. As such, our offerings should not have an impact on their pricing. But who knows… maybe long term we will. Hard to say.

Will the lead time still be 50+ weeks though? My guess is yes.

Huh, I was pretty sure that you need an extra chip in-between. Otherwise, the 4 CPU lanes will just drop to 4.0 level.

Yes you do.

Same.

I upgraded to a 4 TB NVMe drive that theoretically reads/writes at up to 7.7 GB/s, but I only get 3.5 GB/s because my CPU is still an i9-9900K running PCI-E 3.0.

Planning on upgrading once the next Intel generation drops.

You'd need a bunch of those chips to make good use of a bunch of 6.0 lanes on the CPU, and at that point you're paying so much for the converters that I'm skeptical it would almost ever be worth it.

I expect consumer machines to keep doing some conversion and expansion in the chipset, but nowhere else. I expect servers to directly attach almost everything and drop down to smaller lane counts for large numbers of devices.

It's worth noting that when Kioxia first put out PCIe 5.0 EDSFF drives, they were marketing them as being optimized for 2 lanes at the higher speed.