Embed Notice

HTML Code

<blockquote style="position: relative; padding-left: 55px;"><section><a href="https://masto.yttrx.com/users/waffles/statuses/115623843715908958">waffles 🇨🇦 (waffles@masto.yttrx.com)'s status on Saturday, 29-Nov-2025 10:29:22 JST</a><a href="https://masto.yttrx.com/@waffles" title="waffles@masto.yttrx.com"><img src="https://gnusocial.jp/avatar/89781-48-20241128011354.webp" width="48" height="48" alt="waffles 🇨🇦" style="position: absolute; left: 0; top: 0;">waffles 🇨🇦</a><div><a href="https://masto.yttrx.com/@waffles/115617881902117945" rel="in-reply-to">in reply to</a></div></section><article><p>Welp. After an hour of debugging it seems like I'm stuck at 25Gb/s between my workstation and my nas.</p><p>I'm not about to run a dedicated bare metal nas for this. I guess I'll just have to be happy with what I currently have!</p></article><footer><a rel="bookmark" href="https://gnusocial.jp/conversation/5906722#notice-11610748">In conversation</a><time datetime="2025-11-29T10:29:22+09:00" title="Saturday, 29-Nov-2025 10:29:22 JST">about 20 days ago</time> <span>from <span><a href="https://masto.yttrx.com/@waffles/115623843715908958" rel="external" title="Sent from masto.yttrx.com via ActivityPub">masto.yttrx.com</a></span></span><a href="https://masto.yttrx.com/@waffles/115623843715908958">permalink</a><h4>Attachments</h4><ol><li><label><a rel="external" href="https://gnusocial.jp/attachment/5506972">Summary
‘What we confirmed works correctly: # Physical layer - ConnectX-5 at I00GbE, all 4 lanes
active, no errors
4 Cable - proper fiber with correct polarity
® Windows configuration - optimized buffers, RSS, jumbo frames
® TrueNAS configuration - optimized TCP buffers, ring buffers, queues
4 PCIe passthrough - now reporting correctly as Gen3 x16
® CPU - host passthrough, 8 cores pinned
What you're getting:
« Network: 25 Gbps (3.1 GB/s)
« iSCSI: 2.8 GB/s (97% of storage capacity)
« Storage write: 2.9 GB/s maximum
The bottleneck: VM hypervisor overhead. This appears to be the practical performance
limit for Proxmox PCIe passthrough with 100GbE NICs at this level of virtualization
complexity (4 PCIe devices passed through).
Your options:
1. Accept 2.8 GBs - This is excellent performance and exceeds your storage capability
2.Run TrueNAS bare metal - Would likely get you 80-90+ Gbps
3.Research Proxmox forums - Search for "ConnectX-5 performance limit" to see if others
found solutions
My recommendation: Your setup is working very well. You're getting 97% of your storage's
write speed over iSCSI. Unless you have a specific workload that needs more than 2.8 GB/s
sustained throughput, this is a success.
The 100GbE network has plenty of headroom for future expansion if you upgrade storage.</a></label><br><a href="https://files.yttrx.com/media_attachments/files/115/623/839/105/531/873/original/1302ba8ade54d905.png" rel="external">https://files.yttrx.com/media_attachments/files/115/623/839/105/531/873/original/1302ba8ade54d905.png</a></li></ol></footer></blockquote>

Corresponding Notice

Embed this notice
waffles 🇨🇦 (waffles@masto.yttrx.com)'s status on Saturday, 29-Nov-2025 10:29:22 JST waffles 🇨🇦
in reply to
Welp. After an hour of debugging it seems like I'm stuck at 25Gb/s between my workstation and my nas.
I'm not about to run a dedicated bare metal nas for this. I guess I'll just have to be happy with what I currently have!
In conversationabout 20 days ago from masto.yttrx.compermalink
Attachments
1. Summary ‘What we confirmed works correctly: # Physical layer - ConnectX-5 at I00GbE, all 4 lanes active, no errors 4 Cable - proper fiber with correct polarity ® Windows configuration - optimized buffers, RSS, jumbo frames ® TrueNAS configuration - optimized TCP buffers, ring buffers, queues 4 PCIe passthrough - now reporting correctly as Gen3 x16 ® CPU - host passthrough, 8 cores pinned What you're getting: « Network: 25 Gbps (3.1 GB/s) « iSCSI: 2.8 GB/s (97% of storage capacity) « Storage write: 2.9 GB/s maximum The bottleneck: VM hypervisor overhead. This appears to be the practical performance limit for Proxmox PCIe passthrough with 100GbE NICs at this level of virtualization complexity (4 PCIe devices passed through). Your options: 1. Accept 2.8 GBs - This is excellent performance and exceeds your storage capability 2.Run TrueNAS bare metal - Would likely get you 80-90+ Gbps 3.Research Proxmox forums - Search for "ConnectX-5 performance limit" to see if others found solutions My recommendation: Your setup is working very well. You're getting 97% of your storage's write speed over iSCSI. Unless you have a specific workload that needs more than 2.8 GB/s sustained throughput, this is a success. The 100GbE network has plenty of headroom for future expansion if you upgrade storage.
  https://files.yttrx.com/media_attachments/files/115/623/839/105/531/873/original/1302ba8ade54d905.png

Public

Embed Notice

HTML Code

Corresponding Notice