Increase NFS performance: Linux nconnect

Increase NFS performance: Linux nconnect

As we all know, the default behavior for transferring data is usually session based. NFS is also one of those protocols that will try to handle all the traffic through 1 session. On the client side you can manipulate this by using the nconnect option. This is available since linux kernel 5.3.
As a Ubuntu server 20.04 user I’m in luck, as it uses linux kernel version 5.4.

So, now … how does this work and what are the benefits?

First of all, it’s not that hard to use. You just add the nconnect option to the mount option list -o and allow multiple tcp sessions to be created.

example : mount -t nfs -o rw,nconnect=8 /media/user/data

My advice would be here to increase nconnect by the available interface ports on the storage side. So 1 session per port for maximum throughput. The max nconnect session is 16, so keep that in mind when setting up your storage architecture.

Also, don’t worry about any services that are using the nfs mount to access data. That is still just 1 session to the service or application.

Now, the results :

My test setup had 4x 1gig nic’s on the client side and 8x 1gig nic’s on the storage side.

Result : mount -t nfs -o rw /media/user/data

Traffic would flow around a 100MB/s which makes sense.
(1000mbps/8=125MB/s with some overhead)

Result : mount -t nfs -o rw,nconnect=8 /media/user/data

Traffic would flow around a 750MB/s which also makes sense.
(8x 1000mbps/8=1000MB/s with some overhead)

The gain :

Before, single-client bandwidth-intensive services were limited by how much bandwidth 1 interface had on the storage side (or client side) and you had to scale accordingly. Client’s were getting throttled as a result and were unable to take advantage of the full potential of your storage array.

With the nconnect mount option, services or app’s can now easily take full advantage of the infrastructure’s capabilities and deliver faster access to there storage.


2 Responses so far.

  1. In my case, the NFS share becomes completely inoperable, with any value of nconnect, from 2 to 16. When you try to write to a network folder, mounted with this parameter, mc freezes, and it becomes impossible to work with a share until rebooting the system.

  2. There are a couple of things:
    1. How many interfaces does your client and server have?
    2. Do you have jumbo frames enabled on the client/server side and on your switch infra in between?
    3. what is the cpu load on the server side when this happens?
    4. What is your OS

    based on that I can give you a more detailed answer.

Leave a Reply

Your email address will not be published.

4 × one =

This site uses Akismet to reduce spam. Learn how your comment data is processed.