ProFTPD mini-HOWTO - Sendfile Support

## Sendfile Support

What is "sendfile"?
Many Unix kernels provide a function called `sendfile(2)`, a system call which provides a "zero-copy" way of copying data from one file (or socket) descriptor to another. The phrase "zero-copy" refers to the fact that all of the copying of data between the two descriptors is done entirely by the kernel, with no copying of data into userspace buffers. The normal way of copying data between files involves using the `read(2)` system call to read data from the source descriptor into a userspace buffer, then calling the `write(2)` system call to write that buffer to the destination descriptor. This copying of the data twice (once into the userland buffer, and once out from that userland buffer) imposes some performance and resource penalties. The `sendfile(2)` avoids these penalties by avoiding any use of userland buffers; it also results in a single system call (and thus only one context switch), rather than the series of `read(2)`/`write(2)` system calls (each system call requiring a context switch) usually used for data copying.

Unix kernels handle a socket as just another file descriptor; this means that `sendfile(2)` can be used to efficiently copy data from a file on disk to a network socket, e.g. for downloading a file.

Use of `sendfile(2)` in ProFTPD
For the above reasons, ProFTPD will, by default, attempt to use the `sendfile(2)` function for all downloads. However, there are a few cases where ProFTPD will specifically avoid the use of `sendfile(2)`:

- When data are transferred in ASCII mode; note that this automatically applies to directory listings (which, by definition, are ASCII transfers)
- When RFC2228 data channel protection is in effect (e.g. SSL/TLS) When transfers are being throttled via the `TransferRate` directive
- When `MODE Z` data compression is being used (via the `mod_deflate` module)

The use of `sendfile(2)` can also be explicitly configured at run-time by using the following in your `proftpd.conf` file:

      UseSendfile off

Sendfile support in the compiled `proftpd` daemon can also be disabled at compile time, by using the `--disable-sendfile` configure option, e.g.:

      # ./configure --disable-sendfile ...

This is not recommended unless necessary.

Known Issues with `sendfile(2)`
As useful as the `sendfile(2)` function can be, there are unfortunately cases where bad implementations of the function can cause problems. These problems manifest as broken or aborted downloads, or as downloaded data being corrupted, or even as downloaded files being larger than the original file.

The `sendfile(2)` function requires support from both the Unix kernel and from the filesystem drivers which handle the filesystems for the file descriptors being used. The following operating systems have had reports of issues when using `sendfile(2)`:

- AIX 5.2
- Solaris 10 x86
- PA-RISC Linux

There have also been some issues reported from users of virtualization technologies such as OpenVZ and VMware.

There have been cases where it was the filesystems, rather than the kernels, which appeared to have been the culprits in `sendfile(2)` problems:

- Network filesystems (e.g NFS, SMBFS/Samba, CIFS)
- Virtualized filesystems (OpenVZ, VMware, and even Veritas)
- Other filesystems (e.g. NTFS and `tmpfs` on Linux)

Again, if you encounter issues with downloading files from ProFTPD when those files reside on a networked or virtualized filesystem, try using "UseSendfile off" in your `proftpd.conf`.

Bugs in certain network cards have been reported on Linux, where the use of `sendfile(2)` triggers TCP checksum offloading bugs on these cards when using IPv6.

Bug 3081 also demonstrates an interesting `sendfile(2)` issue. It is a special case where the FTP client and server are on the same machine, `sendfile(2)` is used when downloading a file, and the client is downloading the file to the exact same filesystem path from which the server is reading the file. When this happens, `proftpd` will "hang". The fix for this situation is relatively simple: "Don't do that."

If your Unix kernel and your filesystems work together to support the use of `sendfile(2)` properly, you may run into one last potential issue with ProFTPD's `ftptop` and `ftpwho` utilities when `sendfile(2)` is used (at least for versions of `proftpd` prior to 1.3.4rc1). These utilities rely on data in ProFTPD's `ScoreboardFile` for providing their information, including transfer rates. The progress of a file being downloaded is tracked in the `ScoreboardFile` as part of the download process; specifically, as part of the normal `read(2)`/`write(2)` loop. When the `sendfile(2)` function is used, the `ScoreboardFile` is not updated with the progress (since all of the data copying is being handled by the kernel, not by `proftpd` code at that point). Thus for downloads which use `sendfile(2)`, `ftptop` and `ftpwho` often report the transfer rate as "Inf".

The transfer rates for uploads are still reported properly by `ftptop` and `ftpwho`, however.

In `proftpd-1.3.4rc1` and later, the `UseSendfile` directive changed. Now you can use `UseSendfile` in `<Directory>` sections and `.ftpaccess` files. This allows sites to disable use of `sendfile(2)` just for specific directories/filesystems where it might be a problem, e.g.:

      <Directory /path/to/nfs/files>
        UseSendfile off
      </Directory>

Instead of the usual on/off parameters, the `UseSendfile` directive can also take a byte length, or a file size percentage, as parameters. These parameters can be used so that `sendfile(2)` is used for more optimal downloads while still letting `ftptop` and `ftpwho` display download progress rates. To use this, the best way to use specify a file size percentage such as 10% or 25%; this means that `sendfile(2)` will be used for each 10% or 25% percent of the size of the file being downloaded:

      <Directory />
        UseSendfile 10%
      </Directory>

$Date: 2011/02/21 02:57:16 $