Databases Reference
In-Depth Information
File Copy Benchmarks
For the sake of comparison,
Table C-1
shows how quickly we were able to copy a
sample file over a standard 100 Mbps Ethernet link on a LAN. The file was 738 MB
uncompressed and compressed to 100 MB with
gzip
's default options. The source and
destination machines had plenty of available memory, CPU resources, and disk ca-
pacity; the network was the bottleneck.
Table C-1. Benchmarks for copying files across a network
Method
Time (seconds)
rsync
without compression
71
scp
without compression
68
nc
without compression
67
rsync
with compression (
-z
)
63
gzip, scp
, and
gunzip
60 (44 + 10 + 6)
ssh
with compression
44
nc
with compression
42
Notice how much it helped to compress the file when sending it across the network—
the three slowest methods didn't compress the file. Your mileage will vary, however.
If you have slow CPUs and disks and a gigabit Ethernet connection, reading and
compressing the data might be the bottleneck, and it might be faster to skip the
compression.
By the way, it's often much faster to use fast compression, such as
gzip --fast
, than to
use the default compression levels, which use a lot of CPU time to compress the file
only slightly more. Our test used the default compression level.
The last step in transferring data is to verify that the copy didn't corrupt the files. You
can use a variety of methods for this, such as
md5sum
, but it's rather expensive to do
a full scan of the file again. This is another reason why compression is helpful: the
compression itself typically includes at least a cyclic redundancy check (CRC), which
should catch any errors, so you get error checking for free.
