Transferring Large Files - High Performance MySQL - page 716

A Naïve Example
We begin with a naïve example of how to send an uncompressed file securely from one
machine to another, compress it en route, and then decompress it. On the source server,
which we call server1 , we execute the following:
server1$ gzip -c /backup/mydb/mytable.MYD > mytable.MYD.gz
server1$ scp mytable.MYD.gz root@server2:/var/lib/myql/mydb/
And then, on server2 :
server2$ gunzip /var/lib/mysql/mydb/mytable.MYD.gz
This is probably the simplest approach, but it's not very efficient because it serializes
the steps involved in compressing, copying, and decompressing the file. Each step also
requires reads from and writes to disk, which is slow. Here's what really happens during
each of the above commands: the gzip performs both reads and writes on server1 , the
scp reads on server1 and writes on server2 , and the gunzip reads and writes on server2 .
A One-Step Method
It's more efficient to compress and copy the file and then decompress it on the other
end in one step. This time we use SSH, the secure protocol upon which SCP is based.
Here's the command we execute on server1 :
server1$ gzip -c /backup/mydb/mytable.MYD | ssh root@server2"gunzip -c - > /var/lib
>/mysql/mydb/mytable.MYD"
This usually performs much better than the first method, because it significantly re-
duces disk I/O: the disk activity is reduced to reading on server1 and writing on
server2 . This lets the disk operate sequentially.
You can also use SSH's built-in compression to do this, but we've shown you how to
compress and decompress with pipes because they give you more flexibility. For ex-
ample, if you didn't want to decompress the file on the other end, you wouldn't want
to use SSH compression.
You can improve on this method by tweaking some options, such as adding- 1 to make
the gzip compression faster. This usually doesn't lower the compression ratio much,
but it can make it much faster, which is important. You can also use different com-
pression algorithms. For example, if you want very high compression and don't care
about how long it takes, you can use bzip2 instead of gzip . If you want very fast com-
pression, you can instead use an LZO-based archiver. The compressed data might be
about 20% larger, but the compression will be around five times faster.
Avoiding Encryption Overhead
SSH isn't the fastest way to transport data across the network, because it adds the
overhead of encrypting and decrypting. If you don't need encryption, you can just copy