Database Reference
In-Depth Information
stress modern technology, and the entire stream could be kept in main memory, essentially
forever.
Now, give the sensor a GPS unit, and let it report surface height instead of temperature.
The surface height varies quite rapidly compared with temperature, so we might have the
sensor send back a reading every tenth of a second. If it sends a 4-byte real number each
time, then it produces 3.5 megabytes per day. It will still take some time to fill up main
memory, let alone a single disk.
But one sensor might not be that interesting. To learn something about ocean behavior,
we might want to deploy a million sensors, each sending back a stream, at the rate of ten
per second. A million sensors isn't very many; there would be one for every 150 square
miles of ocean. Now we have 3.5 terabytes arriving every day, and we definitely need to
think about what can be kept in working storage and what can only be archived.
Image Data
Satellites often send down to earth streams consisting of many terabytes of images per day.
Surveillance cameras produce images with lower resolution than satellites, but there can be
many of them, each producing a stream of images at intervals like one second. London is
said to have six million such cameras, each producing a stream.
Internet and Web Traffic
A switching node in the middle of the Internet receives streams of IP packets from many
inputs and routes them to its outputs. Normally, the job of the switch is to transmit data and
not to retain it or query it. But there is a tendency to put more capability into the switch,
e.g., the ability to detect denial-of-service attacks or the ability to reroute packets based on
information about congestion in the network.
Web sites receive streams of various types. For example, Google receives several hun-
dred million search queries per day. Yahoo! accepts billions of “clicks” per day on its vari-
ous sites. Many interesting things can be learned from these streams. For example, an in-
crease in queries like “sore throat” enables us to track the spread of viruses. A sudden in-
crease in the click rate for a link could indicate some news connected to that page, or it
could mean that the link is broken and needs to be repaired.
