Databases Reference
In-Depth Information
service scribe extends fb303.FacebookService
{
ResultCode Log(1: list<LogEntry> messages);
}
A sample PHP client message could be like this:
$messages = array();
$entry = new LogEntry;
$entry->category = “test_bucket”;
$entry->message = “a message”;
$messages []= $entry;
$result = $conn->Log($messages);
Available for
download on
Wrox.com
scribe_client.php
Log parsing and management is a very important job in the world of big data and its processing.
Flume is another solution like Scribe.
FLUME
Flume is a distributed service for effi ciently collecting, aggregating, and moving large amounts of
log data. It is based on streaming data fl ows. It is robust and fault tolerant and allows for fl exible
confi gurations. Flume documentation is available online at
http://archive.cloudera
.com/cdh/3/flume/
.
Flume consists of multiple logical nodes, through which the log data fl ows. The nodes can be
classifi ed into three distinct tiers, which are as follows:
Agent tier
— The agent tiers usually are on nodes that generate the log fi les.
Collector tier
— The collector tier aggregates log data and forwards the log to the storage tiers.
➤
➤
Storage tier
— This could be HDFS.
The Agent tier could listen to log data from multiple tiers and sources. For example, Flume agents
could listen to log fi les from syslog, a web server log, or Hadoop JobTracker.
➤
Flume can be thought of as a network of logical nodes that facilitate the fl ow of log data from the
source to the fi nal store. Each logical node consists of a source and a sink defi nition. Optionally,
logical nodes can have decorators. The logical node architecture allows for per-fl ow data guarantees
like compression, durability, and batching. Each physical node is a separate Java process but
multiple logical nodes can be mapped to a single physical node.
CHUKWA
Chukwa is a Hadoop subproject devoted to large-scale collection and analysis. Chukwa leverages
HDFS and MapReduce to provide a scalable infrastructure to aggregate and analyze log fi les.
