Database Reference
In-Depth Information
Flume Sources
One of the chief draws for Flume is its ability to integrate with a variety
of systems. To support this, Flume ships with a number of built-in source
components. It also supports the ability to implement custom sources in
much the same manner as the channel selector discussed in the previous
section.
Avro Source
Avro is a structured data format developed by Yahoo!. Philosophically
similar to other data formats such as protocol buffers from Google and
Thrift from Facebook, it is the native data interchange format for Flume.
All three of these systems use an Interface Definition Language (IDL) to
define structures that are similar to C structs or simple Java classes.
The formats can use this description to encode structures in their native
language to a high-performance binary wire protocol that is easily decoded
by another service with access to the schema defined by the IDL. In the case
of Avro, this IDL is JSON, making it easy to define data structures that can
be implemented by a variety of languages.
Like Thrift, Avro also defines a remote procedure call (RPC) interface. This
allows “methods” defined by Avro to be executed via a command sent over a
TCP/IP connection. It is this RPC interface that Flume implements to pass
events into the agent.
The Avro source in Flume is assigned to a source definition by specifying
the type as avro . It also takes a bind and port property that defines the
network interface and port the agent should listen on for Avro RPC calls.
For example, the following options bind the Avro listener to all network
interfaces defined by the system on port 4141:
agent_1.source.source-1.type=avro
agent_1.source.source-1.bind=0.0.0.0
agent_1.source.source-1.port=4141
The special IP address 0.0.0.0 should be used with care. If the agent is
running on a server that has an external IP address, it binds to that IP
address. In most cases, this is harmless because the server almost certainly
sits behind a firewall that protects it from most mischief. However, it may
also open the port to internal services that should not have access. For
Search WWH ::




Custom Search