Databases Reference
In-Depth Information
Unlike Prolog, however, Datalog represents a truly declarative logic programming lan‐
guage: it expresses the logic for a unit of work without needing to specify its control
flow. This is great for query languages, and Datalog did influence “recursive queries” in
Oracle, “Magic Sets” in DB2, etc. Declarative logic programming is also the basis for
specifying “what” is needed instead of “how” it must be achieved. Within the context
of a functional programming paradigm—and especially within the context of Cascading
workflows and parallel processing—many decisions about “how” can be deferred to the
flow planner to leverage the underlying topology.
Cascalog leverages Datalog and Cascading within Clojure to provide several key
benefits:
• Code is very compact, generally smaller than with other Hadoop abstraction layers.
• An interactive REPL makes development and testing convenient and efficient.
• Generating queries dynamically is easy and idiomatic—unlike in SQL.
• Interspersing queries within the rest of the application logic is trivial—unlike in
Java + SQL.
• Custom operations get defined just like any other function—unlike the UDFs used
in Hive, Pig, etc.
An interesting aspect of having the REPL is that Cascalog provides many of the benefits
of Apache Hive—convenient for ad hoc queries, roll-ups, etc. However, the queries are
more expressive than SQL and, in general, an order of magnitude less code is required
for comparable Cascalog apps. Meanwhile, Cascalog apps provide deterministic work‐
flows within a functional programming paradigm. Apache Hive does not.
There are a number of published articles and case studies about large-scale commercial
deployments of Cascalog. Nathan Marz authored the language while working at Back‐
Type, which was subsequently acquired by Twitter. The publisher analytics team at
Twitter uses Cascalog apps at very large scale for ETL, ad hoc queries, predictive mod‐
eling, etc.
Other interesting deployments include:
Insuring farmers against crop losses due to global warming
Location services, mobile maps, etc.
Price comparison and switching
Geolocation data sets
