Database Reference
In-Depth Information
Moving Forward
We are looking to two major steps to maximize the value from this system more efficiently.
First, we want to create prescriptive practices around the Hadoop ecosystem and its sup-
porting libraries. A number of good practices are defined in this topic and elsewhere, but
they often require significant expertise to implement effectively. We are using and building
libraries that make such patterns explicit and accessible to a larger audience. Crunch offers
some good examples of this, with a variety of join and processing patterns built into the lib-
rary.
Second, our growing catalog of datasets has created a demand for simple and prescriptive
data management to complement the processing features offered by Crunch. We have been
adopting the Kite SDK to meet this need in some use cases, and expect to expand its use
over time.
The end goal is a secure, scalable catalog of data to support many needs in healthcare, in-
cluding problems that have not yet emerged. Hadoop has shown it can scale to our data and
processing needs, and higher-level libraries are now making it usable by a larger audience
for many problems.
