Databases Reference
In-Depth Information
R
Mahout
SAS
Analytics
Sqoop
JDBC
Native-
Drivers
DB
Integration
Landing
Zone
Ingestion
Process
Discover
BI Tools
on Hive
Ta bleau
Spotfire
Qlikvew
Operational
Reporting
Hadoop-HDFS
NoSQL
GFS
Amazon EMR
MapReduce
Hive QL
Python
Ruby
Java
Unix Files
Amazon S3
HDFS
Datameer
Quest Software
Kamasphere
Qlikview
Custom
Scropts,
Tools
Raw Data
Extracts
Extract Data
FIGURE 3.7
Conceptual Big Data processing platform.
●
Three-month rolling average for usage and billing
●
Customer Lifetime value (LTV)
●
Customer profitability tier
The Telco has received complaints from a number of consumers on dropped calls or dropped
packets, resulting in revenue leakage and customer attrition. The IT team believes that inspecting the
data from the cell phone towers can give them a clue on whether the failure is due to poor infrastruc-
ture. The business team also wants to include the data from social media and newsprint about the
quality of service, customer sentiments, competitive comparisons, and product and service reviews.
To process this additional volume of data, the IT team undertook a study on the overall impact, the
results of which are as follows:
●
Current data in data warehouse: 5 TB (three years rolling)
●
New data from towers: 400 MB log/day
●
New data from social media and others: 250 MB/day
●
Content from documents, contracts, and call center data: 200 MB/day
●
Emails, forum posts, and social media: 100 MB/day
Based on the data analysis, the IT team concluded that a database cannot handle the type, volume,
and processing complexity of the new data, for the following reasons:
●
Data is stored in columns and table structures within a database. To process data, you need a finite
structural definition first, called the data model.
●
Relationships between different data structures are represented by data.
●
Data manipulation language and data definition language are the two key mechanisms of
processing data in the database.
●
Databases are highly optimized for handling finite transactions.
