Database Reference
In-Depth Information
Currently, we're using the following hardware specifications. They are con-
stantly evolving as our needs change and we need to adapt.
Nodes:
HP DL 380, 12 cores, 96GB RAM, 5.5TB RAID10 HDD
Dell R620, 12 cores, 128GB RAM, 1TB RAID10 HDD, 10Gbps NIC
Shared storage:
Violin 6200/6600 flash memory array
NetApp-based spindle storage array
Challenges Faced and Lessons Learned
One of the most difficult things we have had to deal with is the Cassandra data
model. Because of the high-scale distributed nature of Cassandra, we lose the flex-
ibility of having efficient ad hoc queries. We had to begin our designs with query
patterns in mind. This method is sometimes painful and time-consuming as many
of the queries may not be known up front. To support multiple query patterns ef-
ficiently, we had to duplicate data in multiple ColumnFamilys. I won't be able to
discuss all the data model challenges we faced here, but for more information, you
can check out our data modeling best practices slides:
www.slideshare.net/jayku-
Another challenge we've faced is to decide proper replication factor (RF) and
consistency level (CL) settings for each application. Choice of RF and CL affects
latency, availability, durability, consistency, and cost. Eventual consistency is great
for system availability. However, designing an application for eventual consisten-
cy takes a bit of extra effort and planning. But the tunable nature of consistency
is a huge plus for Cassandra as we don't need to have eventual consistency for all
use cases and we can alter it on a per-use-case basis.
From the operations perspective, we have also faced many challenges in mak-
ing sure Cassandra can keep up with the pending compactions given the huge
amount of write traffic and data ingested. We have multiple times run into thou-
sands of pending compactions per node. Single-threaded compaction was simply
not able to keep up with the traffic and data size on each node. Reducing data size
from 5TB to under 1TB per node was a big help. Also, manually splitting Colum-
nFamilys aided compactions in running faster for some of our use cases. Garbage
collection pauses were another problem that we ran into quite frequently as a result
