Database Reference
In-Depth Information
Batch inserter / indexer
Similar to LOAD CSV, batch inserter / indexer is also a memory-intensive operation. So it
is imperative for inserter/indexer to tune the memory and JVM parameters. The following
are few considerations for tuning the memory and JVM of batch inserter / indexer:
•
BatchInserter#shutdown
should be called only at the end of each phase of
the batch writing operation, as it will commit all changes and make it visible to the
querying APIs.
• Avoid executing
BatchInserterIndex.flush()
too often during batch in-
dexing as flushing will commit all changes and make it available for the querying
APIs, which could result in performance degradation.
• Depending upon your memory sizes, have big phases for the write operations.
• Enable indexing and caching (
BatchInserterIn-
dex.html#setCacheCapacity
) for common and widely used keys so that it
can increase performance during lookups. Though it will slightly degrade the per-
formance, there will be no major impact unless you care for each millisecond.
• Avoid or keep to a minimum read/lookup operations during batch insertion or in-
dexing.