Insert a large number of records in cassandra

Question

Insert a large number of records in cassandra

Navigation

#1 by (1 votes)

1

I need to insert 1 million records in cassandra, I'm using the datastax driver, I have a correctly mapped teacher class and a list of teachers, then using the mapper I call the save method.

Cluster cluster  = Cluster.builder().addContactPoint("MiServer").build();
    Session session = cluster.connect("prueba");
    Mapper<Profesor> mapper = new MappingManager(session).mapper(Profesor.class);
    List<Profesor> profesors = CSVLogic.readAll("data.csv");
    profesors.forEach(mapper::save);

the insert works well however the performance does not seem good, and I want to know what is the most efficient way to insert 1 million records from java

java cassandra

asked by Julian David Grijalba 13.11.2017 в 17:33

source

1 answer

Remove elements from an array that are already in another array FATAL EXCEPTION: Timer-0

score 1 · Accepted Answer

The performance usually increases if we go to a lower level, maybe you can use something like BatchStatement and group many PreparedStatement. The code would be something like:

PreparedStatement statement= session.prepare("INSERT INTO people (name,age) VALUES (?,?)");
BoundStatement boundStatement = new BoundStatement(statement);
BatchStatement batchStmt = new BatchStatement();
batchStmt.add(boundStatement.bind("User A", "10"));
batchStmt.add(boundStatement.bind("User B", "12"));
session.execute(batchStmt);

With a loop you could put all the data (or in groups of a few thousand, at least).

The idea was taken from a similar question in SO in English