Cassandra is not consistent despite QUORUM consistency level with replication factor 3 -

i have problem cassandras consistency. have 3 cassandra nodes (version 2.0.14.352) in cluster , reading , writting consistency level quorum , replicationfactor 3. if understand this right in case cassandra should consistent, because 2+2>3. wrote test in java, insert data fast cassandra using datastax-driver:

final instant t1 = instant.parse("2000-01-01t00:00:00.000z"); final instant t2 = instant.parse("2000-02-01t00:00:00.000z");  (int = 0; < 100; i++) {     dataprovider.setvalue(t1, new double(1));     //if next line removed, test pass     dataprovider.setvalue(t2, new double(3));      dataprovider.savetodb();     dataprovider.clear();     assertequals("i=" + i, new double(3), dataprovider.getvalue(t2));     assertequals("i=" + i, new double(1), dataprovider.getvalue(t1));      dataprovider.setvalue(t1, new double(2));     dataprovider.savetodb();     dataprovider.clear();     assertequals("i=" + i, new double(2), dataprovider.getvalue(t1));      dataprovider.setvalue(t1, new double(101));     dataprovider.savetodb();     dataprovider.clear();     assertequals("i=" + i, new double(101), dataprovider.getvalue(t1)); } 

with corresponding table

create table keyspace.table(   id text,   year int,   month int,   time timestamp,   value double,   primary key ((id, year, month), time) ) 

dataprovider.setvalue() internaly puts given value navigablemap. dataprovider.savetodb() inserts data cassandra. here tried on 1 hand insert data asynchronous , waited until resultsetfuture finished , on other hand executed statements synchronous. effected performance. in detail save method looks like

final list<resultsetfuture> sets = newlinkedlist(); batch batch = querybuilder.batch(); int batchsize=0; (map.entry<instant, double> entry : valuemap) {     final instant instant = entry.getkey();     final zoneddatetime zoneddatetime = instant.atzone(zoneid.of("utc"));     final date date = date.from(instant);     final insert insert = querybuilder.insertinto(table)             .value(id, id)             .value(year, zoneddatetime.getyear())             .value(month, zoneddatetime.getmonthvalue())             .value(time, date)             .value(value, entry.getvalue());     batch.add(insert);     ++batchsize;     if(batchsize % 200 == 0){         sets.add(cassandraconnector.executeasync(batch));         batch = querybuilder.batch();     } } if(batchsize % 200 != 0) { //es gibt noch nicht abgeschickte statements     sets.add(cassandraconnector.executeasync(batch)); } cassandraconnector.waitforfinish(sets); 

cassandraconnector manages connection. waiting until resultsets finished with

public boolean waitforfinish(list<resultsetfuture> sets) {     resultset result = null;     (final resultsetfuture resultsetfuture : sets) {         // wait until finished         try {             result = resultsetfuture.get();         } catch (interruptedexception e) {             resultsetfuture.cancel(true);             e.printstacktrace();             return false;         } catch (executionexception e) {             e.printstacktrace();             if (result != null) {                 executioninfo executioninfo = result.getexecutioninfo();                 system.out.println("timout server ip: " + executioninfo.gettriedhosts());             }             return false;         }     }     return true; } 

the curiosity is, if remove line under comment, test pass , doesn´t matter how execute it. if run test without removing line fails in first loop, runs 3 loop until fails. furthermore fails @ different lines. example

java.lang.assertionerror: i=0  expected :101 actual   :2 

i got

java.lang.assertionerror: i=2 expected :2 actual   :101 

so seems cassandra wrote 1 , after instead of writting 2 cassandra recovered 101 wrote before 1. have explanation behavior? why test pass if remove line? writting different partitions. tried change consistency level behavior didn´t change.

i solved it. clocks not 100 % synchronous. when create insert statement added .using(timestamp(system.nanotime() / 1000)); , test passes.