scala - spark test on local machine -

i running unit tests on spark 1.3.1 sbt test , besides unit tests being incredibly slow keep running java.lang.classnotfoundexception: org.apache.spark.storage.rddblockid issues. means dependency issue, wouldn't know where. tried installing on new machine, including fresh hadoop, fresh ivy2, still run same issue

any appreciated

exception:

exception in thread "driver heartbeater" java.lang.classnotfoundexception:      org.apache.spark.storage.rddblockid     @ java.net.urlclassloader$1.run(urlclassloader.java:366)     @ java.net.urlclassloader$1.run(urlclassloader.java:355)     @ java.security.accesscontroller.doprivileged(native method)     @ java.net.urlclassloader.findclass(urlclassloader.java:354)     @ java.lang.classloader.loadclass(classloader.java:425)     @ java.lang.classloader.loadclass(classloader.java:358)     @ java.lang.class.forname0(native method)     @ java.lang.class.forname(class.java:270) 

my build.sbt:

librarydependencies ++=  seq(    "org.scalaz"              %% "scalaz-core" % "7.1.2" excludeall exclusionrule(organization = "org.slf4j"),    "com.typesafe.play"       %% "play-json" % "2.3.4" excludeall exclusionrule(organization = "org.slf4j"),    "org.apache.spark"        %% "spark-core" % "1.3.1" % "provided"  withsources() excludeall (exclusionrule(organization = "org.slf4j"), exclusionrule("org.spark-project.akka", "akka-actor_2.10")),    "org.apache.spark"        %% "spark-graphx" % "1.3.1" % "provided" withsources() excludeall (exclusionrule(organization = "org.slf4j"), exclusionrule("org.spark-project.akka", "akka-actor_2.10")),    "org.apache.cassandra"    % "cassandra-all" % "2.1.6",    "org.apache.cassandra"    % "cassandra-thrift" % "2.1.6",    "com.typesafe.akka" %% "akka-actor" % "2.3.11",    "com.datastax.cassandra"  % "cassandra-driver-core" % "2.1.6" withsources() withjavadoc() excludeall (exclusionrule(organization = "org.slf4j"),exclusionrule(organization = "org.apache.spark"),exclusionrule(organization = "com.twitter",name = "parquet-hadoop-bundle")),    "com.github.nscala-time"  %% "nscala-time" % "1.2.0" excludeall exclusionrule(organization = "org.slf4j") withsources(),    "com.datastax.spark"      %% "spark-cassandra-connector-embedded" % "1.3.0-m2" excludeall (exclusionrule(organization = "org.slf4j"),exclusionrule(organization = "org.apache.spark"),exclusionrule(organization = "com.twitter",name = "parquet-hadoop-bundle")),    "com.datastax.spark"      %% "spark-cassandra-connector" % "1.3.0-m2" excludeall (exclusionrule(organization = "org.slf4j"),exclusionrule(organization = "org.apache.spark"),exclusionrule(organization = "com.twitter",name = "parquet-hadoop-bundle")),    "org.slf4j"               % "slf4j-api"            % "1.6.1",     "com.twitter"            % "jsr166e" % "1.1.0",    "org.slf4j"               % "slf4j-nop" % "1.6.1" % "test",    "org.scalatest"           %% "scalatest" % "2.2.1" % "test" excludeall exclusionrule(organization = "org.slf4j")  )  

and spark test settings (of have disabled test it)

(spark.kryo.registrator,com.my.spark.myregistrator)  (spark.eventlog.dir,)  (spark.driver.memory,16g)  (spark.kryoserializer.buffer.mb,512)  (spark.akka.framesize,5)  (spark.shuffle.spill,false)  (spark.default.parallelism,8)  (spark.shuffle.consolidatefiles,false)  (spark.serializer,org.apache.spark.serializer.kryoserializer)  (spark.shuffle.spill.compress,false)  (spark.driver.host,10.10.68.66)  (spark.akka.timeout,300)  (spark.driver.port,55328)  (spark.eventlog.enabled,false)  (spark.cassandra.connection.host,127.0.0.1)  (spark.cassandra.connection.ssl.enabled,false)  (spark.master,local[8])  (spark.cassandra.connection.ssl.truststore.password,password)  (spark.fileserver.uri,http://10.10.68.66:55329)  (spark.cassandra.auth.username,username)  (spark.local.dir,/tmp/spark)  (spark.app.id,local-1436229075894)  (spark.storage.blockmanagerheartbeatms,300000)  (spark.executor.id,<driver>)  (spark.storage.memoryfraction,0.5)  (spark.app.name,count entries 217885402)  (spark.shuffle.compress,false)  

an assembled or packaged jar sent standalone or mesos works fine! suggestions?

we ran same issue in spark 1.6.0 (there bug report it) fixed switching kryo serializer (which should using anyway). appears bug in default javaserializer.

simply following rid of it:

new sparkconf().setappname("simple application").set("spark.serializer", "org.apache.spark.serializer.kryoserializer")