scala - Spark sample code not serializable -


i ran sample code spark tutorial. works fine before second time run it. says it's not serializable. here code

import org.apache.spark.mllib.tree.randomforest import org.apache.spark.mllib.tree.model.randomforestmodel import org.apache.spark.mllib.util.mlutils  // load , parse data file. val data = mlutils.loadlibsvmfile(sc, "/home/liushiqi9/spark-1.3.0-src/data/mllib/sample_libsvm_data.txt") // split data training , test sets (30% held out testing) val splits = data.randomsplit(array(0.7, 0.3)) val (trainingdata, testdata) = (splits(0), splits(1))  // train randomforest model. //  empty categoricalfeaturesinfo indicates features continuous. val numclasses = 2 val categoricalfeaturesinfo = map[int, int]() val numtrees = 3 // use more in practice. val featuresubsetstrategy = "auto" // let algorithm choose. val impurity = "gini" val maxdepth = 4 val maxbins = 32  val model = randomforest.trainclassifier(trainingdata, numclasses, categoricalfeaturesinfo,   numtrees, featuresubsetstrategy, impurity, maxdepth, maxbins)  // evaluate model on test instances , compute test error val labelandpreds = testdata.map { point =>   val prediction = model.predict(point.features)   (point.label, prediction) } val testerr = labelandpreds.filter(r => r._1 != r._2).count.todouble / testdata.count() println("test error = " + testerr) println("learned classification forest model:\n" + model.todebugstring)

it's link:random forest

the logs this:

import org.apache.spark.mllib.tree.randomforest import org.apache.spark.mllib.tree.model.randomforestmodel import org.apache.spark.mllib.util.mlutils data: org.apache.spark.rdd.rdd[org.apache.spark.mllib.regression.labeledpoint] = mappartitionsrdd[735] @ map @ mlutils.scala:98 splits: array[org.apache.spark.rdd.rdd[org.apache.spark.mllib.regression.labeledpoint]] = array(mappartitionsrdd[736] @ randomsplit @ <console>:126, mappartitionsrdd[737] @ randomsplit @ <console>:126) trainingdata: org.apache.spark.rdd.rdd[org.apache.spark.mllib.regression.labeledpoint] = mappartitionsrdd[736] @ randomsplit @ <console>:126 testdata: org.apache.spark.rdd.rdd[org.apache.spark.mllib.regression.labeledpoint] = mappartitionsrdd[737] @ randomsplit @ <console>:126 numclasses: int = 2 categoricalfeaturesinfo: scala.collection.immutable.map[int,int] = map() numtrees: int = 3 featuresubsetstrategy: string = auto impurity: string = gini maxdepth: int = 4 maxbins: int = 32 model: org.apache.spark.mllib.tree.model.randomforestmodel =  treeensemblemodel classifier 3 trees  org.apache.spark.sparkexception: task not serializable     @ org.apache.spark.util.closurecleaner$.ensureserializable(closurecleaner.scala:315)     @ org.apache.spark.util.closurecleaner$.org$apache$spark$util$closurecleaner$$clean(closurecleaner.scala:305)     @ org.apache.spark.util.closurecleaner$.clean(closurecleaner.scala:132)     @ org.apache.spark.sparkcontext.clean(sparkcontext.scala:1891)     @ org.apache.spark.rdd.rdd$$anonfun$map$1.apply(rdd.scala:294)     @ org.apache.spark.rdd.rdd$$anonfun$map$1.apply(rdd.scala:293)     @ org.apache.spark.rdd.rddoperationscope$.withscope(rddoperationscope.scala:148)     @ org.apache.spark.rdd.rddoperationscope$.withscope(rddoperationscope.scala:109)     @ org.apache.spark.rdd.rdd.withscope(rdd.scala:286)     @ org.apache.spark.rdd.rdd.map(rdd.scala:293)     @ $iwc$$iwc$$iwc$$iwc$$iwc$$iwc$$iwc$$iwc$$iwc$$iwc$$iwc$$iwc$$iwc$$iwc$$iwc$$iwc$$iwc$$iwc$$iwc$$iwc$$iwc$$iwc$$iwc$$iwc$$iwc$$iwc$$iwc$$iwc$$iwc$$iwc$$iwc$$iwc$$iwc.<init>(<console>:141)

i don't understand why task fine , didn't work. didn't reinstall spark or anything. thanks!


Comments

Post a Comment

Popular posts from this blog

android - Pass an Serializable object in AIDL -

i have defined aidl file interface imyfeature { void dotask(class clazz); } but need pass class object process. possible in android? if so, how? i tried: interface imyfeature { serializable class; void dotask(class clazz); } but doesn’t work. aidl in android supports custom class implement parcelable interface. in case want send class object not possible. can send class canonical name , retrieve in service. for example have aidl file interface imyfeature { void dotask(string classname); } then call : binder.dotask(myclass.class.getcanonicalname()); and in service binder stub implementation : private final imyfeature mbinder = new imyfeature.stub() { public void dotask(string classname) { try { class<?> clazz = class.forname(classname); constructor<?> ctor = clazz.getconstructor((class[]) null); object object = ctor.newinstance(); ...
Read more

How to provide Authorization & Authentication using Asp.net, C#? -

Image
i have asp.net application in c#. i have provide page authorization specific user/usergroup. i have created 2 tables in database: 1) pagename 2)permissions[insert, update, view, delete] but i'm unable match tables , permissions assigning... can 1 please me, how created tables in database , how coding. please me. you can below explanation. configure access specific file , folder, set forms-based authentication. request page in application redirected logon.aspx automatically. in web.config file, type or paste following code. this code grants users access default1.aspx page , subdir1 folder . <configuration> <system.web> <authentication mode="forms" > <forms loginurl="login.aspx" name=".aspnetauth" protection="none" path="/" timeout="20" > </forms> </authentication> <!-- section denies access files in...
Read more

How to use Authorization & Authentication in Asp.net, C#? -

i using roll management , trying give page , folder access according user or user group, using server created ad group user authentication. i have default1.aspx page default , subdir1 folder give different access separate user group i using below logic in web.config. <location path="subdir1"> <system.web> <authorization> <allow users ="?" /> </authorization> </system.web> </location> i facing problem provide same access 2 or more directory same user should have provide allow user code twice both folder? i can use logic repeating value folder want access providing in 1 logic. i have got answer configure folder/page access, have make different access shown below.. configure access specific file , folder, set forms-based authentication. request page in application redirected logon.aspx automatically. in web.config file, done following code. this code grants user...
Read more