apache spark - Find out the partition no/id -
is there way(a method) in spark find out parition id/no
take example here
val input1 = sc.parallelize(list(8, 9, 10), 3) val res = input1.reduce{ (x, y) => println("inside partiton " + ???) x + y)} i put code in ??? print partition id / no
indeed, mapparitionswithindex give iterator & partition index. (this isn't same reduce of course, combine result of aggregate).
Comments
Post a Comment