Questions tagged as 'apache-spark'

Tagged questions
0
answers

How do I convert a code into scala to pyspark?

Hello I need help with a code that I'm doing, it turns out that I must define a row wn pyspark object, but I have a code in scala that does that but I must do it in python def getRow(x : String) : Row={ val columnArray = new Array[String](95)...
asked by 15.05.2018 / 18:19
0
answers

java.lang.ClassNotFoundException: org.apache.spark.sql.ForeachWriter

When running a Scala file that uses the Spark ForeachWriter type I get the following stack trace:    (run-main-0) java.lang.NoClassDefFoundError:   org / apache / spark / sql / ForeachWriter java.lang.NoClassDefFoundError:   org / apache / sp...
asked by 27.04.2018 / 00:29
0
answers

Error due to insufficient memory to run algorithm in ApacheSpark

The error is:    System memory 259522560 must be at least 471859200. Increase the heap size by using the --driver-memory or spark.driver.memory option in the Spark configuration. This is my code and the error is about the spark variable o...
asked by 20.03.2018 / 18:57
1
answer

convert JavaPairRDDString, Integer to Json

I have the following WordCount example and I would like to get the result to JSONObject: JavaPairRDD<String, Integer> counts = words.mapToPair(new PairFunction<String, String, Integer>() { public Tuple2<String, Integer> c...
asked by 13.02.2018 / 17:47
1
answer

Help with the following problem

Good morning, I am trying to solve the following problems: "Count number of vowels and consonants" using the hadoop platform, MapReduce technique. The question that happens to me is the following: When executing the algorithm I get this outpu...
asked by 15.11.2017 / 21:41
1
answer

Change values of an RDD

How can I change all the letters A to a 9 and all the letters B to an 8 in an RDD with a lambda function. I tried this but it does not work: rdd.map(lambda a: 9 if a == "A" else a == a) rdd.map(lambda a: 8 if a == 'B' else a == a) My exampl...
asked by 22.06.2017 / 20:49
1
answer

anonymous class implements interface; can not have arguments

I'm trying spark on maven but I get this problem I want to show me the port but the runable shows me this error    "anonymous class implements interface; can not have arguments" Code public class HelloWorldSparkStyle { publ...
asked by 04.06.2017 / 17:05