Apache Spark has just passed Hadoop in popolarity on the web (google trends) My first Apache Spark usage was extracting texts from tweets I’ve been collecting in Hadoop HDFS. My python script tweet-texts.py was``` import json

from pyspark import SparkContext

def valid(tweet):   return ’text’ in tweet

def gettext(line):   tweet = json.loads(line)   return tweet[’text’]

sc = SparkContext(appName=“Tweets”) data = sc.textFile(“hdfs://hadoop.redaelli.org:9000/user/matteo/staging/twitter/searches/TheCalExperience.json/*/*/*.gz”)

result = data.filter(lambda line: valid(line))\     .map(lambda tweet: gettext(tweet))

output = result.collect() for text in output:     print text.encode(‘utf-8’) And lunched with spark-1.1.0> bin/spark-submit –master local[4] tweet-texts.py I used Apche Spark 1.1.0 and Apache Hadoop 2.5.2. When I compiled Spark with mvn -Phadoop-2.5 -Dhadoop.version=2.5.2 -DskipTests -Pyarn -Phive package I got an error related to protocolBuffer jar release when I tried to read files from Hadoop HDFS org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$CreateSnapshotRequestProto overrides final method getUnknownFields.()Lcom/google/protobuf/UnknownFieldSet So I changed the pom.xml adding hadoop-2.5 <hadoop.version>2.5.2</hadoop.version> <protobuf.version>2.5.0</protobuf.version> <jets3t.version>0.9.0</jets3t.version>