Enable MySQL Streaming in Cascading / Scalding
Last week I ran into a an ugly problem of Scalding: I needed to read a really large table from MySQL to process it in a certain job. In generall this is trivial: just use a JDBC Source, select your...
View ArticleScalding: unable to compare stream elements in position: 0
I’m currently working quite a bit with Twitter’s Scalding. Recently I split up a job into sub-jobs and suddenly got an Exception in my join: If I had remembered the Fields API in detail, I would have...
View ArticleScalding Exception: diverging implicit expansion for type...
I was just doing a again some scalding jobs and again got an .. interesting exception: In a groupBy operation, I wanted to sum something up using: And was rewarded with this one: WTF?? Solution: Spot...
View ArticleEnable output compression in Scalding
I just wanted to enable final output compression in one of my Scalding jobs (because I needed to reorganize a some-TB-data set). Unfortunately scalding always produced uncompressed files. After some...
View ArticleScalding hiding NPEs in “operator Each failed executing operation”
Yesterday I was surprised by a failing Scalding task. Everything worked fine locally and all I git was like “job failed, see cluster log”. In the cluster log I saw the following: 2014-10-24...
View ArticleWhat to do in case of...
I’m currently gathering my first experiences with Apache Spark and in particular Spark SQL. While I was playing a bit with Spark SQL Joins I suddenly faced an exception like Exception in thread "main"...
View Article
