At an expensive Apache Spark training class. For hands on portion you use their system. When everyone logged on, their system couldn't handle the load. First they blamed the hotel wifi (even though every other site was running perfectly) and then they blamed AWS. It's clearly an issue with their system. It's ridiculous to not be honest with a room full of developers who know what's up.

Fucking shit uni is such a waste of time. We are learning Apache Spark in Big Data module. Fucking losers have Spark 1.6.0 installed while the latest version is 2.2.1 right now.

What a bunch of cunts. We are paying tons of money to study deprecated shits and a degree. A fucking degree that is not even on a piece of paper anymore.

Fuck this shit man.

To be honest, I'm not as excited as I was 6-7 years ago when our tech industry seen a big leap, where these ML/Deep Learning algorithms were out performing humans, Apache Spark out perfomed Hadoop in distributed computing, Docker/Kubernetes are the new phenomenon in software development and delivery, Microservices architecture, ReactJS virtual DOM concepts were so cool.

Really though, I've come realise that these software trends come and go. All you need to do is adapt and go with the flow.

Just finished dumping all ethereum tranasctions into one big 30 GB csv.

Only thing left is to configure Apache Spark cluster.

I'm not sure if it is a dev experience, but definitely boosted my morale.

In 2014, my company (in India) sent me to attend a conference in Boston. The conference was about big data.

When I came back, I wrote a blog post about Apache Spark in my company's blog. Because of the blog, my name got mentioned in a prominent newspaper's article about Apache Spark.

PS: That is my only blog post till date

I am working with Apache Spark on Scala for my undergrad research. It got me thinking: what is Scala used for beside doing stuff with Spark?

Not sure whether it's just me but most posts talking about Scala seem to always mention Spark.

Damn Apache Spark. Instead of supporting just one language that works perfectly well, let's support several languages that give users different results! It's so much fun to combine chunks of each language to get a fully working solution!

I have unsubscribed myself from the Apache Spark mailing list but I keep getting emails from them regarding PRs and comments on the repo. What do I do now? Block them or mark as spam?

One of the things that gives me a lot of satisfaction is doing integrations. One customer has a good architecture, and they chose to put their metrics database with billions of records in Apache Cassandra. Which is pretty cool! A business consultancy that is helping them grow a lot, has implemented PowerBI and the dashboards are really good, but the most important reports will be based on this gigantic Cassandra database. We just delivered a quick integration using Apache Spark. Small project, fast delivery and everyone happy. Is so good. Mission accomplished feeling.

It seems like it never came to the mind of the developers of Apache Spark that people might want to use two of their libraries together to perform graph analysis on streamed Twitter data.

Why can't I simply use the streamed batches to create and extend a graph and perform continuous graph Algorithms on it??