Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Sell talk and buzzwords. Either author has no idea that Hadoop is ecosystem and Spark depends on it or deliberately mix Hadoop and Kubernetes, which aren't much related.

And good luck running Spark without Hadoop ;)



Spark doesn’t have a hard dependency on Hadoop. Spark doesn’t have a storage engine, but you don’t necessarily need one.


Spark still depends on Hadoop for a lot:

- Using Parquet files = parquet-mr which is tied to Hadoop MR https://github.com/apache/spark/tree/master/sql/core/src/mai...

- Using S3 instead of HDFS = Hadoop S3a connector

Even if you don't run HDFS and YARN, you aren't escaping Hadoop. And if some configuration goes wrong, and you'll probably need to look into the Hadoop conf files.

The original comment was about the mass of libraries that Hadoop brings in. Spark isn't a solution that allows you to leave the mess. If you try to dockerize spark, you'll still see that you have 300 MB size images full of JARs that came from wherever.


Yes, but my comment was about serious, production grade setup.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: