Why isn't differential dataflow more popular?

Differential dataflow is a library that lets you write simple dataflow programs and a) then runs them in parallel and b) efficiently updates the outputs when new inputs arrive. Compared to competition like spark and kafka streams, it can handle more complex computations and provides dramatically better throughput and latency while using much less memory.

But I'm only aware of a few companies that use it in production, even though it's been around for 5 years.

Possible explanations:

• It's missing some important feature, like persistence? (Rejected explanation - it doesn't have error resilience builtin - it's sufficiently fast that on all but the largest workloads it will complete well inside the MTBF anyway)
• It's had very little advertising?
• The api is too hard to use?
• The docs / tutorials are not good enough?
• Rust is intimidating?

Even more surprising is that noone has copied the ideas into some enterprise friendly java monstrosity - despite the fact that differential dataflow is open source and is explained in depth in many papers and blog posts.

I'm interested because materialize is expending a huge amount of effort adding a SQL layer on top of differential dataflow. That's all very well for people who like SQL, but I'm curious whether there are also potential users who would have been perfectly happy with javascript bindings and a good tutorial?

If you considered using differential dataflow and decided against, please let me know why.

from Hacker News https://ift.tt/36jTCwL