My observation is that many teams lack strong "technical discipline"; someone that says "no, don't do that", makes the case, and takes a stand. It's easy to let the complexity genie out of the bottle if the team doesn't have someone like this with enough clout/authority to actually make the team pause.
I think the problem is that this microservices vs monolith decision is a really hard one to convince people of. I made a passionate case for ECS instead of lambda for a long time, but only after the rest of the team and leadership see the problems the popular strategy generates do we get something approaching uptake (and the balance has already shifted to kubernetes instead, which is at least better)
I 100% agree with you but also sad fact is that it’s easy to understand why people don’t want to take this role. You can make enemies easily, you need to deliver “bad news” and convince people to put more effort or prove that effort they did was not enough. Why bother when you probably won’t be the one that have to clean it up
I've implemented a distributed worker system on top of this paradigm.
I used ZMQ to connect nodes and the worker nodes would connect to an indexer/coordinator node that effectively did a `SELECT FROM ORDER BY ASC`.
It's easier than you may think and the bits here ended up with probably < 1000 SLOC all told.
- Coordinator node ingests from a SQL table
- There is a discriminator key for each row in the table for ordering by stacking into an in-memory list-of-lists
- Worker nodes are started with _n_ threads
- Each thread sends a "ready" message to the coordinator and coordinator replies with a "work" message
- On each cycle, the coordinator advances the pointer on the list, locks the list, and marks the first item in the child list as "pending"
- When worker thread finishes, it sends a "completed" message to the coordinator and coordinator replies with another "work" message
- Coordinator unlocks the list the work item originated from and dequeues the finished item.
- When it reaches the end of the list, it cycles to the beginning of the list and starts over, skipping over any child lists marked as locked (has a pending work item)
Effectively a distributed event loop with the events queued up via a simple SQL query.
Dead simple design, extremely robust, very high throughput, very easy to scale workers both horizontally (more nodes) and vertically (more threads). ZMQ made it easy to connect the remote threads to the centralized coordinator. It was effectively "self balancing" because the workers would only re-queue their thread once it finished work. Very easy to manage, but did not have hot failovers since we kept the materialized, "2D" work queue in memory. Though very rarely did we have issues with this.
Yeah, but that's like doing actual engineering. Instead you should just point to Kafka and say that it's going to make your horrible architecture scale magically. That's how the pros do it.
Kafka isn't magic, but it's close. If a single-node solution like an SQL database can handle your load then why shouldn't you stick with SQL? Kafka is not for you. Kafka is for workloads that would DDoS Postgres.
Kafka is really not intended to improve on this. Instead, it's intended for very high-volume ETL processing, where a classical message queue delivering records would spend too much time on locking. Kafka is hot-rodding the message queue design and removing guard rails to get more messages thru faster.
Generally I say, "Message queues are for tasks, Kafka is for data." But in the latter case, if your data volume is not huge, a message queue for async ETL will do just fine and give better guarantees as FIFO goes.
In essence, Kafka is a very specialized version of much more general-purpose message queues, which should be your default starting point. It's similar to replacing a SQL RDBMS with some kind of special NoSQL system - if you need it, okay, but otherwise the general-purpose default is usually the better option.
Of course this is not the same as Kafka, but the comment I'm replying to:
> Ah yes, and every consumer should just do this in a while (true) loop as producers write to it. Very efficient and simple with no possibility of lock contention or hot spots. Genius, really.
Seemed to imply that it's not possible to build a high performance pub/sub system using a simple SQL select. I do not think that is true and it is in fact fairly easy to build a high performance pub/sub system with a simple SQL select. Clearly, this design as proposed is not the same as Kafka.
No, I implied that implementing pub/sub with just a select statement is silly because it is. Your implementation accounts for the downfalls of this approach with smart design using a message queue and intelligent locking semantics. Parent of my comment was glib and included none of this.
> ...and internal systems they have to use, whose sole purpose is to make sure nobody does anything
I once had to use Lotus Notes after the company I was at was acquired by the now defunct Computer Sciences Corporation. I decided I would never, ever work for another company that used Lotus Notes.
In a lot of ways Notes was ahead of its time. You could easily have encrypted replicated databases with offline work, which was very handy for traveling users back before high bandwidth connections were widely available, and you could build quite complex apps on top of those databases.
I saw at least one large company that migrated from Notes to exchange and they got the email/calendaring bit done quite easily and were still running notes servers for line of business applications years later.
Notes was pretty decent as a groupware/ nosql platform. Lotus script wasn’t great. I might be biased because my first CS job was to write applications with it.
It felt like they basically tacked on the email functionality to to Notes to sell it, but it always seemed kinda ok to me.
> Normal people, though, hate that. Instead of giving them a tool to get their job done, we've given them a puzzle
Normal people already have a process and mental model of fitting tools into that process. Loosely coupled tools -- as inefficient as they may be -- have the benefit of being able to conform to those existing processes and models. It's why the humble spreadsheet is still so widely used as the glue between people and processes because it is just rigid enough while offering endless flexibility and handling of edge cases.
On the other hand, highly integrated or opinionated tools need to be both easy and flexible enough to fit existing models/processes or it will simply overwhelm new users. Or it has to have some other benefit that is significant enough that users are willing to change their models/processes.
The spreadsheet is a great example. The core mental model is extremely simple: it's the same as a paper spreadsheet but you can put calculations (formulas) in a cell. That's it! That's the mental model.
Once you go beyond that model, people get confused. What's a pivot table? What's the difference between $A1 and A$1? Why is my date showing up as a number?
But it works because 80% of the people only need to know about the basic mental model. If you try to reduce email, notes, meetings into a single mental model, then you're back to the puzzle problem and people have to solve it to do even the simplest thing.
You may or may not remember Lotus Improv, which was effectively a reinvention of the spreadsheet. In Improv, the core mental model was the pivot table, which allowed you to do some amazing things (e.g., "add up the sales for each quarter by product type"). But even the simplest things required people to understand the core concepts of a pivot table (columns, categories, rows, etc.). Tech people loved it. Normal people, not so much.
> What is missing from most documentation is simple Use Case.
There's a lesson to be learned from Microsoft's success with SharePoint in the 2007-2013 time frame where they absolutely sold a ton of deals in various verticals for what is otherwise a super generic productivity platform comprised of docs, calendars, task lists, etc.
Microsoft's strategy was simple: have vertical specific SMEs, sales teams, and solution delivery teams that build solutions that addressed specific use cases for those industries. Life sciences, legal, manufacturing, you name it: Microsoft had a vertical specific team that understood the use cases in those verticals. Their sales teams configured vertical specific demos. They shipped vertical specific templates and configurations for those use cases.
I ended up in life sciences (pharmas, biotechs) by happenstance and we helped customers get SharePoint configured for use cases like operating clinical trials on SharePoint.
The underlying platform is incredibly generic, but having your sales people speak the domain space and pre-configured solutions (or solution delivery partners) made it more palatable.
I think most companies that are building these kind of tools need an "on-ramp" that is industry or use-case specific designed with SMEs or customer observations for those use cases.
I wonder if that is why it worked as opposed to why it sold. At least, as one of the users in that time it seems to me that the itch was the document repository with some features, and the rest was fluff for purchasing to sign off on.
All the calendar, task, website stuff ended up dying away, because what we really needed was a good document management system, with optionally some simple signoff loops and notifications.
That was great. Yes, it was just unix like tools with a window. That is the craigslist of os improvements.
It really was the use case, but the simple one. I saw plenty of power users try to do complex and ultimately fragile uses that died away.
In the case of this user, I love the idea of turning an email to an action, but I also need to add that to my action board and assign it, and check people time, at which point the simple action only makes sense for individuals, not teams or orgs. and I need to add a couple missing actions, and summarize. So suddenly the all in one is a marginally useful tool that is also a straightjacket. And then I go back to outlook and jira and excel and trelli or whatever.
> All the calendar, task, website stuff ended up dying away, because what we really needed was a...
That's part of my point: the sales strategy was brilliant. Whether or not SharePoint's built in calendar or Project Server integration made sense is almost secondary; they understood how to sell it.
> Nonsense. Average Java/C# is an enterprise monkey who barely knows outside of their grotesque codebase.
Netflix is Java. Amazon is mostly Java. Some of the biggest open source projects in the world are Java. Unity and Godot both use C# for scripting.
I don't know where you're getting the impression that Java and C# are somehow only for "enterprise monkey who barely knows outside of their grotesque codebase"
> Netflix is Java. Amazon is mostly Java. Some of the biggest open source projects in the world are Java. Unity and Godot both use C# for scripting.
You can add Meta, Google and Palantir to your list and it won’t change that average Java dev is from an Eastern hemisphere and knows nothing about Java outside of JavaEE/Spring.
> Before about 2016, you could have said the same about Intel.
I'm going to guess that part of the problem is American business culture and ceding high level strategic decisions not to engineers but to MBA types. It's hard to see anyone falling the same way Intel fell looking at companies like Nvidia and AMD whom are both still (outside looking in) very much engineering driven.
Do you have any facts backing up that opinion? Because while I’ll agree that MBAs who ignore engineering nuance can be a problem, engineers are perfectly capable of running an org into the ground all on their own.
In this case, Intel looks like a variant of the Innovators’ Dilemma. Their internal processes, systems, and culture revolve around designing and manufacturing their own chips. Moving to a customer-centric approach is a big switch in culture and I’m not surprised it’s a challenge.
What decisions were MBA instead of engineering decisions? It seems like intel has just made a lot of bad bets or failed to put their mass behind good ones.
The heights nvidia has achieved seem incidental and have depended heavily on the transformer/LLM market materializing.
Intel's biggest problem has been management remaining in denial about their serious engineering problems, and believing that they'll have things sorted out in another few quarters. They were years late to taking meaningful action to adjust their product roadmap to account for their ongoing 10nm failures. Putting all their eggs in the 10nm basket wasn't an engineering decision, and keeping them all there after years of being unable to ship a working chip wasn't an engineering decision.
Intel's in a somewhat better place today because while they continue to insist that their new fab process is just around the corner and will put their foundry back on top, they've started shipping new chips again, using TSMC where necessary.
Stock buybacks and huge sums of capital wasted on mergers and acquisitions (that went nowhere) while not investing in the very expensive EUV fabrication equipment that TSMC had been using for years.
If you can't prove knowledge gained, would that not indicate that the pursuit was fruitless?
Regardless of your endpoint in that pursuit, you should have gained intermediate foundational knowledge along the way, even if you haven't arrived at your endpoint.
If you cannot show mastery of that intermediate knowledge, then any kind of journey for knowledge would have failed.
reply