Disclaimer: I'm a former Red Hat employee, but only speak for myself. I'm also going to speak very frankly.
You got it. From the customer side support from them is cheaper than Red Hat's (sometimes much cheaper). From the supplier side, they don't have to bear much (or any) of the cost of development (beyond infra hosting and re-branding), so they can beat Red Hat's price very easily. I personally find that gross and unethical, but that's just my opinion.
I think (or at least I hope) that anyone who buys a support contract from a third-party company for a third-party distribution understands that they are not getting the same level of support that they would from Red Hat.
There is lots of room in the enterprise sector for third-party support. This is, for example, what almost all consultants effectively are. The third-party vendor can solve a lot of problems that the customer may not have the experience to deal with and can be well worth the price paid. But at the end of the day if there's an actual bug all the way upstream in RHEL, only RHEL can (permanently) fix that.
I've dealt with Red Hat in the past and one good thing about their support is that if you have a particularly thorny problem or a genuine bug, you will often eventually end up talking directly to someone who is either wrote the code or sits next to the person who did.
You make some great points, thanks (side note: I love HN for conversations exactly like this one). You've won me over somewhat. As long as the support vendor isn't misrepresenting what they offer and how it differs from Red Hat, it doesn't seem nearly as unethical as it felt initially. I've only had experience with a handful of vendors, but all of them marketed themselves as "same support as Red Hat, 1/4 to 1/2 the price." I find that gross, but if it were "we'll help you setup and configure your machine" rather than "use us instead of Red Hat" I don't really have an issue with that.
Nobody ever got fired for hiring red hat for red hat support. To the other post’s point those may not be the contracts red hat wants. They want big enterprise where there is 0 reason to not go red hat.
It goes further than that. If you file a bug on Red Hat's bugzilla with or without a support contract, you will quite often get a response if you put the effort into producing a detailed report. No guarantee, sometimes just others with the same issue, but still pretty good. You will also find Red Hat employees on the mailing lists for the OSS projects they contribute or depend on who actively participate in conversations there.
If you have a support contract though and open a case, the level and quality of support is usually very high.
This isn't my experience at all. I've personally had three paid support cases attached to Bugzilla issues. Two of them were for bugs that had one-line fixes that were already committed upstream, along with test cases, and that I just needed Red Hat to backport. They sat for about 6 months before I got any response other than the automated monthly "our engineers are working on the Bugzilla." The third was for a problem that existed upstream too. It was opened over 2 years ago and still isn't fixed.
The components for the first two were NSS (Network Security Services, not Name Service Switch) and GCC, and the third was Kerberos. I don't have the BZ number handy right now.
For the first two it's probably that they were considered not bad enough for an asynchronous release. With CentOS Stream, you would have a better understanding of when the patch was committed and more certainty of whether it would be fixed in the next minor (6-months cadence) release.
Feel free to write to me at pbonzini@redhat.com, I don't work on Kerberos myself but I know people in the team.
I've been involved with companies buying from IBM for over 20 years. Since the Lotus Notes days I've always been impressed by their support chasing down hairy issues. Like, their tier1 would escalate to T3 when mostly needed. Compare to Google where robots lock you out for days. Expensive but never had a client be mad about it.
Former consultant here: There are a lot of entities out there that aren't really a good fit for Red Hat's service offerings, but who may still need the system. As others have said, Red Hat is a great provider if you may need support from someone deep in the bowels of the exact code you're having a problem with.
Having consultants out there using CentOS/Rocky/whatever gets these "not a good fit" customers burden off Red Hat, which is probably an advantage to them. I operated what amounts to a "phone a Linux friend" service for ~18 years, and that sort of help just doesn't really fit into Red Hat's offering.
It's nice in many ways to be able to disconnect from the licensing model and just be able to deploy Linux boxes, but still get help when you need it. I recall one conversation about Red Hat licensing related to a machine I was installing for a client named "The Fedora Project" that went like this:
"I assume you'd like me to put Red Hat on there?" "Yes." "Can you provide me with a license key to use on it?" "Uhhh. Just go ahead and install CentOS".
You're forgetting to mention the part where Rocky Linux is not just a rebrand of RHEL, it's a revival of CentOS, which IBM killed off in what's effectively a bait-and-switch, forcing customers to go through the painful/expensive migration process to another distro, or the less-painful but still expensive migration process to RHEL.
Rocky Linux is a shining example of both a free market and the open source community working to the benefit consumers. I don't see how that's unethical.
Sadly, the decision to kill CentOS was entirely Red Hat's. The self-appointed 'community managers' decided that the community didn't actually want CentOS, they wanted a free version of RHEL-Beta called "CentOS Stream."
Nobody said it was a community choice. Red Hat's own layered products (OKD, RDO, oVirt) needed to be based on a distribution that leads RHEL rather than one that trails it, and that's why CentOS Stream was born.
Dropping CentOS Linux was a completely different thing and one should also acknowledge that there are two very different parts of the CentOS community.
Those that simply needed a free RHEL, didn't have any benefit from CentOS Stream. However, their usecase is filled by Alma/Rocky.
Downstream CentOS distributions however only got benefits from CentOS Stream. There are many private ones, for example Facebook runs on a CentOS derivative, but the most prominent example is Alma itself, which existed (IIRC with another name) even when CentOS Linux existed.
And to be honest, only the latter are really part of the community. Downloading an ISO doesn't make you part of the community. I myself used CentOS Linux on a small EC2 VM but I didn't consider myself to be part of their community (I have since switched to Amazon Linux, for what it's worth).
So all that Red Hat did was basically restructure their collaboration with downstream distros. On one hand they enabled those distros to collaborate even more to RHEL development, which is now public (including individual patches to the kernel, if you remember the circa 2011 kerfuffle). On the other hand release rebuilds are entirely in the hand of the community.
Now, I am not saying everything was perfect. The announcement sucked in many ways, and there still isn't a good solution to use RHEL container images on public CI. People inside Red Hat (including me) will all tell you the same. However, it's intellectually dishonest to ignore that there was and is a CentOS world that goes beyond "I need free Linux and I don't/cannot use Debian", and Red Hat has been very receptive to the needs of that world.
> CentOS world that goes beyond "I need free Linux and I don't/cannot use Debian", and Red Hat has been very receptive to the needs of that world.
There really wasn't a CentOS world beyond that, because it wasn't a true community distribution. Red Hat never attempted to meaningful involve the community, all board members or whatever pretend org CentOS has are RH employees. There were never any kind of community elections or anything of the sort.
There totally was. Facebook had been running CentOS long before Stream, and had several RPM backports to CentOS 7 on GitHub. And guess what, they love CentOS Stream. The default desktop distros for Facebook developers are Fedora and Stream.
Honestly if all you know about CentOS is "download ISOs and report bugs that with some luck will be forwarded to RH Bugzilla", you don't know anything about the CentOS community.
Again: downloading ISOs doesn't make you part of a community.
Facebook is the exact example the self-appointed CentOS leadership used. So you're either part of that cohort, or you're really drinking their koolaide.
Well, it's the classic "We don't need support until we need support" perspective from the CFO or CIO or whomever that's looking to cut costs. I've had clients who always had some excuse for why their mission-critical production box still couldn't justify the cost of a proper RHEL license ("No! If build my PROD box on RHEL, then my DR box will have to be RHEL, and my Test box will have to be RHEL! So you see it's really three licenses I'd have to buy!") and always insisted on CentOS. But despite needing everything to work correctly 100% of the time and mandating several levels of redundancy, they were curiously okay with being up the creek if they ran into an OS problem. Go figure.
FWIW, clients I dealt with were universally thrilled with RedHat support, the ones who had it. I had one tell me over the phone once, "Oh yeah! Red Hat is amazing! If an issue gets assigned to an engineer, it will be resolved by that same engineer. Unlike you guys."
absolutely, I saw that quite a bit as well. I was really glad to see Red Hat change their policy so that non-prod machines are free now. That makes it a lot more affordable for people doing CI/CD with staging and dev environments. The old model disincentivized good practices.
I'd say it's a two-sided sword. There's no question RH employs/founds large parts of Linux development, with only Suse being remotely as involved (maybe historically). OTOH, RH has pushed "innovations" such as systemd purely in their own interest, fragmenting a once-strong and user-centric F/OSS Unix community also including the BSDs into a Linux-only cloud slavedom. Plus, it was IBM/RH who cancelled the CentOS roadmap (after having bought-out the CentOS project and community); they can't now expect to be treated as trusted bona-fide Linux steward or something.
IBM had nothing to do with the CentOS decision. It was long time Red Hat people who made the decision. I don't agree with everything about the decision, but I don't think it's as bad as most people say it is[1].
You are definitely right that RH has pushed things in their own interest, but if those things don't offer value to the broader community, then the community won't adopt them. Red Hat can't force Debian or Ubuntu or Arch to adopt anything. They can push it through Fedora Cent and RHEL, but that's it. The other distros adopted systemd because it offered benefits/improvements over existing things like Upstart. I like firewalld, but that's a good example of something that is only on RH despite RH pushing it. If systemd was really such a negative, then you'd see distros like Devuan take off. A frequent criticism is things like, "Red Hat made Gnome dependent on systemd" which isn't wrong, but they didn't just do it because they could. There were real benefits there.
I think the reality of life is that there will always be people who want things to change, and those who don't want the change. To succeed you have to find a balance.
Also important to remember when decrying "fragmentation" (which I decry also btw), in a massive heterogeneous community like open source, you're gonna have users who have completely different needs and use cases, and both are valid. The beauty of it is the code is open and free, so people can serve niche use cases as well as standard.
While I agree with you, it is fair to say RH has an unusual amount of leverage with respect to forcing things. They directly control a lot of big ticket projects, and have powerful leadership positions in others. They can coordinate major changes across the board and the momentum they can throw behind some decisions can certainly exert a LOT of pressure. This isn't necessarily a bad thing though - as you note it can be good to have a leader, but it's also something that can be detrimental too.
> I don't agree with everything about the decision, but I don't think it's as bad as most people say it is[1].
It was bad because when I chose CentOS for servers at work I thought I could count on letting them live their lives for about 10 years without worrying about them. I have a lot of things to do and taking care of those servers is a very low priority.
Thank God there was a script to convert my CentOSes to Rocky and that it went great, otherwise it would have taken time I don't have to reinstall those servers.
Yes I will 100% grant on that one. Pulling the plug so early on CentOS was low and I don't think they should have done that.
The whole "we never put an EOL date on it" excuse was really weak. It looks to nearly everyone like just an unintentional omission, not a legal hedge. It ended up working out well like you said thanks to Alma and Rocky, but the panic and uncertainty it caused was wholly unnecessary.
They also created quay.io and fragmented docker global public repo by asking sponsored FOSS project to publish ONLY on quay.
I really dislike redhat and won't touch anything related with them.
I really don't understand why companies would pay thousands per server for support. I'm working with Linux systems since 2 decades and never needed to pay a cent in licenses or support.
Red hat didn't create quay, they acquired it. But even if they had made it, I think that's an important and healthy thing to have. You could call it fragmentation, but I see it as taking on a monopoly by offering choice. Docker fully supports alternative registries, so I wouldn't consider that fragmentation either. Remind me a lot of when Apache was the only http server, and nginx came out. Some people said fragmentation, but most said, "cool, we have a choice and some market competition, without which development tends to stagnate"
There are things to dislike about red hat, but quay is one I would praise them for instead.
I appreciate your candor, but what makes this "gross and unethical" exactly? RedHat itself makes money by charging for support on work made by thousands of other coders outside the company (i.e. Linus et al). It's hypocritical if you ask me.
Good point, there's some hypocrisy at play here. I think overall though the "taking" done by RH is on the whole much less than most of the 3rd party support vendors. But my original language was a little harsher than I really feel.
Red Hat funds an enormous amount of the development of projects (like the kernel), and makes Fedora one of the best (IMHO the best) distros for personal computing, so is one of the top contributors to the community at large. Their support also actively fixes bugs and sends them upstream.
Contrast that with many of the cheaper 3rd party supports, who rarely if ever send contributions (beyond bug reports, which are sometimes a positive contribution, but frequently are net drain because the bug reports don't contain enough info to be reproducible or actionable). They also don't do much or any development.
Of course this is a broad stereotype. I'm sure you can find 3rd party support providers that do contribute to the community.
> RedHat itself makes money by charging for support on work made by thousands of other coders outside the company
Linux is not developed by thousands of volunteers on their free time in their garage.
At least 85% of commits to the Linux kernel are from corporates (AMD, Intel, Broadcom, RedHat, etc). Linux is free and open source code, but very largely written by people who are paid to do so, by companies that need to somehow make a profit.
RedHat amounts to roughly 20% of these corporate commits, and that's counting in the share of other corporates that frankly mainly only contribute work for their own drivers.
I see paying for RHEL as being 50% for support / 50% for sponsoring Linux development as a whole.
Unethical would be if the companies selling the support then turn around and use their (one) Red Hat license whenever there's a real difficult problem.
But the vast majority of "support" for Linux isn't engineer-level, it's likely config and setup. Which is where both Red Hat and others try to make their money.
Redhat builds off of a mountain of open source software that they didn't write. To call it gross and unethical is blatantly hypocritical. If Redhat doesn't want others to use software they create in certain ways, then they should change the license of the OS. Oh, that's right, they can't, because it's not all their software.
Yes, this is a good point to make explicitly, thank you. The support is from 3rd parties. Those 3rd parties are probably kicking some funding back, but I don't think Rocky is doing anything unethical by offering their rebuild or by accepting that money.
I think overall Rocky is a net positive for the world and for Red Hat.
and RH doesn't have to bear much of the cost of the development of Linux, GNU, and many other products. This is the spirit of OS.. If you think that's unethical, you should find another employer ;)
edit: ah.. former employee.. so that part was already done :)
> RH doesn't have to bear much of the cost of the development of Linux, GNU, and many other products.
I disagree. RH is one of the top contributors to many of the major projects that make up the distro. Also the process of building/maintaining a distro is itself enormous. The 3rd parties have none of that expense so they can undercut the cost easily. In the end it hurts the whole ecosystem, while benefiting a select few.
If this hurts the whole ecosystem, all community developed distributions must have been hurting it, too, by your bizarre interpretation of free software and/or arrogant take on ethics.
Explain how anything you've said so far in this sub thread can't be summed up to "red hat good, everyone else leeches".
No, it's the same. I think the amount of upstream contributions (i.e. patches) that the entity sends offsets some of the ethicality deficit. At some point it would even go positive if they send enough upstream fixes.
Obviously this is entirely my opinion :-D
There's also an impossible-to-measure factor in the form of eco-system benefit though. For example, I would never have paid for RHEL had I not entered into the eco-system through Fedora and CentOS. So while RH didn't make money from my CentOS usage, it did eventually make them money because I bought RHEL later when it was worth it. I don't know how you would calculate that, but it does offset ethicality deficit somewhat as well
hi - thank you for speaking plainly to a large tech audience. Isnt there some "market correction" due though, overall, since OSS and Linux have become so central, so deeply performant, while the engineers and other "community" repeatedly get zero money.. Although the point of simply cloning and re-selling the work of RedHat, perhaps with support claims, might look bad, we overall have to allow some growth for non-centralized players right?
Yes, I agree there's value in decentralization. And to clarify, I don't think there are is anything unethical about offering a RHEL clone like Rocky and Alma do. I think that's a net positive for everyone, even Red Hat. My beef is more with the people that sell support which directly undermines Red Hat and ultimately hurts all eco-system users because it means less development, less QA, etc.
That said market competition in general is a good thing, and I don't doubt for a minute that Red Hat prices would be a lot higher without the competition. It's a complex equation that's impossible to calculate since the inputs are immeasurable and in many cases theoretical.
Did CentOS not do the same? RH bought them, right? Then changed it to a rolling release experiment. I guess that was a talent acquisition? So RH benefited off the previous attempt to do this.
Also RH packages open source project and also does not necessarily gives back to every project.
The right to help your neighbor is important to open source. I find Rocky does just that.
Yes, IBM/Redhat decided to make CentOS less attractive to those who need a RHEL work alike. This was done on very short notice, leaving people with the uncomfortable decision to A) pay for RHEL B) tolerate a rolling release (not directly related to RHEL 8 or C) switch to another OS.
Rocky and Alma popped up for folks that want a bug for bug compatible RHEL OS.
Out of curiosity, what sort of problems would customers typically run into?
My very naive understanding is that it’s just the OS and then only even a distribution (just to emphasize- very naive), what would they need apart from some networking and to run some software on top that needs so much support?
It really runs the gamut, from bugs in included software to "why are we dropping packets" (usually misconfiguration) to an update broke our existing config or an update failed to install, etc. I saw one ticket that was "missing build dependencies" where they couldn't build their own code because of a package difference, and while Red Hat didn't fix their code, they did give them some helpful pointers.
Your notion of ethics is unequivocally and objectively wrong. Red Hat derives massive value up to and including its very existence and every single penny that passes through its hands from participation in the larger open source ecosystem that comes with the obligation to share their work not only with random bob hacker but with people like Rocky Linux, Alma, and even folks like Oracle.
You cannot acknowledge that they owe everything based on willing participation in an ecosystem where they wherein they openly made a promise and follow that up with the idea that its immoral for anyone to rely on such a promise. It's logically and morally inconsistent.
Gross is just an emotive label you have slapped on after incorrectly concluding that such actions are unethical.
You can't argue a position by linking to a Wikipedia article about a logical fallacy.
It cannot be unethical to rely on a freely given promise that the other side has derived substantial value from.
Red Hat received the entire basis of their business by virtue of their side of that promise they cannot and indeed haven't tried to withhold performance of their obligation in that regard.
Under what ethical theory do you propose that creating downstream distros is unethical? Is it likewise unethical for Ubuntu to exist? If your answer differs why?
RedHat licenses are just too expensive by sticker price. Beancounters haunt us for deploying RedHat because "the Windows license for that box would have been cheaper in our licensing model". That there is support included which we never use doesn't matter to them, the usual suggestion being "buy support for one box, test everything on that one, and open a support case for that one box, replicate the solution everywhere".
If you want to fix this, make a RedHat license be significantly cheaper than the equivalent Windows product. Charge for support by ticket/case and only support licensed boxes. You'll earn a lot more because it'll look cheaper to the beancounters.
I don't think it's fair to be that harsh.
Ultimately Red Hat incurs costs from engineering RHEL. They fund that through support contracts. So obviously a company that doesn't incur these costs can sell support contracts for cheap.
While it's legal, I agree there are grey areas around the ethics. Of course it's OSS, so you can't have expectations, but just imagine if all RHEL customers switched to 3rd party support. Red Hat would stop existing. Then what would these 3rd party support companies do? They'd go under too, as they'd have nothing to support.
I'm just using the exact same words used by the OP...
No one asked them to do that and put it in open source, if they are not happy they can go to closed source or different license... Like Microsoft... Maybe the new OP employer? That would make a lot of sense when I see his attitude
Live by the sword, die by the sword. Red Hat can develop a closed-source OS and sue people who copy it, but then no one would use their product. If one of your major selling points is open source, not to mention benefiting from all of the non-RH developed code that is in RHEL, you can't complain about the perfectly predictable consequences of that.
RH publishes all their sources (including the MIT/BSD/etc ones that they don't have to publish). They're the very opposite of "closed source". I don't know of them suing anybody. And nobody is complaining about people using the code, or about the existence of Rocky or Alma or any other rebuild distros.
You are confused. Read my comment again. I’m just saying, if you don’t want people cloning your software and competing with you for support contracts, then don’t make it open source. It’s absurd to make your product open source, use open source as a marketing point, incorporate millions upon millions of lines of other peoples code into your product, and then complain when people do exactly what your open source license gives them the right to do.
And you literally said:
>I personally find that gross and unethical, but that's just my opinion.
Yep, definitely not illegal, although IMHO there things that are legal but still unethical. And the line of ethicality is highly subjective. There is sort of a limiter in place in that if it got too bad RH could kill it pretty quickly by not publishing all the SRPMs. Although if they did that, I think the ecosystem would fall apart. I would bail.
Part of my irritation is definitely a bad taste in my mouth still from Oracle.
You got it. From the customer side support from them is cheaper than Red Hat's (sometimes much cheaper). From the supplier side, they don't have to bear much (or any) of the cost of development (beyond infra hosting and re-branding), so they can beat Red Hat's price very easily. I personally find that gross and unethical, but that's just my opinion.