The bigger issue is that they count small based on fixed number of parameters, and not the active parameter for MoE, didn't account for any hardware improvements etc. If they counted small based on the price or computational cost, I think they would have seen increase in small models.
I think using total parameters is fair, it correlates well with the RAM prerequisites to run it. Otherwise Kimi K2 would be "small" despite being a trillion parameters!
Depends. Look at the graph of month year of professional hockey player[1]. Player born in first quarter is twice as more likely to be in pro leagues than last quarter. Month of birth's only effect is that it gives 0.5 year extra during junior year to be in spotlight. It shouldn't affect player's performance in any other way. And the effect persists for decades.
If you get supported initially when you aren't the best, the effect of the small support can make you much better player.
In the US, USA Hockey (by far the biggest youth hockey organization) groups players by birth year. So if you are born late in the year, you are among the youngest players on your team. You tend to be smaller, and less experienced, and unless you are exceptional you tend to play less. This impacts you from your first youth teams up until high-school.
> How much money can they make by integrating advertising and/or selling user profiles?
I would say if executed well the revenue per user could be at least an order of magnitude more than Google search ads as the ads could be much more convincing and the information density is higher in chat.
Such a bad (AI written?) article. These kind of introduction to advanced topics feels like how to draw an owl tutorial where they spent so much time diving into what group is.
> The group of all rotations of a ball in space, known to mathematicians as SO(3), is a six-dimensional tangle of spheres and circles.
This is wrong. It's 3D, not 6D. In fact SO(3) is simple to visualize as movement of north pole to any point on the ball + rotation along that.
That is very strange. It's certainly not an academic level explanation, but that's not what the magazine is for. But the blatant incorrect statement is beyond the pale. Dim(SO(N)) = N(N-1)/2. Thus SO(4) has dimension 6.
The quality of this article is par for the course for Quanta Magazine, sadly. I do not need to accuse the author of using AI to explain the data I'm seeing here. It feels like every submission on HN from Quanta garners the exact same discussion: The article is almost worthless because it presents complex ideas in such a cheap, dumbed-down, and imprecise way that it ceases to communicate anything interesting. (Interested readers can fare much better by reading other sources.) It's been this way for years. The phenomenon is almost Wolfram-Derangement-Syndrome-like.
Then why don't they do it? It's the easiest money they can ever make. Even I can litigate the hell out of it if I get $27B, take the money and close the LLC.
If the bank transaction is eventually consistent, it means that the state can flip and the person receiving will "never" be sure. A state that the transaction will be finished later is a consistent state.
Let's say OpenAI signed a commitment contract that they agreed to spend XXX USD in your company over N years. You invest in infrastructure, your contractors sometimes take loans, the construction companies take loans. Countries / Funds lend money to such companies (example: Saudi Arabia fund), these funds themselves raised debt, it can quickly spiral down.
If OpenAI fails, wouldn't their customers just move to other AI providers? So the total hardware demand by AI companies wouldn't dramatically change over a reasonable time frame?
If OpenAI fails, the assumed case is overcapitalisation relative to demand. In such a world, if the reason OpenAI has no liquidity is because there's insufficient demand, there's no customers to move across.
What happens if the OpenAI failure is because of lack of demand from their customers and the market in general? None of the AI providers or hardware providers will survive, no?
Similar to certain banks in 2007/2008, the idea would be “so much investment is tied to one company that if that company went bankrupt, it could have consequences for the broader economy”
The thing is, it is not 2007/2008 any more. The US government is holding record amounts of debt and countries around the world are now trying to become independent of it. This includes its bond markets on which the dollar relies upon to give it its reserve currency status, which in turn is what gives it its power to print money and bail industries out. If something happens that requires Big Tech to be bailed out and international bond holders decide the US is no longer reliable, it could very well end up triggering the collapse of the US dollar as the world's reserve currency and the downfall of the US as we know it.
The stock market will lose faith in AI companies, which will crash the stock price of Google, Microsoft, Oracle, Nvidia and CoreWeave. Investors will lose billions, many of those investors are pension funds. Any AI projects that aren't already profitable will shutdown.
And, because AI is currently what prevents the US economy from being in a recession (at least that what some people speculate), the US economy will stumble, which means that everyone else will to.
Is it unclear? Compared to other times a "too big to fail" industry failed?
If OpenAI crashes, for example funding stops, they go broke, fall behind, nobody buys anything, then all the money they invested for data centers or demand they created for NVIDIA chips and compute collapses. That creates surplus of hardware, causes lots of construction/buildout / stockup orders to get cancelled, and the whole thing ripples as suppliers and construction and data center providers etc etc suddenly lose a ton of anticipated profits.
Share prices drop as people dump to protect their portfolios, anticipating dips in the prices because share prices will drop as people dump to protect their portfolios (I'm not kidding).
Given that the big 7 AI companies are basically _all_ of the market growth lately, it doesn't even take a serious panic / paranoia episode to see the market itself stagnate or significantly regress, as people pull from anything AI related, and then pull from the market itself anticipating the market will fall.
That's different from saying that boeing is too big to fail for example. The US can't accept to lose its only major commercial aircraft manufacturer. Or Intel for similar reasons.
But what you're describing is about keeping the AI bubble from popping. Can a bubble really be too big too fail?
What I'm describing is the scare-quotes too-big-to-fail. Some actually are. But we use that term to mean anything that might cause significant economic trouble nowadays.
You can't "require" manual intervention. Sure you can say that the keys stays on say 2 developers laptops, but personal devices have even more surface area for key leak than CI/CD pipeline. It wouldn't have prevented attacks like this issue in any case where the binary just searched for keys across the system.
One alternative is to do the signing on airlocked system stored in physically safe but accessible location, but I guess that's just way too much inconvenience.
The discussion is not about the product where you can just remove the stuff. The thread was testing in small setting and then moving to oddball setting. If it is required to cover oddball settings, it makes sense to know and plan for oddball setting.
Single point of failure means exactly opposite of what you think it means. If my work depends on 5 services to be up, each service would be a single point of failure, and correlation of failure is good for probability that I can do my work.
"If one thing I need is going to be down, everything might as well be down."
If I have a product with 5 dependencies and one of them is down, there's things I can do to partially mitigate. A circuit breaker would allow my thing to at least stay up and responsive. Maybe I could get a status message up and turn off a feature flag to disable what calls that dependency.
On the other hand, if all my dependencies are down AND the management layer is down AND the AWS portal is not functioning correctly, I'm pretty much SOL.
Massive centralization is never, ever a good thing for anyone other than the ones who are doing the centralizing.
So if you can just run without one service, what's stopping you to remove the dependency altogether. Why would you only want to remove the dependency when service is down.
So e.g. to get real my application depends on AWS's EC2, RDS, EKS, S3 Cloudflare's DNS, and Redis' instance. If any of those stop working it will go down. If everyone is within SLA, they might as well go down together than separately.
This is a really interesting point, because I could see a situation where your application requires integration with say 10 services. If they all run on AWS, they either all go down or all run together. If they're all self-hosted, there's a good chance that at any time one of the ten is down, and so your service can't run.
reply