In my experience, it's a complete myth that requirement gathering can be done in a box.
"Requirement gathering" is almost always influenced by technical capabilities. All but the simplest projects, require back and forth with guestimates about the direction and approach.
This is because its difficult to decouple the code from the requirements. And the full requirements are never contained within a doc. So you need an engineer to look at the code to see how its done. This is going to change.
So someone who doesn’t understand code is going to know whether some AI generated description/summary of existing code is accurate, without being able to verify that in anyway? And then create requirements based on that summary? I use GPT-4 as a coding assistant pretty much every day, but it a. makes mistakes or strange assumptions all the time b. can’t parse the logic of or describe complex codebases c. fails to account for obvious edge cases, like an array being empty instead of populated. I mean, maybe some companies will do this, but I hope never have to use their products or they contain any sensitive information.
Again, where do you guys work? Maybe lets not use software engineering. If I asked you today, to do requirements gathering for building a bridge across the Mississippi river, could you do it? Don't you think you would have to learn about materials, regulations, land procurement, tensile and compression strengths and all the tradeoffs of the above? And once you learned all of that and you can finally present the requirements, would it be fair to call you non-technical?
I have to strongly disagree with this. If someone non-technical does the requirements gathering, the requirements will be incomplete. Developers will then either assume their own requirements or, if they’re thorough and good at their job, check these assumptions with the business. Either way, this wastes the company’s time. Either the assumed requirements will require re-work or the company has paid two employees to do the same job that should’ve been done by one.
> The economic benefits of mining asteroids are also too large
Ironically, that's an example I like to list as "pure sci-fi fantasy, divorced from economic reality."
The total cost of the iron ore that goes into making a new a car is about $200-$300 dollars, depending on various factors (size of the car, ore spot price, etc...).
Even if -- magically -- asteroid mining made not just "iron ore", but specifically the steel alloy used for car bodies literally free, new cars costing $30,000 would now cost... $29,700.
You can save more by skipping the optional coffee cup warmer, or whatever.
In reality: 90% of iron and steel is recycled, and asteroid mining is not magic.
Why would anyone want to mine asteroids for iron, given that it's the most abundant element on Earth? I don't think I ever recall seeing a proposal like that outside of the broader notion of "space factories" (where such mining makes sense to reduce the cost of shipping materials back and forth, not because it's cheaper as such).
It's stuff like platinum and germanium that makes asteroid mining potentially interesting.
On Earth, geological processes concentrate elements into ores, primarily through volcanic and hydrological means. Neither are available in small, cold asteroids devoid of liquid water. Hence asteroids are generally undifferentiated mineralogically, making mining them much less economically viable.
You often see total quantities listed as an amazing thing, glossing over the fact that the Earth has more of everything and in usefully concentrated lumps.
Market competition and innovation in both ML and hardware has consistently driven down the price of AI in the past decade. You only have to look at where we are with capabilities today compared to ten years ago when CIFAR100 classifiers were the state of the art.
Barring a Chinese invasion of Taiwan, these APIs will halve in price over the next year.
What would a Chinese invasion of Taiwan do to "tech", it sounds like it would be awfully catastrophic ?
On the other hand, for all peoples worrying about China, they are pretty restrained given the enormous turmoil the invasion would cause. If they wanted to break the USA, now would probably be the time to do it?
Given what's used right now to my knowledge, the main ones would be Nvidia's tensor cores, Apple's M chips and Google's cloud TPUs. All of that's TSMC I think?
Sometimes I am jealous of these people. They arent doing the corporate grind, and they arent even really doing the startup grind. They are shilling stuff on the internet and avoiding spending their lives working. I know a few, they live great, though frugal lives. I go to my job at prestigious big Co and often wonder who the smart one is, me or them
The genius of all such people, I noticed at the start of my career in the 2000s, is that they take an idea that--for most folk--warrants maybe 3-5 mins of passing conversation at most (in the case of AI, maybe a few minutes more) and somehow twist, wring and hydraulically frack _years_ of blog posts, Twitter threads, speaking tours, books, podcasts, etc. out of it.
Their real brilliance is in identifying worthwhile candidates for prolonged shysterism and sophistic schmoozing, and, like an exceptionally gifted oil and gas industry geologist, to extract that stored potential hype energy to a degree no normal person would think possible.
This happened with Agile, with TDD, with pretty much any technocratic crapola that was laundered through the ostensible prestige of TED, and by god, did it ever happen with "blockchain", NFTs, and anything crypto-related.
Outside our particular sociocultural nexus, you most often see it with sundry species of MLM grifters, in the tradition pioneered by Quixtar-Amway and widely emulated, well, everywhere that lower middle-class consumer eyeballs go.
Don't be fooled by the glamorous facade of such "influencers". It's as tough a grind as any corporate job. There is 24x7 demand for new content. You have to compete with a million others in the space, and it is next to impossible to set yourself apart. Only the top 0.1% have any real viewership and are making money. For the rest, the "living frugally" part you mention is likely not by choice.
And whilst churning out bad takes about AI on Twitter is about the lowest hanging fruit out there for LLMs, it's a fair bet none of these 'experts' on 'AI disruption' have outsourced their tweets to one yet...
Oh I'm 100% sure ChatGPT can do it brilliantly, I just suspect that most of the people trying to be 'AI influencers' (i) actually enjoy coming up with middlebrow takes on AI, reading others' middlebrow takes on AI and monitoring their timeline for retweets and (ii) lack the tech skills to set the ChatGPT API up to tweet on their behalf...
Just think, you could make months of blog posts about your journey to get ChatGPT to generate pro-AI tweets for you.
And then you can follow that up with your journey to get ChatGPT to give you the code for a system that hits CharGPT and then posts the tweet automatically.
And then you go silent and the money keeps rolling in because ChatGPT is grifting your viewers on your behalf now!
Yeah, but they're taking a major risk that there will be a continuous audience willing to spend money on their placebo products. When money becomes tight they'll have to pivot hard to avoid going under, or go back to a "real" job but with little experience to command a higher salary.
It's the difference between investing and day trading. Day traders may get lucky and post massive, envy-inducing gains, but boring long term investors keep building during the downturns that wipe out the day traders, and so win in the long run.
Besides, being forced to live frugally doesn't sound that great to me. You aren't free, you're just constrained by money instead of people. Guess for some people it's worth it, but to me they're just two sides of the same coin. There are many meaningful experiences in life that cannot be had below a certain level of prosperity.
Exactly what I was about to say. They know the shovel saying and said to themselves "shit I can't make or sell a shovel, but you know what I can do? Talk!! A lot!"
Most people ought to be afraid of the AI salesman. They're equally as harmful as the cryptobro, selling some nebulous tech that the layman will always misinterpret. Their profitability layer is being able to confuse and obfuscate the components that go into their business, as well as being able to outwit their customers.
I'll proudly hate and harp on those folks anywhere I go. I won't go as far as to claim they "must be stopped", but internally I'd wager most people are waiting for these guys to shut up already.
I'd love to see an analysis of people who were shilling crypto a few months ago and moved to AI, I'd imagine it's almost a perfect circle. These types just move to the newest hot tech trend and use FOMO to drive engagement
"If you don't learn about every single GPT frontend tool made this week in my twitter thread you will be poor!"
times like these you need to think about the Lindey effect(LLMs, general AI advances) and not the flavor of the month stuff. Similar to software engineering, don't obsess with every new frontend framework, learn fundamentals that will always be useful to know
Have you heard of "non-sponsored influencers"? They're your average person who starts creating videos shilling brands, for free, in the hope that they'll get picked up for a real endorsement deal.
You may know one of the ones that made it for real, but I see a lot of delusional hopefuls simply copying what the others are doing and hoping to strike it rich.
I'm not jealous. People with the exact same skills are landing jobs at McKinsey selling this bullshit to the C-suite. These guys are at best scraping the bottom of the barrel (Twitter impressions) to sell a course.
Lyft was paying engineers 300-450k a year, sometimes with just 2-4 years of experience. This was true even pre 2020. The interviews were just leetcode hards, with an instant reject on not getting the problem right. I feel like a parrot repeating the same thing, but none of these companies hired the talent they thought they were hiring.
EDIT:
For reference I work at a company that requires leetcode medium/hard perfection and am appalled at the money we pay for the engineers relative to the value they bring
One more point: Startups are confused about the role engineers play. A completely divorced product management and engineering workforce will kill new ideas. Engineers actually building the product have insights into what else can be built that product does not. By hiring engineers with zero product sense and treating them like robots, you get no innovation. This doesnt matter at a behemoth like amazon or google, but it does at a smaller company. Clean syntax != good engineer
This seems to be a common trope at HN where the failure of a company must be because of their hiring practice. Your comment also implies that the current failure has been engineering. This couldnt be further from the truth, Lyft had some of the best and smartest engineers. Lyft paid as well as they did because of the risk these engineers took in moving on from Google/Meta/Twitter what have you.
FWIW, I found Lyfts interviews to be the easiest of the companies I interviewed at.
The companies current downturn stems from the combined blow of COVID and focusing only on rideshare at the expense of not diversifying their revenue streams and their hiring or engineering has had very little to do with it.
OP says they don’t think the engineers hired by doing leetcode hard problems (and screening out the folks who can’t) does not engender a good future for the org.
Your argument is “I found the interview easy, and hence it’s not that bad.” Just pointing out that perhaps OP thinks the culture you’re part of is not a good one. I’m not taking sides here but curious what this clarification means to you.
Your point about engineering not being part of the problem is another. It’s partly true - unless your technology is true main selling point (like openai or maybe google search), then sure engineering makes or breaks the future of the company. So in one way you yourself point out the commodity nature of what companies like Lyft do - given how many companies around the world do it as well now that is. Then the question becomes what exactly are you paying top dollar to these engineers for? Create 4 more unnecessary open source projects to make sure the engineers feel like they’re getting exposure?
Airbnb is the epitome of this. Like dudes, you rent out rooms, what are you trying to do creating open source ecosystems in data engineering for?
Elons original thesis of Twitters problems was absolutely correct, why you needed 8000 people to run that org (or whatever the number is for Lyft especially on the engineering side) is beyond me. Overengineered is absolutely the correct term in these cases I think.
My argument is not that 'I found interview easy and hence not so bad'. My argument is that in Lyfts case engineering hasnt been the problem with why its business is failing.
> Then the question becomes what exactly are you paying top dollar to these engineers for?
Again Engineering is one vector in a company's success but not the only one. It might well make sense for Lyft to pay well for good engineering talent there. Maybe not paying well might have doom'd them much faster.
Your take on OSS is also a naive one . OSS helps in off boarding long term maintenance costs (if the project becomes popular enough) and staves off bit rot. It helps attract talent , creates industry standards etc. Engineering is feature multiplier and for some companies it does make sense to OSS.
I’m not against OSS, just frivolous ones funded by companies who should have no business doing that stuff. It’s not news that orgs like Uber were indulging the development and release of such projects so they can keep the fairly jobless highly paid engineers happy. There’s no way you can make an argument that the development and maintenance of airflow had any real bottom line effect on their ability to make a home rental market place. The true technology companies don’t and never did release oss solutions that were not fundamental things they could do well and needed to do well. I’m again happy (maybe I’m not) that airflow exists thanks to airbnbs indulgence, but as a bottom line for the company it doesn’t make sense. Especially in the current environment where you now are suddenly forced to show real value for your spend on engjneeeing.
I am sad that Lyft isn’t going to make it through, not only cause of the engineers but for their contributions for OSS and blogposts. I got some of their work as inspiration for some stuff. However, I acknowledge that some part of this can be a product of a ZIRP period.
Your views were commonplace 3 years ago. Now, the idea of paying for free OSS development with expensive engineer time, at an unprofitable company, in the middle of a tech-recession, is indefensible.
I would be careful not to express luxurious views in future interviews. Saying that Lyft engineering wasn't swollen and engaged in silly side-projects flies in the face of the layoffs announced in the article we're talking about.
An interviewee expressing those views would be seen as out-of-touch, if not stupid.
> Elons original thesis of Twitters problems was absolutely correct, why you needed 8000 people to run that org (or whatever the number is for Lyft especially on the engineering side) is beyond me. Overengineered is absolutely the correct term in these cases I think.
Anybody who read the mythical man-months knows that adding engineers to a project gets be detremental if it leads to a point where the marginal added communication overhead gets higher without really needing more people to achieve the goal.
The usual solution in big companies for the overengineering problem is that people who don't want to fight the org self-select, drink lattes all day long, and let the few people who are needed to solve the engineering problems do the work.
>The usual solution in big companies for the overengineering problem is that people who don't want to fight the org self-select, drink lattes all day long, and let the few people who are needed to solve the engineering problems do the work
Lyft actually did try and create many different revenue streams. Besides ridesharing, they own the bikeshares and scooters in pretty much every major city in US, got into their own car rental business and tried to become a one stop shop for transit. It's not that they didn't try, its that they didn't seem to have taken any of them to be profitable revenue streams.
Also, wouldn't the very high salaries paid to a lot of engineers contribute it Lyft having unsustainable economics?
I mean sure it's more complex than "hurrdurr bad engineers, not enough Rust etc etc" but to succeed a company needs products of a suitable quality level delivered in a suitable time scale to meet the needs of their customers. The failure to do that the buck stops at the engineering group.
If ride share can’t break even very soon then the company is fucked either way. At this point the VC money trucks are long gone and public markets have little interest in unprofitable companies long term.
How is it possible that the money trucks kept coming for nearly 15 years on a business that has no path to profitability?
In finance, such situations usually imply that someone in the middle of the transaction made money while grifting others - is that all that was going on in VC?
The general assumption by their investors was they had a path to profitability. Critically they don’t need to convince most investors just enough to keep going.
If you really want to understand VC’s from a finance perspective it comes down to trying to value intangible assets. Having ongoing customer relationships is inherently valuable. Lyft could for example sell their customers email address to scammers, but the goal is to leverage those relationships more sustainably thus extracting vastly more money.
For a rapidly growing unprofitable company it really just becomes a question is the long term value of those intangible assets (software, customers, patents etc) worth more than the current burn rate. If it is then you have a profitable investment in a company that will eventually become profitable or get bought out by someone else.
When a company’s engineers ship quality code fast, they end up with more bandwidth. Then the company does more shit. What new has Lyft really built? It could be a symptom of slow engineering.
> Your comment also implies that the current failure has been engineering.
Well to an extent it's failure of engineering.
The best engineers are not code monkeys. But people who can map the business domain to code perfectly.
Given this is HN, we've all seen engineers who make business impact. some had degrees, some didn't.
And we've seen engineers - who complicate matters while not actually shipping usable features , but doing things the GRRM martin way of plucking weeds. \
maybe step away from the mega corps and notice impact engineers have on a tech company's business direction.
Stock based compensation is a huge drag on their expenses, to the degree that it has hurt their future. "Lyft had some of the best and smartest engineers", whats the bench mark for this? Good looking syntax in a clean code base isnt a top engineer. Delivering business value through technology is
Seems like that lends more credence to the idea that product strategy was the problem then no? How does a company decide to hire angineers and the products they'll be working on? It's true that engineering roles across the industry were overpaid in a way we're probably never going to see again, but isolated, that was just a symptom of the general amount of money sloshing around the tech ecosystem and doesn't really have anything to do with the reasons Lyft may or may not fail. The business model as it exists (at current pricing levels) is unsustainable. Even Uber is only achieving profitability because of its investments.
Its disingenuous to claim that these engineers werent delivering business value. James Gosling worked on Java at Sun and Sun failed, would you say that Gosling isnt a good engineer ?
Engineering can only work in the bounds of the problems set to it. If the leadership doesnt want to diversify streams , get into delivery for example there is nothing engineering can do there. In the end you need both good business acumen to succeed and engineering is just a multiplier and facilitator there.
Engineers can be passive recipients of assignments from business leaders. Or they can find ways to influence without authority and drive the initiatives that they know are needed. Everyone has that opportunity to some extent, but in some companies it's not worth the effort to break through all the obstacles.
> Isn’t using James Gosling example as a question a fallacy in argument (i.e., loaded question)?
No, it's a clear way to refute the absurd argument that a company struggling or even failing automatically means all their engineers are incompetent morons.
It's pathetic how there are people in this thread throwing blanket accusations of incompetence at engineers being laid off, primarily because they were far better paid and had far better jobs than them. It reeks of envy, and complete lack of empathy.
I talk about this when I do hiring. Our process isn't designed to find the best person for the job. We want to hire the first person who passes our bar for hiring. We just don't have the need or budget for the best and smartest people. But we often hire very excellent and very smart people!
It's kind of like the high jump. You get the same result whether you're an inch over the bar or a foot.
Just due to the law of large numbers, any company at or above the size of Lyft is going to have mostly very average engineers. There just aren't many "10×" engineers out there in the market regardless of how much you pay. The key to effective management is figuring out how to obtain extraordinary value from average people: this can be done but it's largely a matter of organizational culture.
Very average? How do you figure that? If you're paying top dollar, which Lyft did for most of its existence, you're going to picking from a pool of the top few percent. Even 1% of all software developers is many times more than Lyft's headcount.
I'm curious, who do you think is hiring the 10th percentile engineers? What are those people doing?
That's not how it works in the real world. There are a bunch of other companies also targeting the top few percent. Mathematically it isn't possible for them all to succeed in that goal regardless of their hiring process or compensation levels. If you actually talk to those engineers it's clear that beyond ability to solve "leetcode" type problems they are mostly pretty average.
Why not claim you have engineers that deliver the best value to shareholders? Employing the smartest people in the world to make an app to skim money off taxi going from point A to point B can't possibly be a judicious good value use of human resources, especially in the context of the extreme premium those at the absolute top demand.
I hope this downturn gets rid of leetcode. I really have no desire to grind leetcode whenever I need a new job. I am more than happy to do system design and practical coding questions (ex: integrate with some API like Stripe does), but the leetcode stuff is ridiculous for mid-senior+ level engineers.
If you think of Leetcode and similar filters as just veiled IQ tests it makes way more sense - of course those problems don’t have much to do with actual day to day engineering. If companies could just administer IQ tests they would happily do that. Since that’s illegal, they rely on Leetcode.
It's a modified IQ test filtered through the transfer function of having a CS degree. A big part of the shade that this practice gets is from people with backgrounds that are CS adjacent who are highly intelligent, can do math at a university level and perfectly capable of writing quality production ready code are dinged because they don't know some CS 201 puzzle that is never used in practice.
Interviewing isn't just a measure of how good you are at programming in real life. It's how good of an employee you will be. Conformity is a strong indicator of this. Comparing someone who completed a CS degree versus some guy that learned to program in a garage hacking together broken computer parts - the employer is going to choose the CS degree guy, even if the hacker is better at programming. CS degree means you follow direction, you can work within a rigid structure, you follow rules enough to make good grades and not get kicked out, and you're capable of doing monotonous work that sometimes has very little value.
eh, not really. even if you have a CS degree you forget how to do typical algo hard problems quickly after working in industry because you so rarely need to do them. the CS degree lets me _recognize_ when there's a problem with a particular complexity domain that needs a time- or space-efficient solution to scale, but I sure as hell can't recall the exact strategy on the spot like interview scenarios demand.
given a day or two, sure, I can go review, find that solution, and provide an implementation and explanation as to why it matters for a particular problem domain, but an hour-long phone interview isn't gonna check that effectively, unless they really want to see the initial stage of me silently reviewing maths literature to awaken knowledge in cold storage
on contrast, I _can_ explore systems design stuff fine in that space of time, because systems design _is_ what I practice regularly in industry, because that's what I'm regularly asked to do.
You're trying to frame this about college degree vs none. I'm sure that's a valid point and a college degree has value, but that's not relevant if we are talking about a degree in CS vs a degree in EE/Math/Physics. CS degree does not mean that you follow direction or work within a rigid structure any more than a degree in any other mathematical science. The Leet Code question biases the evaluation very strongly to tidbits of CS trivia that may or may not be relevant to the actual job. This will filter out someone with a background in say electrical engineering, even if the job is about doing linear algebra and signal processing (areas where EE is probably a better training than CS).
I detest leetcode-style questions, but system design questions are even worse. At least leetcode-style questions have some type of objective evaluation criteria. System design questions are entirely subjective and whether or not you're successful generally comes down to whether or not your first impression of a problem matches the expectations of your interviewer.
At my last company, we stopped doing system design entirely because we found it was completely uncorrelated to whether or not a candidate did well on the job.
Did you hire people who did poorly on the system design interview and then measure their job performance? Or did you just determine that "passing with flying colors" didn't really improve job performance vs. "just scraping by"?
We moved from leetcode + system design question to project-based interview + system design question to just project based.
Our project-based interviews were the main hiring criteria, and we originally planned on using system design interviews for leveling. What we found is that doing well on the project-based interview was a much better signal than the system design interview. When we used the system design interview for leveling, we under leveled certain people and over leveled others. We discovered that it would have been more accurate to only use the project interview for both making a hiring decision and leveling.
Ultimately, what we had trouble with was formalizing the evaluation criteria for the system design interview. With a project interview, we formalized the requirements of the project and everyone on the loop evaluates candidates against those criteria. With the system design interview, only one person ends up evaluating each candidate and a single person's judgement is highly variable. Unless the candidate did something egregiously wrong, it was very hard to say whether or not something was strictly "bad" or "good" and it was impossible to write down the list of "bad" and "good" qualities ahead of time.
How did you structure your project based interview? Was it live pair coding, or a take home project?
Did you have any trouble with candidates declining the project? My first instinct as a candidate is to turn down take home project based interviews. I almost always say no to them (especially if it’s for the 1st round) for a few reasons:
One: In general they seem like a poor use of my time. I would rather do 5 one hour coding or design interviews at 5 different companies than a single 5 hour project at one company.
Two: I’m skeptical that anyone is actually reviewing the submitted project. And the evaluation criteria are usually unclear. For instance is it worth it to add linters during the build process? Or to introduce caching for api responses? Will the person evaluating the project notice details like that?
Lastly: Every coding or design interview carries over to the next one. Preparing for one is preparing for all of them. In contrast I’ve seen projects (for a backend role) range from “build a simple weather api, any language” to “build a sudoku solver specifically in JavaScript”.
The project interviews are in-person. They take between 4 and 5 hours, so the time commitment is roughly the same as any other on-site interview process. No preparation is required. In fact, I could tell you the question ahead of time and I would still get useful signal out of the interview.
Our process was to have someone "setup" the interview by introducing the project and helping the candidate get set up. Then the candidate would work on their own. Periodically, someone else on the loop would stop in, see how things were going, answer questions, etc. At the end, the last person on the loop would go over their code with them. This was a very important step. Hearing someone talk about their own code is just as important as the code itself. It's also interesting to see how candidates debug things, should any bugs arise. Finally, we would debrief. We'd bring up their code in a meeting room, test their project, ensure it meets the requirements, find bugs, etc. We'd discuss the code, see how clean it was, whether or not it was nonsensical, etc.
Selecting a project for a project-interview is non-trivial. It's definitely possible to pick a bad project. For instance, sudoku solver sounds like a bad project. The evaluation criteria is too strict and it sounds like there is probably an "optimal" answer. Basically, it sounds like one long leet code question. A good project question has a number of qualities:
1. The project needs to be a language of the candidates choice. You want to see how they code, how they think about problems etc. If you give them a skeleton, you're constraining their thinking and will get a weaker signal. It's also very informative to watch a candidate setup a new project. Good engineers will easily setup a new project in the language of their choice. Weaker candidates will suffer.
2. Evaluation criteria needs to be clear. We had 5 requirements that the project needed to meet. It was objectively easy to say whether or not the requirements were met. Failure to meet any of the requirements was essentially an auto-fail. I've seen project interviews where they don't tell you exactly how your code is evaluated and then run a test suite against your code. Forgot to test a negative number? Auto fail. How is that reflective of work done on the job? Such opaque criteria are terrible.
3. The project needs to be well-defined, but still somewhat open ended. We had 5 criteria to meet, but it was amazing the variety of responses we got. New grads would struggle to get all 5 done in time. Experienced engineers would use the time to go above and beyond. Going "above and beyond" wasn't really necessary. We could still judge skill just from the basic requirements, but it was cool to see how far some people took the project in the short time given them. This is a reason something like "sudoku solver" is a bad question.
4. The problem needs to involve some sort of data model / state tracking. A lot of software engineering involves tracking state, so we found it's important to see how people actually do that.
5. The current team needs to do the interview themselves. For instance, going back to a sudoku solver, how many people could sit down and do that in a short period of time? How many people spend their time thinking about discrete math problems like that? Personally, I don't like doing that, so it seems weird that I would require that of a candidate. Everyone on the team should feel comfortable that the project is achievable and reflective of the general work done on the team.
Unfortunately it will likely get worse in the short term. With a huge pool of laid off people fighting for a small number of open positions (especially at the senior level and above) companies can be as demanding and stupid as they want with hiring. Leet code prowess is just another screening that whittles down the work they have to do to hire--only people that study and learn those interviews will pass them so the company has to worry less about actually figuring out if someone is a good fit for the position.
Leetcode is also just an awful signal. It would be fine if it was only problems with false negatives, but the false positive rate is through the roof.
I'm consistently shocked by coworkers I've had, who I knew had to pass challenging leetcode interviews, who seem to not only know nothing about building real world software, but don't really have a firm grasp on algorithms.
Yes they can process a leetcode question requiring dynamic programming in record time, but when mapping a real world problem to a dp (or any other similar solution) are completely at a loss.
I don't think it will be the downturn that eliminates it, so much as it will be LLM-accelerated coding, and the normalization and incorporation of these tools into standard development workflows.
If you think co-pilot is cool now, just wait and see what it does in 5 years. A team of 10 tomorrow can probably replace a team of 50 today.
So maybe the interview process in 10 years is more like: "build an auction site like ebay in the next hour."
Who needs LLM-accelerated coding for that? Leetcode was already comically unrealistic as an evaluation of actual engineering skill. When's the last time you had to solve an algorithm puzzle without any ability to do research or check references?
Shoutout to the interview I had where I was asked a tricky graph question, reasoned my way to the optimal solution, then failed because I "should have recognized" that it was a variant of Djikstra's algorithm from the beginning. As we all know, being able to regurgitate an existing algorithm shows far more promise than being able to logically create it yourself!
> So maybe the interview process in 10 years is more like: "build an auction site like ebay in the next hour."
This is very "Monkey's Paw" and I can absolutely see something like that taking hold, as sloppy and stupid as such a task might be.
I had an interview with a YC company who asked me how to do something, and then I did it, and then they asked "how could I make it a one-liner" - immediately thought was "I'm pretty sure I don't want to maintain a codebase full of fun little one-liners for everything" - managed to get it, but it definitely looked like crap.
If I were to get asked to build eBay in an hour, I'm sure my auction model would be 1) shit, 2) very prone to social engineering type attacks (fake bids, etc)
> If you think co-pilot is cool now, just wait and see what it does in 5 years.
What will it do? It can't write exponentially better code. Maybe 50% better, so that I would need to edit it's outputs less often. The only thing I could see it doing exponentially better would be writing _less_ code. Telling me "stop! close this file. close this project. you are working on the wrong thing!"
Technology can jump sharply but then go relatively flat. We still haven't gone back to the moon, 50 years later. We still don't have at-home nuclear power plants.
It’s a balance, an LC medium isn’t to hard to do offhand. If your business is hard tech, it’s a reasonable filter.
I see diminishing returns from LC hards and greater other than as a guardrail to block management from hiring senior people who are practically no better than the junior people.
From reading recent interview experiences on Reddit, the thing got way worse now that you have thousands of unemployed engineers competing for scarce positions.
Now it's leetcode hards AND takeaway assignment.
How would the downturn get rid of leetcode? If anything it will dramatically increase its use as now there’s 100k more programmers on the market all applying for the same reduced pool of jobs.
Somehow companies are going to have to now filter through a lot more candidates to fill positions, and for right now leetcode-like tests are the quickest way to get a gauge of an engineer’s performance (on leetcode).
Don’t get me wrong: I think leetcode hiring depending on how it is done doesn’t result in getting the best talent.
The problem is that you and the other hundreds of senior engineers would perform the same if it were all design and the other example you gave. There's just not a good solution for companies that must evaluate the number of candidates these big companies get daily.
The Lyft service is rock solid, it's just not a very profitable business. That has nothing to do with engineering talent.
Engineers need to stop being bitter about others making more money than them and start roasting the business people that fail to make the company profitable. Even if they make 1/3 of what engineers do, they fail at their primary job function and need to be let go.
> the business people that fail to make the company profitable
people who own the failure will have more luck developing their skills and becoming more marketable, than people who run away from failure and play the blame game. woe be to the contributor, manager or otherwise, who believes their little bubble of excellence will never pop.
Taking a high salary from a structurally unprofitable company is a massive risk. This has been hidden by the endless amounts of cheap money in the last decade but it's always worth thinking about.
I realised in 2016 that Uber/Lyft were never gonna print cash so I didn't work for them.
And one of the puzzling things to me here is what all those engineers are doing. I mean, if you're making bad hiring decisions, engineers can happily create work for one another and themselves, so I get where the labor could be going. But it does not look to me like Lyft and Uber are particularly feature-filled.
I suspect that if rideshare companies ever get to being profitable, we'll see the emergence of white-label rideshare software that gets made for a tiny fraction of the cost of what Lyft and Uber spend. Because that'll be an organization with a real incentive to keep costs down.
Thanks, I hadn't seen that. And yes, nobody should get into that game until prices stabilize such that the rideshare companies are profitable. When the game is setting money on fire in pursuit of a monopoly, you can't compete with Uber.
As always with large companies, what you see as a consumer is only the tip of the iceberg. For every user-facing line of code you have a hundred or a thousand lines of code made internal-only services, for interacting with partners, for variations due to local restrictions, etc.
Sure. I mean I have some questions about your proportions, but let that pass. We clearly agree that the amount of total code is proportional to the user-facing stuff. Which for Lyft seems to be quite stable. And since the main function of developers is to improve things, that makes me wonder.
I also suspect that with larger companies, a lot of that below-the-waterline code is not ultimately necessary, the kind of stuff that if they were running a tight ship wouldn't be there. I certainly get reports like that from friends, anyhow. But it sounds like that's ultimately not an engineering problem but a management problem.
In my experience, you usually have about 5-10 engineers on a team, reporting to a manager. Manager might report to a director who manages about 5-10 managers. Then VPs managing 5-10 directions, an SVP managing 5-10 VPs, CTO managing 5-10 SVPs, then CEO.
Let's say each project touched about 20-40 people.
What projects were they working on that leadership deemed 1200 employees now useless? Was it... 30 growth projects they shouldn't have been working on? 50? 100? 20? 10? Was there just not enough work going around?
> For reference I work at a company that requires leetcode medium/hard perfection and am appalled at the money we pay for the engineers relative to the value they bring
Have you tried not paying that much and checking how much value you'll get in return?
Anyone who can write code can learn to leetcode (artificial time-pressuee aside). There's nothing that sets aside leetcoders from the rest of the engineering talent pool except that they put in the time (or have a competitive programming hobby). I haven't seen any evidence that leetcoders are worse in executing, compared to those that will not (or can not) put in the time.
Are companies laying off people people's they could not execute at the expected level? Or is it because of leadership strategic errors (over hiring)? Even if your assertion that engineers are overpaid is true - that is a failure of leadership. However, I don't see heads rolling in the C-suites, it's the fault of "macro-economic headwinds" they failed to see coming.
You just said that you see no difference between people that take leetcode and those who don't So why the hell use leetcode as a filter if it doesn't filter for anything?
Filtering for people willing (and able) to Leetcode is the point! When you have a large talent pool, you will have to come up with a filter; and there is also an element of luck involved - but I don't think interviewers or interviewees will be happy for an offer to be hang on a coin-toss.
I think theres a point of compensation that you reach where you just stop caring about the work or the company. If I was being payed that much a year, I'd be thinking about retiring early and trying to leave the working stiff life in as short of time as possible since with that compensation, its actually realistic to become a feudal lord in a few years especially buying into the market in a recession and riding the wave in the following years. 2-4 years of experience tells me they are right out of college too, so if you are fine with that standard of living you could realistically be living off 20k a year or less and the rest is just banked. Then in the back of your head you think if you ever do get laid off, you now have a title that was compensated at that much on your resume and new doors open for you, which you can take time opening with your massive nest egg.
Living off 20k a year in a Hcol, where these salaries primarily occur, is next to impossible. Rent alone anywhere in SF area, for example San Jose, is 2k at a bare minimum for a 1BR, or 3k for a 2BR split between a roommate at 1.5k each. That means rent is 18k-24k, not to mention other living expenses (car, utilities, food, travel to visit family, etc.)
If you went to school at say UCLA you are already used to living with a roommate in your bedroom (sometimes four). That's $1000 in san jose. Say you have a partner, now it doesn't even seem so desperate. Now you are at $12k rent alone. Fly spirit and buy the airfare long out to visit mom and dad when tickets are like $200 (i get emails for $49 spirit fares sometimes). call it $13000 with five visits home a year. Food wise you can easily do $50 a week groceries buying basic ingredients like fresh produce and rice or tortillas and beans. Now we are at about $16000 with all your needs met and about $4000 left over for any oddball expenses. SF area has bussing too, and bike lanes if you didn't want to bother with car expenses.
They didn't care. They needed to show investors a plausibly looking story how they "hire the best" by having a process with >90% rejection rate, and they did.
I doubt the interview was literally just Leetcode. They're screening resumes and making judgment calls about who even gets to interview. They also evaluate the candidate's experience and so on.
It's the big gate though. No amount of experience, resume high points, etc. will overcome a failed tech screen or interview. It has stopped every single interview loop immediately that I've seen and been a part of the process. As soon as there's a "no hire" voice in the loop it's over, and the fastest way to get there is not figuring out an interviewer's tech screen question to their satisfaction.
Why is the "bar" for hiring designed around code puzzles(leetcode). Maybe that is not a good signal for hiring at all. You are optimizing for people who just grind away at these silly puzzles vs. having domain experts who have both depth and breadth across a technology stack. I have worked with many a engineer who aced these exercises who imho was entirely unfit for working at a large tech company, their communication was absolutely terrible, and their thinking was quite flawed from a variety of perspectives when it came to systems design and operational rigor. I think we will look back on this time and think what a mistake it was to optimize for employees who were good at solving leetcode.
I mean, the more senior you get the more focus you have on the stuff you describe - system design interviews, leadership interviews, etc. I know a few people that interviewed at Meta E6 for example, and they were explicitly told the leetcode interviews don't really matter as much and they are just there to make sure you can actually code.
Imo Leetcode is fine for younger hires since at that point you are just looking for aptitude and/or someone who has the work ethic to grind.
With that being said, Lyfts hiring practices is probably NOT why they are laying people off. This is a business issue. No amount of infrastructure or hackernews-style idealistic interviews would be able to fundamentally change their position in their business segment, especially if its just for hiring ICs. The people at lyft are plenty talented, contrary to what OP is saying.
> With that being said, Lyfts hiring practices is probably NOT why they are laying people off
My comment was more a critique of the en-vogue hiring practice of basing a majority of a hiring decision on whether the engineering candidate can pass leetcode level X.
As for why Lyft is struggling is really two-fold, not diversifying themselves besides ride-hailing really pinned them into a corner unlike their competitor Uber which branched into food delivery and other gig businesses. In addition they like many other tech companies over-hired due to pandemic money and herd mentality.
These companies are too large and have to rely on hiring managers versus people who might be domain experts for the actual project (who are probably too busy with said project themselves to take time to vet 500 candidates). It's an unsolvable problem I think for a large company. For much smaller companies, maybe you'd get a lot fewer applicants and it would actually be realistic for a team member to take a week to do some hiring for their team. Of course as small companies get successful, they tend to grow, then they lose this hiring relevancy advantage.
Wow, I would hate to work with you. Blaming the failure oof the company on engineering of all departments. And imagine being your coworker and not know the animosity you have towards them and the compensation they negotiated. Or maybe you do show it?
And companies won't learn! For how quickly most small companies seem to (or be forced to) learn on other facets, they've learned nothing about attracting and hiring the kind of engineers you want and how to keep them happy.
I still see the same mistakes, the same mindset, the same process and the same result with very very few exceptions.
If you think that's bad wait till you see how much people a rung or two above that in management are getting paid without even being grilled on how to solve difficult puzzles.
The problem isn't the engineering, it's the business. Changing the interview process isn't going to make their competition disappear or make them profitable.
there was a crazy spike in hiring and salaries during the pandemic, with many companies hiring whoever they could find, as fast as possible (hence with higher salaries). I'm assuming that's what OP is referring to
OpenAI from a research point of view haven’t really had any “big innovations”. At least I struggle to think of any published research they have done that would qualify in that category. Probably they keep the good stuff for themselves
But Ilya definitely had some big papers before and he is widely acknowledged as a top researcher in the field.
I think the fact that there are no other systems publicly available that are comparable to GPT-4 (and I dont think Bard is as good), points to innovation they havent released
Yeah but ChatGPT can already write Terraform and Cloudformation really well. So this doesn't seem that special besides it's already baked into the pulumi eco system.
There is a "need" vs "want" aspect here, but often the boilerplates that we reach for become out of date. In theory, an LLM producing these could remain up to date without maintenance, key phrase "in theory"
There's no guarantee the boilerplate an LLM spits out will be up to date. It'll almost definitely have some outdated code in it's dataset that it can reference.
but how will it be maintained? "IAC" seems like a bit of a misnomer here, if I'm understanding correctly. Now, if Pulumi generated terraform for me to maintain, that'd be a different matter, but this article seems to just produce cloud resources based on LLM prompts, which are not deterministic at all.
Pulumi is an IAC tool similar to Terraform (it actually usually calls terraform providers over gRPC under the hood), just written in languages like Typescript/Python. I think the intention would be to use this AI prompt thing to generate pulumi and then to insert it into your codebase, just like you'd do with terraform, and then it becomes deterministic.
I've used ChatGPT and other tools to generate Pulumi before, so I'm not totally sure how this specific tool from Pulumi is different, but I'd guess they've somehow trained it more specifically on their sdks/docs or something
It produces pulumi programs which are the equivalent of a terraform module or main.tf file. Whether or not said programs are deterministic is another matter.
Reminds me of UML. Instead of laboriously hand-writing all this nasty code, why not use a tool that can autogenerate it for us? After all, all that’s needed is a spec (UML)!
Of course, the spec is the work. LLMs are then just very potent at translating it to be machine-readable.
I work on a research team in FAANG. What it really feels like is one company made everyone else obsolete. And we are going to work working on NLP models that underperform ChatGPT by a huge margin. Twiddling my thumbs and keeping quiet while no one wants to recognize the elephant in the room.
Also, there is no "working in AI", a few thousand people are doing real AI at most. The rest of us are calling an API.
This reminds me of back in the mid 2000's, there were a lot of smart people working on search algorithms at different companies. But eventually, you'd talk to someone smart working on Yahoo Search, and they would just be kind of beaten down by the frustration of working on a competing search engine when Google was considered to be by far the best. It got harder for them to recruit, and eventually they just gave up.
So... I don't know where you're working. But don't twiddle your thumbs for too long! It's no fun to be in the last half of people to leave the sinking ship.
Why do you think OpenAPI is so far out in front? It's not like there's a lot of secret sauce here - most of this stuff (transformers,etc.) is all out there in papers. And places like Google & Meta must have a lot more computing resources to train on that OpenAI does thus they should be able to train faster. Do you think OpenAI has discovered something they haven't been open about?
> Why do you think OpenAPI is so far out in front?
There is a network effect forming around its models. The strengths of its kit speak for themselves. (It also cannot be understated how making ChatGPT public, something its competitors were too feeble, incompetent and behind the curve to do, dealt OpenAI a massive first-mover advantage.)
But as others note, other models are in the ballpark. Where OpenAI is different is in the ecosystem of marketing literature, contracts, code and e.g. prompt engineers being written and trained with GPT in mind. That introduces a subtle switching cost, and not-so-subtle platform advantage, that–barring a Google-scale bout of incompetence–OpenAI is set to retain for some time.
Doubtful "feeble and incompetent" are fitting considering much of the research behind OpenAI was pioneered and published by those predecessors. More like "unwilling", and perhaps for good reason. Time will tell.
The level of denial people are willing to sink into regarding how good GPT-4 is compared to everything else is truly crazy. Not a single other project is an order of magnitude close to the quantitative and qualitative (actual experiential results, not just benchmarks) results that GPT-4 brings.
I feel that there’s significant insecurity among a lot of coders about GPT-4. A lot of them are ignoring the pace of improvement and highlighting the few off chances where it gets things wrong.
I think there's a lot of people writing boilerplate programs who are going to be freed from these menial tasks (i.e. no more Java enterprise application development, thankfully).
GPT 4 is quite astounding. It might be wrong on occasion, but it will easily point you in the right direction most of the time. It still messes up, but like a twentieth of what 3.5 did. Honestly it is like an incredible rubber ducky for me. Not only can I just talk like I’m talking to a rubber duck but I can get fast, mostly informed, feedback that unblocks me. If I have a bunch of things competing for my attention I can ask gpt about one of them, a hard one, go do something else while it types out its answer, and then come back later and move on with that project.
I'm a recent convert, been experimenting with converting one PL to another.
GPT-3.5 will get the gist of what the code is doing, and then provide what looks like a direct translation but differs in numerous details whilst having a bunch of other problems.
GPT-4 does a correct translation, almost every time.
It kills me that there's a waiting list for the API. I have put together some tools to integrate 3.5 into my workflow and it helps for my current task a lot (for others it's useless). But to really shine it needs to have API access to 4.
I recently finally got access to 4 in the API, it's good. It's much better imo at following the system prompt too. Faster than you see in chatgpt I think, not as fast as 3.5-turbo but definitely less tedious.
My only kind of quantitative answer is that I had 3.5 creating ember templates & data and it would get the templates mostly ok after a couple of iterations of fixing errors, almost never on the first shot (if ever). Often wouldn't quite get it in two, and data structures would often be kinda there but not quite. Required a lot more care with the prompts. 4 gave me working things every time first time (except only where it did things that were fine in a template but not currently supported by the custom framework I'm using them in), and didn't need as much hand holding.
Qualitatively, it's wildly different from gpt-3.5-turbo for discussions. 3.5 feels a little formulaic after a while with some kinds of questions. 4 is much more like talking to an intelligent person. It's not perfect, but I'm flipping between discussing a sporting thing, then medical malpractice, legal issues, technical specifications and it's doing extremely well.
If it's affordable for you, I'd really recommend trying it.
For code 90%+ reduction is easily correct. For text and other content, I can't say, but I would guess it's not that good.
Anything involving reasoning, code, complex logic, GPT-4 is a breakthrough. GPT-3.5 turbo is more than good enough for poetry and the other text generation stuff.
My favorite part about GPT-4 is that if it generates code that is wrong, and you ask it to verify what it just wrote - without telling it whether it's wrong or not, much less pointing out the specific issue - more often than not it will spot the problem and fix it right away.
And yes, it does indeed make an amazing rubber duck for brainstorming.
GPT-4 is leaps ahead, and it's improving with every new release. The latest March 23 release is significantly better than the previous one and does a LOT of heavy lifting for my code at least.
At the very least, it's a massive productivity booster.
I've had decent success with Open Assistant, an open source model. I'd say it's within the order of magnitude of ChatGPT, given the prompts I'm looking at, including reasoning prompts. This, I believe, is due to the overwhelmingly clean data that OA have managed to acquire through human volunteers.
I dont work at Google, I think other FAANGS underinvested in this area as they didnt think it was promising. But I will admit, I am suspicious that Google is incompetent. Probably they can come back given how much money they will be forced to throw at it. But Bard is clearly behind and I dont believe their "abundance of caution" arguments for why Bard cant even code.
Google was built by brilliant people whose mission was to make any information available to anyone, instantly.
Google is run by smart people whose mission is to maximize clicks on ads. If a user finds what they’re looking for quickly, that’s lost revenue.
Google’s profit motives are not aligned with useful AI. The better AI is, the less people need to click through to lots of web pages and ads, the less revenue for Google.
I don’t think they can catch up without a major pivot in business model. It’s very hard to be deeply invested in providing more value if it means reducing your revenue.
FAANG companies have distribution. They can sell anything. Give them a bit of time. They are not companies of a single product, they milk people better than governments.
The impression I get is that they purposely limited Bard in certain ways. It does not seem to be willing to code and makes a lot of excuses when you ask it to (at least as of a couple of weeks ago when I last tried it). Did they put this limitation on it because it's not so good at coding or because they don't want it to be abused in some way? In my experience I had to trick it into coding by telling it to pretend that it had created it's own programming language and then implement an algorithm in it. It seemed to think that it had created Python.
> I am suspicious that Google is incompetent.
Google has put a lot of effort and investment into AI. With Bard I get the feeling they're not showing us what they really have - it's like for some reason they're holding back the good stuff, at least that's my suspicion.
I am still wondering if Google is experiencing it Kodak moment.
They have the dominant product that makes them billions and billions of dollars at 'relatively' low cost.
The new dominant product is on its way, but it costs far more to operate and will net them far less money, so... um no one wants to kill the goose that is still laying golden eggs, even though its days are numbered already.
(moment worth photographing): From an Eastman Kodak Company advertising campaign.
(business's failure to foresee): In reference to the Eastman Kodak Company's decline when cameras and film were overtaken by smartphones and digital technologies.
Noun
Kodak moment (plural Kodak moments)
(informal) A sentimental or charming moment worthy of capturing in a photograph.
(informal) The situation in which a business fails to foresee changes within its industry and drops from a market-dominant position to being a minor player or declares bankruptcy.
Kodak Film Commercial - These are the Moments - Baby (1993):
I don't see ads as a big problem with ChatGPT. You could put a side-bar on the right and on the fly recommend products relevant to the on-going conversation.
The cost of computing these ads would be a lot more than today's keyword-based approach, that's certainly a problem. But think of hyper-relevant ads, based on the chat itself. There's a lot of information there, that beats tracking people's behavior online all day.
That depends on ad publishers, right? If they want to sell A, B and C and I am interested in D, then Google's still showing one of A, B or C to me. D doesn't make profit if there is nobody paying for ads.
Google is advertising things we don't need, that's why ad clicks are so abysmal. LLMs won't change that.
I’ve been infuriated with DuckDuckGo on occasion because it refused to exclude certain results.
In fact when you add an exclusion clause it simply boosts those results further instead of removing them.
I’ve been told this is because the underlying search providers refuse to exclude paying customer even when you explicitly don’t want to hear from them.
I could definitely see this happening in LLM answers too and I don’t expect it to be particularly subtle.
100% this -- a Google search requires multiple input queries, refinement, and scrolling a list of possible answers that are really just links to other web sites. The ChatGPT experience is far superior to this, for the average consumer and getting close for the power user. It's a better way to ask the Internet what it knows with a more natural interface that everyone already knows how to use -- real natural language. Less cognitive overhead, no busy search results that require clicking back and forth and (for now) no ads. That last part is key -- ChatGPT is doing Google's job right now, and not even having to run ads. Google doesn't even offer a premium no ads option for search and if they did I doubt enough people would buy it anyway to matter.
If I was Google I'd be worried. Very worried indeed. They either need to dramatically change their entire company within 18 months, or accept they are going to loose substantial amount of market -- and once its gone, it's gone in a first mover, winner takes all environment like what we have right now. Just ask Google themselves what it felt like back in the early 2000's when they completely destroyed the other search engines.
Now it is. Google used to be good too, until ads started looking like search results, and then the first page became entirely ads.
In the future, when you ask ChatGPT to help you write your resume, it will try to upsell you a premium account in linked in. It will withhold its best resume advice only for LinkedIn premium users after all.
> With Bard I get the feeling they're not showing us what they really have
I highly doubt this. If they had it they would show it because if they don't react swiftly and decisively their brand will be in 'catch up' mode rather than out front where they are used to being.
It's not just about hardware, it's about the software infrastructure to go with it. Other than OpenAI most researchers weren't interested in "merely" scaling things up because that was/is seen as simple engineering, unworthy of the great minds who dream up new algorithms.
I don't know about GPT4, but GPT3.5 I'd bet is pretty traditional and boring. It's power comes from a really good, properly curated dataset (including the RLHF).
GPT3.5 turbo is much more interesting probably, because they seem to have found out how to make it much more efficient (some kind of distillation?).
GPT4 if I had to make a very rough guess, probably flash attention, 100% of the (useful) internet/books for it's dataset, and highly optimized hyperparameters.
I'd say with GPT4 they probably reached the limit of how big the dataset can be, because they are already using all the data that exists. Thus for GPT5 they'll have to scale in other ways.
To be fair, if the opposite were true, it might not be wise to admit. Saturating available high quality training data is one of the few ways anyone can see OpenAI slowing down.
1. They would already be using everything they can get
2. They would easily be able to explain what they're not using, without giving away sensitive secrets.
I wonder if we saw the same video - or maybe it is just ChatGPT being "great" in the wild? I see one guy asking another guy simple questions and getting weaselwords for an answer.
Right, that's totally it. I came away thinking the interviewer seemed way sharper than the interviewee which is pretty rare. The sheer throughput and speed of interesting questions was incredible. Too bad that many of the answers were not.
> I'd say with GPT4 they probably reached the limit of how big the dataset can be
I’m curious about this too; not just on the dataset size, but also the model size. My hunch is that the rapid improvements of the underlying model by making it bigger/giving it more data will slow, and there’ll be more focus on shrinking the models/other optimisations.
I don't think we're anywhere close to the limit of sheer hardware scalability on this. Returns are diminishing, but if GPT-4 (with its 8+ k context window) is any indication, even those diminishing returns are still very worthwhile.
If anything, I wonder if the actual limit that'll be hit first will be the global manufacturing capacity for relevant hardware. Check out the stock price of NVDA since last October.
According to financial reports they are building a $225 million supercomputer for AI. What we can probably expect is the same dataset with even more compute ran on it.
There is a soft limit due to the computation required; the currently used model architectures are quadratic with respect to context size, so if you want ten times larger context size, that's going to need a hundred times more effort.
For a little more than a year I worked in an AI startup doing basically everything other than AI (APIs, webapps, devops...), but from what I've seen there the "secret sauce" to AI success is the training process (dataset, parameters, fine-tuning steps, ...). And OpenAI isn't open about theirs since their beginnings.
> Do you think OpenAI has discovered something they haven't been open about?
They have not, which makes me curious about which company gp works for because the "F" and "G" in FAANG are publicly known to already have LLMs. Not sure about Amazon, but I'm guessing they do too.
As an outsider, the amazing thing about ML/AI research is that you get a revolutionary discovery of a technique or refinement that changes everything, and a few months later another seminal paper is published[0]. My bet is ChatGPT is not the last word in AI, and OpenAI will not have a monopoly on upcoming discoveries that will improve the state of the art. They will have to contend with the fact that Google, Meta & Amazon own their datacenters and can likely train models for cheaper[1] than what Microsoft is paying itself via their investment in OpenAI.
0. In no particular order: Deep learning, GANs, Transformers, transfer learning, Style Transfer, auto-encoders, BERT, LLMs. Betting the farm on LLMs doesn't sound like a reasonable thing to do - not saying that's what OpenAI is doing, but there are a lot of folk on HN who are treating LLMs as the holy grail.
1. OpenAI may get a discount, but my prediction when they burn through Microsoft, they'll end up being "owned" by Microsoft for all intents and purposes.
Issue is the data moat OAI is building. They'll have hundreds of millions of high quality user interactions with ChatGPT they can use to finetune their models. What will anyone else including Google have?
Google has been collecting user interactions since 2007 via GOOG-411, which was a precursor to the Google Assistant - I suspect Google has billions of user interactions on hand through the latter. Facebook has posts and comment, Amazon has products pages, reviews and product Q&As and all of them have billions of dollars to draw upon if they choose to buy high-quality data, or spin-up / increase teams that create and/or categorize training data.
They also have deep roster of AI researchers[1] to potentially obsolete LLMs or make fine-tuning work without access to of ChatGPT records.
1. I suspect Google alone has more AI researchers that OpenAI has employees
I'm not sure how deep that moat is. As soon as you open up the API, anyone can distil ChatGPT (or at least, some smaller part of it) by fine-tuning another model on its outputs[0].
I'm guessing that this is the #1 fear for people inside OpenAI have right now.
[0] For the record, I have zero problem with this.
1. ToS make it hard for a commercial entity to do so. So some third parties would have to collect the data first
2. You won't be able to get the hundreds of millions or more interactions that OAI will have (both due to cost of API as well as it being not easy to figure out a good way to generate that many queries for a good multiturn conversaton). Maybe you can make up for it by querying smartly. We don't know if we can right now.
As people make chat bots with openAI and tie them into existing chat services, organizations that offer these chat services will get their hands on that kind of data too.
A lot of FAANG data folks aren't on the teams that were doing research into this stuff and weren't using the latest fruits of that research.
OpenAI has released a ton more easy-to-use-for-everyone stuff that has really leapfrogged what a lot of "applied" folks everywhere else were trying to build themselves, despite being on-the-face-of-it more "general."
I think it’s the way things go usually. The big players have a business to run so they can’t focus much on innovation. OpenAI has the only purpose right now to push AI and nothing else. Once they have a real business they will also slow down.
They have been collecting human feedback data for 2 years + probably have a lot of data from Copilot + are training with large context models + have invested a ridiculous amount in curating pretraining data -- the kind of stuff that won't get you a ton of pubs (so you won't see Google researchers having focused on it a lot) but apparently turns out to be super important for a good LLM
All of the neural network architecture for human level thinking and processing, including vision, speech, emotion, abstract thought, balance and fine motor skills, everything was publicly released in April 2003, twenty years ago this month. It's a 700 megabyte tarball and sets up an 80b parameter neural network.
What? Huh? Yes the human genome encodes all human level thought.[1] Clearly it does because the only difference between humans that have abstract thought as well as language capabilities and primates that don't is slightly different DNA.
In other words: those slight differences matter.
To anyone who has used GPT since ChatGPT's public release in November and who pays to use GPT 4 now, it is clear that GPT 4 is a lot smarter than 3 was.
However, to the select few who see an ocean in a drop of water, the November release already showed glimmers of abstract thought, many other people dismiss it as an illusion.
To a select few, it is apparent that OpenAI have found the magic parameters. Everything after that is just fine tuning.
Is it any surprise that without OpenAI releasing their weights, models, or training data, Google can't just come up with its own? Why should they when without turning it into weights and models, the human neural network architecture itself is still unmatched (even by OpenAI) despite being digitized twenty years ago?
No, it's no surprise. OpenAI performed what amounts to a miracle, ten years ahead of schedule, and didn't tell anyone how they did it.
If you work for another company, such as Google, don't be surprised that you are ten years behind. After all, the magic formula had been gathering dust on a CD-ROM for 20 years (human DNA which encodes the human neural network architecture), and nobody made the slightest tangible progress toward it until OpenAI brute forced a solution using $1 billion of Azure GPU's that Microsoft poured into OpenAI in 2019.
Is your team using $1 billion of GPU's for 3 years? If not, don't expect to catch up with OpenAI's November miracle.
p.s. two months after the November miracle, Microsoft closed a $10 billion follow-on investment in OpenAI.
Until we see otherwise, don't we have to assume there's some secret sauce? Bard doesn't match GPT4 and it isn't for a lack of trying. (though perhaps that will change, so far that's the case)
Bard would not trick anyone into thinking it was sentient, yet something they have supposedly did. I just think Google has far more to lose than Bing, so they are being more cautious.
> Bard would not trick anyone into thinking it was sentient, yet something they have supposedly did
Bard is overtly a reduced-resources model compared to the best version of the same technology (which, if true, is probably a boneheadedly bad choice for a public demo when everyone is already wowed by the people who got theirs out first, but easily explains that disparity. Though so does “guy who wanted public attention made stuff up well-calibrated to that goal.”)
> Bard is overtly a reduced-resources model compared to the best version of the same technology
There's a scaling problem. ChatGPT/LLM systems cost far more to run per query than the Google search engine. Google can't afford to make those the first line query reply.
A big business model question is whether Google will insist you be logged in to get to the large language model.
At Google scale, these things are going to have to be a hierarchy. Not everything needs to go to a full LLM system. Most Google queries by volume can be answered from a a cache.
> Most Google queries by volume can be answered from a a cache.
And given how aggressively they limit the number of search results (in spite of listing some ridiculous number of results on page #1) that percentage may well be very large.
Google also has Sundar and Ruth who’d rather focus on how to get another ad on the SERP than kill the golden goose. They’re not going slow, they just don’t have the leadership for the moment.
Maybe it is just time to train and also good training data for prompts which openai has gathered for so long already? E.g. there is a bottleneck on how fast you can train and also gather good data.
Possibly, but wouldn't Google and Meta have access to way more compute resources and data than OpenAI? Google has been touting their TPUs for several years now.
OpenAI has access to Microsoft and Azure. That’s bigger than Meta, roughly on par with Google in terms of capability and higher in terms of market cap.
Google has the compute, from the comparisons I have seen Bard smokes GPT-3.5-Turbo on response times. So my guess is that internal politics prevents them from putting out something better. There would have to be immense pressure from the search division to not make them obsolete.
Without Nadella footing the compute bills, nobody would be taking about OpenAI. He’s brilliant, he let the start up take on huge risk to quietly claim the gains for m$.
If you try Bard or Claude or character.ai they are not far behind GPT4. They might even be on par in terms of raw LLM capabilities. ChatGPT has better marketing and in some cases better UX. A lot of this is self-fulfilling. We think it's far ahead, so it appears to be far ahead.
> If you try Bard or Claude or character.ai they are not far behind GPT4
Bard is way behind ChatGPT with GPT-3.5, much less GPT-4. Haven’t tried the others, though.
OTOH, that’s way behind qualitatively, not in terms of time-of-progress. So I don’t think it is at all an insurmountable lead, as much as it is a big utility gap.
Claude and Character AI are great at holding a conversation but they lack the ability to do anything specialized that really makes these LLM’s useful in my day to day life. I ask GPT-4 and ChatGPT questions I would ask in stackoverflow, I can’t do that with Claude or Character AI. Bard actually seems behind even conversationally to the rest
ChatGPT is cool and novel, but FAANG's requirements for ML/AI go far beyond what ChatGPT provides as a product. ChatGPT is good at answering questions based on an older data set. FAANG typically requires up to date real time inference for huge rapidly changing data sets.
Working on the practical side of ML/AI at FAANG, you will probably be working with some combination of feature stores, training platforms, inference engines, and so on - all attempting to optimize inference and models for specific use cases - largely ranking - which ads to show which customers based on feature store attributes, which shows to show which customers - all these ranking problems exist orthogonal to ChatGPT, which is using relatively stale datasets to answer knowledge based questions.
The scaling problems for AI/ML for productionizing these ranking models from training to inference is a huge scaling problem. ChatGPT hasn't really come close to solving it in a general way (and also solves a different class of problems).
Agreed. For my job maintaining real-time models with high business value to be disrupted by a chatbot, an LLM would have to be able to plug into our entire data ecosystem and yield insights in realtime. The backend engineering work required to facilitate this will be immense, and if the answer to that is "an LLM will create a new backend data architecture required to support the front-end prompt systems", then... well, suffice to say I can't see that happening overnight. It will require several major iterative and unpredictable pivots to re-envisage what exactly engineers are doing at the company.
For the time being, I expect LLMs to start creeping their tendrils into various workflows where the underlying engineering work is light but the rate of this will be limited by the slow adaptability of the humans that are not yet completely disposable. The "low hanging fruit" is obvious, but EVPs who are asking "why can't we just replace our whole web experience with a chatbot interface?" may end up causing weird overcorrections among their subordinates.
Isn't this as straightforward as semantic search over an embedded corpus ? Unless i'm missing something, i don't think the backend engineering would take much
I think generating useful embeddings off of a lot of realtime data flows (eg. user clickstream data) is in fact fairly difficult. Furthermore, if you had such embeddings it's unclear if an LLM would add value to whatever inference you're trying to do. If the LLM is not only be used for inference but to actually retrieve data ("find and summarize the clickstream history of his user") then I would not expect this to be doable in realtime.
ChatGPT is human level intelligence, it’s not just novel and cool, it’s the thing.
Remember, GPT-4 training was finished 6 months ago. Listen to people at OpenAI, their concern is: disruption to the world, UBI, getting people used to superintelligence as part of our world. I think they have quite a few things in the pipeline.
So yes ads optimisation/recommendations still need to be reliable for the time being, but for how long?
GPT-4 is not human level intelligence, nor is is above or below. It’s quite a different kind of intelligence not entirely comparable to humans. That’s probably why we’re moving the AGI goalpost; we visualize AGI as a robot human, but these machines may simply be founded on too different principles to ever receive that honor.
I think it’s mostly different because they crippled the public version for now:
no internet access, everything is done in one pass.
In our mind we get an idea, we inspect it, we try different variations, we simulate how it will be perceive(consciousness). In this way we iterate before putting the information out.
This is not difficult and is getting added on to it externally.
Chat GPT is just to get us used to the idea, it’s the toy version.
I would be interested to know which part you feel is implausible, to me it seems inevitable
You have a language model produce an outline with steps and then recursively set agents to consume and iterate on a task until another language model finds the results satisfies the specification.
This includes interactions with the real world (via instructions executed over an API) and using the success of those interactions for reinforcement learning on the model.
But I think they are mostly pointless as OpenAI is so far ahead of everyone external it’s not even funny. Most externals things with the API will be obsolete in a few months.
They had GPT4 6 months ago or more!
They have access to the full model without crippling.
They (for sure) have larger, more powerful models that are not cost effective/safe to release to the public.
Now they have a new data flyweel with people asking millions of questions daily.
Put your speculation hat on and listen attentively to the interviews of Sam Altman and Ilya Sutskever.
You will see were their minds go: UBI, safety, world disruption, etc.
I'm quite surprised at how little progress FAANG companies have made in recent years, as I believe much of what's happening now with ChatGPT was predictable. Here's a slide from a deck for a product I was developing in 2017: https://twitter.com/LechMazur/status/1644093407357202434/pho.... Its main function was to serve as a writing assistant.
Scaling up an LM from 2017 would not achieve what GPT-4 does. It's nowhere near that simple. Of course companies saw the potential of natural language interfaces, there has been billions spent on it over the years and a lot of progress was made prior to ChatGPT coming along.
You're making incorrect assumptions. This project wasn't about scaling any published approaches. It was original neural net research that produced excellent results with a new architecture without self-attention, using a new optimizer, new regularization and augmentation ideas, sparsity, but with some NLP feature engineering, etc. Scaling it up to GPT-2 size matched its performance for English (my project was English-only and it was bidirectional unlike GPT so not a perfect comparison), and very likely scaling it up to GPT-3 size would have matched it as well, since GPT-3 wasn't much of an improvement over GPT-2 besides scale. Unclear for GPT-4 since there is very little known about it. Of course, in the meantime, most of these ideas are no longer SOTA and there has been a ton of progress in GPU hardware and frameworks like PyTorch/TF.
What you're saying about companies recognizing the commercial potential is clearly wrong. It's six years later and Siri, Alexa, and Google Home are still nearly as dumb as they were back then. Microsoft is only now working on adding a writing assistant to Word, and that's thanks to OpenAI. Why do you think Google had to have "code red" if they saw the potential? Low-budget startups are also very slow - they should've had their products out when the GPT-3 API was published, not now.
One thing I didn't expect is how well this same approach would work for code. I haven't even tried to do it.
Do you have any publications to back up your claims about your work? They seem more than a bit grandiose. If you're ideas are as novel and useful as you say then you should publish them.
And I'm sorry, but you're completely wrong about companies recognizing commercial potential. I worked on Alexa for five years, it is a far harder problem than you think. It is nowhere near as simple as "we just weren't looking at the right NN architecture or optimizer!" You're acting like it was a novel idea to think LMs would be extremely useful if the performance was better (in 2017). I'm just trying to tell you that isn't the case.
No, I have no plans to openly publish any of it. Some of my researcher employees have published their own stuff. I've previously written about how it was a huge commercial mistake for Google and others to openly publish their research, and they should stop. Indeed, now OpenAI has not published a meaningful GPT-4 paper, and DeepMind has also become more cautious. This mistake has cost them billions, and for what? Recruiting? Now they lost many people to OpenAI and wasted time and effort on publishing. Publishing is fine for university researchers or those looking to get hired. I did record some research ideas in an encrypted format on Twitter for posterity: https://twitter.com/LechMazur/status/1534481734598811650.
If any of the FAANG companies recognized the commercial potential and still accomplished so little, they must be entirely incompetent. When this 2017 deck was created, I had 50k LOC (fewer would be needed now using the frameworks and libraries) plus Word and Chrome plugins. The inference was still too slow and not quite feature-complete, and it was just a writing assistant with several other features in early testing, but it seems more than enough for me to know quite well how difficult is the task.
The fact that you think creating a writing assistant plugged into Word is equivalent to building a general purpose, always-on voice assistant tells me all I need to know.
What? We were talking about making a language model. I mentioned the plugins in relation to the question of commercializing. I'm very clear about what my project was and was not doing. I get that you're bitter because Alexa became a joke with how little progress was made and the struggles of Amazon in getting the top talent are well known. How much did it cost Amazon to fail like this? Is the whole team gone? Is that the billions you mentioned?
"It's six years later and Siri, Alexa, and Google Home are still nearly as dumb as they were back then". You can't even keep a coherent discussion and you are delusional about the significance of your work. You shared a slide with nothing but generic pie-in-the-sky use-cases and you act like it gives you some credibility on the subject ("let's make an AI system that can do the work of your non-professional employees!"). And to top it off you act like you've been successful here! Again, you shared nothing but a slide with generic use-cases that a 12 year old could think up. I don't know what you think you proved. Enjoy your imaginary pedestal.
Calling an API doesn't mean no value is captured. There are vastly complex integrations of LLM as a small component in larger systems, with their own programming, memory, task models and so on.
If you think GPT is just about chat, you've misunderstood LLMs.
Folks need to start getting over themselves. It's pretty trivial to get GPT4 to explain how transformers work, where the bottlenecks are in both performance and learning, and start modifying pytorch.
It's really not that complicated. Gatekeeping is so over.
Not sure why LLM would make Facebook (ads), Apple (hardware), Amazon (hosting, retail), Netflix (tv) obsolete. It's definitely something Google needs to think about, but there's no reason to think they won't again be the leader soon.
I actually think Apple is in a unique position here again with the hardware/software integration.
Once again, their ability to do computation on device and optimize silicon to do it, is unparalleled.
A huge Achilles heel of current models like GPT-4 is that they can’t be run locally. And there are tons of use cases where we don’t necessarily want to share what we’re doing with OpenAI.
That’s why if Apple wasn’t so behind on the actual models (Siri is still a joke a decade later), they’d be in great shape hardware-wise.
Google has some impressive on device AI software such as Google Translate (translation), Google Photos (object detection, removal, inpainting), and Recorder (multi-speaker speech to text). Most of this is possible without their Tensor chip, but is more efficient with it.
Imagine Walmart launching a ChatGPT interfaced bot for shopping that customers take a liking to. Walmart starts acquiring both new customers as well as high quality data they can use for RLHF for shopping. Eventually Walmart's data moat becomes so big, that Amazon retail cannot catch up and customers start leaving Amazon.
For AWS, if MS starts giving discounts for OAI model usage to regular Azure customers, that's gonna be a strong incentive to switch
For Apple, A Windows integrated with GPT tech may become a tough beast to beat.
> Imagine Walmart launching a ChatGPT interfaced bot for shopping that customers take a liking to.
I can't imagine that, because it doesn't seem to fit the use case. Especially not to the point of bankruptcy of Amazon, maybe as a small novelty? Can you list some killer features that the chat would bring that would make the existing shopping experience irrelevant? Maybe not everything is a nail to the hammer?
sure, this is basically a fancy version of the current FAQ's, customer's QnA and current recommendations (similar products, recommended products). Would it be just so much better in experience that it kills Amazon? i kind of doubt it. Also, we're currently talking model that was trained on pre-2021 data and here we have an inventory in the millions that changes daily, so the tech has to catch up, too.
Asking for your preferences to figure out your product choices or searching for a product for you based on your given requirements is a lot more than just FAQs and Q&As. You can even imagine a fancier version where you describe say what kind of setup you want and it gives you combinations that are nice.
Also this could aggregate information not just on the product page but across multiple pages which is time consuming to do by oneself.
As for 2021 cutoff -- ChatGPT can now browse the internet and if Walmart built a bot that interfaces with ChatGPT, I'm sure they would be feeding it the latest info
I work at a FAANG and our unreleased models are fantastic. Now, there might be panic about how to productize it all, but tech wise i'm pretty surprised how good they are.
Sounds like the same thing that happened with datacenters? No one has ops or hardware sysadmins, no one sets up large networks except a few in those centralized cloud companies and couple other niche uses. website ops job changed
