Over Christmas I met up with a friend who teaches part-time at a local college. He said he’s failed more people this year than the cumulative total of his entire past teaching career due to LLM abuse.
He doesn’t use LLM detection tools, but he says it’s easy to identify papers with warning signs of LLM use. For some reason, using ChatGPT for his specific niche topic overuses a few obscure, rarely-used words that most people wouldn’t even recognize. The ChatGPT abusers some times have these words appearing multiple times through their essays.
He’s also caught people who cited a lot of different works and books in their reports that were outside of the assigned reading, or in some cases books that don’t exist at all. Catching them is as simple as asking them about their sources or where they acquired a copy of the text.
I see a lot of parallels in hiring and talking to junior software engineers right now. We had a take-home problem that was well liked that we used for many years, but now it’s obvious that a majority of young applicants are just using LLMs to get an answer. When we want to talk about their solution in the interview, they “can’t remember” how it works or why they picked their method.
It’s really sad to me as a long time remote worker because I see far more blatant abuse from remote candidates. Like you, bringing people on site for interviews seems to instantly scare away the LLM cheaters, but it’s expensive and time consuming for everyone involved.
I know the on-site is time-consuming and expensive, but so is firing people (at least in United States it is.) I've had a few on-site interviews where having them on-site made us realize we could never work with them. Given how much time they will spend with you, it's totally worth it to spend your resources on hiring.
Firing people isn’t as expensive as people make it out to be. People vastly overestimate the chance of a lawsuit. And they overestimate the chance of a lawsuit that makes it far enough that it costs significant money even more.
Hiring fast and firing fast (for lying or misrepresentation) is almost always a better business decision than being ultra defensive in the hiring process.
The fastest that I can possibly fire somebody is still months from the date I choose to hire them.
I decide they are the best candidate. A recruiter talks with them to negotiate compensation and they accept the offer. This takes a week at best, but can take weeks if they are choosing between multiple offers. Then they choose a start date. They've got a couple weeks at the old job, plus probably some time in between roles before they start. So 2-6 weeks waiting here. Then they join and go through the company-wide onboarding and training processes and set up their equipment. Another week.
The first time I actually get to have them do any work is 4-10 weeks from the date I chose to offer them a job. It now takes me some time to realize they are hopeless and misrepresented themself on their resume. Three weeks would be an extraordinary outcome here, but it more likely that this takes 8+ weeks. Even if the actual process of firing them is instant once I've decided that it was a bad hire, I'm still out 3-5 months from the date I chose to hire them. Any other strong candidates I had in the pipeline now have other jobs and I am starting from scratch.
I can't believe any company would look at this story (which I've heard variations on from multiple peers) and go: "we should save money by not flying candidates out for an on-site and use terrible AI tools to sort our candidates."
But we've just established in this thread that even senior people are having difficulty findings jobs. This has nothing to do with desperation. Temp contract works both ways, if an employee doesn't like the company or finds another job within the 2-3 months, they are free to leave. This is more than fair.
Ignoring that this seems like a bad way to start a hopefully long-term relationship, this would largely limit your pool to people who don’t already have a job. If a senior candidate already has a job, why would they give it up for a sketchy 2-3 month contract and the vague promise of full employment?
Relationships between an individual and a corporation are fundamentally asymmetrical. They can only be made equal by heavily favouring the single human side.
It's not the lawsuit, it's about the time wasted as a manager and salary to the person as you work out if it's actually time to fire. Performance Improvement Plans, a bunch of back-and-forths. I'm not going to be the kind of person that fires quickly, so there's a bunch of sunk cost we have to take. Plus, fast firing creates a cooling effect among everyone there.
And for what? To save money on hiring? Not worth it.
Okay, so the original argument is about whether or not it's worth it to fly people out for an on-site. Hotel and airfare: $2000 absolute max. Salary at $100/hr for one month for me to figure out it's not going to work out, then pull the trigger to fire: $24,000.
I mean, being a manager is hard, but putting in the time and money to hire and then putting in the time to make sure your team doesn't have a morale drag, it's worth it.
The catch is that even in-person interviews are no panacea. I agree that it's worth the time to filter -- I wasn't really responding to that bit -- but from what I've seen, you have to be a very good interviewer not to get a bad hire every so often.
I often wonder how many hiring managers are actually good interviewers, in-person or not, but I digress...
Seeing the truly bad hires dragged along to the detriment of the rest of the team is a sore spot for me, though. It happens way too often in my experience.
Also imagine you are a company with a reputation for hiring people - inducing them to leave their current job - and then often dismissing them quickly afterwards.
That would give many great prospective employees pause before applying to work there, because you are asking them to give up a good thing and take a chance on your company, without commitment.
There's a difference between layoffs and firing. To fire an individual, the company must have documentation to ensure it's not a wrongful termination. Ironically, it's easier to lay-off 100 people because all you need to do is demonstrate their division's project is cancelled.
And that documentation takes time as a manager, which costs money.
But I admit not knowing completely because I haven't had to fire anyone yet. I have talked to legal about the process regarding someone not on my team.
What jjav is referring to is "at will" employment - in almost all US states, employees can be fired for almost any reason, with no recourse. So the fact you're saying that firing people is expensive and time-consuming in the US flies in the face of the actual legal environment there compared with most other relevant countries.
>the company must have documentation to ensure it's not a wrongful termination.
Companies develop documentation processes as they get bigger for myriad reasons, but there is very little to worry about in the US in the way of terminating someone.
The only adverse effect most times is increase in unemployment insurance premiums, if you do not have enough documentation to show you terminated for cause.
Otherwise, 99.9% of the time, the terminated person can claim whatever kind of wrongful termination they want, they probably won’t get anywhere via the courts.
> To fire an individual, the company must have documentation to ensure it's not a wrongful termination.
Not in the US. All you must do is tell them they're gone, walk them out the door and that's that. You must pay them any worked days not yet paid but that's all.
Company HR departments sometimes establish more elaborate procedures for firing, but none of that is required by law, it's just internal company process.
I'm assuming you're talking about "at-will" states, coming from Canada I've heard there are also sane states. And even some at-will states have powerful unions no doubt.
In most places, even with strong labour laws, you can layoff people for any reason in the first 30/60/90 days. And, the US has extremely weak labour laws. Usually, a month of severance for each year of service is customary, but probably not strictly required.
> When we want to talk about their solution in the interview, they “can’t remember” how it works or why they picked their method.
Sweet! That sounds like perfect signal for "used ChatGPT" to answer this question. So, you can send take home test, candidate sends reply (many from ChatGPT), then you do quick follow-up phone/video call to discuss the code. When you get the "signal" (should be quick!), then you immediately close the interview and move to the next candidate.
>It’s really sad to me as a long time remote worker because I see far more blatant abuse from remote candidates. Like you, bringing people on site for interviews seems to instantly scare away the LLM cheaters, but it’s expensive and time consuming for everyone involved.
Technology enables scale and reach, which solves some problems but also creates its own set of issues. I think you're right on with the solution: do things that are anti-scale. If you make things a bit more inconvenient, a bit more costly, and a bit more local, you create an environment where there's space for trust and humanity---values that don't scale.
Overly expressive words like "robust" and words that would appear in the thesaurus for "extremely" seem like tell tale signs in my experience. I've noticed personally that sometimes I have to tell ChatGPT to "sound more human and concise." I'd love to hear if anyone else has had the same experience as me.
Regardless, if ChatGPT is tailored enough, or a custom model is created, I can't think of any way to detect if an LLM has authored something generic. The lazy college student will probably get caught, but the cunning one not so much.
He doesn’t use LLM detection tools, but he says it’s easy to identify papers with warning signs of LLM use. For some reason, using ChatGPT for his specific niche topic overuses a few obscure, rarely-used words that most people wouldn’t even recognize. The ChatGPT abusers some times have these words appearing multiple times through their essays.
He’s also caught people who cited a lot of different works and books in their reports that were outside of the assigned reading, or in some cases books that don’t exist at all. Catching them is as simple as asking them about their sources or where they acquired a copy of the text.
I see a lot of parallels in hiring and talking to junior software engineers right now. We had a take-home problem that was well liked that we used for many years, but now it’s obvious that a majority of young applicants are just using LLMs to get an answer. When we want to talk about their solution in the interview, they “can’t remember” how it works or why they picked their method.
It’s really sad to me as a long time remote worker because I see far more blatant abuse from remote candidates. Like you, bringing people on site for interviews seems to instantly scare away the LLM cheaters, but it’s expensive and time consuming for everyone involved.