I like this... in theory. But I don't think I'm allowed to give you a system diagram and detailed explanation of systems I've worked on, because obviously they're proprietary. That seems like a problem.
How do your candidates normally work around this? Does everyone just talk about hobby or open-source systems they've worked on?
Show me a system you’ve worked on that wasn’t put together from knowledge from previous education and jobs (yours and others), documentation from vendors, stack overflow, code generators , etc.
I would hate to be a stakeholder in a company whose value depends on employees not talking about and reusing the incremental skills and knowledge gained while working there.
Im pretty sure I learned a full set of new skills at my current job.
I don't think knowledge on any particular api is particularly useful compare to the more generic "can you read docs and use apis" skill, which is directly applicable from something like high level hard ware work making an ftdi USB chip go brrr to making dynamoDB work nicely, but I definitely didn't know anything about dynamodb before doing a server/web job
Async distributed systems and FPGA vhdl have some overlap in how you build things, but doing an interview based on my old FPGA skills likely won't show much of my servers and databases systems design skills
Yup. I’m outside DC. Somewhere near half my software engineer (and adjacent fields) friends are under contract to either the military or NSA/CIA. Most of them with TS/poly clearances. Ask one of the what they’re working on and the most you’ll get is something like “dig data on platform X”.
If platform X is something public, they could talk about the platform. But, talking about any project specifics, implementation details, etc would be a felony.
98% of what you'd be inclined to diagram out and explain for a system design interview isn't remotely proprietary. People are generally looking for a high-level overview of a system you've worked on to see a) that you've really worked on it and b) you can explain it such that it makes sense.
Sure, folks will ask you to drill down here and there for more detailed info, but if someone pokes on something where the details really _ARE_ proprietary, it's fine to just say that and offer to drill down somewhere else.
> 98% of what you'd be inclined to diagram out and explain for a system design interview isn't remotely proprietary
Maybe we're just on different wavelengths, but literally every single thing I work on that would be a good candidate for this sort of interview is extremely proprietary...and I'm just working on regular ol' backend services at [generic big tech].
That would fall into the 2%. At least in my experience, the 98% number is in the ballpark. If you are working on backend services in big tech, what you are building is quite different than most software systems.
This comment is a prime example of how the non-technical founder understands the world.
Proprietary systems are company IP. Talking about the design of the system is protected by the NDA, which the exercise described above is asking the candidate to do.
You hire for the skills to develop your own proprietary system. The employer doesn't own the skill. The employer owns anything those skills produced for the employer.
You wouldn't want your employees giving the recipe to your secret sauce away to your competitor, right? That's what everyone is talking about.
[Client]->[Cloudfront]->[Load balancer]->[ECS service in a VPC]->[RDS/other internal systems]
* Replace the above with any other cloud provider, or just commercial/open source tools that do the same job running in a VPS or whatever *
I can't see how any of this could be claimed to be protected by an NDA or similar, as this is just a highly standard architecture.
If you work on a proprietary system that has a specific architecture just don't mention that of course. I think the above allows us to discuss things pretty well, talk about introducing caching, why each layer is there, TLS termination, authentication/authorization concerns and implementation, secrets management...
> you hire for the skills to develop your own proprietary system. The employer doesn't own the skill.
Oh OK! Then I guess a good engineer should be able to make it through a standard systems design interview process, and shouldn't complain about it!
I am glad you agree that it is totally fine to ask engineers these types of questions, and there is no problems with NDAs that will get in the way of them demonstrating basic skills!
That was my point entirely.
Yes, I totally agree that an engineer should be able to go through a tech interview.
And the engineers who are complaining about the fact that they have an NDA are wrong, and they should stop complaining.
If the NDA is so significant that it prevents you from doing an interview, then you are worthless as an employee anyway. But really that should never happen.
If your employees go to a competitor and make a secret sauce that tastes the same, how do you judge what is the source of knowledge?
They can have used their skills to come up with a recipe that tastes the same because the requirements led to that outcome, or they simply copied your recipe.
If somebody from OpenAI goes to a competitor and creates something like ChatGPT, where is the line crossed that shows that company IP is transferred? The employee knows how to create the system because he knows how ChatGPT works. If he recreates a similar system but doesn't say that it is a copy of ChatGPT, is that applied skill or is that a violation of the NDA?
If it is a violation, how could he forget the structure of ChatGPT to genuinely come up with a new structure?
Legislatures write rules, people enter contracts, sue for perceived breaches and courts rule on these disputes. People then adjust behaviors to get desired results without incurring legal liabilities. Results vary across jurisdiction. Such questions can't be answered in principle. California famously won't enforce non-compete clauses, and that apparently helps innovation. China caught up industrially with the West within one generation by flouting every IP rule we have here. On the other side of the debate, patent trolls...
The flaw in this argument is assuming that you could do a good job of diagramming a system without describing the use cases of the system. My web dev skills are not proprietary, but the processes and systems I have worked on are.
The engineering knowledge isn’t under NDA but there’s no way I could fill an hour long interview about the systems I work on without getting into enough specifics about our business processes to violate NDA.
If I were asked a question like this I’d wonder if I were being tested to see how loose-lipped I am with sensitive information.
I get it...I tend to think in point solutions as well, but getting stuck there is the problem and moving past that to abstract point solutions into a systems design that talks about the architecture of the solution and the 'why' behind the design choices is what gets you out of NDA-land. None of that stuff the next level up is proprietary. Specifically how it was applied in that problem domain is.
It’s the difference between being asked “how would you design a Twitter clone?” and “How did you design [current employer’ssystem]?”
The first is generic (unless you happen to be a current Twitter employee). No NDA impact. The second is asking you to reveal details of a proprietary system covered by NDA (or if you’re a Fed contractor, covered by various classification laws).
I'm not a lawyer, but wouldn't anything you or anyone else developed on company time at another company be owned by that company and thus proprietary?
In practice I can't imagine the spirit of any law would be violated by doing an architecture diagram, but it seems likely (to me, at least) the word of the law would be violated.
Sure. But the level of description that you're giving in an interview can be pretty generic/non-proprietary. If that wasn't the case, then you really can't reuse any knowledge from job to job and that's just not so.
Sure, every company is going to have their 'special sauce' components, that ARE proprietary, but nobody's going to expect you to unpack those.
I doubt it's common, but my friend had an experience where they kept trying to direct him towards specific technical topics of particular interest to the business.
He worked for their competitor, but not in the capacity the interviewer was trying to delve into, so in addition to scummy, it was pointless and annoying.
Most people barely consider this. In the old days, before leet code interviews, I had people give me code samples. One guy gave me a code sample from some telecom carrier's billing system. It wasn't good. It was also bad that he gave us a proprietary code sample, so he got no points for that either.
> I'm not a lawyer, but wouldn't anything you or anyone else developed on company time at another company be owned by that company and thus proprietary?
In theory yes, in practice ... if that's true, why would anyone ever hire you for your experience? You wouldn't be allowed to use it.
(coming from the standpoint of a historically non-technical individual contributor who is leaning towards technical now, and asking because I want to learn):
But isn't that exactly why you as an interviewer would pose a problem, set up the context that is potentially similar to the work your project would require, and see how the interviewee navigates that?
Yes, it is. I would want to know how a person solves my problems.
And I sure would hope they bring all their experience to bear! Just because they learned about CDNs at their previous job and “everything is proprietary”, I don’t want them to suddenly forget how CDNs work.
Very little is actually unique between software businesses. We’re mostly just doing data bureaucracy.
System diagram of a CRUD SAAS isn't going to be a critical business secret in the same way it might be for a company that does stream processing of marketing/advertising data off of twitter or video/radar analytics. Or a real-time fintech trading company. Many companies are a Python/Django monolith on Postgres with redis & || memcached. That system diagram isn't going to let you disrupt the online medical services industry but it's what a lot of them use.
We call this out in our pre-interview handout. If you've got an NDA, choose something that isn't included in your NDA or figure out a way to describe it at a high level. We caution candidates that if we dig in and we run into something blocked by the NDA it creates challenges with our evaluation.
It doesn't necessarily have to be something that you're working on. It just has to be something that you understand well enough to describe the design of and how things are inter-connected.
It's a reasonable hurdle; an employee who previously signed an NDA (or any one, for that matter) needs to be able to design new systems without regurgitating verbatim parts of an old one under NDA.
So it's really not a big ask to navigate their own NDA and come up with something interesting to talk about at an interview.
I’m going to be way more strict about confidentiality with an interviewer I don’t know at all than with a trusted colleague. For all I know you get beers every week with the CTO of my current employer’s largest competitor.
Besides which there’s a way more interesting variation on the question, which is “how would you design our product?” Got that one in my last big tech interview and quite enjoyed it.
It’s not a handicap. We’re not asking people to diagram their current systems. We don’t advocate for people to do that. You could just walk us through an HTTPS request and it would work, if you understood HTTPS requests well enough.
There’s no penalty. There are plenty of technologies that you’re working with that aren’t under NDA. We’re not asking you to diagram what you’re working on, we asking you to diagram something technical that you understand well.
There have to be parts of what you work on that are not covered by NDA. Solving problems like this would be indicative of someone we would want on our team.
When I worked at Apple they made it abundantly clear that EVERYTHING was under NDA. Even things like “what version of the standard library do you use”. Or “what are computers for”. And that any minuscule transgression of the NDA would result in me
Being immediately deported, divorced, and likely put in front of a firing squad.
That can’t tell you that you can’t explain how Kubernetes works or NGINX or the JVM. Any one of those things would be perfectly acceptable as a starting point.
The problem you're asking them to solve is a legal one, but you are not interviewing lawyers. I have never worked for a single employer that didn't require an NDA, and the NDAs are always written in a way that can be easily interpreted as "do not tell anyone anything about what you do here".
The question is not tell me what you do in your current job. The question is describe a technology or system that you understand well. Do you understand how the underlying systems work? You don’t have to tell me that your current company uses kubernetes to describe how kubernetes works. Your current job can’t keep you from describing how kubernetes works. Kubernetes is just an example here you could pick any technology that you’re comfortable with.
You're asking them to explain a technology they use while expecting them to tiptoe around what they're actually allowed to say about how they applied the technology. And you say this replaced your system design interviews. It sounds to me like someone explaining what Kubernetes is won't do nearly as well as someone who can explain in-depth how they applied Kubernetes to their job.
When I interviewed with apple they said pretty much everything they do is proprietary and considered secret. I imagine for some of these people working there for years it would be hard to avoid breaking nda
There’s no place where we are asking you to diagram the system that you’re working on. We ask you to diagram a system you understand well. Are you using messages queues, container orchestration, network components, cloud infrastructure, Java/rust/go/Python/etc.? Tell me about one of those components and how they connect together in theory vs the specifics of your system. Everyone who works in technology works with, mostly, complicated interconnected systems. The point of this exercise isn’t to design a system or tell me about a system that you’ve designed as much as it is about helping the team to understand the breadth and depth of your technology knowledge. Good engineers typically find creative ways help us understand their knowledge without disclosing NDA material.
I feel like I'm either losing my mind or people are being deliberately obtuse now.
He's not saying they ask you to give a full overview of the way Apple architects their systems, he's saying they ask you to explain how a basic Kubernetes deployment might look, or how you might go about setting up some kind of backup system, or literally any of a million different technical things that you might have knowledge of and interest in.
How could things like this possibly be under NDA? If you weren't able to use any of this knowledge, you would be completely crippled in your work.
Ok yes because that’s colleges level material. The subject here is Staff Engineer (L6) interviewing. So that will require nearly all the demonstration of value to be of high level professional context.
The only really important details to keep secret are business-specific algorithms, so talking about the overall architecture of a system is usually okay, without those details. If you're building very special architectures where that actually is the business detail, then I suppose you need to figure out how to talk about something else similar or something. I just talk about the systems I worked on in generalities, so I would talk about how we get some telemetry from some devices, process it, store it, and etc., but only the mechanical details and not the meaty algorithmic parts (we use a JSON API to submit the telemetry, then we store it in a table, then we take that data in our other component and process it through our "algorithm", etc.).
We say specifically “diagram a technical system you understand well.” It’s going to be challenging to work in technology if the only technical system you understand well is the one you’re currently working on and only the parts that are under NDA.
I have the opposite question from other people in the thread - this seems pretty open-ended.
Just as an example, I do bioinformatics work, and "technical system" for someone I'm interviewing (as I understand your question) might encompass any of:
- a workflow manager: how it works under the hood, common patterns for modularity/reuse or configuration
- a specific pipeline: the stages of cleaning and quantifying raw data, the rationale for various stages and the tools chosen to execute them (including how to benchmark various methods against each other)
- a specific third-party tool: theoretical details of the algorithm, practical considerations for when to apply one over another
- familiarity with common general-purpose packages or APIs (e.g. pandas, numpy, scikit-learn)
- QC: common metrics for QC'ing various experimental data, how to decide if data is "good enough", how to troubleshoot biological vs technical (lab) vs technical (computational) sources of error
- biology: technical details of an experiment, the technologies generating the data, or the underlying biology
- data analysis: how to choose and make relevant figures, any of many data-science/ML topics (e.g. clustering), connecting data to relevant domain questions
- devops: data storage/management on HPC or cloud, HPC job schedulers, any of many cloud topics (e.g. IAC, setting up a database or cloud execution of a pipeline)
I'm more curious about how you guide a candidate towards selecting a system that gives you the most relevant perspective on their thinking and skillset - do you provide any suggestions, or are there typically pretty evident choices based on the role?
For our more junior roles, we leave it more open ended because candidates may not have a similar knowledge bases to the panel. For more senior roles we advise that if they pick technologies that are pertinent to the role and likely to be understood by senior/lead/staff engineers, the process will go better for candidates and panel members.
If I were in your position, one of understanding many complex systems, I would advise you to pick where you felt both strong and the panel was likely to have some entry point into.
So draw the block diagram of something you worked on in school. Or, I don't know, a radio or something, if you're a hardware person.
Someone who answers, "I can't draw anything for you because everything I know about is somebody's private IP," isn't going to get the job. At least, not if I'm doing the interviewing.
You must know something about something else. Right?
Or just interview some place that doesn't give this kind of interview, like most companies. It's pretty trivial to get hired even if you can't talk about technical specifics along the lines of an architecture diagram.
This is why this question is so informative. There are a ton of people in this thread arguing that they can’t describe any technologies they understand because of NDAs.
"Great, that'll do nicely. Vacuum cleaners are complicated and interesting machines. What can you tell me about how a vacuum cleaner is built and how it works? Also, vacuum cleaners are pretty old tech. Is there room for further innovation in that area? If so, what angles would you explore if you worked at a vacuum cleaner company?"
What is the vector you’re concerned about here? You sketch out a diagram so well that the company you’re interviewing with takes a photo and secretly integrates it into their stack? Then your past employer catches wind of this, realizes you were the only person in the world to understand this system, so they go after you for damages? Sorry man, I don’t see it.
I understand that you see your past work as “proprietary”, but in the real world the vast, vast majority of us do not work on systems shrouded in such secrecy. There’s nothing interesting or proprietary about CRUD infrastructure and I genuinely can’t think of a situation where you’d be exposing yourself to any real risk by explaining it to a new employer.
In my first semester as a freshman in an American university, I took a quiz in the Calculus class that had a question based on American football. Having never played the sports or even knowing the rules all that well, I aced the exam.
Yes, it is not that hard to keep the proprietary business logic separate from the system design.
See the reason ops thing works is this can even be applied to something you haven't worked on but purely as a passion project on the side. If you are Saying "oh what if I don't have side projects" you do realize people spend/waste 3+ months going these interviews + tons of mock interviews with companies they have no intention of joining. What does it say about if you are telling me you'd rather waste valuable personal time that way instead of building something fun?
I'm fact I'd even say that even if you haven't built a side project, I'd still use op's technique and ask the candidate to talk about and white board a system they would like to build and then dive deeper from there. Heck I'd even tell this to them before an interview!
I put in 60 hour weeks at my current job, except for a month before I change jobs, where I put in an additional 2 hours/night to prep for interviews, because that’s what will benefit me for every other company.
I care about my career and my family, so I don’t have the luxury of spending it on passion projects outside of work. I’d rather spend the precious little time I have left on my real hobbies and family/friends.
It doesn’t communicate that I’m a bad engineer. On the contrary it communicates that I give my all at work.
But I never said you had to spend all your life on passion projects. How is leet code fair game but asking how they would build something they would to even if they never did or do somehow an indication that I think you are a bad engineer? Because I see today's state of system design question interviewing pretty much like leetcode no (asking this as a faang interviewer)?
Having said that what you said about spending 60 hours at work tells others you will be a much better employee. Which is probably the signal they are looking for?
I am not going make judgement on people's loyalty. If someone chooses to do 60 hours works more power to them. If people don't want to do side projects that is perfectly fine too. Seriously my motivation wasn't to suggest id penalize those without a side project. It was more from my observation that (from several l5-l7 candidates):
1. Despite the "let us work together" intent it is still an exam where you are penalized for taking too long or making a "wrong" assumption or missing a few bits,
2. Given above or not when confronted with a random question it can be easy to get very nervous unless you have built a similar system from 1 to 1b user scale or you have done tons of practise interviews with companies.
So apart from interviews being hard and what they are purely because of supply and demand I wanted to see if you can learn just as much from a more open book approach (with either a side project or candidate knowing the question ahead of time say the night before). It could even be more inclusive that way!
How do your candidates normally work around this? Does everyone just talk about hobby or open-source systems they've worked on?