One thing I don't understand with Windows Server is that it seems no matter how fast the nvme drives I use, or I pair/pool, I can't get a normal file copy to go faster than around 1.5GB/s (that's local, no network). The underlying disks show multi GB/s performance under crystal disk mark. But I suspect something in the OS must get in the way.
Your system ~3+ years old or so? Your story screams DMI 3.0 or similar PCH/PCIe switch saturation issue. DMI 3.0 caps at ~3.5GB/s but about 1.5-1.7 when bidirectional, such as drive to drive. If 100% reads hit about 3.5 then you’ve all but confirmed it. ~1.5GB/s bidirectional is a super common issue for a super common hardware combination.
It’ll happen if your U.2 ports route through DMI 3.0 PCH/Chipset/PCIe switch rather than directly to the CPU PCIe lanes. Easiest to just check motherboard manual, but you can use hwinfo to inspect the PCI tree and see if your U.2 ports are under a “chipset” labeled node. You might have different ports on the mobo that are direct, or possibly bios changes to explore. Sometimes lots of options, sometimes none. Worst case a direct PCIe adapter will resolve it.
Actually more than 3y old. My servers are EPYC Gen 2 and Gen 3 mostly.
But I don't think it is a hardware thing, as I see in CrystalDiskMark (which I believe bypasses the Windows write cache) performances that are close to the SSD specs. But it is when I do windows copy, the performance goes down the drain.
And it's not just parallelism. If I copy one file from a fast nvme to fast nvme, it caps at about 1.5GB/s. If I copy two files in parallel, they seem to split that speed, even if file1 is copied from diskA to diskB while file2 is copied from diskC to diskD.
In addition to my other comments about parallel IO and unbuffered IO, be aware that WS2022 has (had?) a rather slow NVMe driver. It has been improved in WS2025.
If it's over SMB/Windows file sharing then you might be looking at some kind of latency-induced limit. AFAIK SMB doesn't stream uploads, they occur as a sequence of individual write operations, which I'm going to guess also produce an acknowledgement from the other end. It's possible something like this (say, client waiting for an ACK before issuing a new pending IO) is responsible
What does iperf say about your client/server combination? If it's capping out at the same level then networking, else something somewhere else in the stack.
I noticed recently that OS X file IO performance is absolute garbage because of all the extra protection functionality they've been piling into newer versions. No idea how any of it works, all I know is some background process burns CPU just from simple operations like recursively listing directories
The problem I describe is local (U.2 to U.2 SSD on the same machine, drives that could easily performs at 4GB/s read/write, and even when I pool them in RAID0 in arrays that can do 10GB/s).
Windows has weird behaviors for copying. Like if I pool some SAS or NVMe SSD in storage space parity (~RAID5) the performance in CrystalDiskMark is abyssal (~250MB/s) but a windows copy will be stable at about 1GB/s over terabytes of data.
So it seems that whatever they do hurts in certain cases and severely limits the upside as well.
I think the point is that if you are not Netflix, you can use AV1 as most of your clients devices support hardware acceleration thanks to the big guys using AV1 themselves.
One difference that strikes me with the .com bubble is that I don't remember the .com companies having sustained multi-billions losses / cash burn. They were not profitable but this is quite different. If (or when) the music stops, won't OpenAI go bust immediately? That's quite a counterparty risk those companies are taking.
Cisco was very much not "stuff like pets.com". Most of the money lost in the dotcom crash wasn't in pets.com, it was the infrastructure companies like Cisco and Sun.
I believe we'd have to tease out what proportion of that cash burn is essential to keep serving compute to customers (which I assume to have profitable unit economics), versus what percentage is optional datacenter buildouts that could be paused in this situation.
Massive cash burn was an absolutely key feature of the dotcom boom/bust. Admittedly, it never really went away - there's always been a free->enshittification profit taking cycle since then. It's just the scale that's terrifyingly different this time.
Numbers of km driven in the US has increased by circa 10% [1] over that period while decreased in the EU by circa 10% [2]. Add to that in european cities the multiplication of bike lanes, and the permanent manufactured congestion of certain cities. There are many reasons that can explain the movement, and car design is probably a small factor among many small factors.
That probably is doing a lot of work here. A truck with a driver sitting so high above the street they can't physically see a child or bicycle in front of them is just an inherent risk to pedestrians and cyclists, no matter how you twist it. And don't even get me started on Cybertrucks, which are pretty much designed to cause accidents with casualties.
Even if the causal link is more complex than the numbers make it seem, acting like putting heavier and bigger vehicles with less restrictions on streets won't cause accidents is just plain dishonest.
Only if you need high availability, which is probably not the case for home use.
If one of your drives fails under RAID5, before you even order a new disk, you should do an incremental backup, so that your backup is up to date. Then it doesn't really matter that the rebuild times take long. And if you have more data coming in, just do more incremental backups during the rebuild time.
Synology adds some additional logic to automatically recover data from other RAID drives when the BTRFS layer encounters a checksum failure. I don’t think there’s any way to homebrew that part.
Technically any motherboard can become a NAS, but there are desirable features.
- low idle power consumption since your NAS will be sitting doing nothing most of the time - pretty much any desktop MB will do
- fast networking, 1gbe means ~100MB/s transfers, nicer to have 10gbe. Limited benefits beyond 10gbe in practice.
- enough PCIe lanes to connect enough drives. HDD of course but nice to have a separate fast SSD array plus SSD caching. You might also want a SAS HBA if you are looking enterprise drives or SSDs (and even for SATA SSD you will get a better performance via a HBA than through the motherboard). Some people also want a graphic card for video transcoding
- ECC memory
- IPMI - once you start using it it becomes hard to give up. Allows you to manage the server remotely even when switched off, and access the BIOS via a web interface. Allows you to keep the server headless (i.e. not have to go plug a screen to understand why the server is taking so long to reboot).
I'd say a good candidate for a NAS motherboard would be something like a supermicro X11SSH-LN4F, you can find used ones pretty cheap on ebay.
HDD have to be bought new, as well as anything mechanical (eg fans). But for motherboards, CPU, RAM and SSD, there is great value in buying used enterprise hardware on ebay. It is generally durable hardware that spent a quiet life in a temperature controlled datacentre, server motherboards from 5 years ago are absolute aircraft carriers in term of PCIe lanes and functionalities. Used enterprise SSDs are probably more durable than a new retail SSD, plus power loss protection and better performance.
The only downside is slightly higher power consumption. But just bought a 32 core 3rd gen Xeon CPU + motherboard, 128GB RAM, it idles at 75w without disks which isn't terrible. And you can build a more powerful NAS for a third of the price of a high end Synology. Unlikely that the additional 20-30w idle power consumption will cost you more than that.
I wouldn't say that being new is an absolute requirement. I recently upgraded my ZFS pool from SATA to SAS HDDs. Since SAS HDDs have much better firmware for early error detection and monitoring, I decided to buy 50% refurbished. Even if I lost half of them, I would still be safe. I also have offsite backups. This setup worked really well for me, and I feel completely confident that my data is safe while not wasting unnecessary resources. Whether to use new or used equipment therefore depends on the setup.
Agree, but that's taking a risk with your data (whereas if a MB fails, you likely just need to replace it but your data is fine), and HDD kind of have a finite number of hours in them. Where buying them used I think makes sense is for a backup server, that you leave off except the few hours in a week where you do an incremental backup. Then it doesn't really matter that the drives have already been running for 3 or 4 years.
So you buy used enterprise disks because their error detection is 'better'?
Do you have any source for this claim? Why would be the firmware so different? Software is cheap i don't think they would be that different.
I mean a used enterprise disk gets sold after it was running on heavy load for a long time. Any consumer hdd will have a lot less runtime than enterprise disks.
The protocol is better (SAS). Once I put them in, I immediately noticed transmission issues in a counter going up for several of my HDDs where the SATA showed no errors at all. It was a faulty backplane that introduced these transmission issues that went unnoticed by my SATA drives for several years.
Yes, I am using ZFS (Raidz2), which is exactly why I know the data remained intact despite the transmission errors. Here is the breakdown of what happened during my recent migration 5 weeks ago and why the SAS protocol made the difference:
> Did the HDDs write broken data?
No. ZFS handles end-to-end data integrity via checksums. If the data had been corrupted during the transfer, ZFS would have flagged it as CKSUM errors in zpool status. Because ZFS validates the data after it crosses the wire, I could be confident that the data written to disk was valid, or the write would have been rejected/retried. However, ZFS did not notice any errors because the SATA/SAS firmware error correction were (still) able to handle this through repeated submission of signals.
> SAS vs. SATA
The issue was that my old backplane (BPN-SAS-743TQ) was electrically failing (signal degradation), but not dead yet. For years, my SATA drives likely engaged in silent retries or masked these marginal signal issues. I know I had issues for years, where I suspected bad cables (this had cost me $500 to replace several cables to try solving this). Standard SATA SMART attributes often only flag UDMA_CRC_Error_Count if things get very bad, but they are generally less verbose about link stability to the HBA. As soon as I plugged in the SAS drives, the mpt3sas driver and the drives themselves flooded dmesg and SMART logs with Invalid DWORD count errors. SAS has robust Physical Layer (PHY) error reporting. It wasn't waiting for data corruption. It was alerting me that the signal integrity on the way between the HBA and the drive was compromised.
> Was the backplane just incompatible?
No, it was physically faulty. I performed a standard isolation test:
- I took a "suspect" SAS drive and moved it to a known-good bay -> No errors.
- I took a known-good SAS drive and put it in the "suspect" bay -> Immediate errors.
I did the same with the cables. During both tests (drives & cables), the errors stayed with the specific bays on the backplane. This proved the backplane was introducing signal noise on some bays. The SAS drives screamed about this noise immediately, whereas the SATA drives had been tolerating (and hiding) it.
To sum up, ZFS kept the data safe (logical layer), but switching to SAS enterprise drives exposed a rotting hardware component (physical layer) that consumer SATA drives were ignoring. If I had been using a hardware RAID card with SATA, I likely wouldn't have known until a drive dropped offline entirely.
I later read that backplanes do degrade regularly due to temperature stress etc. I put in a replacement backplane and all was solved.
Maybe 75 W without disk is not terrible but it's not good. My unoptimized ARM servers idle at about 3 or 4 W and add another 1 or 10 W when their SSDs or HDDs are switched on.
75 W probably need active cooling. 4 W do not.
Anyway you can probably do many more things with that 75 W server.
A raspberry pi will go even lower. But as you say, not really comparable in term of functionality. I am comparing this to high end consumer NAS, which will idle at 30-40w minimum.
Well, a Synology NAS would probably consume like 30-40w, so we are talking about an excess of $70-100 a year where I live. Depends on one's budget of course, but not really a big deal for me. And certainly less than what I am saving on the upfront cost.
It's frustrating when a comment is both needlessly belligerent flamebait AND wrong about electricity prices in the US. I guess that's what makes effective flamebait
German electricity prices are around €0.38/kwh based on my quick googling which is roughly $0.44/kwh. I pay $0.12-0.13/kwh in the US so I’m at least right factually up to rounding.
They have failed to have a sensible industrial and energy policy, leading to around net 0 GDP growth since 2019. I’m sure for the degrowth elites though this is not a failure it is working as intended.
After all the USA produces tones of co2, ignores all responsibililties in the world since trump (defunding USAID) and lets not talk about income inequality?
But hey, im pretty sure your cheap energy prices will keep inflation low and living standards high.
Btw. China took over the industry products export 2018 and since 2024 chinese companies primarily buy chines made machines.
The main problem of germany is not energy, its def one topic, but its the market force of a country like china who is huge, has a ton of resources, less worker rights, high work attitude.
China is getting rich now from the same thing which made USA big in the 60is and kept germany high until now.
We are still the 3th biggest country by gdp.
Edit: And germany is so anti innovation its not funny anymore. Even the slow car makers started pushing EVs but then everyone else said "NOOOO". Heat pump? Industry said "yeah ready" and then everyone else "NOOOOO".
While this happend in Germany, in China EVs are normal already...
The hardware has different form factors (19"), two power supplies, very loud, very power hungry.
There are so many good combinations of old and still functional hardware for consumers.
My main pc 6 years ago had a powerful cpu and idle load of 15 watts due to the combination of mainboard and amount of components i had in it (one ram block instead of 2 or so).
And often enough, if you can buy enterpsire hardware, the hardware is so outdated that a current consumer system would beat it without looking at it.
If you then need to replace something, its hard to find or its differennt like the power supply.
The datahoarder community frequently utilizes used hard drives.
That's perfectly fine, if your NAS has redundancy and you can recover from 1 - 2 disk failures, and you're buying the drives from a reputable reseller.
I usually buy used hard drives, but prices are strange for all electronics right now. It's a bad time to buy anything computer-related, but especially used goods which aren't discounted as much as normal are priced higher due to massive inflation (to the point that refurbished drives I bought 5 years ago have a better dollar/GB ratio than refurbs I can buy today).
I am not sure what the stats are but I am certain they left behind a huge number of projects. Probably a population that won't be very vocal on HN but that doesn't mean they don't exist.
Even for new projects there are problems I run into that force me to use 4.8. Like if you build an Excel formula, the mandatory async/await approach just doesn't work. It's not an asynchronous operation and runs into a UI context where you deadlock if you wait. They also broke a lot of the windows integration, where in a corporate environment the same network calls with the same syntax authenticate successfully with 4.8 but fail with core.
And because they broke backward compatibility on so many libraries, it's a non trivial effort to convert a complex code base to core.
It's great that they focus on performance, but the .net framework fossilised in term of functionality. It took like 15 years before they added a json serializer to the standard library, and don't even think about support for any new major image format (webp, heic). Everything has become complicated, there isn't a day where I don't see a crash message in visual studio. I used to be a big .net fan but I miss Anders' leadership.
On images... I think it might be best to handle a lot of it outside the core... possibly a nuget package that embeds ImageMagick, OptiPng and some other utilities for the purpose of image manipulation in general. The .Net experience itself has always been somewhat poor here.
As to the breaking changes... .Net Core 1.0 was nearly a decade ago... I understand that some people may want to continue running Windows 7 too, but eventually the world moves on and you either update or use a stale version.
The shift to Core and the breakup of a lot of bits comes down to the need to better support running on platforms outside windows... or, .Net itself would have likely died off completely. Most of the .Net projects I've worked on for close to a decade now have been deploying to Linux/Docker... If it weren't for Core/5+ as a shift, it would have been under another language/toolkit entirely.
> Like if you build an Excel formula, the mandatory async/await approach just doesn't work. It's not an asynchronous operation and runs into a UI context where you deadlock if you wait.
Last time I did excel interop it was COM based and there wasn't any async part of it. I'm curious if you were using COM interop also? Also, async/await was explicitly designed to work in UI contexts like Winforms and WPF where there is only a single UI thread...?
> It took like 15 years before they added a json serializer to the standard library..
That isn't really true. DataContractJsonSerializer [0] landed in .NET 3.5 which was in 2007. Admittedly, it kinda sucked but it was there and was usable. And also JSON.Net was around by that point and was/is excellent.
> ...and don't even think about support for any new major image format (webp, heic).
Image support on windows was historically provided by WIC [1] and does support the formats you talked about. But you are correct that native .NET support for many image formats is non-existent.
> And because they broke backward compatibility on so many libraries, it's a non trivial effort to convert a complex code base to core.
This is very true, and I felt it firsthand. My employer still has a codebase on .NET Framework (compiled against 4.5.2 but deployed against 4.8). It is WCF based and the jump to Core was a massive break. But in the end, I think the break was a good decision. There were just too many design mistakes, bad assumptions and underlying system changes to keep compat across the big leap to a modern multi-platform framework. .NET today is faster, more flexible and has more capabilities than .NET Framework ever did. Even if it did take a long time to get here.
And besides, even if new features are not coming to .NET Framework anymore, Microsoft has support .NET 3.5.1 until 2029! [2] Isn't 22 years of support enough? (.NET 4.8's EOL hasn't even been announced yet!)
For writing excel formulas, while I think you can use VSTO/COM, you will get a poor performance if you do. The "right" way to do it is with ExcelDNA/ManagedXLL. But the excel calculation engine was never designed to support async/await, and your calculations operate within a UI thread. So if you need to make a network call in one of your excel functions, you quickly run out of options.
RTD functions are a different thing. It may be useful for certain cases but in my usage would break the calculation model, as the calls must be made in a given order, and must be refreshed by the user deterministically. I can also build some more complex workaround, like calling a different process, though the antivirus in the corporate environment I work in is unlikely to like that.
reply