It took so long because IDE _is_ good old PC AT ISA bus with all of its warts. History wall of text:
First PC hard drives were Seagate ST-412 with matching MFM controller plugged into 8bit ISA slot of original IBM XT PC. Drive connected to controller in pretty much exact same way as a floppy - signals controlling STEP/DIRection, selecting HEAD/Drive, and finally pre-processed READ/WRITE data in form of impulses (instead of raw analog signal from/to head, https://github.com/raszpl/sigrok-disk for way too much info). This with later ST-506 was enshrined as PC ~standard with all Bioses hardcoding support.
Then came RLL controllers, same thing but with tighter tolerances allowing for denser encoding (7.5Mbit) at same flux rate (5Mhz).
ESDI is where someone said wait a minute, that 30 cm cable transmitting impulses going to/from media mated to random controller is not all that optimal, lets put our highly tuned RLL Endec (encoder/decoder) chip on the drive itself and talk synchronous serial to it. This combined with probably better magnetic media allowed for another boost to flux rate (10-20Mbit) and capacity.
So what do you do when whole PC industry standardizes on one vendors (Seagate ST-412) product? You emulate it. Western Digital, who started making big bucks thanks to 1976 FD1771 floppy controller and later ST-506 compatible solutions, worked since 1984 to glue whole ST-506 ISA controller onto the drive itself and connect it to ISA bus over the ribbon.
>"Compaq bought 90% of Conner’s drive output in 1987, the first year of production. Conner sales went from $10 million in Q1-1987, to $30 million in Q2, and finished 1987 at over $113 million, then over $256 million in 1988. Conner reached $1.337 billion in sales within four years, a record growth for a startup."
Finally in ~1994 (unclear who first https://www.os2museum.com/wp/the-secret-history-of-atapi/) CDROMS piggybacking on IDE and taking advantage of ATAPI standard started shipping. Packetized SCSI, development again driven by Western Digital! started in 1992, first draft 1993.
As all historical 'it was a good idea at the time' hacks go IDE is not a nice interface. Consider the case of two IDE devices on same ribbon, those are in effect two ISA disk controllers pretending to be _one and the same_ ISA disk controller!
>Device 0 has to act differently depending on whether Device 1 is present; if Device 1 is present, Device 0 has to let it respond to register accesses directed to it, but if there is no Device 1, then Device 0 has to respond to simulate a two-drive controller with no second drive attached.
TLDR: SCSI you have nice synchronous clocks and exchange sane commands. IDE you need to know how to talk to raw PC AT CPU bus. Luckily Polpo is the creator of PicoGus and thankfully got convinced in 2022 to abandon full Raspberry Pi 3 in favor of $5 Pico https://www.vogons.org/viewtopic.php?p=1078867#p1078867 One of my finest retro hobby moments :P
>I assumed you would need pico RP2040 with PIO to respond fast enough. Add ~$2 8MB SPI PSRAM APS6404L and you could do any sound card emulation on pico
>If you can handle ISA then one could do IDE (Gotek like Optical Driver Emulator - Is it possible?) reusing your code
Emulating IDE is the next logical step once you know how to emulate ISA device.
I can only say that from purely user/tinkerer POV (not some low-level system programmer) the best memory of IDE/PATA I have is when the PCs have finally moved on to SATA/AHCI. Phew.
First PC hard drives were Seagate ST-412 with matching MFM controller plugged into 8bit ISA slot of original IBM XT PC. Drive connected to controller in pretty much exact same way as a floppy - signals controlling STEP/DIRection, selecting HEAD/Drive, and finally pre-processed READ/WRITE data in form of impulses (instead of raw analog signal from/to head, https://github.com/raszpl/sigrok-disk for way too much info). This with later ST-506 was enshrined as PC ~standard with all Bioses hardcoding support.
Then came RLL controllers, same thing but with tighter tolerances allowing for denser encoding (7.5Mbit) at same flux rate (5Mhz).
ESDI is where someone said wait a minute, that 30 cm cable transmitting impulses going to/from media mated to random controller is not all that optimal, lets put our highly tuned RLL Endec (encoder/decoder) chip on the drive itself and talk synchronous serial to it. This combined with probably better magnetic media allowed for another boost to flux rate (10-20Mbit) and capacity.
So what do you do when whole PC industry standardizes on one vendors (Seagate ST-412) product? You emulate it. Western Digital, who started making big bucks thanks to 1976 FD1771 floppy controller and later ST-506 compatible solutions, worked since 1984 to glue whole ST-506 ISA controller onto the drive itself and connect it to ISA bus over the ribbon.
It all came together when Compaq started pushing IDE adoption. First with some Compaq Portables using Western Digital WD1003-IWH IDE to MFM adapters screwed into Miniscribe MFM drives, and then mayor 1987 deployment in volume with Conner CP341 (https://www.os2museum.com/wp/can-this-conner-talk/ http://s3.computerhistory.org/groups/ds-conner-cp340-family-...).
>"Compaq bought 90% of Conner’s drive output in 1987, the first year of production. Conner sales went from $10 million in Q1-1987, to $30 million in Q2, and finished 1987 at over $113 million, then over $256 million in 1988. Conner reached $1.337 billion in sales within four years, a record growth for a startup."
In the mean time in 1984 DEC shipped first computer CDROM RRD-50 (Philips LMSI CM100) using weird Philips LMSI synchronous serial interface (https://github.com/AkBKukU/CM153-Repro). Later Sony/Panasonic/Mitsumi opted to copy IDE solution by mating 8 bit parallel bus straight to ISA (https://www.chiark.greenend.org.uk/~theom/electronics/panaso...) visible to computer as four IO mapped ports.
Finally in ~1994 (unclear who first https://www.os2museum.com/wp/the-secret-history-of-atapi/) CDROMS piggybacking on IDE and taking advantage of ATAPI standard started shipping. Packetized SCSI, development again driven by Western Digital! started in 1992, first draft 1993.
As all historical 'it was a good idea at the time' hacks go IDE is not a nice interface. Consider the case of two IDE devices on same ribbon, those are in effect two ISA disk controllers pretending to be _one and the same_ ISA disk controller!
>Device 0 has to act differently depending on whether Device 1 is present; if Device 1 is present, Device 0 has to let it respond to register accesses directed to it, but if there is no Device 1, then Device 0 has to respond to simulate a two-drive controller with no second drive attached.
This use case was only fully standardized in late nineties (https://www.os2museum.com/wp/the-dual-drive-ide-hell/).
--------------------------------------------------------
TLDR: SCSI you have nice synchronous clocks and exchange sane commands. IDE you need to know how to talk to raw PC AT CPU bus. Luckily Polpo is the creator of PicoGus and thankfully got convinced in 2022 to abandon full Raspberry Pi 3 in favor of $5 Pico https://www.vogons.org/viewtopic.php?p=1078867#p1078867 One of my finest retro hobby moments :P
>I assumed you would need pico RP2040 with PIO to respond fast enough. Add ~$2 8MB SPI PSRAM APS6404L and you could do any sound card emulation on pico
>If you can handle ISA then one could do IDE (Gotek like Optical Driver Emulator - Is it possible?) reusing your code
Emulating IDE is the next logical step once you know how to emulate ISA device.