The original MacOS was vastly more complex than contemporary 8-bit operating systems. I think Apple produced very useful documentation with Inside Macintosh - the Human Interface Guidelines that came a bit later are still useful today.
However, I think there was a mismatch of expectations, since the Apple ][ was documented down to the last bit, including the schematics and ROM source code (excluding the Applesoft BASIC developed by Microsoft).
The creators of the Mac wanted to force developers to adhere to the abstractions Apple provided with the OS and to discourage developers from using low-level tricks. This enabled Apple to evolve the Mac much further than the Apple ][ series. The last model of the Apple ][ series, the //gs, had to provide a lot of hardware to enable backwards compatibility with software written from 1977 on.
In contrast, the Mac survived a relatively pain-free transition to the PowerPC in the '90s, which was one of the first wide-spread uses of binary translation in commercial systems (later used for the transition of OS X to intel and now to Arm) - even significant parts of the kernel were still written in 68k assembler on PPC systems.
Of course, there were some significant problems to overcome in the 68k era. One well-known example is that developers (even Apple's developers themselves) abused the most significant 8 bits of addresses to store data. The original 68000 had 32 bit address registers but only a 24 bit external address bus, so the 8 MSBs were ignored by hardware. This was no longer true on the 68020 and later CPUs and caused lots of problems...
Talking about the state of documentation of macOS, even the old NeXT documentation was better and more in-depth - though there was no detailed information on the hardware, which causes problems for the developers of the Previous emulator today. Of course, the NeXT systems were much less complex than it is today, even though you already had to cope with coprocessors such as the Motorola 56001 DSP and the i860 on the Dimension color graphics card (which unfortunately could not be programmed directly)...
The original Inside Macintosh was quite decent. The main problem, I think, was that it was hard to understand just pieces of it; if you read it cover to cover it was pretty good. But by the time Inside Macintosh VI rolled around with its 1750 pages to cover the System 7 additions, that approach had run its course.
Apple then reorganized and rewrote the entire documentation along functional lines, producing separate, smaller, volumes for e.g. Memory, File Management, etc. That New Inside Macintosh series was, in my opinion, the highest quality documentation Apple ever produced.
It was also assembly and pascal for the longest time, even when most folks had switched over to C.
The heyday was an app that had digitized all the documentation and iirc the code examples were in C. It had hyperlinks between pages. Quite nice pre web reference material.
Their code examples were also helpful, but things like low level double buffered sound recording or high performance animation was left as an exercise to the reader.
I thought it was very clear, and I kept it on my bookshelf long after it had become irrelevant. I only chucked it to make space for other things on the shelf.
