It, according to Eric Lippert, was at least partially to fix the problem of vexing exceptions[1] that plagued Int32.Parse. Parse was first and it was kind of an unfortunate design decision to make a almost expected result (parsing to fail) throw an exception.

It's worth noting that nowadays Parse is just a wrapper around TryParse that throws (although this is an implementation detail)

[1]: https://ericlippert.com/2008/09/10/vexing-exceptions/