It's not talking to an LDAP server, it's the functionality for talking to an LDAP server that is causing the issue. Even if you don't need LDAP you're still vulnerable when a client can inject information in a log message.

Why is this functionality needed in the first place? I want to write log, some kind of string, into some kind of files, with rotation, maybe even send it somewhere that expect logs.

Why parse whatever is in the logs, at all?

Imagine the same stuff in your SSH client, it would parse the content before sending them over because a functionality requires it to talk to some server somewhere, it's insanity.

Log4j contains a very big collection of extensions for just about anything including inserting data from various sources. Of course it's overkill for lots of situation, but nobody ever uses all functionality. It's just that nobody can agree on which functionality is useless ;)