You need to have access to the docker socket, if you're using sockets to connect. That's not necessarily only root.