That's why I just use cpan2deb. There's a few modules that won't package properly with it, I should file some bugs.

I also use cpanm when I have a need to be separate from vendor, but generally in the worst-case admins prefer to only maintain one set of perl/cpan rather than two.