There seem to be a few bug reports all around the same phenomenon. See also JRUBY-6162. I started digging into it a bit last week, but I was in the middle of a release and didn't have time to get very far. My working hypothesis is that the external process is getting killed off by JRuby before it has time to finish executing. The (hacky) workaround is to make your code wait a bit so the process has time to complete. Faster JRuby execution might only make the problem worse :-), but I'm looking forward to playing with it!
6162 does sound very like my issue. I must try sticking a sleep inside the block passed to popen and see if that gets past my problem. It's not a solution as such, as the process I am calling could take anywhere from a second to many minutes to execute, but it will be interesting to try. If you manage to solve these bugs it will be a great help to me, so good luck!