Here are the relevant parts of my .xkb file. Note that the "osfUp" code is code I found elsewhere (don't remember where). Just repeat it for Left, Right and Down.

I first added this new "THREE_HYPER" using three levels (you can add more levels if you fancy that):

    type "THREE_HYPER" {
        modifiers= Shift+Hyper;
        map[Shift]= Level2;
        map[Hyper]= Level3;
        map[Shift+Hyper]= Level3;
        level_name[Level1]= "Base";
        level_name[Level2]= "Shift";
        level_name[Level3]= "Hyper3";
    };

Then the "interpret" does the magic:

    interpret osfUp {
        repeat= True;
        action = RedirectKey(keycode=<UP>, clearmodifiers=Hyper);
    };

And for each key, assign the arrow key redirection you want (showing only "i" for up here):

    key <AD08> { type= "THREE_MESH_HYPER", symbols[Group1]= [ i, I, osfUp ] };

I'd say that it's not a job for xmodmap but for .xkb.

Also I don't use xmodmap at all: I set everything, including all the modifiers, from my .xkb file.

My xmodmap output for Hyper (which I set, from .xkb, as mod5):

    ... $  xmodmap | grep mod5 
    mod5        Hyper_L (0xcf)

You'll want to read on .xkb but this should get you started.