The problem is that the collision is never detesced fo the walls but I managed to do something (regardless of how stupid it sounds).
I made the .x file by first making a 3ds and then converting it to X. For that model the walls worked but the floor and ceiling didn't
. So now I broke it up into 2 objects. I make the floor and ceiling object as before directly and I make the walls object via a 3ds. So now it all works. From that I'm guessing the problem is in the model data and not in the code (since I didn't change a thing in the code). Something to do with the normals data.... maybe... well isn't this exciting.
Now it's onto sliding collisions, yay fun.