im no expert on animation, but it looks like you are trying to manipulate the arm in an unnatural manner. try it with your own arm and study how it moves. you can freely rotate about the y axis as seen on your left side pic, this is how the arm is built, the motion provided by deltoids and trapezieus among others.
once you have it up around 90 degrees on the y you can rotate forward on the z, but if you try to twist, rotating on the x you can see that the shoulder joint has almost no ability to do that. most of the twisting comes lower down, somewhat at the elbow, primarily at the wrist. this twisting is possible do to the forearm being made of two bones which share a mostly fixed base at elbow. the shoulder just isnt built for this kind of twisting.
basically, it is not necessarily an issue of your mesh or rig, but an issue of how you are using them.