For me the hardest part is adapting. Let me explain - I can design a game in my head, or watch another game, and instantly know how it's done. I imagine a lot of people here are like that - we don't need design documents, we need left alone to figure it out, then a leash during the whole implimentation stage. It's not solely a thing that you learn from writing games, I think it comes from education. I studied technical drawing, in fact I was the very last person to study technical drawing at my school - I missed the exam in 4th year, so took the class again in 5th year, and obliterated the final exam

. Anyway, technical drawing teaches you to imagine 3D objects in your head based on line drawings. 3D artists will have this as well, just through practice. I'm not sure what they call it, spacial cognition or something hoity-toity like that. Anyway, it applies to game design perfectly, instead of imagining how cool your game will look, imagine how it works, how things move, how things react. Spend enough time thinking like that, and your game eventually gets set in stone, all done bar the actual development. You already know how the game engine has to work, you already know what data the particles need, you already know how to make the sky look like you decided how it should look. (anuerism)
My point is that the most important thing in game development is practice. Things might seem daunting, until you program them a few times - then it becomes instinct... you can imagine every facet of your game idea. The problems arise when you try to explain it to someone negative, or you have to plan it - that's when programmers need a little organisational help sometimes. Without that help, someone to help push the project to completion, well procrastination sets in, and that spells doom. Obviously a game project can be finished, and this can all be disregarded - but what I'm talking about it realising that original game idea in full, and not settle for the game you end up making, spend another year on it, go big or go home, why are you crying?...
Last paragraph I promise. Maybe game development is a bit like getting your hair waxed... like people compliment you and congratulate you, like waxy honey stuff smothered over your bits... but then sometimes it's good to have an honest (potentially bitter) person come along, and rip the hair right off your back. Quick, panful, and honest critique is a valuable thing - don't squander it by looking for compliments... dig for more details, let them tell you what they really think, and damn if you don't secretly think those things as well. It leads to a much more solid game at the end of the day.
So I'd say, the most difficult part of game dev is all the stuff that happens after you think you've finished writing your game. That and finding decent sound effects.

Health, Ammo, and bacon and eggs!
