Exactly, the host is the factor. Here's the other confusion ...
For someone that comes into the room when the two boxes remain and doesnt know which is which, he has a 50/50 chance of choosing the right box.
For the person who owns the box and has the decision of STICK or SWAP, his odds are improved if he swaps.
You can visualise this to an extent by asking someone to pick a number from 1 to 1000 and they'll tell you if its the one they're thinking of. You choose 562. You can see you'll have a 1 in 1000 chance of getting it right. If they then say it's not 1, not 2, not 3, not 4 and so on until they've said it's not everything single number between 1 and 1000, except 97 and the one you chose. Do you still honestly think you had a 50% chance of choosing the right number, or do you think it's probably 97 - the only number they left after they eliminated all the others?
It's the same principle, but reduced to such a simple example that it's hard to visualise the odds. In the example above, you have a 999/1000 if you swap and a 1/1000 if you stick. In the original example, you have a 2/3 chance if you swap and a 1/3 chance if you stick.