It's not that difficult:
1st time. All the pieces are in position X and it is white's turn to move.
2nd time. All the pieces are in position X and it is white's turn to move. This is a repetition.
3rd time. All the pieces are in position X and it is black's turn to move. This is not a repetition because the conditions are different.
The same thing would apply if for the first two times your rook was on a1 and you could castle, but for the third time your rook had moved to b1 and then back to a1. The pieces are on the same squares but the position is different because your castling options have changed.