Elio dislikes the phrase “If not later, when?” because it implies two things at once. On the one hand, it suggests that something is bound to happen between him and Oliver, an idea that simultaneously excites and scares him. On the other hand, though, the only actual answer to the question is less optimistic. After all, if something doesn’t happen “later,” then it will
never happen. In this way, Elio senses Oliver’s conflicting desire to move forward with their relationship and stop it before it develops into anything serious.