Rufus does not think a change of scenery will help him for two reasons: first, his race will always pose a problem, no matter where he goes; second, the thing he struggles with most is his past behavior, which will follow him wherever he goes. Additionally, if he moves, he will have no friends and family, much like Leona when she first came to New York. This passage also serves as build up for the reveal of what Rufus did to Leona. Both he and Vivaldo have mentioned it several times at this point without saying it explicitly, suggesting that whatever it was, it was pretty bad.