Was it racism? No. He certainly experienced it, but he was used to that.
Was it poverty and wage labour? No. He was certainly poor, but he was able to cobble a living together and was used to lean times.
What constantly stressed him out was the freedom to make his own life choices. In Cuba, he tested well in math so they made him an engineer. They told him what university to attend, what his job would be, and where to live. Now he had to make these decisions himself.