Carl is easily annoyed, but doesn't always gets roused into anger. It's why his reaction is him rolling his shoulders and continue on his way. "Does it matter to you?"

He remembers going on the road for a long, long time, often times not knowing where his next shelter will be. This will be no different.