There is a pretty far cry between low-paid labour and being bred for the sole purpose of being enslaved, having your body manipulated for the benefit of your oppressors and ultimately murdered to be eaten. 

It may not be beneficial for someone in a dire situation to refuse any kind of work, no matter how exploited or poorly paid they are; but, at the very least, they have a choice. Non-human animals do not.

Vegans aren’t the only people who eat vegetables and grains. Carnists eat them, too. Let’s not forget that a lot of exploited labour is in places like slaughterhouses and other undesirable jobs that those with privilege can easily turn down. Then there is the fact that, on top of what grains/veg they consume directly: carnists are responsible for the food eaten by the livestock that they eat, which accounts for a majority of the world’s food.

I am not saying that veganism is a cure for the mis-allocation of food and the injustices in the global agricultural system. The people on the bottom are on the bottom and exploited because the people on the top want it that way for financial gain. It is the responsibility of everyone to be an ethical consumer (both for human and non-human benefit; vegans don’t just care about animals) and veganism is a step in the right direction as it takes away your money, your vote, from a product that is exploitative.