Back in the day, Jews were used as tax collectors. And guess what....tax collectors aren't popular people, even today.
Ancient Tax Collector (Bible History Online)
Another reason might be a failure to assimilate in the communities they inhabited, throughout the years, in the same way we see Muslims in America wearing their hijabs and purposefully not melding with the cultural norm. Then there is the question of loyalty. Remember when JFK was running for president? People rightly wondered whether his allegiance was to the USA, or was it to the Holy Roman Empire and the Pope? Look at Rahm Emmanuel. He's a dual citizen of the US and Israel. Where were his loyalties when he was O's Chief of Staff?
I'm not saying any of that is right(except the dual citizen thing....Rahm, Ted Cruz....I'm looking at y'all), but I can understand it. After all, I doubt people who work for the IRS are popular at parties.