These two things suggest gear noise.
A bearing, axle, etc would not make noise in only one direction.. The meshing of the differential gears, and which side of the teeth are being loaded, would.
Thought it's quite rare to hear of a factory (Getrag) assembled diff becoming noisy from incorrect tooth mesh, it's not impossible. If it was a tuner-built box, then it's a more common complaint.
Finding a good, used differential isn't that costly. Especially for an A4 car. Lots of people with A4s upgrade to 3.42 ratio, leaving the 2.73 and 3.15 boxes somewhat undesirable (and cheap). Changing the diff can be done in 2-3 hours without the need to fully remove the drivetrain.
If it turns out to be this, you also may wish to consider upgrading to a 3.42 diff out of a 6spd for a little added acceleration kick, but when doing this, a higher stall tq converter becomes a recommended compliment, as is some minor tuning to adjust shift points. Not absolutely essential, but recommended.
Being that it's impossible to ID a noise of this nature over a forum, I would find a trusted mechanic near you to have a listen and advise accordingly before proceeding or just throwing parts at it.
