|but it's through 2 gigabit switches|
Are they daisy chained
router - switch - switch
or both plugged into your router separately
router - switch
This latency, is it between clients on your LAN or or between a client and an external source out on the internet?
What are you doing on your LAN that you think you need to use QoS? QoS is usually reserved for VoIP and videoconferencing traffic.