I would be more concerned about the seed plants pollinating the ones that don't have seeds than I would be about the seeds reducing quality... and yes they will as the plant will dedicate more resources to developing the seeds and thus have less left for thc and bud development
it's kind of like putting norton on your computer...
it still works but it's slow and annoying...