Yes your buds will keep growing and keep getting bigger. They are not the biggest at week 6. Not sure who is saying that or why? But, yes, your plants will keep growing and the buds will keep growing in size and filling out.
Like xtsho says, worry about pm and bud rot now while your buds keep...