I wouldnt be too concerned, as many people have ran that manifold without port matching, etc...
Ideally, youde like to have both ports the same size, but the head is "upstream" so to say...
But with that being said, After being told my tuner shop that those particular manifolds are super turbulent, to the point where exhaust pulses are making their way up their neighboring runners due to short unequal runner lengths, Ide say that you could actually be dealing with heat going upstream and heating up the head just like you had mentioned. Its all theory from me, but my tuner shop has told me time and time again that the FP race manifold has some flaws to its design. Its hard to say exactly whats going on without some extreme testing. A flow bench wouldnt work either. Youde need to have it hooked up to reproduce the pulses that the exhaust gives off, to properly represent what the exhaust manifolds characteristics are while on an engine.
If you ever get a chance, port match. But i dont think you have to be overly worried. It would definitely be an issue if the header ports where smaller than the heads ports.