Unfortunately Goku, we observe and measure everything from our point in time and space, which does not always provide the real picture. You will have seen the balloon effect, where any two points on the surface of the balloon move apart much faster than the air molecules in the centre of the balloon. However in our case the inflation is rather like filling a balloon with some gas and placing it in a vacuum chamber and gradually increase the vacuum. The balloon will expand, but now the gas inside the balloon changes density, the molecules also spreading out, so we now not only have different rates of expansion between the centre and the surface, we also have a changing 'fabric' (dark matter). Despite the conditions locally being relatively constant, in other localities the rate of change is huge.
My understanding of the expansion is that it is caused by dark energy, which I thought is supposed to have a relatively consistent density throughout the universe which is related to the universe's isotropy and homogeneity. I subscribe to the lambda CDM model by the way. So the idea that the universe has a center and that the edges expand faster than the center is foreign to me. It's not that gas is expanding to fill a vacuum, rather space itself is expanding and the gas is just along for the ride. I personally prefer the bread dough analogy to the balloon, but both are just analogies and shouldn't be taken too literally in my opinion.
I am confused on your use of dark matter. My understanding is that dark matter (once you account for MACHOs that are hard to detect which artificially raises the amount of dark matter in cosmological surveys) is a non-baryonic substance(s) of matter that does not interact with the electromagnetic force. So I don't know what you mean by dark matter being related to the fabric of space, can you expand on what you're talking about there?