Just like in reality, in the Harry Potter world people feared magic. I don’t think it was explicitly said, but the magical community is much smaller than the muggle one. Witches and wizards are just afraid of being murdered.