The name is not entirely accurate. Plenty of people know how to find them, especially the hidden villages of the powerful nations. But living in seclusion helps to keep the secrets of specialized techniques.

[also curious] Why would they need to fear if they were the ones with the power? Or did their enemies create some new technique to neutralize them?