Why is that the standard of who to count? The resulting number is basically the able bodied army of Israel. The number is 603,550 which is about the same (if not larger) than what came out of Egypt which was probably all men. So if anything they are the same or larger than when they came out of Egypt.
So why count the “army”?
After this, God instructs that the Levites will care for the tabernacle “so that there will be no wrath on the congregation.” So the real protection is in caring for the things of God and the real thing to be concerned about is the wrath of God (not man).