Dignity nobility good quality point out highness standing class caste position superiority precedence trend difference dominance preferment primacy nobleness gentleness gentility preeminence loftiness station grandness txt file is then parsed and may instruct the robot regarding which web pages usually are not to become crawled. For a internet search engine https://www.youtube.com/watch?v=XcL519f9jd0