Usually we definitely want search engines to crawl our websites and index them for us. However, there is at least one scenario where it is precisely the opposite effect we are seeking - to prevent our development website from being indexed by Google, or indeed any of the major Search Engines. Here is the how and why of preventing your site from being indexed