Should the White House be allowed to do this? My opinion: no. There might be an innocent technical explanation, but I can’t think of one. I suppose, on the long list of crimes this administration has committed, preventing Google, Yahoo, and other search spiders from indexing public materials is low on the list. But it’s nonetheless wrong to deliberately make it harder for people to find content on the White House site.
For non-techies: a robots.txt file gives instructions to search engines as to which links they may and may not follow (and therefore index.) This file is instructing all search engines to NOT follow and NOT index any directory that follows a “disallow” statement. A cursory glance tells me they’re blocking just about everything. Most interesting? All budget materials, NSC materials, and press releases.
I do believe a domain owner has the right to control where search spiders are allowed to go. However, in this case the domain owner is the American People. So, search engines: this is one robots.txt file I encourage you to ignore.