from HTYP, the free directory anyone can edit if they can prove to me that they're not a spambot
Revision as of 11:24, 14 February 2017 by Woozle (talk | contribs) (→‎Notes)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search


computing: software: wget


wget is a command-line utility for downloading pages and files from the web.

Subject Pages


Apparently there is an undocumented option which tells wget to ignore ROBOTS.TXT:

  • -e robots=off
  • "wget -r <url>" will supposedly retrieve all of a site recursively, but sometimes it mysteriously doesn't work.