[TUHS] Mirror with link

Jose R. Valverde jrvalverde at cnb.csic.es
Sat Oct 15 01:59:53 AEST 2011

That is already included: -m == --mirror

But --mirror (or -m) does not include -k (convert links to local after
the transfer) nor -np (do not follow links upwards the parent directory),
nor an instruction to ignore 'robots.txt'.

The magic incantation I submitted will only download down the hierarchy,
in spite of robots.txt and fixing links, all three problems reported in
the thread.

Of course it is not polite to ignore robots.txt, but sometimes it may be


On Thu, 13 Oct 2011 11:43:37 -0700
Larry McVoy <lm at bitmover.com> wrote:
> On Thu, Oct 13, 2011 at 08:37:27PM +0200, Jose R. Valverde wrote:
> > Just for the record.
> > 
> > The correct way to mirror a site with links corrected is
> > 
> > 	wget -c -m -k -np -e robots=off URL
> > 
> > Seems most people have problem remembering this incantation.
> Wouldn't it be nice if it were
> 	wget --mirror URL
> ?
> -- 
> ---
> Larry McVoy                lm at bitmover.com           http://www.bitkeeper.com

		Scientific Computing Service
	Solving all your computer needs for Scientific


More information about the TUHS mailing list