I wonder if this spider adheres to the robots.txt convention. And how to react if, for example, auction sites block it.

At heise online news there's the original article.
I wonder if this spider adheres to the robots.txt convention. And how to react if, for example, auction sites block it.

At heise online news there's the original article.