Alright, so in this sort of business, keyword scraping is slightly black hat, …. but what the hell.

So there I was last night sat looking at the output of the current scrape my application is running. It was sat at about 8.5K words, after something like 140 scrapes. Now, the way my app runs is that it will scrape all those 8500 words, plus any new ones that appear in those extra 8360 scrapes, and so on, until it’s drilled down to EVERY word that it’s scraped.

But that’s just stupid. The vast majority of those scrapes will return 0 new words. But… those last few words returned will be ones that no-one else has scraped.

So where do you draw the line?

Do you keep on scraping, hoping to gain words that no-one else is using?

Or do you say “enough is enough” and move onto your next root keyword?

I’ve decided on a compromise. I keep the number of new words found by the last 100 scrapes, and when the average drops below 1, I quit scraping that keyword and start the next one.

Any thoughts, Dear Reader????

Technorati

No responses yet