Thumbnail Scraper for Cartographer's Choice

[Feralspirit]

So... I feel very selfish, bringing this up. This is a fantastic site already. Has anyone ever discussed the idea of a thumbnail scraper for the Cartographer's Choice Forum?

Edit; To clarify, I'm not even sure what all is involved in such an endeavor. It's simply that the thumbnails in the finished maps forum is so cool, but there isn't a similar page on the works we have chosen as exceptional.

[Redrobes]

Thats a good idea. I will see what I can do.

[Redrobes]

Is this ok now ?

[Feralspirit]

Wow, that was a quick response. It's awesome, TY.

[Immolate]

A search through Google revealed to me that yes, there are indeed a whole variety of tools and implements that were invented and produced to scrape one's thumbnails, but little that helps me to understand what exactly a thumbnail scraper does. Can one of you be so kind as to explain the concept to the graphically-challenged?

[Feralspirit]

Look here (new, as of today) and here and the challenge forums.

[Redrobes]

A scraper is a program that requests documents (like pages off the web) and scrapes off some information on them to build up another document or page. Usually this is for stuff like scraping the daily financial data to build up historical finance data.

We started doing monthly challenges before I arrived here but at the time there were about 4 or 5 entries per month. Then as time went by we started to get more people enter each month and it started getting hard to work out what the state was of all the entrants 'work in progress'. So I write a WIP scraper and published a page of entries. Then this was used to make up the finished maps thread and now there's a Cartographers Choice thumbnails page too.

I also make a members introductory post index, a keyword index, and the CWBP place index and these are also scraped from the guild threads.

I also do the members map and the Ansium overview image but these are generated with external data and not scraped.

[Robbie]

ummm...I think I may have broke it or something by moving it out of cartographers choice...I would REALLY like to implement some kind of built in thumbnailer similar to CGTalk. (mentioned in another thread)

I've had a pretty crappy weekend and this is probably going to be a pretty crappy week, so I may be scarce this week, but I'm definitely willing to work some kind of cool thumbnail function.

Redrobes, I'm tempted at this point to give you a cgi-bin ftp account you can host stuff for us on so we're not killing yer bandwidth. What would you require to get all of the awesome stuff you've built for us hosted here with access for you to continue to work on it as necessary?

[Redrobes]

It shouldn't have broken it as it just points to a page that I host its not on relative address or anything. Still, ill look in a mo...

If I had ftp I could host the thumbnail images on the main server but all of them finished, choice and all the challenges are only 70Mb so from a bandwidth point of view its not a biggie.

With cgi ability I could have the script run on the server which would be good because it could possibly fetch the image and resize it to the thumb quicker and also maybe get the pages faster but again since its only done now and again and only fetches the image if its changed then thats not going to make a big difference.

The area that would provide a great ability would be to have the indexer and scrapers all running off of a cron job so we could have them all updated daily. Having them run on the server and save the attachments there would mean that theres no login or ftp of the results so it makes sense to do that on the server. The other option is to have a web page with buttons on it where we can stab one to get it to update the thumbs. Thats a bit easier than running the script and uploading.

The requirements of the indexer etc is that it has the perl modules for some of the things it uses. One in particular is the image magick one which is what I use to resample the images. I have the modules locally of course but I know my ISP for my site has it installed so its quite likely.

The members map could be sent to the guild web server directly since its just running a straight up CGI bin thing for the map. No problem there. I think there are some vBulletin members map plugins tho if we wanted to change to that. I kinda like the current one as its nice and simple. I would be cool to have the link to it as a members only link too. Theres nothing at mo to stop anyone adding themselves in.

From the point of view of the site I think it would look nicer to have the info hosted from the save server keeping the same domain name but there's some small uncertainties about how we can update them all more automatically than we do currently. You should have all these files and ability without reliance on me anyway.

EDIT -- and to actually answer the question... ho hum... yep, just ftp to the cgi area, some temp space which might be in the cgi area for files in progress and members locations etc and some directory to host the scraped info / thumb images.

[Redrobes]

Ok, first of them comming through now...

http://www.cartographersguild.com/ut...ChoiceMaps.htm

Anyone have any problems with this ?