23Dec
I know I haven’t posted to this site in a very long time, but I have been working on some email related projects for quite some time.
I’ve been writing an open source email (SMTP,IMAP and POP) server in Erlang called ErlMail at http://erlsoft.org which has been happily taking up all of my spare time.
http://erlsoft.org is the main website for the ErlMail project as well as a DNS server, VoIP server and a web server with it’s own markup language.
All of the projects are on Google Code now as well at the following URLs:
Well, the lets you know where I’m at and what I’ve been up to for the past few months.
22Mar
It is old news that Google uses the sitemap protocol which allows webmasters to tell Google’s spider, Googlebot, which web pages have been updated recently. The greatest advantage is that Google will use the sitemap to make sure they spider your web site in a more intelligent way that you control within the bounds of the sitemap protocol.
The sitemap protocol has grown beyond just Google these days. You can find out more information about the protocol itself at sitemaps.org. They give a good overview of the protocol, which is pretty simple, and in the FAQ they imply that other search engines are using the sitemaps protocol.
That fact turns out to be true. For some time now Yahoo has been using the sitemaps protocol as well. On their Site Explorer web site you can let them know about your sitemaps in much the same way that you would inform Google and they also take feeds, such as RSS feeds, in addition to the sitemap protocol.
MSN is also suppose to be using the sitemap protocol, but I haven’t found the page to submit any sitemaps to.
At this point the only thing that I can think of that would make this better is if there was a way to for auto discovery of sitemaps. Something similar to the auto discovery options for RSS feeds, a META tag that point to one or more sitemaps on your domain.
With this type of auto discovery method any search engine could use the sitemap protocol and have an intelligent way to spider your site without having to login to all of the different services that use sitemaps.
sitemaps, Google, Yahoo
Technorati Tags: sitemaps, Google, Yahoo
23Feb
For the past several weeks I’ve been redesigning the site and I recently redirected the RealTimeSpiderList.com to point back here to SpiderHunter.com.
I’ve been moving my servers around and I am no longer hosting the servers at my home. I eventually plan to host these servers on my own hardware sometime off in the future, but my own personal plans making hosting my own server nearly impossible at the moment.
So I’ve recreated the site as best I can on a WordPress blogs. I’m still reworking some of the content to make sure the URL are as similar as possible, but some of the content will be lost in the transition.
If you notice something that you want back, email me and I’ll re-post the content in blog form instead of articles or whatever format they were in.
Spider Hunter
Technorati Tags: Spider Hunter
07Jan
I was wandering through my local book store and I happen to come across a book with an interesting title IP Location. It turns out to be a book on geolocation from several different points of view. I flipped through it and I’m planning on purchasing it soon, I also put it up on my amazon.com wish-list in case anyone wants to make my week.
04Nov
For the past month I have been working on the Netflix Prize, trying to create some collaborative filters in pure Erlang. While I am no where near a winning score, at the moment, I have learn a lot about how I want to create my own websites in a pure Erlang environment. One of the more interesting things is going to be the fact that I will be needing my own spider, and in the standard Erlang way I have named it SpidErl. Erlang people like putting Erl into the names they give things.
SpidErl will mostly just fetch things in the first few versions, I’m going to be collecting RSS feeds and some Amazon affiliate data for the most part. I think I’m going to write a module that collects AdSense data as well, obviously you need a user name and password to get that data. In the long run I hope to create a full fledged Search Engine around SpidErl’s information store, but that will be month or years from now.
In the mean time look for some notes about building my own spider and a site redesign taking advantage of the information that the spider collects 
19Jun
In the most interesting Google and cloaking related story I’ve found lately, as you might notice from the lack of blogging, it seems that the New York Times is using cloaking technology on their web sites. I’m not sure who actually broke this story, but Danny Sullivan from search engine watch is quoted as saying that what the New York Times is doing looks like cloaking in his opinion.
What in fact is happening is that Googlebot is able to see the contents of the paid area for the New York Times without actually paying. This is shown by the fact that when someone searches for content on the new York times web site the information displayed in the Google pages is different than what a non subscriber would get if they went to the New York Times website.
This has always been one of the arguments I have seen in favor of cloaking. The ability to have your content indexed by certain IP addresses for search engines spiders, while not allowing people to see the content without at least some sort of registration. In my mind this is 100% in cloaking technology, but this is the type of cloaking that should be allowed. Keeping that in mind, the Google terms of service does not allow cloaking of any kind. So while I completely support this use of cloaking technology it seems only fair that Google would have to ban all of the web pages from the New York Times website. This is an obvious violation of terms of service and if Google makes an exception for a predominant web site such as the New York Times they leave themselves open two potential legal actions from site owners who have been banned under this policy.
Personally I don’t like either of those options. My personal preference would be to see the Google terms of service modified in such a way that cloaking technologies were allowed so long as certain criteria are met, with the intention behind those criteria to be displaying the same or extremely similar content to both the search engines spider and the end user. Of course that opens up another completely different set of problems where Google would end up having to view the contents of both a cloaked and uncloaked pages and then make their decision on whether or not the cloaking technology was the appropriately. That gets us into a whole subjective realm of what constitutes similar content.
So the bottom line here is that it would appear that Google is damned if they do anything and if they don’t. The chances of getting the New York Times to stop using the cloaking technology would appear to be slim at best and now that this is public knowledge if Google does nothing they may see other consequences.
http://blog.outer-court.com/archive/2006-06-19-n18.html
18May
I’m a big fan of Jenstar from webmaster world as well as her JenSense web site. For the most part I’m interested in the information about AdSense and other contextual advertising mediums, occasionally she surprises me with information that is usually a bit more technical than I would expect. (Not that Jen isn’t technical, but her site is usually about advertising)
About a month ago she made a post in which she shows that her log entries verified that the Google AdSense media partner GoogleBot was adding information into the Google search engine. While Google maintains that they have different search engines spiders for different purposes, it certainly does not surprise me that the information is being used across their different services.
In fact it would be the best use of their resources to have a single search engine spider that collected all information about a website and then the each of their different services used that one huge database differently. I do understand Google not wanting people think that having an AdSense account would get them into the search engine results there are faster in any way. It just seems to me to be a huge waste of resources.
http://www.jensense.com/archives/2006/04/adsense_mediapa.html
17May
Google announced recently on one of their blogs that they are now using META tags for site map validation. I personally think this is a fantastic idea and wish they would have done this from the beginning.
The Google site map program seems to be obsessed with proving that you own the page that you’re tracking. Which all in all is not a bad thing, but initially they do not have enough ways to prove you’re the owner of the site. While they have been adding more ways to validate your site in the past few months the initial way of validation was a file that could be put in the root of your home page.
In my case the site map service would not validate the file because my 404 error messages were not always returning 404 error codes. This was done on purpose and was something that I was unwilling to change.
Interestingly enough I had several of my sites that validated just fine. All the sites were on the same server and all the sites are configured in the exact same way, but for some reason about 25% validated fine while the rest were denied for not returning 404 error codes.
I have not tried the new META tag for site map validation yet as I am currently redesigning the core of many of my sites, but during this redesign I will be implementing the new META tag validation and I look forward to many other new features from site maps as well.
http://sitemaps.blogspot.com/2006/05/more-about-meta-tag-verification.html
20Mar
I’ve had a few requests to have access to my JavaMail IMAP CFC so I’m releasing it for anyone who wants to play with it.
This is free open source code. No need for any licensing at all. There is no warranty and I will not be providing any technical support.
I highly recommend that you use the code as a basis for your own implementation and not use this code directly. Especially in a production environment.
I know of some issues that i has regarding how it does log-ins and I had always intended to somehow cache the log-in so that it would be able to do multiple transactions at once. If you find the code helpful, please put up some comments about it and if you feel like re-releasing the code I’d love to hear about it.
Download JavaMail IMAP CFC
18Mar
I just went through and cleaned up the comments across this site. Seems some people felt it necessary to spam the blog comments, if I notice this anymore I’ll have to rework the comment code to limit or require registration, which I do not what to do…
And for the record, the comment sections have code that tells spider to not follow any of the links and the Google AdSense code to not use the comments as part of the page they use to target the ads. So from a spammers perspective, putting up your link on this site will not help you in any way …