Python Web Crawler

Spiders and webs

Web Spiders can be very useful to the Website Administrator, in addition to indexing your sites, you can load test the server and an intelligent web crawler can even simulate normal -> moderate -> high levels of web traffic, allowing you to benchmark your website and server performance information.

While there are potentially malicious applications for any web technology, the utility of the Web Crawler makes having one handy a good idea.  Here is my implementation, using Python multiprocessingurllib, and socket.

Continue reading “Python Web Crawler”

New mailer.py

The Python smtplib module offers a high-level interface to what is typically a very low-level process, offering handy dictionary returns and several convenience methods to process and send SMTP e-mail messages.  Anyone who has worked with e-mail knows it can be a beast, with effective diagnosis of issues often involving time-consuming review of cryptic logs and a cascading list of potential “gotchas” to miss or misinterpret; this tool is part one in a suite of utilities that will help to streamline the scripted sending of messages and provide a more human-readable alerts system based on scraping e-mail headers of received messages.

Continue reading “New mailer.py”

Installing Dwarf Fortress on a Debian Droplet

I recently began playing this gem of a game again, and I decided I wanted to set it up on my Debian droplet so I can access the same game from any location.  I recently installed Rogue and this seemed like a logical next step.  I went ahead and downloaded the archive and extracted it.

Continue reading “Installing Dwarf Fortress on a Debian Droplet”