Python Web Crawler

Spiders and webs

Web Spiders can be very useful to the Website Administrator, in addition to indexing your sites, you can load test the server and an intelligent web crawler can even simulate normal -> moderate -> high levels of web traffic, allowing you to benchmark your website and server performance information.

While there are potentially malicious applications for any web technology, the utility of the Web Crawler makes having one handy a good idea.  Here is my implementation, using Python multiprocessingurllib, and socket.

Continue reading “Python Web Crawler”

New mailer.py

The Python smtplib module offers a high-level interface to what is typically a very low-level process, offering handy dictionary returns and several convenience methods to process and send SMTP e-mail messages.  Anyone who has worked with e-mail knows it can be a beast, with effective diagnosis of issues often involving time-consuming review of cryptic logs and a cascading list of potential “gotchas” to miss or misinterpret; this tool is part one in a suite of utilities that will help to streamline the scripted sending of messages and provide a more human-readable alerts system based on scraping e-mail headers of received messages.

Continue reading “New mailer.py”

New PokeyConfig Release!

In an effort to get away from custom-derived formats, I’ve retired legacy delimited files in favor of more universal JSON and YAML formats (either one!).  This will require the PyYaml module to be added in most cases.  Using pip :

The feature list for the next major release will include some sort of dependency management.

Continue reading “New PokeyConfig Release!”

WorldGenerator Update

The first obstacle to tackle with the PokeyGame idea was world generation – how do you generate a randomized world that can be traversed?  I ultimately decided to build a procedural level generator capable of churning out levels on demand, or according to a predefined structure and saved to a template file that can be read later.  This lead to the WorldGenerator class I’ll be demonstrating.  This is my first iteration, I’ve decided to come up with my own method, and then investigate other methods once I am happy with mine to see how it compares.

Continue reading “WorldGenerator Update”

site_backups alpha version released!

Alpha Version

PokeyHost v0.1-alpha

The website backups utility is functioning in a limited capacity, there are few a still bugs to iron out but runs are succeeding without error and the results are as expected.

While I was intending to use the MySQLdb module to handle all database activities, I opted to use it instead only for the connectivity and diagnostic testing portion of the application.  When it comes to actually dumping the data, mysqldump is a simpler (and faster in some cases) solution.

Continue reading “site_backups alpha version released!”

PokeyWorks Update

Check out the newly added PokeyWorks PokeyConfig description at the project page.

This class is at the core of PokeyWorks and was the initial reason I started the framework.  It offers a file-based configuration superclass, enabling a familiar route to configuring an application, and the flexibility of multiple separate configurations.

See my GitHub repository, I will have a Readme available soon.