PokeyCrawl version 0.1.1a5 is now available in the Python Package Index or PyPi! See the Github page for details on installation and usage, and the detailed post for an explanation of the mechanisms used.
Spiders and webs
Web Spiders can be very useful to the Website Administrator, in addition to indexing your sites, you can load test the server and an intelligent web crawler can even simulate normal -> moderate -> high levels of web traffic, allowing you to benchmark your website and server performance information.
While there are potentially malicious applications for any web technology, the utility of the Web Crawler makes having one handy a good idea. Here is my implementation, using Python multiprocessing, urllib, and socket.
In an effort to get away from custom-derived formats, I’ve retired legacy delimited files in favor of more universal JSON and YAML formats (either one!). This will require the PyYaml module to be added in most cases. Using pip :
#sudo pip install --upgrade pip
#sudo pip install pyyaml
The feature list for the next major release will include some sort of dependency management.
The first obstacle to tackle with the PokeyGame idea was world generation – how do you generate a randomized world that can be traversed? I ultimately decided to build a procedural level generator capable of churning out levels on demand, or according to a predefined structure and saved to a template file that can be read later. This lead to the WorldGenerator class I’ll be demonstrating. This is my first iteration, I’ve decided to come up with my own method, and then investigate other methods once I am happy with mine to see how it compares.
The Pathfinder class is used in the WorldGenerator functionality to ensure each level is passable, and I plan on integrating with the enemy AI for random level traversal and encounters.
The website backups utility is functioning in a limited capacity, there are few a still bugs to iron out but runs are succeeding without error and the results are as expected.
While I was intending to use the MySQLdb module to handle all database activities, I opted to use it instead only for the connectivity and diagnostic testing portion of the application. When it comes to actually dumping the data, mysqldump is a simpler (and faster in some cases) solution.
Check out the newly added PokeyWorks PokeyConfig description at the project page.
This class is at the core of PokeyWorks and was the initial reason I started the framework. It offers a file-based configuration superclass, enabling a familiar route to configuring an application, and the flexibility of multiple separate configurations.
See my GitHub repository, I will have a Readme available soon.
I took a moment to knock out a description of my ProjectEuler.net solutions. Check it out! ProjectEuler