Wordpress stuff, a statistics plugin, and jello

I’m not sure how to say this, so I think I’ll just spit it out; there are some major changes on the horizon for kStats. These are good changes, and I will go into more detail farther on, but they are changes that I may negatively impact current users of the plugin. Out of 6,000+ downloads, this could mean anywhere between a 2-3 dozen web sites! </tongue-in-cheek>

Humble Beginnings

As I’ve mentioned a few hundred times, kStats began as a simple fork of StatPress Reloaded to speed things up and create a plugin more suited to larger applications of Wordpress.

Due to the nature of how StatPress chose to store statistics and report on them, a StatPress table had a tendency of growing extremely large, extremely fast. Not only did this approach create a severe bottleneck when visitors tried to access your site, but it was sadly even worse when you tried to retrieve the resulting data on the administrative side.

Due to this, I figured the best approach for kStats was to restructure the existing format to use aggregated data. Much smaller more accessible records that gave you a fast look at real time numbers combined with past totals.

Statistics are important

This approach worked great at first. kStats was fast, it recorded data quickly and it reported it to the site administrator just as fast. But over time, it’s starting to show its weakness. It gives you the numbers, but what about the meat of your stats? What if you want to see what happened last month? What about 3 weeks ago? Or 16 weeks ago?

I started to develop a new aggregate process which would store almost four times as much data, allowing the existing system to remain in place, and the reports to grow in features considerably. I could still sense impending doom with this approach though. What about further down the road? Would there be a ceiling to how much data I could store in an aggregate form? Would I just move the bloat from one storage method to another, winding up in the same pit that StatPress did?

Learning from our mistakes

In the end, I think StatPress was on the right track. I think I was too, but instead of forking down a completely different road, I think the best approach would be to learn a lesson from both approaches, and to turn that into a single new approach.

I was drowning in workload over the holidays, while simultaneously trying to keep up with family functions. I didn’t get a lot of time to work on kStats, and I must apologize to anybody I left out to dry as a result. There were some bug fixes desperately needed, and while they’re out now, they should’ve been out then. However while I didn’t have as much time to work on kStats as I would have liked, I did find some time here and there in the mornings and evenings to get some reading in.

It’s all about the database

I consider myself fairly competent when it comes to MySQL, and more importantly database design. The more I learn though (as with life), the more I realize I don’t know. SQL by itself, without the various layers that bring it to life on the web, is an art form all by itself.

I’ve been studying up on the subject, because I know it’s a very important part of this design process, as well as the design and development of a few other projects I’m working on right now, and I think I’ve drawn an outline for a new database structure that will allow kStats to break the mold a little when it comes to statistics recording and analysis for Wordpress.

Using a combination of MySQL engines, such as the ARCHIVE engine, and a completely redesigned schema, I think I can bring the speed without sacrificing any data. There’s no reason you shouldn’t be able to look up historical periods of time and see exactly what occurred. There’s no reason you shouldn’t be able to click a button and produce a detailed report using any piece of recorded data as the focus (as opposed to being restricted to predefined reports). This is what statistics are for after all, right?

The bad news

While the plugin is still technically considered a beta, I would rather not release a version that destroys all the data that’s been recorded to date. This may be unavoidable to some degree though.

While data that currently exists in the raw table of your kStats install is easy enough to move over to the new format with no data loss, the data that’s already been summarized might not be so easy to transfer.

Before I dive headlong into this new strategy I’m going to do everything in my power to determine how we can avoid such a loss. It may be as easy as providing a legacy utility which will store the historical data in a different format, and retrieve it when building certain reports. The problem I’m facing is that this aggregate data may be unusable in producing certain reports that require a particular level of accuracy.

It’s tough. I have a project board dedicated to brainstorming this, so I’ll be sure to keep everybody up to date on how it’s going to go down, well before it does.

kStats Reloaded just passed 5,000 downloads!

Thank you to everyone who has been part of the development, by supporting the plugin through using it, writing about it in your blogs, sending me feedback via comments, email or bug reports, and anything I missed mentioning! It’s been a lot of fun, and I hope to see it continue growing in popularity.

I’ve got a list a mile long of new features that are up and coming, and I’m currently working on yet more improvements to the database structure as we speak in further attempts to ensure that this is not only an accurate and feature rich plugin, but a lil’ speed demon too. Again, thank you all, and don’t be shy about sending me your criticisms or suggestions!

Mod What?

I just recently installed Mod Security on my server in an attempt to reduce the number of attacks on my blog and the ridiculous referrer spam that I’ve been getting lately. I’m sorry, but I have no interest in taking a screenshot of the latest kStats and showing everybody that 5 out of ten of my top referrers are coming from some stupid subdomain of a**f***d****.com (you’ll notice the most recent screenshot has the top referrers chart collapsed for a reason).

While it deals mainly in HTTP and regular expressions, of which I’m familiar with, the syntax is completely new to me. I hope I haven’t turned on any rules that result in any odd behaviour for anyone; If you notice anything out of the ordinary while trying to perform common tasks such as leaving comments, please let me know so that I can fix it asap. I already had to rewrite a few rules because I managed to make it believe using phpMyAdmin was some form of attempted SQL injection attack…

I have yet to track down the source of the problem. It’s hard to fix a problem that you can’t a) reproduce and b) even if you could, since the process is run deep behind the scenes you can’t step through it.

So other than some changes I made to the bar graph, I’m putting a complete freeze on adding new features to kStats until I’ve got this one nailed down. I’m going to go through each and every file and line of code one by one, and see where I can make improvements, fixes, or just finish commenting if nothing else.

Tracking down the beast

At the same time, since the only feasible source of the problem I can see is the nightly cleanup routines (as nothing else interacts directly with the totals table), I’ve taken a few steps. I’m now running a logger on it, so every time it trips I have a neat little text file that spits out every query, object and variable that’s part of the routine. I’m also running the wp-cron hook on an hourly basis, instead of nightly like regular users, so that it runs in a day what would normally take almost a month.

So far I haven’t seen the data get truncated again, so it may have been as simple as adding the call to ignore_user_abort().

In the meantime

There may be a few rapid fire bugfix releases of the 0.6.x series in the next few days.

Due to the feature freeze, they will not include any database changes or major updates that require you to run the upgrade utility. Each one should drop into place and self-update without any headache caused on your part.

kStats Reloaded 0.4.1 for Wordpress is now available for download!

This version finally brings a lot of stability to the platform! A variety of bugs have been fixed, including any probability of fatal errors (I’ve been running it non-stop since last night in full debug to catch any possibles) that may have caused serious issues for people who have downloaded previous beta versions of the plugin.

View the changelog for the short list.

New Interface

The statistics reporting and general administration interface has gone through a complete overhaul. I’ve added a new tabbed navigation system to help organize the data in easy to use groupings, and included a new options page which will grow with the plugin.

Along with the new interface and due to the administrative options page I decided to go ahead and integrate the StatPress Reloaded conversion script. This way people upgrading from StatPress will now find it easy to convert all their data into kStats. If you don’t need to convert, or have already done so, simply delete the file /kstats-reloaded/lib/convert.php and the option removes itself.

Using a similar style interface, there is now an integrated upgrade process that will degrade gracefully all the way back to 0.1.0. Now no matter what version you started with, you will always be able to update to the latest with minimal fuss.

Database Changes

I completely restructured the way the aggregate tables store information. This has not only improved the performance of the plugin during regular operation, but it also allows for much more scalability and future options, including the ability to go back a year and view your monthly statistics, or go back a few months to view daily/weekly activity.

The potential here is to continue in the direction of kStats not only being the fastest statistics recording plugin available, without sacrificing any data.

Nightly cleanup

Since version 0.1.0, I’ve been reinventing my own wheel, and with little success ironically. My nightly aggregate function was a complete rewrite of another function I was already using to gather data from the aggregate tables and raw tables to display real-time information. The problem was that I spent too much time focusing on how to make sure all the new data was dropped in the right spots that I completely saw past the possibility of using the same function (with slight additions) for both purposes.

This has been fixed in 0.4.1, and the nightly cleanup is now not only working flawlessly (I say that with great trepidation, mixed with confidence), but has been clocked on various test runs with tables up to 30,000 rows in well under half a second. That means whomever trips the nightly cleanup shouldn’t half to blink twice before the page loads.

Odds and ends

I’ve combed over every file and every function – there’s a few that I’ve deprecated in favor of better more versatile functions. There was a bug where stats were being misreported at night between 12:00am and the cleanup being triggered due to the fact that the datetime class was reporting the wrong days for a period of an hour, but it has been fixed as well.

Let me know what you think! I feel good about the path from here to version 1.0.0 myself, but what I think doesn’t matter anywhere near as much as what you think!

There are quite a few reasons why developers may choose to use PHP 5 when building their applications. For example, PHP 5’s implementation of object oriented programming is much more robust and in depth than version 4’s almost an afterthought of an implementation. For another, there’s a plethora of new functions that you may find necessary to accomplish a certain task.

Amongst the list of new features, functions, constructs and other ’stuff’ is the simple fact that PHP 4 has been declared dead by its own development team.

How does this affect Wordpress?

The problem facing Wordpress plugin developers is that despite PHP 5 being available for as many years as it is versioned, and PHP 4 being declared dead, is the fact that a lot of hosts out there still use it as their default installation of PHP. In my personal opinion they should all be chastised for this, but in their defense most do have the option to upgrade to PHP 5. You just have to ask (nicely).

If you decide you need to incorporate PHP 5 specific functionality in your plugin, you’re going to have to also allow it to die gracefully for those who are still running under PHP 4.

Trust me, no matter how much you make note of it in your Readme, or display notifications on your web site, people will still be confounded by the fatal error that they are faced with when they attempt to activate your plugin. Parse error: syntax error, unexpected T_STRING, expecting T_OLD_FUNCTION or T_FUNCTION or T_VAR or ‘}’ in /plugin/plugin.php on line 29 means absolutely nothing to somebody who isn’t a developer, and cannot by itself point them in the right direction to fixing their newly discovered situation.

How to point them in the right direction

There are a few ways to go about this, and I’ll show you one of the most direct methods here.

The best way to ensure that your end users won’t install and activate your plugin, only to run into some previously unknown problem a day later (and after spending all that time configuring and implementing it) is to check during the activation process. If the check is run there, and halts activation, you can inform your user of the problem and even help them out in fixing it without ever having to leave your couch.

So therefore the first step, if you didn’t already have one, is to add an activation hook to your plugin. From within this hook we’re going to run a version comparison on the users installed copy of PHP, against the version you are expecting to find.

If you already have an activation hook for your plugin, all you need to add is lines 5 and 6 at the top of your existing callback function.

1
2
3
4
5
6
7
8

register_activation_hook( __FILE__, 'myplugin_activate' );
 
function myplugin_activate() {
 
	if ( version_compare( phpversion(), '5.0', '<' ) )
		wp_die( '<strong>We apologize, but kStats Reloaded requires PHP 5 or above for normal operation.</strong><br />PHP 4 is outdated <a href="http://www.php.net/archive/2007.php#2007-07-13-1">by over 2 years now</a>, and you should contact your host and request that they upgrade your server immediately.' );
 
}

Feel free to get more creative, but at the very root of the situation this will do the trick.

Checking for the existence of…

The check we’ve just run is very simple in nature, due to the fact that I’m only using PHP 5 OOP and not a specific function or functionality that was introduced in a later version (such as 5.1.0, 5.2.0, etc). You can improve on this by adding specific checks with function_exists() or any manner that you see fit.

Related Blog Posts

You may find the following posts on other blogs to be helpful in clarifying this article. Both are well written and explore more in depth methods of providing a solution to the situation.

Nothing But Words: How to Make a PHP 5 WordPress Plugin Die Gracefully in PHP 4
Nerdlife: Wordpress Plugin Installation Hackery