Wordpress stuff, a statistics plugin, and jello

It’s been awhile since I’ve written anything on here or been able to do much work on kStats, and I must apologize for the absence. The holiday season came, and paid projects piled up for a short period that kept me far away from getting some much needed updates out.

Now that the holidays are nearing an end, I’m still finding myself under a heavy workload. While this is of course a priority to put some food on the table, I am going to do my best to balance some free time in to continue development of kStats – I have some big plans that I hope to start implementing soon, of which I will write more about in a future post.

The fixes

I have a new permanent note written on my project board; ‘Check compatibility’. I chose with the last release to make use of PHP’s built in filter_var() functionality, in lieu of reinventing the wheel for a validation routine. This method however wasn’t introduced until PHP 5.2.0, and caused the plugin to not function properly for anybody using it previous to that release. Fixed.

There was also a problem with the charts not displaying correctly on new installs. Despite the setting on the options page, a chart would only display as many days as it had data for, and didn’t fill in the blanks. This has been fixed, and should now display a full chart, regardless of data for a particular day.

Download Here   Changelog

As I was writing my kStats introductory post for 0.7.1 I concurrently received a bug report which should be fixed now.

The problem was with the htmlentities() php function I was using — all information coming from the database should be trustworthy, due to sanitization on the way in. However I figured it couldn’t hurt to wrap it on the way out again, and make sure on both sides of the equation.

Since PHP 5.2.3 htmlentities() has allowed for a fourth argument, which if set to false won’t encode already encoded html entities. By running this on data to be displayed I figured it would help catch any mistakes that slipped by on the way in and ensure no malicious javascript could be injected into your dashboard. The problem is I forgot to read the changelog on the function and didn’t realize at first that it was only available on 5.2.3 and up, causing an error to be displayed for anybody running an earlier version.

The wrapper has been updated with a version check. If you’re running 5.2.3 and up it runs with the flag set. If you’re running an earlier version, it simply decodes the string first then encodes it again to make sure all html entities are caught.

Remember to upgrade your copy of PHP, or harass your sysadmin to do so for you! Not just to cover my blind spots (though it doesn’t hurt!) but for the sake of your own security. Keep up to date. (Disclaimer: I realize this responsibility is most often supposed to be that of the host. Hosts, despite providing otherwise exceptional service, can be dinosaurs when it comes to upgrading. Harass them.)

Thanks for catching that one Jake.

Download

I have yet to track down the source of the problem. It’s hard to fix a problem that you can’t a) reproduce and b) even if you could, since the process is run deep behind the scenes you can’t step through it.

So other than some changes I made to the bar graph, I’m putting a complete freeze on adding new features to kStats until I’ve got this one nailed down. I’m going to go through each and every file and line of code one by one, and see where I can make improvements, fixes, or just finish commenting if nothing else.

Tracking down the beast

At the same time, since the only feasible source of the problem I can see is the nightly cleanup routines (as nothing else interacts directly with the totals table), I’ve taken a few steps. I’m now running a logger on it, so every time it trips I have a neat little text file that spits out every query, object and variable that’s part of the routine. I’m also running the wp-cron hook on an hourly basis, instead of nightly like regular users, so that it runs in a day what would normally take almost a month.

So far I haven’t seen the data get truncated again, so it may have been as simple as adding the call to ignore_user_abort().

In the meantime

There may be a few rapid fire bugfix releases of the 0.6.x series in the next few days.

Due to the feature freeze, they will not include any database changes or major updates that require you to run the upgrade utility. Each one should drop into place and self-update without any headache caused on your part.

I’ve had reports of it for many releases now – data is lost from the totals table, for no apparent reason. I’ve seen it in the search terms people use to find my site and the kStats plugin, and I’ve had bug reports and numerous comments about it. Until now, I’ve never seen it happen and therefore had no basis to start my search in fixing it.

Well, after an entire week of testing 0.6.0 every day on my own blog to try and have kStats first major bug free release, I thought I was good to go. So I tagged it and dropped it in the repository earlier this evening, thinking “Finally!”.

Not an hour later, it happened.

My data was gone. All except for yesterday and the new hits coming in after midnight. Nothing was missing from the raw table or charts, something happened to truncate the totals (aggregate) information. The problem is, I still have no better clue what it was. It happened right after the wp-cron event for kStats was tripped, so I presume it has to be occurring during the cleanup routine. So I went in and debugged every query that was going on.

Every single query ran fine. I tripped the cron 7 times. Each and every time worked perfectly. So what the HECK is going on?

I’ve been banging my head against the walls and my dining room table for a few hours now trying to track it down. What good is a stats plugin that arbitrarily truncates data?! The only possibility I’ve come up with so far that seems remotely feasible is that it happens when a search engine or bot trips the cleanup – maybe they’re hitting a page and leaving so fast that it’s only getting part way into the process before it gets canceled?

So I just finished tagging 0.6.1, hopefully before too many people have already upped to 0.6.0. I added (and probably should have since day one anyways) ignore_user_abort() to the primary cleanup function to ensure that even if the page is exited before the script is done, it will follow through. I had this in the collector, I’m not sure why I didn’t put it there before.

If that doesn’t fix the problem, I’m going to turn the cron feature off and make it a utility on the Options page until the reason is tracked down. I would like to release a stable version. I can’t until this is worked out.

*sigh*

kStats Reloaded v0.2.5 is now available for download! Despite having an issue with tagging the new release on SVN, the plugin repository itself has updated from trunk and anybody using 0.1.x (if anybody is after the fatal error fiasco) should upgrade immediately to 0.2.5

The changelog has been updated, but I’ll go over it quickly here;

New Improved SQL Queries

A lot of the functionality has been updated with new SQL queries that reduce the need for extraneous PHP routines. This primarily focuses on the aggregate (nightly cleanup) functions, however it has by proxy extended to Views module and in particular the overview page which shows all the Top XXXX lists. All data is updated in real time.

Fatal Errors

The Datetime class has been fixed, so no more Fatal Errors from that department (and no more in general I hope!) At the same time I updated a few of the methods to run faster, by removing unnecessary routines that could be simplified with a little thought on my part.

Nightly Cleanup

I rewrote the nightly cleanup from the ground up, as my previous model was based on flawed logic in other departments. Information accumulated before this update is probably severely skewed, and all tables should be dropped before continuing to gather information. This can be done by deactivating the plugin and reactivating it, or manually if you are so inclined.

I plan on making releases all the way through 0.1.x to 0.9.x before declaring it stable, though I’m hoping to have this area in particular solid by 0.3.x and bullet proof by or before 0.5.x – All your continued support through filing bug reports and providing me with feedback via comments is greatly appreciated!