Humble Beginnings

As I’ve mentioned a few hundred times, kStats began as a simple fork of StatPress Reloaded to speed things up and create a plugin more suited to larger applications of Wordpress.

Due to the nature of how StatPress chose to store statistics and report on them, a StatPress table had a tendency of growing extremely large, extremely fast. Not only did this approach create a severe bottleneck when visitors tried to access your site, but it was sadly even worse when you tried to retrieve the resulting data on the administrative side.

Due to this, I figured the best approach for kStats was to restructure the existing format to use aggregated data. Much smaller more accessible records that gave you a fast look at real time numbers combined with past totals.

Statistics are important

This approach worked great at first. kStats was fast, it recorded data quickly and it reported it to the site administrator just as fast. But over time, it’s starting to show its weakness. It gives you the numbers, but what about the meat of your stats? What if you want to see what happened last month? What about 3 weeks ago? Or 16 weeks ago?

I started to develop a new aggregate process which would store almost four times as much data, allowing the existing system to remain in place, and the reports to grow in features considerably. I could still sense impending doom with this approach though. What about further down the road? Would there be a ceiling to how much data I could store in an aggregate form? Would I just move the bloat from one storage method to another, winding up in the same pit that StatPress did?

Learning from our mistakes

In the end, I think StatPress was on the right track. I think I was too, but instead of forking down a completely different road, I think the best approach would be to learn a lesson from both approaches, and to turn that into a single new approach.

I was drowning in workload over the holidays, while simultaneously trying to keep up with family functions. I didn’t get a lot of time to work on kStats, and I must apologize to anybody I left out to dry as a result. There were some bug fixes desperately needed, and while they’re out now, they should’ve been out then. However while I didn’t have as much time to work on kStats as I would have liked, I did find some time here and there in the mornings and evenings to get some reading in.

It’s all about the database

I consider myself fairly competent when it comes to MySQL, and more importantly database design. The more I learn though (as with life), the more I realize I don’t know. SQL by itself, without the various layers that bring it to life on the web, is an art form all by itself.

I’ve been studying up on the subject, because I know it’s a very important part of this design process, as well as the design and development of a few other projects I’m working on right now, and I think I’ve drawn an outline for a new database structure that will allow kStats to break the mold a little when it comes to statistics recording and analysis for Wordpress.

Using a combination of MySQL engines, such as the ARCHIVE engine, and a completely redesigned schema, I think I can bring the speed without sacrificing any data. There’s no reason you shouldn’t be able to look up historical periods of time and see exactly what occurred. There’s no reason you shouldn’t be able to click a button and produce a detailed report using any piece of recorded data as the focus (as opposed to being restricted to predefined reports). This is what statistics are for after all, right?

The bad news

While the plugin is still technically considered a beta, I would rather not release a version that destroys all the data that’s been recorded to date. This may be unavoidable to some degree though.

While data that currently exists in the raw table of your kStats install is easy enough to move over to the new format with no data loss, the data that’s already been summarized might not be so easy to transfer.

Before I dive headlong into this new strategy I’m going to do everything in my power to determine how we can avoid such a loss. It may be as easy as providing a legacy utility which will store the historical data in a different format, and retrieve it when building certain reports. The problem I’m facing is that this aggregate data may be unusable in producing certain reports that require a particular level of accuracy.

It’s tough. I have a project board dedicated to brainstorming this, so I’ll be sure to keep everybody up to date on how it’s going to go down, well before it does.

Now that the holidays are nearing an end, I’m still finding myself under a heavy workload. While this is of course a priority to put some food on the table, I am going to do my best to balance some free time in to continue development of kStats – I have some big plans that I hope to start implementing soon, of which I will write more about in a future post.

The fixes

I have a new permanent note written on my project board; ‘Check compatibility’. I chose with the last release to make use of PHP’s built in filter_var() functionality, in lieu of reinventing the wheel for a validation routine. This method however wasn’t introduced until PHP 5.2.0, and caused the plugin to not function properly for anybody using it previous to that release. Fixed.

There was also a problem with the charts not displaying correctly on new installs. Despite the setting on the options page, a chart would only display as many days as it had data for, and didn’t fill in the blanks. This has been fixed, and should now display a full chart, regardless of data for a particular day.

Download Here   Changelog

We’ve got a little christmas shopping to do after, as an earlier planned trip in the other direction is now highly improbable, but we will be home by Sunday. At that time I promise to immediately get on top of responding to all the comments I have received here since then, as well as to my email which I don’t currently have access to. I apologize for any inconvenience, and I’ll talk to you all again when we’re back home!

Thank you to everyone who has been part of the development, by supporting the plugin through using it, writing about it in your blogs, sending me feedback via comments, email or bug reports, and anything I missed mentioning! It’s been a lot of fun, and I hope to see it continue growing in popularity.

I’ve got a list a mile long of new features that are up and coming, and I’m currently working on yet more improvements to the database structure as we speak in further attempts to ensure that this is not only an accurate and feature rich plugin, but a lil’ speed demon too. Again, thank you all, and don’t be shy about sending me your criticisms or suggestions!

Mod What?

I just recently installed Mod Security on my server in an attempt to reduce the number of attacks on my blog and the ridiculous referrer spam that I’ve been getting lately. I’m sorry, but I have no interest in taking a screenshot of the latest kStats and showing everybody that 5 out of ten of my top referrers are coming from some stupid subdomain of a**f***d****.com (you’ll notice the most recent screenshot has the top referrers chart collapsed for a reason).

While it deals mainly in HTTP and regular expressions, of which I’m familiar with, the syntax is completely new to me. I hope I haven’t turned on any rules that result in any odd behaviour for anyone; If you notice anything out of the ordinary while trying to perform common tasks such as leaving comments, please let me know so that I can fix it asap. I already had to rewrite a few rules because I managed to make it believe using phpMyAdmin was some form of attempted SQL injection attack…

The problem was with the htmlentities() php function I was using — all information coming from the database should be trustworthy, due to sanitization on the way in. However I figured it couldn’t hurt to wrap it on the way out again, and make sure on both sides of the equation.

Since PHP 5.2.3 htmlentities() has allowed for a fourth argument, which if set to false won’t encode already encoded html entities. By running this on data to be displayed I figured it would help catch any mistakes that slipped by on the way in and ensure no malicious javascript could be injected into your dashboard. The problem is I forgot to read the changelog on the function and didn’t realize at first that it was only available on 5.2.3 and up, causing an error to be displayed for anybody running an earlier version.

The wrapper has been updated with a version check. If you’re running 5.2.3 and up it runs with the flag set. If you’re running an earlier version, it simply decodes the string first then encodes it again to make sure all html entities are caught.

Remember to upgrade your copy of PHP, or harass your sysadmin to do so for you! Not just to cover my blind spots (though it doesn’t hurt!) but for the sake of your own security. Keep up to date. (Disclaimer: I realize this responsibility is most often supposed to be that of the host. Hosts, despite providing otherwise exceptional service, can be dinosaurs when it comes to upgrading. Harass them.)

Thanks for catching that one Jake.

Download

What’s new in 0.7.1?

You won’t notice any major visual changes or fancy new features in this release. I fixed a possible vulnerability in the way that some of the data was stored and retrieved and added a new opt-in program which benefits the plugin and another program, both of which I’ll go into further detail on below.

I did however bump up the versioning from 0.6.x to 0.7.x because there’s something new going on behind the scenes that will be a long term benefit to kStats and the people who use it on their blogs.

The Old Way

The aggregate is tripped every night by somebody visiting your web site. Long story short, this would be better accomplished via a cron process run directly off the server, but due to the nature of plugins and Wordpress, expecting a user to set such a thing up just to use kStats would be asking a little too much.

When the aggregate was tripped, previous to this release, the process would run fast as fast can be and sort your data from the raw table into the seperate totals and charts tables. This of course allows kStats to run faster on a regular basis, and store more information with a much smaller footprint than its predecessor did. The pitfall was that the poor sap who tripped the process had to wait anywhere from 1-3 seconds extra for their page to load (possibly even longer on high traffic web sites).

In this age of broadband expectations, 3 seconds is an eternity.

The New Way

kStats now uses what is called an Asynchronous HTTP Request to run the aggregate. When the scheduled time comes, kStats fires off an HTTP request to an interface that runs the whole process in the background. This means that poor sap we were talking about above no longer notices a delay in their page load, no matter what the size of your database is or how much traffic you’re getting.

I promised when I started this project that the primary focus, regardless of features and capabilities, was to bring you the fastest plugin I could. I believe this update goes a long way to solidifying the groundwork of that promise.

Odds and Ends

There’s a new opt-in program that can be found on the Options page under the Definitions Utility – while I’m still looking for a more reliable Geolocation API (hint, hint), the user agent facility (determining OS, Browser, etc) is powered by the API provided by user-agent-string.info.

Should you choose to participate, what happens is when kStats stumbles across a user agent it can’t identify, it will immediately fire it off to user-agent-string.info so that they can identify it and include it in the next update of their API. The more user agents we can identify, the more accurate the process will be in determining exactly what people are using when they visit your site.

In addition, a possible security vulnerability has been closed up in the way that some data was being stored and returned from the database. The upgrade process will clean your current database and all information entered from now on is completely verified and sanitized. Please note that this was not an SQL injection vulnerability but instead a much smaller XSS vulnerability.

Download Changelog

So other than some changes I made to the bar graph, I’m putting a complete freeze on adding new features to kStats until I’ve got this one nailed down. I’m going to go through each and every file and line of code one by one, and see where I can make improvements, fixes, or just finish commenting if nothing else.

Tracking down the beast

At the same time, since the only feasible source of the problem I can see is the nightly cleanup routines (as nothing else interacts directly with the totals table), I’ve taken a few steps. I’m now running a logger on it, so every time it trips I have a neat little text file that spits out every query, object and variable that’s part of the routine. I’m also running the wp-cron hook on an hourly basis, instead of nightly like regular users, so that it runs in a day what would normally take almost a month.

So far I haven’t seen the data get truncated again, so it may have been as simple as adding the call to ignore_user_abort().

In the meantime

There may be a few rapid fire bugfix releases of the 0.6.x series in the next few days.

Due to the feature freeze, they will not include any database changes or major updates that require you to run the upgrade utility. Each one should drop into place and self-update without any headache caused on your part.

Well, after an entire week of testing 0.6.0 every day on my own blog to try and have kStats first major bug free release, I thought I was good to go. So I tagged it and dropped it in the repository earlier this evening, thinking “Finally!”.

Not an hour later, it happened.

My data was gone. All except for yesterday and the new hits coming in after midnight. Nothing was missing from the raw table or charts, something happened to truncate the totals (aggregate) information. The problem is, I still have no better clue what it was. It happened right after the wp-cron event for kStats was tripped, so I presume it has to be occurring during the cleanup routine. So I went in and debugged every query that was going on.

Every single query ran fine. I tripped the cron 7 times. Each and every time worked perfectly. So what the HECK is going on?

I’ve been banging my head against the walls and my dining room table for a few hours now trying to track it down. What good is a stats plugin that arbitrarily truncates data?! The only possibility I’ve come up with so far that seems remotely feasible is that it happens when a search engine or bot trips the cleanup – maybe they’re hitting a page and leaving so fast that it’s only getting part way into the process before it gets canceled?

So I just finished tagging 0.6.1, hopefully before too many people have already upped to 0.6.0. I added (and probably should have since day one anyways) ignore_user_abort() to the primary cleanup function to ensure that even if the page is exited before the script is done, it will follow through. I had this in the collector, I’m not sure why I didn’t put it there before.

If that doesn’t fix the problem, I’m going to turn the cron feature off and make it a utility on the Options page until the reason is tracked down. I would like to release a stable version. I can’t until this is worked out.

*sigh*

Would it be funny if I just left the screenshots from 0.1.0 up forever?

The path to stable

So far I’ve been managing to hit all of my goals, even surpassing a few during the beta period of kStats. I fubbed up the nightly aggregates for the first little bit, until I realized half my problem was my attempt to reinvent my own wheel. The other half fell under the same umbrella, I was just making things complicated that could have been simplified. Once I crested that hill, things starting falling into place.

There have been other bumps too. Never before this have I developed a publicly released plugin, and had to deal with automatic upgrades, collective groups of database upgrades and other such freakiness. Before now, if something broke, it wasn’t a big deal. I could just fix it.

Now I have to make sure nothing breaks, because the general public is playing with my potential disasters. I rushed a few releases I really shouldn’t have, and had to tag bug fixes within minutes to hours of those releases. Talk about red in the face. I’m fine-tuning my release process, and this time around I’ve been running the latest version for over a week on my own blogs to try and throw everything I could at it. It was well worth the effort too, I discovered over a dozen bugs that might have been serious deterrents to people making use of my plugin.

We’re over halfway through the beta process now, and I want at least four version releases without a single (major) problem before I finally tag the 1.0.0 release. It’s looking up.

What’s new?

kStats has come quite a ways since 0.1.0 and it’s single page of statistics. Over the past few releases I’ve added both a dashboard and blog widget, organized all the statistical data into much easier to navigate pages, and developed an Options page which allows you, the user, to gain some control over how kStats works.

With this release, I’ve been listening.

The database

I’ve actually had quite a bit of help and feedback from people named Rene. One of the Rene’s dug some skeletons out of my closet and pointed out the fact that I hadn’t really updated the SQL schema since porting from StatPress. While I’ve still got some future plans to improve the capability and structure of the current schema, I took heed!

The tables have been updated considerably to reduce size and increase speed of access. IPs are now being translated to long so that they can be stored in a more appriopriate INT field. I dropped the ID column for now as it has yet to server a purpose, though it may come back if I go through with adding meta data tables to further improve the speed of the main raw data table. I took two fields that have set data in them and reduced them to ENUMs, and the rest are now VARCHAR.

The only stipulation is that you have to be running MySQL 5.0.3 or greater to take full advantage of the VARCHAR fields (and lack of disk writes that come with them in most queries). The url, referrer, and user agent columns need to handle data greater than 255 characters in length, so there is a version check in place to determine the version of your MySQL server – if you’re lower than 5.0.3 you should upgrade immediately kStats will automatically set these columns up as TEXT.

File system structure

The whole structure of the plugin has been modified to maintain the highest level of organization possible, while keeping the door wide open for future expansion in a variety of areas. A few new methods have also been added, including one that permits the Options page to load utility files. Each utility is a self contained feature that can be removed as easily as deleting the file. On the flip side, new utilities can be added (and developed) at the drop of a hat.

I’m not entirely sure there’s a need for plugins on a plugin – the idea was mainly to make part of my life easier and for the plugin to operate better in your lives, but it’s just another possibility for the future of kStats.

Access Permissions

Another request I had that was already on my todo list; the ability to allow other blog users to view the stats has been added. From the Options page of the plugin you can now set varying levels (based on the built in Wordpress user roles) of access to either view statistics or manage options for them.

Eventually I would like to upgrade this facility to deny access from everyone (including other administrators with the exception of the blog creator) if desired, or add people on a user by user basis regardless of their level. This function will also extend to MU and the looming merge to include site wide options, per blog options and their respective permissions.

Deactivation != Destruction

The deactivation hook of kStats no longer destroys all the accumulated data. I’ve moved this to the uninstall hook, to accommodate situations where you may want to or need to deactivate the plugin without losing everything that’s been recorded. The whole upgrade/installation process has been highly modified to streamline events.

If you work by hand, like I do, there is also an uninstall utility located on the Options page which will manually remove the database tables and any options added by the plugin.

StatPress Conversion

The Conversion utility has been upgraded again. Instead of manually backing up your old StatPress table, the utility creates a temporary table that it works from, so nothing extra is required on your part to ensure data integrity. I’ve also stopped my silliness of mangling the data directly inside of the StatPress table to populate all the kStats tables. The table is only modified as needed to transfer the data into your raw table, at which time the aggregate methods are run on the raw table to populate your totals and charts tables.

The option has been added to run the kStats user agent identifier on your StatPress table. While this feature will provide a much higher level of accuracy in identifying bots, browsers and operating systems, be warned! I ran it on a small table of 40 thousand some odd rows and it took over 2 minutes to complete. So if you’re going to use it, be prepared to go find yourself a snack.

I’ve also had the pleasure of testing it on a much larger database (over three hundred thousand rows), and though the process can be time consuming, it works!

Internacionalización

I should have been doing this from day one. It’s been a tedious process, but I’ve completed a huge portion of the work necessary to provide i18n support for kStats. I’d have to say tentatively that about 90% of the plugin is now running Wordpress i18n functionality, and by the next release (and many more coffee’s) I should have support completely in place and ready to go for translations.

I’m not sure if there’s anybody out there that will be interested, and honestly even when it’s fully in place I’d suggest that anybody willing would wait until the stable version 1.0.0 release. Things just change so much right now that it might be a bit of a stressful project to keep up with.

Odds and Ends

Long enough post for you? I’m sorry, I’ll try and wrap it up.

The interface now runs postbox’s. You can move things, collapse them, and generally it makes everything look prettier. I want this plugin to look like it belongs on Wordpress, instead of trying too hard to make it look stylish.

Recent Hits is now Recent Pageviews, and you can group the results by IP or by the page viewed. There’s also a search feature, where you can type in an IP or IP fragment to locate specific entries. You’ll also notice a couple of new icons down the right side. When you see these, it means that the view either comes with a referrer attached, or from a search engine. Hover over them to find out.

I forget the rest…

What’s coming next?

I’ve got some plans for improving the collector. I want to start running it against a session to trap even more information such as time spent on a page, time spent on the site as a whole, bounce rates, and more. This would eventually evolve into a system (I hope) where by you could visually map each visitors path through the site.

Thanks to the recent (and persistent) RFI attacks on my blog, I’m thinking of adding an exploit scanner to the plugin. Basically this would check the visitor against a list of known exploits, attempt to determine the threat level, and notify you of such attacks. Possible attacks would be highlighted in the recent pageviews list, as well as maintain their own list. Features that may extend from this would be the ability to block such people from accessing your site either intentionally (by you) or if the threat level detected is high enough, the plugin could reject them itself.

I just discovered the Wordpress built in Snoopy class. I’m hoping to integrate this for use by the definitions update utility, once again reducing the additional code presented by kStats and using already available features of WP.

I’m also going to split up the top agents charts into two – top OS and top Browser. Right now it’s running against the User Agent string itself, and thus a lot of the results seem duplicated, when in reality they’re not. It however does seem more logical to group by the extracted names rather than the string itself, as the same version of the same product can produce different UA strings dependent on outside factors.

As for other possible upgrades, as usual, I’m waiting to hear what you’d like to see! I hope you enjoy the plugin.

Screenshots   Changelog

Does your grandma know what you’re doing in her rent-free basement?

I’m sorry, I realize most of you probably live at home with mommy and daddy, and I shouldn’t be bringing your grandmother’s into this, but seriously. Half of you tried to attack my site with a phpBB vulnerability. I’m sorry, last time I checked, I was running Wordpress.

Put down the porn, sign off your favorite MMORPG, change your clothes and shower for once in your life and walk to the front of your house. There’s a big rectangle there with a little round knobby thing on it. This is called the front door. If you open it, there’s a life somewhere out there for you. Go find it.

You’re not getting root on my box, trust me. I’m sure a skilled hacker could if they put enough effort into it, but you’re not.

I don’t know who amongst my readers might have the skills and the time, but if you want to slap some children around with a big trout, here’s a few IPs from my logs;

61.63.10.150
64.6.232.171
194.105.193.46
209.216.213.119

Have a nice day!