Wordpress stuff, a statistics plugin, and jello

I don’t know why, but this blog has been really hard to write. Could be the fact that I’m still extremely sore from ripping the garage apart and cleaning it top to bottom, or the fact that I’m bummed out about my new intake for my car not being in the mail today, but I just don’t find writing easy at the moment. So I’ll just try and spit it out, and eventually it will get lost in my archives anyways…

What’s new in 0.7.1?

You won’t notice any major visual changes or fancy new features in this release. I fixed a possible vulnerability in the way that some of the data was stored and retrieved and added a new opt-in program which benefits the plugin and another program, both of which I’ll go into further detail on below.

I did however bump up the versioning from 0.6.x to 0.7.x because there’s something new going on behind the scenes that will be a long term benefit to kStats and the people who use it on their blogs.

The Old Way

The aggregate is tripped every night by somebody visiting your web site. Long story short, this would be better accomplished via a cron process run directly off the server, but due to the nature of plugins and Wordpress, expecting a user to set such a thing up just to use kStats would be asking a little too much.

When the aggregate was tripped, previous to this release, the process would run fast as fast can be and sort your data from the raw table into the seperate totals and charts tables. This of course allows kStats to run faster on a regular basis, and store more information with a much smaller footprint than its predecessor did. The pitfall was that the poor sap who tripped the process had to wait anywhere from 1-3 seconds extra for their page to load (possibly even longer on high traffic web sites).

In this age of broadband expectations, 3 seconds is an eternity.

The New Way

kStats now uses what is called an Asynchronous HTTP Request to run the aggregate. When the scheduled time comes, kStats fires off an HTTP request to an interface that runs the whole process in the background. This means that poor sap we were talking about above no longer notices a delay in their page load, no matter what the size of your database is or how much traffic you’re getting.

I promised when I started this project that the primary focus, regardless of features and capabilities, was to bring you the fastest plugin I could. I believe this update goes a long way to solidifying the groundwork of that promise.

Odds and Ends

There’s a new opt-in program that can be found on the Options page under the Definitions Utility – while I’m still looking for a more reliable Geolocation API (hint, hint), the user agent facility (determining OS, Browser, etc) is powered by the API provided by user-agent-string.info.

Should you choose to participate, what happens is when kStats stumbles across a user agent it can’t identify, it will immediately fire it off to user-agent-string.info so that they can identify it and include it in the next update of their API. The more user agents we can identify, the more accurate the process will be in determining exactly what people are using when they visit your site.

In addition, a possible security vulnerability has been closed up in the way that some data was being stored and returned from the database. The upgrade process will clean your current database and all information entered from now on is completely verified and sanitized. Please note that this was not an SQL injection vulnerability but instead a much smaller XSS vulnerability.

Download Changelog

I have yet to track down the source of the problem. It’s hard to fix a problem that you can’t a) reproduce and b) even if you could, since the process is run deep behind the scenes you can’t step through it.

So other than some changes I made to the bar graph, I’m putting a complete freeze on adding new features to kStats until I’ve got this one nailed down. I’m going to go through each and every file and line of code one by one, and see where I can make improvements, fixes, or just finish commenting if nothing else.

Tracking down the beast

At the same time, since the only feasible source of the problem I can see is the nightly cleanup routines (as nothing else interacts directly with the totals table), I’ve taken a few steps. I’m now running a logger on it, so every time it trips I have a neat little text file that spits out every query, object and variable that’s part of the routine. I’m also running the wp-cron hook on an hourly basis, instead of nightly like regular users, so that it runs in a day what would normally take almost a month.

So far I haven’t seen the data get truncated again, so it may have been as simple as adding the call to ignore_user_abort().

In the meantime

There may be a few rapid fire bugfix releases of the 0.6.x series in the next few days.

Due to the feature freeze, they will not include any database changes or major updates that require you to run the upgrade utility. Each one should drop into place and self-update without any headache caused on your part.

I’ve had reports of it for many releases now – data is lost from the totals table, for no apparent reason. I’ve seen it in the search terms people use to find my site and the kStats plugin, and I’ve had bug reports and numerous comments about it. Until now, I’ve never seen it happen and therefore had no basis to start my search in fixing it.

Well, after an entire week of testing 0.6.0 every day on my own blog to try and have kStats first major bug free release, I thought I was good to go. So I tagged it and dropped it in the repository earlier this evening, thinking “Finally!”.

Not an hour later, it happened.

My data was gone. All except for yesterday and the new hits coming in after midnight. Nothing was missing from the raw table or charts, something happened to truncate the totals (aggregate) information. The problem is, I still have no better clue what it was. It happened right after the wp-cron event for kStats was tripped, so I presume it has to be occurring during the cleanup routine. So I went in and debugged every query that was going on.

Every single query ran fine. I tripped the cron 7 times. Each and every time worked perfectly. So what the HECK is going on?

I’ve been banging my head against the walls and my dining room table for a few hours now trying to track it down. What good is a stats plugin that arbitrarily truncates data?! The only possibility I’ve come up with so far that seems remotely feasible is that it happens when a search engine or bot trips the cleanup – maybe they’re hitting a page and leaving so fast that it’s only getting part way into the process before it gets canceled?

So I just finished tagging 0.6.1, hopefully before too many people have already upped to 0.6.0. I added (and probably should have since day one anyways) ignore_user_abort() to the primary cleanup function to ensure that even if the page is exited before the script is done, it will follow through. I had this in the collector, I’m not sure why I didn’t put it there before.

If that doesn’t fix the problem, I’m going to turn the cron feature off and make it a utility on the Options page until the reason is tracked down. I would like to release a stable version. I can’t until this is worked out.

*sigh*

I have to stop making changes to the user interface. Every time I’m about ready to tag a new release of kStats, I finish checking in the last of the files to Trunk, and just as I start typing svn cp trunk tags/x.x.x… Oops, forgot the new screenshots.

Would it be funny if I just left the screenshots from 0.1.0 up forever?

The path to stable

So far I’ve been managing to hit all of my goals, even surpassing a few during the beta period of kStats. I fubbed up the nightly aggregates for the first little bit, until I realized half my problem was my attempt to reinvent my own wheel. The other half fell under the same umbrella, I was just making things complicated that could have been simplified. Once I crested that hill, things starting falling into place. Read the rest of this entry »

I doubt any of you script kiddiez even look at the sites you attempt to hack with your little botnet and RFI scripts, but in case you do, I just have one question for the lot of you;

Does your grandma know what you’re doing in her rent-free basement?

I’m sorry, I realize most of you probably live at home with mommy and daddy, and I shouldn’t be bringing your grandmother’s into this, but seriously. Half of you tried to attack my site with a phpBB vulnerability. I’m sorry, last time I checked, I was running Wordpress.

Put down the porn, sign off your favorite MMORPG, change your clothes and shower for once in your life and walk to the front of your house. There’s a big rectangle there with a little round knobby thing on it. This is called the front door. If you open it, there’s a life somewhere out there for you. Go find it.

You’re not getting root on my box, trust me. I’m sure a skilled hacker could if they put enough effort into it, but you’re not.

I don’t know who amongst my readers might have the skills and the time, but if you want to slap some children around with a big trout, here’s a few IPs from my logs;

61.63.10.150
64.6.232.171
194.105.193.46
209.216.213.119

Have a nice day!