ApexSQL Log 2005.04, part I

As I announced back in March ApexSQL Log 2005.04 is now in QA. It took more time that I thought back then but for a good reason: we went back to our keyboards and did more damage on new issues reported by some customers - especially scaling for very large transaction log files (50 Gb and greater.) In any case 2005.04 version is in QA right now, it's looking great and I hope it will be out soon. In the meantime I'm going to do a small series of posts on improvements that 2005.04 brings starting right now.

Reconstruction of UPDATE operations

Here's the problem of UPDATE reconstructions in a nutshell:
1. In general case, when logging an UPDATE statement, SQL Server just logs what was changed and into what.
2. These before/after state can correspond to everything from sub-field parts to cross-field parts of a row.

So from 1 and 2 comes the following definition of the problem:

To reconstruct fully what happened on a field level in an UPDATE statement one needs to know the state of the row in which the UPDATE statement occurred.

The difficult part of UPDATE reconstruction is finding that original state of the row. We have greatly improved this in version 2005.04 and I will blog more on this later on together with 2005.03/2005.04 examples.

Memory footprint and performance scaling

Storage is getting cheaper, processing power abounds, bandwidths are improving and this all leads to larger databases and higher transactions counts. Both of these increments - larger databases lead to larger MDF files which we use in our recovery process and more transactions lead to larger transaction logs - are beginning to weight on our "last year's" technology. Hence we felt it necessary to redesign the way memory is used by the application in order to allow greater scaling. We also wanted to improve the user experience with the application playing nicely with system resources (well, with memory and disk space at least - as with most other applications we want as much CPU and I/O as we can get.) In this kind of software there is a constant tension between just how much memory we should use (so that we don't have to re-read a lot - which slows things down of course) vs. how does our memory and I/O usage affect the ability of the system on which it is running and its own ability to successfully finish auditing. All of these things (better scaling in memory and performance, playing nicely with the rest of the applications) we have improved greatly in 2005.04. I plan to blog in detail on this soon.

Support for transaction log backups converted from 3rd party backups

Use of 3rd party backup tools is getting more frequent and some of the people using 3rd party backups are also our customers (or want to become one.) However, there is a problem with some 3rd party backups and their converters to MTF (Microsoft Tape Format) files as the converted MTF files do not always match files that SQL Server would have produced. This used to confuse ApexSQL Log but starting with version 2005.04 the application handles correctly these inconsistencies and will now read backups that doesn't perfectly match SQL Server's backups.

Support for reading transaction logs of SQL Server 7/2000 under SQL Server 2005 and vice versa

Format of transaction log changed significantly between SQL Server 2000 and 2005. In ApexSQL Log versions prior to 2005.04 it was not possible to read SQL Server 7/2000 transaction logs when connecting to SQL Server 2005 server or vice versa. However, as migration toward SQL Server 2005 is accelerating (which I believe from anecdotal evidence) there is more and more need for auditing of old transaction logs on newly migrated servers. This situation happens in two instances:
1. When migration is done by detaching the db files from old SQL Server and then attaching them to SQL Server 2005. In this case old transactions are still in the transaction log file but in the format of the previous version of SQL Server.
2. When the need arises to audit old transaction log backups (or detached transaction log files) and the new server is all that we have left available.
Starting with SQL Server 2005.04 we handle both of these cases seamlessly. We also handle reading of SQL Server 2005 transaction logs on SQL Server 7/2000 - just in case anyone ever needs that.

Auditing progress in GUI

With the current version of the software users can't really tell how much more will they have to wait before the results come in. This is , especially for large data sets that we are processing, and I'm sorry we never got around to fixing this prior to 2005.04. But the good news is - it's fixed and I think that audit progress bar is now informative and helpful. The progress is split into two parts:
1. First 50% of the progress are dedicated to initial processing of the transaction log sources we are auditing. Here's a typical shot:


2. Second 50% of the progress are dedicated to filtering of transaction log sources according to the parameters set by the user. However, even when the there is no progress in the number of matching entries, the time in log that is currently being analyzed is shown. Here's a typical shot:

This makes it easy to understand what's going on, where the application is and just how much more (approximately) there is to go.

I think that's it for today. But this isn't all - I'll blog more on 2005.04 soon.