Moderating comments

I just deleted the first comment spam to appear on this blog. Since I have no plans on wasting much time on moderating the comments I have decided to change the comments policy in two ways:
1. I have enabled word verification option so posting will require entering CAPTCHA displayed by the blog site. This will raise the bar on ease of commenting but I hope it will be acceptable.
2. I have enabled posting comments by non-members of blogspot. This is intended to balance the raised bar on ease of commenting due to the 1st policy change.

On the move...

On Sunday I moved to new house. It's my first house and I'm very happy for it. But only today my ISP connected me back to Internet so I'm finally back working at home! You should have seen me hunting down some very weak WiFi signal the other day at 5AM.... but on the second thought it's probably better that no one saw me!

Eric Sink on coding addiction

Hi, my name is Ivan, and I'm a coding addict.

Eric's post is really about WPF but he starts with admiting his addiction so I thought I do the same. I have to admit that I feel ill when I don't write enough code during a day. Couple of days like that and I'm ready to go nuts. I remember after my 8th grade (I was 14 years old at the time) I went to a vacations with my family and each day there I wrote Zilog Z80 assembler programs on paper (and of course I read loads of SF) The absolutely first thing I did when we came back home was to turn on my faithful ZX Spectrum and type it all in. I think it was an intro (as demos were called back then) that did some cool image manipulation (or it seemed cool at the time :) IIRC it translated correcly and worked right from the start but I had to tweak it some more to get an even cooler effect (or it seemed cooler at the time :)

Cryptonomicon 2 and 3?!

Yesterday a friend pointed out to me that Cryptonomicon 2 and 3 were available on Amazon. Since Cryptonomicon is one of my favorites books this left me with my mouth watering almost literaly.... until I realized that this is really a publishing strategy imployed by the publisher of Cryptonomicon in Spanish language: splitting one book into three and selling them separately. I heard about this before but didn't remember it immediately. It did however remind me on just how hard it was to come by SF and other reading material (say books on programming?) before the advent of Amazon.com and credit cards (I'm talking about situation in then Yugoslavia in the '80s and '90s) Most of people used to totally depend on the few and sorry translations of local publishers (I was one of the lucky ones - but more on that in another post) From what I hear situation is much better these days exactly because the local publishers these days have to compete with Amazon.com and the likes.

Hitting the maximum simultaneously open files at the stdio - Part II

As you may have read in the I part of this post in certain cases we have hit a limit of maximum simultaneously open files in Visual C++ stdio library. Since axLog engine is implemented in Visual C++ and uses iostream library to abstract IO this means that we had to raise this limit somehow and _setmaxstdio helped there. However, the hard upper limit of open files is 2,048 and since during recovery we are opening one file per table, recovery for databases with more than 2,048 tables would fail (actually less than that since there are always other open files like standard streams but you get my meaning). The solution was to use Win32 API for reading files and thus deal with OS and not library limitations. In both Windows 2000 and Windows XP the default number of open user handles per process is set to 10,000 which seemed a much more reasonable limit. And if this limit is ever reached it can be changed as described here and here (as both articles warn this should not be done lightly)

During implementation phase the most important thing to us was keeping IO stream library abstraction intact so that we didn't have to change anything elsewhere in the code (which uses throughout IO stream library interfaces) This left us with only once choice: implementing IO streams classes for files using Win32 API file managment functions. The clearest inspiration came from the handle_stream library that I found researching previous art. We couldn't use this library directly since it didn't support wide characters, it wasn't working correctly in VC++ 7.1 (VS.NET 2003) and its license status was not clear (it seems public domain but we couldn't confirm it) But it was enough to understand that what we really wanted to do is implement IO streams abstraction based on Window's handle concept. This allowed us to move opening of files to a higher level (conceptually) and not complicate further the interface of our IO stream classes (e.g. with all parameters for CreateFile) I will post implementation details in the next post.

The Declaration of Independence

My warmest regards and best wishes for their Republic to my US friends and colleagues. I'm neither US citizen nor resident but I'm deeply grateful to be living in the world where political ideals set in the Declaration are prevaling. To the gentlemen that wrote and ratified this document and pledged their Lives, Fortunes and sacred Honour goes my eternal gratitude and respect.

Hitting the maximum simultaneously open files at the stdio

We are working on new version of axLog that will sport some rather important changes in recovery model. Probably the biggest one is switching from single recovery file to multiple recovery files (one per table) We have a customer with immediate need for this so I sent them a modified version that was opening one file per recovery table. And it worked fine until it would suddenly fail to open a file with well formed file name, with no limitations on the directory and with enough free space on the drive. After taking several shots at solving the issue in my own code ("when you see hoofs think horses, not zebras") I finally had a moment of clarity to realize that this could be a library limitation (Log's engine is implemented in Visual C++) on the number of simultaneously open files - so I tested for it. And sure enough by default I couldn't create more than 508 file streams. I checked the results using CreateFile and that of course worked fine for any number of files that I threw at it. I started searching on the net for similar problems and soon found the following function _setmaxstdio from Microsoft Run-Time Library. As you can see this function allows setting "new maximum for the number of simultaneously open files at the stdio level." Great... except that the hard upper limit is 2,048 files and there is no going beyond that. We lucked out with this particular customer as they had to recovery a bit above 1,700 tables. But to solve this generically we had to come up with something else - more on that in next post.

Btw, since this is Microsoft Visual C++ Run-Time library specific limitation, if you use a development tool that uses STDIO from MSVCRT internally - you have this same limitation as well.

Mounting CD-ROM images in Windows XP

If you ever need to mount a CD-ROM image as a virtual drive instead of burning it, try this unsupported Microsoft utility. The utility supports ISO and several other formats and works great. You can find a short description on how to use it here.

You can also check DAEMON Tools. I haven't tested it but seems to support more formats, works with DVDs and even works on Windows 2000. It's freeware for personal use.

Unix command line utilities for Windows

I'm not a very experienced UNIX developer by any stretch of imagination but I enormously appreciate its command line utilities and just how much they make my life easier. I grew up using DOS command line (and before that ZX Spectrum's line editor!) and to this day the task I most commonly run (Win+R) is "cmd". On top of that, the very first thing I install on a new box is "Open Command Window Here" Explorer context menu option! So even though I got exposed to UNIX relatively late in my career (so far) I quickly learned to love its tools (except vi!) Since I've been exposed to them I had to start using UNIX command line tools on Windows as well. The port I use is called UnxUtils and can be found here. It's not a full port but it works for me. Actually seems to be a set of ports of GNU utilities and as we all know "GNU's Not Unix" but whatever - it works great for me.

Btw, I'm looking forward very much to Windows PowerShell (The Shell Formerly Known As Monad) A very very cool piece of technology (with now a very lame name but that's a lack of geekiness in MS marketing divisions for you) Microsoft is finally getting its game up for command line. I just wish this was released already!

"The Power of the Marginal" by Paul Graham

A new essay by Paul Graham titled "The Power of the Marginal" is up (you can find it here) As always Paul is a joy to read. Even when I don't agree with him (usually I do and very much so) his esseys never fail to make me think through the issues he's writing about.

New info on upcoming version of ApexSQL Log

I have just left a small post on over on our corporate blog about three major features that will be available in the upcoming version of ApexSQL Log (a.k.a. the-only-SQL-Server-2005-transaction-log-reading-tool amongst those in the known)

ApexSQL Log 2005.02 has been released

We have posted the new version of axLog - 2005.02. Here are the release notes for it:

----------------------------------------------------
RELEASE  2005.02
DATE:  26 June 2006
DESCRIPTION: Major Enhancement/Fix release
----------------------------------------------------

Enhancements:

- Added "User Id" and "Login Time" columns to grid.
 "User Id" column shows either UID or SID depending on
 SQL Server version.
 "Login Time" column shows the time of the login of the
 connection on which the transaction was made.
- Added selection of available transaction log backups.
- Added selection of heterogenous log sources to GUI.
- Added in Row Details icons indicating columns that are
 part of table's clustered index or primary key.
- Added difference viewer for UPDATE operations.
- Added internal viewer of column values for INSERT and
 DELETE operations.
- Added column level regular expressions filter for tables
 in log operations grid.
- Added support for user defined types (non-CLR).
- Added milliseconds to to/from filters in CLI.

Changes:

- Limited the number of rows recovered by Recovery Wizard
 during product evaluation.

Fixes:

- A problem with user filtering (MAJOR FIX)
- A major performance issue (MAJOR FIX)
- A problem with CLR user defined types (MAJOR FIX)
- A problem with IS NULL row filter when used from GUI.
- A problem with reading of aborted transactions.
- A problem with showing application tab in task bar.
- A problem with opening new filtered log.
- A rare problem with remote installation failing on
 local server.
- A rare problem with UPDATE reconstruction.
- A problem with remote online transaction log auditing
 with files over 2Gb in size.
- A rare problem with server side logs not being found
 in system32 folder.
- An unhandled exception thrown by Field Values Filter
 with no continuation operator.
- A problem with BLOB fields appearing in UNDO/REDO
 statements.
- A problem with REDO/UNDO not taking into account the
 primary key on tables with no clustered index.
- A rare problem with row reconstruction.
- A rare problem with dropped table recovery when
 recovering from online database data.
- A problem with adding servers with expired trial version
 to the server tree.
- A problem with copying from REDO/UNDO tabs to clipboard.
- A problem with closing log window.
- Several minor GUI issues.

This release was all about fixing any problems that were reported in the last 3 and so months since the relase of 2005.01 but we also added some enhancements. Two biggest enhancements are User ID and Login Time columns in the audit grid and support of heterogenous transaction log sources in GUI (it was already suppored in CLI) The next major release will be all about new features and improvements. I can tell you that this next version has been in development for some time now and should be out in Q3. I will blog more on upcoming features soon - there is a lot of stuff already done and much more in the works.

Recoverying data from large dbs

As you can see from this forum thread some of ApexSQL Log (aka axLog) users are having performance problems with recovery of large dbs from detached (and in this case seemingly corrupted) MDF files. In this case it's a db with 220gb of data split in 4 files. In the current version of axLog (2005.02, which is comming out tomorrow) recovery process makes one pass through MDF files per recovered table. On the other hand we don't offer selection of tables to be recovered. Put these two things together and you get performance problems on very large dbs. We have been aware of this for a while so in next major version we will offer:

1. Selection of tables to recover.
2. Output of recovered data for each table into separate files.

This will allow users to select only what they need to recover and have it recovered in only one pass. We will offer this for recovery from detached MDF files and for the upcoming recovery from database backups.

SQL Server Storage Engine blog

Folks from SQL Server Engine Team have been blogging since recently. So far mosts posts are by Paul Randal, Lead Program Manager. The content is really top notch and in these 3 weeks since they started blogging I can honestly say that it has become a required reading for those of us that are interested in internal workings of SQL Server.

Output to console from .NET GUI apps

I have posted on ApexSQL's blog an article on output to console from .NET GUI apps.