Hitting the maximum simultaneously open files at the stdio - Part II

As you may have read in the I part of this post in certain cases we have hit a limit of maximum simultaneously open files in Visual C++ stdio library. Since axLog engine is implemented in Visual C++ and uses iostream library to abstract IO this means that we had to raise this limit somehow and _setmaxstdio helped there. However, the hard upper limit of open files is 2,048 and since during recovery we are opening one file per table, recovery for databases with more than 2,048 tables would fail (actually less than that since there are always other open files like standard streams but you get my meaning). The solution was to use Win32 API for reading files and thus deal with OS and not library limitations. In both Windows 2000 and Windows XP the default number of open user handles per process is set to 10,000 which seemed a much more reasonable limit. And if this limit is ever reached it can be changed as described here and here (as both articles warn this should not be done lightly)

During implementation phase the most important thing to us was keeping IO stream library abstraction intact so that we didn't have to change anything elsewhere in the code (which uses throughout IO stream library interfaces) This left us with only once choice: implementing IO streams classes for files using Win32 API file managment functions. The clearest inspiration came from the handle_stream library that I found researching previous art. We couldn't use this library directly since it didn't support wide characters, it wasn't working correctly in VC++ 7.1 (VS.NET 2003) and its license status was not clear (it seems public domain but we couldn't confirm it) But it was enough to understand that what we really wanted to do is implement IO streams abstraction based on Window's handle concept. This allowed us to move opening of files to a higher level (conceptually) and not complicate further the interface of our IO stream classes (e.g. with all parameters for CreateFile) I will post implementation details in the next post.