Hitting the maximum simultaneously open files at the stdio

We are working on new version of axLog that will sport some rather important changes in recovery model. Probably the biggest one is switching from single recovery file to multiple recovery files (one per table) We have a customer with immediate need for this so I sent them a modified version that was opening one file per recovery table. And it worked fine until it would suddenly fail to open a file with well formed file name, with no limitations on the directory and with enough free space on the drive. After taking several shots at solving the issue in my own code ("when you see hoofs think horses, not zebras") I finally had a moment of clarity to realize that this could be a library limitation (Log's engine is implemented in Visual C++) on the number of simultaneously open files - so I tested for it. And sure enough by default I couldn't create more than 508 file streams. I checked the results using CreateFile and that of course worked fine for any number of files that I threw at it. I started searching on the net for similar problems and soon found the following function _setmaxstdio from Microsoft Run-Time Library. As you can see this function allows setting "new maximum for the number of simultaneously open files at the stdio level." Great... except that the hard upper limit is 2,048 files and there is no going beyond that. We lucked out with this particular customer as they had to recovery a bit above 1,700 tables. But to solve this generically we had to come up with something else - more on that in next post.

Btw, since this is Microsoft Visual C++ Run-Time library specific limitation, if you use a development tool that uses STDIO from MSVCRT internally - you have this same limitation as well.