Showing posts with label SQL Server. Show all posts
Showing posts with label SQL Server. Show all posts

Wednesday, January 23, 2008

CAM world *intersecting* SQL Server world

I never thought that my current area of work (in general auditing and recovery of SQL Server databases) would somehow *intersect* (pun intended - see below) with my previous efforts in 2D CAM (Computer Aided Manufacturing.) But SQL Server 2008: Spatial indexes article by Paul Randal proved me wrong. In the article Paul explains how SQL Server 2008 spatial indexes work but what was most interesting to me was that the solution he described matches a solution I applied in a 2.5D CAM application some 10 or more years ago. Don't get me wrong, that doesn't make me rocket scientist since and anyone who had to work with intensive 2D geometry calculations sooner or later figures out that calculating intersection of curves is expensive (sometimes more, sometimes less: depends on the curves) so avoiding it in advance by using "bounding boxes" is, in general case, much better than blindly trying to calculate every intersection. I remember that one great advantage in my case was that we were able to use comparison of integers instead of floating points which further sped up the algorithm. Also, in our case there was just one "bounding box" per curve which was more coarse but didn't use that much memory and was easily calculated.

The 2.5D CAM application that I refer to above was called "Impakt!" It was used to generate G-code programs for 2.5D CNC (Computer Numerical Control) mills. Half of D means that mill could only position itself in height without really orienting the tool in three dimensions. Thus the geometrical problems were reduced to creating optimal 2D cutting paths and then additionally repositioning the tool between different 2D depth levels.

We started development of Impakt! in Delphi 2.0 sometime in 1996. Impakt! worked like this:
1. You would load 2D geometry from a file in Autodesk's DXF format.
2. You would define depth of each contour by choosing each area and then setting the depth (this could then be saved in our own file format that kept depth and other information)
3. You would define your tool set for the job (we kept a DB of tools and users were able to define their own tools)
4. Application would then generate tool paths by going through the following process:
4.1 Take the tool with the largest diameter in the tool set and start with contours on the highest level of the defined geometry.
4.2 Analyze the contours and identify "lakes" (outward contours) and "islands" (inner contours)
4.3 Grow "islands" outward and shrink lakes inward by the radius of the current tool.
4.4 Solve intersections of the grown/shrank contours thus defining tool path. This really solves one particular path through Voronoi diagram of the 2D contours available on the given depth level (I can't remember why we didn't just calculate Voronoi diagrams and then generate paths... I think that there were cases we couldn't or didn't know how to handle with Voronoi diagrams and I don't think we even had math for it, at first anyway)
4.5 Continue with the process in 4.2 and 4.3 until all contours would fold on themselves.
4.6 Take next smaller tool in tool set, take left over contours from the last successful iteration of 4.5 and then start again at 4.3.
4.7 Once there are no more available tools in the tool set go to the next depth as defined by geometry and start again at 4.2.
4.8 Once you have generated geometry for all depth levels you are finished.
5. You would then define cutting parameters for the job (this only influenced G-code generation)
6. G-code (or some other type of CNC code depending on loaded modules) would be generated from tool paths and cutting parameters. Unlike geometry generation where paths for all tools were calculated at once for each depth level, G-code was generated by following the paths of the largest tool on all depth levels, then changing the tool to the next smaller one and following all its paths on all depth levels and so on until the smallest tool in the tool set.
7. If I remember correctly we had a separate tool for transferring G-code (through RS-232!) to CNC machines.

Heh, more and more stuff is starting to crawl out from the back of my brain. I just remembered that once a contour was identified as an "island" or a "lake" we would reorient it if necessary so that it would go in counter-clock-wise direction if it was an "island" or in clock-wise direction if it was a "lake" (or vice-versa, it doesn't really matter so long as they had different orientations.) This allowed us to grow/shrink the contours with the same parameters ignoring the orientation as they would then "naturally" either grow or shrink depending on the orientation we gave it.

I hope that someday we will make open code source for "Impakt!" I would probably be rather embarrassed with the quality of the code but maybe somebody could find some use for it.

Monday, April 30, 2007

ApexSQL Log 2005.04, part II: Smoother user experience

This is the 2nd part of my "ApexSQL Log 2005.04" series which I started here. In the first one I blogged about some of the most important new features and fixes, this time I'm going to mention most important things we did to improve user experience (especially for new users)

Integrated client-side and server-side setups into one setup

Starting with version 2005.03 we have provided a standalone server-side setup for our customers. However, this hasn't been convenient enough so we went one step further and we will now provide one unified setup for client and server-side components.

There are now three setup options:
1. Client application (GUI and CLI) and server-side components on a local server
2. Client application (GUI and CLI)
3. Server-side components on a local server

The only thing worth noting here is that setup can install server-side components only on a local server (this includes virtual servers in failover cluster) and not on a remote server.


Problems with SQL Server run by non-administrator account

In previous versions of the software our server-side components needed high level of privileges which of course led to problems in environments with restricted privileges of the account running SQL Server service. In version 2005.04 we took this problem head on and have significantly lowered the level of privileges needed for server-side components. In the process we have also solved another of problems springing from inability of the account running SQL Server to back-connect to the server itself. There is one known issue left here: Connection Monitor still needs login privileges for the account. In future versions we will allow manual configuration of Connection Monitor connection parameters.

We also had problems with logging on server-side components when lacking privileges. In previous versions server-side logs were stored in System32 (or SysWOW64) folder but in some configurations the account running SQL Server service lacks privileges for writing into system folders. Version 2005.04 stores all server-side logs into "LOG" subdirectory of SQL Server (which, in retrospect, is the ideal place for log files!)

For the record, I think that restricting privileges to SQL Server service (and other services) to bare minimum is a great security practice and one that we certainly try to encourage.

Problems with bad @@SERVERNAME

ApexSQL Log uses @@SERVERNAME to identify server's real name in some situations. But this leads to problems if you change server's network name after SQL Server was already installed. When that happens @@SERVERNAME continues to return the old server name even after the service restart so ApexSQL Log doesn't have access to real machine name which in turn leads to all sorts of problems. This is surprisingly (for me) common situation and we often had to help users fix @@SERVERNAME values. Starting with version 2005.04 this has been fixed. We still use @@SERVERNAME for some internal stuff but connection is now always done through server name as input by user. This also solved the problem with accessing servers available only through IP address or listening on a port other than 1433.

We also solved the same problem but on server-side with Connection Monitor. Connection Monitor has to back connect to SQL Server and used to obtain its server name from @@SERVERNAME. It doesn't any more - it now always connects to 127.0.0.1 with instance name if available retrieved directly from command line that ran SQL Server.

In case you are wondering, to fix @@SERVERNAME so that it returns the correct machine name you can do the following:
1. Execute the following script

sp_dropserver ''
go
sp_addserver '', local
go

2. Restart SQL Server service

Saturday, April 28, 2007

ApexSQL Log 2005.04, part I

As I announced back in March ApexSQL Log 2005.04 is now in QA. It took more time that I thought back then but for a good reason: we went back to our keyboards and did more damage on new issues reported by some customers - especially scaling for very large transaction log files (50 Gb and greater.) In any case 2005.04 version is in QA right now, it's looking great and I hope it will be out soon. In the meantime I'm going to do a small series of posts on improvements that 2005.04 brings starting right now.

Reconstruction of UPDATE operations

Here's the problem of UPDATE reconstructions in a nutshell:
1. In general case, when logging an UPDATE statement, SQL Server just logs what was changed and into what.
2. These before/after state can correspond to everything from sub-field parts to cross-field parts of a row.

So from 1 and 2 comes the following definition of the problem:

To reconstruct fully what happened on a field level in an UPDATE statement one needs to know the state of the row in which the UPDATE statement occurred.

The difficult part of UPDATE reconstruction is finding that original state of the row. We have greatly improved this in version 2005.04 and I will blog more on this later on together with 2005.03/2005.04 examples.

Memory footprint and performance scaling

Storage is getting cheaper, processing power abounds, bandwidths are improving and this all leads to larger databases and higher transactions counts. Both of these increments - larger databases lead to larger MDF files which we use in our recovery process and more transactions lead to larger transaction logs - are beginning to weight on our "last year's" technology. Hence we felt it necessary to redesign the way memory is used by the application in order to allow greater scaling. We also wanted to improve the user experience with the application playing nicely with system resources (well, with memory and disk space at least - as with most other applications we want as much CPU and I/O as we can get.) In this kind of software there is a constant tension between just how much memory we should use (so that we don't have to re-read a lot - which slows things down of course) vs. how does our memory and I/O usage affect the ability of the system on which it is running and its own ability to successfully finish auditing. All of these things (better scaling in memory and performance, playing nicely with the rest of the applications) we have improved greatly in 2005.04. I plan to blog in detail on this soon.

Support for transaction log backups converted from 3rd party backups

Use of 3rd party backup tools is getting more frequent and some of the people using 3rd party backups are also our customers (or want to become one.) However, there is a problem with some 3rd party backups and their converters to MTF (Microsoft Tape Format) files as the converted MTF files do not always match files that SQL Server would have produced. This used to confuse ApexSQL Log but starting with version 2005.04 the application handles correctly these inconsistencies and will now read backups that doesn't perfectly match SQL Server's backups.

Support for reading transaction logs of SQL Server 7/2000 under SQL Server 2005 and vice versa

Format of transaction log changed significantly between SQL Server 2000 and 2005. In ApexSQL Log versions prior to 2005.04 it was not possible to read SQL Server 7/2000 transaction logs when connecting to SQL Server 2005 server or vice versa. However, as migration toward SQL Server 2005 is accelerating (which I believe from anecdotal evidence) there is more and more need for auditing of old transaction logs on newly migrated servers. This situation happens in two instances:
1. When migration is done by detaching the db files from old SQL Server and then attaching them to SQL Server 2005. In this case old transactions are still in the transaction log file but in the format of the previous version of SQL Server.
2. When the need arises to audit old transaction log backups (or detached transaction log files) and the new server is all that we have left available.
Starting with SQL Server 2005.04 we handle both of these cases seamlessly. We also handle reading of SQL Server 2005 transaction logs on SQL Server 7/2000 - just in case anyone ever needs that.

Auditing progress in GUI

With the current version of the software users can't really tell how much more will they have to wait before the results come in. This is , especially for large data sets that we are processing, and I'm sorry we never got around to fixing this prior to 2005.04. But the good news is - it's fixed and I think that audit progress bar is now informative and helpful. The progress is split into two parts:
1. First 50% of the progress are dedicated to initial processing of the transaction log sources we are auditing. Here's a typical shot:


2. Second 50% of the progress are dedicated to filtering of transaction log sources according to the parameters set by the user. However, even when the there is no progress in the number of matching entries, the time in log that is currently being analyzed is shown. Here's a typical shot:

This makes it easy to understand what's going on, where the application is and just how much more (approximately) there is to go.

I think that's it for today. But this isn't all - I'll blog more on 2005.04 soon.

Friday, April 27, 2007

What's the difference between database version and database compatibility level?

Paul Randal, over at SQL Server Storage Engine blog, discusses in his post the difference between database version and database compatibility level. When working with ApexSQL Log this difference is important since any db on SQL Server 2005, even with compatibility level of 70 (SQL Server 7) or 80 (SQL Server 2000), still has the same structure of system tables as a db with the level of 90 (SQL Server 2005). This matters in three cases:
1. When we are doing DDL analysis/recovery since we have to look for changes in different tables under SQL Server 7/2000 and SQL Server 2005 due to complete redesign of system tables in the latter version.
2. When users are directly auditing changes made to system tables (which is sometimes necessary.)
3. When auditing transaction log backups from one version of SQL Server on another. In ApexSQL Log 2005.03 transaction logs from SQL Server 7/2000 could not be read at SQL Server 2005 or vice versa. With the upcoming 2005.04 version this will be done seamlessly.

Of all of these cases only the 3rd is really problematic since we depend on SQL Server to provide us with meta-data for all the tables including system tables. So when we see an operation on say "sysobjects" table from a SQL Server 2000 transaction log attached to SQL Server 2005, we aren't able to reconstruct it since SQL Server 2005 lacks meta-data for "sysobjects" table. This case is very rare but even so we will try to solve it after 2005.04 by building in meta-data for system tables of all three versions of SQL Server that we support.

Btw, in case anyone is wondering how you can audit SQL Server 2005 transaction logs on SQL Server 7/2000 database (considering that it cannot be attached), it can be done by auditing transaction log backups or detached transaction logs.

Wednesday, March 28, 2007

Turn AUTO_SHRINK off!!

Paul Randal (Principal Lead Program Manager, SQL Storage Engine) over at SQL Server Storage Engine blog, has posted today on why everybody should turn AUTO_SHRINK off for their production dbs. He enumerates three reasons why AUTO_SHRINK should be turned off. He's an authority on the SQL Server Storage Engine so I would heed what he says.

I have however one more reason to add: in case of catastrophic data loss (either through db corruption, DELETE without WHERE, DROP TABLE or TRUNCATE TABLE) from which you can't recover by restoring a backup (either because you don't have it or you don't have up to date transaction log backups or whatever), you really really *really* don't want SQL Server going in and shrinking the database files before you had a chance to recover the data. What you want to do instead is:
1. Put database in read-only mode immediately so that it's left in the state as close as possible to the state it was in in the moment of the data loss (if you experienced a hardware failure or something such your db files are now detached - just leave them like that for now)
2. Download ApexSQL Log and install it. If you need to analyze a database that's online install ApexSQL Log's server-side components on the server. It doesn't matter if you have or don't have transaction logs for the database - the application will try to recover data from what you have (besides, transaction logs can help only with the recovery of delete data)
3. Run ApexSQL Log's Recovery Wizard and chose the recovery option most adequate for the scenario. Recovery Wizard will recover all the data (including BLOB) it can still find in the database and create a recovery script.
4. Run recovery script on another database to check the data. If everything is fine - great! If there's a problem or you think that the software should have recovered more data, please contact us at support@apexsql.com and we will help you out.

Friday, December 29, 2006

ApexSQL Log 2005.03 released

Today we have released ApexSQL Log 2005.03. It's been six months since the last release and there are several major features that we packed in 2005.03, of which the most important, and definitely the most anticipated by our users, is full x64 support. This includes support for SQL Server 2005 x64 and for 32-bit SQL Server 7/2000/2005 running on Windows x64 operating systems (including online transaction log reading). Beside this we have added BLOB support to our Recovery Wizard (only when recovering from MDF files though, not from transaction log files) and we have greatly improved performance of MDF recovery. There is a host of other smaller features accompanied with another host of really important bug fixes.

I almost forgot that we have also created a separate installer for server-side components. This is now our recommended way to install server-side components since it allows uninstallation directly from Add/Remove Programs. Of course, remote installation/uninstallation from the client is still supported in GUI and CLI.

So, if you are interested in top-of-the-line transaction log reader (reading online and detached transaction logs and transaction log backups) and recovery tool (allowing recovery of deleted, dropped, truncated and even corrupted data and database objects from transaction log files or database files) for SQL Server 7, 2000 and 2005 (as of 12/29/2006 the only such tool on the market to support SQL Server 2005) that works equally well on your 32-bit and 64-bit SQL Servers, you may want to download here a 30 day evaluation version of ApexSQL Log 2005.03 and give it a go. While you are at it, maybe you will also want to check out our Universal Studio, packing all 9 of our tools with 1 year of full support and 1 year of major upgrades for just USD 1,999.

The full list of features and fixes of ApexSQL Log 2005.03 follows.

----------------------------------------------------
RELEASE 2005.03.0417
DATE: 29 December 2006
DESCRIPTION: Major Enhancement/Fix release
----------------------------------------------------

Enhancements:
- Added support for Windows 64-bit and SQL Server 2005 64-bit for x64 CPUs (MAJOR ENHANCEMENT)
- Added BLOB support in recovery of deleted rows and rows in truncated and dropped tables when using database sources for recovery (online database files and MDF files) (MAJOR ENHANCEMENT)
- Significantly improved performance of data recovery from MDF files, online or detached (MAJOR ENHANCEMENT)
- Added support for online transaction log and database files on drives mounted as NTFS folders.
- Added support for in-row values for BLOB types.
- Added SERVER_NAME column to Connection Monitor table allowing for filtering by the name of the server where transaction log was created.
- Added xp_ApexSqlLogMonitor_Enable and xp_ApexSqlLogMonitor_Disable extended procedures.
- In generation of UNDO/REDO statements:
- Added SET IDENTITY_INSERT ON/OFF for INSERT operations.
- Added transaction wrapping for DELETE operations.
- Changed preference for construction of WHERE statments to primary key over clustered index.
- For heap tables, added generation of WHERE clause from values of all non-BLOB fields.
- Added skipping of BLOB and TIMESTAMP fields.
- In recovery from detached MDF files, added loading of meta-data from a live database when such is unaivalable in MDF files.
- For recovery on SQL Server 2005, modified algorhitms to improve chances of success in suboptimal recovery scenarios.
- Added SET NOCOUNT ON to recovery scripts.

Changes:
- Last used filter now is stored per server and database.
- Added server name to login information captured by Connection Monitor.

Fixes:
- A problem with conversion of _variant_t to __int64 (MAJOR FIX)
- A problem with memory over-consumption (MAJOR FIX)
- A problem with activation not taking immediate effect on the server side (MAJOR FIX)
- A problem with old ID mapping feature (MAJOR FIX)
- A problem with multiple value switches in CLI (MAJOR FIX)
- Parsing of table names in command line interface (table names can now be specified with within angular brackets)
- A problem with NULL values in WHERE statements.
- A problem with UNICODE table names and table names with quotes in CLI and Recovery Wizard.
- A problem with Connection Monitor with automatic restart.
- A problem with Connection Monitor sometimes failing to flush rows from temporary login table.
- A rare problem with table names on case-sensitive databases.
- A problem with user ID showing as N/A even when it's known.
- A problem with refreshing of transaction log operations grid.
- A problem with NULL/NOT NULL flag in table DDL scripts.
- A rare problem with transaction log reading.
- A rare unhandled NULL exception.
- A rare problem of mistaken table ID for dropped tables on SQL Server 2005.
- An exception in About box when not connected to any server.
- Several errors in captions and messages.
- Several icons in the Log Filter dialog.