Question : Sybase 12.5: Compare Table Load Performace between Dev and Prod box

Hi Experts,

I have job that moves data from DB2 to Sybase (4mil records),using Informatica ETL .  On Dev box it takes 23 min (drop indexes, load, create indexes).
On Prod box it takes around 2-3 hours. DBAs are saying because of SRDF disks prod will always be slower , but why performance is SOO much different? Table has ~ 10 columns  and 2indexes , that are recreated during job run . Creating indexes takes ~ from 20 to 50 min on Prod box . Why ?

Appreciate your help
Julia

Answer : Sybase 12.5: Compare Table Load Performace between Dev and Prod box

Something is very wrong; the problem is that there are a bunch of possibilities.  There are three or four possibilities in a generic sense:
 - Insufficient memory, either available on the machine or configured (properly) in Sybase
 - Paging of memory
 - High disk I/O latency and/or low disk bandwidth -- the most likely cause with SRDF.

For the purposes of this discussion, I am going to assume that the PROD box is equal or greater in capability than the DEV box.  I will also assume you are using the Bulk Load facility from Informatica.  Since you are dropping the indexes and recreating them, I will also assume you are getting fast bulk copy to work.  If, in fact, you are doing INSERTS from Informatica, you will be beating the heck out of the Log and that narrows down your problem space.

Unfortunately, all put one of the things you might try fall squarely in the DBA/Sys Admin territory so there may not be much you can do.

 - Start by comparing the Sybase server configurations.  In particular, make certain the memory configuration is set properly on PROD.  Not only do you have to set the memory that Sybase can use, you have to increase the size of the default buffer cache as well.  I have seen this mistake more times than I can count.
 - Make certain that the PROD server actually has enough memory to prevent paging of Sybase.  Paging kills the database.  I have seen a box where the very smart BIOS disables a bad memory chip (and it's pair) and no one noticed.  It may also be that there are other processes running on the PROD box that are squeezing Sybase out of RAM.  Sybase should be configured to allocate all of the memory at startup and lock it into RAM.  It will throw an error at startup if it cannot do what you ask of it.
 - Now run TOP and compare the two machines under load.  In particular, look at the amount of "I/O WAIT" in the CPU times.  If you are badly disk bound, as I suspect you are, you will see it here.

Since you are using EMC SRDF, one possibility is that your DBAs have it set up in synchronous mode.  That means that when the server issues and I/O, it is not reported as completed until the write occurs on the far end and an acknowledgment is returned.  This adds whatever network latency for transmitting the data blocks and receiving the ACK to each I/O.  Could that slow things down by a factor of 5; absolutely.

Most of the time, you probably don't see the SRDF impact too badly because the system is not loaded to 100 percent, at least not in a sustained fashion.  Most small writes (with the exception of LOGs) are cached and don't actually hit the disk until the checkpoint writer process pushes dirty pages to disk.  Since this happens in the background, you don't see how long it is taking.

Your load process has a boat load of data, fills up the cache, and the dirty pages need to get flushed to disk before you can put more data in.  That puts the large SRDF latencies in the execution path for your job.  You may be able to mitigate this by increasing the amount of memory available to Sybase.  You are probably only loading 400-800MB of data so even a 32 bit installation should be able to assign that much cache to the database server.

If you are doing INSERTs from Informatica, your Log writes are the limiting factor.  If you have a large enough installation, that is at least 4 CPUs with 4 Sybase engines lit up, you can turn on asynchronous log writes.  This will decouple your process from the disk latencies to some extent.  You might also consider increasing the size of your user log pages to 8 or 16KB.  SRDF adds more latency per I/O than it does per KB transfered so making fewer, larger I/Os should help throughput.

While it won't help your INDEX builds, you might see significant benefit on the PROD box by modifying the Informatica job to use the Sybase Bulk facility.  I don't know which version of Informatica you are running but I thought (could be wrong) that later versions supported the bulk interface directly.  If not, you change the job by writing out a delimited flat file and firing off BCP (the Sybase bulk copy utility) to actually load the data.  With no indexes and triggers, the load is "minimally logged".  What happens is that the rows are formatted and directly written to table pages.  The only logging is that required to keep track of space utilization.  Bulk loading along with having enough memory/cache setup to contain the majority of the new data should eleviate the SRDF bottleneck.  It is also the one thing you have complete control over.

Another thing you might want to talk with the DBAs about next time they do an upgrade is to increase the Sybase page size.  The default of 2KB is way to small for the way disk hardware works these days.  Even on small machines, I always set the page size to 8KB.  Again, fewer larger I/Os will help in your situation.

BTW, the tempdb devices should not be SRDFed at all.  tempdb is recreated each time the database server starts up.  In addition, many systems pound the heck our of tempdb so that last thing you want is to put it on a slow device.  I have actually had good success putting tempdb on tmpfs (ramdisk) file systems.

You can ask the DBAs/Sys Admins to take the volumes that Sybase lives on and change the SRDF setup to run asynchronously however, that brings about it's own set of issues.  It will mitigate your performance issues but may result in an unrecoverable database at the far end under certain failure modes.  Databases like to control, or at least know the order in which things are written to disk.  I would call Sybase tech support and see what they say about SRDF and similar remote disk mirroring and whether there is a recommended (or at least relatively safe) way to configure it for asynchronous operation.

A better solution for remote hot standby of a Sybase database is to use Replication Server.  Rep Server is very unobtrusive the to PROD server because it reads the transaction log rather than adding triggers to tables.  It reverses the operations that took place from the log and transmits them to the remote server where an agent then applies them.  Rep Server also has an internal store-and-forward component that ensures that nothing gets dropped on the ground in any failure mode of server or network.  It also has other clever optimizations such as replicating stored procedure calls rather than all of the changed data that the stored procedure might produce.  Depending on your application, this can be quite an advantage.

Hope this help out.

Regards,
Bill