Chained/unchained refers to how transactions are handled.
In chained mode - the ANSI standard - you are *always* in a transaction. When a client connects to the server, it issues an invisible implicit BEGIN TRAN, and when the client issues a COMMIT TRAN or ROLLBACK TRAN it immediately issues a new BEGIN TRAN.
This means in chained mode, you never issue BEGIN TRANs. Effectively you close transactions but not open them.
In unchained mode - the Sybase ASE standard - any individual command is atomic (so an update of 1 million rows will update all million or none)... but you must explicitly issue a BEGIN TRAN before you are in a multi-statement transaction.
So the set chained on|off lets you vary this, remembering Sybase ASE is unchained. Clearly this can have a drastic effect on transaction management for your code, since any code written for one mode will not work as intended in the other.
eg/ consider the code:
INSERT [...]
INSERT [...]
COMMIT TRAN
In chained mode, this generates a transaction of two inserts, and with appropriate error handling (which I have left out) you will get either both of them or neither of them. But in unchained mode, you are not in a transaction until you issue an explicit BEGIN TRAN, and that final COMMIT TRAN does nothing.
This is potentially such a problem that ASE includes what the current session-level chained/unchained mode was when stored procedures are created, and by default will throw an error if you try to call a proc from the wrong mode.
However... if you have written code clever enough to (say) do different things whether it is run in chained or unchained mode, you don't care about that error, so you can tell ASE to not care about it either by setting the proc to "anymode".
Which should you use? I think you get clearer code if you follow the Sybase standard, since then you have paired BEGINs and COMMITs. The ANSI way (chained) always looks to me like unpaired closed parentheses.
What really matters is that you pick one and stick with it. I suggest unchained but it doesn't really matter, just be consistent.
If you use one consistently, you'll never need to worry about procedure transaction modes, since they will always be the same as your current session's mode.
