About Lotus Notes and Domino and more... provided by the Emcon company.

DAOS issues with backup

Fredrik Malmborg  June 10 2010 02:15:32 PM
This week I implemented DAOS at another customer. They were about to run out of disk space.
DAOS did as I expected, it released close to 50% of the disk used by Domino. So now they don't have to buy more disks and backup completes faster.

But next morning users could not open attachments in mail. In server log it says:
The database d:\Lotus\Domino\Data\mail\abcdef.nsf was unable to open or read the file Document attachment is invalid
That was not what I wanted to see. Then I found out that the backup was still running. Had it halted, but still no change.
Run RESYNC on DAOS, but still same error.
Did an emergency update from 8.5.1 FP2 with HF for DAOS, to 8.5.1 FP3 (was released the other day). And now the error messages were gone. Felt great.

But they came back. Suspected the backup for being rude against Domino. They use Ahsay for backup. Found out that it was working overtime taking backup of the domino\data folder.
Now we reconfigured the backup to not include data\domino because it is not critical in this installation. It can be restored from Lotus Domino setup.
After running for 2 days without errors I conclude that the backup application does something with the files in the DAOS folder (using default location under data). Backup now finishes long time before the users get on.

In the log we also have this nice error:
Process C:\Program Files\AhsayOBM\jvm\bin\bJW.exe (3572/0xDF4) has terminated abnormally
It is the Process Monitor in Domino that senses that a program did not tell Domino that it was about to stop. Usually it is not a problem. But I find it annoying with messages like that in the log.

To sum up, get the latest fixpack and make sure to monitor your backup while using DAOS. You also have to put some extra effort on planning backup with DAOS enabled. If not, it can become a real pain when you have to restore a maildatabase.
Comments

1bu11frogg  06/11/2010 1:12:20 PM  DAOS issues with backup

We had similar issues with CA's ArcServe. DAOS and ArcServe worked fine for weeks. Then, occasionally, backup would take far longer than it should or fail altogether (not good!). I never thought to stop it from backing up the data\domino directory (I'm merely an admin, and Domino is just one of dozens of things I look after), but instead I opted to disable DAOS and wait for CA to update their software.

Then the real problem hit us: Disabling DAOS doesn't completely disable DAOS. We had nights with the log full of hundreds of lines like the one you mentioned above...and that was with DAOS supposedly "off". On Windows task manager, we could see 'ndaosmgr.exe' clearly running. Pull DAOS out of the notes.ini, and it reappeared after restart. Backups might succeed, but would occasionally still fail -- in other words, it behaved the same with DAOS off as it did when DAOS was on. I had an open PMR with Lotus for a month or so, and only closed it when CA finally issued a new version with DAOS compatibility.

My guess is that these backup tools may not be fully using the Domino API, but I'm not really sure.

At least now we can turn DAOS back on...and hopefully Arcserve behaves.

2Fredrik Malmborg  06/15/2010 1:21:20 PM  DAOS issues with backup

"Disabling DAOS doesn't completely disable DAOS" - What it does is to stop new attachments from being extracted and recreated as a link. It is still needed to support attachments already put into DOAS folder. If you want to remove DAOS completely you must also disable DAOS on each database and create new copies of them. That will migrate attachments back into the database. When no files are left in DAOS folder it is safe to fully remove DAOS. BUT don't forget to check what you have in backup. When you have removed DAOS you cannot restore old backups that used DAOS for attachments !