Open Mic - Troubleshooting & Best Practices for Read/Unread Marks Manisha Parida and Sandeep R Deshpande 29 Feb

2012

OPEN MIC LOTUS TEAM

Manisha Parida - Lotus Technical support engineer Presenter Sandeep R Deshpande - Lotus Technical support engineer Presenter Hansraj Mali - Lotus Technical Advisor Focussing on Notes/Domino, LotusLive Ranjit Rai - Lotus Technical Advisor Focussing on entire Notes/Domino Jayavel Rajendran - Lotus Technical Advisor Focussing on entire Notes/Domino Vinayak Tavargeri Lotus Support Facilitator for Open Mics

Agenda

Unread Marks Architecture How are Unread Marks Updated? How are Unread Marks Maintained? Tips for troubleshooting unread mark issues Best Practices
Q/A

What is Unread ID Table ?

Unread marks are a list of unread documents within a Notes database. Use note ID tables to track all unread documents. If a note ID is in this table,it will appear as a UNREAD document. If it is not in the table it appears as READ. Each user that accesses a database has an individual Unread ID table in a database How the Unread ID table in a database looks like ?

What is Unread Mark Journal ?

Notes Client has unread mark journal which tracks individual operations for each database replica ID by UNID and it is present in cache.ndk Journal operations are applied to unread table on database open Journal is circular and has a limit on the total operations it can retain Mark All operations are treated as Mark All Before This Time

Unread Mark journal can be disabled with notes.ini parameters: For databases that replicate unread marks
DEBUG_LIMIT_UNREAD_JOURNAL=1

For databases that do not replicate unread marks

DEBUG_DONT_PLAY_UNREAD_JOURNAL

Single Replica scenario

Single client - Single replica & Multiple client Single replica

Unread mark table is created when a user first opens the database All note Ids are initially added to the user's Unread ID table Read a document, remove a note id On next db open, newly modified docs are added to the table Works well for single replica-multiple client machines scenario

Single Client -Multiple Replica

User reads documents in replica on Server A Unread Mark Journal tracks operations User opens database on Server B Journal operations are then applied to table in replica B

Multiple Clients -Multiple Replicas

Most of the unread mark issues are seen in this scenario. Notes.INI parameter introduced to synchronize unread mark tables during client-side replication REPLICATOR_SYNC_UNREAD=2 will sync Unread ID table every one hour (half of the 2) Or

REPLICATOR_SYNC_UNREAD=-1 will sync the Unread ID table on each replication.

How other non-Notes clients update unread marks ?

Lotus iNotes iNotes directly uses the Unread ID table to mark the document as read (It does not use Unread mark journal like Notes Client.)

Blackberry, Traveler: Essentially use their own systems that wrap/integrate with the Unread ID table and thereby directly modifying the Unread ID table in database.

What are Unread Activity Logs ?

Unread Activity logs are stored per user in the NSF database Log entries include timestamp of operation These logs will replicate! Data organized into chunks to help minimize I/O. Only necessary chunks are loaded/replicated to avoid performance issue. Old data is purged (Replication Cutoff) This is how Unread Activity log looks like -

Unread Activity Log replication

Replicates individual chunk entries between databases Uses adjusted activity time for sequence Minimizes server bounceback Client-initiated replication only processes user's activity logs Do not mark modified documents unread should be selected

Unread Log replication

Unread Activity log playback

Playback starts from the oldest modified log entry since playback.

Note IDs are added/removed from Unread ID table

Do not mark modified documents unread ignores modified docs since read operation

Unread log playback

Unread marks - Tips

Q.- What is role of cache.ndk in unread marks ? Ans:- Cache.ndk stores unread mark journal which tracks all unread activity of the user. From 8.5 version, there is less dependency on cache.ndk as the the Unread ID table in database is directly updated. However still unread mark journal tracks unread activity of the user and if you open another replica of the database immediately after closing the first database, then unread journal is played back to another replica and hence another replica will display correct unread count.

Unread marks - Tips

Q.- What should be the basic steps taken by Administrator if any user is facing unread mark issue ? Ans:- If your only one or two users are facing the unread marks issue, then do these basic steps. Your problem may get resolved for 70% of the times with these steps.
Mark all mails as read first (you may move the unread mails to one folder, so that later you can know which mails were unread)

Repalce the design of mail file. Shut down the Client and rename cache.ndk.

After starting the Notes Client, go to workspace and unstack replica icons for mail file and then select both icons and then exchange unread marks by going to Edit > Unread marks > Exchange unread marks. So unread marks from first selected replica will be pushed to second selected replica.

Run Fixup -f and Compact -c on mail file of the user.

Unread marks - Tips

Q.- I have a cluster environment. In my primary mail server the unread marks are always correct but in secondary server, they are often incorrect. This is happening with most of the users. What could be the problem ? Ans:- This problem normally happens in cluster environment when you create new replicas of user mail file on secondary server by selecting them all and Create > New Replica (i.e.- via AdminP process). You must have the following notes.ini parameter in source server ADMINP_EXCHANGE_ALL_UNREAD_MARKS = 1 so that unread marks will be replicated to new replica. If you have already created new replicas without this parameter and therefore facing unread marks problems in cluster servers, then you should normally create new replicas again as far as possible.

Unread marks - Tips

Q.- I have so many users and I have already created new replicas on my secondary server in cluster without adding the parameter ADMINP_EXCHANGE_ALL_UNREAD_MARKS = 1 in source server. Now is there any way, I can force unread marks to replicate without creating all new replicas again ? Ans:- Yes, there is one notes.ini parameter available for Notes/Domino version 8.0.2 and above as REPL_SYNC_ALL_UNREAD=1 which will force to sync all the unread marks from source server to destination server every time when the replication happens. This parameter should be added to only one server (generally primary server where the unread marks are correct) and it will push the unread marks for all databases on each replication. This parameter puts some extra overhead on replication task.

Unread marks - Tips

Q.- What precautions should be taken if I want to use the parameter REPL_SYNC_ALL_UNREAD=1 ? Ans:- This parameter should not be added to both servers else both replicas may end up with incorrect unread marks. It should be added only to one server where unread marks are correct. This is very resource intensive parameter, so once your unread marks are in sync in both replicas, then remove this parameter. Be sure to go for Domino 8.5.3, if you are going to use this parameter as previous versions have some known issues with this parameter.

Unread marks - Tips

Q.- I have a user whose unread marks are not properly synchronized between Notes Client and Lotus iNotes, but for other users it is working fine. Recreating the cache.ndk of Notes Client is also not helping in this case. What should I do ? Ans:- This kind of problem can happen if the user is recently renamed and for some reason, rename process has not properly updated the person document. If there are any errors with user rename process then address them, otherwise just make sure that, User name field of person document contains the latest name and user id file is also updated with latest name. The first entry in the User name field in person document should be same as that of id file.

Unread marks - Tips

Q.- For most of the users, unread marks are shown differently in Notes Client and iNotes. If they read any mail from iNotes then the same is not shown as read in Notes Client and vice-versa. We have tried manually exchanging unread marks between replicas but it is difficult to instruct users to manually do it every time. What should we do ? Ans:- This kind of problem is observed if you have a separate iNotes server and users are using Notes Client from another server. It seems to be your most of the replicas are out of sync in unread table. So you should try using the parameter REPL_SYNC_ALL_UNREAD=1 in your primary server where the users are connected via Notes Client. This will initiate a full sync of unread table for all mail files. If this parameter fails to resolve the issue then you should create new replicas on iNotes server by first adding the parameter ADMINP_EXCHANGE_ALL_UNREAD_MARKS = 1 in source server.

Unread marks - Tips

Q.- We are using a custom mail template and we are facing unread marks issues for some users. Everytime some new users are impacted. Are there any known issues of unread marks with custom mail template ? Ans:- Starting from Lotus iNotes 8.5.1, folders like Inbox and personal folders show unread count like Notes Client. So there are major design changes for unread marks in 8.5.1 version. It has been observed that, a custom mail template based on 8.5 version can give unread marks problems. So please use the custom template based on at least 8.5.1 version.

Unread marks - Tips

Q.- One of our VIP user has a strange issue of unread marks. New mails come in Inbox and they are unread at that time. But after 5 minutes they automatically become read. We have recreated cache.ndk, reinstalled Notes Client but nothing is helping. What could be the issue here ? Ans:- If you have configured POP3 client, then this type of issue can occur because POP3 client will periodically poll the server for new mails and will mark the mails as read if you have the notes.ini parameter POP3MarkRead in your Domino server. So once the mails are downloaded by POP3 client, they will be marked as read and thus this issue happens.

Unread marks - Tips

Q.- Our Blackberry users are sometimes facing unread marks problems. Do you have any known issues or troubleshooting steps for Blackberry related problems ? Ans:- There are several reasons for having unread marks issues in Blackberry like below. They are well explained at Blackberry site.
- The BlackBerry Enterprise Server and the BlackBerry smartphone user do not have Manager and Editor rights. - Unread marks replication of the mail database is not configured. - The canonical name does not match the first item in the User Name field in the Person document. More information can be found on Blackberry site at the given link.

http://btsc.webapps.blackberry.com/btsc/search.do? cmd=displayKC&docType=kc&externalId=KB05253&sliceId=SAL_Public&dialogI D=8052832&stateId=0%200%204714709

Unread marks - Tips

Q.- We have one user whose unread count in Inbox is always shown to some positive number even if there are no unread mails in Inbox. We have already tried marking all mails as read and even recreated cache.ndk few times but problem still persists. This happens only in Standard Notes Client while Basic client shows correct unread count. What could be the issue here ? Ans:- Try renaming the bookmark.nsf. It has been observed that sometimes corruption in bookmark.nsf will cause such kind of issue in Standard Client. In some cases, renaming the workspace folder in data directory of Notes Client has helped to solve the issue. This is especially applicable if the unread count is correctly displayed in Basic client but incorrect in Standard client.

Unread marks - Tips

Q.- We have one mail file and the new mails are becoming as Unread during night time or at particular time of day. We have already done fixup, compact on mail file, recreated cache.ndk, reinstalled Notes Client but still the problem is there. What could the possible reason in our case ? Ans:- In multiple cases, it was found that, Database property > Do not mark modified documents as unread was not selected and there was a scheduled agent running on that mail file which was adding some field to documents and thereby modifying the documents. So the problem was resolved after selecting the above option.

Unread marks - Tips

Q.- Are there any debug parameters for troubleshooting unread marks ? Ans:- There are debug parameters for unread marks and can be used if required. The parameter Debug_Unread=xx can be used with below values.
Bit 0 ( not used ) - Unused Bit 1 (value: 2) - Dump log on view refresh Bit 2 (value: 4) - Log replication debug info Bit 3 (value: 8) - Log playback debug info Bit 4 (value: 16) - Display log management (allocs, resizes, etc) debug info Bit 5 (value: 32) - Extreme logging - adds more debug info to other settings */ --continue..

Unread marks - Tips

Bit 6 (value: 64) - Log admin activity - i.e. renames/deletes Bit 7 (value: 128) - Purge (cutoff time) logging Bit 8 (value: 256) - OpBlock logging Bit 9 (value: 512) - Debug for messages marked Read/Unread

e.g.- DEBUG_UNREAD=50 (32 + 16 + 2) then extreme logging related to management and those for view refresh are captured. DEBUG_UNREAD=512 for document mark read/unread logging

Unread marks - Tips

This is how the Debug_Unread=512 log looks like (when the document is opened by user and therefore automatically marked as read).

Unread marks - Tips

This is how the Debug_Unread=512 log looks like (when the document is marked as read by user without opening it).

Best Practices:

If you are upgrading your Domino servers in cluster, then always upgrade both servers to same version, if you just upgrade one server, then you might face unread mark issues. If you are creating a new server in cluster then always make sure to have the parameter ADMINP_EXCHANGE_ALL_UNREAD_MARKS =1 in source server before you create new replicas on secondary server. Use the parameter REPL_SYNC_ALL_UNREAD=1 carefully, if you add it to server where unread marks are incorrect, then all replicas may end up with incorrect unread marks.
Make sure to select Database Properties > Advanced > Replicate unread marks to All Servers for all your mail files on server and on local. If using a custom mail template, use the template based on at least 8.5.1 or above as using older template like 8.5 may cause some unread mark problems. If unread marks are getting affected during weekend then probably some third party software if running on weekend might be responsible for corrupting NSF databases, so try to exclude some databases from it or try to disable it for some time and check again.

Best Practices:

There are some unread mark issues fixed in Domino 8.5.2 FP2 version, so if you are using an earlier version of Domino, then upgrade to Domino 8.5.2 FP2 (preferably 8.5.3) to fix these issues. If the cache.ndk of the users is frequently getting corrupted and recreating the cache.ndk is resolving the problem, then something might be wrong in your mail template. This generally happens with custom mail templates, so try using a standard mail template for few users and monitor. Use the Debug parameters like Debug_Unread=xx at last if nothing is helping at your side. You need to give the logs to IBM for analysis or if you are interested you can have a look at logs yourself and you may get a clue what is going wrong. While exchanging unread marks between 2 replicas from workspace, always select the replica with correct unread marks first and then select the replica with incorrect unread marks.

Additional resources

The Architecture of Unread Marks in Lotus Notes (Technote #7002920) http://www.ibm.com/support/docview.wss?uid=swg27002920

The enhanced unread functionality in Notes/Domino (Technote #1140018) http://www.ibm.com/support/docview.wss?uid=swg21140018

Unread Marks and Domino Web Access (#7003399) http://www.ibm.com/support/docview.wss?uid=swg27003399

Unread Mark problems in clustered environments (Technote #1100309) http://www.ibm.com/support/docview.wss?uid=swg21100309

Unread count for a folder is greater than actual number of unread documents Technote #1110573) http://www.ibm.com/support/docview.wss?uid=swg21110573

Questions?