AIE HOME
Directory Replication
Author: Jens Moller

Directory Replication is a complex topic, because of this, I've tried to keep the examples as simple and to the point as possible. All of the existing Production quality Directory Server vendors have created their own replication solutions and in some cases, its possible to replicate bits and pieces of Directory information on an 'As Needed' basis. There are efforts to create a standardize replication model (you'll hear the term LDUP mentioned when people discuss this), but as of yet its not something that you should count on.

First, realize that very few Directories that have replication capabilities that are compatible with other Directories Replication models. Even the Standards Based Models often do a few things in an inconsistent fashion. This does not make it impossible to synchronize things, however it often means that it requires an understanding of where the sources of record are and how to sync that data in an appropriate way.

There are a wide range of solutions that the vendors provide. Interestingly enough, many people write their own replication functions because they don't like the solution provided by the vendor, or they need some functionality that the vendor didn't provide.

Replication is never immediate - there is always a time lag from the time one system is updated before the other systems pick up the changes to the data. Its safe to assume that changes will take at least 5 minutes to show up on remote systems, however, if there are thousands of updates being done, it may take hours before a change is replicated to a remote system. This is normal operations and something to consider when designing an application (all the more reason to avoid storing frequently changing things in a Directory).

Recovery from failure modes will determine how well any replication model really works. Over time the Directory Server will encounter most of the possible failure conditions that will cause data consistency issues. If you have very few writes/updates/deletes and mostly reads, there will be few opporitunities for a failure to occur. Most applications do far more writes/updates/deletes than people think they do.

Master Servers allow Adds, Deletes, Modifies and general Read against the Directory. All non-Master Servers operate in a Read Only fashion.

The concept of a GUID (Global User Id) is gaining popularity in Multi-Master Directory Servers. It is a unique identifier that is created on a Master Directory Server and it will never be the same as a GUID created on different Master Directory Server from the same software vendor. When a Directory entry is replicated, the GUID follows the entry. If duplicates show up but the GUID values are different, this usually indicates a failed re-synch operation as a result of an outage of one of the other master Directory Servers in a Multi Master environment.

Note: the Multi-Master examples discuss simplified implementations of specific vendors operations. These are operational models that people have made available in their products and are listed here to show how they might work. Also listed are related problems

Single Master Model

By far the most common, and easiest to implement. Many vendors offer it as an option, and often, each implementation includes other capabilities.

How it Works

A transaction log file is written to by one process, while one or more other processes read thru the transaction log file and attempt to update one or more remote systems. The transaction log file is the single master list of updates that must take place on other servers. Some systems offer replication agreements that allow subsets of the updates in the transaction log to be replicated to different systems.

If the Master becomes unavailable, updates cannot be made to any Directory. Some Servers will queue up modifications on the Read-Only Servers, however this is often not a good idea since random Read-Only queues may attempt to update the same entry and you have no idea what order any of these will occur in.

Who Uses this model

  • iPlanet (using LDAP)
  • OpenLDAP (using LDAP)
  • DCL (using DAP)
  • OpenWave LDAP Server (using LDAP)

Good things about it

Simple to implement. As long as the network is reliable and there is not an excessive amount of data to sync, it works remarkably well.

If you had to sync a bunch of unrelated Directory Servers, such as DCL, Active Directory and iPlanet with each other at the same time, this would be the easiest method to use, and is reliable enough to put into a production environment.

Problem areas

Over time, you tend to lose occasional transactions. This occurs as a result of many environmental things. There is a dependency that the 'DN' (Distinguished Name) remain consistent on all of the nodes, otherwise the transaction log file (which typically has only the DN and the changes made to that DN in it) won't match records on all nodes. This is not always easy to enforce, even with controls on the sources of record on the Master system.

There are a multitude of actions that could cause one system to stop replication to a remote node,or instances where the remote node was accidently (or intentionally) updated by another process and suddenly you get duplicate entries because the DNs don't exactly match each other.

Another common issue is where one of the nodes is intentionally shutdown and all the other nodes sync data as normal, however the shut down system is unable to be updated. If the Master Systems's Transaction Log file is somehow truncated before the node is brought back on-line, then any additions, modifications or deletions that occurred on the other system can not be delivered to the node once it comes available again. Some of the Directory Servers manage the truncation of the log files for you and you might never even know that this happened. Since some of the replication tools are designed to retry a limited amount of times before skipping a transaction in the log file, you might not have any indication that anything ever went wrong until something else gives you an indication of the inconsistency.

If iPlanet V 4.x detects that something is out of sync on a remote node, it disables processing on the Master and tries to re-sync everything based on its replication model - In theory, a wonderful idea, however, in reality, it may lock up the Master database for hours as it tries to resolve issues that can only be resolved by manual intervention.

Safe Operation

You must assume that over time if you have more than one node, it will have occasional inconsistencies that require some sort of intervention. You need have a module that runs occasionally (once a week, for example) that compares entries on nodes (and it needs to be run from the remote nodes) to see if any differences exist and logs them.

Dual Master Model

This is the next step above a Single Master. Usually implemented because it is generally better than a Single Master Model, and it offers the potential that if the Primary Master goes down, you can still handle updates with a Secondary Master.

How it Works

Its very similar to the Single Master Model in that transaction Log files written to and replication agreements are drawn up on the Master Directory Servers as to what pieces are to go where. There is usually a Primary Master and a Secondary Master. The Primary Master is always the master if it is operational. The Secondary Master takes over when the Primary Master is not available.

Unlike the Single Master, writes need to be directed to a Proxy that decides which of the Masters are running and who gets Write/Update/Delete requests, since only one of them should ever be in control at any one time. This whole arrangement is substantially more complex to manage.

Typically, out on the Nodes where the replicated directorys are, there is a process that allows updates to occur, but they actually get queued up and get sent to the Write/Update/Delete proxy. Reads are handled locally.

Who Uses this model

  • iPlanet

Good things about it

You are never without a Master Directory server as long as one of the two Masters are available. The Secondary Master will attempt to re-sync data based on timestamps (that are part of each entry) with the Primary Master Directory Server.

Problem areas

If the network holds together, this is more reliable than Single Master, however it suffers from many of the same problems. It introduces a new problem when the network causes it to appear that the Primary Master has failed, when in reality, the network links between the Primary and Secondary master fails and each of them assume the Master Roles. If the Proxy is on multiple systems (for redundancy), any system on either side of the network will operate as if a failover occurred. In these instances, the 2 Masters happily allow Adds, Updates and Deletes, syncing data to nodes that they are able to access. The problem comes when the 2 Masters are able to talk to each other again - since both were allowing updates at the same time, the resulting changes that appear in the Primary Master directory are a mix of each servers changes. This can leave quite a mess, often causing duplicate DN entries that don't show up as obvious duplicates after an attempt at resolution by the Primary Master.

To allow the Read-Only systems to appear as if they allow writes, they capture update requests and attempt to process them. These requests are dropped into a local queue (on the same physical system as the Directory Server) then sent to a proxy that decides which of the Multi Master Directory Servers does the updates. If there are a lot of updates, the delays introduced by the additional processing may take quite a while to actually be replicated back to the system that the write appeared to occur on. So, 2 things happen:

  1. Write/Update/Delete operations are usually quite slow

  2. Reads of the data that was just Written/Updated/Deleted have not been pushed back out to the system that the change occurred on yet (depending on what else is happening, it could be hours before the update actually shows up). Typically a 5 minute delay should be expected.

This confuses most applications and people that thought that they had just successfully done an update, and they request the data and its not there. Account Creation and Password resets are common places where this is noticed. This is really not any different than would happen in a Single Master system, however, the Multi-Master tends to be slower because there are more processes involved.

Casual alteration of the network can have a major impact on this Replication Model.

Safe Operation

Operate this in the same way that you would a Single Master Directory. A close monitoring of the networks is quite important. Schedule all network changes at a high level and keep the Directory Management people informed.

Pseudo Multi Master Model

Novell NDS systems allow parts of a directory to be partitioned to various different NDS Directory Servers. When an update occurs, the change goes to the Directory Server than manages that given piece of the Directory. A GUID (Global User ID) is used to track the entry and maintain its uniqueness in case of a duplicate entry (the entries will always have a unique GUID).

Multiple servers that hold replicas of the same partition compose a replica ring. NDS automatically synchronizes the servers in a replica ring.

How it Works

  1. Master Replica

    The first NDS server installed on the network holds the master replica by default. Partitions have only one master replica. Other replicas are created as read/write replicas by default. The "creating a new partition" and "creating a new replica" operations require that a master replica is available. If the master replica is on a server that is down, NDS cannot create new partitions or replicas until the server is up again.

  2. Read/Write Replica

    NDS can access and change information in a read/write replica and then propagate those changes to the other replicas. If users cross slow WAN links or busy routers, you can place a read/write replica locally to them to speed their access of network resources. Keep in mind that the more read/write replicas you create, the more network traffic necessary to keep the replicas synchronized.

  3. Read-only Replica

    Network clients can read network information from read-only replicas, but they cannot change them. NDS synchronizes read-only replicas with the changes from read/write and master replicas.

  4. Subordinate Reference

    Subordinate references are system-generated and contain only enough information to allow NDS to resolve names across partition boundaries. The system creates subordinate references on servers that hold a replica of a parent partition, but not of its child partitions. This allows NDS to walk the tree to replicas of the parent partition's child partitions on other servers. The system automatically deletes the subordinate reference, if a replica of the child partition is copied to the server holding a replica of the parent partition.

Who Uses this model

  • Novell NDS

Good things about it

The NDS database is loosely consistent, which means that replicas are not guaranteed to hold the latest changes to the database at any specific moment in time. To ensure database integrity, NDS automatically synchronizes all replicas. So, the database is guaranteed to completely synchronize. If there is any period of time during which no database changes occur, the database will completely synchronize and all replicas will hold the most recent information.

A good example of this occurs whenever users change their passwords. If they attempt to login with their new password immediately after they change it, the authentication often fails. But, if they wait a few minutes and then login with the new password, the system allows them in. This is because NDS had time to synchronize the password with all the replicas. Fast synchronization, which is synchronization of all object modifications, occurs every 10 seconds. Slow synchronization, which is synchronization of the attributes dealing with login time and network addresses, occurs every 5 minutes.

Instead of synchronizing all information for every object, NDS only synchronizes the updated (delta) information. Each attribute has a time stamp associated with it. NDS updates this time stamp whenever the attribute is updated.

Problem areas

The partitions that you put things can't hold very much. Novell recommends never going over 1250 objects per NDS container and per NDS partition. This is unrealistically low for most Directory instances.

Too many things depend upon NetWare. The support applications are not well integrated with LDAP based tools.

Safe Operation

NDS has build a reliable system by reducing the amount of data that any one partitioned area can hold at a time, and by forcing you to have replica rings if you need 100% uptime. Without the Replica rings, you could easily lose Master access to pieces of your Directory.

These issues make it hard to utilize NDS as a Directory Server for anything outside of general Novell created applications. In this case, the replication models complexity and need for large amounts of systems is a limiting factor for any directory instance.

Multi Master Model

How it Works

Windows 2000 Active Directory provides multi-master replication. Multi-master replication means that rather than having a single point where changes can be made, we actually have multiple points where changes can be made. In Windows 2000, all DCs are equal with each containing a read-write copy of the database. If one of the domain controllers (another word for Directory tree running on a Windows 2000 server in Microsofts eyes) goes down, you can make modifications to the Active Directory database on any other domain controller.

Who Uses this model

  • Microsoft Active Directory

Good things about it

If Administrator A is modifying an attribute for a user object on one DC while Administrator B is modifying the same attribute on a different DC, one of the updates will have to end up overriding the other update. Windows 2000 AD relies on something called USNs (Update Sequence Numbers) to determine which update to use. If the USNs are identical, Windows 2000 AD employs other tie-breaking methods.

GUIDs are also used to determine the uniqueness of the elements.

There is no specific limit to the amount of entries allowed.

In general Microsoft had the benefit of looking at the problems that the other Directory Server systems had in place and trying to pick out the good things and avoid the bad things. It is generally successful, except that it modeled itself as a replacement for the previous Domain Controller rather than a general purpose Directory Server.

Problem areas

The structure of Active Directory assumes a great many things, many of which are not really X.500 compliant. You also get attributes that you were not expecting (because AD simply requires them).

Microsoft limits what can be part of the Multi-Master replication scheme. Certain domain and enterprise-wide operations not well suited to multi-master placement reside on a single domain controller in the domain or forest. The advantage of single-master operation is to prevent the introduction of conflicts while an operation master is offline, rather than introducing potential conflicts and having to resolve them later.

Most actions require Microsoft specific tools to operate against them. Most of these tools only support known (to Active Directory) Object Classes, so if you extend the directory, and you don't alter an existing Object Class, you won't be able to use Microsoft tools to see that the data is getting replicated. If you extend the schema and you don't set the attributes properly at that time, you cannot correct your errors without re-installing and rebuilding your Active Directory (its a feature).

Safe Operation

Do all development and testing in a test environment. You will not be allowed to 'unextend' an Active Directory Server or its replication agreements for any attributes that you add.

Summary

  1. Replication cannot be a time critical function of your application - It is very probable that replication will usually occur quickly, however at times, replicated data may take many hours to get to all of the systems. This does not make the Directory Inconsistant, it is simply how it works in the real world.

  2. Replication success depends on how well the Directories manage failures.

  3. Problems can occur when there are network outages or network performance issues.

  4. Validation of the data on remote systems should be a task that the Directory Manager will do at a given interval. The question is not If, rather, it is When things get out of sync. A plan must be in place to deal with the sync errors.

  5. You cannot automate everything for all possible failure and recovery modes.

  6. Identifying a Directory entry by using GUID's may help to resolve issues where resyncing appears to introduce duplicates - at least you will know if they variants of existing entries or new ones.

  7. Don't trust anyone who tells you that the Directory Server will always recover from all system outages. Eventually, you will find an instance that it will not recover from. It may not happen often, but be prepared.

  8. There are no such things as transactions on a Directory Server. There is no way to Roll Back a change, you have to do the appropriate update/delete/write steps to put things back the way they were.

  9. All of the above methods work well to provide replication services. Many organizations run well with Single Master systems, however it will never be completely redundant in case of an outage. Multi Master systems make things more complex, but strive to provide better redundancy and scalability.

Comments? Questions? Contact Support@AIE-Services.com