AIE HOME
Meta Directories
Author: Jens Moller

Some applications need to look at diverse data sets. This data probably already exists in your organization, however it may not be accessible in a way that ties it to the other organizations that may want or need to use it.

The reason for this is simple - organizations solved their business problems by building point solutions. Often, these are customized very specifically for the business need and this allows it to provide the best possible data for that organization.

The problem is that there are often many instances of data that is related to another instance of data, but these data stores cannot casually be merged without substantially impacting the performance and usefulness of the other systems. In reality, the issue is not typically one or 2 systems that need to be tied together, but rather dozens of systems that have been created specifically for the business need or highly customized software packages. There was no business reason to make them compatible at the start, but business models have changed and there is a need to bring this data together.

What are Meta Directories?

The concept of 'Federated' databases is central to Meta Directory implementation. 'Federated' databases basically allow you to join tables from Relational Databases that are distinct and separate from each other. These are all SQL based and can be very complex to setup and manage. These 'Federated' accesses normally are read only, because the data that they tend to use is scattered within an organization and cannot be 'locked' in any consistent fashion without causing major performance problems.

A Meta Directory looks at this same issue, but approaches it in a slightly different way. SQL is not the basis anymore, but the access is abstracted out further to LDAP (Lightweight Directory Access Protocol) and/or XML (Extensible Markup Language). The actual access still tries to do similar functions as an SQL Based 'Federated' database, but no longer enforces SQL rules on the transactions, opening the access up to any form of database, not just those that can be configured to provide support for 'Federated' access.

Basically, a Meta Directory Engine is a dynamic database access tool that can be adapted to any existing set of databases to retrieve information and/or provide translation services for the information returned. It should be able to talk to practically any existing database that provides a TCP/IP access point and be able to use it without requiring that the database, or the applications that currently use it to be altered.

Typically a Meta Directory is a series of applications (usually called 'connectors' or 'listeners') that provide the services. the Services are typically defined using Policies, where the Policies are stored within an LDAP Directory server (Meta Directory Policy Store).

Why use a Meta Directory

  • Consolidating Data

    Organizations often have multiple data stores, frequently scattered across the country or the world and need a way to connect this information together. They may also have a desire to get a better handle on the status of existing production or resource utilization. These resources can be data or they could be functions of any sort - for example, finding out the thermostat settings in all of their buildings, or determining where someone is today.

    Many Meta Directory applications are used in Real-Time control applications. Others are used in reporting. The most common use is to insert a Meta Directory function in between applications, having the Meta Directory application intercept the communications and act upon them. Most of the time the data exists in a data store that the Meta Directory accesses. In some instances, unique data will exist as a Meta Data within the Meta Directory Policy Store. These Policies are used to implement business rules.

  • Data is owned by different groups

    Often, key portions of data belong to groups that have little interest in what other groups might be doing. Rather than duplicating all of the data (which usually requires syncing and can be problematic when trying to enforce as business rules accross diverse organizations), relationships are defined and the data is mapped. The Meta Directory Engine appears to be a single system in itself to the end users who access it, but in reality, it allows controlled access of many sources of data.

  • When data needs be transformed to look consistent.

    Any organization who can define the structure of the data they need to resolve a specific business need, and identify that it needs to be the result of data that appears in unique and seperate data sources, is a candidate for Meta Directory applications.

How are they Different than LDAP Directory Servers or Relational Databases?

Most data stores are entities into themselves. Their data, while accessible to the outside world thru various API's or applications, are still bound as a point solution database. A Meta Directory is not really a single place where data resides, rather its a method by which multiple databases appear as a single data source.

Meta Directories don't care what the data is, only that they can access it and return it as some part of a request for data. In this sense, the data that it accesses is virtual, and can be from anything that can be connected to a network.

Typically, a Meta Directory speaks any of LDAP, XML or SQL as needed. The front end processing is most often done using LDAP or XML.

Policies.

By default, most Meta Directory Applications are re-usable for many different functions and until they have read their policies, they usually don't do anything. The reason that they don't do anything is to provide a level of security to their operations. Unless The Meta Directory applications can identify their data sources and provide a security infrastructure to access of that data, they should do practically nothing.

The Policies are read by the Meta Directory application at start-up as well as at any interval that the application developers deem necessary.

More about Policies

Configuration Management.

When you create a data map and define how a Meta Directory application will support this, it should be committed to a safe application management system.

The Meta Directory functions are highly dependent on their policy data and will not operate as expected without it. Other than configuration specific data within a Policy, this data should never be altered casually by any users. Any changes to Policy must be done by Engineers who understand the various systems and their access.

Benefits - Pros

  • Since a Meta Directory application enables existing data, its ROI appears very early on.

  • Unlike data that is related can be tied to one-another. This allows greater access and control of assets and resources.

  • Data that is similar from different existing systems can operate as if it was on a single unified data store. This alone often highlights inconsistencies in the different systems and allows it to be addressed in a controlled fashion - as a result, the data quality usually improves over time.

  • Data Warehousing takes on a different perspective as Meta Directory Application actions can be expanded very quickly by a very small staff. if your business changes, your Meta Directory can adapt very quickly.

Issues - Cons

  • The Meta Directory is sensitive to network availability - Its functionality is directly tied to being able to access its data sources.

  • Someone needs to take ownership of data issues. Inconsistencies will appear once data is made visible; someone (organization) will need to address this, otherwise the data will not be trusted.

  • The Meta Directory applications will be highly distributed. Managing this is a larger effort than point solution applications. Some one who understands the whole system must be involved in resolving production issues.

  • Interfaces to existing systems must be well documented. This is not always available for existing systems that the Meta Directory is attempting to utilize. Minor uncoordinated changes in the point solution applications may greatly affect the ability of the Meta Directory applications to resolve data.

  • Requires strong management backing - most point solution application developers see Meta Directories as an invasion of their environment. Often the biggest stumbling blocks are political rather than technical.

  • Monitoring of applications that are used by the Meta Directory becomes far more critical to operations. An outage in one area may bring the entire Meta Functions down during that time. Some Meta Applications cache data for these situations, however it is often impossible to cache all of the data required for all operations. Redundant data stores may be required for some applications, and this must be considered in any implementation.

Summary

Meta Directory applications provide a service that is layered upon your existing systems. Done correctly, it allows an organization to access data in ways that previously were very hard to do and not adaptable to changes in the business models.

Meta Directory solutions need not be expensive, or extensive. They are best built out in controlled portions and adapted to organizational needs. Typically, the communications and access methods need to be designed and supported by the various data sources. Interface documents need to be created and Service Level Agreements need to be hammered out to make sure that the data that is required is there when its needed.

It all comes down to a simple fact, A Meta Directory becomes an infrastructure solution. When its working correctly, most people won't even know its there. Infrastructure is often hard to justify, because it becomes an invisible operations layer. Once in place, it can invigorate, or if poorly defined, it won't be used.

Any Meta Directory solution requires buy in from the highest levels of management, otherwise it is destined for failure.


Comments? Questions? Contact Support@AIE-Support.com