Troubleshooting eDirectory™ and Configuring Large eDirectory Environments

Mangal Shriram

eDirectory Engineering

Novell, Inc.

Rick Killpack

World Wide Support (eDirectory)

Novell Inc.

Steve Lindsey

Solutions Interoperability Lab

Novell, Inc.

Jason Taylor

Solutions Interoperability Lab

Novell Inc.

Agenda

eDirectory Health Checks

Health Check Report

Troubleshooting with NDS iMonitor

eDirectory Agent Configuration

Disaster Recovery

SESSION BREAK

Agenda - Continued

Large Scale eDirectory™ Design

Management Tools

Tree Architecture Design

Operating Systems

Hardware Considerations

Performance and Scalability Tuning

Data Access

eDirectory Health Checks

Basic Health Check

Run the Treewide Server Information Report with the Health Check option enabled

Complete Health Check

Basic Health Check

Against each server in the Tree

Run Obituary Report

Run Object Statistics Report – Check Unknown objects

Run Schema Compare

http://www.novell.com/documentation/edir87/index.html

Administration Guide -> Maintaining Novell eDirectory -> Keeping eDirectory Healthy

iMonitor Health Check Report

this slide is for full bleed images…

(delete this text box)

iMonitor Health Check Report

Scheduled Reports run as .[Public].

Agent Health Sub-Report

iMonitor Health Check Report

iMonitor Health Check Configuration

Self documenting configuration file ndsimonhealth.ini or ndsimonhealth.conf

time_sync-active: WARN

time_delta-active: WARN | MARGINAL | SUSPECT

time_delta-Min_Warn: -60

time_delta-Min_Suspect: -30

time_delta-Min_Marginal: -20

time_delta-Max_Marginal: 20

time_delta-Max_Suspect: 30

time_delta-Max_Warn: 60

ring_readable-Min_Warn: 1

perishable_data-active: off

Troubleshooting Guidelines

Don’t panic

Look for root causes, not symptoms

After taking steps to correct a problem, make sure
the correct background processes run to completion

Get training

General rules

Solve communication problems first

Solve synchronization issues first (Schema & Object)

Make sure your system is correctly time-synced

Run the correct (latest) support pack

Enabling Advanced Mode

Health Troubleshooting

Obituary Troubleshooting

iMonitor Obituary Report

Obituary Help

. . .

Schema Troubleshooting

Agent Configuration

Disaster Recovery

Backup and Restore Options

eDirectory Backup eMTool

Clone DIB Set

eDirectory Backup eMTool

Hot Continuous Backup

Supports the following:

Full Backups

Incremental Backups

Transactional Recovery

On-line Backups

Can Run in Batch Mode

Cross Platform Tool

Very Quick in Backup and Restore

Clone DIB Set

Can Be Used as a Disaster Recovery in Large Environments

Makes a Clone of a Server

Creates the NCP Server Object

Modifies the replica rings and adds the new server

End result is a duplicate of the original server

Disadvantages

Restores a snap shot

Requires another server with same information to clone

Does not restore all objects

LDAP, SAS, NMAS, SNMP, HTTP objects.

Clone DIB Set – Extend Schema

. . .

Clone DIB Set – Create a Clone

. . .

Clone DIB Set - Status

SESSION BREAK

Large Scale eDirectory Design
Identity Management Systems and LDAP

Agenda

Large Scale eDirectory Design

Management Tools

Tree Architecture Design

Operating Systems

Hardware Considerations

Performance and Scalability Tuning

Data Access

Large Trees
Browsing Containers With a Lot of Objects

Modify the URL for Quicker Browse

Novell iManager
Browse vs. Search

Novell iManager
Browse vs. Search

iManager - Query For Objects

DSRepair Strategies in Large Trees
Single Object Repair

DSRepair Strategies in Large Trees
Local Database Repair

DSRepair Strategies in Large Trees
Deselect “Check Local References”

DSRepair Strategies in Large Trees
Repair Single Replicas

Groups in Large Environments
Dynamic Groups

Doesn't need static members defined.

Define a filter

If an object meets the filter requirement, it is part of the group

Security not calculated on login

Dynamic Group filter only applied if resource is needed

Groups in Large Environments
Dynamic Groups

Groups in Large Environments
Dynamic Groups

A Quick Look at a Typical IDM System

eDirectory Servers

Web Policy Enforcement Servers

LDAP Enabled App Servers

Web Servers

Proxy Server(s)

Custom Middleware Servers

Associates

Customers

Other Data Stores

Nsure® IDM

Internet

Replica

Replica

Replica

Replica

Tree Architecture Design
Traditional Trees

The tree is primarily an organization of your physical network resources.

Design the top of the tree to mirror your WAN infrastructure.

Design the bottom of the tree to mirror sites, divisions, departments, and workgroups.

Place your network's physical resources close to their users.

Partition top of the tree based on the WAN infrastructure

Do NOT create a partition that spans the WAN (split across a WAN link)

Partition around the local servers in each geographic area

Partition top of the tree based on size (1000 - 100,000's of users).

OU=PAYROLL

[Root]

O=ACME

Geographic

(Regional)

Geographic

(Location)

Operational

Project/Department

OU=PACRIM

OU=AMERC

OU=EUROPE

OU=HKNG

OU=LON

OU=PAR

OU=SALES

OU=RECRUIT

OU=CHI

OU=HR

Tree Architecture Design
Identity Management Trees

Keep the Tree as flat as possible.

When dealing with millions of users do not try to excessively organize.

Use O and OU objects as access containers for both security and ease of middleware design.

Be frugal with the use of ACLs and more reliant on the use of attributes.

Partition Organizational Units (OU) into 2 Million objects

Use partitions more for manageability instead of geographic locations of replicas (although geographic locations may still be relevant).

Shorten DSRepair times

Partition around the local servers in each geographic area

Partition top of the tree based on size (1000 - 100,000's of users).

T=ACME_Inc

O=Administration

Admin

Servers

OU=Cust1

CN=user1

CN=user2000000

OU=Cust2

CN=user2000001

CN=user4000000

OU=Cust50

CN=user98000001

CN=user100000000

O=Customers

CN=employee1

CN=employee100000

O=Employees

Operating Systems
NetWare 6.5

Strengths

Very mature network operating system

Handles day to day business of the file system access, printing, identity management and web applications simultaneously better than any other network operating system

Handles complex operations (I.E. LDAP complex queries) better than other operating systems

Optimized for multi-purpose, day to day tasks

Novell eDirectory utilities (I.E. DSRepair) more efficient on Netware than other operating systems

Weaknesses

Does not handle multi-processors as well as Solaris or Linux

Does not handle thread management as well as Solaris or Linux

Not 64-bit Enabled

Operating Systems
SuSE Linux Enterprise 9

Strengths

Handles simple operations (I.E. simple LDAP queries) faster than other operating systems

Very stable

Good native memory management

Performs well with multi-processors

Highly scalable

64-bit enabled to take advantage of AMD Opteron and Intel EM64T processors

Highest Price to Performance ratio of all Operating Systems for LDAP Operations

Weaknesses

Relatively new in the corporate world

Does not handle complex operations as well as other operating systems

Operating Systems
Solaris

Strengths

Utilizes multi-processors much more efficiently than other operating systems

Very stable architecture

Mature in the market place in the corporate world

Manages large amounts of connections efficiently

64-bit processing and infrastructure

Weaknesses

More expensive than other operating systems

Hardware is proprietary

Limited driver support and peripherals

LDAP search speeds and import speeds not as fast as Linux

File and print capabilities are not as strong as other platforms

Sample LDAP Performance Comparison

Hardware Considerations
Eliminating the bottleneck

Disk

I/O is the key

CPU

Quantity or Quality

Memory

How much is too much

Network

Too little and too much

Disk Considerations
Eliminating the bottleneck

Some actions which cause eDirectory to write to disk

Modify/Add/Delete operations

Authentication with “Login Update Enabled” on

RAID Level 1+0 is recommended

SAN vs. DAS vs. NAS vs. internal

Disk Cache plays a role

Calculate your overall disk I/O speed

Sequential Writes Versus Random Writes

The Key is Disk Aggregation!

CPU Considerations
Eliminating the bottleneck

The General Rule:

Quantity (SMP)

More CPU's = more load (bandwidth, connections)

Quality (Speed)

Faster CPU = more transactions/second(throughput)

Although it is more complicated than this, you should be able to anticipate CPU performance behavior using this general rule.

CPU Considerations
Eliminating the bottleneck

2 Processors

8 Processors

6 Processors

4 Processors

Processor scaling can be forcast using Amdahl's Law:

Scalar = (1-f) + f / # of Procs

MPTrans/sec = SPTrans/sec * Scalar

Actual performance tests validate calculations

Memory Considerations
Eliminating the bottleneck

NDSD/DHOST Memory Usage

eDirectory memory consumption can be split into 3 areas:

Flaim Database Cache

NDSD/DHOST Process

LDAP, Nsure Audit, IDM Manager

Worker Threads

eDirectory Memory Configuration

Database Cache

Static

Dynamic

Dirty Cache

Network Considerations
Eliminating the bottleneck

Network

10,000 ldap client requests to 4 replicas caused 80Mbs of traffic across a Gigabit Ethernet network.

The bigger the requested LDAP attributes (ie. returning all of the attributes of 2 Million users under an OU) the more bandwidth that is consumed.

SSL

Our tests have shown that using LDAP over SSL results in an average of 10% performance hit due to CPU encryption latency.

Cache Settings

Database

Max DirtyCache

Min DirtyCache

Replication threads

Worker Process threads

Large Tree Performance and
Scalability Tuning

Database Cache
eDirectory Optimization Settings

Dynamic Cache

Static Cache

Cache Size

Cache Allocation

Make Changes Permanent

iMonitor-> Agent Configuration-> Database Cache

Static Cache is suggested in larger environments

More Block cache favors write operations while less Block Cache favors read operations

Cache Size should be kept between 250MB and 1GB

Dirty Cache Settings
eDirectory Optimization Settings

eDirectory DIB directory-> _NDSDB.INI

Upper Limit of Dirty Cache

Lower Flush Limit of Dirty Cache

Max/Min Dirty Cache Settings
How Disk Speed Affects Response Time

1

3

5

7

9

12

15

18

21

24

27

30

33

36

39

42

45

48

0

10

20

30

40

50

60

70

80

90

100

110

120

MB

Dirty Cache Builds Up

Dirty Cache Flush

51

54

57

60

63

66

69

72

75

78

81

84

87

90

93

96

99

103

108

113

118

123

128

133

138

Seconds

Time Period when DIB is locked and all writes are suspended

maxdirtycache=120MB

mindirtycache=0MB

Replication Threads
eDirectory Optimization Settings

iMonitor-> Agent Configuration-> Agent Synchronization

Number of Sync Threads

Thread association

Worker Threads
eDirectory Optimization Settings

ndsconfig get

Thread Timeout (millisecs)

Minimum Threads in Pool

Minimum Threads in Pool at Startup

Maximum Number of Threads in Pool

More on Worker Threads
eDirectory Optimization Settings

n4u.server.max-threads

This parameter lists the maximum number of threads that will be started by the eDirectory server. This is the number of concurrent operations that can be done within the eDirectory server. (Default 64)

For LDAP, each eDirectory Worker Thread can handle up to 256 concurrent connections. Therefore, with 64 LDAP Worker Threads eDirectory can handle up to 16,384 concurrent LDAP connections.

type “ndstrace -c threads” to see how many active threads are running

n4u.server.idle-threads

This parameter lists out the maximum number of idle threads that are allowed in the eDirectory server. (Default 8)

This parameter should be set to a value that can handle the average concurrent connection load your system will have. By default eDirectory is set to receive 2,048 concurrent LDAP connections.

Bulk Loading

LDIF Files

Comma delimited files (great for Database migrations)

MetaData Convergence

Nsure Identity Manager (formerly DirXML)

Multi-Master

Horizontally Scale with multiple write replicas

LDAP Best Practices

Indexing

Rely more on attributes than groups to associate users.

If large groups are necessary then use Dynamic Groups

Data Access

Questions and Answers

Appendix
Troubleshooting Unknown Objects

Unknown Object Troubleshooting

iMonitor Object Statistics Report

Method Server Installation

Biologon

Digipass

Change Password PLM

Method Client Installation

Biologon

Digipass

Change Password

pcProx identity method

Method Enrollment

Digipass

Token Import

Token assignment

NDS password

Remove NDS Password

Assign NDS Password

pcProx

Token assignment for identity

Biologon

Self-Enrollment

Login Sequences Definition

Biologon and Digipass and NDS

Biologon or Digipass

Biologon and Digipass and Change Password

Login sequences Deletion

Biologon

NDS

User Assignment

Default all sequences allowed

Assign allowed sequences

Default login sequence

Login Demonstration

default sequence

Biologon or Digipass

Biologon and Digipass and NDS

Biologon and Digipass and Change Password

Login sequence not authorized for user

Undefined login sequence

Method Server Installation

Biologon

Digipass

Change Password PLM

Method Client Installation

Biologon

Digipass

Change Password

pcProx identity method

Method Enrollment

Digipass

Token Import

Token assignment

NDS password

Remove NDS Password

Assign NDS Password

pcProx

Token assignment for identity

Biologon

Self-Enrollment

Login Sequences Definition

Biologon and Digipass and NDS

Biologon or Digipass

Biologon and Digipass and Change Password

Login sequences Deletion

Biologon

NDS

User Assignment

Default all sequences allowed

Assign allowed sequences

Default login sequence

Login Demonstration

default sequence

Biologon or Digipass

Biologon and Digipass and NDS

Biologon and Digipass and Change Password

Login sequence not authorized for user

Undefined login sequence

Method Server Installation

Biologon

Digipass

Change Password PLM

Method Client Installation

Biologon

Digipass

Change Password

pcProx identity method

Method Enrollment

Digipass

Token Import

Token assignment

NDS password

Remove NDS Password

Assign NDS Password

pcProx

Token assignment for identity

Biologon

Self-Enrollment

Login Sequences Definition

Biologon and Digipass and NDS

Biologon or Digipass

Biologon and Digipass and Change Password

Login sequences Deletion

Biologon

NDS

User Assignment

Default all sequences allowed

Assign allowed sequences

Default login sequence

Login Demonstration

default sequence

Biologon or Digipass

Biologon and Digipass and NDS

Biologon and Digipass and Change Password

Login sequence not authorized for user

Undefined login sequence

Method Server Installation

Biologon

Digipass

Change Password PLM

Method Client Installation

Biologon

Digipass

Change Password

pcProx identity method

Method Enrollment

Digipass

Token Import

Token assignment

NDS password

Remove NDS Password

Assign NDS Password

pcProx

Token assignment for identity

Biologon

Self-Enrollment

Login Sequences Definition

Biologon and Digipass and NDS

Biologon or Digipass

Biologon and Digipass and Change Password

Login sequences Deletion

Biologon

NDS

User Assignment

Default all sequences allowed

Assign allowed sequences

Default login sequence

Login Demonstration

default sequence

Biologon or Digipass

Biologon and Digipass and NDS

Biologon and Digipass and Change Password

Login sequence not authorized for user

Undefined login sequence

Method Server Installation

Biologon

Digipass

Change Password PLM

Method Client Installation

Biologon

Digipass

Change Password

pcProx identity method

Method Enrollment

Digipass

Token Import

Token assignment

NDS password

Remove NDS Password

Assign NDS Password

pcProx

Token assignment for identity

Biologon

Self-Enrollment

Login Sequences Definition

Biologon and Digipass and NDS

Biologon or Digipass

Biologon and Digipass and Change Password

Login sequences Deletion

Biologon

NDS

User Assignment

Default all sequences allowed

Assign allowed sequences

Default login sequence

Login Demonstration

default sequence

Biologon or Digipass

Biologon and Digipass and NDS

Biologon and Digipass and Change Password

Login sequence not authorized for user

Undefined login sequence

Method Server Installation

Biologon

Digipass

Change Password PLM

Method Client Installation

Biologon

Digipass

Change Password

pcProx identity method

Method Enrollment

Digipass

Token Import

Token assignment

NDS password

Remove NDS Password

Assign NDS Password

pcProx

Token assignment for identity

Biologon

Self-Enrollment

Login Sequences Definition

Biologon and Digipass and NDS

Biologon or Digipass

Biologon and Digipass and Change Password

Login sequences Deletion

Biologon

NDS

User Assignment

Default all sequences allowed

Assign allowed sequences

Default login sequence

Login Demonstration

default sequence

Biologon or Digipass

Biologon and Digipass and NDS

Biologon and Digipass and Change Password

Login sequence not authorized for user

Undefined login sequence

Method Server Installation

Biologon

Digipass

Change Password PLM

Method Client Installation

Biologon

Digipass

Change Password

pcProx identity method

Method Enrollment

Digipass

Token Import

Token assignment

NDS password

Remove NDS Password

Assign NDS Password

pcProx

Token assignment for identity

Biologon

Self-Enrollment

Login Sequences Definition

Biologon and Digipass and NDS

Biologon or Digipass

Biologon and Digipass and Change Password

Login sequences Deletion

Biologon

NDS

User Assignment

Default all sequences allowed

Assign allowed sequences

Default login sequence

Login Demonstration

default sequence

Biologon or Digipass

Biologon and Digipass and NDS

Biologon and Digipass and Change Password

Login sequence not authorized for user

Undefined login sequence

Method Server Installation

Biologon

Digipass

Change Password PLM

Method Client Installation

Biologon

Digipass

Change Password

pcProx identity method

Method Enrollment

Digipass

Token Import

Token assignment

NDS password

Remove NDS Password

Assign NDS Password

pcProx

Token assignment for identity

Biologon

Self-Enrollment

Login Sequences Definition

Biologon and Digipass and NDS

Biologon or Digipass

Biologon and Digipass and Change Password

Login sequences Deletion

Biologon

NDS

User Assignment

Default all sequences allowed

Assign allowed sequences

Default login sequence

Login Demonstration

default sequence

Biologon or Digipass

Biologon and Digipass and NDS

Biologon and Digipass and Change Password

Login sequence not authorized for user

Undefined login sequence

[Close]

Method Server Installation

Biologon

Digipass

Change Password PLM

Method Client Installation

Biologon

Digipass

Change Password

pcProx identity method

Method Enrollment

Digipass

Token Import

Token assignment

NDS password

Remove NDS Password

Assign NDS Password

pcProx

Token assignment for identity

Biologon

Self-Enrollment

Login Sequences Definition

Biologon and Digipass and NDS

Biologon or Digipass

Biologon and Digipass and Change Password

Login sequences Deletion

Biologon

NDS

User Assignment

Default all sequences allowed

Assign allowed sequences

Default login sequence

Login Demonstration

default sequence

Biologon or Digipass

Biologon and Digipass and NDS

Biologon and Digipass and Change Password

Login sequence not authorized for user

Undefined login sequence

Method Server Installation

Biologon

Digipass

Change Password PLM

Method Client Installation

Biologon

Digipass

Change Password

pcProx identity method

Method Enrollment

Digipass

Token Import

Token assignment

NDS password

Remove NDS Password

Assign NDS Password

pcProx

Token assignment for identity

Biologon

Self-Enrollment

Login Sequences Definition

Biologon and Digipass and NDS

Biologon or Digipass

Biologon and Digipass and Change Password

Login sequences Deletion

Biologon

NDS

User Assignment

Default all sequences allowed

Assign allowed sequences

Default login sequence

Login Demonstration

default sequence

Biologon or Digipass

Biologon and Digipass and NDS

Biologon and Digipass and Change Password

Login sequence not authorized for user

Undefined login sequence

This means that no work is lost with either the Novell Nsure UDDI corporate or testing solution.

Enterprises that build their UDDI solutions with Novell Nsure UDDI Server, are not only leveraging the deep experience Novell Identity Management foundation brings to the standard, they are securing their own future by building on an open standard implementation that will grow and scale with their corporate needs.

2003-12-08T12:43:39