Novell Nsure® Identity Manager 2 Troubleshooting Tips and Tricks: Net Solution Support Training

Sandy Swanson

WWS Engineer

Novell, Inc.

Rick Moore

IDM Engineering

Novell, Inc.

Agenda

Troubleshooting Tools

Trace Commands & Trace Files

NXSL Processor

Troubleshooting Methodology

Troubleshooting Sample

How to read a trace log

DXCMD & Designer

Trace Commands & Trace Files

Trace Options

Dstrace

Easy and familiar

It can slow down IDM® if used excessively

iMonitor

Easy to use and will display real time activity remotely

May truncate large XML documents (versions prior to 1.5) so it may not be suitable for detailed troubleshooting

Javatrace File

Doesn't show eDirectory™ event system or cache events

Good option when you want a trace file

With IDM you can now set the log file max size

Trace Commands & Trace Files (cont…)

Trace levels are set on the driver-set or the driver

Driver Trace Level is used to troubleshoot driver issues

XSL Trace Level is used to troubleshoot XSLT issues (Don’t use unless you are developing or troubleshooting stylesheet code)

How to Enable Trace Levels

XSL Trace Level and Driver Trace Level are enabled using iManager in the following instructions

In iManager select the DirXML® Management task and click overview

Click search to find your driverset(s)

Choose a driverset and click on it's link.

Click on the “misc” tab.

Enter 0 to 3 in the text box for “Driver trace level” or “XSL trace level”

XSLT Trace Level details

Trace the above plus template instantiations

3

Trace the above plus rule matching and select expression details

Trace the above plus rule instantiations

2

Trace source node processing, output to standard error

No trace messages are displayed or logged

Description

Value

4

1

0

DVRS Trace Level details

Level 2 messages and extensive rule processing messages are displayed and logged

3

Level 1 messages and the contents of XML documents used during event processing are displayed and logged

2

Basic trace messages are displayed and logged

No trace messages are displayed or logged

Description

Value

1

0

Trace Commands & Trace Files (cont…)

DSTrace Commands

NetWare®

LOAD DSTRACE

DSTRACE –ALL

DSTRACE + DVRS (to troubleshoot Drivers)

DSTRACE + DXML (to troubleshoot XSLT Code)

DSTRACE file on (to log to sys:system\dstrace.dbg)

DSTRACE screen on (to display live trace on server console)

DSTrace Commands

Windows NT / 2000 / 2003

Start DSTrace.dlm from NDS® Services (NDSCONS.EXE)

Edit, Options, Clear All

Check DirXML Drivers	(to troubleshoot Drivers)

Check DirXML	(to troubleshoot XSLT Code)

File, New, enter

File Name	(to log to file)

Trace Commands & Trace Files (cont…)

DSTrace Commands

Unix / Linux

start ndstrace -l

set dstrace = nodebug	(to clear all other trace options)

set dstrace = + dvrs	(to troubleshoot Drivers)

set dstrace = + dxml	(to troubleshoot XSLT Code)

Dstrace file on	(to log to file /var/nds/dstrace.log)

Trace Commands & Trace Files (cont…)

How to enable Java Trace File

In iManager select the DirXML Management task and click overview

Click search to find your driverset(s)

Choose a driverset and click on it's link.

Click on the “misc” tab.

Enter the path in the Java Trace File box

On NetWare use volume:/path/filename (i.e. SYS:TEMP\javatrace.txt)

On Win32 use Drive:\path\filename (i.e. C:\temp\javatrace.txt)

On Unix use /path/path/filename (i.e. /var/nds/javatrace.txt)

Trace Commands & Trace Files (cont…)

NXSL Processor

The NXSL Processor provides the ability to use Novell’s XSL Processor outside DirXML

To run the NXSL processor, perform the following steps

Install JRE 1.4.X from Sun Microsystems

Copy rt.jar, nxsl.jar, collections.jar and xp.jar into a directory

Copy the following text into a batch file in your NXSL Directory

@echo off

setlocal

rem set JAVA_HOME to your JRE or JDK directory

SET JAVA_HOME=C:\jdk1.3.1_01\jre

SET NXSL_JARS=C:\novell\xsl

rem set NXSL_JARS to the location containing nxsl.jar, xp.jar, and collections.jar

set CLASSPATH=%JAVA_HOME%\lib\rt.jar

set CLASSPATH=%CLASSPATH%;%NXSL_JARS%\nxsl.jar;%NXSL_JARS%\collections.jar

set CLASSPATH=%CLASSPATH%;%NXSL_JARS%\xp.jar

set nxsl=%JAVA_HOME%\bin\java -classpath %CLASSPATH% com.novell.xsl.nxsl

%nxsl% -e1 %1 %2 %3 %4 %5 %6 %7 %8

endlocal

Use NXSL Processor to troubleshoot the following

NXSL Processor

Validate XSLT Logic

Check for well formed XML/XSLT documents

Test stylesheet in different event types & scenarios to ensure that all events are handled

Test stylesheets without affecting production/test tree

Don’t Use NXSL Processor to troubleshoot the following

Stylesheets with Queries in them

Policy based rules

Check for well formed XDS documents

Test Rules/Stylesheet interaction

Not valid for end to end testing

Only testing the XSLT logic

Not testing Application/eDirectory™ specific issues

NXSL Processor (cont.)

To process a source XML document with a stylesheet:

nxsl source.xml stylesheet.xsl

Will output Result tree to Standard Out

nxsl –o result.xml source.xml stylesheet.xsl

Will output Result tree to file result.xml

nxsl –d[1-4] source.xml stylesheet.xsl

Will generate detailed trace messages (see XSLTTracelevel for description)

NXSL Processor (cont.)

Troubleshooting Methodology

Identify General Issue

Understand IDM Environment

Understand general driver functionality

Generate trace file

Review trace file carefully

Identify error and it’s cause

Identify possible solutions

At any point, if you have a strong hunch (based on your experience or knowledge) spend a few minutes following that hunch but always return to the method if that hunch does not pan out. Don’t get side tracked by it!

Remember to use the KIS principle “Keep It Simple” most problems when broken down are one to many simple problems

Identify General Issue

Driver not starting

Driver stopping unexpectedly

Policies not functioning as expected

Stylesheets not functioning as expected

Unexpected output/results

Unexpected eDirectory Error Codes

Identify / isolate what action causes the issue (Important!)

Starting driver

Stopping driver

Creating users

Deleting users

Moving Users

Running driver for specific amount of time

Troubleshooting Methodology (cont.)

Understand IDM Environment

Identify Platform/Product versions in use

Identify IDM Drivers in use

Understand basic data-flow

Identify outstanding eDirectory issues (if applicable)

Understand Replica Placement

Obtain Driver Exports and import into Test Tree for review

Troubleshooting Methodology (cont.)

Understand general driver functionality

Review driver exports / documentation

Understand functionality that each stylesheet/policy provides

Understand filter configuration

Match action that caused error with driver that caused it

Identify Channel that causes the error

Identify the stylesheet/policy that is likely causing the error

Troubleshooting Methodology (cont.)

Generate trace file

Identify trace method (dstrace, javatrace or iMonitor)

Identify trace flags (DXML or DVRS)

Identify trace level that is needed

If eDirectory to eDirectory or remoteloader, you will need traces from both sides

Don’t assume default configuration is used, always ask

Isolate trace to specific driver

Turn off any drivers that are not causing the issue

Try and limit the trace to to specific action that causes the issue (as identified previously)

Perform action that causes issue and then stop trace

Troubleshooting Methodology (cont.)

Review trace file carefully

Find error message in trace and see if you can determine the cause from that message alone. If not, continue below.

Find beginning of action that caused the error

Compare contents with stylesheets/policies as they are instantiated

Try and understand each segment of the trace before moving to the next segment

Don’t assume it’s OK just because the error happens later

Don’t assume it’s normal unless you are certain that it is

If you don’t see the error or cause then you may need to review the steps taken to isolate the trace (may have been too restrictive)

Troubleshooting Methodology (cont.)

After identifying error and it’s cause

Identify if error is caused by

Application

eDirectory

Driver

Policy

Stylesheet

Environment (replica placement, infrastructure etc…)

If possible, review findings with developer of solution

Identify if the issue is a fundamental flaw in solution design or is it a simple oversight

Troubleshooting Methodology (cont.)

Identify possible solutions

Identify all possible methods of resolving the issue

Identify ramifications of each resolution

If issue is fundamental design flaw be sure that the appropriate people are notified and involved in the resolution

Make sure you are qualified to determine proper solution (If not, you must involve appropriate individuals)

Make sure you plug your resolution into the overall picture

Once you’ve chosen a solution TEST, TEST and TEST some more before making the change in production

Troubleshooting Methodology (cont.)

Error is

reported

Yes

Identify

general

issue

No

Yes

Carefully

review trace

File with

stylesheets

No

Any ideas?

Any ideas?

Yes

No

Follow your

hunch and

verify

Were you

correct? Have

you tested it?

No

Yes

Any ideas?

Understand

DirXML

environment

Understand

General

Driver

Functionality

Generate

isolated

trace log

Glance at

error

message

Review

trace

Isolation

Too Narrow?

Troubleshooting Methodology (cont.)

NDS

Interface

DirXML Architecture

DirXML

Join Engine

Apply

to NDS®

Rules and Stylesheets

Rules and Stylesheets

NDS

Datastore

Application

or Directory

or Database

Application Shim

Publisher Channel

Subscriber Channel

C1.1

P

C

M

E

S

I

C1.1

P

C

M

E

S

O

NDS

Event

ANF

formatted

XML

ANF

formatted

XML

App

Event

Apply to App

Filter

xxxxx.tao

NDS

formatted

XML

ANF XML

to

NDS XML

NDS XML

to

ANF XML

Filter

NDS

formatted

XML

Troubleshooting Sample

Customer reports an error

Ask customer to describe error (identify general issue)

Customer is sometimes getting -609 error when creating users

Does that mean anything to you?

-609 error means Missing Mandatory Attributes, however, we don’t know what drivers are in use and we don’t know why it only happens ‘sometimes’

Ask customer to describe environment & driver functionality

Customer is using NDS to NDS flat (no custom policies or stylesheets) on NetWare 6.5/IDM 2.0.1

Are you ready to get a trace?

No! Customer said ‘sometimes’ and you have not determined when

Ask customer to describe when error happens

Customer says users created in Tree-A are not created in Tree-B. Users created in Tree-B are created in Tree-A, however, modifications work both ways

Troubleshooting Sample (Cont…)

What trace do you think you need?

Since it only happens when creating users in Tree-A, let’s start with a trace on Tree-B side, of a user being created in Tree-A. Using DVRS and DXML flags

DirXML Log Event -------------------

Driver: \PALM\novell\DirXML\Driver Set\eDirectory Driver-Brainshare

Channel: Publisher

Object: \NW65RONDXML_TREE\CORP\Users\Macaroni (novell\users\Macaroni)

Status: Error

Message: Code(-9010) An exception occurred: novell.jclient.JCException: createEntry -609 ERR_MISSING_MANDATORY

Troubleshooting Sample (Cont…)

Any Clues?

You should have guessed that since it’s NDS 2 NDS, both drivers will have the same required attributes so the <surname> was certainly there when the user was created in Tree-A. Something between the create in Tree-A and Tree-B is removing the <surname>

Next Steps?

Since Filters are generally used to remove specific attributes, your first step would be to check them. Since the error is only when creating users in Tree-A, you should check the Subscriber Filter in Tree-A and the Publisher Filter in Tree-B.

You can also check the same log and see if the initial document had the <surname> which would indicate that the Tree-B Publisher Filter removed it. If the initial document did not have the <surname> then the Subscriber Filter in TREE-A removed it

Troubleshooting Sample (Cont…)

Check Publisher Filter to confirm your hunch

Oops!

Our hunch was wrong! Surname is present in the filter! We need to review trace some more

Troubleshooting Sample (Cont…)

Glance at trace. Does anything catch your eye?

eDirectory Driver-Brainshare PT: No match found.

eDirectory Driver-Brainshare PT: Applying object creation policies.

eDirectory Driver-Brainshare PT: Applying object placement policies.

eDirectory Driver-Brainshare PT: Applying command transformation policies.

eDirectory Driver-Brainshare PT: Applying XSLT policy.

eDirectory Driver-Brainshare PT: Filtering out notification-only attributes.

eDirectory Driver-Brainshare PT: Pumping XDS to eDirectory.

eDirectory Driver-Brainshare PT: Performing operation add for novell\users\Macaroni.

eDirectory Driver-Brainshare PT: Adding entry novell\users\Macaroni.

eDirectory Driver-Brainshare PT: Creating RDN Macaroni in context novell\users.

eDirectory Driver-Brainshare PT:

Troubleshooting Sample (Cont…)

Carefully review trace from top

Upon careful review you find the following line in your trace file

eDirectory Driver-Brainshare PT: Applying XSLT policy.

The statement was that “they had no custom policies” !!!

Yet we see IDM applying an XSLT Policy from within the

Output Transformation rule!

Troubleshooting Sample (Cont…)

<?xml version="1.0" encoding="UTF-8"?>

<xsl:stylesheet version="1.0"

xmlns:xsl="http://www.w3.org/1999/XSL/Transform">

<xsl:template match="/">

<xsl:apply-templates select="node()|@*"/>

</xsl:template>

<xsl:template match="node()|@*">

<xsl:copy>

<xsl:apply-templates select="node()|@*"/>

</xsl:copy>

</xsl:template>

<xsl:template match="add-attr[@attr-name='Surname']"/>

</xsl:stylesheet>

Carefully review driver export of Tree-B

Upon careful review you find the following XSLT in the Output Transformation Rule

Somebody added an Output Transformation Rule that filters out add add-attr elements for Surname

This is causing your -609

Troubleshooting Sample (Cont…)

Identify possible Solutions

Remove that template from the Output Transformation Rule

Pro: Eliminate the root cause

Con: Do you know why it was placed there to begin with? If not, then you can’t simply
remove it

Synthesize the Surname from another attribute

Pro: None

Con: Still don’t know the reason it was coded so any work around is suspect. In addition, this particular solution is a kludge because you added significant code to work around a seemingly simple issue

Are you qualified to implement the solution

Unless you are the developer, you are not qualified to implement the solution above since you don’t know what the initial intention was

Contact the Developer

Contact developer that was responsible for the initial deployment and engage them in a discussion

How to read a trace log

When carefully reviewing a trace, keep these in mind

Don’t assume anything is OK

You will learn with experience which errors are normal

Until then, investigate every error

Understand every line before proceeding to the next

Again, you will learn with experience what each line means

The best way to learn how the IDM2.0.1 engine works is to analyze the trace logs

Have the driver exports at hand

Have the driver-exports handy as you review the traces

Correlate the actions in the trace logs with the policies/stylesheets in the export

The next few slides will show a sample tracelog with no errors and explain the content

Sample trace log

Once the user is created in the DIB, DirXML issues an event for that ADD

<nds dtdversion="2.01" ndsversion="8.x">

<source>

<product version="2.0.5.51">DirXML</product>

<contact>Novell, Inc.</contact>

</source>

<input>

<add class-name="User" src-dn="\Palm\novell\Users\James Dean" src-entry-id="33151“>

<add-attr attr-name="Surname" timestamp="1010424096#15">

<value timestamp="1010424096#15" type="string">Dean</value>

</add-attr>

<add-attr attr-name="Given Name" timestamp="1010424096#17">

<value timestamp="1010424096#17" type="string">James</value>

</add-attr>

<add-attr attr-name="CN" timestamp="1010424096#28">

<value timestamp="1010424096#28" type="string">James Dean</value>

</add-attr>

</add>

</input>

</nds>

Sample trace log – Subscriber (cont…)

IDM now takes that add Document and sends it down the Subscriber channel, processing each polcy:

874fe160:217 AD ST: Subscriber processing add for \PALM\novell\Users\James.

874fe160:217 AD ST: Applying object matching policies.

874fe160:217 AD ST: Applying policy: Matching.

874fe160:217 AD ST: Applying to add #1.

874fe160:217 AD ST: Evaluating selection criteria for rule 'Mirrored Match: Users - setup'.

874fe160:217 AD ST: (if-class-name equal "User") = TRUE.

874fe160:217 AD ST: (if-src-dn in-subtree "novell\users") = TRUE.

874fe160:217 AD ST: Rule selected.

874fe160:217 AD ST: Applying rule 'Mirrored Match: Users - setup'.

874fe160:217 AD ST: Action: do-veto-if-op-attr-not-available("Full Name").

874fe160:217 AD ST: Policy returned:

422DEBD9:365:874fe160:217 Active Directory basic ST:

(

Sample trace log – Subscriber (cont…)

DXCMD

Command line tool for managing IDM

Command-line mode and an interactive, menu-driven mode

Will work with DirXML 1.x as well as IDM 2.0.1

Access to all DirXML subverbs

Can perform common driver management tasks

Command to launch

Windows: \Novell\Nds\dxcmd.bat

NetWare: sys:\system\dxcmd.ncf

UNIX: /usr/bin/dxcmd

Designer

Client based tool for designing, debugging, deploying, and documenting your identity management solutions

Obtain at http://www.novell.com/coolsolutions/dirxml/designer/

Policy Simulator

Policy Builder

XML/XSL Editor

Document Generation

IDE for IDM

Demo's in the Solutions Lab

Question and Answer

General Disclaimer

This document is not to be construed as a promise by any participating company to develop, deliver, or market a product. Novell, Inc., makes no representations or warranties with respect to the contents of this document, and specifically disclaims any express or implied warranties of merchantability or fitness for any particular purpose. Further, Novell, Inc., reserves the right to revise this document and to make changes to its content, at any time, without obligation to notify any person or entity of such revisions or changes. All Novell marks referenced in this presentation are trademarks or registered trademarks of Novell, Inc. in the United States and other countries. All third-party trademarks are the property of their respective owners.

No part of this work may be practiced, performed, copied, distributed, revised, modified, translated, abridged, condensed, expanded, collected, or adapted without the prior written consent of Novell, Inc. Any use or exploitation of this work without authorization could subject the perpetrator to criminal and civil liability.

This means that no work is lost with either the Novell Nsure UDDI corporate or testing solution.

Enterprises that build their UDDI solutions with Novell Nsure UDDI Server, are not only leveraging the deep experience Novell Identity Management foundation brings to the standard, they are securing their own future by building on an open standard implementation that will grow and scale with their corporate needs.

2003-12-08T12:43:39