FedEx:
Protecting Clustered Data on NetWare,
SLES and OES

Trey Ray	Chris Stewart

Technical Support Advisor	Technical Support Advisor

FedEx	FedEx

Winston W. Hait

Senior Marketing Engineer

Syncsort, Inc.

Today’s Agenda

Backup Express Innovations with Novell

Syncsort Incorporated

Backup Express, Data Protection and Business Continuance including Clustering

FedEx Case Study

Cluster Backup and Restore Challenges and Criteria

Features of TSAFS

BrainShare Demo description

Backup Express Innovations with Novell

First data protection application to support:

cluster failover

SCSI reserve / release

device persistency

GroupWise TSA (GWTSA)

TSAFS

First LIVE fail-over demo of a Fibre Channel-based cluster backup

- BrainShare 2003

First LIVE fail-over demo of an iSCSI-based NetWare cluster backup

- BrainShare 2004

First LIVE fail-over demo of an iSCSI-based OES cluster backup

- BrainShare 2005

The Performance Software Experts

Backup Express

Top service & support

Selected by 96% of Fortune 100

Established in over 12,000 customer sites

Over 30 years of high performance software development

Backup Express: Data Protection and Business Continuance

Adapts to Evolving Networks

Single catalog for the entire network

Backup to Tape or Disk – NetWare, OES, SLES, Unix, Windows, NAS

Backup Express three tier architecture

Jukebox

Unix, Linux

or Windows

Unix /

Linux

Win XP

Win 2003

Master Server

SLES/OES

M/S Exchange

GroupWise

NAS

NetWare/OES

How Backup Express Works in a Cluster

Backup Express provides automatic failover and task retry support so backups are uninterrupted. Without cluster support, the job will fail.

Backup Express is cluster aware and displays the shared resources appropriately in the cluster. This simplifies defining backup jobs.

VOL3:

VOL4:

Tape Library

FC Switch

Vnode CLUS

Node “Right”

Node “Left”

FedEx: Protecting NW Cluster Data

A 3-phase implementation of NetWare Clusters for

Data Protection

FedEx: Protecting NW Cluster Data

Who is FedEx?

FedEx is the world's largest express transportation company, providing fast and reliable delivery to every U.S. address and to more than 220 countries and territories. FedEx Express uses a global air-and-ground network to speed delivery of time-sensitive shipments, usually in one to two business days with the delivery time guaranteed.

Who is the FedEx Air Operations Division?

The Air Operations Division is the airline component of FedEx and is comprised of 9,000 employees at hundreds of locations around the world

4,000 Pilots

2,000 Aircraft Maintenance Technicians

3,000 Administrative and Support Staff

FedEx: Protecting NW Cluster Data

Phase 1 – Adoption (2002)

Opportunity

Newly installed 4-node NW 5.1 cluster (one of first in production)

Current backup solution not working

Need: one that can ease burden and “follow the data”

Configuration

One 4-node NW 5.1 cluster, two 2-node NW 5.1 clusters

One Dell 136T Tape library with 2 LTO-1 drives

One Dell 130T Tape library with 2 DLT7000 drives

Dell SAN with Dell storage on both

FedEx: Protecting NW Cluster Data

Dell 130T

Drives: 2- DLT 7000

Slots: 30

Dell 136T

Drives: 2- LTO-1

Slots: 60

Dell SAN
Storage

Backup Express Win2K Master Server

4-node NetWare 5.1 cluster

2-node NetWare 5.1 cluster

2-node NetWare 5.1 cluster

Phase 1

Dell SAN
Storage

FedEx: Protecting NW Cluster Data

Phase 2 – Expansion (2003)

Opportunity

Move from stand alone servers to 2-node cluster for fault tolerance and redundancy

Greater data coverage: both volume and geography

Need centralized control

Configuration – Adding to Phase 1 configuration:

seven remotes sites with 2-node NetWare 6.0 cluster plus three 2-node Windows clusters

One Dell 136T tape library with LTO-1 drives and 60 slots

Be able to protect the data on NetWare and Windows Servers on the LAN through the 6 clusters

FedEx: Protecting NW Cluster Data

Windows
Servers

NetWare
Servers

7 Remote Sites each with a 2-Node NetWare cluster with local tape devices using a single, central catalog at the Main Office

Dell SAN
Storage

2-Node NetWare Cluster

4-Node NetWare 6 Cluster

2-Node NW 6 Cluster

2-Node NW 6 Cluster

2-Node MS Cluster

2-Node MS Cluster

Dell 130T

Drives: 2- DLT 7000

Slots: 30

Dell 136T

Drives: 2- LTO-1

Slots: 60

Dell 136T

Drives: 4- LTO-1

Slots: 60

Win2K Master Server

Phase 2

2-Node MS Cluster

FedEx: Protecting NW Cluster Data

Phase 3 – Recognition (2004)

Opportunity

World Headquarters doing server consolidation and installing new SAN with 4-node NetWare 6.5 cluster

Over 1 TB of shared storage

They knew success that Air Operations had and asked for presentation about how it was done

Ask for recommendations on design, installation and protection

Configuration

One 4-node NW 6.5 cluster, all new Dell SAN including servers, storage, switch

Dell 136T Tape library with 4 LTO-2 drives and 60 slots

FedEx: Protecting NW Cluster Data

Dell SAN
Storage

Dell 136T

Drives: 2 - LTO-2

Slots: 60

Backup Express Win2K Master Server

4-node NetWare 6.5 cluster

Phase 3 - World Head Quarters

FedEx: Protecting NW Cluster Data

Benefits Achieved from using Backup Express

Technical Support that is second to none

Ease of Install

Ease of Use

Media Handling - Good media management (i.e. Daily, weekly, and monthly)

Fail Over Handling - Backup Express completes no matter what the resource state (Failed or Normal)

FedEx: Protecting NW Cluster Data

Benefits Achieved from using Backup Express

Media Library Support – Library support for the Dell 136T, 130T and 120T that are utilized in the Air Operations Complex

Speed of Backup – Completely remained within the Fibre of the rack (Utilization of the shared Fibre resources)

No Customer Interruptions – Allows daily maintenance without intruding on our customers at any business hour (Server abends)

Remains independent of of peer cluster nodes

Supports all platforms used within FedEx

FedEx: Protecting NW Cluster Data

Future Plans

Recently deployed Novell Nterprise Linux Services (NLS) on SUSE Linux Enterprise Server 8 (SLES8) for hosting iFolder and NetStorage services.

Backing up the NNLS server with Syncsort’s Backup Express to Dell 136T library connected to 4-node NetWare cluster.

Will evaluate Novell's Open Enterprise Server (OES) and weigh pros and cons of moving to Linux kernel.

Upgrade to new Backup Express version on all 11 clusters

Access catalog for restores while backups are taking place

Cluster Backup & Restore Challenges and Criteria

Cluster enabled volumes can be on any node at any time

Backup Needs To:

Follow resource to any node

Migration can occur before, during or after backup

Ensure each volume gets backed up only one time

No data doubling or missed volumes

Define one backup definition to get resource on any node

Obtain proper base / incremental / differential

Restore Needs To:

Automatically know where resource was hosted in order to restore

Be able to restore to original location, without user knowing original or current volume location

Backup Express Cluster Backup Features

Backup Express is a fully cluster aware application

BEX heartbeat NLM

Each clustered resource is backed up once, even if migration occurs at any time

Backup to disk or tape / LAN or SAN

LAN free backup with Dynamic Tape Drive sharing in SAN

OES / NetWare Cluster Configuration with Backup Express

Backup Express client on all physical cluster nodes

Cluster client (cmagent) runs on MASTER cluster node

Backup Express brings up cluster agent on master node

Backup Express creates a Virtual Node (Master Cluster Node) -provides single view of all Virtual Servers

Simple configuration with “Cluster Lookup” function

Automatic or manual “scan” tool is available for physical and virtual nodes – adds nodes to Backup Express configuration

GUI is “Cluster Aware” and represents the Virtual Node with the cluster enabled volumes under it

GUI will show n+1 cluster nodes

Physical View of an OES/NetWare Cluster

VOL3:

VOL4:

Tape Library

SYS:

VOL1:

SYS:

VOL1:

Local Disk

Heartbeat Cable

Node
“Left”

Node “Right”

FC Switch

Logical View of an OES/NetWare Cluster

VOL3:

VOL4:

VOL1:

SYS:

VOL1:

Local Disk

LAN Connection

Node
“Left”

Node “Right”

Node “CLUS”

SYS:

Virtual Node

Backup Express Screen Shot

Backup Express Architecture Backup Process

Local Physical volume backups

Connect to TSA on physical node and backup VOLUME

Data mover runs on this physical node

Cluster Enabled Volume backups

Connect to Backup Express Virtual Node

Cluster API provides physical host of the Virtual Server

Connect to TSA on physical node hosting the cluster enabled volume

Data mover runs on this physical node

Backup During Cluster Migration / Failover

Cluster volume may be hosted anywhere

Backup will ultimately succeed even if the resource migrates multiple times during backup

Cluster volume migration to a new node causes task retry

Node failure on physical node will cause task retry

Data will backup through new physical node

Backup Express Flexibility with Cluster Volume Backups

Multiple cluster volumes can backup in parallel to multiple tape devices to increase performance

LAN free backup with Dynamic Tape Drive Sharing

SAN Resource Sharing (SRS)

SCSI reserve / release in SAN

Protects devices from being reset when in use

Implements Floating Tape Method

Payne & Wipfel (2002). Novell's Guide to Storage Area Networks and Novell Cluster Services

Backup Express Flexibility
with Cluster Volume Restores

Define cluster volume restore through virtual node

No need to know where volume was hosted during backup

No need to know where volume is during recovery

Data can be restored to original location or new location, with one TSAFS exception

Cluster Support Summary

Cluster enabled volumes can be on any node at any time

With Backup Express, backup will:

Follow resource to any node

Migration can occur before, during or after backup

Ensure each volume gets backed up only one time

No data doubling or missed volumes

Define one backup definition to get resource on any node

Obtain proper base / incremental / differential

With Backup Express, restore will:

Automatically know where resource was hosted in order to restore

Be able to restore to original location, without user knowing original or current volume location

Cluster Support Summary

Cluster Support for OES, SLES, Netware 5.x & 6.x, Windows and Unix

Simple Configuration with “Cluster Lookup” Function

GUI is “Cluster Aware” and represents the Virtual Node with the shared resources under it (clustered disks, SQL Cluster, Exchange Cluster)

If failover occurs during a backup job, the task will fail, but will automatically retry and succeed on another physical node.

Application support including GroupWise, MS Exchange, SQL Server, Oracle, etc…

Configurable retry time for failover

Features of TSAFS

Supported on NetWare, SLES and OES

Replaces GWTSA

File level backup/restore

GroupWise database lock/backup/unlock

Timestamping of user databases

Latest file system code and related development/support resources

Features of TSAFS

Cluster support - job continue on failover (don't have to restart job from beginning)

Cluster support - use cluster name rather than node name (facilitates smooth failover/resume)

Performance/read-ahead - Take advantage of higher-end hardware

Backup Express BrainShare Demo

Master Server

SYS:

VOL1:

NW-based OES Cluster
Node1

SLES-based OES Cluster Node2

FC Switch

SLES-based OES Cluster Node1

iSCSI Target and Source Disk

iSCSI-based cluster failover during back up

Backup Express, Data Protection and Business Continuance

Storage Area Network Support

Dynamic drive sharing (SRS)

Broad Platform Support

FC and iSCSI Support

LAN-free protection and recovery

Single M/S for SAN and LAN

Certified with leading SAN vendors

NAS

SLES/OES, Unix, Linux

Windows

Cluster

RAID

Tape Library

SAN

SAN

LAN

FC Switch

OES / NetWare Cluster

Clients on the LAN can be backed up to the tape library in the SAN through any of the SAN Device Servers

Linux/SLES/

OES, Unix

NetWare/OES

Cluster

Windows

RAID

Tape Library

LAN-100 BaseT Ethernet

SAN Device Server

SAN Device Server

SAN Device Servers

LAN Clients (5)

FC Switch

Storage Area Network Support

Backup Express, SANs and OES Clustering

Database and Application Interfaces

Novell GroupWise 6.x

eDirectory 8.7 and higher

NetWare 6.5 SnapShot Support

Syncsort Open File Manager

Oracle 8.x, 9.x and 10.x and 10g with RMAN (32 and 64-bit)

SAP R/3 Certified for Oracle, DB2 and SQL Server

Microsoft Exchange 5.5, 2000, 2003

Microsoft SQL Server 6.x, 7.x, 2000

Microsoft Share Point Server 2001 and 2003

Lotus Notes / Domino Server 5.x and 6.x

Sybase Backup Server 11.5 and higher (32 and 64-bit)

IBM DB/2 32- and 64-bit versions using Universal Database EE/EEE 6.3+

Pre- and Post-Scripting for automated backup of applications

Live demo of iSCSI-based mixed OES cluster failover during back up

Master Server

SYS:

VOL1:

NW-based OES Cluster
Node1

SLES-based OES Cluster Node2

GigE Switch

SLES-based OES Cluster Node1

iSCSI Target and Source Disk

See it live in

booth #P500

Summary

Backup Express Innovations with Novell

Syncsort Incorporated

Backup Express, Data Protection and Business Continuance

FedEx Case Study

Cluster Backup and Restore Challenges and Criteria

Features of TSAFS

BrainShare Demo description

Mr. Winston W. Hait

Senior Marketing Engineer

Syncsort, Inc.

201-573-5179

whait@syncsort.com

www.syncsort.com

Thank You!

Mr. Trey Ray

Technical Support Advisor

FedEx

trey.ray@fedex.com

www.fedex.com

Mr. Chris Stewart

Technical Support Advisor

FedEx

chris@fedex.com

www.fedex.com

Benefits of the solution

- Saved money

- Recognition of corporate

- Able to restore lost server the same day

- True NetWare cluster support

- Leverage distributed architecture

- Centralized management, even with 9 remote sites

- Easy to configure / use / manage

- Complete LAN-Free backup of NetWare 6 and Windows clustered servers

- Great technical support - there when we needed a person

Things that the GWTSA does that the TSAFS does not do

Run on NetWare 5.x and 6.x

Expose GroupWise Post Office, Domains, and Libraries as high-level backup objects

Multiple pass of OFFILES

Multiple pass of GWDMS directories (prevents documents from being orphaned)

Filtering/ignoring certain files

Library location

Cluster volume may be hosted anywhere

Backup will ultimately succeed even if the resource migrates multiple times during backup

Cluster volume migration to a new node causes task retry

Node failure on physical node will cause task retry

Data will backup through new physical node

Define cluster volume restore through virtual node

No need to know where volume was hosted during backup

No need to know where volume is during recovery

Data can be restored to original location or new location, with one TSAFS exception

Cluster enabled volumes can be on any node at any time

With Backup Express, backup will:

Follow resource to any node

Migration can occur before, during or after backup

Ensure each volume gets backed up only one time

No data doubling or missed volumes

Define one backup definition to get resource on any node

Obtain proper base / incremental / differential

With Backup Express, restore will:

Automatically know where resource was hosted in order to restore

Be able to restore to original location, without user knowing original or current volume location

Cluster Support for OES, SLES, Netware 5.x & 6.x, Windows and Unix

Simple Configuration with “Cluster Lookup” Function

GUI is “Cluster Aware” and represents the Virtual Node with the shared resources under it (clustered disks, SQL Cluster, Exchange Cluster)

If failover occurs during a backup job, the task will fail, but will automatically retry and succeed on another physical node.

Application support including GroupWise, MS Exchange, SQL Server, Oracle, etc…

Configurable retry time for failover

For TSAFS, no /home switches needed, but should load with /enableGW=yes

Cluster volume may be hosted anywhere

Backup will ultimately succeed even if the resource migrates multiple times during backup

Cluster volume migration to a new node causes task retry

Node failure on physical node will cause task retry

Data will backup through new physical node

Define cluster volume restore through virtual node

No need to know where volume was hosted during backup

No need to know where volume is during recovery

Data can be restored to original location or new location, with one TSAFS exception

Cluster enabled volumes can be on any node at any time

With Backup Express, backup will:

Follow resource to any node

Migration can occur before, during or after backup

Ensure each volume gets backed up only one time

No data doubling or missed volumes

Define one backup definition to get resource on any node

Obtain proper base / incremental / differential

With Backup Express, restore will:

Automatically know where resource was hosted in order to restore

Be able to restore to original location, without user knowing original or current volume location

Cluster Support for OES, SLES, Netware 5.x & 6.x, Windows and Unix

Simple Configuration with “Cluster Lookup” Function

GUI is “Cluster Aware” and represents the Virtual Node with the shared resources under it (clustered disks, SQL Cluster, Exchange Cluster)

If failover occurs during a backup job, the task will fail, but will automatically retry and succeed on another physical node.

Application support including GroupWise, MS Exchange, SQL Server, Oracle, etc…

Configurable retry time for failover

For TSAFS, no /home switches needed, but should load with /enableGW=yes

BUS 290 FedEx: Protecting Clustered Data on NetWare, SLES and OES.sxi

McKay Brown

2002-04-25T15:40:43

Winston Hait