Advanced Clustering with
SUSE® LINUX Enterprise Server 9

Tony Kubinski – Novell

Troy Keur - Novell

Linux Impact Team

Agenda

Market Overview

High Performance Computing – HPC

HPC Overview

Building a Validated HPC Configuration

High Availability Clustering

HA Fail-over - Active/Passive Fail-over Clusters

Geographically Disbursed HA Clusters

Data Center Clustering - Cluster File System (CFS)

Grid computing for the mission critical data center

Technology Demonstrations

Question and Answer

Novell Linux Framework

SUSE LINUX Enterprise Server

Novell Open

Enterprise

Server

Novell Small
Business Suite

SUSE LINUX

Retail Solution

Data

Center

Solutions

Data Center

Computing

Novell Linux

Desktop

Enterprise

Desktop

Workgroup

Computing

Small Business

Computing

Retail POS

Applications

High Performance

Compute Clusters

Massive

Computing

Solutions

Resource Management (ZENworks)

Identity Management

Market Overview

Value Proposition

Enabling performance, availability and manageability from industry standard servers and storage

What if you could drastically reduce IT expenses, and at the same time maintain or even improve performance, application availability, and systems management?

What is a Cluster?

In its simplest form, a cluster is two or more computers that work together to provide a solution.

The idea behind clusters is to join the computing powers of the nodes involved to provide more combined computing power, and/or to build in redundancy for high availability.

Clusters of computers must be somewhat self-aware, that is, the work being done on a specific node often must be coordinated with the work being done on other nodes.

This can result in complex connectivity configurations and sophisticated inter-process communications between the nodes of a cluster.

In addition, the sharing of data between the nodes of a cluster through a common file system is almost always a requirement.

There are many other complexities that are introduced by clusters, such as the operational considerations of dealing with a potentially large number of computers as a single resource.

Linux Clustering

Clustering is the new compute model for all server-based applications

There are now multiple types of clusters:

High Performance Technical Compute Clusters

Storage (NAS) Clusters

Simple Fail-over Clusters

Geographical Fail-over Clusters

Enterprise Data-Center Clusters

Linux is the leading platform for Cluster Computing

Novell has strong technology and offerings in all areas of clustering technology

High Performance Computing

What’s High Performance Computing (HPC)

HPC problems characterized by:

Computational, data-intensive, or numerically intensive tasks

Complex computations with large data sets requiring exceptionally fast throughput

Scientific Research

Geo

Sciences

Electronic Design

Automation

Finance

High-End Film

and Video

Life & Materials

Sciences

Product Lifecycle

Management

/ Informatics

Mechanical Engineering /

Virtual Prototyping

Government

Classified / Defense

High Availability Compared to High-Performance Computing

All clusters basically fall into two broad categories:

High availability

High-performance computing

High availability clusters provide:

Reliable services, focusing on rapid recovery methods in case of component or system failure, protecting applications and data

High-performance computing:

Provides greater computational power through compute clusters than one computer alone could

Parallel Computation

Parallel computation may be achieved in several ways:

The ability of many independent threads of control to make progress simultaneously toward the completion of a task, or

The simultaneous use of more than one computer to solve a problem

Parallelism is based on the connection and use of the memory subsystem

Flynn’s taxonomy classifies parallel computing as:

When the process executes the same instruction on multiple data at the same time

Called Single Instruction stream Multiple Data stream (SIMD)

When the processor executes different instructions on multiple data

Called Multiple Instruction stream Multiple Data stream (MIMD)

Strategy: “Divide and conquer”

Parallel-processing system divides a complex problem into smaller computer component tasks

Tasks are then assigned to the system’s nodes which tackle the components simultaneously

Efficiency of parallel processing depends largely on the nature of the problem

Parallel Computing Memory Architectures

Distributed Memory

Distributed-Shared

Memory

Distributed memory

Loosely coupled network of processors

Derived from the term message passing multiprocessors

Processors do not share a single address space

Processors communicate by

passing messages by way of an interconnection network

Difficult to program

Cannot share memory

Message passing is error prone

Typically scales better than a shared memory system

Shared memory

Small shared memory multiprocessors

Processors share a single address space

Processors communicate through memory

Shared Memory

P

P

P

P

P

P

P

M

P

Systems for High Performance Computing- “The Need for Speed”

HPC systems require:

Fast processors and large cache

Typically 2P/2GB for most application segments

Interleaved if possible; but not interleaved for the right price

At least 1 PCI Slot for high speed interconnect

Usually Linux based and ISV application support

Large sites have in house developed code

Unique networking for server/node communications

Most CAE Apps Ported to Linux Platforms

Dramatic Growth of Clusters

Source: IDC 2004

HPC CAGR
6.5%

Clusters CAGR
16.8%

Source: IDC 2004

What customers and users say?

Reasons for implementing clusters

Better Price/Performance

TCO (Total Cost of Ownership)

Ability to do new more/better science

Ability to run larger problems

Greater system throughput

Improved capacity management

To improve competitiveness

Challenges to implementing clusters

System management capability

Application availability/ Lack of maturity of the solution

Complexity of implementing parallel algorithms

Facility issues, space, power, cooling, noise

Interconnect Bandwidth/ Latency

I/O performance

Interconnect complexity/cable, cards, switches

3rd Party Software costs

Source: High Performance Technical Computing Cluster Multi-Client Study: End User Summary Report, IDC 2004

Build vs. Buy

Commercial tools tend to help you get more out of your cluster investment with less effort.

Management Tools

Variety of Open Source tools will get the job done with some effort.

Support should be 24/7.

Vendor eats mistakes.

Support

Support via Web, newsgroups.

You eat mistakes.

Vendors (should) have good knowledge of best practices.

Research & Expertise

You research your components, software.

Money

Time vs. Money

Time

Buy

Factor

Build

Novell Validated Configuration Program

Focused on reducing the deployment risk related to complex, multi-application, integrated stacks

Delivers fully tested, fully integrated reference architectures that solve enterprise business problems (Custom Validated Configs where appropriate)

Provides customers with the technical and business details necessary for customers to model and to accelerate similar solution deployments within their enterprise

Jointly developed and marketed with key ISV partners

VALIDATED 'BILL OF MATERIALS'

IDENTIFIED

ISSUES & RISKS

PERFORMANCE BENCHMARKS

DEPLOYMENT

BEST PRACTICES

DEVELOPMENT

BEST PRACTICES

SYSTEM

MANAGEMENT

BEST PRACTICES

COST METRICS

Initial Focus Areas:

High Performance Computing

High Availability

Virtualization

Systems Management and Monitoring

Application Hosting

Security & Identity

Novell Validated Configuration Program

compute middleware can be customized by the vertical need of the customer, a superset of the software is represented here

proposed: a slimmed down HPC version of SLES is included in this stack

customer applications will not be included in the stack as there is much variation based on customer need, certification SWAT efforts are underway

only one interconnect is required, leading options are shown

SuSE Linux Enterprise Server 9* - HPC Version

MVICH*

hardware

high-speed interconnect

operating system

compute middleware

x86 / Opteron / EM64T Server Platform

LAM/MPI*

MAUI*

OpenSSH*

OpenSSL*

OpenMP*

LUI*

SI Suite*

PBS*

PVM*

C3*

MPICH*

compilers

Opteron

Portland Group C, C++, Fortran

Intel

Intel C, C++, Fortran compilers

GNU Collection*

GNU C++, GCC, GNU Fortran (g77)

BLACS

SuperLU

WSMP

PETC

SCALAPACK

BLAS

Lustre*/PVFS* OpenGFS*/PVFS*

clustered file system

ATLAS

LAPACK

FFTW

Infiniband

Quadrics

GigE

MyriNet

Customer applications

(customized by vertical)

compute libraries

(customized by need)

application security

Application Firewalling / Class Protection

* = open source

Novell Validated Configuration Example
(Open Source HPC Stack)

SuSE Linux Enterprise Server 9 - HPC Version

hardware

high-speed interconnect

operating system

compute middleware

x86 / Opteron / EM64T Server Platform

EnFuzion

PBS - Pro

Platform

LSF

Meiosys

MetaCluster

Scali

Manage

NCAPI

ROCKS

SCALI MPI

compilers

Opteron

Portland Group C, C++, Fortran

Intel

Intel C, C++, Fortran compilers

GNU Collection*

GNU C++, GCC, GNU Fortran (g77)

BLACS

SuperLU

WSMP

PETC

SCALAPACK

BLAS

PolyServe Matrix Server

clustered file system

ATLAS

LAPACK

FFTW

Infiniband

Quadrics

GigE

MyriNet

Customer applications

(customized by vertical)

compute libraries

(customized by need)

application security

Application Firewalling / Class Protection

* = open source

Novell Validated Configuration Example
(Commercial HPC Stack)

Lustre

actual required drivers will be determined by the supported hardware/storage support matrix

proposed: a slimmed down HPC version of SLES is included in this stack

customer applications will not be included in the stack as there is much variation based on customer need, certification SWAT efforts are underway

only one interconnect is required, leading options are shown

Novell Validated Configuration Program

Novell | HP Cluster Alliance

MyriNet

Novell SuSE Linux Enterprise Server 9

servers

high-speed

interconnects

operating system

compute middleware

HP Opteron, EM64T, BladeSystem

compilers & libraries

Gig Ethernet

Infiniband

Scali

Manage and MPI Connect

Cluster Management

United Devices

Grid MP

Data Synapse

Altair

PBS Pro

Meiosys

MetaCluster

TurboWorx

PolyServe

Matrix Server

Processor Specific Compilers & Libraries

Industry Vertical Compute Libraries

end user applications

Fluent	Mentor	Ansys

Synopsys	Landmark	Tripos

fully customizable by vertical

all combinations fully tested and validated

pre-configured options available for:

Financial Services

Oil & Gas

Manufacturing / CAE

Life Sciences

EDA

addresses all HPC components including the hardware and the entire software stack (operating system, drivers, tools, and applications)

provides a tested and proven platform

customer ready near the end of Q2

available through HP and Novell channels

Novell Validated Configuration Program

MyriNet

Novell SuSE Linux Enterprise Server 9

servers

high-speed

interconnects

operating system

compute middleware

HP Opteron, EM64T, BladeSystem

compilers & libraries

Gig Ethernet

Infiniband

Scali

Manage and MPI Connect

Cluster Management

United Devices

Grid MP

Data Synapse

Altair

PBS Pro

Meiosys

MetaCluster

TurboWorx

PolyServe

Matrix Server

Processor Specific Compilers & Libraries

Industry Vertical Compute Libraries

end user applications

Fluent	Mentor	Ansys

Synopsys	Landmark	Tripos

fully customizable by vertical

all combinations fully tested and validated

pre-configured options available for:

Financial Services

Oil & Gas

Manufacturing / CAE

Life Sciences

EDA

addresses all HPC components including the hardware and the entire software stack (operating system, drivers, tools, and applications)

provides a tested and proven platform

customer ready near the end of Q2

available through HP and Novell channels

Novell Validated Configuration Program

HP Knows HPC

H1 CY2004 HPC by Vendor

IBM

30.3%

Sun

17.0%

Dell

7.5%

Other

6.9%

HP

31.5%

Hitachi

0.1%

Fujitsu

0.4%

Cray

0.9%

NEC

1.3%

SGI

4.1%

HP’s HPC Platforms

HP ProLiant DL140/DL145

HP ProLiant DL360 G4p
DL385
DL380

HP ProLiant DL585

HP ProLiant BL20p G3
BL25p
BL30p
BL35p

choice

performance

manageability

HP Integrity

rx2620

HP Integrity

rx1620

HP Integrity rx4640

HP Integrity
Superdome

HP Integrity rx7620

HP Integrity rx8620

HP Cluster

Platform 4000

HP Cluster

Platform 3000

HP Cluster

Platform 6000

PolyServe NAS Clusters Benefits

Scalable

Supports thousands of compute nodes

Delivers gigabytes-per-second throughput

Full cache coherency across NAS cluster

Fully symmetric

Scales both reads and writes

Scales random and sequential I/O

Scales file manipulation and data I/O

Easy to operate

Global namespace

Full high availability

Cluster-wide management

Easy to grow

Add servers for I/O throughput

Add storage to grow data footprint

PolyServe NAS Clusters—Hardware

NAS Cluster Servers

Up to 16 industry-standard Intel/AMD servers per file system

NAS Cluster Storage

Up to 1024 LUNs on FibreChannel storage arrays

Networks

One or more IP networks for intra-cluster communication

One or more IP networks for client-to-cluster file serving traffic (or combined)

Compute nodes

Anything that supports NFS

Scalable, Fault Tolerant NAS Services

Client IP Network

PolyServe NAS Clusters—Software

PolyServe Matrix Server

Enterprise-class shared data clustering software

Symmetric cluster file system

Integrated high-availability

Cluster-wide management

NAS Cluster layer atop Matrix Server

Cluster-wide exports

Single virtual server from client view

High-availability for NFS

Health monitor

Server failover

Leverages standard NFS server

Scalable, Fault Tolerant NAS Services

Client IP Network

PolyServe HPC NFS Write Performance

PolyServe HPC Server to Storage Write Performance

Cluster Size (Nodes)

Complexity of Clustering Software

IA32

Fast E

MPICH

PowerFlow

xCAT

StarCD

SCore

LAM

GbE

IA64

Fluent

CSM

MPICH-GM

Myrinet

Opteron

LS-Dyna

Scali

MPI PRo

InfiniBand

Hardware

Platform

Fluent

OSCAR

Scali

SCI

Hardware

Platform

RH 8.0

SLES 8

SLES 9

SuSE 8.2

RH 7.3

Reduced Complexity of Clustering

IA32

MPI

Cluster

Mgmt

IA64

Opteron

Hardware

Platform

Hardware

Platform

PowerFlow

StarCD

Fluent

LS Dyna

Fluent

Fast E

GbE

Myrinet

InfiniBand

SCI

Single, fully supported solution stack

Designed for performance, reliability and ease-of-use.

Profiled for specific application requirements.

SLES 9

Scali Manage

Scali MPI Connect

Scali MPI Connect™

Scali MPI Connect enables companies to
standardize on a single cross-enterprise MPI solution

Reduces complexity and TCO

Optimized for performance across full range of cluster sizes

Reduces the number of software builds needed to support multiple interconnects, hardware platforms and third-party applications

Dynamic link,
Common API

Runtime selection of
interconnect w/ failover

Applications

IBA

Ethernet

SMP

Myrinet

SCI

Scali's MPI

Clustering Tasks: a perspective from sys-admin and application users

Submit

Jobs

Scheduling

policies

Monitor

Job

Install

Configure

Rapid Deployment

Manage

Users &

Groups

Job

Queue

Mgt

Filesystem

Manage

System

Images

Tune

Kernel

Tuning

Deploy

New

Servers

Patch

Mgt

SW

Updates

Maintain & Adapt

Cluster

Health

Predictive

Diagnostics

Event

Cluster System Monitoring

Workload Management

Scali Manage Highlights

Enables single point of central management of cluster resources

Mix, match and merge across hardware platforms

Comprehensive functionality for rapid installation, cluster control, monitoring and fault handling, remote administration

Scales up > 1000 nodes

Complex
Environments

Enterprise-Class Installation and Configuration Flexibility

Multiple installation options

Express, Custom, Unattended

Includes automated installation of 3rd party apps and libraries

Processor Support

X86, EM64T, Opteron, Itainium

32, 32-64, 64 Bit support

Interconnect Support

Ethernet, Myrinet, Infiniband and SCI

Supported across all processors

Heterogeneous Configuration Support

Processors, OS’s, Kernels

Server form factors ( Blades and Servers)

Industry Standard Enterprise Linux distributions

Consolidation Support

Easily add into existing support

Enterprise-Class Monitoring Capabilities

Resource Utilization

CPU, memory, disk and I/O

Hardware Status

Temperature, fan speed, Interconnect

Interconnect & Network Status

Global view, status, performance, protocol, interfaces per node, input/output, auto routing/rerouting (SCI)

Work Load Status

Job queues and node activity

Integrates with LSF, PBS Pro

Graphical & text based interface

2D and 3D views

Compact, historic & compound views

SNMP Support and Remote Management

PBS Pro - The Portable Batch System

Flexible workload management and job scheduler

Unified interface to all computing resources

Supports all major UNIX, Linux (32 bit & 64 bit), Windows 2000 & Windows XP

Supports heterogeneous environment, SMP’s & clusters and parallel jobs (MPI)

Single interface handles both interactive and batch processing

GUI tools for users, managers and administrator

Computational Grid Support

Supports system-specific features (IBM SP PSSP, SGI Origin “cpusets”, HP MPI Integration Q4 2004)

POSIX batch standard

Fully configurable scheduler module & options

Fair-share, load balancing, priorities, back-filling, meta-scheduling, peer to peer

Scheduling

User
interface

Jobs

Commands

PBS Pro Functional Components

Batch
Server

Compute

Nodes

Scheduler

Site Specific
Policies for
Sharing

Resource
Monitor

Job
Execution
Manager

PBS Pro V5.4 has the Reliability, Availability and Serviceability features to satisfy production enterprise customers with massive computational Infrastructures

Major new RAS features in V5.4

PBS Pro Server/Scheduler Failover

Automatic Job Migration upon failures

Topology aware Scheduling

Default Directories are automatically created and permissions set at Install

PBS Pro Functional Components

Technology Demo

HPC Demo

A ProLiant Cluster

BL20P, BL30P Blades

DL360 Head Node

MSA1000 Storage Array

Software Configuration

SLES 9

Scali Manage/Connect

PBS Pro Job Queue Mgr.

PolyServe Matrix Server

Fluent CFD Application

High Availability on Linux

High Availability Clustering

Purpose: High-availability clusters are built with the intention of	providing a fail-safe environment through redundancy.

Architecture: Active / Active / Passive

There may be n nodes

For each application, one node is primary

A primary for one application may be backup for another

No two nodes access the same data set at the same time

Ownership of application data is transferred at failover time

client

client

Failed

App Server A

App Server B

Server A Application Data

Server B Application Data

Heartbeat Signaling

High Availability Clustering

HA Cluster Technology: What Happens

Heartbeat monitors application health

If monitor does not respond, failover is initiated

SCSI reserve/release used to migrate storage ownership

IP addresses moved from one node to another

Application restarted on backup node

Average failover time: 45 seconds to several minutes

Users: Typically aware that a failure has occurred

HA Offerings on Linux

HA-Linux*

Steeleye LifeKeeper -

Constant Data -

High Availability Clustering

Components High-Availability systems need a whole range of components in order to provide complete solutions covering every need.

These components are:

*Membership services

*Communication Services

*Cluster management

*Resource (I/O) fencing

*Resource Monitoring

*Storage Sharing/Replication Storage sharing can be accomplished as:

*Storage Sharing (shared SCSI, FDDI, etc.)

*Storage Replication

*Application protocol (DNS, NIS, etc.)

*rsync, etc.

*drbd, nbd, etc.

Not every deployment needs all these components typically.

High Availability Clustering
What Linux-HA can do today – built-in to SLES9

Code ("heartbeat") which implements serial, UDP, and PPP/UDP heartbeats together with IP address takeover including a nice resource model including resource groups. It currently supports multiple IP addresses and a simple two-node primary/secondary model. It is both extremely useful and quite stable at this point in time. The following types of applications are typical:

*Web servers

*LVS director servers

*Mail servers

*Database servers

*Firewalls

*File servers

*DNS servers

*DHCP servers

*Proxy Caching servers

client

client

Failed

App Server A

App Server B

Server A App Data

Server B App Data

Heartbeat Signaling

High Availability Clustering

open source HA stack

IBM

DB2 UDB

Apache

MySQL

cluster edition

Jboss cluster edition

Mono .Net Apps.

NFS / CIFS (for non-database stacks)

SuSE Linux Enterprise Server 9

X86 / Operon / EM64T Server Platform

Qlogic HBA

NIC

EMC PowerPath

SCSI

Zen Linux Management

Cluster Resources Mgmt. NCS on OES / Heartbeat

Clustered File System OCFS2 / GPFS / GFS

Dynamic Kernel

Module Support (DKMS)

hardware

drivers*

operating system

ha components

file services

Validated applications (one per stack)

“Open Source HA Stack”

HA platform layer

application layer

Geographically Distributed HA Clustering

Block Level Replication In Linux (DRBD)

SAN Based Replication

Complete Geo site fail-over

Protect against disasters

System available in seconds

Los Angeles

San Bernardino

Switches

Switches

Application Servers

SB Production Storage System

LA Duplicate Storage System

SB Duplicate Storage System

LA Production Storage System

Server A

Server B

Server D

Server C

NAS - Adaptive Resource Switches

Client Presentation

Enterprise-wide access point (global namespace)

Transparent storage provisioning and consolidation

File

Server

NAS

File Storage Resources

User Community

Data Distribution

Geographically distributed data placement for access and availability

Centralized management

Storage Management

Resource pooling

Capacity management

Heartbeat/DRBD based NAS

IP Network Infrastructure

WAN

WAN

File Server

NFS / CIFS

Linux-HA Success Stories

http://wiki.linux-ha.org/SuccessStories

DataCenter Clustering

Linux Data Center Clusters

The New Model for Mission Critical Data Center Architecture

Provides full HA like traditional failover clusters

Supports scale out compute farms

Adds storage virtualization

Allows both servers and storage to be pooled for fault tolerance and application performance

Allows multiple servers to be managed as a single logical cluster:

Shared application binaries

Shared Linux root file system

Shared databases

Shared identity management and security

High Availability

Application Monitoring

Application Failover

Cluster File System

All nodes can read and write all storage

Cluster Volume Management

Storage can be virtualized across all nodes

Identity Management

View the cluster as a single entity for authentication and security

Data Center Requirements
Tier One Linux Data Center Requirements

Applications

CFS

CVM

IDM

HA

SUSE Enterprise Linux

Backup

Storage System

Storage System

Shared Data

Allows all servers to “own” all data with speed & integrity

High Availability

Application Failover

Capacity Maintenance

Inserts & upgrades servers and storage as needed, with no service disruption

Application Management

Matches applications to required capacity - dynamically

Storage System

Storage System

Storage System

Storage System

Consolidated Storage SAN

High Availability

Application Management

Storage System

Storage System

Storage System

Storage System

Shared Data

Storage System

Storage System

Storage System

Storage System

Shared Data

Capacity Maintenance

The Clustered Data Center

What is Matrix Server?

Software that allows multiple servers to work together and access one pool of data concurrently

Consolidates storage management

Consolidates management of multiple servers

Integrated High Availability

Integrated application management

Matrix Server Major Components

Cluster File System (CFS)

All servers share read/write access to files with no separate meta-data server

High Availability (HA) services

Automatic application fail over/fail back

Cluster and application management

Applications can run on one or more servers on demand

Multi-path IO (MPIO) optional

No single point of failure, load balance, increase throughput

Central Management Console

Multi-path I/O

Cluster File System

High Availability Services

PolyServe Matrix Server

SUSE Linux Enterprise Server

Cluster and App Mgmt

Database High Availability

Where Does Matrix Server Play?

Matrix Server allows the mid-market to enterprise data center to reliably address four main needs:

Oracle RAC/HA deployments

Database and web server and storage consolidation

Scale up database High Availability needs

High performance, economical file and print serving

Oracle RAC deployments

Database consolidation

File and print serving

Matrix Server Benefits

Scale performance to meet needs

Up to 16 nodes with 1 – N CPUs/node

Fully distributed lock manager ensures linear scalability

2GB/s file serving throughput per array

Reduce management complexity

Installs in less than two hours

Management console controls all servers and storage access

Easily add servers and storage to a cluster on demand

Higher availability for files, databases and applications

Cost saving and ability to leverage industry standard products

Reduce three year TCO by > 25 – 35%

Reduce CAPEX and OPEX

Dramatically improve ROI, utilization on servers and storage

“With PolyServe Matrix Server, we can chain together less expensive Intel architecture–based servers and get more processing power at a much lower cost … than previous proprietary hardware. The PolyServe environment is just as manageable as a large UNIX server.”

Bill Ubelacker

Director of System Services

Customer Success:

Goals

Migrate entire data center from IBM/Sequent UNIX to Linux clusters

Migrate a variety of applications to Linux

40 mission-critical Oracle databases

Custom J2EE Java applications

Custom file-based applications

Solution

Multiple clusters of Oracle9i RAC

Oracle9i Application Server for J2EE

File-based applications scale out with shared data repositoryOracle9i RAC

Results

Lower TCO with better performance

Gained system management and availability comparable to UNIX

Scalability to keep pace with retail volume

Fault tolerance to guarantee uptime

GRID Computing

Grid computing enables the virtualization of distributed

computing and data resources such as processing, network

bandwidth and storage capacity to create a single system image,

granting users and applications seamless

access to vast IT capabilities.

From a business perspective this:

Improves price for performance

Improves flexibility and time-to-market

Improves robustness

Improves scalability

What is Grid Computing?

Basic Grid
Services

Resource Discovery

Uniform Data Access

Monitoring & Events

Portals

Resource brokering

Workflow management

--------

Fault management

Authorization

--------

Accounting

Data replication and metadata management

--------

Grid MPI

--------

CORBA, DCOM, …

Encapsulation as Web Services, as Script Based Services, as Java Based Services

Portals that are Web Services based, shell scripts, specialized

(e.g. high end vis workstations, PDAs)

Advanced Services

Applications

Architecture of a Grid

Scientific Instruments, Clusters, Pools of Workstations, Tertiary Storage, National Supercomputer Facilities

Distributed
Resources

Resource access
& functionality

Resource access
& functionality

Resource access
& functionality

Resource access
& functionality

Resource access
& functionality

Uniform
Computing Access

Resource
Scheduling

Visualization

Data analysis

--------

Data integration

--------

Collaboration tools

Authentication

Encapsulation as Web Services, as Script Based Services, as Java Based Services

space-based networks

optical networks

Internet

Identity Credentials

Communications

job initiation, event generators, GridFTP servers

Grid Services

Application Services

Grid Communication Functions (transport services, security services)

LAN/WAN

Operational Support

Complementary technologies to grid computing are emerging that will enable both deeper and broader grid usage such as;

Web services architectures imply loosely coupled collections of functionality rather than monolithic applications, which addresses challenges both in messaging complexity and in parallelizing applications***

Oracle 10g enables databases as well as applications to be run on grids eliminating potential I/O bottlenecks

Shared memory space tools support common memory across distributed servers addressing challenges both in messaging complexity and in parallelizing applications

Server virtualization tools allow applications to be deployed to servers in near real time, enable rapid expansion and re-configuration of grids. Individually and in combination, these technologies will enable a wide variety of applications to leverage grid platforms.

*** Insert Novell ExteNd Suite

Market Perspective - GRID / SOA

Grid and Web Services Standards Merge

Convergence of Core Technology Standards allows common base for business and technology services

Grid

OGSi

GT2

GT1

Web

HTTP

WSDL,

SOAP

WS-*

Have been

converging

WSRF

Started far apart in applications and technology

XML

BPEL

WS-I Compliant

Technology

Stack

Relationship:
SOA, Web services and Grid Computing

Services Oriented Architecture (SOA) is an architectural design style for business application systems.

Web Services is a set of standards that can be used to create a Service Oriented Architecture.

Grid Computing is a system management strategy whose goal is to maximize the use of hardware resources.

As SOA / GRID evolve we will see lead to support next gen technologies such as autonomic computing, self-managing systems and will serve as the basis for advanced capabilities in Web Services, such as the WS-Trust and Federated Identity Management.

Novell GRID Strategy

Complexity

The Evolution of Resource Virtualization

Privacy

Social

Political

Contractual Liability

Confidentiality

Example Novell Technology Opportunities

Cluster Services

Red Carpet / Zen

SUSE LINUX

NLS

(Security, Trusted Computing, RBAC) eDirectory

Identity Manager (SAML, Liberty)

Data Grids

Clusters

Devices

Departmental

Enterprise Grids

Global, Partner or B2B Grids

Evolution

Technology Demo

Linux HA File System Matrix

SLES9 Ready

Clustered FS

Comments

Oracle

OCFS2

Novell

NCS/NSS for Linux

Lustre

LustreFS

IBM

GPFS

InterMezzo

Veritas

Requires Metadata Server . approximately 100 commercially supported deployments

at the end of 2004, including 8 of the 16 largest Linux clusters in the world.

Stemmed from Coda. Strong ties to Lustre

GPFS is a solution to a requirement for a high-performance cluster file system with fault tolerance

characteristics. GPFS has been used for parallel processing applications, wide area file serving, non-database commercial applications, digital media/library applications and many others.

Open Source

Red Hat

GFS

Requires Metadata Server .

Requires Red Hat Linux. Based upon Sistina software.

n/a

Linux HA ISV Competitive Matrix

SLES9 Ready

Clustered FS

HA Monitoring

Fail-over

Data Replication

Scalability

( # of servers)

HA Toolkit

Storage Mgmt

Cluster-wide Mgmt

Multi platform

IA32 only

32

16

IA32 only

32

16

8

Linux – HA

HB, DRDB, LVS

SGI

Linux Failsafe

PolyServe

Matrix Server

SteelEye

LifeKeeper

Veritas

Cluster Server

Constant Data

Constant Replicator

Red Hat

Cluster Suite

We've spent months doing research on Linux in the data center. We've talked with customers all over the US and Europe.

We've validated our findings with reseller partners ISV's and IHV's then created a market response by articulating a data center solution strategy. We've socialized this strategy with the likes of:

- Ron Hosephian

- Bob Coutour

- Nancy Faigen

- Joe Wagner

- John Dragoon

- John Hogan

- Scott Lewis

- Brian Green

We talked with category specialists in the US and Europe. And, everyone agrees that this is the right strategy.

What Is High Performance Computing (HPC)

High Performance Computing is a division of computer science and represents compute- or data-intensive systems. It developed from various technologies and focuses on the creation of supercomputers and the applications that run on them. The need for an HPC environment grew as customers began to seek computers with:

Higher performance

More reliable platforms

Lower cost

HPC has traditionally placed three primary demands on system architectures:

Floating point computation (specifically 64-bit double-precision floating point)

Memory bandwidth

Memory size

Companies and institutions who use HPC continue to invest in HPC technologies. Most companies realize that they must continually invest in HPC to remain competitive even in difficult economic times. In 2001, HP did not see any significant cancellation of HPC purchases. Other HPC users are mostly government or government-funded institutions whose funding is not subject to economic conditions (though may be subject to political conditions). HPC systems save manufacturing companies millions of dollars by enabling them to virtually prototype their products before building physical models. In fact, more-and-more, physical prototypes are built only to validate the computer-generated models. In some cases, it is impossible to test potential designs except through virtual prototyping. Companies use HPC not only to design new products, but also to improve current designs.

HPC systems are also used to simulate very complex systems. For example weather forecasting is a very computationally intensive activity. Almost every government in the world has a meteorological center. Many military applications also require HPC systems. Lastly, major university and government research institutions will be engaged in research activities that require HPC systems, such as fundamental physics, genetics, drug discovery, and scientific engineering.

High Availability Compared to High-Performance Computing

All clusters basically fall into two broad categories:

High availability

High-performance computing

High availability clusters strive to provide extremely reliable services, focusing on rapid recovery methods in case of component or system failure.

High-performance computing is a cluster configuration designed to provide greater computational power than one computer alone could provide. High-performance computing is important because of its lower cost, compared to monolithic, proprietary solutions and because it is implemented in sectors where distributed parallel computing is needed to:

Solve large, scientific problems

Store and process large amounts of data

Parallel Computation

Following the Turing Machine concept, parallel computation, also called parallel computing or parallelism, is defined as:

The ability of many independent threads of control to make progress simultaneously toward the completion of a task, or

The simultaneous use of more than one computer to solve a problem

Both definitions are true because parallelism may appear in several ways.

Use of Memory

The definition of parallelism is based on the connection and use of the memory subsystem. Processors may either run independently or cooperate in solving a problem. In either case, they are usually under the control of another processor, which distributes the workload and collects results from them. Flynn’s taxonomy classifies parallel computing as:

When the process executes the same instruction on multiple data at the same time

Called Single Instruction stream Multiple Data stream (SIMD)

When the processor executes different instructions on multiple data

Called Multiple Instruction stream Multiple Data stream (MIMD)

The concept of parallelism is important to keep in mind when designing and selecting the components for a Beowulf compute cluster, which will covered in more detail shortly.

Parallelism Strategy

The strategy behind parallel-processing is to “divide and conquer”. A divide and conquer is a technique which splits a problem into two smaller problems whose solutions can be simply combined to solve the larger problem. This splitting is continued recursively until problems are so small that they are easy to solve. Tasks are then assigned to the system’s nodes which tackle the components simultaneously. The efficiency of parallel processing depends largely on the nature of the problem.

Parallel Computing Memory Architectures

Shared Memory

Shared memory is an attractive programming model for designing parallel and distributed applications. In a shared memory architecture, processors connect to a global memory store through an interconnect or bus. In this architecture, one processor is assigned to write data and another processor is assigned to read the data. The programming ease and portability of these systems cut parallel software development costs. The main disadvantage with the shared memory architecture is that bottlenecks can occur. Shared-memory multiprocessors typically suffer from increased contention and longer latencies in accessing the shared memory, which degrades peak performance and limits scalability compared to distributed systems.

Distributed Memory

In a distributed memory architecture, each processor contains local memory, and only memory that is directly connected to the processor can be accessed. In this architecture, the processors must communicate with each other, and one processor must receive memory from another processor. If the remote processor sending the memory is far away, data might be accessed slowly. Also, process migration imposes problems because of different address spaces. Compared to shared-memory systems, hardware problems are easier, and software problems more complex in distributed-memory systems.

Distributed-Shared Memory

A relatively new concept, distributed shared memory (DSM) combines the advantages of the two approaches. The DSM system hides the remote communication mechanism from the application writer, preserving the programming ease and portability typical of shared-memory systems. DSM systems allow relatively easy modification and efficient execution of existing shared-memory system applications, which preserves software investments while maximizing the resulting performance. In addition, the scalability and cost-effectiveness of underlying DSM systems are also inherited. Consequently, DSM systems offer a viable choice for building efficient, large-scale multiprocessors.

The face of the HPC market is changing. All HPC segments are demanding increased performance and price/performance, driving market growth via borader adoption of scale-out(cluster) computing. There is a bit of circular reasoning here – The better price of clusters is causing expectations on pricing to drop, while demands for higher performance without higher price go hand in hand. Bottom line, everyone wants more for less, the best way to do that is clusters – thus their broad adoption.

Clusters have become a major part of the overall business, and is driving the growth. In fact, cluster sales have been growing at 50% per year since 1999, according to IDC. They currently represent 30% of all HPC sales. By 2008, they are expected to account for a full 50% of the HPC market.

With the strategy and Unified Cluster Portfolio that we are announcing on November 8, HP is targetting our HPC cluster business to be $1 billion by 2006.

And HP is #1 in HPTC overall and in clusters, also the Top HPTC vendor in the world

Market share leader for HPTC and for Linux

Largest Linux Itanium cluster

Extensive, in-depth knowledge of clustering and its use for HPTC

CAE, Life Sciences, Materials, EDA, Science and Research

Close relationships with ISVs and developers

Expertise center: a proving ground for industry-standard technology

HP’s cluster solutions leverages outstanding portfolio of products and technology

Integrity, ProLiant servers

StorageWorks

Fully supported operating systems and middleware

FYI: Bright clusters are clusters bought from a vendor as clusters. This is an IDC definition. In contrast, a Dim Cluster is bought as a bunch of components not necessarily at the same time, and later assembled by a customer into a cluster.

A side point is that the lower entry cost for clusters is expanding the traditional supercomputing market into smaller customers, making supercomputing available to many more customers, which is driving the growth in the HPC market.

This info is actually from IDC Multiclient study, but confirms our experience in meeting customers, discussing their requirements. Does it map to what you see/hear?

Now that we have agreement on what the market is demanding, let’s look at how best to bridge the gap between the old HPC market and the new HPC market. We’ll also show how this change is a huge benefit to you in acquiring new customers and reducing your cost to application development and maintenance.

This is the state of our industry today, and it is a mess. ISVs like yourself have an ugly knot of different technologies to support in order to deliver basic capabilities to your customers. But are current technology stacks limit your market share, limits customer choice for using your products, and drives you into the very expensive processes of trying to support everything.

And supporting “everything” is a huge issue. One of our HPC partners notes that they have about 90 different variations of their products. Until they saw the light behind the standards-based HPC, their cost for maintaining all these different variations keeps them from adding new features and creating new value for their customers.

That is one of the enormous added values provided by Novell and Scali. By driving standards-based HPC into the market, we are providing customers with hardware platform choice and keeping they safe from hardware and technology lock-in. We also give them great flexibility in creating heterogeneous clusters with not only different servers, but different networking topologies. And if they ever decided to change either of those components, they don’t have to change their operation system, their MPI layer, their cluster management suite, or their application.

You have limited horizons because of the current alignment of hardware and middleware. Novell and Scali open new markets for you, and bring new customers to your door through standards-based HPC.

SMP - shared memory processing

MPI - message passing interface

POSIX - US government standards committee that sets standards for high performance computing

Scheduling options

fair-share - prevents monopolization of resources

back-filling - filling small gaps of capacity with small jobs

meta-scheduling - peer to peer (i.e.. grid, i.e.. “meta” by NASA) scheduling

ACL - Access Control List

Relate the three areas of workload management to the three daemons in PBS.

Show CAE computing problems and goals

Show basic principles of workload management

Show some building blocks (scheduling policy, automation) using two case studies

1. Form a team to focus on the Data Center Linux

Need the focus to enter this new market

Reaches all parts of the organization (sales, marketing, PBUs, consulting)

Define the data center product that sits above the distribution

Alliances (Application Certification - ISV’s, IHV’s)

Certify SUSE Linux on all major line of business and vertical applications

Provide testing environment for application certification

Expand development resources with key partners (Oracle, DB2 and others) for on-site consultation and development

Marketing/Consulting

Publish detailed white papers, migration practices, and benchmarks

Coordinate the delivery and implementation of open source management tools

Unix to Linux, and Windows to Linux documentation

Positioning Novell in the Data Center as the market leader

Determine correct licensing/pricing structure

Neutralize RedHat market momentum

Determine correct support structure

4. Sales

Different sales pitch and structure than in work group space

Training within partner sales organizations is critical

5. Security

Publish policies and best practices

Default to Off

Need to Add Ron's Edit's here

Problems with SANs without PolyServe:

No shared data

No shared binaries

High availability is hard because of SCSI reservation issues

Shifting workloads is cumbersome, causing poor server utilization

PolyServe realizes the potential of SANs and industry standard servers

Fundamentally changes the relationship between servers and storage and creates an “any to any” relationship

Allows servers as storage to be viewed as interchangeable but unified resources

Matrix Server maintains data integrity in the face of high speed, concurrent data reads and writes with little overhead

Matrix Server is built on top of a high availability infrastructure that absorbs server, storage, network, and application failures and eliminates, or minimizes, user impact

Since all servers can own all data with no fear of data corruption, access to data transitions quickly and certainly to designated back up servers

Matrix Server allows clusters to be comprised of multiple server types: Vendors, processor counts, clock speeds, memory compositions and Operating systems, providing maximum utilization of server capacity

Unlike some high availability solutions

Provides flexibility in moving applications between servers during peak and no-peak load situations

Because of the high availability infrastructure, and the data sharing and data integrity measures, server and storage capacity can be added to applications without affecting other servers, storage, or applications

Even the application to which it is being added if it is a scale-out application like file serving

With less than 90 seconds of disruption if it is not a scale-out application, like SQL Server

Matrix Server provides a Matrix-wide console showing the status of all servers in respect to applications running on servers, and the status of the servers

Applications can be shifted between servers with one click

Capacity can be added to applications with one click

2003-12-08T12:43:39