Overview of Grid Technology

Tony Hey and Geoffrey Fox

Originally given at CERN March 30-April 4 2003

Lectures and Streaming Video version available at
http://academia.web.cern.ch/academia/lectures/grid/
Given July 2003 by Geoffrey Fox
 Part 2

Overview of Web service and Grid

Technology
 What is a Grid I?
• Collaborative Environment (Ch2.2,18)
• Combining powerful resources, federated computing and a security
structure (Ch38.2)
• Coordinated resource sharing and problem solving in dynamic multi-
institutional virtual organizations (Ch6)
• Data Grids as Managed Distributed Systems for Global Virtual
Organizations (Ch39)
• Distributed Computing or distributed systems (Ch2.2,10)
• Enabling Scalable Virtual Organizations (Ch6)
• Enabling use of enterprise-wide systems, and someday nationwide
systems, that consist of workstations, vector supercomputers, and
parallel supercomputers connected by local and wide area networks.
Users will be presented the illusion of a single, very powerful
computer, rather than a collection of disparate machines. The system
will schedule application components on processors, manage data
transfer, and provide communication and synchronization in such a
manner as to dramatically improve application performance. Further,
boundaries between computers will be invisible, as will the location
of data and the failure of processors. (Ch10)
 What is a Grid II?
• Supporting e-Science representing increasing global collaborations of
people and of shared resources that will be needed to solve the new
problems of Science and Engineering (Ch36)
• As infrastructure that will provide us with the ability to dynamically
link together resources as an ensemble to support the execution of
large-scale, resource-intensive, and distributed applications. (Ch1)
• Makes high-performance computers superfluous (Ch6)
• Metasystems or metacomputing systems (Ch10,37)
• Middleware as the services needed to support a common set of
applications in a distributed network environment (Ch6)
• Next Generation Internet (Ch6)
• Peer-to-peer Network (Ch10, 18)
• Realizing thirty year dream of science fiction writers that have spun
yarns featuring worldwide networks of interconnected computers that
behave as a single entity. (Ch10)
 What is Grid Technology?
• Grids support distributed collaboratories or virtual
organizations integrating concepts from
• The Web
• Distributed Objects (CORBA Java/Jini COM)
• Globus Legion Condor NetSolve Ninf and other High
Performance Computing activities
• Peer-to-peer Networks
• With perhaps the Web being the most important for
“Information Grids” and Globus for “Compute Grids”
• Use Information Grids and not usual Data Grids as
“distributed file systems” (holding lots of data!) are
handled in Compute Grids
 Grid Politics
• There is a Global Grid Forum meeting 3 times per year with about
700 attendees per meeting
– Exchange information and define standards for “everything” not done in
W3C and OASIS
– e.g. Grid Service, Security, What is a Job, Database, Computer, How to build
portals ….
• There is a large project called Globus developing software largely
for “compute/file” Grids
• There are some 50 Grid projects (mainly in Europe and USA)
developing software and applications as well as installing
infrastructure
– Some are “deployment”: EDG NMI VDT …..
• There are related initiatives called CyberInfrastructure (NSF USA)
and e-Science (UK)
• There is a proposed OMII (Open Middleware Infrastructure
Institute) – an international Alliance of separately funded projects
with common coordination
 SERVOGrid Caricature
Repositories Sensor Nets
Federated Databases Streaming Data

Database Database

Analysis and
Loosely Coupled Visualization
Filters Closely Coupled Compute Nodes

Solid Earth Research Virtual Observatory

 SERVOGrid Grid Requirements
• Seamless Access to Data repositories and large scale
computers
• Integration of multiple data sources including sensors,
databases, file systems with analysis system
– Including filtered OGSA-DAI
• Rich meta-data generation and access with SERVOGrid
specific Schema extending industry standards
• Portals with component model for user interfaces and web
control of all capabilities
• Collaboration to support world-wide work
• Basic Grid tools: workflow and notification
 OGSA/OGSI Top Level View
http://www.gridforum.org/Meetings/ggf7/docs/default.htm
http://www.globusworld.org/globusworld_web/jw2_program_tut.htm

• OGSA is the set of

models
Domain -specific services

Other
“core” Grid services
More specialized services: data
– Stuff you can’t live replication, workflow, etc., etc.

Models for resources

& other entities
without Broadly applicable services: registry,
– If you built a Grid authorization, monitoring, data
access, etc., etc.
you would need to
Web Services and OGSI
invent these things
Hosting Environment
Network
 Taxonomy of Grid Functionalities
Name of Grid Type Description of Grid Functionality

Compute/File Grid Run multiple jobs with distributed compute and data
resources (Global “UNIX Shell”)
Desktop Grid “Internet Computing” and “Cycle Scavenging” with secure
sandbox on large numbers of untrusted computers

Information Grid Grid service access to distributed information, data and

knowledge repositories
Complexity or Hybrid combination of Information and Compute/File Grid
Hybrid Grid emphasizing integration of experimental data, filters and
simulations
Campus Grid Grid supporting University community computing

Enterprise Grid Grid supporting a company’s enterprise infrastructure

Note: Term Data Grid not used consistently in community so avoided

Taxonomy of Grid Operational Style
Name of Grid Style Description of Grid Operational or
Architectural Style
Semantic Grid Integration of Grid and Semantic Web meta-data
and ontology technologies
Peer-to-peer Grid Grid built with peer-to-peer mechanisms

Lightweight Grid Grid designed for rapid deployment and minimum

life-cycle support costs
Collaboration Grid Grid supporting collaborative tools like the Access
Grid, whiteboard and shared applications.

R3 or Autonomic Fault tolerant and self-healing Grid

Grid Robust Reliable Resilient R3
 Underlying Grid Principles
• We will start from the Web view and assert that
basic paradigm is
• Meta-data rich Web Services communicating via
messages
• These have some basic support from some runtime
such as .NET, Jini (pure Java), Apache
Tomcat+Axis (Web Service toolkit), Enterprise
JavaBeans, WebSphere (IBM) or GT3 (Globus
Toolkit 3)
– These are the distributed equivalent of operating
system functions as in UNIX Shell
• Called Hosting Environment or platform
 Some Basic Observations
• Grids manage and share asynchronous resources in a rather
centralized fashion
• Peer-to-peer networks are “just like” Grids with different
implementations of services like registration and look-up
• Web Services interact with messages
– Everything (including applications like PowerPoint will be a
WS?) – see later short discussion
• Computers are fast and getting faster. One can afford many strategies
that used to be unrealistic
– All messages can be publish/subscribe
– Software message routing
• XML will be used for most interesting data and meta-data
– One will store/consider data and meta-data separately but often
use same technology to manage both of them.
• Need Synchronous and Asynchronous Resource Sharing
– Integrate Grid and Collaboration technology
 Classic Grid Architecture Resources

Database Database

Content Access
Composition Middle Tier
Netsolve Brokers
Security Service Providers
Collaboration
Computing

Middle Tier becomes Web Services

Clients Users and Devices

 What is a Web Service I
• A web service is a computer program running on either the local
or remote machine with a set of well defined interfaces (ports)
specified in XML (WSDL)
• In principle, computer program can be in any language (Fortran ..
Java .. Perl .. Python) and the interfaces can be implemented in
any way what so ever
– Interfaces can be method calls, Java RMI Messages, CGI Web
invocations, totally compiled away (inlining) but
• The simplest implementations involve XML messages (SOAP)
and programs written in net friendly languages like Java and
Python
• Web Services separate the meaning of a port (message) interface
from its implementation
• Enhances/Enables Re-usable component model of ANY electronic
resource
Raw Raw Data Raw Data
Resources
(Virtual) XML Data Interface

Web Service (WS)

WS WS WS

XML WS to WS Interfaces etc.

WS WS WS WS WS

(Virtual) XML Knowledge (User) Interface

Render to XML Display Format

(Virtual) XML
Clients Rendering Interface
 What is a Web Service II
• Web Services have important implication that ALL
interfaces are XML messages based. In contrast
• Most Windows programs have interfaces defined as
interrupts due to user inputs
• Most software have interfaces defined as methods which
might be implemented as a message but this is often NOT
explicit
Payment
WSDL interfaces Credit Card

Security Catalog

Warehouse
WSDL interfaces shipping
 What is a Web Service III
• “Everything electronic” is a resource
– Computers; Programs; People
– Data (from sensors to this presentation to email to databases)
• “Everything electronic” is a distributed object
• All resources have interfaces which are defined in XML for both
properties (data-structure) and methods (service, function,
subroutine) (Resources are Services)
– We can assume that a data-structure property has
getproperty() and setproperty(value) methods to act as
interface
• All resources are linked by messages with structure, which must
be specifiable in XML
• All resources have a URI such as unique://a/b/c …….
 WSDL Abstractions
• WSDL abstracts a program as an entity that does
something given one or more inputs with its results
defined by streams on one or more outputs.
• Functions are defined by method name and parameters
methodname(parm1,parm2, … parmN)
– Where parameters are “Input” “Output” or both
• In WSDL, we will have a Web Service which like a
(Java or CORBA Program) can be thought of as a
(distributed) object with many methods
– Instead of a function call, the “calling routine” sends an XML
message to the Web Service specifying methodname and
values of the parameters
– Note name of function is just another parameter
 Details of WSDL Protocol Stack
• UDDI finds where programs are
UDDI or WSIL
– remote( (distributed) programs are
just Web Services WSFL
– (not a great success)
• WSFL links programs together WSDL

(under revision as BPEL4WS)

SOAP or RMI
• WSDL defines interface (methods,
parameters, data formats) HTTP or SMTP
• SOAP defines structure of message or IIOP or RMTP
including serialization of information
TCP/IP
• HTTP is negotiation/transport protocol
• TCP/IP is layers 3-4 of OSI Physical Network
• Physical Network is layer 1 of OSI
 Education as a Web Service
• Can link to Science as a Web Service and substitute educational
modules
• “Learning Object” XML standards already exist from IMS/ADL
http://www.adlnet.org – need to update architecture
• Web Services for virtual university include:
• Registration
• Performance (grading)
• Authoring of Curriculum
• Online laboratories for real and virtual instruments
• Homework submission
• Quizzes of various types (multiple choice, random parameters)
• Assessment data access and analysis
• Synchronous Delivery of Curricula
• Scheduling of courses and mentoring sessions
• Asynchronous access, data-mining and knowledge discovery
• Learning Plan agents to guide students and teachers
 What are System and Application Services?
• There are generic Grid system services: security, collaboration,
persistent storage, notification, workflow
– OGSA (Open Grid Service Architecture) is implementing these as
extended Web Services
• An Application Web Service is a capability used either by another
service or by a user
– It has input and output ports – data is from sensors or other
services
• Consider Satellite-based Sensor Operations as a Web Service
– Satellite management (with a web front end)
– Each tracking station is a service
– Image Processing is a pipeline of filters – which can be grouped
into different services
– Data storage is an important system service
– Big services built hierarchically from “basic” services
• Portals are the user (web browser) interfaces to Web services
 Application Web Services
Prog1 Prog2
• Filter1 Filter2
Note Service model Filter3sensor analysis, simulations
integrates sensors, and people
WS WS WS WS WS
• An Application Web Service is a capability used either by another service or by a
user Build as multiple
Build as multiple Filter Web Services
interdisciplinary
– It has input and output ports – data is from users, sensors or other services
– Big services built hierarchically from “basic” servicesPrograms

Sensor Data Simulation WS

as a Web
service (WS)

Data
Analysis WS

Sensor Visualization WS
Management
WS
 The Application Service Model
• As bandwidth of communication (between) services increases
one can support smaller services
• A service “is a component” and is a replacement for a library in
case where performance allows
• Services (components) are a sustainable model of software
development – each service has documented capability with
standards compliant interfaces
– XML defines interfaces at several levels
– WSDL at Service interface level and XSIL or equivalent for
scientific data format
• A service can be written as Perl, Python, Java Servlet, Enterprise
JavaBean, CORBA (C++ or Fortran) Object …
• Communication protocol can be RMI (Java), IIOP (CORBA) or
SOAP (HTTP, XML) ……
 Application with W3C DOM Structure as a Web Service
Resource Data
Facing Ports

Application as a Web service

MVC Application Model
M: Model Remaining W3C DOM Semantic Events

Control User Facing View

Ports
C:Control Events as Rendering as
Messages Messages

Selected W3C DOM Semantic Events

Application
V: View View
and W3C DOM Raw(UI) Events
Selected
Control W3C DOM User Interface
 7 Primitives in WSDL
• types: which provides data type definitions used to describe the
messages exchanged.
• message: which represents an abstract definition of the data
being transmitted. A message consists of logical parts, each of
which is associated with a definition within some type system.
• operation– an abstract description of an action supported by the
service.
• portType: which is a set of abstract operations. Each operation
refers to an input message and output messages.
• binding: which specifies concrete protocol and data format
specifications for the operations and messages defined by a
particular portType.
• port: which specifies an address for a binding, thus defining a
single communication endpoint.
• service: which is used to aggregate a set of related ports
<?xml version="1.0" encoding="UTF-8"?>
<wsdl:definitions>
<wsdl:message name="execLocalCommandResponse">
<wsdl:message name="execLocalCommandRequest">
<wsdl:portType name="SJwsImp">
<wsdl:operation name="execLocalCommand" parameterOrder="in0">
<wsdl:input message="impl:execLocalCommandRequest"
name="execLocalCommandRequest"/>
<wsdl:output message="impl:execLocalCommandResponse"
name="execLocalCommandResponse"/>
</wsdl:operation>
</wsdl:portType>
<wsdl:binding name="SubmitjobSoapBinding" type="impl:SJwsImp">
<wsdlsoap:binding style="rpc" transport="http://schemas.xmlsoap.org/soap/http"/>
<wsdl:operation name="execLocalCommand">
<wsdlsoap:operation soapAction=""/>
<wsdl:input name="execLocalCommandRequest">
<wsdl:output name="execLocalCommandResponse">
</wsdl:operation>
</wsdl:binding>
<wsdl:service name="SJwsImpService">
<wsdl:port binding="impl:SubmitjobSoapBinding" name="Submitjob">
</wsdl:service>
</wsdl:definitions>
 Discussion of 7 WSDL Primitives
• types specify data-structures which are equivalent to arguments of
methods
• message specifies collections of types and is equivalent to set of
arguments in a method call. Note that it is an “abstract method” in
Java terminology
• operation is a a collection of input output and fault messages; there
are 4 types of operation one-way(service just receives a message),
request-response(RPC), solicit-response, notification (services pushes
out a message)
• portType represents a single channel that can support multiple
operations. It is “abstract” as specified as a set of operations. It is
equivalent to a “interface or abstract class” in Java
• binding tells you transport and message format for a porttype (which
can have multiple bindings to reflect say performance-portability
trades)
• port combines a binding and an endpoint network address (URL) and
is like a “class instance”
• service consists of multiple ports and is equivalent to a “program” in
Java