Post on 13-Mar-2016
description
Arquiteturas Avançadas de SGBDs
• Por que processamento em paralelo?• Consultas 'paralelizáveis'• Formas de processamento em paralelo• Critérios de escolha de processamento em
paralelo• Solução Oracle p/ processamento em
paralelo• Conclusões
Por que processamento em paralelo?
• Contar os registros de um imenso arquivoSQL> SELECT degree, instances FROM user_tables WHERE table_name = 'Faturamento'; DEGREE INSTANCES 1 1
SQL> SET TIMING ONSQL> SELECT COUNT(*) FROM Faturamento;
COUNT(*) 12384004 Elapsed: 00:58:49.01
Por que processamento em paralelo?
• Contar os registros de um imenso arquivoSQL> ALTER TABLE Faturamento PARALLEL (DEGREE 8); Table altered.SQL> SELECT degree, instances FROM user_tables WHERE table_name = 'Faturamento'; DEGREE INSTANCES 8 1
SQL> SET TIMING ONSQL> SELECT COUNT(*) FROM Faturamento;
COUNT(*) 12384004 Elapsed: 00:08:13.23
CLIENT
QC
PQ1 PQ2 PQ3
QC - Query CoordinatorPQ - Parallel Query Slave
0 5 10 15 20
0 2
SEQUENCIAL
PARALELO
segundos
0.2 segundos
0 15minutos
0 2
SEQUENCIAL
PARALELO
horas
10 segs
10 segs
Requisitos de Consultas com Paralelização
• Resultados simples– Facilidade de integrar subresultados
• Subconsultas não-correlacionadas– Uma subconsulta não passa/recebe parâmetro
de outra (subconsultas independentes)• Subconsultas não-concorrentes
– Uma subconsulta não disputa recursos com outra
Formas de Processamento em Paralelo
• Consulta– "Table full scan"
• "Dynamic partitioning"– "Indexed range scan"
• "Static Partitioned table"• "Static Partitioned index"
– Consultas distribuídas• SGBD
– “Parallel query”– “Parallel server”– “Distributed DB”
Formas de Processamento em Paralelo
• Hardware Paralelo– "Uniprocessor system“, com “threads”– "Symetric multiprocessing (SMP) system"– "Massively parallel processing (MPP) system"
Hardware Paralelo
CPU
SYSTEM BUS
MEMORYI/O
CONTROLLERI/O
CONTROLLER
DISK DISK
Uniprocessor System
CPU
SYSTEM BUS
MEMORYI/O
CONTROLLERI/O
CONTROLLER
DISK DISK
CPU
SMP System
CPU
SYSTEM BUS
MEMORYI/O
CONTROLLERI/O
CONTROLLER
DISK DISK
NÓ
NÓ-1
NÓ-3 NÓ-4
NÓ-2
SYST
EM IN
TER
CO
NN
E CT
MPP
Sy s
tem
Hardware para consultas distribuídas
1.2 Terabytes ofDisk Storage16 Node MPP
Cost: $10M
1.2 Terabytes ofDisk Storage
4 Node MPPCost: $3M
300 GB ofDisk Storage
300 GB ofDisk Storage
300 GB ofDisk Storage
300 GB ofDisk Storage
4 SMP ServersCost: 4x$1M = $4M
TotalHardwareCost: $7M
MáquinaMPP
MáquinaSMP
MáquinaSMP Máquina
SMP
MáquinaSMP
Internet(Extranet)(Intranet)
Critérios de Escolha de Hardware para Processamento em Paralelo
Hardware “ParallelQuery”
“ParallelServer”
“Uniprocessor” Talvez Não
SMP Sim (a melhor opção)
Não
MPP Sim Sim
Critérios de Escolha de Hardware para Processamento em Paralelo
Hard-ware
Confia-bilidade
Disponi-bilidade
Gerên-cia
Desem-penho
Escala
“Uniprocessor”
Excelente Pobre Excelente Pobre Pobre
SMP Bom Bom Bom Excelente Bom
MPP Pobre Bom Pobre Bom Excelente
Solução Oracle p/ Consulta em Paralelo
Opção “Parallel Query”
Consulta em Paralelo: Full Table Scan
• Arquitetura SMP• Tabelas espalhadas em vários discos
Create tablespace exemploDatafile `datafile_on_disk1´ size 536879004,Datafile `datafile_on_disk2´ size 536879004,Datafile `datafile_on_disk3´ size 536879004,Datafile `datafile_on_disk4´ size 536879004 ...
• Grau de paralelismo > no. de CPUs– No. de “threads” por CPU
9/133/13
1/13
CLIENT
QC
PQ1 PQ2 PQ3
Full Table ScanDynamic Partitioning Bloco cheio
Bloco vazio
9/133/13
1/13
CLIENT
QC
PQ1 PQ2 PQ3
Full Table ScanDynamic Partitioning
ORACLESERVER
PROCESS(QC)
PARALLELQUERY SLAVE
PROCESS(PQ)
REQUESTS
DATA REQUESTS
DATAFull Table Scan: Dynamic Partitioning
QUERY
INSERT
TABLE 2
TABLE 1
QC
CREATE
Full Table Scan: Dynamic Partitioning
SET TIMING ONDELETEFROM faturamentoWHERE data_fat < ´01-JAN-96´;
1803773 rows deleted.Elapsed: 22:44:55.03
SET TIMING ONCREATE TABLE n_faturamentoPARALLEL (DEGREE 8)TABLESPACE fatSTORAGE (INITIAL 4M NEXT
4M PCTINCREASE 0)AS
SELECT *FROM faturamentoWHERE data_fat ≥ ´01-JAN-96´;
Elapsed: 00.50.03.43
RENAME faturamento TO ant_faturamento;
Elapsed: 00:00:01.01
RENAME n_faturamento TO faturamento;
Elapsed: 00:00:02.03
CREATE TABLE ... AS SELECT (pCTAS)CREATE INDEX
PQ3PQ2PQ1
high-water mark (HWM)
unused space
EXTENT 1 EXTENT 2 EXTENT 3 EXTENT 4 EXTENT 5
TEMPORARYSEGMENT
TEMPORARYSEGMENT
TEMPORARYSEGMENT
TABLESPACE SEGMENT
QC
P003P002P001
PQSlaveProcess
Local(Non/Prefixed, Prefixed)IndexPartitions
Table Partitions
Indexed Range ScanStatic Partitioned TableStatic Partitioned Index
Partitioned Table
CREATE TABLE Vendas(Mes NUMBER(6), Prod_id NUMBER, Loja_id NUMBER, Faturamento NUMBER, Custo NUMBER)STORAGE (...)PARALLEL (DEGREE 8)...
PARTITION BY RANGE (Mes)(PARTITION 01 VALUES LESS THAN (`199901´)
TABLESPACE Vendas_1999Jan,PARTITION 02 VALUES LESS THAN (`199902´)
TABLESPACE Vendas_1999Fev,...PARTITION 12 VALUES LESS THAN (`199912´)
TABLESPACE Vendas_1999Dec );
Partitioned Index
CREATE INDEX Mes-Loja ON Vendas (Mes, Loja_id)STORAGE (...)PARALLEL (DEGREE 8)...LOCAL
(PARTITION I-01 TABLESPACE I_1999Jan, PARTITION I-02 TABLESPACE I_1999Fev, ... PARTITION I-12 TABLESPACE I_1999Dez);
Index Range Scan
SELECT degree FROM user_tablesWHERE table_name = `Vendas´;
DEGREE 8
SELECT Loja_id, SUM(Faturamento)FROM Vendas WHERE Mes IN (`199902´, `199903´, `199904´)GROUP BY Loja_id;
Full Scan em Tabelas Fragmentadas
CREATE INDEX Loja_Mes ON Vendas (Loja_id, Mes)STORAGE (...)PARALLEL (DEGREE 8)...LOCAL
(PARTITION I-01 TABLESPACE I_1999Jan, PARTITION I-02 TABLESPACE I_1999Fev, ... PARTITION I-12 TABLESPACE I_1999Dez);
SELECT COUNT(*) FROM VendasWHERE Loja_id = 08;
PARTITION 3PARTITION 2PARTITION 1
QC
PQ3PQ2PQ1
QUERYQUERYQUERYUPDATE
ORDELETE
UPDATEOR
DELETE
UPDATEOR
DELETE
TABLE X
Indexed Range ScanStatic Partitioned TableStatic Partitioned Index
Consulta em Paralelo
Opção “Parallel Server”
AccountingDepartmentEnd-users
MarketingDepartmentEnd-users
Database Instance 1 Database Instance 2
Node 1 Node 2
Oracle Parallel Server Database
NODE 1 NODE 2 NODE 3
1 32 4 125 87 11106 9PQPQPQPQPQ PQPQPQ PQPQPQPQ
PARALLEL (DEGREE 4 and INSTANCES 3)
Table Full ScanSELECT degree FROM user_tablesWHERE table_name = `Vendas´;
DEGREE INSTANCES 1 1
SELECT COUNT(*)FROM Vendas;
COUNT(*) 12384004Elapsed: 00:58:49.01
SELECT degree, instances FROM user_tablesWHERE table_name = `Vendas´;
DEGREE INSTANCES 4 3
SELECT COUNT(*)FROM Vendas;
COUNT(*) 12384004Elapsed: 00:05:13.23
Parallel Cache Management
Instância A Instância B
2
Oracle Parallel Server Database
Attempt toobtain
EXCLUSIVELOCK
DLM transmitslock attempt to
all maching instances
Parallel Cache Management
Instância A Instância B
1. May Iexclusively
lock thisblock?
1
2
2. First, let mewrite my
changes to thedatafile
3OK, now
you may havethe lock
4
4. Read blockinto cache
Oracle Parallel Server Database
Consulta em Paralelo
BD Distribuído
AccountingDepartmentEnd-users
MarketingDepartmentEnd-users
Database Instance 1 Database Instance 2
Database 1 Database 2
Occasional
Occasional
Transactions
Transactions
Node 1 Node 2
abc
CLIENT
Oracle InstanceLOCL
Oracle InstanceRMOT
Select @RMOTSelect
DATABASE SERVER #2DATABASE SERVER #1
abc
def
A REMOTE QUERY
BD Oracle Distribuído
CREATE [PUBLIC] DATABASE LINK rmotCONNECT TO username IDENTIFIED BY passwordUSING `connect_string´;
CREATE PUBLIC SYNONYM projetosFOR projetos@rmot;
SELECT *FROM projetos;
abd
CLIENT
Oracle InstanceLOCL
Oracle InstanceRMOT
Select @LOCL,@RMOT
Select
DATABASE SERVER #2DATABASE SERVER #1
abc
def
A DISTRIBUTED QUERY
CREATE VIEW todos_os_projetos ASSELECT *
FROM projetosUNION ALLSELECT *
FROM projetos@europaUNION ALL...;
SELECT nome_projeto, data_inicioFROM todos_os_projetos;
BD Oracle Distribuído
Propriedades de uma Transação
• Transação Centralizada• Transação Distribuída• Propriedades ACID
– Atomicidade a Falhas (A)– Consistência (C)– Isolação (I)– Durabilidade (D)
a´b´
CLIENT
Oracle InstanceLOCL
Oracle InstanceRMOT
Update @RMOTUpdate
DATABASE SERVER #2DATABASE SERVER #1
a´b´c
def
A REMOTE TRANSACTION
a´b´d´
CLIENT
Oracle InstanceLOCL
Oracle InstanceRMOT
Update @LOCL,@RMOT
Updade
DATABASE SERVER #2DATABASE SERVER #1
a´b´c
d´ef
A DISTRIBUTED TRANSACTION
Protocolo de Validação de Transações Distribuídas
• Protocolo de Validação em Duas Fases
Standard Distributed Two-Phase Commit Protocol
Coordinator Logic
Local prepare. Invoke each local resource manager to prepare for commit.Distributed prepare. Send a prepare request on each of the transaction’s
outgoing sessions.Decide. If all resource managers vote yes and all outgoing sessions respond yes,
then durably write the transaction commit log record containing a list of participating resource maanagers and transaction managers.
Commit. Invoke each participating resource manager, telling it the commit decision. Send “commit” message on each of the transaction’s outgoing sessions.
Complete. When all local resource managers and all outgoing sessions acknowledge commit, write a completion record to the log indicating that phase 2 completed. When the complete record is durable, deallocate the live transaction state.
Standard Distributed Two-Phase Commit Protocol
Participant Logic
Prepare()Local prepare. Invoke each local resource manager to prepare for commit.Distributed prepare. Send prepare requests on the outgoing sessions.
Decide. If all resource managers vote yes and all outgoing sessions respond yes, then the local node is almost prepared.
Prepared. Durably write the transaction prepare log record containing a list of participating resource managers, participating transaction managers, and the coordinator transaction manager.
Respond. Send yes as response (vote) to the prepare message on the incoming session.
Wait. Wait (forever) for a commit message from coordinator.
Standard Distributed Two-Phase Commit Protocol
Participant Logic
Commit()Commit. Invoke each participating resource manager, telling it the commit decision.Complete. When all local resource managers and all outgoing sessions
acknowledge commit decision, write a completion record to the local log indicating that phase 2 completed.
Acknowledge. When the completion record is durable, send acknowledgement of commit to coordinator and deallocate local transaction state.
Servidores de Transações Distribuídas
• “Middle-tier” de Arquiteturas “Three-tier”• Padrão de Interoperabilibade de Processadores de
Transações Distribuídas– OSI-TP– OSI-CCR– X/Open DTP
• Produtos– Microsoft Transaction Server (MTS)– Enterprise Java Beans (EJB – Sun)
SQLSQLServerServerMSMQMSMQOracleOracle
UNIXUNIXCICS/CICS/IMSIMS
MTSMTSUpdateUpdate
AnalysisAnalysis
QueryQuery
Browsers Clients
PresentationPresentation
IIS/ASPIIS/ASPDCOMDCOM
Presentation
ApplicationLogic
Data andResources
Windows NT ServerWindows NT Server
Conclusões
• Estamos no limiar da explosão do hardware e do software paralelo
• Prepare-se para esta nova etapa da evolução das tecnologias de informática!