Rails Summit 2008 : Histórias de Guerra - Enfrentando problemas reais em aplicações Web 2.0

Uma história de guerra: Enfrentando problemas reais em aplicações Web 2.0

Manoel Lemos

Ronaldo Ferraz

Nando Vieira

Rails Summit Latin America 2008São Paulo, Brasil, 16.Outubro

1983

1984

1983

1984

1993

1983

1984 1994

1993

1983

1984 1994

1993

1997

1983

1984 1994

1993

1997

1999

1983

1984 1994

1993

1997 2000

1999

1983

1984 1994

1993

1997 2000

1999 2006

Abr.06

Abr.06

Nov.06

Abr.06

Nov.06

Fev.07

Abr.06

Nov.06

Fev.07

Fev.08

Abr.06

Nov.06

Fev.07

Ago.08

Fev.08

BlogBlogs Status

Business Numbers

110.000 usuários

200.000 blogs

2.000.000 posts

250 usuários / dia

150 blogs / dia

5.000 posts / dia

Traffic Numbers

5.2M PageViews

2.6M Uniques

Top 4000 Alexa Global

Top 120 Alexa Brasil

Top 102 IBOPE Brasil

50M Widget Impressions

Crescer é um processo complicado e doloroso!

É impossível de se prever com precisão

Existem gargalos escondidos

Às vezes você precisa voltar atrás

Às vezes você precisa mudar tudo

Não existem “silver-bullets”

A complexidade vai crescer com você

Crescimento real nunca acontece em um só eixo!

Tráfego : mais usuários, transações, imagens...

Tamanho : banco, memória, cache...

Complexidade : algoritmos, arquitetura, infraestrutura...

Organização : equipe, comunicação, processos...

Negócio : modelos, parceiros, fornecedores...

Na maioria dos casos os problemas crescem mais que linearmente!!!

Hoje1.5 Milhões

por dia

httpdmongrel

mysql

httpdmongrel

mysql

mongrelmongrel

httpdmongrel

mysql

mongrelmongrel

memcached

httpdmongrel

mysql

mongrelmongrel

memcached

httpdmongrelmongrelmongrel

memcached

load balancing

httpdmongrel

mysql

mongrelmongrel

memcached


memcached


memcached

load balancing


memcached


memcached


memcached

load balancing

mysql




load balancing

mysql

memcached memcachedmemcached




load balancing

mysql


mongrel mongrelmongrel

mysql




load balancing






load balancing 1 (web)



mysql










load balancing 2 (widgets)



mysql













mysql (slave)

mysql (master)







mysql







mysql (slave)

mysql (master)






mysql





memcached memcachedmemcached memcached memcachedmemcached


mysql (slave)

mysql (master)






mysql






mysql (slave)

mysql (master)

memcached memcached

memcached memcached






mysql






mysql (slave)

mysql (master)

memcached memcached

memcached memcached

mongrel mongrelmongrel mongrel mongrelmongrel






mysql






mysql (slave)

mysql (master)

memcached memcached

memcached memcached

Robot

Robot


Robot

Robot






mysql






mysql (slave)

mysql (master)

memcached memcached

memcached memcached

Robot

Robot


Robot

Robot

Indexer(Search)

Indexer(Search)






mysql






mysql (slave)

mysql (master)

memcached memcached

memcached memcached

Robot

Robot


Robot

Robot

Indexer(Search)

httpdmongrelmongrelmongrelmongrelmongrel

httpdmongrelmongrelmongrelmongrelmongrel

Indexer(Search)

load balancing 3 (API)


mysql


mysql (slave)

mysql (master)

memcached memcached

memcached memcached

Robot

Robot

mongrels

Robot

Robot

Indexer(Search)

Indexer(Search)

load balancing 3 (API)

mongrels mongrels mongrels mongrels mongrels

mongrels mongrels

load balancing

memcached

Robot

mongrels

Indexer(Search)

Indexer(Search)

mongrels

mongrels

mongrels

mongrels

mongrels

mongrels

mongrels

Robot

memcached

memcached

memcached

mysql (master)

mysql (master)

load balancing

memcached

Robot

mongrels

Indexer(Search)

Indexer(Search)

mongrels

mongrels

mongrels

mongrels

mongrels

mongrels

mongrels

Robot

memcached

memcached

memcached

mysql (master)

mysql (master)

load balancing

memcached

Robot

mongrels

Indexer(Search)

Indexer(Search)

mongrels

mongrels

mongrels

mongrels

mongrels

mongrels

mongrels

Robot

memcached

memcached

memcached

mysql (master)

mysql (master)

load balancing

memcached

Robot

mongrels

Indexer(Search)

Indexer(Search)

mongrels

mongrels

mongrels

mongrels

mongrels

mongrels

mongrels

Robot

memcached

memcached

memcached

mysql (master)

mysql (master)

load balancing

memcached

Robot

mongrels

Indexer(Search)

Indexer(Search)

mongrels

mongrels

mongrels

mongrels

mongrels

mongrels

mongrels

Robot

memcached

memcached

memcached

mysql (master)

mysql (master)

Eureca...Idéias...Single Sign On

Shared ResourcesIntegrated Search

E=MC2

load balancing

memcached

Robot

mongrels

Indexer(Search)

Indexer(Search)

mongrels

mongrels

mongrels

mongrels

mongrels

mongrels

mongrels

Robot

memcached

memcached

memcached

mysql (master)

mysql (master)

load balancing

memcached

Robot

mongrels

Indexer(Search)

Indexer(Search)

mongrels

mongrels

mongrels

mongrels

mongrels

mongrels

mongrels

Robot

memcached

memcached

memcached

mysql (master)

mysql (master)

load balancing

memcached

Robot

mongrels

Indexer(Search)

Indexer(Search)

mongrels

mongrels

mongrels

mongrels

mongrels

mongrels

mongrels

Robot

memcached

memcached

memcached

mysql (master)

mysql (master)

load balancing

memcached

Robot

mongrels

Indexer(Search)

Indexer(Search)

mongrels

mongrels

mongrels

mongrels

mongrels

mongrels

mongrels

Robot

memcached

memcached

memcached

mysql (master)

mysql (master)

E ai, #comofaz ?

load balancing

mongrels

mongrels

mongrels

mongrels

mongrels

mongrels

sinatra

mongrels

load balancing

mongrels

mongrels

mongrels

mongrels

mongrels

mongrels

mongrels

mongrels

load balancing

mongrels

mongrels

mongrels

mongrels

mongrels

mongrels

sinatra

mongrels

Robot

Robot

memcached

memcached

memcached

memcached

mysql (master)

mysql (slave)

load balancing

mongrels

Indexer(Search)

Indexer(Search)

mongrels

mongrels

mongrels

mongrels

mongrels

mongrels

mongrels

Robot

Robot

memcached

memcached

mysql (slave)

Indexer(Search)

Indexer(Search)

ActiveDNA (REST)

load balancing

mongrels

mongrels

mongrels

mongrels

mongrels

mongrels

sinatra

mongrels

Robot

Robot

memcached

memcached

memcached

memcached

mysql (master)

mysql (slave)

load balancing

mongrels

Indexer(Search)

Indexer(Search)

mongrels

mongrels

mongrels

mongrels

mongrels

mongrels

mongrels

Robot

Robot

memcached

memcached

mysql (slave)

Indexer(Search)

Indexer(Search)

ActiveDNA (REST)

ActiveDNA (REST)

load balancing

mongrels

mongrels

mongrels

mongrels

mongrels

mongrels

sinatra

mongrels

Robot

Robot

memcached

memcached

memcached

memcached

mysql (master)

mysql (slave)

load balancing

mongrels

Indexer(Search)

Indexer(Search)

mongrels

mongrels

mongrels

mongrels

mongrels

mongrels

mongrels

Robot

Robot

memcached

memcached

mysql (slave)

DNACredentials

(PHP)

Indexer(Search)

Indexer(Search)

ActiveDNA (REST)

ActiveDNA (REST)

load balancing

mongrels

mongrels

mongrels

mongrels

mongrels

mongrels

sinatra

mongrels

Robot

Robot

memcached

memcached

memcached

memcached

mysql (master)

mysql (slave)

load balancing

mongrels

Indexer(Search)

Indexer(Search)

mongrels

mongrels

mongrels

mongrels

mongrels

mongrels

mongrels

Robot

Robot

memcached

memcached

mysql (slave)

DNACredentials

(PHP)

Indexer(Search)

Indexer(Search)

DNAProfile(PHP)

ActiveDNA (REST)

ActiveDNA (REST)

load balancing

mongrels

mongrels

mongrels

mongrels

mongrels

mongrels

sinatra

mongrels

Robot

Robot

memcached

memcached

memcached

memcached

mysql (master)

mysql (slave)

load balancing

mongrels

Indexer(Search)

Indexer(Search)

mongrels

mongrels

mongrels

mongrels

mongrels

mongrels

mongrels

Robot

Robot

memcached

memcached

mysql (slave)

DNACredentials

(PHP)

Indexer(Search)

Indexer(Search)

DNAProfile(PHP)

DNAAvatars(PHP)

ActiveDNA (REST)

ActiveDNA (REST)

load balancing

mongrels

mongrels

mongrels

mongrels

mongrels

mongrels

sinatra

mongrels

Robot

Robot

memcached

memcached

memcached

memcached

mysql (master)

mysql (slave)

load balancing

mongrels

Indexer(Search)

Indexer(Search)

mongrels

mongrels

mongrels

mongrels

mongrels

mongrels

mongrels

Robot

Robot

memcached

memcached

mysql (slave)

DNACredentials

(PHP)

Indexer(Search)

Indexer(Search)

DNAProfile(PHP)

DNAAvatars(PHP)

DNAScoring(C++)

DNASocialGraph

(Python)

ActiveDNA (REST)

ActiveDNA (REST)

load balancing

mongrels

mongrels

mongrels

mongrels

mongrels

mongrels

sinatra

mongrels

Robot

Robot

memcached

memcached

memcached

memcached

mysql (master)

mysql (slave)

load balancing

mongrels

Indexer(Search)

Indexer(Search)

mongrels

mongrels

mongrels

mongrels

mongrels

mongrels

mongrels

Robot

Robot

memcached

memcached

mysql (slave)

DNACredentials

(PHP)

Indexer(Search)

Indexer(Search)

DNAProfile(PHP)

DNAAvatars(PHP)

DNAScoring(C++)

DNASocialGraph

(Python)

DNAMessaging

(Erlang)

DNAMessaging

(Erlang)

DNAMessaging

(Erlang)

DNAMessaging

(Erlang)

DNAMessaging

(Erlang)

ActiveDNA (REST)

ActiveDNA (REST)

load balancing

load balancing

load balancing

mongrels

mongrels

mongrels

mongrels

mongrels

mongrels

sinatra

mongrels

Robot

Robot

memcached

memcached

memcached

memcached

mysql (master)

mysql (slave)

load balancing

mongrels

Indexer(Search)

Indexer(Search)

mongrels

mongrels

mongrels

mongrels

mongrels

mongrels

mongrels

Robot

Robot

memcached

memcached

mysql (slave)

DNACredentials

(PHP)

Indexer(Search)

Indexer(Search)

DNAProfile(PHP)

DNAAvatars(PHP)

DNAScoring(C++)

DNASocialGraph

(Python)

DNAMessaging

(Erlang)

DNAMessaging

(Erlang)

DNAMessaging

(Erlang)

DNAMessaging

(Erlang)

DNAMessaging

(Erlang)

load balancing

load balancing

ActiveDNA (REST)

ActiveDNA (REST)

load balancing

mongrels

mongrels

mongrels

mongrels

mongrels

mongrels

sinatra

mongrels

Robot

Robot

memcached

memcached

memcached

memcached

mysql (master)

mysql (slave)

load balancing

mongrels

Indexer(Search)

Indexer(Search)

mongrels

mongrels

mongrels

mongrels

mongrels

mongrels

mongrels

Robot

Robot

memcached

memcached

mysql (slave)

DNACredentials

(PHP)

Indexer(Search)

Indexer(Search)

DNAProfile(PHP)

DNAAvatars(PHP)

DNAScoring(C++)

DNASocialGraph

(Python)

DNAMessaging

(Erlang)

DNAMessaging

(Erlang)

DNAMessaging

(Erlang)

DNAMessaging

(Erlang)

DNAMessaging

(Erlang)

load balancing

load balancing

ActiveDNA (REST)

ActiveDNA (REST)

load balancing

mongrels

mongrels

mongrels

mongrels

mongrels

mongrels

sinatra

mongrels

Robot

Robot

memcached

memcached

memcached

memcached

mysql (master)

mysql (slave)

load balancing

mongrels

Indexer(Search)

Indexer(Search)

mongrels

mongrels

mongrels

mongrels

mongrels

mongrels

mongrels

Robot

Robot

memcached

memcached

mysql (slave)

DNACredentials

(PHP)

Indexer(Search)

Indexer(Search)

DNAProfile(PHP)

DNAAvatars(PHP)

DNAScoring(C++)

DNASocialGraph

(Python)

DNAMessaging

(Erlang)

DNAMessaging

(Erlang)

DNAMessaging

(Erlang)

DNAMessaging

(Erlang)

DNAMessaging

(Erlang)

humm... PHP...WTF??!!

DNACredentials

(PHP)

load balancing

load balancing

ActiveDNA (REST)

ActiveDNA (REST)

load balancing

mongrels

mongrels

mongrels

mongrels

mongrels

mongrels

sinatra

mongrels

Robot

Robot

memcached

memcached

memcached

memcached

mysql (master)

mysql (slave)

load balancing

mongrels

Indexer(Search)

Indexer(Search)

mongrels

mongrels

mongrels

mongrels

mongrels

mongrels

mongrels

Robot

Robot

memcached

memcached

mysql (slave)

DNACredentials

(PHP)

Indexer(Search)

Indexer(Search)

DNAProfile(PHP)

DNAAvatars(PHP)

DNAScoring(C++)

DNASocialGraph

(Python)

DNAMessaging

(Erlang)

DNAMessaging

(Erlang)

DNAMessaging

(Erlang)

DNAMessaging

(Erlang)

DNAMessaging

(Erlang)

humm... PHP...WTF??!!

Active DNA

Problema: Várias features necessárias (Profile, SSO, Avatar, etc) e pouco tempo

Serendipity: Companhia irmã com vários componentes REST

Resultado: Active DNA, uma biblioteca REST flexível mais cache e serviços

Active DNA

Por que não Active Resource

Interface fixa

Somente Rails sem overrides

Pouca integração com external services

Busca

Ferret: FAIL

Sphinx: Muito limitado

Lucene / Solr: Java

Xapian: Interessante

Mas, e se precisarmos mudar?

Busca

Solução: Sapien

Cliente / encapsulamento de servidor de busca

Driver-based, naturalmente distribuído

Drivers podem ser mudados e o índice refeito em um passo

Brasigo Query Language

API

Controle de Acesso

HTTP Auth (email & senha)

OAuth para suportar usuários de OpenID e outros

Restrições de uso

Frameworks mais leves

Sinatra

Thin

Dia-a-dia (SCRUM)

Scrum

Várias equipes

Coordenação e priorização

Rotacionamento de equipes

Socialização de conhecimento

Cultura compartilhada

Dia-a-dia (Git)

Git

Vários repositórios internos, todos Git

Branches por estória

Repositório sempre releasable

Dia-a-dia (Sharing)

Cultura de bibliotecas

Plugins

Gems / Gem server interno

Funcionalidade compartilhada via API

Dia-a-dia (Testes)

Test all the f***ing time

Sem cobertura, não há conhecimento

Socialização de conhecimento

Integração contínua

Continuous Integration

Dedicated integration environment with automated integration and tests execution with the most up to dated code base.

Automated Reporting

Test Cases Results

Tests Coverage

Tests, tests & more tests

Continuous Integration

Venha trabalhar com a gente!!!

[email protected]

mailto:[email protected]


[email protected]

[email protected]

[email protected]

http://brasigo.com.br/http://blog.brasigo.com.br/

http://blogblogs.com.br/http://blog.blogblogs.com.br/







http://brasigo.com.br

http://brasigo.com.br

http://blog.brasigo.com.br

http://blog.brasigo.com.br

http://blogblogs.com.br

http://blogblogs.com.br

http://blog.blogblogs.com.br

http://blog.blogblogs.com.br

Rails Summit 2008 : Histórias de Guerra - Enfrentando problemas reais em aplicações Web 2.0

Technology

Transcript of Rails Summit 2008 : Histórias de Guerra - Enfrentando problemas reais em aplicações Web 2.0