Post on 30-Oct-2014
description
Uma história de guerra: Enfrentando problemas reais em aplicações Web 2.0
Manoel Lemos
Ronaldo Ferraz
Nando Vieira
Rails Summit Latin America 2008São Paulo, Brasil, 16.Outubro
1983
1983
1984
1983
1984
1993
1983
1984 1994
1993
1983
1984 1994
1993
1997
1983
1984 1994
1993
1997
1999
1983
1984 1994
1993
1997
1999
1983
1984 1994
1993
1997 2000
1999
1983
1984 1994
1993
1997 2000
1999 2006
Abr.06
Abr.06
Nov.06
Abr.06
Nov.06
Fev.07
Abr.06
Nov.06
Fev.07
Fev.08
Abr.06
Nov.06
Fev.07
Ago.08
Fev.08
Abr.06
Nov.06
Fev.07
Ago.08
Fev.08
BlogBlogs Status
Business Numbers
110.000 usuários
200.000 blogs
2.000.000 posts
250 usuários / dia
150 blogs / dia
5.000 posts / dia
Traffic Numbers
5.2M PageViews
2.6M Uniques
Top 4000 Alexa Global
Top 120 Alexa Brasil
Top 102 IBOPE Brasil
50M Widget Impressions
Crescer é um processo complicado e doloroso!
É impossível de se prever com precisão
Existem gargalos escondidos
Às vezes você precisa voltar atrás
Às vezes você precisa mudar tudo
Não existem “silver-bullets”
A complexidade vai crescer com você
Crescimento real nunca acontece em um só eixo!
Tráfego : mais usuários, transações, imagens...
Tamanho : banco, memória, cache...
Complexidade : algoritmos, arquitetura, infraestrutura...
Organização : equipe, comunicação, processos...
Negócio : modelos, parceiros, fornecedores...
Na maioria dos casos os problemas crescem mais que linearmente!!!
Hoje1.5 Milhões
por dia
httpdmongrel
mysql
httpdmongrel
mysql
mongrelmongrel
httpdmongrel
mysql
mongrelmongrel
memcached
httpdmongrel
mysql
mongrelmongrel
memcached
httpdmongrel
mysql
mongrelmongrel
memcached
httpdmongrel
mysql
mongrelmongrel
memcached
httpdmongrelmongrelmongrel
memcached
load balancing
httpdmongrel
mysql
mongrelmongrel
memcached
httpdmongrelmongrelmongrel
memcached
httpdmongrelmongrelmongrel
memcached
load balancing
httpdmongrelmongrelmongrel
memcached
httpdmongrelmongrelmongrel
memcached
httpdmongrelmongrelmongrel
memcached
load balancing
mysql
httpdmongrelmongrelmongrel
memcached
httpdmongrelmongrelmongrel
memcached
httpdmongrelmongrelmongrel
memcached
load balancing
mysql
httpdmongrelmongrelmongrel
memcached
httpdmongrelmongrelmongrel
memcached
httpdmongrelmongrelmongrel
memcached
load balancing
mysql
httpdmongrelmongrelmongrel
httpdmongrelmongrelmongrel
httpdmongrelmongrelmongrel
load balancing
mysql
memcached memcachedmemcached
httpdmongrelmongrelmongrel
httpdmongrelmongrelmongrel
httpdmongrelmongrelmongrel
load balancing
mysql
memcached memcachedmemcached
mongrel mongrelmongrel
mysql
httpdmongrelmongrelmongrel
httpdmongrelmongrelmongrel
httpdmongrelmongrelmongrel
load balancing
memcached memcachedmemcached
mongrel mongrelmongrel
mysql
httpdmongrelmongrelmongrel
httpdmongrelmongrelmongrel
httpdmongrelmongrelmongrel
load balancing
memcached memcachedmemcached
mongrel mongrelmongrel
httpdmongrelmongrelmongrel
httpdmongrelmongrelmongrel
httpdmongrelmongrelmongrel
load balancing 1 (web)
memcached memcachedmemcached
mongrel mongrelmongrel
mysql
httpdmongrelmongrelmongrel
httpdmongrelmongrelmongrel
httpdmongrelmongrelmongrel
load balancing 1 (web)
memcached memcachedmemcached
mongrel mongrelmongrel
httpdmongrelmongrelmongrel
httpdmongrelmongrelmongrel
httpdmongrelmongrelmongrel
load balancing 2 (widgets)
memcached memcachedmemcached
mongrel mongrelmongrel
mysql
httpdmongrelmongrelmongrel
httpdmongrelmongrelmongrel
httpdmongrelmongrelmongrel
load balancing 1 (web)
memcached memcachedmemcached
mongrel mongrelmongrel
httpdmongrelmongrelmongrel
httpdmongrelmongrelmongrel
httpdmongrelmongrelmongrel
load balancing 2 (widgets)
memcached memcachedmemcached
mongrel mongrelmongrel
mysql (slave)
mysql (master)
httpdmongrelmongrelmongrel
httpdmongrelmongrelmongrel
httpdmongrelmongrelmongrel
load balancing 1 (web)
memcached memcachedmemcached
mongrel mongrelmongrel
mysql
httpdmongrelmongrelmongrel
httpdmongrelmongrelmongrel
httpdmongrelmongrelmongrel
load balancing 2 (widgets)
memcached memcachedmemcached
mongrel mongrelmongrel
mysql (slave)
mysql (master)
httpdmongrelmongrelmongrel
httpdmongrelmongrelmongrel
httpdmongrelmongrelmongrel
load balancing 1 (web)
memcached memcachedmemcached
mongrel mongrelmongrel
mysql
httpdmongrelmongrelmongrel
httpdmongrelmongrelmongrel
httpdmongrelmongrelmongrel
load balancing 2 (widgets)
memcached memcachedmemcached
mongrel mongrelmongrel
mysql (slave)
mysql (master)
httpdmongrelmongrelmongrel
httpdmongrelmongrelmongrel
httpdmongrelmongrelmongrel
load balancing 1 (web)
mongrel mongrelmongrel
mysql
httpdmongrelmongrelmongrel
httpdmongrelmongrelmongrel
httpdmongrelmongrelmongrel
load balancing 2 (widgets)
memcached memcachedmemcached memcached memcachedmemcached
mongrel mongrelmongrel
mysql (slave)
mysql (master)
httpdmongrelmongrelmongrel
httpdmongrelmongrelmongrel
httpdmongrelmongrelmongrel
load balancing 1 (web)
mongrel mongrelmongrel
mysql
httpdmongrelmongrelmongrel
httpdmongrelmongrelmongrel
httpdmongrelmongrelmongrel
load balancing 2 (widgets)
mongrel mongrelmongrel
mysql (slave)
mysql (master)
memcached memcached
memcached memcached
httpdmongrelmongrelmongrel
httpdmongrelmongrelmongrel
httpdmongrelmongrelmongrel
load balancing 1 (web)
mongrel mongrelmongrel
mysql
httpdmongrelmongrelmongrel
httpdmongrelmongrelmongrel
httpdmongrelmongrelmongrel
load balancing 2 (widgets)
mongrel mongrelmongrel
mysql (slave)
mysql (master)
memcached memcached
memcached memcached
mongrel mongrelmongrel mongrel mongrelmongrel
httpdmongrelmongrelmongrel
httpdmongrelmongrelmongrel
httpdmongrelmongrelmongrel
load balancing 1 (web)
mongrel mongrelmongrel
mysql
httpdmongrelmongrelmongrel
httpdmongrelmongrelmongrel
httpdmongrelmongrelmongrel
load balancing 2 (widgets)
mongrel mongrelmongrel
mysql (slave)
mysql (master)
memcached memcached
memcached memcached
Robot
Robot
mongrel mongrelmongrel mongrel mongrelmongrel
Robot
Robot
httpdmongrelmongrelmongrel
httpdmongrelmongrelmongrel
httpdmongrelmongrelmongrel
load balancing 1 (web)
mongrel mongrelmongrel
mysql
httpdmongrelmongrelmongrel
httpdmongrelmongrelmongrel
httpdmongrelmongrelmongrel
load balancing 2 (widgets)
mongrel mongrelmongrel
mysql (slave)
mysql (master)
memcached memcached
memcached memcached
Robot
Robot
mongrel mongrelmongrel mongrel mongrelmongrel
Robot
Robot
Indexer(Search)
Indexer(Search)
httpdmongrelmongrelmongrel
httpdmongrelmongrelmongrel
httpdmongrelmongrelmongrel
load balancing 1 (web)
mongrel mongrelmongrel
mysql
httpdmongrelmongrelmongrel
httpdmongrelmongrelmongrel
httpdmongrelmongrelmongrel
load balancing 2 (widgets)
mongrel mongrelmongrel
mysql (slave)
mysql (master)
memcached memcached
memcached memcached
Robot
Robot
mongrel mongrelmongrel mongrel mongrelmongrel
Robot
Robot
Indexer(Search)
httpdmongrelmongrelmongrelmongrelmongrel
httpdmongrelmongrelmongrelmongrelmongrel
Indexer(Search)
load balancing 3 (API)
load balancing 1 (web)
mysql
load balancing 2 (widgets)
mysql (slave)
mysql (master)
memcached memcached
memcached memcached
Robot
Robot
mongrels
Robot
Robot
Indexer(Search)
Indexer(Search)
load balancing 3 (API)
mongrels mongrels mongrels mongrels mongrels
mongrels mongrels
load balancing
memcached
Robot
mongrels
Indexer(Search)
Indexer(Search)
mongrels
mongrels
mongrels
mongrels
mongrels
mongrels
mongrels
Robot
memcached
memcached
memcached
mysql (master)
mysql (master)
load balancing
memcached
Robot
mongrels
Indexer(Search)
Indexer(Search)
mongrels
mongrels
mongrels
mongrels
mongrels
mongrels
mongrels
Robot
memcached
memcached
memcached
mysql (master)
mysql (master)
load balancing
memcached
Robot
mongrels
Indexer(Search)
Indexer(Search)
mongrels
mongrels
mongrels
mongrels
mongrels
mongrels
mongrels
Robot
memcached
memcached
memcached
mysql (master)
mysql (master)
load balancing
memcached
Robot
mongrels
Indexer(Search)
Indexer(Search)
mongrels
mongrels
mongrels
mongrels
mongrels
mongrels
mongrels
Robot
memcached
memcached
memcached
mysql (master)
mysql (master)
load balancing
memcached
Robot
mongrels
Indexer(Search)
Indexer(Search)
mongrels
mongrels
mongrels
mongrels
mongrels
mongrels
mongrels
Robot
memcached
memcached
memcached
mysql (master)
mysql (master)
load balancing
memcached
Robot
mongrels
Indexer(Search)
Indexer(Search)
mongrels
mongrels
mongrels
mongrels
mongrels
mongrels
mongrels
Robot
memcached
memcached
memcached
mysql (master)
mysql (master)
load balancing
memcached
Robot
mongrels
Indexer(Search)
Indexer(Search)
mongrels
mongrels
mongrels
mongrels
mongrels
mongrels
mongrels
Robot
memcached
memcached
memcached
mysql (master)
mysql (master)
load balancing
memcached
Robot
mongrels
Indexer(Search)
Indexer(Search)
mongrels
mongrels
mongrels
mongrels
mongrels
mongrels
mongrels
Robot
memcached
memcached
memcached
mysql (master)
mysql (master)
Eureca...Idéias...Single Sign On
Shared ResourcesIntegrated Search
E=MC2
load balancing
memcached
Robot
mongrels
Indexer(Search)
Indexer(Search)
mongrels
mongrels
mongrels
mongrels
mongrels
mongrels
mongrels
Robot
memcached
memcached
memcached
mysql (master)
mysql (master)
load balancing
memcached
Robot
mongrels
Indexer(Search)
Indexer(Search)
mongrels
mongrels
mongrels
mongrels
mongrels
mongrels
mongrels
Robot
memcached
memcached
memcached
mysql (master)
mysql (master)
load balancing
memcached
Robot
mongrels
Indexer(Search)
Indexer(Search)
mongrels
mongrels
mongrels
mongrels
mongrels
mongrels
mongrels
Robot
memcached
memcached
memcached
mysql (master)
mysql (master)
load balancing
memcached
Robot
mongrels
Indexer(Search)
Indexer(Search)
mongrels
mongrels
mongrels
mongrels
mongrels
mongrels
mongrels
Robot
memcached
memcached
memcached
mysql (master)
mysql (master)
E ai, #comofaz ?
load balancing
mongrels
mongrels
mongrels
mongrels
mongrels
mongrels
sinatra
mongrels
load balancing
mongrels
mongrels
mongrels
mongrels
mongrels
mongrels
mongrels
mongrels
load balancing
mongrels
mongrels
mongrels
mongrels
mongrels
mongrels
sinatra
mongrels
Robot
Robot
memcached
memcached
memcached
memcached
mysql (master)
mysql (slave)
load balancing
mongrels
Indexer(Search)
Indexer(Search)
mongrels
mongrels
mongrels
mongrels
mongrels
mongrels
mongrels
Robot
Robot
memcached
memcached
mysql (slave)
Indexer(Search)
Indexer(Search)
ActiveDNA (REST)
load balancing
mongrels
mongrels
mongrels
mongrels
mongrels
mongrels
sinatra
mongrels
Robot
Robot
memcached
memcached
memcached
memcached
mysql (master)
mysql (slave)
load balancing
mongrels
Indexer(Search)
Indexer(Search)
mongrels
mongrels
mongrels
mongrels
mongrels
mongrels
mongrels
Robot
Robot
memcached
memcached
mysql (slave)
Indexer(Search)
Indexer(Search)
ActiveDNA (REST)
ActiveDNA (REST)
load balancing
mongrels
mongrels
mongrels
mongrels
mongrels
mongrels
sinatra
mongrels
Robot
Robot
memcached
memcached
memcached
memcached
mysql (master)
mysql (slave)
load balancing
mongrels
Indexer(Search)
Indexer(Search)
mongrels
mongrels
mongrels
mongrels
mongrels
mongrels
mongrels
Robot
Robot
memcached
memcached
mysql (slave)
DNACredentials
(PHP)
Indexer(Search)
Indexer(Search)
ActiveDNA (REST)
ActiveDNA (REST)
load balancing
mongrels
mongrels
mongrels
mongrels
mongrels
mongrels
sinatra
mongrels
Robot
Robot
memcached
memcached
memcached
memcached
mysql (master)
mysql (slave)
load balancing
mongrels
Indexer(Search)
Indexer(Search)
mongrels
mongrels
mongrels
mongrels
mongrels
mongrels
mongrels
Robot
Robot
memcached
memcached
mysql (slave)
DNACredentials
(PHP)
Indexer(Search)
Indexer(Search)
DNAProfile(PHP)
ActiveDNA (REST)
ActiveDNA (REST)
load balancing
mongrels
mongrels
mongrels
mongrels
mongrels
mongrels
sinatra
mongrels
Robot
Robot
memcached
memcached
memcached
memcached
mysql (master)
mysql (slave)
load balancing
mongrels
Indexer(Search)
Indexer(Search)
mongrels
mongrels
mongrels
mongrels
mongrels
mongrels
mongrels
Robot
Robot
memcached
memcached
mysql (slave)
DNACredentials
(PHP)
Indexer(Search)
Indexer(Search)
DNAProfile(PHP)
DNAAvatars(PHP)
ActiveDNA (REST)
ActiveDNA (REST)
load balancing
mongrels
mongrels
mongrels
mongrels
mongrels
mongrels
sinatra
mongrels
Robot
Robot
memcached
memcached
memcached
memcached
mysql (master)
mysql (slave)
load balancing
mongrels
Indexer(Search)
Indexer(Search)
mongrels
mongrels
mongrels
mongrels
mongrels
mongrels
mongrels
Robot
Robot
memcached
memcached
mysql (slave)
DNACredentials
(PHP)
Indexer(Search)
Indexer(Search)
DNAProfile(PHP)
DNAAvatars(PHP)
DNAScoring(C++)
DNASocialGraph
(Python)
ActiveDNA (REST)
ActiveDNA (REST)
load balancing
mongrels
mongrels
mongrels
mongrels
mongrels
mongrels
sinatra
mongrels
Robot
Robot
memcached
memcached
memcached
memcached
mysql (master)
mysql (slave)
load balancing
mongrels
Indexer(Search)
Indexer(Search)
mongrels
mongrels
mongrels
mongrels
mongrels
mongrels
mongrels
Robot
Robot
memcached
memcached
mysql (slave)
DNACredentials
(PHP)
Indexer(Search)
Indexer(Search)
DNAProfile(PHP)
DNAAvatars(PHP)
DNAScoring(C++)
DNASocialGraph
(Python)
DNAMessaging
(Erlang)
DNAMessaging
(Erlang)
DNAMessaging
(Erlang)
DNAMessaging
(Erlang)
DNAMessaging
(Erlang)
ActiveDNA (REST)
ActiveDNA (REST)
load balancing
load balancing
load balancing
mongrels
mongrels
mongrels
mongrels
mongrels
mongrels
sinatra
mongrels
Robot
Robot
memcached
memcached
memcached
memcached
mysql (master)
mysql (slave)
load balancing
mongrels
Indexer(Search)
Indexer(Search)
mongrels
mongrels
mongrels
mongrels
mongrels
mongrels
mongrels
Robot
Robot
memcached
memcached
mysql (slave)
DNACredentials
(PHP)
Indexer(Search)
Indexer(Search)
DNAProfile(PHP)
DNAAvatars(PHP)
DNAScoring(C++)
DNASocialGraph
(Python)
DNAMessaging
(Erlang)
DNAMessaging
(Erlang)
DNAMessaging
(Erlang)
DNAMessaging
(Erlang)
DNAMessaging
(Erlang)
load balancing
load balancing
ActiveDNA (REST)
ActiveDNA (REST)
load balancing
mongrels
mongrels
mongrels
mongrels
mongrels
mongrels
sinatra
mongrels
Robot
Robot
memcached
memcached
memcached
memcached
mysql (master)
mysql (slave)
load balancing
mongrels
Indexer(Search)
Indexer(Search)
mongrels
mongrels
mongrels
mongrels
mongrels
mongrels
mongrels
Robot
Robot
memcached
memcached
mysql (slave)
DNACredentials
(PHP)
Indexer(Search)
Indexer(Search)
DNAProfile(PHP)
DNAAvatars(PHP)
DNAScoring(C++)
DNASocialGraph
(Python)
DNAMessaging
(Erlang)
DNAMessaging
(Erlang)
DNAMessaging
(Erlang)
DNAMessaging
(Erlang)
DNAMessaging
(Erlang)
load balancing
load balancing
ActiveDNA (REST)
ActiveDNA (REST)
load balancing
mongrels
mongrels
mongrels
mongrels
mongrels
mongrels
sinatra
mongrels
Robot
Robot
memcached
memcached
memcached
memcached
mysql (master)
mysql (slave)
load balancing
mongrels
Indexer(Search)
Indexer(Search)
mongrels
mongrels
mongrels
mongrels
mongrels
mongrels
mongrels
Robot
Robot
memcached
memcached
mysql (slave)
DNACredentials
(PHP)
Indexer(Search)
Indexer(Search)
DNAProfile(PHP)
DNAAvatars(PHP)
DNAScoring(C++)
DNASocialGraph
(Python)
DNAMessaging
(Erlang)
DNAMessaging
(Erlang)
DNAMessaging
(Erlang)
DNAMessaging
(Erlang)
DNAMessaging
(Erlang)
load balancing
load balancing
ActiveDNA (REST)
ActiveDNA (REST)
load balancing
mongrels
mongrels
mongrels
mongrels
mongrels
mongrels
sinatra
mongrels
Robot
Robot
memcached
memcached
memcached
memcached
mysql (master)
mysql (slave)
load balancing
mongrels
Indexer(Search)
Indexer(Search)
mongrels
mongrels
mongrels
mongrels
mongrels
mongrels
mongrels
Robot
Robot
memcached
memcached
mysql (slave)
DNACredentials
(PHP)
Indexer(Search)
Indexer(Search)
DNAProfile(PHP)
DNAAvatars(PHP)
DNAScoring(C++)
DNASocialGraph
(Python)
DNAMessaging
(Erlang)
DNAMessaging
(Erlang)
DNAMessaging
(Erlang)
DNAMessaging
(Erlang)
DNAMessaging
(Erlang)
humm... PHP...WTF??!!
DNACredentials
(PHP)
load balancing
load balancing
ActiveDNA (REST)
ActiveDNA (REST)
load balancing
mongrels
mongrels
mongrels
mongrels
mongrels
mongrels
sinatra
mongrels
Robot
Robot
memcached
memcached
memcached
memcached
mysql (master)
mysql (slave)
load balancing
mongrels
Indexer(Search)
Indexer(Search)
mongrels
mongrels
mongrels
mongrels
mongrels
mongrels
mongrels
Robot
Robot
memcached
memcached
mysql (slave)
DNACredentials
(PHP)
Indexer(Search)
Indexer(Search)
DNAProfile(PHP)
DNAAvatars(PHP)
DNAScoring(C++)
DNASocialGraph
(Python)
DNAMessaging
(Erlang)
DNAMessaging
(Erlang)
DNAMessaging
(Erlang)
DNAMessaging
(Erlang)
DNAMessaging
(Erlang)
humm... PHP...WTF??!!
Active DNA
Problema: Várias features necessárias (Profile, SSO, Avatar, etc) e pouco tempo
Serendipity: Companhia irmã com vários componentes REST
Resultado: Active DNA, uma biblioteca REST flexível mais cache e serviços
Active DNA
Por que não Active Resource
Interface fixa
Somente Rails sem overrides
Pouca integração com external services
Busca
Ferret: FAIL
Sphinx: Muito limitado
Lucene / Solr: Java
Xapian: Interessante
Mas, e se precisarmos mudar?
Busca
Solução: Sapien
Cliente / encapsulamento de servidor de busca
Driver-based, naturalmente distribuído
Drivers podem ser mudados e o índice refeito em um passo
Brasigo Query Language
API
Controle de Acesso
HTTP Auth (email & senha)
OAuth para suportar usuários de OpenID e outros
Restrições de uso
Frameworks mais leves
Sinatra
Thin
Dia-a-dia (SCRUM)
Scrum
Várias equipes
Coordenação e priorização
Rotacionamento de equipes
Socialização de conhecimento
Cultura compartilhada
Dia-a-dia (Git)
Git
Vários repositórios internos, todos Git
Branches por estória
Repositório sempre releasable
Dia-a-dia (Sharing)
Cultura de bibliotecas
Plugins
Gems / Gem server interno
Funcionalidade compartilhada via API
Dia-a-dia (Testes)
Test all the f***ing time
Sem cobertura, não há conhecimento
Socialização de conhecimento
Integração contínua
Continuous Integration
Dedicated integration environment with automated integration and tests execution with the most up to dated code base.
Automated Reporting
Test Cases Results
Tests Coverage
Tests, tests & more tests
Tests, tests & more tests
Tests, tests & more tests
Continuous Integration
Continuous Integration
Continuous Integration
Continuous Integration
Continuous Integration
Continuous Integration
Continuous Integration
Continuous Integration
Continuous Integration
Continuous Integration
Continuous Integration
Continuous Integration
Continuous Integration
Continuous Integration
Continuous Integration
Continuous Integration
Venha trabalhar com a gente!!!
vagas@blogblogs.com.br
Obrigado!mlemos@webcointernet.com
rferraz@webcointernet.com
fvieira@webcointernet.com
http://brasigo.com.br/http://blog.brasigo.com.br/
http://blogblogs.com.br/http://blog.blogblogs.com.br/