Friday, May 31, 2013

MYSQL master and slave replication considerations and failover

I recently saw an interesting question on a forum which asked how to best set up a MYSQL master slave environment. The answer was, 'take a look at this link!'. The link details some really interesting information about how best to set up your master slave environment, but also how to scale your application in a MYSQL master / slave setup.

Here is an interesting diagram to look at first.



 


















This is pretty simple stuff. A web app, in this case, requests a write to the MySQL master, which replicates the information to each of the slaves. 10/10!

You need to run your slaves with the --log-bin option and without --log-slave-updates. What does this mean exactly? Well these are optional values you can use when start the mysql. 

In linux flavours, you can edit the file /etc/my.sql and set these values in this file that get loaded when mysql starts. You can also specify some values using a mysql client, connecting to the database and specifying the values required. I would recommend using /etc/my.sql as a one stop shop.

In Windows, MySQL programs read startup options from the following files, in the following order:















First of all, what do we need to set?

For --log-bin, this needs a filename value. See here:










So Type = file name and the default is off. If we need to set up replication, the mysql /etc/my.cnf file will look like this:

[mysqld]

log-bin=mysql-bin

For  --log-slave-updates, well this is slightly easier. Why? Take a look at this:














The default is set to false anyhow so unless the config has explicitly set this to true, we don't need to do anything. For good measure:

[ mysqld]

log-slave-updates=false


However, what happens in a failure here?  Each MySQL Slave (Slave 1, Slave 2, and Slave 3) is a slave running with --log-bin and without --log-slave-updates. Because updates received by a slave from the master are not logged in the binary log unless --log-slave-updates is specified, the binary log on each slave is empty initially. If for some reason MySQL Master becomes unavailable, you can pick one of the slaves to become the new master. For example, if you pick Slave 1, all Web Clients should be redirected to Slave 1, which will log updates to its binary log. Slave 2 and Slave 3 should then replicate from Slave 1.

The reason for running the slave without --log-slave-updates is to prevent slaves from receiving updates twice in case you cause one of the slaves to become the new master. Suppose that Slave 1 has --log-slave-updates enabled. Then it will write updates that it receives from Master to its own binary log. When Slave 2 changes from Master to Slave 1 as its master, it may receive updates from Slave 1 that it has already received from Master.


However there is some intervention required to do all of this. It's not automated. Here is what you would need to do in a case of a MYSQL failure:

Make sure that all slaves have processed any statements in their relay log. On each slave, issue STOP SLAVE IO_THREAD, then check the output of SHOW PROCESSLIST until you see Has read all relay log. When this is true for all slaves, they can be reconfigured to the new setup. On the slave Slave 1 being promoted to become the master, issue STOP SLAVE and RESET MASTER.

On the other slaves Slave 2 and Slave 3, use STOP SLAVE and CHANGE MASTER TO MASTER_HOST='Slave1' (where 'Slave1' represents the real host name of Slave 1). To use CHANGE MASTER TO, add all information about how to connect to Slave 1 from Slave 2 or Slave 3 (user, password, port). In CHANGE MASTER TO, there is no need to specify the name of the Slave 1 binary log file or log position to read from: We know it is the first binary log file and position 4, which are the defaults for CHANGE MASTER TO. Finally, use START SLAVE on Slave 2 and Slave 3. 

There is an interesting scaling solution you could consider which I like. Take a look at this diagram:



In this scenario, the web client are directed to slaves for read purposes only, and any writes are directed to the MySQL master server. This takes the read load off from the MySQL master server. Reads generate more requests than writes generally. However, you need to have the application aware of the slave setup. This is generally done by creating MYSQL connectors that are aware of read only functions. These connectors would then utilise a slave server rather than a master server. For more info, see this link https://dev.mysql.com/doc/refman/5.7/en/replication-solutions-scaleout.html.

Happy playing!

Wednesday, May 29, 2013

Cloudstack db.properties password encryption

I was recently asked to update the db password in the db.properties files in Cloudstack. I kind of knew the guy who asked me was still in Cloudstack 2.2 land where passwords were in plain old text, but as of Cloudstack 3.x, the passwords in db.properties file and other passwords for that fact are encrypted. Lets have a closer look at the db.properties file.



The following file on each cloudstack and cloudplatform server contains the db connection details
 
/etc/cloud/management/db.properties

In this file there is information about the db username and password details.
 
# CloudStack database tuning parameters
db.cloud.password=ENC(57nNfvJaN9X54lbJi0pjugc9YylyRo8c)
db.cloud.username=cloudadmin
db.cloud.host=10.0.26.16
db.cloud.validationQuery=SELECT 1
db.cloud.testOnBorrow=true

In cloudplatform and cloudstack version 3 +, the db.cloud.password is encrypted. To change this, run the following file to work out the encrypted password
 
java -classpath /usr/share/java/cloud-jasypt-1.8.jar 
org.jasypt.intf.cli.JasyptPBEStringEncryptionCLI 
encrypt.sh input="Password12" 
password="$(cat /etc/cloud/management/key)" verbose=true

Where
 
input=Password12

is the password you wish to encrypt.

The output should look something like this.
 
$ java -classpath /usr/share/java/cloud-jasypt-1.8.jar 
org.jasypt.intf.cli.JasyptPBEStringEncryptionCLI 
encrypt.sh input="Password12" 
password="$(cat /etc/cloud/management/key)" verbose=true
 
----ENVIRONMENT-----------------

Runtime: Sun Microsystems Inc. OpenJDK 64-Bit Server VM 20.0-b12



----ARGUMENTS-------------------

verbose: true
input: Password12
password: password



----OUTPUT----------------------

57nNfvJaN9X54lbJi0pjugc9YylyRo8c

The 57nNfvJaN9X54lbJi0pjugc9YylyRo8c is the encrypted password for Password12. Use this as the new password in the db.properties file

If you wish to decrypt the password, run the following command:
 
java -classpath /usr/share/java/cloud-jasypt-1.8.jar 
 org.jasypt.intf.cli.JasyptPBEStringDecryptionCLI 
 decrypt.sh input="57nNfvJaN9X54lbJi0pjugc9YylyRo8c" 
 password="$(cat /etc/cloud/management/key)" verbose=false

The password is then displayed, in this case Password12

Friday, May 10, 2013

Summary of VMUG




VMUG in London was good - these VMUG's always are. In fact I must say. if you get the chance to go, do it.

2 talks stood out for me and here is my review:



1 – Software-Defined Control – Transforming IT Operations, Shmuel Kliger, VMTurbo

Shmuel Kliger, who is the founder, gave a talk about using VMturbo and the benefits of using it in your environment. It’s a tool that gathers 1000’s (well 10,000’s) of metrics from all component resources and calculates the optimum state for your environment. It’s good for the corporate IT world and a bit like the visioncore products that got bought out by Quest Software. It’s pretty much useless in a cloud environment if you ask me unless you like looking at charts and graphs and saying, ooh that looks good but ‘now what?’. Having said that, it can integrate with CloudPortal business manager and if it works well, it may be worth trying to offer this as a service bundle to offer vmturbo to the customer as a service offering.

2 – Don't Do Virtual Networking, Do Cloud Networking - Greg Ferro, packetpushers.net

This was by far the best talk presented by a guy who was to the point, lots of knowledge about networks and opinionated views. A very high summary is the discussions were about how traditional networks will turn in to more overlay networks and how SDN technology will glue overlay networks all together using encapsulation managed by SDN controllers using openflow as the management protocol. In fact, this was also something discussed at the HP technology conference last year so there seems to be a consistent message from these events. The opinion is SDN is still not ready but should start to progress in a few years’ time, circa 2015. The feeling was Cisco hasn’t yet made a move in the SDN space yet and there are no real standards. The only real standard are open vswitch, which seems to be the choice for the virtualization network stack but that’s not news, and openflow. I feel that the technology is all over the shop a bit at the moment but is gaining pace and needs some standardisations.


I didn't make vBeers but would have if I had the chance. It was great meeting with some of the guys there and talking about VMware and Puppet. Overall, a good day out.

Thursday, April 18, 2013

VMUG in London on 25th April







VMUG London is happening on the 25th April 2013 and is only one week away. The location is here:


London VMUG Meeting
Thursday 25 April 2013 8:30 - 17:00
Location London Chamber of Commerce and Industry
33 Queen Street
London, EC4R 1AP



... and this is the agenda.



Sign up here http://www.vmug.com/e/in/eid=781
 
Lots of interesting topics and you'll learn a lot, that is for sure. I'm going and getting excited!

Monday, April 15, 2013

Cloudstack deploying cloudstack








 







I was talking at the Cloudstack European Meet up the other day about getting Cloudstack to deploy Cloudstack.

Well its more than that - I was talking about the concepts rather than what I was doing, as really, Cloudstack deploying Cloudstack is a bit of a gimick.

Here is my github link to the project

https://github.com/oliverleach/cloudstack-puppet


However, I thought I would include some scribble on how the project works in case anyone is interested:

Cloudstack 4.0.1 deploy

This project enables a running Cloudstack instance 3.x and above to deploy Cloudstack 4.0.1 instances. This could be to demo Cloudstack 4.0.1 or used for testing purposes, by easily spinning up Cloudstack 4.0.1 instances. It could also be used to provision private clouds using Cloudstack.

How to use this project

The python script and puppet manifests in this repo deploys cloudstack 4.0.1 using Puppet and the Cloudstack API.

For this to work, you need to use the python scripts and 2 puppet templates. One template is the puppet-master, and one template is the puppet-agent.

You need to configure the python parameters in the puppet-deploy.py script as per your environment. This includes:

1 - your API key
2 - your secret key
3 - your service offering
4 - your Cloudstack zone ID
5 - your network ID of your user account
6 - The Cloudstack provisioning host IP and port
7 - The Cloudstack protocol used (http or https)


You will also need your 2 template IDs for the puppet master and puppet agent

These templates can be downloaded using the following link:

Puppet Agent


https://dl.dropbox.com/u/79905244/puppet-agent.7z

Puppet Master

https://dl.dropbox.com/u/79905244/puppet-master.7z

Both templates are running CentOS 6.4 64-bit. The puppet master is configured as a puppet-ca, a puppet server and a puppet agent. The puppet agent has the puppet agent installed and runs a userdata.rb script on startup. See the rc.local file for further information.

Once the templates are unzipped and uploaded in to your provisioning Cloudstack instance, update the puppet-deploy.py script with the template IDs.

To run the python scripts, use an interpreter like idle, open and run the puppet-deploy.py file, or simply run from the command line where an instance of python is installed; e.g. python puppet-deploy.py. This has been tested against python 2.7.3.

The git repo holds the puppet manifests which are included in the puppet-master template.
They are just for reference.

This project is just for fun, could be better written and I hold no responsibility if you use the project. You are allowed to do so.

Saturday, April 13, 2013

Cloudstack 4.1 new features



Here is a list I have put together which discusses the Cloudstack 4.1. new features. I have put some comments down against some of the features. Some comments are missing, simply because I do not know much about them!





  • AWS-Style Regions

    This is the biggest change which involves placing a new layer above Zones. This is so we can scale cloudstack better with the ability of having a cloudstack management server per availablity zone. This is great if you customers are geographically apart across continents. There are some big challenges around this change I suspect.

  • F5 & SRX inline mode

    This is an improvement. Basically having a firewall in front of the F5 load balancer so your customer can have the advantage of IDP and IDS filtering.

  • AutoScale in the Netscalers

    This has been in CloudPlatform and requires NS firmware 10+, but its cool and it works. You do need a netscaler, and the cheapest option is a VPX device.
 
  • Advanced Search UI

    Required as the product is becoming so more powerful, with the ability to handle so many devices now. Tick!
 
  • S3-backed Secondary Storage

    Well S3 has about 15% of the worlds entire storage, so why not :)

  • EC2 Query API support

    Well EC2 has about 15% of the worlds entire compute force, so why not :)
 
  • API Request Throttling

    This has been outstanding for a while. You can actually DDOS a CSM server if you wanted to. This will stop the potential.
 
  • Enhanced baremetal servers support on Cisco UCS

    This is a good. Automating physical hosts with Cisco UCS platform, allowing Cloudstack to manage the lifecycle of the physical host.
 
  • Events framework to publish/subscribe to CloudStack events

    Good for developers who want to hook in to events. Can be used for emailing when
    something happens which isn't really possible in CS 4 

  • OVS support in KVM

    I would say this is a pit of plumbing for some future networking functionality coming our way with KVM. A step in the right direction.
 
  • Reset SSH Key to access VM (similar to reset password)

    Great stuff. Easy and now usable.
 
  • Security Groups Isolation in Advanced Zone

    I am sure this is only KVM using etables. We'll see though.
 
  • Site-to-Site VPN: Monitoring of VPN Tunnels 
 
  • Persistent Networks without running a VM

    This is actually fairly important if using cloudstack to burst or scale into. You have vlan dedicated to you so persistent config can now be applied to your external devices connecting in to cloudstack.
 
  • Egress firewall rules for guest network

    I need to see this. This is in CloudPlatform but we need to have more control over outbound traffic, i.e., what we can allow through a back door. For example, if I wanted to connect to a kms server for licensing or a monitoring server for my customers.
 
  • Additional VMX Settings

    _not_yet_sure_what_this_is_
 
  • Resize volumes feature

    This is root and data volumes. Great if you have deployed a VM and want to resize it.
 
  • Add/remove network on VM

    Yes - and finally. So you need a NIC? You can now add on.
 
  • Limit API Queries

    1,2,3,4,5 that's all you can have Mr.
 
  • Improve API Performance / Add Search Capabilities

    With so many APIs now in Cloudstack, we can get a bit lost with it all. This will help.

  • Allow for same vlan on different physical nics

    Helpful if you want to define your networks a bit more over physical devices. For example, coming inbound via an MPLS link.
   
  • ApiDiscoverService: Implement a plugin mechanism that exposes the list of APIs through a discovery service on the management server

    Oh, this one again. So what was the other one then?

  •  Implement L3 Router functionality in Nicira Nvp Plugin

    No more vlans.. no more vlans.. yeah.
 
  • Mash up marvin into an interactive auto-completing API shell for CloudStack (aka Cloudmonkey CLI) 
 
  • Baremetal kickstart

    Good for physical host deployments of say Windows :)
 
  • Nicira NVP/KVM

    SDN not my game.. yet - I need it to settle down a bit more before I get a grip :)
 
  • Netscaler plugin

    More netscaler intergration

  • API changes for IPv6 Support

    IP6 stuff coming in to play more.

Lots of new features. Can't wait to have a play and will report back when I do. Leave a comment if you get the chance to muck around - be good to know feedback!

Puppet manifests for cloudstack and co.

Here are some of my puppet manifests for Cloudstack and co. You can also find more information here:

https://github.com/oliverleach/cloudstack-puppet

I also recently did a presentation at the Cloudstack European meetup in London on the 11th April about deploying applications using puppet and Cloudstack. The talk was well received so I thought I would jot down the manifests that I wrote as part of the talk.

Here is the Cloudstack manifest:

# Class cloudstack
#
# This class installs cloudstack
#
# Author: Oliver Leach
# Date : 18th March 2013
#
# Actions:
# Installs the cloudstakc package
#
# This includes, the cloudstack-client, mysql-server and any install commands
#
# Currently the clodustack class is monlithic for demo purposes.
#
# Requires:
# Centos - other operating systems are not supported by this manifest
#
# Sample Usage:
# include cloudstack
#

class cloudstack ($mysql_password) {

 case $::operatingsystem {

  'centos': {
   $supported  = true
  }
 }

 if ($supported == true) {

  yumrepo { 'cloudstack':
      baseurl  => "http://cloudstack.apt-get.eu/rhel/4.0/",
   descr   => "Cloudstack 4.0.1 yum repo",
      enabled  => 1,
      gpgcheck => 0,
  }

  package { "cloud-client": 
   ensure  => "installed",
   require => Yumrepo["cloudstack"],
   }

  exec { "cloud-setup" :
   command => "/usr/bin/cloud-setup-databases cloud:cloud@localhost 
              --deploy-as=root:${mysql_password} && /bin/sleep 3 && 
              /usr/bin/cloud-setup-management && /bin/sleep 3 && 
              /usr/bin/cloud-setup-databases cloud:cloud@localhost 
              --deploy-as=root:${mysql_password} && /bin/sleep 3 && 
              /usr/bin/cloud-setup-management",
   creates => '/var/lib/mysql/cloud',
   require => Package["cloud-client"],
  }

        service { "cloud-management":
      ensure  => "running",
      enable  => "true",
      require => Exec["cloud-setup"],
  }
 }
}

MySQL server manifest


# Class myslqserver
#
# This class installs cloudstack
#
# Author: Oliver Leach
# Date : 20th March 2013
#
# Actions:
# Installs the mysql server package
#
# This includes, the mysql-server package
#
# Requires:
# Centos - other operating systems are not supported by this manifest
#
# Sample Usage:
# include mysql::server
#

class mysql::server ($mysql_password) {

 case $::operatingsystem {
  "centos": {
   $supported  = true
  }
 }

 if ($supported == true) {

        file { "flag-files" :

            ensure  => directory,
            path    => "/var/lib/puppet/flags/",
            owner   => root,
            group   => root,
            mode    => "0644",
        }

  package { "mysql-server":
   ensure  => installed,
   require => File["flag-files"], 
  }

        file { "/etc/my.cnf" :
            owner   => "root",
            group   => "root",
            source  => "puppet:///modules/mysql/my.cnf",
   require => Package["mysql-server"],
        }

  service { "mysqld" :
   enable  => true,
   ensure  => running,
         require => File["/etc/my.cnf"],
  }

        $mysql_server_exec_1 = "0000000000001"
        $mysql_server_file_1 = "/var/lib/puppet/flags/mysql_server_exec_1"

  exec { "restart-mysqld" :
            command => "/sbin/service mysqld restart && echo 
                       \"$mysql_server_exec_1\" > \"$mysql_server_file_1\"",
            unless  => "/usr/bin/test \"`/bin/cat $mysql_server_file_1 
                       2>/dev/null`\" = \"$mysql_server_exec_1\"",
            require => Service["mysqld"],
            }

  exec { "set-mysql-password" :
   unless  => "/usr/bin/mysqladmin -uroot -p${mysql_password} status",
   command => "/usr/bin/mysqladmin -uroot password ${mysql_password}",
   require => Exec["restart-mysqld"],
  }

  $mysql_server_exec_2 = "0000000000001"
  $mysql_server_file_2 = "/var/lib/puppet/flags/mysql_server_exec_2"  

  exec { "remove-test-dbs" :
   command => "/usr/bin/mysql -u root -p${mysql_password} -e 
              \"DELETE FROM mysql.db WHERE Db='test' OR Db='test
              \\_%';\" && echo \"$mysql_server_exec_2\" > 
              \"$mysql_server_file_2\"",
   unless  => "/usr/bin/test \"`/bin/cat $mysql_server_file_2 2>
              /dev/null`\" = \"$mysql_server_exec_2\"",
   require => Exec["set-mysql-password"],
   }

     $mysql_server_exec_3 = "0000000000003"
        $mysql_server_file_3 = "/var/lib/puppet/flags/mysql_server_exec_3"

  exec { "remove-remote-root-access" :
   command => "/usr/bin/mysql -u root -p${mysql_password} -e 
              \"DELETE FROM mysql.user WHERE User='root' AND 
              Host NOT IN ('localhost', '127.0.0.1', '::1');
              \" && echo \"$mysql_server_exec_3\" > \
              "$mysql_server_file_3\"",
   unless  => "/usr/bin/test \"`/bin/cat $mysql_server_file_3 
              2>/dev/null`\" = \"$mysql_server_exec_3\"",
   require => Exec["set-mysql-password"],
  }
 }
}


Apache manifest


class apache {

 $snat_ip = $snat_ipaddress

 case $operatingsystem {

  'CentOS', 'RedHat' : {
   $supported = true
  }
 } 

 if ($supported == true) {

  file { 'httpd_conf_d' :
   ensure  => directory,
         path   => '/etc/httpd/conf.d/',
         recurse => true,
         purge   => true,
         force   => true,
         owner   => "root",
         group   => "root",
         mode    => 0644,
   require => Package['httpd'],
  }

  file { 'cloudstack_conf' :

   path => '/etc/httpd/conf.d/cloudstack.conf',
   owner  => root,
   group => root,
   mode  => '0644',
   content => template('apache/cloudstack.conf.erb'),
   require => File['httpd_conf_d'],
  }

        package { 'httpd' :
   ensure  => present,
  }

        package { 'mod_ssl' : 
   ensure  => present,
  }

       service { 'httpd' :
         ensure  => 'running',
         enable  => true,
   require => File['cloudstack_conf'],
  }
 }
} 
 
and the ERB template: 

LoadModule ssl_module modules/mod_ssl.so

NameVirtualHost *:80

<VirtualHost *:80>
   ServerName <%= snat_ip %>
   DocumentRoot /var/www/html/
   Redirect permanent / https://<%= snat_ip %>/client
</VirtualHost>


Listen 443
NameVirtualHost *:443
<VirtualHost *:443>
   SSLEngine On
   #SSLCertificateFile /var/lib/puppet/ssl/certs/ca.pem
   #SSLCertificateKeyFile /var/lib/puppet/ssl/certs/cert001.pem

   SSLCertificateFile /etc/pki/tls/certs/localhost.crt
   SSLCertificateKeyFile /etc/pki/tls/private/localhost.key

   RewriteEngine On
   ProxyRequests On
   ProxyVia On
   <Proxy *>
     Order deny,allow
     Deny from all
     Allow from all
   </Proxy>

   Redirect permanent / /client

   ProxyPass /client http://localhost:8080/client
   ProxyPassReverse /client http://localhost:8080/client

</VirtualHost> 
 
And the iptables manifest

class iptables {
 
    $ipv4_file = $operatingsystem ? {
        "debian"          => '/etc/iptables/rules.v4',
        /(RedHat|CentOS)/ => '/etc/sysconfig/iptables',
    }
 
    exec { "purge default firewall":
        command => "/sbin/iptables -F && /sbin/iptables-save > 
                   $ipv4_file && /sbin/service iptables restart",
        onlyif  => "/usr/bin/test `/bin/grep \"Firewall configuration 
                   written by\" $ipv4_file | /usr/bin/wc -l` -gt 0",
        user    => 'root',
    }
 
    /* Make the firewall persistent */
    exec { "persist-firewall":
        command     => "/bin/echo \"# This file is managed by puppet. 
                       Do not modify manually.\" > $ipv4_file && 
                       /sbin/iptables-save >> $ipv4_file", 
        refreshonly => true,
        user        => 'root',
    }
 
    /* purge anything not managed by puppet */
    resources { 'firewall':
        purge => true,
    }
 
    firewall { '000 INPUT allow related and established':
        state  => ['RELATED', 'ESTABLISHED'],
        action  => 'accept',
        proto  => 'all',
    }
 
    firewall { "001 accept all icmp requests":
        proto  => 'icmp',
        action  => 'accept',
    }
 
    firewall { '002 INPUT allow loopback':
        iniface => 'lo',
        chain   => 'INPUT',
        action  => 'accept',
    }

    firewall { '100 allow ssh':
        state  => ['NEW'],
        dport  => '22',
        proto  => 'tcp',
        action  => 'accept',
    }
    firewall { '100 allow port 80':
        state  => ['NEW'],
        dport  => '80',
        proto  => 'tcp',
        action  => 'accept',
    }
    firewall { '100 allow port 443':
        state  => ['NEW'],
        dport  => '443',
        proto  => 'tcp',
        action  => 'accept',
    }
    firewall { '100 allow 9090':
        state  => ['NEW'],
        dport  => '9090',
        proto  => 'tcp',
        action  => 'accept',
    }
 
    firewall { '100 allow 8250':
        state  => ['NEW'],
        dport  => '8250',
        proto  => 'tcp',
        action  => 'accept',
    }

    firewall { '100 allow 7080':
        state  => ['NEW'],
        dport  => '7080',
        proto  => 'tcp',
        action  => 'accept',
    }

    firewall { '100 allow 8080':
        state  => ['NEW'],
        dport  => '8080',
        proto  => 'tcp',
        action => 'accept',
    }

    firewall { "998 deny all other requests":
        action  => 'reject',
        proto  => 'all',
        reject  => 'icmp-host-prohibited',
    }
 
    firewall { "999 deny all other requests":
        chain   => 'FORWARD',
        action  => 'reject',
        proto   => 'all',
        reject  => 'icmp-host-prohibited',
    }
}
 


Wednesday, March 20, 2013

Cloudstack 4.1 - Regions are coming to town!

 







Here a list of new features in Cloudstack 4.1 from a presentation that Chip Chandler gave.











 


There are a few new features, but I though I would talk about AWS style regions:

Uhem.. AWS style regions you say? This is a BIG move. It's taking Cloudstack zones and adding a new layer on top. Well why would you want to do that? Cloudstack zones do not spread well geographically. In fact multiple Cloudstack zones are often deployed within a data center to work around the 4095 vlan limit by keep the vlans within a rack and using L3 routing to move to another rack. That is ok, but customers cannot easily route between zones (yet). Having regions will allow additional segregation, with one important new feature which is splitting up the Cloudstack management server, so each CSM will manage one region. This is perfect is your customers are spread geographically.

Here are the requirements around regions in Cloudstack:

• Admin should be able to install a management server for a Region and then connect to management servers in other Regions. Each Management Server in each region should know about all other regions and their management servers. The following operation allows for each CSM server to know about the other CSM in the other region:
 

Connect To Region
 

• The CS Admin should be able to create multiple Regions within a CloudStack cloud. I.e., Cloudstack should manage multiple regions within a cloud. The following operations should be supported:
 

Create Region
Delete Region
List Regions
 

• Admin should be able to create multiple zones within a Region. Management server cluster manages all the zones within the region. Following Zone operations should be supported on per regions basis:

Create Zone
Delete Zone
List Zones

• Management Server cluster for each region should only control the infrastructure within
that region. In other words, if a management server cluster fails, then only the region managed by that server cluster should be inaccessible for admin and users. It should not have any impact on the other Regions in the cloud
 

• Each Region should have access to an object store. The object store should be common to all Zones within the Region. I.e., any object stored from a Zone should be visible across
all the other zones within the region
 

• EIP function should be available across both basic and advance Zones within a Region
 

• ELB function should be available across both basic and advance Zones within a Region
 

• The administrative hierarchy  (Users, Accounts, Domains, Projects) - should be valid across all the regions. I.e., if admins create domains, accounts and users etc. in one region it should be reflected in other regions as well.

This is key. I create a user in AZ 1 and I want that user to be able to create a VM in AZ2 using the same credentials, api and secret key.

• Authentication credentials – username, password, keys – should be valid across all
regions


^ Oh, I just said that! ^

• Switching between Regions should not require the user to sign-on again (SSO should be supported).

• Resource management should be extended to Region level
 

Available compute, storage, network (IPs, VLANs etc.) resources that are currently tracked
should be aggregated and displayed at region level


Appropriate global parameters should be available at Region level while the remaining would be available at Zone level
 

• Usage: All the (per account) usage records should be aggregated to Regions level

This is important for billing purposes.

• Global Configurations: All of the administrative hierarchy (Domains, Accounts, Users, Projects) related Global Configurations should have consistent values across all regions and has to be propagated when a user modifies a configuration on one of the regions.
• Each region should maintain a list of all other regions and their region endpoints.


As you can see, this really is a big change. How people will be able to upgrade will also be a big area of work. However this is way the product needs to move to ensure that it can keep delivering a top class service.

Here are a couple of links that you can look at to find out how the guys did it. This is why opensource is so great. No hiding, which means we can all get a better idea of how this beast will work

https://issues.apache.org/jira/browse/CLOUDSTACK-241

https://issues.apache.org/jira/browse/CLOUDSTACK-815