The Relenta CRM blog

It goes without saying that we are constantly looking for ways to perfect Relenta. We want the best speed and uptime for you, reduced costs and maintenance for us, and peace of mind for all. Greedy? Unrealistic? Impossible? These words aren’t in our vocabulary.

Sergey Gondrya, CTO at Relenta

To our astonishment, we quickly found the perfect solution that meets our seemingly conflicting requirements (its name is Slicehost). On the other hand, convincing ourselves that this is the right choice took quite a bit of time. And that’s what this post is all about.

“We” really means Sergey Gondrya, Relenta’s CTO and system architect extraordinaire. I just wrote it all down.

We’ll start with our wish list, talk about the Relenta system architecture, the process of elimination we went through, and the tests we ran to justify our decision.

What matters to us

Because Relenta hosts your email and sensitive business data, we take availability, security, and performance much more seriously than providers of less mission-critical software services might be able to afford.

As Winston Churchill, we are “easily satisfied with the very best.” Here is our wish list. In no particular order, we want to…

• Focus on developing and servicing the application instead of worrying about hardware monitoring and maintenance.
• Be in control and do almost whatever we want with the system without contacting support. This includes provisioning additional capacity.
• Have storage that is fast, redundant and scalable right out of the box. Cheap, too.
• Spend less on our monthly hosting bill and increase performance at the same time.
• Know exactly how much we’ll pay for hosting on the next billing date. Estimates won’t do.
• Have the best possible data center connectivity for customers across the globe, not just the US.
• Be powered by a brand that is respected by geeks and recognized by the majority of our prospective customers.
• Scale up easily, inexpensively and in a heartbeat.

It all comes down to the right balance between freedom, control and money. Doesn’t it always?

Relenta system architecture

Inspired by Google’s infrastructure, Relenta is designed to run on generic, non-specialized, cheap servers with no dependence on hardware type or manufacturer. The system can also be run easily on a single server. (In the near future, you will be able to download, install and run Relenta on any x86 hardware on your premises.)

Relenta is a single-instance, multi-tenant application. Customer data is sharded. Relenta is written primarily in PHP 5 with Java modules and shell scripts.

Currently Relenta runs on RedHat Enterprise Linux 5 OS, nginx and fastCGI webservers, and MySQL 5.1 database. The system is webserver-agnostic and can be easily modified to work with any other database server.

Relenta architecture

• Master database contains meta information about customer databases. Replicated for high availability.
• Data nodes are horizontally partitioned and may contain one or more customer databases which can be easily moved to a different node. Each node is replicated for high availability.
• File storage is used to store information that doesn’t require or is not suited for relational database, such as documents, email attachments, originals of incoming email, data export files, etc.
• Search. We use Apache Lucene for email search and will soon expand its capabilities to contact search.
• Fetchers retrieve email from remote POP and IMAP servers. In the future this functionality will be expanded to include social network messaging (Twitter, Facebook, LinkedIn) as well as RSS.
• System queue is an asynchronous internal messaging and task executing system.

SoftLayer was awesome

Our servers have been hosted at SoftLayer for a few years. It’s been a fantastic experience. Hardware upgrades aside, we’ve never had any need to contact support. SoftLayer’s uptime record is impeccable.

One thing that perplexed us at SoftLayer was its monitoring system, which is unpredictable and inconsistent (to be fair, we haven’t applied any effort to fix or adjust it either). Instead, we used Pingdom, which provides great value and quality monitoring from a dozen or so locations across the globe.

Other than monitoring, we’ve had no gripes with SoftLayer. In fact, we learned quite a few things from the way this company operates. We highly recommend it to anyone who is looking for self-managed dedicated hosting.

As for Relenta, we simply wanted to see if we can delegate the hardware management responsibility without paying a king’s ransom for managed hosting.

The process of elimination

There are only three hosting options, really: shared, virtual (VM for virtual machine or VPS for virtual private server) and dedicated. Both flavors of the dedicated option are a no-go for us:

1. Self-managed, the kind we’re moving away from.
2. Managed, the kind that is way too costly.

Then there is much hoopla about cloud hosting. The term cloud is used as a metaphor, based on the way internet is pictured on the network diagrams. It represents an abstraction for the concealed hosting infrastructure, and for that reason it would seem reasonable to apply the term only to shared and virtual environments.

As much as we dislike vague buzzwords, we have to admit, the word cloud has a very nice ring to it. Whoever came up with the term is a genius. Or was it the wisdom of the crowd?

Anyway, a quick look at the shared cloud hosting solutions, such as Rackspace’s CloudSites, revealed functional limitations similar to those of shared hosting. Because Relenta requires Java, shell scripts and custom server-side components, shared-type cloud hosting turned out to be unsuitable for us. Nevertheless, CloudSites is a major step up from a familiar shared hosting environment, because it offers load balancing, clustering, redundant storage and easy scalability straight out of the box.

What’s left is a virtual-style cloud hosting. Back in the day when I was deeply involved with the web hosting industry, we called it VPS. A broader term that seems to be preferred today is VM. The truth is, there isn’t much difference either way.

Virtualization technology has come a long way since we first tried it a few years ago for another project. At the time, the overhead of virtualization was too high. The technology seemed to be not quite there, so we decided to give it some time to ripen. And so it has.

Among VM providers we quickly narrowed our choices to Rackspace’s Slicehost and Amazon’s EC2, then almost instantly discarded the latter and finalized on Slicehost. The rest of the time was spent on testing and convincing ourselves that our decision was spot on.

Rackspace’s Slicehost vs. Amazon’s EC2

Slicehost is a VPS company “built for people who know their stuff.” They use customized Xen VPS back-end to manage slices ranging from 256MB/10GB for $20 per month to 15.5GB/620GB for $800 per month. If you use your slices for only a part of the billing cycle, they will be prorated based on the number of days used. Simple and straightforward. In October 2008 Slicehost was acquired by Rackspace. Our slices are physically located in the mothership’s data center, which was one of our critical decision-making factors.

EC2 stands for Elastic Compute Cloud that “lets you run on Amazon’s proven computing environment.” EC2 “changes the economics of computing” by charging you for the actual usage. How much? We gave up trying to figure it out. If you want a good example of what simplicity is not, take a look at EC2′s “simple calculator.”

Slicehost gave us a temporary free slice to play with. We didn’t ask EC2 for a free test drive, because we didn’t want to embark on time-intensive research not knowing what our costs will be. We wanted a fixed monthly per-unit fee. With EC2, the exact fees are impossible to determine without setting up and running the system under full payload. Life is too short for that.

Another level of uncertainty for us was related to support. To our minds, Amazon is in the business of selling stuff and gathering knowledge about our buying behavior along the way. Are they as qualified to offer hosting services to the public as Slicehost, a division of Rackspace who built its reputation on fanatical support over the years? We didn’t want to find out the hard way.

Long story short, as much as we were intrigued by Amazon EC2, we didn’t feel like opening the Pandora’s box.

The Slicehost experience

Slicehost did well on two benchmarking tests, MySQL sysbench and Bonnie disk. Our servers at SoftLayer had SATA drives in RAID5 configuration. Slicehost uses undisclosed drive types (we strongly suspect SAS) in RAID10 which would have been cost-prohibitive for us in a dedicated server environment.

MySQL sysbench

Transactions per second:

* as reported by RightScale

Bonnie disk

Private network speed

The speed of private network is 5x faster at SoftLayer compared to Slicehost (20-25MB/sec vs 4-5MB/sec), but we can live with that since it isn’t a bottleneck for Relenta performance.

Support

We didn’t require much support during the setup and migration process. Any questions, regardless of their perceived or actual importance are answered by Slicehost within minutes regardless of time of day. For completeness sake we will mention one issue that Slicehost dealt with proactively and swiftly. We were notified that our slices were initially placed on the same physical machine:

Typically our load balancing algorithm will select distinct hardware on creation, so its likely that at the moment you added those servers we were in the process of adding capacity and not enough spare servers were available to properly space them out.  If you’d like to have those slices relocated to different hardware, please reply to this message and we should be able to take care of it right away.

This was indeed taken care of within minutes.

SLA

One other thing we feel compelled to mention is SLA (service level agreement), which Slicehost doesn’t offer. Because they are a part of Rackspace’s brand and data center, we are not concerned. The way we look at it, if a hosting service goes down for more than a few hours, any SLA issues will be the least of our worries. Any agreement is only as good as the company that provides it. In fact, one can argue that if a hypothetical hosting company goes down for say 10 hours and as a result doesn’t collect one month worth of its revenue, it might eventually put customers at greater risk because of the disrupted cash flow.

Like a glove

All things considered, Slicehost fits all of our criteria almost like a glove:

• We can forget about the hardware. Yeah!
• Our expenses and performance are predictable.
• It takes two minutes to provision a new slice from the control panel.
• For storage, we can use Rackspace’s CloudFiles which offers functionality and pricing similar to Amazon CloudFront. For detailed comparison, see Mosso’s blog.
• We conservatively estimate that we end up paying anywhere from 30% to 70% less for the same end result compared to the dedicated servers at Softlayer (the variation is due to the multitude of configuration options and a few wildcards). This is the hard cost that does not include indirect savings.
• Traceroutes that we ran from Latvia, Australia, Germany, UK and several locations across the US showed between one and five fewer hops compared to SoftLayer.
• We ran informal brand recognition tests when talking to prospects during the past few weeks. Whenever Relenta’s strategy for reliability and uptime would come up, we would mention that we’re in the process of migrating from SoftLayer to Slicehost, a Rackspace company. A typical response was “Rackspace, good for you.” Unfortunately, not many people heard of Softlayer before. Could this be because our prospects are small businesses, while SoftLayer targets enterprise customers?

That’s it! As with any hosting provider, only time will tell how good of a fit it is. We have good feelings about Slicehost and hope you will do too.