• If you are citizen of an European Union member nation, you may not use this service unless you are at least 16 years old.

  • Stop wasting time looking for files and revisions. Connect your Gmail, DriveDropbox, and Slack accounts and in less than 2 minutes, Dokkio will automatically organize all your file attachments. Learn more and claim your free account.


The Hardware Dilemma

Page history last edited by Elizabeth Leddy 10 years, 4 months ago

So you aren't the hippest kid on the block and you want to run your system on real hardware instead of in the cloud/vm/blah. Don't be ashamed - there are plenty of reasons to do this* and if you got it you might as well flaunt it! Whether you are in the cloud or in the server room, you have to decide how to distribute the software amongst your boxes get the best bang for your buck when it comes to hardware utilization.


Ideally, we can all sit down in our comfy desk chairs and think about all the software that we want to use and all the beautiful high availability plans and then we can go get all the hardware we need to fulfill these yada yada yada dreams at the local grocery store. In reality, many of us are limited in our hardware budgets and we have to start with hardware and fit the software to that purpose. For this discussion, I will assume that's the case but feel free to adopt this simple technique for the rest of your evil planning.


The easiest way to optimize resources is to draw a simple diagram, putting your desired/potential software stack in the diagram like so:



And just like that you see why squid could be a drain on your system if you only have one box! If you have two boxes laying around you may want to consider putting zope and apache on different boxes to have better control over the drive of the cpu. Grab a piece of paper and play along at home!


Please note that if your zeo and zope instances are on the same box, you will reduce your disk overhead because you no longer have the need for a local db cache. This isn't necessarily possible for all setups, especially larger ones, but small and medium setups should take advantage of this schema.


Relating to Requests/Second

There is also the question of how man requests/second you need to be supporting.  This way you can get an idea as to how many boxes, real or virtual, you need to handle the response. This is a harder question, and I personally think most benchmarks are impossible to compare/trust/believe because they aren't necessarily measuring "real" page renders. A "real" page render is not a gif or js or css and it is NOT cached - it's a moderately complex page render. Remember, zope should only serving up pages in your system that need zope to be rendered. If you get your info from a consistent source you are better off but everything varies so much its just about impossible to get a realistic number. If you have tools setup - yay - use them. For those that are getting started, here's a quick and dirty guestimation technique that's worked really well for me in the past. This is definately an overestimate but at least you shouldn't get any nasty surprises this way.


First let's think about what types of pages we are rendering and classify them according to complexity. If you are doing any custom development, chances are you are generally doing moderately complex page renders (very complex if you have crappy coders). Edits are very complex page renders. Searches are anywhere from moderately complex to disgustingly complex. Don't even think about counting reporting. In general, views are simple. 


Now think about what percentage of each you will have. In a tough app that gets a decent number of writes, you may look at 40% very complex, 30% moderately complex, 30% easy renders. With a Plone 2.5 baseline (theoretically subtracting 30% for each major release after that if you follow the trend lines) you can safely assume 1 second to render a very complex view, .5 second for a moderately complex, and .25 seconds for a simple view. If you had 100 requests then, it would take 40 + 15 + 7.5 = 62.5 seconds of server rendering time, or .625 requests/second per zope instance. Seems accurate for the 2.x series, no? (Note: these numbers are artifically high because every system is different - don't get mad if you get more or less).


Almost there. Now think about your PEAK load - how may users will be using the system simultaneously? For simplicity, assume they can view one page every 2 seconds. If you have 100 simultaneous users always rendering a fresh page view, you then have 50 requests per second. To handle this load, you need 50 * .625 = 31 zope instances. If you oversubscribe your zopes 2:1 to the number of CPU's and each one of your boxes has 4 CPUs, then you are looking at 7-8 machines to handle those zope renders. That seems like a lot of hardware, but remember this is the raw rendering of zope application logic and when you incorporate all the caching and other system components, your total system throughput is going to be many more requests/second. 


Remember, zope is not apache and its doing some amazing logic behind the scenes. Be patient with her and keep in mind that each release is getting better and better w.r.t speed.


* There is a nice long discussion on virtualizing Plone in google wave. For the life of me I don't know how to give you a url for it but its public and just search for "Virtualizing Plone" to get the skinny. I'm not sure if I sure dig out the pros and cons of a virtual environment here - anyone interested in that?


Services, Caching, and Maximizing RAM

In the past I have preached a lot about having all service and async requests routing to a non-user servicing zope. There is a lot of merit to this and I saw the difference in response time immediately when this got changed. HOWEVER, it also meant that the servicing zopes had a cache for every client. In our case, certain zopes service certain clients (thanks HAProxy!) in order to optimize the speed of response by hitting an already filled cache. Makes sense. But then the service zopes had to have a cache for everyone, one that was often 4 times the size of any other zope. That became a waste of memory that was better utilized for client based zope and a bunch of disk requests to fill a cache. Additionally, services responded much slower because they were waiting for disk instead of utilizing an already existing cache. In the end, we switch back to having services on the same zopes as user requests.


If you have a lot of services or async requests, consider the advantages and disadvantages of having a zope just for handing services. Long lived requests will slow down your end users. That makes us respond with - ok we'll but them on there own zope. But then, the requests become even longer because of cache priming and your RAM gets chowed by service zopes. In the end, the best solution is to dump those long live requests by any means necessary. In the case that you can't, measuer carefully and consider all options.

Comments (0)

You don't have permission to comment on this page.