| 
  • If you are citizen of an European Union member nation, you may not use this service unless you are at least 16 years old.

  • Files spread between Dropbox, Google Drive, Gmail, Slack, and more? Dokkio, a new product from the PBworks team, integrates and organizes them for you. Try it for free today.

View
 

Tuning

Page history last edited by Jean 8 years, 1 month ago Saved with comment

Tuning zope.conf

 

How many threads?

One thread! The GIL isn't a very good context switcher, so just let the OS do what it does best.  To add more concurrency then, just add more Zope instances.

 

The only time where you should use more threads is in the case that there is the potential for a web service deadlock. For example, Plone contacts service X which needs more info from Plone ... and deadlock. You shouldn't code your services for this, but it happens to the best of us in unexpected ways. If you know there is the potential for this to happen, use 2 threads to be safe (this way you can handle 2 requests at any given time).

 

So what the real question is here, is how many Zopes? Simply put, as many as possible but not a single one more. Start with by oversubscribing 2 Zopes per CPU core. So, if you have a 4-core CPU, start with 8 Zopes. The goal is to utilize 50% of load on average so that if things get crazy the server can spike to 100%. If you aren't at 50% load on average, add more Zopes. If you are higher than that, reduce the number.

 

An even better question is how many Zopes do I need to handle X requests per second? Glad you asked! It depends. IMHO, a lot of the figures on Plone performance are very misleading since they assume you are utilizing caching and/or serving silly things like user.gif. "Requests per second" should really be called "Rendered requests per second" and any long term Plone administrator will tell you that is a much lower number. For example, with Plone 2.5 and custom archetypes, we squeeze out 2.5 rendered requests per second on average. Of course some pages will be faster and some (especially writes) will be slower and you, Mr or Mrs Sys Admin, need to measure that. Be realistic and don't expect a lot. If you want to handle 10 requests per second with an average of 2.5 requests per second, then you are looking at managing 4 Zopes.

 

The goal in any web app should be to render EVERY page in 200-300 milliseconds. If your page is rendered in 300 milliseconds, it will get to the user in about 1 second, which is the limit of most users' attention span. If you have pages which take significantly more time (i.e. reports), consider setting some Zopes aside to specifically handle long running requests. You want fast to remain fast, and those long requests will affect your system.

 

How much ZEO cache?

The ZEO cache is a disk buffer (usually in /tmp) of all the objects you are accessing from a ZEO client (i.e. a Zope instance). This is especially useful in the case that you are not running Zope and ZEO on the same box and you have to make a network request to get your db objects. If you run your ZEO and Zope on the same box, then the answer is: none! No need to waste /tmp space. (I think that you can't technically set it to 0, but set it close to that and it shouldn't make a difference). If you are NOT on the same box, then set it as high as you can without blowing your tmp directory, keeping in mind that the OS will utilize available RAM to cache that disk movement.

 

How many ZODB cache objects?

The simple answer is: as many as you can. This is different than the ZEO cache in that its a cache of the objects, entirely in memory for performance on the SAME machine. If you have been a good boy or girl and separated your portal_catalog on a new mount, try to keep the entire catalog in memory. For example, set this to 100,000 and move up from there. These objects are so small that you won't see memory fill up too much.  

 

As far as non-catalog mounts are concerned, the answer is not so clear. Start as high as you can, e.g. 40,000, and monitor performance. Keep lowering until performance is impacted.  If you immediately get worse performance, then turn around and start adding more. If you are worried about a RAM-hungry process, setup monit to restart your Zope once it reaches x% or RAM. It's better to restart and have performance than to be proud of a long-running, under-performing service. 

 

* Any other suggestions here?

 

Tuning the operating system for ZEO

Do any lookup on tuning Linux for hosting databases, read, understand, and apply.  I don't think you are going to get any huge performance gains from any of these (I didn't see any noticeable difference) but they didn't hurt so... here they are. In general, you will get lots of tips from googling "TUNE ORACLE LINUX". If an OS can handle Oracle, it can handle ZEO.

 

Changing Disk Algorithms

Of all the things you can tweak the OS itself, changing the scheduler algorithm on the zeo server box is a good place to start. Myroslav has some info to put here....

 

Messing with noatime

Linux marks every access to files in the system, which can hurt database performance. Changing Data.fs et al to not record this is simple and easy.

 

Optimizing with multiple disks/SANs

Of course, the best thing you can do is shard your dbs and put them on different disks, hopefully as part of a bigger/faster/awesomer disk array. At minimum, put your catalog on a different disk than the rest of your dbs and do NOT log on the same disk either. That's 3 disk partitions/mounts minimum. Plan for this from the beginning and your life will be much easier in the long term. 

 

Do you RAID?

If you use RAID 5, none of this will help with the pain you are about to suffer. If you can afford it, go RAID 10 and if you can't, stick to RAID 1. 

 

Setting Check Intervals

http://pypi.python.org/pypi/jarn.checkinterval

 

Tuning Zope and Plone for speed

 

Modules to install

There are some modules that improve certain aspects of Plone performance, for certain versions of Plone. Depending on your usage and version, these may be applicable:

  • experimental.aggressiveopaquespeedup
    This helps with the traversal through the site. In Plone 3 if you traverse through a folder with a large number of objects you wake up all objects in that folder. Typically this occurs when you have something like a 'news' folder and you select 5 items for the homepage. You actually load al items in that folder from the ZODB.
  • experimental.catalogqueryplan (you will need to pin to the last Plone 3.x version which IIRC is 3.0.2)
  • experimental.contentcreation
    This speeds up the act of initially creating new content (e.g. clicking 'page' from the add new menu). It puts in a 'dummy' catalog in portal_factory which prevents the object being indexed/unindexed as it is being created.
  • experimental.daterangeindexoptimisations
    You will need to re-index your catalog after this to take effect, but it optimises the date indexes for you.
  • Run the catalog optimise script:  https://github.com/hannosch/scripts/commits/master/catalogoptimize.py
    This will optimise the BTree buckets in your catalog

     

Monitoring

  • Look into collective.stats for monitoring ZODB level activities (not sure how well it plays with ZEO). This will tell you if your 'reads' are actually doing 'writes' to your database and how many objects are loaded and stored for each request.
  • You can get some information about the use of the ZODB cache (the most important cache relevant to ZODB usage) in the ZMI via "Control_Panel" --> "Database Managemant" --> "main" --> "Activity".
    It shows use information about number of writes, reads and connections. If the number of reads is high with respect to the number of connections (= requests), then increasing the ZODB cache size may be advisable.
  • If you heavily customized your Plone: use ZopeProfiler or PTProfiler for profiling the rendering of a typical "slow" page.
  • You must get an idea how many requests per second you receive in peak times and check how long processing takes (Zope's requestprofiler may help here).

Zope cluster configuration

  • Isolate editors to a particular load balancing pool.  The editor pool should be different from the public/anonymous pool.  
http://scalingplone.pbworks.com/w/users/#view=all_users
you also might want to pin zope.i18nmessageid to 3.5.1. This helped a lot on our

Comments (4)

(account deleted) said

at 11:16 am on Dec 19, 2009

Starting with Plone 4, we changed some of the defaults (via plone.recipe.zope2instance):

python-check-interval is now configurable directly and defaults to 1000 instead of Python's default value of 100. This gives the main benefit and experimenting in the 1000-2000 range usually doesn't have much of an impact anymore.

zserver-threads now defaults to two instead of four. One is better for production, but won't be optimal in simple a non-ZEO setup.

(account deleted) said

at 11:19 am on Dec 19, 2009

Regarding the ZODB cache, using plone.app.blob (or Plone 4 ones it's out) changes the game quite dramatically. With large binary content no longer in the ZODB and more importantly no longer in the ZODB cache at all, you'll be able to crank up the cache size number to a much larger value. The key here is to have monitoring in place and reevaluate the cache size after the update.

(account deleted) said

at 12:21 pm on Apr 20, 2010

Regarding memory: Read http://n2.nabble.com/Severe-memory-leak-in-zope-i18nmessageid-fixed-td4917950.html#a4917950 and update immediately to zope.i18nmessageid 3.5.1 or later. At this moment this is only included in the yet unreleased Plone 4.0b3. The 3.5.1 version works with any Zope 2.10 / Plone 3.0 combo. Saved us memory growth of 1gb / day. Much more RAM to allocate to zodb caches and no more scheduled restarts :)

Elizabeth Leddy said

at 1:14 pm on Apr 20, 2010

thanks so much for fixing that!

You don't have permission to comment on this page.