Bugs:

- deferred mail was used as default FIXED (now default is send instantly)

- sender email adress was not correct set FIXED

Enhancements + features:

- upgrade to qooxdoo 1.4

- rather deals with contact email if available, fallback to email, if not available

- automatic polling for new e-mails

- change polling interval in settings

- message history changed to message center

- show inbox bevor outbox

- doubleclick on an email or a line in the mail table - large view / normal view

- added button for maximize the view

- allow copy and paste in read mail window

- quote the incoming mail into the mail answer

- set state of e-mail to read after 1 second

NoSQL - a new hype? 

The idea of NoSQL databases is not so new. Document-oriented databases like Lotus Notes are not so far away from what we call today a NoSQL (not only SQL) database.

I followed a lot of discussion, if NoSQL databases in general are better than the conventional relational databases. In which cases one should use them and in what circumstances it really needs a relational database.

Some new web projects are based on NoSQL databases, like couchDB, Redis or mongoDB. And mostly project leaders and developers seemed to be much exited.

My practical problem

I own a shop system. No, it's not really a shop system; it's more like a big mySQL table with currently round about 800.000 data rows. Nightly my server downloads a bunch of product data from Affiliate networks and tries to add categories and subcategories. There are some light relations, e.g. between the product table and the category table. Also a fulltext index is added to give users the possibility to search e.g. for all products, having a special substring in fields like productname, description, shortdescription, brand or so.

Although I used some tricks like deactivating all indexes while download and insert, the whole process of downloading and indexing of these data takes some hours.

I saw the limit with this technique: I would run into problems, if I would add some more new partners with a bigger product portfolio.

In production environment my server is very busy to find and deliver selected data, especially if one uses fulltext search.

I decided to give a chance to a NoSQL database. I heard, they are good for handling large amount of mostly data. And that is my use case.

Selecting the right database system

There are some database systems out there, and they are good depending on your tasks and environment.

If you mainly want to access the database with REST, e.g. with you JavaScript RIA application, then CouchCB could be the system of your choice. Have a look at cassadra, if have to handle many writes to your database.

MongoDB is very fast in reading (the main task for most web applications) and has excellent horizontal scalability. Search for mongoDB autosharding, if you want to know more.

If you want to know more about the differences between mongoDB and CouchDB please follow this link.

http://www.mongodb.org/display/DOCS/Comparing+Mongo+DB+and+Couch+DB

I liked the idea, to save and access JSON data - and mongoDB uses the BSON format - that's a more compact JSON format. I looked for the availability of native drivers for my projects language, and PHP is as good supported as many, many others. I had a view on performance and scalability. Main focus is the web application. 

Installation and first contact

MongoDB was easy to setup, it was not hard to find a good management console, written in PHP, it's easy to access the database directly in the mongo console, using JavaScript. I installed mongoDB on two different Linux distributions, and there where no problems. I installed it on my Windows, and I had no problem. With a 32 bit system you only can access 2 GB databases.

Transferring the schema

What does that mean? MongoDB is like all NoSQL databases schemaless.

In SQL databases you have a fixed set of fields in a table.

Comparing to a table in a SQL database, in mongoDB we have a collection. But such a collection may have datasets (called documents) of every structure.

In practice you would save your data of one type in one collection. But often you would embed things like simple lookups.

These days I’ve seen a database table with contact entries. One contact in this table has 115 fields, it’s not a joke. There are phone1, phone2, phone3, private_phone1, private_phone2, private_phone3, adress1, address2… and – may be the designer forgot some fields there are 20 fields for free usage.

In mongoDB you just fill the fields you need. And if you add a new field later, so what? You don’t have to change your schema. All contacts without this new information are not affected.

The second difference is: You often embed a simple lookup into your collection.

I’ll give a simple example:

You have a table of staffs and a second table where the children names are saved. The children table contains the id of the associated employee.

In mongoDB you could create a collection "staffs".

The data of that collection could be look as follows:

{

  _id : ObjectId("4f9d9183eabc88c012000118"),

  name : "Smith",

  surname : "Hajo",

  children : {

     [

 { name : "Paula" },

 { name : "Paul" }

     ]

  }

}

The _id is given by mongoDB and is used as primary key, but you can define your own if you need to.

Can you see, how easy it is, if you get more information about Paula? You simply update the document to this:

{

  _id : ObjectId("4f9d9183eabc88c012000118"),

  name : "Smith",

  surname : "Hajo",

  children : {

     [

 { name : "Paula", birthday : "2011-02-02"},

 { name : "Paul" }

     ]

  }

}

One more thing you can see: you should use short names for the keys to keep the database storage space as small as possible. I would change the structure to:

{

  _id : ObjectId("4f9d9183eabc88c012000118"),

  n : "Smith",

  sn : "Hajo",

  c : {

     [

 { n : "Paula", bd : "2011-02-02"},

 { n : "Paul" }

     ]

  }

}

That saves a lot of disk space.

Notes from praxis

Inserts:

Downloading and inserting of round about 1 Mio of data rows, creating the necessary indexes including fulltext takes round about 3 or 4 hours with mySQL.

Now I insert the data in round about half an hour into my mongoDB. Creating some additional order indexes takes also half an hour.

Queries:

Searching for indexed keys is very fast. But note: you should have enough memory. MongoDB is very fast as long as the database size fits in memory size.
It's very easy to create the queries, you only have to specify key / value pairs for the "WHERE" criteria.

As a result you will get a cursor, which you can iterate. The cursor also may have additional properties like sort.

Are mongoDB or other NoSQL database like Cassandra interesting for elgg?

In large Elgg communities we see some performance problems. One challenge is to give Elgg system managers a more performance stable system independent of the amount of users.

Using of NoSQL databases may be hype, but it would be worth to find out if it's possible the best way for the future for Elgg large communities, the best technique to handle huge databases.

In Elgg you have many relationships between several tables to archive one goal: Saving of more or less schemaless information. Look at the user's profile: If you want to add an additional field, Elgg saves it similar like mongo does. Not in the same structure, but Elgg creates e new key / value pair and saves it in relation to his entity.

I assume Elgg is an ideal candidate for a schemaless NoSQL database, much more suitable for that task than mySQL or any other relational databases. As in Elgg you can archive to add or remove a property without changing the (not existing) database schema.

MongoDB has autosharding build in - that means: Scale horizontally about as much machines as you like without compromising functionality.

Moving Elgg from mySQL to noSQL database would be possible and worth, but I can't give estimation for effort. May be, it's not so complicated, if only the database layer has to be replaced. But I fear that SQL queries are built on several parts inside of Elgg core or plugins.

This is release 0.9 of my plugin, which already gives some nice options for managing your users. Did you know, how many users in your communiy use the registration form, but never do a click on the validation link in there e-mail? Simply look at "Not activated users" and delete these dead records, if you like.

Download or visit related Page at elgg.org

The plugin is for admins and for normal users; admins of course have options, like delete or edit user data.

The instruction manual comes in form of some screenshots with comments. Please have a view at the following images.

The last picture shows a planned feature, the SMS functionality of the plugin.

This small extension shows a download counter on the right hand side of the file download button. (Default Elgg file plugin) To install, download the zip. Unpack in the mod file and activate it. Take care, it is below the file plugin.

Actual version is 1.6. Download here or on elgg.org

You will be added as a member after my confirmation of your order.

Please select

1 website $33,00 USD 5 websites $99,00 USD

Please enter your username

Don't want to use Paypal? Please drop me a note for other payment options.

Please note: There is also a free version of this software available under elgg.org. The free version has some restrictions, e.g. you are not able to manipulate elgg entities with this plugin.

WestorElggMan Professional Facts:

Pricing:

• 33,- USD (1 website)
• 99,- USD (5 websites)

If you like to order an license for more then 5 websites, please contact me.

License:

This software may be used under the terms of the Eclipse Public License (EPL). Please read full license info.

Features:

This plugin is helpful for admins and for normal users. A comfortable AJAX interface helps to manage users, friends, external contacts, groups and messages. One can send bulk e-mail and SMS to a bunch of users or friends. You can easily search in your messages history.

Admins have more options then normal users, like delete or edit user data, block or unblock users, activate or deactivate them.

As an admin you can inspect Elgg entities with all their properties and metadata.

The professional versions also allows the manipulation of an elgg entity.

Terms of use:

You agree to the license conditions. See license item.

I track the websites where you want to use this software. You have to enter the websites urls in the groups bookmarks section - instructions are given inside.

In some cases you may get free updates for a bugfix or for compatibility with a new elgg release. But please note: There is no right for free updates, and it may depend on the effort, if I can offer a free fix or if I have to charge for that work.

I disavow any responsibility for possible data or function loss when using this software. Please test it thoroughly for your needs and make backups of your data.

Ordering now:

To order the product now, you have to do make your payment here: 

Please select

1 website $33,00 USD 5 websites $99,00 USD

Please enter your username