Erlang ETS types read/write performance tests

Categories Erlang

Run:

 

A simple tcp server in Erlang

Categories Erlang

Run:

Test:

 

 

Receive messages with priority in Erlang

Categories Erlang

 

Compile and run:

 

 

 

A simple gen_fsm example

Categories Erlang

My ongoing journey in the wonderful world of Erlang has been with the help of 2 books. I started off with Learn You Some Erlang For Good which is surely the best beginner guide you can ever get on Erlang language.

This is a rewrite of one of the examples from the above book. The original code is at http://learnyousomeerlang.com/static/erlang/dog_fsm.erl which we will convert to a module with gen_fsm behaviour.

From http://learnyousomeerlang.com/finite-state-machines :

A finite-state machine (FSM) is not really a machine, but it does have a finite number of states. I’ve always found finite-state machines easier to understand with graphs and diagrams. For example, the following would be a simplistic diagram for a (very dumb) dog as a state machine:

Here the dog has 3 states: sitting, barking or wagging its tail. Different events or inputs may force it to change its state. If a dog is calmly sitting and sees a squirrel, it will start barking and won’t stop until you pet it again. However, if the dog is sitting and you pet it, we have no idea what might happen. In the Erlang world, the dog could crash (and eventually be restarted by its supervisor). In the real world that would be a freaky event, but your dog would come back after being ran over by a car, so it’s not all bad.

NOTE: I assume that you already know what gen_server is and how it works, since we will be making couple of comparisons with it later.

Let us start with the diagram given above. It is very evident that at any given point in time, our dog will be in one of these states: sitting / barking / wagging tail. For each of these states, gen_fsm callbacks should be exported. The callback function will handle events that occur when our dog is in that particular state.

For example, callback functions sitting/2 (StateName/2) and sitting/3 (StateName/3) handles all events when the dog is “sitting”. Compared to a gen_server implementation, sitting/2 is same as gen_server’s Module:handle_cast/2 and sitting/3 is equal to gen_server’s Module:handle_call/3, i.e., sitting/2 contains code to handle events that are called asynchronously, and sitting/3 contains code to handle events that are called synchronously when our dog is in sitting state.

In any of the FSM states, an event can be triggered in 2 ways: synchronously (using gen_fsm:sync_send_event/[2,3]), or asynchronously (via gen_fsm:send_event/2). These functions can be related to gen_server:call/[2,3] and gen_server:cast/2 respectively.

Speaking of states, it is important to know that the FSM states we discussed above is different from the “State” of a gen_server which is used to store data related to that particular process. A FSM process does have the “State” to facilitate storing of data, but also has another “state” which indicates whether our dog is currently sitting or barking or wagging its tail. The state information of a running FSM process can be known via sys:get_state/1 function (more on that later).

Coming back to the initial diagram, notice that change in our dog’s state happen due to 3 actions: gets petted / sees squirrel / wait (timeout). The first two are result of external actions, so we export API functions for it, namely pet/0 and squirrel/0. Timeout will be discussed later.

Now, let us start coding dog_fsm.erl:

Nothing fancy here. We declare our module “dog_fsm” as an implementation of “gen_fsm” behaviour. Then we export API calls start_link/0 and stop/0 to … well, start and stop the FSM. As we discussed above, in order to change our dog’s state, we either pet it or show him a squirrel. This functionality will be made available via pet/0 and squirrel/0.

Now let’s come to gen_fsm callbacks. The below functions are required for gen_fsm behaviour to work and cannot be left out, so we export them:

init/1, handle_event/3, handle_sync_event/4, handle_info/3, terminate/3, code_change/4.

It is now time to export our states – barking/2, wagging_tail/2 and sitting/2. These functions will handle their state events that are called asynchronously. It means that I will be using gen_fsm:send_event/2 to trigger those events.

NOTE: Had I planned to trigger some events synchronously, I would have also exported barking/3, wagging_tail/3 and sitting/3. And then, instead of using gen_fsm:send_event/2, I would opt for gen_fsm:sync_send_event/[2,3]. We won’t be doing any synchronous stuff in this module though, so I have skipped them.

Let’s move on and define our API functions now.

start_link/0 calls gen_fsm:start_link/[3,4] which will perform a synchronous call to init/1 (below).

squirrel/0 and pet/0 triggers an event asynchronously (since it calls gen_fsm:send_event/2), and this event is processed based on the current state of the FSM process. For example, if the current dog state is “barking” and I call pet/0, then the event ‘pet’ will be sent to barking/2. Likewise, if the dog state was “sitting” when I called pet/0, then the event ‘pet’ would have been handled via sitting/2.

stop/0 sends the event ‘stop’ asynchronously using gen_fsm:send_all_state_event/2. We handle this event using handle_event/3 later.

Our init/1 function returns a tuple with the following values to gen_fsm:start_link/[2,3]:

  1. ok – Notifies that the process initialization is complete
  2. barking – The initial state of the FSM process. This means that our dog switches to ‘barking’ state as soon as the process starts, and hence any further events will be handled by barking/2.
  3. [] – State variable. We do not need one in our case. Normally there would be a record defined (like: -record(state, {name, age}). ) that will be initialized here as State variable.
  4. 0 – Timeout, optional. We set Timeout to 0 deliberately so that a timeout event is passed to barking/2 as soon as the process switches to the ‘barking’ state. Inside barking/2, we handle the timeout event by printing the dog bark text in a loop (see below).

We know that the process switched to ‘barking’ state during init/1 itself. So all further events will be handled by barking/2 as long as the state remains ‘barking’.

As evident from above code snippet, ‘timeout’ event will call bark/0 (which will print the text “Dog says: BARK! BARK!”) and then switch to ‘barking’ state (no major change there since the state was already ‘barking’), but also sets timeout value to 2 seconds. This means that ‘timeout’ event will trigger again within 2 seconds, which in turn will print the text on screen, and this goes on and on until you send in another event (‘pet’, maybe?).

When someone trigger the ‘pet’ event, wag/0 is called which prints a message to screen and then switches the process state to ‘wagging_tail’ for the next 30 seconds. Notice that the process state changes from ‘barking’ to ‘wagging_tail’ instantly, and the ‘timeout’ event in ‘wagging_tail’ state will be triggered after 30 seconds.

After 30 seconds, the ‘timeout’ event in ‘wagging_tail’ state gets triggered, which in turn switches the process back to ‘barking’ state.

Now let’s say that we triggered the ‘pet’ event in ‘wagging_tail’ state before it timed out (i.e., call pet/0 within 30 seconds). As you can see from above snippet, sit/0 is then called which just prints out another line of text on screen and then switches the state to ‘sitting’. Notice that there is no timeout value mentioned here – which means that the process will wait endlessly for the next event to come its way, and will handle it in sitting/2.

When our FSM process enters ‘sitting’ state, the only valid event it will entertain is ‘squirrel’. This event can be triggered via squirrel/0 after which the process state switches to ‘barking’.

With this we conclude the functions that handle events which are defined up front. Now let us look at some other interesting callbacks.

There are cases when you will want to trigger an event irrespective of the state that our FSM process is currently in. Stoping an FSM process is one classic example. You should be able to stop the process by triggering ‘stop’ event without caring whether its current state is ‘barking’ or ‘wagging tail’ or ‘sitting’.

Worry not, because gen_fsm behaviour provides handle_event/3 callback to handle such events in an asynchronous way. Events that we handle via handle_event/3 are triggered via gen_fsm:send_all_state_event/2. If you scroll up and check our stop/0 function, you will notice that we used the same gen_fsm:send_all_state_event/2 to send ‘stop’ event.

FYI, there is another gen_fsm callback named handle_sync_event/4 which can handle events sent to a FSM process synchronously. The events are sent via gen_fsm:sync_send_all_state_event/[2,3] but we will stay away from synchronous calls for our dog FSM.

However, it is necessary to write code for handle_sync_event/4 callback so we do it anyway. All other messages received by the FSM process can be handled using handle_info/3 but we ignore that here. Usual terminate/3 and code_change/4 completes our gen_fsm callback code.

There are a few internal functions which simply prints out the dog action. Internal functions are not exported to the outside world.

Phew! That completes our example. Let’s compile it and take it for a spin. In between we will use sys:get_state/1 to check the current state of our FSM process.

Further reading:

How to extend LVM on qcow2 images

Categories Virtualization

Preparing to add some more space for VM’s swap and /. Commands listed below [note: host = node, VM = virtual machine]

Make sure that the VM is stopped before trying this out.

Current VM space layout (compare with final):

 

* Extend image to required size

* Boot up the VM,  then log into it, delete and recreate LVM partition

* Reboot then VM, and login back to resize physical volume

* I have to extend swap as well, so better unmount swap which is on LVM before extending it

* Now extend swap LVM

* Make Swap and mount it back again

New Swap = 2GB [success!!]

* Now extend root LVM so that it has ~10GB+ space

* Online resize (WARNING: for ext4 only)

Done!

So, final VM layout is as follows:

 

Where the hell is “HYPERVISOR_MEM” in OpenNebula 4.6?

Categories Virtualization

This was the question I asked myself when I saw no mention of HYPERVISOR_MEM in file /etc/one/sched.conf. It is the percentage of memory reserved for Hypervisor use.

# HYPERVISOR_MEM: Fraction of total MEMORY reserved for the hypervisor.
# E.g. 0.1 means that only 90% of the total MEMORY will be used

HYPERVISOR_MEM=0.4

.. will reserve 40% of the total memory for OpenNebula (only 60% RAM will be available for guest VMs). You can choose to over-commit resources by simply specifying a negative value:

HYPERVISOR_MEM=-0.4

This  is  was the preferred method to over-commit resources in OpenNebula.

Well, now things have got even more easier since OpenNebula v4.6. HYPERVISOR_MEM is deprecated now – instead set RESERVED_MEM and RESERVED_CPU attributes to the host via Sunstone -> Infrastructure -> Hosts -> [host]

  • RESERVED_CPU in percentage. It will be subtracted from the TOTAL CPU
  • RESERVED_MEM in KB. It will be subtracted from the TOTAL MEM

These values can be negative, in that case you’ll be actually increasing the overall capacity so over-commiting host capacity

 

Restrict VM network Speed in OpenNebula

Categories Virtualization

Environment:

OpenNebula + KVM

Requirement:

Automatically set a cap on VM network speed to 1MB/s for all VMs when it start.

.. and then:

# chmod 755 /var/lib/one/remotes/hooks/set_vm_network_speed.sh

Resize KVM images in OpenNebula

Categories Virtualization

Although I am doing this for OpenNebula, this can be followed for any KVM setup.

I am using a CentOS 6.2 image downloaded from OpenNebula Marketplace. This is of size 10GB, and my requirement is to extend root partition (/dev/sda1) to 20GB.

All commands on KVM host:

Make a backup of the existing disk image before proceeding..

I wish to add 10 more GB to the current partition (so, total size = 20GB). So, create a new raw disk image of size 20GB first, and then clone the contents from original image file to new image file.

The filesystem should be ready by now, but just in case you wish to run “resize2fs” manually:

Now include the new image in OpenNebula, use it in the VM template and start a new instance again.

NOTE: There are many more ways to resize, this is just one of those methods (like http://docs.openstack.org/image-guide/content/ch_modifying_images.html , or even live-resize using “qemu-img resize”).

For example: to extend the entire disk in order to create additional partitions (/dev/vda2, /dev/vda3 etc), “qemu-img resize” command should be enough.

 

How to allow irsync to use different config files

Categories Backup

Irsync ( https://www.rfxn.com/projects/irsync-incremental-rsync/ ) is a great backup application for Linux admins. It uses rsync to take backups and can be configured to maintain backups over a period of time.

However there are cases when you will want to run multiple irsync cron jobs for taking backups of different directory paths. In such cases, having an option to set different config file would be great, which can be achieved quite easily.

This is for irsync version 1.1:

Proposed patch:

Proceed with patching:

… and now, you can set different config file for each cron, like:

Git url: https://github.com/vimal7370/irsync

Git download: # git clone git@github.com:vimal7370/irsync.git

Haraka – Superfast mail server based on node.js

Categories Mail Servers

From Haraka’s official website:

Haraka is an open source SMTP server written in Node.js which provides extremely high performance coupled with a flexible plugin system allowing Javascript programmers full access to change the behaviour of the server.

Admit it – Javascript is everywhere. Node.js brings the power of event-driven asynchronous programming to server-side, allowing programmers to use Javascript to write daemons, servers, and what not. Stuffs which were meant to be done with C/C++/Perl is now being completed in Javascript.

Haraka is a SMTP server written in node.js. It is suuuuuuuper-fast, just ask the guys at Craigslist.

In Web hosting industry dominated by Cpanel, Haraka can co-exist with Exim and can be used as an alternative mail server for clients with large mailing lists. Haraka consumes negligible memory while easing the job off Exim which can cater to normal clients hosted on a Cpanel server.

This guide describes steps to configure Haraka as outgoing mail server only. If you wish to accept emails and deliver it locally you will need to refer the official docs.

1. Install Node.js binary (http://nodejs.org/download/)

2. Install git and Haraka

3. Create symlinks once the installation is complete

4. Also copy over the start-up script

5. Initialize Haraka config directory

6. make changes to config files

/etc/haraka/config/smtp.ini:

/etc/haraka/config/plugins:

/etc/haraka/config/auth_flat_file.ini:

 /etc/sysconfig/haraka:

7. Fix permissions:

8. Configure chkconfig:

9. And finally.., start Haraka

In case haraka failed to start for any reason, check its log file at /var/log/haraka.log.

If you encounter the following error while starting Haraka:

.. then edit /usr/local/haraka/bin/haraka, and modify its first line to:

.. and Haraka will start fine:

Now that Haraka is up and running, use the following settings to send email:

On a Cpanel server, run Haraka on another port and use it as outgoing email server for bulk mailing/newsletter tasks and spare Exim from the pain!

Further reading: