Alarm Caching

There are many cases in which a component can send an alarm repeatedly, many times in a short period of time. If the AlarmSender actually sent an alarm every time a component asked to send the same alarm message, the channels could quickly overwhelm the recipients. Having your inbox overflowing with hundreds of emails containing the same alarm message is not a very effective notification.

To avoid this, the AlarmSender has an internal cache, which keeps count of the last time every alarm message is sent. When a component wants to send an alarm, the AlarmSender uses the cache together with each channel's minimum resend interval, to determine if the alarm should be resent over each channel or not. So depending on the minimum resend interval of each channel, the same message will be sent more or less frequently.

Configuration

jAlarms includes three caching mechanisms. The default one is used transparently if you don't configure one explicitly and it only consists of a Map which stores the last time each message is sent. This is very simple and doesn't use a lot of memory; if your application sends 100 different alarm messages, then the cache will have at most 100 keys in memory, each one storing a Long.

The other two mechanisms use (or at least can use) external storage which can be shared among several applications. One is memcached, the other one is ehcache.

memcached

Sometimes you may have two or more applications working independently but sharing some components and maybe they send the same alarm messages if one of the shared components or resources fail (for example a connection to the internet if both apps are on the same server, or the database if both use the same one). In those cases, if the AlarmSender is configured identically in each app, then multiple identical alarms will be sent, one from every application that detects the failure of the shared component or resource.

This can be avoided if you use memcached, which is an external memory-only cache that can be used by several applications concurrently. You only need to setup the memcached alarm cache to connect to the available memcached instances and it will be used to store short keys representing the alarm messages; this way, if two or more apps want to send the same alarm message over and over again, only one of them will do it because the rest of the apps will detect the alarm key in memcached.

Setting up the memcached alarm cache

First of all, you need access to at least one instance of memcached. Right now only unprotected memcached access is supported. You need to create an instance of AlarmMemcachedClient and set the servers property, which is a list of strings, each one specifying a memcached server. You can specify just the IP or hostname if memcached is listening on the default port 11211, otherwise you need to specify the port, separating it from the IP by colons (e.g. 192.168.0.1:12345).

Then you need to set this cache in the AlarmSender through the alarmCache property and that's it, the AlarmSender will use memcached to determine if a message must be resent through a given channel.

But how does it work?

The memcached client creates a key for each message that is passed to it in the store() or shouldResend methods. The format of the key is jalarms:channelID:source:message_hash, where

  • jalarms is a fixed prefix for all keys (this is helpful when you use stats tools for memcached).
  • channelID is the channel's ID because each channel has a different resend interval. This value is generated by using the channel's hashCode().
  • source is the alarm source. If no alarm source is specified then an empty string is used here.
  • message_hash is a hex-encoded MD5 hash of the alarm message. This is to avoid using the whole message as part of a key, since there's a limit to key lengths in memcached.

ehcache

There is an AlarmEhcacheClient component which will use ehcache to store the alarms. The way it works is similar to the memcached client, except ehcache is used instead.

To use ehcache, the AlarmEhcacheClient needs to be set up first. It has a configPath property which must be set to the path to an echache XML configuration file located in the classpath. The default value is "jalarms_ehcache.xml". It also needs the cache name, since the config file can define several caches. The default is "jalarms". This is an example ehcache configuration file for jAlarms, which will only use memory storage:

<?xml version="1.0" encoding="UTF-8"?>

<ehcache xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:noNamespaceSchemaLocation="ehcache.xsd" name="testCaches"
    updateCheck="false" monitoring="off"
    dynamicConfig="true">

  <cache name="jalarms_test" maxElementsInMemory="1024" maxElementsOnDisk="0"
    eternal="false" timeToIdleSeconds="120" timeToLiveSeconds="180"
    overflowToDisk="false" memoryStoreEvictionPolicy="LRU"
    transactionalMode="off" statistics="false" />

</ehcache>

The ehcache client is really more useful if you have access to a Terracotta ehcache server, in which case it can be used similar to the memcached client, which can be shared by several applications.