Kraken Log API

Kraken Log API provides centralized log aggregation and normalization platform. You can manage various loggers and configurations, and attach log pipe to loggers. You can load or unload logger factory and logger dynamically.

Author

• xeraph ( xeraph@nchovy.com)

Command

• logapi.loggerFactories
  • List all logger factories.

    run result from windows sentry agent
    
    kraken> logapi.loggerFactories
    Logger Factories
    ---------------------
    fullname=local\disk-usage, type=Disk Usage Logger, description=Check Disk Usage
    fullname=local\cpu-usage, type=CPU Usage Logger, description=Check CPU Usage
    fullname=local\windows-event-logger, type=Windows EventLog, description=Windows EventLog
    fullname=local\memory-usage, type=Memory Usage Logger, description=Check Memory Usage
    fullname=local\network-usage, type=Network Usage Logger, description=Check Network Usage
    fullname=local\process-check, type=Process Check, description=Check process running status periodically
    

• logapi.loggers
  • List all logggers. You can check logger running status.

    run result from server
    
    kraken> logapi.loggers
    Loggers
    ----------------------
    name=8a262b97-05fc-4db0-8108-1a8cac7ae1ba\syslog-16, factory=local\netscreen-isg, log count=0, last log=null, last run=2010-12-26 17:04:03
    name=8d809039-8cad-4f47-b55e-593ae18ee850\network-usage, factory=local\network-usage, log count=4482140, last log=2010-12-26 17:03:56, last run=2010-12-26 17:03:56
    

• logapi.startLogger [logger name] [interval]
  • Start the logger
• logapi.stopLogger [logger name] [max wait time]
  • Stop the logger
• logapi.createLogger [logger factory name] [logger namespace] [logger name] [description]
  • Create a logger using specified logger factory.
• logapi.removeLogger [logger fullname]
  • Remove the logger
• logapi.parsers
  • List all log parsers

    kraken> logapi.parsers
    Log Parsers
    ---------------------
    fortigate
    netscreen-isg
    

• logapi.normalizers
  • List all log normalizers

    kraken> logapi.normalizers
    Log Normalizers
    ---------------------
    fortigate
    netscreen-isg
    

• logapi.parse
  • Parse log for test
• logapi.normalize
  • Normalize log for test

Interfaces

Logger

Logger has identifier, running state, and statistics. LoggerFactory creates Logger with configuration. All loggers are uniquely identified by fullname. Logger fullname is "namespace\name". In general, local loggers use namespace local, and remote loggers use host identifiers. For example, kraken sentry (SIEM agent) notifies local loggers to kraken base, and kraken base registers proxy loggers to logger registry. You can receive remote logs as if it exists in local system.

public interface Logger {
	// identifier and related logger factory
	String getFullName();
	String getNamespace();
	String getName();
	String getFactoryFullName();
	String getFactoryName();
	String getFactoryNamespace();
	String getDescription();

	// get received log count, last run date, and last log date,
	Date getLastRunDate();
	Date getLastLogDate();
	long getLogCount();

	// get running state and running interval
	boolean isRunning();
	int getInterval();

	// start/stop logger
	void start(int interval);
	void stop();
	void stop(int maxWaitTime);

	// attach/detach log pipe and receive logs
	void addLogPipe(LogPipe pipe);
	void removeLogPipe(LogPipe pipe);

	// attach/detach event listener and receive logger event callbacks
	void addEventListener(LoggerEventListener callback);
	void removeEventListener(LoggerEventListener callback);
}

If you want to receive logs from specific logger, you should search logger registry and attach log pipe to logger. Because OSGi environment has dynamicity, your target logger may not exists at that time. In this case, you should add logger registry event listener to logger registry and monitor registration event.

LogPipe is look like this:

public interface LogPipe {
	void onLog(Logger logger, Log log);
}

When you create new logger class, you should extends AbstractLogger class. It implements all required members, and, of course, it dispatches log to connected log pipes when you call AbstractLogger.write(log) method. Typically, logger is implemented like this:

public class CpuUsageLogger extends AbstractLogger {
	public CpuUsageLogger(String namespace, String name, String description, LoggerFactory loggerFactory) {
		super(namespace, name, description, loggerFactory);
	}

	@Override
	protected void runOnce() {
		try {
			SystemTime s = new SystemTime();

			Map<String, Object> m = new HashMap<String, Object>();
			m.put("idle", s.getIdlePercent());
			m.put("kernel", s.getKernelPercent());
			m.put("user", s.getUserPercent());

			String msg = String.format("cpu usage: %d%%", s.getUsage());
			Log log = new SimpleLog(new Date(), getFullName(), "system", msg, m);
			write(log);
		} catch (InterruptedException e) {
		}
	}
}

runOnce() is periodically called with configured interval milliseconds. You should not throw any exception out of runOnce(). When it throws exception except InterruptedException, logger will be stopped.

You can monitor logger running status changes using LoggerEventListener callback:

public interface LoggerEventListener {
	void onStart(Logger logger);

	void onStop(Logger logger);
}

Because callback method contains Logger argument, you can monitor many loggers with just one event tracker.

LoggerFactory

Logger factory models specific logger type, and it creates and destroyes logger instance. When logger factory creates new logger or destroyes existing logger, it register new logger to logger registry, or unregister logger from logger registry.

Logger factory is also identified by fullname, but you probably need also user friendly name and description for UI control. You can display localized name and description using LoggerFactory.getDisplayName(locale) and LoggerFactory.getDescription(locale). LoggerConfigOption also has localization support.

public interface LoggerFactory {
	// identifier
	String getFullName();
	String getNamespace();
	String getName();

	// display names
	Collection<Locale> getDisplayNameLocales();
	String getDisplayName(Locale locale);

	// descriptions
	Collection<Locale> getDescriptionLocales();
	String getDescription(Locale locale);

	// required or optional config options for logger
	Collection<LoggerConfigOption> getConfigOptions();

	// creates logger (namespace is 'local' by default)
	Logger newLogger(String name, String description, Properties config);
	Logger newLogger(String namespace, String name, String description, Properties config);
	
	// destroyes logger (namespace is 'local' by default)
	void deleteLogger(String name);
	void deleteLogger(String namespace, String name);

	// callback control
	void addListener(LoggerFactoryEventListener callback);
	void removeListener(LoggerFactoryEventListener callback);
}

LoggerConfigOption has own type, name, and required/optional properties. You should implement parse() and validate() because all logger configurations are saved in string format:

public interface LoggerConfigOption {
	String getType();
	String getName();
	boolean isRequired();

	Collection<Locale> getDisplayNameLocales();
	String getDisplayName(Locale locale);

	Collection<Locale> getDescriptionLocales();
	String getDescription(Locale locale);

	Object parse(String value);
	void validate(Object value);
}

If you don't want to support localization, just return Arrays.asList(Locale.ENGLISH) for getDisplayNameLocales() or getDescriptionLocales(), and return only english name or description for getDisplayName(locale) or getDescription(locale).

When you create new logger factory class, you should extends AbstractLoggerFactory class. Typically, logger factory is implemented like this:

@Component(name = "cpu-usage-logger-factory")
@Provides
public class CpuUsageLoggerFactory extends AbstractLoggerFactory {
	@Override
	public String getName() {
		return "cpu-usage";
	}

	@Override
	public Logger createLogger(String namespace, String name, String description, Properties config) {
		return new CpuUsageLogger(namespace, name, description, this);
	}

	@Override
	public String getDisplayName(Locale locale) {
		return "CPU Usage Logger";
	}

	@Override
	public String getDescription(Locale locale) {
		return "Check CPU Usage";
	}
}

In above example, logger factory's fullname is local\cpu-usage. It doesn't support localization, and returns only english text. It creates cpu usage logger with no configuration. This logger factory is an iPOJO component named cpu-usage-logger-factory. Logger factory registry component tracks LoggerFactory services, therefore it will be registered to logger factory automatically. You can see registered logger factories using logapi.loggerFactories command.

LoggerRegistry

Logger registry manages all registered loggers and dispatches logger registration events. See LoggerRegistry interface:

public interface LoggerRegistry {
	boolean isOpen();

	// fetch all loggers
	Collection<Logger> getLoggers();

	// search logger
	Logger getLogger(String fullName);
	Logger getLogger(String namespace, String name);

	// logger registration
	void addLogger(Logger logger);
	void removeLogger(Logger logger);

	// log pipe control
	void addLogPipe(String loggerFactory, LogPipe pipe);
	void removeLogPipe(String loggerFactory, LogPipe pipe);

	// event callback control
	void addListener(LoggerRegistryEventListener callback);
	void removeListener(LoggerRegistryEventListener callback);
}

LoggerRegistryImpl component class implements LoggerRegistry interface. You can see logger-registry component instance using ipojo.instances command. This component connect log pipe to all registered loggers, receives onLog() callback, and dispatches log to connected log pipes. In other words, you can receive all logs which have common logger factory fullname.

For example, if you want to receive all cpu-usage logs:

// assume that 'this' class implements LogPipe
loggerRegistry.addLogPipe("cpu-usage", this);

When a logger factory is unregistered, all related loggers are also removed.

LoggerFactoryRegistry

Logger factory registry manages all logger factories:

public interface LoggerFactoryRegistry {
	// fetch all factories
	Collection<LoggerFactory> getLoggerFactories();

	// search logger factory
	LoggerFactory getLoggerFactory(String name);
	LoggerFactory getLoggerFactory(String namespace, String name);

	// create new logger
	Logger newLogger(String factoryName, String loggerNamespace, String loggerName, String description, Properties config);

	// event callback control
	void addListener(LoggerFactoryRegistryEventListener callback);
	void removeListener(LoggerFactoryRegistryEventListener callback);
}

LoggerFactoryRegistryImpl component class implements LoggerFactoryRegistry interface. This component tracks all logger factories using  OSGi service tracker. It dispatches logger factory registration event to callbacks:

public interface LoggerFactoryRegistryEventListener {
	void factoryAdded(LoggerFactory loggerFactory);

	void factoryRemoved(LoggerFactory loggerFactory);
}

You can monitor all logger factory registration events using one monitor instance.

LogParserRegistry

public interface LogParserRegistry {
	void register(LogParser parser);
	void unregister(LogParser parser);
	
	Collection<String> getNames();
	LogParser get(String loggerFactoryName);

	void addEventListener(LogParserRegistryEventListener callback);
	void removeEventListener(LogParserRegistryEventListener callback);
}

LogNormalizerRegistry

public interface LogNormalizerRegistry {
	void register(LogNormalizer normalizer);
	void unregister(LogNormalizer normalizer);

	Collection<String> getNames();
	LogNormalizer get(String loggerFactoryName);

	void addEventListener(LogNormalizerRegistryEventListener callback);
	void removeEventListener(LogNormalizerRegistryEventListener callback);
}

Msgbus Calls

Query LogParserFactory list

• Request

  org.krakenapps.log.api.msgbus.LoggerPlugin.getParserFactories
  

• Response

  {
      "factories": [
          {
              "display_name": "Apache Web Log Parser Factory",
              "description": "Create apache httpd log parser with log format option",
              "name": "httpd",
              "options": [
                  {
                      "display_name": "Log Format",
                      "default_value": "%h %l %u %t "%r" %>s %O "%{Referer}i" "%{User-Agent}i"",
                      "description": "Apache Log Format",
                      "name": "log_format",
                      "required": true,
                      "type": "string"
                  }
              ]
          },
          {
              "display_name": "OpenSSH Log Parser Factory",
              "description": "Create redhat openssh log parser",
              "name": "openssh",
              "options": []
          }
      ]
  }
  

See Also

• Kraken Sentry

History

• 1.4.0 (2011-12-10)
  • added log normalizer factory instead of normalizer component
• 1.0.0 release (2010-12-15)