FIX Analyser 2 Netprobe User Guide

Intended Audience

This document is intended for users who wish to install and configure the FIX Analyser 2 plug-in to implement FIX message monitoring as part of their existing Geneos installation.

The purpose of this document is to describe how to install, use and configure the FIX Analyser 2 plug-in. The latter portion of the document also contains a technical reference for more experienced users.

The document assumes that users have some understanding of the FIX Protocol (see http://www.fixprotocol.org), as well as a basic knowledge of how to configure Geneos components such as adding a new Netprobe or sampler to a Gateway configuration.

Architecture

image0

This diagram shows the recommended deployment of Fix Analyser 2.

FIX logs produced by application processes (such as a FIX engine) are hosted on an Application server. These log files are read by a lightweight Geneos File Agent process and forwarded to FIX Analyser 2 which is hosted on a Monitoring server. This deployment ensures that parsing and analysis of the FIX messages do not impact on the performance of any applications on the Application server.

FIX Analyser 2 uses a SQLite in-process database to store order information for the FIX messages it is processing. The data contained in this database is stored as a file on the local disk and is used for tracking order states and latency data.

Message data can optionally be stored by FIX Analyser 2 to a MySQL database. This data is used to augment HTML-based order reports to display drill-down message-level data for each order, and enable the message querying feature. Where possible we recommend installing this database for enhanced plug-in functionality.

As with previous versions of FIX Analyser, files available locally to the FIX Analyser 2 process can be read directly, without the use of a Geneos File Agent. However, since this involves deploying the analyzer process on an application server, we recommend this configuration only for lower volume message rates (depending upon the host hardware capabilities).

Installation

Component Prerequisites

The components of the FIX Analyser 2 solution are FIX Analyser 2 Netprobe, File Agent, and MySQL. Please refer to the Geneos Compatibility Matrix for the list of supported platforms.

Note:

As FIX Analyser 2 Netprobe is only supported on Linux 64-bit OS versions, a File Agent process is required to monitor files hosted on other platforms.

With a configuration consisting of 5 dataviews, the FIX Analyser 2 Netprobe requires on average 500MB of disk space for the SQLite database file, when processing 1,000,000 orders per day.

The FIX Analyser 2 Netprobe host must have the MySQL 64-bit client drivers installed if using the optional message database.

MySQL message database content will require on average 10GB of disk space, for 3 days of persisted data containing ~9 million messages (approximately 1 million orders assuming an average of 8 acknowledgement messages per order).

Netprobe, File Agent and MySQL processes all require a TCP/IP listen port for communication. If any of these processes are behind a firewall, a rule will need to be added to allow access to these ports.

FIX Analyser 2 Netprobe binary

The FIX Analyser 2 plug-in is hosted within a specialised Netprobe binary. Unlike a standard Netprobe this binary contains only the FIX Analyser 2 plug-in. CPU and memory usage of this probe is expected to be higher than for normal monitoring tasks due to the analytics being performed (dependent upon the message rate being processed). This probe allows plug-ins to run with a higher maximum memory threshold to cater for this.

Because of this change, other plug-ins such as CPU or FKM (for example) are not available in the FIX Analyser 2 binary, therefore other monitoring tasks will still require a standard Netprobe. If you are using the suggested architecture (see diagram above) you should consider running a standard Netprobe on the monitoring server for Geneos self-monitoring.

The default listen port for a FIX Analyser 2 Netprobe is now 7030. This can be changed as normal via the -port command-line argument or the NET_PORT environment variable.

Other Netprobe features such as executing remote commands, running action scripts, or floating / self-announcing run modes are unchanged. Command-line configuration options not relating to plug-ins are also unchanged. Please refer to the Netprobe User Guide for further details.

There are cases when the Fix Analyser machine and the File Agents or FIX logs may be on different time zones. Therefore, the time zone of FIX Analyser Netprobe is adjusted with the TZ environment variable in the start-up script.

Warning: Time zone differences between the FIX log timestamps and FIX Analyser machine can affect the displayed value of secondsSinceLastActivity and the actual calculation of time-out for the SessionStatus view.

If the FIX logs are in UTC and the FIX Analyser Netprobe is on a different time zone:

export TZ=UTC

Packaging

The FIX Analyser 2 Netprobe comes packaged as a tarball (a GZip'd TAR file). This package contains the following files:

File Description
fix-analyzer2-netprobe.linux_64 The main FIX Analyser 2 Netprobe binary.
libsqlite3.so SQLite 3 dynamic library.
libcrypto.so* OpenSSL library, required by libcurl.so to provide cryptographic functions for authentication.
libssl.so* OpenSSL library, required by libcurl.so to provide support for HTTPS URLs.
LICENCE Licence declarations for 3rd party software libraries used by FIX Analyser 2 Netprobe.
NOTICES Notices of 3rd party library licence declarations.
compat/libstdc++.so.6 C++ runtime dynamic library for users running on RHEL 5 who cannot install the appropriate compatibility RPM package.
resources/db-schema/fix_analyzer_schema_install.sh Database schema install script.
resources/db-schema/mysql-fix-analyzer-v1.0-v1.1-schema-upgrade.sql SQL statements to upgrade a v1.0 schema to a v1.2 schema.
resources/db-schema/mysql-fix-analyzer-v1.1-v1.2-schema-upgrade.sql SQL statements to upgrade a v1.1 schema to a v1.2 schema.
resources/db-schema/mysql-fix-analyzer-v1.2-db-schema.sql SQL statements to create a v1.2 schema message database in MySQL.
templates/geneos-fix-analyzer2.xml Template setup file.

The minimum installation of the FIX Analyser 2 Netprobe binary consists of the Netprobe executable and the libsqlite3.so dynamic library.

Users must use the provided libsqlite3.so dynamic library (by setting the LD_LIBRARY_PATH environment variable if the file is not located in the Netprobe working directory) as this version (3.7.11) is typically more up-to-date than versions installed at system level. The database file may become corrupted if Netprobe is run against the same database file using different SQLite versions.

The scripts in resources/db-schema are related to installation of the MySQL message database. Once installation has been completed (or if you do not intend to use this feature) the files can be removed.

File Agent binary

To monitor FIX log files hosted on supported platforms other than Linux 64-bit RHEL 5+, a File Agent must be installed on the machine.

For installation details please see the Geneos File Agent User Guide documentation.

MySQL message database

FIX Analyser 2 can optionally log FIX message data to a MySQL database. This data is used to augment HTML-based order reports to display drill-down message-level data for each order, and allow parameterised message queries. To install a MySQL database for use by FIX Analyser 2 please follow the steps below.

Server installation

The free MySQL database can be installed using the package manager for your Linux distribution or downloaded from the MySQL website. For licensing reasons, this software cannot be distributed by ITRS as part of any Geneos product.

Installing the database

Please refer to the MySQL installation instructions for your platform available using the link below: http://dev.mysql.com/doc/refman/5.1/en/installing.html

FIX Analyser 2 requires a minimum of MySQL 5.1.6 for the MySQL event scheduler feature. This scheduler is used to automate data housekeeping tasks allowing for low-maintenance operation of the message database.

Securing the database

The newly installed database must be secured. This is achieved using the mysql script:

/usr/bin/mysql_secure_installation

This script will give you the following options with which we provide recommendations:

Option Recommendation
Set root password Strongly recommended
Remove anonymous user Required for correct permission settings
Disallow root login remotely Recommended
Removed test database and access to it Recommended
Reload privilege table Required for correct permission settings

Persisting the event scheduler

The FIX Analyser 2 schema install script will enable the event scheduler; however, to ensure the setting persists across restarts of the MySQL server process, it must be set to ON via a command-line option or in the options file. For example, on a Linux system containing a standard MySQL package installation add the line event-scheduler=ON to the /etc/my.cnf configuration file.

Please see the following link for full details: http://dev.mysql.com/doc/refman/5.1/en/server-system-variables.html#sysvar_event_scheduler

Database schema installation

FIX Analyser 2 comes packaged with a database schema file which can be used to create a database instance on a MySQL database server.

The current schema version is 1.2, which adds an automatic data housekeeping function to the previous 1.1 version. This function utilises the MySQL event scheduler feature, which is present in MySQL versions 5.1.6 and onwards.

There are several methods for installing this schema as described below.

Attended scripted installation

A schema installation / upgrade script is provided in the resources/db-schema folder of the FIX Analyser 2 package. The script should be run from within this folder and will require database administrator (DBA) access to the database. If remote DBA access has been disabled, then the script will need to be executed on the database server host itself (or remote access granted).

To execute the script and create a new FIX Analyser 2 installation, run the following command:

cd resources/db-schema
./fix_analyzer_schema_install.sh -n

The script will prompt for the following information during the installation process:

  • Database administrator (DBA) username
    • This account will perform administrative functions on the database
  • DBA password
  • Hostname for the MySQL database server
  • Port for the MySQL database server
  • Name of the FIX Analyser 2 database to create
    • This database should not already exist on the server
  • FIX Analyser user account name to create
    • This account will be given permissions to write to the new database
  • FIX Analyser user account password
  • Time to run the daily data housekeeping
    • Default time is 00:15
  • Number of days to retain message data
    • Default is 3 days of data

Unattended scripted installation

The installation script described can also be run in unattended (non-interactive) mode.

To do this, the requisite information the script requires must be provided as environment variables. The table below describes the environment variables the script expects when installing a new database.

Environment Variable Description
ROOT_USER MySQL database administrator (DBA) account username
ROOT_PASS MySQL DBA account password
MYSQL_HOST MySQL server hostname or IP address
MYSQL_PORT MySQL server listen port
DB_NAME Name of the database to create
FIX_USER FIX Analyser 2 database account username to create and permission
FIX_PASS FIX Analyser 2 database account password
USE_EXISTING_ACCOUNT If the FIX Analyser 2 database account already exists, and should be given permissions on the new database, specify the value y for this variable.
HK_TIME The time of day to execute the daily housekeeping script, in the format HH:MM. This should be a time of day where (ideally) no FIX sessions are operating or when no messages are sent. The default time when running an attended installation is 00:15.
HK_DAYS Number of days of message data to retain in the database. Expired data will be cleared by the housekeeping function. The recommended number is 3.

Manual installation

If you are unable to or do not wish to install the database schema using the installation script, you can install the schema manually using the following steps. The SQL statements below should be executed as a database administrator (DBA) account on the MySQL database server.

  1. Connect to the MySQL database using the MySQL command-line client. See the reference (http://dev.mysql.com/doc/refman/5.1/en/mysql.html) for full details.
mysql -u dba_username -p dba_password [ -h db_host -P db_port ]
  1. Enable the MySQL event scheduler.

Note: You should also enable the scheduler permanently as detailed in the server installation section of this document.

SET GLOBAL event_scheduler = ON;
SHOW PROCESSLIST;

The output of the SHOW PROCESSLIST command should return a row with the text event_scheduler if it was successful.

  1. Create a new database for use by the FIX Analyser 2 plug-in (e.g. FIX_DB).
CREATE DATABASE FIX_DB;
  1. Create a user for FIX Analyser 2 to connect to the MySQL server with. We allow this fix_db user to connect from any hostname with the password fix_pw.
CREATE USER fix_db@'%' IDENTIFIED BY 'fix_pw';
  1. Grant the user appropriate permissions to access the new database.
GRANT event,execute,select,insert,update,delete on FIX_DB.* TO fix_db@'%';
  1. Load the schema into the new database created earlier. From within mysql run:
source mysql-fix-analyzer-v1.2-db-schema.sql

Or from the shell pipe the schema into the mysql command-line client as follows:

mysql connection_parameters FIX_DB < mysql-fix-analyzer-v1.2-db-schema.sql
  1. Schedule the housekeeping event. In the following SQL, HH:MM is the daily execution time of the event and DAYS the number of days of message data to retain.

It is recommended that the housekeeping be run at a time of day where (ideally) no FIX sessions are operating or when no messages are sent. The installation script uses default values of 00:15 and 3 respectively.

CREATE EVENT FIX_DB_housekeeping
ON SCHEDULE
EVERY 1 DAY
STARTS DATE_ADD(CURRENT_DATE(), INTERVAL 'HH:MM' HOUR_MINUTE)
DO
CALL FIX_DB.removeExpiredFiles(DAYS);

Message Queue Limits

Fix Analyser 2 maintains an internal queue of message to be logged to the MySQL database. Should the database server connection become slow or unavailable during operation, the program will cache up to 1.5 million inserts before beginning to discard new data.

Message Processing

This chapter discusses how FIX Analyser 2 processes incoming FIX messages and generates data views based on the user configuration. This information is useful background knowledge when discussing column configurations and filtering in the plug-in setup.

Data flow

image11

This diagram shows the internal data flow of the FIX Analyser 2 plug-in.

The main stages in this data flow are as follows:

  1. FIX log files are read as raw textual data.
  2. Input text is optionally filtered before being further processed.
  3. The data is parsed as a FIX message allowing access to FIX field values.
  4. Order tracking creates a new order record for the message (if required) or updates the state of an existing order record. The FIX message is augmented with additional order state information before further processing. See section Message Processing.
  5. Messages seen in sequence (see the out-of-order messages section below) are processed immediately. For each configured view in the FIX Analyser 2 setup, a table of analytics data is maintained. All tables are updated as appropriate by the message.
  6. When the plug-in samples, the latest data from the analytics tables is published as dataviews, which are then processed by Gateway and displayed in Active Console 2.
  7. Unsequenced messages (see the out-of-order messages section below) are processed by the out-of-order message table, in the same way that analytics tables are normally updated. When the "missing" message is seen (or a timeout elapses) this data is re-integrated back into the analytics tables making it available for sampling.
  8. FIX messages are optionally logged to the MySQL message database.

Order tracking

The FIX Analyser 2 plug-in tracks the state of FIX (protocol version 4.2) orders, for use by configured order columns in a user view definition. By tracking orders information crossing multiple FIX messages can be captured, including latency information (the time for acknowledgement and fills) and the latest value of FIX fields referenced by order columns, for example the LeavesQty (FIX field 151).

Orders are inferred by examining all messages which are processed by the plug-in, using the following logic.

  • All FIX messages of type Single Order (35=D), Cancel Request (35=F) or Cancel/Replace Request (35=G) are considered to be an "order".
  • The first reply seen for an order acknowledges the order, and is used to calculate the ackLatency time. The reply can be an Execution Report (35=8) or Order Cancel Reject (35=9) coming from the opposite message direction to the order request.
  • Order acknowledgements for which the plug-in has not seen a corresponding request are handled by the out-of-order message processing. If no request message is seen then the acknowledgement latency is 0ms.
  • An order may skip states depending on the message received. For example, a Single Order (35=D) message followed by an Execution Report (35=8) message where the order is filled immediately will move straight from the unacknowledged to filled states. In this situation the ackLatency and fillLatency will be identical.

There are several possible states an order can be in, which can be used for filtering using the order-column match states setting. These states are also used internally and are displayed in the diagram below. The diagram also shows, for each order type, the conditions (i.e. FIX message contents) under which the order would transition to this state. See Technical Reference guide for more details.

image13

The FIX Analyser 2 plug-in also uses an internal tracking state for each order, which can (if desired) be configured for display by a value-type order-column or for output in a HTML report. These tracking states map to the order states above in the following way:

  • New - an order not seen previously by FIX Analyser 2, corresponding to the order state unacknowledged.
  • Acknowledged - an acknowledged order, corresponding to order states New and Partially filled.
  • Finished - a completed order for which we expect no further messages. Corresponds to the remaining order states Filled, Replaced, Cancelled and Rejected

Out-of-order messages

FIX Analyser 2 can handle a specific case of out-of-order (or unsequenced) FIX messages when tracking orders, where the initial message in an order is seen after some (or all) acknowledgement messages for that order. This case assumes that the acknowledgement messages are still ordered with respect to each other, and is intended to handle processing of FIX engine log files where the sent and received messages are logged to separate files.

Messages for sequenced orders (where the FIX messages are seen in normal ordered sequence) are processed immediately by configured views as described in the data flow section above. See section Message Processing for more details.

Out-of-order messages are processed by creating an order-specific row in the out-of-order message table. Data stored in this table is retained and not published to dataviews when the plug-in samples, and is said to be "deferred". An order will cease being deferred if:

  • The initial order creation message is seen.
  • 15 seconds has elapsed since the order was deferred (by absolute system time, not using message timestamps).
  • There are more than 1000 order entries in the out-of-order message table (the oldest such entries will cease being deferred).

When an order stops being deferred, the stored data is re-integrated back into the views and will become visible in a dataview on the next sample. Any further FIX messages for the order once re-integrated will be processed directly by the views.

Filters configured for user view order columns may work slightly differently when messages are seen out-of-order. Orders re-integrated because the initial order creation message was seen will evaluate filters on order columns against this initial message. The remaining re-integration cases will evaluate the filters against the first acknowledgement message seen.

Session state tracking

If a session analysis view or session status columns of a user view has been configured, the view will display the current state of the FIX session for each unique pair of SenderCompId / TargetCompId (session parties) FIX field values. The session state is inferred based on the content of the FIX messages processed by the plug-in.

For best results when displaying session states, it is recommended that FIX administrative messages are made available to FIX Analyser 2, including logon, logoff and heartbeat messages.

The logic used to determine the connection state is described below:

  • A session's status is Up if the plug-in has seen a logon message (35=A) from an initiating party and the corresponding reply from the accepting party (an explicit logon). This means that the session is in the connected state.
  • A session will also be considered connected if the plug-in detects a conversation already in progress (an implicit logon). This is detected by observing messages sent and received by both parties (that are not rejected).
  • A session is considered to disconnected (displayed as Down) if a logout message and acknowledgement is seen. This indicates that the connection has been terminated normally.
  • A connection is also considered disconnected if no messages are sent by either party for longer than the heartbeat interval defined for that session. In this case, the session status will still be displayed as Down, and the logoff column will additionally display TimedOut.
  • Sessions for which the plug-in has seen no FIX messages at all, or only an initial logon message but not the acknowledgement, displays the status Pending.
  • A session's status is Pending if an Explicit Logon (MessageType: 35=A) is followed by an Explicit Logoff (MessageType: 35=5). However, logonState is SentLogoff. Sample FIX messages:
  • 35=A
    2016-06-01 05:34:38,039 INFO [Fix message assembler] 
    in.PARTY01_PARTY02 - <29 Logon 
    (8=FIX.4.29=6235=A49=PARTY0156=PARTY0234=2952=20160531-21:
    34:38108=3098=010=081)
    
    35=5
    2016-06-01 05:34:38,043 INFO [Protocol] 
    out.PARTY01_PARTY02 - >2774 Logout 
    (8=FIX.4.29=11135=549=PARTY0256=PARTY0134=277452=20160531-
    21:34:3858=MsgSeqNum too low, expecting 2760 but received 
    29 Logon10=077)
  • A session's status is Down if an Explicit Logon (MessageType: 35=A) is followed by an Explicit Logoff (MessageType: 35=5) and is followed again by an Explicit Logoff (MessageType: 35=5), but in the opposite direction. Sample FIX messages:
  • 35=A
    2016-06-01 05:34:38,039 INFO [Fix message assembler] 
    in.PARTY01_PARTY02 - <29 Logon (8=FIX.4.29=6235=A49=
    PARTY0156=PARTY0234=2952=20160531-21:34:38108=3098=010=
    081)
    
    35=5
    2016-06-01 05:34:38,043 INFO [Protocol] 
    out.PARTY01_PARTY02 - >2774 Logout (
    8=FIX.4.29=11135=549=PARTY0256=PARTY0134=277452=20160531-
    21:34:3858=MsgSeqNum too low, 
    expecting 2760 but received 29 Logon10=077)
    
    35=5 (Opposite direction as symbolized by "<")
    2016-06-01 05:34:38,050 INFO [Fix message assembler] 
    in.PARTY01_PARTY02 - <2774 Logout (8=FIX.4.29=009935=534=
    277449=PARTY0156=PARTY0252=20160531-21:34:3858=00100000001
    o8YD_4204- Logout accepted10=021)
    
  • A session's status is Down if an Explicit Logon (MessageType: 35=A) is followed by an Explicit Logoff (MessageType: 35=5) and is followed again by an Explicit Logon, and an Explicit Logoff. Sample FIX messages:
  • 35=A
    2016-06-01 05:34:38,039 INFO [Fix message assembler] 
    in.PARTY01_PARTY02 - <29 Logon 
    (8=FIX.4.29=6235=A49=PARTY0156=PARTY0234=2952=20160531-21:
    34:38108=3098=010=081)
    
    35=5
    2016-06-01 05:34:38,043 INFO [Protocol] 
    out.PARTY01_PARTY02 - >2774 Logout 
    (8=FIX.4.29=11135=549=PARTY0256=PARTY0134=277452=20160531-
    21:34:3858=MsgSeqNum too low, 
    expecting 2760 but received 29 Logon10=077)
    
    35=A
    2016-06-01 05:34:40,039 INFO [Fix message assembler] 
    in.PARTY01_PARTY02 - <30 Logon 
    (8=FIX.4.29=6235=A49=PARTY0156=PARTY0234=3052=20160531-21:
    34:40108=3098=010=081)
    
    35=5
    2016-06-01 05:34:40,043 INFO [Protocol] 
    out.PARTY01_PARTY02 - >2775 Logout 
    (8=FIX.4.29=11135=549=PARTY0256=PARTY0134=277552=20160531-
    21:34:4058=MsgSeqNum too low, 
    expecting 2760 but received 29 Logon10=077)
    
    

Data persistence

During operation each FIX Analyser 2 sampler creates and maintains a SQLite database stored locally as a file on disk. This database is used to persist information about the current state of the sampler including:

  • Order data - data for all FIX orders seen used by the order tracking functionality.
  • File offsets - The current read position of input files FIX Analyser 2 is processing.
  • View data - computed values as shown in the published dataviews.

This persistent storage allows FIX Analyser 2 to resume processing from where it left off, if the sampler is stopped and restarted (as opposed to re-processing FIX log files from the start). Offsets of remote files read via a File Agent process are also persisted.

Creation and maintenance of this database is intended to be (almost) invisible to the user. During normal operation no user involvement should be required.

Persistence of data for setup changes is maintained on a best effort basis. Data for unaltered areas of configuration will be retained, whereas other data may be cleared. For example, if a new dataview column is added the existing column values will be retained. Any new data will then update both the old values in existing columns and create new values for new columns. Changing a dataview column definition will result in stored values for the column to be lost - values are not recomputed against the new configuration.

FIX Analyser 2 can be made to forget its current state by manually moving or deleting the SQLite database. This is done by removing the database file which by default is named managedEntityName.samplerName.db and is located in the Netprobe working directory. This file should not be removed while FIX Analyser 2 is still running - either stop the Netprobe or disable / unconfigure the sampler prior to file deletion.

The order and view content of the database file are cleared daily at the specified reset time (by default at 00:00 - midnight).

User Guide

This chapter describes how to configure and use the most common features of the FIX Analyser 2 plug-in. Users are expected to have some basic experience with using the Gateway Setup Editor to edit Geneos configurations.

Configuring Files

Inputs to the FIX Analyser 2 plug-in are configured in the form of paths to log files containing FIX messages. These log files can either be available locally on the FIX Analyser 2 Netprobe host, or accessed from a remote host using TCP/IP via a Geneos File Agent process.

Message direction

At a minimum, files must be configured with a file path and a message direction. The messageDirection setting describes to the plug-in whether the file contains FIX messages in two directions (the sendAndReceive option) or a single direction only.

Single direction only (the send or receive settings) should be used in the case where the FIX engine (or other logging process) splits the two sides of a FIX conversation into different log files. In this case it is not important which file is configured with which direction, but the files must have different directions (see the example below). It is recommended the direction is chosen to provide the least confusion to users maintaining the configuration.

image14

Figure 1 - Gateway Setup Editor view of FIX Analyser 2 files configuration.

Files containing FIX messages in both directions should be configured with the sendAndReceive message direction. This configuration also requires a regular expression (or regex for short) that specifies which messages in the file are in the receive direction - messages not matching this regex are assumed to be in the send direction.

As per single-direction send or receive files, it is not important which messages are chosen to be in the receive direction and which in the send: the regex should be chosen to be simple to maintain.

image15

Figure 2 - Gateway Setup Editor view of FIX Analyser 2 send-and-receive message direction configuration.

File paths

Paths used in a file definition in FIX Analyser 2 can be configured to be either absolute, or relative to the working directory of the FIX Analyser 2 Netprobe (or File Agent if this file is hosted remotely). Path configurations may also contain wildcard characters (* and ?) which by default will monitor all matching files found.

The file properties settings can be used to monitor only the latest modified file, which makes the behaviour more like other Geneos plug-ins such as FKM (see image below). These properties also control features such as reading FIX messages spanning multiple file lines (multi-line messages).

image16

Figure 3 - Gateway Setup Editor view of FIX Analyser 2 file properties configuration.

FIX Analyser 2 also supports date patterns (today, yesterday and tomorrow) for configuring paths, again in common with the Geneos FKM plug-in. See the filename setting and the time formatting codesAppendix for more details.

Timestamp parsing

When a new file definition is created in the FIX Analyser 2 plug-in configuration, timestamp parsing for FIX messages from this file is automatically set to sendingTime, which parses the content of the SendingTime field (52).

Depending upon the FIX engine or process sending the FIX messages, the precision of this field may only be to second resolution. FIX Analyser allows custom parsing of timestamp information using other information available in the same file line to obtain more precise timestamps.

To do this, users should examine the monitored log file and locate the timestamps for FIX message lines. A custom regex to capture the timestamp portion of the line should then be configured, where the first parenthesised expression will be the timestamp text. Many log files place the timestamp information at the start of a line, so a regex of the form ^(.{N}) where N is the length of the time text is generally a good starting point.

Example File line text Regex
1 2010/04/0700:00:00:598:FIXPump... ^(.{23})
2 lowinformation2010-04-2619:45:56.626... ^lowinformation(.{23})
3 22-JUN-2010,16:16:17:790,TX ... ^(.{24})

Once the text is extracted, it must then be interpreted using a format string. This information tells FIX Analyser 2 which parts of the timestamp text correspond with years, hours or seconds etc… using time formatting codes.

Example Timestamp text Pattern
1 2010/04/0700:00:00:598 %Y/%m/%d%H:%M:%S:%Of
2 2010-04-2619:45:56.626 %Y-%m-%d%H:%M:%S%f or %Y-%m-%d%H:%M:%S.%Of
3 22-JUN-2010,16:16:17:790 %d-%b-%Y,%H:%M:%S:%Of

The configuration for example 2 above appears in the GSE as follows:

image17

Figure 4 - Gateway Setup Editor view of FIX Analyser 2 custom timestamp configuration.

Reading remote files using Geneos File Agent

FIX Analyser 2 supports reading a log file remotely via a Geneos File Agent. The configuration for this mode is very simple. Assuming the File Agent has been installed on the target machine, simply configure a file definition as you could for a normal file, then set the File Agent settings on the file to point to the File Agent process hostname (and port number, if not using the default port).

image18

Figure 5 - Gateway Setup Editor view of FIX Analyser 2 file agent configuration.

Admin View

The administration (or Admin) view of the FIX Analyser 2 plug-in displays information regarding the FIX log file inputs to the plug-in.

Monitored files are represented as rows in the dataview output. If these files are located via a wildcard pattern in the default ('match all') mode, then they will appear as sub-rows of the pattern row. A non-wildcarded or 'match latest' path will be displayed as a single row. An example view is shown below.

image19

Figure 6 - An admin view showing various file conditions.

Headline Legend

Name Description
fileCount The number of files currently monitored by the plug-in.
iniFile The location of the conversations configFile if specified.
storeFile The location of the SQLite data persistence file.
messageStoreQueueSize* Size of the queue of messages to be logged to the message database.
messageStoreLogged* Number of messages written to the message store since the last reset. Counts both data and new file events.
messageStoreConnStatus* Status of the connection to the message database. Can display either Up or Down.

*Headlines only published when message database logging is enabled.

Table Legend

Name Description
id A row identifier generated by the plug-in - typically a variation of the file pattern configured, with the file agent connection details (if supplied).
file The actual name of a file being monitored.
Inode The inode number of the monitored file.
sizeKB The size of the monitored file in KB.
dbID The MySQL message database id (if enabled) for the file. Messages from this file will be logged with the displayed id.
localId A sequential id allocated to the file and used internally and in some messages logged to standard output.
offset The number of bytes of data that have been processed from this file.
msgsLastSample The number of messages read / processed from this file during the last sample.
msgsDay The number of messages read / processed from this file since the last reset. This count is zeroed at the reset time. The count is also reset if the Netprobe is restarted or the connection to the file agent (if configured) is lost.
fileAgent Displays the host and port of the file agent (if configured) that the file will be read from. An empty value indicates the file is being read from the local (FIX Analyser 2 host) machine.
status

The adapter status. For local files this displays OK when a file is readable or an error message otherwise.

If a file agent is configured for this adapter, the column will additionally display the connection state of the remote agent. The following table describes the possible fields:

  • Connecting - Waiting for connection to be established.
  • OK - Connection is established (and files detected correctly).
  • Rejected - Connection was rejected by remote process: either it is not a file agent or it already has a connection to a Netprobe. |
  • RejectRetry - The connection has been rejected however a retry is in progress. |
  • Disconnected - The connection could not be established.
expectedFilename Displays the expected filename, if no files could be opened. This is useful for file patterns configured with the <today>, <yesterday> or <tomorrow> syntax, to show what these fields have been expanded to.

Session Analysis View

The session analysis view displays the current status for the conversations on a monitored FIX engine. This view is useful for ensuring that specific connections remain up throughout the trading day.

Configuration

Configuration of the view consists simply of enabling the session analysis section in the views configuration list (see below). For additional information relating to session timeouts and expected status, the session analysis view can optionally be configured with a list of named conversations either directly or via an external INI-file.

image20

Figure 7 - Gateway Setup Editor view of FIX Analyser 2 session analysis configuration.

/PLUGIN:FIX-analyzeR2:sessionInfo
Path Fix-analyzer/Session info
Long Description Shows the comments associated with the message.
Effect This command displays the information related to a session in the SessionAnalysis view.
   

Target

/geneos/gateway/directory/probe/managedEntity/sampler[param('PluginName')='FIX-analyzeR2']/dataview[param('ViewType')='SessionAnalysis']/rows/row/cell

Arguments

Description Arg Type (Default) Value
Row name 1 XPath @rowname
Dataview name 2 XPath ancestor::dataview/@name
       

Output

image21

Figure 8 - A session analysis view showing 2 sessions: Columns 1-8 visible.

image22

Figure 9 - A session analysis view showing 2 sessions: Columns 9-14 visible.

Table Legend

Name Description
id The FIX conversation identifier; typically composed of the two parties involved in the conversation.
status

The current session status as detected by the plug-in.

Possible values are:

  • Up - Session is up.
  • Pending - The plug-in has not detected any messages for this connection, or has seen messages sent by one party but no corresponding messages returned by the other party.
  • Down - The conversation was terminated via a log-off message (or timeout).
logon Displays the logon time of a conversation, which is the time of the first logon message sent. If the connection was inferred by seeing messages between parties, the time shown is the time of this message and will be prefixed with the text "before"
logoff The logoff time. This may display "timed out" if a session timeout was detected.
expectedState This setting displays what the expected session status should be, as determined by the session start and end times. This is to aid in setting rules on the session status.
partyOne The name of party one - the "sender" as configured by the file message direction setting.
partyTwo The name of party two - the "receiver" as configured by the file message direction setting.
secondsSinceLastActivity The number of seconds elapsed since a message was last seen for this conversation.
protocol The FIX protocol version used.
partyOneSeqNumber Message sequence number for party one - the "sender" as configured by the file message direction setting.
partyTwoSeqNumber Message sequence number for party two - the "receiver" as configured by the file message direction setting.
adminMsg This column displays session administration events as detected by the plug-in. See the description of these events below.
rejectedMsgCount The number of rejected messages.
timeConnected The time in seconds the conversation has been Up.
partyOneMsgCount Number of messages sent by party one - the "sender" as configured by the file message direction setting.
partyTwoMsgCount Number of messages sent by party two - the "receiver" as configured by the file message direction setting.

Session administration events

The adminMsg column for a session analysis view displays information about administrative events detected by the plug-in. Output displayed will be in the format "time: event" where time is the timestamp of the FIX message and event may be one of the following (see below).

Event Description
Party attempting to logon Set when a logon message is sent by Party but the corresponding reply is not yet seen.
Party Logged Out Set when a successful session logoff is seen.
Party sent test request Set when a test request is sent by Party.
Party Rerequested N to M Set when a re-request is sent by Party for message sequence numbers N to M.
Party sent session reject Set when a session reject message is seen (additional reason text is displayed at the end of the event text).
   

The adminMsg column for each session row will be cleared (set to empty) when two minutes have passed since the FIX message timestamp relating to the event, except for session reject or logout messages which remain until replaced.

Message Detector Views

The message detector view checks all incoming FIX messages against a filter configured for the view. Messages which pass the filter are processed and a new row representing this message is added to the view. Users have control over the filter used, how the messages are grouped and what is extracted from the message for display (user-configured columns).

Configuration

The configuration example below shows the "Cancels" view from the example template configuration. The view uses the filter "cancel" which is defined in the template to match FIX messages where 35=8 (MsgType=ExecutionReport) and 150=4 (ExecType=Cancelled). Therefore, every row added to the dataview will represent an order being cancelled.

The messages in the output dataview are categorised by the uniqueId configuration. In the example configuration the fields chosen are FIX fields 49 and 56 (SenderCompId and TargetCompId). Therefore, each group of messages in the dataview represent the cancelled orders for each FIX conversation. Multiple messages seen for a connection will be displayed as sub-rows under a main group row, up to the configured messageCount limit.

The contents of each row are defined by the columns section. In this example, the values for each column (except Time) are extracted using the specified FIX field ids from the cancel message (fields 11, 55, … 38, 1). The Time column itself uses the named field TimeStamp which is one of an additional set of message properties provided by FIX Analyser 2.

image23

Figure 10 - Gateway Setup Editor view of FIX Analyser 2 message detector view template configuration.

Output

image24

Figure 11 - A message detector view. One group of messages is visible, with its associated sub-rows.

Headline Legend

Name Description
messageGroups The number of message groups (sets of messages) currently displayed by the view.
totalMessages The number of messages currently displayed in the view.
   

Table Legend

Name Description
id A unique identifier assigned to the message.
msgCount Number of messages detected for the corresponding group.
<other> Additional columns in the view are configured by the user.
   

Clearing message rows

Rows in the message detector view will be cleared automatically once the messageTimeout period (if configured) has elapsed. Messages can also be removed manually using the "Fix Analyser > Accept Message" right-click menu option. A whole message group can be removed by using the same option on a group row instead of a message sub-row.

image25

Figure 12 - Using the Accept message right-click menu option to remove a row from a message detector view.

/PLUGIN:FIX-analyzeR2:acceptMessage
Path Fix-analyzer/Accept Message
Long description Accepts this message.
Effect This command removes a message row from the message detector view. Using the same command on a group row, instead of a message sub-row, removes a whole message group.
   

Target

/geneos/gateway/directory/probe/managedEntity/sampler[param(\"PluginName\")='FIX-analyzeR2']/dataview[param('ViewType')='MessageDetector']/rows/row/cell

Arguments

Description Arg Type (Default) Value
Row name 1 XPath @rowname
Dataview name 2 XPath ancestor::dataview/@name
       

User Views

The user view allows users to define their own views by applying select statistical operations to process messages monitored by the plug-in.

User views are configured in the views section of the plug-in setup. Multiple views can be configured each of which will process the same input data as read by the file configuration. Views can be defined directly in the plug-in setup, or referenced from a common section (using the Gateway Setup Editor's static variables feature) to promote configuration sharing.

image26

Figure 13 - Gateway Setup Editor view of FIX Analyser 2 user view configuration area.

Categorisation (row configuration)

Messages are categorised into 1- or 2-level groups using the row identifier options by specifying primary (and optionally secondary) FIX fields to categorise on. These categorised are displayed as rows (and sub-rows for 2-level groups) in the output dataview. A field chosen for categorisation should be present in the initial message of a FIX order and its value should not change over the life of an order (although it may be omitted from subsequent messages in the order).

image27

Figure 14 - Gateway Setup Editor view of FIX Analyser 2 user view template configuration.

The example screenshots below show the difference between 1- and 2-level categorisation using row identifiers, against the same input data.

image28

Figure 15 - A user view with a primary row identifier of SenderCompID (FIX field 49).

image29

Figure 16 - A user view with a primary-secondary row identifier of SenderCompID (FIX field 49) and TargetCompID (FIX field 56).

Statistical Calculations (column configuration)

Statistical calculations on the input data are defined by configuring columns. There are two different types of column; message columns which take input from individual FIX messages and order columns which use values from order data computed by the plug-in.

Both column types are configured in the columns table in the User View configuration (see screenshot below).

The simplest type of column to configure is a message count column. To configure this, simply give the column a short understandable name, then click the "Message…" button and select count from the combo-box. This configuration will display a message count for each category row in the output dataview.

Other types of message column available are sum (which sums the values of the specified FIX field over all input messages) or value (which takes the latest value seen of the specified field). See Technical Reference guide for more details.

image30

Figure 17 - Gateway Setup Editor view of a user view with message columns being configured.

Message and order column data can be refined by adding either a filter or a rolling period or both. A filter attached to a column definition means the column will only process messages passing the filter. In the example above, a count message column "businessMsgCount" is configured with a filter BusinessMessage, which only passes business messages (ignoring FIX administration messages). The result is that the column only counts this type of message, rather than all FIX messages seen like the "msgCount" column.

A rolling period will cause the column to output only values seen within the defined period. The "msgCount_5Mins" column above will display a count of all messages seen in the preceding 5 minutes of each sample. If the BusinessMessage filter were additionally applied to this definition, the result would be a count of all business messages seen in the last 5 minutes.

Order columns behave much like message columns, but take data from order records rather than individual messages. The order tracking feature of FIX Analyser 2 tracks the latest value of FIX fields referenced by order columns per order, in addition to computing derived properties such as the ackLatency. This allows metrics such as the total LeavesQty (FIX field 151) to be calculated for each client.

Calculating this value using message columns would not be possible, as each successive ExecutionReport message would give decreasing values for the LeavesQty, which if summed would be incorrect. Using a value message column would give the correct value for a single order, but fail when messages for multiple orders are summarised by a single row in the view.

Order columns also have an additional built-in filter defined by the matchStates setting. This filter determines whether the FIX field value from the specified order will contribute to the output value or not, based on the current order state.

image31

Figure 18 - Gateway Setup Editor order column configuration dialog.

The screenshot above shows an order column configuration using match states. This definition produces a count of orders in the rejected state. Filters and rolling periods (not shown in the screenshot) may further adjust the resulting value.

Output

image32

Figure 19 - Example user view output.

Table Legend

Name Description
id The row identifier constructed as per configuration.
<other> Additional columns in the view are configured by the user.

Using the supplied template setup file

FIX Analyser 2 is packaged with a template for a Gateway Setup Editor (GSE) include file. This provides:

  • Several static variables defining useful user views, message detector views and column filters.
  • An example FIX Analyser 2 sampler definition using these views and filters. The sampler configures its input file(s) via a number of references to variables.
  • An environment definition giving example values for the file configuration variables.
  • A managed entity type referencing the example sampler.

Many of the examples in the previous sections are based on this template.

Having referenced this template from the Includes section of your Gateway setup, you can make use of it in a number of ways, depending on how closely the definitions in the template match your immediate requirements.

To make full use of the template, you can add the "Fixanalyzer2" type provided by the template to a new or existing managed entity, and reference an environment definition providing the symbols required by the template. The "Fixanalyzer2 Example" environment in the template defines these symbols for a particular format of input file; you can either reference it directly or use it as a model:

image33

Figure 20 - A managed entity definition referencing the template type and using a custom environment.

image34

Figure 21 - The custom environment, defining the variables required by the template sampler.

Alternatively, you can reference the views and/or filters defined by the template in samplers located either in your main Gateway setup file or in another include file.

Configuring the Message Database connection

Settings for the MySQL message database are contained in the Advanced configuration section. Users will need to configure the database server hostname, FIX Analyser 2 database instance name, and optional username and password used to authenticate with the database.

These details will be found (or configured) during database installation.

image35

Figure 22 - Gateway Setup Editor view of a MySQL database configuration.

Web-based reports

Show Orders report

Using the report

All FIX Analyser 2 dataviews except for the Admin view provide a right-click context-menu command ShowOrders… on rows. This command produces a HTML-based report giving additional details on the orders associated with the row, and optional message drill-down functionality assuming the MySQL message database has been configured.

To use the command simply right-click on a row and execute it.

Note: Individual rows in a message detector view will typically only return one order, although a message group row may return more data.

image36

Figure 23 - A "Show Orders" command being executed on a User View row.

/PLUGIN:FIX-analyzeR2:showOrders
Path Fix-analyzer/Show orders
Long description Shows the associated orders for this row.
Effect This command produces an HTML-based report with details on the orders associated with the row, and optional message drill-down functionality assuming the MySQL message database has been configured.
   

Targets

/geneos/gateway/directory/probe/managedEntity/sampler[param('PluginName')='FIX-analyzeR2']/dataview[param('ViewType')='User']/rows/row/cell
/geneos/gateway/directory/probe/managedEntity/sampler[param('PluginName')='FIX-analyzeR2']/dataview[param('ViewType')='MessageDetector']/rows/row/cell

Arguments

Description Arg Type (Default) Value
Row name 1 XPath @rowname
Dataview name 2 XPath ancestor::dataview/@name
Maximum number of orders to return 3 User Input: Integer 1000
       

Executing the command will display the following dialog. The number of results returned for a report is limited to maintain performance. By default the most recently updated 1000 orders are returned. The number can be changed as desired in the dialog.

image37

Figure 24 - A "Show Orders" command order limit dialog.

When the command is executed in Active Console 2, the report will be displayed using the default browser on the client machine. Example report output is displayed below. Orders associated with the selected row will be listed in a table.

This table supports several options for controlling its content, such as the number of entries per "table page", or whether completed orders will be hidden (they are shown by default). The columns of this table can be sorted by clicking on the column headers, and the table content can be filtered dynamically by typing in the search field.

image38

Figure 25 - An example "Show Orders" report.

If an order row is selected, then a FIX message drill-down table will be displayed below the order table. This feature requires the MySQL message database to be configured and available. The message table displays the individual FIX messages detected that made up the order, including chained messages where orders are changed or replaced.

As per the order table, the message table can also be sorted by clicking on the table headings. The raw FIX message contents (the full input log file line) is available by clicking the "Show/HideFIX messages" button.

image39

Figure 26 - An example "Show Orders" report showing message drill-down output.

Customising report output

The columns in the order table of the report output can be configured using the reportColumns setting for both user views and message detector views. This allows fields to be added or removed on a per-dataview basis so that order reports for different views can be more relevant to the data being displayed for that view.

For example, three additional columns have been added to the default columns using the following setup:

image40

Figure 27 - Gateway Setup Editor configuration showing custom order report columns for a user view template.

This configuration produces the following output:

image41

Figure 28 - An example "Show Orders" report showing additional customized columns.

FIX Message Query

If the MySQL message database has been configured, right-clicking on any FIX Analyser 2 dataview will provide a right-click context-menu command Query…

image42

Figure 29 - A "Message Query" command being executed on a FIX Analyser 2 dataview.

/PLUGIN:FIX-analyzeR2:queryMessage
Path Fix-analyzer/Query
Long description Queries FIX Messages.
Effect

This command provides an HTML-based report of messages matching the query parameters selected. The FIX messages results must match all provided parameter values.

If the StartTime and EndTime parameters specify an empty range (i.e. the times are the same), then it ignores the values. The ClientID and OnBehalfOfCompID fields are added for MySQL database schema version 1.1.

   

Target

/geneos/gateway/directory/probe/managedEntity/sampler[param('PluginName')='FIX-analyzeR2']/dataview

Arguments

Description Arg Type (Default) Value
SenderCompID 1 User Input: SingleLineString  
TargetCompID 2 User Input: SingleLineString  
ClOrdID 3 User Input: SingleLineString  
MsgType 4 User Input: SingleLineString  
Symbol 5 User Input: SingleLineString  
SecurityID 6 User Input: SingleLineString  
OrderQty 7 User Input: SingleLineString  
ClientID 8 User Input: SingleLineString  
OnBehalfOfCompID 9 User Input: SingleLineString  
StartTime 10 User Input: TimeInSecs  
EndTime 11 User Input: TimeInSecs  
LimitRows 12 User input: Integer 200
       

Executing this command will display the following dialog window allowing query parameters to be selected:

image43

Figure 30 - A "Message Query" command parameter dialog.

This command provides a HTML-based report of messages matching the query parameters selected. Resulting FIX messages must match all provided parameter values. If the StartTime and EndTime parameters specify an empty range (i.e. the times are the same) then the values will be ignored. The CliendID and OnBehalfOfCompID fields were added for MySQL database schema version 1.1.

When the query is executed, the report output will look similar to the following. As for an order report the output can be sorted by table heading. The number of entries per "table page" is changeable, and the table dynamically searchable using the search field.

image44

Figure 31 - An example "Message Query" report.

Altering the report URL

The reporting URL sent to Active Console 2 clients is automatically generated by the FIX Analyser 2 Netprobe. By default, this URL contains the hostname reported by the host machine and the Netprobe listen port. In some circumstances (particularly if accessing the Netprobe from a different domain) the hostname in the URL may fail to resolve.

To solve this issue, users may directly configure the URL that FIX Analyser 2 will report using the reportURLPrefix setting. The correct URL may either be directly specified, or indirectly specified via variables in the Gateway setup.

We recommend specifying the variable via a macro, such that the hostname used is taken from the Gateway probe configuration (and therefore more likely to contain the fully qualified domain name). An example of this is shown in the following screenshot.

Note: These variables can be configured anywhere permissible - they do not have to be configured specifically on an entity.

image45

Figure 32 - Gateway Setup Editor managed entity environment with macro variables.

The XML configuration for these variables is as follows:

<var name="probeHost">
<macro>
<netprobeHost></netprobeHost>
</macro>
</var>
<var name="probePort">
<macro>
<netprobePort></netprobePort>
</macro>
</var>

Once configured, the variables can be referenced in the reportURLPrefix setting:

image47

Figure 33 - Gateway Setup Editor reportURLPrefix setting using macro variables.

Direct web access to reports

Report URLs generated by FIX Analyser 2 contain any necessary data required to re-run the command. Message query URLs contain the query parameters, while order queries contain the name of the view and row originally clicked on when the command was issued. Refreshing or reloading the URL will regenerate the same report using the most recent data.

In addition, reports may be generated from a simple top-level page allowing users to specify report parameters as form input. This can be useful for making additional Message Queries (although less useful for Order Reports). The page can be reached at the following URL, and in the case of multiple FIX Analyser 2 samplers running will use the first configured instance to service requests.

http://probeHost:probePort/Fixanalyzer/

The access page appears as follows:

image49

Figure 34 - Simple web-based reports index page.

Technical Reference

This section describes the configuration settings and options available for the FIX Analyser 2 plug-in. Configuration for the plug-in is placed in the Gateway setup file under the fix-analyzer2 section of a sampler descriptor.

A FIX Analyser 2 configuration consists of 4 major parts:

  • Files - the set of files to be monitored.
  • Views - the statistical output to be displayed.
  • Filters - referenced by the view configuration to filter messages.
  • Database - optional database configuration for FIX message logging.

Detailed descriptions of these sections are described below.

All settings which specify regular expressions are evaluated using PCRE, the Perl 5 compatible regular expression library.

Files

The files configuration section lists all the FIX log files that the plug-in will monitor during operation. These files will be scanned for FIX messages, and any messages found will then be parsed and passed to all configured views for processing.

File names can be specified with absolute or relative paths, and may contain wildcard or date-based pattern. Files can also be streamed from a remote host by using a Geneos File Agent.

At least one file must be specified in order for the plug-in to operate.

Mandatory: Yes

files > file > filename

This setting specifies the path of a file (or files) to monitor. The path can be specified either as a relative path from the Netprobe working directory, or as an absolute path.

If the path contains wildcard characters (* or ?) then Netprobe will evaluate this path to find files matching the pattern. Either the latest or all of these files will then be monitored, subject to the value of the wildcardMode setting for the file. Monitoring all files can be useful for FIX engines which create a separate log file per FIX session.

The path may also contain the special tags <today>, <yesterday> or <tomorrow>. These tags will be expanded each day to a numerical representation of the date, YYYYMMDD.

For example, on 27th May 2011 the filename fix_<today>_file.log will be expanded to fix_20110527_file.log. Yesterday and tomorrow tags will expand to 20110526 and 20110528 respectively.

The numerical format can be controlled by inserting time formatting codes inside the tag. For example, on the same date the filename fix-<today%m-%d-%Y>.log expands to fix-05-27-2011.log.

Mandatory: Yes

files > file > fileProperties > wildcardMode

The wildcard mode specifies how files configured with a wildcard (* or ? characters) will behave. Available wildcard modes are as follows:

Name Description
MATCH_ALL All files matching the wildcard will be read by Fix-analyzer.
MATCH_LATEST_MODIFIED Only the file with the latest modification time will be read by Fix-analyzer.
MATCH_LARGEST_GROWING
The file to have grown the most between this and the last check will be selected.

Note

(1): On the first check after the sampler (or file agent, if applicable) is started, the file with the latest modification time will be read. Thereafter, the matching file which has grown the most between file checks will be read; if the file being read becomes unreadable and no other file is growing, an error will be reported and no file will be read until a matching file increases in size.
(2): This mode is provided as a work-around for the behaviour of some (Windows) file systems that do not correctly maintain the modification time for files. It should not be used unless MATCH_LATEST_MODIFIED behaviour is required, but the file system in question has that limitation.

Mandatory: No
Default: MATCH_ALL

files > file > fileProperties > maxLinesPerCheck

This setting specifies the maximum number of lines to process, each time the file is checked for data. Files are checked every second. If the maximum limit is reached but the file still contains lines to read, the remaining lines will not be processed until the next check. This setting can be used to fine-tune how file reading operates, in addition to the file control options.

This setting is not applied when files are being read via a file agent.

Mandatory: No
Default: unlimited

files > file > fileProperties > multiLine

Specifies settings for reading log multiple lines from a log file, and treating this as a single message to process.

Mandatory: No
Default: Each individual file lines is treated as a single message

files > file > fileProperties > multiLine > rtsLog

Shorthand setting to specify the multi-line configuration for an RTS log file. This is the equivalent of setting a custom configuration with the values:

Mandatory: One of rtsLog or custom must be specified.

files > file > fileProperties > multiLine > custom

Allows users to specify a custom multi-line message format, using regular expressions.

Lines are read from the input log file, and then checked against the startRegex to find a start of a message. When inside a message, lines are also checked against the dataRegex (if specified) to extract a specific part of the line, and the endRegex (if specified) to determine the end of the message.

If a line is found to match the startRegex while "inside" a message, then the current message data is passed for processing, and a new message is started. Lines matching the endRegex will also be added to the message, and end the message at that point.

Mandatory: One of rtsLog or custom must be specified.

files > file > fileProperties > multiLine > custom > startRegex

Specifies the start of a multi-line message. A line matching this regex will start a new multi-line message, and pass any previous message onwards for processing.

Mandatory: Yes

files > file > fileProperties > multiLine > custom > dataRegex

The data regex allows users to specify how to extract the "data" part of a file line to add to a multi-line message. The regex must contain a named capture-group called data. Only the part of the line matching this capture-group will then be added to the message.

For example, with the regex:

EMSOutput.java:420\) (?<data>.*)

And the file line:

2010-03-04 18:04:23,691 INFO (EMSOutput.java:420) 8=FIX.4.2 9=460 ... 10=090

The data added to the message would be:

8=FIX.4.2 9=460 ... 10=090
Mandatory: No
Default: The whole line is added to the message.

files > file > fileProperties > multiLine > custom > endRegex

The end regex allows users to specify a pattern for a line that indicates the end of a multi-line message. When this line is seen it is then added to the message, and the completed message is passed onwards for processing.

Mandatory: No
Default: A message remains "open" until the next line matching the startRegex is seen.

files > file > fileProperties > multiLine > custom > stripNewlines

The strip newlines setting controls whether newlines are stripped from the file lines, when appending new file lines to the multi-line message. If set to true, newlines will be stripped so the end message will seem to be one long line of data. If false, newlines from the file will be kept.

Mandatory: No
Default: false (newlines are retained)

files > file > fileAgent

If configured, this setting specifies that the file should be read from a remote machine via a TCP/IP connection to a Geneos File Agent process running on that machine.

The hostname of this machine must be specified.

The listen port of the file agent is optional, and if not specified will use the default port 7030.

Mandatory: No

files > file > timestamp

The timestamp setting describes how to obtain a message time, for data read from the file(s) the setting is specified against. In addition to displaying the timestamp in user-configured views, the time is also required for performing order matching latency calculations, and is used in the session analysis view when detecting FIX conversations.

Mandatory: Yes

files > file > timestamp > sendingTime

This setting configures the plug-in to read the SendingTime field (FIX field 52) of FIX messages from the file to obtain the message timestamp.

Note: This field may only contain timestamps to second resolution, depending upon the exchange or FIX engine being used. It is recommended that clients check the log files when configuring the plug-in, and choose the most accurate timestamps available. See the custom timestamp setting in the Technical Reference guide for more details.

Mandatory: Either sendingTime or custom timestamp must be configured per file.

files > file > timestamp > custom

A custom timestamp configuration should be used when the FIX SendingTime field (field 52) of a FIX message is either not present, or does not contain the message timestamp to the desired level of accuracy.

Using this custom timestamp setting, users can configure a regex pattern to extract the timestamp portion of a file line. The plug-in will then interpret this time using the configured format.

Mandatory: Either sendingTime or custom timestamp must be configured per file.

files > file > timestamp > custom > regex

This regular expression (regex) is used to extract the timestamp component of the message, which is then interpreted by the format. The first capture group (the regex component in parenthesis) is taken to be the timestamp.

Example File line text Regex
1 2010/04/0700:00:00:598:FIXPump... ^(.{23})
2 lowinformation2010/04/2619:45:56.626... ^lowinformation(.{23})
3 22-JUN-2010,16:16:17:790,TX ... ^(.{24})

In example 1, the regex extracts the first 23 characters of the file as the timestamp.

Example 2 is similar; it extracts 23 characters of timestamp following the text "low information", which must occur at the start of the file.

For the last example (3) again we extract characters from the start of the file, but this time 24 characters are required as the timestamp is slightly longer.

The regex can optionally be configured with flags that change the pattern matching behaviour. By default no flags are set. Available flags are:

Flag Name Description
i Case insensitive
The regex matching is done in a case-insensitive manner.
e.g. The letter "A" in the regex pattern will match both "A" and "a" (in both upper and lower case).
s Dot matches all
The dot (.) special character matches all characters.
I.e. It will additionally match the newline character.

Mandatory: Yes

files > file > timestamp > custom > format

The format is used to interpret the timestamp once it has been extracted by the timestamp regex. Timestamps are interpreted using a string containing special time formatting code characters.

Example Timestamp text Pattern
1 2010/04/0700:00:00:598 %Y/%m/%d%H:%M:%S:%Of
2 2010/04/2619:45:56.626 %Y/%m/%d%H:%M:%S%f or %Y/%m/%d%H:%M:%S.%Of
3 22-JUN-2010,16:16:17:790 %d-%b-%Y,%H:%M:%S:%Of
4 WedJul 2815:17:53:103 %a%b %d%H:%M:%S:%Of

The fields of the timestamp in example 1 are read directly in the pattern.

For example 2 there is a choice on how to read the milliseconds. The format code %f will read both the dot (.) and the millisecond digits, or the code %Of will read only the digits and so needs to be prefixed by dot.

The timestamp for example 3 contains a textual month description instead of a numerical value. This is read by using the format code %b to read the month as text (locale dependant).

Finally, for example 4 the date does not contain the year. When interpreting this pattern, the plug-in will fill in the missing fields (in this case only the year) using the system date and time. For files containing only a time (and not the date) in the timestamp, the plug-in will fill in the missing date fields (the entire date) using the current date.

Mandatory: Yes

files > file > messageDirection

Each message processed by FIX Analyser 2 has an associated direction, either SEND or RECEIVE. The direction is assigned to the message according to the value configured for this setting.

The message direction is used by FIX Analyser 2 in order to determine whether messages apply to same FIX session, and to invert paired FIX fields for grouping in a user view.

For example, if SenderCompId (FIX field 49) has been configured as a grouping, then messages in the SEND direction will be categorised using this field. However, messages in the RECEIVE direction will use values from the paired field TargetCompId (FIX field 56) for grouping instead.

Mandatory: Yes. One of send, receive or sendAndReceive must be specified.

files > file > messageDirection > send

This setting indicates that all FIX messages from this file have a SEND direction.

Mandatory: Mutually exclusive (one of send, receive or sendAndReceive must be specified).

files > file > messageDirection > receive

This setting indicates that all FIX messages from this file have a RECEIVE direction.

Mandatory: Mutually exclusive (one of ref:send <send>, receive or sendAndReceive must be specified).

files > file > messageDirection > sendAndReceive

The send-and-receive message direction indicates that the monitored log file contains FIX messages in both directions; the exact direction for each message read will be set using the configured regex.

Messages are marked as having a RECEIVE direction if they match the receiveId regex pattern. FIX messages which do not match this pattern are assumed to have a SEND direction. Possible FIX messages that are neither send nor receive messages (e.g. error reports) should be excluded using inputTextFilters.

The receiveId pattern can additionally be configured with flags that change its behaviour. Available flags are:

Flag Name Description
i Case insensitive
The regex matching is done in a case-insensitive manner.
e.g. The letter "A" in the regex pattern will match both "A" and "a" (in both upper and lower case).
s Dot matches all
The dot (.) special character matches all characters.
I.e. It will additionally match the newline character.

Mandatory: Mutually exclusive (one of send, receive or sendAndReceive must be specified).

files > file > fieldSeparator

This setting specifies the field separator character used within a FIX message. This may be of use if the FIX engine logs messages using a different separator character (e.g. | or #).

This character can be specified directly (e.g. #) or using hexadecimal notation (e.g. \x23) or octal notation (e.g. \043). Hexadecimal or octal notation is useful when specifying non-printing or whitespace characters, as these will not display directly in the Gateway Setup Editor (GSE).

Mandatory: No
Default: The Start-Of-Header (SOH) special character as defined by the FIX protocol, which is ASCII code 001.

Filters

Filters are used to select subsets of FIX messages, primarily when computing statistics for user-defined columns. Because the column types are generic (e.g. a count column is just an incrementing integer value) filters are used to make the columns more specific. For example, by attaching a filter to a count column to only process "New Order" messages, the column value then displays a count of all "New Order" messages seen.

Filters can either be defined in the filters configuration section or in a separate filter list which is then referenced by the plug-in. Individual filters are then referenced by name from other parts of the setup.

At present there are three types of filter; Comparison filters, Regex filters and Boolean filters (and, or and not).

Mandatory: No

filters > filter > name

This setting specifies the name of the filter. Filters are referenced by the name from other parts of the plug-in configuration, and so are required to be unique.

Mandatory: Yes

Comparison Filters

Comparison filters compare the content of a FIX message against a set of preconfigured values. This is the most common type of filter and can be used (for example) to filter by message type or order volume.

To configure a comparison filter 3 settings are required; the FIX field ID to compare, a comparison operation and a set of values to compare against.

When using an equality and inequality operators, if both the FIX message value and the configured value are numeric (i.e. are integer or floating-point values) the comparison will be performed numerically. In all other cases comparisons are performed as a case-sensitive string comparison.

When using the other relational operators (i.e. <, <=, > and >=), comparisons will only be performed if the values are the same data type. That is, these operators will only be performed for strings against strings (using dictionary order), or numbers against numbers (numerical comparison). Any other comparisons will result in a "false" output. (i.e. "A" < 23 gives false, as does "A" >= 23).

filters > filter > comparison

The comparison section of a filter marks the filter as being a comparison-type filter. This filter compares a FIX message field value against a set of pre-configured values.

Mandatory: Yes (if specifying a comparison filter).

filters > filter > comparison > fieldId

This setting specifies the FIX field id which will be compared for a comparison filter. The value of this field will be extracted to compare against the configured values. If the message does not contain a field with this id, then an empty value will be used for comparison.

Mandatory: Yes

filters > filter > comparison > comparison

Using this setting you can specify the type of comparison operation which will be performed for a comparison filter. The table below lists the available comparisons:

Setting Description
EQUALS The FIX message passes this filter if the message field value is equal to one of the configured comparison values.
NOT_EQUALS Pass if the field value is not equal to any of the values.
LESS_THAN Pass if the field value is less-than any of the values (therefore only the largest configured value counts towards the result).
LESS_THAN_EQUALS Pass if the field value is less-than or equal to any of the values.
GREATER_THAN Pass if the field value is greater-than any of the values (therefore only the smallest configured value counts towards the result).
GREATER_THAN_EQUALS Pass if the field value is greater-than or equal to any of the values.

Mandatory: Yes

filters > filter > comparison > values

This setting specifies values which the FIX field value will be compared against for this comparison filter.

If configuring LESS_THAN or GREATER_THAN operators, only a single value needs to be configured. During comparison all values defined will be checked, however for LESS_THAN operators the value of the largest value will determine whether the message passes or fails the filter. Similarly for GREATER_THAN operators it is the smallest value which is important.

Mandatory: Yes - at least one value must be configured per comparison filter.

Regex filters

filters > filter > regex

Regex filters compare the raw message content (i.e. the un-parsed text containing the FIX message) against a configured regular expression to check for a match.

Regex filters require a minimum of a regex pattern to be specified. If the message text matches the regex, then the FIX message will pass the filter and be processed.

The optional invert setting reverses the result of the filter. If enabled (set to true) then a message which matches the regex pattern will not pass the filter.

Mandatory: Yes (if specifying a regex filter).

Boolean filters

Boolean filters allow multiple sub-filters to be combined together using Boolean logic to produce more complex and meaningful filters. Sub-filters of a Boolean filter can either be a name-reference to another filter, or a contained inline filter definition. Boolean filters can be made up of other Boolean filters if desired.

Three types of Boolean filter can be configured - and, or and not filters.

filters > filter > and

The andBoolean filter combines the results of its configured sub-filters. In order for a message to pass the filter, it must pass all configured sub-filters. If any one of the sub-filters fail to pass the message then this filter also fails. If no sub-filters are configured, the and filter will pass all messages.

Any number of sub-filters can be configured, and will consist either of a name-reference to another filter or be a new filter defined inline at the point-of-use.

Mandatory: Yes (one of and, or, not must be specified).

filters > filter > or

The orBoolean filter combines the results of its configured sub-filters. In order for a message to pass the filter, it must pass any one of the configured sub-filters. If all of the sub-filters fail to pass the message, then this filter also fails. If no sub-filters are configured, the or filter will pass all messages.

Any number of sub-filters can be configured, and will consist either of a name-reference to another filter or be a new filter defined inline at the point-of-use.

Mandatory: Yes (one of and, or, not must be specified).

filters > filter > not

The notBoolean filter inverts the result of the configured sub-filter. A FIX message which passes the sub-filter will fail the not filter, and vice versa.

The sub-filter can be either a name-reference to another filter, or be a new filter defined inline at the point-of-use.

Mandatory: Yes (one of and, or, not must be specified).

Filter Lists

Filter lists allow filter definitions to be shared between samplers. This can help reduce configuration burden (fewer filters need to be configured) and also increase maintainability by providing a single location to check when filters need to be updated.

Filters lists are configured in the top-level Staticvariables section of the gateway setup, inside the Fix-analyzertemplates sub-section.

To use a filter list in a FIX Analyser 2 plug-in the list must be referenced by name. Filters defined in this list are then available for use by other parts of the plug-in setup, and are referenced by name in the same was as for normal filters defined directly in the plug-in setup.

Filter names inside a single sampler definition must be unique. If a filter in a filter list has the same name as another filter (either defined in the same filter list, another referenced filter list, or directly in the plug-in configuration) then this is an error. The Netprobe log will contain diagnostic output in this case.

Views

The views configuration section defines the different views that the plug-in will produce. Each view is passed all FIX messages read in from the configured files, and the output of the view is defined by the user-configuration.

There are three types of view which can be configured:

  • User view - a custom view created by the user, which can generate statistics on a per-message or per-order basis.
  • Message Detector view - rows are created dynamically when a message passing some user-configured criteria is detected. Selected values from the FIX message are extracted and placed in the row.
  • Session Analysis view - the session analysis view monitors FIX sessions (or conversations) based on the FIX messages being processed by the plug-in. Session state information and statistics on these sessions are output to the dataview. See Technical Reference guide for more details.

User view

views > user

User views are a customised view configured by the user. These views contain configuration settings for the rows (categories) and columns (statistical values) to be computed for the input FIX messages and displayed in the dataview. Column values can be driven using order data or values from individual FIX messages as specified by the column type.

The user view can be configured either by specifying the view settings directly in the plug-in configuration, or by referencing a user view template static variable. If the view is specified as a static variable, the setup can be shared by a number of plug-in instances.

Mandatory: No

views > user > fix-analyzer-user-viewTemplate

This setting references a user view from a static variable.

Mandatory: No
See the Gateway 2 Reference Guide for more information on static variables.

views > user > data

This setting specifies a user view configuration directly in the plug-in setup.

Mandatory: No

views > user > data > viewName

Specifies the name of the user view. View names must be unique across all views for a sampler, so that each view can be uniquely referenced.

Mandatory: Yes

Row configuration

views > user > data > rowIdentifier

The row identifier setting specifies how FIX messages are categorised when computing statistical data in the output dataview. The values of the specified FIX fields are extracted from the FIX message being processed, and used to form the name of the row which will process updates from the message.

This process works slightly differently depending upon the column type as follows.

Message columns

For each FIX message processed, the value of the rowIdentifier FIX field(s) are extracted and used to generate a row name. If this row does not already exist a new row is added to the view. Message columns in the named row are then updated with the content of the FIX message.

If a message does not contain the specified FIX field (or has an empty value) then the value of the field as seen by the associated order for this message (if any) is used. If there is no order, or the order does not contain a value, then the text "<none>" is used as the row name.

Order columns

Operation is similar to the above, except that row names are generated directly from the order information associated with the FIX message. The appropriate row is found (and created if necessary) and order columns updated with any changes to the order state caused by the FIX message.

Note: Order updates are limited to a single dataview row. As a consequence, the FIX fields used as the rowIdentifier (for categorisation) should be chosen with care. Fields which change value during the lifetime of an order are unsuitable as rowIdentifiers.

It is allowable however, that the first message of an order contains a value for the rowIdentifier field, and that the remaining messages do not contain this FIX field at all. In this case the field value will be cached and all FIX messages for the order correctly processed by the same dataview row.

Mandatory: Yes

views > user > data > rowIdentifier > custom

The custom row identifier allows the user to specify the FIX field tags, which will be extracted and used as the row name. FIX Analyser 2 allows 1 or 2 levels of identifier tags; primary and secondary. Secondary tags can only be specified once a primary tag has been configured.

If both primary and secondary row identifiers are configured, the result is a view with 2 levels of row name. Row names of the form primaryValue#secondaryValue are value rows, and the statistics (column values) on this row display values for those messages with matching identifier fields. Row names of the form primaryValue are summary rows, and summarise the data of the value sub-rows with the same starting prefix.

See the user guide configuration section for example screenshots.

Mandatory: Yes - one of custom or senderTarget must be configured.

views > user > data > rowIdentifier > senderTarget

The senderTarget setting specifies that the view should use the FIX fields SenderCompId (49) and TargetCompId (56) for the row identifier.

If using these fields you may also optionally enable the session status columns, which allows you to create a customised session analysis view.

Mandatory: Yes - one of custom or senderTarget must be configured.

views > user > data > rowIdentifier > senderTarget > showSessionStatus

If enabled, this setting adds the columns status, logon and logoff to the user view. These columns are normally displayed in a session analysis view, so enabling these allows the user to configure a customised session analysis view using the user view.

Mandatory: No
Default: false - session status columns are not displayed.

views > user > data > rowIdentifier > senderTarget > showSessionStatus > sessionTimeout

The session timeout specifies the default maximum period of inactivity for a session in seconds. If no messages are sent or received for this period of time, the session is considered to be timed out. This is displayed in the logoff column.

When timeout delay setting is non-zero, the maximum period is increased by the value specified. This setting can be used to cater for delays in writing or processing FIX messages which would otherwise incorrectly display a session as timed out.

Timeouts are calculated by comparing the last message timestamp for that session against the local time of the Netprobe host.

Mandatory: No
Default: 0 - no timeout

views > user > data > rowIdentifier > senderTarget > showSessionStatus > timeoutDelay

The timeout delay setting specifies an extra period (in seconds) of allowable inactivity for a FIX session. If a session is idle for a period equal to sessionTimeout +timeoutDelay seconds, then the session is considered to be timed out. This setting is ignored if timeouts are disabled.

Expected use of this setting is to cater for any delays between FIX messages being written to log files and being fully processed by FIX Analyser 2. Additional time can be granted to prevent the session being incorrectly shown as timed out; typically 1 or 2 seconds should be enough depending on normal message processing rates.

Mandatory: No
Default: 0 - no timeout delay.

views > user > data > rowIdentifier > senderTarget > showExpectedStatus

This setting controls whether the expectedStatus column is displayed in the view. This column is normally present in a session analysis view if named FIX conversations have been configured. It can optionally be added to a user view to allow users to configure a customised session analysis view.

The expectedStatus column displays whether a session (senderTarget row) is expected to be Up or Down based on the time information configured in the conversation setting. Values in this column will be empty unless conversations have been configured.

Mandatory: No
Default: false- expectedStatus column is not displayed.

Column configuration

The column configuration defines the data to be displayed for each row in the dataview output. Users can extract message or order values and compute selected statistics from these for display, using the configuration described below.

User views currently support two types of column; message columns and order columns.

Mandatory: Yes

views > user > data > columns > column > displayName

Specifies the name of the column as it will appear in the dataview. The name must be unique among all other columns configured for the view.

Mandatory: Yes

views > user > data > columns > column > rollingPeriod

This setting allows a rolling period to be configured for the column. This allows you to (for example) configure a column that shows a count of messages received in the last 5 minutes.

Values are specified as an integer in 3 different units: seconds, minutes and hours. To configure a non-integer period of 1.5 hours for example, configure the period as 90 minutes instead.

The period unit-type also specifies the granularity of the period. Values will be placed into "buckets" containing all values seen for a particular time. The time is taken from the FIX message timestamp truncated to the unit specified. The overall value of the period is then a statistical operation on the most recent X buckets (where X is the specified period length).

e.g. When using a period of minutes:

Message Timestamp Rolling period "bucket"
2012-01-1608:45:12:713 2012-01-1608:45
2012-01-1608:45:53:061 2012-01-1608:45(samebucketasprevious)
2012-01-1608:46:04:288 2012-01-1608:46

Therefore, for more accurate rolling period values users should choose a more granular unit type when only using a small number of units. For example, a rolling period of 60 minutes would be better than a period of 1 hour.

Note: There is a memory cost to more granular rolling periods however, as more values need to be stored.

Mandatory: No

views > user > data > columns > column > filter

References a filter from the filters section by name. Filters are applied to input messages before the column processes any data. This allows the creation of (for example) a count of Cancel messages, by configuring a count column and filtering out all non-Cancel messages.

See the message column and order column sections in the Technical Reference guide for more details.

Mandatory: No
Default: No filter - column will process all updates.

Message columns

views > user > data > columns > column > columnType > message

Message columns process FIX messages individually when computing their statistics. There are 3 types of message column which can be configured; sum, count and value columns.

If a message column is configured with a filter, then only messages which pass the filter are processed by the column. Therefore, a count column (for example) would count only the messages which passed the filter. This could be used to create a count of Cancel messages, by applying a filter that accepts only Cancel messages.

When aggregating an empty data range, the sum and value message columns will output an empty call in the dataview display (count columns will display as 0). This behaviour is intended and allows for improved data processing when used in conjunction with the database logging or charting features of Geneos.

Mandatory: Yes

views > user > data > columns > column > columnType > message > type > sum

The sum message column sums the values of the specified FIX field, for all messages processed by this column (i.e. messages that passed the filter, if configured).

Summary rows for sum columns will display a sum of all values displayed for the sub-rows.

A sum message column could be used to obtain the total size (in bytes) of FIX messages processed for a particular party, by summing FIX field 9 (BodyLength).

Sum columns will produce an empty value when no data is present. This may occur when summing messages that do not contain the configured FIX field, if a rolling period is configured and no values have been received within the configured period, or if the column has processed no FIX messages at all.

Mandatory: Yes - one of sum, count or value must be configured.

views > user > data > columns > column > columnType > message > type > count

The count message column is an integer value which starts at zero and increments by one for each message processed by the column (i.e. messages that pass the filter, if configured).

Summary rows for count columns will display a sum of all counts displayed for the sub-rows.

A count column could be used to could all Reject messages for a particular client, by configuring a filter to match on FIX field 35=9 (OrderCancelReject).

Mandatory: Yes - one of sum, count or value must be configured.

views > user > data > columns > column > columnType > message > type > value

The value message column stores the extracted FIX field or named value of the latest message processed by the column (i.e. messages that passed the filter, if configured).

Named property values which can be selected are as follows:

Property Description
TimeStamp
The timestamp of the message in a human readable format
E.g. Thu Aug 23 14:55:02.148 2010.
SecondsSince The number of seconds since the timestamp value, using the current local time of the Netprobe host.
MinutesSince The number of minutes since the timestamp value, using the current local time of the Netprobe host.

Summary rows for value columns will display the latest value (using the message timestamp) of the values displayed by the sub-rows. If no messages with the configured field are received (within the rolling period if configured) then an empty dataview cell is produced. See the message column description for a brief summary on this behaviour in the Technical Reference guide.

A message value column could be used to show the latest error for a particular session, by extracting FIX field 58 (Text). This field is typically used to describe the error for a Don't Know Trade (35=Q) or an order reject execution (35=8 and 150=8).

Mandatory: Yes - one of sum, count or value must be configured.

Order columns

views > user > data > columns > column > columnType > order

Order columns display a statistical summary for all orders applicable to the given dataview row. For example if the row configuration specifies that rows equate to exchanges, then order columns for a specific row will display statistics for orders to that exchange.

Available column types are:

  • Value
  • Sum
  • Min
  • Max
  • Average
  • Standard deviation

Order statistics can be filtered by the current order status, as specified by the match states setting. See the order tracking section in Message Processing for more details.

Filters for order columns are applied to the first message of the order seen for each dataview row, so that the order is only filtered once for that column.

Order columns (with the exception of the count column type) may return an empty cell value when aggregating an empty data range. This behaviour is intentional and allows for improved data processing when used in conjunction with Geneos database logging or charting features. See the specific reference section for each column-type for further details.

Mandatory: Yes

views > user > data > columns > column > columnType > order > type > matchStates

This setting specifies the states of orders which will contribute towards the column value. Orders with a state that has not been configured here will not be considered by the column.

Possible states are as follows:

State Description
unacknowledged
An order which has not been acknowledged by the other party.
E.g. A Single Order message (35=D) for which no corresponding Execution message (35=8) has been seen.
new An acknowledged order.
partiallyFilled An order which has been partially filled.
filled An order which has been completely filled. This state will also apply to a successful Order Cancel Request (35=F).
replaced An order which was replaced by a successful Order Cancel / Replace Request (35=G).
cancelled An order which was cancelled by a successful Order Cancel Request (35=F).
rejected An order which was rejected, or an Order Cancel Request (35=F) which was rejected by an Order Cancel Reject message (35=9).
finished Provided for convenience of configuration - this state will match all filled, replaced, cancelled or rejected orders.

See the order tracking section for more details on state changes.

For example, a count order column configured for the unacknowledged state would show a count of all unacknowledged orders. Orders which were subsequently acknowledged would then be removed from the count. By adding a filter to this count, we could (for example) count all unacknowledged orders of a particular type.

Mandatory: Yes

views > user > data > columns > column > columnType > order > type > count

The count column shows the number of orders which pass both the configured filter and order match states, or 0 when there are no matching orders. An example usage of this column could be to display the number of open orders (i.e. states "new" or "partiallyFilled") for that row.

Mandatory: Yes - a valid order-column type must be configured.

views > user > data > columns > column > columnType > order > type > value

The value column shows the latest updated value for the specified FIX field or named property, from orders which pass the configured filter and order match states.

Value columns will display an empty if no matching values are present. This may occur if messages for matching orders do not contain the configured FIX field, if a rolling period is configured and no order updates have occurred within this time, or if no orders were processed at all. See the order column description for a brief summary on this behaviour.

Mandatory: Yes (a valid order-column type must be configured).

views > user > data > columns > column > columnType > order > type > sum

The sum column sums values for the specified FIX field or named property extracted for all applicable orders, subject to the filters and order match states selected. Non-numeric values processed by this column will be converted to a number if they begin with a numeric prefix, otherwise they will be treated as zero.

An example usage for this column could be to sum FIX field 151 (LeavesQty) for all orders in states new or partiallyFilled. The result would be the total quantity for all orders, which are the units remaining to be executed (per dataview row).

Sum columns will display an empty value if there are no matching orders or values. This may occur if messages for matching orders do not contain th configured FIX field, or if a rolling period is configured and no order updates have occurred within this time, or if no orders were processed at all. See the order column description for a brief summary on this behaviour.

Mandatory: Yes (a valid order-column type must be configured).

views > user > data > columns > column > columnType > order > type > max

The max column displays the maximum value of all FIX field or named property values extracted for applicable orders, subject to the filters and order match states selected. Non-numeric values processed by this column will be converted to a number if they begin with a numeric prefix, otherwise they will be treated as zero.

A column of this type could be configured to display the maximum acknowledgement-latency for all orders of a dataview row.

Max columns will display an empty value if there are no matching orders or values. This may occur if messages for matching orders do not contain the configured FIX field, or if a rolling period is configured and no order updates have occurred within this time, or if no orders were processed at all. See the order column description for a brief summary on this behaviour.

Mandatory: Yes (a valid order-column type must be configured).

views > user > data > columns > column > columnType > order > type > min

The min column displays the minimum value of all FIX field or named property values extracted for applicable orders, subject to the filters and order match states selected. Non-numeric values processed by this column will be converted to a number if they begin with a numeric prefix, otherwise they will be treated as zero.

A column of this type could be configured to display the minimum acknowledgement-latency for all orders of a dataview row.

Min columns will display an empty value if there are no matching orders or values. This may occur if messages for matching orders do not contain the configured FIX field, or if a rolling period is configured and no order updates have occurred within this time, or if no orders were processed at all. See the order column description for a brief summary on this behaviour.

Mandatory: Yes (a valid order-column type must be configured).

views > user > data > columns > column > columnType > order > type > average

The average column displays the average of the FIX field or named property values extracted for orders, subject to the filters and order match states selected. Non-numeric values processed by this column will be converted to a number if they begin with a numeric prefix, otherwise they will be treated as zero.

A column of this type could be configured to display the average acknowledgement-latency for all orders of a dataview row.

Average columns will display an empty value if there are no matching orders or values. This may occur if messages for matching orders do not contain the configured FIX field, or if a rolling period is configured and no order updates have occurred within this time, or if no orders were processed at all. See the order column description for a brief summary on this behaviour.

Mandatory: Yes (a valid order-column type must be configured).

views > user > data > columns > column > columnType > order > type > stdDev

The standard deviation column computes the deviation of all FIX field or named property values extracted for applicable orders, subject to the filters and order match states selected. Non-numeric values processed by this column will be converted to a number if they begin with a numeric prefix, otherwise they will be treated as zero.

A column of this type could be configured to display the standard deviation acknowledgement-latency for all orders of a dataview row.

Standard deviation columns will display an empty value if there are no matching orders or values. This may occur if messages for matching orders do not contain the configured FIX field, or if a rolling period is configured and no order updates have occurred within this time, or if no orders were processed at all. See the order column description for a brief summary on this behaviour.

Mandatory: Yes (a valid order-column type must be configured).

views > user > data > columns > column > columnType > order > fields > fieldId

The fieldId setting specifies a FIX field id, which will be used to extract values from FIX messages for use by an order column.

Mandatory: Yes - one of fieldId or named must be specified.

views > user > data > columns > column > columnType > order > fields > named

The named setting specifies a property which will be extracted from an order for use by an order column. Available properties are as follows:

State Description
ackLatency
The acknowledgement latency in milliseconds. The time taken for an order to go from the "unacknowledged" state to the "new" state.
E.g. The time between a 35=D FIX (single order) message and the first 35=8 (execution) reply message.
fillLatency The fill latency in milliseconds. The time taken for an order to go from the unacknowledged state to a finished state.
Status The current order status, which will be checked against the match states configured for each order column.
Tracking
The current tracking status, used internally by Fix-analyzer.
One of New, Acknowledged or Finished.

For more detail on the order status, see the order tracking in the Message Processing section.

Mandatory: Yes - one of fieldId or named must be specified.

Custom Order Report output

views > user > data > reportColumns

The report columns setting specifies the set of columns to display in the "Show Orders" command output when run for this dataview. If no report columns have been specified for a view, the default set of columns is used:

Title Configuration Description
Symbol fieldId = 55 The "human representation" of a security name.
OrderQty fieldId = 38 The order quantity.
CumQty fieldId = 14 The total (cumulative) quantity of shares filled.
Status named = "Status" The current order status.
ackLatency (ms) named = "ackLatency" The acknowledgement latency time (in ms).

Mandatory: No

Message Detector view

views > messageDetector

Message detector views extract and display values from "interesting" FIX messages. They do this by dynamically adding new rows to the view when a message is detected which matches a configured filter. Each row in the view represents a message which passed the filter.

The values displayed for the rows are determined by the column configuration as detailed below. The message detector view can be configured either by specifying the settings directly in the plug-in configuration, or by specifying a message detector template as a static variable. If specified as a static variable, the view setup can be shared by a number of plug-in instances.

Mandatory: No

views > messageDetector > fix-analyzer-message-detector-viewTemplate

This setting references a message detector view from a static variable.

Mandatory: No
See the Gateway 2 Reference Guide for more information on static variables.

views > messageDetector > data

This setting specifies a message detector configuration directly in the plug-in setup.

Mandatory: No

views > messageDetector > data > viewName

Specifies the name of the message detector view. View names must be unique across all views for a sampler, so that each view can be uniquely referenced.

Mandatory: Yes

views > messageDetector > data > filter

The filter setting specifies the filter used to determine "interesting" messages. FIX messages which pass the filter will be added to the message detector view as a new row.

The filter is referenced by name from the filters available to the plug-in. To filter on multiple "interesting" message types, combine multiple filters using a Boolean filter.

Mandatory: Yes

views > messageDetector > data > messageTimeout

The message timeout setting specifies the maximum age a message can be (in seconds) before it is removed from the message detector view.

If the difference between the current time (obtained from the local time of the Netprobe host) and the message timestamp is larger than this threshold, then the row representing this message is removed from the view.

Mandatory: No
Default: 0 seconds - no timeout

views > messageDetector > data > messageCount

The message count setting controls the maximum number of messages which will be displayed in a message group. If the count it set to 1, then the group summary row in the dataview will be omitted.

A message group is a set of messages which share the same unique id. If the maximum limit is reached, a new incoming message will replace the oldest message in the group.

Mandatory: No
Default: No per group maximum

Row configuration

views > messageDetector > data > uniqueId

The uniqueId setting specifies how to construct a unique row identifier (row name) from a FIX message for display in the message detector view.

The uniqueId is made up of one or more id elements, each of which can extract a FIX field or named property from the message. If using more than one id, the field values will be compounded together separated by dashes (the - character) to form the uniqueId.

If several messages are found with the same id, the message detector view will display these all as sub-rows under a group row named with the id. This set of messages is then called a message group. The maximum number of messages per group can be configured using the messageCount setting.

Mandatory: Yes

views > messageDetector > data > uniqueId > id > fieldId

The fieldId setting specifies which FIX field to extract from a message. This value will then be used to form the uniqueId (the row name) in the message detector view.

Mandatory: Yes - one of fieldId, named or autoGenerated must be specified.

views > messageDetector > data > uniqueId > id > named

The named setting specifies a named FIX message property to extract. This value will then be used to form the uniqueId (the row name) in the message detector view.

Available properties are as follows:

Property Description
TimeStamp
The timestamp of the message formatted as follows:
Mon Sep 06 11:42:37.084 2010
SecondsSince

The number of seconds elapsed since the message timestamp.

Note: This value is only computed at the point when the row is created. When used in a row name this property will not update to reflect time changes dynamically.


Mandatory: Yes - one of fieldId, named or autoGenerated must be specified.

views > messageDetector > data > uniqueId > id > autoGenerated

The autoGenerated setting generates a series of incrementing numbers starting from 1. These values will then be used to form the uniqueId (the row name) in the message detector view.

Users should use this setting if they are unsure whether the FIX fields they have specified as identifiers will be unique. If using only autoGenerated to create a unique id, it is recommended to disable the message group feature via the messageCount setting.

Mandatory: Yes - one of fieldId, named or autoGenerated must be specified.

Column configuration

views > messageDetector > data > columns

This setting specifies the columns which will be displayed in this message analyzer view. For each message which passes the filter, field values from the FIX message are extracted as per the column configuration and displayed as a new dataview row.

Mandatory: Yes

views > messageDetector > data > columns > column > displayName

Specifies the name of this column as it will appear in the dataview. Column names must be unique amongst all other columns configured for this message detector view.

Mandatory: Yes

views > messageDetector > data > columns > column > value

This setting specifies the source of the column value.

If a FIX field id is specified, then the value will be extracted from the FIX message. If the specified field does not exist in the message an empty value will be displayed.

Users may alternatively specify a FIX message property rather than a field value. The available properties are as follows:

Property Description
TimeStamp
The timestamp of the message formatted as follows:
Mon Sep 06 11:42:37.084 2010
SecondsSince

The number of seconds elapsed since the message timestamp.

Note

This value is only computed at the point when the row is created. When used in a row name this property will not update to reflect time changes dynamically.


Mandatory: Yes

Custom Order Report output

views > messageDetector > data > reportColumns

The output of the "Show Orders Report" command can be customised for message detector views, in the same way as for user view. Please see the user view custom order report section in the Technical Reference guide.

Session Analysis view

views > sessionAnalysis

The session analysis view displays the current status for the conversations on a monitored FIX engine. This view is useful for ensuring that specific connections remain up throughout the trading day. For more information, please refer to the section on session state tracking.

Each Fix-analyzer 2 plug-in can output a single session analysis view, which is created when this setting is enabled.

Expected FIX conversations can be configured using the conversations setting. This also allows other properties such as an alias to be specified. A start and end time can be specified for a conversation during which time the session is expected to be up.

The session analysis view can be configured either by specifying configuration directly in the plug-in configuration or by specifying a session analysis view template static variable. If specified as a static variable, the view setup can be shared by a number of plug-in instances.

Mandatory: No
Default: No session analysis view is created.

views > sessionAnalysis > fix-analyzer-session-analysis-viewTemplate

This setting references a session analysis view from a static variable.

Mandatory: No
See the Gateway 2 Reference Guide for more information on static variables.

views > sessionAnalysis > data

This setting specifies a session analysis view configuration directly in the plug-in setup.

Mandatory: No

views > sessionAnalysis > data > viewName

Specifies the name of the session analysis view. View names must be unique across all views for a sampler, so that each view can be uniquely referenced.

Mandatory: Yes

views > sessionAnalysis > data > showSessionStatus

The sessionTimeout setting specifies a default timeout period for a FIX session conversation. This period will be overridden by the timeout (if specified) of the corresponding named conversation if available.

Combined with the timeoutDelay setting, this defines a default maximum period of (sessionTimeout+timeoutDelay) seconds of inactivity for a session. If no messages are sent or received for this period of time, the session is considered to be timed out. This condition is displayed in the logoff column.

Timeouts are calculated by comparing the last message timestamp for that session against the local time of the Netprobe host.

The expected use of the timeoutDelay setting is to cater for any delays between FIX messages being written to log files and being fully processed by FIX Analyser 2. Additional time can be granted to prevent the session being incorrectly shown as timed out; typically 1 or 2 seconds should be enough for normal message processing rates.

Mandatory: No
Defaults: 0 seconds session timeout (no timeout), 0 seconds timeout delay (no delay).

Named FIX conversations

conversations

The conversations section allows users to define a set of named FIX conversations between known parties. This list is used both in the session analysis view and in user views where the showSessionStatus setting has been enabled.

The named conversations are then always displayed in those views, which can be used to detect unseen conversations which are down (without configuring sampler expect-rows).

Conversation entries can also be used to add more readable names, as well as configure a specific timeout time or a pair of times between which the conversation should be up to assist with Gateway rules and alerts.

Mandatory: No

conversations > configFile

This setting specifies that the list of conversations is defined in an external file. At present this file is expected to have the same format as the Appia FIX engine configuration file (named appia.ini by default). The Netprobe must be able to read this file in order to load the conversation settings.

Conversations are detected within the appa.ini file by looking for connection configuration details. The keys local_firm_id and remote_firm_id are used as the SenderCompId and TargetCompId settings for the conversation configuration.The heartbeat_interval key becomes the timeout.

The trading periods for the session are detected by resolving the trading_session key value, and extracting the trading_session_dow, trading_session_start_time and trading_session_end_time values. The comments setting is populated with the text from that section of the ini file in case any other settings are useful.

The INI-file is only read once at when FIX Analyser 2 is started, therefore a Netprobe restart or plug-in setup change is required to pick up any changes made to this configuration file.

An example except from an INI file is shown below. The highlighted text indicates fields used by FIX Analyser 2 when parsing the file.

[MAIN]
connection = [BANK-EXCHANGE]
trading_session = [NORMAL_HOURS]
[BANK-EXCHANGE]
connection_type = FIX_44
contact = 24 hour support contact number +44 (0)20 xxx yyyy
description = Live Connection from Bank to Exchange
heartbeat_interval = 30
local_firm_id = BANK
remote_firm_id = EXCHANGE
trading_session = [NORMAL_HOURS]
[NORMAL_HOURS]
trading_session_auto_connect = OFF
trading_session_auto_disconnect = OFF
trading_session_auto_eod = ON
trading_session_auto_retry = OFF
trading_session_dow = MONDAY
trading_session_dow = TUESDAY
trading_session_dow = WEDNESDAY
trading_session_dow = THURSDAY
trading_session_dow = FRIDAY
trading_session_end_time = 21:10:00
trading_session_start_time = 21:40:00

Mandatory: Yes - one of conversations or configFile must be specified.

conversations > conversations

This setting specifies the list of conversations as entries defined inside the plug-in configuration.

Mandatory: Yes - one of conversations or configFile should be specified.

conversations > conversations > conversation > compIdA

Specifies a host for one side of a FIX conversation. The pair of hosts specified should uniquely identify the conversation for which the other settings (e.g. the timeout) should be applied. compIdB must also be specified.

Mandatory: Yes

conversations > conversations > conversation > compIdB

Specifies a host for one side of a FIX conversation. The pair of hosts specified should uniquely identify the conversation for which the other settings (e.g. the timeout) should be applied. compldA must also be specified.

Mandatory: Yes

conversations > conversations > conversation > timeout

This setting specifies the timeout interval (in seconds) for this particular FIX conversation. If a value of 0 is specified this session will not time out.

If this setting is not specified, then the default value is used instead. For a session analysis view the default is specified by the sessionTimeout setting. For a user view please see the showSessionStatus sessionTimeout setting in the Technical Reference guide.

Mandatory: No
Default: Uses sessionTimeout setting applicable to the view type.

conversations > conversations > conversation > alias

Specifies an alternate name (an alias) for the conversation. This name will be displayed in the dataview instead of the sender and target ids.

Mandatory: No
Default: If no alias is specified the format "Sender: Target" is used to generate the row name.

conversations > conversations > conversation > comments

Allows additional details to be added for this conversation. These details can be displayed by issuing the "Show session info" context-menu command in the session analysis view.

Mandatory: No
Default: No comment

conversations > conversations > conversation > tradingSessions > tradingSession

Specifies a trading session for this conversation. Trading sessions are used to populate the expectedState session-analysis column. A conversation has an expected state of Up if the local time on the Netprobe host falls inside the times specified by a trading session.

The trading session must have a startTime and an endTime specified in the form HH:MM or HH:MM:SS using 24-hour clock. The end time may be before the start time.

A trading session definition must also specify the days the session is active for. At least one day must be specified.

Mandatory: No
Default: No trading session. The expectedState column will be empty for this conversation.

Advanced settings

filterPlugin

This deprecated setting controls the plug-in level message filtering feature. Users are recommended to switch to using the more powerful inputTextFilters replacement feature. Messages filtered at this level are not processed by any views configured in the plug-in, and are also not used for order tracking or database logging.

Messages must match all configured positive filters, and not match any configured negative filters to be processed.

Mandatory: No
Default: No filters - all messages are processed

filterPlugins > positiveFilter

If positive filters are configured, then messages must pass the positive filters before they are processed by the plug-in. Positive filters are specified as a plain string, which must then be present in any raw lines read in from the source file(s). Regex syntax is not supported here.

Mandatory: No

filterPlugins > negativeFilter

If negative filters are configured, then any message which matches a negative filter is discarded and not processed by the plug-in. Negative filters are specified as a plain string, which must then not be present in any raw lines read in from the source file(s). Regex syntax is not supported here.

Mandatory: No

inputTextFilters

The inputTextFilters setting provides global message filtering on all input data. These filters are applied before any FIX message parsing is performed, and so will prevent processing of the message by any configured views, order tracking or database logging.

Filters are configured as an ordered list of regular expressions. Each input message read by the plug-in is checked against these filters in turn. The input is compared against each regular expression in order, and if a match is found the message will be either included (processed by the plug-in) or excluded (ignored) as defined by the filter configuration.

If a message does not match the filter regular expression, the next filter in the list will be checked. If no filters in the list match a message then:

  • When only "include" filters are configured, the message is excluded (ignored),
  • Otherwise, the message is included (processed).

This list of filters compares similarly to a list of ACCEPT/REJECT rules of a firewall configuration.

The regex for a filter can be configured via a variable, instead of being directly configured in the sampler definition. If using variables then the referenced variable must be of "regex" type.

Filter regexes can be made case insensitive by enabling the optional "i" flag. By default they are case sensitive.

Filter regexes can also be optionally "inverted". This means that if a message does not match the pattern then it will be included or excluded according to the filter configuration. This setting should be used when you want to further filter any remaining messages that do not match the regex.

E.g. "Exclude messages not matching (i.e. invert) regex "8=FIX4.4" to only process FIX4.4 messages but still allow further filtering on these messages.

Mandatory: No
Default: No filters - all FIX messages are processed.

showOrders > reportURLPrefix

Report output generated by the "Show Orders Report" and "Show Messages" commands are published by FIX Analyser 2 as a link to an internally-generated HTML webpage.

The reportURLPrefix setting controls the hostname component of the link URLs. By default the link contains the hostname as obtained from the local machine. Users can use this setting to provide a different (or more specific) hostname to use, which may be required if the Netprobe is hosted in a different domain or DNS environment from users attempting to view the reports.

For example, to use the same hostname value as configured in the gateway setup file, configure variables of type "macro > netprobeHost" and "macro > netprobePort" on your managed entity. These values can then be referenced in the reportURLPrefix, e.g. "$(FIXHOST):$(FIXPORT)"

Mandatory: No
Default: Uses the Netprobe machine hostname and Netprobe listen port.

resetTime

The reset time setting controls the time of day at which FIX Analyser 2 will reset its internal state. When a reset occurs, all stored order data is cleared and all output views are blanked. If any named conversations have been configured, then session analysis views (or user views configured to show session data) will contain empty rows representing the sessions.

It is recommended that the reset time be set to a time of day where (ideally) no FIX sessions are operating or when no messages are sent. If possible, the reset time should also be different to the MySQL message database housekeeping time. The default housekeeping time is 00:15.

Mandatory: No
Default: 00:00 (midnight)

Database settings

connection

The connection section specifies the database connection details for the FIX Analyser 2 message database. The message database is required for the message query report, and the message-level drilldown in the orders report.

Mandatory: No

connection > enabled

This Boolean setting specifies whether message database logging is enabled. This flag allows the feature to be disabled without un-configuring the database details.

Mandatory: Yes

connection > database > mysql

This setting specifies the MySQL connection details.

Setting Description
Server name The (required) MySQL server hostname or IP address.
Port Optional MySQL server listen port, defaults to 3306.
Database name The (required) FIX Analyser 2 database instance name as hosted by the MySQL server.

Mandatory: Yes - if the database setting has been specified.

connection > database > username

The username to use when connecting to the database. This user must have the appropriate privileges to the configured FIX Analyser 2 database instance.

Mandatory: No
Default: No username

connection > database > password

An optional password to use when authenticating with the database server as the specified username.

Mandatory: No
Default: No password

connection > filter

Optional filter. If specified, only FIX messages that pass the filter will be logged to the MySQL message database.

Mandatory: No
Default: No filter

Local cache settings

localCache

The local cache section contains optional settings controlling the local order store, which FIX Analyser 2 uses to maintain intra-day order records. Currently the local cache is implemented by an on-disk SQLite database instance, as described in the Data persistence in the Message Processing section.

localCache > database > sqlite > file

Specifies a custom filename used to store the SQLite order database. The file will be created and initialised automatically if it does not exist. The file must remain accessible while FIX Analyser 2 is running.

Mandatory: No
Default: The default filename is of the form managedEntityName.samplerName.db. Characters in the entity or sampler names which cannot be used in a file path will be replaced by an underscore _ character.

localCache > orderCacheSize

The order cache size specifies how much memory space (in megabytes) to allocate for caching order records in FIX Analyser 2. This caching helps reduce how frequently data must be read or written to disk, thereby increasing performance.

Using the default of 5MB allows approximately 13k order records to be stored in memory, as each order record is just under 400 bytes in size.

Note: This figure relates only to order state, other information such as field values for reporting will increase the memory footprint of FIX Analyser 2.

Mandatory: No
Default: 5 (MB), the minimum allowable value. Maximum is 2048 MB.

localCache > writeBackCount

The write back count controls how many orders are written out to disk at a time. This typically occurs when an order needs to be placed in the cache and there is no free space available.

Larger values mean that more orders are written leaving a larger amount of free cache space and thus improving performance for subsequent insertion operations. However, the written-out orders may potentially be required at a later time and so will need to be cached-in again which would decrease performance.

Mandatory: No
Default: 10

Debug settings

debug

The settings in the debug section enable different log outputs in FIX Analyser 2, to assist in diagnosing any problems. The available settings are:

Setting Description
cacheStatistics Hit/miss statistics for the in-memory order cache are output to the log each sample.
inputFilter Filter matching debug is output to the log file.
inputFilterDisableImplicit Disables the implicit filter that excludes lines not matching "8=FIX". The implicit filter avoids attempting to parse lines that do not appear to contain a FIX message, which improves performance.
multilineMatching Enable multi-line file processing logging.
ooomTableTrace Enable logging for out-of-order message table events.
ooomTableDisplay Enable a dataview displaying the contents of the out-of-order message table.
orderDatabaseQuery Output SQL queries being made to the SQLite order database.

Mandatory: No (the debug section and each setting inside are optional)

Appendices

Appendix A - Time Format Codes

The following time formatting codes can be used with this plug-in. These codes work both for the time format (when reading a timestamp from a log file) and also when configuring a filename to read from, where the filename contains the current date.

Code Meaning
%a Abbreviated weekday name (e.g. Mon, Tue, Wed)
%A Full weekday name (e.g. Monday, Tuesday)
%b Abbreviated month name (e.g. Aug, Sep)
%B Full month name (e.g. August, September)
%c Date and time representation (locale dependant)
%d Day of the month (01-31)
%f Milliseconds. Reads a period (decimal point) followed by 0-3 digits. To read seconds and milliseconds (e.g. 08.410) use %S%f. If you do not want the period included in the code use %Of which reads just the digits. (e.g. 08:410 can be read by %S:%Of).
%g Microseconds. Reads a period (decimal point) followed by 0-6 digits. If you do not want the period included, use %Og.
%H Hour in 24h format (00-23)
%I Hour in 12h format (01-12)
%j Day of the year (001-366)
%m Month as a decimal number (01-12)
%M Minute (00-59)
%p AM or PM designation
%qd Day of the month (1-31) (no preceding 0 or space)
%qm Month as a decimal number (1-12) (no preceding 0 or space)
%S Second (00-61)
%U Week number with the first Sunday as the first day of week one (00-53)
%w Weekday as a decimal number with Sunday as 0 (0-6)
%W Week number with the first Monday as the first day of week one (00-53)
%x Date representation (local dependant)
%X Time representation (local dependant)
%y Year, last two digits (00-99)
%Y Year, last all digits (ie 2008)
%Z,%z Timezone name or abbreviation (ie CDT)
%% A % sign

Some examples:

Timestamp text Format code pattern
2010/04/0700:00:00:598 %Y/%m/%d%H:%M:%S:%Of
2010/04/2619:45:56.626 %Y/%m/%d%H:%M:%S%f or %Y/%m/%d%H:%M:%S.%Of
22-JUN-2010,16:16:17:790 %d-%b-%Y,%H:%M:%S:%Of
WedJul 2815:17:53:103 %a%b %d%H:%M:%S:%Of

Appendix B - MySQL message database schema

Version 1.2 schema

A file containing the latest MySQL message database schema is packaged with the FIX Analyser 2 Netprobe. The section below describes the latest schema version (1.2) at the time of writing for reference. See the database installation instructions for how to install a new database.

-- --------------------------------------------------------
-- Table structure for table `data`
--
CREATE TABLE IF NOT EXISTS `data` (
`TimeStamp` decimal(20,6) NOT NULL,
`FileId` int(10) unsigned NOT NULL,
`SenderCompID` varchar(128) COLLATE latin1_bin NOT NULL,
`ClientID` varchar(128) COLLATE latin1_bin DEFAULT NULL,
`TargetCompID` varchar(128) COLLATE latin1_bin NOT NULL,
`OnBehalfOfCompID` varchar(128) COLLATE latin1_bin DEFAULT NULL,
`MsgType` varchar(4) COLLATE latin1_bin NOT NULL,
`ClOrdID` varchar(256) COLLATE latin1_bin DEFAULT NULL,
`BuySideInst` varchar(128) COLLATE latin1_bin DEFAULT NULL,
`BuySideField` int(11) DEFAULT NULL,
`OrigClOrdID` varchar(256) COLLATE latin1_bin DEFAULT NULL,
`Symbol` varchar(128) COLLATE latin1_bin DEFAULT NULL,
`SecurityID` varchar(128) COLLATE latin1_bin DEFAULT NULL,
`ExecType` varchar(4) COLLATE latin1_bin DEFAULT NULL,
`OrdStatus` varchar(4) COLLATE latin1_bin DEFAULT NULL,
`OrderQty` float DEFAULT NULL,
`CumQty` float DEFAULT NULL,
`FixMessage` varchar(2048) COLLATE latin1_bin NOT NULL,
KEY `FileId` (`FileId`),
KEY `MsgType` (`MsgType`),
KEY `Order` (`ClOrdID`,`BuySideInst`,`BuySideField`),
KEY `OrigOrder` (`OrigClOrdID`)
	) ENGINE=MyISAM DEFAULT CHARSET=latin1 COLLATE=latin1_bin;
-- --------------------------------------------------------
-- Table structure for table `files`
--
CREATE TABLE IF NOT EXISTS `files` (
`FileId` int(11) unsigned NOT NULL AUTO_INCREMENT,
`Offset` bigint(20) NOT NULL,
`Date` date NOT NULL,
`File` varchar(1024) COLLATE latin1_bin NOT NULL,
`Host` varchar(255) COLLATE latin1_bin NOT NULL,
`UniqueFileID` bigint(20) DEFAULT NULL,
PRIMARY KEY (`FileId`)
) ENGINE=MyISAM  DEFAULT CHARSET=latin1 COLLATE=latin1_bin AUTO_INCREMENT=1;
-- --------------------------------------------------------
-- Table structure for table `version`
--
CREATE TABLE IF NOT EXISTS `version` (
`major` int(11) NOT NULL,
`minor` int(11) NOT NULL
) ENGINE=MyISAM DEFAULT CHARSET=latin1 COLLATE=latin1_bin;

INSERT INTO `version` (`major`, `minor`) VALUES
(1, 2);
-- --------------------------------------------------------
-- Data management procedure.
-4- Culls data older than the specified threshold.
delimiter $$
CREATE PROCEDURE `removeExpiredFiles`(IN `daysOld` int)
BEGIN
-- Remove data and files,
-- where file date is older then the threshold
DELETE FROM `files`, `data`
USING `files`
LEFT JOIN `data` ON `data`.`FileId` = `files`.`FileId`
WHERE TIMESTAMPDIFF(DAY, `files`.`Date`, NOW()) > `daysOld`;
END$

Upgrading the database schema

FIX Analyser 2 comes packaged with database schema upgrade files which can be used to upgrade a MySQL database instance to a later version while retaining contained data.

The current schema version is 1.2, which adds an automatic data housekeeping function to the previous 1.1 version. This function utilises the MySQL event scheduler feature, which is present in MySQL versions 5.1.6 and onwards. The event scheduler must be enabled (to persist with server restarts) using the same steps as described for MySQL server installation.

If a MySQL server upgrade is required, please see the appropriate instructions on the MySQL website: http://dev.mysql.com/doc/refman/5.1/en/upgrading.html.

To upgrade the schema follow the steps listed in the appropriate section below.

Attended Scripted Upgrade

A schema installation / upgrade script is provided in the resources/db-schema folder of the FIX Analyser 2 package. The script should be run from within this folder and will require database administrator (DBA) access to the database. If remote DBA access has been disabled, then the script will need to be executed on the database server host itself (or remote access granted).

To execute the script to upgrade an existing FIX Analyser database, run the following command:

cd resources/db-schema
./fix_analyzer_schema_install.sh -u

The script will prompt for the following information during the installation process:

  • Database administrator (DBA) username
    • This account will perform administrative functions on the database
  • DBA password
  • Hostname for the MySQL database server
  • Port for the MySQL database server
  • Name of the FIX Analyser 2 database to update
    • This database must already exist on the server
  • Time to run the daily data housekeeping
    • Default time is 00:15
  • Number of days to retain message data
    • Default is 3 days of data

Unattended Scripted Upgrade

The upgrade script described above can also be run in unattended (non-interactive) mode.

To do this, the requisite information the script requires must be provided as environment variables. The table below describes the environment variables the script expects when upgrading an existing database installation.

Environment Variable Description
ROOT_USER MySQL database administrator (DBA) account username
ROOT_PASS MySQL DBA account password
MYSQL_HOST MySQL server hostname or IP address
MYSQL_PORT MySQL server listen port
DB_NAME Name of the database to upgrade
HK_TIME The time of day to execute the daily housekeeping script, in the format HH:MM. This should be a time of day where (ideally) no FIX sessions are operating or when no messages are sent. The default time when running an attended installation is 00:15.
HK_DAYS Number of days of message data to retain in the database. Expired data will be cleared by the housekeeping function. The recommended number is 3.

Manual Upgrade

If you are unable or do not wish to install the database schema using the installation script, you can install the schema manually using the following steps. The SQL statements below should be executed as a database administrator (DBA) account on the MySQL database server.

  1. Connect to the MySQL database FIX_DB using the MySQL command-line client. See the reference (http://dev.mysql.com/doc/refman/5.1/en/mysql.html) for full details.
mysql -u dba_username -p dba_password [ -h db_host -P db_port ] FIX_DB
  1. Enable the MySQL event scheduler. Note, you should also enable the scheduler permanently as detailed in the server installation section of this document.
SET GLOBAL event_scheduler = ON;
SHOW PROCESSLIST;

The output of the SHOW PROCESSLIST command should return a row with the text event_scheduler if it was successful.

  1. Verify the current schema version.
SELECT * FROM version;
  1. Upgrade the database schema using the upgrade SQL files. To do this from within mysql use the command:
source sql_file

Or from the shell pipe the schema into the mysql command-line client as follows:

mysql connection_parameters FIX_DB < sql_file

The exact files to use will depend on the version of database being upgraded. For a version 1.0 schema first upgrade to version 1.1 using the SQL file:

mysql-fix-analyzer-v1.0-v1.1-schema-upgrade.sql

Then upgrade from version 1.1. to version 1.2 using the SQL file:

mysql-fix-analyzer-v1.1-v1.2-schema-upgrade.sql
  1. Schedule the housekeeping event. In the following SQL, HH:MM is the daily execution time of the event and DAYS the number of days of message data to retain.

    It is recommended that the housekeeping be run at a time of day where (ideally) no FIX sessions are operating or when no messages are sent. The installation script uses default values of 00:15 and 3 respectively.

CREATE EVENT FIX_DB_housekeeping
ON SCHEDULE
EVERY 1 DAY
STARTS DATE_ADD(CURRENT_DATE(), INTERVAL 'HH:MM' HOUR_MINUTE)
DO
CALL FIX_DB.removeExpiredFiles(DAYS);