Custom Script Monitoring Documentation - Updated

Oracle GoldenGate
Custom Script Based Monitoring
Documentation

Custom Script Monitoring Overview

Two files make up the custom script based monitoring solution.

One file is a text properties file which contains the monitoring settings/metric thresholds, and the other is the Perl script code that parses the properties file and performs all the process checks on the GoldenGate instance according to the settings/metrics defined in the properties file. 

The monitoring script (code) was developed in Perl to be compatible across all UNIX/Linux, Microsoft Windows and mainframe/mini (AS400) systems (customers wanted the same monitoring across all platforms and customizable for each instance in terms of metrics/thresholds.   As GoldenGate is a file system/OS centric replication solution, monitoring GoldenGate using a scripted solution involves issuing commands to the GGSCI utility and parsing that output to analyze status, when not using any Oracle products to monitor GoldenGate (Director, Grid Control OGG Plug-in or GoldenGate Standalone Monitor).

The only requirement for this script-based monitoring solution is to have the Perl interpreter installed on the local system running GoldenGate and Perl accessible (executable) to the user executing the monitoring.    The Perl installation does not require any special Perl modules to function;   the current design of the monitoring solution ensures the monitoring script is easily run in all versions of Perl and across all platforms.  Many Perl installations are part of software installs (OS or Oracle installs) and typically use older versions of Perl that do not have many add-on modules installed.  Therefore, this script has no references to any Perl add-on modules as to avoid the Perl version and module compatibility/version issues or requirements across all platforms.  This makes the Perl code about 10-12% longer, but guarantees a smooth rollout across many platforms. The script code will also localize the path structure for each platform.

Installation/configuration steps are described in detail in the next section, however, a quick overview is covered here.

The two files that make up the monitoring solution should be deployed to each Oracle GoldenGate implementation (server) in a directory created for holding the monitoring files. The files are:

·      “parms.ini”   - goes in $GGS_HOME/dirprm/

·      “gg_monitor.pl”   - goes in $GGS_HOME/scripts/

The Perl script can produce an output message file and a log file for debugging purposes if desired. The script must also have access to an Oracle_Home for the “ggsci” utility to function correctly. The “ggsci” utility requires access to the Oracle XDK libraries as part of the library loading mechanism for the platform.  The script uses “ggsci” commands to get process status and information at each invocation.

Each GoldenGate installation to be monitored has its own directory structure (path to the OGG binaries) and functions as a single GoldenGate implementation. The two script files needed could be centrally located and pushed to the individual server(s) running GoldenGate, as long as the files are present on the system at invocation time.  The properties file is a server/implementation specific file (it localizes the specific environment for this GoldenGate installation). It could also be centralized if the metrics do not change from one implementation to another (same file system paths and monitoring settings/metrics). If each implementation needs localized values, then copy the properties file to each system and customize on that system and do not overwrite.  The Perl code script could be centralized and deployed; the code is generic and functions based on settings in the properties file.

The monitoring script can be run in an endless loop (continuously), or set to run one time (one iteration) only. These are controlled by parameters in the $GGS_HOME/dirprm/parms.ini.  When configured as a single run, it is usually run from cron or some other monitoring client which executes the script at a regular interval and expects a return result or stdout message to parse.   When run with an endless loop, it can be run via nohup and detached from the shell.

The two files are listed again here:

1.      “parms.ini”

a.      Properties file that contains the parameters (properties) which are parsed at runtime to control the monitoring performed, the metrics to be evaluated and the related notification settings

2.      “gg_monitor.pl”

a.      File that contains the Perl code logic

b.     The code is completely variablized and all functions/checks/thresholds are controlled by values set in the “parms.ini” file

A unique feature of the monitoring solution is the customized metrics and customized process checks/exclusions that can be implemented. These are described below:

·         Global and/or per process metrics for DML lag and checkpoint delay/lag

o         LAG and CHECKPOINT have global metrics (which apply to all processes), or per process custom metrics that can be enabled which override the global metric for that specific process

§         This helps prevent monitoring “noise” with processes which might have unique situations at various times but do not indicate a problem with replication

·         Blackout listing

o        Enables you to exclude a process by name from all monitoring. Simply add the process name to the blackout list and it will be excluded. When done, remove the entry from the blackout list

·         Blackout for DML or Checkpoint lag times

o        Enables you to exclude a process by name from a particular LAG check. Simply add the process name to the blackout list for that check and it will be excluded. When done, remove the entry from the blackout list for that check

§         If the process needs a custom DML or lag check metric, use the customization for that process instead of the blackout feature

For example, you may set the global DML lag to 5 minutes for all processes. However on Friday night two processes experience lag up to 20 minutes (for a short period of time) due to heavy data maintenance activities being replicated (2 million deletes or updates).  In this case, you can set the global to 5 minutes and set custom per process DML lag to 20 minutes for these two processes only.  This enforces a DML lag check of 20 minutes for every check for those two processes only. All other processes will be evaluated at the 5 minute lag.

This monitoring solution is also easily deployed using any script scheduling/running mechanism. The script can be run manually from the command-line, from cron, or by another system-based scheduler such as a third party monitoring tool such as OpenView, Tivoli or Oracle Grid Control/OEM (as a metric extension) or a database scheduler that can execute a file system script.

The monitoring solution sends email (by default) for alert notifications and/or output to standard output or a file with alert details. This solution can also be easily integrated with other monitoring tools that offer command-line interfaces (API) or with SNMP (sending SNMP Traps). In this case, the script will call the third-party API to submit messages to the enterprise monitoring system using a command-line call or SNMP.  The only requirement is the Perl script is called and one argument must be passed to it (location of the OGG home for that server).  When running from a central location, many OGG installations use different paths, the argument passed to the script at invocation time allows the script to find the properties file and set up what is needed.

Installation/Configuration Steps

1)      On each server with GoldenGate to be monitored, create a “scripts” sub-directory in the $GGS_HOME,  and place the “gg_monitor.pl” file in the new “scripts” directory:

 $GGS_HOME is the base directory for the GoldenGate binaries, for example:

  $GGS_HOME/               /u01/app/oracle/product/ogg                      (OGG binaries are here, OGG base directory)
  $GGS_HOME/scripts   /u01/app/oracle/product/ogg/scripts         (script goes here)

Note:

The monitoring directory can be anything name you prefer under the $GGS_HOME, but its location must be identified in the properties file (parms.ini).

2)      Create another sub-directory under the $GGS_HOME/scripts directory called “monitor”

Note:
The “$GGS_HOME/scripts/monitor” sub-directory is a temporary working directory during script executions and the data location for any *.csv files created containing the replication lag data points collected and saved.  In a CRS/RAC cluster environment, this directory also contains the PID file for the monitoring script controlled/called by clusterware (ggmon resource).

The resulting directory structure should look like the following:

 $GGS_HOME/                                 /u01/app/oracle/product/ogg/                    (OGG binaries are here, base directory)
$GGS_HOME/dirprm                     /u01/app/oracle/product/ogg/dirprm/     (properties files goes here)
 $GGS_HOME/scripts                     /u01/app/ oracle/product/ogg/scripts/     (Perl script file goes here)
 $GGS_HOME/scripts/monitor  /u01/app/ oracle/product/ogg/scripts/monitor/

3)      Copy the properties file “parms.ini” file to the $GGS_HOME/dirprm directory
This is the central locations for all process parameter files.

$GGS_HOME/dirprm/

Note:
At runtime, the Perl script parses the “parms.ini” file from $GGS_HOME/dirprm directory to set the appropriate values for monitoring control and evaluations/metrics. The script uses variables internally for easy configuration without having to modify the Perl code itself. This script may be deployed and run by any means (manually, from cron, or any third-party schedulers such as OpenView, Tivoli,  OEM/Grid Control, etc) and portability between the different platforms/systems using Oracle GoldenGate is critical for a single monitoring solution. In addition, the properties file is needed to localize the script for that particular implementation/instance.  By setting properties to control and tune evaluation/metrics, the script does not need to change from one server to another.

4)      Edit the properties file (parms.ini) to set some required parameters for the “gg_monitor.pl” script.
Some changes are needed which reflect the local path information for each/this implementation:

GGS_HOME
ORACLE_HOME

Note:
ORACLE_HOME is needed by “ggsci” utility to load the Oracle XDK libraries from an Oracle home.

When running the script from cron or from CRS or any third-party scheduler, the environment will not have the environment set for the OGG and must be set for GGSCI to function correctly during script execution.
The script sets up the environment, but needs to know where GGS_HOME and ORACLE_HOME are located to properly setup the environment for that run.

5)      Modify the “parms.ini” file to setup/customize monitoring specific to the environment.

The “parms.ini” file is used to customize and tailor the monitoring script to the environment and its behavior, by specifying and adjusting variables for various environments and runtime checks. 

Monitoring can be tweaked to prevent excessive alerts from the process checks by using the global or custom DML lag/Checkpoint delay thresholds and controls what information is sent in alerts or stdout.
The monitoring script behavior is controlled by the properties in the “parms.ini”.

The following parameters must be set in the “parms.ini” file in $GGS_HOME/dirprm/:

a)      GGS_DEBUG

i)        Required

ii)       This sets the debug log level for the script. The log will produce a log file at each run if GGS_DEBUG is 2 or 3.  The log file produced is reused for each run, so tere will be no accumulation of log files to have to deal with. 

GGS_DEBUG = 0 turns off the logging completely.

(1)    To toggle the level of debug produced during script execution:

ggs_debug=”0|2|3”  (0=no debug information, 3=lots of detailed debugging information)

iii)     

b)      ggshome

i)        Required

ii)       This sets the path to the Oracle GoldenGate Home. The script needs to know where the OGG home is located, and this can be different for each system being monitored. 

c)       ORACLE_HOME

i)        Required

ii)       This sets the path to the Oracle home that will be used for “ggsci” utility. Ggsci requires an Oracle home when using Oracle databases.

d)      ggs_name

i)        Optional

ii)       This sets the log file name when running multiple instances of Oracle GoldenGate on the same system and using CRS to manage all OGG processes

e)      systemname

i)        Optional

ii)       This sets the subject line on all emails to a friend name for this instance of the monitoring script

iii)     You can use this to set a friendly name for this instance.

f)       ggs_mode

i)        Required

ii)       This sets the type of monitoring for this instance meaning to check only extract process , only replicat processes or both

(1)    GoldenGate instance type (SOURCE or TARGET or BOTH)

ggs_mode=BOTH

g)      process_error_status

i)        Required

ii)       This sets the process status values that will flag a process as an error if state matches

process_error_status=ABENDED|STOPPED

h)      alert_count

extract_count

replicat_count

i)        Required

ii)       This configures the process count checks.

(1)    Monitor number of processes, and number of each process type to monitor:

alert_count=Y           (N to not monitor process counts)

extract_count=<number of extract processes - extracts and pumps>

replicat_count=<number of replicat processes>

i)        metrictype

i)        Required

ii)       This sets the type of metric used for evaluations (seconds or minutes) for all lag or checkpoint delay calculations (to determine if metric thresholds are surpassed)

metrictype=seconds or minutes

metrictype=minutes

j)        lag_time

i)        Required

ii)       Global threshold for replication DML lag

lag_time=<number of seconds/minutes of DML replication lag allowed

k)      xlag_time

i)        Optional

ii)       Number of seconds of DML replication lag for a custom (per process lag check)

iii)     custom (per process)  thresholds for replication DML lag and checkpoint latency:

iv)     To add multiple processes with custom thresholds for each process, separate with a space:

xlag_time="RPTST:66 RPXX2A:11 PMP02A:20"

l)        xchk_time

i)        Required

ii)       Global thresholds for replication checkpoint latency of a process

chk_time=<number of seconds or minutes of Checkpoint latency for a process allowed

m)    xchk_time

i)        Optional

ii)       Number of seconds of DML replication lag

iii)     custom (per process)  thresholds for replication DML lag and checkpoint latency:

iv)     To add multiple processes with custom thresholds for each process, separate with a space:

xchk_time="RPTST:33 RPXX1A:44 EXT01A:10"

n)   Blackout lists (to exclude processes from process/lag checks)

blackout="none"

blackout="RPAPX6A RPBPX1A"

# custom lag/chk blackout of processes

blackoutlag="RPAPX6A|RPBPX1A"

o)      Monitoring data table inserts:

i)        Optional

ii)       This is not used at this time, used to insert check records into a table in the target or source

insertdata=<Y|N>

truncatemonitortable=<Y|N>

monitoring_tbl="<table name to insert monitoring data>"  

p)   writedatapoints

i)   Required

ii)           This instructs the script to write out the lag/chk data points to a CSV file (one file per process) for graphing

iii)      Not used at this time

q)   parse_error_log

i)   Required

ii)           This configures the ggserr.log parsing on each run

iii)      Scripts keeps track of byte size of log and reads from last byte offset from previous run

parse_error_log=Y

r)       Outputmessage

i)        Required

ii)       output status information to stdout for integration with command-line tools or APIs for third party monitoring tools

outputmessage=Y

s)       Print_dashboard

i)        Required

ii)       This has the script print the informational dashboard, print the information collected for the run (ggsci info all output)

 

print_dashboard=Y

t)   notifyflag

i)   Required

ii)           Notify when an alert is found

notifyflag=N

u)   notifyonlyonerrors

i)   Required

ii)           This controls whether the output includes “STATUS = OK” on each run if no errors are found.

(1)                       Some monitoring tools expect an output status flag on each run

(2)                       This can be customized to format the output however needed

v)      fromuser

i)        Required

ii)       This sets the From email address for all emails sent by the script

fromuser=oracle@system_name_you_choose.com

w)    mailuser

i)        Required

x)      This sets the email address list for notification via email (separate with spaces)

mailuser=<email address #1>  <email address #2>

y)      Sleeptime and loopinterval

i)        # Loop control when for nohup   # (0 = one time run, 1= for endless loop, and set sleeptime for looping)

ii)       loopinterval is a switch/flag to run in endless loop or single run

iii)     sleeptime is in seconds

sleeptime=600

loopinterval=1

6)      To run the script, call it with a single parameter that is the path to the OGG home for this specific instance:

a)      # perl gg_monitor.pl /u01/app/oracle/product/ogg
--or--

b)      # perl gg_monitor.pl C:\oracle\app\oracle\product\ogg

Note:
Every GoldenGate installation to be monitored may be located at a different system path on each server. For GGSCI to function correctly, the local path information must be known.  The script will use this path to locate the properties file for this server.

7)      For non-RAC implementations, schedule the script to run from “cron” or run from any type of scheduler available on the system. Script may be run manually as well at any time.
   sh ./gg_monitor.pl

For a RAC clustered system with HA, configure the monitoring in the cluster software as a application script.  For Oracle Clusterware, the monitor script can be setup as a CRS resource (ggmon) and execution/failover is controlled by the Clusterware properties on the “ggmon” resource itself.  See the install document for the varaibles set during CRS resource creation.

8)      (Optional) Create a symbolic in the “scripts” directory for easy viewing of the “parms” file
ln –s $GGS_HOME/dirprm/parms $GGS_HOME/scripts/parms

Description of Monitoring files


Sample parms.ini:

This file contains all the properties (parameters) used to configure the properties/variables for runtime execution and process checks for a real environment. These properties are parsed on each run as the script is not persistent from run to run.

The properties (variables) can be adjusted for each environment to customize for that environment:

# Sets the debug mode for debugging statements in the log file produced
# GGS_DEBUG=0 (no logging, only status info returned), GGS_DEBUG=2 (full detail logging)
GGS_DEBUG=3

# Path to ggshome
ggshome=/u03/ogg12c

# Path to Oracle Home (GGSCI needs Oracle libraries to run) - in restricted environments ORACLE_HOME is not set
ORACLE_HOME=/u01/app/oracle/product/12.1.0/dbhome_3

# Name of the log file prefix in CRS
ggs_name=gg_mon

# Set friendly system name
systemname=oel - Badass

# OGG instance type (BOTH, SOURCE, TARGET)
# allows you to check only EXTRACT or only REPLICAT processes or BOTH)
ggs_mode="BOTH"

# Process status that triggers error status
process_error_status=ABENDED|STOPPED

# Enable a process count check
# Alert (messages if counts do not match configured counts in properties file)

alert_counts=N
extract_count=6
replicat_count=1

# Set metric for all lag checks (minutes or seconds)
metrictype=seconds

# Global DML lag and custom lag (global or custom, lag_time is global - for all processes)
# Processes with custom lag will use custom for check
lag_time=15
xlag_time="RPTST:66 EXTEST2:91 ODIT1P:92"

# Checkpoint delay (global or custom, chk_time is global - for all processes)
# Processes with custom lag will use custom for check

chk_time=15
xchk_time="RPTST:33 EXTEST2:63 PMTEST:44 ODIT1P:98 EXRTEST:99 PMRTEST:8"

# Set a blackout list - excludes processes(s) from all checks - use to reduce noise during long maintenance periods for certain processes
# Set a blackout list to "none" or to the name of OGG processes, separate with a space 'EXTR1 EXTR2 PEXT1'

# process check blackout list (excludes a process from all checks)
#blackout="none"
#blackout="RPXXX6A EXXXST2 RPRTEST ODIT*"
blackout="RPXXX6A EXXXST2 PMTES* RPRTEST"

# DML lag blackout list (by process) - (just for DML lag checks)
#blackoutlag="RPBXX1A EXXXXT2 ODIT1P"
blackoutlag="RPBXX1A EXXXXT2"

# CHK lag blackout list (by process) - (just for CHK lag checks)
#blackoutchk="RPAXXXA EXXXXT2 ODIT1P"
blackoutchk="RPAXXXA EXXXXT2"

#  Insert monitor info and lag data points to a table
insertdata=N
truncatemonitortable=N
monitoring_tbl=wowdba.monitor_ggs

# Output CSV data points per run
writedatapoints=Y

# Parse "ggserr.log" looking for errors (Y or N)
parse_error_log=Y

# COntrols whether script outputs status to STDOUT (will return OK or ERROR messages)
outputmessage=Y

# Print info dashboard to stdout
print_dashboard=Y

# Notification flags (for email, SNMP, CLI integration)
notifyflag=Y
notifyonlyonerrors=N

# Email notification (list of email addresses, separate with spaces)
fromuser=oracle@badlands.com

#Mailuser=nobody
mailuser=gthomas@enkitec.com gnt.architect@gmail.com

gg_monitor.pl:

This script monitors the OGG processes (Manager, Extract and Replicat processes) and will send a notification if any process violates a configuration setting/threshold for lag or process status.   Two variables are needed by this script to function correctly: .  $GGS_HOME and $ORACLE_HOME. These two variables allow the “ggsci” utility to run to get process status when running in a restricted shell such as cron or CRS.

The types of checks are as follows:

·     
   Manager process is running (OS process and “RUNNING” status in GGSCI

·         GoldenGate process are running (status is not ABENDED or STOPPED)

·         Lag in the replication or lag since last checkpoint (checkpoint latency)

o        Global metric (for all processes)

o        Per Process Custom metric

·         Blackout list for process checks and lag checks

Note:
If any process name is listed in the “blackout” list in the “parms” file, those processes will be excluded from monitoring until they are removed from the blackout list.  Once added to the blackout list, the next run will consider the blackout list changes.

·         Blackout list for DML lag checks (for long running processes which lag is encountered by alerting is not required)

·         Blackout list for CHK delay (lag)  (for long running processes which lag is encountered by alerting is not required)

·         Notify or not

·         Output status information to stdout or to a message file

·         Email list for alerts, can be customized to send SNMP alerts or integrated with command-line API for third-party monitoring tools.

Note:
This script sends an email to an email list when an event such as an ABEND is detected, or the Manager is not running for the instance; it may also be modified to integrate in with SNMP or other monitoring command-line API agents on the server.

Run via cron, enterprise scheduler or via Oracle Clusterware.

Oracle Clusterware Resources

When run by Clusterware, this script must be copied to the GRID_HOME/crs/script or when creating the resource specify the location on a shared file system and copy the script to that location.
Below is an example of the monitoring running as a CRS resource:

---------------------------------------------
Cluster Resources
---------------------------------------------
gg_vip
      1        ONLINE  ONLINE       maexdb01
ggapp
      1        ONLINE  ONLINE       maexdb01
ggmon
      1        ONLINE  ONLINE       maexdb01

As mentioned earlier, the “gg_monitor.pl” script requires a few paths defined in its properties file; these are used to establish the path information of this system and this paltform. Define the following for each implementation:

   GGSHOME=
   ORACLE_HOME=

These variables are needed anytime the script is run through cron, via clusterware from a rsh or su shell with a restricted environment or any third-party scheduler where the path information may not be set in the shell or command-line environment.

Sample Output #1:

This example shows the output with most output message types enabled.

oel1:/u03/ogg_src/scripts ogg$ perl /u03/ogg_src/scripts/gg_monitor.pl /u03/ogg_src

  INFO

      Server: oel1.badlands.com

 Date: Mon Sep 22 08:57:54 2014

      ----------------------------------

  ERROR EXTRACT process (EXRTEST) has status 'ABENDED'
  ERROR EXTRACT process (EXRTEST) with status (ABENDED) has exceeded CHKP delay (custom / in seconds): 578636 (160:43:56) > 99)

  ERROR EXTRACT process (EXTEST) has status 'ABENDED'
  ERROR EXTRACT process (EXTEST) with status (ABENDED) has exceeded CHKP delay (global / in seconds): 578633 (160:43:53) > 8)

  ERROR EXTRACT process (ODISC) has status 'ABENDED'
  ERROR EXTRACT process (ODISC) with status (ABENDED) has exceeded CHKP delay (global / in seconds): 578630 (160:43:50) > 8)

  ERROR EXTRACT process (ODIT1P) has status 'ABENDED'
  ERROR EXTRACT process (ODIT1P) with status (ABENDED) has exceeded CHKP delay (custom / in seconds): 578588 (160:43:08) > 98)

  ERROR EXTRACT process (PMRTEST) has status 'ABENDED'
  ERROR EXTRACT process (PMRTEST) with status (ABENDED) has exceeded CHKP delay (custom / in seconds): 578589 (160:43:09) > 8)

  ERROR REPLICAT process (RPTEST) has status 'ABENDED'
  ERROR REPLICAT process (RPTEST) with status (ABENDED) has exceeded CHKP delay (global / in seconds): 578630 (160:43:50) > 8)

   ------------ ggserr.log ERRORS ---------

  ------------- Information Dashboard -------------

   INFO Server: oel1.badlands.com

 Date: Mon Sep 22 08:57:54 2014

   Process List from Server

MANAGER     STOPPED
EXTRACT     ABENDED     EXRTEST     00:00:00      160:43:56
EXTRACT     ABENDED     EXTEST      00:00:00      160:43:53
EXTRACT     ABENDED     ODISC       00:00:00      160:43:50
EXTRACT     ABENDED     ODIT1P      00:00:00      160:43:08
EXTRACT     ABENDED     PMRTEST     00:00:00      160:43:09
EXTRACT     ABENDED     PMTEST      00:00:00      160:43:03
REPLICAT    ABENDED     RPRTEST     00:00:00      160:43:59
REPLICAT    ABENDED     RPTEST      00:00:00      160:43:50

Sample Output #2:

This example shows the output with most output message types disabled except for any error messages.
This example has a process that has exceeded the checkpoint delay threshold of 8 seconds:

oel1:/u03/ogg_src/scripts ogg$ perl /u03/ogg_src/scripts/gg_monitor.pl /u03/ogg_src

  INFO

      Server: oel1.badlands.com

 Date: Mon Sep 22 09:06:30 2014

      ----------------------------------

  ERROR REPLICAT process (RPTEST) with status (RUNNING) has exceeded CHKP delay (global / in seconds): 9 (00:00:09) > 8)

Sample Output #3:

Example of a check with no issues. This is an example of when the script can output a message

oel1:/u03/ogg_src/scripts ogg$ perl /u03/ogg_src/scripts/gg_monitor.pl /u03/ogg_src

STATUS->OK

Sample Output #4:

An example with no issues, but the process dashboard is printed in the output.

oel1:/u03/ogg_src/scripts ogg$ perl /u03/ogg_src/scripts/gg_monitor.pl /u03/ogg_src

STATUS->OK

  ------------- Information Dashboard -------------

   INFO Server: oel1.badass.com

 Date: Mon Sep 22 09:36:22 2014

   Process List from Server

MANAGER     RUNNING

EXTRACT     RUNNING     EXRTEST     00:00:00      00:00:03
EXTRACT     RUNNING     EXTEST      00:00:00      00:00:02
EXTRACT     RUNNING     ODISC       00:00:00      00:00:05
EXTRACT     RUNNING     ODIT1P      00:00:00      00:00:00
EXTRACT     RUNNING     PMRTEST     00:00:00      00:00:08
EXTRACT     RUNNING     PMTEST      00:00:00      00:00:01
REPLICAT    RUNNING     RPRTEST     00:00:00      00:00:05
REPLICAT    RUNNING     RPTEST      00:00:00      00:00:00