====== Transactional RFC (tRFC) Monitoring ======
  Transactional RFC (tRFC) is the asynchronous, guaranteed-delivery RFC mechanism used by SAP NetWeaver
  to exchange IDocs and function calls between systems (transaction SM58).
  This monitor supervises the open tRFC entries of an ABAP system, in both directions, and raises alarms
  when entries fail or stay in process for too long.

===== What Can Be Monitored =====

You can detect and alert on:
  * tRFC entries in an error state (''CPICERR/CONFAIL'', ''SYSFAIL'', ''SYSLOAD'', ''RETRY/ARETRY/ANORETRY'')
  * tRFC entries stuck in process (recorded or sent but never confirmed)
  * The number of entries matching a filter set over a time window (aggregate mode)
  * Individual failed or aged entries, one alarm per entry (non-aggregate mode)
  * The volume of tRFC entries per state, direction and destination (metric)

Each surveillance row targets a subset of entries using filters on direction, destination, function/program, user and state.

===== Monitored Data =====

The monitor reads the tRFC state tables through ''RFC_READ_TABLE'':

^ Direction ^ SAP table ^ Content ^
| ''OUTBOUND'' | ''ARFCSSTATE'' | tRFC calls sent by the monitored system to a destination. |
| ''INBOUND'' | ''ARFCRSTATE'' | tRFC calls received by the monitored system. |

Only open entries are read (''ARFCRETURN'' is empty). The following fields are collected:

^ Field ^ Description ^
| ''ARFCDEST'' | RFC destination of the call. Used by the **Destination** filter. |
| ''ARFCSTATE'' | Current state of the entry (''SYSFAIL'', ''CPICERR'', ''RECORDED'', ...). Used by the **status** filter. |
| ''ARFCFNAM'' | Function module or program name. Used by the **Program name** filter. |
| ''ARFCUSER'' | User who created the entry. Used by the **User** filter. |
| ''ARFCDATUM'' + ''ARFCUZEIT'' | Date and time of the entry, converted using the time zone of the SAP system. Used for the time window and the age evaluation. |
| ''ARFCMSG'' | Error message of the entry. Included in per-entry alarms. |

The connector user needs the authorization to call ''RFC_READ_TABLE'' on both tables.

==== tRFC states ====

The **status** filter accepts the following values:

^ status ^ Matches ^
| ''ANY'' | Every entry, whatever its state. |
| ''IN_ERROR'' | Entries in any error state: ''CPICERR/CONFAIL'', ''SYSFAIL'', ''SYSLOAD'', ''RETRY/ARETRY/ANORETRY''. Unknown states are also treated as errors. |
| ''IN_PROCESS'' | Entries that are not in an error state: ''RECORDED/VBRECORD'', ''SENDED'', ''HOLDEXE'', ''HOLDDEL'', ''EXECUTED'', ''FINISHED'', ''CONFIRMD'', ''WCONFIRM''. |
| ''CPICERR/CONFAIL'' | Communication error or connection failure with the destination. |
| ''SYSFAIL'' | Error raised in the target system. |
| ''SYSLOAD'' | No resources available in the target system. |
| ''RETRY/ARETRY/ANORETRY'' | Entry waiting for an automatic retry, or for which retries are exhausted. |
| ''RECORDED/VBRECORD'' | Entry recorded but not executed yet. |
| ''SENDED'' | Entry sent, waiting for confirmation. |

===== How It Works =====

==== Collection window ====

At each execution the monitor computes, per direction, a **collection start date**:
  * The reference date is the previous successful execution of the monitor, limited to **60 minutes** in the past.
  * If some active rows define a **Period**, the largest Period of the direction is used instead, when it is longer.

The collection start date drives the query sent to SAP:
  * With **Use period for collection depth** enabled (default), only open entries whose ''ARFCDATUM'' is within the last [period + 24h] of the collection start date are read. The collection depth is therefore the Period (or the previous execution), extended by 24h in the SAP system time zone.
  * With **Use period for collection depth** disabled, **all open entries** of the state tables are read, whatever their age. This gives a complete view of stuck entries, at the cost of reading the full tables at each execution.

Every entry returned by SAP is kept for evaluation. Entries older than the row window are never counted for **Max Entries**, but they are evaluated by the **Max entry age** check.

==== Row evaluation ====

Rows are evaluated in the order of the surveillance table. For each active row:
  * Entries are filtered by **Direction**, **Destination**, **Program name**, **User** and **status**.
  * The row time window is ''now - Period'', extended back to the previous successful execution when that is earlier. With **Period** = ''0'' the window starts at the previous successful execution.
  * ''count'' is the number of matching entries whose timestamp is inside the row window.
  * ''old count'' is the number of matching collected entries whose age exceeds **Max entry age**, inside or outside the row window.
  * If the **Severity** of **Max entries** is ''Disabled'', the matching entries are removed from the list and the row raises nothing: the row acts as an **exclusion filter** for the following rows.
  * If **Exclusive** is enabled, the matching entries are removed from the list after evaluation so that the following rows do not evaluate them again.

===== Configuration Hints =====

  * Filters support ''*'' (all), a case-insensitive substring (''PRD'' matches ''PRD_CLNT100''), ''!'' for negation (''!TEST''), a comma for OR (''PRD,QAS'') and ''+'' for AND (''PRD+CLNT100'').
  * Unused thresholds must be set to ''0''.
  * **Max entries** is only used when **Aggregates** is enabled. In non-aggregate mode every matching entry of the window generates its own alarm.
  * **Max entry age** accepts a number with an optional unit: ''d''/''day''/''days'', ''h''/''hour''/''hours'', ''m''/''min''/''minute''/''minutes''. Units can be combined (''1h 30m''). Without unit the value is read as **hours** (''6'' means 6 hours), which keeps existing configurations valid.
  * To detect entries older than the previous execution with **Max entry age**, either set a **Period** at least as long as the threshold on the row, or disable **Use period for collection depth** so that every open entry is read (see [[#collection_window|Collection window]]).
  * In non-aggregate mode, entries older than **7 days** are reported as one grouped alarm per row instead of one alarm per entry, to avoid flooding the alarm console with historical entries.
  * Put exclusion rows (Severity ''Disabled'' or **Exclusive**) **before** the rows they must protect: rows are evaluated top to bottom.
  * **Auto clear** controls whether alarms are cleared automatically when the condition recovers.
  * Metrics are global to the monitor (**Send metrics**) and do not depend on the surveillance rows.

===== Configuration =====

==== Monitoring Parameters ====

^ Parameter ^ Type ^ Required ^ Default ^ Description ^
| Send metrics | Boolean | Yes | ''true'' | Emits the ''TRFC_ENTRIES_COUNT'' metric for every collected entry (see [[#generated_metrics|Generated Metrics]]). |
| Use period for collection depth | Boolean | Yes | ''true'' | If enabled, the ''RFC_READ_TABLE'' query only returns open entries whose ''ARFCDATUM'' is on or after the day of the collection start date. If disabled, all open entries are read whatever their age, so that **Max entry age** can detect entries stuck for days or weeks without a large **Period**. Reading the full tables can be slow on systems with many open entries. |

==== Surveillance Table ====

^ Field ^ Required ^ Default ^ Description ^
| Active | Yes | ''true'' | Enables/disables this surveillance row. |
| Period | Yes | ''0'' | Time window, in minutes, in which entries are counted for **Max entries**. ''0'' means "since the previous successful execution" (60 minutes at most). The largest Period of a direction also defines the collection start date (see [[#collection_window|Collection window]]). |
| Direction | Yes | ''OUTBOUND'' | ''OUTBOUND'' (table ''ARFCSSTATE'') or ''INBOUND'' (table ''ARFCRSTATE''). |
| Destination | Yes | ''*'' | Filter on the RFC destination (''ARFCDEST''). |
| Program name | Yes | ''*'' | Filter on the function module or program name (''ARFCFNAM''). |
| User | Yes | ''*'' | Filter on the user who created the entry (''ARFCUSER''). |
| status | Yes | ''IN_ERROR'' | State of the entries to evaluate (see [[#trfc_states|tRFC states]]). |
| Max entries | Yes | ''0'' | Number of matching entries in the window from which the aggregated alarm is raised (''count >= Max entries''). ''0'' disables this threshold. Ignored when **Aggregates** is disabled. |
| Severity (Max entries) | Yes | ''Major'' | Severity of the "entries count" alarm(s). ''Disabled'' turns the row into an exclusion filter: matching entries are removed and ignored by the following rows. |
| Aggregates | Yes | ''true'' | If enabled, one alarm per row carrying the number of entries. If disabled, one alarm per matching entry. |
| Max entry age (d/h/m) | Yes | ''0'' | Age threshold of the entries, as a number with an optional time unit (''2h'', ''90m'', ''1d 12h''). Without unit the value is in **hours**. Alarm when the age of an entry is greater than the threshold. Evaluated on every collected entry, inside or outside the row window. ''0'' disables this threshold. |
| Severity (Max entry age) | Yes | ''Major'' | Severity of the "entries older than" alarm(s). ''Clear'' or ''Disabled'' suppresses the per-entry age alarms. |
| Auto clear | Yes | ''true'' | Alarms generated by this row are clearable when the condition recovers. |
| Prefix | No | (empty) | Alarm tag and message prefix. ''%MSG%'' can be used to place the generated message inside a custom text (''my_prefix %MSG% my_suffix''). A value starting with ''!'' sets the tag without altering the message. |
| Exclusive | No | ''false'' | If enabled, entries matched by this row are removed from the list and are not evaluated by the following rows. |
| Volatile | No | ''false'' | Applies to the aggregated entries count alarm when **Period** > ''0'': every execution generates a new alarm instead of updating the previous one. |
| Alarm | No | ''true'' | Enables/disables alarm generation for this row. |
| Report | No | ''false'' | If enabled, the entry count and severity of this row are shown in the daily report. |

===== Alarm Conditions =====

For each active surveillance row with **Alarm** enabled:
  * Entries are matched by **Direction**, **Destination**, **Program name**, **User** and **status**.
  * **Entries count alarm (Aggregates enabled)**: triggered when **Max entries** > ''0'' and ''count >= Max entries''. One alarm per row, with the severity of **Max entries**.
  * **Per-entry alarm (Aggregates disabled)**: one alarm for every matching entry updated inside the row window, with the severity of **Max entries**. **Max entries** is not used.
  * **Entries age alarm (Aggregates enabled)**: triggered when **Max entry age** is not ''0'' and at least one matching entry is older than the threshold. One alarm per row carrying the number of aged entries, with the severity of **Max entry age**.
  * **Per-entry age alarm (Aggregates disabled)**: one alarm for every matching entry older than **Max entry age**, when the threshold is greater than ''0'' and the severity of **Max entry age** is above ''Clear''. Entries older than **7 days** are not reported one by one: they are grouped into a single alarm per row giving their number, with the severity of **Max entry age**.
  * Alarms are attached to the ''TRFC_OUT_ENTRIES_COUNT'' or ''TRFC_IN_ENTRIES_COUNT'' metric depending on the row **Direction**.
  * Aggregated alarms are identified per row; per-entry alarms are identified per direction, destination, program name and user.

===== Generated Metrics =====

When **Send metrics** is enabled, the monitor emits one time series for all collected entries, independently of the surveillance rows:

^ metricId ^ metricUnit ^ Tags ^ metricDescription ^
| ''TRFC_ENTRIES_COUNT'' | count | ''status'', ''direction'', ''destination'' | Number of open tRFC entries updated since the previous execution, aggregated on 5-minute slots per state, direction and destination. |

''TRFC_IN_ENTRIES_COUNT'' and ''TRFC_OUT_ENTRIES_COUNT'' (formerly ''TRFC_IN_ENTRIES'' and ''TRFC_OUT_ENTRIES'') are the metric identifiers carried by the alarms.

===== Generated Alarms =====

Alarm messages include the tRFC context, for example:
  * Entries count (aggregate): ''12 OUTBOUND TRFC entries with ERROR state (>=10) [Destination PRD_CLNT100] [Between 10:15 and 10:30 09/11/2026 CEST]''
  * Entries age (aggregate): ''3 INBOUND TRFC entries in process older than 2h''
  * Per-entry: ''OUTBOUND TRFC with SYSFAIL state : <error message> [Dest: PRD_CLNT100] [User:PIAPPLUSER] [Function/Prog: IDOC_INBOUND_ASYNCHRONOUS]''
  * Per-entry age: ''OUTBOUND TRFC in process since 1 day 5 hours 3 min [<error message>] [Dest: PRD_CLNT100] [User:PIAPPLUSER] [Function/Prog: IDOC_INBOUND_ASYNCHRONOUS]''
  * Grouped old entries (non-aggregate mode): ''5 OUTBOUND TRFC entries with ERROR state older than 7 days [Destination PRD_CLNT100]''

The state wording depends on the row **status**: nothing for ''ANY'', ''with ERROR state'' for ''IN_ERROR'', ''in process'' for ''IN_PROCESS'', and ''with <STATE> state'' for a specific state.

===== Examples =====

^ Active ^ Period ^ Direction ^ Destination ^ Program name ^ User ^ status ^ Max entries ^ Severity ^ Aggregates ^ Max entry age (d/h/m) ^ Severity ^ Auto clear ^ Prefix ^ Exclusive ^ Volatile ^ Alarm ^ Report ^
| true | 0 | OUTBOUND | ''SANDBOX'' | ''*'' | ''*'' | ANY | 0 | Disabled | true | 0 | Major | true | | false | false | true | false |
| true | 15 | OUTBOUND | ''*'' | ''*'' | ''*'' | IN_ERROR | 10 | Major | true | 0 | Major | true | | false | false | true | false |
| true | 1440 | INBOUND | ''*'' | ''*'' | ''*'' | IN_PROCESS | 0 | Major | true | ''2h'' | Warning | true | | false | false | true | true |
| true | 0 | OUTBOUND | ''PRD_CLNT100'' | ''*'' | ''*'' | SYSFAIL | 0 | Critical | false | ''1d'' | Major | true | ''SAP-PI %MSG%'' | false | false | true | false |

**Row 1 effect**: exclusion row. OUTBOUND entries whose destination contains ''SANDBOX'' are removed and ignored by the following rows. No alarm is raised.

**Row 2 effect**: sends one MAJOR alarm when 10 or more OUTBOUND entries in an error state were updated during the last 15 minutes. The alarm is cleared automatically when the count drops below 10.

**Row 3 effect**: INBOUND entries from the last 24 hours are collected. Sends one WARNING alarm giving the number of INBOUND entries still in process for more than 2 hours. **Max entries** is ''0'', so no count alarm is raised. The row is also shown in the daily report. With **Use period for collection depth** disabled, the same row with **Period** ''0'' would check every open INBOUND entry, whatever its age.

**Row 4 effect**: sends one CRITICAL alarm per SYSFAIL OUTBOUND entry to destination ''PRD_CLNT100'' updated since the previous execution, with the message prefixed by ''SAP-PI''. Entries in SYSFAIL for more than one day raise one MAJOR alarm each; those older than 7 days are grouped into a single MAJOR alarm giving their number.