:EP: 31
:Title: Enabling Telemetry for EVCs provisioned with Kytos-ng mef-eline
:Authors:
    - Jeronimo Bezerra <jbezerra AT fiu DOT edu>
    - Italo Valcy <idasilva AT fiu DOT edu>
    - Vinicius Arcanjo <vindasil AT fiu DOT edu>
:Created: 2022-08-24
:Updated: 2024-04-11
:Kytos-Version: 2024.1
:Status: Accepted

****************************************
EP031 - Telemetry-enabled EVCs
****************************************


Abstract
========

Blueprint 31 presents requirements and features to support In-band Network Telemetry (a.k.a. INT or telemetry) for EVCs created by the Kytos **mef-eline** napp.


Motivation
==========

INT enables per-packet monitoring and it is specified by the P4.org consortium as one of P4 use cases. To enable full network visibility, **all** EVCs must be enabled with INT. For the network devices available at AmLight, INT will be performed by the network data plane when instructed to do so via OpenFlow NoviFlow Experimenter's actions.

Supporting INT with the current implementation of the NoviFlow's NoviWare OS won't be trivial due to many conditions imposed by NoviWare. This blueprint will focus on enabling a MINIMALLY functional support for INT over Kytos. Once this blueprint is implemented, new blueprints will be created addressing more complex requirements.


I. Requirements
===============

This blueprint has the following characteristics:

  1. There will be no concerns about number or location of the INT Hops switches. Currently, an INT Sink switch can only export up to 10 metadata stacked by the network. If more than 10 switches add metadata, the INT Sink will export the LAST 10 metadata added to the packet. The other will be discarded. At AmLight, the longest route planned is currently composed of 9 switches.
  2. There will be no concerns about MTU restrictions. NoviWare and Tofino support frames as large as 10KB which is more than enough. However, any legacy device in the middle must be previously identified since legacy devices usually have a MTU of up to 9,216 Bytes. The INT Source Switch adds a header of 12 Bytes plus 24 Bytes of metadata. Each INT hop in the path will add 24 bytes only.
  3. There will be no concerns if INT Sink switches have loops for INT reports (Section III). This blueprint assumes physical loops are already deployed by the operators **during** the commissioning phase, which means, before the **telemetry_int** napp is deployed.
  4. There will be no concerns about proxy ports and their mappings. These were addressed in Blueprint EP034.
  5. There is no need for persistent data. The **mef_eline** and **flow_manager** napps will persist their entries accordingly since **telemetry_int** will leverage **flow_manager**.
  6. This version won't require changes to the way the **mef_eline** napp works. However, a new value will be added each EVC's metadata attribute.
  7. This specification assumes the data plane's pipeline is ready for INT, with multiple tables, and it assumes that **mef_eline** uses table 0 (but it will follow ``of_multi_table`` mef_eline table groups as they're set). **telemetry_int** aims to use any table with an ID higher than **mef_eline**, for instance in this document, table 2 and table 3, by default, for EVPLs and EPLs respectively.

II. How INT works with NoviWare
===============================

Currently, the **mef_eline** napp creates Ethernet Virtual Circuits (EVCs) using *matches* `IN_PORT` and `VLAN_VID` and actions `PUSH/POP VLAN`, `SET FIELD`, `SET_QUEUE`, and `OUTPUT`. All EVCs' flow entries have the same priority based on their type: Ethernet Private Line (EPL) or Ethernet Virtual Private Line (EVPL). Currently, INT is only supported for IPv4 traffic with TCP and UDP protocols, which means that more specific match entries will be needed to match IPv4 and TCP/UDP. Non-IPV4 or non-TCP/UDP traffic will continue using the existing flow entries created by **mef_eline**. The **telemetry_int** napp only adds and removes flows for INT. Any other flows created by the **mef_eline** won't be affected. Example:

- Current by **mef_eline**:

  - 1 flow per direction (ignoring failover protection. See section IX)
  - priority: DEFAULT FOR EVPL

    - match: <in_port=10,vlan_vid=20>

- New with **mef_eline** and **telemetry_int**:

  - 3 flows per direction
  - priority: Higher (mef_eline + 100) (created by **telemetry_int**)

    - match: <in_port=10,vlan_vid=20,eth_type=0x800,ip_proto=6> for TCP
  - priority: Higher (mef_eline + 100) (created by **telemetry_int**)

    - match: <in_port=10,vlan_vid=20,eth_type=0x800,ip_proto=17> for UDP
  - priority: DEFAULT (DEFAULT FOR EVPL) (originally created by **mef_eline**)

    - match: <in_port=10,vlan_vid=20> for everything else


This new approach requires 3x more flows to manage, so scalability and a new pipeline could become a concern in the future. However, those concerns are out of the scope of Blueprint 31.

Another change NoviWare requires to support INT is new OpenFlow actions. The Kytos **NoviFlow** napp already instantiates four new OpenFlow experimenter actions: `push_int`, `add_int_metadata`, `send_report`, and `pop_int`.  The IPv4+TCP and IPv4+UDP flows need the following workflow to support INT:

1. The first NoviFlow switch in the path (a.k.a. INT Source switch) needs to execute two operations: `push_int` to create the INT header and `add_int_metadata` to add a per-hop telemetry data. However, due its implementation, these actions have to be executed in different tables, this example is using table 2:

   1. Table 0 is where `push_int` is executed

   2. Table 0+N (N>0) is where `add_int_metadata` is executed.

   3. Example:

      - table 0

        - priority: Higher (mef_eline + 100)
        - match: <in_port=10,vlan_vid=20,eth_type=0x800,ip_proto=6> # TCP ( IP protocol 6 )
        - instructions:

          - action: push_int
          - action: goto_table 2

        - priority: Higher (mef_eline + 100)
        - match: <in_port=10,vlan_vid=20,eth_type=0x800,ip_proto=17> # UDP ( IP protocol 17 )
        - instructions:

          - action: push_int
          - action: goto_table 2


      - table 2

        - priority: Any
        - match: <in_port=10,vlan_vid=20>  # Just in_port and vlan_vid

        - instructions:

          - action: add_int_metadata
          - action: <all original actions (set_queue, output, push/pop/swap vlan, etc.)>

   - Note: `add_int_metadata` has to be added to the same flow entry where `output` action is, otherwise INT field "egress_id" will be set to 0.


2. The last NoviFlow switch in the path (a.k.a. INT Sink switch) needs to execute two operations: `send_report` to send all metadata content previously added to an INT collector and `pop_int` to remove the INT header and INT metadata, and return the packet to its initial configuration, including DSCP. However, `send_report`, and `pop_int` must be executed in different tables:

   1. Table 0 is where `send_report` is executed
   2. Table 0+N (N>0) is where `pop_int` is executed.
   3. Example:

      - table 0

        - priority: Higher (mef_eline + 100)
        - match: <in_port=10,vlan_vid=20,eth_type=0x800,ip_proto=6>. # TCP
        - instrutions:

          - action: send_report
          - action: goto_table 2

        - priority: Higher (mef_eline + 100)
        - match: <in_port=10,vlan_vid=20,eth_type=0x800,ip_proto=17>. # UDP
        - instrutions:

          - action: send_report
          - action: goto_table 2

      - table 2

        - priority: Any
        - match: <in_port=10,vlan_vid=20>  # Just in_port and vlan_vid
        - instructions:

          - action: pop_int
          - action: <all original actions (set_queue, output, push/pop/swap vlan, etc.)>

  - The choice between adding telemetry or not at the INT Sink Switch will be discussed in Section III.
  - There are other steps for the INT Sink to be discussed later in Section III.


3. NoviFlow switches in the path (a.k.a. INT Hop switch) will only need to add telemetry data to IPv4/TCP/UDP packets.

   1. Example:

      - table 0

        - priority: Higher (mef_eline + 100)
        - match: <in_port=10,vlan_vid=20,eth_type=0x800,ip_proto=6>  # TCP
        - instrutions:

          - action: add_int_metadata
          - action: <all original actions (set_queue, output, push/pop/swap vlan, etc.)>

        - priority: Higher (mef_eline + 100)
        - match: <in_port=10,vlan_vid=20,eth_type=0x800,ip_proto=17>. # UDP
        - instrutions:

          - action: add_int_metadata
          - action: <all original actions (set_queue, output, push/pop/swap vlan, etc.)>

  - There are other options to handle the INT matches at INT Hop switches that could save flow entries. However, this optimization will be addressed in a future blueprint, for instance matching on IP DSCP or other field in the TCP/IP header.

III. Adding INT metadata at the INT Sink switch
===============================================
The NoviWare's INT implementation requires `send_report` action to be executed in Table 0. `send_report` is executed with higher priority than other INT actions, which means adding INT metadata at the INT Sink has to be performed before `send_report` which is not possible on the same set of flow actions. To add INT metadata at the INT Sink, the packets have to be re-injected into the pipeline using external connections via physical loops.

To illustrate the challenge, consider an EVC terminating on INT Hop Z on port 23. The user packet with INT metadata comes from port 11. **mef_eline** would create the following flows (for simplicity, just one direction is presented):

  0. **met_eline** default behavior:

    - match:

      - priority: DEFAULT (DEFAULT FOR EVPL)
      - match: <in_port=11,vlan_vid=20>

    - instruction:

      - action [set_queue, pop_vlan, etc.]
      - action: output to port 23.

To enable INT, first a physical loop has to be deployed. For this example, on INT Hop Z, port 1 is connected to port 2 by a physical fiber patch cord (done during commissioning). Then, the following flows need to be ADDED to the pipeline:

  1. Adding INT metadata:

    - match:

      - table 0

        - priority: Higher (mef_eline + 100)
        - match: <in_port=\ **11**,vlan_vid=20,eth_type=0x800,ip_proto=6>  # TCP
        - instrutions:

          - action: add_int_metadata
          - action: output port **1** (loop)

        - priority: Higher (mef_eline + 100)
        - match: <in_port=\ **11**,vlan_vid=20,eth_type=0x800,ip_proto=17>. # UDP
        - instrutions:

          - action: add_int_metadata
          - action: output port **1** (loop)

  2. Send Report and pop INT data (traffic is coming from port 2 that's the loop with port 1). Only INT data gets into the loop.

    - match:

      - table 0

        - priority: Higher (mef_eline + 100)
        - match: <in_port=\ **2**,vlan_vid=20>
        - instrutions:

          - action: send_report
          - action go to table 2


      - table 2

        - priority: Higher (mef_eline + 100)
        - match: <in_port=\ **2**,vlan_vid=20>
        - instrutions:

          - action: pop_int
          - action [set_queue, pop_vlan, etc.]
          - action: output port **23** (original port)


IV. How to enable INT for EVCs
==============================

The goal for the **telemetry_int** napp is to enable telemetry for ALL EVCs. However, it must support enabling and disabling telemetry for a single EVC or ALL EVCs. This is the approach:

  1 . The **telemetry_int** napp will start operating once **mef_eline** is loaded and EVCs and their flows are pushed to the data plane.

  2. **telemetry_int** will listen for events *kytos/mef_eline.(redeployed_link_(up|down)|deployed|undeployed|deleted|error_redeploy_link_down|created)*

  3. For each EVC identified, **telemetry** will
    1. use EVC's cookie to get all flow entries created by **flow_manager** IF telemetry is not already enabled.
    2. push more specific flows as described in Section II. (See Section IX for information on the cookie ID to be used.)
    3. add a key in the EVC's metadata called "telemetry" with value "enabled". key "telemetry" will be "disabled" once telemetry is disabled for an EVC.

V. Events
==========

  1. Listening
    1. *kytos/mef_eline.(redeployed_link_(up|down)|deployed|undeployed|deleted|error_redeploy_link_down|failover_deployed|failover_link_down)*
    2. *kytos/topology.link_up|link_down*

The following table specifies expected **mef_eline** and **telemetry_int** actions when producing or handling certain events. A redeploy operation means remove and install the flows:

+----------------------------------+--------------------------------------------------------------------+-------------------------------------------------------+
| kytos/mef_eline.<name> event     | mef_eline EVC action                                               | telemetry_int EVC action                              |
+==================================+====================================================================+=======================================================+
| ``undeployed``                   | remove flows                                                       | remove flows; deactivate                              |
+----------------------------------+--------------------------------------------------------------------+-------------------------------------------------------+
| ``deployed``                     | redeploy                                                           | if requested INT, enable if first time or redeploy    |
+----------------------------------+--------------------------------------------------------------------+-------------------------------------------------------+
| ``deleted``                      | remove flows; delete; archive                                      | remove flows; disable                                 |
+----------------------------------+--------------------------------------------------------------------+-------------------------------------------------------+
| ``redeployed_link_down``         | redeploy                                                           | same                                                  |
+----------------------------------+--------------------------------------------------------------------+-------------------------------------------------------+
| ``redeployed_link_up``           | redeploy                                                           | same                                                  |
+----------------------------------+--------------------------------------------------------------------+-------------------------------------------------------+
| ``error_redeploy_link_down``     | remove flows; deactivate                                           | same                                                  |
+----------------------------------+--------------------------------------------------------------------+-------------------------------------------------------+
| ``failover_link_down``           | install ingress flows; publish new flows                           | generate subset flows; install flows                  |
+----------------------------------+--------------------------------------------------------------------+-------------------------------------------------------+
| ``failover_old_path``            | remove old flows; publish old flows (old current_path or failover) | generate subset flows; remove flows                   |
+----------------------------------+--------------------------------------------------------------------+-------------------------------------------------------+
| ``failover_deployed``            | remove failover; install failover flows; publish old and new flows | generate subset flows; remove old; install new        |
+----------------------------------+--------------------------------------------------------------------+-------------------------------------------------------+
| ``uni_active_updated``           | deactivate or activate                                             | same                                                  |
+----------------------------------+--------------------------------------------------------------------+-------------------------------------------------------+

Major challenges to be aware when dealing with **mef_eline** events convergence: a) ensure failover fast convergence, and b) **telemetry_int** isn't differentiating which **mef_eline** path each flow belongs to. Whenever it's desirable that **telemetry_int** only perform a side effect on a subset of the flows, **mef_eline** should facilitate ideally by publishing upfront the set of the flows since **mef_eline** owns the follows and **telemetry_int** essentially follows with its own INT equivalent higher priority flows matching UDP and TCP. In general, the final outcome when handling these events is supposed to only add a few extra milliseconds on top of the existing 2023.2 **mef_eline** flows convergence, the biggest expected latency will be from sending the flow mods in the TCP OpenFlow channel. **mef_eline** will implement these new events:

- **mef_eline** should publish ``kytos/mef_eline.failover_link_down`` right after installing the ingress flows and publishing the new failover flows. Currently, **mef_eline** is publishing a `redeployed_link_down <https://github.com/kytos-ng/mef_eline/blob/master/main.py#L893>`_, but it should be replaced with ``kytos/mef_eline.failover_link_down`` in this case, just so ``telemetry_int`` will be able to efficiently get the flows upfront during this hot path event handling and install ingress related INT flow. 
- **mef_eline** should publish ``kytos/mef_eline.failover_old_path`` when an EVC failover related old path gets removed.
- **mef_eline** should publish ``kytos/mef_eline.failover_deployed`` whenever a new failover is successfully removed and installed, both the old and new failover flows should be published.
- **mef_eline** should publish ``kytos/mef_eline.uni_active_updated`` whenever an EVC active state is updated due to a UNI going up or down.

There's also opportunity to minimize certain deletion FlowMods, especially when **mef_eline** deletes all flows on a switch for a given cookie ``0xA8<7bytes>``, for those cases, it could also mask the adjacent **telemetry_int** cookie ``0xAA<7bytes>``, which would save extra FlowMods to be sent for **telemetry_int**, but sometimes **mef_eline** also only deletes with a specific match, and for those cases it wouldn't be able to delete the other **telemetry_int** flows. This idea might be explored in the future as the network convergence is stress tested depending on the results.

VI. REST API
=============

  - ``POST /telemetry_int/v1/evc/enable`` body evc_ids: [] for bulk insertions, if empty, then enable all. If invalid or non-existing EVC_ID are provided, abort the entire operation with 4XX status code.
  - ``POST /telemetry_int/v1/evc/disable`` body evc_ids: [] for bulk removals, if empty, then remove all. If invalid or non-existing EVC_ID are provided, abort the entire operation with 4XX status code.
  - ``GET /telemetry_int/v1/evc`` list all INT-enabled EVCs.
  - ``GET /telemetry_int/v1/evc_compare`` list and compare which telemetry_int flows are still coherent with EVC metadata status
  - ``PATCH /telemetry_int/v1/evc/redeploy`` body evc_ids: [] to force a redeploy

VII. Dependencies
=================
 * flow_manager
 * mef_eline
 * noviflow
 * topology


VII. New EVC attribute
======================

The **telemetry_int** napp will leverage the EVC's metadata attribute to create a new item, called `telemetry`. This new item will be a dictionary will the following values:

  * "enabled": true|false
  * "status": "UP|DOWN"
  * "status_reason": ["some_error"]
  * "status_updated_at": utc string "%Y-%m-%dT%H:%M:%S" of when the status was updated or null if never.

IX. Failover integration
========================

For EVCs that have failover path pre-provisioned, INT flows will be created to optimize failover. This might lead to flows being created where the EVC is not active and use more table entries.

X. Cookies
==========

The **telemetry_int** napp must use a different cookie ID to help understanding flow ownership and saving IO operations. The cookie prefix assigned to **telemetry** is 0xA8.

XI. Consistency
===============

The **telemetry_int** napp will deploy a routine to evaluate the consistency of the telemetry flows as performed by the **mef_eline** napp. This implementation will be defined via field experience with Kytos. The consistency check will rely on ``sdntrace_cp`` and follow the same pattern as ``mef_eline``, except that also when trying to trace, it should test both UDP and TCP payloads, if any fails after a few attempts, then it should disable telemetry int and remove the flows for now, falling back to mef_eline flows. In the future, the consistency check process might evolve, but for now if it fails, it will fail safely falling back to mef_eline flows.

The consistency check will be implemented after version ``2024.1`` when **mef_eline** implements its enhanced consistency check, and when it's been battle tested for some time, which is expected to also check for active EVCs. **mef_eline** enhanced consistency check details will be specified in a new blueprint. But, in general, it's expected to:

- Run periodically. The seconds interval will be explored, it might stay the same as the existing one every 60 seconds.
- No false positives. It should prioritize stability, it doesn't need to run immediately.
- Only execute and make a decision when no flows have been updated recently.
- **telemetry_int** will implemented a similar consistency check, except it'll run periodically a bit slower, just so if **mef_eline** consistency has to perform any side effects, **telemetry_int** will have a chance to first also react to it, before running its consistency check.


XII. Pacing
===========

The **telemetry_int** napp must wait a *settings.wait_to_deploy* interval before sending instructions to the flow_manager after EVCs are created/modified/redeployed to avoid overwhelming the switches. The goal is to create batch operations.

XIII. Implementation details ``v1``
===================================

The following requirements clarify certain details and expected behavior for ``telemetry_int`` v1 that will be shipped with Kytos-ng ``2023.2``:

- ``mef_eline`` EVC ``telemetry`` metadata is managed by ``telemetry_int``, **only ``telemetry_int`` is supposed to write or delete it**. If you enable or disable INT you should call ``POST /telemetry_int/v1/evc/enable`` or ``POST /telemetry_int/v1/evc/disable``  endpoints. ``telemetry_int`` will not listen for EVC metadata changes since it'll manage it.

- Once ``mef_eline`` creates an EVC, it can optionally request that INT be provisioned. For this case, a ``telemetry_request: dict`` needs to be set in the metadata, currently no keys are needed, but as more options are supported in the future, they can be set. If ``telemetry_int`` can't provision ``telemetry_int``, then it'll set the ``telemetry: {"status": "disabled", "status_reason": ["<reason>"]}`` metadata, updating the status and filling out the reason accordingly.

- Currently, EVCs are always bidirectional. ``telemetry_int`` v1 iteration, will also follow the bidirectional flows as described in the prior sections. In the future, when ``mef_eline`` starts to support unidirectional flows, then following the flows should be mostly seamless, this facilitates implementation and code maintenance without having to try to derive the direction of all flows and maintain a structure that ``mef_eline`` still doesn't support.

- ``telemetry_int`` will require a looped link on each source sink for both intra and inter EVCs, if it's not present, then ``telemetry_int`` will not enable INT, which implies that in this v1 iteration, you'll need to always have a proxy port (check out EP033 for more information) associated with both UNIs since the EVC is bidirectional. Although the EVC is bidirectional, the looped ports are used unidirectionally for each INT source. This explicitness of always knowing that both UNIs will need a proxy port facilitates to keep track when a proxy port changes and performing a side effect.

- If an UNI's proxy port value changes to another port, then ``telemetry_int`` should reinstall the specific associated EVC sink flows accordingly. Similarly, if ``proxy_port`` is removed, it should remove all associated telemetry int flows. Essentially, changing a ``proxy_port`` metadata acts like an update as far as a EVC telemetry enabled is concerned.

- If any other NApp or client, end up accidentally deleting or overwriting ``telemetry`` metadata, it might result in flows being permanently installed in the database. If this ever happens, the following approaches can be used to fix it: a) ``POST /telemetry_int/v1/evc/enable`` and ``POST /telemetry_int/v1/evc/disable`` will allow a ``force`` boolean flag which will ignore if an EVC exist or not, so it'll either provision or decommission accordingly. b) It'll also expose a ``GET /telemetry_int/v1/evc_compare`` which will compare which ``telemetry_int`` flows still have the metadata enabled or not, and generate a list indicating inconsistencies, then you can use it with the option a) endpoints with ``force`` option to either enable or disable again. It will not try to auto remediate.

- When configuring the proxy port, it always needs to be the lower looped interface number (which is also guaranteed by LLDP loop detection), e.g., if you have a loop between interface port number 5 and 6, you need to configure 5 as the proxy port. By this convention, the lower port will be the outgoing port for an incoming NNI traffic.

- Once an EVC is redeployed, ``telemetry_int`` will also redeploy accordingly. Also, to ensure fast convergence when handling link down for EVCs that have failover, it's expected that a typical query to stored flows since it's querying indexed fields will not add significant latency, this point will be observed, and we'll see if it'll perform as expected or if more optimization will be needed from ``telemetry_int`` perspective.

- If a proxy port link goes down, telemetry_int should be disabled and flows removed, falling back to mef_eline flows. Once a proxy port link goes up it should redeploy INT flows if the underlying EVC is active, otherwise it try to deploy again once a new mef_eline deployment event is received.

- If an EVC is deleted or removed and it has INT enabled the flows should be removed.

- The only supported ``table_group`` for ``of_multi_table`` will be ``evpl`` and ``epl``, which represents all EVPL and EPL flows on table 2 and 3 by default respectively. All the other flows will follow the ``table_group`` ``mef_eline`` uses. Also, since NoviWare's INT implementation requires ``send_report`` to be executed in table 0, and ``telemetry_int`` is following ``mef_eline`` then only table 0 should be allowed on ``of_multi_table`` when setting the pipeline if ``telemetry_int`` is also being set. So, in practice, in this iteration, you'll always need to have ``telemetry_int`` on table 0 + table X, where X > 0, and by default it will be on table 2 as documented.

XIV. Open Questions / Future Work
=================================

  1. Error codes, for instance, flows were not instance, there is no proxy ports
  2. Support QFactor (where INT is also extended to the hosts). In this case, the source and the sink should behave like a INT hop only using the `add_int_metadata` action.
  3. Support unidirectional EVCs
  4. Potentially support a specific different "source" and "sink"
