########################################################################################################
'''
 ddrclass.py

******************************************************************************************
******************************************************************************************

  Revision History

 1.0.02 - July-13-2022 petervh 'Added delay time for ddr-control based on runWait'

 1.0.02 - July-13-2022 petervh 'Added delay time for ddr-control based on runWait'
 1.0.01 - July-02-2022 petervh 'Tested and updated ddr rule functions'

******************************************************************************************
******************************************************************************************

 This module defines the DDR class used to instantiate instances of the DDR expert system 
 machine reasoning framework

 Each instance of the DDR class implements an independent machine reasoning environment with
 indepedent instances of the CLIPs expert system.  Multiple instances can be run simultaneously.
 A common way to deploy DDR is to instantiate an DDR instance for each use case

 DDR can be run "on-box" in the guestshell on XE and NX-OS platforms and in a Docker container
 on XR eXR enabled platforms in the app-hosting environment

 DDR can be run "off-box" on PC, MAC or Linux platforms

 DDR execution is controlled by content contained in a collection of files loaded by DDR on startup:

   facts - This file contains information used by DDR to collect "FACTS" from the device.  Also contains
           "control" flags used to control DDR operating modes and execution
   rules - This file contains the CLIPS rules and other CLIPS constructs used to implement the use case
   flags - This file contains control flags for DDR execution
   devices - This file defines the devices that are used by the workflow
   tests - This optional file contains FACTS used to enable execution of CLIPS using pre-defined fact data
           instead of collecting FACTS from the device.  When 'test-facts' is enabled DDR does not interact
           with devices and instead reads and asserts lists of facts from the file.  This can be used
           for testing and demonstrating DDR use cases
   log - This optional file when 'debug-logging' is enabled can be used to persist messages generated by DDR in
         a log file on the device, typically stored on bootflash

******************************************************************************************
******************************************************************************************

 Running DDR

 DDR is run on-box or off-box using a command of the form below.  The relative paths to the control files are parameters. 
 If no parameters are specified DDR finds files in the directory DDR is started in with names 'ddr-facts', 'ddr-rules.',
 'ddr-flags', 'ddr-tests', and 'ddr-log'

   $ python ddrrun.py --facts=input/assurance/memory-leak.txt --rules=input/assurance/memory-leak.clp  
                       --tests=input/assuracne/tests.txt --log=bootflash/ddr-log

 Execution is stopped by CTRL-/ in the window where the script is started

 For information contact: Peter Van Horne petervh@cisco.com

******************************************************************************************
******************************************************************************************

 DDR Functions Called from ddr-rules during workflow execution

 The following modules are called from the execution section in a CLIPs rule
 When a RULE is triggered 0 or more of these action functions can be called
 Action functions can perform any operation desired by the developer
 Typical action functions collect additional data and assert additional FACTS

    run_action - This action function invoked from a RULE causes collection of FACTs defined in the "control_facts_list" in the FACTs file
    run_apply_config - This function called from a RULE applies an edit-config operation to the device
    run_assert_message - Write a message into the DDR log
    run_assert_message_syslog - Generate a Syslog message for XE device without timestamp
    run_assert_sim_fact - Assert a selected FACT from the "ddr-sim" file in /bootflash/guest-share/ddr and assert the FACT
    run_clear_selected_facts - Clear facts in the knowledge base for the template passed in the argument.  Can be used by RULEs to remove types of FACTs
    run_cli_command - Runs a CLI command on the host using the Python "cli" package implemented for the host
    run_cli_parameter - Runs any cli command on the host.  The CLI command can include up to 3 parameters
    run_copy_file - RULEs can copy a file from the device and save the file.  This gives the RULE control over persistence (e.g. prevent wrapping/deletion)
    run_command - Runs a CLI command on the host using the Python "cli" package.  No result is data is collected
    run_ddr- This action function runs the CLIPs engine when called from a rule. Normally called when a RULE asserts new FACTs to trigger any RULEs satisfied by new FACTs
    run_decode_btrace_log - RULE function processes btrace log, selects and asserts FACTs
    run_delay - Rule function that delays execution of rules for specified number of seconds
    run_delete_file - Rule function that deletes a file or files from /bootflash/guest-share on the device running DDR
    run_delete_remote_file - Rule function that deletes a file or files from /bootflash/guest-share on other usecase devices
    run_logging_trigger - Rule function that parses the content of the Syslog logging buffer to
            and asserts facts when buffered log messages contain specific content.
    run_nc_fact - This function called from a RULE collects data using the NETCONF interface and asserts a FACT
    run_ping_action - This action function called from CLIPs rules runs a ping command, collects results and asserts a FACT
    run_process_file - Rule function that parses the content of "filename" stored in /bootflash/guest-share and asserts FACTs
    run_read_control_file - Read a file from /bootflash/guest-share containing FACTs in string form and assert the FACTs
    run_rule_timer - This action function starts a thread to generate a timer to assert a FACT on a regular time interval
    run_set_rule_step - Rule function used to directly set the (rule-step (step STEPNAME)) FACT to change execution flow
    run_set_runmode - Rule function used to change the DDR runmode, 0 for timed trigger, 1 for Syslog trigger (not available on XE), 2 for RFC5277 Notification trigger (not available on XR)
    run_show_fact - Run show command defined in the 'show_facts_list' in the facts file using the rule action function run_show_fact
    run_show_parameter - Runs a show command on the host using either the Python "cli" library or an SSH connection
                         Accepts up to 3 parameters from the CLIPs knowledge base to insert into the show command
    run_show_parameter_index - Runs a show command on the host using either the Python "cli" library or an SSH connection
                         Accepts up to 3 parameters from the CLIPs knowledge base to insert into the show command
                         Accepts a "device_index" parameter to select on which device to perform the operation
    run_suffix - Remove the first part of a value and return the remainder, e.g. get "tunnel-id100", remove "tunnel-id" return 100
    run_trace - This action function called from CLIPs rules runs a traceroute command
    run_write_to_syslog - Send a Syslog message from XE device
    run_xr_send_syslog - Send a Syslog message from XR device
    
    
'''
###################################################################################################################
import ncclient.manager
from ncclient.xml_ import to_ele
from lxml import etree
import xml.dom.minidom
import xml.etree.ElementTree as ET
import clips
import datetime
from datetime import datetime
import time
import os
import sys
import logging
import logging.handlers
from logging.handlers import RotatingFileHandler
import imp
import json
import pprint
import copy
import xmltodict
import re
from threading import Timer
import pexpect
import yaml
#
# Imports for syslog message receiver
#
import queue
import socketserver
import threading
try:
    import resource
except:
    pass
#
# If DDR is running on a device, import the python cli library for running cli commands in guestshell
#
try:
    import cli
except:
    pass

#
# Import library of parser Classes to translate show commands into python dictionaries that are
# translated into CLIPs FACTS using FACT definitions
#
try:
    from genie_parsers import *
except:
    pass
#
# Import parsers that are specific to the usecase if present.  The parsers will be in the ddrparsers.py file
# in the usecase directory
#
try:
    from ddrparsers import *
except:
    pass

############################################################################################
#
# Syslog server for receiving syslog messages from devices that can't send
# NETCONF notfications for syslog messages
# The SyslogUDPHandler runs in a thread and when a Syslog message is received from the device
# puts the Syslog message on the "mesq".  The Syslog mesq is processed by the DDR main loop
#
############################################################################################
#
# This class defines the server for syslog messages
# The handle function is called whenever a message is received
#
class SyslogUDPHandler(socketserver.BaseRequestHandler):
    global mesq
    def handle(self):
        try:
            data = bytes.decode(self.request[0].strip())
            socket = self.request[1]
            mesq.put(str(data).split(':', 1)[-1])
        except Exception as e:
            print("\n%%%%% Error processing Syslog message queue")
#
# Function to put a syslog message in the queue - Runs in thread
# The SyslogUDPHandler is invoked when a Syslog message is received
# UDP server is started to receive messages on PORT
# The device(s) generating Syslog messages are configured "logging host a.b.c.d" to
# direct Syslog messages to the device hosting the Syslog receiver.  UDP messages on
# port 514 are forwarded to the Syslog receiver in eMRE
#
def queue_syslog(mesq, HOST, PORT):
    try:
        server = socketserver.UDPServer((HOST,PORT), SyslogUDPHandler)
        server.serve_forever(poll_interval=0.5)

    except Exception as e:
        print("%%%%% Syslog listener exception: " + str(e))
        print("%%%%% {host,port}: {" + HOST + "," + str(PORT) + "}")
    except KeyboardInterrupt:
        print ("Crtl+C Pressed. Shutting down.")
        sys.exit(0)

    #############################################################################
    #############################################################################
    #############################################################################
class DDR:
    global mesq
    mesq = queue.Queue()
    #
    # DDR entry point - Call this function to load and start CLIPs execution
    #
    #############################################################################
    #############################################################################
    #############################################################################
    def ddr(self,flist, model_control, single_run):
        self.ddr_init(flist, model_control, single_run)
    # 
    # Argument definitions:
    # ~~~ flist - list of file names containing FACT definitions, RULEs, test FACTs and log file name
    # ~~~ model_control - If set to "1" the script reads configuration from a YANG model
    # ~~~ single_run - If set to "1" run the main processing loop once and exit
    #
    # Maintain a counter for the notification logs generated during the session
    # The "control" dictionary stores configuration and state information
    #  
        self.control["session-log-count"] = 1
   
    ######################################################################################    
    ######################################################################################
    # while loop runs until interrupted by CTRL-/ or an error occurs that causes the
    # DDR function to return.
    # Loop continuosly looking for triggering events which include a syslog message
    # containing a string that matches syslog_triggers, an RFC5277 notification including
    # a string that matches the content of a notification or timed execution
    ######################################################################################
    ######################################################################################
        while True:
            self.engine_process()

    def engine_process(self):
        ######################################################################################
        ######################################################################################
        # MAIN RULES ENGINE PROCESSING LOOP
        ######################################################################################
        ######################################################################################

    # 
    # Triggering conditions cause execution of the FACT collection and 
    # RULEs engine execution
    #
        syslog_trigger = False
        timed_trigger = False
        notification_trigger = False
        control_trigger = False

#######################################################################################
#
# control["run-mode"] = 0 Run after time delay
#
#######################################################################################
        if self.control["run-mode"] == 0:
            self.print_log("\n**** DDR Notice: Wait for Trigger Event")
            time.sleep(self.control["run-wait"]/1000)
            timed_trigger = True
            timestamp =  datetime.now().strftime("%m-%d-%Y_%H:%M:%S.%f")
            
            self.print_log("\n**** DDR Notice: Trigger Event: Timed Execution at: " + str(timestamp))

#######################################################################################
#
#   control["run-mode"] = 1 waits for a Syslog message to trigger execution
#   The DDR script runs a Syslog listener on port 9514 (default Syslog port is 514)
#   Devices sending "logging" Syslog messages should use the DDR script IP address and
#   port 9514.  Typicall port forwarding must be configured using NAT to map
#   Packets addressed to the hosting device on Port 9514 to forward to the receiver in DDR 
#   Example XR configuration: logging 10.24.121.9 vrf default severity notifications port 9514
#   Example XE configuration: logging host 10.24.72.167 vrf Mgmt-vrf transport udp port 9514
#
#######################################################################################

        if self.control["run-mode"] == 1:
#
# mesq.get() checks to see if there is a message available on the Queue that
# contains device Syslog messages received and processed by "queue_syslog" function.
# If there are no syslog messages in the queue, do not wait
#
# script waits to receive a syslog message sent the IP and port defined for the syslog listener
#
            syslog = mesq.get()
            timestamp =  datetime.now().strftime("%m-%d-%Y_%H:%M:%S.%f")
            
            if self.control["debug-syslog"] == 1:
                self.print_log("\n**** DDR Debug: Syslog Message Queued at: " + str(timestamp))
                self.print_log(syslog)
#
# Loop through all match strings for the Syslog message.  All of the strings must be found
# in the syslog or the trigger condition is not satisfied
#
#   element 0 - List containing strings that must all be matched
#   element 1 - Optional statically defined FACTs with no parameters.  Each FACT in the list is asserted
#   element 2 - 'True' if the FACT defined in element 3 should be asserted using content parsed from the Syslog message
#   element 3[0] - Regular expression used to extract fields from the syslog message and insert into variables.
#   element 3[1] - List variable names that will be used to populate the FACT
#   element 3[2] - Prototype FACT template name
#   element 3[3] - Syslog event-type
#   element 3[4] - device name
#
# The example below will assert a "rule-step" FACT normally used to trigger usecase execution
# A bgp-event FACT will also be generated
#
# ddr-fact content:
#   syslog_triggers = 
#    [['BGP-5-ADJCHANGE', 'Down'], 
#     ['(rule-step (step bgp-adjacency-syslog-received))'], 
#     'True', 
#     ['.*neighbor.{1}(?P<neighbor>(\S+)).{1}(?P<state>(\S+)).*\(VRF\: (?P<vrf>(\S[^)]+)).*', 
#      [['device', 'str'], ['event-time', 'str'], ['event-type', 'str'], ['neighbor', 'str'], ['state', 'str'], ['vrf', 'str']], 'bgp-event', 'NEIGHBOR-DOWN']]
#
            syslog_trigger = False
            for trigger in self.control["syslog-triggers"]:
                if self.control["debug-syslog"] == 1:
                    self.print_log("\n**** DDR Debug: Syslog trigger: " + str(trigger))

#
# Loop through all match string for the syslog trigger entry.  All of the strings must be found
# in the syslog or the trigger condition is not satisfied
#
                if all([val in str(syslog) for val in trigger[0]]):
                    syslog_trigger = True
                    try:
                        if trigger[2] == 'True':
                            if self.control["debug-syslog"] == 1:
                                self.print_log("\n**** DDR Debug: Assert Syslog FACT")
                            self.assert_syslog_fact(syslog)
#
#  If additional FACTs should be asserted assert each static FACT in the list
#
                        if trigger[1] != []:
                            try:
                                for syslog_fact in trigger[1]:
                                    if self.control["debug-syslog"] == 1:
                                        self.print_log("\n**** DDR Debug: Assert static Syslog FACT: " + str(syslog_fact))
                                    self.env.assert_string(str(syslog_fact))
                            except Exception as e:
                                self.print_log("\n%%%% DDR Error: Asserting static Syslog fact: " + str(trigger[1]) + "\n" + str(e))
                            break
                    except:
                        self.print_log("\n%%%% DDR Error: Asserting syslog fact for: " + str(syslog))
                        break
#
# Extract FACTs from the notification message and assert a FACT in CLIPs
# Process in a try clause for backward compatibility with older FACT files that do not have this option
#
                    if trigger[3] != []:
                        mdata = trigger[3]
                        SlogClean = syslog.replace(',',' ')
                        if self.control["debug-syslog"] == 1:
                            self.print_log("**** DDR Debug: Clean Syslog message: " + str(SlogClean) + "\nRegex: " + str(mdata[0]))
                        try:
                            p1 = re.compile(str(mdata[0])) #generate regex object
                        except Exception as e:
                            self.print_log("\n%%%% DDR Error: Syslog fact regex: " + str(mdata[0]) + "\n" +str(e))
                            break
#
# Build the notification FACT using data parsed from the notification text
# This code supports selecting up to 6 values from the Syslog message to assert in the FACT
# 
                        timestamp =  datetime.now().strftime("%m-%d-%Y_%H:%M:%S.%f")
                        try:
                            try:
                                results = p1.match(str(SlogClean)) #extract fields from message
                                if str(results) == 'None':
                                    self.print_log("**** DDR Debug: Syslog no data found: " + str(mdata[0]) + "\nSyslog Message: " + str(SlogClean))
                                    break

                                if self.control["debug-syslog"] == 1:
                                     self.print_log("**** DDR Debug: Syslog regex results: " + str(results))

                            except Exception as e:
                                self.print_log("\n%%%% DDR Error: Syslog parsing: " + str(SlogClean) + "\n" +str(e))
                                break
#
# create a python dictionary containing the Syslog message FACT data - mdata
#  ['.*', '.*Ethernet ring (?P<ring>\\S+).*', 'ASR903_4', [['ring', 'str']], 'PENDING-CHANGE', 'ring-event']
#                                         
                            if self.control["debug-syslog"] == 1:
                                 self.print_log("**** DDR Debug: Syslog match data: " + str(mdata))
                            try:
                                keys = mdata[3] #get the slot names
                                template = self.env.find_template(str(mdata[5])) # get an empty template fact for event
                                fact1 = {}
                                fact1["device"] = str(mdata[2])
                                fact1["event-type"] = str(mdata[4]) # event type defined for this Syslog message
                                fact1["event-time"] = str(timestamp)
                            except Exception as e:
                                self.print_log("\n%%%% DDR Error: Syslog fact template: " + str(e))
#  ['ring', 'str']

                            try:
                                results = re.search(mdata[1], str(SlogClean))
                                group = results.groupdict() #dictionary contains key/values for parsed message slots

                                for fact in mdata[3]:
                                    if fact[1] == 'str':
                                        fact1[str(fact[0])] = str(group[str(fact[0])])
                                    elif fact[1] == 'int':
                                        fact1[str(fact[0])] = int(group[str(fact[0])])
                                    elif fact[1] == 'flt':
                                        fact1[str(fact[0])] = float(group[str(fact[0])])
                                    elif fact[1] == 'none':
                                        fact1[str(fact[0])] = group[str(fact[0])]
                            except Exception as e:
                                self.print_log("\n%%%% DDR Error: Syslog fact parameters: " + str(e) + " fact: " + str(fact) + " " + str(group[str(fact[0])]))

                            try:
                                if self.control["debug-fact"] == 1:
                                    self.print_log("**** DDR Debug: syslog fact dictionary: " + str(fact1))
                                template.assert_fact(**fact1)

                            except Exception as e:
                                if self.control["debug-fact"] == 1:
                                    self.print_log("\n%%%% DDR Exception: syslog fact: " + str(e) + " " + str(fact1))
                                pass

                        except Exception as e:
                                self.print_log("\n%%%% DDR Error: Syslog processing error :" + str(SlogClean) + " " + str(e))

                    if syslog_trigger == True: break

#######################################################################################
#
# control["run-mode"] = 2 Trigger on NETCONF notification
#                         The notification is generated with the DMI infrastructure generates
#                         an RFC5277 notification using the content of Syslog message or SNMP trap
#
#######################################################################################
        if self.control["run-mode"] == 2:
#
# Check for NETCONF notifications from the device
# If netconf notification is received, check if the content of the notification
# Matches one of the "triggers".
# Check to see if the notification contains content that should be used to directly assert a fact
#
#######################################################################################
#
# block and wait until an RFC5277 NETCONF notification is received
#
            if self.control["debug-notify"] == 1:
               self.print_log("**** DDR Debug: Wait for Notification Event")

            if self.control["single-notify"] == 1:
                input("\n\nHit Enter to accept one notification event\n\n")  # useful when manually debugging

            try:
                if self.control["run-mode"] == 2:
                    notification_found = False
                    notif = self.notify_conn.take_notification(block=True) # block on IO waiting for the NETCONF notification
                    notify_xml = notif.notification_xml
                if self.control["debug-notify"] == 1:
                    self.print_log("**** DDR Debug: RFC5277 notification received: " + notify_xml)
            except Exception as e:
                self.print_log("\n%%%% DDR Error: In notification event processing: " + str(e))
                notify_xml = ''

######################################################################################
# Test to see if the snmpevents notification included any of the strings that
# are included in the notification_triggers definitions in the FACTs file.  If so, run the rules engine
######################################################################################

            if notify_xml != '':

                for trigger in self.control["notification-triggers"]:
                    notification_trigger = False
#
# Loop through all match strings for the notification.  All of the strings must be found
# in the notification or the trigger condition is not satisfied
#
#   element 0 - List containing strings that must all be matched.  Strings appear in the NETCONF Notification
#   element 1 - Optional statically defined FACT with no paramters if triggering conditions in element 0 are satisfied
#   element 2 - 'True' if the the content in element 3 should be used to generate a FACT from the message content
#   element 3[0] - The regular expression in this entry extracts all of the text from the indicated xml tag for processing
#   element 3[1] - Regular expression used to extract fields from the message and insert into variables.
#   element 3[2] - List variable names that will be used to populate the FACT
#   element 3[3] - Prototype FACT template.  The variables are inserted in order into the template and the FACT is asserted
#   element 3[4] - RULE template name.  Name of the RULE in ddr-rules used to assert the FACT
#
# notification_triggers = [
#      [['ADJCHANGE', 'Down'], 
#       ['(assert (sample-fact (sample-slot sample-value)))'], 
#       'false', 
#       ['<clogHistMsgText>(.*)</clogHistMsgText>', 
#        'neighbor.{1}(?P<neighbor>(\S+)).{1}(?P<state>(\S+)).{1}(?P<message>(.+))', 
#        ['neighbor', 'state', 'message'],
#        '(bgp-event (device {0}) (event-time {1}) (event-type NEIGHBOR-DOWN) (neighbor {2}) (state {3}) (message {4})  (syslog-time {5}))']
#     ]
# ]
#
                    if all([val in str(notify_xml) for val in trigger[0]]): 
                        notification_trigger = True                        
                        try:
                            timestamp =  datetime.now().strftime("%m-%d-%Y_%H:%M:%S.%f")
                            self.print_log("\n**** DDR Notice: Triggering Notification Event Found: " + str(trigger[0]) + " at: " + str(timestamp))
                            for fact in trigger[1]:
                                if self.control["debug-fact"] == 1:
                                    self.print_log("**** DDR Debug: Notification static FACT: " + str(fact))  
                                try:                                  
                                    self.env.assert_string(str(fact))
                                except Exception as e:
                                    self.print_log("\n%%%% DDR Error: Asserting notification static FACT: " + str(fact) + " " + str(e))
                                    break

#
# Get the event time - Extract syslog mesasge time from notification and translate into seconds since midnight
# Insert the seconds into the generated syslog FACT
# Sample time: <eventTime>2020-09-29T09:47:38+00:00</eventTime>
#
                            try:
                                event_seconds = 0
                                event_content = re.search('<eventTime>(.*)</eventTime>', str(notify_xml)) #extract event_time field from message
                                p1 = re.compile('.{11}(?P<mtime>(.{8}))') #regex to extract time in seconds
                                results = p1.match(str(event_content.group(1))) #extract seconds field from message
                                syslog_time = str(event_content.group(1))
                                group = results.groupdict() #dictionary contains objects
                                times = str(group["mtime"]).split(":")
                                event_seconds = sum(int(x) * 60 ** i for i, x in enumerate(reversed(times)))
                            except Exception as e:
                                self.print_log("\n%%%% DDR Error: Generating notification time: " + str(event_content.group(1)) + " " + str(e))
                                break
#
# Extract FACTs from the notification message and assert a FACT in CLIPs
#
                            if trigger [2] == "True" and trigger[3] != [] :
                                try:                                
                                    SlogClean = notify_xml.replace(',',' ')
                                except Exception as e:
                                    self.print_log("\n%%%% DDR Error: Processing notification message: " + str(notify_xml) + " " + str(e))
                                    break
#
# Build the notification FACT using data parsed from the notification text
# trigger[3][0] is used to select content from the Syslog message
#
                                timestamp =  datetime.now().strftime("%m-%d-%Y_%H:%M:%S.%f")
                                try:
                                    if self.control["debug-notify"] == 1:
                                        self.print_log("**** DDR Debug: Syslog trigger regex: \n" + str(trigger[3][0]) + " " + str(trigger[3][1]))
                                    message_content = re.search(trigger[3][0], str(notify_xml))

                                    if self.control["debug-notify"] == 1:
                                        self.print_log("**** DDR Debug: Syslog trigger message: \n" + str(message_content.group(1)))

                                    p1 = re.compile(trigger[3][1]) #regex to extract fields from the message content
                                    results = p1.match(str(message_content.group(1))) #extract fields from message

                                    if results == None:
                                        self.print_log("\n%%%% DDR Error: No Notification match for: \n" + str(message_content.group(1)) +  " " + str(trigger[3][1]) )
                                        break

                                    if self.control["debug-notify"] == 1:
                                        self.print_log("**** DDR Debug: Syslog FACT string: \n" + str(results))
                                        
                                except Exception as e:
                                    self.print_log("\n%%%% DDR Error: Notification parsing: Check for errors or missing regex definitions\n "  +str(e) + "\n" + str(trigger))
                                    break

############################################################################################################
#
# notification_triggers are triggered by matching a list of strings contained in RFC5277 notifications generated by the device.
# If the notification first element of the list, the trigger is matched, eMRE asserts a triggers eMRE execution
# The notification_trigger can also be used to extract data from the Syslog message and assert a FACT.
# In the examples below two notifications are used, one for neighbor up and one for neighbor down.
#   element 0 - List containing strings that must all be matched
#   element 1 - Optional statically defined FACT with no paramters
#   element 2 - 'True' if the FACT defined in element 3 should be asserted
#   element 3[0] - The regular expression in this entry extracts all of the text in the indicated xml tag for processing
#   element 3[1] - Regular expression used to extract fields from the message and insert into variables.
#   element 3[2] - List variable names that will be used to populate the FACT
#   element 3[3] - Value to put into notification FACT 'event-type' slot
#   element 3[4] - deftemplate name in ddr-rules file to contain the generated FACT
#
# Example: *Feb 26 07:13:33.842: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/12, changed state to down
#     <clogHistMsgText>Line protocol on Interface GigabitEthernet1/0/11, changed state to down</clogHistMsgText>
#notification_triggers = [
#  [['LINEPROTO', 'changed state to down'],
#   ['(notify-event (event-type "INTERFACE-DOWN") (name GE12))'],
#   'True',
#  ['<clogHistMsgText>(.*)</clogHistMsgText>',
#    'Line protocol on Interface (?P<name>(\S+))',
#  [['name', 'str']], 'INTERFACE-DOWN', 'notify-event']
#  ]
#]
#
# create a python dictionary containing the Notification message FACT data
# Create a FACT and insert the slot values for device, event-time, and event-type defined in ddr-facts 
#
                                fact1 = {"notification-not-found"} # return in error message if FACT is not generated

                                try:
                                    try:
                                        template = self.env.find_template(str(trigger[3][4])) # get an empty template fact for event
                                    except Exception as e:
                                        self.print_log("\n%%%% DDR Error: Notification template not found: " + str(fact1) + " " + str(e))
                                        break

                                    fact1 = {}
                                    fact1["device"] = str(self.control["mgmt-device"][4])
                                    fact1["event-type"] = str(trigger[3][3]) # event type defined for this Syslog message
                                    fact1["event-time"] = str(timestamp)
                                    if self.control["debug-fact"] == 1:
                                        self.print_log("**** DDR Debug: notification fact initialization: " + str(fact1))

                                except Exception as e:
                                    self.print_log("\n%%%% DDR Error: Notification FACT initialization error: " + str(fact1) + " " + str(e))
                                    break
#
# Insert the FACT slot values defined in ddr-facts for this notification
#
                                try:
                                    if results == None:
                                        self.print_log("\n%%%% DDR Error: No Notification match for fact slots in: " + str(message_content))
                                                                          
                                    group = results.groupdict() #dictionary contains key/values for parsed message slots
                                    try:
                                        keys = trigger[3][2] #get the slot names
                                        if self.control["debug-notify"] == 1:
                                            self.print_log("**** DDR Debug: Notification FACT keys: " + str(keys))
                                       
                                        for fact in keys:
                                            if fact[1] == "int":
                                                fact1[fact[0]] = int(group[str(fact[0])])
                                            elif fact[1] == "flt":
                                                fact1[fact[0]] = float(group[str(fact[0])])
                                            elif fact[1] == "str":
                                                fact1[fact[0]] = str(group[str(fact[0])])
                                            else:
                                                fact1[fact[0]] = group[str(fact[0])]

                                    except Exception as e:
                                        self.print_log("\n%%%% DDR Error: notification fact slots: " + str(fact) + " " + str(e))

                                    if self.control["debug-fact"] == 1:
                                        self.print_log("**** DDR Debug: notification fact dictionary: " + str(fact1))
                                    template.assert_fact(**fact1)
                                        
                                except Exception as e:
                                    if self.control["debug-fact"] == 1:
                                        self.print_log("\n%%%% DDR Error: Notification fact: " + str(e) + " " + str(fact1))
                                    pass

                        except Exception as e:
                            self.print_log("\n%%%% DDR Error: Processing notification fact: " + str(fact1) + " " + str(e))
                            break

                    if notification_trigger == True: break

#######################################################################################
#
# control["run-mode"] = 3 Run CLIPs immediately - Subsequent triggers caused by
#                         control files written by external systems to /bootflash/guest-share/ddr
#
#######################################################################################
        if self.control["run-mode"] == 3:
            self.print_log("\n**** DDR Notice: Trigger from external application control-file write to device")
            run_delay = self.control["run-wait"]/1000
            self.run_read_control_file(self.control["control-file"], run_delay)
            control_trigger = True
            timestamp =  datetime.now().strftime("%m-%d-%Y_%H:%M:%S.%f")
            
            self.print_log("\n**** DDR Notice: Trigger Event: Enable external application trigger at: " + str(timestamp))
            
###################################################################################
#
# Determine if DDR is in the "active" mode with standby-active set to 1
# If standby-active is set to 0, skip processing for this execution cycle
# The standby-active state is updated at the end of each execution loop
#
###################################################################################

        if self.control["standby-active"] == 1:
         
###################################################################################
#
# Test to see if any triggering event was found
#
###################################################################################

            if syslog_trigger or timed_trigger or notification_trigger or control_trigger:

######################################################################################
#
# For each fact in the fact_list get the operational or configuration data required
# to populate the facts required by the expert system
# Assert the facts in the expert system
# Measure the amount of time required to collect the facts from the device
#
######################################################################################

#####################################################################
#####################################################################
#
# Assert all FACTS here
#
####################################################################
####################################################################
                starttime = time.time()
#
# If test facts are being used, get the facts from the test_data list 
# loaded from the 'ddr-tests' file
# The test FACTs are loaded into memory so the FACTs can be "played"
# to DDR for testing when the actual use case environment is not available
#
                if self.control["test-facts"] == 1:
                    starttime = time.time()
                    if self.control["debug-fact"] == 1:
                        self.print_log("**** DDR Debug: Test FACTs: " + str(self.test_data))

                    self.assert_test_facts(self.test_data)
#
# Log time to load test facts
#
                    endtime = time.time()
                    fact_runtime = endtime - starttime
                    self.ddr_timing(' **** DDR Time: Load and assert test facts(ms): %8.3f', fact_runtime, "load-test-facts")
#
# If use case requires asserting a FACT to advance the "period" in a state machine in the rules, assert the
# template fact "update-period-fact" at the end of each run of CLIPS
# Asserting this FACT will cause rules in the RULEs file that include this FACT as a Condition to fire
#
                    if self.control["update-period"] == 1:  
                        try:
                            self.env.assert_string("(update-period-fact (update TRUE))")
                        except: pass #ignore case where FACT already exists    
#
# If control["test-facts"] not set, read the FACTs from the devices
#
                else:
                    starttime = time.time()
                    try:
                        for fact in self.control["fact-list"]:
                            if self.control["debug-fact"] == 1:
                                self.print_log("**** DDR Debug: Main fact loop: " + str(fact))
                            if fact["fact_type"] == "show_and_assert":
                                if "log_message_while_running" in fact:
                                    self.print_log(fact["log_message_while_running"])
                                self.show_and_assert_fact(fact, self.control["device-list"][0][6])
                            elif fact["fact_type"] == "multitemplate":
                                self.get_template_multifacts_index(fact["data"], 'none', 'none', self.control["device-list"][0][6])
                            elif fact["fact_type"] == "multitemplate_protofact":
                                self.get_template_multifacts_protofact_index(fact, 'none', 'none', self.control["device-list"][0][6])
                            else:
                                self.print_log("\n%%%% DDR Error: Error in ddr-facts fact_list [] - Invalid fact type: " + str(fact))
                    except Exception as e:
                    
                        self.print_log("\n%%%% DDR Error: Error in ddr-facts fact_list [] - Fact type selection error: " + str(e))
#
# If required assert fact to advance the "period" state in the knowledge base
#
# If use case requires asserting a FACT to advance the "period" in a state machine in the rules, assert the
# template fact "update-period-fact" at the end of each run of CLIPS
#
                    if self.control["update-period"] == 1:  
                        try:
                            self.env.assert_string("(update-period-fact (update TRUE))")
                        except: pass #ignore case where FACT already exists    

                    endtime = time.time()
                    fact_runtime = endtime - starttime
                    self.ddr_timing(' **** DDR Time: Get and process FACTS from devices(ms): %8.3f', fact_runtime, "get-device-facts")

#
# If control facts are being used, get the facts from the ddr-control file
# The FACTs are loaded into memory so the FACTs can be used
# control DDR exection
#
                if self.control["control-facts"] == 1:
                    starttime = time.time()
                    self.read_control_facts()
                    
#
# Log time to load ddr-control facts
#
                    endtime = time.time()
                    fact_runtime = endtime - starttime
                    self.ddr_timing(' **** DDR Time: Load and assert control-facts(ms): %8.3f', fact_runtime, "load-ddr-control")
#
# If use case requires asserting a FACT to advance the "period" in a state machine in the rules, assert the
# template fact "update-period-fact" at the end of each run of CLIPS
# Asserting this FACT will cause rules in the RULEs file that include this FACT as a Condition to fire
#
                    if self.control["update-period"] == 1:  
                        try:
                            self.env.assert_string("(update-period-fact (update TRUE))")
                        except: pass #ignore case where FACT already exists    
#
# log the FACTS asserted and triggered RULES on the agenda before CLIPs runs 
#
                self.print_clips_info()
#
# If required return clips facts and a dictionary representation of facts in the Service Impact Notification
# FACTs are sent before running clips to capture all facts that were present before the rules were run
#
                self.clips_facts = ''
                self.dict_facts = ''
                if self.control["send-before"] == 1:
                    self.send_clips_info()
                    self.save_dict_facts()

                for item in self.env.activations():
                    root_cause_rule = str(item)
                    break

##################################################################################
#
# env.run() causes execution of any action functions on the RHS of rules
# These control["actions"] can include: asserting additional FACTS, calling a python function
# adding a message to the service-impact notification
# Compute the run time for the CLIPs rules
#
##################################################################################
                starttime = time.time()
                try:
                    self.memory_use(" **** DDR Memory: Before Running Inference Engine in KBytes: ", "before-clips-run")
                    if self.control["debug-fact"] == 1:
                        self.print_log("**** DDR Debug: Run CLIPs to process FACTs")
                    
                    self.env.run()
                except Exception as e:
                    self.print_log("\n%%%% DDR Error: Exception when running inference engine " + str(e))
                    self.close_sessions()
                    sys.exit(0)
                self.memory_use(" **** DDR Memory: After Running Inference Engine in KBytes: ", "after-clips-run")

                endtime = time.time()
                clips_runtime = endtime - starttime
                self.ddr_timing(' **** DDR Time: Run Inference Engine(ms): %8.3f\n', clips_runtime, "clips-runtime")
#
# Add facts to service impact notification after CLIPs is run if required
#
                if self.control["send-after"] == 1:
                    self.clips_facts = ''
                    self.dict_facts = ''
                    self.send_clips_info()
                    self.save_dict_facts()
#
# Generate service-impact-notification message 
# Include all messages asserted by rules.  Optionally include the CLIPS facts in
# CLIPS fact format and/or in Python dictionary format
#
                try:
                    event_time = datetime.now().strftime("%m-%d-%Y_%H_%M_%S.%f")
                    impactMessage = ''.join(self.impactList)
                    self.impactList = []
#
# Generate RFC5277 formated Service Impact notification message
#
                    try:
                        si_notification = self.control["service-impact"].format("DDR Service Impact Notification: " + str(self.control["use-case"]), impactMessage, self.clips_facts, self.dict_facts, str(self.control["session-time"]), datetime.now().strftime("%m-%d-%Y_%H_%M_%S.%f"))
                        if self.control["json-notify"] == 1:
                            data_dict = xmltodict.parse(si_notification)
                            si_notification = json.dumps(data_dict, indent=2, sort_keys=True)
                    except Exception as e:
                        self.print_log("\n%%%% DDR Error: Content error: " + str(e))
      
                    if self.control["show-notification"] == 1:
                        self.print_log("\n################################ DDR Service Impact Notification ############################\n")
                        self.print_log(si_notification)
#
# Send RFC 5277 notification if platform with "send-notification" model implementation
# The RFC 5277 notification is not currently included in the Polaris implementation
#
                    if self.control["send-notification"] == 1 or self.control["send-notification"] == 3:
                        if self.control["debug-notify"] == 1:
                            self.print_log("\n**** DDR Debug: Notification Message: \n" + str(si_notification))
                        try:
                            self.netconf_connection.dispatch(to_ele(si_notification))
                        except Exception as e:
                            self.print_log("\n%%%% DDR Error: Notification request send error: " + str(e))
#
# Send Syslog message using IOX infrastructure if required
# This Syslog message will have the following form:
#
#    *Nov 3 17:22:13.514: %IM-5-IOX_INST_NOTICE: Switch 1 R0/0: ioxman:  
#                         IOX SERVICE guestshell LOG: DDR_MESSAGE BGPIC_Event: Log: /bootflash/guest-share/BGPIC_Event_11-03-2020_17:20:36.896019.1
#
# The Syslog indicates the message came from the IOX infrastructure which hosts the guestshell.  The first section is generated by # # the device Syslog feature.  The second section (shown on a separate line) is generated by DDR from content provided by 
# RULEs and FACTs.  The "Log:" contains the full text of the service impact message.
# There can be multiple logs generated for an DDR execution.  The timestamp on the log is the time the session started.  The ".1" increment is added to filename  for each additional log
                    if self.control["send-notification"] == 2:
                        try:
                            filenum = self.control["session-log-count"]
                            self.control["session-log-count"] = self.control["session-log-count"] + 1
                            filename = str(self.control["log-path"]) + str(self.control["use-case"]) + "_TS_" + str(self.control["session-time"]).replace(":", "_") + "." + str(filenum)
                            syslog_notification = "DDR_MESSAGE " + str(self.control["use-case"]) + ": Log: " + str(filename)
                            self.run_write_to_syslog(syslog_notification, 0)
                        except Exception as e:
                            self.print_log("\n%%%% DDR Error: Error sending Syslog notification: " + str(e))
                        if self.control["debug-notify"] == 1:
                            self.print_log("\n**** DDR Debug: Syslog Notification Message: \n" + str(syslog_notification))
#
# Write notification log file to guest-share if syslog message notification mode selected
#
                        try:
                            with open(filename, "a+") as fd:
                                fd.write(str(si_notification))
                                fd.write("\n")
                            if self.control["debug-notify"] == 1:
                                self.print_log("\n**** DDR Debug: Syslog Notification Log: \n" + str(filename))
                        except Exception as e:
                            self.print_log("\n%%%% DDR Error: Error writing Notification log: " + str(e) + "\n" + str(filename))

                except Exception as e:
                    self.print_log("\n%%%% DDR Error: Notification send error: " + str(e))

#
# log the FACTS asserted and RULES on the agenda after CLIPs runs
#
                self.print_clips_info()
#
# Show memory statistics
#
            if self.control["show-memory"] == 1:
                self.print_log("\n**** DDR Memory: Memory use statistics (kb)")
                self.print_log(self.memory)
                self.assert_statistics_fact("ddr-memory", self.memory)

#
# Show timing statistics
#
            if self.control["show-timing"] == 1:
                self.print_log("\n**** DDR Timing: Execution time statistics (ms)")
                self.print_log(self.timing)
                self.assert_statistics_fact("ddr-timing", self.timing)
                self.timing = {
                "load-initial-facts" : 0,
                "load-test-facts" : 0,
                "load-ddr-control" : 0,
                "get-device-facts" : 0,
                "clips-runtime" : 0}    
#
# If required clear selected template facts before running CLIPS again
# A list of FACTs to clear is optionally included in the FACT definitions
#
            self.clear_selected_facts()
#
# If using test-facts and all facts have been used exit the main loop
# test-facts loop above sets standby-active to force exit
#
            if self.control["standby-active"] == 2:
                sys.exit(0)
#
# If "run-one" is set, exit DDR after running only one iteration
#
            if self.control["run-one"] == 1:
                self.print_log("\n!!!!!!!!!!!!!!!!!!!!!!!!!!!!! DDR Execution Cycle Completed !!!!!!!!!!!!!!!!!!!!!!!!!!!\n")
                sys.exit(0)
#
# If manual execution control wait for operator input
#
            if self.control["user-control"] == 1:
                self.print_log("\n!!!!!!!!!!!!!!!!!!!!!!!!!!!!! DDR Execution Cycle Completed !!!!!!!!!!!!!!!!!!!!!!!!!!!\n")
                continueRunning = input("**** DDR Notice: Enter 1 continue, 0 to exit: ")

                if continueRunning == str(1):
                    pass
                else:
                    sys.exit(0)
#
# call set_ddr_runmode to apply any changes to DDR execution applied
# to the facts file after DDR is started
# if standby-active has been set to 2, exit DDR
#
        self.set_ddr_runmode()
        if self.control["standby-active"] == 2:
            sys.exit(0)

################################################################
################################################################
################################################################
#
# End of DDR While loop
#
################################################################
################################################################
################################################################


########################################################################
########################################################################
#
# Class Methods
#
########################################################################
########################################################################

    ###########################################################################################
    #
    # save_si_notification - Save service impact notification in ddr-control model list
    # list if required.  Save up to "max-notifications" service impact notifications in the 
    # saved-notifications list
    #
    ###########################################################################################

    def save_si_notification(self, si_notification):
        if self.control["save-notifications"] == 1:
            if self.control["notify-index"] > self.control["max-notifications"]:
                self.control["notify-index"] = 1
            try:
                store_notification = self.control["save-notification"].format(self.control["ddr-id"], self.control["notify-index"], datetime.now().strftime("%m-%d-%Y_%H_%M_%S.%f"), 'new', si_notification.replace("<", "&lt;").replace(">", "&gt;"), self.nc_facts)
                store_notification_text = store_notification
                if self.control["debug-notify"] == 1:
                    self.print_log("\n**** DDR Debug: Saved Notificaion: \n" + str(store_notification_text))
                self.netconf_connection.edit_config(store_notification_text, target='running')
                self.control["notify-index"] = self.control["notify-index"] + 1

            except Exception as e:
                self.print_log("\n%%%% DDR Error: Notification save error: " + str(e))

    ###########################################################################################
    #
    # clear_selected_facts - Clear facts in the knowledge base for templates in the clear-filter list
    #
    # NOTE: All FACTS in the knowledge-base that have a template name in the clear-filter list
    #       all cleared at the end of the DDR execution cycle
    #
    ###########################################################################################
    def clear_selected_facts(self):
        if self.control["clear-selected"] == 1:
            for cfilter in self.control["clear-filter"]:
                notify_found = True
                while notify_found:
                    for fact in self.env.facts():
                        notify_found = False
                        try:
                            if str(cfilter) in str(fact):
                                notify_found = True
                                fact.retract()
                                break
                        except Exception as e:
                            self.print_log("\n%%%% DDR Error: Error clearing fact: " + str(e))

########################################################
#
#  ddr_init - initialize ddr runtime
#
########################################################
    def ddr_init(self, flist, model_control, single_run):
    #
    # Flags that control DDR execution are read from the cisco-ios-xe-ddr-control.yang model
    # The values of the control flags are read when DDR is started and after the
    # execution of each DDR pass (while loop execution)
    # The default values for the flags are shown below
    # These values will be updated by reading the ddr-control model
    # If the 'control["sync"]' flag in the ddr-control model is set, the values in the model
    # are updated in the control dictionary and selected values applied to DDR control variables
    #
        self.control = dict()
    #
    # If model_control which is passed in when DDR is started is == 1, control DDR using
    # the ddr-control.yang model
    #
        try:
            self.control.update({"model-control" : model_control})
            self.control.update({"send-messages" : 0})
            self.control.update({"debug-logging" : 0})
            self.control.update({"session-time" : ''})
            self.control["session-time"] = str(datetime.now().strftime('%m-%d-%Y_%H_%M_%S.%f'))
            self.control.update({"session-log-count" : 0})
    #
    # variable to collect facts that are converted from dictionary format and saved
    #
            self.nc_facts = ''
    #
    # Dictionary for collecting timing information
    #
            self.timing = {
            "load-initial-facts" : 0,
            "load-test-facts" : 0,
            "load-sim-facts" : 0,
            "load-ddr-control" : 0,
            "get-device-facts" : 0,
            "clips-runtime" : 0
            }
    #
    # Dictionary for collecting memory use information
    #
            self.memory = {
            "entry" : 0,
            "before-test-facts" : 0,
            "after-test-facts" : 0,
            "before-sim-facts" : 0,
            "after-sim-facts" : 0,
            "before-ddr-control" : 0,
            "after-ddr-control" : 0,
            "before-clips-env" : 0,
            "after-clips-env" : 0,
            "before-rules-load" : 0,
            "after-rules-load" : 0,
            "before-load-init-facts" : 0,
            "after-load-init-facts" : 0,
            "after-assert-init-facts" : 0,
            "before-clips-run": 0,
            "after-clips-run": 0
            }
    #
    # impactList is used to collect messages to include in a service-impact notification
    #
            self.impactList = []
    #
    # Template for saving notifications
    #
            self.control["save-notification"] = '''
 <config xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
  <ddr-control xmlns="http://cisco.com/ns/yang/cisco-ios-xe-ddr-control">
    <instances>
      <instance>
        <ddr-id>{0}</ddr-id>
        <saved-notifications>
          <notifications>
            <notification>
              <id>{1}</id>
              <datetime>{2}</datetime>
              <status>{3}</status>
              <content>{4}</content>
              <dictionary-facts>{5}</dictionary-facts>
            </notification>
          </notifications>
        </saved-notifications>
      </instance>
    </instances>
  </ddr-control>
 </config>'''
        except Exception as e:
            self.print_log('%%%% DDR Error: ddr_init Intialization error: ' + str(e))
            self.print_log('%%%% DDR Error: flist: ' + str(flist))
            sys.exit(0)

    ###################################################################################################
    #
    # Check to see if the use case FACTS file is available in /bootflash/guest-share/ddr-facts
    # If the ddr-facts file is present, read all DDR control files from guestshare:
    #
    #    ddr-facts - FACT collection and use case control definitions
    #    ddr-rules - RULE definitions for CLIPs execution
    #    ddr-flags - Control flags for DDR execution
    #    ddr-devices - Devices used for use case
    #    ddr-tests - FACT definitions to assert when DDR is running if 'test-facts' is set
    #    ddr-init - File containing collection of FACTs to assert on ddr startup if 'initial-facts' is set
    #    ddr-control - File containing facts defined by external application used to control ddr execution
    #    ddr-action - File containing collection of FACTs to assert on
    #    ddrstartup if 'initial-facts' is not  set
    #    
    # If the files are not present in /guest-share use the the files provided as command line arguments
    #
    ###################################################################################################

        try:
            if os.path.exists('/bootflash/guest-share/ddr' + 'ddr-facts'):
                self.control.update({"facts-file" : '/bootflash/guest-share/ddr' + 'ddr-facts'})
                self.control.update({"rules-file" : '/bootflash/guest-share/ddr' + 'ddr-rules'})
                self.control.update({"flags-file" : '/bootflash/guest-share/ddr' + 'ddr-flags'})
                self.control.update({"devices-file" : '/bootflash/guest-share/ddr' + 'ddr-devices'})
                self.control.update({"test-file" : '/bootflash/guest-share/ddr' + 'ddr-tests'})
                self.control.update({"init-facts-file" : '/bootflash/guest-share/ddr' + 'ddr-init'})
                self.control.update({"control-file" : '/bootflash/guest-share/ddr' + 'ddr-control'})
                self.control.update({"sim-file" : '/bootflash/guest-share/ddr' + 'ddr-sim'})
                self.control.update({"parser-file" : '/bootflash/guest-share/ddr' + 'ddr-parsers.py'})
            else:
                if not os.path.exists(flist[0]):
                    self.print_log("\n%%%% DDR Error: FACTS file not found")
                if not os.path.exists(flist[1]):
                    self.print_log("\n%%%% DDR Error: RULES file not found")
                self.control.update({"facts-file" : flist[0]})
                self.control.update({"rules-file" : flist[1]})
                self.control.update({"flags-file" : flist[2]})
                self.control.update({"devices-file" : flist[3]})
                self.control.update({"test-file" : flist[4]})
                self.control.update({"init-facts-file" : flist[5]})
                self.control.update({"control-file" : flist[6]})
                self.control.update({"sim-file" : flist[7]})
                self.control.update({"parser-file" : flist[8]})
    #
    # Read the ddr control flags from the ddr-flags
    #
            fd = open(self.control["flags-file"])
            inputdata = imp.load_source('inputdata', self.control["flags-file"], fd)
            fd.close()
        except Exception as e:
            self.print_log('%%%% DDR Error: initialization files read error: ' + str(e))
            sys.exit(0)
    #
    # DDR instance control parameters will be loaded
    # from the ddr-flags file to set the 'control' dictionary parameters
    #
        try:
            self.control.update({"initial-facts" : inputdata.initialFacts})
            self.control.update({"nc-timeout" : inputdata.ncTimeout})
            self.control.update({"log-file" : inputdata.logFile})
            self.control.update({"ddr-id" :  1})
            self.control.update({"run-mode" : inputdata.runMode})
            self.control.update({"run-one" : inputdata.runOne})
            self.control.update({"user-control" : inputdata.userControl})
            self.control.update({"run-wait" : inputdata.runWait})
            self.control.update({"update-period" : inputdata.updatePeriod})
            self.control.update({"actions" : inputdata.actions})
            self.control.update({"clear-facts" : inputdata.clearFacts})
            self.control.update({"clear-selected" : inputdata.clearSelected})
            self.control.update({"test-facts" : inputdata.testFacts})
            self.control.update({"sim-facts" : inputdata.simFacts})
            self.control.update({"show-facts" : inputdata.showFacts})
            self.control.update({"show-dict" : inputdata.showDict})
            self.control.update({"show-rules" : inputdata.showRules})
            self.control.update({"show-messages" : inputdata.showMessages})
            self.control.update({"send-messages" : inputdata.sendMessages})
            self.control.update({"show-notification" : inputdata.showNotification})
            self.control.update({"show-memory" : inputdata.showMemory})
            self.control.update({"show-timing" : inputdata.showTiming})
            self.control.update({"debug-action" : inputdata.debugAction})
            self.control.update({"debug-CLI" : inputdata.debugCLI})
            self.control.update({"debug-notify" : inputdata.debugNotify})
            self.control.update({"debug-syslog" : inputdata.debugSyslog})
            self.control.update({"debug-fact" : inputdata.debugFact})
            self.control.update({"debug-config" : inputdata.debugConfig})
            self.control.update({"debug-parser" : inputdata.debugParser})
            self.control.update({"debug-file" : inputdata.debugFile})
            self.control.update({"debug-logging" : inputdata.debugLogging})
            self.control.update({"logging-null" : inputdata.loggingNull})
            self.control.update({"send-notification" : inputdata.sendNotification})
            self.control.update({"save-notifications" : inputdata.saveNotifications})
            self.control.update({"save-dict-facts" : inputdata.saveDictFacts})
            self.control.update({"max-notifications" : inputdata.maxNotifications})
            self.control.update({"send-before" : inputdata.sendBefore})
            self.control.update({"send-after" : inputdata.sendAfter})
            self.control.update({"send-clips" : inputdata.sendClips})
            self.control.update({"send-dict" : inputdata.sendDict})
            self.control.update({"syslog-address" : inputdata.syslogAddress})
            self.control.update({"syslog-port" : inputdata.syslogPort})
            self.control.update({"notify-index" : 1})
            self.control.update({"service-impact" : inputdata.service_impact})
            self.control.update({"use-case" : inputdata.useCase})
            self.control.update({"log-path" : inputdata.logPath})
            self.control.update({"single-notify" : inputdata.singleNotify})
            self.control.update({"fact-timer" : inputdata.factTimer})
            self.control.update({"timer-fact-name" : inputdata.factName})
            self.control.update({"notify-path" : inputdata.notifyPath})
            self.control.update({"local-path" : inputdata.localPath})
            self.control.update({"json-notify" : inputdata.jsonNotify})
            self.control.update({"retry-count" : inputdata.retryCount})
            self.control.update({"retry-time" : inputdata.retryTime})
            self.control.update({"startup-delay" : inputdata.startupDelay})
            try:
                self.control.update({"control-facts" : inputdata.controlFacts})
                self.control.update({"delete-control-facts" : inputdata.deleteControlFacts})
            except:
                self.control.update({"control-facts" : 0})
                self.control.update({"delete-control-facts" : 0})
    #
    # Add standby-active flag to control whether DDR should execute during the next
    # execution cycle or should be in "standby" mode, not perform FACT collection
    # and wait to standby-active set to cause full execution
            try:
                self.control.update({"standby-active" : inputdata.standbyActive})
            except:
                self.control.update({"standby-active" : 1})
    #
    # Set logging to null device if required
    #
            self.stdout_save = sys.stdout
            if self.control["logging-null"] == 1:
                f = open(os.devnull, 'w')
                sys.stdout = f
        except Exception as e:
            self.print_log('%%%% DDR Error: ddr-flags file missing content: ' + str(e))


    ###################################################################################################
    #
    # Check to see if device configuration information is available in /bootflash/guest-share/ddr-devices
    # If the file is present, update the "device-list" and "mgmt-device" with configurations from ddr-devices
    #
    ###################################################################################################

        try:
    #
    # if ddr-devices.yaml is in the local directory use the yaml file instead of ddr-devices
    #
            if os.path.exists('./ddr-devices.yaml'):
                with open('ddr-devices.yaml', 'r') as file:
                    devices = yaml.safe_load(file)
                self.print_log("**** DDR Notice: ddr-devices.yaml in local directory used to set device information")
                device_list = []
                device_instance = []
                device_dictionary = [] # Save each device definition dictionary entry to assert as an initial FACT
                for key, device in devices.items():
                    device_instance = list(device.values())
                    if key == "mgmt":
                        self.control["mgmt-device"] = device_instance
                    else:
                        device_list.append(device_instance)
                        device_dictionary.append(device) 
                self.control["device-list"] = device_list
    #
    # if ddr-devices.yaml is not in the local directory use ddr-devices
    #
            else:
                self.print_log("**** DDR Notice: ddr-devices in usecase directory used to set device information")
                fd = open(self.control["devices-file"])
                devicedata = imp.load_source('devicedata', self.control["devices-file"], fd)
                self.control.update({"device-list" : devicedata.device_list})
                self.control.update({"mgmt-device" : devicedata.mgmt_device})

        except Exception as e:
            self.print_log('\n%%%% DDR Error: DDR ddr-devices file read error: ' + str(e))
            sys.exit(0)
            
    ###########################################################################
    #
    # Read the FACTs definitions from ddr-facts
    #
    ###########################################################################
        try:
            fd = open(self.control["facts-file"])
            inputdata = imp.load_source('inputdata', self.control["facts-file"], fd)
            fd.close()
    #
    # Read the FACT collection instructions from the ddr-fact file
    #
            self.control.update({"initial-facts-list" : inputdata.initial_facts})
            self.control.update({"fact-list" : inputdata.fact_list})        
            self.control.update({"nc-fact-list" : inputdata.nc_fact_list})
            self.control.update({"show-fact-list" : inputdata.show_fact_list})
            self.control.update({"show-run-facts" : inputdata.showRunFacts})
            self.control.update({"show-parameter-fact-list" : inputdata.show_parameter_fact_list})
            self.control.update({"file-fact-list" : inputdata.file_fact_list})
            self.control.update({"decode-btrace-fact-list" : inputdata.decode_btrace_fact_list})
            self.control.update({"logging-trigger-list" : inputdata.logging_trigger_list})
            self.control.update({"fact-filter" : inputdata.fact_filter})
            self.control.update({"dict-filter" : inputdata.dict_filter})
            self.control.update({"clear-filter" : inputdata.clear_filter})
            self.control.update({"notification-triggers" : inputdata.notification_triggers})
            self.control.update({"syslog-triggers" : inputdata.syslog_triggers})
            self.control.update({"edit-configs" : inputdata.edit_configs})
    #
    # optional configuration content
    #
            try:
                self.control.update({"action-fact-list" : inputdata.action_fact_list})

                self.telem_list = inputdata.telem_list
                self.telemetry_config = inputdata.telemetry_config
                self.cli_cmd = inputdata.cli_cmd
                self.translation_dict = inputdata.translation_dict
                self.append_slot_dict = inputdata.append_slot_dict
            except: pass
        except Exception as e:
            self.print_log('\n%%%% DDR Error: ddr-facts file read error: ' + str(e))
            sys.exit(0)

        self.memory_use("**** DDR Memory: On Entry(kb): ", "entry")
    #
    # If this DDR instance is run on a platform with the 'cisco-ios-xe-ddr-control.yang" model
    # control configurations can be read from the device NETCONF datastore
    # DDR uses the device credentials contained in the FACTS file to connect to the device NETCONF instance
    #
        if self.control["model-control"] == 1:            
            self.get_ddr_control()

    ######################################################################################
    ######################################################################################
    #
    # If test-facts=0, fact data is collected from devices and device connections must
    # be established.  If test-facts=1, simulated facts are read from the ddr-tests file
    #
    ######################################################################################
    ######################################################################################
        if self.control["test-facts"] == 0:
    #
    # If required delay before attempting the first device connection
    #
            if self.control["startup-delay"] != 0:            
                self.print_log("**** DDR Notice: Delay Seconds before Connecting to Device: " + str(self.control["startup-delay"]))
                time.sleep(self.control["startup-delay"])
                self.print_log("**** DDR Notice: Startup Connect Delay Complete")
    #
    # Connect to device identified as the management device
    #
            retry_count = 1
            connect_success = False
            while retry_count <= self.control["retry-count"]:
    #
    #    Try multiple times to connect (may be delayed for DHCP lease)
    #    If using passwordless NETCONF the ip address will be 127.0.0.1 and connect is different
                try:
                    if self.control["mgmt-device"][0] == '127.0.0.1':
                    
                        self.netconf_connection = (ncclient.manager.connect(host=self.control["mgmt-device"][0], port=self.control["mgmt-device"][1],
                            username=self.control["mgmt-device"][2],
                            ssh_config=True,
                            hostkey_verify=False,
                            look_for_keys=False,
                            allow_agent=False,
                            timeout=self.control["nc-timeout"]))

                    else:
                        self.netconf_connection = (ncclient.manager.connect_ssh(host=self.control["mgmt-device"][0], port=self.control["mgmt-device"][1],
                            username=self.control["mgmt-device"][2],
                            password=self.control["mgmt-device"][3],
                            hostkey_verify=False,
                            look_for_keys=False,
                            allow_agent=False,
                            timeout=5))
    #
    # If connections successful break out of the retry loop
    #
                    connect_success = True
                    break
    #
    # If connection failed, retry after waiting "retry-time" seconds
    #
                except Exception as e:
                    timestamp =  datetime.now().strftime("%m-%d-%Y_%H:%M:%S.%f")
                
                    self.print_log("\n%%%% DDR Error: Unable to connect to management device: " + str(self.control["mgmt-device"][0]) + " retry: " + str(retry_count) + " at: " + str(timestamp) + " " + str(e))
                    retry_count = retry_count + 1
                    time.sleep(self.control["retry-time"])
    #
    # If device connections failed exit
    #
            if connect_success == False:
                timestamp =  datetime.now().strftime("%m-%d-%Y_%H:%M:%S.%f")
            
                self.print_log("\n%%%% DDR Error: Unable to connect management device: " + str(self.control["mgmt-device"][0]) + " at: " + str(timestamp))
                self.close_sessions()
                sys.exit(0)
            timestamp =  datetime.now().strftime("%m-%d-%Y_%H:%M:%S.%f")
        
            self.print_log("**** DDR Notice: Connected to management device: " + str(self.control["mgmt-device"][0]) + " at: " + str(timestamp))

    ################################################################################################
    ################################################################################################
    #
    # run-mode = 1 : Syslog receiver in DDR receives Syslog messages from the hosting device
    # and inspects each Syslog message to determine if the Syslog should trigger usecase execution.
    #
    # The syslog_triggers list in ddr-facts defines the Syslog messages that are used to trigger
    # execution.  The general form of syslog_triggers is:
    #
    #    syslog_triggers = [[[LIST_OF_STRINGS in message], [Facts to assert], True/False to assert facts]]
    #    syslog_triggers = [[['MGBL-SYS-5-CONFIG_I'], [], 'False']]
    #
    #
    ################################################################################################
    ################################################################################################
    #
            if (self.control["run-mode"] == 1):
                if self.control["syslog-address"] != "none":
                    try:

    ######################################################################
    #
    # Start the syslog server and use threading to recieve and queue
    # syslog messasges for processing
    #
    #####################################################################

                        t = threading.Thread(target=queue_syslog, args = (mesq, self.control["syslog-address"], self.control["syslog-port"]))
                        t.deamon = True
                        t.start()

                    except Exception as e:
                        self.print_log("\n%%%% DDR Error: Unable to start syslog receiver: " + str(e))   

    ################################################################################################
    ################################################################################################
    #
    # If run-mode requires waiting for notifications create NETCONF listener for snmpevents stream
    # Create a notification listener for run-mode
    #
            if (self.control["run-mode"] == 2):
                retry_count = 1
                connect_success = False
                while retry_count <= self.control["retry-count"]:
    #
    #    Try multiple times to connect for notifications (may be delayed for DHCP lease)
    #
                    try:
                        if self.control["mgmt-device"][0] == '127.0.0.1':

                            self.notify_conn = (ncclient.manager.connect(host=self.control["mgmt-device"][0], port=self.control["mgmt-device"][1],
                                username=self.control["mgmt-device"][2],
                                ssh_config=True,
                                hostkey_verify=False,
                                look_for_keys=False,
                                allow_agent=False,
                                timeout=self.control["nc-timeout"]))
                            self.notify_conn.async_mode = False
                            self.notify_conn.create_subscription(stream_name='snmpevents')

                        else:
                            self.notify_conn = (ncclient.manager.connect_ssh(host=self.control["mgmt-device"][0], port=self.control["mgmt-device"][1],
                                username=self.control["mgmt-device"][2],
                                password=self.control["mgmt-device"][3],
                                hostkey_verify=False,
                                look_for_keys=False,
                                allow_agent=False,
                                timeout=self.control["nc-timeout"]))
                            self.notify_conn.async_mode = False
                            self.notify_conn.create_subscription(stream_name='snmpevents')
    #
    # If connections successful break out of the retry loop
    #
                        connect_success = True
                        break
    #
    # If connection failed, retry after waiting "retry-time" seconds
    #
                    except Exception as e:
                        timestamp =  datetime.now().strftime("%m-%d-%Y_%H:%M:%S.%f")
                    
                        self.print_log("\n%%%% DDR Error: Unable to connect to notification stream from: " + str(self.control["mgmt-device"][0]) + " retry: " + str(retry_count) + " at: " + str(timestamp) + " " + str(e))
                        retry_count = retry_count + 1
                        time.sleep(self.control["retry-time"])
    #
    # If device connection failed exit
    #
                if connect_success == False:
                    timestamp =  datetime.now().strftime("%m-%d-%Y_%H:%M:%S.%f")
                
                    self.print_log("\n%%%% DDR Error: Unable to connect to notification stream from: " + str(self.control["mgmt-device"][0]) + " at: " + str(timestamp))
                    self.close_sessions()
                timestamp =  datetime.now().strftime("%m-%d-%Y_%H:%M:%S.%f")
            
                self.print_log("**** DDR Notice: Connected to notification stream from: " + str(self.control["mgmt-device"][0]) + " at: " + str(timestamp))

    ###########################################################################
    #
    # Connect to devices used to implement the use case
    # Create a list with entries for each device containing the
    # ncclient connection object.  The indexes into the 'device' list
    # are one of the parameters in configurations in the 'facts.txt' instructions
    # for collecting FACT information from devices
    # If control["test-facts"] is not set, connect to the devices
    #
    ###########################################################################
            retry_count = 1
            connect_success = False
            while retry_count <= self.control["retry-count"]:
            
                try:
                    self.device = []
                    for device_dat in self.control["device-list"]:
                        if device_dat[0] == '127.0.0.1':
                            self.device.append(ncclient.manager.connect(host=device_dat[0], port=device_dat[1],
                                        username=device_dat[2],
                                        ssh_config=True,
                                        hostkey_verify=False,
                                        look_for_keys=False,
                                        allow_agent=False,
                                        timeout=self.control["nc-timeout"]))

                        else:
                            self.device.append(ncclient.manager.connect_ssh(host=device_dat[0], port=device_dat[1],
                                        username=device_dat[2],
                                        password=device_dat[3],
                                        hostkey_verify=False,
                                        look_for_keys=False,
                                        allow_agent=False,
                                        timeout=self.control["nc-timeout"]))

    #
    # If connections successful break out of the retry loop
    #
                    connect_success = True
                    break
    #
    # If connection failed, retry after waiting "retry-time" seconds
    #
                except Exception as e:
                    timestamp =  datetime.now().strftime("%m-%d-%Y_%H:%M:%S.%f")
                
                    self.print_log("\n%%%% DDR Error: Unable to connect to device-list entry: " + str(device_dat[0]) + " retry: " + str(retry_count) + " at: " + str(timestamp) + " " + str(e))
                    retry_count = retry_count + 1
                    time.sleep(self.control["retry-time"])
    #
    # If device connections failed exit
    #
            if connect_success == False:
                timestamp =  datetime.now().strftime("%m-%d-%Y_%H:%M:%S.%f")
            
                self.print_log("\n%%%% DDR Error: Unable to connect to device-list entries at: " + str(timestamp))
                self.close_sessions()
                sys.exit(0)
            timestamp =  datetime.now().strftime("%m-%d-%Y_%H:%M:%S.%f")
        
            self.print_log("**** DDR Notice: Connected to all device-list entry devices at: " + str(timestamp))

    ################################################################################################
    ################################################################################################
    #
    # Read test facts to assert if 'test-facts' is configured
    # The list of list of facts will be available in self.test_data
    #
    ################################################################################################
    ################################################################################################

        self.test_index = 0
        if self.control["test-facts"] == 1:
            self.memory_use("**** DDR Memory: Before Loading Test Facts(kb): ", "before-test-facts")

            try:
                if not os.path.exists(self.control["test-file"]):
                    self.print_log('\n%%%% DDR Error: Test fact file does not exist')

                else:
                    with open(self.control["test-file"]) as file:
                        self.test_data = file.readlines()
                        self.test_data = [line.rstrip() for line in self.test_data]
                        if self.control["debug-fact"] == 1:
                            self.print_log("**** DDR Debug: test_data file content: \n" + str(self.test_data))

            except Exception as e:
                self.print_log('\n%%%% DDR Error: Test fact file read error: ' + str(e))
            self.memory_use("**** DDR Memory: After Loading Test Facts(kb): ", "after-test-facts")

    ################################################################################################
    ################################################################################################
    #
    # Read sim-facts to assert in rules if 'sim-facts = 1' is configured
    # The list of list of sim-facts will be available in self.sim_data
    #
    ################################################################################################
    ################################################################################################

        if self.control["sim-facts"] == 1:
            self.memory_use("**** DDR Memory: Before Loading Sim Facts(kb): ", "before-sim-facts")
            try:
                if not os.path.exists(self.control["sim-file"]):
                    self.print_log('\n%%%% DDR Error: sim fact file does not exist')

                else:
                    with open(self.control["sim-file"]) as file:
                        self.sim_data = file.readlines()
                        self.sim_data= [line.rstrip() for line in self.sim_data]
                        if self.control["debug-fact"] == 1:
                            self.print_log("**** DDR Debug: sim_data file content: \n" + str(self.sim_data))

                        fd.close()
            except Exception as e:
                self.print_log('\n%%%% DDR Error: Sim fact file read error: ' + str(e))
            self.memory_use("**** DDR Memory: After Loading Sim Facts(kb): ", "after-sim-facts")
    
    ###########################################################################################
    #
    # Initialize CLIPS environment
    #
    ###########################################################################################
        try:
            self.memory_use("**** DDR Memory: Before Creating CLIPs env(kb): ", "before-clips-env")
            self.env = clips.Environment()
            if self.control["debug-action"] == 1:
                self.print_log("**** DDR Debug: CLIPs Environment: " + str(self.env))

        except:
            self.print_log("\n%%%% DDR Error: Failed to clear inference engine")
            sys.exit(0)
        self.memory_use("**** DDR Memory: After Creating CLIPs env(kb): ", "after-clips-env")

    ###########################################################################################
    #
    # action_functions - Names of any python functions used as control["actions"] in CLIPS rules
    # The functions in the list must be defined before being referenced below
    # The functions could be defined in a library that is imported by the basic script.
    # As new action functions are created they are added to the library
    # dynamically load action functions
    ###########################################################################################
        try:
            action_functions = self.get_action_functions()
            for function in action_functions:
                if self.control["debug-action"] == 1:
                    self.print_log("**** DDR Debug: intialize action functions: " + str(function))
                self.env.define_function(function)
        except Exception as e:
            self.print_log("\n%%%% DDR Error: registering action_functions: " + str(e))
            self.close_sessions()
            sys.exit(0)
    #
    # Load the clips constructs file including rules deftemplates and deffacts
    #
        try:
            self.memory_use("**** DDR Memory: Before load CLIPS rules in KBytes: ", "before-rules-load")
            self.env.load(self.control["rules-file"])
            self.memory_use("**** DDR Memory: After load CLIPS rules in KBytes: ", "after-rules-load")

            self.print_log("**** DDE Notice: Inference Engine rules file loaded: " + self.control["rules-file"])
        except Exception as e:
            self.print_log("\n%%%% DDR Error: failed to load inference engine rules" + str(e))
            self.close_sessions()
            sys.exit(0)

        try:
            self.env.reset() # Initialize any "deffacts" defined in the ddr-rules CLIPS file
            self.print_log("**** DDR Notice: Inference Engine Reset")
        except:
            self.print_log("\n%%%% DDR Error: failed resetting inference engine")
            self.close_sessions()
            sys.exit(0)

    #
    # Assert initial facts defined in the FACTS configuration file
    #
        starttime = time.time()
        if self.control["initial-facts-list"] != []:
            self.get_initial_facts()
    #
    # Read and assert facts in the init-facts-file
    #
        if self.control["initial-facts"] == 1:
            self.memory_use("**** DDR Memory: After Loading Initial Facts(kb): ", "before-load-init-facts")

            try:
                fd = open(self.control["init-facts-file"], 'r')
                facts = fd.readlines()
                fd.close()
                self.memory_use("**** DDR Memory: After Loading Initial Facts(kb): ", "after-load-init-facts")

    #
    # Assert each initial fact
    #
                for fact in facts:
                    if self.control["debug-fact"] == 1:
                        self.print_log("**** DDR Debug: Assert init-file fact: " + str(fact))
                    try:
                        self.env.assert_string(str(fact))
                    except Exception as e:
                        self.print_log('\n%%%% DDR Error: init-file-fact assert error' + str(e) + " " + str(fact))
                        sys.exit(0)

            except Exception as e:
                self.print_log('\n%%%% DDR Error: init-file-fact read error: ' + str(e))
                sys.exit(0)
    #
    # Log time to load any inital facts
    #
        endtime = time.time()
        fact_runtime = endtime - starttime
        self.ddr_timing('**** DDR Time: to load and assert initial facts(ms): %8.3f', fact_runtime, "load-initial-facts")
        self.memory_use("**** DDR Memory: After Asserting Initial Facts(kb): ", "after-assert-init-facts")
    #
    # Assert FACTs for each device in the device_list
    # This must be applied after CLIPs is initialized
    #
        try:
            for device in device_dictionary:
                if self.control["debug-fact"] == 1:
                    self.print_log("\n**** DDR Debug: Assert FACT for each device: " + str(device))
                template = self.env.find_template("devices")
                template.assert_fact(**device)
        except Exception as e:
            if self.control["debug-fact"] == 1:
                self.print_log("%%%% DDR Exception: DDR initialization device_dictionary FACTs: " + str(e))
            pass


    ##############################################################################################
    #
    # set_ddr_runmode - 
    #  This function checks the facts file at the end of each execution loop to determine if the runninng
    #  mode of DDR should be changed and to change debug logging settings
    #
    #  To change these options, update the FACTs file on the device with new settings for the parameters
    #  At the end of the main execution loop, the FACT file will be read and these parameters updated
    # 
    #        standbyActive - 0/do not perform action, 1/perform normal DDR actions, 2/terminate DDR
    #        runMode - 0/run continously using the updatePeriod delay between cycles
    #        updatePeriod - Set the continuous running mode delay before the next cycle starts in ms
    #        debugCLI - 1/log CLI command processing
    #        debugSyslog - 1/log Syslog processing actions
    #        debugFact - 1/log FACT generation actions
    #        debugConfig - 1/generate debug logging for configuration actions               
    #
    ##############################################################################################
    def set_ddr_runmode(self):
    #
    # Read the facts file and update parameters that control DDR execution behavior
    #
        try:
    #
    # Read the FACTs definitions from the facts-file
    #
            fd = open(self.control["facts-file"])
            inputdata = imp.load_source('inputdata', self.control["facts-file"], fd)
            fd.close()
        except Exception as e:
            self.print_log('\n%%%% DDR Error: DDR Intialization files read error: ' + str(e))
            sys.exit(0)
    #
    # If this DDR instance control parameters will be loaded
    # from files set the 'control' dictionary parameters
    #
        try:
            self.control.update({"standby-active" : inputdata.standbyActive})
        except:
            self.control.update({"standby-active" : 1})
    #
    # Update any changes in other running control flags
    #            
        self.control.update({"debug-CLI" : inputdata.debugCLI})
        self.control.update({"debug-notify" : inputdata.debugNotify})
        self.control.update({"debug-syslog" : inputdata.debugSyslog})
        self.control.update({"debug-fact" : inputdata.debugFact})
        self.control.update({"debug-config" : inputdata.debugConfig})

    ##############################################################################################
    #
    # memory_use - Measure and display memory used by the DDR Python script
    #
    ##############################################################################################
    def memory_use(self, message, key):
        if self.control["show-memory"] == 1:
            try:
                usage = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
                self.print_log(message + str(usage))
                self.memory[key] = usage
            except:
                pass

    ##############################################################################################
    #
    # timing - Report execution time used by DDR Python script
    #
    ##############################################################################################
    def ddr_timing(self, message, runtime, key):
        if self.control["show-timing"] == 1:
            try:
                self.print_log(message%(runtime*1000))
                self.timing[key] = runtime*1000
            except:
                pass

    #########################################################################################
    #
    # Create a logging file
    #
    #########################################################################################
    def make_logger(self, path):

        logger = logging.getLogger('ddr Logger')
        logger.setLevel(logging.DEBUG)

        handler = logging.handlers.SysLogHandler('/var/log')
        handler_rot = RotatingFileHandler(path, maxBytes=10*1024*1024, backupCount=5)

        l_format = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
        handler.setFormatter(l_format)

        logger.addHandler(handler_rot)

        return logger

    ########################################################################################
    #
    # get_template_multifacts_protofact_index - For each element in the element_list substitute the element
    # value, for example the name of an interface, as the key in a template used to read
    # multiple facts from a device
    #
    ########################################################################################

    def get_template_multifacts_protofact_index(self, fact, hard_slot, hard_slot_value, device_index="none"):
      '''
      The test_get_template_multifacts_protofact function is used in Python scripts to test the implementation
      of FACT definitions used to translate the content of NETCONF get responses into CLIPs FACTs.
 
      The get_template_multifacts_protofact function is called from the main DDR loop or when the DDR "run_nc_fact"
      function is called from a CLIPs RULE.

      The FACT dictionary entry in the nc_fact_list section of the ddr-facts file contains:
 
         "fact_type": for NETCONF operations the multitemplate_protofact is used to generate FACTs from lists in get responses

         "assert_fact_for_each": identifies the list in the response for which a FACT is generated for each list entry.
                          In the example a FACT will be generated for each DyPeer-list entry

         "element_list": {"ip": ["204.1.1.1", "201.1.1.1"]} element_list filters out FACTs and only asserts a FACT if
                   in this example, the "ip" slot in the rule contains the one of the ip addresses shown in the list.  The element_list is a dictionary
                   so additional filtering can be specified where the dictionary element key, e.g. "ip" is used to select values
                   from the list associated with the key to verify that a listed value is also present in the data.
                   If "element_list" is not included in the FACT, no filtering is performed
                          
         path: This is the path used in the NETCONF request to the device.  Optionally parameters can be inserted into
          the get request template shown in the path entry.  The parameters are defined in the run_nc_fact and
          run_nc_fact_index functions called from CLIPs rules
          In the run_nc_fact example below the arguments are as follows:
          (run_nc_fact_index 3 ?device 1 ?edId none none)
          3 - Index in to the nc_fact_list in the ddr-facts file for the FACT definition to use
          ?device - A variable used to select which device to access
          1 - Indicates the number of parameters to insert into the "path" (0, 1, 2, or 3)
          ?epId - Variable that will be inserted into the "path" to replace {0} (note parameter 2 replaces {1} and 3 replaces {2})
          none - No value for this optional parameter
          
         protofact: This entry describes how to extract data from the get_response and where to insert the data
               into the FACT.  
               "template:" is the name of the deftemplate definition in the ddr-rules file
               The left-hand entry in "slots:" defines the slots or fields in the deftemplate that will recieve the data
               The right-hand entry defines how to extract the data from the get_response.
               In the definition below "nve": 'Ep-list/epId' selects the value of the epId in the Ep-list in the
               get_response and places the value in an nve-peer-list FACT in the nve slot.
               "primary-ip" and "admin-state" are also selected from the Ep-list.
               The next 3 slot definitions ip, state, mac, because there is no "list-name/" are used to create
               a FACT for each entry in the DyPeer-list.
               
         types: This second defines the type for each value that will be populated into the FACT.

      For the example get_response and fact_list entry below, the following FACTs are generated:

         (nve-peer-list (tname nve-peer-list) (device "n9k") (nve 1) (admin-state "enabled") (primary-ip "204.1.1.1") (peer-ip "201.1.1.1") (peer-state "Up") (peer-mac "00:00:00:00:00:00"))
         (nve-peer-list (tname nve-peer-list) (device "n9k") (nve 1) (admin-state "enabled") (primary-ip "204.1.1.1") (peer-ip "1.2.3.4") (peer-state "Down") (peer-mac "10:00:00:00:00:00"))
     nc_fact_list = [
     {"fact_type": "multitemplate_protofact", 
        "device_index": 0,
        "device": 'n9k-test-1',
        "assert_fact_for_each": 'DyPeer-list',
        "hardcoded_list": ["device"],
        "element_list": {"peer-ip": ["1.2.3.5", "201.1.1.1"], "peer-mac": ["00:00:00:00:00:00", "10:00:00:00:00:00"]},

     "path": '
      <System xmlns="http://cisco.com/ns/yang/cisco-nx-os-device">
        <eps-items>
          <epId-items>
            <Ep-list>
              <epId>{0}</epId>
              <adminSt/>
              <primaryIp/>
              <peers-items>
                <dy_peer-items>
                  <DyPeer-list>
                    <ip/>
                    <state/>
                    <mac/>
                  </DyPeer-list>
                </dy_peer-items>
              </peers-items>
            </Ep-list>
          </epId-items>
        </eps-items>
      </System>',
     "protofact": {"template": 'nve-peer-list',
              "slots": {"device": 'None',
                        "nve": 'Ep-list/epId',
                        "primary-ip": 'Ep-list/primaryIp',
                        "admin-state": 'Ep-list/adminSt',
                        "peer-ip": 'ip',
                        "peer-state": 'state',
                        "peer-mac": 'mac'
                       },
              "types": {"device": 'str',
                        "nve": 'int',
                        "primary-ip": 'str',
                        "admin-state": 'str',
                        "peer-ip": 'str',
                        "peer-state": 'str',
                        "peer-mac": 'str'
                        } 
        }
}
]
      '''
    #
    # If device_index = 'none' select the device id from the fact definition in ddr-facts
    # If not 0, select the device using device_index variable
    #
      try:
        if device_index == 'none':
            device_id = self.device[fact['device_index']]
            device_name = fact['device']
        else:
            device_info = self.control["device-list"][int(device_index)]
            device_id = self.device[int(device_index)]
            device_name = device_info[4]

        get_result = device_id.get(filter=('subtree', str(fact['path'])))
        if self.control["debug-fact"] == 1:
            self.print_log("\n**** DDR Debug: get_template_multifacts_protofact NETCONF get result: \n\n" + str(get_result) + "\n")

        instances = xml.dom.minidom.parseString(get_result.xml).getElementsByTagName(fact["assert_fact_for_each"])

        # The instances in the response that will be used to generate FACTs can be filtered
        # If the fact definition includes a dictionary key "filtered_instances" the following code includes
        # only instances where the value in the selected slot matches a valued in "filtered_instances"
        # For example: "element_list": {"peer-ip": ["1.2.3.5", "201.1.1.1"], "peer-mac": ["00:00:00:00:00:00", "10:00:00:00:00:00"]},
        # FACTs will only be generated if the FACT contains one of the listed peer-ip AND one of the listed peer-mac
        #
        if "element_list" in fact:
            filtered_instances = []
            for each in instances:
                found = False
                slots = fact['protofact']["slots"]
                for slot in slots:
                    if found: break
                    if ("hardcoded_list" in fact) and (slot in fact["hardcoded_list"]): continue
                    node_list = each.getElementsByTagName(slots[slot])
                    if node_list:
                        node = node_list[0]
                    elif "/" in slots[slot]:
                        node = self.get_upper_value(each, slots[slot])
                    else: continue
                    for category in fact["element_list"]:
                        if slot == category:
                            if self.control["debug-fact"] == 1:
                                self.print_log("\n**** DDR Debug: get_template_multifacts_protofact element_list: \n" + "Instance value and element_list: " + str(node.firstChild.nodeValue).replace(" ", "") + " " + str(fact["element_list"][category]) + "\n")
                            if (str(node.firstChild.nodeValue).replace(" ", "") in fact["element_list"][category]):
                                filtered_instances.append(each)
                                found = True
                                break
        else:
            filtered_instances = instances

        # Generate a FACT for each instance in the get_response
        # The protofact element in the FACT dictionary is copied and the slot values read from the get_response are
        # written into the copy of the protofact dictionary
        # For example, after execution the new "protofact" variable could contain:
        #
        #    protofact: {'template': 'nve-peer-list', 'slots': {'device': 'n9k', 'nve': 1, 'primary-ip': '204.1.1.1', 'admin-state': 'enabled', 'peer-ip': '201.1.1.1', 'peer-state': 'Up', 'peer-mac': '00:00:00:00:00:00'}
        #
        # The protofact dictionary generated below is used to assert the FACT into CLIPs
        #
        for each in filtered_instances:
            protofact = copy.deepcopy(fact['protofact'])
            slots = fact['protofact']["slots"] # list of slots (fields) that will have values assigned
            for slot in slots:
                if ("hardcoded_list" in fact) and (slot == fact["hardcoded_list"][0]):
                    if (hard_slot_value == None): # Get the slot value from the FACT definition
                        protofact["slots"][slot] = fact["device"]                
                        continue
                    else: # Get the slot value passed in by the function call 'hard_slot_value'
                        protofact["slots"][slot] = hard_slot_value                
                        continue

                # create a list XML nodes that for the "slot" in the FACT, for example "ip"
                # There will only be one element in the list
                # If the value has a / then a second lookup is required to get value 'Ep-list/primaryIp'

                try:
                    node_list = each.getElementsByTagName(slots[slot])
                    if node_list:
                        node = node_list[0]
                        value = node.firstChild.nodeValue
                    elif "/" in slots[slot]:
                        # for upper tags, node_list would have been empty above
                        node = self.get_upper_value(each, slots[slot])
                        value = node.firstChild.nodeValue
                    else: value = "nil" # if no value provided for the slot, set the value in the FACT to "nil"
                except Exception as e:
                    self.print_log("\n%%%% DDR Error: get_template_multifacts_protofact_index: slots, node_list " + str(slots[slot]) + " " + str(node_list) + " " + str(e))

                # set the type of value saved in the protofact dictionary using the types defined in the FACT section "types"
                
                if protofact["types"][slot] == "int":
                    protofact["slots"][slot] = int(value)
                elif protofact["types"][slot] == "flt":
                    protofact["slots"][slot] = float(value)
                else:
                    protofact["slots"][slot] = str(value).replace(" ", "_")

                if self.control["debug-fact"] == 1:
                    self.print_log("**** DDR Debug: slot: " + str(node))
                    self.print_log("**** DDR Debug: value: " + str(value))

            if self.control["debug-fact"] == 1:
                self.print_log("**** DDR Debug: get_template_multifacts_protofact_index: " + str(protofact))
    #
    # Assert the FACT in the CLIPs instance using the deftemplate "template"
    # and the FACT content (protofact) generated above.  For example:
    #    protofact: {'template': 'nve-peer-list', 'slots': {'device': 'n9k', 'nve': 1, 'primary-ip': '204.1.1.1', 'admin-state': 'enabled', 'peer-ip': '201.1.1.1', 'peer-state': 'Up', 'peer-mac': '00:00:00:00:00:00'}
    # CLIPs provides a function that asserts a FACT using the content of a dictionary
    #
                     
            self.assert_template_fact(protofact, hard_slot, hard_slot_value)
      except Exception as e:
          self.print_log("\n%%%% DDR Error: get_template_multifacts_protofact_index: " + str(e))
          
    ##############################################################################
    #
    # get_upper_value - Get the value for model nodes above the target nodes
    #                   These nodes are normally keys in a nested list above the
    #                   target nodes
    #
    ##############################################################################
    def get_upper_value(self, node, upper_tag_key_combo):
        if self.control["debug-fact"] == 1:
            self.print_log("**** DDR Debug: in get_upper_value node, tags: " +  str(node) + " " + str(upper_tag_key_combo))       
        tag_key_list = upper_tag_key_combo.split("/")
        upper_tag = tag_key_list[0]
        key = tag_key_list[1]
        cur_node = node
        while cur_node.tagName != upper_tag:
            try:
                cur_node = cur_node.parentNode
            except:
                self.print_log("\n%%%% DDR Error: Protofact could not find parent: " + str(upper_tag_key_combo))
                return
        return cur_node.getElementsByTagName(key)[0]

    def get_template_multifacts_index(self, fact, slot, slot_value, device_index):
        '''
            Use a NETCONF get operation to read operational or configuration model content.
            If the data is in a list, create a fact for each list entry optionally filtered using a list of key names.
            Extract identified leaf values for the response and insert into the facts

            {"fact_type": "multitemplate",
            "data": ["multitemplate", 0, "CAT9K-24", # fact type, index into device_list, device name
            """<interfaces xmlns='http://cisco.com/ns/yang/Cisco-IOS-XE-interfaces-oper'> # cut and paste from a YANG view of the model (suggest use YangSuite)
                    <interface>
                      <name/>
                      <statistics>
                        <in-errors/>
                        <in-crc-errors/>
                        <in-errors-64/>
                      </statistics>
                    </interface>
                  </interfaces>""",
            "interface-stats", "interface", #first element is fact template name, 2nd element is list name given by user
            ["name", "in-errors", "in-crc-errors", "in-errors-64"], # names of leafs that end in /> generated if /> at end
            [["name", "str"], ["in-errors", "int"], ["in-crc-errors", "int"], ["in-errors-64", "int"]], # names of the slots in the interface-stats fact generate deftemplate slot names with same values as leaf names
            [['fact-name1', 'value1'], ['fact-name2', 'value2']], # list of facts to assert when this template is executed
            ['GigabitEthernet1', 'GigabitEthernet2']] # list of key filters, only return data if key (name) matches a list entry
            }

            :param fact: List of parameters used to control fact collection
               [0] - Fact type
               [1] - device_list index for device to perform operation
               [2] - device name to include in generated fact
               [3] - RPC content for get operation to get the required leafs
               [4] - name of deftemplate for the fact that will be generated
               [5] - name of the 'key' leaf.  One fact will be generated for each instance of the key
               [6] - parameter names in the RPC which will be extracted and asserted as slots in the fact
               [7] - slot names in the deftemplate to receive the parameters extracted from the get result
               [8] - optional list of additional facts to assert when this template is executed
               [9] - list of key values to use as filters to generate facts only for keys that match the list, if empty [] generate facts for all keys
        
        Usage::

              get_template_multifacts_index(self, test_fact, device, 'CAT9K-24', 0)

            :raises none:

        '''

        try:
    #
    # If device_index = 'none' select the device id from the fact definition in ddr-facts
    # If not 0, select the device using device_index variable
    #
            if device_index == 'none':
                device_id = self.device[fact[1]]
                device_name = fact[2]
            else:
                device_info = self.control["device-list"][int(device_index)]
                device_id = self.device[int(device_index)]
                device_name = device_info[4]
            path =  fact[3]
            deftemplate = fact[4]
            key = fact[5]
            leafs = fact[6]  
            facts = fact[7]
            assert_list = fact[8] # list of facts to assert when this template is executed
            element_list = fact[9]
            totaltimestart = time.time()
        except Exception as e:
            self.print_log("\n%%%% DDR Error: Get template multifact_index test_fact error: \n" + str(fact) + "\n" + str(e))
       
        try:
            if self.control["debug-fact"] == 1:
                self.print_log("**** DDR Debug: get_fact parameters: \n" + str(path) + "\nKey: " + str(key) + " " + str(leafs) + " " + str(facts) + " " + str(assert_list) + " " + str(element_list))
    #
    # If there are facts that must be asserted for this call assert all facts in the required templates
    # The list contains entries of this form [[template_name, slot_name, slot_value]]
    #
            for assert_fact in assert_list:
                try:
                    self.env.assert_string("(" + str(assert_fact[0]) + " (" + str(assert_fact[1]) + " " + str(assert_fact[2]) + "))")
                except: pass #ignore case where FACT already exists
    #
    # get data for multiple instances
    # instances will be a list of the objects which match the "key"
    #
            get_result = device_id.get(filter=('subtree', str(path)))
            if self.control["debug-fact"] == 1:
                self.print_log("\n**** DDR Debug: get_template_multifact NETCONF get result: \n" + str(get_result))
            instances = xml.dom.minidom.parseString(get_result.xml).getElementsByTagName(key)
            if self.control["debug-fact"] == 1:
                self.print_log("**** DDR Debug: instances: " + str(instances))
    #
    # Filter entries in multitemplate list if a filter is specified to limit fact collection
    #
            if element_list != []:
                filtered_elements = []
                filtered_instances = []
                for each in instances:
                    instance = []
                    fact = facts[0]
                    if fact[1] == 'str':
                        for node in each.getElementsByTagName(fact[0]):
                            if str(node.firstChild.nodeValue).replace(" ", "") in element_list:
                                if self.control["debug-fact"] == 1:
                                    self.print_log("**** DDR Debug: Multitemplate instance in list: " + str(node.firstChild.nodeValue))
                                filtered_elements.append(str(node.firstChild.nodeValue).replace(" ", ""))
                                filtered_instances.append(each)
                    if fact[1] == 'int':
                        for node in each.getElementsByTagName(fact[0]):
                            if int(node.firstChild.nodeValue) in element_list:
                                if self.control["debug-fact"] == 1:
                                    self.print_log("**** DDR Debug: Multitemplate instance in list: " + str(node.firstChild.nodeValue))
                                filtered_elements.append(int(node.firstChild.nodeValue))
                                filtered_instances.append(each)
                element_list = filtered_elements
                instances = filtered_instances

            instance_list = []
            for each in instances:
                if self.control["debug-fact"] == 1:
                    self.print_log("**** DDR Debug nodes: " + str(each.childNodes))
                instance = []
                for leaf in leafs:
                    if self.control["debug-fact"] == 1:
                        self.print_log("**** DDR Debug: slot: " + str(leaf))
                        self.print_log("**** DDR Debug: value: " + str(each.getElementsByTagName(leaf)))
    #BUGFIX 2.7.23
    # If no value is returned for the get request the value is []
    # A value of 'nil' must be filled in if no value is returned from the get request
    #
                    if len(each.getElementsByTagName(leaf)) == 0:
                        instance.append('nil')
                        if self.control["debug-fact"] == 1:
                            self.print_log("**** DDR Debug get_template_fact node: " + 'nil')
                    else:
                        for node in each.getElementsByTagName(leaf):
                            if self.control["debug-fact"] == 1:
                                self.print_log("**** DDR Debug get_template_fact node: " + str(node.firstChild.nodeValue))
                            instance.append(node.firstChild.nodeValue)
                if instance != []:
                    instance_list.append(instance)
                instance = []
    #
    # For each instance in the instance_list assert the information read as facts
    # Create a python dictionary with each key:value pair where the key is the slot
    # and value is the slot value
    # Template facts may specify the type of the value stored into the slot
    #  [["five-seconds", "int"], ["one-minute", "int"], ["five-minutes", "int"]]
    #
            for each in instance_list:
                template = self.env.find_template(str(deftemplate))
                fact1 = {}
                fact1["device"] = str(device_name)
                j = 0

                try:
                    for fact in facts:
                        if len(fact) == 2:
                            if fact[1] == "int":
                                fact1[str(fact[0])] = int(each[j])
                            elif fact[1] == "str":
                                fact1[str(fact[0])] = str(each[j]).replace(" ", "_")
                            elif fact[1] == "flt":
                                fact1[str(fact[0])] = float(each[j])
                            else:
                                self.print_log("%%%% DDR Exception: Invalid fact type " + str(fact))
                                break

                            j = j + 1
                        
                        else:
                            if isinstance(each[j], int):
                                fact1[str(fact)] = int(each[j])
                            else:
                                fact1[str(fact)] = str(each[j]).replace(" ", "_")
                                j = j + 1
                except Exception as e:
                    self.print_log("\n%%%% DDR Error: get_template_multifacts fact generation error: " + str(fact))                      
    #
    # Assert the FACT defined by a Python dictionary
    #
                if self.control["debug-fact"] == 1:
                    self.print_log("**** DDR Debug: get_template_multifacts_index fact to assert: " + str(template) + " " + str(fact1))
                   
                try:
                    template.assert_fact(**fact1)
                except Exception as e:
                    if self.control["debug-fact"] == 1:
                        self.print_log("%%%% DDR Exception: get_template_multifacts_index assert_fact: " + str(e))
                    pass

        except Exception as e:
            self.print_log("\n%%%% DDR Error: get_template_multifacts_index error: " + str(e))
            self.print_log("\n%%%% DDR Error: No value found in get_template_multifacts_index: " + str(fact1))
        return

##############################################################################
#
#
# show_and_assert_fact -
#            Execute a show command, parse the show command output,
#            and assert FACTS based on the output
#            Support using ssh for device access (for offbox use cases) or use the
#            Python cli package.
#            Function is backward compatible with older FACT files that do not have the
#            the "access-method" dictionary entry in the FACT definition
#
#############################################################################
    def show_and_assert_fact(self, fact, device_index):
    #
    # Determine if ssh should be used to access device
    # Execute in try clause for backward compatibility with older FACT files
    #
        try:
            access_type = self.control["device-list"][int(device_index)][5]
            if access_type == 'ssh':
                try:
                    device_info = self.control["device-list"][int(device_index)]
                    device_address = str(device_info[0])
                    device_user = str(device_info[2])
                    device_pass = str(device_info[3])
                except Exception as e:
                    self.print_log("\n%%%% DDR Error: Exception in ssh show_and_assert_fact invalid device definitions: " + str(e))
                    return
                
                options = '-q -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oPubkeyAuthentication=no'
    #
    # Select the device to use
    #
                ssh_cmd = 'ssh %s@%s %s "%s"' % (device_user, device_address, options, str(fact["command"]))
                if self.control["debug-fact"] == 1:
                    self.print_log("**** DDR Debug: run_show_and_assert SSH command: " + str(ssh_cmd) + "\n")
                try:
                    child = pexpect.spawn(ssh_cmd, timeout=15, encoding='utf-8')
                    child.delaybeforesend = None
                    if self.control["debug-CLI"] == 1:
                        child.logfile = sys.stdout
                    child.expect(['\r\nPassword: ', '\r\npassword: ', 'Password: ', 'password: '])
                    child.sendline(device_pass)
                    child.expect(pexpect.EOF)
                    response = child.before
                    if self.control["debug-fact"] == 1:
                        self.print_log("**** DDR Debug: run_show_and_assert SSH execution response: " + str(response))
                except Exception as e:
                    child.close()
                    self.print_log("\n%%%% DDR ERROR: show_and_assert ssh or timeout Error: " + str(ssh_cmd) + "\n")
                    return

                child.close()
    #
    # process response and generate FACTs
    #
                try:
                    if "error" in response:
                        self.print_log("\n%%%% DDR Error: error in ssh show_and_assert show command response")
                        return
                    parser = genie_str_to_class(fact["genie_parser"])
                    if parser == None:
                        self.print_log("\n%%%% DDR Error: Exception in ssh show_and_assert_fact: parser not found: \n" + str(fact["genie_parser"]) + " \n")
                        return
    #
    # Parse data if the parser is available
    #
                    try:
                        parsed_genie_output = parser.parse(output=response)
                    except Exception as e:
                        self.print_log("\n%%%% DDR Error: Exception in ssh show_and_assert_fact: parser not found: " + str(fact["genie_parser"]))
                        return
                    if self.control["debug-parser"] == 1:
                        self.print_log("\n*** DDR Debug: parsed genie output: " + str(parsed_genie_output))
                    if parsed_genie_output == {}:
                        self.print_log("\n%%%% DDR Error: show_and_assert_fact no parser output")
                        return
                    sub_dictionary_list = self.find(fact["assert_fact_for_each_item_in"], parsed_genie_output)
                    import copy
                    for sub_dictionary in sub_dictionary_list:
                        for item in sub_dictionary:
                            protofact = copy.deepcopy(fact["protofact"])
                            for slot in protofact["slots"]:
                                value = protofact["slots"][slot]
	#
	# insert the device name into the fact
	#
                                if value == "device":
                                    protofact["slots"][slot] = value.replace("device", self.control["device-list"][int(device_index)][4])
                                elif type(value) == str and "$" in value:
                                    protofact["slots"][slot] = value.replace("$", item)
                            self.assert_template_fact(protofact, 'device', self.control["device-list"][int(device_index)][4], sub_dictionary)
                    return
                except Exception as e:
                    self.print_log("\n%%%% DDR Error: Exception in ssh show_and_assert_fact response processing: " + str(e))
                    return


            elif access_type == 'cli':
                try:
                    response = cli.cli(str(fact["command"]), "30")
                    if self.control["debug-fact"] == 1:
                        self.print_log("**** DDR Debug: show_and_assert cli command result \n" + str(response))
                    if "error" in response:
                        self.print_log("\n%%%% DDR Error: show_and_assert show command response")
                        return

                    parser = genie_str_to_class(fact["genie_parser"])
                    if parser == None:
                        self.print_log("\n%%%% DDR Error: Exception in show_and_assert_fact: parser not found: \n" + str(fact["genie_parser"]) + " \n")
                        return
    #
    # convert show command response to parsed dictionary
    #
    #
    # Parse data if the parser is available
    #
                    try:
                        parsed_genie_output = parser.parse(output=response)
                    except Exception as e:
                       self.print_log("\n%%%% DDR Error: Exception in show_and_assert_fact: parser not found: " + str(fact["genie_parser"]))
                       return

                    if self.control["debug-parser"] == 1:
                        self.print_log("\n*** DDR Debug: parsed genie output: " + str(parsed_genie_output))
                    if parsed_genie_output == {}:
                        self.print_log("\n%%%% DDR Error: show_and_assert_fact no parser output")
                        return

                    sub_dictionary_list = self.find(fact["assert_fact_for_each_item_in"], parsed_genie_output)
                    import copy
                    for sub_dictionary in sub_dictionary_list:
                        for item in sub_dictionary:
                            protofact = copy.deepcopy(fact["protofact"])
                            for slot in protofact["slots"]:
                                value = protofact["slots"][slot]
	#
	# insert the device name into the fact
	#
                                if value == "device":
                                    protofact["slots"][slot] = value.replace("device", str(self.control["device-list"][int(device_index)][4]))
                                elif type(value) == str and "$" in value:
                                    protofact["slots"][slot] = value.replace("$", item)
                            self.assert_template_fact(protofact, 'device', str(self.control["device-list"][int(device_index)][4]), sub_dictionary)
                    return
                except Exception as e:
                    self.print_log("\n%%%% DDR Error: Exception in cli show_and_assert_fact: " + str(e))
                    return
    #
    # If "access-method" not supported, return error
    #
            else:
                self.print_log("\n%%%% DDR Error: Exception in show_and_assert_fact: Invalid access-method")
                return


    #
    # catch exception if using old FACT file without the "access-method" dictionary element
    # Process show_and_assert using the python cli package
    #
        except:
            try:
                response = cli.cli(str(fact["command"]), "30")
                if self.control["debug-fact"] == 1:
                    self.print_log("\n*** DDR Debug: show_and_assert_fact cli response: \n" + str(response))
                if "error" in response:
                    self.print_log("\n%%%% DDR Error: show_and_assert show command response")
                    return

                parser = genie_str_to_class(fact["genie_parser"])
                if parser == None:
                    self.print_log("\n%%%% DDR Error: Exception in show_and_assert_fact: parser not found: \n" + str(fact["genie_parser"]) + " \n")
                    return
    #
    # Parse data if the parser is available
    #
                try:
                    parsed_genie_output = parser.parse(output=response)
                except Exception as e:
                    self.print_log("\n%%%% DDR Error: pException in show_and_assert_fact parser not found: " + str(fact["genie_parser"]))
                    return
                     
                if self.control["debug-parser"] == 1:
                    self.print_log("\n*** DDR Debug: show_and_assert_fact parsed genie output: " + str(parsed_genie_output))
                if parsed_genie_output == {}:
                    self.print_log("\n%%%% DDR Error: show_and_assert_fact no parser output")
                    return

                sub_dictionary_list = self.find(fact["assert_fact_for_each_item_in"], parsed_genie_output)
                import copy
                for sub_dictionary in sub_dictionary_list:
                    for item in sub_dictionary:
                        protofact = copy.deepcopy(fact["protofact"])
                        for slot in protofact["slots"]:
                            value = protofact["slots"][slot]
	#
	# insert the device name into the fact
	#
                            if value == "device":
                                protofact["slots"][slot] = value.repla("device", str(self.control["device-list"][int(device_index)][4]))
                            elif type(value) == str and "$" in value:
                                protofact["slots"][slot] = value.replace("$", item)
                        self.assert_template_fact(protofact, 'device', str(self.control["device-list"][int(device_index)][4]), sub_dictionary)
                return
            except Exception as e:
                self.print_log("\n%%%% DDR Error: Exception in show_and_assert_fact: " + str(e))
                return

##############################################################################
#
# find - return nested dictionary value given a dictionary (j) and a string
#		 element (element) in the form of "Garden.Flowers.White"
#
#############################################################################
    def find(self, element, j):
        if element == "":
            return j
        keys = element.split('+')
        rv = j
        for i, key in enumerate(keys):
            if key == '*':
                new_list = []
                new_keys = copy.deepcopy(keys[i+1:]) # all the keys past the * entry
                for entry in rv: # for each entry in the * dictionary
                    new_rv = copy.deepcopy(rv[entry])
                    for new_key in new_keys:
                        new_rv = new_rv[new_key]
                    for e in new_rv:
                        new_rv[e]["upper_value"] = entry
                    new_list.append(new_rv)
                return new_list
            else:
    # normal stepping through dictionary
                rv = rv[key]
        return [rv]

##############################################################################
#
# assert_template_fact - given a protofact, assert the fact into the clips system
#
# This function generates a python dictionary from the FACT data
# The python dictionary contains all of the slots that are applied to a FACT template
# The clipspy template.assert_fact method creates a CLIPs FACT using a python dictionary produced by this function
#
#############################################################################
    def assert_template_fact(self, protofact, add_slot, slot_value, sub_dictionary=None):
        try:
            template = self.env.find_template(protofact["template"])
            fact1 = {}
            for slot, value in protofact["slots"].items():
    #
    # If the "value" is in a subdirectory, look up the final value in the sub_dictionary
    #
                if type(value) is str and "+" in value:
                    try:
                        value = self.find(value, sub_dictionary)[0]
                    except Exception as e:
                        if self.control["debug-fact"] == 1:
                            self.print_log("**** DDR Debug: assert_template_fact: No value found for slot: " + str(e))
                        continue
    #
    # Use "types" in the protofact to determine the type to assert in the FACT
    #
                try:
                    if protofact["types"][slot] == "int":
                        fact1[slot] = int(value)
                    elif protofact["types"][slot] == "flt":
                        fact1[slot] = float(value)
                    else:
                        fact1[slot] = str(value).replace(" ", "_")
                except Exception as e:
                    self.print_log("\n%%%% DDR Error: Exception assert_template_fact: type error: " +  str(protofact["template"]) + " slot: "  + str(slot) + " " + str(e))
    # 
    # Append extra "slot" if there is a slot and slot_value passed in to the function
    # The added slot will always be type string
    #  
            if add_slot != 'none':
                fact1[add_slot] = str(slot_value)                
    #
    # Assert the FACT defined by a Python dictionary
    #
            try:
                if self.control["debug-fact"] == 1:
                    self.print_log("**** DDR Debug: assert_template_fact: " + str(protofact["template"]) + " " + str(fact1))
                template.assert_fact(**fact1)
            except Exception as e:
                if self.control["debug-fact"] == 1:
                    self.print_log("%%%% DDR Exception: assert_template_fact: " + str(e))
                pass
        except Exception as e:
            self.print_log("\n%%%% DDR Error: Exception assert_template_fact: " + str(e))

    ##############################################################################
    #
    # assert_syslog_fact - Convert the syslog message passed in to a
    #            to a FACT and assert in CLIPS
    #
    # The device syslog message has this form:
    # XR: RP/0/RSP0/CPU0:Apr 26 20:30:07.567 UTC: ifmgr[257]:
    #                    %PKT_INFRA-LINK-3-UPDOWN : Interface Loopback5, changed state to Down
    #
    #    The syslog message fact has this form:
    #       (deftemplate syslog-message
    #         (slot device)
    #         (slot source)
    #         (slot date)
    #         (slot time)
    #         (slot component)
    #         (slot syslog)
    #         (slot content)
    #       )
    #
    #############################################################################
    def assert_syslog_fact(self, fact):
        try:
            if self.control["mgmt-device"][7] == 'XR':
                parser = genie_str_to_class("ParseXRSyslogMessage")
                if parser == None:
                    self.print_log("\n%%%% DDR Error: Exception in assert_syslog_fact: parser not found: \n" + str(fact["genie_parser"]) + " \n")
                    return
    #
    # Parse data if the parser is available
    #
                try:
                    parsed_genie_output = parser.parse(output=response)
                except Exception as e:
                    self.print_log("\n%%%% DDR Error: Exception in assert_syslog_fact: parser not found: " + str(fact["genie_parser"]))
                    return
                    
                if self.control["debug-parser"] == 1:
                    self.print_log("**** DDR Debug: assert_syslog_fact result: \n\n" + str(parsed_genie_output) + "\n")

            elif self.control["mgmt-device"][7] == 'XE':
                parser = genie_str_to_class("ParseXESyslogMessage")
                if parser == None:
                    self.print_log("\n%%%% DDR Error: Exception in assert_syslog_fact: parser not found: \n" + str(fact["genie_parser"]) + " \n")
                    return

    #
    # Parse data if the parser is available
    #
                try:
                    parsed_genie_output = parser.parse(output=response)
                except Exception as e:
                    self.print_log("\n%%%% DDR Error: Exception in assert_syslog_fact: parser not found: " + str(fact["genie_parser"]))
                    return
                    
                if self.control["debug-parser"] == 1:
                    self.print_log("**** DDR Debug: assert_syslog_fact result: \n\n" + str(parsed_genie_output) + "\n")

            else:
                self.print_log("\n%%%% DDR Error: Exception in assert_syslog_fact: Invalid device type in mgmt-device: \n" + str(fact["genie_parser"]) + " \n")
                return
                                
            if self.control["debug-syslog"] == 1:
                self.print_log("*** DDR Debug: parsed Syslog output: " + str(parsed_genie_output))
            sub_dictionary = parsed_genie_output["syslog-message"]
            assert_fact = "(syslog-message (device " + str(self.control["mgmt-device"][4]) + ")"

            for key, value in sub_dictionary.items():
                if self.control["debug-syslog"] == 1:
                    self.print_log("**** DDR Debug: key,value: " + str(key) + " " + str(value))
                if self.control["mgmt-device"][5] == 'XE':
                    if key == 'source': value = self.control["mgmt-device"][4]
                assert_fact = assert_fact + " (" + str(key) + " " + str(value) + ")"

            assert_fact = assert_fact + "))"
            if self.control["debug-syslog"] == 1:
                self.print_log("*** DDR Debug: assert_syslog_fact: " + str(assert_fact))
    #
    # Assert the syslog FACT
    #
            try:
                self.env.assert_string(assert_fact)
            except: pass #ignore case where FACT already exists
            return
        except Exception as e:
            self.print_log("\n%%%% DDR Error: Exception assert_syslog_fact: " + str(e))
            return

    ##############################################################################
    #
    # assert_rfc5277_fact - Convert the RFC5277 notification message generated for a Syslog message 
    #            to a FACT and assert in CLIPS
    #
    # The device syslog message has this form:
    # XR: RP/0/RSP0/CPU0:Apr 26 20:30:07.567 UTC: ifmgr[257]:
    #                    %PKT_INFRA-LINK-3-UPDOWN : Interface Loopback5, changed state to Down
    #
    #    The syslog message fact has this form:
    #       (deftemplate rfc5277-message
    #         (slot device)
    #         (slot source)
    #         (slot date)
    #         (slot time)
    #         (slot component)
    #         (slot syslog)
    #         (slot content)
    #       )
    #
    #############################################################################
    def assert_rfc5277_fact(self, fact):
        try:
            if self.control["mgmt-device"][5] == 'XR':
                parser = genie_str_to_class("ParseRFC5277Message")
                if parser == None:
                    self.print_log("\n%%%% DDR Error: Exception in assert_rfc5277_fact: parser not found: \n" + str(fact["genie_parser"]) + " \n")
                    return

                parsed_genie_output = parser.rfc5277(message=fact)
                if self.control["debug-parser"] == 1:
                    self.print_log("**** DDR Debug: assert_rfc5277_fact result: \n\n" + str(parsed_genie_output) + "\n")

            if self.control["mgmt-device"][5] == 'XE':
                parser = genie_str_to_class("ParseRFC5277Message")
                if parser == None:
                    self.print_log("\n%%%% DDR Error: Exception in assert_rfc5277_fact: parser not found: \n" + str(fact["genie_parser"]) + " \n")
                    return

                parsed_genie_output = parser.rfc5277(message=fact)
                if self.control["debug-parser"] == 1:
                    self.print_log("**** DDR Debug: assert_rfc5277_fact result: \n\n" + str(parsed_genie_output) + "\n")

            if self.control["debug-notify"] == 1:
                self.print_log("*** DDR Debug: parsed RFC5277 output: " + str(parsed_genie_output))
            sub_dictionary = parsed_genie_output["rfc5277-message"]
            assert_fact = "(rfc5277-message (device " + str(self.control["mgmt-device"][4]) + ")"

            for key, value in sub_dictionary.items():
                if self.control["debug-notify"] == 1:
                    self.print_log("**** DDR Debug: key,value: " + str(key) + " " + str(value))
                if self.control["mgmt-device"][5] == 'XE':
                    if key == 'source': value = self.control["mgmt-device"][4]
                if self.control["mgmt-device"][5] == 'XR':
                    if key == 'source': value = self.control["mgmt-device"][4]
                assert_fact = assert_fact + " (" + str(key) + " " + str(value) + ")"

            assert_fact = assert_fact + "))"
            if self.control["debug-notify"] == 1:
                self.print_log("*** DDR Debug: assert_rfc5277_fact: " + str(assert_fact))
    #
    # Assert the syslog FACT
    #
            try:
                self.env.assert_string(assert_fact)
            except: pass #ignore case where FACT already exists
            return
        except Exception as e:
            self.print_log("\n%%%% DDR Error: Exception assert_rfc5277_fact: " + str(e))
            return

    ##############################################################################
    #
    # assert_statistics_fact
    #
    #############################################################################
    def assert_statistics_fact(self, template_name, statistics):
        try:
            template = self.env.find_template(template_name)
            fact = {}
            for key, value in statistics.items():
                fact[key] = value

            if self.control["debug-fact"] == 1:
                self.print_log("**** DDR Debug: assert_statistics_fact: " + str(fact))
            try:
                template.assert_fact(**fact)
            except Exception as e:
                if self.control["debug-fact"] == 1:
                    self.print_log("**** DDR Exception: assert_statistics_fact: " + str(e) + " " + str(fact))
                pass
        except Exception as e:
            self.print_log("\n%%%% DDR Error: Exception assert_statistics_fact: " + str(e))

    ##############################################################################
    #
    #  print_clips_info - Print facts and rules from main loop
    #
    #############################################################################
    def print_clips_info(self):
        try:
            if self.control["show-facts"] == 1:
                if self.control["fact-filter"] == ['all']:
                    self.print_log("\n####################### FACTs(all) Main - CLIPS Format ########################\n")
                    for fact in self.env.facts(): self.print_log(fact)

                elif self.control["fact-filter"] != ['none']:
                    self.print_log("\n####################### FACTs(filtered)  Main - CLIPS Format ########################\n")
                    for fact in self.env.facts():
                        for ffilter in self.control["fact-filter"]:
                            if str(ffilter) in str(fact): 
                                self.print_log(fact)

            if self.control["show-rules"] == 1:
                self.print_log("\n####################### DDR Activated RULES Main ########################\n")
                for item in self.env.activations(): self.print_log(item)

#
# Show all facts in dictionary encoded format without filtering if dict_filter specifies all FACTS
#
            if self.control["show-dict"] == 1 and self.control["dict-filter"] == ['all']:
                self.print_log("\n####################### FACTs(all) Main - Python Dictionary Format ########################\n")
                for fact in self.env.facts():
                    try:
                        cleanfact = str(fact).split('  ')[-1]
                        templatename = cleanfact.split('(')
                        template = templatename[1].lstrip('(')
                        tempdict = "{'template': '" + str(template) + "', "
                        strfact = str(dict(fact))
                        stripquotes = strfact.split('  ')[-1].strip('"').lstrip("{")
                        finalfact = tempdict + stripquotes
                        self.print_log(finalfact.strip('"'))
                    except Exception as e:
                        self.print_log("\n%%%% DDR Error: Print Dictionary exception: " + str(e))
                        pass
#
# Show dictionary encoded facts filtered by the dict_filter list unless 'none' is specified in the filter
#
            elif self.control["show-dict"] == 1 and self.control["dict-filter"] != ['none']:
                self.print_log("\n####################### FACTs(filtered) Main - Python Dictionary Format ########################\n")
                for fact in self.env.facts():
                    try:
                        cleanfact = str(fact).split('  ')[-1]
                        templatename = cleanfact.split('(')
                        template = templatename[1].lstrip('(')
                        for dfilter in self.control["dict-filter"]:
                            if str(dfilter) in template: 
                                tempdict = "{'template': '" + str(template) + "', "
                                strfact = str(dict(fact))
                                stripquotes = strfact.split('  ')[-1].strip('"').lstrip("{")
                                finalfact = tempdict + stripquotes
                                self.print_log(finalfact.strip('"'))
                    except Exception as e:
                        self.print_log("\n%%%% DDR Error: Print Dictionary exception: " + str(e))
                        pass

        except Exception as e:
            self.print_log("\n%%%% DDR Error: Exception print_clips_info: " + str(e))

    ##############################################################################
    #
    #  print_clips_info_run - Print facts and rules from run_ functions
    #
    #############################################################################
    def print_clips_info_run(self, when_run):
        try:
            if self.control["show-run-facts"] == 1:
                if self.control["fact-filter"] == ['all']:
                    self.print_log("\n####################### FACTs(all) Run Action " + str(when_run) +"- CLIPS Format ########################\n")
                    for fact in self.env.facts(): self.print_log(fact)

                elif self.control["fact-filter"] != ['none']:
                    self.print_log("\n####################### FACTs(filtered) Run Action " + str(when_run) +"- CLIPS Format ########################\n")
                    for fact in self.env.facts():
                        for ffilter in self.control["fact-filter"]:
                            if str(ffilter) in str(fact): 
                                self.print_log(fact)

            if self.control["show-rules"] == 1:
                self.print_log("\n####################### DDR Activated RULES Run Action ########################\n")
                for item in self.env.activations(): self.print_log(item)

#
# Show all facts in dictionary encoded format without filtering if dict_filter specifies all FACTS
#
            if self.control["show-dict"] == 1 and self.control["dict-filter"] == ['all']:
                self.print_log("\n####################### FACTs(all) Run Action - Python Dictionary Format ########################\n")
                for fact in self.env.facts():
                    try:
                        cleanfact = str(fact).split('  ')[-1]
                        templatename = cleanfact.split('(')
                        template = templatename[1].lstrip('(')
                        tempdict = "{'template': '" + str(template) + "', "
                        strfact = str(dict(fact))
                        stripquotes = strfact.split('  ')[-1].strip('"').lstrip("{")
                        finalfact = tempdict + stripquotes
                        self.print_log(finalfact.strip('"'))
                    except Exception as e:
                        self.print_log("\n%%%% DDR Error: Print Dictionary exception: " + str(e))
                        pass
#
# Send dictionary encoded facts filtered by the dict_filter list unless 'none' is specified in the filter
#
            elif self.control["show-dict"] == 1 and self.control["dict-filter"] != ['none']:
                self.print_log("\n####################### FACTs(filtered) Run Action - Python Dictionary Format ########################\n")
                for fact in self.env.facts():
                    try:
                        cleanfact = str(fact).split('  ')[-1]
                        templatename = cleanfact.split('(')
                        template = templatename[1].lstrip('(')
                        for dfilter in self.control["dict-filter"]:
                            if str(dfilter) in template: 
                                tempdict = "{'template': '" + str(template) + "', "
                                strfact = str(dict(fact))
                                stripquotes = strfact.split('  ')[-1].strip('"').lstrip("{")
                                finalfact = tempdict + stripquotes
                                self.print_log(finalfact.strip('"'))
                    except Exception as e:
                        self.print_log("\n%%%% DDR Error: Print Dictionary exception: " + str(e))
                        pass

        except Exception as e:
            self.print_log("\n%%%% DDR Error: Exception print_clips_info: " + str(e))

    ##############################################################################
    #
    #  send_clips_info - Send filtered facts and rules in the service impact notification
    #
    #############################################################################
    def send_clips_info(self):
        try:
            if self.control["send-clips"] == 1:
                if (self.control["fact-filter"] == ['all'] or self.control["fact-filter"] == []):
                    for fact in self.env.facts(): self.clips_facts = self.clips_facts + "      <cfact>" + str(fact) + "</cfact>\n"

                elif self.control["fact-filter"] != ['none']:
                    for fact in self.env.facts():
                        for ffilter in self.control["fact-filter"]:
                            if str(ffilter) in str(fact): 
                                self.clips_facts = self.clips_facts + "      <cfact>" + str(fact) + "</cfact>\n"
#
# Send all facts in dictionary encoded format without filtering if dict_filter specifies 'all'
#
            if self.control["send-dict"] == 1 and (self.control["dict-filter"] == ['all'] or self.control["dict-filter"] == []):
                for fact in self.env.facts():
                    try:
                        cleanfact = str(fact).split('  ')[-1]
                        templatename = cleanfact.split('(')
                        template = templatename[1].lstrip('(')
                        tempdict = "{'template': '" + str(template) + "', "
                        strfact = str(dict(fact))
                        stripquotes = strfact.split('  ')[-1].strip('"').lstrip("{")
                        finalfact = tempdict + stripquotes
                        self.dict_facts = self.dict_facts + "      <dfact>" + json.dumps(finalfact).strip('"') + "</dfact>\n"
                    except Exception as e:
                        if str(fact).find("retract") == -1:
                            self.print_log("\n%%%% DDR Error: Send Dictionary exception: " + str(fact))
                        pass
#
# Send dictionary encoded facts filtered by the dict_filter list
#
            elif self.control["send-dict"] == 1 and self.control["dict-filter"] != ['none']:
                for fact in self.env.facts():
                    try:
                        cleanfact = str(fact).split('  ')[-1]
                        templatename = cleanfact.split('(')
                        template = templatename[1].lstrip('(')
                        for dfilter in self.control["dict-filter"]:
                            if str(dfilter) in template: 
                                tempdict = "{'template': '" + str(template) + "', "
                                strfact = str(dict(fact))
                                stripquotes = strfact.split('  ')[-1].strip('"').lstrip("{")
                                finalfact = tempdict + stripquotes
                                self.dict_facts = self.dict_facts + "      <dfact>" + json.dumps(finalfact).strip('"') + "</dfact>\n"
                    except Exception as e:
                        if str(fact).find("retract") == -1:
                            self.print_log("\n%%%% DDR Error: Send Dictionary exception: " + str(fact))
                        pass

        except Exception as e:
            self.print_log("\n%%%% DDR Error: Exception send_clips_info: " + str(e))


    ##############################################################################
    #
    #  save_dict_facts - Save dictionary facts in the ddr-control dictionary-facts list
    #  (interfact-error-state (slot value) (slot value) (slot value))
    #
    #############################################################################
    def save_dict_facts(self):
        try:
            count = 1
            if self.control["save-dict-facts"] == 1 and self.control["dict-filter"] == ['all']:
                self.nc_facts = '<facts>'
                count = 1
                for fact in self.env.facts():
                    try:
                        cleanfact = str(fact).split('  ')[-1]
                        template = cleanfact.split('(')
                        name = template[1].lstrip('(').rstrip(' ')
                        templeaf = "<template>" + str(name) + "</template><slot-list>"
                        self.nc_facts = self.nc_facts + "<fact><id>" + str(count) + "</id>"
                        self.nc_facts = self.nc_facts + str(templeaf)
                        fact_len = len(template)
    #
    # add each fact to the slot-list
    #
                        index = 2
                        while index <= fact_len - 1:
                            strfact = str(template[index]).rstrip(' ').rstrip(')')
                            stripfact = strfact.split(' ')
                            self.nc_facts = self.nc_facts + "<slots><slot>" + str(stripfact[0]) + "</slot><value>" + str(stripfact[1]) + "</value></slots>"
                            index = index + 1
                        count = count + 1
                        self.nc_facts = self.nc_facts + "</slot-list></fact>"

                    except Exception as e:
                        self.print_log("\n%%%% DDR Error: save_dict_facts unfiltered: " + str(e))
                        pass
                self.nc_facts = self.nc_facts + '</facts>'
#
# Send dictionary encoded facts filtered by the dict_filter list
#
            elif self.control["save-dict-facts"] == 1 and self.control["dict-filter"] != ['none']:
                self.nc_facts = '<facts>'
                count = 1
                for fact in self.env.facts():
                    try:
                        cleanfact = str(fact).split('  ')[-1]
                        template = cleanfact.split('(')
                        name = template[1].lstrip('(').rstrip(' ')
                        for dfilter in self.control["dict-filter"]:
                            if str(dfilter) == name:
                                templeaf = "<template>" + str(name) + "</template><slot-list>"
                                self.nc_facts = self.nc_facts + "<fact><id>" + str(count) + "</id>"
                                self.nc_facts = self.nc_facts + str(templeaf)
                                fact_len = len(template)
    #
    # add each fact to the slot-list
    #
                                index = 2
                                while index <= fact_len - 1:
                                    strfact = str(template[index]).rstrip(' ').rstrip(')')
                                    stripfact = strfact.split(' ')
                                    self.nc_facts = self.nc_facts + "<slots><slot>" + str(stripfact[0]) + "</slot><value>" + str(stripfact[1]) + "</value></slots>"
                                    index = index + 1
                                count = count + 1
                                self.nc_facts = self.nc_facts + "</slot-list></fact>"

                    except Exception as e:
                        self.print_log("\n%%%% DDR Error: save_dict_facts filtered: " + str(e))
                        pass
                self.nc_facts = self.nc_facts + '</facts>'

        except Exception as e:
            self.print_log("\n%%%% DDR Error: Exception save_dict_facts: " + str(e))

    def get_initial_facts(self):
        '''
        Initial facts are defined in ddr use cases to provide basic configuration 
        information required before ddr starts execution
        
        Initial facts may be defined in a "python dictionary" format or may be defined
        as a string of strings to maintain backward compability with older use cases.
        The content of the initial_facts is used to assert facts defined in deftemplates in 
        the ddr use case rules file.  The 'key" for each dictionary entry in the name of the 
        the fact template and the items in the dictionary key/value pairs are used to initialize
        slots in the fact.  For example:
            
           initial_facts = [
           {'devices': {'device': 'CAT9K-24'},
            'thresholds': {'max-threshold': '0', 'used-threshold': '1', 'percent-used-threshold': '1'},
            'reporting': {'healthy': '0', 'degraded': '1'}}]
            
        results in the assertion of these facts in ddr:
        
            (devices (device CAT9K-24))
            (thresholds (max-threshold 0) (used-threshold 1) (percent-used-threshold 1))
            (reporting (healthy 0) (degraded 1))

        The list of lists format for initial facts directly defines the strings that will be asserted as facts
        
            initial_facts_list = [
            '(devices (device CAT9K-24))',
            '(thresholds (max-threshold 0) (used-threshold 1) (percent-used-threshold 1))',
            '(reporting (healthy 0) (degraded 1))'
        '''

    #
    # If the initial facts are in dictionary form generate facts from dictionary content
    #    
        initial_dict = self.control["initial-facts-list"][0]
        if type(initial_dict) is dict:
            for key in initial_dict:
                flist = ["(", str(key)]
                slot_dict = (initial_dict[key])
                try:
                    for slot, value in slot_dict.items():
                        flist.append(" (")
                        flist.append(str(slot))
                        flist.append(" ")
                        flist.append(str(value))
                        flist.append(")")
                    flist.append(")")
                except Exception as e:
                    self.print_log("\n%%%% DDR Error creating initial fact from dictionary: " + str(key) + " " + str(e))
                    sys.exit(0)
    #
    # Assert the fact
    #
                fact = ''.join(flist)
                if self.control["debug-fact"] == 1:
                    self.print_log("**** DDR Debug: Assert initial dictionary fact: ")
                    self.print_log(fact)
                try:
                    self.env.assert_string(fact)
                except Exception as e:
                    self.print_log("**** DDR Debug: Assert initial fact error: " + str(fact) + " " + str(e))
                    sys.exit(0)
    #
    # If the initial-fact-list contains a list of strings assert each string as a fact
    #
        else:
            try:
    #
    # assert a fact for each instance in the fact list
    #
                for fact in self.control["initial-facts-list"]:
                    if self.control["debug-fact"] == 1:
                        self.print_log("**** DDR Debug: Assert initial string fact: ")
                        self.print_log(fact)
                    try:
                        self.env.assert_string(fact)
                    except Exception as e:
                        self.print_log("**** DDR Debug: Assert initial fact error: " + str(fact) + " " + str(e))
                        sys.exit(0)

            except Exception as e:
                self.print_log("\n%%%% DDR Error: Error asserting initial string fact: " + str(fact))
                sys.exit(0)

##############################################################################################
#
# assert_test_facts
#
##############################################################################################
    def assert_test_facts(self, test_fact):
#
# assert a fact for each instance in the fact list
#
        try:
            fact_count = len(test_fact)
            if self.control["debug-fact"] == 1:
                self.print_log("**** DDR Debug: assert_test_facts - fact_count: " + str(fact_count))
            for i in range(int(fact_count)):
                  try:
                      if self.control["debug-fact"] == 1:
                          self.print_log("**** DDR Debug: assert_test_facts - test_fact: " + str(test_fact[i]))
                      self.env.assert_string(str(test_fact[i]))
                  except Exception as e: 
                      self.print_log("**** DDR Debug: assert_test_facts - assert_string: " + str(e))
                      pass #ignore case where FACT already exists  
        except Exception as e:
            self.print_log("\n%%%% DDR Error: assert_test_facts - Asserting test facts: " + str(e))
            sys.exit(0)
        return

    #############################################################################
    #
    # Close open netconf device sessions and other open connections
    #
    #############################################################################
    def close_sessions(self):
        try:
            self.netconf_connection.close_session()
            self.notify_conn.close_session()
            for dev in self.device:
                dev.close_session()
        except Exception as e:
            self.print_log("\n%%%% DDR Error: Closing NETCONF sessions: DDR Initialization Failed")
            sys.exit(0)

    ##############################################################################
    #
    #  print_log - log message to debug control["log-file"] if enabled otherwise only display
    #              logging messages on command line
    #     debug-logging controls logging behavior
    #        0 - Display log messages on stdout
    #        1 - Write log messages to the path defined in the FACT file with a timestamp added
    #        2 - Write log messages to stdout and log file path
    #
    #############################################################################
    def print_log(self, logMessage):
        try: 
            if self.control["debug-logging"] == 0 : print(logMessage)
            if self.control["debug-logging"] >= 1: 
                with open(self.control["log-path"] + self.control["log-file"] + "_TS_" + str(self.control["session-time"]), 'a+') as debuglog_file:
                    debuglog_file.write(str(logMessage) + "\n")
                if self.control["debug-logging"] == 2: print(logMessage)
        except Exception as e:
            print("\n%%%% DDR Error: Error writing to log file: " + str(e))

    def read_control_facts(self):

        """
            The read_control_facts function is used to read ddr-control file content from the device guest-share.
            External systems write FACTs into the ddr-control file and save the file in a shared externally visible directory.
            The ddr-control file contains ddr-clips FACTS that are defined in the ddr-rules file.
            Each FACT is read from the ddr-control file and asserted into the CLIPs knowledge-base.
            These FACTs can be used for several purposes:
               * Provide FACTs to trigger usecase execution, e.g. a "state-machine" FACT to control rule execution
               * initial FACTs used to instruct the DDR usecase what data/objects to process
               * Instructions from controllers, e.g. DNAC could provide a list of resources to monitor and KPIs for the resources
               * EEM action Python script can generate FACTs used to trigger usecase execution and provide instructions to the DDR usecase
               
           After the ddr-control file is read from the device: /bootflash/guest-share/ddr/ddr-control is the default path,
           DDR will delete the ddr-control file.
           The external application should interact with DDR as follows:
               * Check for existence of ddr-control file.  If file exists, wait until it has been deleted by DDR
               * When ddr-control file no longer exists, create and write the ddr-control file to the device
               * The DDR usecase will be operating in the "time-based" runMode (0) waiting runWait MS between executions
               * When DDR runs, if controlFacts = 1, DDR will look for the ddr-control file
               * If ddr-control is present, the contents of the file are read into DDR and asserted as FACTs defined as deftemplates in ddr-rules
               * DDR then deletes the ddr-control file
                        
            :param ddr-control file facts: (example)
              (tunnel-event (device "XR123") (tunnel-id "10.1.2.2") (state "Down"))
              (dnac-interface-ip-address (device "cat9k-24") (dnac-ip "19.0.0.1") (dnac-protocol "up"))
        
        Usage::

              read_control_facts()

            :raises none:

        """

        try:
            with open(self.control["control-file"]) as file:
                self.action_data = file.readlines()
                self.action_data = [line.rstrip() for line in self.action_data]
                if self.control["debug-fact"] == 1:
                    self.print_log("**** DDR Debug: read_control_facts file content: \n" + str(self.action_data))
        #
        # assert each fact in the action_data file content
        #
                for line in self.action_data:
                    try:
                        if self.control["debug-fact"] == 1:
                            self.print_log("**** DDR Debug: control-fact: " + str(line))                                         
                        self.env.assert_string(str(line))
                    except Exception as e: 
                        if self.control["debug-fact"] == 1:
                            self.print_log("\n%%%% DDR Error: read_control_facts: \n" + str(line) + "\n" + str(e))
                        pass # ignore if fact already exists in knowledge base
        #
        # delete the ddr-control file after it is read
        # External systems will not write to the ddr-control file if it already exists
        # External systems will verify there is no ddr-control file present and will then create and write new content
        #
                if self.control["delete-control-facts"] == 1:
                    if self.control["debug-fact"] == 1:
                        self.print_log("**** DDR Debug: Deleting ddr-control file")                                         
                    file.close()
                    os.remove(self.control["control-file"])

        except Exception as e:
            self.print_log('\n%%%% DDR Error: read_control_facts error: ' + str(e))
 
######################################################################################################
######################################################################################################
#
# RULE action functions
#
# The following modules are called from the execution section in a CLIPs rule
# When a RULE is triggered 0 or more of these action functions can be called
# Action functions can perform any operation desired by the developer
# Typical action functions collect additional data and assert additional FACTS
#
#    run_action - This action function invoked from a RULE causes collection of FACTs defined in the "control_facts_list" in the FACTs file
#    run_apply_config - This function called from a RULE applies an edit-config operation to the device
#    run_assert_message - Write a message into the DDR log
#    run_assert_message_syslog - Generate a Syslog message for XE device without timestamp
#    run_assert_sim_fact - Assert a selected FACT from the "ddr-sim" file in /bootflash/guest-share/ddr and assert the FACT
#    run_clear_selected_facts - Clear facts in the knowledge base for the template passed in the argument.  Can be used by RULEs to remove types of FACTs
#    run_cli_command - Runs a CLI command on the host using the Python "cli" package implemented for the host
#    run_cli_parameter - Runs any cli command on the host.  The CLI command can include up to 3 parameters
#    run_copy_file - RULEs can copy a file from the device and save the file.  This gives the RULE control over persistence (e.g. prevent wrapping/deletion)
#    run_command - Runs a CLI command on the host using the Python "cli" package.  No result is data is collected
#    run_ddr- This action function runs the CLIPs engine when called from a rule. Normally called when a RULE asserts new FACTs to trigger any RULEs satisfied by new FACTs
#    run_decode_btrace_log - RULE function processes btrace log, selects and asserts FACTs
#    run_delay - Rule function that delays execution of rules for specified number of seconds
#    run_delete_file - Rule function that deletes a file or files from /bootflash/guest-share on the device running DDR
#    run_delete_remote_file - Rule function that deletes a file or files from /bootflash/guest-share on other usecase devices
#    run_logging_trigger - Rule function that parses the content of the Syslog logging buffer to
#            and asserts facts when buffered log messages contain specific content.
#    run_nc_fact - This function called from a RULE collects data using the NETCONF interface and asserts a FACT
#    run_nc_fact_index - This function called from a RULE collects data using the NETCONF interface and asserts a FACT including an index to the ddr-devices file for the target device
#    run_ping_action - This action function called from CLIPs rules runs a ping command, collects results and asserts a FACT
#    run_process_file - Rule function that parses the content of "filename" stored in /bootflash/guest-share and asserts FACTs
#    run_read_control_file - Read a file from /bootflash/guest-share containing FACTs in string form and assert the FACTs
#    run_rule_timer - This action function starts a thread to generate a timer to assert a FACT on a regular time interval
#    run_set_rule_step - Rule function used to directly set the (rule-step (step STEPNAME)) FACT to change execution flow
#    run_set_runmode - Rule function used to change the DDR runmode, 0 for timed trigger, 1 for Syslog trigger (not available on XE), 2 for RFC5277 Notification trigger (not available on XR)
#    run_show_fact - Run show command defined in the 'show_facts_list' in the facts file using the rule action function run_show_fact
#    run_show_parameter - Runs a show command on the host using either the Python "cli" library or an SSH connection
#                         Accepts up to 3 parameters from the CLIPs knowledge base to insert into the show command
#    run_show_parameter_index - Runs a show command on the host using either the Python "cli" library or an SSH connection
#                         Accepts up to 3 parameters from the CLIPs knowledge base to insert into the show command
#                         Accepts a "device_index" parameter to select on which device to perform the operation
#    run_suffix - Remove the first part of a value and return the remainder, e.g. get "tunnel-id100", remove "tunnel-id" return 100
#    run_trace - This action function called from CLIPs rules runs a traceroute command
#    run_write_to_syslog - Send a Syslog message from XE device
#    run_xr_send_syslog - Send a Syslog message from XR device
#
    def run_action(self, action_fact_index, device_index):
        """
            Rule file function call: (run_action 1)
            
            Invoked in the "RHS" of a triggered rule.
            This function is called from the CLIPS RULES to cause the execution of fact collection operations.
            This function can be used to run fact collection using show commands or NETCONF

            The action_fact_list in the ddr-facts file can have one or more entries defining fact collection operations.
            The run_action function selects the required collection list dictionary from action_fact_list and executes the collection.

            The instructions are the same as those used in the nc_fact_list and show command fact collection.  For example: 
            
               {"fact_type": "multitemplate",
                "data": ["multitemplate", 0, "leaf2",
                      '''<native xmlns="http://cisco.com/ns/yang/Cisco-IOS-XE-native">
                           <router>
                             <bgp xmlns="http://cisco.com/ns/yang/Cisco-IOS-XE-bgp">
                               <neighbor>
                                 <id>{0}</id>
                                 <timers>
                                   <holdtime/>
                                 </timers>
                               </neighbor>
                             </bgp>
                           </router>
                         </native>
                ''',
                "nbr-hold-time", "timers",
                 ["holdtime"], 
                 ["hold-time"], [],
                 []]}
                 
               {"fact_type": "show_and_assert",
                   "device": "cat9k-24",
                   "genie_parser": "ShowLoggingLast",
                   "assert_fact_for_each_item_in": "log_instance",
                   "protofact": {"template": "log-instance",
                                "slots": {"device": "device",
                                          "datetime": "$+datetime",
                                          "facility": "$+facility",
                                          "level": "$+level",
                                          "message": "$+message",
                                          "note": "$+note"
                                         }
                               }
                 }
                        
            :param action_fact_index: 0 origin index into the action_fact_list list in the ddr-facts file
            :param device_index: index into ddr-devices to select the device and communication type ssh or cli
        
        Usage::

              (run_action 1)

            :raises none:

        """
        try:
            index = int(action_fact_index)
            fact = self.control["action-fact-list"][index]
            if self.control["debug-fact"] == 1:
                self.print_log("**** DDR Debug: run_action facts: " + str(fact))
            if fact["fact_type"] == "show_and_assert":
                self.show_and_assert_fact(fact, int(device_index))
            elif fact["fact_type"] == "multitemplate":
        # use device access type, ssh or loc using management interface ddr-devices entry
                self.get_template_multifacts_index(fact["data"], 'none', 'none', int(device_index))
            elif fact["fact_type"] == "multitemplate_protofact":
        # use device access type, ssh or loc using management interface ddr-devices entry
                self.get_template_multifacts_protofact_index(fact, 'none', 'none', int(device_index))
            else:
                self.print_log("\n%%%% DDR Error: Error in ddr-facts definition: Invalid fact type: " + str(fact))

        except Exception as e:
            self.print_log("\n%%%% DDR Error: Exception in run_action: " + str(e))

    def run_apply_config(self, config_id):
        """
            Rule file function call: (run_apply_config config_id)
            
            Invoked in the "RHS" of a triggered rule.
            This function is called from the CLIPS RULES
            if an edit-config action is required.  'config_id' is the 0 origin index for the 
            edit-configs list in the ddr-facts file.

            If the device uses a candidate data store and a commit is required, the commit/RPC is sent.
            Configuration operations are stored as list entries.  This is an example.  Note that the namespaces must be
            included with the <config> tag:

              edit_configs = [[0, 'none',
                <config xmlns:xc="urn:ietf:params:xml:ns:netconf:base:1.0" xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
                 <native xmlns='http://cisco.com/ns/yang/Cisco-IOS-XE-native'>
                  <interface><Loopback>
                   <name>111</name>
                   <description>LB111</description>
                  </Loopback></interface>
                 </native>
                </config>'''

             The first list element is the index of the device in the "devices" list used by ncclient
             The second argument is 'commit' if the device uses the candidate datastore and requires 'commit/'
             the argument is 'none' if the device uses the 'running-configuration'
             The third argument is the 'config' section of the edit-config operations, for example copied from YangSuite
            
            :param config_id: 0 origin index into the edit_configs list in the ddr-facts file
        
        Usage::

              run_apply_config(config_id)

            :raises none:

        """
        try:
            config_key = int(config_id)
            edit_msg = self.control["edit-configs"][config_key]
            device = self.device[int(edit_msg[0])]
            if self.control["actions"] == 1:
#
# Use candidate datastore and "commit" for XR configurations
#
                if edit_msg[1] == 'commit':
                    datastore = 'candidate'
                    try:
                        result = device.edit_config(str(edit_msg[2]), target=datastore)
                        if self.control["debug-config"] == 1:
                            self.print_log("**** DDR Debug: apply configuration:\n" + str(edit_msg[2]))
                            self.print_log("**** DDR Debug: edit_config result\n" + str(result.xml))
                        result = dedvice.commit()
                    except Exception as e:
                       self.print_log("%%%% DDR Error applying configuration: \n", str(config_key) + "\n" + str(e))
#
# Use running configuration for XE and NX-OS if commit_required = 0
#
                if edit_msg[1] == 'none':
                    datastore = 'running'
                    try:
                        result = device.edit_config(str(edit_msg[2]), target=datastore)
                        if self.control["debug-config"] == 1:
                            self.print_log("**** DDR Debug: apply configuration:\n" + str(edit_msg[2]))
                            self.print_log("**** DDR Debug: edit_config result\n" + str(result.xml))
                    except Exception as e:
                        self.print_log("%%%% DDR Error applying configuration: \n" + str(edit_msg[2]) + "\n" + str(e))

        except Exception as e:
            self.print_log("%%%% DDR Error in run_apply_config: " + str(e))

    def run_assert_message(self, string):
        """
            Rule file function call: (run_assert_message)
            
            Invoked in the "RHS" of a triggered rule.
            This function called from a RULE writes a message string to the DDR log
            
            :param string - String containing the message to write to the log
        
        Usage::

              (run_assert_message ?message)

            :raises none:

        """

        if self.control["show-messages"] == 1:
            self.print_log("#### DDR Rule Message: " + string)
        if self.control["send-messages"] == 1:
    #
    # Add the message to a list that will be inserted into the service impact notification
    #
            try:
                self.impactList.append("      <message>")
                self.impactList.append(string)
                self.impactList.append("</message>")
                self.impactList.append("\n")
            except Exception as e:
                self.print_log("\n%%%% DDR Error: run_assert_message error: " + str(e))
        return

    def run_assert_message_syslog(self, string):
        """
            Rule file function call: (run_assert_message_syslog string)
            
            Invoked in the "RHS" of a triggered rule.
            This function called from a RULE generates and sends a Syslog message
            
            :param string - String containing the message to include in Syslog
        
        Usage::

              (run_assert_message_syslog ?message)

            :raises none:

        """

        if self.control["show-messages"] == 1:
            self.print_log("#### DDR Rule Message: " + string)
        if self.control["send-messages"] == 1:
    #
    # Add the message to a list that will be inserted into the service impact notification
    #
            try:
                self.impactList.append("      <message>")
                self.impactList.append(string)
                self.impactList.append("</message>")
                self.impactList.append("\n")
            except Exception as e:
                self.print_log("\n%%%% DDR Error: run_assert_message error: " + str(e))
    #
    # Send a Syslog message
    #
            try:
                syslog_notification = "DDR_MESSAGE " + str(self.control["use-case"]) + ": " + string
                self.run_write_to_syslog(syslog_notification, 0)
            except Exception as e:
                self.print_log("\n%%%% DDR Error: run_assert_message_syslog error: " + str(e))
        return

    def run_assert_sim_fact(self, fact_index):
        """
            Rule function call: (run_assert_sim_fact fact_index)
            
            Invoked in the "RHS" of a triggered rule.
            This function asserts a fact in the "sim_data" list that is used
            to simulate the collection of fact data when facts can't be read from a real device.
            This allows inserted simulated facts at any step in the rule workflow execution
                        
            :param fact_index: 0 origin index into the self.sim_data list optionally loaded 
                               when the usecase locaes
        
        Usage::

              (run_assert_sim_fact 5)

            :raises none:

        """

        try:
            int_index = int(fact_index)
            if self.control["debug-fact"] == 1:
                self.print_log(self.sim_data[int_index])
            try:
                if self.control["debug-fact"] == 1:
                    self.print_log("**** DDR Debug: Assert sim FACT: " + str(self.sim_data[int_index]))
                self.env.assert_string(str(self.sim_data[int_index]))
            except Exception as e:
                if self.control["debug-fact"] == 1:
                    self.print_log('\n%%%% DDR Error: run_assert_sim_fact fact assert error: ' + str(self.sim_data[int_index]) + str(e))
                pass # Ignore error caused by asserting an existing fact
             
        except Exception as e:
            self.print_log('\n%%%% DDR Error: run_assert_sim_fact function error: ' + str(self.sim_data[int_index]) + str(e))

    def run_clear_selected_facts(self, template):
        """
            Rule file function call: (run_clear_selected_facts interface-status-facts)
            
            Invoked in the "RHS" of a triggered rule.
            Deletes (retracts) all facts of the type passed in template from the knowledge base  
            
            :param template: deftemplate name in rule file which will have all existing facts deleted
        
        Usage::

              (run_clear_selected_facts interface-status-facts)

            :raises none:

        """
        for fact in self.env.facts():
            try:
                if str(template) in str(fact):
                    fact.retract()
            except Exception as e:
                self.print_log("\n%%%% DDR Error: run_clear_selected_facts: " + str(e))

    def run_cli_command(self, cli_index, filename, parameter1, parameter2, addtime):
        """
            Rule file function call: (run_cli_command 0 file_name)

            Invoked in the "RHS" of a triggered rule.
            Runs a CLI command in the guestshell on the management device using the
            Python "cli" package installed by default in the guestshell.  
            
            The response from the command can optionally be redirected,
            for example to bootflashthe path for the redirected output is provided.
            
            The CLI command executed is defined in the 'cli_cmd' list in the ddr-facts file.
            
            cli_cmd = [['flash:guest-share/', 'show device-tracking events | redirect RUN_FILE_NAME ']]
               first element - Prefix for the filename used to store CLI command response if saved to file
               second element - CLI command to execute including optional redirect to a file RUN_FILE_NAME
               RUN_FILE_NAME - string included in rule statement with name for file to subsitute in redirect command
               third element - 

            :param cli_index: 0 origin index into the cli_cmd list for command to execute
            :param filename: Name to include in file name which is prepended with the path argument in the cli_cmd entry and with a timestamp appended
        
        Usage::

            (run_cli_command 0 file_name)

            :raises none:

        """
        if self.control["actions"] == 1:
            command_idx = int(cli_index)
            timestamp =  datetime.now().strftime("%m-%d-%Y_%H:%M:%S.%f")
            
        #######################################################################
        #
        # Process each CLI command in the list identified in the RULE callback
        # If filename is not equal to 'none' include the filename argument in the data file name
        #
        #######################################################################
            try:
                cmdline = self.cli_cmd[command_idx]
                if self.control["debug-CLI"] == 1:
                    self.print_log("**** DDR Debug: run_cli_cmd: cmdline: " + str(cmdline))
                timestring = ""
                if filename == 'none':
                    outfile = cmdline[0] + timestring
                else:
                    outfile = cmdline[0] + "_" + str(filename) + timestring
        #
        # substitute parameters in command string if included in call from rule
        #
                if cmdline[3] == 0:
                    command = str(cmdline[1])
                else:
                    translated_value = self.translation_dict.get(str(parameter1))
                    if not translated_value:
                        return
                    if cmdline[3] == 1:
                        first = translated_value
                        command = str(cmdline[1]).format(first)
                    elif cmdline[3] == 2:
                        (first, second) = translated_value
                        command = str(cmdline[1]).format(first, second)
                    elif cmdline[3] == 3:
                        (first, second) = translated_value
                        third = parameter2
                        command = str(cmdline[1]).format(first, second, third)
        #
        # if command result is written to a file in nvram insert the file name into the cli command
        #
                if cmdline[2] == 'none':
                    out_command = command
                else:
                    out_command = command.replace('RUN_FILE_NAME', str(outfile))
        #######################################################################
        #
        # If the location to save the data is 'nvram' only write the command results to the device flash/bootflash
        #
        #######################################################################
                try:
                    if self.control["debug-CLI"] == 1:
                        self.print_log("**** DDR Debug: run_cli_cmd: out_command: " + str(out_command))

                    if addtime == 0:
                        cli.cli(out_command, "30")
                    else:
                        timeout = 30 + int(addtime)
                        cli.cli(out_command, str(timeout))
                    
                    if self.control["debug-CLI"] == 1:
                        self.print_log("**** DDR Notice: CLI Command Executed: " + str(out_command) + " at: " + str(timestamp))
                except Exception as e:
                    self.print_log("\n%%%% DDR ERROR: run_cli_cmd Error: device cli command timeout:" +  str(e) + "\n" + str(cmdline) )
            except Exception as e:
                self.print_log("\n%%%% DDR ERROR: run_cli_cmd Error: Exception command processing: " + str(e) + "\n" + str(cmdline))

    def run_cli_parameter(self, device_index, show_template, pcount, par1, par2, par3):
        """
            Rule file function call: (run_cli_parameter ssh "monitor capture CAP interface {0} out" 1 ?port none none)
            
            Invoked in the "RHS" of a triggered rule.
            Runs an exec command in the guestshell on the management device using the
            Python "cli" package installed by default in the guestshell or ssh connection to a device  
            
            This method is used only to execute commands.  No facts are generated as a result of execution

            :param access_type: device index in ddr-devices file containing 'cli' to use the Python cli package in the guestshell or 'ssh' for ssh device access
            :param show_template: show command template with parameters identified by {0}, {1}, {2}
            :param pcount: Number of parameters to substitute in the show_template 0 to 3
            :param parx: Values for parmeters 1, 2 and 3 which can be variables from the rule or defined values
        
        Usage::

              (run_cli_parameter 0 "monitor capture CAP interface {0} out" 1 ?port none none)

            :raises none:

        """
        if self.control["actions"] == 1:
            timestamp =  datetime.now().strftime("%m-%d-%Y_%H:%M:%S.%f")
            
        #######################################################################
        #
        # Generate the show command with parameters
        #
        #######################################################################
            try:
                access_type = str(self.control["device-list"][int(device_index)][5])
                cmdline = str(show_template)
                if self.control["debug-CLI"] == 1:
                    self.print_log("**** DDR Debug: run_show_parameter: cmdline: " + str(cmdline))
        #
        # substitute parameters in command string if included in call from rule
        #
                if int(pcount) == 0:
                    command = str(cmdline)
                else:
                    if int(pcount) == 1:
                        command = str(show_template).format(str(par1))
                    elif int(pcount) == 2:
                        command = str(show_template).format(str(par1), str(par2))
                    elif int(pcount) == 3:
                        command = str(show_template).format(str(par1), str(par2), str(par3))
                try:
                    if self.control["debug-CLI"] == 1:
                        self.print_log("**** DDR Debug: run_cli_parameter: command: " + str(command))
        #
        # If access-type is cli use the Python CLI package to run command
        #
                    if str(access_type) == 'cli':
                        cli.cli(command, "30")
                        if self.control["debug-CLI"] == 1:
                            self.print_log("**** DDR Notice: CLI Command Executed: " + str(command) + " at: " + str(timestamp))
                    else:
        #
        # Use SSH to run the show command
        #
                        try:
                            options = '-q -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oPubkeyAuthentication=no'
                            ssh_cmd = 'ssh %s@%s %s "%s"' % (self.control["mgmt-device"][2], self.control["mgmt-device"][0], options, command)
                            if self.control["debug-CLI"] == 1:
                                self.print_log("**** DDR Debug: run_cli_parameter SSH command: " + str(ssh_cmd) + "\n")
                            child = pexpect.spawn(ssh_cmd, timeout= 20, encoding='utf-8')
                            child.delaybeforesend = None
                            if self.control["debug-CLI"] == 1:
                                child.logfile = sys.stdout
                            child.expect(['\r\nPassword: ', '\r\npassword: ', 'Password: ', 'password: '])
                            child.sendline(self.control["mgmt-device"][3])
                            child.expect(pexpect.EOF) #result contains the ping result
                            response = child.before

                        except Exception as e:
                            child.close()
                            self.print_log("\n%%%% DDR ERROR: run_cli_parameter SSH or timeout Error: " + str(ssh_cmd) + "\n")
                        child.close()

                except Exception as e:
                    self.print_log("\n%%%% DDR ERROR: run_cli_parameter Error: Exception sending show command: " + "\n" + str(command) + "\n" + str(e) + "\n")
            except Exception as e:
                self.print_log("\n%%%% DDR ERROR:  run_cli_parameter Error: Exception generating show command: " + "\n" + str(show_template) + "\n" + str(e) + "\n")

    def run_copy_file(self, infile, outfile):
        """
            Rule file function call: (run_copy_file "bootflash:/core/coredump" "bootflash:/guest_share/saved-coredump")
            
            Invoked in the "RHS" of a triggered rule.
            Copies a file from infile location to outfile location.
            Typical use would be to save a log, debug or capture file for further processing or to archive.  
            
            :param infile: full pathname for file to copy
            :param outfile: full pathname for file destination
        
        Usage::

              (run_copy_file "/bootflash/guest-share/test1" "/bootflash/guest-share/test2")
              (run_copy_file ?filename-variable "/bootflash/guest-share/filename")

            :raises none:

        """
        if self.control["actions"] == 1:
            
            try:
                with open(str(infile), 'r') as rfd:
                    indata = rfd.read()
                    rfd.close()
                out_path = str(outfile)
                with open(out_path, 'w') as wfd:
                    wfd.write(indata)
                    wfd.close()
                if self.control["debug-CLI"] == 1:
                    self.print_log("**** DDR Debug: run_copy_file infile: " + str(infile) + " outfile: " + out_path)
            except Exception as e:
                self.print_log("\n%%%% DDR ERROR: run_copy_file Error: Exception command processing: " + str(e))

    def run_copy_file(self, infile, outfile):
        """
            Rule file function call: (run_copy_file "bootflash:/core/coredump" "bootflash:/guest_share/saved-coredump")
            
            Invoked in the "RHS" of a triggered rule.
            Copies a file from infile location to outfile location.
            Typical use would be to save a log, debug or capture file for further processing or to archive.  
            
            :param infile: full pathname for file to copy
            :param outfile: full pathname for file destination
        
        Usage::

              (run_copy_file "/bootflash/guest-share/test1" "/bootflash/guest-share/test2")
              (run_copy_file ?filename-variable "/bootflash/guest-share/filename")

            :raises none:

        """
        if self.control["actions"] == 1:
            
            try:
                with open(str(infile), 'r') as rfd:
                    indata = rfd.read()
                    rfd.close()
                out_path = str(outfile)
                with open(out_path, 'w') as wfd:
                    wfd.write(indata)
                    wfd.close()
                if self.control["debug-CLI"] == 1:
                    self.print_log("**** DDR Debug: run_copy_file infile: " + str(infile) + " outfile: " + out_path)
            except Exception as e:
                self.print_log("\n%%%% DDR ERROR: run_copy_file Error: Exception command processing: " + str(e))

    def run_copy_from_device(self, infile, device):
        """
            Rule file function call: (run_copy_from_device "flash:ddr_data_file" 0)
            
            Invoked in the "RHS" of a triggered rule.
            Copies a file from infile location on the device into the directory where ddr is running offbox.
            Typical use would be to copy a log file generated by IOS features, e.g. 'redirect'/'append' from
            CLI command execution to a location off the device where DDR is running offbox.  
            
            :param infile: full pathname for file to copy on the device
            :param device: index in the devices list where the file is located
                    
        Usage::

              (run_copy_from_device "flash:ddr_data_file" 0)

            :raises none:

        """
          
        if self.control["actions"] == 1:
            
        #######################################################################
        #
        # run the scp copy command from the device
        # scp admin@10.77.92.62:ddr-g8032-show-logging .
        #######################################################################
            try:
                device_index = int(device)
                device_dat = self.control["device-list"][device_index]
                command = ":" + str(infile) + " ."
                options = '-q -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oPubkeyAuthentication=no'
                ssh_cmd = 'scp %s@%s%s' % (device_dat[2], device_dat[0], command)
                if self.control["debug-CLI"] == 1:
                    self.print_log("**** DDR Debug: run_copy_from_device command: " + str(ssh_cmd) + "\n")
                child = pexpect.spawn(ssh_cmd, timeout= 20, encoding='utf-8')
                child.delaybeforesend = None
                if self.control["debug-CLI"] == 1:
                    child.logfile = sys.stdout
                child.expect(['\r\nPassword: ', '\r\npassword: ', 'Password: ', 'password: '])
                child.sendline(device_dat[3])
                child.expect(pexpect.EOF)
                response = child.before
                if self.control["debug-CLI"] == 1:
                    self.print_log("**** DDR Notice: copy_from_device Command Response:\n " + str(response))
            except Exception as e:
                self.print_log("\n%%%% DDR ERROR: run_copy_from_device Error: Exception command processing: " + str(e))

    def run_command(self, command, delay):
        """
            Rule file function call: (run_command "clear counters" 1000)
            
            Invoked in the "RHS" of a triggered rule.
            Runs the CLI command based in as an argument and delays for the number of ms in delay
            before continuing 
            
            :param command: cli command to execute
            :param delay: number of milliseconds to delay after running the command
        
        Usage::

              (run_command "clear counters" 1000)

            :raises none:

        """
        if self.control["actions"] == 1:
            try:
                if self.control["debug-CLI"] == 1:
                    self.print_log("**** DDR Debug: run_command: " + str(command))
                result = cli.cli(command, "30")
                if self.control["debug-CLI"] == 1:
                    self.print_log("**** DDR Debug: run_command result: " + str(result))
                if int(delay) > 0:
                    if self.control["debug-CLI"] == 1:
                        self.print_log("**** DDR Debug: run_command delay ms: " + str(delay))
                    time.sleep(int(delay)/1000)
                    if self.control["debug-CLI"] == 1:
                        self.print_log("**** DDR Debug: run_command delay complete")
            except Exception as e:
                self.print_log("\n%%%% DDR ERROR: run_command Error: Exception: " + str(e) + "\n")

    def run_ddr(self):
        """
            Rule file function call: (run_ddr)
            
            Invoked in the "RHS" of a triggered rule.
            This function runs the CLIPs runtime to enable execution of actions for rules that are "triggered" by added facts.
            If a rule asserts or modifies a fact, the CLIPs system evaluates the state of all rules that are dependent on the added or changed fact.
            If the added or changed fact results in all of the conditions on the lefr-hand-side of rule being satisifed, the rule is "triggered".
            When a rule is triggered it is marked by CLIPs as ready to execute the right-hand-side actions.
            In order to execute the rule's actions, the CLIPs runtime must be invoked to run again.
            
            This function is typically used in a workflow where rule 1 asserts a fact and "knows" that another rule depends on the fact and needs to be
            executed immediately to implement the worklfow

            :param none: 
        
        Usage::

              (run_ddr)

            :raises none:

        """
        try:
            self.print_clips_info_run(' BEFORE ')
            self.env.run()
            self.print_clips_info_run(' AFTER' )
        except Exception as e:
            self.print_log("\n%%%% DDR Error: Exception when running inference engine in run_action" + str(e))
            self.close_sessions()
            sys.exit(0)

    def run_decode_btrace_log(self, fact_index, device_index, show_template, pcount, par1, par2, par3, device_id):

        """
            Rule file function call: (run_decode_btrace_log 0 ssh "show platform software trace message {0} {1}" 2 dmiauthd "switch active R0" none 1)
            
            Invoked in the "RHS" of a triggered rule.
            Runs a show command in the guestshell on the management device using the
            Python "cli" package installed by default in the guestshell or ssh connection to a device
            
            The show command uses 'show platform software trace..." to decode the current btrace log
            file for a process and extracts information from the file to assert facts.  The 'genie_parser' searches
            the lines in the decoded btrace log and asserts facts for each line that match the expressions defined in the parser 
            
            The actions performed by the method are defined in the 'decode_btrace_fact_list' list in the ddr-facts file.
            Each entry in the list is indexed by the fact_index parameter as a Python dictionary
              fact_type - The type of action function
              device - optionally directly specify the device name otherwise management device used by default
              genie_parser - Name of parser class in genie_parsers.py for processing this show command
              assert_fact_for_each_item_in - Assert a fact for each dictionary entry key at this level in the dictionary
              protofact - prototype describing how to translate parser python dictionary output to a CLIPs fact
              template - name of the deftemplate in ddr-rules defining the fact
              slots - names of the slots/fields in the deftemplate for the fact
              $+transaction_id - the value of the key in the dictionary.  In the example below the NETCONF transaction_id is the key and the slot 'transaction-id' is set to the key value
              $+date - set the date slot to the parser dictionary entry for 'date'
              etc.
                  {"fact_type": "show_and_assert",
                   "device": "cat9k-24",
                   "genie_parser": "BtraceDmiauthdConfigI",
                   "assert_fact_for_each_item_in": "config_transaction",
                   "protofact": {"template": "bt-dmi-config",
                                    "slots": {"device": "device",
                                          "transaction-id": "$+transaction_id",
                                          "date": "$+date",
                                          "time": "$+time",
                                          "method": "$+method",
                                          "config-by": "$+config_by"
                                             }
                               }
                 }

            :param fact_index: 0 origin index into the decode_btrace_fact_list with execution instructions
            :param device_index: index into ddr-devices for device access method 'cli' to use the Python cli package in the guestshell or 'ssh' for ssh device access
            :param show_template: show command template with parameters identified by {0}, {1}, {2}
            :param pcount: Number of parameters to substitute in the show_template 0 to 3
            :param parx: Values for parmeters 1, 2 and 3 which can be variables from the rule or defined values
            :param device_id: index into self.devices to select device for decode operation
        
        Usage::

              (run_decode_btrace_log 0 0 "show platform software trace message {0} {1}" 2 dmiauthd "switch active R0" none 0)

            :raises none:

        """

        if self.control["actions"] == 1:
            index = int(fact_index)
            access_type = str(self.control["device-list"][int(device_index)][5])
            fact = self.control["decode-btrace-fact-list"][int(index)]

            timestamp =  datetime.now().strftime("%m-%d-%Y_%H:%M:%S.%f")
          
        #######################################################################
        #
        # Generate the show command with parameters
        #
        #######################################################################
            try:
                cmdline = str(show_template)
                if self.control["debug-CLI"] == 1:
                    self.print_log("**** DDR Debug: run_decode_btrace_log: cmdline: " + str(cmdline))
        #
        # substitute parameters in command string if included in call from rule
        #
                if int(pcount) == 0:
                    command = str(cmdline)
                else:
                    if int(pcount) == 1:
                        command = str(show_template).format(str(par1))
                    elif int(pcount) == 2:
                        command = str(show_template).format(str(par1), str(par2))
                    elif int(pcount) == 3:
                        command = str(show_template).format(str(par1), str(par2), str(par3))
                try:
                    if self.control["debug-CLI"] == 1:
                        self.print_log("**** DDR Debug: run_decode_btrace_log: command: " + str(command))
        #
        # If access-type is cli use the Python CLI package to run command
        #
                    if str(access_type) == 'cli':
                        response = cli.cli(command, "30")
                        if self.control["debug-CLI"] == 1:
                            self.print_log("**** DDR Notice: Decode Btrace log Command Executed: " + str(command) + " at: " + str(timestamp))
                            
                    else:
        #
        # Use SSH to run the show command
        # Use mgmt-device if device_id is 0, otherwise use device selected from self
        #
                        try:
                            device_ip = self.control["device-list"][int(device_id)][0]
                            device_user = self.control["device-list"][int(device_id)][2]
                            device_pass = self.control["device-list"][int(device_id)][3]
                            
                            options = '-q -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oPubkeyAuthentication=no'
                            ssh_cmd = 'ssh %s@%s %s "%s"' % (device_user, device_ip, options, command)
                            if self.control["debug-CLI"] == 1:
                                self.print_log("**** DDR Debug: run_decode_btrace_log SSH command: " + str(ssh_cmd) + "\n")
                            child = pexpect.spawn(ssh_cmd, timeout= 60, encoding='utf-8')
                            child.delaybeforesend = None
                            if self.control["debug-CLI"] == 1:
                                child.logfile = sys.stdout
                            child.expect(['\r\nPassword: ', '\r\npassword: ', 'Password: ', 'password: '])
                            child.sendline(device_pass)
                            child.expect(pexpect.EOF)
                            response = child.before
                            if self.control["debug-CLI"] == 1:
                                self.print_log("**** DDR Debug: run_decode_btrace_log: \n" + str(response))

                        except Exception as e:
                            child.close()
                            self.print_log("%%%% DDR run_decode_btrace_log SSH or timeout Error: " + str(ssh_cmd) + "\n")
                        child.close()

    ##########################################################################################
    #
    # Generate facts using log content
    #
    ##########################################################################################
                    fact_found = False
                    try:
                        parser = genie_str_to_class(fact["genie_parser"])
                        if parser == None:
                            self.print_log("\n%%%% DDR Error: Exception in run_decode_btrace_log: parser not found: \n" + str(fact["genie_parser"]) + " \n")
                            return
    #
    # Parse data if the parser is available
    #
                        try:
                            parsed_genie_output = parser.parse(output=response)
                        except Exception as e:
                            self.print_log("\n%%%% DDR Error: Exception in run_decode_btrace_log: parser not found: " + str(fact["genie_parser"]))
                            return
                            
                        if self.control["debug-parser"] == 1:
                            self.print_log("**** DDR Debug: run_decode_btrace_log parser result: \n\n" + str(parsed_genie_output) + "\n")
                        if parsed_genie_output == {}:
                            self.print_log("\n%%%% DDR Error: run_decode_btrace_log no parser output")
                            return
        #
        # Get the starting point in the Python dictionary containing the parsed content
        #
                        sub_dictionary_list = self.find(fact["assert_fact_for_each_item_in"], parsed_genie_output)
                        import copy
                        for sub_dictionary in sub_dictionary_list:
                            for item in sub_dictionary:
                                protofact = copy.deepcopy(fact["protofact"])
                                for slot in protofact["slots"]:
                                    value = protofact["slots"][slot]
	#
	# insert the device name into the fact
	#
                                    if value == "device":
                                        protofact["slots"][slot] = value.replace("device", str(self.control["device-list"][int(device_index)][4]))
                                    elif type(value) == str and "$" in value:
                                        protofact["slots"][slot] = value.replace("$", item)
                                fact_found = True
                                self.assert_template_fact(protofact, 'device', str(self.control["device-list"][int(device_index)][4]), sub_dictionary)

                            return

                    except Exception as e:
                        self.print_log("\n%%%% DDR Error: Exception in run_decode_btrace_log response processing: " + str(e))
                        return

                except Exception as e:
                    self.print_log("%%%% DDR run_decode_btrace_log Error: Exception sending show command: " + str(e) + "\n")
            except Exception as e:
                self.print_log("%%%% DDR run_decode_btrace_log Error: Exception generating show command: " + str(e) + "\n")

    def run_delay(self, seconds):
        """
            Rule file function call: (run_delay 5)
            
            Invoked in the "RHS" of a triggered rule.
            This function called from a rule delays the specified number of seconds before continuing the execution of rule actions.
            This function can be called from a rule to insert a fixed delay in a worklow, for example, "delay 60 seconds after starting packet capture"
            could be implemented by:
               Start packet capture
               (run_delay 60)
            
            :param mode - cli/ssh run using python cli package or using ssh 
            :param vrf - vrf-name or none - do not include a vrf in the ping command
            :param neighbor - ip address of neighbor to ping
            :param count - number of pings to run
            :param min-success - minimum % of pings that must succeed
            :param src-interface - interface to run ping command through
            :param pkt_size - ping payload size in bytes
            :param fact_result - name of template to store results
        
        Usage::

              (run_delay 5)

            :raises none:

        """
        if self.control["actions"] == 1:
            if self.control["debug-action"] == 1:
                timestamp =  datetime.now().strftime("%m-%d-%Y_%H:%M:%S.%f")
                self.print_log("\n**** DDR Debug: run_delay seconds: " + str(seconds) + " starting at: " + str(timestamp))
            time.sleep(float(seconds))

    def run_delete_file(self, filepath):
        """
            Rule file function call: (run_delete_file /bootflash/guest-share/ddr-dmi-auth)
            
            Invoked in the "RHS" of a triggered rule.
            Deletes a file or files at the specified path. Wildcard * can be used  
            
            :param filepath: full pathname for file to delete
        
        Usage::

              (run_delete_file /bootflash/guest-share/ddr-dmi-auth)
              (run_delete_file /bootflash/guest-share/ddr/logs/*)

            :raises none:

        """
        if self.control["actions"] == 1:
            
            try:
                if self.control["debug-CLI"] == 1:
                    self.print_log("**** DDR Debug: run_delete_file: " + str(filepath) )
                os.remove(filepath)
            except Exception as e:
                if self.control["debug-CLI"] == 1:
                    self.print_log("**** DDR Debug: file doesnt exist: " + str(filepath) )
                pass #ignore error if file did not exist

    def run_delete_remote_file(self, device_id, filepath):
        """
            Rule file function call: (run_delete_remote_file 1 flash:guest-share/ddr-dmi-auth)
            
            Invoked in the "RHS" of a triggered rule.
            Deletes file(s) on the remote device identified by the device_id.
            Sample command executed on remote device:
                #del /force flash:guest-share/ddr-btrace-to-file
  
            :param device_id: Index into the device list defined in ddr-devices           
            :param filename: full pathname for file to delete
        
        Usage::

              (run_delete_remote_file 1 flash:guest-share/ddr-dmi-auth)
              (run_delete_remote_file 1 flash:guest-share/ddr/logs/*)

            :raises none:

        """
        if self.control["actions"] == 1:           
        #
        # Use SSH to run the command on a different device
        # Use mgmt-device if device_id is 0, otherwise use device selected from self
        #
            try:
                command = 'del /force ' + str(filepath)
                device_ip = self.control["device-list"][int(device_id)][0]
                device_user = self.control["device-list"][int(device_id)][2]
                device_pass = self.control["device-list"][int(device_id)][3]
                            
                options = '-q -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oPubkeyAuthentication=no'
                ssh_cmd = 'ssh %s@%s %s "%s"' % (device_user, device_ip, options, command)
                if self.control["debug-CLI"] == 1:
                    self.print_log("**** DDR Debug: run_delete_remote_file SSH command: " + str(ssh_cmd) + "\n")
                child = pexpect.spawn(ssh_cmd, timeout= 60, encoding='utf-8')
                child.delaybeforesend = None
                if self.control["debug-CLI"] == 1:
                    child.logfile = sys.stdout
                    child.expect(['\r\nPassword: ', '\r\npassword: ', 'Password: ', 'password: '])
                    child.sendline(device_pass)
                    child.expect(pexpect.EOF)
                    response = child.before
                    if self.control["debug-CLI"] == 1:
                        self.print_log("**** DDR Debug: run_delete_remote_file: \n" + str(response))

            except Exception as e:
                child.close()
                self.print_log("%%%% DDR run_delete_remote_file SSH or timeout Error: " + str(ssh_cmd) + "\n")
                
            child.close()

    def run_logging_trigger(self, device_index, logging_trigger_index, number_lines, facility):

        """
            Rule file function call: (run_logging_trigger ssh 0 4 DMI)
            
            Invoked in the "RHS" of a triggered rule.
            Rule function that parses the content of the Syslog logging buffer to
            and asserts facts when buffered log messages contain specific content.
            This function can be used by a rule to trigger when specific Syslog messages
            with specific content are generated.  This may be used if the device or the deployment
            can't support the RFC5277 notification trigger method for DDR execution triggering.
            This function can also be used by a workflow that decides it needs to search the logging history
            to determine if an event occurred.   
            
            The actions performed by the method are defined in the 'logging_trigger_list' list in the ddr-facts file.
            The logging_trigger_list is a list of dictionary entries that provide instructions to the main DDR

              *Mar 29 17:01:20.953: %SEC_LOGIN-5-LOGIN_SUCCESS: Login Success [user: admin] [Source: 172.20.86.186] [localport: 22] at 17:01:20 UTC Mon Mar 29 2021

            Each entry in the list is indexed by the fact_index parameter as a Python dictionary
              fact_type - The type of action function
              device - optionally directly specify the device name otherwise management device used by default
              genie_parser - Name of parser class in genie_parsers.py for processing this show command
              assert_fact_for_each_item_in - Assert a fact for each dictionary entry key at this level in the dictionary
              protofact - prototype describing how to translate parser python dictionary output to a CLIPs fact
              template - name of the deftemplate in ddr-rules defining the fact
              slots - names of the slots/fields in the deftemplate for the fact
              $+datetime - Date and time extracted from the syslog message
              $+facility - type of syslog message, e.g. "SEC_LOGIN"
              $+level - Syslog level 0 - 7. e.g 5
              $+message - text to right of the level, e.g. "LOGIN_SUCCESS"
              $+note - free from text with remainder of syslog message 

                  {"fact_type": "show_and_assert",
                   "device": "cat9k-24",
                   "genie_parser": "ShowLoggingLast",
                   "assert_fact_for_each_item_in": "log_instance",
                   "protofact": {"template": "log-instance",
                                "slots": {"device": "device",
                                          "datetime": "$+datetime",
                                          "facility": "$+facility",
                                          "level": "$+level",
                                          "message": "$+message",
                                          "note": "$+note"
                                         }
                               }
                 }
             
            :param device_index: Index into ddr-devices to determine device access method "cli" runs command using guestshell Python cli package, "ssh" uses SSH connectin
            :param logging_trigger_index: 0 origin index into the logging_trigger_list
            :param number_lines: Number of lines in the syslog buffer to process (starting with most recent)
            :param facility: String containing the Syslog "facility" name, e.g. "SEC_LOGIN" for the example above
        
        Usage::

              (run_logging_trigger ssh 0 4 DMI) ;use ssh, first entry in fact list, process 4 lines of the syslog buffer looking for "DMI" facility messages

            :raises none:

        """
        if self.control["actions"] == 1:
            try:
                access_type = str(self.control["device-list"][int(device_index)][5])
                command_idx = int(logging_trigger_index)
                cmdline = "show logging last {0}"
                command = cmdline.format(int(number_lines))

                if self.control["debug-CLI"] == 1:
                    self.print_log("\n**** DDR Debug: run_logging_trigger command: " + str(command))
        #
        # If access-type is cli use the Python CLI package to run command
        #
                if access_type == 'cli':
                    response = cli.cli(str(command), "30")
                    if self.control["debug-CLI"] == 1:
                        self.print_log("**** DDR Debug: run_logging_trigger cli response: \n" + str(response))
        #
        # Use SSH to run the show command
        #
                else:
                    try:
                        options = '-q -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oPubkeyAuthentication=no'
                        ssh_cmd = 'ssh %s@%s %s "%s"' % (self.control["mgmt-device"][2], self.control["mgmt-device"][0], options, command)
                        if self.control["debug-CLI"] == 1:
                            self.print_log("**** DDR Debug: run_logging_trigger SSH command: " + str(ssh_cmd) + "\n")
                        child = pexpect.spawn(ssh_cmd, timeout= 20, encoding='utf-8')
                        child.delaybeforesend = None
                        if self.control["debug-CLI"] == 1:
                            child.logfile = sys.stdout
                        child.expect(['\r\nPassword: ', '\r\npassword: ', 'Password: ', 'password: '])
                        child.sendline(self.control["mgmt-device"][3])
                        child.expect(pexpect.EOF) #result contains the ping result
                        response = child.before
                        if self.control["debug-CLI"] == 1:
                            self.print_log("**** DDR Debug: run_logging_trigger ssh response: \n" + str(response))

                    except Exception as e:
                        child.close()
                        self.print_log("%%%% DDR run_logging_trigger SSH or timeout Error: " + str(ssh_cmd) + "\n")
                        return
                        
                    child.close()
         #
         # Assert the syslog logging facts using the response
         #
                fact_found = False
        #
        # Read specified number of lines in Syslog logging buffer and assert facts
        #
                try:
                    fact = self.control["logging-trigger-list"][command_idx]
                    parser = genie_str_to_class(fact["genie_parser"])
                    if parser == None:
                        self.print_log("\n%%%% DDR Error: Exception in run_logging_trigger: parser not found: \n" + str(fact["genie_parser"]) + " \n")
                        return
    #
    # Parse data if the parser is available
    #
                    try:
                        parsed_genie_output = parser.parse(num_lines=int(number_lines), output=response, facility_filter=str(facility))
                    except Exception as e:
                        self.print_log("\n%%%% DDR Error: Exception in run_logging_trigger parser not found: " + str(fact["genie_parser"]))
                        return

                    if self.control["debug-parser"] == 1:
                        self.print_log("**** DDR Debug: run_logging_trigger parser result: \n\n" + str(parsed_genie_output) + "\n")
                except Exception as e:
                    self.print_log("\n%%%% DDR Error: Exception in run_logging_trigger parser processing: \n" + str(parsed_genie_output) + "\n" + str(e))
                    return
               
                sub_dictionary_list = self.find(fact["assert_fact_for_each_item_in"], parsed_genie_output)
                import copy
                for sub_dictionary in sub_dictionary_list:
                    for item in sub_dictionary:
                        protofact = copy.deepcopy(fact["protofact"])
                        for slot in protofact["slots"]:
                            value = protofact["slots"][slot]
	#
	# insert the device name into the fact
	#
                            if value == "device":
                                protofact["slots"][slot] = value.replace("device", str(self.control["device-list"][int(device_index)][4]))
                            elif type(value) == str and "$" in value:
                                protofact["slots"][slot] = value.replace("$", item)
                        self.assert_template_fact(protofact, 'device', str(self.control["device-list"][int(device_index)][4]), sub_dictionary)
            except Exception as e:
                self.print_log("\n%%%% DDR Error: Exception in run_logging_trigger assert_fact response processing: " + str(fact) + "\n" + str(e))

    def run_nc_fact(self, *args):

        """
            Rule file function call: (run_nc_fact 0 1 neighbor ?neighbor none ?neighbor) ; use netconf to get hold-time for the bgp-neighbor
            
            Invoked in the "RHS" of a triggered rule.
            This function executes NETCONF operations defined in ddr-facts nc_fact_list entries
            For example: (run_nc_fact 0 1 neighbor ?neighbor none ?neighbor)
              This example runs a NETCONF operation using index "0" in the nc_fact_list.  The 1 indicates there is one
              parameter that will provided by the rule and will substituted into the NETCONF message.
              "none" indicates that splitting the contents of variable is not required 

            Two types of FACTs can be defined in ddr-facts, multitemplate and multitemplate_protofact
            
              nc_fact_list = [
               {"fact_type": "multitemplate",
                "data": ["multitemplate", 0, "leaf2",
                      '''<native xmlns="http://cisco.com/ns/yang/Cisco-IOS-XE-native">
                           <router>
                             <bgp xmlns="http://cisco.com/ns/yang/Cisco-IOS-XE-bgp">
                               <neighbor>
                                 <id>{0}</id>
                                 <timers>
                                   <holdtime/>
                                 </timers>
                               </neighbor>
                             </bgp>
                           </router>
                         </native>
                ''',
                "nbr-hold-time", "timers",
                 ["holdtime"], 
                 ["hold-time"], [],
                 []]},

		{"fact_type": "multitemplate_protofact", 
		        "device_index": 0,
		        "device": 'NVE-EVPN-leaf1',
		        "assert_fact_for_each": 'Ep-list',
		        "hardcoded_list": ["device"],
		"path": '''
		      <System xmlns="http://cisco.com/ns/yang/cisco-nx-os-device">
		        <eps-items>
		          <epId-items>
	            	    <Ep-list>
		              <epId/>
		              <adminSt/>
		              <primaryIp/>
		              <peers-items>
		                <dy_peer-items>
		                  <DyPeer-list>
		                    <ip/>
		                    <state/>
		                    <mac/>
		                  </DyPeer-list>
  		                </dy_peer-items>
   		              </peers-items>
     		              </Ep-list>
		            </epId-items>
		        </eps-items>
		      </System>''',
		"protofact": {"template": 'nve-peer-list2',
		              "slots": {"device": 'NVE-EVPN-leaf1',
		                        "name": 'epId',
		                        "primary-ip": 'primaryIp',
		                        "admin-state": 'adminSt',
		                        "peer-ip": 'ip',
		                        "peer-state": 'state',
		                        "peer-mac": 'mac'
		                       },
		              "types": {"device": "str",
		                        "name": "str",
		                        "primary-ip": "str",
		                        "admin-state": "str",
		                        "peer-ip": "str",
		                        "peer-state": "str",
		                        "peer-mac": "str"
		                        } 
		        }
		}                                  
            
            :param *args: Pointer to structure with passed arguments defined in function content below
                    args[0]: index = int(argval)
                    args[1]: num_params = int(argval)
                    args[2]: slot = argval is the slot name
                    args[3]: slot_value = argval is the slot value
                    args[4]: split = argval #This argument = 'split' if the next argument is an interface name
       
        Usage::

              (run_nc_fact 0 1 neighbor ?neighbor none ?neighbor) ;This use netconf to get hold-time for the bgp-neighbor
              (run_nc_fact 2 1 neighbor ?neighbor none ?neighbor) ;This use netconf to get session-state for the bgp-neighbor
              (run_nc_fact 1 1 neighbor ?neighbor none ?neighbor) ;This use netconf to routing state the bgp-neighbor

            :raises none:

        """

        if self.control["debug-fact"] == 1:
                self.print_log("**** DDR Debug: run_nc_facts facts: " + str(args))
        try:
    #
    # Extract the arguments from the call from the RULE
    # 1st argument is index into "nc-fact-list"
    # 2nd argument is the number of parameters to substitute into the NETCONF request
    # Remaining arguments are inserted into the NETCONF request using string.format method
    #            
            sub_params = []
            j = 0
            split = 'none'
            try:
                for argval in args:
                    if j == 0: index = int(argval)
                    if j == 1: num_params = int(argval)
                    if j == 2: slot = argval
                    if j == 3: slot_value = argval
                    if j == 4: split = argval #This argument = 'split' if the next argument is an interface name
                    if j > 4:
                        if (j == 5) and (split == 'split'):
                            interface = argval.partition('Ethernet')
                            sub_params.append(interface[0] + interface[1])
                            sub_params.append(interface[2])
                            split = 'none'
                            num_params = num_params + 1
                        else: sub_params.append(str(argval))
                    j = j + 1
            except Exception as e:
                self.print_log("\n%%%% DDR Error: run_nc_fact argument error: " +str(sub_params) + " " + str(e))
                return

        #
        # process multitemplate and multitemplate_protofact FACT types
        #
            raw_fact = copy.deepcopy(self.control["nc-fact-list"][index])
            if raw_fact["fact_type"] == "multitemplate":
                fact_data = raw_fact["data"]
                fact_msg = fact_data[3]
            else: #fact is multitemplate_protofact
                fact_msg = raw_fact["path"]
#
# Substitute arguments from the rule function call into the Netconf operation
#
            try:
                if num_params == 0:
                   formatted = str(fact_msg)
                elif num_params == 1:
                   formatted = str(fact_msg).format(str(sub_params[0]))
                elif num_params == 2:
                   formatted = str(fact_msg).format(str(sub_params[0]), str(sub_params[1]))
                elif num_params == 3:
                   formatted = str(fact_msg).format(str(sub_params[0]), str(sub_params[1]), str(sub_params[2]))
                elif num_params == 4:
                   formatted = str(fact_msg).format(str(sub_params[0]), str(sub_params[1]), str(sub_params[2]), str(sub_params[3]))
                else:
                    self.print_log("\n%%%% DDR Error: run_nc_fact invalid # arguments: " + str(num_params))
                    return

            except Exception as e:
                self.print_log("\n%%%% DDR Error: run_nc_fact format message: " + str(e))
               
            if self.control["debug-fact"] == 1:
                self.print_log("**** DDR Debug: run_nc_facts facts: " + str(fact_msg))

            #
            # update FACT netconf request including any parameters provided
            #
            if raw_fact["fact_type"] == "multitemplate":
                fact_data[3] = formatted
                raw_fact["data"] = fact_data
                fact = raw_fact
            else: #fact is multitemplate_protofact
                raw_fact["path"] = formatted
                fact = raw_fact

            # insert device name into get calls to insert device name into generated facts
            if fact["fact_type"] == "multitemplate":
                self.get_template_multifacts_index(fact["data"], "device", fact["data"][2], 'none')
            elif fact["fact_type"] == "multitemplate_protofact":
                self.get_template_multifacts_protofact_index(fact, "device", fact["data"][2], 'none')
            else:
                self.print_log("\n%%%% DDR Error: run_nc_facts Error in ddr-facts definition: Invalid fact type: " + str(fact))

        except Exception as e:
            self.print_log("\n%%%% DDR Error: Exception in run_nc_fact: " + str(e))

    def run_nc_fact_index(self, index, device_index, num_params, par1, par2, par3):

        """
            Rule file function call: (run_nc_fact_index 0 ?device_index 1 neighbor ?neighbor none ?neighbor) ; use netconf to get hold-time for the bgp-neighbor
            
            Invoked in the "RHS" of a triggered rule.
            This function executes NETCONF operations defined in ddr-facts nc_fact_list entries
            For example: (run_nc_fact 0 ?device_index 1 neighbor ?neighbor none ?neighbor)
              This example runs a NETCONF operation using index "0" in the nc_fact_list.  ?device_index selects which device in the 'device_list' defined in ddr-devices to perform the operation on.  The 1 indicates there is one
              parameter that will provided by the rule and will substituted into the NETCONF message.
              "none" indicates that splitting the contents of variable is not required 
         
            Two types of FACTs can be defined in ddr-facts, multitemplate and multitemplate_protofact
            
              nc_fact_list = [
               {"fact_type": "multitemplate",
                "data": ["multitemplate", 0, "leaf2",
                      '''<native xmlns="http://cisco.com/ns/yang/Cisco-IOS-XE-native">
                           <router>
                             <bgp xmlns="http://cisco.com/ns/yang/Cisco-IOS-XE-bgp">
                               <neighbor>
                                 <id>{0}</id>
                                 <timers>
                                   <holdtime/>
                                 </timers>
                               </neighbor>
                             </bgp>
                           </router>
                         </native>
                ''',
                "nbr-hold-time", "timers",
                 ["holdtime"], 
                 ["hold-time"], [],
                 []]},

		{"fact_type": "multitemplate_protofact", 
		        "device_index": 0,
		        "device": 'NVE-EVPN-leaf1',
		        "assert_fact_for_each": 'Ep-list',
		        "hardcoded_list": ["device"],
		"path": '''
		      <System xmlns="http://cisco.com/ns/yang/cisco-nx-os-device">
		        <eps-items>
		          <epId-items>
	            	    <Ep-list>
		              <epId/>
		              <adminSt/>
		              <primaryIp/>
		              <peers-items>
		                <dy_peer-items>
		                  <DyPeer-list>
		                    <ip/>
		                    <state/>
		                    <mac/>
		                  </DyPeer-list>
  		                </dy_peer-items>
   		              </peers-items>
     		              </Ep-list>
		            </epId-items>
		        </eps-items>
		      </System>''',
		"protofact": {"template": 'nve-peer-list2',
		              "slots": {"device": 'NVE-EVPN-leaf1',
		                        "name": 'epId',
		                        "primary-ip": 'primaryIp',
		                        "admin-state": 'adminSt',
		                        "peer-ip": 'ip',
		                        "peer-state": 'state',
		                        "peer-mac": 'mac'
		                       },
		              "types": {"device": "str",
		                        "name": "str",
		                        "primary-ip": "str",
		                        "admin-state": "str",
		                        "peer-ip": "str",
		                        "peer-state": "str",
		                        "peer-mac": "str"
		                        } 
		        }
		}                                  

            :param index - Index into the nc_fact list id ddr-facts
            :param device_index - Index to the device_list defined in ddr-devices device_list
            :param num_params - Number of parameters to insert into NETCONF request
            :param par1 - First parameter
            :param par2 - Second parameter
            :param par3 - Third parameter
       
        Usage::

              (run_nc_fact_index 0 0 1 GigabitEthernet0/0 none none) ;This use netconf to get hold-time for the bgp-neighbor

            :raises none:

        """

        try:
            try:
                raw_fact = copy.deepcopy(self.control["nc-fact-list"][index])
        #
        # process multitemplate and multitemplate_protofact FACT types
        #
                try:
                    if raw_fact["fact_type"] == "multitemplate":
                        fact_data = raw_fact["data"]
                        fact_msg = fact_data[3]
                    else:
                        fact_msg = raw_fact["path"]
                except:
                    self.print_log("\n%%%% DDR Error: run_nc_fact_index 4405 fact copy error list index: " + str(index) + str(self.control["nc-fact-list"][index]))
                    return
                    
            except:
                self.print_log("\n%%%% DDR Error: run_nc_fact_index missing fact in nc_fact_list in ddr-facts list index: " + str(index))
                return
        #
        # substitute parameters in command string if included in call from rule
        #
            try:
                
                if num_params == 0:
                   formatted = str(fact_msg)
                elif num_params == 1:
                   formatted = str(fact_msg).format(str(par1))
                elif num_params == 2:
                   formatted = str(fact_msg).format(str(par1), str(par2))
                elif num_params == 3:
                   formatted = str(fact_msg).format(str(par1), str(par2), str(par2))
                else:
                    self.print_log("\n%%%% DDR Error: run_nc_fact_index invalid # arguments: " + str(num_params))
                    return

            except Exception as e:
                self.print_log("\n%%%% DDR Error: run_nc_fact_index format message: " + str(e))
                return
               
            if self.control["debug-fact"] == 1:
                self.print_log("**** DDR Debug: run_nc_facts_index device_index, facts: \ndevice_index:" + str(device_index) + " " + str(formatted))

            #
            # update FACT netconf request including any parameters provided
            #
            if raw_fact["fact_type"] == "multitemplate":
                fact_data[3] = formatted
                raw_fact["data"] = fact_data
                fact = raw_fact
            else: # fact type is multitemplate_protofact
                raw_fact["path"] = formatted
                fact = raw_fact
            
            try:
                device_info = self.control["device-list"][int(device_index)]
                device_id = str(device_info[4])
                if self.control["debug-fact"] == 1:
                    self.print_log("**** DDR Debug: run_nc_facts_index device_info " + str(device_info))
            except:
                self.print_log("\n%%%% DDR Error: run_nc_fact_index missing device definition in  ddr-devices index value: " + str(device_index))
                return

            if fact["fact_type"] == "multitemplate":
                self.get_template_multifacts_index(fact["data"], "device", device_id, device_index)
            elif fact["fact_type"] == "multitemplate_protofact":
                self.get_template_multifacts_protofact_index(fact, "device", device_id, device_index)
            else:
                self.print_log("\n%%%% DDR Error: run_nc_facts_index Error in ddr-facts definition: Invalid fact type: " + str(fact))

        except Exception as e:
            self.print_log("\n%%%% DDR Error: Exception in run_nc_fact_index: " + str(e))

    def cancel_ping(self, neighbor):
        """
            Helper function to generate a ping-cancel FACT to allow usecase to continue if ping fails
            :param neighbor - ip address of neighbor to ping
        
        Usage::

              self.cancel_ping(neighbor)

            :raises none:

        """
        template = self.env.find_template("ping-cancel") # get an empty template fact for event
        fact1 = {}
        fact1["neighbor"] = str(neighbor)
        fact1["cancel-ping"] = "TRUE" # event type defined for this Syslog message
        try:
            template.assert_fact(**fact1)
        except Exception as e:
            self.print_log("\n%%%% DDR ERROR: run_ping_action cancel error: " + str(e) + "\n")
        return
       
    def run_ping_action(self, device_index , vrf, neighbor, ping_count, min_success, src_interface, pkt_size, fact_result):
        """
            Rule file function call: (run_ping_action cli vrf ?neighbor count ?min-success ?src-interface 100 action-ping)
            
            Invoked in the "RHS" of a triggered rule.
            This function executes a ping to a remote address and asserts the results of the ping in a fact.
            The "ping result fact" can be used by additional rules in the workflow to make decisions.
            
            :param device_index - Use device index in ddr-devices to select mode cli/ssh run using python cli package or using ssh 
            :param vrf - vrf-name or none - do not include a vrf in the ping command
            :param neighbor - ip address of neighbor to ping
            :param count - number of pings to run
            :param min-success - minimum % of pings that must succeed
            :param src-interface - interface to run ping command through
            :param pkt_size - ping payload size in bytes
            :param fact_result - name of template to store results
        
        Usage::

              (run_ping_action none ?neighbor count ?min-success ?src-interface 100 action-ping-2)

            :raises none:

        """
        if self.control["actions"] == 1:
            try:
                access_type = str(self.control["device-list"][int(device_index)][5])
                if vrf == 'none':         
                    cmdline = 'ping ' + str(neighbor) + " source " + str(src_interface) + " size " + str(pkt_size) + " repeat " + str(ping_count)
                else:
                    cmdline = 'ping vrf ' + str(vrf) + ' ' + str(neighbor) + " source " + str(src_interface) + " size " + str(pkt_size) + " repeat " + str(ping_count)
                if self.control["debug-CLI"] == 1:
                    self.print_log("**** DDR Debug: run_ping_action: cmdline: " + str(cmdline))
    #
    # Use ssh to the device to perform the ping
    #F
                if access_type == 'ssh':
                    options = '-q -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oPubkeyAuthentication=no'
                    ssh_cmd = 'ssh %s@%s %s "%s"' % (self.control["mgmt-device"][2], self.control["mgmt-device"][0], options, cmdline)
                    if self.control["debug-CLI"] == 1:
                        self.print_log("**** DDR Debug: ssh run_ping_action command: " + str(ssh_cmd) + "\n")
                    try:
                        child = pexpect.spawn(ssh_cmd, timeout= 20, encoding='utf-8')
                        child.delaybeforesend = None
                        if self.control["debug-CLI"] == 1:
                            child.logfile = sys.stdout
                        child.expect(['\r\nPassword: ', '\r\npassword: ', 'Password: ', 'password: '])
                        child.sendline(self.control["mgmt-device"][3])
                        child.expect(pexpect.EOF) #result contains the ping result
                        result = child.before
                        child.close()

                    except Exception as e:
                        child.close()
                        self.print_log("\n%%%% DDR ERROR: ssh run_ping_action or timeout Error: " + str(ssh_cmd) + "\n")

                        self.cancel_ping(neighbor)
                        return
    #
    # ping is successful
    #
                    if self.control["debug-CLI"] == 1:
                        self.print_log("**** DDR Debug: ssh ping command result \n" + str(result)) #successful ping
                    try:
                        regex = 'Success rate is.{1}(?P<percent>(\d+))'
                        p1 = re.compile(regex)

                        for line in result.splitlines():
                            line = line.strip()
                            try:
                                results = p1.match(line)
                                group = results.groupdict()
                                percent = group['percent']
                                if self.control["debug-CLI"] == 1:
                                    self.print_log("**** DDR Debug: ssh run_ping_action: percent: " + str(percent))

                            except:  #skip lines that don't match the expected ping result
                                if self.control["debug-CLI"] == 1:
                                    self.print_log("**** DDR Debug: ssh run_ping_action: skipline: " + str(line))

                                if result.find("% Invalid source interface") != -1:
                                    self.cancel_ping(neighbor)
                                return
    #
    # Report ping success
    #
                        template = self.env.find_template(str(fact_result)) # get an empty template fact
                        fact1 = {}
                        fact1["neighbor"] = str(neighbor)
                        fact1["percent"] = int(percent)
                        if int(percent) >= int(min_success):
                            fact1["result"] = "TRUE" # event type defined for this Syslog message
                        else:
                            fact1["result"] = "FALSE" # event type defined for this Syslog message
                        try:
                            template.assert_fact(**fact1)
                        except Exception as e:
                            self.print_log("\n%%%% DDR ERROR: ssh run_ping_action cancel error: " + str(e) + "\n")
                        return
                        
                    except Exception as e:
                        self.print_log("\n%%%% DDR ERROR ssh run_ping_action command Error: " + str(e) + "\n" + str(message))
                        self.cancel_ping(neighbor)
                        return
    #
    # Use python cli package to perform ping
    #
                elif access_type == 'cli':
                    try:
                        if vrf == 'none':         
                            cmdline = 'ping ' + str(neighbor) + " source " + str(src_interface) + " size " + str(pkt_size) + " repeat " + str(ping_count)
                        else:
                             cmdline = 'ping vrf ' + str(vrf) + ' ' + str(neighbor) + " source " + str(src_interface) + " size " + str(pkt_size) + " repeat " + str(ping_count)
                        if self.control["debug-CLI"] == 1:
                            self.print_log("**** DDR Debug: cli run_ping_action: cmdline: " + str(cmdline))
                        try:
                            regex = 'Success rate is.{1}(?P<percent>(\d+))'
                            p1 = re.compile(regex)

                            result = cli.cli(cmdline, "30") # Run ping command to get ping results
                            for line in result.splitlines():
                                line = line.strip()
                                try:
                                    results = p1.match(line)
                                    group = results.groupdict()
                                    percent = group['percent']
                                    if self.control["debug-CLI"] == 1:
                                        self.print_log("**** DDR Debug: cli run_ping_action: percent: " + str(percent))
                                except:  #skip lines that don't match the expected ping result
                                    if self.control["debug-CLI"] == 1:
                                        self.print_log("**** DDR Debug: cli run_ping_action: skipline: " + str(line))

                                    if result.find("% Invalid source interface") != -1:
                                        self.print_log("\n%%%% DDR ERROR: cli run_ping_action command Error: Invalid source interface")
                                        self.cancel_ping(neighbor)
                                        return
                                    else: continue
    #
    # Report ping success
    #
                            template = self.env.find_template(str(fact_result)) # get an empty template fact
                            fact1 = {}
                            fact1["neighbor"] = str(neighbor)
                            fact1["percent"] = int(percent)

                            if int(percent) >= int(min_success):
                                fact1["result"] = "TRUE" # event type defined for this Syslog message
                            else:
                                fact1["result"] = "FALSE" # event type defined for this Syslog message
                            try:
                                template.assert_fact(**fact1)
                            except Exception as e:
                                if self.control["debug-fact"] == 1:
                                    self.print_log("**** DDR Debug: Error cli Assert run_ping_action FACT: " + str(message) + "\n" + str(e))
                                self.cancel_ping(neighbor)
                                return

                        except Exception as e:
                            self.print_log("\n%%%% DDR ERROR: run_ping_action CLI command Error: " + str(e) + "\n" + str(message))
                            self.cancel_ping(neighbor)
                            return

                    except Exception as e:
                        self.print_log("\n%%%% DDR ERROR: cli run_ping_action command Error: " + str(e) + "\n")
                        message = '(' + str(fact_result) + ' (neighbor ' + str(neighbor) + ') (percent 0) (result "FALSE"))'
                        if self.control["debug-fact"] == 1:
                            self.print_log("**** DDR Debug: cli Error Assert run_ping_action FACT: " + str(message))

                        self.cancel_ping(neighbor)
                        return
   #
   # else invalid mode 
   #
                else:
                    self.print_log("\n%%%% DDR ERROR: invalid run_ping_action type ")
                    self.cancel_ping(neighbor)
                    return

            except Exception as e:
                self.print_log("\n%%%% DDR ERROR: run_ping_action command Error: " + str(e) + "\n")
                self.cancel_ping(neighbor)
                return

    def run_process_file(self, file_fact_index, filename, device_index):
        """
            Rule file function call: (run_process_file 0 /bootflash/guest-share/emre_show-tech-acl)
            
            Invoked in the "RHS" of a triggered rule.
            Rule function that parses the content of "filename" stored in /bootflash/guest-share
            using the file_fact_list entry identified by file_fact_index to assert
            FACTs from the file content. A parser selects data from the file
            
            Each entry in the list is indexed by the file_fact_index parameter as a Python dictionary
             fact_type - The type of action function
             device - optionally directly specify the device name otherwise management device used by default
             genie_parser - Name of parser class in genie_parsers.py for processing this show command
             assert_fact_for_each_item_in - Assert a fact for each dictionary entry key at this level in the dictionary
             protofact - prototype describing how to translate parser python dictionary output to a CLIPs fact
             template - name of the deftemplate in ddr-rules defining the fact
             slots - names of the slots/fields in the deftemplate for the fact
             $+datetime - Date and time extracted from the syslog message
             $+facility - type of syslog message, e.g. "SEC_LOGIN"
             $+level - Syslog level 0 - 7. e.g 5
             $+message - text to right of the level, e.g. "LOGIN_SUCCESS"
             $+note - free from text with remainder of syslog message 
           
              {"fact_type": "run_process_file",
               "device": "CAT9K-24",
               "genie_parser": "ShowTechAclPlatform",
               "assert_fact_for_each_item_in": "acl_platform",
               "protofact": {"template": "acl-tech-platform-info",
                               "slots": {"switch": "$",
                                         "model": "$+model",
                                         "serial": "$+serial",
                                         "mac": "$+mac",
                                         "hwver": "$+hwver",
                                         "swver": "$+swver"}
                            }    
              },
            
            :param file_fact_index: 0 origin index into the file_fact_list in the ddr-facts file
            :param filename: full pathname for processed to generate facts
            :param device_index: index for device to use in ddr-devices
        
        Usage::

              (run_process_file 0 /bootflash/guest-share/emre_show-tech-acl 0) ; Get device image
              (run_process_file 1 /bootflash/guest-share/emre_show-tech-acl 0) ; Get platform information
              (run_process_file 2 /bootflash/guest-share/emre_show-tech-acl 0) ; Get ACL names
              (run_process_file 3 /bootflash/guest-share/emre_show-tech-acl 0) ; Get ACL counters

            :raises none:

        """
        if self.control["actions"] == 1:
            access_type = str(self.control["device-list"][int(device_index)][5])
            command_idx = int(file_fact_index)
            timestamp =  datetime.now().strftime("%m-%d-%Y_%H:%M:%S.%f")
            
        # Read the contents fo the file
        #
            try:
                with open(filename, "r") as fd:
                    file_data = fd.read()
                    fd.close()
                    if self.control["debug-file"] == 1:
                        self.print_log("\n**** DDR Debug: run_process_file content: \n" + str(file_data))
            except Exception as e:
                self.print_log("\n%%%% DDR: run_process_file: Error reading file: " + str(e) + "\n" + str(filename))
                return

    #
    # process file content and generate FACTs
    #
            try:
                fact = self.control["file-fact-list"][command_idx]
                parser = genie_str_to_class(fact["genie_parser"])
                if parser == None:
                    self.print_log("\n%%%% DDR Error: Exception in run_process_file: parser not found: \n" + str(fact["genie_parser"]) + " \n")
                    return
                try:
                    parsed_genie_output = parser.parse(output=file_data)
                except Exception as e:
                    self.print_log("\n%%%% DDR Error: Exception in run_process_file parsing: \n" + str(output) + " \n" + str(parsed_genie_output))
                
                if self.control["debug-parser"] == 1:
                    self.print_log("\n*** DDR Debug: run_process_file parsed output:\n" + str(parsed_genie_output))

                sub_dictionary_list = self.find(fact["assert_fact_for_each_item_in"], parsed_genie_output)
                import copy
                for sub_dictionary in sub_dictionary_list:
                    for item in sub_dictionary:
                        protofact = copy.deepcopy(fact["protofact"])
                        for slot in protofact["slots"]:
                            value = protofact["slots"][slot]
	#
	# insert the device name into the fact using name in ddr-devices
	#
                            if value == "device":
                                protofact["slots"][slot] = value.replace("device", str(self.control["device-list"][int(device_index)][4]))

                            elif type(value) == str and "$" in value:
                                protofact["slots"][slot] = value.replace("$", item)
                        self.assert_template_fact(protofact, 'device', str(self.control["device-list"][int(device_index)][4]), sub_dictionary)
            except Exception as e:
                self.print_log("\n%%%% DDR Error: Exception in run_process_file assert_fact response processing: " + str(fact) + "\n" + str(parsed_genie_output) + "\n" + str(e))

    def run_read_control_file(self, control_file, delay):
        """
            Rule function call: (run_read_control_file "filename" delay)
            
            Invoked in the "RHS" of a triggered rule.
            This function looks in the /bootflash/guest-share/ddr with the name 'filename'
            This file is written to the guest-share by an external system and results in assertion
            of facts defined in the file.
            These facts are asserted into the CLIPs knowledge base and processed by the usecase CLIPs rules.
            This function would typically be used to look for information/instructions used to control
            the execution of the facts.
            If no file is present, the function returns with no error.  Typically the rule implementation would
            run_wait for a number of seconds then look for external facts again.
                        
            :param action_file: string with name of file in the guest-share/ddr directory
            :param delay: delay in seconds to wait before returning
        
        Usage::

              (run_read_control_file "ddr-control" 10)

            :raises none:

        """
        #
        # Wait for the ddr-control file to exist
        #
        try:
            while not os.path.exists(str(control_file)):
                if self.control["debug-fact"] == 1:
                    self.print_log("**** DDR Debug: wait for ddr-control file exists: " + str(os.path.exists(str(control_file))))
                time.sleep(delay)
        
        except Exception as e:
            self.print_log('\n%%%% DDR Error: run_read_control_file wait for ddr-control error: ' + str(control_file) + " " + str(e))
        
        try:
            with open(str(control_file)) as file:
                self.action_data = file.readlines()
                self.action_data = [line.rstrip() for line in self.action_data]
                if self.control["debug-fact"] == 1:
                    self.print_log("**** DDR Debug: run_read_control_file file content: \n" + str(self.action_data))
                file.close()
        #
        # assert each fact in the file content
        #
                for line in self.action_data:
                    try:
                        if self.control["debug-fact"] == 1:
                            self.print_log("**** DDR Debug: control-fact: " + str(line))                                         
                        self.env.assert_string(str(line))
                    except Exception as e: 
                        if self.control["debug-fact"] == 1:
                            self.print_log("**** DDR Debug: run_read_control_file exceptionF: \n" + str(line) + " " + str(e))
                        pass # ignore if fact already exists in knowledge base

        except Exception as e:
            self.print_log('\n%%%% DDR Error: run_read_control_file error: ' + str(control_file) + " " + str(e))
            
        if delay == 0:
            return
        else:
            time.sleep(delay)
            return

    def run_rule_timer(self):
        """
            Rule file function call: (run_rule_timer)
            
            Invoked in the "RHS" of a triggered rule.
            This action function starts a thread to generate a timer to assert a FACT on
            a regular time interval. When the timer is running FACTs are generated of the form:
            
              (timer-fact-name (timer TRUE) (time 4596)
              
              Where timer-fact-name is the name of a deftemplate defining a fact that should be asserted automatically 
              based on the timer execution.  "timer-fact-name" is defined in ddr-flags in the "timerFactName" parameter                       

            :param none:
        
        Usage::

              (run_rule_timer)

            :raises none:

        """

        try:
            timestamp =  datetime.now().strftime("%m-%d-%Y_%H:%M:%S.%f")
            
            hms = datetime.now().strftime("%H:%M:%S")
            h, m, s = [int(i) for i in hms.split(':')]
            timer_time = int(3600*h + 60*m + s)
        #
        # Build timer fact
        #
            message = '(' + str(self.control["timer-fact-name"]) + ' (timer TRUE) (time ' + str(timer_time) + '))'
            if self.control["debug-fact"] == 1:
                self.print_log("**** DDR Debug: run_time fact: " + str(message))
            try:
                self.env.assert_string(message)
                self.env.run() #Run CLIPs to execute RULEs activated as a result of asserting the timer FACT
            except Exception as e:
                self.print_log("\n%%%% DDR: Exception in run_time: " + str(e))
    #
    # Run the timer
    #
            Timer(float(self.control["fact-timer"]), self.rule_timer).start()
        except Exception as e:
            self.print_log("\n%%%% DDR: run_timer exception: " + str(e))

    def run_set_rule_step(self, rule_step):
        """
            Rule file function call: (run_set_rule_step step-name)
            
            Invoked in the "RHS" of a triggered rule.
            Directly sets the next step in the rule-step FACT  
            
            :param step-name: string with name of the next step to execute in ddr-rules
        
        Usage::

              (run_set_rule_step step-1)

            :raises none:

        """
        if self.control["actions"] == 1:
           
            try:
                step_fact = '(rule-step (step ' + str(rule_step) + '))'
                if self.control["debug-fact"] == 1:
                    self.print_log("**** DDR Debug: run_set_rule_step: " + str(step_fact))
                self.env.assert_string(str(step_fact))  
            except Exception as e:
                self.print_log("\n%%%% DDR ERROR: run_set_rule_step: Exception setting step: " + str(step_fact) + " " + str(e))

    def run_set_runmode(self, runmode, runone):
        """
            Rule file function call: (run_set_runmode RUNMODE RUNONE)
            
            Invoked in the "RHS" of a triggered rule.
            This function sets the ddr run-mode control variable
            The function allows a rule to change the execution mode, for example from "0" for timed execution
            to "2" for trigger off NETCONF notification
            
            :param runmode: 0/timed execution, 1/trigger on syslog, 2/trigger on NETCONF notification 
            :param runone: 0/run until terminated, 1/run until main loop runs DDR 
        
        Usage::

              (run_set_runmode 0 1) Set run-mode to 0 for timed triggering and set run-one to 1 to terminate execution after next cycle

            :raises none:

        """
        try:
            if self.control["debug-fact"] == 1:
                self.print_log("**** DDR Debug: run_set_runmode change mode to: " + str(runmode))
            self.control["run-mode"] = int(runmode)
            self.control["run-one"] = int(runone)
        except Exception as e:
            self.print_log("%%%% ERROR run_set_runmode change mode value: " +str(runmode) + " " + str(e))        
               
    def run_show_fact(self, fact_index):
        """
            Rule file function call: (run_show_fact fact_index)

            Invoked in the "RHS" of a triggered rule.
            Runs a show command defined in the ddr-facts file "show_fact_list".  The show_fact_list entry defines the 
            show command to execute and how to generate FACTs from the result

            :param fact_index: 0 origin index into the show_fact_list with execution instructions
        
        Usage::

              (run_show_fact_list 0)

            :raises none:

        """

        if self.control["debug-fact"] == 1:
            self.print_log("**** EMRE Debug: run_show_facts entry: " + str(fact_index))
        fact = self.control["show-fact-list"][int(fact_index)]
        if self.control["debug-fact"] == 1:
                self.print_log("**** EMRE Debug: run_show_facts fact: " + str(fact))
        self.show_and_assert_fact(fact)
      
    def run_show_fact(self, fact_index, device_index):
        """
            Rule file function call: (run_show_fact fact_index, device_index)

            Invoked in the "RHS" of a triggered rule.
            Runs a show command defined in the ddr-facts file "show_fact_list".  The show_fact_list entry defines the 
            show command to execute and how to generate FACTs from the result

            :param fact_index: 0 origin index into the show_fact_list with execution instructions
            :param device_index: index into the ddr-devices list to used to select local cli or ssh device access
        
        Usage::

              (run_show_fact_list 0 0)

            :raises none:

        """

        if self.control["debug-fact"] == 1:
            self.print_log("**** EMRE Debug: run_show_facts entry: " + str(fact_index))
        fact = self.control["show-fact-list"][int(fact_index)]
        if self.control["debug-fact"] == 1:
                self.print_log("**** EMRE Debug: run_show_facts fact: " + str(fact))
        self.show_and_assert_fact(fact, device_index)
      
    def run_show_parameter(self, fact_index, device_index, show_template, pcount, par1, par2, par3, no_fact, yes_fact):
        """
            Rule file function call: (run_show_parameter 3 0 "show platform software fed active vp key {0} {1}" 2 ?key ?vlan none no-fact no-fact)

            Invoked in the "RHS" of a triggered rule.
            Runs a show command in the guestshell on the management device using the
            Python "cli" package installed by default in the guestshell or ssh connection to a device  
            
            The actions performed by the method are defined in the 'show_parameter_fact_list' list in the ddr-facts file.
            Each entry in the list is indexed by the fact_index parameter as a Python dictionary
              fact_type - The type of action function
              device - optionally directly specify the device name otherwise management device used by default
              genie_parser - Name of parser class in genie_parsers.py for processing this show command
              assert_fact_for_each_item_in - Assert a fact for each dictionary entry key at this level in the dictionary
              protofact - prototype describing how to translate parser python dictionary output to a CLIPs fact
              template - name of the deftemplate in ddr-rules defining the fact
              slots - names of the slots/fields in the deftemplate for the fact
              $ - the value of the key in the dictionary.  In the example below local-host is the key and the slot 'local-host' is set to the key value
              $+neighbor - set the neighbor slot in the bgp-keepalive-messages fact to the value of 'neighbor' in the current dictionary entry
            
                {"fact_type": "run_show_parameter",
                 "device": "513E.C.24-9300-2",
                 "genie_parser": "ShowMonitorCapture",
                 "assert_fact_for_each_item_in": "bgp_keepalive",
                 "protofact": {"template": "bgp-keepalive-messages",
                                   "slots": {"local-host": "$",
                                   "neighbor": "$+neighbor",
                                   "message": "$+message"
                                            }
                              }
            }

            :param fact_index: 0 origin index into the show_parameter_fact_list with execution instructions
            :param device_index: index into ddr-devices to select access type 'cli' to use the Python cli package in the guestshell or 'ssh' for ssh device access
            :param show_template: show command template with parameters identified by {0}, {1}, {2}
            :param pcount: Number of parameters to substitute in the show_template 0 to 3
            :param parx: Values for parmeters 1, 2 and 3 which can be variables from the rule or defined values
            :param no_fact: Optional fact to assert if this function did not assert a fact that met rule conditions
            :param yes_fact: Optional fact to assert if this function did assert a fact that met rule conditions
        
        Usage::

              (run_show_parameter 3 ssh "show monitor capture CAP buffer brief" 0 none none none "(fact-found (value false))" "(fact-found (value true))")
              (run_show_parameter 3 ssh "show platform software fed active vp key {0} {1}" 2 ?key ?vlan none no-fact no-fact)

            :raises none:

        """
        if self.control["actions"] == 1:
            access_type = str(self.control["device-list"][int(device_index)][5])
            index = int(fact_index)
            fact = self.control["show-parameter-fact-list"][int(index)]
            timestamp =  datetime.now().strftime("%m-%d-%Y_%H:%M:%S.%f")
            
        #######################################################################
        #
        # Generate the show command with parameters
        #
        #######################################################################
            try:
                cmdline = str(show_template)
                if self.control["debug-CLI"] == 1:
                    self.print_log("**** DDR Debug: run_show_parameter: cmdline: " + str(cmdline))
        #
        # substitute parameters in command string if included in call from rule
        #
                if int(pcount) == 0:
                    command = str(cmdline)
                else:
                    if int(pcount) == 1:
                        command = str(show_template).format(str(par1))
                    elif int(pcount) == 2:
                        command = str(show_template).format(str(par1), str(par2))
                    elif int(pcount) == 3:
                        command = str(show_template).format(str(par1), str(par2), str(par3))
                try:
                    if self.control["debug-CLI"] == 1:
                        self.print_log("**** DDR Debug: run_show_parameter: command: " + str(command))
        #
        # If access-type is cli use the Python CLI package to run command
        #
                    if str(access_type) == 'cli':
                        response = cli.cli(command, "30")
                        if self.control["debug-CLI"] == 1:
                            self.print_log("**** DDR Notice: CLI Command Executed: " + str(command) + " at: " + str(timestamp))
                            self.print_log("**** DDR Notice: CLI Command Response: \n" + str(response))
                    else:
        #
        # Use SSH to run the show command
        #
                        try:
                            options = '-q -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oPubkeyAuthentication=no'
                            ssh_cmd = 'ssh %s@%s %s "%s"' % (self.control["mgmt-device"][2], self.control["mgmt-device"][0], options, command)
                            if self.control["debug-CLI"] == 1:
                                self.print_log("**** DDR Debug: run_show_parameter SSH command: " + str(ssh_cmd) + "\n")
                            child = pexpect.spawn(ssh_cmd, timeout= 20, encoding='utf-8')
                            child.delaybeforesend = None
                            if self.control["debug-CLI"] == 1:
                                child.logfile = sys.stdout
                            child.expect(['\r\nPassword: ', '\r\npassword: ', 'Password: ', 'password: '])
                            child.sendline(self.control["mgmt-device"][3])
                            child.expect(pexpect.EOF) #result contains the ping result
                            response = child.before
                            if self.control["debug-CLI"] == 1:
                                self.print_log("**** DDR Notice: CLI Command Response:\n " + str(response))

                        except Exception as e:
                            child.close()
                            self.print_log("\n%%%% DDR ERROR: run_show_parameter SSH or timeout Error: " + str(ssh_cmd) + "\n")
                        child.close()
         #
         # Assert the FACTs for using the response
         #
                    fact_found = False
                    try:
                        if "error" in response:
                            self.print_log("\n%%%% DDR Error: error in run_show_parameter show command response")
                            return

                        parser = genie_str_to_class(fact["genie_parser"])
                        if parser == None:
                            self.print_log("\n%%%% DDR Error: run_show_parameter: parser not found: \n" + str(fact["genie_parser"]) + " \n")
                            return
    #
    # Parse data if the parser is available
    #
                        try:
                            parsed_genie_output = parser.parse(output=response)
                        except Exception as e:
                            self.print_log("\n%%%% DDR Error: Exception in run_show_parameter parser not found: " + str(fact["genie_parser"]))
                            return
                            
                        if self.control["debug-parser"] == 1:
                            self.print_log("**** DDR Debug: run_show_parameter parsing result: \n\n" + str(parsed_genie_output) + "\n")

                        if self.control["debug-CLI"] == 1:
                            self.print_log("**** DDR Debug: run_show_parameter genie: " + str(parsed_genie_output) + "\n")
                        if parsed_genie_output == {}:
                            self.print_log("\n%%%% DDR Error: run_show_parameter: no parser output")
                            return
        #
        # Get the starting point in the Python dictionary containing the parsed content
        #
                        sub_dictionary_list = self.find(fact["assert_fact_for_each_item_in"], parsed_genie_output)
                        import copy
                        for sub_dictionary in sub_dictionary_list:
                            for item in sub_dictionary:
                                protofact = copy.deepcopy(fact["protofact"])
                                for slot in protofact["slots"]:
                                    value = protofact["slots"][slot]
	#
	# insert the device name into the fact
	#
                                    if value == "device":
                                        protofact["slots"][slot] = value.replace("device", str(self.control["device-list"][int(device_index)][4]))
                                    elif type(value) == str and "$" in value:
                                        if isinstance(value.replace("$", item), int):
                                            protofact["slots"][slot] = int(value.replace("$", item))
                                        else:                                      
                                            protofact["slots"][slot] = str(value.replace("$", item))
                                fact_found = True
                                self.assert_template_fact(protofact, 'device', str(self.control["device-list"][int(device_index)][4]), sub_dictionary)

                    except Exception as e:
                        self.print_log("\n%%%% DDR Error: Exception in run_show_parameter response processing: " + str(e))
                        return

                except Exception as e:
                    self.print_log("\n%%%% DDR ERROR: run_show_parameter Error: Exception sending show command: " + str(e) + "\n")
            except Exception as e:
                self.print_log("\n%%%% DDR ERROR: run_show_parameter Error: Exception generating show command: " + str(e) + "\n")    

    def run_show_parameter_index(self, fact_index, device_index, show_template, pcount, par1, par2, par3, no_fact, yes_fact):
        """
            Rule file function call: (run_show_parameter_index 3 1 "show platform software fed active vp key {0} {1}" 2 ?key ?vlan none no-fact no-fact)

            Invoked in the "RHS" of a triggered rule.
            Runs a show command in the guestshell on the management device using the
            Python "cli" package installed by default in the guestshell or ssh connection to a device  
            
            The actions performed by the method are defined in the 'show_parameter_fact_list' list in the ddr-facts file.
            Each entry in the list is indexed by the fact_index parameter as a Python dictionary
              fact_type - The type of action function
              device - optionally directly specify the device name otherwise management device used by default
              genie_parser - Name of parser class in genie_parsers.py for processing this show command
              assert_fact_for_each_item_in - Assert a fact for each dictionary entry key at this level in the dictionary
              protofact - prototype describing how to translate parser python dictionary output to a CLIPs fact
              template - name of the deftemplate in ddr-rules defining the fact
              slots - names of the slots/fields in the deftemplate for the fact
              $ - the value of the key in the dictionary.  In the example below local-host is the key and the slot 'local-host' is set to the key value
              $+neighbor - set the neighbor slot in the bgp-keepalive-messages fact to the value of 'neighbor' in the current dictionary entry
            
                {"fact_type": "run_show_parameter",
                 "device": "513E.C.24-9300-2",
                 "genie_parser": "ShowMonitorCapture",
                 "assert_fact_for_each_item_in": "bgp_keepalive",
                 "protofact": {"template": "bgp-keepalive-messages",
                                   "slots": {"local-host": "$",
                                   "neighbor": "$+neighbor",
                                   "message": "$+message"
                                            }
                              }
            }

            :param fact_index: 0 origin index into the show_parameter_fact_list with execution instructions
            :param device_index: 0 origin index devices list to target execution, only ssh access_type if not 0
            :param show_template: show command template with parameters identified by {0}, {1}, {2}
            :param pcount: Number of parameters to substitute in the show_template 0 to 3
            :param parx: Values for parmeters 1, 2 and 3 which can be variables from the rule or defined values
            :param no_fact: Optional fact to assert if this function did not assert a fact that met rule conditions
            :param yes_fact: Optional fact to assert if this function did assert a fact that met rule conditions
        
        Usage::

              (run_show_parameter 3 1 "show monitor capture CAP buffer brief" 0 none none none "(fact-found (value false))" "(fact-found (value true))")
              (run_show_parameter 3 0 "show platform software fed active vp key {0} {1}" 2 ?key ?vlan none no-fact no-fact)

            :raises none:

        """
        if self.control["actions"] == 1:
            index = int(fact_index)
            fact = self.control["show-parameter-fact-list"][int(index)]
            timestamp =  datetime.now().strftime("%m-%d-%Y_%H:%M:%S.%f")
        #
        # Get the device access information using the device_index
        #
            device_info = self.control["device-list"][int(device_index)]
            device_address = str(device_info[0])
            device_user = str(device_info[2])
            device_pass = str(device_info[3])
            device_id = str(device_info[4])
            device_access = str(device_info[5])
            
        #######################################################################
        #
        # Generate the show command with parameters
        #
        #######################################################################
            try:
                cmdline = str(show_template)
                if self.control["debug-CLI"] == 1:
                    self.print_log("**** DDR Debug: run_show_parameter_index: cmdline: " + str(cmdline))
        #
        # substitute parameters in command string if included in call from rule
        #
                if int(pcount) == 0:
                    command = str(cmdline)
                else:
                    if int(pcount) == 1:
                        command = str(show_template).format(str(par1))
                    elif int(pcount) == 2:
                        command = str(show_template).format(str(par1), str(par2))
                    elif int(pcount) == 3:
                        command = str(show_template).format(str(par1), str(par2), str(par3))
                try:
                    if self.control["debug-CLI"] == 1:
                        self.print_log("**** DDR Debug: run_show_parameter_index: command: " + str(command))
        #
        # If access-type is cli use the Python CLI package to run command
        # The device_index must be 0 to run on the local device
        #
                    if device_access == 'cli':
                        if device_index == 0:
                            response = cli.cli(command, "30")
                            if self.control["debug-CLI"] == 1:
                                self.print_log("**** DDR Notice: CLI Command Executed: " + str(command) + " at: " + str(timestamp))
                                self.print_log("**** DDR Notice: CLI Command Response: \n" + str(response))
                        else:
                            self.print_log("\n%%%% DDR ERROR: run_show_parameter_index: device_index not 0 for local access: " + str(device_index))
                            return                         
                    else:
        #
        # Use SSH to run the show command
        # The device_index is used to select the target device
        #
                        try:
                            options = '-q -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oPubkeyAuthentication=no'

                            ssh_cmd = 'ssh %s@%s %s "%s"' % (device_user, device_address, options, command)
                            if self.control["debug-CLI"] == 1:
                                self.print_log("**** DDR Debug: run_show_parameter_index SSH command: " + str(ssh_cmd) + "\n")
                            child = pexpect.spawn(ssh_cmd, timeout= 20, encoding='utf-8')
                            child.delaybeforesend = None
                            if self.control["debug-CLI"] == 1:
                                child.logfile = sys.stdout
                            child.expect(['\r\nPassword: ', '\r\npassword: ', 'Password: ', 'password: '])
                            child.sendline(device_pass)
                            child.expect(pexpect.EOF) #result contains the ping result
                            response = child.before

                        except Exception as e:
                            child.close()
                            self.print_log("\n%%%% DDR ERROR: run_show_parameter_index SSH or timeout Error: " + str(ssh_cmd) + "\n")
                            return
                            
                        child.close()
         #
         # Assert the FACTs for using the response
         #
                    fact_found = False
                    try:
                        if "error" in response:
                            self.print_log("\n%%%% DDR Error: error in run_show_parameter_index show command response")
                            return
                        parser = genie_str_to_class(fact["genie_parser"])
                        if parser == None:
                            self.print_log("\n%%%% DDR Error: run_show_parameter_index: parser not found: \n" + str(fact["genie_parser"]) + " \n")
                            return
    #
    # Parse data if the parser is available
    #
                        try:
                            parsed_genie_output = parser.parse(output=response)
                        except Exception as e:
                            self.print_log("\n%%%% DDR Error: Exception in run_show_parameter_index parser not found: " + str(fact["genie_parser"]))
                            return
                        if self.control["debug-parser"] == 1:
                            self.print_log("**** DDR Debug: run_show_parameter_index parsing result: \n\n" + str(parsed_genie_output) + "\n")

                        if self.control["debug-CLI"] == 1:
                            self.print_log("**** DDR Debug: run_show_parameter_index genie: " + str(parsed_genie_output) + "\n")
                        if parsed_genie_output == {}:
                            self.print_log("\n%%%% DDR Error: run_show_parameter_index no parser output")
                            return
        #
        # Get the starting point in the Python dictionary containing the parsed content
        #
                        sub_dictionary_list = self.find(fact["assert_fact_for_each_item_in"], parsed_genie_output)
                        import copy
                        for sub_dictionary in sub_dictionary_list:
                            for item in sub_dictionary:
                                protofact = copy.deepcopy(fact["protofact"])
                                for slot in protofact["slots"]:
                                    value = protofact["slots"][slot]
	#
	# insert the device name into the fact
	#
                                    try:
                                        if value == "device":
                                            protofact["slots"][slot] = value.replace("device", str(self.control["device-list"][int(device_index)][4]))
                                        elif type(value) == str and "$" in value:
                                            if isinstance(value.replace("$", item), int):
                                                protofact["slots"][slot] = int(value.replace("$", item))
                                            else:                                      
                                                protofact["slots"][slot] = str(value.replace("$", item))
                                    except Exception as e:
                                        self.print_log("\n%%%% DDR Error: Exception in run_show_parameter_index slot replace: " + str(protofact["slots"][slot]) + " " + str(e))

                                fact_found = True
                                self.assert_template_fact(protofact, 'device', str(self.control["device-list"][int(device_index)][4]), sub_dictionary)

                    except Exception as e:
                        self.print_log("\n%%%% DDR Error: Exception in run_show_parameter_index response processing: " + str(e))
                        return

                except Exception as e:
                    self.print_log("\n%%%% DDR ERROR: run_show_parameter_index Error: Exception sending show command: " + str(e) + "\n")
            except Exception as e:
                self.print_log("\n%%%% DDR ERROR: run_show_parameter_index Error: Exception generating show command: " + str(e) + "\n")    

    def run_suffix(self, fact_string, fact_prefix):
        """
            Rule function call: (run_suffix fact_string fact_prefix)
            
            Invoked in the "RHS" of a triggered rule.
            This function accepts a fact_string and removes the fact_prefix from the string returning the suffix
                                    
            :param fact_string: Text string variable containing the full fact string
            :param fact_prefix: Text string variable containing the prefix to remove from the fact_string
        
        Usage::

              (run_suffix tunnel-id100 tunnel-id)

            :raises none:
            :returns: suffix or remainder of string after removing the prefix or ERROR

        """

        try:
            result = fact_string.split(fact_prefix)
            if len(result) == 2:
                return result[1]  # return the suffix
            else:
                return
                
        except Exception as e:
             self.print_log("\n%%%% DDR ERROR: run_prefix string/prefix: " + str(fact_sring) + " " + (fact_prefix) + "\n" + str(e))

    def cancel_trace(self, neighbor):
        """
            Helper function to generate a trace-cancel FACT to allow usecase to continue if traceroute fails
            :param neighbor - ip address of neighbor to ping
        
        Usage::

              self.cancel_trace(neighbor)

            :raises none:

        """
        template = self.env.find_template("trace-cancel") # get an empty template fact for event
        fact1 = {}
        fact1["neighbor"] = str(neighbor)
        fact1["cancel-trace"] = "TRUE"
        try:
            template.assert_fact(**fact1)
        except Exception as e:
            self.print_log("\n%%%% DDR ERROR: run_trace cancel error: " + str(e) + "\n")
        return

    def trace_failed(self, neighbor):
        """
            Helper function to generate a traceroute failed FAC
            :param neighbor - ip address of neighbor to ping
        
        Usage::

              self.trace_failed(neighbor)

            :raises none:

        """
        template = self.env.find_template("action-trace") # get an empty template fact
        fact1 = {}
        fact1["neighbor"] = str(neighbor)
        fact1["result"] = "DOWN"
        try:
            template.assert_fact(**fact1)
        except Exception as e:
            self.print_log("\n%%%% DDR ERROR: ssh run_trace failed fact assert error: " + str(e) + "\n")
        return
  
    def run_trace(self, mode, vrf, trace_type, neighbor, src_interface, probe, timeout, ttl, trace_timeout, all_addresses, fact_result):
        """
            Rule file function call: (run_trace ?mode ?vrf ?trace-type ?neighbor ?src-interface ?probe ?timeout ?ttl ?trace-timeout ?all-addresses ?syslog-time action-trace)
            
            Invoked in the "RHS" of a triggered rule.
            This function executes a trace route to a remote address and asserts the results of the trace route in a fact.
            Sample command to send to device: traceroute [vrf Mgmt-vrf] ip 172.24.115.25 source GigabitEthernet0/0 probe 1 timeout 1
            The "action_trace fact" can be used by additional rules in the workflow to make decisions.
            Sample trace FACT: 
              (action-trace (neighbor 10.1.7.2) (hop 8) (last-ip 10.1.7.2) (result TRUE))
            
            :param mode: cli/ssh run using python cli package or using ssh 
            :param vrf: vrf-name or none - do not include a vrf in the ping command
            :param trace_type: default is 'ip'
            :param neighbor: ip address of trace neighbor endpoint
            :param src-interface: interface to run trace command through
            :param probe: integer typically 1 for number of times to try each path before failing
            :param timeout: timeout in seconds for complete trace
            :param trace-timeout: timeout in seconds for each trace hop
            :param ttl: maximum trace hop length
            :param all_addresses: TRUE to return all addresses in path, FALSE to return last address only
            :param fact_result: name of deftemplate that will be used to generate the trace route result fact
        
        Usage::

              (run_trace ?mode ?vrf ?trace-type ?neighbor ?src-interface ?probe ?timeout ?ttl ?trace-timeout ?all-addresses ?syslog-time action-trace)

            :raises none:

        """
        if self.control["actions"] == 1:
            try:
                if vrf == 'none':         
                    cmdline = 'traceroute ' + str(trace_type) + " " + str(neighbor) + " source " + str(src_interface) + " probe " + str(probe) + " timeout " + str(timeout) + " ttl 1 " + str(ttl)
                else:
                    cmdline = 'traceroute vrf ' + str(vrf) + " " + str(trace_type) + " " + str(neighbor) + " source " + str(src_interface) + " probe " + str(probe) + " timeout " + str(timeout) + " ttl 1 " + str(ttl)
                if self.control["debug-CLI"] == 1:
                    self.print_log("**** DDR Debug: run_trace: cmdline: " + str(cmdline))
    #
    # Use ssh to the device to perform the traceroute
    #
                if mode == 'ssh':
                    options = '-q -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oPubkeyAuthentication=no'
                    ssh_cmd = 'ssh %s@%s %s "%s"' % (self.control["mgmt-device"][2], self.control["mgmt-device"][0], options, cmdline)
                    if self.control["debug-CLI"] == 1:
                        self.print_log("**** DDR Debug: run_trace SSH command: " + str(ssh_cmd) + "\n")
                    try:
                        child = pexpect.spawn(ssh_cmd, timeout= trace_timeout, encoding='utf-8')
                        child.delaybeforesend = None
                        if self.control["debug-CLI"] == 1:
                            child.logfile = sys.stdout
                        child.expect(['\r\nPassword: ', '\r\npassword: ', 'Password: ', 'password: '])
                        child.sendline(self.control["mgmt-device"][3])
                        child.expect(pexpect.EOF) #result contains the traceroute result
                        result = child.before
                    except Exception as e:
                        child.close()
                        self.print_log("\n%%%% DDR ERROR: run_trace ssh or timeout Error: " + str(ssh_cmd) + "\n")
                        self.cancel_trace(neighbor)
                        return

                    child.close()
                    if self.control["debug-CLI"] == 1:
                        self.print_log("**** DDR Debug: ssh traceroute command result \n", result) #successful traceroute execution
    #
    # Use python cli package to perform the traceroute
    #
                elif mode == 'cli':
                    try:
                        result = cli.cli(cmdline, "30") # Run trace command
                    except Exception as e:
                        self.print_log("\n%%%% DDR ERROR: run_trace cli Error: " + str(cmdline) + "\n" + str(e))

                        self.trace_failed(neighbor)
                        self.cancel_trace(neighbor)
                        return
    #
    # If invalid traceroute mode is selected
    #
                else:
                    self.print_log("\n%%%% DDR ERROR: run_trace Error: Invalid mode" + str(mode))
                    self.trace_failed(neighbor)
                    self.cancel_trace(neighbor)
                    return

    #############################################################
    #
    #  The traceroute reply is now in the "result" string variable
    #  Select information and build the result FACT
    #
    # ddmi-cat9300-2#traceroute vrf Mgmt-vrf ip 172.27.255.23 source GigabitEthernet0/0 probe 1 timeout 1 ttl 1 3
    # Type escape sequence to abort.
    # Tracing the route to 172.27.255.23
    # VRF info: (vrf in name/id, vrf out name/id)
    #   1 172.27.255.23 3 msec
    #
    #############################################################
                ipaddress = 'none'
                hop = 'none'
                domain = 'none'
                last_address = 'none'
                p1 = re.compile('(?P<hop>(\d+)).{1}(?P<address>(\S+)).*')
                p2 = re.compile('(?P<nohop>(\d+)).{1}(?P<failed>(\S+)).*')

                if self.control["debug-CLI"] == 1:
                    self.print_log("**** DDR Debug: run_trace: result: " + str(result))
                for line in result.splitlines():
                    line = line.strip()
                    try:
                        results = p1.match(line)
                        group = results.groupdict()
                        hop = group['hop']
                        ipaddress = group['address'].strip().lstrip('(').rstrip(')')
                        if str(ipaddress) != '*':
                            last_address = ipaddress        
                        if self.control["debug-CLI"] == 1:
                            self.print_log("**** DDR Debug: run_trace: hop address: " + str(hop) + " "+ str(ipaddress))

                    except Exception as ex: #skip lines that don't match the expected trace result
                        if self.control["debug-CLI"] == 1:
                            self.print_log("**** DDR Debug: run_trace: skipline: " + str(line))
                        continue
    #
    # After parsing the response for the traceroute results, check to see if the interface was down
    #
                if result.find("% Invalid source interface") != -1:
                    message = '(' + str(fact_result) + ' (neighbor ' + str(neighbor) + ') (result "DOWN"))'
                    if self.control["debug-fact"] == 1:
                        self.print_log("**** DDR Debug: Assert run_trace Interface Down: " + str(message))
                    self.trace_failed(neighbor)
                    self.cancel_trace(neighbor)
                    return

                else: 

    ###########################################################################
    #
    # Test to see if neighbor is found as the last element in the trace list
    #
    ###########################################################################
                    failed = 'none'
                    nohop = 'none'
                    try:
                        last_result = p2.match(line)
                        group = results.groupdict()
                        failed = group['failed']
                        nohop = group['nohop']
                    except Exception as e:
                        if self.control["debug-fact"] == 1:
                            self.print_log("\n%%%% DDR ERROR: run_trace command Error: no regex match for 'last element' " + str(e) + "\n" + str(line))
                        message = '(' + str(fact_result) + ' (neighbor ' + str(neighbor) + ') (hop "none") (last-ip ' + str(last_address) + ') (result "FALSE"))'
                        if self.control["debug-fact"] == 1:
                            self.print_log("**** DDR Debug: Assert run_trace no hop found")

                        try:
                            self.env.assert_string(message)
                        except Exception as e:
                            if self.control["debug-fact"] == 1:
                                self.print_log("**** DDR Debug: Assert run_trace: last element " + str(message) + "\n" + str(e))
                        return
    #
    # test for timeout as the last hop reported which indicates trace failed to complete
    #
                    if str(failed) == '*':
                        message = '(' + str(fact_result) + ' (neighbor ' + str(neighbor) + ') (hop ' + str(nohop) + ') (last-ip ' + str(last_address) + ') (result "FALSE"))'
                    else:
    #
    # return the last hop ip address
    #
                        template = self.env.find_template(str(fact_result)) # get an empty template fact
                        fact1 = {}
                        fact1["neighbor"] = str(neighbor)
                        fact1["hop"] = str(hop)
                        fact1["last-ip"] = str(last_address)
                        fact1["result"] = "TRUE"
                        try:
                            template.assert_fact(**fact1)
                        except Exception as e:
                            self.print_log("\n%%%% DDR ERROR: run_trace failed fact assert error: " + str(e) + "\n")
                            self.trace_failed(neighbor)
                            self.cancel_trace(neighbor)
                            return
                        return

            except Exception as e:
                self.print_log("\n%%%% DDR ERROR: run_trace command Error: " + str(e) + "\n")
                message = '(' + str(fact_result) + ' (neighbor ' + str(neighbor) + ') (result "FALSE"))'
                if self.control["debug-fact"] == 1:
                    self.print_log("**** DDR Debug: Assert run_trace Error: " + str(e))
                self.trace_failed(neighbor)
                self.cancel_trace(neighbor)
                return
            return
    
    def run_write_to_syslog(self, syslog_message, delay_ms):
        """
            Rule function call: (run_write_to_syslog SYSLOG_MSG_STRING DELAY)
            
            Invoked in the "RHS" of a triggered rule.
            This function sends a Syslog message on IOS-XE platforms
            This function sends the input syslog_message to the host through serial device /dev/ttyS2 as described here: 
            https://developer.cisco.com/docs/iox/#!app-hosting-loggingtracing-services/iox-logging-tracing-facility
                                    
            :param syslog_message: Text string to send as a Syslog message
            :param delay_ms: Delay in ms to wait before returning
        
        Usage::

              (run_write_to_syslog "%DDR_SYSLOG-5-MESSAGE: Configured by Console" 3000)

            :raises none:

        """

        try:
            if int(delay_ms) != 0: time.sleep(delay_ms/1000)
            timestamp =  datetime.now().strftime("%m-%d-%Y_%H:%M:%S.%f")               
            outfile = str(self.control["notify-path"]) # Write to the syslog "device"
            if self.control["debug-fact"] == 1:
                self.print_log("**** DDR Debug: Notification log file name: " + str(outfile))
            tty_fd = os.open(outfile, os.O_WRONLY | os.O_CREAT)
            numbytes = os.write(tty_fd, str.encode(syslog_message + "\n"))
            os.close(tty_fd)
        except Exception as e:
            self.print_log('\n%%%% DDR Error: Writing Syslog notification ' + str(e))

    def run_xr_send_syslog(self, syslog_message):
        """
            Rule function call: (run_xr_send_syslog SYSLOG_MSG_STRING)
            
            Invoked in the "RHS" of a triggered rule.
            This function sends a Syslog message on IOS-XR platforms using the "logmsg"
            CLI command.
                                    
            :param syslog_message: Text string to send as a Syslog message
        
        Usage::

              (run_xr_send_syslog "%DDR_SYSLOG-5-MESSAGE: Configured by Console")

            :raises none:

        """
        #
        # Add the Service Impact Notification filename to the Syslog message
        #
        full_message = str(syslog_message) + " Log Location: " + str(self.control["log-path"]) + str(self.control["use-case"]) + "_TS_" + str(self.control["session-time"])
        try:
            options = '-q -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oPubkeyAuthentication=no'
            ssh_cmd = 'ssh %s@%s %s "%s"' % (self.control["mgmt-device"][2], self.control["mgmt-device"][0], options, str(full_message))
            if self.control["debug-CLI"] == 1:
                self.print_log("**** DDR Debug: run_xr_send_syslog: " + str(ssh_cmd) + "\n")
            child = pexpect.spawn(ssh_cmd, timeout= 20, encoding='utf-8')
            child.delaybeforesend = None
            if self.control["debug-CLI"] == 1:
                child.logfile = sys.stdout
            child.expect(['\r\nPassword: ', '\r\npassword: ', 'Password: ', 'password: '])
            child.sendline(self.control["mgmt-device"][3])
            child.expect(pexpect.EOF)
            response = child.before
            timestamp =  datetime.now().strftime("%m-%d-%Y_%H_%M_%S_%f")
            self.print_log("**** DDR Notice: XR logmsg sent: " + str(syslog_message) + " at: " + str(timestamp))

        except Exception as e:
            child.close()
            self.print_log("\n%%%% DDR ERROR: run_xr_send_syslog or timeout Error: " + str(ssh_cmd) + "\n")
        child.close()

    #################################################################################################
    #
    # get_action_functions - return a list of all action functions defined for DDR
    #                        The functions are registered with the CLIPs runtime so the functions
    #                        can be called from RULEs 
    #
    #################################################################################################
    def get_action_functions(self):
        action_functions = [self.run_action, self.run_apply_config, self.run_assert_message, self.run_assert_message_syslog, self.run_assert_sim_fact, self.run_clear_selected_facts, self.run_cli_command, self.run_cli_parameter, self.run_copy_file, self.run_copy_from_device, self.run_command, self.run_ddr, self.run_decode_btrace_log, self.run_delay, self.run_delete_file, self.run_delete_remote_file, self.run_logging_trigger, self.run_nc_fact, self.run_nc_fact_index, self.run_ping_action, self.run_process_file, self.run_read_control_file, self.run_rule_timer, self.run_set_rule_step, self.run_set_runmode, self.run_show_fact, self.run_show_parameter, self.run_show_parameter_index, self.run_suffix, self.run_trace, self.run_write_to_syslog, self.run_xr_send_syslog]
        return action_functions


