root/tags/v1.0/change.py

"""
A changeset is a sequence of changes that can be applied to a managed element.
A changeset will move the element from one state, A, to a destination state, B.
A changeset may contain other changesets. These changesets will be implemented
in a specific order.
The successful implementation of a changeset must be testable.
If the test fails, implementation of the changeset has failed, and a backout can
be executed.
A changeset contains the necessary backouts.

If a sub-changeset fails implementation, the containing changeset also fails.

Imagine the following changeset:

useradd justin

This adds a user to the system. If something goes wrong, how do we back out? 
Default backout behaviour is to alert a human that something went wrong.

Backout may depend on the exact failure. The 'did it work' test should permit
selecting from a list of possible backout procedures, depending on the failure
that was detected.

This example would be a "Add User" changeset, and could be different on different
destination systems.

Note that a "Remove User" changeset would be different from the backout procedure
for a failed "Add User" changeset.

For a changeset that runs a sequence of commands (a CommandChangeset, perhaps?),
the default behaviour should be to detect if an individual command in the sequence
failed, and report the error to the backout method detection code. This will
provide the most generic way of testing for failure without having to run a mini-audit
after each command is executed. However, if such paranoid auditing is required, you
could implement the changeset as a series of changesets, rather than a single,
multi-command changeset. This would provide much finer grained error detection and
possible correction capabilities.

A changeset is a set of changes to be applied to a set of devices.
A change can be applied to multiple devices, eg: loading the same configuration file onto all 70 hosts.
Does the 'load configuration file to host list' change happen all at once, or as part of a sequence
of changes? How do the changes inter-relate? Contrast "Change all 70 hosts, then do the next thing", to
"Change the first host, then do this other thing, then change the next host".

A Change is the minimum transaction size. This is the smallest element that can be applied,
or backed out. So, to implement the above scenarios, you could:

- Define a Change that should be applied to a device list. The provisioning method may be different
  (ssh vs rsh) but only to a certain degree. A ZAPI commandset is fundamentally different to an ssh/rsh
  commandset, for example.
- Define a Change that should be applied to a device list of length 1.
- Define the order that changes should occur in.

So.. first we define a list of devices and their attributes (ssh key locations, etc). This would be
a single, global area for all devices/entities under change management.

Then, we define a changeset, which is a list of changes. Each change defines the device(s) it applies to,
and how to apply the change to it.
"""
import logging
import debug

log = logging.getLogger('modipy')

from zope.interface import Interface, implements

from twisted.internet import defer

CHANGE_STATE = {
    'success': 0,
    'pending': 1,
    'partial_failure': 2,
    'total_failure': 3,
    'backout_ok': 4,
    'backout_failed': 5,
    'retry': 6,
    }

class ChangeConditionFailure(Exception):
    """
    Raised if an expectset condition fails eval()
    """

class IChange(Interface):
    """
    How to implement a specific change.
    """

    def __init__(self, name, devices=[], serial_mode=True, backout_all=False, on_fail_continue=False, **kwargs):
        """
        @param devices: a list of devices that this change should be applied to.
        @param serial_mode: Boolean, defines whether the change should be applied
        to each device one after the other, or if all devices can have the change
        applied to them in parallel.
        @param backout_all: Boolean. If set to True, if the application of a
        L{Change} to a device fails, the L{Change} will be backed out for all other
        devices that have so far succeeded. This makes a L{Change} atomic.
        @param on_fail_continue: Boolean. If set to True, if application of a
        L{Change} to a device fails, the change will be backed out from that device,
        but the system will continue attempting to apply the change to the rest
        of the devices that have not yet been processed. Overrides backout_all.

        When you create a L{Change} object, you can supply a list of parameters
        that can be used within the L{Change} object to parameterise the
        change itself.

        This is handy for substituting the username, for example, for
        a standard user adding mechanism.

        If a L{Change} should be applied to some devices in parallel, and other
        devices in series, this should be broken up into 2 separate L{Change}s.
        """
        self.devices = devices
        self.serial_mode = serial_mode
        self.backout_all = backout_all
        self.namespace = kwargs

    def pre_apply_check(self, provisioner, namespace):
        """
        Perform any pre-implementation checks you may need to perform
        before attempting to implement the change.
        """
    
    def apply(self, provisioner, namespace):
        """
        Implement the change via the provisioner.

        @param provisioner: A L{Provisioner} instance that this change
        should be applied via.
        """

    def test_apply_success(self, provisioner, result, namespace):
        """
        Test that the change was successfully implemented.
        @param result: Holds a result structure that was
        returned from the apply().

        @returns: True on success, False otherwise
        """

    def backout(self, provisioner, result, namespace):
        """
        Back out this change via the provisioner.

        @param result: The result of a change apply() that failed.
        """

    def test_backout_success(self, provisioner, namespace):
        """
        Test that the backout worked.

        @returns: True on success, False otherwise
        """

class ChangeFailed(Exception):
    """
    Execution of a Change failed for some reason.
    """

class CommandChange:
    """
    A CommandChange implements a change by using a supplied
    CommandProvisioner to execute the change by running a
    sequence of commands on the Provisioner target.

    There are several phases to change processing:
    * Pre-change checking. This phase runs checks to see if
      the change should even start.
    * Change implementation. This runs the actual change.
    * Post-change verification. This makes sure the change
      did what it was supposed to do.
    * Backout. This backs out the change if it fails to get
      implemented correctly, or if post-change verification fails.
    
    """
    implements( IChange, )

    def __init__(self, name, devices=[], serial_mode=True, backout_all=False,
                 on_fail_continue=False,
                 on_fail_retry=False,
                 max_retries=3,
                 pre_impl=None,
                 impl=None,
                 post_impl=None,
                 pre_backout=None,
                 backoutset=None,
                 post_backout=None,
                 pre_requisites=[],
                 **kwargs):

        self.name = name

        # must use a copy or weird things happen
        self.devices = devices[:]
        self.serial_mode = serial_mode
        self.backout_all = backout_all
        self.on_fail_continue = on_fail_continue
        self.on_fail_retry = on_fail_retry
        self.max_retries = max_retries
        self.retries = 0
        self.pre_impl = pre_impl
        self.impl = impl
        self.post_impl = post_impl
        self.pre_backout = pre_backout
        self.backoutset = backoutset
        self.post_backout = post_backout
        self.pre_requisites = pre_requisites[:]

        self.namespace = kwargs

        self.state = CHANGE_STATE['pending']
        pass

    def __repr__(self):
        return '<%s: %s>' % ( self.__class__, self.name )

    def copy(self):
        """
        Create a copy of myself.
        """
        newchange = CommandChange('Copy of %s' % self.name,
                                  self.devices,
                                  self.serial_mode,
                                  self.backout_all,
                                  self.on_fail_continue,
                                  self.on_fail_retry,
                                  self.max_retries,
                                  self.pre_impl,
                                  self.impl,
                                  self.post_impl,
                                  self.pre_backout,
                                  self.backoutset,
                                  self.post_backout,
                                  self.pre_requisites,
                                  )

        # set the namespace as a copy of mine
        if self.namespace is not None:
            newchange.namespace = self.namespace.copy()
            pass

        return newchange
        
    def set_state(self, state_string):
        self.state = CHANGE_STATE[state_string]

    def run_expectset(self, provisioner, expectset, namespace):
        """
        Execute an expectset and verify it against the conditions, if any.
        """
        log.debug("running expectset: %s", expectset)
        if expectset is not None:
            commands, conditions = expectset

            # If this change has an iterator, configure the iteration values
            # into the namespace and set up a callback chain to run the
            # commands in order.
            if getattr(self, 'iterator', None) is not None:
                log.debug("I am iterating some commands!")

                d = defer.succeed(None)

                for ns in self.iterator:
                    log.debug("Adding namespace entry of: %s", ns)

                    # Make sure we only update a copy, so that this
                    # update isn't permanent
                    newnamespace = ns.copy()
                    newnamespace.update(namespace)
                    log.debug("Namespace is now: %s", newnamespace)

                    d.addCallback(provisioner.run_commands, commands, newnamespace)
                    d.addCallback(self.check_conditions, conditions, newnamespace)
                    pass

                return d
                pass

            else:
                d = provisioner.run_commands(None, commands, namespace)
                d.addCallback(self.check_conditions, conditions, namespace)
                return d
        else:
            return defer.succeed(None)

    def check_conditions(self, result, conditions, namespace={}):
        """
        Check the result of running expectset commands against a set
        of conditions, if any.
        """
        log.debug("condition checking result: %s against conditions: %s", result, conditions)

        # FIXME: include all namespaces when building this namespace
        namespace['exitcode'] = result[0]
        namespace['cmdoutput'] = result[1]

        # Default to failure if exitCode is != 0
        if len(conditions) == 0:
            conditions.append('exitcode == 0')

        for cond in conditions:
            # perform a string substitution on the condition first
            cond = cond % namespace

            # create a real dict out of what may be a Namespace object
            ns = {}
            ns.update(namespace)
            ok = eval( cond, ns, {} )
            if not ok:
                log.error("%s: condition failed: %s", self.name, cond)
                raise ChangeConditionFailure("change failed condition check: %s" % cond)
            else:
                log.debug("condition check passed")

    def run_commands_failure(self, failure):
        """
        Called if the command fails utterly to run, such as with connection problems.
        """
        log.error("Command failure: %s: %s")
        return failure
        
    def pre_apply_check(self, provisioner, namespace):
        log.debug("running pre_apply_check")
        return self.run_expectset(provisioner, self.pre_impl, namespace)

    def apply(self, provisioner, namespace):
        """
        Use provisioner to implement me.
        """
        return self.run_expectset(provisioner, self.impl, namespace)
    
    def test_apply_success(self, provisioner, results, namespace):
        log.debug("testing success of change. results: %s", results)
        return self.run_expectset(provisioner, self.post_impl, namespace)

    def backout(self, provisioner, namespace):
        """
        Back out a change from a device.
        """
        log.debug("Running backout")
        return self.run_expectset(provisioner, self.backoutset, namespace)

    def test_backout_success(self, results, provisioner, namespace):
        log.debug("testing success of backout. results: %s", results)
        return self.run_expectset(provisioner, self.post_backout, namespace)

    def parse_node_preimpl(self, node):
        """
        When passed a 'preimpl' node, parse it and add it to my commands.
        """
        expectset, conditions = self.build_expectset(node)
        log.debug("set pre_impl node to: %s", (expectset, conditions))
        self.pre_impl = (expectset, conditions)

    def parse_node_impl(self, node):
        """
        When passed a 'impl' node, parse it and add it to my commands.
        """
        expectset, conditions = self.build_expectset(node)
        log.debug("set post_impl node to: %s", (expectset, conditions))
        self.impl = (expectset, conditions)

    def parse_node_postimpl(self, node):
        """
        When passed a 'postimpl' node, parse it and add it to my commands.
        """
        expectset, conditions = self.build_expectset(node)
        self.post_impl = (expectset, conditions)

    def parse_node_prebackout(self, node):
        """
        When passed a 'prebackout' node, parse it and add it to my commands.
        """
        expectset, conditions = self.build_expectset(node)
        self.pre_backout = (expectset, conditions)

    def parse_node_backout(self, node):
        """
        When passed a 'backout' node, parse it and add it to my commands.
        """
        expectset, conditions = self.build_expectset(node)
        self.backoutset = (expectset, conditions)

    def parse_node_postbackout(self, node):
        """
        When passed a 'postbackout' node, parse it and add it to my commands.
        """
        expectset, conditions = self.build_expectset(node)
        self.post_backout = (expectset, conditions)

    def build_expectset(self, node):
        """
        Build an expectset from a node that contains a set of expectset commands.
        """
        known_children = [ 'command', 'condition' ]

        expectset = []
        conditions = []
        
        for child in node.xpath('*'):
            if child.tag not in known_children:
                raise ValueError("'%s' is an unknown child nodename of '%s'" % (child.tag, node.tag))
            else:
                if child.tag == 'command':
                    expect_tuple = self.parse_command(child)
                    log.debug("adding command to expectset: %s", expect_tuple)
                    expectset.append(expect_tuple)
                elif child.tag == 'condition':
                    conditions.append( self.parse_condition(child) )
                pass
            pass

        log.debug("built expectset: %s", (expectset, conditions))
        return expectset, conditions

    def parse_command(self, element):
        """
        Parse a command element to extract the expect/send pairs.
        """
        if element.tag != 'command':
            raise ValueError("element is not a command node")

        # find the expect string, if it exists
        expectNodes = element.findall('expect')
        if len(expectNodes) == 1:
            expect_str = expectNodes[0].text

        elif len(expectNodes) > 1:
            log.error(">0 expectNodes found: %s", expectNodes)
            raise ValueError("More than one 'expect' tag found!")

        else:
            expect_str = None

        sendNodes = element.findall('send')
        if len(sendNodes) == 1:
            send_str = sendNodes[0].text

        elif len(sendNodes) > 1:
            log.error(">0 sendNodes found: %s", sendNodes)
            raise ValueError("More than one 'send' tag found!")

        else:
            send_str = None

        log.debug("returning: %s, %s", expect_str, send_str)
        return (expect_str, send_str)

    def parse_condition(self, node):
        """
        Parse a condition node, and add it 
        """
        return node.text

    def fetch_namespace(self, provisioner):
        """
        Build a unified namespace merging my namespace with the
        device, provisioner and global namespaces.
        """
        log.info("Fetching namespace...")

        namespace = self.namespace

    def can_retry(self):
        """
        Check to see if this change is allowed to retry after failing.
        """
        if self.on_fail_retry:
            log.debug("Onfail-retry is enabled")
            if self.retries < self.max_retries:
                log.debug("Retries: %d is less than retry max: %s. Will retry change.", self.retries, self.max_retries)
                self.retries += 1
                return True
            else:
                log.debug("Too many retries for change. Will not retry again.")
            pass
        return False
        
if __name__ == '__main__':
    log.debug('Testing base provisioner')

    # Define a changeset
    changeset = []
    commands = [ "echo 'hello there!'", ]
    change = CommandChange(commandlist=commands)

    changeset.append(change)

    # Add a provisioner
    prov = CommandProvisioner()

    for change in changeset:
        prov.perform_change(change)