Quickunderstanding Feature Dev

Here we are, after all the claiming that this framework enhances Ignite to a rapid feature development tool! I hope I’m mot wrong… personally, I experience implementing new fully integrated, shortcutted, auto-categorized features/transitions/bso-containers (you name it) with tons of individual events a.k.a. state objects feels exactly like implementing a simple standard python class… the framework base classes deliver the full integration for free. The only differences are:

• You define your state objects (a.k.a. custom events) as simple class attribute and select the desired state object types which add even more functionalities for free.
• There is a simple order for __init__ when to e.g. initialize the base class and when to set the callbacks and so on.
• For full integration simple naming rules for input & output attributes must be observed.

In the following I will quickly discuss demo features but I would also recommend to have a look at already existing bso-containers and trainsitions which are recommonden below the demo code blocks. Based on the demos the basic rules for feature implementation (quickunderstanding_feature_dev.py) are presented followed by some more detailed sections.

Note

The framework offers really quick (& dirty) hacks to get whatever idea up & running very quickly. You can declare this a feature of the framework & you are welcome to mess up what is needed to get fast results in your script. To guarantee a long-term stable architecture that keeps running even when messing up the application scripts, clear design rules for new features are mandatory and are part of the architecture like the implementation itself.

In progress

The “How-to-implement-feautres” guideline is just being written in this very moment meaning everything presented here is half cooked. Everybody is welcome to contribute, question, change & progress everything… if anything is not clear enough, I either haven’t documented it clearly or haven’t thought of or implemented it clearly enough.

Simple feature demos

Below, two very simple examples are given to demonstrate a low-level feature (e.g. transition like a metric, or an output handler) handling a single task and a high level feature (e.g. pipeline or composition like a supervised trainer or a metrics comparision chart generator) handling multiple low-level features/tasks sequentially or in parallel. Underneath each example proper implementations of real features are listed for study and comparison.

# STATE
from ignite_framework.states import state

# BASE CLASS
from ignite_framework.transitions import BaseTransition

# UTILS
from ignite_framework.feature_dev_tools.state_objects import StateDevice, StateVariable
from ignite_framework.feature_dev_tools.utils import get_caller_refs, create_bso_ctr_name_prefix_from_input_ref


# SELECT THE BASE CLASS
# NOTE:
# - The base class will define the state container to which
#   the transition/feature is automatically attached to.
# - Here `BaseTransition` will assign `DoubleTheInput` to `state.transitions`

class DoubleTheInput(BaseTransition):
    # Set up state object (current Ignite: "custom event")
    output = StateVariable()
    device = StateDevice()

    def __init__(self, double_name, input_ref, caller_refs='',
                 device=state.configs.default_configs.device_for_modules_ref):
        # ARGUMENT CHECKING
        # NOTE:
        # - mainly `input_ref(s)` and `caller_ref(s)` are reformatted
        #   (e.g. as list) or set to default values etc.
        # - the `caller_ref(s)` default values is(are) normally the `input_ref(s)`
        caller_refs = get_caller_refs(caller_refs=caller_refs, input_refs=input_ref)

        # AUTOMATIC OR ASSISTED NAMING
        name = '_'.join([create_bso_ctr_name_prefix_from_input_ref(input_ref=input_ref), double_name])

        # BASE CLASS INITIALIZATION
        # NOTE: Initializing the base class start framework support, therefore
        #       to order when it's initialized MATTERS!
        super().__init__(name=name)

        # ASSIGN REFERENCES
        # Assigned input references which will be tracked by the framework
        self._input_ref = input_ref
        # NOTE:
        # - `device` could also be a fixed value, e.g. `gpu` or `cpu'
        # - passing in a reference synchronizes `self.device` with the ref, otherwise fixed value is set
        self.device = device

        # Everything else
        self._double_factor = 2

        # APPEND METHODS TO CALLER REFS CALLBACKS
        # NOTE: The callback appending can be individualized as required
        self._set_callbacks(caller_refs=caller_refs)

    def run(self):
        self.output = self._double_factor * self._input_ref.caller_name

    def _set_callbacks(self, caller_refs):
        for caller_ref in caller_refs:
            # Append the methods as desired to the callbacks
            caller_ref.caller_name = ('short description is helpful', self.run)
            # NOTE:
            # - if bso-container has method `run` then run will be detected
            #   and append when simply `self` is assigned, see below
            # - alternative implementations with automatic description
            #   generation: `caller_ref.caller_name = self`

Note that most code lines are comments, the real transition code is really short!

Good examples of useful feature are e.g. Engine (in comparison to current Ignite Engine), BaseMetric, Metrics which may also look familiar.

# BASE CLASS
from ignite_framework.pipelines import BasePipeline

# FEATURE CLASSES
from ignite_framework.output_handlers import OutputHandler
from ignite_framework.metrics import OutputMetric


class CurrentIgniteMetricWithOutputHandler(BasePipeline):
    """
    Demo pipeline dowing nothing but piping e.g. the engine output through an output handler and a metric.
    """
    def __init__(self, name, input_ref, caller_refs='', metric_device=''):
        # No Auto or assisted naming

        # Argument checking
        caller_refs = get_caller_refs(caller_refs=caller_refs, input_refs=input_ref)

        # Initialize base class
        super().__init__(name=name, device=metric_device)

        # SAVE NAME OF PIPELINE BSO-CONTAINERS TO `self.composed_bso_ctrs` ordered dictionary
        # NOTE: The name of each bso-container must be related to the pipeline `name`
        self.composed_bso_ctrs['output_handler'] = OutputHandler(input_refs=input_ref,
                                                                 output_handler_name=name,
                                                                 caller_refs=caller_refs).name
        self.composed_bso_ctrs['metric'] = OutputMetric(metric_name=name + '_metric',
                                                        input_ref=input_ref,
                                                        caller_refs='')

        # All callbacks are set by `OutputHandler` and `OutputMetric`

Basic feature implementation rules

Try to use this as a checklist when implementing new features. That’s why details of the checklist are discussed separately in successive sections:

• never use Frame* classes: feature development and framework development are strictly separated

• define feature scope and resulting Base*-class:

  • single task feature: choose according base class to assign the feature to the correct state container, e.g. BaseMetric to assign to state.metrics, BaseCharts for state.tensorboard or BaseOutputHandler for state.output_handlers etc.
  • multi task feature: choose BasePipeline or Base*Composition explained here

• state objects defined as simple class attribute:

  • define state objects before __init__ (overview)
  • only attributes references by other features should be defined as state object (otherwise regular instance attribute)

  • naming:

    • output reserved for feature output within the model dataflow path, i.e. value calculated from input_ref
    • currently no further restrictions

  • currently no further restrictions/rulse

• __init__ arguments and command sequence:

  • __init__ arguments:

    • notes:

      • any arguments requiring a state object reference as value must have argument name with suffix _ref
      • default value placeholder is '', not None
      • data references must be assigned during __init__

    • bso container name:

      • name: requires manual bso ctr name setting, no assisted prefix addition, no automatic naming
      • name='': automatic naming, optional manual bso ctr name setting, no assisted prefix addition
      • *_name: requires manual bso ctr name setting, with assisted prefix addition, no automatic naming
      • *_name='': automatic naming, optional manual bso ctr name setting with assisted prefix addition

    • input_ref(s):

      • only input data originating from model output is named input_ref(s) (for dataflow analysis)
      • input_ref(s) must be assigned to self._input_ref(s) (for dataflow graph analysis)

  • __init__ command sequence (when do what in __init__)`:

    1. arguments checking and reformating
    2. automatic and/or assisted name creation *_name –> name (if required)
    3. base class initialization with automatic/assisted/manual name (launches reference tracking and shortcut creation)
    4. reference assignment, e.g. self._input_ref(s) input_ref(s)
    5. everything else

    6. set callbacks self._set_callbacks:

       • default value: caller_ref(s)='' defaults to caller_ref(s)=input_ref(s)

• output state object:

  • only output data originating from model output is set as ouput (for dataflow analysis)
  • successive bso containers/features may reference the output as SuccessiveFeature(.., input_ref=this_feature.output_ref,...)

• everything else unrestricted:

  • any methods may be arbitrarily implemented
  • any methods may be arbitrarily appended to any callbacks (just appending needs to happen in self._set_callbacks())
  • any number of state objects allowed (but please don’t trash state with useless state objects)

Basic feature design rules

The following rules try to guide multiple processes:

• correct application feature implementation
• guideline for standardizing and enhancing the process of application feature implementation
• guideline for enhancing the framework features and architecture and prevent the framework from degradation.

While coding an application or framework feature or refining the framework architecture, keep these aspects in mind:

• a bso container/feature only performs exactly one task:

  • low-level, e.g.: transform output, calculate metric
  • high-level, e.g.: coordinate multiple low-level (or high-level) features, also just one task

• a feature is encapsulated:

  • never holds other features, despite state components (e.g. dataloader in engine)
  • above means: no more passing in a output handler into a metric as in current Ignite
  • data transfer between feature exclusively with state objects references which are set at initialization
  • method calls between features exclusively with callbacks, despite assigned state component methods (e.g. trainer engine gets next batch form trainer dataloader)
  • above means: time sequential alignment of feature processing is managed exclusively with state object callbacks
  • internal methods of a feature can of course call its own internal methods

• maximum user overview:

  • every feature is categorized in a state container, guaranteed by base classes and encapsulation

• high individualization and provision of manual setting options
• automation always builds on low-level features and should never exclude the possibility of manual individualizations
• designed for complete user spectrum: researchers, engineers, feature developers, application developers

• designed for any user level: from experts to beginners

  • experts works with low-level features (actually with any feature)
  • beginner works with high-level features
  • everybody benefits from state overview and feature encapsulation

• PYTHONIC (hope it still is!)

State objects types

T.B.A.

Bso container types and base class selection

T.B.A.

I’m not quite sure how clear the naming of the framework objects is, so just for recapitulation:

• Feature here is often used as synonym for base state object container (bso-container) because it is more graphic. But it also is a little ambiguous w.r.t it’s general meaning and features provided by the framework (e.g. shortcuts). Still Feature Dev is really about developing all kinds of features, not only transitions, metrics and output handlers, but also state plugins, new engines, useful analysis and visualization tools etc., all new features that are based on bso containers.
• Transitions are the most common bso containers, e.g. engines, metrics, output handler, but not e.g. state components, default configs container etc.. This is why we are talking about feature dev and not of transition dev.
• State components…

The namings above also relate to the (super-)types of the feature classes:

3 main state container types all inherited from supertypeclass MetaStateContainer: MetaBaseTransitionsContainer, MetaStateComponentsContainer, MetaStateValueContainer

Transitions

T.B.A.

State Components

T.B.A.

State value containers

T.B.A.

Pipelines & Compositions

T.B.A.

ContainerIDs

T.B.A.