🐍 Pluggable Architecture with Python
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

pytest has open sourced their amazing plugin framework , it allows library authors to give their users a way to modify the libaries behavior without needing...

Date: January 23, 2021

pytest has open sourced their amazing plugin framework pluggy, it allows library authors to give their users a way to modify the libaries behavior without needing to submit a change that may not make sense to the entire library.

Previous Experience
───────────────────

My experience so far as a plugin user, and plugin author has been great. Building and using plugins are incredibly intuitive. I wanted to dive a bit deeper and see how they are implemented inside of a library and its a bit of a mind bend the first time you try to do it.

Plugins vs. Hooks
─────────────────

A hook is a single function that has a specific place that it is ran by the PluginManager.

A Plugin is a collection of one or more hooks.

Layers
──────

- library author
- plugin author
- end user

Using a plugin
──────────────

For a plugin to be registered is must be registered by the PluginManager which is implemented by the library author. It is the job of the library author to determine what plugins are actively registered or disabled. There are two common ways that I have seen that plugins are registered, through entrypoints or configuration.

Using a plugin - entrypoints
────────────────────────────

Plugins that are implemented with entrypoints are the simplest for the user. They are simply activated by pip install plugin or deactivated by pip uninstall plugin. The library author will show an entrypoint in their docs which tells plugin authors how to setup entrypoints so that they will be loaded autommatically.

Using a plugin - config
───────────────────────

Another way to configure plugins is through configuration. This may come in the form of a list in a python module or listed in a text file in the config. This route requires the user to add the plugin to a list or import it into a python module.

Examples
────────

I really stuggled to find a good example of pluggy to get started. I found the best way for me to understand was to create one myself. the pluggy repo has one simple example <https://github.com/pytest-dev/pluggy/blob/master/docs/examples/toy-example.py>, but it is unclear who owns each piece from the example. The whole point of pluggy is to pass ownership of implementation from the library author to the plugin author.

Floris Bruynooghe
─────────────────

https://www.youtube.com/watch?v=zZsNPDfOoHU

Floris Bruynooghe has a great talk from EuroPython 2015 <https://www.youtube.com/watch?v=zZsNPDfOoHU> where he shows how to build a project thats plugins all the way down. His slides <https://devork.be/talks/pluggy> are also available.

Kedro
─────

Kedro is a data pipelining framekwork that includes a hooks based architecture that allows users to modify the behavior of the framework at different points through the lifecycle. There is a hooks <https://github.com/kedro-org/kedro/tree/dc1ee8e06b255d4d5a4348ad8a2e78048c547279/kedro/framework/hooks> module that implements everything, and a test_plugin <https://github.com/kedro-org/kedro/blob/dc1ee8e06b255d4d5a4348ad8a2e78048c547279/features/steps/test_plugin/plugin.py> that is used for testing, but also serves as a good example.

palantir/python-language-server
───────────────────────────────

Another example is the palantir python language server. Check out their hookspec <https://github.com/palantir/python-language-server/blob/91a13687dbd5247374253b245124befb8d9c60c9/pyls/hookspecs.py> module.

Tutorial
────────

Plugin Components
─────────────────

- project_name

  - implemented by the library author
  - gives a namespace for pluggy to store hooks
- hookspec

  - created and used by libary author
- hookimpl

  - created by libary author
  - used by plugin author
- PluginManager

  - implementation of plugins in the library

hookspec
────────

_empty hooks created by the library author

[code]
  # hookspec.py
  import pluggy

  hookspec = pluggy.HookspecMarker("printer")

  class PrinterHooks:
      @hookspec
      def pre_print(msg):
          "pre print hook"
          pass

      @hookspec
      def post_print(msg):
          "pre print hook"
          pass

hookimpl
────────

used by the plugin author

Implementations of plugins much match the name of the spec exactly. They can include some or all of the arguments listed in the spec, but no others. They can be implemented as a module with functions that match the name of the spec or as a class with methods that match the name of the spec.

### Class Style Plugin

[code]
  # plug.py
  # would be imported from the library authors hookspec
  from hookspec import hookimpl


  class Pre:
      @hookimpl
      def pre_print(msg):
          msg = msg.upper()
          return "BEFORE"


  class Post:
      @hookimpl
      def post_print(msg):
          print(f"\033[A\033[2Knot today")

### Module Style Plugin

[code]
  # plug/Pre.py
  from hookspec import hookimpl


  @hookimpl
  def pre_print(msg):
      msg = msg.upper()


  # plug/Post.py
  class Post:
      @hookimpl
      def post_print(msg):
          print(f"\033[A\033[2Knot today")

note These plugins only implement one hook. Each plugin may implement one or more hooks, a plugin is not required to only implement one hook.

Plugin Manager
──────────────

implementing the hooks into the library

### Simple Example

[code]
  import pluggy
  import importlib

  from hookspec import PrinterHooks
  from plug import Pre

  pm = pluggy.PluginManager("printer")
  pm.add_hookspecs(PrinterHooks)
  pm.register(Pre)

  def printer(msg):
      pm.hook.pre_print(msg=msg)
      print(msg)
      pm.hook.post_print(msg=msg)

Running the library
───────────────────

Now if we run the printer function as a user we will see this output.

[pycon]
  >>> printer('hello world')

  <!--markata-attribution-->
  HELLO WORLD

Adding Post
───────────

Now if we register the Post plugin we will see the following output.

[code]
  from plug import Pre, Post

  pm.register(Pre)
  pm.register(Post)

[pycon]
  >>> printer('hello world')

  <!--markata-attribution-->
  not today

The Post plugin wipes away the last line from the console and prints out "not today"

Plugin Manager - with dynamic imports
─────────────────────────────────────

In a real library we might want to allow the user to configure their plugins through a config file. If we do this we will need to reach for importlib to handle the imports based on a string.

[code]
  import pluggy
  import importlib

  # from hookspec import hookspec
  from hookspec import PrinterHooks

  # from hookspec import hookimpl

  plugins = ["plug.Pre", "plug.Post"]
  pm = pluggy.PluginManager("printer")
  pm.add_hookspecs(PrinterHooks)

  for plug in plugins:
      if isinstance(plug, str):
          # plug is a str representing a module to import
          try:
              # module style plugins
              plugin = importlib.import_module(plug)
          except ModuleNotFoundError as e:
              # class style plugins
              if "." in plug:
                  mod = importlib.import_module(".".join(plug.split(".")[:-1]))
                  plugin = getattr(mod, plug.split(".")[-1])
              else:
                  raise e
      else:
          # plug is a module that is already imported
          plugin = plug

      pm.register(plugin)


  def printer(msg):
      pm.hook.pre_print(msg=msg)
      print(msg)
      pm.hook.post_print(msg=msg)

EntryPoint plugins
──────────────────