Archive

With the latest releases of kedro 0.17.x, it is now possible to run kedro pipelines from within scripts. While I would not start a project with this technique, it will be a good tool to keep in my back pocket when I want to sprinkle in a bit of kedro goodness in existing projects.

What is Kedro

If your just learning about kedro check out this post walking through it

...

Kedro can have a chatty logger. While this is super nice in production so see everything that happened during a pipeline run. This can be troublesome while trying to implement a cli extension with clean output.

First, how does one silence a python log? Python loggers can be retrieved by the logging module’s getLogger function. Then their log level can be changed. Much of kedro’s chattiness comes from INFO level logs. I don’t want to hear about anything for my current use case unless it’s essential, i.e., a failure. In this case, I set the log levels to ERROR as most errors should stop execution anyways.

Getting a python logger is straightforward if we know the name of the logger. The following block will grab the logger object for the logger currently registered under the name passed in.

...

Running multiple processes using the same diskcache object can cause issues with locks. As I was trying to setup a rich Live display for markata I ran into issues where each part could not nun simultaneusly. As I had followed the instructions from discache it was not directly aparant to me, so I had to make a simple example to experiment and play with at a small scale.

Minimum reporducible error is one of my superpowers in development. I do this very often to sus out what is really happening. My day to day work is processing data with python, I keep a number of very small data sets handy to break and fix. This helps separate complexities of the project and the problem.

Markata has a lot going on. It’s a plugins all the way down static site generator built in python. Trying to find the root cause through the layers of plugin and cli modules can be a pain, but in this case building a very simple minimum reporducible error was much easier.

...

In 2021 I changed the way I navigate between tmux sessions big time. Now I can create, kill, switch with ease, and generally keep work separated into logical groups.

Since making this post, I have made ~20 other posts in short form that all have a YouTube video to go along with them you can find them all on my tmux-playlist.

I took Chris’s tmux course in December 2020 and it was fantastic. Even as a seasoned tmux user, I learned quite a bit. Before the course, I was proficient in...

...

When a python module is called it is assigned the __name__ of __main__ otherwise if it’s imported it will be assigned the __name__ of the module.

Let’s create a module to play with __name__ a bit. We will call this module nodes.py. It is a module that we may want to run by it’self or import and use in other modules.

#!python # nodes.py if __name__ == "nodes": import sys import __main__ print(f"you have imported me {__name__} from {sys.modules['__main__'].__file__}") if __name__ == "__main__": print("you are running me as main")

I have set this module up to execute one of two if statements based on whether the module it’self is being ran or if the module is being imported.

...

Creating a directory that is a union of several directories can be achieved with a few symlinks at the command line.

Here is how I am creating a virtual directory of all my projects that is a combination of both work and not-work projects. I am creating symlinks for every directory under ~/work and ~/git.

rm -rf ~/projects mkdir ~/projects ln -sf ~/work/* ~/projects ln -sf ~/git/* ~/projects

⚠ Notice that first I am recreating the directory each time. This will ensure that any project that is deleted from their actual directory is removed from the virtual gallery.

...

I really like using conda (miniconda) as my python virtual environment manager of choice. It’s simple and it includes its own python interpreter using the version that I specify at creation.

from their readme

Mamba is a reimplementation of the conda package manager in C++.

...

I’ve long used neovim from within windows wsl, and for far too long, I went without a proper way to get text out of it and into windows.

wsl can access clip.exe. You can do some cool things with it, such as cat a file into the clipboard, sending output from a command to the clipboard, or set an autocmd group in vim to send yank to the windows clipboard.

Let’s say you want to send a teammate the tail of a log file over chat. You can tail the file into clip.exe.

...

I am trying to see what an embeded replit

Testing print/log statements in pytest can be a bit tricky, capsys makes it super easy, but I often struggle to find it.

capsys is a builtin pytest fixture that can be passed into any test to capture stdin/stdout. For a more comprehensive description check out the docs on capsys

Simply create a test function that accepts capsys as an argument and pytest will give you a capsys opject.

Draft Post

I’ve really been digging @willmcgugan’s rich library for creating TUI like interfaces in python. I’ve only recently started to take full advantage of it.

I am working on a project in which I want to have a dev server running continuously in the background. I really like dev servers theat automatically chooose an unused port and list out the running pid so that I can kill it if I need to.

...