Published

With the latest releases of kedro, it is now possible to run kedro pipelines from within scripts. While I would not start a project with this technique, it will be a good tool to keep in my back pocket when I want to sprinkle in a bit of kedro goodness in existing projects. New to Kedro # What is Kedro If your just learning about kedro check out this post walking through it No More Rabbit Hole of Errors # as of 0.17.2 I’ve tried to do this in kedro and it turned into a rabbit hole of errors. First kedro needed a directory, if you tried to fake one in it would then ask for logging setup. These errors just kept coming to the point it wasnt worth doing and I might as well use a proper template for real projects and stick to simple function calls for things that are not a kedro project. Kedro in a script # To get kedro running, you will need a pipeline, catalog, and runner at a minimum. Those who have used kedro before the pipeline will look very similar to what you are familiar with, but…

Kedro can have a chatty logger. While this is super nice in production so see everything that happened during a pipeline run. This can be troublesome while trying to implement a cli extension with clean output. Silence a Python log # First, how does one silence a python log? Python loggers can be retrieved by the module’s function. Then their log level can be changed. Much of kedro’s chattiness comes from INFO level logs. I don’t want to hear about anything for my current use case unless it’s essential, i.e., a failure. In this case, I set the log levels to ERROR as most errors should stop execution anyways. python logging levels # Get or Create a logger # Getting a python logger is straightforward if we know the name of the logger. The following block will grab the logger object for the logger currently registered under the name passed in. 🔥 If a logger doesn’t exist under the passed in name, it will create one for you. Set Level # Once we get the logger, we need to silence it by sett…

Running multiple processes using the same diskcache object can cause issues with locks. As I was trying to setup a rich Live display for markata I ran into issues where each part could not nun simultaneusly. As I had followed the instructions from discache it was not directly aparant to me, so I had to make a simple example to experiment and play with at a small scale. Minimum reproducible error # Minimum reporducible error is one of my superpowers in development. I do this very often to sus out what is really happening. My day to day work is processing data with python, I keep a number of very small data sets handy to break and fix. This helps separate complexities of the project and the problem. Let’s break it # Markata has a lot going on. It’s a plugins all the way down static site generator built in python. Trying to find the root cause through the layers of plugin and cli modules can be a pain, but in this case building a very simple minimum reporducible error was much easier. 📝 i…

In 2021 I changed the way I navigate between tmux sessions big time. Now I can create, kill, switch with ease, and generally keep work separated into logical groups. Update # Since making this post, I have made ~20 other posts in short form that all have a YouTube video to go along with them you can find them all on my tmux-playlist. Chris Toomey’s Tmux Course # I took Chris’s tmux course in December 2020 and it was fantastic. Even as a seasoned tmux user, I learned quite a bit. Before the course, I was proficient in navigating within each of my tmux sessions but rarely started more than one session. A few months later, I have adopted a lot of what I learned from Chris and made it my own. I am now keeping projects to their own session and can move between them fluidly with just a few keystrokes. For high-traffic projects, I have them bound to a single keystroke for instant switching. This change has been a game-changer from the mess of windows I used to have and the nightmare it was to…

Trim branches no longer on origin # Find branches already merged #

When a python module is called it is assigned the of otherwise if it’s imported it will be assigned the of the module. Concrete example # Let’s create a module to play with a bit. We will call this module. It is a module that we may want to run by it’self or import and use in other modules. I have set this module up to execute one of two if statements based on whether the module it’self is being ran or if the module is being imported. Note it is not common to have a block, this is just for demnonstration purposes. running python nodes.py # Running a python script with the command will execute your script top to bottom. This will print out How to Install miniconda on linux (from the command line only) If you don’t already have python installed try using miniconda or replit.com running./nodes.py # You can also simply execute the script from bash if you first set the module to be executable. is only needed one time, even if you edit the file. pipeline.py # Let’s create a second module and…

Creating a directory that is a union of several directories can be achieved with a few symlinks at the command line. Creating a Virtual File Gallery # Here is how I am creating a virtual directory of all my projects that is a combination of both work and not-work projects. I am creating symlinks for every directory under and. ⚠ Notice that first I am recreating the directory each time. This will ensure that any project that is deleted from their actual directory is removed from the virtual gallery. Updating the gallery # Since links are always kept up to date without any extra work, all the data is still in the same place it started. But as new directories are added to any project directory they will not be automatically added to the virtual gallery. cron bashrc/zshrc If you’re concerned about system resources, you can add it to a cron job to run at a regular schedule that makes sense to you. For me, I just popped those 4 lines right in my. It’s a bit overkill, maybe bloat, but it runs…