Posts tagged: python

Kedro can have a chatty logger. While this is super nice in production so see everything that happened during a pipeline run. This can be troublesome while trying to implement a cli extension with clean output. Silence a Python log # First, how does one silence a python log? Python loggers can be retrieved by the module’s function. Then their log level can be changed. Much of kedro’s chattiness comes from INFO level logs. I don’t want to hear about anything for my current use case unless it’s essential, i.e., a failure. In this case, I set the log levels to ERROR as most errors should stop execution anyways. python logging levels # Get or Create a logger # Getting a python logger is straightforward if we know the name of the logger. The following block will grab the logger object for the logger currently registered under the name passed in. 🔥 If a logger doesn’t exist under the passed in name, it will create one for you. Set Level # Once we get the logger, we need to silence it by sett…

Setting up python with the native nvim>0.5 lsp was mr lsp-config # https://github.com/neovim/nvim-lspconfig pyls#190 # https://github.com/palantir/python-language-server/issues/190 mypy # Getting mypy working with lsp was tricky for me. I had some issues trying to run mypy in ci and pyright in my editor and I really wanted them to match.

Running multiple processes using the same diskcache object can cause issues with locks. As I was trying to setup a rich Live display for markata I ran into issues where each part could not nun simultaneusly. As I had followed the instructions from discache it was not directly aparant to me, so I had to make a simple example to experiment and play with at a small scale. Minimum reproducible error # Minimum reporducible error is one of my superpowers in development. I do this very often to sus out what is really happening. My day to day work is processing data with python, I keep a number of very small data sets handy to break and fix. This helps separate complexities of the project and the problem. Let’s break it # Markata has a lot going on. It’s a plugins all the way down static site generator built in python. Trying to find the root cause through the layers of plugin and cli modules can be a pain, but in this case building a very simple minimum reporducible error was much easier. 📝 i…

Add current file and commit with diff in a split #:on[ly] # C-W o:on[ly] will make the current buffer the only one on the screen. This is super helpful as many of fugitive commands will open in a split by default. C-I C-O # cycle through the jumplist This one has nothing to do with fugitive, but is a native vim feature that makes fugitive glorious. Before I realized how to utilize and, I would get completely lost when using fugitive. Digging deep into the log, opening a file from a specific commit, then no way to get back where I was in the log. C-i jump:jump[s] # show the jumplist The jumplist is sorted Oldest to newest:Telescope jumplist # When navigating the jumplist with, it will add a new entry to the jumplist and let you get back to where you were with a.:Telescope jumplist adds to the jumplist C-W J / C-W L #:G log # Ggrep # Unlike you don’t need to specify a file glob. Staging Hunks # In the file you want to stage hunks of run. Then use vim’s diff commands (,,,) to move the hun…

When a python module is called it is assigned the of otherwise if it’s imported it will be assigned the of the module. Concrete example # Let’s create a module to play with a bit. We will call this module. It is a module that we may want to run by it’self or import and use in other modules. I have set this module up to execute one of two if statements based on whether the module it’self is being ran or if the module is being imported. Note it is not common to have a block, this is just for demnonstration purposes. running python nodes.py # Running a python script with the command will execute your script top to bottom. This will print out How to Install miniconda on linux (from the command line only) If you don’t already have python installed try using miniconda or replit.com running./nodes.py # You can also simply execute the script from bash if you first set the module to be executable. is only needed one time, even if you edit the file. pipeline.py # Let’s create a second module and…

DRAFT -

I really like using conda () as my python virtual environment manager of choice. It’s simple and it includes its own python interpreter using the version that I specify at creation. Mamba # from their readme Mamba is a reimplementation of the conda package manager in C++. parallel downloading of repository data and package files using multi-threading libsolv for much faster dependency solving, a state of the art library used in the RPM package manager of Red Hat, Fedora and OpenSUSE core parts of mamba are implemented in C++ for maximum efficiency At the same time, mamba utilize the same command line parser, package installation and deinstallation code and transaction verification routines as conda to stay as compatible as possible. Installing Micromamba # Similar to miniconda micromamba can be installed with a few lines of bash Creating Environments with Micromamba # Creating new environments with micromamba is pretty similar to using conda. -c is required # I was unable to figure out…

I am trying to see what an embeded replit

Zev Averbach, Frustrated spreadsheet jockey to software developer at 36 Q: Tell me about your journey as a spreadsheet jockey into Data Engineering? A: First of all, it’s hilarious that I accidentally found your questions for this interview by Googling myself. 😊 I’ve always been a frustrated software user, and that frustration led me to be a “power user” (keyboard shortcuts etc) of my most used applications, as well as a “visual coder” using desktop automation like Alfred and Keyboard Maestro (Mac). Now that I’ve met data analysts and finance people that use Excel all day, I don’t think I’d claim to have been a true “spreadsheet jockey” in comparison to them. However, hitting up against the limitations of spreadsheets for running my transcription business – specifically for bookkeeping – created a new frustration for me: As the business grew I was spending more and more time copying entries from Google Sheets to the Freshbooks web app for invoicing purposes. I tried to automate that pr…

Testing print/log statements in pytest can be a bit tricky, capsys makes it super easy, but I often struggle to find it. capsys # capsys is a builtin pytest fixture that can be passed into any test to capture stdin/stdout. For a more comprehensive description check out the docs on capsys using capsys # Simply create a test function that accepts capsys as an argument and pytest will give you a capsys opject.

Draft Post I’ve really been digging @willmcgugan ’s rich library for creating TUI like interfaces in python. I’ve only recently started to take full advantage of it. Dev Server # I am working on a project in which I want to have a dev server running continuously in the background. I really like dev servers theat automatically chooose an unused port and list out the running pid so that I can kill it if I need to. automatic port number auto-restart display ( port, pid, uptime) finding the port # I am very novice at best when it comes to sockets, the following function came from searching StackOverflow for how to tell if a port is in use. I recursively check if a port is being used, if it is I increment by one until I find an unused port to return. The Dev Server # I am going super basic here and just running. It works for what I need it for, it hosts my files for the browser to display, and if I try a route without an index. html it gives me a decent file list. Optionally if you wanted a…

Final Result #

I have automated my process to create virtual environments in my python projects, here is how I did it. I’ve really been digging my new tmux session management setup. Now I have leveled it up by adding direnv to my workflow. It will execute a shell script whenever I cd into a directory. One thing I wanted to add to this was, automatic activation of python environments whenever I cd into a directory, or create a new environment if one does not exist. https://waylonwalker.com/tmux-nav-2021/ Direnv # You can learn more about direnv on their website. There are several libraries that seem to do a similar concept, run a bash script when I cd into a directory direnv is the one that I chose to use, but you can likely do the same concept in other ones. Installation # direnv is in many package repos, or can be installed from binary builds with a one liner. See their installation instructions for more information..envrc # direnv works by executing the contents of known safe files. Here are some o…

I have started doing more regular PR’s on my teams Kedro pipelines. I generally take a two phase approach to the review in order to give the reviewee both quick and detailed feedback. What is Kedro initial scan (Phase1) # passing ci Variable Names Antipatterns No commented out code Docsttrings generally make sense Phase1 is typically a quick scan over the PR right within the PR window in my browser. Passing CI # flake8 black isort interrogate pytest build The very first thing that needs to happen is automated CI. We use things like flake8, black, isort, interrogate to ensure that everyone follows generic style guides like pep8. The project does a build within the PR, but no deploy. Variable Names # I strugle really hard to not impose my own opinion into the PR at this point, and sometimes really want to change a lot of variable names. Typically I make sure they don’t grow longer than necessary, too short, misspelled, or inconsistent. I make sure that I can follow the flow without getti…

With the realease of came a new module in the project template. Here are some notes that I learned while playing with this new module. migrating to # create a file create a function in that mirrors the register_pipelines method from your module do not bring the decorator remove register_pipelines method on your class You should now have something that looks like this in your. pipeline_registry only works in Conflict Resolution # What happens If I register pipelines in both places I was not able to find any official documentation on how conflict resolution worked so I stepped into a project and added to both my and file. I noticed that it would pick up pipelines from both modules, but pipelines from always take precedence. The entire duplicate pipeline will be over written by the one from. kedro automatically merges pipelines from both hooks.py takes precedence Ready to update # In my experience there were no issues upgrading from to. I would reccomend only having one so decide to migra…

pytest has open sourced their amazing plugin framework, it allows library authors to give their users a way to modify the libaries behavior without needing to submit a change that may not make sense to the entire library. Previous Experience # My experience so far as a plugin user, and plugin author has been great. Building and using plugins are incredibly intuitive. I wanted to dive a bit deeper and see how they are implemented inside of a library and its a bit of a mind bend the first time you try to do it. Plugins vs. Hooks # A hook is a single function that has a specific place that it is ran by the PluginManager. A Plugin is a collection of one or more hooks. Layers # library author plugin author end user Using a plugin # For a plugin to be registered is must be registered by the PluginManager which is implemented by the library author. It is the job of the library author to determine what plugins are actively registered or disabled. There are two common ways that I have seen that…

There are so many tutorials out there for creating OG images with or, but I have yet to see many using python. PIL # Template # Text # Overlays #

There are various ways to configure python tools, config files, code, or environment variables. Let’s look at a few projects that allow users to configure them through the use of config files and how they do it. Motivation # This will not include how they are implemented, I’ve looked at a few and its not simple. This will focus on where config is placed and the order in which duplicates are resolved. The motivation of this article is to serve as a bit of a reference guide for those who may want to create their own package that needs configuration. Flake8 # Global # User settings can exist in the users file to configure how flake8 runs on their machine. Per-Project # Only One project config file will be considered, but allows for several options. These files all use the format and must have a section header to be consideered. Selection of the config file can also be overridden by the cli option. An extra config file may be selected as. It will be read in last and take highest precedence…

How small can a minimum kedro pipeline ready to package be? I made one within 4 files that you can pip install. It’s only a total of 35 lines of python, 8 in and 27 in. 📝 Note this is only a composable pipeline, not a full project, it does not contain a catalog or runner. Minimal Kedro Pipeline # I have everything for this post hosted in this gihub repo, you can fork it, clone it, or just follow along. Installation # Caveats # This repo represents the minimal amount of structure to build a kedro pipeline that can be shared across projects. Its installable, and drops right into your or modules. It is not a runnable pipeline. At this point I think the config loader requires to have a logging config file. This is a sharable pipeline that can be used across many different projects. Usage # Implemantation # This builds on another post that I made about creating the minimal python package. I am not sure if it should be called a package, it’s a module, but what do you call it after you build…

This is a post that may be a work in progress for awhile, Its a collections of thoughts on managing my blog, but could be translated into anythiung that is just a collection of markdown. Listing things # posts tags draft posts data # frontmatter filepath content template html render content # Markdown.Markdown support extentsions frontmatter cleaning. # provide ways to hook in or clean up the frontmatter Markata.Markata methods # load render save Markata.Post methods # load render save Markata plugins # before_load before_post_load after_load after_post_load before_save before_post_save after_save after_post_save Markata plugins # cleanse_frontmatter html_feed json_feed rss_feed save_posts CLI #