Archive

Click primarily takes two forms of inputs Options and arguments. I think of options as keyword argument and arguments as regular positional arguments.

**From the Docs

To get the Python argument name, the chosen name is converted to lower case, up to two dashes are removed as the prefix, and other dashes are converted to underscores.

...

See all of my kedro related posts in [[ tag/kedro ]].

I am tweeting out most of these snippets as I add them, you can find them all here #kedrotips.

Below are some quick snippets/notes for when using kedro to build data pipelines. So far I am just compiling snippets. Eventually I will create several posts on kedro. These are mostly things that I use In my everyday with kedro. Some are a bit more essoteric. Some are helpful when writing production code, some are useful more usefule for exploration.

...

|-|-| |github: |https://github.com/zaxr/bulwark|

I definitely want to try this out with kedro.

Bulwark is a package for convenient property-based testing of pandas dataframes, supported for Python 3.5+.

Pathlib is an amazing cross-platform path tool.

from pathlib import Path

Create path object ¶ #

Current Directory

cwd = Path('.').absolute()

Users Home Directory

...

Good for method chaining, i.e. adding more methods or filters without assigning a new variable.

# is skus.query('AVAILABILITY == " AVAILABLE"') # is not skus.query('AVAILABILITY != " AVAILABLE"')

masking ¶ #

general purpose, this is probably the most common method you see in training/examples

# is skus[skus['AVAILABILITY'] == 'AVAILABLE'] # is not skus[~skus['AVAILABILITY'] == 'AVAILABLE']

isin ¶ #

capable of including multiple strings to include

...

I love digital ocean for it’s simplicity and its commitment to open source.

tqdm is one of my favorite general purpose utility libraries in python. It allows me to see progress of multipart processes as they happen. I really like this for when I am developing something that takes some amount of time and I am unsure of performance. It allows me to be patient when the process is going well and will finish in sufficient time, and allows me to 💥 kill it and find a way to make it perform better if it will not finish in sufficient time.

for more gifs like these follow me on twitter @waylonwalker

Add a simple Progress bar!

...

If you are a regular listener of TalkPython or PythonBytes you have hear Michael Kennedy talk about Named Tuples many times, but what are they and how do they fit into my data science workflow.

As you graduate your scripts into modules and libraries you might start to notice that you need to pass a lot of data around to all of the functions that you have created. For example if you are running some analysis utilizing sales, inventory, and pricing data. You may need to calculate total revenue, inventory on hand. You may need to pass these data sets into various models to drive production or pricing based on predicted volumes.

Here we setup functions that can load data from the sales database. Assume that we also have similar functions to get_inventory and get_pricing.

...

This post is intended as an extension/update from background tasks in python. I started using background the week that Kenneth Reitz released it. It takes away so much boilerplate from running background tasks that I use it in more places than I probably should. After taking a look at that post today, I wanted to put a better data science example in here to help folks get started.

This post is intended as an extension/update from background tasks in python. I started using background the week that Kenneth Reitz released it. It takes away so much boilerplate from running background tasks that I use it in more places than I probably should. After taking a look at that post today, I wanted to put a better data science example in here to help folks get started.

I use it in more places than I probably should

...

Bash is super powerful.

Show Remaining Space on Drives

df -h

show largest files in current directory

...

I have used %autoreload for several years now with great success and 🔥 rapid reloads. It allows me to move super fast when developing libraries and modules. They have made some great updates this year that allows class modules to be automatically be updated.

🔥 Blazing Fast

💥 Keeps me in the comfort of my text editor

...

Throw that mouse Away its time to setup some keyboard shortcuts.

These sortcuts were the baseline for switching from tmux/vim to vscode. Most folks posts I was able to find gave great tips on replacing vim, but very few have focused on the hackability of tmux. tmux allows me to rapidly fire up a workspace, create new windows and splits. Then When I switch tasks I can leave that workspace open and and jump right back in later exactly where I left off. There is nothing quite like it. The shortcuts listed here make the transition a bit better. The worst thing I found when using vscode at first was no way to switch between the terminal and editor without the mouse. This first set of keybindings solve that issue.

The worst thing I found when using vscode at first was no way to switch between the terminal and editor without the mouse.

...

My git workflow based on real life. Its not always clean and simple.

sometimes things get messy

pull 👉 branch 👉 format 👉 work👉 add 👉 commit 👉 pull 👉 rebase 👉 push

...

Testing out forestry.io

I still ♥️ your product dont be

forestry is simple

...