Archive
All published posts
cmd.exe tips
I spend a lot of my time at the terminal for my daily work, mostly in Linux or wsl. One big reason for using wsl over cmd.exe is the ease of walking through history that fzf provides. This week we had a windows bug in a cli and I was stuck in vanilla cmd.exe 😭
First off if you are stuck using cmd.exe, do yourself a favor and get cmder. It makes life just a bit easier. It is super confugurable and comes with several power ups that make it a bit more enjoyable than cmd.exe.
F7 - Scroll through history
...
I’m really excited about python-c2f, an amazing project by grantjenks. It’s worth exploring!
Cython for All with GitHub Actions
Check out wesbos and their project beginner-javascript.
Slam Dunk JavaScript
If you’re into interesting projects, don’t miss out on awesome-react-components, created by brillout.
Curated List of React Components & Libraries.
What is something you should have learned or understood earlier?
Mine is the python debugger. I was a long holdout thinking that print statements were sufficient. That was untill I started having errors crop up in functions that took minutes to run. The thing that I most notably wish I would have known about is post_mortem.
[ins] In [4]: def repeater(msg, repeats=1): ...: "repeats messages {repeats} number of times" ...: print(f'{msg}\n' * repeats) [ins] In [5]: repeater('hi', 3) hi hi hi [ins] In [6]: repeater('hi', 'a') --------------------------------------------------------------------------- TypeError Traceback (most recent call last) <ipython-input-6-0ec595774c81> in <module> ----> 1 repeater('hi', 'a') <ipython-input-4-530890de75cd> in repeater(msg, repeats) 1 def repeater(msg, repeats=1): 2 "repeats messages {repeats} number of times" ----> 3 print(f'{msg}\n' * repeats) 4
Debug with...
...
Supercharge Zsh Startup
I have been using oh-my-zsh successfully for about 2 years now. But lately my startup time has been really bothersome. It has grown to the point where it was taking about 5.5s to startup a shell! This is ok if I am going to spend some time in here for awhile and do some work that benefits from all of the autocompletions, plugins, and shortcuts that oh-my-zsh brings. But to only jump in to run a handful of commands is infuriating.
I believe the real issue is io speed on wsl. I have some remote servers with similar configs that are 10x faster or more, loading in 100s of milliseconds rather than seconds. Sourcing all of the individual plugin files are just too much for it.
Quick side note: your zsh config is controled by your ~/.zshrc file. This file can source other files, load plugins, or run literally anything.
...
Keep Location List Closed
Vim’s (neovim in my case) location list can provide some very useful information while developing. Mine gives me information about linting and type checking errors with fairly little config. Generally, it sits nicely at the bottom of the screen and barely affects me. Other times, especially while zoomed way in during a presentation, it just gets in the way.
Location List eating up the screen while I am zoomed in and trying to live code
Through some google search I found the culprit was syntastic. It has an auto_loc_list feature. We can turn it off by setting syntastic_auto_loc_list=0.
...
SqlAlchemy Models
I came across jumpcutter from carykh, and it’s packed with great features and ideas.
Automatically edits vidx. Explanation here: https://www.youtube.com/watch?v=DQ8orIurGxw
Check out great-expectations and their project great_expectations.
Always know what to expect from your data.
I like asmeurer’s project removestar.
Tool to automatically replace ‘import *’ in Python files with explicit imports
Check out lolcommits and their project lolcommits.
📷 git-based selfies for software developers
I’m really excited about mask, an amazing project by jacobdeichert. It’s worth exploring!
🎭 A CLI task runner defined by a simple markdown file
Check out pyjanitor-devs and their project pandas_flavor.
The easy way to write your own flavor of Pandas
Building Cli apps in Python
Click primarily takes two forms of inputs Options and arguments. I think of options as keyword argument and arguments as regular positional arguments.
**From the Docs
To get the Python argument name, the chosen name is converted to lower case, up to two dashes are removed as the prefix, and other dashes are converted to underscores.
...
I recently discovered git-history by pomber, and it’s truly impressive.
Quickly browse the history of a file from any git repository
Kedro
See all of my kedro related posts in [[ tag/kedro ]].
I am tweeting out most of these snippets as I add them, you can find them all here #kedrotips.
Below are some quick snippets/notes for when using kedro to build data pipelines. So far I am just compiling snippets. Eventually I will create several posts on kedro. These are mostly things that I use In my everyday with kedro. Some are a bit more essoteric. Some are helpful when writing production code, some are useful more usefule for exploration.
...
Check out vscode-git-semantic-commit by nitayneeman. It’s a well-crafted project with great potential.
💬 A Visual Studio Code extension which enables to commit simply by the semantic message conventions
awesome-streamlit by MarcSkovMadsen is a game-changer in its space. Excited to see how it evolves.
The purpose of this project is to share knowledge on how awesome Streamlit is and can be
I’m impressed by js13k-2019 from bencoder.
xx142-b2.exe. An entry for js13kgames 2019
Just starred death-to-ie11 by gabLaroche. It’s an exciting project with a lot to offer.
Countdown for IE11 end of support
📝 Packages to Investigate Notes
|-|-| |github: |https://github.com/zaxr/bulwark|
I definitely want to try this out with kedro.
Bulwark is a package for convenient property-based testing of pandas dataframes, supported for Python 3.5+.
I came across awesome-data-engineering from igorbarinov, and it’s packed with great features and ideas.
A curated list of data engineering tools for software developers
I’m really excited about vscode-python, an amazing project by microsoft. It’s worth exploring!
Python extension for Visual Studio Code
Debugging Python
Just Use Pathlib
Pathlib is an amazing cross-platform path tool.
from pathlib import Path
Create path object #
Current Directory
cwd = Path('.').absolute()
Users Home Directory
...
Custom Python Exceptions
Filtering Pandas
Good for method chaining, i.e. adding more methods or filters without assigning a new variable.
# is skus.query('AVAILABILITY == " AVAILABLE"') # is not skus.query('AVAILABILITY != " AVAILABLE"')
masking #
general purpose, this is probably the most common method you see in training/examples
# is skus[skus['AVAILABILITY'] == 'AVAILABLE'] # is not skus[~skus['AVAILABILITY'] == 'AVAILABLE']
isin #
capable of including multiple strings to include
...
Digital Ocean
I love digital ocean for it’s simplicity and its commitment to open source.
If you’re into interesting projects, don’t miss out on Recreation-of-Nature, created by Kashu7100.
ALife simulation with Python: patterns, behavior, and cognition.
Quick Progress Bars in python using TQDM
tqdm is one of my favorite general purpose utility libraries in python. It allows me to see progress of multipart processes as they happen. I really like this for when I am developing something that takes some amount of time and I am unsure of performance. It allows me to be patient when the process is going well and will finish in sufficient time, and allows me to 💥 kill it and find a way to make it perform better if it will not finish in sufficient time.
for more gifs like these follow me on twitter @waylonwalker
Add a simple Progress bar!
...
I’m impressed by bake from kennethreitz.
Bake — the strangely familiar workflow utility.
Clean up Your Data Science with Named Tuples
If you are a regular listener of TalkPython or PythonBytes you have hear Michael Kennedy talk about Named Tuples many times, but what are they and how do they fit into my data science workflow.
As you graduate your scripts into modules and libraries you might start to notice that you need to pass a lot of data around to all of the functions that you have created. For example if you are running some analysis utilizing sales, inventory, and pricing data. You may need to calculate total revenue, inventory on hand. You may need to pass these data sets into various models to drive production or pricing based on predicted volumes.
Here we setup functions that can load data from the sales database. Assume that we also have similar...
...
Background Tasks in Python for Data Science
This post is intended as an extension/update from background tasks in python. I started using background the week that Kenneth Reitz released it. It takes away so much boilerplate from running background tasks that I use it in more places than I probably should. After taking a look at that post today, I wanted to put a better data science example in here to help folks get started.
This post is intended as an extension/update from background tasks in python. I started using...
...
alttch has done a fantastic job with rapidtables. Highly recommend taking a look.
Super fast list of dicts to pre-formatted tables conversion library for Python 2/3
📝 Bash Notes
Bash is super powerful.
Show Remaining Space on Drives
df -h
show largest files in current directory
...
Autoreload in Ipython
I have used %autoreload for several years now with great success and 🔥 rapid reloads. It allows me to move super fast when developing libraries and modules. They have made some great updates this year that allows class modules to be automatically be updated.
🔥 Blazing Fast
💥 Keeps me in the comfort of my text editor
...
If you’re into interesting projects, don’t miss out on promote-open-source-project, created by zenika-open-source.
📄 How to promote my open source project?
Check out watchtower by kislyuk. It’s a well-crafted project with great potential.
Python CloudWatch Logging: Log Analytics and Application Intelligence