Posts tagged: python

What is Kedro This is my original what-is-kedro article. There is a brand new one Kedro is an open source data pipeline framework. It provides guardrails to set your project up right from the start without needing to know deeply how to setup your own python library for data pipelining. It includes really great ways to manipulate and. This article will cover the 10K view of kedro, future articles will dive deper into each one. kedro is an open-source data pipeline framework. It provides guardrails to set your project up right from the start without needing to know deeply how to set up your own python library for data pipelining. It includes great ways to manipulate and. This article will cover the 10K view of kedro, future articles will dive deeper into each one. Libraries # Currently, kedro is broken down into 3 different libraries. 💎 kedro 📉 kedro-viz 🏗 kedro-docker kedro # kedro is the core of the ecosystem. It provides the docs, getting started, templates, and the core library inclu…

🏷️ Long variable names are a good thing. Self documenting code is more important than poorly documented code. Simply adding a few characters to your variable names can go a long ways. Containers are plural # Aliases are welcome # Scope is important

cli tools are super handy and easy to add to your python libraries to supercharge them. Even if your library is not a cli tool there are a number of things that a cli can do to your library. Example Ideas # Things a cli can do to enhance your library. 🆚 print version 🕶 print readme 📝 print changelog 📃 print config ✏ change config 👩‍🎓 run a tutorial 🏗 scaffold a project with cookiecutter 🖱 Click # Click is the most popular python cli tool framework for python. There are others, some old, some new comers that make take the crown. For now Click is the gold standard if you want to make a powerful cli quickly. If you are dependency conscious and dont need a lot of tooling, use argparse. Project Structure # ❯ cli.py # ✨ init.py # For our simple_click library can be left empty. It is here purely to signify that simple_click is a library. It is likely that you will import other modules here that need to reside at the top level of your library api, your cli does not need to be at the top of of…

Make a connection # Make a session # Make a Base Class # Make your First Model # Make your own Base Class to inherit From # Use the Custom Base Class #

Packages # Click # Inputs # Click primarily takes two forms of inputs Options and arguments. I think of options as keyword argument and arguments as regular positional arguments. Option # typically aliased with a shorthand (’-v’, ‘–verbose’) **From the Docs To get the Python argument name, the chosen name is converted to lower case, up to two dashes are removed as the prefix, and other dashes are converted to underscores. Argument # positional required no help text supplied by click Yaspin # Click Help Colors # # Colorama # Colorama Example Click DidYouMean #

See all of my kedro related posts in [[ tag/kedro]]. #kedrotips # I am tweeting out most of these snippets as I add them, you can find them all here #kedrotips. 🗣 Heads up # Below are some quick snippets/notes for when using kedro to build data pipelines. So far I am just compiling snippets. Eventually I will create several posts on kedro. These are mostly things that I use In my everyday with kedro. Some are a bit more essoteric. Some are helpful when writing production code, some are useful more usefule for exploration. 📚 Catalog # CSVLocalDataSet # python yaml CSVHTTPDataSet # HDFLocalDataSet # HDFS3LocalDataSet # JSONLocalDataSet # ParquetLocalDataSet # PickleS3DataSet SQLTableDataSet SQLQueryDataSet TextLocalDataSet ExcelLocalDataSet ⏳ Loading Data # Simple Loading # list all datasets # Saving Data # 🔍 Finding data # simple keyword search see on #kedrotips multi keyword serch see on #kedrotips 🐒 monkey patch it _see on #kedrotips 🤙 YOLO # You Only Load Once simple more refined 🍷 r…

jmespath Tabnine Bulwark # |-|-| |github: | https://github.com/zaxr/bulwark | I definitely want to try this out with kedro. Bulwark is a package for convenient property-based testing of pandas dataframes, supported for Python 3.5+. Example #

Using pdb #

Pathlib is an amazing cross-platform path tool. Import # Create path object # Current Directory Users Home Directory module directory Others Let’s create a path relative to our current module. Check if files exist # Make Directories # rename files # List files # Glob Files # recursively Write #

query # Good for method chaining, i.e. adding more methods or filters without assigning a new variable. masking # general purpose, this is probably the most common method you see in training/examples isin # capable of including multiple strings to include contains # Good For partial matches MASKS # anything that we put inside of square brackets can be set as a variable then passed in. Operators # & - and ~ - not | - or AVAILABLE and NAME # AVAILABLE or NAME # AVAILABLE and not NAME #

I have been using pyspark since March 2019, here are my thoughts.

Tools Sphinx # Portray # I just started using portray and it is amazingly simple to use! Methodology

tqdm is one of my favorite general purpose utility libraries in python. It allows me to see progress of multipart processes as they happen. I really like this for when I am developing something that takes some amount of time and I am unsure of performance. It allows me to be patient when the process is going well and will finish in sufficient time, and allows me to 💥 kill it and find a way to make it perform better if it will not finish in sufficient time. @waylonwalker Add a simple Progress bar! convenience TQDM also has a convenience function called trange that wraps the range function with a tqdm progress bar automatically. notebook support There is also notebook support. If you are bouncing between ipython and jupyter I recomend importing from the auto module. Autoreload in Ipython If you are using notebooks you should enable ipython autoreload 👆

If you are a regular listener of TalkPython or PythonBytes you have hear Michael Kennedy talk about Named Tuples many times, but what are they and how do they fit into my data science workflow. Example # As you graduate your scripts into modules and libraries you might start to notice that you need to pass a lot of data around to all of the functions that you have created. For example if you are running some analysis utilizing,, and data. You may need to calculate total revenue, inventory on hand. You may need to pass these data sets into various models to drive production or pricing based on predicted volumes. Load data # Here we setup functions that can load data from the sales database. Assume that we also have similar functions to and. Create Metrics # Here we create our first function to calculate some metrics. There are likely many of these functions that repeat a similar pattern. They use similar data and have their own custom logic for calculations and joins. Furthermore these…

This post is intended as an extension/update from background tasks in python. I started using the week that Kenneth Reitz released it. It takes away so much boilerplate from running background tasks that I use it in more places than I probably should. After taking a look at that post today, I wanted to put a better data science example in here to help folks get started. This post is intended as an extension/update from background tasks in python. I started using the week that Kenneth Reitz released it. It takes away so much boilerplate from running background tasks that I use it in more places than I probably should. After taking a look at that post today, I wanted to put a better data science example in here to help folks get started. I use it in more places than I probably should Before we get into it, I want to make a shout out to Kenneth Reitz for making this so easy. Kenneth is a python God for all that he has given to the community in so many ways, especially with his ideas in bu…

Bash is super powerful. File System Full # Show Remaining Space on Drives show largest files in current directory Move files then symlink them Fuzzy One Liners # edit in vim cat a file bash execute git add git reset Kill a process Finding things # Files # fd-find is amazing for finding files, it even respects your file 😲. Install with. ++Vanilla Bonus Content # ** show matching text ** ++Vanilla Bonus ** show file names only ** ++Vanilla Bonus Recursively Replace text # ++Vanilla Bonus Extending or There are so many options inside of,, and that you could many an enormous amount of these if you really wanted to, but I like to keep it simple. These cover 90% of my usage. If I wanted to change something in the second half I would just paste in this command and edit it. More often though I want to limit the input, say only replace word1 to word2 inside of markdown files. Limited Scope ++Vanilla Bonus Large Refactor At The Command Line I use these replace commands heavily when doing large r…

I have used for several years now with great success and 🔥 rapid reloads. It allows me to move super fast when developing libraries and modules. They have made some great updates this year that allows class modules to be automatically be updated. What I like about autoreload # 🔥 Blazing Fast 💥 Keeps me in the comfort of my text editor 👏 Allows me to use Jupyter when I need 👟 Extremely Reliable One of the biggest benefits that I find is that it shortens the distance between my module/library code and test code inside of a terminal/notebook. Now I primarily use jupyter notebooks for the presentation aspect. I develop code from the comfort of my editor with all of the tools I have setup, and run the functions in a notebook to get the output. From there I might do some aggregations or plots, but the 🥩 meat of development is done outside of jupyter. Now I primarily use jupyter notebooks for the presentation aspect. Enabling Autoreload # 📐 config This is a short script that I use to setup ip…

Dictionaries # Unpacking # **kwargs func(**input) locals().update(d) #

Generating Readme Tables From Pandas # I commonly have a need to paste the first few lines of a dataset into a markdown file. I use two handy packages to do this, and. Lets say I have a Pandas DataFrame in memory as already. All I would need to do to convert the first 5 rows to markdown and copy it to the clipboard is the following. This is a super handy snippet that I use a lot. Folks really appreciate it when they can see a sample of the data without opening the entire file.