Published

Packages # Click # Inputs # Click primarily takes two forms of inputs Options and arguments. I think of options as keyword argument and arguments as regular positional arguments. Option # typically aliased with a shorthand (’-v’, ‘–verbose’) **From the Docs To get the Python argument name, the chosen name is converted to lower case, up to two dashes are removed as the prefix, and other dashes are converted to underscores. Argument # positional required no help text supplied by click Yaspin # Click Help Colors # # Colorama # Colorama Example Click DidYouMean #

See all of my kedro related posts in [[ tag/kedro]]. #kedrotips # I am tweeting out most of these snippets as I add them, you can find them all here #kedrotips. 🗣 Heads up # Below are some quick snippets/notes for when using kedro to build data pipelines. So far I am just compiling snippets. Eventually I will create several posts on kedro. These are mostly things that I use In my everyday with kedro. Some are a bit more essoteric. Some are helpful when writing production code, some are useful more usefule for exploration. 📚 Catalog # CSVLocalDataSet # python yaml CSVHTTPDataSet # HDFLocalDataSet # HDFS3LocalDataSet # JSONLocalDataSet # ParquetLocalDataSet # PickleS3DataSet SQLTableDataSet SQLQueryDataSet TextLocalDataSet ExcelLocalDataSet ⏳ Loading Data # Simple Loading # list all datasets # Saving Data # 🔍 Finding data # simple keyword search see on #kedrotips multi keyword serch see on #kedrotips 🐒 monkey patch it _see on #kedrotips 🤙 YOLO # You Only Load Once simple more refined 🍷 r…

jmespath Tabnine Bulwark # |-|-| |github: | https://github.com/zaxr/bulwark | I definitely want to try this out with kedro. Bulwark is a package for convenient property-based testing of pandas dataframes, supported for Python 3.5+. Example #

Using pdb #

Pathlib is an amazing cross-platform path tool. Import # Create path object # Current Directory Users Home Directory module directory Others Let’s create a path relative to our current module. Check if files exist # Make Directories # rename files # List files # Glob Files # recursively Write #

Custom Exceptions #

query # Good for method chaining, i.e. adding more methods or filters without assigning a new variable. masking # general purpose, this is probably the most common method you see in training/examples isin # capable of including multiple strings to include contains # Good For partial matches MASKS # anything that we put inside of square brackets can be set as a variable then passed in. Operators # & - and ~ - not | - or AVAILABLE and NAME # AVAILABLE or NAME # AVAILABLE and not NAME #

I love digital ocean for it’s simplicity and its commitment to open source.

tqdm is one of my favorite general purpose utility libraries in python. It allows me to see progress of multipart processes as they happen. I really like this for when I am developing something that takes some amount of time and I am unsure of performance. It allows me to be patient when the process is going well and will finish in sufficient time, and allows me to 💥 kill it and find a way to make it perform better if it will not finish in sufficient time. @waylonwalker Add a simple Progress bar! convenience TQDM also has a convenience function called trange that wraps the range function with a tqdm progress bar automatically. notebook support There is also notebook support. If you are bouncing between ipython and jupyter I recomend importing from the auto module. Autoreload in Ipython If you are using notebooks you should enable ipython autoreload 👆