Posts tagged: kedro

I recently started streaming on twitch.tv/waylonwalker and it’s been a blast so far. python kedro Data Science Data Engineering webdev digital gardening Kedro Spaceflights # It all started with kedro/issues/606, Yetu called out for users of kedro to record themselves doing a walk through of their tutorials. I wanted to do this, but was really stuck at the fact that recording or editing somewhat polished vide is quite time consuming for me. Inspiration # My introduction to twitch came from twitch.tv/theprimeagen. I watched him on YouTube, and then decided to drop into a stream. It was so fun to watch him live that I started following others in the science and tech category. twitch.tv/teej_dv Brilliant neovim core dev, I learn a bunch about nvim every time I watch. twitch.tv/cmgriffing Super Chill and engaging chat. twitch.tv/cassidoo Fantastic discussion/chat. twitch.tv/anthonywritescode Building the python ecosystem. twitch.tv/kierisi Exploring data in r-studio, taking xgboost to Slice…

I'm no longer streaming As much as I would really love to make streaming work, its really hard for my family situation to make large blocks of time work for me. https://stackoverflow.com/questions/16720541/python-string-replace-regular-expression I am starting to stream 3 days per week, before I start work in the morning. These streams will likely be me just talking through things I am already doing. Making DAGs do 🔮Magical Things | Open Source 🐍Python | kedro plugins | # Science & Technology | Every Monday • 7:00 AM - 9:00 AM CDT On Monday’s I am going to be working on open source packages/plugins for kedro. kedro-diff test kedro-diff on piplines with history setup deploy pipeline deply to pypi 🌱 Digital Gardening | Blogging with 🐍Python | Building 🔮Markata a static site generator in python for waylonwalker.com # Science & Technology | Every Wednesday • 7:00 AM - 9:00 AM CDT On Wednesday morning I will be working on my personal website and the static site generator that I built for it…

This was my seconf time ever streaming on twitch.tv/waylonwalker, and I completely botched my mic 2x. https://youtu.be/_7MwgKu-844 Links # Spaceflights Tutorial my spaceflights repo Notes to get started #

This was my first time ever streaming on twitch.tv/waylonwalker. I am excited to get going. I have been streaming early in the morning while I am still waking up, so still a bit groggy as I go. https://youtu.be/Y07UBr9Ccjs Kedro Spaceflights # It all started with kedro/issues/606, Yetu called out for users of kedro to record themselves doing a walk through of their tutorials. I wanted to do this, but was really stuck at the fact that recording or editing somewhat polished vide is quite time consuming for me. Notes #

The Kedro node is an essential part of the pipeline. It defines what catalog entries get passed in, what function gets ran, and the catalog entry to save the results under. does this link work? # https://waylonwalker.com/what-is-kedro/ 👆 Unsure what kedro is? Check out this post. The node function # The node function is the most common and reccomended way to define kedro nodes. It is a function that constructs and returns objects for you. Creating your first kedro node # function # The passed into node can be any callable that accepts the inputs yout have specified, and returns the correct output that you specify as your output. any callable a function you write a function from a library class constructor lambda function partial function literally any callable https://waylonwalker.com/kedro-inputs/ For more information on how kedro passes inputs into your functions check out this post Using a lambda as a function # I keep my nodes short and sweet. They do one thing and do it well. I tu…

I had an idea come to me via twitter. Short one word name packages are becoming hard to find on pypi. Short one word readable package names that are not a play on words are easy to remember, easy to spell correctly, and quick to type out. Simple index # I started with the simple index. Pypi provides a single page listing to every single package hosted on pypi via the simple-index

With the latest releases of kedro, it is now possible to run kedro pipelines from within scripts. While I would not start a project with this technique, it will be a good tool to keep in my back pocket when I want to sprinkle in a bit of kedro goodness in existing projects. New to Kedro # What is Kedro If your just learning about kedro check out this post walking through it No More Rabbit Hole of Errors # as of 0.17.2 I’ve tried to do this in kedro and it turned into a rabbit hole of errors. First kedro needed a directory, if you tried to fake one in it would then ask for logging setup. These errors just kept coming to the point it wasnt worth doing and I might as well use a proper template for real projects and stick to simple function calls for things that are not a kedro project. Kedro in a script # To get kedro running, you will need a pipeline, catalog, and runner at a minimum. Those who have used kedro before the pipeline will look very similar to what you are familiar with, but…

Kedro can have a chatty logger. While this is super nice in production so see everything that happened during a pipeline run. This can be troublesome while trying to implement a cli extension with clean output. Silence a Python log # First, how does one silence a python log? Python loggers can be retrieved by the module’s function. Then their log level can be changed. Much of kedro’s chattiness comes from INFO level logs. I don’t want to hear about anything for my current use case unless it’s essential, i.e., a failure. In this case, I set the log levels to ERROR as most errors should stop execution anyways. python logging levels # Get or Create a logger # Getting a python logger is straightforward if we know the name of the logger. The following block will grab the logger object for the logger currently registered under the name passed in. 🔥 If a logger doesn’t exist under the passed in name, it will create one for you. Set Level # Once we get the logger, we need to silence it by sett…

Add current file and commit with diff in a split #:on[ly] # C-W o:on[ly] will make the current buffer the only one on the screen. This is super helpful as many of fugitive commands will open in a split by default. C-I C-O # cycle through the jumplist This one has nothing to do with fugitive, but is a native vim feature that makes fugitive glorious. Before I realized how to utilize and, I would get completely lost when using fugitive. Digging deep into the log, opening a file from a specific commit, then no way to get back where I was in the log. C-i jump:jump[s] # show the jumplist The jumplist is sorted Oldest to newest:Telescope jumplist # When navigating the jumplist with, it will add a new entry to the jumplist and let you get back to where you were with a.:Telescope jumplist adds to the jumplist C-W J / C-W L #:G log # Ggrep # Unlike you don’t need to specify a file glob. Staging Hunks # In the file you want to stage hunks of run. Then use vim’s diff commands (,,,) to move the hun…

DRAFT -

I am trying to see what an embeded replit

With the realease of came a new module in the project template. Here are some notes that I learned while playing with this new module. migrating to # create a file create a function in that mirrors the register_pipelines method from your module do not bring the decorator remove register_pipelines method on your class You should now have something that looks like this in your. pipeline_registry only works in Conflict Resolution # What happens If I register pipelines in both places I was not able to find any official documentation on how conflict resolution worked so I stepped into a project and added to both my and file. I noticed that it would pick up pipelines from both modules, but pipelines from always take precedence. The entire duplicate pipeline will be over written by the one from. kedro automatically merges pipelines from both hooks.py takes precedence Ready to update # In my experience there were no issues upgrading from to. I would reccomend only having one so decide to migra…

How small can a minimum kedro pipeline ready to package be? I made one within 4 files that you can pip install. It’s only a total of 35 lines of python, 8 in and 27 in. 📝 Note this is only a composable pipeline, not a full project, it does not contain a catalog or runner. Minimal Kedro Pipeline # I have everything for this post hosted in this gihub repo, you can fork it, clone it, or just follow along. Installation # Caveats # This repo represents the minimal amount of structure to build a kedro pipeline that can be shared across projects. Its installable, and drops right into your or modules. It is not a runnable pipeline. At this point I think the config loader requires to have a logging config file. This is a sharable pipeline that can be used across many different projects. Usage # Implemantation # This builds on another post that I made about creating the minimal python package. I am not sure if it should be called a package, it’s a module, but what do you call it after you build…

Docs # https://kedro.readthedocs.io/en/stable/04_kedro_project_setup/01_dependencies.html?highlight=install pip-tools # pip-compile # requirements # requirements.in requirements.txt

In python data science/engineering most of our data is in the form of some sort of table, typically a DataFrame from a library like pandas, spark, or dask. DataFrames are the heart of most pipelines # These containers for data contain many convenient methods to manipulate table like data structures. Sometimes we leverage other data types, namely vanilla types like lists and dicts, or even numpy data types. What is Kedro unfamiliar with kedro, check out this post Sometimes datasets are not tables # There are times when our data doesn’t fit nicely into a DataFrame. Lucky for us Kedro has pickle support out of the box. Pickle is a way to store any python object to disk. Beware that pickle files coming from an unknown source can run malicous code and are considered unsafe. For the most part though when you read and write your own pickle files they are a good tool to consider. See more about pickle from python.org. Cataloging Pickle # I may have a dictionary that describes some cars. In the…

Lint/Format/Doc black flake8 interrogate mypy Pipeline Assertions pipeline constructs pipeline as expected nodes pipeline has minimum nodes test minimum tags test alternate tags Catalog Assertions test catalog follows naming structure Node Tests test function does the correct operations on test data Great Expectations

There are many reasons that you should be using kedro. If you are on a team of Data Scientists/Data Engineers processing DataFrames from many data sources should be considering a pipeline framework. Kedro is a great option that provides many benefits for teams to collaborate, develop, and deploy data pipelines What is Kedro Starter Template # Kedro makes it super easy to get started with their cli that utilizes cookiecutter under the hood. Create New Kedro Project read more about how to start your first kedro project here Collaboration # Kedro provides many tools that help teams collaborate on a single codebase. While writing monolithic scripts it can be easy to pin yourself in a corner where it is difficult to have multiple people making changes to the notebook/script at the same time. Kedro helps guide your team to break your project down into small pieces that different members of the team can work on in parallel. sharable catalog # Kedro makes it easy to collaborate with members wh…

Reasons to Kedro # collaboration Sharable catalog small nodes over monolithic notebooks catalog easily load anything without needing to run No need to write read/write code pipeline No need to keep execution order in your head easily run a slice of a pipeline plugins pip install make your own hooks flexible expandable cli Reasons Not to Kedro # Already utilizing another DAG framework Data is not in a widely supported format Micro short-lived project Large Project / Deadline Use a lower profile project to learn first Team not willing to change Need minimal dependencies God Project - kedro owns everything??

Kedro 0.16.6 is out! Let’s take a look through the release notes Deployment Docs # This is really exciting to see more deployment options coming from the kedro team. It really shows the power of the framework. The power of some of these orchestrations options is incredible. Argo Prefect Kubeflow Batch SageMaker Most of them hinge on a sweet combination of the kedro cli, docker image, and the pipeline knowing your nodes dependencies. Argo, Prefect, and Kubeflow have an interesting technique where they translate the pipeline and its dependencies from kedro to their language. Batch uses the aws cli to submit jobs, one node per job, and listen for them to complete. It will submit all nodes with completed dependencies at once, meaning that we can get some massive parallelization. I did a quick and dirty test of one of these by simulating the technique in a bash script and saw a 40 hr pipeline finish in about 1 hour. I am excited to get this working in my production workflow. Spaceflight sta…

I started making stories as kind of a brain dump a few times per day and posting them to [LinkedIn]( https://www.linkedin.com/in/waylonwalker/(https://www.linkedin.com/in/waylonwalker/). Here are the last 11 days of stories. I store all the stories on my website with the hopes of doing something with them on my own platform eventually. For now it makes it easy to make these posts. Stories 10-10-2020 - 10-21-2020 #