The work on cookiecutter-data-science [1] by drivendataorg [2].
A logical, reasonably standardized, but flexible project structure for doing and sharing data science work.
References:
[1]: https://github.com/drivendataorg/cookiecutter-data-science
[2]: https://github.com/drivendataorg
Publishing rhythm
Site Down During Build
Recently I noticed a new netlify site of mine was down while I was checking to
see if new content was live. Later found out this was consistent after each
and every push the site would go gown as soon as I hit push, and would not come
back until the build finished.
Is this normal? # [1]
Do other Netlify sites go down during build???
Short Answer NO. All of my google fu lead me to believe I was alone and none of my other sites do this.
Digging into my build # [2]
My deploy script ends with the following. After resetting keys and watching it build half a dozen
times I determined that everything was working as normal here.
- name: Deploy to Netlify
uses: nwtgck/[email protected]
with:
publish-dir: "./markout"
production-branch: markout
production-deploy: true
deploy-message: "Deploy markout from GitHub Actions"
env:
NETLIFY_AUTH_TOKEN: ${{ secrets.NETLIFY_AUTH_TOKEN }}
NETLIFY_SITE_ID: ${{ secrets.NETLIFY_SITE_ID }}
Opening the Nelify Console # [3]
After poking at t...
Kedro pipeline_registry.py
With the realease of kedro==0.17.2 came a new module in the project template
pipeline_registry.py. Here are some notes that I learned while playing with
this new module.
migrating to pipeline_registry.py # [1]
- create a src/<package-name>/pipeline_registry.py file create a
- register_pipelines function in pipeline_registry.py that mirrors the
- register_pipelines method from your hooks.py module do not bring the
- hook_impl decorator remove register_pipelines method on your ProjectHooks
- class
You should now have something that looks like this in your
src/<package-name>/pipeline_registry.py.
"""Project pipelines."""
from typing import Dict
from kedro.pipeline import Pipeline
def register_pipelines() -> Dict[str, Pipeline]:
"""Register the project's pipelines.
Returns: A mapping from a pipeline name to a ``Pipeline`` object.
"""
return {"__default__": Pipeline([])}
pipeline_registry only works in kedro>=0.17.2
Conflict Resolution # [2]
What happens If I register p...
Check out smitajit [1] and their project bufutils.vim [2].
bufutils.vim provide utilities to open, close, refresh, move, resize, zoom buffers faster
References:
[1]: https://github.com/smitajit
[2]: https://github.com/smitajit/bufutils.vim
I like Rigellute’s [1] project spotify-tui [2].
Spotify for the terminal written in Rust 🚀
References:
[1]: https://github.com/Rigellute
[2]: https://github.com/Rigellute/spotify-tui
I like swyxio’s [1] project technical-community-builders [2].
companies hiring technical community builders
References:
[1]: https://github.com/swyxio
[2]: https://github.com/swyxio/technical-community-builders
Just starred Minyus [1] by Minyus [2]. It’s an exciting project with a lot to offer.
No description available.
References:
[1]: https://github.com/Minyus/Minyus
[2]: https://github.com/Minyus
I like uranusjr’s [1] project simpleindex [2].
No description available.
References:
[1]: https://github.com/uranusjr
[2]: https://github.com/uranusjr/simpleindex
I came across lorenabalan [1] from lorenabalan [2], and it’s packed with great features and ideas.
Personal repo
References:
[1]: https://github.com/lorenabalan/lorenabalan
[2]: https://github.com/lorenabalan
I came across vim.wasm [1] from rhysd [2], and it’s packed with great features and ideas.
Vim editor ported to WebAssembly
References:
[1]: https://github.com/rhysd/vim.wasm
[2]: https://github.com/rhysd
I recently discovered de1-python [1] by dataengineerone [2], and it’s truly impressive.
Curated collection of DE1’s favorite kedro pieces.
References:
[1]: https://github.com/dataengineerone/de1-python
[2]: https://github.com/dataengineerone
I recently discovered markserv [1] by markserv [2], and it’s truly impressive.
🏁 serve markdown as html [3] (GitHub style), index directories, live-reload as you edit
References:
[1]: https://github.com/markserv/markserv
[2]: https://github.com/markserv
[3]: /html/
facelessuser [1] has done a fantastic job with pymdown-extensions [2]. Highly recommend taking a look.
Extensions for Python Markdown
References:
[1]: https://github.com/facelessuser
[2]: https://github.com/facelessuser/pymdown-extensions
Check out dirsync [1] by tkhyn [2]. It’s a well-crafted project with great potential.
No description available.
References:
[1]: https://github.com/tkhyn/dirsync
[2]: https://github.com/tkhyn
Check out andrewlin12 [1] and their project markdown2png [2].
Render markdown to PNG (or other formats)
References:
[1]: https://github.com/andrewlin12
[2]: https://github.com/andrewlin12/markdown2png
Stand With Your Team
People who are quick to toss team members under a bus are not well trusted or
highly thought of and it will lead to some toxic team dynamics.
Building Steam # [1]
While collaborating on any project there are going to be decisions made that
aren’t necessarily your favorite, during a summer internship my mentor made
some decisions that I was not on board with, but I accepted his wisdom and
moved forward with little push back.
Full Spead Ahead # [2]
During a review, leadership showed interest in the option that I wanted to go
towards. I was quick to jump up and say I told you so right then and there
and pitch reasons why my idea was so much better.
The Aftermath # [3]
I walked out so proud that leadership recognized my idea, but I will never
forget the look on my teammate’s face and what he told me after the meeting.
“Wow quick to throw me under the bus in front of everyone 😢”. I
~ my teammate
Realizing what Happened # [4]
It was at this moment that I threw my mentor, a good ...
Blogging For Me
I create this blog with one person in mind, me.
There are others like me # [1]
This is not completely selfish, as there are likely many others out there that
think similarly to me. Everyone comes from different backgrounds and varying
levels of experience. In no way do you need to be an expert to create content
others will benefit from.
Accurate # [2]
I am as accurate as possible. I don’t know everything, and If I waited for
that to happen I would never post, or write at such a high level no one else
(including me) would ever want to read.
Generally I know that I am directionally accurate for most of the article. In
fact there is often a part of the article that I don’t feel the best about. It
may have been some code that could have been better. I try to point these
things out. Often when I am and point out where I know there is
improvement to be had I get the most welcoming comments, “that was just fine
how you did that”, or “I tried that myself and saw a better abstraction.
...
I recently discovered twint [1] by twintproject [2], and it’s truly impressive.
An advanced Twitter scraping & OSINT tool written in Python that doesn’t use Twitter’s API, allowing you to scrape a user’s followers, following, Tweets and more while evading most API limitations.
References:
[1]: https://github.com/twintproject/twint
[2]: https://github.com/twintproject
I like pytest-dev’s [1] project pluggy [2].
A minimalist production ready plugin system
References:
[1]: https://github.com/pytest-dev
[2]: https://github.com/pytest-dev/pluggy
to-mc [1] has done a fantastic job with checksumdir [2]. Highly recommend taking a look.
Simple package to compute a single deterministic hash of the file contents of a directory.
References:
[1]: https://github.com/to-mc
[2]: https://github.com/to-mc/checksumdir