Posts tagged: python

I’m poking a bit into gamedev. Partly to better understand, partly because it’s stretching different parts of my brain/skillset than writing data pipelines does, but mostly for the experience of designing them with my 9yo Wyatt.

I’ve seen several pygame boilerplate templates, but they all seem to rely heavily on globl variables. That’s just not how I generally develop anything. I want a package that I can pip install, run, import, test, all the good stuff.

What currently have is a single module starter package that is on github so that I can install it and start building games with very little code.

Since it’s a package on GitHub you can install it with the git+ prefix.

...

My personal Site build went down last week, and I was unable to publish a new article. This is the process I went through to get it back up and running quickly.

Classic IT fix, rerun it and see if you get the same error. Everyone is busy and when you have your build go down you are probably busy doing something else. My first step is often to simply click rerun right from GitHub actions. Sometimes this will fix it, and sometimes it doesn’t. It’s an easy fix to run in the meantime you are not focused on fixing it.

Also worth a check to see if GitHub is having a hiccup or not. This error felt pretty obviously not GitHub’s fault, but it’s a good one to check when you run into a weird unexplainable error.

Check github status for any downtime issues with actions.

...

I ran into a PR this week where the author was inheriting what BaseException rather than exception. I made this example to illustrate the unintended side effects that it can have.

Try running these examples in a .py file for yourself and try to kill them with control-c.

Since things such as KeyboardInterrupt are created as an exception that inherits from BaseException, if you except BaseException you can no longer KeyboardInterrupt.

from time import sleep while True: try: sleep(30) except BaseException: # ❌ pass

except from Exception or higher #

If you except from exception or something than inherits from it you will be better off, and avoid unintended side effects.

...

When I need a consistent key for a pythohn object I often reach for hashlib.md5 It works for me and the use cases I have.

Yesterday we talked about setting up a persistant cache with python diskcache. In order to make this really work we need a good way to make consistent cache keys from some sort of python object.

How I setup a sqlite cache in python

does not work

...

When I need to cache some data between runs or share a cache accross multiple processes my go to library in python is diskcache. It’s built on sqlite with just enough cacheing niceties that make it very worth it.

Install diskcache into your virtual environement of choice using pip from your command line.

python -m pip install diskcache

setup the cache #

There are a couple of different types of cache, Cache, FanoutCache, and DjangoCache, you can read more about those in the docs

from diskcache import Cache cache = FanoutCache('.mycache', statistics=True)

Adding to the cache #

Adding to the cache only needs a key and value.

...

The easiest way to speed up any code is to run less code. A common technique to reduce the amount of repative work is to implement a cache such that the next time you need the same work done, you don’t need to recompute anything you can simply retrieve it from a cache.

The easiest and most common to setup in python is a builtin functools.lru_cache.

from functools import lru_cache @lru_cache def get_cars(): print('pulling cars data') return pd.read_csv("https://waylonwalker.com/cars.csv", storage_options = {'User-Agent': 'Mozilla/5.0'})

when to use lru_cache #

Any time you have a function where you expect the same results each time a function is called with the same inputs, you can use lru_cache.

when same *args, **kwargs always return the same value

...

I keep a small cars.csv on my website for quickly trying out different pandas operations. It’s very handy to keep around to help what a method you are unfamiliar with does, or give a teammate an example they can replicate.

I recently switched hosting from netlify over to cloudflare. Well cloudflare does some work to block certain requests that it does not think is a real user. One of these checks is to ensure there is a real user agent on the request.

This breaks my go to example dataset.

pd.read_csv("https://waylonwalker.com/cars.csv") # HTTPError: HTTP Error 403: Forbidden

But requests works??? #

What’s weird is, requests still works just fine! Not sure why using urllib the way pandas does breaks the request, but it does.

...

Python’s requests library is one of the gold standard apis, designed by Kenneth Reitz. It was designed with the user perspective in mind first and implementation second. I have heard this called readme driven development, where the interface the user will use is laid out first, then implemented. This makes the library much mor intuitive than if it were designed around how it was easiest to implement.

Requests is on pypi and can be installed into your virtual environtment with pip.

python -m pip install requests

Getting the content of a request #

Requests makes getting content from a web url as easy as possible.

import requests r = requests.get('https://waylonwalker.com/til/htmx-get/') article = r.content

html">requests is not limited to html

...

I recently gave a talk at python web conf 2022, and one of the things I did when I should have been working on my presentation was workig on how my presentation looked… classic procrastination technique.

Lets use this section to show what it looks like as I change my styles.

from markata import Markata Markata() markata.run()

☝ This is how my website is built

This is what the above slide looks like in lookatme.

...

I use a package eyeseast/python-frontmatter{.hoverlink} to load files with frontmatter in them. Its a handy package that allows you to load files with structured frontmatter (yaml, json, or toml).

It’s on pypi, so you can install it into your virtual environment with pip.

python -m pip install python-frontmatter

🙋 What’s Frontmatter #

Frontmatter is a handy way to add metadata to your plain text files. It’s quite common to have yaml frontmatter in markdown. All of my blog posts have yaml frontmatter to give the post metadata such as post date, tags, title, and template. dev.to is a popular developer blogging...

...

Today I was watching the python web conf 2022 and saw @davidbujic use the new Dict Union Operator Live on stage during his Functional Programming talk. This operator was first introduced into python 3.9 with pep584.

I’ve long updated dicts through the use of unpacking. Note that the last item always wins. It makes it pretty easy to make user overrides to default configurations. With pep584 landing in python 3.9 we can now leverage the | operator to achieve the same result.

default_config = {'url': 'https://example.com', 'assets_dir': 'static' } user_config = {'url':...