Background Tasks in Python for Data Science
===========================================

This post is intended as an extension/update from background tasks in python. I started using the week that Kenneth Reitz released it. It takes away so much...

Date: September 10, 2019

This post is intended as an extension/update from [background tasks in
python](https://waylonwalker.com/background-1/). I started using `background`
the week that Kenneth Reitz released it. It takes away so much boilerplate
from running background tasks that I use it in more places than I probably
should. After taking a look at that post today, I wanted to put a better data
science example in here to help folks get started.

This post is intended as an extension/update from [background tasks in python](https://waylonwalker.com/background-1/). I started using `background` the week that Kenneth Reitz released it. It takes away so much boilerplate from running background tasks that I use it in more places than I probably should. After taking a look at that post today, I wanted to put a better data science example in here to help folks get started.

> I use it in more places than I probably should

Before we get into it, I want to make a shout out to Kenneth Reitz for making this so easy. Kenneth is a python God for all that he has given to the community in so many ways, especially with his ideas in building stupid simple api's for very complicated things.

## Installation

### install via pip

 pip install background

### install via github

I believe one of the later pr's to the project fixes the way arguments are passed in. I generally clone the repo or copy the module directly into my project.

**clone it**

 git clone https://github.com/ParthS007/background.git
 cd background
 python setup.py install

**copy the module**

 curl https://raw.githubusercontent.com/ParthS007/background/master/background.py > background.py

## 🐌 The Slow Function

Imagine that this function is a big one! This function is fairly realistic as it takes in some input and returns a DataFrame. This is what a good half of my fuctions do in data science. The internals of this function generally will include a sql query, load from s3 or a data catalog, an aggregation from another DataFrame. In general it should do one simple thing.

**Feel Free to copy this "boilerplate"**

``` python
import background
from time import sleep
import pandas as pd

@background.task
def long_func(i):
 """
 Simulates fetching data from a service
 and returning a pandas DataFrame.

 """
 sleep(10)
 return pd.DataFrame({'number_squared': [i**2]})
```

## Calling the Slow Function

_it's the future calling 🤙_

If we were to call this function 10 times it would take 100s. Not bad for a dumb example, but detrimental when this gets scaled up💥. We want to utilize all of our available resources to reduce our development time and get moving on our project.

Calling `long_func` will return a future object. This object has a number of methods that you can read about in the [cpython docs](https://docs.python.org/3/library/concurrent.futures.html#future-objects). The main one we are interested in is `result`. I typically call these functions many times and put them into a list object so that I can track their progress and get their results. If you needed to map inputs back to the result use a dictionary.

``` python
%time futures = [long_func(i) for i in range(10)]

CPU times: user 319 µs, sys: 197 µs, total: 516 µs
Wall time: 212 µs
```

## Do something with those `results()`

Simply running the function completes in no time! This is because the future objects that are returned are non blocking and will run in a background task using the `ProcessPoolExecutor`. To get the result back out we need to call the `result` method on the future object.`result` is a blocking function that will not realease until the function has completed.

``` python
%%time
futures = [long_func(i) for i in range(10)]
pd.concat([future.result() for future in futures])

CPU times: user 5.38 ms, sys: 3.53 ms, total: 8.9 ms
Wall time: 10 s
```

Note that this example completed in `10s`, the time it took for only one run, not all 10! 😎

## n

😫 _crank it up_

By default the number of parallel processes wil be equal to the number of cpu threads on your machine. To increase the number of parallel processes (`max_workers`) set increase `background.n`.

``` python
background.n = 100
```

## Is it possible to overruse @background.task?

I use this essentially anywhere that I cannot vectorize a python operation and push the compute down into those fast 💨 c extended libraries like numpy, and the operation takes more than a few minutes. Nearly every big network request I make gets broken down into chunks and multithreaded. Let me know... is is possible to overruse `@background.task`? Let me know your thoughts [@_WaylonWalker](https://twitter.com/_WaylonWalker).

## Repl.It

Play with the code here! Try different values of background.n and n_runs.