!https://simonwillison.net/2024/Jul/23/sqlite-jiff/
Posts tagged: sql
All posts with the tag "sql"
!https://stackoverflow.com/questions/8988915/sqlite-count-slow-on-big-tables
!https://kerkour.com/sqlite-for-servers
!https://sqlite-utils.datasette.io/en/stable/python-api.html#full-text-search
!https://stackoverflow.com/questions/4186062/sqlalchemy-order-by-descending
!https://simonwillison.net/2023/Jul/24/sqlite-utils-plugins/
!https://docs.sqlalchemy.org/en/14/core/defaults.html#server-invoked-ddl-explicit-default-expressions
!https://sqlite-utils.datasette.io/en/stable/cli.html#querying-data-directly-using-an-in-memory-database
Stepping Up My SQL Game
In 2018 I transitioned from a Product Engineering (Mechanical) role to a Data Scientist Role. I entered this space with strong subject matter expertise with our products, our data, munging through data in pyhon, and data visualization in python. My sql skills were lacking to say the least. I had learned what I needed to know to get data from our relational databases, then use pandas to do any further analysis. Just run something like the following and you have data.
SELECT * FROM Table Where col_1 = 'col_1_filter'
This technique works great for small data sets that you only need to run once. There is no shame to pull in a big dataset and start munging with it in pandas to get some results, and make decisions. The problem becomes when your dataset becomes too big or you need to run the query on a frequent basis. Doing the aggregations on the server run much quicker, as it reduces the time spent in io. My longest running steps are currently io related. Reducing these steps have improved my workflow. At the point that I was getting server timeout errors, or using the same long running query in many places I would be searching for examples online, because I just did not have the experience with many more techniques. I decided it was time to put away the cheat sheets, step away from Stack Overflow, and improve my speed.
SQL is far from the hot topic in 2018, AI, Deep Learning, BIG data, Machine Learning, Natural Language Processing take the win here....
...