As someone who works in a Fortune 100 company, the number of spreadsheets we have for the vast majority of our tasks…
The biggest issue I’ve seen is how do you get a bunch of data to look and behave between a bunch of users who have different skillsets and varying knowledge about how the data connects to other data?
You could build a web page with a database backend. But this takes hours when plopping the data into a spreadsheet is minutes.
I work in the accounting department of an insurance company doing python development, killing excel spreadsheet processes one by one. I hate when the sources came in excel because polars lazyframes dosen’t work as nice that with other formats like csv or parquet. My outputs are always parquets is they are going to be used by other processes or excel if they for humans.
Interesting reading. I’m an actuarie in an insurance company and everything I do is in python, is easy to maintain because I’m a “solo developer” building custom tools for me and my team (with pyinstall to create GUIs of the programs so they can used them), but my internal libraries have started to grow up.
About the comments the author had about pandas, I just started to move away from it to polars because the databases I’m working now have easy 50M+ rows, and as they say came for the speed stay for the syntax. I’m debating myself if make my intern learn pandas first, or just go for polars from the begging.
Thanks for sharing - that’s actually pretty interesting. I knew about Polars, but I didn’t know it performed better. I know about that in passing from folks that are in the Cloud “Data” space, who use SaaS platforms that are heavily Python based. That includes Pandas and Polars, but also Jupyter. That really threw me for a loop, but the more I think about it, the more sense it makes.
In the first project that I had to use polars because the databases couldn’t be processed, I moved from spending 40min just uploading to memory one of the bases on pandas to 10min doing all the process on polars.
That convinced me to move everything forward to polars.
As someone who works in a Fortune 100 company, the number of spreadsheets we have for the vast majority of our tasks…
The biggest issue I’ve seen is how do you get a bunch of data to look and behave between a bunch of users who have different skillsets and varying knowledge about how the data connects to other data?
You could build a web page with a database backend. But this takes hours when plopping the data into a spreadsheet is minutes.
Accountants are learning python to parse spreadsheets now
I work in the accounting department of an insurance company doing python development, killing excel spreadsheet processes one by one. I hate when the sources came in excel because polars lazyframes dosen’t work as nice that with other formats like csv or parquet. My outputs are always parquets is they are going to be used by other processes or excel if they for humans.
Meanwhile Banks are using… Bank Python.
Interesting reading. I’m an actuarie in an insurance company and everything I do is in python, is easy to maintain because I’m a “solo developer” building custom tools for me and my team (with pyinstall to create GUIs of the programs so they can used them), but my internal libraries have started to grow up.
About the comments the author had about pandas, I just started to move away from it to polars because the databases I’m working now have easy 50M+ rows, and as they say came for the speed stay for the syntax. I’m debating myself if make my intern learn pandas first, or just go for polars from the begging.
Thanks for sharing - that’s actually pretty interesting. I knew about Polars, but I didn’t know it performed better. I know about that in passing from folks that are in the Cloud “Data” space, who use SaaS platforms that are heavily Python based. That includes Pandas and Polars, but also Jupyter. That really threw me for a loop, but the more I think about it, the more sense it makes.
In the first project that I had to use polars because the databases couldn’t be processed, I moved from spending 40min just uploading to memory one of the bases on pandas to 10min doing all the process on polars.
That convinced me to move everything forward to polars.