5d (edited) • 💻 Technical Skills
New Online Course: How Big data from Smart Meters are processed efficiently
I’ve just published a new online course about Memory-Efficient Processing of Big Data. This course teaches real-world skills as they are used in practice.
Smart meters measure the electricity-consumption data every hour, and store the information in CSV files. These files eventually become very large (big data).
The new online course is called "Smart Meter Big Data Efficient Processing" and it is in the Classroom in 1.36. This online course teaches a Python methodology that is used by energy companies in practice to read extremely large datasets (Big Data). Without this technique such files cannot be read because they cause a memory (RAM) error.
Companies that sell electricity to consumers are known as 'Retailers' or 'Suppliers'. Such companies have CSV files with hundreds of millions of rows, where each row is the hourly kWh electricity consumption.
If they try to load these CSV files, their computers will run out of RAM and crash.
So these companies process these files using Python iterators, which enable a memory-efficient and fast processing method.
In this course, I show you the industry-standard solution: using Python Iterators to process Big Data in "chunks". See the attached image; this is analysed in detail in the course.
22
8 comments
New Online Course: How Big data from Smart Meters are processed efficiently
powered by
skool.com/software-school-for-energy-7177
A beginner-friendly training program for a career in the energy sector. Regardless of your location, age, sex, education or experience level.
Suggested communities
Powered by