Hello. Thank you in advance.
I am sure this is a silly question but I am not an expert.
I have 3 files per week that contain between 500K-975k rows the files have the same column headers and type. I need to combine a years worth of data. So i have about 20M rows for a years worth of files. I copy and pasted data into as many CSV files as I could to reduce the number of files. So i went from 150+ files to 30 files. I linked 10 files to power query to do a test and saved the files wasn't too large. I then appended all the files into one and it casued the file to go over 1,296,372 KB.
My goal is to manage all the data in these files and use in Powe BI to do reporting. Yes i need all the files it is trackcing shippments for a large company. Can i get all the data into a text file and upload? What is the best way to work with 220M rows of data and keep the excel file fairly small so i can use in Power BI. Thank you
Hi Chad,
I don't understand why you're preforming these intermediary steps. Why don't you simply get the CSV files with Power BI and import it direct to the Power BI model, rather than going through the appending into another Excel file? Excel files are always going to be way bigger than CSV files. I would do away with the Excel step. After all, the exact same Power Query tool is build into Power BI.
I hope that points you in the right direction.
Mynda
I am still learning some of the details of Power BI and Power Pivot. I prefer to get all of the data into a table so i don't have to have a number of tables in Power BI. My thought was to load all the data into Power Pivot/Quary append those tables into one and then link to Power BI. I have the basic Power BI so i am limited with space. Can i get 20 million records in Power BI without exceeding the limit?
Hi Chad,
Let's rewind. In Excel you have Power Query which you're wanting to use to create a single table, correct? What I'm trying to point out is that you have the exact same Power Query tool in Power BI too. Therefore, you don't need to use Excel at all. Instead, you can do exactly the same Power Query data gathering and transformations inside of Power BI, and load a single table into the Power Pivot model in Power BI. There's no need to have multiple tables in Power BI. i.e. You use Power Query inside of Power BI to get the data, consolidate it into one table and then 'close & apply' (the equivalent of Close & Load in Power BI) to the Power Pivot model in Power BI.
All Power BI licences (free and paid) allow you to build models containing 100's of millions of rows of data. It's when it comes time to sharing your reports that you'll need a paid/pro licence, as will those you want to view your reports.
So, you can stop trying to use Excel for the data gathering and cleaning and instead go straight to Power BI and do it there.
Mynda
First of all thank you for the responses. It is really helpful. So I have linked all my files in Power BI. I have 12 files that are just connections and not tables. I then used the Append to New function to combine all of the tables into 1. It may the table but will not apply and create the table. Assuming the appended table is too large? Or could it be due to limited memory on my PC? Thank you once again for your time.
The file is 16KB currently.
Error Message
FinalTest
failed to save modifcations to the server. error returned: there's not enough memory to complete this operation. Please try again later when there may be more memory available.
Hi Chad,
It sounds like you haven't got enough RAM in your PC. You should also check you have the 64-bit version of Power BI installed as this will be essential to working with this much data.
Other things to consider is if your dataset has a lot of columns e.g. > 15, then this has a significant impact on performance.
And try disabling "Allow data preview to download in the background" from options > options and settings > data load.
Mynda
