• Skip to main content
  • Skip to header right navigation
  • Skip to site footer

My Online Training Hub

Learn Dashboards, Excel, Power BI, Power Query, Power Pivot

  • Courses
  • Pricing
    • Free Courses
    • Power BI Course
    • Excel Power Query Course
    • Power Pivot and DAX Course
    • Excel Dashboard Course
    • Excel PivotTable Course – Quick Start
    • Advanced Excel Formulas Course
    • Excel Expert Advanced Excel Training
    • Excel Tables Course
    • Excel, Word, Outlook
    • Financial Modelling Course
    • Excel PivotTable Course
    • Excel for Customer Service Professionals
    • Excel for Operations Management Course
    • Excel for Decision Making Under Uncertainty Course
    • Excel for Finance Course
    • Excel Analysis ToolPak Course
    • Multi-User Pricing
  • Resources
    • Free Downloads
    • Excel Functions Explained
    • Excel Formulas
    • Excel Add-ins
    • IF Function
      • Excel IF Statement Explained
      • Excel IF AND OR Functions
      • IF Formula Builder
    • Time & Dates in Excel
      • Excel Date & Time
      • Calculating Time in Excel
      • Excel Time Calculation Tricks
      • Excel Date and Time Formatting
    • Excel Keyboard Shortcuts
    • Excel Custom Number Format Guide
    • Pivot Tables Guide
    • VLOOKUP Guide
    • ALT Codes
    • Excel VBA & Macros
    • Excel User Forms
    • VBA String Functions
  • Members
    • Login
  • Blog
  • Excel Webinars
  • Excel Forum
    • Register as Forum Member

Power Pivot to Power Bi|Power Query|Excel Forum|My Online Training Hub

You are here: Home / Power Pivot to Power Bi|Power Query|Excel Forum|My Online Training Hub

vba course banner

Avatar
sp_LogInOut Log In sp_Registration Register
sp_Search Search
Advanced Search
Search
Forum Scope




Match



Forum Options



Minimum search word length is 3 characters - maximum search word length is 84 characters
sp_Search Search
sp_RankInfo
Lost password?
sp_CrumbsHome HomeExcel ForumPower QueryPower Pivot to Power Bi
sp_PrintTopic sp_TopicIcon
Power Pivot to Power Bi
Avatar
Chad Wood

Active Member
Members
Level 0
Forum Posts: 4
Member Since:
January 25, 2021
sp_UserOfflineSmall Offline
1
January 25, 2021 - 3:28 pm
sp_Permalink sp_Print sp_EditHistory

Hello.  Thank you in advance.  

I am sure this is a silly question but I am not an expert. 

I have 3 files per week that contain between 500K-975k rows the files have the same column headers and type.  I need to combine a years worth of data.  So i have about 20M rows for a years worth of files.  I copy and pasted data into as many CSV files as I could to reduce the number of files.  So i went from 150+ files to 30 files.  I linked 10 files to power query to do a test and saved the files wasn't too large.  I then appended all the files into one and it casued the file to go over 1,296,372 KB.  

My goal is to manage all the data in these files and use in Powe BI to do reporting.  Yes i need all the files it is trackcing shippments for a large company.  Can i get all the data into a text file and upload?  What is the best way to work with 220M rows of data and keep the excel file fairly small so i can use in Power BI.  Thank you

Avatar
Mynda Treacy
Admin
Level 10
Forum Posts: 4519
Member Since:
July 16, 2010
sp_UserOnlineSmall Online
2
January 25, 2021 - 4:22 pm
sp_Permalink sp_Print

Hi Chad,

I don't understand why you're preforming these intermediary steps. Why don't you simply get the CSV files with Power BI and import it direct to the Power BI model, rather than going through the appending into another Excel file? Excel files are always going to be way bigger than CSV files. I would do away with the Excel step. After all, the exact same Power Query tool is build into Power BI.

I hope that points you in the right direction.

Mynda

Avatar
Chad Wood

Active Member
Members
Level 0
Forum Posts: 4
Member Since:
January 25, 2021
sp_UserOfflineSmall Offline
3
January 26, 2021 - 5:41 am
sp_Permalink sp_Print

I am still learning some of the details of Power BI and Power Pivot.  I prefer to get all of the data into a table so i don't have to have a number of tables in Power BI.  My thought was to load all the data into Power Pivot/Quary append those tables into one and then link to Power BI.  I have the basic Power BI so i am limited with space.  Can i get 20 million records in Power BI without exceeding the limit?

Avatar
Mynda Treacy
Admin
Level 10
Forum Posts: 4519
Member Since:
July 16, 2010
sp_UserOnlineSmall Online
4
January 26, 2021 - 9:33 am
sp_Permalink sp_Print

Hi Chad,

Let's rewind. In Excel you have Power Query which you're wanting to use to create a single table, correct? What I'm trying to point out is that you have the exact same Power Query tool in Power BI too. Therefore, you don't need to use Excel at all. Instead, you can do exactly the same Power Query data gathering and transformations inside of Power BI, and load a single table into the Power Pivot model in Power BI. There's no need to have multiple tables in Power BI. i.e. You use Power Query inside of Power BI to get the data, consolidate it into one table and then 'close & apply' (the equivalent of Close & Load in Power BI) to the Power Pivot model in Power BI.

All Power BI licences (free and paid) allow you to build models containing 100's of millions of rows of data. It's when it comes time to sharing your reports that you'll need a paid/pro licence, as will those you want to view your reports. 

So, you can stop trying to use Excel for the data gathering and cleaning and instead go straight to Power BI and do it there.

Mynda

Avatar
Chad Wood

Active Member
Members
Level 0
Forum Posts: 4
Member Since:
January 25, 2021
sp_UserOfflineSmall Offline
5
January 30, 2021 - 1:44 am
sp_Permalink sp_Print

First of all thank you for the responses.  It is really helpful.  So I have linked all my files in Power BI.   I have 12 files that are just connections and not tables.  I then used the Append to New function to combine all of the tables into 1.  It may the table but will not apply and create the table.  Assuming the appended table is too large?  Or could it be due to limited memory on my PC?  Thank you once again for your time.

The file is 16KB currently.  

Error Message 

FinalTest

failed to save modifcations to the server.  error returned: there's not enough memory to complete this operation.  Please try again later when there may be more memory available. 

Avatar
Mynda Treacy
Admin
Level 10
Forum Posts: 4519
Member Since:
July 16, 2010
sp_UserOnlineSmall Online
6
January 30, 2021 - 1:50 pm
sp_Permalink sp_Print sp_EditHistory

Hi Chad,

It sounds like you haven't got enough RAM in your PC. You should also check you have the 64-bit version of Power BI installed as this will be essential to working with this much data.

Other things to consider is if your dataset has a lot of columns e.g. > 15, then this has a significant impact on performance.

And try disabling "Allow data preview to download in the background" from options > options and settings > data load. 

Mynda

sp_Feed
Go to top
Forum Timezone: Australia/Brisbane
Most Users Ever Online: 245
Currently Online: Mynda Treacy, Mohamed Jelle Hussein
Guest(s) 8
Currently Browsing this Page:
1 Guest(s)
Top Posters:
SunnyKow: 1432
Anders Sehlstedt: 873
Purfleet: 414
Frans Visser: 346
David_Ng: 306
lea cohen: 222
Jessica Stewart: 218
A.Maurizio: 202
Aye Mu: 201
jaryszek: 183
Newest Members:
Aminu Sule
Samuel Ramirez
Abdifatah Osman
mohad borhom
Abayomi Adedeji
Ganesh MVS
Gilbert Lemek
Ashleigh Farquharson
Jayz Luu
Fred Smith
Forum Stats:
Groups: 3
Forums: 24
Topics: 6364
Posts: 27828

 

Member Stats:
Guest Posters: 49
Members: 32346
Moderators: 3
Admins: 4
Administrators: Mynda Treacy, Philip Treacy, Catalin Bombea, FT
Moderators: MOTH Support, Velouria, Riny van Eekelen
© Simple:Press —sp_Information

Sidebar

Blog Categories

  • Excel
  • Excel Charts
  • Excel Dashboard
  • Excel Formulas
  • Excel PivotTables
  • Excel Shortcuts
  • Excel VBA
  • General Tips
  • Online Training
  • Outlook
  • Power Apps
  • Power Automate
  • Power BI
  • Power Pivot
  • Power Query
microsoft mvp logo
trustpilot excellent rating
Secured by Sucuri Badge
MyOnlineTrainingHub on YouTube Mynda Treacy on Linked In Mynda Treacy on Instagram Mynda Treacy on Twitter Mynda Treacy on Pinterest MyOnlineTrainingHub on Facebook
 

Company

  • About My Online Training Hub
  • Disclosure Statement
  • Frequently Asked Questions
  • Guarantee
  • Privacy Policy
  • Terms & Conditions
  • Testimonials
  • Become an Affiliate

Support

  • Contact
  • Forum
  • Helpdesk - For Technical Issues

Copyright © 2023 · My Online Training Hub · All Rights Reserved. Microsoft and the Microsoft Office logo are trademarks or registered trademarks of Microsoft Corporation in the United States and/or other countries. Product names, logos, brands, and other trademarks featured or referred to within this website are the property of their respective trademark holders.