• Skip to main content
  • Skip to header right navigation
  • Skip to site footer

My Online Training Hub

Learn Dashboards, Excel, Power BI, Power Query, Power Pivot

  • Courses
  • Pricing
    • Free Courses
    • Power BI Course
    • Excel Power Query Course
    • Power Pivot and DAX Course
    • Excel Dashboard Course
    • Excel PivotTable Course – Quick Start
    • Advanced Excel Formulas Course
    • Excel Expert Advanced Excel Training
    • Excel Tables Course
    • Excel, Word, Outlook
    • Financial Modelling Course
    • Excel PivotTable Course
    • Excel for Customer Service Professionals
    • Excel for Operations Management Course
    • Excel for Decision Making Under Uncertainty Course
    • Excel for Finance Course
    • Excel Analysis ToolPak Course
    • Multi-User Pricing
  • Resources
    • Free Downloads
    • Excel Functions Explained
    • Excel Formulas
    • Excel Add-ins
    • IF Function
      • Excel IF Statement Explained
      • Excel IF AND OR Functions
      • IF Formula Builder
    • Time & Dates in Excel
      • Excel Date & Time
      • Calculating Time in Excel
      • Excel Time Calculation Tricks
      • Excel Date and Time Formatting
    • Excel Keyboard Shortcuts
    • Excel Custom Number Format Guide
    • Pivot Tables Guide
    • VLOOKUP Guide
    • ALT Codes
    • Excel VBA & Macros
    • Excel User Forms
    • VBA String Functions
  • Members
    • Login
    • Password Reset
  • Blog
  • Excel Webinars
  • Excel Forum
    • Register as Forum Member

1 Million Row Limit|General Excel Questions & Answers|Excel Forum|My Online Training Hub

You are here: Home / 1 Million Row Limit|General Excel Questions & Answers|Excel Forum|My Online Training Hub
Avatar
sp_LogInOut Log In sp_Registration Register
sp_Search Search
Advanced Search|Last Search Results
Search
Forum Scope




Match



Forum Options



Minimum search word length is 3 characters - maximum search word length is 84 characters
sp_Search Search
sp_RankInfo
Lost password?
sp_CrumbsHome HomeExcel ForumGeneral Excel Questions & Answe…1 Million Row Limit
sp_PrintTopic sp_TopicIcon
1 Million Row Limit
Avatar
PaulFogel
Member
Members
Level 0
Forum Posts: 81
Member Since:
July 7, 2016
sp_UserOfflineSmall Offline
1
March 20, 2019 - 3:27 am
sp_Permalink sp_Print

I have two tables of Excel data which I need to combine. Together they are more than a million rows, which Excel cannot accept. I tried to create a Pivot Table from them in the hope that it would summarize to many less rows. But the nature of the data is such that each row is unique, so the resulting Pivot table used the same number of rows. Next, I tried Power Query, but Close and Load still filled up the same million plus rows, which wouldn't work.

What might be the solution?

Paul

Avatar
Catalin Bombea
Iasi, Romania
Admin
Level 10
Forum Posts: 1807
Member Since:
November 8, 2013
sp_UserOfflineSmall Offline
2
March 20, 2019 - 6:26 pm
sp_Permalink sp_Print sp_EditHistory

In the query, you can add an index column, based on that index column you can calculate and add a new column: rows 1 to 999.999 will be part of the 1 Million group, 2.000.000 to 2.999.999 will be part of the 2 Million group, and so on.

This should be the formula for the new column, to calculate groups based on the index column:

=1000000*(1+Number.RoundTowardZero([Index]/1000000))

Load the query as connection only, then insert a pivot table based on that connection.

Add a slicer for that Group field, and you will be able to see the data 1 million rows at a time, no matter if data has 100 million rows. Of course, you should not select in the slicer more than 1 group.

Obviously, you can set the group size to a smaller value, if you want to see only 100.000 rows, or 500.000 rows, just adjust the formula to desired group size.

If the group size is smaller, you will be able to select more than 1 group in the slicer, as long as the cumulated groups are not exceeding excel sheet rows limits.

Avatar
PaulFogel
Member
Members
Level 0
Forum Posts: 81
Member Since:
July 7, 2016
sp_UserOfflineSmall Offline
3
April 17, 2019 - 4:47 am
sp_Permalink sp_Print

Catalin,

I wanted to get back to you on this. I'm greatly encouraged there's a way to do this, but I couldn't get past the custom column, above, without running into errors. I'm just not experienced enough with PQ.

Is it possible for you to do the steps above with the sample data, attached? I've limited the rows to fit within the file size limits on this site.

Paul

Avatar
Catalin Bombea
Iasi, Romania
Admin
Level 10
Forum Posts: 1807
Member Since:
November 8, 2013
sp_UserOfflineSmall Offline
4
April 17, 2019 - 5:07 am
sp_Permalink sp_Print

Here is it, i just did what I already said: added an index column, then a custom column with the formula provided.

Load the query as a connection only (not to worksheet table), then add a pivot based on that connection, with a slicer for the group. I used a 5000 records group size.

sp_Feed
Go to top
Forum Timezone: Australia/Brisbane
Most Users Ever Online: 245
Currently Online: Brian Pham, drsven
Guest(s) 9
Currently Browsing this Page:
1 Guest(s)
Top Posters:
SunnyKow: 1432
Anders Sehlstedt: 870
Purfleet: 412
Frans Visser: 346
David_Ng: 306
lea cohen: 219
A.Maurizio: 202
Jessica Stewart: 202
Aye Mu: 201
jaryszek: 183
Newest Members:
drsven
Annie Witbrod
wahab tunde
Cong Le Duc
Faisal Bashir
Ivica Cvetkovski
Blaine Cox
Shankar Srinivasan
riyepa fdgf
Hannah Cave
Forum Stats:
Groups: 3
Forums: 24
Topics: 6205
Posts: 27211

 

Member Stats:
Guest Posters: 49
Members: 31880
Moderators: 3
Admins: 4
Administrators: Mynda Treacy, Philip Treacy, Catalin Bombea, FT
Moderators: MOTH Support, Velouria, Riny van Eekelen
© Simple:Press —sp_Information

Sidebar

Blog Categories

  • Excel
  • Excel Charts
  • Excel Dashboard
  • Excel Formulas
  • Excel PivotTables
  • Excel Shortcuts
  • Excel VBA
  • General Tips
  • Online Training
  • Outlook
  • Power Apps
  • Power Automate
  • Power BI
  • Power Pivot
  • Power Query
microsoft mvp logo
trustpilot excellent rating
Secured by Sucuri Badge
MyOnlineTrainingHub on YouTube Mynda Treacy on Linked In Mynda Treacy on Instagram Mynda Treacy on Twitter Mynda Treacy on Pinterest MyOnlineTrainingHub on Facebook
 

Company

  • About My Online Training Hub
  • Disclosure Statement
  • Frequently Asked Questions
  • Guarantee
  • Privacy Policy
  • Terms & Conditions
  • Testimonials
  • Become an Affiliate

Support

  • Contact
  • Forum
  • Helpdesk - For Technical Issues

Copyright © 2023 · My Online Training Hub · All Rights Reserved. Microsoft and the Microsoft Office logo are trademarks or registered trademarks of Microsoft Corporation in the United States and/or other countries. Product names, logos, brands, and other trademarks featured or referred to within this website are the property of their respective trademark holders.