Log In
Register
Home
July 7, 2016
Offline
I have two tables of Excel data which I need to combine. Together they are more than a million rows, which Excel cannot accept. I tried to create a Pivot Table from them in the hope that it would summarize to many less rows. But the nature of the data is such that each row is unique, so the resulting Pivot table used the same number of rows. Next, I tried Power Query, but Close and Load still filled up the same million plus rows, which wouldn't work.
What might be the solution?
Paul
November 8, 2013
Offline
In the query, you can add an index column, based on that index column you can calculate and add a new column: rows 1 to 999.999 will be part of the 1 Million group, 2.000.000 to 2.999.999 will be part of the 2 Million group, and so on.
This should be the formula for the new column, to calculate groups based on the index column:
=1000000*(1+Number.RoundTowardZero([Index]/1000000))
Load the query as connection only, then insert a pivot table based on that connection.
Add a slicer for that Group field, and you will be able to see the data 1 million rows at a time, no matter if data has 100 million rows. Of course, you should not select in the slicer more than 1 group.
Obviously, you can set the group size to a smaller value, if you want to see only 100.000 rows, or 500.000 rows, just adjust the formula to desired group size.
If the group size is smaller, you will be able to select more than 1 group in the slicer, as long as the cumulated groups are not exceeding excel sheet rows limits.
July 7, 2016
Offline
Catalin,
I wanted to get back to you on this. I'm greatly encouraged there's a way to do this, but I couldn't get past the custom column, above, without running into errors. I'm just not experienced enough with PQ.
Is it possible for you to do the steps above with the sample data, attached? I've limited the rows to fit within the file size limits on this site.
Paul
November 8, 2013
Offline
Here is it, i just did what I already said: added an index column, then a custom column with the formula provided.
Load the query as a connection only (not to worksheet table), then add a pivot based on that connection, with a slicer for the group. I used a 5000 records group size.
1 Guest(s)
