• Skip to main content
  • Skip to header right navigation
  • Skip to site footer

My Online Training Hub

Learn Dashboards, Excel, Power BI, Power Query, Power Pivot

  • Courses
  • Pricing
    • Free Courses
    • Power BI Course
    • Excel Power Query Course
    • Power Pivot and DAX Course
    • Excel Dashboard Course
    • Excel PivotTable Course – Quick Start
    • Advanced Excel Formulas Course
    • Excel Expert Advanced Excel Training
    • Excel Tables Course
    • Excel, Word, Outlook
    • Financial Modelling Course
    • Excel PivotTable Course
    • Excel for Customer Service Professionals
    • Excel for Operations Management Course
    • Excel for Decision Making Under Uncertainty Course
    • Excel for Finance Course
    • Excel Analysis ToolPak Course
    • Multi-User Pricing
  • Resources
    • Free Downloads
    • Excel Functions Explained
    • Excel Formulas
    • Excel Add-ins
    • IF Function
      • Excel IF Statement Explained
      • Excel IF AND OR Functions
      • IF Formula Builder
    • Time & Dates in Excel
      • Excel Date & Time
      • Calculating Time in Excel
      • Excel Time Calculation Tricks
      • Excel Date and Time Formatting
    • Excel Keyboard Shortcuts
    • Excel Custom Number Format Guide
    • Pivot Tables Guide
    • VLOOKUP Guide
    • ALT Codes
    • Excel VBA & Macros
    • Excel User Forms
    • VBA String Functions
  • Members
    • Login
  • Blog
  • Excel Webinars
  • Excel Forum
    • Register as Forum Member

Unpivot tables combined from separate files in PowerQuery doesn't seem to unpivot a new column|Power Query|Excel Forum|My Online Training Hub

You are here: Home / Unpivot tables combined from separate files in PowerQuery doesn't seem to unpivot a new column|Power Query|Excel Forum|My Online Training Hub

vba course banner

Avatar
sp_LogInOut Log In sp_Registration Register
sp_Search Search
Advanced Search|Last Search Results
Search
Forum Scope




Match



Forum Options



Minimum search word length is 3 characters - maximum search word length is 84 characters
sp_Search Search
sp_RankInfo
Lost password?
sp_CrumbsHome HomeExcel ForumPower QueryUnpivot tables combined from separa…
sp_PrintTopic sp_TopicIcon
Unpivot tables combined from separate files in PowerQuery doesn't seem to unpivot a new column
Avatar
John Doe

New Member
Members
Level 0
Forum Posts: 2
Member Since:
June 18, 2021
sp_UserOfflineSmall Offline
1
June 18, 2021 - 4:50 pm
sp_Permalink sp_Print sp_EditHistory

Hi there everyone!

This may be a slightly stupid question, as I am very much out of my depth! I'm a lawyer who has been tasked with finding a way of tracking time recording of employees internally. 

I've been doing a bit of research and stumbled across PowerQuery 

have a particular problem with the "Unpivot Other Columns" function.

Despite my best efforts to convince people further up the chain to record data in an unpivoted format, they are insisting 'in the interests of user friendliness' that each employee time records by client in a new column in this format:

 

Date |  Client A | Client B | Client C | Client D etc etc

 

The file name of each employee will identify the employee.

Thanks to Mynda's videos on unpivot/cleaning up poorly structured data, I thought I had the solution. PowerQuery the folder with all the employee excel files having tables in the format above, add the employee's name using the file name so the table starts to look like this:

Employee Name | Date |  Client A | Client B | Client C | Client D etc etc

 

select "Employee Name" and "Date" and then "Unpivot Other Columns".

 

Now, this actually seems to work, which was awesome for me! But I've hit a stumbling block.

To test the resilience of this, I just added a "Client E" to just one of the employee files. For some reason, PowerQuery was not picking this up as an extra column at all when all the files are combined into a single table (i.e. the step called "Expanded Table Column1"). This is despite the fact that when I open that speciifc Employee's File by expanding just that table in PowerQuery, the extra column for Client E is there. So what I'm able to deduce (I think!) is that something is going wrong at the "Expanded Table Column1" step.  

Am I doing something wrong? Or is it not possible to have extra columns added to individual files then be combined into a single table after the first PowerQuery is completed? 

 

Thank you!

Avatar
Philip Treacy
Admin
Level 10
Forum Posts: 1529
Member Since:
October 5, 2010
sp_UserOfflineSmall Offline
2
June 21, 2021 - 3:25 pm
sp_Permalink sp_Print

Hi John,

You've already mentioned the solution to this which is to record data in the correct, tabular format.

As it is, when PQ imports data in columns it uses one of the files in the folder as its 'template' for how the data is laid out.  PQ expects that the number of columns, and the column names, are the same for each file.

The purpose of loading files from a folder is to append them together so if the number of columns is different it doesn't know what to do.  If one file has a column for Customer D and the other files don't, you'll end up with a column for Customer D that is mostly empty because only 1 employee recorded time for that customer.  Multiply that for many customers and employees and you'll end up with lots of empty/null values allover the place.  Very messy.

When a file does not fit the template then you get problems like the ones you are experiencing.

I don't understand how entering data in a tabular format is not user friendly?

Whilst it may be a hard sell to those people 'up the chain' a tabular format is the correct way to record data.  If it's not you'll just keep running into issues like this.

Regards

Phil

Avatar
John Doe

New Member
Members
Level 0
Forum Posts: 2
Member Since:
June 18, 2021
sp_UserOfflineSmall Offline
3
June 21, 2021 - 8:08 pm
sp_Permalink sp_Print

Ah, thank you so much Phil! I suspected this would be more an input issue rather than a PowerQuery one!

Time for me to butt heads then!

Avatar
Velouria
London or thereabouts
Moderator
Members


Trusted Members

Moderators
Level 4
Forum Posts: 648
Member Since:
November 1, 2018
sp_UserOfflineSmall Offline
4
June 22, 2021 - 1:05 am
sp_Permalink sp_Print sp_EditHistory

If you do the unpivot in the sample function that is created, I would think that your process should work since the additional client will just be represented by additional rows in the aggregation steps, not additional columns.

sp_Feed
Go to top
Forum Timezone: Australia/Brisbane
Most Users Ever Online: 245
Currently Online: Riny van Eekelen, Lavina Dhanwani, Malcolm Sawyer
Guest(s) 10
Currently Browsing this Page:
1 Guest(s)
Top Posters:
SunnyKow: 1432
Anders Sehlstedt: 873
Purfleet: 414
Frans Visser: 346
David_Ng: 306
lea cohen: 222
Jessica Stewart: 216
A.Maurizio: 202
Aye Mu: 201
jaryszek: 183
Newest Members:
Melanie Ford
Isaac Felbah
Adele Glover
Hitesh Asrani
Rohan Abraham
Anthony van Riessen
Erlinda Eloriaga
Abisola Ogundele
MARTYN STERRY
Rahim Lakhani
Forum Stats:
Groups: 3
Forums: 24
Topics: 6355
Posts: 27792

 

Member Stats:
Guest Posters: 49
Members: 32324
Moderators: 3
Admins: 4
Administrators: Mynda Treacy, Philip Treacy, Catalin Bombea, FT
Moderators: MOTH Support, Velouria, Riny van Eekelen
© Simple:Press —sp_Information

Sidebar

Blog Categories

  • Excel
  • Excel Charts
  • Excel Dashboard
  • Excel Formulas
  • Excel PivotTables
  • Excel Shortcuts
  • Excel VBA
  • General Tips
  • Online Training
  • Outlook
  • Power Apps
  • Power Automate
  • Power BI
  • Power Pivot
  • Power Query
microsoft mvp logo
trustpilot excellent rating
Secured by Sucuri Badge
MyOnlineTrainingHub on YouTube Mynda Treacy on Linked In Mynda Treacy on Instagram Mynda Treacy on Twitter Mynda Treacy on Pinterest MyOnlineTrainingHub on Facebook
 

Company

  • About My Online Training Hub
  • Disclosure Statement
  • Frequently Asked Questions
  • Guarantee
  • Privacy Policy
  • Terms & Conditions
  • Testimonials
  • Become an Affiliate

Support

  • Contact
  • Forum
  • Helpdesk - For Technical Issues

Copyright © 2023 · My Online Training Hub · All Rights Reserved. Microsoft and the Microsoft Office logo are trademarks or registered trademarks of Microsoft Corporation in the United States and/or other countries. Product names, logos, brands, and other trademarks featured or referred to within this website are the property of their respective trademark holders.