• Skip to main content
  • Skip to header right navigation
  • Skip to site footer

My Online Training Hub

Learn Dashboards, Excel, Power BI, Power Query, Power Pivot

  • Courses
  • Pricing
    • Free Courses
    • Power BI Course
    • Excel Power Query Course
    • Power Pivot and DAX Course
    • Excel Dashboard Course
    • Excel PivotTable Course – Quick Start
    • Advanced Excel Formulas Course
    • Excel Expert Advanced Excel Training
    • Excel Tables Course
    • Excel, Word, Outlook
    • Financial Modelling Course
    • Excel PivotTable Course
    • Excel for Customer Service Professionals
    • Excel for Operations Management Course
    • Excel for Decision Making Under Uncertainty Course
    • Excel for Finance Course
    • Excel Analysis ToolPak Course
    • Multi-User Pricing
  • Resources
    • Free Downloads
    • Excel Functions Explained
    • Excel Formulas
    • Excel Add-ins
    • IF Function
      • Excel IF Statement Explained
      • Excel IF AND OR Functions
      • IF Formula Builder
    • Time & Dates in Excel
      • Excel Date & Time
      • Calculating Time in Excel
      • Excel Time Calculation Tricks
      • Excel Date and Time Formatting
    • Excel Keyboard Shortcuts
    • Excel Custom Number Format Guide
    • Pivot Tables Guide
    • VLOOKUP Guide
    • ALT Codes
    • Excel VBA & Macros
    • Excel User Forms
    • VBA String Functions
  • Members
    • Login
    • Password Reset
  • Blog
  • Excel Webinars
  • Excel Forum
    • Register as Forum Member

Extracting tables and another data from folder path from multiple workbooks|Power Query|Excel Forum|My Online Training Hub

You are here: Home / Extracting tables and another data from folder path from multiple workbooks|Power Query|Excel Forum|My Online Training Hub
Avatar
sp_LogInOut Log In sp_Registration Register
sp_Search Search
Advanced Search|Last Search Results
Search
Forum Scope




Match



Forum Options



Minimum search word length is 3 characters - maximum search word length is 84 characters
sp_Search Search
sp_RankInfo
Lost password?
sp_CrumbsHome HomeExcel ForumPower QueryExtracting tables and another data …
sp_PrintTopic sp_TopicIcon
Extracting tables and another data from folder path from multiple workbooks
Avatar
jaryszek
Member
Members
Level 0
Forum Posts: 183
Member Since:
February 1, 2019
sp_UserOfflineSmall Offline
1
December 30, 2019 - 4:38 pm
sp_Permalink sp_Print sp_EditHistory

Hi Guys,

I have folder and in this folder i have multiple the same structure workbooks with the same tables which i want to merge into one big summary table.Screenshot_22.pngImage Enlarger

In first tab i have tableToMerge, in worksheet Decription there is one cell named range "Description" with string inside for example "Workbook 1 User" and i have also mapping table where i have IndexTopo Column mapped for each Topology from TableToMerge worksheet. 

What i want to get is summary workbook like here:

Screenshot_24.pngImage Enlarger

On red i marked columns which i want to add with PQ. 

So from Summary workbook i want to:

1. Build PQ to loop through multiple workbooks in folder path provided (worksheet "FolderPathParameter")

2. Merge all tables t_MergeTable from worksheets tabletoMerge from each workbook

3. Provide also workbook path, description and IndexTopo from mapping table into Summary workbook.

I am attaching example files.

Please help,
Thank you in advance,
Best,
Jacek

sp_PlupAttachments Attachments
  • sp_PlupImage Screenshot_22-1.png (272 KB)
  • sp_PlupImage Screenshot_24.png (316 KB)
sp_AnswersTopicSeeAnswer See Answer
Avatar
Guenter Weber
Graz, Austria
Member
Members
Level 0
Forum Posts: 8
Member Since:
July 24, 2019
sp_UserOfflineSmall Offline
2
January 3, 2020 - 7:39 am
sp_Permalink sp_Print

You can find my solution in the attachment.

I used functions to

- read your parameter "File Path" 

- read Table1 on sheet TableToMerge of each file and

- read Named Range "Description" on sheet "Description" of each file

to get the result

best regards,

Guenter

sp_AnswersTopicAnswer
Answers Post
Avatar
jaryszek
Member
Members
Level 0
Forum Posts: 183
Member Since:
February 1, 2019
sp_UserOfflineSmall Offline
3
January 3, 2020 - 5:30 pm
sp_Permalink sp_Print

Thank you very much!

Your solution is very nice ! But i have to analyze it more and i will come back with questions.

Best,
Jacek

Avatar
jaryszek
Member
Members
Level 0
Forum Posts: 183
Member Since:
February 1, 2019
sp_UserOfflineSmall Offline
4
January 6, 2020 - 8:25 pm
sp_Permalink sp_Print sp_EditHistory

Hi Guenter Weber,

i analyzed your queries and they are awesome. 

I have one more question for you.
I want to add function which will be retriving range from worksheet.

What i achieved is writing standard query:

let
DynamicSource = "C:\Users\admin\Desktop\Newest Pull request\Vnomic-Generic-Azure-Master\PQ Example Cost Summary\Master\Sample.xlsm",
Source = Excel.Workbook(File.Contents(DynamicSource), null, true),
#"Filtered Rows" = Table.SelectRows(Source, each ([Name] = "Deployment Costs")),
#"Expanded Data" = Table.ExpandTableColumn(#"Filtered Rows", "Data", {"Column1", "Column2", "Column3", "Column4", "Column5", "Column6", "Column7"}, {"Column1", "Column2", "Column3", "Column4", "Column5", "Column6", "Column7"})
in
#"Expanded Data"

How can i write function to read from specific worksheet name from specific workbook from folder?
I do not know why PQ treats this sheets as table - there is also formatted range in a shape of table but there is no table object, just data. 

I am attaching workbooks with sample worksheet as range "Deployment Costs".

To sum up:

I want to create a function fnGetWorksheet which will be get parameters like Folder path & workbook name, worksheetName to read data from worksheet from specific workbook and folder.

Please help,
Jacek

Avatar
Guenter Weber
Graz, Austria
Member
Members
Level 0
Forum Posts: 8
Member Since:
July 24, 2019
sp_UserOfflineSmall Offline
5
January 8, 2020 - 12:34 am
sp_Permalink sp_Print

Hi Jacek,

only a small modification is required. Then you can use it as a function (as long neither worksheet name and number of columns are changing!)

---------------------------

(DynamicSource as text) =>

let
  // DynamicSource = "C:\Users\admin\Desktop\Newest Pull request\Vnomic-Generic-Azure-Master\PQ Example Cost",

  Source = Excel.Workbook(File.Contents(DynamicSource), null, true),
  #"Filtered Rows" = Table.SelectRows(Source, each ([Name] = "Deployment Costs")),
  #"Expanded Data" = Table.ExpandTableColumn(#"Filtered Rows", "Data", {"Column1", "Column2", "Column3", "Column4", "Column5", "Column6", "Column7"}, {"Column1", "Column2", "Column3", "Column4", "Column5", "Column6", "Column7"})
in
  #"Expanded Data"

-------------------------

if you also want to use different worksheet names, then change

(DynamicSource as text) =>  to (DynamicSource as text, WorksheetName as text) =>

and ...each ([Name] = "Deployment Costs")), to ... each ([Name] = WorksheetName)),

 

but be aware that your column names or number of columns may also change. You can overcome this by using here also a dynamic aproach for column names and numbers.

regards,

guenter 

Avatar
jaryszek
Member
Members
Level 0
Forum Posts: 183
Member Since:
February 1, 2019
sp_UserOfflineSmall Offline
6
January 8, 2020 - 5:59 pm
sp_Permalink sp_Print

Thank you very much! 

O wow this is working like a charm.

You can overcome this by using here also a dynamic aproach for column names and numbers.

How can i do number of columns dynamic when range change? Something like used range? 

Best Wishes,
Jacek

Avatar
jaryszek
Member
Members
Level 0
Forum Posts: 183
Member Since:
February 1, 2019
sp_UserOfflineSmall Offline
7
January 8, 2020 - 9:03 pm
sp_Permalink sp_Print sp_EditHistory

And one more issue i have with your poqwer query:

#"Added TopoIndex" = Table.AddColumn(#"Removed Columns1", "TopoIndex", each "Index" & [Topology]),

Here you are avoiding merging tables but i have to do this. In sample there is easy workaround like you did but whole purpose is to merge tables. 
But when i am mergind them instead of writing workaround - i am getting :

Formula.Firewall: Query (step 'Import each TableToMerge') references other queries or steps, so it may not directly access a data source. Please rebuild this data combination.

I can not definitely workaround this because in merged tables i can have a lot of other information which i can not put in the functions...

How can i avoid this error?

In attachement sample workbook.

Best,
Jacek

Avatar
Guenter Weber
Graz, Austria
Member
Members
Level 0
Forum Posts: 8
Member Since:
July 24, 2019
sp_UserOfflineSmall Offline
8
January 10, 2020 - 4:13 am
sp_Permalink sp_Print

Hi Jacek,

ad 6) as long as your tables look the same you can directly use the column names because they will not change (I assume) and therefore also the number is always the same. With this direct approach you are also able to modify some data directly.

If you use a more generic approach you may not know now how data manipulation will look like and therefore must be done outside of this function.

As every time it is easier if you have a specific example than talking very generic. 

 

ad 7)

Here you are avoiding merging tables but i have to do this. In sample there is easy workaround like you did but whole purpose is to merge tables. 

This I do not understand. What do you mean with workaround?

What is your expected result? Can you put a sample in you summary file?

Please also give a sample for the final table of the deployment costs (I think you also have something in mind because of your trial and question about your query change to a function)

For the Firewall problem: I do not know which tables you want to merge and I do not get this error. There are a lot of explanations about root causes and solutions on the internet.

I can not definitely workaround this because in merged tables i can have a lot of other information which i can not put in the functions...

too generic for me. Give me some more information/examples, then I may be able to help you. 

 cheers,

Guenter

Avatar
jaryszek
Member
Members
Level 0
Forum Posts: 183
Member Since:
February 1, 2019
sp_UserOfflineSmall Offline
9
January 13, 2020 - 5:45 pm
sp_Permalink sp_Print

Hi Guenter,

thank you.

Let's focuc first on ad 7)

This I do not understand. What do you mean with workaround?

You added custom columns with formula and i need just do Merge queries (left join) in order to combine tables.

What i need to get is table like here:

Screenshot_1.pngImage Enlarger

So in source workbooks i have only 2 tables. 
And in result table i have to join them. 

let
pFolder=fnGetParameter("File Path"),
Source = Folder.Files(pFolder),
#"Filtered Rows" = Table.SelectRows(Source, each not Text.StartsWith([Name], "~") and Text.StartsWith([Extension], ".xls")),
#"Import each TableToMerge" = Table.AddColumn(#"Filtered Rows", "Imported", each fnSingleFileImport([Folder Path] & [Name], "t_MergeTable", "Table")),
#"Removed Columns" = Table.RemoveColumns(#"Import each TableToMerge",{"Content", "Extension", "Date accessed", "Date modified", "Date created", "Attributes"}),
#"Expanded Imported" = Table.ExpandTableColumn(#"Removed Columns", "Imported", {"Topology", "Server"}, {"Topology", "Server"}),
#"Removed Columns1" = Table.RemoveColumns(#"Expanded Imported",{"Name", "Folder Path"}),
#"Merged Queries" = Table.NestedJoin(#"Removed Columns1", {"Topology"}, t_MappingTable, {"Imported.Topology"}, "t_MappingTable", JoinKind.LeftOuter)
in
#"Merged Queries"

In step #Merged Queries i am getting this firewall error. 
What will be the solution in this case? 

I know that there is a lot solutions but i can not refer them to our specific example. 

What did you do? Just turn of privacy setting in Excel? 
Maybe there is another work around?

Best,
Jacek

sp_PlupAttachments Attachments
  • sp_PlupImage Screenshot_1.png (389 KB)
Avatar
Guenter Weber
Graz, Austria
Member
Members
Level 0
Forum Posts: 8
Member Since:
July 24, 2019
sp_UserOfflineSmall Offline
10
January 16, 2020 - 8:28 am
sp_Permalink sp_Print

I used your example files and may be the info there was too simple. in your latest attachment I can only see a result without source (sample files).

Firewall: Up to now (I have no "external" sources) I only had this problem when I copied a workbook to another computer. I always turned off the privacy settings . What you can do is that you rename/remove this function and redo it step by step by yourself (same code) but I do not know if the really helps. Sometimes it worked for me.

I also read that sometimes this error is dependend on the version of Excel (build) and Power Query.  YOu only can try all solutions you find in the WEB.

Avatar
jaryszek
Member
Members
Level 0
Forum Posts: 183
Member Since:
February 1, 2019
sp_UserOfflineSmall Offline
11
January 16, 2020 - 4:53 pm
sp_Permalink sp_Print

Thank you Guenter,

I only had this problem when I copied a workbook to another computer

Exactly. When i sharing file in somebody else computer there can be an issue. 

I will create separate topic about this. 

Thank you for your help and support,
Best wishes,
Jacek

sp_Feed
Go to top
Forum Timezone: Australia/Brisbane
Most Users Ever Online: 245
Currently Online: Kim Knox, Jeanine Hagge, Nada Perovic
Guest(s) 10
Currently Browsing this Page:
1 Guest(s)
Top Posters:
SunnyKow: 1432
Anders Sehlstedt: 871
Purfleet: 412
Frans Visser: 346
David_Ng: 306
lea cohen: 219
Jessica Stewart: 205
A.Maurizio: 202
Aye Mu: 201
jaryszek: 183
Newest Members:
Othman AL MUTAIRI
Misael Gutierrez Sr.
Attif Ihsan
Kieran Fee
Murat Hasanoglu
Brett Dryland
Saeed Aldousari
Bhuwan Devkota
Kathryn Patton
Maria Conatser
Forum Stats:
Groups: 3
Forums: 24
Topics: 6222
Posts: 27293

 

Member Stats:
Guest Posters: 49
Members: 31916
Moderators: 3
Admins: 4
Administrators: Mynda Treacy, Philip Treacy, Catalin Bombea, FT
Moderators: MOTH Support, Velouria, Riny van Eekelen
© Simple:Press —sp_Information

Sidebar

Blog Categories

  • Excel
  • Excel Charts
  • Excel Dashboard
  • Excel Formulas
  • Excel PivotTables
  • Excel Shortcuts
  • Excel VBA
  • General Tips
  • Online Training
  • Outlook
  • Power Apps
  • Power Automate
  • Power BI
  • Power Pivot
  • Power Query
microsoft mvp logo
trustpilot excellent rating
Secured by Sucuri Badge
MyOnlineTrainingHub on YouTube Mynda Treacy on Linked In Mynda Treacy on Instagram Mynda Treacy on Twitter Mynda Treacy on Pinterest MyOnlineTrainingHub on Facebook
 

Company

  • About My Online Training Hub
  • Disclosure Statement
  • Frequently Asked Questions
  • Guarantee
  • Privacy Policy
  • Terms & Conditions
  • Testimonials
  • Become an Affiliate

Support

  • Contact
  • Forum
  • Helpdesk - For Technical Issues

Copyright © 2023 · My Online Training Hub · All Rights Reserved. Microsoft and the Microsoft Office logo are trademarks or registered trademarks of Microsoft Corporation in the United States and/or other countries. Product names, logos, brands, and other trademarks featured or referred to within this website are the property of their respective trademark holders.