• Skip to main content
  • Skip to header right navigation
  • Skip to site footer

My Online Training Hub

Learn Dashboards, Excel, Power BI, Power Query, Power Pivot

  • Courses
  • Pricing
    • Free Courses
    • Power BI Course
    • Excel Power Query Course
    • Power Pivot and DAX Course
    • Excel Dashboard Course
    • Excel PivotTable Course – Quick Start
    • Advanced Excel Formulas Course
    • Excel Expert Advanced Excel Training
    • Excel Tables Course
    • Excel, Word, Outlook
    • Financial Modelling Course
    • Excel PivotTable Course
    • Excel for Customer Service Professionals
    • Excel for Operations Management Course
    • Excel for Decision Making Under Uncertainty Course
    • Excel for Finance Course
    • Excel Analysis ToolPak Course
    • Multi-User Pricing
  • Resources
    • Free Downloads
    • Excel Functions Explained
    • Excel Formulas
    • Excel Add-ins
    • IF Function
      • Excel IF Statement Explained
      • Excel IF AND OR Functions
      • IF Formula Builder
    • Time & Dates in Excel
      • Excel Date & Time
      • Calculating Time in Excel
      • Excel Time Calculation Tricks
      • Excel Date and Time Formatting
    • Excel Keyboard Shortcuts
    • Excel Custom Number Format Guide
    • Pivot Tables Guide
    • VLOOKUP Guide
    • ALT Codes
    • Excel VBA & Macros
    • Excel User Forms
    • VBA String Functions
  • Members
    • Login
  • Blog
  • Excel Webinars
  • Excel Forum
    • Register as Forum Member
  • Login

Power Query Performance Issues|Power Query|Excel Forum|My Online Training Hub

You are here: Home / Power Query Performance Issues|Power Query|Excel Forum|My Online Training Hub
Avatar
sp_LogInOut Log In sp_Registration Register
sp_Search Search
Advanced Search
Search
Forum Scope




Match



Forum Options



Minimum search word length is 3 characters - maximum search word length is 84 characters
sp_Search Search
sp_RankInfo
Lost password?
sp_CrumbsHome HomeExcel ForumPower QueryPower Query Performance Issues
sp_PrintTopic sp_TopicIcon
Power Query Performance Issues
Avatar
Alan Jones

New Member
Members
Level 0
Forum Posts: 1
Member Since:
January 6, 2021
sp_UserOfflineSmall Offline
1
January 7, 2021 - 4:57 am
sp_Permalink sp_Print

I have query that pulls data from an Excel sheet and transforms to grab a few columns and a max of 2 rows (email and street address).  Then I converted this process to a custom function.  I then created a new query to run this function on almost every sheet in the Excel file (around 300). The query runs and is slower than I would think it should be. However that is not the issue. The odd thing is I then built a  new query that references (uses the first query as a source) and does some minor changes to the output.  Then I did another referencing the new query. 

So each query builds off the first.

Original query on 300 sheets <- new query for minor cleanup <- new query with extra column <- 3 new query that each only show a subset and trans form

I could see my first query being slow but the other queries are built off next so I am confused. In watching them load in the Queries and Connections box (Excel) it is almost like each query is reloading/transforming what the previous query did.

In searching and reading I could not find much that seemed to help. I played some with the queries but nothing helped. I did try using Table.Buffer

    BufferedSource = Table.Buffer(Source)

That did not seem to work. I even read that putting it in the wrong query can hurt not help. So even played with moving it from the first to the 2nd query. Still no luck.

 

Anyone have any thoughts or ideas?

 

Thanks

Alan

Avatar
Catalin Bombea
Iasi, Romania
Admin
Level 10
Forum Posts: 1844
Member Since:
November 8, 2013
sp_UserOfflineSmall Offline
2
January 7, 2021 - 1:51 pm
sp_Permalink sp_Print sp_EditHistory

Hi Alan,

The query results are not stored in memory, dependent queries will run again the initial query to load data.
Try loading the first query to sheet, then for the second query, load the table produced by the first query, instead of referencing the query, will be much faster.

Or, try to use Table Buffer to load into memory the entire list of data tables from sheets, just before adding the new column to transform them.

sp_Feed
Go to top
Forum Timezone: Australia/Brisbane
Most Users Ever Online: 245
Currently Online: Susan VanRiper, Pedro Llobell, Lorna Henning, Joaquim Amorim, Shoua Lee
Guest(s) 10
Top Posters:
SunnyKow: 1432
Anders Sehlstedt: 880
Purfleet: 414
Frans Visser: 346
David_Ng: 306
lea cohen: 237
Jessica Stewart: 219
A.Maurizio: 213
Aye Mu: 201
jaryszek: 183
Newest Members:
David von Kleek
Ronald White
Ginette Guevremont
Taryn Ambrosi
Mark Davenport
Christy Nichols
Harald Endres
Ashley Hughes
Herbie Key
Trevor Pindling
Forum Stats:
Groups: 3
Forums: 24
Topics: 6528
Posts: 28594

 

Member Stats:
Guest Posters: 49
Members: 32817
Moderators: 2
Admins: 4
Administrators: Mynda Treacy, Philip Treacy, Catalin Bombea, FT
Moderators: Velouria, Riny van Eekelen
© Simple:Press —sp_Information

Sidebar

Blog Categories

  • Excel
  • Excel Charts
  • Excel Dashboard
  • Excel Formulas
  • Excel Office Scripts
  • Excel PivotTables
  • Excel Shortcuts
  • Excel VBA
  • General Tips
  • Online Training
  • Outlook
  • Power Apps
  • Power Automate
  • Power BI
  • Power Pivot
  • Power Query
microsoft mvp logo
trustpilot excellent rating
Secured by Sucuri Badge
MyOnlineTrainingHub on YouTube Mynda Treacy on Linked In Mynda Treacy on Instagram Mynda Treacy on Twitter Mynda Treacy on Pinterest MyOnlineTrainingHub on Facebook

Sign up to our newsletter and join over 400,000
others who learn Excel and Power BI with us.

 

Company

  • About My Online Training Hub
  • Disclosure Statement
  • Frequently Asked Questions
  • Guarantee
  • Privacy Policy
  • Terms & Conditions
  • Testimonials
  • Become an Affiliate
  • Sponsor Our Newsletter

Support

  • Contact
  • Forum
  • Helpdesk - For Technical Issues

Copyright © 2023 · My Online Training Hub · All Rights Reserved. Microsoft and the Microsoft Office logo are trademarks or registered trademarks of Microsoft Corporation in the United States and/or other countries. Product names, logos, brands, and other trademarks featured or referred to within this website are the property of their respective trademark holders.