• Skip to main content
  • Skip to header right navigation
  • Skip to site footer

My Online Training Hub

Learn Dashboards, Excel, Power BI, Power Query, Power Pivot

  • Courses
  • Pricing
    • Free Courses
    • Power BI Course
    • Excel Power Query Course
    • Power Pivot and DAX Course
    • Excel Dashboard Course
    • Excel PivotTable Course – Quick Start
    • Advanced Excel Formulas Course
    • Excel Expert Advanced Excel Training
    • Excel Tables Course
    • Excel, Word, Outlook
    • Financial Modelling Course
    • Excel PivotTable Course
    • Excel for Customer Service Professionals
    • Excel for Operations Management Course
    • Excel for Decision Making Under Uncertainty Course
    • Excel for Finance Course
    • Excel Analysis ToolPak Course
    • Multi-User Pricing
  • Resources
    • Free Downloads
    • Excel Functions Explained
    • Excel Formulas
    • Excel Add-ins
    • IF Function
      • Excel IF Statement Explained
      • Excel IF AND OR Functions
      • IF Formula Builder
    • Time & Dates in Excel
      • Excel Date & Time
      • Calculating Time in Excel
      • Excel Time Calculation Tricks
      • Excel Date and Time Formatting
    • Excel Keyboard Shortcuts
    • Excel Custom Number Format Guide
    • Pivot Tables Guide
    • VLOOKUP Guide
    • ALT Codes
    • Excel VBA & Macros
    • Excel User Forms
    • VBA String Functions
  • Members
    • Login
    • Password Reset
  • Blog
  • Excel Webinars
  • Excel Forum
    • Register as Forum Member

How to speed up refreshing times for my query with many custom columns|Power Query|Excel Forum|My Online Training Hub

You are here: Home / How to speed up refreshing times for my query with many custom columns|Power Query|Excel Forum|My Online Training Hub
Avatar
sp_LogInOut Log In sp_Registration Register
sp_Search Search
Advanced Search|Last Search Results
Search
Forum Scope




Match



Forum Options



Minimum search word length is 3 characters - maximum search word length is 84 characters
sp_Search Search
sp_RankInfo
Lost password?
sp_CrumbsHome HomeExcel ForumPower QueryHow to speed up refreshing times fo…
sp_PrintTopic sp_TopicIcon
How to speed up refreshing times for my query with many custom columns
Avatar
job schouenberg

New Member
Members
Level 0
Forum Posts: 1
Member Since:
February 28, 2023
sp_UserOfflineSmall Offline
1
February 28, 2023 - 1:41 am
sp_Permalink sp_Print

Hello everybody,

Since i have a dataset with millions of rows i started using power query in excel instead of standard excel. Normally I build long formulas in excel (combinations of IF formulas, lookups and index formulas) but because of the big dataset i thought it may be a better idea to build these formulas into power querry. 

But now i have a query with multiple custom columns, that contain my formulas which takes like 30 hours to refresh. Because i am new to powerquery the formulas used in the query are build up like a normal excel formula. Which i think is one of the reasons why it takes so long to refresh the query.

 

Some examples of formulas used for the custom columns

=Text.Replace(Text.Replace(Text.Replace(Text.Replace(Text.Replace([delivery zip], "-", ""), "*", ""), "**", ""), ".", "")," ", "")

=if [delivery country]= "AL" or [delivery country] = "BE" or [delivery country] = "BA" or [delivery country] = "BG" or [delivery country] = "DK" or [delivery country] = "DE" or [delivery country] = "EE" or [delivery country] = "FI" or [delivery country] = "FR" or [delivery country] = "GR" or [delivery country] = "HU" or [delivery country] = "IT" or [delivery country] = "HR" or [delivery country] = "LV" or [delivery country] = "LT" or [delivery country] = "MK" or [delivery country] = "NL" or [delivery country] = "NO" or [delivery country] = "AT" or [delivery country] = "PT" or [delivery country] = "RO" or [delivery country] = "SI" or [delivery country] = "SK" or [delivery country] = "CZ" or [delivery country] = "TR" or [delivery country] = "SE" or [delivery country] = "ES" then Text.Start([Special.char.removed delivery zip],2) else if [delivery country] = "RS" then Text.Start([Special.char.removed delivery zip],3) else if [delivery country] = "AM" then "AM" else if [delivery country] = "GE" then "GE" else if [delivery country] = "KO" then "KO" else if [delivery country] = "LU" then "LU" else if [delivery country] = "SM" then "SM" else if [delivery country] = "MD" then "MD" else "ontbreekt nog"

=if [delivery country]= "PL" then List.PositionOf(Hulpcellen_PL,
Text.Start([Special.char.removed delivery zip],3)) else if [delivery country]= "CH" then List.PositionOf(Hulpcellen_CH,
Text.Start([Special.char.removed delivery zip],3)) else if [delivery country]= "ME" then List.PositionOf(Hulpcellen_ME,
Text.Start([Special.char.removed delivery zip],3)) else -2

=try if [loading country] = "NL" and Value.FromText([Lookup zip],"") >= 10 and Value.FromText([Lookup zip],"") <= 13 or [loading country] = "NL" and Value.FromText([Lookup zip],"") >= 20 and Value.FromText([Lookup zip],"") <= 73 then "zone 1" else if [loading country] = "NL" and Value.FromText([Lookup zip],"") >= 14 and Value.FromText([Lookup zip],"") <= 19 or Value.FromText([Lookup zip],"") >= 74 and Value.FromText([Lookup zip],"") <= 99 then "zone 2" else if [loading country] = "BE" then "zone 2" else -2 otherwise "NB"

=if [PL zip hulp] = -1 then Text.Start([Special.char.removed delivery zip],2) else if [PL zip hulp] >= 0 then Text.Start([Special.char.removed delivery zip],3) else if [delivery country]= "ES" and [Custom] = "zone 2" then [Lookup zip]&"x" else [Lookup zip]

=if Text.Start([#"PL zip."],1) = "0" or Text.Start([#"PL zip."],1) = "1" or Text.Start([#"PL zip."],1) = "2" or Text.Start([#"PL zip."],1) = "3" or Text.Start([#"PL zip."],1) = "4" or Text.Start([#"PL zip."],1) = "5" or Text.Start([#"PL zip."],1) = "6" or Text.Start([#"PL zip."],1) = "7" or Text.Start([#"PL zip."],1) = "8" or Text.Start([#"PL zip."],1) = "9" then [delivery country]&[#"PL zip."] else [#"PL zip."]

=try Boektarief_zones_Export[Zone boektarief]{List.PositionOf(Boektarief_zones_Export[Postcode], [Zip indentification])} otherwise "niet beschikbaar"

Any suggestions about how to improve the refreshing speed of this query or how to write the formulas in a more suitable way for powerquery?

Thanks in advance,

 

I am using windows 10 and excel 365 version 2212.

Avatar
Riny van Eekelen
Örnsköldsvik, Sweden
Moderator
Members


Trusted Members

Moderators

Power BI
Level 0
Forum Posts: 440
Member Since:
January 31, 2022
sp_UserOfflineSmall Offline
2
February 28, 2023 - 5:06 am
sp_Permalink sp_Print

Hi Job,
It's difficult to diagnose such a query without having some data to play with. No need to share thousands of rows, but could you upload a file with a representative sample of your data so that we can see what you are dealing with. Many country codes seem to result in different results. Please include the entire query as well.

sp_Feed
Go to top
Forum Timezone: Australia/Brisbane
Most Users Ever Online: 245
Currently Online:
Guest(s) 10
Currently Browsing this Page:
1 Guest(s)
Top Posters:
SunnyKow: 1432
Anders Sehlstedt: 870
Purfleet: 412
Frans Visser: 346
David_Ng: 306
lea cohen: 219
A.Maurizio: 202
Jessica Stewart: 202
Aye Mu: 201
jaryszek: 183
Newest Members:
Ivica Cvetkovski
Blaine Cox
Shankar Srinivasan
riyepa fdgf
Hannah Cave
Len Matthews
Kristine Arthy
Michelle Neven
Andrew Kuhn
Angela Paul
Forum Stats:
Groups: 3
Forums: 24
Topics: 6206
Posts: 27202

 

Member Stats:
Guest Posters: 49
Members: 31875
Moderators: 3
Admins: 4
Administrators: Mynda Treacy, Philip Treacy, Catalin Bombea, FT
Moderators: MOTH Support, Velouria, Riny van Eekelen
© Simple:Press —sp_Information

Sidebar

Blog Categories

  • Excel
  • Excel Charts
  • Excel Dashboard
  • Excel Formulas
  • Excel PivotTables
  • Excel Shortcuts
  • Excel VBA
  • General Tips
  • Online Training
  • Outlook
  • Power Apps
  • Power Automate
  • Power BI
  • Power Pivot
  • Power Query
microsoft mvp logo
trustpilot excellent rating
Secured by Sucuri Badge
MyOnlineTrainingHub on YouTube Mynda Treacy on Linked In Mynda Treacy on Instagram Mynda Treacy on Twitter Mynda Treacy on Pinterest MyOnlineTrainingHub on Facebook
 

Company

  • About My Online Training Hub
  • Disclosure Statement
  • Frequently Asked Questions
  • Guarantee
  • Privacy Policy
  • Terms & Conditions
  • Testimonials
  • Become an Affiliate

Support

  • Contact
  • Forum
  • Helpdesk - For Technical Issues

Copyright © 2023 · My Online Training Hub · All Rights Reserved. Microsoft and the Microsoft Office logo are trademarks or registered trademarks of Microsoft Corporation in the United States and/or other countries. Product names, logos, brands, and other trademarks featured or referred to within this website are the property of their respective trademark holders.