• Skip to main content
  • Skip to header right navigation
  • Skip to site footer

My Online Training Hub

Learn Dashboards, Excel, Power BI, Power Query, Power Pivot

  • Courses
  • Pricing
    • Free Courses
    • Power BI Course
    • Excel Power Query Course
    • Power Pivot and DAX Course
    • Excel Dashboard Course
    • Excel PivotTable Course – Quick Start
    • Advanced Excel Formulas Course
    • Excel Expert Advanced Excel Training
    • Excel Tables Course
    • Excel, Word, Outlook
    • Financial Modelling Course
    • Excel PivotTable Course
    • Excel for Customer Service Professionals
    • Excel for Operations Management Course
    • Excel for Decision Making Under Uncertainty Course
    • Excel for Finance Course
    • Excel Analysis ToolPak Course
    • Multi-User Pricing
  • Resources
    • Free Downloads
    • Excel Functions Explained
    • Excel Formulas
    • Excel Add-ins
    • IF Function
      • Excel IF Statement Explained
      • Excel IF AND OR Functions
      • IF Formula Builder
    • Time & Dates in Excel
      • Excel Date & Time
      • Calculating Time in Excel
      • Excel Time Calculation Tricks
      • Excel Date and Time Formatting
    • Excel Keyboard Shortcuts
    • Excel Custom Number Format Guide
    • Pivot Tables Guide
    • VLOOKUP Guide
    • ALT Codes
    • Excel VBA & Macros
    • Excel User Forms
    • VBA String Functions
  • Members
    • Login
    • Password Reset
  • Blog
  • Excel Webinars
  • Excel Forum
    • Register as Forum Member

Incremental data loads or alternative|Power Query|Excel Forum|My Online Training Hub

You are here: Home / Incremental data loads or alternative|Power Query|Excel Forum|My Online Training Hub
Avatar
sp_LogInOut Log In sp_Registration Register
sp_Search Search
Advanced Search|Last Search Results
Search
Forum Scope




Match



Forum Options



Minimum search word length is 3 characters - maximum search word length is 84 characters
sp_Search Search
sp_RankInfo
Lost password?
sp_CrumbsHome HomeExcel ForumPower QueryIncremental data loads or alternati…
sp_PrintTopic sp_TopicIcon
Incremental data loads or alternative
Avatar
John Murphy
Member
Members
Level 0
Forum Posts: 6
Member Since:
January 28, 2021
sp_UserOfflineSmall Offline
1
January 28, 2021 - 2:17 am
sp_Permalink sp_Print

Hi all,

Self taught user here so please excuse if I am using alternative or inefficient methods - willing to learn if you can help

I use Power Query to gather management information, which is then presented (currently via Excel but hopefully soon with Power BI). 

The information I extract uses multiple queries, and looks back over 365 days, typically refreshed weekly.  The problems this gives me, is that each week, the data is loading (today-365) days of data which slows the loads down, and that the historical data is not held.  Is there a way to tell power query to refresh only the most recent data, and to append this to the previous results?  

The alternative I am considering is currently getting power query to refresh weekly, using VBA to push that data into MS Access, and then reading from the access tables in order to present in Excel, which seems a bit backwards.

Any help would be appreciated.

Thanks

John

Avatar
Mynda Treacy
Admin
Level 10
Forum Posts: 4443
Member Since:
July 16, 2010
sp_UserOfflineSmall Offline
2
January 28, 2021 - 10:43 am
sp_Permalink sp_Print

Hi John,

Welcome to our forum!

Power Query in Excel does not have incremental refresh, but Power BI does. When you move to Power BI you'll be able to avoid refreshing ALL data each week.

Mynda

Avatar
John Murphy
Member
Members
Level 0
Forum Posts: 6
Member Since:
January 28, 2021
sp_UserOfflineSmall Offline
3
January 28, 2021 - 9:03 pm
sp_Permalink sp_Print

Hi Mynda,

 

Thanks for your quick reply, can you confirm if this will also allow me to store the historic data that I need?

 

I'm uncertain as/when we will be able to use PowerBI, is there an alternative way to do this using Power Query at the moment?

Avatar
Mynda Treacy
Admin
Level 10
Forum Posts: 4443
Member Since:
July 16, 2010
sp_UserOfflineSmall Offline
4
January 29, 2021 - 2:10 pm
sp_Permalink sp_Print

Hi John,

The incremental refresh only gets the data you connect the query to. When you refresh a query it doesn't get that same data again, instead it only gets new data. If you remove the connection to the original data then it will be removed from your model. e.g.

If you get data from a folder containing 5 files numbered 1 to 5, and then next month you add another file numbered 6, the incremental refresh will only get file number 6 as it already has the data from files 1 through 5 from when you built the query. If you later remove files 1 through 5 from that folder, and then refresh the query, it will remove the data for files 1 through 5 from the model.

There's no way to perform an incremental refresh in Excel using Power Query. If your final load location for the data is Power Pivot, then you can load that data to Power Pivot directly, thus avoiding Power Query and any refresh performance issues.

Mynda

Avatar
jim hubbard

Active Member
Members
Level 0
Forum Posts: 3
Member Since:
April 10, 2022
sp_UserOfflineSmall Offline
5
April 10, 2022 - 6:54 am
sp_Permalink sp_Print

What about trying this -->  

Will that do incremental refreshing for Excel?

Avatar
Mynda Treacy
Admin
Level 10
Forum Posts: 4443
Member Since:
July 16, 2010
sp_UserOfflineSmall Offline
6
April 10, 2022 - 12:17 pm
sp_Permalink sp_Print

Hi Jim,

That's not incremental refresh in the true sense. The purpose of incremental refresh is to prevent Power Query getting all the data every time you refresh. With this technique it is still getting all the data again, albeit from another table as opposed to other files. 

Mynda

sp_Feed
Go to top
Forum Timezone: Australia/Brisbane
Most Users Ever Online: 245
Currently Online: Brian Pham, Chandler Davis, Valentyn Kristioglo
Guest(s) 8
Currently Browsing this Page:
1 Guest(s)
Top Posters:
SunnyKow: 1432
Anders Sehlstedt: 870
Purfleet: 412
Frans Visser: 346
David_Ng: 306
lea cohen: 219
A.Maurizio: 202
Jessica Stewart: 202
Aye Mu: 201
jaryszek: 183
Newest Members:
drsven
Annie Witbrod
wahab tunde
Cong Le Duc
Faisal Bashir
Ivica Cvetkovski
Blaine Cox
Shankar Srinivasan
riyepa fdgf
Hannah Cave
Forum Stats:
Groups: 3
Forums: 24
Topics: 6205
Posts: 27212

 

Member Stats:
Guest Posters: 49
Members: 31880
Moderators: 3
Admins: 4
Administrators: Mynda Treacy, Philip Treacy, Catalin Bombea, FT
Moderators: MOTH Support, Velouria, Riny van Eekelen
© Simple:Press —sp_Information

Sidebar

Blog Categories

  • Excel
  • Excel Charts
  • Excel Dashboard
  • Excel Formulas
  • Excel PivotTables
  • Excel Shortcuts
  • Excel VBA
  • General Tips
  • Online Training
  • Outlook
  • Power Apps
  • Power Automate
  • Power BI
  • Power Pivot
  • Power Query
microsoft mvp logo
trustpilot excellent rating
Secured by Sucuri Badge
MyOnlineTrainingHub on YouTube Mynda Treacy on Linked In Mynda Treacy on Instagram Mynda Treacy on Twitter Mynda Treacy on Pinterest MyOnlineTrainingHub on Facebook
 

Company

  • About My Online Training Hub
  • Disclosure Statement
  • Frequently Asked Questions
  • Guarantee
  • Privacy Policy
  • Terms & Conditions
  • Testimonials
  • Become an Affiliate

Support

  • Contact
  • Forum
  • Helpdesk - For Technical Issues

Copyright © 2023 · My Online Training Hub · All Rights Reserved. Microsoft and the Microsoft Office logo are trademarks or registered trademarks of Microsoft Corporation in the United States and/or other countries. Product names, logos, brands, and other trademarks featured or referred to within this website are the property of their respective trademark holders.