• Skip to main content
  • Skip to header right navigation
  • Skip to site footer

My Online Training Hub

Learn Dashboards, Excel, Power BI, Power Query, Power Pivot

  • Courses
  • Pricing
    • Free Courses
    • Power BI Course
    • Excel Power Query Course
    • Power Pivot and DAX Course
    • Excel Dashboard Course
    • Excel PivotTable Course – Quick Start
    • Advanced Excel Formulas Course
    • Excel Expert Advanced Excel Training
    • Excel Tables Course
    • Excel, Word, Outlook
    • Financial Modelling Course
    • Excel PivotTable Course
    • Excel for Customer Service Professionals
    • Excel for Operations Management Course
    • Excel for Decision Making Under Uncertainty Course
    • Excel for Finance Course
    • Excel Analysis ToolPak Course
    • Multi-User Pricing
  • Resources
    • Free Downloads
    • Excel Functions Explained
    • Excel Formulas
    • Excel Add-ins
    • IF Function
      • Excel IF Statement Explained
      • Excel IF AND OR Functions
      • IF Formula Builder
    • Time & Dates in Excel
      • Excel Date & Time
      • Calculating Time in Excel
      • Excel Time Calculation Tricks
      • Excel Date and Time Formatting
    • Excel Keyboard Shortcuts
    • Excel Custom Number Format Guide
    • Pivot Tables Guide
    • VLOOKUP Guide
    • ALT Codes
    • Excel VBA & Macros
    • Excel User Forms
    • VBA String Functions
  • Members
    • Login
    • Password Reset
  • Blog
  • Excel Webinars
  • Excel Forum
    • Register as Forum Member

Append and Update issue|Power Query|Excel Forum|My Online Training Hub

You are here: Home / Append and Update issue|Power Query|Excel Forum|My Online Training Hub
Avatar
sp_LogInOut Log In sp_Registration Register
sp_Search Search
Advanced Search|Last Search Results
Search
Forum Scope




Match



Forum Options



Minimum search word length is 3 characters - maximum search word length is 84 characters
sp_Search Search
sp_RankInfo
Lost password?
sp_CrumbsHome HomeExcel ForumPower QueryAppend and Update issue
sp_PrintTopic sp_TopicIcon
Append and Update issue
Avatar
Sue Bristow
Member
Members
Level 0
Forum Posts: 7
Member Since:
April 3, 2020
sp_UserOfflineSmall Offline
1
April 3, 2020 - 8:15 pm
sp_Permalink sp_Print

I want to automate a report that displays information about our leavers during a school year - whether the students are vulnerable, Special needs etc.  Unfortunately our MIS "Hides" this information once a leaving date is entered so a simple leaving report won't work.  The data is still there but there isn't the functionality within the reporting system to get to it.  Changing the way the MIS works is not an option.

My plan was to:

Export a file dated on the first day of September to include all students on roll on that date to get all of the vulnerable data in a base table including the unique number of each student

Export a file every night with all students on roll and append to the base data table to give one "All Students" table with all information about every student who has been on roll during the academic year.

Export a file every night with all leavers since the start of the year including their unique number.

I could then merge the leavers table with the required columns from the All Students table to allow the completion of the report.

Issues that I can see but don't know how to solve:

Appending the daily student roll will keep duplicating the students, I only want to append the student if they are not already in the All Students table.

The vulnerable information does change throughout the year so I would like to update the All Students table with some of the data from the over night table.

Am I asking too much from power query?

Many thanks in advance.

sp_AnswersTopicSeeAnswer See Answer
Avatar
Mynda Treacy
Admin
Level 10
Forum Posts: 4450
Member Since:
July 16, 2010
sp_UserOfflineSmall Offline
2
April 5, 2020 - 10:07 am
sp_Permalink sp_Print

Hi Sue,

Welcome to our forum!

You can remove duplicates from your dataset in Power Query. However, you will no doubt want to choose which records you keep, i.e. the latest record or the original record etc., and this will introduce more complexity.

One issue I foresee is Power Query for Excel can't do an incremental refresh i.e. every refresh will get ALL data again, and if you're creating a new file every day, that's a lot of files and it will end up being very slow to update.

Power Query in Power BI has got incremental refresh, so you might like to consider using that instead of Excel.

I hope that points you in the right direction.

Mynda

Avatar
Sue Bristow
Member
Members
Level 0
Forum Posts: 7
Member Since:
April 3, 2020
sp_UserOfflineSmall Offline
3
April 6, 2020 - 6:05 pm
sp_Permalink sp_Print

Hi Mynda

I had thought about the incremental refresh issue after I posted.  I don't yet have Power BI and as a school we cannot afford the power BI licence for all the people that will use the report.  I know it is the way to go but until the pricing comes down I think I am stuck!

Thanks anyway.

Sue

Avatar
Sue Bristow
Member
Members
Level 0
Forum Posts: 7
Member Since:
April 3, 2020
sp_UserOfflineSmall Offline
4
April 6, 2020 - 6:22 pm
sp_Permalink sp_Print

Just a thought

We will be supplied with one Power BI licence per school by the powers above.  It is likely that I will get that licence for checking data that they send back to us.  

Could I use the Power BI licence to get the data file updated with an incremental append and then read that file in power pivot / power query?

Avatar
Mynda Treacy
Admin
Level 10
Forum Posts: 4450
Member Since:
July 16, 2010
sp_UserOfflineSmall Offline
5
April 6, 2020 - 9:09 pm
sp_Permalink sp_Print

Hi Sue,

You could use Power BI to get the data and model it with Power Pivot. Then you can 'analyze in Excel' which gives you the data model inside of Excel, but not the raw data. I'm sure that will work for your needs though, but it will mean also learning Power Pivot.

Mynda

sp_AnswersTopicAnswer
Answers Post
Avatar
Sue Bristow
Member
Members
Level 0
Forum Posts: 7
Member Since:
April 3, 2020
sp_UserOfflineSmall Offline
6
April 7, 2020 - 7:44 pm
sp_Permalink sp_Print

Thank you - As I don't have access to Power BI yet I am doing everything in power pivot and power chart so that is really good news.

 

Thank you

 

Sue

sp_Feed
Go to top
Forum Timezone: Australia/Brisbane
Most Users Ever Online: 245
Currently Online:
Guest(s) 8
Currently Browsing this Page:
1 Guest(s)
Top Posters:
SunnyKow: 1432
Anders Sehlstedt: 871
Purfleet: 412
Frans Visser: 346
David_Ng: 306
lea cohen: 219
Jessica Stewart: 205
A.Maurizio: 202
Aye Mu: 201
jaryszek: 183
Newest Members:
stuart burge
Bruce Tang Nian
Scot C
Othman AL MUTAIRI
Misael Gutierrez Sr.
Attif Ihsan
Kieran Fee
Murat Hasanoglu
Brett Dryland
Saeed Aldousari
Forum Stats:
Groups: 3
Forums: 24
Topics: 6223
Posts: 27295

 

Member Stats:
Guest Posters: 49
Members: 31920
Moderators: 3
Admins: 4
Administrators: Mynda Treacy, Philip Treacy, Catalin Bombea, FT
Moderators: MOTH Support, Velouria, Riny van Eekelen
© Simple:Press —sp_Information

Sidebar

Blog Categories

  • Excel
  • Excel Charts
  • Excel Dashboard
  • Excel Formulas
  • Excel PivotTables
  • Excel Shortcuts
  • Excel VBA
  • General Tips
  • Online Training
  • Outlook
  • Power Apps
  • Power Automate
  • Power BI
  • Power Pivot
  • Power Query
microsoft mvp logo
trustpilot excellent rating
Secured by Sucuri Badge
MyOnlineTrainingHub on YouTube Mynda Treacy on Linked In Mynda Treacy on Instagram Mynda Treacy on Twitter Mynda Treacy on Pinterest MyOnlineTrainingHub on Facebook
 

Company

  • About My Online Training Hub
  • Disclosure Statement
  • Frequently Asked Questions
  • Guarantee
  • Privacy Policy
  • Terms & Conditions
  • Testimonials
  • Become an Affiliate

Support

  • Contact
  • Forum
  • Helpdesk - For Technical Issues

Copyright © 2023 · My Online Training Hub · All Rights Reserved. Microsoft and the Microsoft Office logo are trademarks or registered trademarks of Microsoft Corporation in the United States and/or other countries. Product names, logos, brands, and other trademarks featured or referred to within this website are the property of their respective trademark holders.