I want to automate a report that displays information about our leavers during a school year - whether the students are vulnerable, Special needs etc. Unfortunately our MIS "Hides" this information once a leaving date is entered so a simple leaving report won't work. The data is still there but there isn't the functionality within the reporting system to get to it. Changing the way the MIS works is not an option.
My plan was to:
Export a file dated on the first day of September to include all students on roll on that date to get all of the vulnerable data in a base table including the unique number of each student
Export a file every night with all students on roll and append to the base data table to give one "All Students" table with all information about every student who has been on roll during the academic year.
Export a file every night with all leavers since the start of the year including their unique number.
I could then merge the leavers table with the required columns from the All Students table to allow the completion of the report.
Issues that I can see but don't know how to solve:
Appending the daily student roll will keep duplicating the students, I only want to append the student if they are not already in the All Students table.
The vulnerable information does change throughout the year so I would like to update the All Students table with some of the data from the over night table.
Am I asking too much from power query?
Many thanks in advance.
Hi Sue,
Welcome to our forum!
You can remove duplicates from your dataset in Power Query. However, you will no doubt want to choose which records you keep, i.e. the latest record or the original record etc., and this will introduce more complexity.
One issue I foresee is Power Query for Excel can't do an incremental refresh i.e. every refresh will get ALL data again, and if you're creating a new file every day, that's a lot of files and it will end up being very slow to update.
Power Query in Power BI has got incremental refresh, so you might like to consider using that instead of Excel.
I hope that points you in the right direction.
Mynda
Hi Mynda
I had thought about the incremental refresh issue after I posted. I don't yet have Power BI and as a school we cannot afford the power BI licence for all the people that will use the report. I know it is the way to go but until the pricing comes down I think I am stuck!
Thanks anyway.
Sue
Just a thought
We will be supplied with one Power BI licence per school by the powers above. It is likely that I will get that licence for checking data that they send back to us.
Could I use the Power BI licence to get the data file updated with an incremental append and then read that file in power pivot / power query?
Hi Sue,
You could use Power BI to get the data and model it with Power Pivot. Then you can 'analyze in Excel' which gives you the data model inside of Excel, but not the raw data. I'm sure that will work for your needs though, but it will mean also learning Power Pivot.
Mynda
Thank you - As I don't have access to Power BI yet I am doing everything in power pivot and power chart so that is really good news.
Thank you
Sue
