• Skip to main content
  • Skip to header right navigation
  • Skip to site footer

My Online Training Hub

Learn Dashboards, Excel, Power BI, Power Query, Power Pivot

  • Courses
  • Pricing
    • Free Courses
    • Power BI Course
    • Excel Power Query Course
    • Power Pivot and DAX Course
    • Excel Dashboard Course
    • Excel PivotTable Course – Quick Start
    • Advanced Excel Formulas Course
    • Excel Expert Advanced Excel Training
    • Excel Tables Course
    • Excel, Word, Outlook
    • Financial Modelling Course
    • Excel PivotTable Course
    • Excel for Customer Service Professionals
    • Excel for Operations Management Course
    • Excel for Decision Making Under Uncertainty Course
    • Excel for Finance Course
    • Excel Analysis ToolPak Course
    • Multi-User Pricing
  • Resources
    • Free Downloads
    • Excel Functions Explained
    • Excel Formulas
    • Excel Add-ins
    • IF Function
      • Excel IF Statement Explained
      • Excel IF AND OR Functions
      • IF Formula Builder
    • Time & Dates in Excel
      • Excel Date & Time
      • Calculating Time in Excel
      • Excel Time Calculation Tricks
      • Excel Date and Time Formatting
    • Excel Keyboard Shortcuts
    • Excel Custom Number Format Guide
    • Pivot Tables Guide
    • VLOOKUP Guide
    • ALT Codes
    • Excel VBA & Macros
    • Excel User Forms
    • VBA String Functions
  • Members
    • Login
    • Password Reset
  • Blog
  • Excel Webinars
  • Excel Forum
    • Register as Forum Member

Removing duplicate values based on results from two columns|Power Query|Excel Forum|My Online Training Hub

You are here: Home / Removing duplicate values based on results from two columns|Power Query|Excel Forum|My Online Training Hub
Avatar
sp_LogInOut Log In sp_Registration Register
sp_Search Search
Advanced Search|Last Search Results
Search
Forum Scope




Match



Forum Options



Minimum search word length is 3 characters - maximum search word length is 84 characters
sp_Search Search
sp_RankInfo
Lost password?
sp_CrumbsHome HomeExcel ForumPower QueryRemoving duplicate values based on …
sp_PrintTopic sp_TopicIcon
Removing duplicate values based on results from two columns
Avatar
Tracey Hartley
Member
Members

Dashboards

Power Query

Power Pivot
Level 0
Forum Posts: 8
Member Since:
May 26, 2022
sp_UserOfflineSmall Offline
1
March 6, 2023 - 11:08 pm
sp_Permalink sp_Print

Hi all.  I receive monthly payroll data and one of the tables I use is dd every month.  The data is staff structure, so there are always duplicate values.  I need the query to remove duplicates from Personal Reference (which needs to be a Unique Identifier) and only keep the version where the Date is the most recent date.  How do I do this?

Avatar
Lionel Baijot
Member
Members
Level 0
Forum Posts: 114
Member Since:
September 9, 2020
sp_UserOfflineSmall Offline
2
March 6, 2023 - 11:35 pm
sp_Permalink sp_Print

Hi Tracey Hartley,

Can you attach a file with anonymised data that shows the basic data to be processed and then the result you want to achieve?

BR,

Lionel

Avatar
Tracey Hartley
Member
Members

Dashboards

Power Query

Power Pivot
Level 0
Forum Posts: 8
Member Since:
May 26, 2022
sp_UserOfflineSmall Offline
3
March 7, 2023 - 12:13 am
sp_Permalink sp_Print

Hi.  Please find attached the output I get from my Power Query.

 

I need it to remove duplicates on Personal Reference:People (column I) so that I can use this as the Unique Identifier.  I need it to remove all those versions and just keep the latest version.

So, you can see that Personal Reference:People number 0000025 appears each mt, and I would only like to keep the latest version (so date 01/02/23):

 

01/01/2023 0000025 KB
01/02/2023 0000025 KB
01/04/2022 0000025 KB
01/05/2022 0000025 KB
01/06/2022 0000025 KB
01/07/2022 0000025 KB
01/08/2022 0000025 KB
01/09/2022 0000025 KB
01/10/2022 0000025 KB
01/11/2022 0000025 KB
01/12/2022 0000025 KB

 

In this example, Personal Reference:People number 5006718 last appeared on the 01/12/2022 file so this would be the record we would need to keep:

 

01/04/2022 5006718 EB  
01/05/2022 5006718 EB  
01/06/2022 5006718 EB  
01/07/2022 5006718 EB  
01/08/2022 5006718 EB  
01/09/2022 5006718 EB  
01/10/2022 5006718 EB  
01/11/2022 5006718 EB  
01/12/2022 5006718 EB 05/12/2022
Avatar
Alan Sidman
Steamboat Springs, CO
Member
Members


Trusted Members
Level 0
Forum Posts: 130
Member Since:
October 18, 2018
sp_UserOfflineSmall Offline
4
March 7, 2023 - 3:50 am
sp_Permalink sp_Print

If I understood you correctly, I first grouped your table by Personal Reference and Max Date.  I then merged (joined) that table back onto the original table to give you only the data for the Max Date for the Personal Reference

See the attached.  I was unable to open your query as it was linked to your PC.

Avatar
Tracey Hartley
Member
Members

Dashboards

Power Query

Power Pivot
Level 0
Forum Posts: 8
Member Since:
May 26, 2022
sp_UserOfflineSmall Offline
5
March 8, 2023 - 11:57 pm
sp_Permalink sp_Print

That is amazing, Alan!  Thank you so much for your help.

Sadly, there's one last bug.  PAYROLL_Structure returns 169 unique rows, but that increases to 178 rows with Merge1.  Looking at it, that's because 8 out of 169 Personal Reference:People are not unique as those 8 people have more than one role within the organisation (something I hadn't realised before doing this!)

Can you suggest any way around this, because otherwise I'm stuck without a Unique Identifier for example, is it possible to add a _1 or _2 to that final data set where there's a duplicate?

Many thanks!

sp_Feed
Go to top
Forum Timezone: Australia/Brisbane
Most Users Ever Online: 245
Currently Online: Mynda Treacy
Guest(s) 8
Currently Browsing this Page:
1 Guest(s)
Top Posters:
SunnyKow: 1432
Anders Sehlstedt: 870
Purfleet: 412
Frans Visser: 346
David_Ng: 306
lea cohen: 219
A.Maurizio: 202
Jessica Stewart: 202
Aye Mu: 201
jaryszek: 183
Newest Members:
Ivica Cvetkovski
Blaine Cox
Shankar Srinivasan
riyepa fdgf
Hannah Cave
Len Matthews
Kristine Arthy
Michelle Neven
Andrew Kuhn
Angela Paul
Forum Stats:
Groups: 3
Forums: 24
Topics: 6206
Posts: 27202

 

Member Stats:
Guest Posters: 49
Members: 31875
Moderators: 3
Admins: 4
Administrators: Mynda Treacy, Philip Treacy, Catalin Bombea, FT
Moderators: MOTH Support, Velouria, Riny van Eekelen
© Simple:Press —sp_Information

Sidebar

Blog Categories

  • Excel
  • Excel Charts
  • Excel Dashboard
  • Excel Formulas
  • Excel PivotTables
  • Excel Shortcuts
  • Excel VBA
  • General Tips
  • Online Training
  • Outlook
  • Power Apps
  • Power Automate
  • Power BI
  • Power Pivot
  • Power Query
microsoft mvp logo
trustpilot excellent rating
Secured by Sucuri Badge
MyOnlineTrainingHub on YouTube Mynda Treacy on Linked In Mynda Treacy on Instagram Mynda Treacy on Twitter Mynda Treacy on Pinterest MyOnlineTrainingHub on Facebook
 

Company

  • About My Online Training Hub
  • Disclosure Statement
  • Frequently Asked Questions
  • Guarantee
  • Privacy Policy
  • Terms & Conditions
  • Testimonials
  • Become an Affiliate

Support

  • Contact
  • Forum
  • Helpdesk - For Technical Issues

Copyright © 2023 · My Online Training Hub · All Rights Reserved. Microsoft and the Microsoft Office logo are trademarks or registered trademarks of Microsoft Corporation in the United States and/or other countries. Product names, logos, brands, and other trademarks featured or referred to within this website are the property of their respective trademark holders.