• Skip to main content
  • Skip to header right navigation
  • Skip to site footer

My Online Training Hub

Learn Dashboards, Excel, Power BI, Power Query, Power Pivot

  • Courses
  • Pricing
    • Free Courses
    • Power BI Course
    • Excel Power Query Course
    • Power Pivot and DAX Course
    • Excel Dashboard Course
    • Excel PivotTable Course – Quick Start
    • Advanced Excel Formulas Course
    • Excel Expert Advanced Excel Training
    • Excel Tables Course
    • Excel, Word, Outlook
    • Financial Modelling Course
    • Excel PivotTable Course
    • Excel for Customer Service Professionals
    • Excel for Operations Management Course
    • Excel for Decision Making Under Uncertainty Course
    • Excel for Finance Course
    • Excel Analysis ToolPak Course
    • Multi-User Pricing
  • Resources
    • Free Downloads
    • Excel Functions Explained
    • Excel Formulas
    • Excel Add-ins
    • IF Function
      • Excel IF Statement Explained
      • Excel IF AND OR Functions
      • IF Formula Builder
    • Time & Dates in Excel
      • Excel Date & Time
      • Calculating Time in Excel
      • Excel Time Calculation Tricks
      • Excel Date and Time Formatting
    • Excel Keyboard Shortcuts
    • Excel Custom Number Format Guide
    • Pivot Tables Guide
    • VLOOKUP Guide
    • ALT Codes
    • Excel VBA & Macros
    • Excel User Forms
    • VBA String Functions
  • Members
    • Login
  • Blog
  • Excel Webinars
  • Excel Forum
    • Register as Forum Member

Complex remove duplicates |General Excel Questions & Answers|Excel Forum|My Online Training Hub

You are here: Home / Complex remove duplicates |General Excel Questions & Answers|Excel Forum|My Online Training Hub

vba course banner

Avatar
sp_LogInOut Log In sp_Registration Register
sp_Search Search
Advanced Search
Search
Forum Scope




Match



Forum Options



Minimum search word length is 3 characters - maximum search word length is 84 characters
sp_Search Search
sp_RankInfo
Lost password?
sp_CrumbsHome HomeExcel ForumGeneral Excel Questions & Answe…Complex remove duplicates
sp_PrintTopic sp_TopicIcon
Complex remove duplicates
Avatar
Eric French
Member
Members
Level 0
Forum Posts: 7
Member Since:
April 4, 2019
sp_UserOfflineSmall Offline
1
October 12, 2022 - 6:01 am
sp_Permalink sp_Print

Hi -

I'm trying to solve what (to me at least) is a complex remove duplicates problem.

The data contains forecasted costs by month for unique ID numbers.  The forecasts are updated monthly and the data includes a manually generated forecast name ("Name" in the sample data file), which is always the year and month they are updated.  For example, a forecast updated in July 2022 would have the name "2022-07."  The system I'm pulling from also automatically creates a version number ("Version" in the sample data file) for each forecast, i.e. 1, 2, 3, etc.

I recently discovered that forecasts deleted by users in the system of record are still being extracted in the data set, resulting in a duplicate combination of ID and forecast name.  In other words, the data is erroneously giving me duplicate forecasts.  Thankfully, the system-generated version number is not duplicated so there is a differentiator.  I cannot find a way to stop the system of record from including the deleted forecasts so I'm trying to figure out a way to do that in the Excel file that I download from the system.

I only want to keep the records/rows where the combination of ID and forecast name are associated with the maximum version number of that combination.  All other instances of that combination should be removed.  So in the sample data I've uploaded, I want the rows where the combination of ID 1234567 and Name 2022-07 include anything less than Version "5" removed from the data set.  The other sample data included needs to remain because it has a different combination of ID and Name.

I'm using Excel 2016 on a PC.

Avatar
Riny van Eekelen
Örnsköldsvik, Sweden
Moderator
Members


Trusted Members

Moderators

Power BI
Level 0
Forum Posts: 494
Member Since:
January 31, 2022
sp_UserOfflineSmall Offline
2
October 12, 2022 - 2:46 pm
sp_Permalink sp_Print

Hi Eric,

I believe Power Query (PQ) is ideal for this kind of task. Connect to the "blue table" that comes from the system. Then PQ can group all rows by ID and keep only the rows with the highest version number for each individual ID. So, if one ID only had 4 updates so far, the 4th version will be taken for that particular ID.

If you are not familiar with PQ, it's well worth learning. The attached file now includes such a PQ solution (the "green table") and should work in your Excel version.

sp_Feed
Go to top
Forum Timezone: Australia/Brisbane
Most Users Ever Online: 245
Currently Online: Atos Franzon, Louis Muti
Guest(s) 9
Currently Browsing this Page:
1 Guest(s)
Top Posters:
SunnyKow: 1432
Anders Sehlstedt: 873
Purfleet: 414
Frans Visser: 346
David_Ng: 306
lea cohen: 222
Jessica Stewart: 218
A.Maurizio: 202
Aye Mu: 201
jaryszek: 183
Newest Members:
Blair Gallagher
Brandi Taylor
Hafiz Ihsan Qadir
Gontran Bage
adolfo casanova
Annestine Johnpulle
Priscila Campbell
Jeff Mikles
Aaron Butler
Maurice Petterlin
Forum Stats:
Groups: 3
Forums: 24
Topics: 6369
Posts: 27852

 

Member Stats:
Guest Posters: 49
Members: 32359
Moderators: 3
Admins: 4
Administrators: Mynda Treacy, Philip Treacy, Catalin Bombea, FT
Moderators: MOTH Support, Velouria, Riny van Eekelen
© Simple:Press —sp_Information

Sidebar

Blog Categories

  • Excel
  • Excel Charts
  • Excel Dashboard
  • Excel Formulas
  • Excel PivotTables
  • Excel Shortcuts
  • Excel VBA
  • General Tips
  • Online Training
  • Outlook
  • Power Apps
  • Power Automate
  • Power BI
  • Power Pivot
  • Power Query
microsoft mvp logo
trustpilot excellent rating
Secured by Sucuri Badge
MyOnlineTrainingHub on YouTube Mynda Treacy on Linked In Mynda Treacy on Instagram Mynda Treacy on Twitter Mynda Treacy on Pinterest MyOnlineTrainingHub on Facebook
 

Company

  • About My Online Training Hub
  • Disclosure Statement
  • Frequently Asked Questions
  • Guarantee
  • Privacy Policy
  • Terms & Conditions
  • Testimonials
  • Become an Affiliate

Support

  • Contact
  • Forum
  • Helpdesk - For Technical Issues

Copyright © 2023 · My Online Training Hub · All Rights Reserved. Microsoft and the Microsoft Office logo are trademarks or registered trademarks of Microsoft Corporation in the United States and/or other countries. Product names, logos, brands, and other trademarks featured or referred to within this website are the property of their respective trademark holders.