OK, I think the best way to get around the multiple insert problem is to do this in an after trigger. Unfortunately despite my try-and-avoid-updating-within-an-update-trigger rule, that is exactly what is needed to be done.

BTW, you can't modify trigger.new records in an after trigger like you can in a before trigger.

So here is how I would tackle your problem (as a start):

trigger CheckDuplicateBatchNumber on Prod_Detail__c (after insert, after update) {
    
   //Get IDs of related Prods then get all Details belonging to them.
   Set<Id> prodID = new Set<Id>();
   for (Prod_Detail__c pd : Trigger.new) {
        prodID.add(pd.prod__c);
   }
   List<Prod_Detail__c> allRelatedPDs = [SELECT ID, Name, Quantity__c, Prod__c, Prod__r.Id FROM Prod_Detail__c WHERE Prod__r.Id IN :prodID];

   //Create a master map (ie the PDs to be updated if there are duplicates) and a matching set (for easy comparison).
   Map<String, Prod_Detail__c> pdMasterMap = new Map<String, Prod_Detail__c>();  
   Set<Prod_Detail__c> pdMasterSet = new Set<Prod_Detail__c>();  
   for (Prod_Detail__c pd : allRelatedPDs) {
       String pdName = pd.Name.toUpperCase();
       pdMasterMap.put((String)pd.Prod__r.Id + pdName,pd);
    }
    pdMasterSet.addAll(pdMasterMap.values());
   
    //Create an update map and delete list.
    Map<Id,Prod_Detail__c> pdToUpdateMap = new Map<Id, Prod_Detail__c>();
    List<Prod_Detail__c> pdToDeleteList = new List<Prod_Detail__c>();
    
    //Populate
    for (Prod_Detail__c pd : allRelatedPDs) {
  
        String ref = (String)pd.Prod__r.Id + pd.Name.toUpperCase();
        //why won’t "pdMasterSet.contains(pd)" work?
        Boolean inMasterset = false;
        for (Prod_Detail__c mastPd : pdMasterMap.values()) {
            if (mastPd.Id == pd.Id) inMasterset = true;
        }
    
        if (! inMasterset) {
             // String ref = (String)pd.Prod__r.Id + pd.Name.toUpperCase();
             Id masterID = pdMasterMap.get(ref).Id;
             if (! pdToUpdateMap.containsKey(masterID)) {
                 pdToUpdateMap.put(masterID,pdMasterMap.get(ref));
             }
             pdToUpdateMap.get(masterID).Quantity__c = pdToUpdateMap.get(masterID).Quantity__c + pd.Quantity__c;
             pdToDeleteList.add(pd);
          }
    }

    //Commit to Database.
    if (pdToDeleteList != null) {
        delete(pdToDeleteList); // This must be done first to stop the update from causing an infinite loop.
        update(pdToUpdateMap.values());
    }
}

 Two things I can think of immediately.

1. Although this code won't go into an infinite loop (because I separated the pdToUpdateMap from the allRelatedPDs), it will run through completely twice, which is a bit of a waste. Wrapping it with a recurring trigger test will save you some CPU time.
2. This, as it is currently written, will create an "ugly" user experience.The way I am creating the master map is pretty random so sometimes the record the user is updating may be the one selected to be deleted—so although the result will be correct, the user will go "huh?". Also, even if not deleted there is no feedback to the user as to why the quantity changed. Assuming of course there is any user interaction with these objects.

I'll leave these to you to modify as you see fit. 

Good luck.

MellowRen