Inplace conversion from WikiSon to List of Dicts

[WolfgangFahl]

As part of the #243 activities there is now a nice conversion between WikiSon/WikiText notation to python List of Dicts so that then handling in different back end storage mechanisms via pyLodStorage will be enabled. See my Workdocumentation of today for the resulting new code (copied below for your convenience and for readers who do not have access to our Requirments/Usecase Wiki.

Personally i consider the "inplace" conversion of the records harmful. We might end up with a mixture of attributes/properties form different sources that are incompatible. I'd rather see creating a copy of the original record and in cases where some kind of check is needed comparing the records for consistency.

We also might want to add an option to issue warnings if the mapping is incomplete and there are unknown properties in the mapping. For the ormigration that will be pretty much standard due to the dozens of properties that are declared and sometimes never used in the environment.

I am going to add another issue in this project that links to tholzheim/wikirender#9 for the general issues we have with making sure the different representation of entities/instances in our Semantic MediaWiki environment are aligned.

What do you think?

Example Result of current approach:

{
  'Acronym': 'ICSME 2020',
  'Title': '36th IEEE International Conference on Software Maintenance and Evolution',
  'Series': 'ICSME',
  'Ordinal': '36th',
  'Type': 'Conference',
  'Field': 'Software engineering',
  'Start date': '27th Sept, 2020',
  'End date': '2020/10/03',
  'Homepage': 'https://icsme2020.github.io/',
  'City': 'Adelaide',
  'State': 'Online',
  'Country': 'Australia',
  'Abstract deadline': '2020/05/22',
  'Paper deadline': '28th May, 2020',
  'Notification': '2020/08/04',
  'Camera ready': '2020/08/25',
  'Has host organization': 'Institute of Electrical and Electronics Engineers',
  'Has coordinator': 'Sebastian Baltes',
  'has general chair': 'Christoph Treude, Hongyu Zhang',
  'has program chair': 'Kelly Blincoe, Zhenchang Xing',
  'has demo chair': 'Mario Linares Vasquez, Hailong Sun'
} {
  'Acronym': 'AISB 2009',
  'Title': 'AISB Symposium: New Frontiers in Human-Robot Interaction',
  'Type': 'Conference',
  'Field': 'Uncategorized',
  'Start date': '2009/04/08',
  'End date': '2009/04/09',
  'Submission deadline': '2009/01/05',
  'Homepage': 'homepages.feis.herts.ac.uk/~comqkd/HRI-AISB2009-Symposium.html',
  'City': 'Edinburgh',
  'Country': 'United Kingdom',
  'Notification': '2009/02/02',
  'Camera ready': '2009/02/23'
}
[{
  'Series': 'ICSME',
  'Type': 'Conference',
  'Field': 'Software engineering',
  'Abstract deadline': '2020/05/22',
  'Paper deadline': '28th May, 2020',
  'Notification': '2020/08/04',
  'Camera ready': '2020/08/25',
  'Has host organization': 'Institute of Electrical and Electronics Engineers',
  'Has coordinator': 'Sebastian Baltes',
  'has general chair': 'Christoph Treude, Hongyu Zhang',
  'has program chair': 'Kelly Blincoe, Zhenchang Xing',
  'has demo chair': 'Mario Linares Vasquez, Hailong Sun',
  
  'acronym': 'ICSME 2020',
  'ordinal': '36th',
  'homepage': 'https://icsme2020.github.io/',
  'title': '36th IEEE International Conference on Software Maintenance and Evolution',
  'startDate': '27th Sept, 2020',
  'endDate': '2020/10/03',
  'country': 'Australia',
  'region': 'Online',
  'city': 'Adelaide'
}, {
  'Type': 'Conference',
  'Field': 'Uncategorized',
  'Submission deadline': '2009/01/05',
  'Notification': '2009/02/02',
  'Camera ready': '2009/02/23',
  'acronym': 'AISB 2009',
  'homepage': 'homepages.feis.herts.ac.uk/~comqkd/HRI-AISB2009-Symposium.html',
  'title': 'AISB Symposium: New Frontiers in Human-Robot Interaction',
  'startDate': '2009/04/08',
  'endDate': '2009/04/09',
  'country': 'United Kingdom',
  'city': 'Edinburgh'
}] {
  'Acronym': 'AAAI',
  'Title': 'Conference on Artificial Intelligence',
  'Logo': 'Aaai-logo.jpg',
  'has CORE2017 Rank': 'A*',
  'Field': 'Artificial intelligence',
  'Period': '1',
  'Unit': 'year',
  'Homepage': 'www.aaai.org/Conferences/AAAI/aaai.php',
  'WikiDataId': 'Q56682083',
  'has CORE2018 Rank': 'A*',
  'has Bibliography': 'dblp.uni-trier.de/db/conf/aaai/',
  'has CORE2014 Rank': 'A*',
  'DblpSeries': 'aaai'
}
[{
  'Logo': 'Aaai-logo.jpg',
  'has CORE2017 Rank': 'A*',
  'Field': 'Artificial intelligence',
  'WikiDataId': 'Q56682083',
  'has CORE2018 Rank': 'A*',
  'has Bibliography': 'dblp.uni-trier.de/db/conf/aaai/',
  'has CORE2014 Rank': 'A*',
  
  'acronym': 'AAAI',
  'homepage': 'www.aaai.org/Conferences/AAAI/aaai.php',
  'title': 'Conference on Artificial Intelligence',
  'dblpSeries': 'aaai',
  'period': '1',
  'unit': 'year'
}]

class OREntity(JSONAble):
    '''
    base class for OpenResearch entities
    '''
    def __init__(self):
        '''
        Constructor
        '''
        
    @staticmethod
    def fromWikiSonToLod(record:dict,lookup:dict):
        '''
        convert the given record from wikiSon to list of dict with the given lookup map
        
        Args:
            record(dict): the original record in WikiSon format
            lookup(dict): the mapping of keys/names for the name/value pairs
        '''  
        # replace name,value pairs with lookup[name],value pairs
        for key in lookup:
            if key in record:
                newKey=lookup[key].get('name')
                if newKey is not None:
                    record[newKey]=record[key]
                    del record[key]


class OREntityList(JSONAbleList):
    '''
    OpenResearch entity list
    '''
...
    def fromWikiFileManager(self,wikiFileManager):
        """
        initialize me from the given WikiFileManager
        """
        self.wikiFileManager= wikiFileManager
        wikiFileList=wikiFileManager.getAllWikiFiles()
        lod=self.wikiFileManager.convertWikiFilesToLOD(wikiFileList.values(),self.getEntityName())
        self.normalizeLodFromWikiSonToLod(lod)
        self.fromLoD(lod)
        
    @classmethod    
    def getPropertyLookup(cls)->dict:
        '''
        get my PropertyLookupList as a map 
        
        Returns:
            dict: my mapping from wiki property names to LoD attribute Names or None if no mapping is defined
        '''
        lookup=None
        if 'propertyLookupList' in cls.__dict__:
            propertyLookupList=cls.__dict__['propertyLookupList']
            lookup,_duplicates=LOD.getLookup(propertyLookupList, 'prop')
        return lookup
      

    @classmethod
    def normalizeLodFromWikiSonToLod(cls, lod:list):
        '''
        normalize the given LOD to the properties in the propertyLookupList
        
        Args:
            lod(list): the list of dicts to normalize/convert
        '''
        lookup=cls.getPropertyLookup()
        if lookup is not None:
            # convert all my records (in place)
            for record in lod:
                if not isinstance(record, dict):
                    continue
                OREntity.fromWikiSonToLod(record,lookup)

Unit Test

    def testWikiSonToLodPropertyMapping(self):
        '''
        test the mapping
        '''
        wikiFileManager=Corpus.getWikiFileManager()
        for entityList,entity,templateName in [
                (EventList,Event,"Event"),
                (EventSeriesList,EventSeries,"Event series")
            ]:
            #propertyLookup=entityList.getPropertyLookup()
            #print(propertyLookup)
            listOfSampleWikiSon=entity.getSampleWikiSon()
            lod=[]
            for sampleWikiSon in listOfSampleWikiSon:
                wikiFile=WikiFile(name="noname",wikiFileManager=wikiFileManager,wikiText=sampleWikiSon)
                #print(str(wikiFile))
                record=wikiFile.extract_template(templateName)
                print(record)
                lod.append(record)
            entityList.normalizeLodFromWikiSonToLod(lod)            
            print(lod)
        pass

[MusaabKh]

The property check seems like a good idea specially when we consider the mapping of the properties to entities. An extra check could provide a more reliable interface to work on.

[tholzheim]

Yeah a warning for unknown properties seems reasonable.
Do we also need a mapping for the other direction?
Since Template:Event uses not exactly the property names
e.g. inEventSeries is Series in the Template

Or are we using the template names as default property names in the Event samples?

[WolfgangFahl]

Differences between labels, property names and template parameter names are a major source of confusion and OpenResearch has lots of such confusions. In the generated environment that we intend to create such problems are mitigated because no humans are involved any more in making the links work. Still we'll have to make sure the migration works and that might need two different mappings or a carefully stepwise fixing of templates. As long as we have two different versions of OpenResearch live as we do at this moment things are even more complicated. As you know i am not in favor of such complications but for simple, understandable procedures. Let's make the migration work by working on tholzheim/wikirender#9 and we'll know where the detail problems are.