Diamond Digital Marketing Group

Tag: Meta System Square

  • What is Metadata Repository

    What is Metadata Repository

    Definition

    Metadata Repository is a centralized database which stored all the metadata (i.e schema) of all other system or database within an organisation. You can regard it as the library of metadata of metadata in an organisation

    What Problem Pattern the Metadata Repository Solves

    Duplicated Schema definition in different system

    Imagine a scenario when an organisation has a CRM and eDM system which both have system properties First Name and Last Name , Middle Name and Gender . First of all, there will be a duplication time on defining the schema of the system properties (e.g. DataType , Validation Rules, Conditional Logic etc.). Besides, as the option values for Gender will also be the same (Male , Female, Unisex , Transgender), it will be spent duplicated time if we create the single dropdown list in the Gender field in both 2 systems.

    If we can just define the concept (e.g. Gender) once and utitize as much as we want throught all the systems inside an organsiation, it will save lots of time.

    Schema Definiton

    But the point is , what if these 2 different systems are built by 2 different teams? How can they figure out the concept Gender is already defined by other teams in another system?

    The Metadata Repository is acting as a metadata map to provide a bird view for the system developers and identify how to leverage the effort from other systems when they are building on their own.

    Synonym and Word Variations in Schema Defintion

    To reflect the intricacies of the real world, think about what if in the field Last Name in the CRM system is naming as Surnamein the eDM system, while in fact they are referring to the exactly same concept?

    Duplicated records in different system.

    To extend the example above, there will be different scenarios to record a new person in the CRM and eDM system, which means the maintenance cost will be increased as we are managing 2 sets of dataset which are overlapped and some are different.

    How can we know that the person Elon Reeve Musk in the CRM is the same person as Elon Musk shown in the eDM due to the fact that the Middle Name is recorded in the CRM while there is no Middle Name field in the eDM system?

    By using a Metadata Repository, the Relationships between these “Field” (i.e. Node) will be addressed and defined.

    Functions of a Metadata Repository

    As a Schema Template

    In practical, it is the duty of the System Analyst to interview the client and address the business objects and its associated properties and pass them to the programmar to build the system.

    For example, when building a lite version of a CRM , the System Analyst is expected to pass the following Schema Rules to the programmar:

    1. First Name
      • Datatype = String
        • UI = Single Text Field
      • Max Length = 50 Characters
      • Allow Duplicate = [Yes / No]
        • UI = Toggle
      • Placeholder Text = “Insert First Name”
      • Default Value = NIL
      • Index Key = [Yes / No]

    Imagine you are working for a company which need to builds tens of different of Systems for hundreds of different of clients, each system may comprise hundreds of Business Objects, and each Business Object may have tens of Properties, and each of the properties has tens of Schema Rules, not to mention that the systems of each different client might have a slightly different versions of Properties and Schema Rules.

    How can you , as a System Analyst, to pass the Schema Rules to the Programmar for coding will be time consuming process and big challenge.

    A Metadata Repository is expected to store the metadata of all these different systems in 1 place as a template for future use, as well as speed up the communication among different parties

    Metadata Harvesting

    On the contrary, what if the client already had a in-house self-hosted and CRM up and running , and would like to build a eDM by leveraging the existing properties of the CRM?

    Opposite to the function “As a Schema Template”, Metadata Harvesting is referring to collecting metadata from the existing system and turning them into the Schema Tempate.

    Identify the Relationships among Properties of different Business Objects

    After we have collected all the metadata from different data source, we have to find out the relationships among :

    1. Business Object (BO) to Business Object
      • e.g. : 1 Client (BO) HAD_PURCHASED Many Product (BO)
    2. Business Object (BO) to Business Object Properties (BOP)
      • e.g. : 1 Client (BO) HAS 1 Birthday Date (BOP)
    3. Properties (BOP) to Properties (BOP)
      • e.g. : Last Name (BOP) in CRM IS_SAME as Surname (BOP) in eDM

    Tools of Building Metadata

    In order to speed up the development cycle of system building, following technology stacks are used to fulfill the objectives.

    1. Graph Database – to record the relationships among Business Objects (BO) and its Properties (BOP)
    2. API – to fetch the schema defintions from different systems.
    3. Graph Data Science (Python) Library – To evaluate or predict the relationships among Business Objects (BO) and its Properties (BOP)
    4. Graph Visualizer – to visualize the result in different prospective for different users.
    5. Semantic Search Bar – to semantically input the search terms in in the Graph Visualizer in order to cater any search problem including : infix search / prefix search / synonym / typo / partial search and etc.

    Conclusion

    Metadata Repository and Graph Database will be the game changer in any industry as it can scale up and speed up the Business Management System development cycle significantly.

  • Build a Business Process Management System – Meta System Square

    Introduction

    Recall the article Build a Business Process Management System – System Index Concepts that there are 3 system index levels including System, Meta System and Meta System2 , this article is go to discuss the Meta System2 level.

    Problems in Relational Database

    To understand why there is a need in Meta System2, it should start by going through the problem patterns that we faced when using relational database to build the Business Process Management System.

    Recall the article Relationship between Human Learning and Data Structure , Relational Database is the database which is built on top of the relational data structure. For decades, even nowsaday it is the most common and popular way for us to build any the Business Process Management System, which works fine.

    But work fine does means there is no problem, it only means limited by technology, we can do nothing in old days and so we used to embrace with the problems in the relational database. And it’s new to review the problems.

    Let’s start the story with a typical business case – an Invoicing System.

    In the simplified model of an Invoicing System, at least there are 4 Tables inside the relational database :

    1. Sales Invoice
    2. Sales Invoice Order Line
    3. Product
    4. Client

    Each of the Tables will have some simple basic Properties as below:

    Sales Invoice Table’s Properties

    1. Sales Invoice #
    2. Invoice Date
    3. Total Invoice Amount
    4. Payment Due Date
    5. Client#

    Sales Invoice Order Line Table’s Properties

    1. Sales Invoice Order Line #
    2. Sales Invoice #
    3. Product Item Quantity
    4. Product Item Unit
    5. Product Item Unit Price
    6. Product Item Sub Total
    7. Product Item Name

    Product

    1. Product#
    2. Product Name
    3. Product Description
    4. Product Unit

    Client

    1. Cilent#
    2. Client Name
    3. Client Address
    4. Client Phone Number

    The Properties in the above 4 Tables are good enough for us to build a simple but functional Invoicing System. It work properly and smoothly until one day your shop is going to expand your business line from retail to wholesale. You want to have a Quotation Function on top of the Invoicing System in order to cater the request for quotation from the wholesale client.

    As an experienced system engineer like you , your intuitive thinking is that you simply add a Sales Quotation Table in your relational database. Yes it is.

    However, due to the fact that instead of an system engineer , you are also the boss of the business that you cannot stay in front of the laptop and open the No-Code platform to expand the current Invoicing System, the only way you can do is to use your mobile phone to enumerate the Properties and the metadata of the Properties (i.e. We called it Table Schema) which a Sales Quotation should have.

    After 10 hours of work, you are exhausted. You want to postpone the Sales Quotation Table expansion project to the weekend which you think to be less busy. Of course, instead of 1 weekend, the task is delayed to 4 weekends , which the wholesale order is overwhelming and you spend most of the time on tackling the back and forth of the wholesale order.

    It is ironic that while the main purpose of your business is to make profit, the overwhelming amount of the incoming sales order exactly fit your business purpose, and at the same time also is the main reason that you cannot expand your business due to the fact that these incoming sales order burnt out all your development time.  

    Of course this is a happy problem. Whether you want to overcome it or not depends on your level of ambition. If in case you are ambitous that you are not satified with the current scale and want to scale up to national level, you should have built the system to systemise and automate the workflow.

    Co-ordination in Business Process Management System Development

    Rather than the pursuit of perfectionism, due to the fact that your new wholesale business line thrives, therefore you compromise and believe that you can afford to hire a system engineer to do the hands-on coding tasks for you, and you focus on designing the blueprint of the system instead.

    When the system engineer – Anna is on board, you are excited to share to him the mighty vision of your retail and wholesale business, which he showed his support to you by asking you to provide him the blueprint and documentation the current system that you have built, so that he don’t need to reinvent the wheel.

    As you don’t want to cool down the enthusiasm of Anna, you finger crossed to provide him the blueprint within a week, which in turn for sure failed the promise again due to the fact that the wholesale incoming new business is overwhelming , as said before.

    Thinking of Building Meta System

    As expected , after Anna quit for a week, you review yourself and realise that Anna did not do anything wrong. The bottleneck of building a system is now shifting from the hands-on coding task to designing the blueprint of the system. Even a seasoned system engineer cannot acquire the knowledge domain of your industry comprehensively like you do. It is not reasonble for you to expect that the system engineer can grow the blueprint from scratch, at least not in a system engineer whom the salary you can afford.

    To summarize your failure , you deduced the problem pattern of building a system:

    1. A person can go fast, but a team can go far.
    2. If you want a team, you should place communication in 1st prority.
    3. Communication protocol is the prerequisite of communication.
    4. How to fetch the expertise of your knowledge domain from your brian, and fit them into the communication protocol is the key to speeding up.

    Having concluded these widsom in expense of your failure, you relieved and transformed. You start reading the books on how to build a meta system. And thanks to Diamond Digital Marketing Group, you figured out the BFs-WAITER Pivot Table is exactly what you are seeking for.

    After 6 months reading and practising, you master the essense of BFs-WATIER Pivot Table. You have identified much more concepts than the Invoicing System you built before. You enumerate some of the concetps as below:

    Entity

    1. Retail Client
    2. Wholesale Client
    3. Cashier
    4. Salesperson
    5. Book-keeper
    6. Shipping Agent
    7. Vendor

    Business Rules

    1. Tiered Pricing base on wholesale’s membership role
    2. Bundled Pricing to bundle the product with low stock turnover
    3. Role based discount rules based on Retail Client

    Workflow

    1. Issue Sales Quotation
    2. Sales Quotation signed back from Client
    3. Convert Sales Quotation into Sales Invoice
    4. Shipping to Client

    Information Flow

    1. Sales Invoice
    2. Blank Sales Quotation
    3. Paymenet Receipt
    4. Signed Quotation
    5. Delivery Note
    6. Thank You Email

    The Tables that are needed to be created in your relational database increase exponentially due to your ability of modelling your business process, which is a good sign.

    After your turn you enumerate all the concepts and turn them into Tables, you finally realise that there are more than 50 Tables that you should creating. Below are extract of the Tables and its extracted Properties lists below:

    1. SalesInvoice
    2. SalesQuotation
    3. SalesInvoiceOrderLine
    4. SalesQuotationOrderLine
    5. Client
    6. ClientsStaff
    7. ClientAddress
    8. ClientPhone
    9. ClientBank
    10. ClientCreditLine
    11. Product
    12. Vendor
    13. Vendor Staff
      • JobTitle
        • BDM
        • COO
        • CEO
        • MD
      • Salutation
        • Mr.
        • Ms.
      • Gender
        • M
        • F
    14. Vendor Address
    15. Vendor Bank
    16. Vendor Phone
    17. Vendor Credit Line
    18. Country
    19. District
    20. Zip Code
    21. …

    Based on the extract of the Table Lists, you address the pattern that:

    1 Table = 1 Business Object

    Table is in fact an business object. Each Table represents 1 Business Object.

    Object to Property = 1 to 1 Relationship

    Each Property (i.e. Column) of an Object (i.e. Table) has a 1 to 1 relationship with the Object (i.e. Table) itself.

    When to create a new Table

    Whenever there is a 1 to Many relationships between the Object and the Property, a new Tables should be created. For example while 1 Person only have 1 Last Name, the Property LastName is 1 to 1 relationshiop between the object Person, which make the Property LastName can stay in the same Table Person. (i.e. Person.LastName)

    However , while 1 Person has Many Phone Number, a new Object Phone Number is identified and a associated new child Table PhoneNumber should be created in order to link up with the parent Table Person.

    Number of Tables increased when Business Object increased

    The relational database is weak on recording the relationships among Objects. Every time if I want to record a 1 to Many relationship , a new child Table should be created, which means that the workload of operating inside the Database will be increased.

    If there are 1,000 business objects are needed to be recorded, it means 1,000 Tables should be created. The Entity Relationship Diagram for these 1,000 Tables will be like a maze which leads the system engineer get lost very soon.

    Moreover, if there is a new system engineer onboard, it is really hard for him/her to check if the Tables that he/she want to build already existed, due to the fact that every single person may use different name to name the Table. For example, If your colleage created a Table Entity, you may have no idea what it is and you probably will create another Table called Person, which in fact is referring to the same business object.

    There is a demand on a Table of Table to record the hierarchical relationship among business objects (i.e. Tables). You start feeling the birth of system of system (i.e. Meta System2)

    Number of SQL Statement increased when Business Object increased

    On top of that , as a system engineer, you realise that you need to rewrite the SQL statement whenever a new Table is referred. For example, when you want to enumerate all the Phone Number from All Person, you should write the SQL statement in below:

    SELECT Person.Name, PhoneNumber.Number
    FROM Person
    JOIN PhoneNumber ON Person.PersonID = PhoneNumber.PersonID;

    If there are 1,000 relationships among 1,000 Tables, you may need to write 1,000 SQL statements like this because the Table Name inside each SQL statement will be different.

    Challenge on recording objects with hierarchical relationship by Relational Database.

    Even worse, what if you want to record a hierarchical relationship among Objects which the relationship chains have no start and no end?

    For example, if I want to record the categories of the products.

  • Build a Business Process Management System – System Index Concepts

    Build a Business Process Management System – System Index Concepts

    Definition

    What is Business Process Management System

    Business Process Management System is the system which you want to standardize and automate literally any business procedures happened in your business routine. While some of the Business Process Management System, for example , a CRM System (Customer Relationship Management) is mature and well modularized which you can subscribe any canned CRM SaaS easily, some of the business routine, for example , a factory owner who wants to standardize the routine of the production line, should be customized and hence a Business Process Management System is needed to be built on purpose.

    What is a System

    Before we dive into the definition of a system, allow me to start the paragraph with the citation from a ancient wisdom.

    The whole is greater than the sum of its parts

    Aristotle

    While the statement from Aristotle is a genius idea exactly addressing the synergy which is derived from the system, the statement by itself does not have any executable guide for people to follow.

    Input-Process-Output Model

    In practical, you can take the definition of system as an  Input-Process-Output (IPO) model which is a framework used to describe the structure of a system or process. Here’s a breakdown of each component:

    1. Input: These are the resources, data, or materials that enter the system. Inputs can include raw materials, information, energy, or human effort.
    2. Process: This is the transformation stage where inputs are converted into outputs. It involves the steps, actions, or operations performed on the inputs.
    3. Output: These are the final products, results, or outcomes that exit the system after the processing stage. Outputs can be goods, services, information, or any other result of the process.

    In a daily example, when you want to withdrawal the cash from ATM machine, you insert the debit card and input the password and withdrawal amount (Input), then the ATM machine do the calculation on deducting the balance from your bank account (Process), and finally the currency note come out from the slot (Output). 

    Whatever if when you input the same thing, it will always process the same thing and output the same thing, this is call a System.

    A system is not necessarily related to “computer” or “digital”. For example , all of us in our body have a respiration system, digestive system , nervous system and so on. 

    Systems Chain

    After you get the Cash from the ATM machine, and then go to the train station and put the cash note into the self-serve Ticketing machine to buy the ticket. Immediately after you chose the destination and number of tickets, the ticket dropped out from the ticket slot.

    You can realise that in fact the Cash that you withdrawal from the ATM Machine before, is in fact the at the same time BOTH Output of the ATM Machine (i.e. the 1st System) and the Input of the Ticketing Machine (i.e. the 2nd System).

    The Output of a system is always the Input of another.

    So if the story goes on, you may realise that you can link up literally anything , i mean anything, no matter material or immaterial ,in this world by the System Chain concept. In fact, it is a philosophy of how you interpret the world. When you think like this , you will become a Philosopher. When you act like this , you will become an Engineer.   

    Scope of a System

    So we back to our ATM machine example. To recall your memory , a System implies you will always get the same output if you provide the same input. As long as the following the criteria are satisifed , you will always get the cash note:

    1. The Debit Card is functional
    2. Password is correct
    3. Your bank account have sufficient fund
    4. ATM Machine is functional
    5. ATM Machine cash slot has sufficient cash note

    I will congratulate you if after you read and you want to challenge me that there is nothing can be “always” except “the Change” itself. In fact, deciding what components are related to the system among the infinity number of concepts from the world , is as important as building the components inside the system.

    What if there is out of electricity supply during the transaction?

    What if the operating system inside the computer of the machine is down?

    What if the internet connection between the ATM machine and the terminal server is broken?

    in the world of Business Process Management System development, while Programmer is more on automating the system via coding for the components and functions within the system, an Engineer is more about observing the world to address what components are related to the system, as well as figuring out the interaction between the environment of the system and the system itself. 

    Coverage of the System

    What if you hungover and input the wrong passwords 3 times? If this is the case , the ATM machine will eat the card and then the screen will show up asking you to contact the staff of your bank, and then of course the user (i.e. you) will kick the ATM machine and growl a FXXX.

    When you start thinking like this, to think of all possible scenario which may be happened during the Usage of an ATM machine, you may realise that this is the most difficult part of building a system due to the fact if you are trying to identity all the if-then-else scenario and chain them up based on their dependencies , you will visualize a if-then-else decision tree diagram which finally enumerate almost infinity number of potential outcome that might be happened, that is why Dr. Strangeⓘ said he considered 14,000,605 possibilities in Avengers 4ⓘ. The parallel universe begins.

    Decision Tree Diagram
    Decision Tree Diagram

    If you start doing the job to test the possibilies and user scenario of a software, you become a UATⓘ Tester. If you start thinking of how to find a way to scientifically enumerate all the possiible outcome among all the if-then-else scenario, you will become a Mathetican in the aspect of Statistic, Graph Theory and Optimisation Theory.

    A good System caters all the scenarios after interacted with the environment

    Finally, before ending this session , back to the ATM example, you can consider to put the hidden microphone inside the ATM machine in order to record the FXXX after the user kicked the ATM machine because of Debit Card being eaten by the ATM machine, which is one of the possible scenario that you can think of, despite out of the scope of ATM machine software development. 

    What is Meta System

    Build a System to build another System

    As the ATM Machine metaphor continues, instead of the ATM machine user, now you become the engineer who built the ATM Machine. Based on the reading and you realise that to build an ATM Machine (i.e. a System) you need to cater all the possibie if-then-else use cases.

    One day, you successfully created a ATM Machine system which is very useful, and the CTO of the bank is going to bring your design to mass production. The next challenge is, how to justify the output quality of the ATM Machine?

    It is quite inevitable and reasonable to think of building another system to build the ATM Machine. And therefore here comes another System which builds the ATM Machine, which we name this kind of system as Meta System.

    What is System Index

    If you are crazy and idle enough, you may wonder what if I can build a System that can build a system to build a system (notes : there is no typo)? When in our daily conversation it is really hard to tell which level of System we are referring to, allow me to name the 3 Level of system as below:

    Level 1

    System : To standardize and automate a specific function

    Level 2

    Meta System : A system to build the System in Level 1 which search a specific function (i.e. build a system to build a system)

    Level 3

    Meta System2 : A generic system which can build any kinds of Meta Systemt to build a System for specific function. (i.e. build a system to build a system to build a system)

    For better communication, we, Diamond Digital Marketing, named this concept of System Level as System Index.

    While you can still understand why it is a need for the Meta System, you may in this moment consider Meta System2 as excessive. I understand your feeling and will suggest you to be patience to read the comprehend the whole idea in this or upcoming articles, which in turn will let you realising why there is a must for a Meta System2 and how many hundreds of hours and manpower you can save during the system development cycle.

    Suggested Reading

    Related Topics
Diamond Digital Marketing Group