Data Platform Architecture Principles and Evaluation Criteria

so I am going to start with a quick introduction to myself uh I am Puja kalgaar I am working as a senior data architect with Rackspace technology I’m a specialist in a data modernization implementation specifically on Google Cloud platform and snowflake I have been working in a data domain for past 16 years I am a quick learner a tech blogger are a tech evangelist my hobbies include reading listening Indian classical music so we are going to let’s go to the agenda and see what what we are going to discuss in today’s session so we are going to talk about the architecture principles so what are the principles which help you to design your data platform we will take a use case of a data modernization and apply those principles and see what are the phases contribute to the principles design over your data platform what are the various offerings are there in the market and the most important part how we can evaluate them so we are going to see various pillars of the evaluation criteria and various inputs which would help you to build your own set of metrics and the evaluation chart to choose the right pick for your application modernization and at the end we will also go through a use case where I’ll walk you through the evaluation inputs and we are going to talk the talk about the final result and what happens as a next part of the evaluation criteria so the very first let’s go ahead and see what are the principles so there are five basic principles of architecture design framework so these principles are security scalability availability efficiency and operation Excellence so these are the five key pillars of the architecture design so this applies to your data platform design your data applications design and this also applies to the application design even you are considering designing of any application apart from the data oriented applications so let’s talk about the very first pillar security so security has various aspects of it so rabbitr access control your data protection data governance your compliance obviously so the security pillars takes care of all the security aspects and have the right set of compliances and policies implemented for your data platform the second one is calabity so scalability of course being being the need to have support to the data grown over a period of time time we need to have our applications which are scalable so unlike to the Legacy systems where we were not able to scale our applications with the cloud platform implementations we can very well implement the scalability so be it your horizontal scalability or a vertical scalability I am setting up the right Auto scaling policies will help you to design this pillar very well the next pillar is the availability so this talks about two important aspects of your application which is a resiliency and reliability so what time your application is up and running and this has a direct impact on your business and your end users so as an architect we have to always ensure that a data is resilient our applications are resilient and they are reliable enough so so that we have our applications running up up and running 24×7 and there is no impact to the business at all the next layer is the next pillar is efficiency the efficiency have two sides of the same coin that is a performance efficiency and a cost efficiency so when we talk about application design which is a performance efficient we cannot have the cost efficiency neglected by having a performance efficient implementation so this goes hand in hand so we have to consider while we Implement a fish performance efficient application your cost is not at a toss so this goes hand in hand and we have to put it in such a way that with the scaling a scaling policy I am the right right choice of a platform which would give your flexibility to have a pay uh pay as your usage the next important pillar is the operational excellence so operational excellence ties back to the maintainability and serviceability of your application so this is generally not paid much attention or very taken at is very easy go when we start designing the application but according to me and based on my experience this is the most critical part on this most critical pillar of architecture where you need to have the clear picture when you design the application so once you design application once you build the application you put it in a production what happened next so once your application is live you need to ensure that your Ops is not a high cost Ops you you if you don’t consider it initially you generally tend tend to put up more efforts maintaining your applications so with the cloud cloud implementations with a various data platform implementations it’s very easy these days to have a automated Ops and a predictive apps also implemented so that way these five pillars contribute to the architecture principles now let’s see how they are related to the most common use case of today’s Journey so that is the data modernization so this is the most common use case of a digital transformation where we talk we see five different phases of a modernization data Discovery data architecture and assessment net architecture and engineering actually migrating and modifying converting your data pipelines and finally your go live so these are the typical five phases of a modernization journey and when we talk about implementing the architecture principles implementing the evaluation criteria these two pay uh play a major role in the data Discovery and assessment phase where the discovery phase consists of various assessment questionnaire which would help you to generate the evaluation inputs and the architecture and assessment will help you to actually assess your existing system and see what are the challenges are there and what it takes to improvise your system and those actually becomes your evaluation criteria input and we will see as we go along though so the next important thing is how it all get starts what are the various personas which player role and how it gets initiated and when when it happens so the typical personas which play are important role in the discovery and assessment are The Architects engineering operations and business teams so these are the typical teams that play a key role providing inputs to the existing system and providing insights on what needs to be modified as we go and start designing on to the new data platform architecture and when it typically happens so there are various reasons of the drive it can be a business drive it can be a Technology Drive or it can be a management and engagement drive so depending on what what your organization or what your customers are being engaged or committed to the service offerings so generally these are the typical drives which drives a modernization journey and as an end result we see various assessment phases are done and tied back to the principle architecture principles now let’s go ahead and see what are the various platforms been available so there are there are many Cloud native platforms available so the big three players of a public cloud in the market Google Cloud AWS and azir so they have various data services being offered which are the native and manager services on top of Cloud they help us to set up Olt piano lab system they also help us to set up the data warehouse and italic systems we can also have a SQL and nosql systems building together and go hand in hand on top of the same cloud and using the set of limit set of the services offered by these Cloud platforms so along with these three major players in the market there are also various open source and the SAS offerings which are built on scratch for data on the cloud so for example snowflake so snowflake has been built from scratch to sub to have data on cloud so snowflake is the one of the best SAS example to implement a data warehouse and a data Lake irrespective of which Cloud it is you can have it very well set it up set it up on Google cloud or AWS or Azure so depending on your business need you can have it integrated with any of the cloud platform without impacting your system usage there are also various open source available in the market or the licensed one which we can use to have a data platform setup now the most important part is how these these play a role in the evaluation so let’s go ahead and see the evaluation criteria so evaluation again tying it back to the phases that we have seen in the modernization journey it falls under a assessment phase so let’s see what what are the prerequisites of the evaluation so as I say prerequisite what it mean it means that there are there are the drives so we have seen what what are the different drives to the modernization journey it can be a business or it can be a technology or an engagement so depending on what kind of what kind of privities we need to have the prerequisites set up so for example let’s consider a scenario where your customer have half of their applications run on cloud and half of their applications are still on Legacy and now they are looking forward to have a green field on cloud and they want to migrate all their legacy systems to the cloud platform so in that case this becomes as of top layer of a filter for your evaluation criteria so this help you to focus on what are the data platform to be focused on what are the data services to be chosen to evaluate as there has numerous data offerings in the market we cannot go ahead and evaluate all the numerous applications so this act as a prerequisite which which is a top layer of a filter so once you have a platform offering set up the another important aspect to it is the license tools so most of the cases these Legacy applications are built along with the set of license tools so for example your scheduler circuit orchestrators your bi tools your ETL tools even you are Ops tools or the ticketing tools so most of the cases because a customer or even as an architect I feel that we need to consider the license Tooling in such a way that what it needs to have the application set up in a less complex way so if for example if your ETL tool is just playing a simple dumping activity I would recommend to throw it out and then convert it everything into a elt phase so these prerequisites play a critical role and help you to derive the inputs so let’s see what inputs it derives to so during your Discovery phase we we have already gathered the stats of the existing system assessment phase and given a prerequisite we are trying to see what are the inputs which goes to the evaluation so capex versus implementation you are looking forward to so definitely the with the cloud it’s going to be a capex instead of Opex what kind of operations are being run are the CPU intents or a memory intense application what are the requirements to the system whether you need need in any systems which are a high CPU or high main operations typically what are the data challenges you have have you seen any tremendous data growth with the various data patterns which which your existing system is unable to support so these are all various factors becomes inputs to the evaluation and help you to look out for the checkpoints so let’s see what are the checkpoints such checkpoints are nothing but these These are the various layers of your evaluation criteria which will help you to segregate and derive the metrics for your evaluation So based on my my experience so far I have come up with this pyramid of a checkpoint so this pyramid consists of five layers the bottom most layer is of a data pit platform checkpoint so what it says is it says go and evaluate your existing system for a type of data its support what are the need of data changes what is the frequency of the data what is the total volume of the data and the projected ones the next layer is the processing checkpoint which talks about what are the Integrations of your system what kind of data pipelines you have is it a hybrid or it is just based on some tooling or it is a mix mix and match of the tools and the native scripts whether it is a patch and streaming how much percentage of the pipeline contribute to the business critical applications the next layer is the business checkpoint layer so business checkpoint let’s ties back to the business requirements where we talk about the SL is so what are the SL is when your end users are our customers are looking for a data availability so this plays a important role when we choose the data platform the next layer is a data analytics checkpoint so this is nothing but a checkpoint where you talk about the analytics what kind of analytics been run are the prescriptive descriptive or predictive if they are predictive what kind of ml models are being used is there any any need to have a GPU intense machines allocated to run those ml leads are they being used by end users or integrated with any of the bi tool so the data analytics checkpoint help us to gather these inputs the topmost layer is again the Ops and the business critical requirements so once you have your data platform checkpoints gathered now the topmost layer is the Ops layer where you talk about what are the what are the manage data pipeline management options are there how those are being maintained now right now and what are the requirements whether it’s a 24×7 monitoring whether it’s a 18×7 monitoring and what are the critical applications so let’s say you have 10 different applications running on a data platform all 10 might not be business critical few might be business critical if you might have a different slas depending on what kind of data type process and not entire applications would be even exposed to the end users so depending on the type of applications you are designing the type of Integrations require these all checkpoints play a vital role in building the metrics now let’s say how we relate it to the various metrics so let’s revise back quickly so evaluation has first point as a prerequisite which is the topmost layer to filter out the data platform Services the next layer is inputs where we talk about what are the various inputs we need to provide when we evaluate any of the data platform the next layer is of a checkpoint so whether all your checkpoints be laid out what are the typical challenges listed in each checkpoint and what is it that you are trying to achieve and the next layer is of a matrix so these metrics tied back to the checkpoint pyramid where each of these checkpoint have a various categories Associated and each category ties back to the set of metrics so this Matrix you see it on the screen might not be the exhaustive list of all the layers of the checkpoints these are some of the most critical and widely used metrics which would help you to evaluate your platforms now how it happens so the moment you funnel it down to the prequisite in a glass of light since you decide to choose three different data platforms to be evaluated then you end up doing a proof of concept on each of these data platform and these metrics will help you to result them down across each other so you would run the same set of metrics you would run the same set of slas and standardization across these three platforms and compare the results and once your results are compared you would have the end result of your evaluation and that’s where the probability to go with those set of data platform or the DB Services depending on your prerequisite so now let’s go ahead and see the sample use case of a data platform evaluation we are going to see a very simple use case where I have taken a use case customer is looking forward to have everything converted to cloud and they would still like to go ahead with their Ops tools so what are the inputs to the evaluation so the inputs are the application is of a retail domain and it runs on a data warehouse which is the teradata Appliance on the Legacy system it uses a ETL tool that is a data stage and what because what it is what it looks like now in terms of Legacy system is it has a data size of 120 terabytes which is a 70 terabyte active 50 terabytes passive and a daily volume which is expected to be of a one terabyte with the 80 of batch and a 20 of a streaming so and it also supports the two types of a data data types which is a structured and semi-structured data types and a customer have a typical data pipelines orchestrated and scaled through control M they are using a ticketing tool which is a jira and they are having a monitoring dashboard setup and alerting Via slack and email so they uh so the Ops part we can still go ahead and evaluate and see if we can keep the same Ops part on top of what customer now coming to the evaluation criterias so here I have given a scenario as the Legacy system was a teledata data warehouse and we are looking for a warehouse modernization so I’m comparing a bigquery versus Asia synapse here so of course these two are the two are the data warehouse offerings onto different Cloud platforms and they the tally is back to all our checkpoint layers so all the five checkpoint layers are being satisfied and we can have them implemented either on a big query or on synapse so what it takes to compare these two so remember we have seen the first layer of or the first funnel that we have is of a prerequisite and with a given use case we have a prerequisite where customer is already having a existing platform and they want to have rest of their applications moved to the same platform so the platform let’s consider the platform is of a Google Cloud platform so in that case there won’t be any uh any sense to go and evaluate a bigquery and synapse all together because when we know we are committed to have it migrated to Google Cloud then I would not have any efforts spent on to evaluating any other cloud cloud offerings instead what we will do is we will go ahead and evaluate the Google cloud services itself and see what different design approaches we can look forward to so for example we are looking for a data warehouse modernization on Google Cloud platform so bigquery is the databad Hub service which supports the structured and semi-structured data which also supports the batch and streaming and because of the performance efficiency of a bigquery we can have it used for for any petabytes of the data as well so it satisfies the the prerequisites it satisfies the inputs that are required for evaluation now while putting up in a form of a proof of concept there are two approaches that I’m proposing out here here for the given use case remember the customer have a warehouse run on an ETL which fades in data to the data platform so what what approaches we can do is the first approach is a lift and shift where your existing tools and everything would remain as is only the teradata warehouse would be replaced with a bigquery all your application will be hosted on the Google Cloud platform and everything feeds back to the bigquery with the minimal changes so all your connectors to the bi tools analytical tools here ETL tools will be modified to point it to bigquery and your application should be up and running so this could be the shortest way or done in a shortest plan as you you would continue to have the same license tools carried to the cloud second approaches as you are moving to Cloud does it make any sense to have any additional tools and the any additional licensing tools to be run on cloud why can’t we raise the power of the Native Services of cloud itself so what can we do let’s convert all the existing ETL jobs through alt have them converted to leverage the utilities of the Native services so let’s have them converted to a bigquery elt jobs bigquery SQL jobs then we can have all the data splitted between active and passive have the passive data on GCS all active on bigquery and from an analytics per se if you’re you’re keen on having the same reporting or bi tool used instead of a looker we can still go ahead and have a tablet used so the Tableau can still connect to the bigquery and have the purpose solved so with these two approaches what we will do is we will end up doing two proof of Concepts one on the approach one the second one on Apple approach 2 which would give a fair idea in terms of efforts required to migrate modernize convert the pipelines and it would help you to evaluate your approach so this is the final report of the evaluation which would help to evaluate the platform and in this case we are not evaluating a platform as the platform was the prerequisite given to us we have evaluated the services of the cloud platform and come up with the approach to user bigquery as a final service so that’s all I had so thank you for your time and uh attending the session stay in touch thank you [Applause]