Data! Data! Data! with Francesco Tisiot

Matty: [00:00:00] See, this is where I just have to change my whole way I think about the world because I’m still stuck back in 1999. It’s time for Arrested DevOps, the podcast that helps you achieve understanding, develop good practices, and operate your team and organization for maximum DevOps awesomeness. I’m Matt Stratton. I have A really exciting show for you today. They’re always exciting shows. Have I ever said we’ve got a really boring episode of Arrested DevOps coming? We never tell you that. It may happen, but it’s not happening today. We are going to be talking about the wonderful world of data. But before we dig into that, a word from our sponsors. Collecting compliance evidence shouldn’t involve spreadsheets and scavenger hunts. With automated controls and over 75 integrations, Drata automates the process without needing to be an expert. Drata supports 14 frameworks including SOC 2, GDPR, HIPAA, and ISO 27001. Companies like Notion and LemonAid have shared how Drata simplifies audits through automated evidence collection. Don’t let audits slow down your dev cycle. Request a demo today and get 10% off along with waived implementation fees at drata.com/partner/arresteddevops. Feeling like you have too many alerts, overwhelmed by vulnerabilities, and at the end of the day, not deploying apps as quickly as you would like? Sysdig hears you. Security in the cloud can be overwhelming and security posture is suffering. You need a way to prioritize what matters so that you can move faster. Shift left is the right operating principle, but you must shift left the right way. Sysdig roots everything it does in runtime insights. By knowing what is running in production, you can prevent, detect, and respond to threats, and do it at cloud speed. To learn more about Sysdig, visit sysdig.com/arresteddevops. I am joined today by my colleague and friend Francesco. We’re going to be talking about data, data platforms, data resiliency. I don’t know, we’re just going to see where it goes. So, but before we get into that, Francesco, why don’t you introduce yourself to the Arrested DevOps audience?

Francesco: [00:02:20] Hi, Matti. Thanks for having me. Yes, I believe you said something interesting there. I’ve been for 15 years of my life into data, but as I would classify, I do daily work on data because I need someone to pay me for my main mission in the world, which is trying to tell people how to properly eat Italian dishes. And, you know, doing with data pays the bill quite well.

Matty: We’ll have to dig into this a little bit. We’ve, we’ve had Tim Banks on the show before and had a long riff about what is chili and what is not chili. And maybe, maybe we, we’ll, we’ll get into this data thing, but let’s, let’s, you know, let’s, let’s follow this up for a second here. So what, what are, what are a couple of the, what are some of the ways that people eat Italian dishes incorrectly? Like what are the mistakes to avoid?

Francesco: Just saying that you have the carbonara with cream or that you put pineapple on pizza or that you have lunch or dinner while drinking a cappuccino. Now, if you enjoy it, it’s up to you. But if you try to do that in Italy, I don’t believe the waiter will be happy to serve you.

Matty: [00:03:30] So thinking about that, you know, rules being important, process being important, consistency being important, you know, data is a pretty big topic, right? And I guess if we kind of think about how to break this up, and this is one of these things that I’d love to kind of get your take on how to folks who work maybe not in a completely data-focused role. So, when I think about when I was coming up and I was a sysadmin and the DBA, like the database stuff, like we didn’t have to worry, but like we gave them a server to run it on and there was just sort of this group of the DBAs and people became a DBA because they didn’t like people and they didn’t want to have to talk to people. People and they could just hide in a closet. And that’s something if the sysadmin is saying that the DBAs are the people who don’t like people. And I feel like that’s maybe not really as applicable, right? You have to understand at all kinds of aspects of a technology role because data’s become so critical. Like what’s some of the evolution you’ve seen in your career about this sort of cross-functional approach to data?

Francesco: [00:04:38] I believe you touch an interesting point, which is also where I started my career. I’m a computer science engineer, but then I wasn’t by far the smarter in the room at my university, but I like to engage with people. And when I finished the university and started looking for a job, I tried to find a job where Yes, I could apply part of my computer science knowledge, but then I wanted to stay in the middle of computers and people. And I find that the data is the perfect place where to be, because yes, maybe the DBA role is some sort of hidden person in a closet, but most of the other data jobs a data analyst, a data scientist, a data engineer, they need somehow to translate human expectations, business expectations into technology, into a document, into a model that is used in the company. So I found myself into this kind of mix between talking with people, understanding various businesses. And I mean, I’ve done 12 years of consultancy across a huge variety of sectors. And that is, I believe, the beauty. Data is data, but how you think about it, how you make sense of the data, changes every time you interact with different people. And this is where you mix the computer side with the human side. And this is the part that I personally like a lot about being in the data space.

Matty: [00:06:21] So we kind of talked about a bunch of different roles, right? You talked about being a data scientist, if you’re a data engineer, You know, we think about the operational pieces of that. Obviously, there’s a lot that goes into that. I think when we kind of could dig into where those places go, but even data itself contains multitudes, right? Like, we can think about operational data stores, we think about if it’s event streaming, if it’s analytical data without— and then even thinking about machine learning pieces and parts of that. Kind of how would you, if we wanted to sort of take this from the perspective, and listeners, maybe you are a data expert, so just bear with us here, but maybe to sort of set some ground pieces, like what are some of the main ways when people talk about like the data systems that run their organization, what do those generally look like in their function?

Francesco: Yep. Okay. Let’s start by saying that data is not something stable or that takes only one shape. It’s a journey. Every data point in the company makes a journey. Usually the first step that a data point makes is that it’s produced. And for example, if you have a company selling goods on internet, the first step is usually someone purchases a product and you end up in a record in a database. That was true like 50 years ago, is also true now, and actually is one of the best ways of storing the data into a database, which is called transactional database, because you purchase one product or you retrieve one order from your website. But this is just the beginning of the journey, because if you have a company that produces data, like almost any company, now the data point starts being interesting for the company because you need to make sense of that data point in the big picture. So you will have various methods to take the data point from the transactional system, which is where you initially store the data, through all the steps until it arrives to basically the business analyst or the business desk. Where it becomes a KPI saying, okay, yesterday you sold 20% less shoes than 2 days ago. What happened? So there is an entire journey from that single, I purchased one pair of shoes, to the pie chart or trend chart or line chart or whatever that the business owner has that It’s made by several technologies, made by several different people, and understanding all the little details of this journey can basically tell you a lot about, for example, also the state of the company. Because if you think about, for example, a startup, the first thing that you need to do in a startup is basically to find a way to store your product orders. And you do that by just putting all the data in a transactional database. If you are a startup, the next thing that you want to do is to do analytics, to be data-driven. So what you do is, for example, run queries against the same transactional database in order to say how many shoes did I sell today compared to yesterday. And this works for small-medium system. This works really well because the amount of data is not too much and because maybe you are okay in analyzing this morning all the shoes that you sold yesterday. But then if your company is successful, the data starts increasing. But not only the data starts increasing, and then your analytical query basically are making the transactional database suffer a little bit. So you want maybe to keep the transactional database, but then try to move the data into some analytical database that will serve all the analytical queries without having— creating any additional load to the transactional database. And this is where, from one single piece of technology that solves all the company needs, you start piping a set of technologies one next to the other. And this is where, for example, change data capture, event-driven event— event-driven architecture, analytical databases start taking shape. And they follow pretty much also the evolution of companies, the growth from a startup to a consolidated business, the growth of the data, the amount of data as well.

Matty: [00:11:33] As an audience, we have folks from all sorts of different walks of life. You know, we have folks who are very infrastructure focused. You know, some of our software engineers might be saying like, hey, we work with these systems all the time. Some folks who are more on the infrastructure side might be like, Well, I know we host this thing, but I don’t really know what it does. So, you kind of talked about, like, event streaming. I know this is a topic that comes up a lot, like, at least you hear a lot about it, right? You know, you got things like Kafka, you got all these events. So, where does this come in, like, in the flow of an application, in the flow of things that happen in my system? Like, what do we actually mean by event streaming?

Francesco: Okay, let me do a little bit of history there. And let’s go back to our example of selling shoes. In the old days, we were basically creating batch systems, which meant that this morning I was able to check the data of yesterday. Because what we were doing is we were waiting for the night where our website was offline and then do a batch extraction of all the data from our transactional database, push them in the analytical database where we were building all the data marts, all the data warehouses, all the data models that were feeding our reporting layer. For some businesses, this is still true. And I mean, if you just need to, you know, report on last month data, waiting one day is not a big issue. On the other side, we have a new set of businesses, a new set of use cases where you cannot wait 12 hours, you cannot wait 6 hours, you cannot wait 10 minutes, you need to react immediately to changes. Let’s say that, you know, we want to sell shoes, but we don’t want to have a huge inventory. So we need to make sure that if we sell 10 pair of shoes, we need to go back to the people furnishing the inventory and to order other 10 or other 15 because we see an upsell. In those cases where we need a rapid interaction between what happens in the outside world and what happens in, in our world, these are the cases where we need to go away from batch and into what is called real-time or near real-time. So we need to work not by what happened in the last 2 hours, but we need to start working per event. Every time I order 5 pair of shoes, because, you know, I love those green pair of shoes, I click OK, I purchase on the website, the order gets pushed into the database, but then with the current technologies like Kafka, which is a streaming technology, I can take the change that is happening in the database, and propagate this change to all the other downstream technologies. And the other downstream technologies could be an analytical database, could be another application that checks what is the current inventory stock for the green shoes compared to a prediction that a data scientist created via model and tells me, look, that you have already to start calling the producer of those green shoes because we will run out of stocks in 3, in 3 weeks. So all this kind of interaction where we need rapid sequence of actions based on a user click, a purchase, or whatever, is what makes batch really old on one side, and what requires really real-time interaction and technologies like Apache Kafka. One thing that I mentioned before, I called out a technology which is called change data capture. I believe it’s interesting to dig a little bit more why this is important. One thing that I said at the beginning is that in a startup world, what you want to do at the beginning is to have an app and then a database behind the app. Then you increase the volumes and you need to propagate the state or the changes that are happening in the database to other technologies. If you apply change data capture process, this means that you are able to track any change happening in the database without having to continuously query the database. So there are various systems that allow you to check the database logs and take any change that is happening in the primary database and move it to other technologies. This is really helpful because it allows you to evolve, passing from a batch system into an event-driven system, without having to touch the transactional database and the application in front. So the business can continue running as it was before, but in the meantime, you can completely evolve your backend processes. And go from batch to real-time.

Matty: [00:16:46] As this stuff grows more and more complex, what are some of the challenges of, like, especially in a large organization when you’re creating these interaction points to solve different problems? What do people do to kind of keep all that stuff aligned?

Francesco: And I like a lot this question because I feel that this is where my previous job and my current job mix a lot. In my previous role, I’ve been doing a lot of enterprise, big enterprise consulting for like 10 years of my career. And now I’m working a lot with startup or small-medium businesses that are still in the startup phase where they want to build a lot of things really fast. But I’ve been on the other side of the pond working with huge enterprises where at a certain point you were building, building, and building, but then you start realizing that you don’t know about a data mart which is sitting there. You don’t know what the data pipeline called ABC is doing. You don’t know who built or who purchased an entire stack of tools because the people come and go and company remains and data remains and, you know, bills remain. But it’s very easy to build up, then managing, keeping up to date, documenting, or understanding the documentation This is where the challenge is. And specifically now with huge companies with a lot of data flowing around, with empowering more and more non-technical users to build business intelligence tools and all the analytics with point-and-click solutions, the risk is that We have so many views of the reality that there is not a single reality anymore. I’ve been trying to battle against people extracting data and creating an Excel report out of transactional system for 10 years. And I’m sorry to say that I lost my battle, but I don’t believe there will be one that will eventually win that battle. Because at a certain point, someone will find that the task that they have to do with a specific tool doesn’t offer a specific function that they need. But by the way, Excel offers it, or they know in Excel how to do that in 10 steps. What I believe is crucial though, is for companies to always try to have the global eye on what data assets they have, which data assets they have, which data pipelines they are building, and how those are connected. Because if you don’t have the map, you are lost. If you don’t have the map, you don’t know who is using what. You don’t know who is, you know, checking the inventory of shoes. And if you don’t know that, you could spend extra money. You could have another project, 6 months, hiring a new consulting firm to solve exactly the same shoes inventory problem that you already solved 6 months ago. But just because you didn’t know that that was already solved, you waste a lot of money. And even worse, if you basically allow 2 teams to solve the same problem but without giving them way of talking with each other, the end result will always be different. Because there is a small detail in a KPI definition, and then everything else becomes different. But then I believe the most critical part of letting data pipelines and data assets grow without control is that it’s not only a technical challenge. Now we are also talking financially. Because in Europe we have GDPR. But even if GDPR wasn’t a thing, if you don’t control strictly where your data goes and where it lands and who can do what, you’re opening yourself to the risk of Francesco, by mistake, takes an export of a database into CSV and dumps it into an OpenStreetMap bucket. And we saw that this happened over and over again. If you limit the ability to do that, if you have a way to more or less control where your data is and where it goes and who can do what, you know, creating a perfect solution, it will be extremely hard because there is always the human factor. But at least having the knowledge of, on business as usual, my data pipeline goes from here to there, to there, to there, is something that allows you to reply to a lot of questions really easily.

Matty: [00:22:14] I imagine that getting to that becomes much more challenging exponentially at scale, right? Because the more systems you have, the more teams that you have, the more We talk about this, just as true of non-data systems, right? The days of being able to hold the whole architecture in your head are gone, right? That was back in— we lost that with the LAMP stack days, right? Which is still a totally legit way to build an application. But do you see anybody really doing this well, or is it aspirational, or are there— and you don’t have to name names necessarily, but What are some of the ways that people have been maybe accomplishing this, having the view of everything, right? When you have these disproportionate lines of business or different sprawl throughout a large enterprise.

Francesco: So, there are 2 ways that I’ve seen this somehow working. The first way is kind of what I call the traditional way. It’s relying on documentation. Every time you were purchasing, creating something, updating, creating a new data pipeline, you, like everyone, was forced to write some docs about it. Which may be now where ChatGPT could be a thing. Because, you know, the problem of writing documentation is not only having to write the documentation, it’s also being able to parse the documentation and find out the insights that you’re looking for. But still, relying on documentation is a bit risky because, as we know, documentation is usually an afterthought. You first build a pipeline, you check that the pipeline is working, you are at the green light, it’s working in production. Now it’s maybe time to write documentation if, for example, we didn’t finish all the budget for our consulting firm. So that is a huge risk. The other part where I saw this working is where you were using one unique tool not to store all your data, but to do all the transformations on top of your data. So if you were using a tool— I’ve been using a tool called Informatica to do a lot of the data pipelines— and if you were just using that tool, problem solved. The tool itself could give you, for example, data lineage. So it could give you all the steps column by column almost that were telling you where the data was coming and going. The problem now is that data spans very different technologies, data spans very different teams that are in charge of building the data pipelines, and you don’t have anymore a single tool that, you know, can manage them all. On the other side, that’s the other bit of the equation. Going back to the Excel example, you want to empower users to use the best tool to solve a problem. Because if you don’t empower them, they will find the Excel way. They will find a way to export the data and to feed the target platform manually. And this opens a security hole in your, in your data pipeline because you magically don’t know how the data goes from here to there. So what you want to do is to build resilient, scalable, secure data pipeline that guarantee that the correct data goes from here to there, and I can manage who can see what and who can do what. And like, I don’t want, you know, Dave, my colleague, to take an export of my database and to feed OpenSearch with it. So being able to build those data pipelines and even more being able to monitor those data pipelines and those data transformation and those data assets, I don’t think it’s a problem that is solved currently because it’s a very hard problem.

Matty: [00:26:23] I was going to say, because like you said, there’s, you know, there’s kind of 2 ways you approach the scenario, right? You can either say we do this with documentation/training, right? You know, or you do it with guardrails. And the, the problem with the former is that’s the absolute worst way, especially for something that people don’t spend a lot of time in all the time. You know, I mean, we run into that with just process things. I mean, I’ve been beating my head against the wall with Asana at work because I want to be able to make it easy. I don’t, I don’t want people to have to read a bunch of documentation to know how to do things the way we want to do it. I want, the way that we do it to be the easy way to do that. So, on the other side, like you said, and I think this is where maybe when it comes to like these data systems, it becomes a lot more challenging because we have a lot of great stuff in other things we do that can be self-documenting, that can be generated. You know, we can generate API documentation automatically. We can do all this, but data is so, like you said, even if it’s all within one tool, You might have the opportunity to be able to do some auto-gen, something of that, but these things don’t live in their own little vacuum and they’re treated so differently than every other problem we’re trying to solve, right? Kubernetes is great until you want to throw some data, right? Containers, awesome, but what about state? Those are minor issues around the data side and the speed at which things happen. It’s the problems that have to be solved completely differently than everything else we’re doing with these computers of ours.

Francesco: [00:28:03] Yeah, I believe that there is a possibility there though, because one interesting thing that data systems have is, as you said, if you work in a single data technology, you can export the metadata into more or less, if you think about a database like Postgres, you can see all the, what are called the catalog views, which allow you to, for example, list all the tables, list all the users, list who can see what, who can write what, which views are created on top of which tables, and so on and so forth. And this works. I mean, it’s what DBAs and titles like that have been checking and verifying since 15 years. But now the challenge is when the data goes above, goes between 2 different technologies, how do you close the gap? And I believe we can close the gap by including a lot of technical knowledge of both systems. And a lot of technical knowledge of how systems work together. Because if you think, for example, of the integration between a database and Kafka, usually this is done with a tool called Kafka Connect. Kafka Connect, it’s basically, you configure Kafka Connect with JSON payload, where you tell Kafka Connect where the data comes from and where it goes. So if with a tool you are basically able to parse this JSON, extract this information, you can link the 2 worlds. It’s not an easy job, but has the potential of being able to solve the data linkage across technologies. You have to remember one thing though. Any kind of automated way to basically describe data processes, data flows, will maybe be able to tell you the how a data pipeline has been built, but will never be able to tell you the why. For that, you still require solid documentation on why you calculated the KPI of churning customers only keeping the last 6 months instead of the last 3 months of sales. This needs to be explained in documentation. While the fact that the SQL routine or the routine calculated on top of the last 6 months is something that you can get from the definition of the ETL batch job. Itself.

Matty: [00:31:03] This thinking about the pipeline of data, you know, and the platform upon which you’re presenting this, when— and I’m sure there’s no one way to do anything. Well, apparently there’s one way to make carbonara, but, you know, pipelines may be a little bit bigger. When we’re thinking about this and we’re trying to design these or reason about them, you talked about, you know, security being a factor. You know, we think about, about having them be robust? Because again, this is, this is critical stuff, right? And especially the more complex— it’s like, it’s every distributed system problem times 100. Like, what are some of the things we should be thinking about, right, when we’re saying, okay, so these ideas, but like, maybe get a little bit more into some— I hesitate to say advice, but just things to keep in mind and where that approach is and kind of what’s your philosophy maybe. Around this?

Francesco: Yeah, this is an interesting question because again, it boils down to the question is about what’s next. It’s not only good to start with a technology, you have to ask yourself if the technology can grow with the growth of your business. So I tried to come up with a way to define robust data platforms based on 4 different, basically, directions. The first direction is defining the scalability of a data solution. The scalability is not only the technical scalability that allows you to go from 1 node with 4 GB of RAM to a cluster of 50 nodes, each one of them having 20 GB of RAM. That is one part that you need to, of course, evaluate, but also you need to evaluate if the technology, the set of technologies allows you to segment all the various business cases that you want to implement with that technology. And also if you can scale the humans behind the technology, because you will need people that will support, will develop your solution. And if you If the technology doesn’t have a talent pool at all, well, this maybe is not a problem now, but will be a problem in 5 years. Also because data platforms usually are sticky. I’ve seen a lot of companies and talks about moving from one database to the other. That is always a huge, huge effort. Let’s say, Very few were successful projects that I saw about backend changes. And then one other scaling thing that you may want to double-check is the financial scaling. Because you could find the perfect technological solution that allows you to scale technologically perfectly, allows you to segment all the business cases, it has a perfect community, but But then once you start running, it costs you like 2 times per user the amount of money that you make from the user. So even if it scales perfectly, it’s not affordable. And then this basically solves the scalability problem. The other bit, the other direction that I’m checking is observability, because Building data platforms that are observable usually is not the first thing that you might think about, because you may want to just build and build and build. But then being able to monitor how your data platform works, check metrics and the integration, have alerting in place, have notification in place, becomes relevant, because you are not only building data pipeline for the sake of building it, but you are offering a service within your company. So once you start offering the service, you are responsible to offer a good service. And then going back to what we were saying before, you need to be able, if you want, if you want to keep control over the data flow, to recreate this bird’s-eye view because it allows you to understand how your data flows across the company, who is using what, If someone comes with GDPR queries, well, if you have this bird’s-eye view, maybe you don’t know all the details, but you know exactly who to ask in order to have all the details. And then, since we have been used a lot about using versioning for software, why shouldn’t we use versioning also for data? Or having a way to quickly replay yesterday’s data if we made a mistake in the app. Of course, we are talking about data, and most of the time, real-time or streaming, we want our data pipelines, data technologies to be fast. And fast can go into the time that it takes to develop a new solution, the times that it takes to deliver the result, or to recover from errors. Fast is not just because I I can create a data pipeline or a certain dashboard in one day that it works. Because if it takes one day to build it, but you have always to wait 23 hours before checking the right data into it, it’s not worth it. And this basically drives to the last axis, that is, we want to create a trustworthy solution. And this is specifically a problem specifically for companies that are growing up. Because if you have only one analyst, the Excel file that that analyst has is the unique source of truth. If now you have 55 analysts, each one having an Excel file, now you don’t have a number anymore. You don’t have a KPI anymore. So you want to create the unique source of truth within your company, the golden truth in your company. And also, you want to always provide the data on time as it is expected. Because if one day the KPI is correct at 9 AM and the day after is correct only at 1 PM, then, you know, you’re not building trust on your data pipelines. You’re not building trust on your data. This is when people will go back to extracting from Excel. Into Excel. And one last thing, together with providing accurate data, correct data, you need to provide data in a secure way. Building data pipelines which are observable and trustworthy means that I know that Francesco cannot touch the data, cannot alter the data. Nobody that shouldn’t see the data has access to the data. So these are all directions. Basically, the 4 directions are scalability, observability, fast, and trustworthy. In my opinion, stuff that you should think about when designing, when implementing data pipelines. Some of them will bite you immediately, some of them will bite you in the long term, but at least having the eyes on all the possible directions allows you to compare different set of technologies into all this direction and make a more, I would say, a better decision, a more, how do you say, 360 degrees decision on top of your data pipelines.

Matty: [00:38:52] So that’s a lot to unpack. So I’ve got a couple of questions that something that’s, again, and maybe thinking back to kind of my history, a little bit of, so this sort of data versioning and this replayability and everything, I mean, that seems Seems really, really hard, right? Because especially as we move away from batch, like how does stuff like that actually happen? What are some of the capabilities around that? Like, are there products that help with this? Are there approaches? You know, I mean, I sit there and think about like, geez, even in the batch world, like going, like, you know, restoring the data back to a certain point was well-nigh impossible because so many other things are constantly changing.

Francesco: Let’s say that recreating the data from scratch is never an easy task, or recreating what was yesterday was never an easy task. As you said, there are backups, there are other possibilities. What tools like Kafka allow you to do? Kafka, it’s an amazing but really simple tool. Kafka, it’s basically a log. You write into Kafka, and you write one event after the other in a log. And you can write, you know, shoes now, a pizza after, you know, your next talk after it. Kafka doesn’t care, it just writes one event after the other. But then what you can also do on top of this is to read. If you read from Kafka, Kafka doesn’t delete the messages after you read. This means that I can also read the same messages if I need, because we are 2 different applications. If then you add on top of it the concept of schemas, this means that basically when you write to Kafka, you have to write the messages in a certain format, otherwise Kafka will refuse them. And then there is this concept of schema evolution that allows you to say, I can alter the schema, but I can be sure that all the existing consumers will still be able to parse the message even if I alter— if I— even if I change the schema. And this is a typical scenario where you change something in the app. Maybe, you know, for the shoes, we add a new field which tells It is a boolean if you want your initials on the side of the shoes, yes or no. It’s a new feature that we add in our website, so we change the schema in order to add the new feature. What the consumer, the existing consumer, for example, for billing purposes, they don’t care if there is this new flag or not. If you design your schemas correctly, even if you add a new feature, all the existing consumers can still read the data. The new consumer that will be, for example, the people printing the initials, they will need to use this new flag so they will read the new schema. So going back to your problem, how do I replay events? Let’s say that we added the new schema, but then we discovered that, well, it wasn’t false or true, it was probably black, gray, or white. So we did something wrong there. Well, with the ability of Kafka of replaying all the log, because it can store the log, the data, for, you know, 5 days, for 2 years, for 5 centuries if we want, and with the ability of saying, okay, I want to use the schema 1 or schema 2 or schema version 3 or schema version 4, we are able to feed a new set of consumers with the data that was generated since yesterday. So we have this possibility of replaying the dataset as it was yesterday or 5 hours ago or 1 hour ago.

Matty: [00:43:05] Interesting. See, this is where I just have to change my whole way I think about the world because I’m still stuck back in 1999. But that’s why it’s a good thing that no one’s putting me in charge of their infrastructure architecture anymore. So this has been really interesting. I think there’s a lot more to dig into. I wanted to see, as we kind of come to wrap up here, what’s one of the things though? So I’ve had my misconceptions I’ve sort of talked about. What do you think when we think about most software developers or infrastructure folks or folks who don’t dig in data really deeply, What are maybe a couple of the misconceptions or kind of, I hesitate to say incorrect, but just kind of not quite right perceptions of reasoning about data and data systems that people tend, the traps people tend to fall into?

Francesco: I believe the biggest misconception that I had in my entire career is people may think that data system databases, data platforms is all old and boring. The reality is that the data world is pretty hot. There are new technology, new concepts, new ways of doing the same things or completely new things that are based on data. If you think ChatGPT is data, AI is data. You see, a lot of the innovation now is coming from data. That’s the good thing, but also the bad thing, because you can do innovation with data, but the secret about doing innovation with data is having quality data. And that usually is kind of the biggest problem to solve everywhere. Data is cool. Data pipelines are cool. There is a little bit of work to cleanse the data, but, you know, that’s what we as data professionals are here to do.

Matty: [00:45:06] Well, I think that’s why we’re here. So, this has been great. Head over to arresteddevops.com/data/data/data for this episode’s show notes. And if you go to arresteddevops.com/itunes, And if you leave us a review in the iTunes Store, that helps other people find the podcast. I know it’s not been called iTunes for years, but I don’t want to change the redirect on the website, so it will be iTunes for me forever, just like it’s still the Sears Tower, not the Willis Tower. You can also subscribe to us on Spotify, iHeartRadio, Audible, all those other places where fine and less fine podcasts can be found. So Francesco, thanks joining us today. You’ve given us some things to start thinking about as we maybe dig a little bit more into where our data pieces go. Where might people find you on the internet or elsewhere?

Francesco: You can find me everywhere, everywhere on internet. I’m on Twitter, @ftiziot. I’m pretty consistent with my username. It’s almost @ftiziot everywhere apart from, I believe LinkedIn is the only one where it’s Francesco Tiziot. Also because I love consistency. I’m pretty a data geek. Yeah, find me everywhere. And my DMs are mostly open everywhere.

Matty: [00:46:21] Excellent. Well, thanks for joining us today, Francesco. It’s been a good time. This is Arrested DevOps. And remember, there is always DevOps in the banana stand.