Learn Ilum.
Ship data products fast.

Florian: I'm excited to be here with the real MVPs: The Ilum developers. Hi Dominic and hi Stan!

Stan: Hi Florian nice to meet you!

Dominic: Nice to meet you as well!

Florian: It's great to meet you where are you actually based?

Stan: I'm based in the US LA area, I don't want to be too specific but I know it's a big area, but California.

Florian: And you woke up pretty early today, yes?!

Stan: Especially for you and for this podcast. I already got my morning run in, I'm ready for the day.

Florian: How about you Dominic how are you feeling?

Dominic: Thanks. Feeling great, feeling excited to have this interview. First time for me doing something this a little bit stressed but I think it would be great.

Florian: I think it's super fine to be stressed. For me, also when I started making the course I was like “My gosh, making videos is much work” – and it really is but you get used to it. And the best thing is to interact with my students or in your case with your clients. Maybe we start with a short introduction. Who are you and can you briefly describe what you're doing at Ilum? Who wants to go first?

Stan: Maybe Dominic cause he's working longer here.

Dominic: I can go first. My name is Dominic and I'm a senior engineer at Ilum, backend developer mostly working with Apache Spark that's what Ilum is about. All the messy but fun parts of integrating big data platforms, big data systems. The big part of my job is I would say building the glue connecting Ilum with the external systems. I say databases, object storages, data warehouses, other Spark-related solutions; Airflow for example or even some ML tools: MLflow, Kubeflow, that's most of my work. I also spend a lot of time thinking about how my team can get better. That’s it. Some repeatable pipelines without anything more special. What is cool about Ilum if we talk about what am I doing there is that we're not trying to reinvent the Spark ecosystem. We are helping people to use it better, faster, and in the way that they make it the most easy. Not only a cloud vendor but also on-prem environments, even local machines. I think that's it.

Florian: One of the first modules of the course is actually about MinIO. Was this you and were you the guy behind this integration?

Dominic: I think of MinIO as our base object storage. I've done a big part of this, not alone of course.

Florian: It's great to have you here. And now over to you, Stan, what are you doing at Ilum?

Stan: Thanks, I'm more on the business side of Ilum. I had some engineering background but that's not relevant now because I focus mostly on strategy and growth. Making that we're building something that people want. That means talking to customers, getting the feedback from the early adopters, spotting patterns also and in some pain points they face. Because you tend to see that a couple of clients seem to be struggling with one thing and then we find out that " this is the start of the issue" we're trying to find it out early we're trying to make it better, more marketable and more approachable for people especially since we're still not a we're not a gigantic company we're trying to make people familiar with our product and I also handle a lot of public conversations exploring the integrations opening up new markets trying everything to market Ilum and constantly looking how to expand and where to expand cause at the end of the day it's about connecting the tech we're building with real world problems and making Ilum something that scales but not only technically but also commercially. this is where I fit in.

Florian: It's interesting, because we have the two big parties of a company here: "the engineers versus the sales people". Any conflicts or fun times you've had far?

Stan: Actually we've never met and we didn't have a chance to really spend time together on site of course. We talked a couple of times with Dominic but that's also that takes some – a new approach because we're spread out a bit and... But we've we had some fun. We had obviously some bad moments as every development team and you remember both of those. But mostly the good parts. And it is actually really fun working here. I know that it may sound kind of cliche but it's a new product I've worked on startups before I've never worked on something that.

Florian: It's also fun to make the course I feel maybe it's a little group therapy for the two of you for me joining you because we also haven't met and of course for all the students in the course. Stan how long did it take you to understand the Ilum value proposition and how do you then translate it to the world that you're working with the clients?

Stan: Actually the story was that Ilum was actually born for a specific case and that's when we actually started to notice – the guys before – ‘cause we talked I'm not here since the beginning I started to notice "but we're using it it..." ... how do we make it more approachable? How other people can use it? Because people have different tastes in their data platforms. We had to figure out let's get some more things on this. Let's make more plug ins, let's give people more functionalities and the problem of it was that it eventually became very bloated. We had too much of it and it wasn't really you couldn't upkeep it. Obviously, we had to find some solution to it. The solution was quite easy actually ‘cause we went to modular. And in that way, we allowed people to: “I don't want that, I want only this and that”… People can actually modify and set up Illum however they want and I think that's how we got from the from this frustration that Dominic was talking about and made it a more, how to say ... a broader experience for a broader party. Not only us can use it on but other people, too, and we're listening to this feedback, we're constantly trying to give you guys and everybody who's using our platform the things that you need for your work.

Florian: I know. This is something that I've seen. There's many different features and now it's all unicorns and rainbows.

Stan: Ha, ha. Not everything but we had to find something that is open source also to avoid, … and that is used broadly. We're actually using stuff that is, let's call it, industry standard. You could call it "rainbowy" "unicorny" kind of a thing. I'm not on the engineering side anymore too strong. For me it's all new funny names to be honest.

Florian: Saves people, unicorns, rainbows ... but I know from working in a startup too behind the scenes sometimes some stuff is very difficult to solve. What is the challenge that you faced when building Ilum and how did you solve it?

Dominic: In terms of people, I really feel very good very good in this team. I have a very good , great connection with my boss and with all of my other colleagues. I can't complain on this part. Maybe in terms of developing it is my first bigger project that I'm working on and I faced a lot of situations where there wasn't really a place to find the knowledge to solve the problem. Doing a research in some different places was an issue but always with some help of my team and myself, I would say it always came out, I would say. And when building something, I get a lot of satisfaction out of technical solutions that end up working in the end.

Florian: Can you remember something where that you're really proud of when you build Ilum and you're "We finally did it! This is awesome!"

Dominic: Yes, I can. Doing some tricky things gives me a lot of fun sometimes that's ... something shouldn't work but it does at the end. This is when you are trying to integrate with different tools sometimes this isn't allowed out of the box, you need to do some custom stuff, a custom job that isn't available anywhere. That's the part of this job that satisfies me the most. For example, integrating with Spark, replacing Spark's internal mechanism that you can use… Using Spark as it is but putting that into our system. For me it is the best. And when you do something like this and you do this successfully.

Florian: What's a place in the Ilum UI where people can really see and appreciate all this engineering work you did? Do you have any particular place where you're "this you should know, this was a lot of work but we did it and here it is”?

Dominic: I don’t know if what I will say is correct but I think that it is the best when the user doesn't know that you did it. Something looks native and when you use it you don't see that. But because for example it was native for Kubernetes and I did it not with Spark and I did it not in native Spark but in Ilum, in our system, the way that you don't really recognize the changes. I think I would turn this around and say something like that. I think that's good?!

Florian: That's good to know. Stan, you talk a lot to customers who may be getting in touch with Ilum for the first time. What's a tip you would give to somebody who's getting started with Ilum?

Stan: The easiest way to actually to get help is through the ticketing platform that we have. You can find it on our website ilum.cloud. It's open to everyone and we're actually monitoring it. It's... I think it's the best way. However, if you're not into the ticketing system, you don't want to check that extra website to check up on, you can send us an email to support@ilum.cloud. We're gonna answer this way, too. It's up to you, really: Add a ticket in our platform or email us directly. Actually, as you might see that we're a pretty hands-on team and we will try to respond quickly when it's a setup issue, feature question or something that is not behaving the way you want it and we're actually pretty keen on hearing this feedback. If you've got something then please let us know. We're very interested in hearing this.

Florian: And you're also monitoring the conversation in the course community inside the course. That's also great and I love that. Thank you for that. Dominic, do you have a tip for people who are getting started with Ilum?

Dominic: From my side, the developer side or the user of Ilum side, because I'm also a user of Ilum, of course, during the development phases and also after this my No. 1 tip would be to get to know Kubernetes a little bit more than you currently do. Getting knowledge about kubectl, using the kubectl better, even some basic commands: Listing pods, describing the existing deployments, … it will give you a lot more confidence in using our platform. Because, of course, we work hard to make it as easy as possible to install and run Ilum but it's still a Kubernetes-native data platform. If something goes wrong or you're curious about what's going on under the hood, simple commands will let you fix those problems. And if it's not a problem, if you want to know the platform better, it will also help you. I suggest to spend an hour or two because it's not any rocket science. It will pay off in big, I think.

Florian: I had some students struggling in with installing Ilum because you need to install Kubernetes and all that stuff. Do you have a resource that you can recommend to refresh your knowledge on Kubernetes?

Dominic: I think I will not go out of the Kubernetes documentation. Of course, we use Helm as a package manager, it is also crucial to know something about Helm because without this knowledge, you are practically unable to use the platform. And if you get this knowledge, that will take an hour or two, I guarantee you it will make a huge change. That's good to know, people should really get more deeper into Kubernetes and understand what's going on to leverage Ilum.

Florian: I had a different experience because you made a cloud deployment for me and I didn't have to touch Kubernetes to get started and I found it incredibly easy and smooth. Thank you for that. The technological possibilities are endless out there especially when it comes to open source, especially in the data world. I think there's a lot to look forward to. But I'm interested to hear from you or maybe Stan: What are you most looking forward to in the Ilum world in the next couple of months?

Stan: We're hoping to score some big clients and we're hoping to spread Ilum and that's our approach. I know it sounds simple but it isn't, obviously. But I'm actually pretty thrilled because I joined a year ago and this product that we're currently marketing it's a, I wanna say, it's a completely different story but the amount of work that's been put in by Dominic and other engineers, that's tremendous and I noticed this. I have read, I believe, from most of the release notes ever since I joined and what the amount of features the amount of thoroughness that that the guys are… the time spent on this – it's incredible work and I'm really proud to be a part of it. Next month I really hope to look forward to other features that we're introducing and we'll see how that goes.

Florian: Dominic says "Stan is happy, my work is done, I'm gonna chill on the beach now." All the difficult tasks have been solved – is there something that you're particularly looking forward to from the technical perspective Dominic?

Dominic: Of course, there is always place to improve. I am currently working on those things. I think what I'm really excited about is expanding this integration layer. Connecting more catalogues, more Spark-related tools or data-related tools. Because as a lot of things work in complex environments where data lives in different places and if we could smoothly plug all of those into Ilum and let users use it from this one place, I think it makes everything a lot easier for users. That's the direction I'm most excited about .

Florian: That sounds exciting indeed. I think we have two great perspectives here: The outside-facing and the inside-facing and I really wanna thank you for that. Do you have any parting words, anything we maybe didn't talk about in this interview or I didn't ask you that you really wanna say?

Stan: I think I'm good. That was really good questions, thanks for that. I wanna say "Try Ilum!" It doesn't cost anything, it's free, why not?

Florian: Dominic, do you have any parting words?

Dominic: I think I'll tell the same from my side. Try Ilum because I would like to build the product that is widely used. That always gives you a satisfaction when your work means something and someone in the other place of the world is working with your solution.

Florian: Yes, thank you so much. This is beautiful and I thank you very much for having this interview with me and for giving the students of the course also a chance to learn more about you: The people behind Ilum and people can reach you at info@ilum.cloud. Thank you so much!

Stan: Thank you so much, Florian, it was a pleasure.

Dominic: Thank you as well!

Florian: Hello and a warm welcome to this interview, Widy. I'm so excited to have you here. Who are you?

Widy: Okay, thanks for the opportunity. My name is Widyawan my complete name. You can call me Widy. I'm from Indonesia, it's a country in Southeast Asia. I am a teacher, I am a university lecturer in a city called Yogyakarta. Yogyakarta is kind of one hour flight from capital, which is, Jakarta in the, Java, island. I teach in engineering faculty in my department. It's called electrical engineering and information technology department. So that's, short, introduction of myself.

Florian: Thank you so much. I think it's super awesome that Ilum is being used in a university. And you said to me before the meeting that Ilum is already being used in student research. Can you share some examples of how students are applying Ilum in their projects?

Widy: Okay. Previously, we used another platform for big data. Mainly we focus on open source platform, right? But if I'm allowed to name a brand, we used to work with Ambari. But as we know it, it is discontinued. So, because we have to provide infrastructure for the students. So we try to search in the internet, of course. Is there any kind of easier platform? Because when you set up Ambari, it's kind of not easy, you know, I ever installed myself. When I search, the keywords is like Kubernetes, big data platform, then ilum.cloud is one on top of it. Yeah, so actually it is kind of, okay, what is this? I don't know. Then I tried to email the contact email in the website and surprisingly the next day they answer, you know. Actually, yeah. My question is simple. What is Ilum and how to use it? And then, and then, okay, you can download from this, this, this, and then, I download and then, we have to enter a kind of, serial numbers. I email again. What's the serial number? How do you, how do I get it? And then, okay.

Florian: They're very responsive, yeah?

Widy: They email again, okay. Surprisingly, it's very responsive for, I'm not a customer, right? And I'm not, not paying. So there's no immediate commercial incentive for them. So that's interesting. Maybe, it's suitable for academic purpose. I, I just say research. And then I, install in my small cluster in my university and we chat quite a bit and they also asked what kind of technology that should include in Ilum? So they asked, I was wondering, okay, like before, I'm kind of big, so why you ask me? I'm not your customer, but I give some advice, okay, you have to, maybe for academic point of view, it's interesting to install because that time I'm working with MPD, mobile positioning data. Maybe it's interesting to include kind of library to, deal with, spatial data. It's called Apache Sedona. I tell them. Maybe you need to install that one. Then, later on, maybe it's interesting to install deep learning library in Spark. And yeah, the email exchange is quite a lot, I think. Maybe more than 20 emails, through and forth. Okay. And then they offer me, I mean my university, offer me some kind of cooperation, can be in education and research. So we were signing the MOU thing like that for research and education. And we start with research first, later on education because it's related to the semester schedule, I think, of the course, but we start with this.

Florian: Okay, so first research and then education?

Widy: Yes, yes. First research and then education. And what kind of research? Because, yeah, the contact is, you see, it's with me first. So that time, my laboratory, my research group, we have collaboration with oneof the operator in Indonesia. But one of the biggest three, we have collaboration, they share their data. They share their data and the data is quite big at that time. More than 100 terabytes. There's a lot of data. Transfer to our university data center. Firstly, like I said before, we deal with Ambari platform using that data with Hadoop, but in bare machine, know, it's kind of difficult to maintain, especially if you're in the university, we don't have many people dealing with those platforms, right? Not many engineers.

Florian: And you can do that with Ilum.

Widy: Yes, yes. mean, personally, I'm trying myself. Okay. I install Ilum. Okay. I try to set up and then try to read data. Not all the data, not in this yet, but at least, okay, I can do it myself because for us, or at least for me, I have to try myself first before I tell the student do this and do that. And then after it's installed, come the students for research.

So your question, what kind of research? So it's still in the early phase because it's like we use for maybe one semester or six months. But firstly, we have students it's a master student now who deal with infrastructure means compare. Actually, you see, compare. Usually, we use Python, in single node to do ETL or data extracting and cleansing, maybe some transform. Mobility data, MPD. So mobility data is like you have ID and you have time and you have latitude, longitude.

She tried to do ETL in a single machine with maybe library, scikit-learn and et cetera. And he compared with the, if we use it in big data platform like Ilum, so using distributed system, using Spark So I think the first research is about infrastructure or base.

Florian: And do you already have results about that research published somewhere?

Widy: Yeah, yeah. So she finished the study, she write the thesis about publication, I have to check about that one, the first student. But the second student has publication.

Florian: Maybe we can link that also together with this podcast.

Widy: Sure. I have to look for it first, the publication. I went in international conference somewhere in Bali, I think. Yeah. And then, so there's a two about infrastructure and it asks that the third is about database. So database comparisons. So Ilum has, I think MongoDB there. There's a MongoDB in Ilum. There's a Hive. my third student is database performance comparisons. a database who, who in the Ilum platform with other database, so for instance, compared to, I don't know, maybe Postgres or a click house, things like that. And that's the third. The fourth is, is more about analytics, like I said, because we have mobility data. So one of students is trying to predict next position based on the previous data. So because we have that –

Florian: And is that in a distributed way like with Spark or how is that implemented?

Widy: Yeah. Yeah. Usually, we compare with Spark and without Spark, to predict, next position with machine learning and deep learning. And we'll see the result. Still, an ongoing research.

Florian: That's interesting. You have like the inside out view of Ilum. You look deep inside how the technology works and then also how you can apply it. That's pretty cool. And you said earlier that yes, you do some research, but next semester you're planning to use it also for teaching. How do you envision Ilum can support your teaching goals?

Widy: Yeah, yeah, yeah. So, yeah, why next semester? Because data related course is in the first semester. We have course of data engineering. And also, we have course of big data. Yeah, that's the main two courses, a course for, undergrade student, and master student, have like a data warehouse and machine learning as well. So there's at least three courses. Ilum give us, what's it called? Cloud platform for, for free. Yeah. And then I can assign my students to using that platform. So, it's quite helpful. So how we use it, we provide students access to the cloud platform provided by Ilum. So we don't have to maintain the platform ourselves. And then, firstly, I think next semester, we can assign user account to lecturer and the student. So we made account for the teacher and also the student. In data engineering course, we teach them how to extract data, how to clean data, how to know the data, or maybe make some transformation, simplification, dimension reduction and et cetera. So first, of course, we have to give introduction. What is data? Theoretical-wise, and then, the best thing to choose, I think, the student has to experience themselves. They have to do it themselves. Maybe they, they can try with their laptop, maybe with Jupyter net notebooks, something like that.

Florian: But it's not the same, right? Like cloud platform versus local notebooks, there is a difference, especially when it comes later to terabytes of mobile communications data. You can't do that on your laptop, right?

Widy: Sure. Yeah. I mean, they tried with small data as well first. And then, I think in Ilum platform, also, they have Jupyter Notebook and also small data. So they can try the small data first. Then, later on, I mean, in maybe half of the, course, we should talk about distributed system, right? Distributed system, distributed data engineering system. I think Ilum will come to the course like that. So in data engineering, start with small first and then with the big part of data and also the Ilum part. That's how I think we envision. And I think secondly, it's important for students to have exposed to enterprise grid tools. So those which is used by enterprise. So I think it's good for them.

Florian: Yeah, I agree with you. think, you know, when I was a student, it was really hard for me to understand how stuff works in the real world. And I think with something like Ilum, you can use it all the way through with small data, big data and beyond. And especially in the environment that you're in, I think you really get this inside out view because you also have the research part. 

Florian: In your experience so far, also you said really that you played around or worked with Ilum yourself. What do you think, which benefits does Ilum provide to your students in that regard?

Widy: Difficult question. Maybe I should ask my students. But I think that distributed computing platform and also distributed file system platform. I think will make a difference and will give greatest benefit to students because it's not easy for them to set up by themselves, know. Student of course has laptop, but I mean to set up a distributed platform where you have many servers, where you can exchange data between server, exchange computation. I don't think they can do it by themselves in their home. So, I think that's the greatest benefit for students. I think secondly, because it's proven in terms of enterprise, so they will have confidence, Maybe sometimes in the future, they will ask by their employer, what kind of platform we should use in their company.

Florian: Yeah, no, I totally see that. I think knowing how everything fits together is very valuable. Deep and wide: Somebody is just getting started with Ilum, which tip would you give them?

Widy: Which tips to give them? It's interesting, you know, when I have a colleague, I give him access to Ilum Cloud and said, whoa, this, you know, it's kind of, it's powerful, many features, there's many, modules, many capabilities, which is sometimes says, kind of overwhelmed for when you come as you know.

My advice is you read Ilum first, what is Ilum? They have in the website and you go to the Jupyter Notebook part. In Jupyter Notebook, they have kind of good documentation. Also there's an example of code, how to first extract code and then some play with the data because they have architecture to retrieve and to analyze. So usually I advise go to the Jupyter notebook, try this notebook and let's see, you can start from there.

Florian: Yeah, really, I'm excited to hear what your learnings are in one year when you've one year of students going through this and then like even more time of research. Thank you so much for your time today.

Widy: Sure, sure.

Florian: It has been a pleasure interviewing you and where can people get in touch with you?

Widy: They can email me. Of course, they can WhatsApp me if, they want but email is, good email at [redacted]. You, you email me. So, right. You see, I, reply if I'm not in holiday.

Florian: Okay, okay. Fair enough. Thank you so much, Widy!

Widy: Okay, thanks.

Florian: That's interesting. You have like the inside out view of Ilum. You look deep inside how the technology works and then also how you can apply it. That's pretty cool. And you said earlier that yes, you do some research, but next semester you're planning to use it also for teaching. How do you envision Ilum can support your teaching goals?

Widy: Yeah, yeah, yeah. So, yeah, why next semester? Because data related course is in the first semester. We have course of data engineering. And also, we have course of big data. Yeah, that's the main two courses, a course for, undergrade student, and master student, have like a data warehouse and machine learning as well. So there's at least three courses. Ilum give us, what's it called? Cloud platform for, for free. Yeah. And then I can assign my students to using that platform. So, it's quite helpful. So how we use it, we provide students access to the cloud platform provided by Ilum. So we don't have to maintain the platform ourselves. And then, firstly, I think next semester, we can assign user account to lecturer and the student. So we made account for the teacher and also the student. In data engineering course, we teach them how to extract data, how to clean data, how to know the data, or maybe make some transformation, simplification, dimension reduction and et cetera. So first, of course, we have to give introduction. What is data? Theoretical-wise, and then, the best thing to choose, I think, the student has to experience themselves. They have to do it themselves. Maybe they, they can try with their laptop, maybe with Jupyter net notebooks, something like that.

Florian: But it's not the same, right? Like cloud platform versus local notebooks, there is a difference, especially when it comes later to terabytes of mobile communications data. You can't do that on your laptop, right?

Widy: Sure. Yeah. I mean, they tried with small data as well first. And then, I think in Ilum platform, also, they have Jupyter Notebook and also small data. So they can try the small data first. Then, later on, I mean, in maybe half of the, course, we should talk about distributed system, right? Distributed system, distributed data engineering system. I think Ilum will come to the course like that. So in data engineering, start with small first and then with the big part of data and also the Ilum part. That's how I think we envision. And I think secondly, it's important for students to have exposed to enterprise grid tools. So those which is used by enterprise. So I think it's good for them.

Florian: Yeah, I agree with you. think, you know, when I was a student, it was really hard for me to understand how stuff works in the real world. And I think with something like Ilum, you can use it all the way through with small data, big data and beyond. And especially in the environment that you're in, I think you really get this inside out view because you also have the research part. In your experience so far, also you said really that you played around or worked with Ilum yourself. What do you think, which benefits does Ilum provide to your students in that regard?

Widy: Difficult question. Maybe I should ask my students. But I think that distributed computing platform and also distributed file system platform. I think will make a difference and will give greatest benefit to students because it's not easy for them to set up by themselves, know. Student of course has laptop, but I mean to set up a distributed platform where you have many servers, where you can exchange data between server, exchange computation. I don't think they can do it by themselves in their home. So, I think that's the greatest benefit for students. I think secondly, because it's proven in terms of enterprise, so they will have confidence, Maybe sometimes in the future, they will ask by their employer, what kind of platform we should use in their company.

Florian: Yeah, no, I totally see that. I think knowing how everything fits together is very valuable. Deep and wide: Somebody is just getting started with Ilum, which tip would you give them?

Widy: Which tips to give them? It's interesting, you know, when I have a colleague, I give him access to Ilum Cloud and said, whoa, this, you know, it's kind of, it's powerful, many features, there's many, modules, many capabilities, which is sometimes says, kind of overwhelmed for when you come as you know.

My advice is you read Ilum first, what is Ilum? They have in the website and you go to the Jupyter Notebook part. In Jupyter Notebook, they have kind of good documentation. Also there's an example of code, how to first extract code and then some play with the data because they have architecture to retrieve and to analyze. So usually I advise go to the Jupyter notebook, try this notebook and let's see, you can start from there.

Florian: Yeah, really, I'm excited to hear what your learnings are in one year when you've one year of students going through this and then like even more time of research. Thank you so much for your time today.

Widy: Sure, sure.

Florian: It has been a pleasure interviewing you and where can people get in touch with you?

Widy: They can email me. Of course, they can WhatsApp me if, they want but email is, good email at [redacted]. You, you email me. So, right. You see, I, reply if I'm not in holiday.

Florian: Okay, okay. Fair enough. Thank you so much, Widy!

Widy: Okay, thanks.