On this episode of Gradient Dissent our guests are Angela Bassa and Danielle Dean!
Angela is an expert in building and leading data teams. An MIT-trained and Edelman-award-winning mathematician, she has over 15 years of experience across industries—spanning finance, life sciences, agriculture, marketing, energy, software, and robotics. Angela heads Data Science and Machine Learning at iRobot, where her teams help bring intelligence to a global fleet of millions of consumer robots. She is also a renowned keynote speaker and author, with credits including the Wall Street Journal and Harvard Business Review.
Follow Angela on twitter: https://twitter.com/angebassa
And on her website: https://www.angelabassa.com/
Danielle Dean, PhD is the Technical Director of Machine Learning at iRobot where she is helping lead the intelligence revolution for robots. She leads a team that leverages machine learning, reinforcement learning, and software engineering to build algorithms that will result in massive improvements in our robots. Before iRobot, Danielle was a Principal Data Scientist Lead at Microsoft Corp. in AzureCAT Engineering within the Cloud AI Platform division.
Follow Danielle on Twitter: https://twitter.com/danielleodean
Angela: It's really nice to be here talking with you guys. At iRobot, we've had an interesting evolution of how we approach machine learning within the organization. And it's really quite significant to have this partnership because Danielle is just amazing. And I really, really, really love working with her. The two of us have really good complementary styles. My background is a bit more towards theoretical and applied Math and I grew up in modeling of systems; so modeling soybean trade introgression in agricultural settings, modeling epidemiology processes within certain geographies or disease spaces. All of that is very distinct from robotics but it turns out that a lot of the tooling, and I have to bring in agriculture again, but there is a lot of cross pollination that's very useful. But Danielle also has a really different background that I'll let her talk about which gives robustness to how we approach these problems.
Danielle: I'm super excited to be here today, and I thank Angela for the introduction. It's been great working at iRobot and thinking about building the machine learning organization. I think one thing that's been really great is thinking about how we bring different perspectives and different people into the skillset. So people from computer science background, people from statistics background, people from geology, biology and chemistry who have had really different experiences. It makes thinking about building a team that can solve real world problems for our customers really exciting.
Lukas: So for example, when you two work together, can you think of a time that you brought different approaches to a problem? Or how would you explain the difference between the way you think through things?
Angela: I grew up working in investment banking and strategy consulting, and then marketing organizations. So my background is a little bit unorthodox, but very real world oriented, which has been helpful. And one anecdote that I like to highlight to this end isn't my personal one, it is another leader on our team; Theresa Borcuch who manages the data science team and has a marine biology background. When we have been thinking about how is it that we analyze data that's coming from teaming missions, we have robots that can work in tandem. There's the mopping robot and the vacuum robot. So you can have the vacuum robot your room and then go to a different place. And then the mopping robot comes back afterwards and completes the mission. And so she had a really interesting way of thinking about that problem because she had done a lot of research with pods of dolphins. So she's looking at the artifacts of that system because dolphins can't tell you what their intent was, they can't tell you what they are trying to do, but they are acting and through sensors and data collection, you can get the artifacts of that and then do analysis to try to to derive deeper insights. Theresa was able to bring all of that knowledge. That literature exists, right? That knowledge exists and she was able to think about it differently and really enrich the way that our team dealt with that. So we do have some bias towards real world experience and ways in which we understand that we're looking at fossils. And so when we project what the dinosaur looks like, our dinosaurs tend to not have cartilage and feathers because we wouldn't have known that just from the fossils. But we know that the real world is richer than what we're looking at and I think that allows us to build more solid answers to how we use these ML applications.
Lukas: Interesting. Do you have anything to add?
Danielle: That was such a good story, I just can't add to that.
Lukas: It's funny. I mean, just as an aside, I remember the first machine learning team I worked on, my boss, John Mark, he always said that he'd like to hire a biologist because they look at the specific examples versus. He actually was a physicist, and he's like, I hate having other physicists because they always try to look in generalities instead of being specific. And he felt like what the team really needed was more people to look at the actual training data, the specific things that were feeding into the model. So it's funny, I hadn't thought about that in fifteen years, maybe, but he always was. He was always talking about that.
Angela: Yeah. And it's huge in our domain because we have these robots deployed literally in every timezone globally, all across the world. And you know, I've never lived in every place in the world so we will look at the data, we will see information, and I will construct a mental model of what might be happening. But if we don't have a team that's robust and can challenge a lot of our heuristics and baked-in assumptions and go, "I don't think that means what we think it means"... And that enforces exactly what you're talking about, descending into the particulars and really examining what it is that the data is telling us, not what we hope it aligns with, because we have this perfect first principle's model of what it should be.
Lukas: So iRobot has been around for a long time building robots and I think robotics, traditionally didn't do a lot of machine learning. I mean, I think it's always obviously relevant but it wasn't machine learning first until maybe recently. I'm curious about the evolution of, if you know it, of iRobots thinking about ML. When did you start to think, "Okay, we need to build ML models and get them in production"?. When did you start building an ML skill set at iRobot?
Danielle: I think it's an interesting thing about iRobot history when they first started over 30 years ago now. Almost every single part of the solution had to be made by iRobot thinking about, from the navigation stack to the hardware stack, to all the software pieces, to how the robot navigates the world and how it uses all that different sensor information. But as the field has changed a lot, and especially as the AI field has changed and deep learning has really transformed and the quality of solutions that can come from machine learning have really transformed. IRobots looked at, "hey, there's solutions out in the industry that we can leverage on and we can improve our products using those skill sets and expertise". So I think machine learning has been at iRobot for quite a while now, mainly started in research areas and thinking about how can we use that research, how we can start these research projects and think about how we can improve the quality of our navigation stack? how do we improve the quality of our cleaning solutions,how do we improve the quality of our robotics overall? But just recently, in the last few years, it's moved from a research stage to actually being in production solutions. Everywhere, from improving our digital experiences in the applications to improving our hardware solutions for things like navigation and cleaning experiences. So it's been interesting to see the journey. And I think machine learning is starting to be a bigger piece of the iRobot solutions moving forward.
Lukas: When do you think it first became a production thing? And what was the impetus like? What what was the application that drove it?
Angela: Initially, I think the first application that we could really call ML, if not ML adjacent, really ML, is Slam. Which is the localization and the mapping of how the robot navigates its space and that uses a lot of the same methods and a lot of the same, now modern tooling. Back then, a lot of the tooling had to be invented and by iRobot too. And so I think the legacy of ML applications in production at iRobot, if you include the same component, is actually a quite longstanding one but in terms of the more typically defined ML, I think it really wasn't something that could have been as important a part of the strategy as it is right now until the robots became connected. So for a very long time, these robots were completely self-sufficient, closed systems. They came out of the manufacturing line, onto an inventory somewhere, into a customer's home, and then it just worked and there was no way to either collect any information off it or to send information to it to modify its behavior. In late 2015, when we started having the IOT connection component to the stack, that really opened the door to improved data collection, which is really the thing without which you can't have meaningful ML that actually does something that feels auto magical to the customer experience. So I think that's, from a historic perspective, where iRobot started using ML in the production environment and I also think that one thing that shapes the iRobot story quite deeply is exactly what Danielle was saying; is that because a lot of the initial tooling was built in-house, we have had a really interesting blend of standard tool choices versus the deep customization that's really part of the DNA of the company. So that's one of the things that has been really interesting over the last two years, especially once this really proved itself internally. And we've been able to demonstrate just how valuable this is, both for our internal use just to improve our own software development capability, but also to help shape where the strategic roadmap might go, as well as all the other ways in which we can use ML answers to improve the business.
Lukas: That's so interesting. So it was really Internet connectivity that made ML applications possible.
Angela: I'm not sure if it's "possible", but it's what finally made the case that even if it had been possible before, it might have been prohibitive to collect enough training and validation data for these things to be robust in a way that will work on an apartment in Singapore, the same way that it will work on a ranch in Texas, right? So the ability to really collect a complete picture of what it is that these machine learning algorithms are going to be interacting with, it really didn't make sense because we wouldn't have been able to meet the magical delightful expectation that our global customers might have had.
Lukas: Cool. So tell me about your team, what are the different people doing? What are the relative sizes of investment and different modeling, deployment.? Are you mostly researchers? Are you mostly engineers?. How do you think about that? What are the different functions and how do they all work together?
Danielle: The team has changed a lot over the years as we've shifted from more of the research side of proven concepts of machine learning application that we can do, to thinking about production applications. So we have a team that's dedicated to thinking about machine learning algorithms and models, thinking about how we develop those models? What is the appropriate type of data to feed into those models and how do we improve them over time?
Lukas: So that team would essentially be machine learning practitioners. I mean, how would it? Apparently they're using ML mostly?
Danielle: This is the only part of the team that will be stereotypical ML. So I'll talk a little bit more now of the other parts of the team. When you talk about a machine learning team, everybody thinks, OK, that's people doing algorithms, right? We do have some of those people. But its like..
Lukas: Can you say a little bit about the relative size of these teams. I think it's just really interesting that you're in production and doing the stuff.
Danielle: Yeah. So the modeling team out of the machine learning team all up is actually of a small size, I think that they're about a quarter of the size of the full team. So the modeling team, they're doing the algorithm development and then there's complementary teams that work beside them. Another team that's working is on more of the integration side. So how do they take the algorithms and actually deploy them to the robots? They are thinking through what is the supporting infrastructure with the bottle conversion process to run on this limited hardware. Obviously, we need to not use the same types of models that you can do with big GPU machines so thinking about the integration of these algorithms onto the robot,
Lukas: What do you call that team? Is that like ML ops?
Danielle: We actually call it an integration team. Integrating into the robot software and the surrounding infrastructure around it.
Lukas: Are these people kind of hardware-ish people?
Danielle: They're still mostly software but have an understanding of the hardware too and they can work with folks in hardware on specific applications. And one interesting part about doing machine learning applications on robots is it's not just the ML algorithm, but it's how it interacts in the system as a whole. So these are the folks that really need to think through the end-to-end solution of how and what the metrics of the system are? How does it affect performance of the robot at the end of the day, things like we want cleaning coverage, we want robots to clean all parts of their house, we want to maximize these metrics from the customer's point of view, not the algorithm point of view.
Lukas: So are some of these people more business focused?
Danielle: They're not but they're thinking about how to capture the metrics to report to the business folks and thinking about what those metrics are. So thinking about the system level metrics rather than just the individual algorithm metrics.
Angela: And just to complement that, it's really iterative. So we have an R&D organization and then a product organization and they're sister organizations. So Danielle and I are on the R&D side of the equation and there are the teams that she's going to continue to talk about. There's the four teams within the machine learning proper, and then each one of those has a counterpart within R&D that's attached to a program. So if it's a specific robot or a specific feature or a specific customer deliverable, there's somebody that's integrating that machine learning component with all of the other R&D components that are necessary to deliver the complete final box solution, either as an over the air update, an OTA or to the manufacturing frame work. On the third leg to that stool is the product team. So we have business counterparts who own that product feature deliverable from the business standpoint. They may not know what's technically feasible or they're actually incredibly robust at iRobot because we do have a bias for technologists so that tends to not truly be the case but they have these R&D partners who can help them iterate on how do we define these metrics. Do we care only about end states and lack of interstitial errors or do we care about that within the context of cleaning coverage and cleaning efficiency and all of the other low abandonment rates, all of the holistic experience of what this thing is supposed to deliver? And so that spans the gamut but for instance, on the first team that Danielle was talking about, the modeling team, the way that they talk to their business counterpart is almost similar to the way that they talk to the integration team. So they can write models and write papers and then tell the integration team that "this is the aha, this is the thing, the solution" but they are not as constrained with, "What's the compute accounting? How much budget do we have to be able to run this as the robot is doing all of these other things that it actually has to do as well?" And so it truly is a really well integrated and iterative process. It's not that they go into their nerd cave, they come up with a solution and they come out and toss it over the fence.
Danielle: Yeah, definitely. So after the integration team, there's a team that's focused on actually what you mentioned before. Look at the platform and operations teams that's separate from the integration team and this is the team that is helping with the infrastructure that supports both modeling and integration work. So things like, how do we scale out our compute? How do we train our models and store models and things like that?
Lukas: So how do you think about the different skillset between that team and the integrations team?
Danielle: So the integration team is a lot in our case because we're deploying on a robot, therefore they also have often C++ knowledge to be able to integrate the models into that environment. They have built system experience around the robot software. So the integration team is more focused on how do we make this stuff real on the robot versus the platform team is how do we build supporting infrastructure to support both of those other teams? And the platform team is, generally speaking, more cloud focused. We use AWS at iRobot for a majority of our applications and so how do we build serverless applications on the cloud to support those different applications.
Angela: We found that having an integration team separate from the model and separate from the architecture and separate from advanced solutions is really important. Because of this, the last mile of deploying machine learning to production is really long. And we have all of the added complexity that our last mile is incredibly restricted, automated robotic sensor, autonomous platform that's deployed in environments we can't control and we can't really modify. So having a team dedicated to focusing on just how hard that last mile is has really paid off in terms of, yes, if it's added complexity, having a whole another team added specialization like we, that's not free, but it's definitely been a net positive.
Lukas: And what's the last team?
Danielle: So the last team is a reinforcement learning team, they specialize in reinforcement learning applications. And they work closely with the integration team and the platform team for the same applications as the modeling team, just the modelling team is focused on other applications.
Lukas: Like a team for researchers essentially.
Danielle: Yeah. Researchers and making reinforcement learning real for iRobot.
Lukas: Interesting. So among those teams, which is the hardest team to hire for?
Danielle: Oh, that's tough. Oftentimes it's these little gaps in between the teams and these things that you don't expect you need and then you find you need in going through a project, and one often overlooked part is aspects around how do we deal with data and how the data feeds into the rest of the systems. So even folks who are supporting the team in how do we use the data and how we curate the data in such a way that our models can improve and our systems can integrate. Often it is the work in between the cracks that's the hardest to figure out. How do we fill these gaps when there's no clear work that the other teams are working towards?
Angela: One thing that I will note, though. Two things actually. One, we are hiring. So if you have experience in embedded systems and machine learning, toolchain, automation, let me know, we're very much hiring. But also, one thing that's amazing about iRobot is that it's actually not that hard to hire. I mean, yes, it is, because this is a very specialized skill set that we're looking for, given that we've reached a level of scale where that specialization pays off for us. But iRobot is really a cool place that's really well connected. We have a lot of really smart people working alongside us who have very broad networks. So we actually get a lot of really smart, really competent folks reaching out to us a lot, which I'm very grateful for, because as you've noted, these tend to be really hard skill sets to find. But also, everybody, even six year old gets tickled when they get to answer, "What do you do for a living?" "I build robots." That really doesn't suck.
Lukas: I'm curious. Do you notice that there's any challenges with cultural differences or miscommunications between these teams? I think you actually do hire fairly different people, has there been any kind of lessons learned from trying to get everyone to work together to build something?
Danielle: I think one really important thing in thinking about building machine learning teams, especially when you get to the point where you have different specializations and different goals of these sub teams, is thinking about where the boundaries are between teams and how handover happens. So one way that we've tried to address this at iRobot is we've built virtual teams that work across team boundaries. So for a particular application, there's somebody from the modeling team working on it, there's somebody from the integration team working there, somebody from the platform team working on it and so they're there like a virtual team that works across boundaries. Because if otherwise, there could be some gaps in handover between teams. So trying to build these integration points between the teams is really helpful in getting an end application out the door.
Angela: Also it's really helpful to have these teams not feel like there are these arbitrary walls in between them so the virtual teaming that Danielle described is one way that we do that. But making sure that the folks get a lot of time together, discussing scientific papers, getting together and talking about recent developments, attending conferences together so that there's really this strong culture that allows for a lot of that connective tissue to happen so that the gaps don't exist. Things do fall through the cracks, there's no way to avoid that. But the fact that we know it will happen, we create a culture where folks are attentive to it and intentional in seeking those things out to catch them rather than going, "Well, it's not my problem". So I think the fact that Danielle is really intentional about building that kind of cohesiveness, that I'm really intentional about paying attention to that kind of cohesiveness is one of the things that has been paying off in spades for us in terms of the speed with which we can get these things to happen and not have to constantly take two steps forward, one step back of "what didn't make it this last time?"
Lukas: Is it a challenge for you to weigh the priorities of what the company needs in the models? The company needs versus paper writing and academic conferences, has that been an issue for you?
Angela: Imagine..But that has been the case for 30 years; you hired the best and the brightest. You hire people with a bias towards nerdery. And obviously, all of us want to be big bad nerds, and big bad nerdery doesn't necessarily always pay the bills. So that's why having that iterative component and having a sister organization in the products team, that is well plugged into what the end goal is, because I think once we reframe how the team talks about it and we stop the dichotomy of "are we doing cool research or are we doing boring product work?", when we reframe it to 'are we solving a real problem?' Not an abstraction. Not a theoretical articulation of a problem. But are we actually seeing in the data that we're collecting back, the telemetry that's aggregated for that we can look at? Are we seeing qualitative impact into how humans behave and how they behave with respect to the behavior of their robot? I think getting that and celebrating that and highlighting that, packaging that and communicating that back to the team so that they get jazzed about the fact that what they're doing, that last long mile once you cross it, you can actually see a meaningful feedback. I think that gets the team really excited to be able to play that game of research and bottom line work in a much lighter way than I had been exposed to previously.
Lukas: That's cool. So like measuring and communicating.
Angela: And having, dashboards across the office where you can see how things are moving, see how the models converging both in test, but also whenever it's deployed in production and seeing the impact on the fleet and communicating that out to everybody else in the company and hearing all of our partner nerds who nerd on different things celebrate that success. I think it just really becomes an interesting virtuous cycle.
Lukas: When you interview ML practitioners and those adjacent, do you interview for something a little bit different than other companies, like Microsoft or Google? Is it the same stuff or is there something different about iRobot that you look for?
Angela: Good question. I do, but I've never hired for Microsoft or Google so I wouldn't know if I would have had some pushback but I was already at iRobot when Danielle joined. So I was really happy to welcome her to the team. But it didn't seem like there were too many fundamental differences with the way the transition happened, is that right?
Danielle: Yeah. Yeah. I can't think of any major differences.
Angela: I think a bias towards a love of hardware helps,
Angela: I think if you don't love the fact that these things are going to live outside of your own realm, it's just not going to be as exciting. And, you know, everybody wants to work with somebody who's excited about what they're doing.
Lukas: I think that it does seem your kind of quality control and testing must be pretty rigorous because it's probably hard to go back and fix things. Can you say a little bit about how you approach that?
Danielle: Model quality control and testing is a huge part of what we do and as much as we can. A lot of these different layers we've automated so that when we're developing software, we have automated tests that run even automated tests that run on the robots. We think about it at almost every single layer of the system. So from model algorithm development, thinking about how do we validate our models off-line and even that very first step is tricky, right? Because we're building robots that run in the real world across, millions of homes, all the different variety of homes in the world. So even that first part of how do we test our models off-line to make sure that they actually will work in the real world? Even that is tricky. But after we get over that hurdle, we then think about the next step of deploying on the robot and making sure all those pieces that we need are together. So from the model conversion process to what pre-processing happens, to making sure that the hardware that goes into the machine learning algorithms is the same as the applications that are developed when we develop the machine learning model. So thinking about the variety and hardware that goes into the data collection aspects and then where the machine learning model is running, that the software is running correctly.. So metrics for off-line, metrics for on the robot, and then also once we deploy the model in production, making sure that we monitor, continue to monitor that model and continue to understand feedback and improvements so that we can also send updates to the model when necessary. So there's a lot of different layers to that. And it gets really complicated because we are thinking about how we can generalize to the real world and the real homes; and there's a lot of variety in that.
Angela: There's a component to testing that goes beyond just model quality and model applicability, because these are robots that are in consumers' homes. So we also have a regulatory and a compliance component to the testing. So the Roomba, the Braava, they have a set of testing. But also, when you start thinking about things like the Tarot, which is the lawnmower, that has a completely different kind of fail stop mechanism because it's got literal blades that are cutting grass and you definitely don't want it going over fido. So we have this testing culture and mentality that goes all the way from how the hardware works, to how the software works, navigation, and then through all of this, the machine learning features that are deployed through these products, be they, you know, recommender systems or computer vision types of applications.
Lukas: Have there been any interesting surprises? Like as you've gone from these specific unit tests to more bigger picture tests of production, have there been any things where you are like 'we didn't see that coming'?
Danielle: I mean, I'd say there is.... It is amazing that you continue to find little aspects and you continue to realize throughout the journey how important testing is and not just for finding things, but actually helping you move more efficiently. So having checks at the different layers also helps you debug faster. It's not only knowing that something is going wrong and being proactive, but knowing something's wrong and knowing where something is wrong in the system because we have tests at each layer so that we can really hone into where the problem is. So I think that's something that I've really enjoyed seeing over time - how building in these different layers helps us move more, move faster in the long run.
Lukas: So they've all worked perfectly to prediction.
Danielle: Definitely not. We've built up tests over time.
Angela: But it's really interesting because we've also learned through this process the navel gazing process of how do we test, how do we ensure and how we root cause. We also noticed that across iRobot, our team is getting called to help other teams root cause what they're going through because of the way that we interact with data. We have brilliant technologists, roboticists and algorithmcists, but they are not all trained in the gospel of data and dealing with really large data sets that require imputation that are gappy, that aren't necessarily fully representative, comprehensive of what it is that a software developer has access to when they're looking at how their IDE is responding to them. So we are able to bring that and help different teams that sometimes have nothing to do with machine learning; to help them look through the information that our robots are sending back at telemetry, whatever it may be, and then help them build tooling that looks at fleet data to help them towards their specific scenarios, even completely unrelated to machine learning altogether.
Lukas: How do you think about testing models that are inherently non-deterministic or at least, you can't be sure that they're going to be accurate every single time? Do you have some threshold or do you look for distribution drift?
Danielle: This is something that we actually think about a lot because of our systems. We also have data deletion requests that come in and obviously we comply and think about how we build in user trust and user privacy into our system. So we think about this a lot because our source data actually changes as a result of data deletion requests. And so thinking about how to build reproducible pipelines that we know, what are the components that went into that pipeline so that we can recreate training datasets based on both the distribution and based on what data went into that model originally, minus the data that was deleted or things that came into the system and new data that feeds in as well, because we also want to supplement our data sources with any new data so we can improve over time and and get better features in the end. So we think a lot about how to create reproducible pipelines, how do we track work?
Lukas: This is really fascinating. So when you say reproducible pipeline, but the data is changing, what exactly does that mean?
Danielle: It means understanding exactly what distribution we're pulling from and what sources we're pulling from and being able to reproduce the processing that pulls and creates the data set that goes into the model.
Lukas: So I see, the process is reproducible.
Danielle: Yes, exactly.
Angela: But also, whenever we do reach some convergent model that the interpretability and the explainability of that model while all of the training data is still there as intended and designed is really important because of the ephemerality of that data. So if the underlying data and the underlying training set does change. What it ultimately generated, even if we have to change things in order to be respectful and sensible stewards of our customers' data, we ensure that whenever we do have a model in hand that the learning of it can be reimplemented in addition to to the reproducibilty of the pipeline so that we can't have just blind black boxes that we just retrain at will because of the regulatory and compliance environment within which we operate.
Danielle: And for the reproducible pipelines, we also think about the metrics that come out of those. So some of the metrics we were talking about earlier of the metrics offline, the metrics on the robot, all of those assets where we're working towards full automation so that we can have these reproducible pipeline and reproducible metrics at each layer of the system that we know
Lukas: That's interesting. So these metrics, when you say these metrics we talked about, you're referring to the end-to-end product or business metrics or do you mean something specific to that model?
Danielle: Both. So with model metrics that come out as well as how that model performs in the bigger system, which there could be a whole other system that it's interacting with on the robot, you need to think through both the model metrics and we might need to actually tune the model differently in order to optimize the system metrics. So thinking about those different layers to be able to reproduce, capturing metrics and then creating those reproducible pipelines and how we think about approaching that problem.
Lukas: Interesting. And have you seen modern techniques? How much has modeling improvements improved these metrics? Has that been important to you? I mean, some people say, "oh, it's just about the data". Some people say "we'll know deeper, more sophisticated models". They really matter. Some people think hyper brain research is a really good idea and some application seems like it doesn't help very much. Where do you stand on those types of things?
Danielle: So I'd say data first. For our use cases, because we're trying to hit all of the users, all of the houses, we are able to generalize to the world and all the variety of the world and making sure that our data is generalizable in that way and is the most important.
Danielle: By the physical world.
Danielle: The physical world. Yeah.
So threshold or threshold between one room and the other... The architecture of 1800 Germany is something that you really can't modify. Like houses in Berlin will be the way that they are and so it's really a different world. When you're talking about a robot that we're going to be approving these models, we're going to be sending new models over air update. But the computer that robot has available to it and the environment within which it operates is static. And so that's the constraint, which is why we value the data collection so much because of that sort of constraint, essentially.
Danielle: Absolutely. And the other aspect is the algorithms that run on low compute. I think the advancements that happen in that space, there are little things that you can do that make a big difference. So thinking about that area, I think there is a lot of room to go in terms of how much algorithm improvements can make a difference.
Lukas: Did they matter to you over the last couple of years, like the stuff that's been really meaningful?
Danielle: Yes. Especially advances in inference time as well, because we are running in such constrained compute environments. The small differences can make a big deal.
Lukas: And this is beyond the hardware improvements. This is actual model improvements.
Danielle: Right, Yep.
Lukas: Cool. Just being really specific, I'm curious. When you think about hyper parameter searches, that's an auto ML, is that something you do a lot of? Does that help you much?
Danielle: Yeah, we definitely leverage hyper parameter search and auto ML type capabilities to see what the space is out there and see how we can improve on data. Augmentation approaches are also very useful and thinking about how we can supplement the current data that we have to try to make sure that it is generalizable to the world.
Lukas: So data augmentation approaches vs. model architecture, which do you think is more important?
Danielle: That's a tough one. Both.
Lukas: Obvious. Fair
Angela: I mean, if we had infinite researches, I would agree. But I think looking back at what we've actually had to overcome over the last year and a half, I'd say that data augmentation has been more of a throttling or a bottleneck to us. I'm not sure that that is always going to be the case. But specifically.
Lukas: It's been a problem for you or it's been helpful to you?
Angela: It's been helpful. So it solved the thing that was a real problem for us, a bigger problem. So the ability to understand every environment from day zero, right? Once a robot starts going around a space, a customer, somebody who just bought it, they don't want to give that robot a month to get good. They want if you wait a month, they're going to stop using it and then you lose the chance of having the light of that customer. So starting off with a really solid day zero solution was really the big important objective for us. But I'm not sure that that is always going to be the case which is why I think both if we had infinite resources.
Lukas: Its early 2020, things change for sure. Have you focusd on one deep learning framework? Is that Tensor flow or PyTorch or do you love Scikit? Or is it all of the above?
Angela: It depends on the team. When we're talking more about the reinforcement learning team or the modeling team, I think they get more leeway in experimenting because that's the role. Like, it is the stuff that we've settled on - continuing to work for what we want and will age pave the way for what we are going to want. So those folks test a little bit more, go a little bit more wild into the research and what's available and what are the state of the art improvements and how can we make those applicable within the environments that we operate in? I think when we're talking about the integration and the infrastructure teams, they're focused on different things. So they are much more focused on that last mile and so re-architecting all of our tooling, unless there is a really important benefit that we get from that, because at that point you're impacting all of the platforms that may make use of this one deliverable and a lot of those platforms are already in homes. So if we have to re-architect how all of that delivery happens, there are a lot of teams and there's a lot of integration that needs to take place. And so is it worthwhile to spend those resources? The farther into the process we get, the harder it is to stay, to modify and to tinker with that stuff. But I think the modeling team. Yes. And I think they've done Tensor flow, and SciKit line is always a favorite of all. I think we've dabbled in PyTorch and figured out it isn't working for us. But once it gets down farther into the pipeline, it is much more into what's already working and 'is it worthwhile to throw a bomb in there'?
Lukas: Can you say what's already working?
Angela: Not yet. But we look forward to coming back and sharing a more details on all of that.
Lukas: Cool. Can't wait. I want to be respectful of your time. I think we've gone a little over. I'm hoping to end with a couple of questions, if you don't mind. We've been asking everyone. So here's my first one; what is the one underrated aspect of machine learning that you think people should pay more attention to?
Danielle: Oh, the data. The data is so important for making sure that it's generalizable, making sure that actually figuring out your metrics are all going to be based on the data that you're using. And so how do you know the metrics of the model or the metrics of the system are actually metrics that you believe are useful to represent things? Because at the end of the day, it's all about the data that goes into the system.
Angela: And it's not just the volume. I mean, one hundred percent. It's the data. But it's not just the volume, the variety or the velocity of it, like the fancy 3D. But it's the quality of it. So are we only looking at data that's streaming back from us? From customers who are already happy? Which means we're not solving the problem for the folks who aren't already happy to begin with. We're not solving the problems for the folks who aren't using the product that they've already paid for and we're not delighting them. So it's not just that we have data, but is that data reflective of the whole? And how do we ensure that the next thing that we do isn't just improving marginally the experience of a fraction of our customers, but yes data. 100% data. Underlined.
Lukas: Why do you think it's so hard for the industry to realize how it could still be underrated? Seems blindingly obvious from where I sit.
Angela: I don't think that it's underrated. I think when you ask this question, you probably get the same answer. It's not like it's underrated. It's just hard. It's known to be valuable. It's also known to be really hard.
Lukas: They know a few companies that will collect high quality data.
Angela: No doubt.
Lukas: So, next question. What is the biggest challenge in making machine learning actually work in the real world from where you sit? When you start a research team coming up with something and some crazy framework to actually getting it deployed, what is the hardest part of that? What makes you the most nervous?
Angela: So a really tough learning for me was not at iRobot. It was at the company that I was at before. We had this amazing solution; a machine learning algorithm that was for energy efficiency and it predicted when there might be a spike on the utilization of the energy grid or something of that nature. And it worked really well. It was really fantastic. We were all proud of it and it didn't sell. And so the really important thing is communicating to the right people. What is it that they're getting and what is the value? Because I think and I was blinded by this, too. I'm not saying that I was smart and I knew it and nobody did. Nobody else did. I wasn't above it. I fell for the mirage of how delightful and wonderful and sexy and how performant and correct it all was. And that doesn't matter. Like if it doesn't actually solve a real problem that a real human is willing to part money with in order to have that problem solved for them, it doesn't matter how sophisticated it is or how much data you have on solving the right problem. Danielle, I don't know if you have a different anecdote or a different experience.
Danielle: So one of the biggest challenges that I see with machine learning systems in the real world and getting them out there is, there's a lot of different pieces that need to go in to make this right. With the data collection piece, the training part, the processing, the model serving, the software pieces, the hardware pieces, the hardware. In our case, the hardware changes over time and we also need to make sure the data reflects that hardware changes over time. There's the different homes, the different customers, making sure that it's actually generalizable. So I think there's so many pieces making sure that you have quality checks in place on those different pieces so that when things don't work as well in general or for a specific customer, we have ways to make it better so that the quality at the end of the day is what the customer is expecting. So I think that's the biggest challenge, is just connecting all those pieces together and making it real.
Lukas: Cool. Final question, you know if people enjoyed this, I bet a lot of people are really going to enjoy this, and they wanted to learn more about your work or reach out to you, is there any anything you'd like to link to or a Twitter account that you use or anything like that?
Angela: Well, first, you should definitely go to LinkedIn and you should look at the iRobot page you should apply, you should come to work with us. That's the number one. The iRobot.com website.
Lukas: Seems like it will be really fun to work with.
Angela: It is going to be super fun. I am not at all biased because I love the people that I work with and I love the work that we do.But that's one place the iRobot.com website actually has a lot of things for the consumer, but it also goes a little bit into what we do and who we are. So that's useful. Personally for me, you can go to my website. I'm at AngelaBassa.com. I'm also prolific on Twitter, which is a problem, I'm working on it! @angebassa.
Danielle: And I'm on Twitter as well. @DanielleODean.
Lukas: Awesome. Well, thanks so much. That was super fun!
Angela: Thank you!