David Rosen

Thanks for coming. We're excited to be here.

I'm David Rosen. I joined Huawei about a year ago. I'm based here in Santa Clara in California, now leading our R&D technology strategy team, which is basically looking into the future of R&D as it applies to Huawei, and helping Huawei progress and guide Huawei as we're changing as a company, which we'll talk about.

Part of my role and our team is to help Huawei in that change and in that transformation.

Edward Pershwitz

My name is Edward Pershwitz. I joined Huawei a couple of years ago, in 2013, and my role is architecture of the tools infrastructure. So I'm leading a small team of architects based in Plano, Texas.

The main goal is enablement of change through technology, through our architecture.

David Rosen

Okay. Just a few minutes around Huawei. Some of you may not know what Huawei is. So it's just a small company out of China of about 175,000 people with 75,000 engineers. The majority of work happens in China, but they have presence all over the world with some 16, 17 R&D centers in the US, in Europe.

Huawei is currently experiencing rapid, massive growth. So 2014, $46, $47 billion. This year it's going to be way over 50. So year over year, 15, 20% growth, which is an interesting situation to be in at that scale.

Huawei has basically three major business areas. The traditional one is selling into telecom operators: massive, large-scale telecom or mobile infrastructure, selling to carriers. The second major business is into enterprise IT, so compute, storage, networking. That is now also transforming into a cloud business. And then the third business, which is experiencing very rapid growth at the moment, is mobile phones and handsets, where Huawei is now number three in the world, trailing Samsung and Apple. And just some numbers the other day, it was announced that Huawei will now pass 100 million devices, possibly 110 million this year. So again, very rapid growth in that area.

It's all bundled into what we call ICT. And really how we look at ICT and where ICT is going is towards enabling what Huawei says in their marketing material, the fourth revolution, which is really around intelligentization, where everything is connected, everything has an opportunity to be intelligent.

So that's really the topic of our talk today: how we're thinking about the big mantra of Huawei in terms of intelligentization. How can we apply that in the context of product engineering? Which is, I'd say, a hot emerging topic, hot emerging research topic. You don't have that much commercial presence in terms of vendors, in terms of solutions.

It was interesting to hear HP in their talk this morning. I think they were alluding to intelligentization. But it's very much an emerging topic, as evidenced by this well-known science and research publication called Elsevier, which in January will have a big publication around artificial intelligence techniques specifically focused for product engineering. So again, interesting and emerging topic.

So for us, as we look at this in the context of product engineering, it's very much around decision-making, and how can we help make Huawei R&D decision-making more efficient and more effective at scale.

And really that boils down to basically, in very simplistic terms, two different types of decisions. One is focused on what should be done and why, which is more or less a management decision. And the second one is around how should it be done and why, which is more of an engineering decision.

That should, though, be thought of in the context of everything from individual daily tasks to large-scale or longer-term visions and missions of teams. So we're thinking around decision-making in that whole two-dimension spectrum.

Some of the inputs to a decision are basically coming from three different angles. One is your cognitive system, basically making up your heuristics. What's your experience and what do you know from your past? Then you have an environment that is affecting the decision. And thirdly, you have technology, and that's really obviously changing massively today with the evolution of everything from big data to search and so on.

On that note, I can recommend a very interesting book, which is really around cognitive system, cognitive thinking, and decision-making. It's Thinking, Fast and Slow by Daniel Kahneman, who was a Nobel Prize winner a few years ago. And he's really thinking around the cognitive system as two types, where type one is the quick intuitive decision, which really is all based upon your heuristics. And then you have the type two, which requires more effort, requires more thought.

And as you read through the book, you would agree and associate yourself with a lot of situations that's being brought up in the book. So highly recommend it.

Another interesting character in the context of decision-making is really Herb Simon, who is by many regarded as a grandfather of business decision-making. And what he came up with is this term called bounded rationality. And really what that's all about is the decision that you're making is really bounded by your context. So you can't make a decision that would affect something outside of your context.

And the way we look at this is that information is really acting as the constraint on that decision or on that boundary. So if we could provide more information, more data, more insights, chances are we could actually improve decision-making.

In the title, it says Economics of Decision-Making, and we were really adding that before we actually sort of knew what to talk about in that particular topic. So it turns out there is not a lot of work done around understanding and estimating and modeling the economics of decision-making.

So some anecdotes in the industry is from Chevron, where they claim by doing what's called decision analysis, they've been able to save billions of dollars per year with a marginal investment in terms of execution and in terms of training. There are a few other notes out there, but not a lot. So this is, again, not a hot and active area of research, yet the savings and the impact that you could have into your normal organization, we believe are massive.

So the right side of this slide just shows a very simplistic model in terms of, as an organization grows in terms of number of employees, just the amount or potential value improved decision-making can have. And really, as you get to the thousands or tens of thousands of employees, you're talking tens of or hundreds of millions of potential improvement or potential impact if you were to improve your decision-making.

So I'm by no means claiming that that model is accurate, but just a very rough estimate. And there is more work on our end to actually work on this in the next year to fully understand and to better understand what would be the economical impact as we roll out some of the initiatives that we're working on.

So we talked about the two different types of R&D decision-making. One would be around what should we do and why, and the other one would be how should we do this and why.

So if you look at the sort of the managerial aspect, there's a lot of talks these days about running lean, lean startup, working on MVPs, minimum viable product. So if you try to take that concept and formalize it into more of a process, as would be necessary if this would be scaled out in this large-scale environment such as Huawei's.

One attempt at this is coming out of a university in Sweden, where they are thinking about requirements as hypotheses, and then they try to validate those hypotheses in the field with very cheap ways of validating it and then providing that feedback back. So basically allowing for a continuous prioritization or deprioritization of these hypotheses.

So I'm not sure how much you can derive out of that graph on the left-hand side, but that basically just shows you a model of this QCD, which is the model that they're proposing, where you take any number of customer feedback techniques, which could be cheap prototypes, it could be surveys, it could be actual interviews, or basically very cheap customer validation techniques. And you feed that into your model of validating hypotheses, and then you overlay that with your overall environment and your list of hypotheses, and then the output of this would be then this constantly prioritized backlog, or prioritization of these hypotheses.

So again, this is just some thoughts around how we're looking at this and what we're looking to productize. And this is our current attempt at formalizing these, or scaling some of these lean startup concepts.

And with that, I'm going to hand over to Edward.

Edward Pershwitz

So David mentioned the HP presentation this morning, and just like him, when I was listening to it, I thought there are many parallels between what we're doing and what HP is doing.

Just like HP, we're a huge company with very diverse products ranging from mobile apps to carrier-grade solutions. And so how can you have something that is all size fits all? In this environment, there is no way we can have that.

And so our solution to that is creating an infrastructure, creating an ecosystem, and then let product lines actually model their own workflows, their own delivery pipelines. And so from there, we could provide a toolchain, we could provide standardized services all throughout that process, but we do not control how different product lines do their business.

If we try to push that, then all we can get back is resistance. So the only thing that we can do is enable the change and then watch it happen.

So when we're talking about the delivery pipeline, and this shows a classic view of the delivery pipeline that starts with continuous integration, and all throughout continuous delivery goes to functional test, then integration, then performance, then acceptance, then production.

Well, we can expand that. We can say that it starts sooner than that. It starts from the requirements entering the system, and then there is product line management that deals with those requirements, and then there are solution architects that deal with solutions, then there are domain architects, then there are designers, then there are developers. Then we actually create code, then we check it in, and this is when continuous integration starts. So the pipeline actually starts way before continuous integration.

And all throughout this pipeline, decisions are made. And these are very diverse decisions. And so how can we improve the R&D efficiency by assisting this decision-making?

Some of those decisions can be made by individuals, by humans. Some of these decisions could be made by technology. Some of these decisions could be suggested by technology and then eventually made by individuals.

So what we're trying to do is we're trying to expand the notion of a delivery pipeline into the notion of an intelligent, adaptive, risk-driven development pipeline to where decisions along the way are assisted based on evidence.

And so I have some examples of that. But this is the common theme, that decisions cannot-- They need to be substantiated, whoever makes those decisions. And so how can we substantiate the decision without having a feedback loop and then processing that feedback data and then turning that into decision-making?

And here's an example. So I'm running a test, and what does it mean for the test to fail? What does it mean to different actors in that delivery pipeline? What would a design and development person do with that failure?

Well, perhaps they will evaluate which modules are hit. Perhaps they will look at when that same test passed last time. Perhaps they will look at the changes that happened in the environment where this test passed last time and what the environment is now, or the code or the test itself might have changed. They would look at whether this change is consistent or the failure is consistent or intermittent.

And so how can the technology be utilized to help with these problems? Well, we can collect data of which modules the test hit. We can run this test multiple times and see if it's intermittent or consistent. We can make a lot of these decisions for the human and then provide that designer with the diffs of what happened last time the test passed and now, and then assist them in fixing the bug.

So that's one example, and another example is for QA. So we're running a test and the test is failing, and can we collect the data to figure out how often that particular test breaks the code?

And then especially with the DevOps transformation, the changes come very rapidly, and on the ops side particularly, changes are flying in your face much faster than you can deal with. But you have test suites to run. Hopefully, you have automation to help you with that.

But now you don't have the time or the capacity to run a complete full product regression every time there is a change. So you need to figure out which tests to run because you have time constraints on it. So there needs to be a strategy of some sort: which tests to run when we get a change.

So can that frequency of a particular test breaking the code be factored into that strategy? Well, sure it can. So could that strategy be automated? Sure it can be automated, or it could be a hybrid solution to where an expert system makes a recommendation for the human to make that decision.

So the pipeline needs to be not only intelligent, but it also needs to be adaptive. So the model, the workflow that we model, or that the product line would model, needs to be capable of self-adjustment. And that self-adjustment is also based on evidence, is also based on the feedback from the field, from acceptance test.

So here's an example, and it looks like the slide is completely messed up. And I looked at it last time on the PC, so maybe that's a Mac thing. But I apologize for that.

So here's a simplistic example: code coverage. How much is enough? How do you engineer your unit test, say, for example, to say, "Okay, well, this is done. We can ship it to QA. We can deliver it to QA."

Well, again, we can have an expert opinion say, "Okay, well, 80% is enough," or, "70% is enough." But can we actually correlate that to what comes back from the field?

So this is the second example. So how does it correlate with the data? How does it correlate with failures coming from the field? How does it correlate with architecture erosion metrics, say, for example, static analysis, say, for example?

So the fact that code passes the test doesn't necessarily mean that it's ready to ship. What if it creates a huge maintenance problem going forward? What if it erodes architecture to where it increases coupling, decreases cohesion, and then we have technical debt on our hands? So can that be made a part of your delivery pipeline as well, in addition to just running tests?

And then the risk-driven. And for right now, we don't have all the answers, but I think that a starting point to getting answers is asking the right questions. So here I'm listing decision factors and risks assigned to them, and they're primarily expert opinion-based.

But again, with ranging products, with ranging release cycles, software and hardware complexity, the workflows that are generated are different. The pipelines are different. And so as we assign different risk factors to all of these different dimensions, and presumably they should be orthogonal--they're not really orthogonal, but this is the best we can do at the moment--we can then come up with the actual delivery pipeline.

So what about human intelligence? Do we consider human intelligence a part of the pipeline or separate from the pipeline?

Well, we can have a view to where human intelligence is deployed to the pipeline. Human intelligence is just a part of it. We deploy applications to the cloud. Why do we do that? It's because we have a comprehensive service layer there, because we have an on-demand environment there that our application can run.

Well, how is human intelligence different? We can create an ecosystem to where we can deploy human intelligence to, and then present that human with a comprehensive service layer, with the comprehensive integrated UI that they can use to contribute their intelligence to the pipeline. Because in the end, it's still product delivery pipeline that we're talking about.

So I'm running out of time, unfortunately. There are some considerations about technology and frameworks that we need to be able to implement this. And we need to have an integration platform, naturally, that would support R&D services, workflows, resources, and data, and link them together.

We need to have an expert system that will be evidence-based, that will collect knowledge, create new knowledge using an inference engine.

We need to have a big data platform and services because there are massive amounts of data generated from this delivery pipeline that we need to process and be able to turn into decision-making.

We need to have a comprehensive service layer.

We need to have workflows and a framework that will support modeling and deployment and execution of those workflows.

Toolchain. No one knows everything. We need to be open. We need to have an ecosystem open to other stakeholders contributing to it.

We need to have resource management. We have lab. There are vast amount of resources that we need to manage.

And naturally for human intelligence, we need a comprehensive integrated UI framework. I'm running out of time. I have a couple of more slides to cover. I guess the presentation will be available online. I will now--

David Rosen

Let's just cover the architecture a little bit.

Edward Pershwitz

Well, this is--

David Rosen

Yeah.

Edward Pershwitz

Very briefly, this is the actual slide of what we're working on. We start with a workflow framework, and this is the actual delivery pipeline workflows.

Going up, there are multiple roles, because people play multiple roles in this delivery process, from management to architects to developers to testers to support.

There are core services that the system doesn't really function without. There are ecosystem services. These are contributed.

There are workflows that are core workflows that are part of the system functioning. There are ecosystem workflows, and these are the ones that are modeled by stakeholders, by different product lines.

There are big data platform, big data services that is used to collect and process data. And then, at the bottom, there is supporting lab layer, which is a lab-as-a-service concept.

And we're out of time.

David Rosen

Let's just bring up the last slide. Okay. Do your conclusions. Yeah.

Just very quickly on the conclusions and some of the top takeaways. So, like we said in the beginning, we believe the effectiveness and efficiency of decision-making is really set up for disruption and can have a tremendous impact in terms of R&D productivity.

And it's like Edward was talking about, it's very much around evidence-based data gathering and using that data to augment or guide human intelligence as part of the delivery pipeline.

So we certainly don't have all the answers today. We're actively working on this, investing in this, researching it. It is an emerging area. It doesn't have a lot of commercial vendors yet. I believe that's going to change in the coming year or two.

And we'd be happy, if this sounds interesting and you're thinking about the same, we'd be happy to have conversations about how we can collaborate, how we can partner.

Edward Pershwitz

For sure.

David Rosen

Something we certainly are open to. And that's it.

Great.