Michael Winslow and Nisha Parkash

Michael Winslow: Good afternoon, everyone. I am Michael Winslow from Comcast, and today we are going to talk to you about how Comcast and Sky were able to use the dojo format to work together even across the seas. I am joined here with Nisha Parkash. Nisha, do you want to say hi?

Nisha Parkash: Hi, everyone. As Mike said, I am Nisha Parkash from Sky. I am the Online Compliance and Brand Protection Manager there.

Michael Winslow: Great. Why do we not jump into our presentation? You did give a brief introduction, so why do you not tell people a little bit more about yourself?

Nisha Parkash: Sure. I am from London, born and raised. I obviously love the city. I grew up in south London, and I now live in West London. I am a big fan of tea. I think most British people are. I think every task from a British person starts with a good cup of tea, so I had to get that right. I went and took a proper British Brew tea-making class to get that right. Just showing a little tip: the idea on a good cup of tea is to put your teabag in the hot water, do not touch the teabag for two and a half minutes, and then put your milk in, and you will get a good cup of tea.

Another fun fact about me: I am completely obsessed with kitchen gadgets. I try and find anything available to make food look fun, and I think the best thing I have found so far is turning a mango into spaghetti.

Michael Winslow: Oh, wow.

Nisha Parkash: So there about me.

Michael Winslow: That is cool. A little bit about me. I am from New Jersey, a little town called Mount Holly. For those of you across the ocean in London, I know a lot of you are familiar with Newark Airport. I would say I am probably about an hour away from there.

My other fun fact is that I spent a semester working in Amsterdam. There I wanted to be able to order my favorite Burger King meal, so I learned how to do it in Dutch: "Ik wil heb een Whopper, met kaas, zonder ui, zonder mayo, met Coca Light en jumbo frites." That is a little thing that I picked up along the way.

On this day, I would say that it is very important to say that I support Black Lives Matter. I hope you do as well. It is a very important movement, and it is a very happy time right now that so many allies are stepping forward. I really appreciate all of you.

Our story starts almost exactly a year ago today. I traveled over to London to visit the Sky UK offices with Nithya Ruff for a Tech Women talk. I did not know that I was going to be able to meet the DNS brand protection managers over there, which happens to be very close to what my team does with DNS. It was great, and serendipitous, to actually meet that team. Here you can see basically what it was like over there: amazing offices, The Simpsons Kwik-E-Mart bus stop for the shuttle bus. We took this picture a year ago, and it is so appropriate today: our differences are only skin deep, but our sames go down to the bone.

A couple of months later, we were able to invite Nisha and Dawn Shackleton out to Comcast, where we were having our Comcast DevOps event, our fifth-year event. It was so great to have them there, to be able to get on stage and talk to Comcasters. It is such a huge part of DevOps to share, and they were able to come and share to our entire community in Philadelphia.

Another great thing about that October event was that Gene Kim was the keynote speaker. It was so great to have him talking about the five ideals, just before "The Unicorn Project" came out. It was a real exciting time, and we were glad we were able to get him over there. He was very personable, talked to everybody, and participated the whole day. It was a great event.

The conference was nice. We are doers, I can tell you that much about Nisha and me. While we had a great time at that conference, the immediate question we had afterwards was, "How do we work together?" We thought, how about we take the application that my team is in charge of and see if we can get it working at Sky? Can we get VinylDNS working at Sky?

Let me quickly tell you what VinylDNS is, since we mentioned it a couple of times. VinylDNS is a DNS management system. It is very important within the walls of Comcast because we have the second-largest DNS footprint in the world. Without giving exact numbers, we have over 200,000 zones that we manage, and we have over 40 million DNS records within those zones. That is a very large footprint.

With something like that, it is great that our application is able to enforce standards for all the DNS entries. We are able to give fine-grained access control to individual groups, which is very important in this industry, so it does not all flow through a single entity. We also provide a robust API for integrations, so for people who do not want to use our front-end portal, there is a headless option to put automation in place, which is huge in DevOps.

We were thinking, why would Sky not want to take a look at this? When you think about an application that needs to support a DNS footprint of this size, there are only a few groups that can actually take advantage of it. I would say that Sky was definitely one of those groups. Right, Nisha?

Nisha Parkash: Definitely.

Michael Winslow: There we go. We had what we wanted to do: basically get VinylDNS working at Sky.

Nisha Parkash: That is great. Let us just do it. Let us just give access to Nisha and she can start working on it.

Michael Winslow: Well, we found out that we had a little bit of a problem there. While we had all of our applications and our communication abilities for Comcast in Philadelphia, Slack channels, GitHub, our application, and our internal cloud, we found out that Sky just did not have the ability to connect with us in those ways. Even something as simple as being able to come together in a Slack channel, we were not able to do. That made it very difficult. We had to figure out how we were going to actually make this happen.

After thinking about it for a while, we realized that VinylDNS was open source. We had taken the step of making sure that this application was something we gave back to the community. In this case, it helped us so much, because now the firewalls meant nothing. It was out on the public internet and public GitHub. We could treat it as if we were two different citizens who wanted to use VinylDNS, and just work with the team to do it.

While we could not get to our own private cloud internally, we found out that Sky, Nisha, and the team over there were users of Azure. We thought this was an exciting challenge because we had never put our application out on Azure before. We were ready to accept this challenge. We were a go.

For anybody who wants to visit and take a look at VinylDNS, you can go to vinyldns.io. From there, you can link to our GitHub account and see all of our documentation.

Now we are talking about the dojo. That is the way that we wanted to come together. What is a dojo, for those who do not know already? The Comcast dojo is an upskilling program, and it is designed to be an immersive learning experience where full-stack teams come together and learn modern engineering, product, and agile practices.

A lot of people in the DevOps community are familiar with dojos already, thanks to the pioneers before us: the Ross Clantons of the world, the Jackie DiGiamos. To put it this way: when you step into a school, you know that your reason for being there is to learn. When you step into an office building, you pretty much know that your reason to be there is to work. What is unique about the dojo is that when you step in, you should know that you are there to both learn and work. You want to get things done.

I think probably the pioneer of the modern-day dojo from a tech standpoint was at Target. You can go to dojo.target.com. They are very upfront and open about how they run their dojos, and it is a huge part of their culture right now. The other thing, and this is what we did when we got started, is to go and check out the book that was put out last year, "Getting Started with Dojos," by Ross Clanton, Jaclyn Damiano, Carmen DeArdo, John Esser, and Eric Passmore. They did a great job really laying out what you need to do in order to successfully get a dojo set up in your own environment.

What can you expect once you get a dojo set up? You bring your team and your own backlog, and that is what we mean when we say we are there to work. You are not there to figure out exercise files that you will throw away after you learn. You are bringing your real work to the table, and you have experts in place who will help you get through the project and do it in a way that is product-focused, as well as using good agile techniques and DevOps techniques.

What often happens is that you create a six-week dojo experience with two-and-a-half-day micro-sprints, where you can fit 12 in through the entire life cycle of it. You want to come out with a better understanding and grasp of agile and product. You want to have better DevOps, and you want to be cloud native when you do these things. You do not have to be; this is not set in stone. But these are things that you can take while you have the time and the people dedicated to learn them. You can take that time to make everyone a little bit stronger in their individual expertises. After six weeks, you will have improved tech skills. In our case, we were also looking to get faster time to value.

The one thing we had to do, because this was not an official program funded by Comcast yet or Sky, was that nobody was going to let us take these people for six weeks and dedicate it to this. We are working to convince everyone that this should be a program that we accept. So we worked more with one- and two-day sprints at this point, just to get some value out of it so that we can go back and say, "Why do we not go ahead and take a chance and make it a much longer timeline and really use the dojo the way it is supposed to be used?"

When we started the dojo with Sky, we went in with a plan. The plan, when you are talking about dojos, is the working charter. We had a working charter between Sky and Comcast. The blank charter looked something like this, and this is what we ended up with when we filled it out. Nisha, do you want to walk everybody through the parts of the charter so people know what it is all about?

Nisha Parkash: Absolutely. The important bit is obviously to name it. In this instance, we named it VinylDNS at Sky, for obvious reasons. It gives you something to focus on. Then you move on to the elevator pitch. The idea would be that if someone walked into the room, they would know exactly what we were doing. In this instance, if somebody joined our digital room, they could see what we were doing, what we were doing with our day, and then they could walk straight out knowing that we were getting on with something. I will leave software strategy elements to you, Mike.

Michael Winslow: This one is somewhat specific to Comcast. We have a group of 15 high-level software strategy elements. One of the ways that we wanted to convince leadership that this is something we could do was, in our charter, to mark down which software strategy elements we were addressing when we came together and had this dojo. We have redacted that because it is Comcast material, but this is a good open space to put maybe your software strategy elements there.

Nisha Parkash: Goals and measures, as it states: you have a goal, and how is it measurable? Mike, if you want to elaborate on that, because you do it so well.

Michael Winslow: I appreciate that. You want to come into the dojo with goals. So many times we create goals, and I do not know if we make them measurable. It is very important in this case that you make the goals measurable. You want to have some wildly important goals that you want to complete, and you want to have measures to make sure that you know those things were complete. Teams work with KPIs, OKRs, things like that. It is basically the same idea, but here we have goals and measures.

Nisha Parkash: The skills matrix defines what technically each person would be learning, or in this instance upskilling, by the end of the dojo sprint. One thing I love about the skills matrix, setting it out from the start, is that you can see that everybody is going to learn something new. You feel confident, regardless of what level you are at, that you are not dealing with somebody who has everything or all the knowledge that they need. Actually, everybody will learn something new from this.

Michael Winslow: I love that part as well.

Nisha Parkash: The working agreements for the dojo are really important, just to set your day out so you know exactly where everyone needs to be throughout the day and you get the full day in. Of course, everybody needs certain types of breaks in a normal working day, so you set the tone from the start. Are there any other meetings that people in the dojo need to attend? What breaks are you going to take for respite, a coffee break, a lunch break? You let everybody know where you are going to be during the six or seven hours that you are doing your sprint, just to set it out for the day so we know what to expect.

Michael Winslow: This is super important when it is international and you have to work with those time differences: being able to come together and agree on what time periods are going to overlap between the two groups. This was very important.

When we came together, we were agile. I am going to go back to January. This was not when Sky and Comcast came together; this was an internal project we did at Comcast using the dojo model that was very successful. We had a scrum master and physical scrum boards. That was something we could not do, number one with the pandemic that came along, but also because all of a sudden we were working with an international crew. We needed something that would help us come together and collaborate even when we could not be co-located.

We used a tool called MURAL, which I think we were very happy with. What did you think of the tool MURAL, Nisha?

Nisha Parkash: I loved it. It looked like the whiteboards that we would be standing next to if we were in a physical room. I loved the fact that you would stick it on like you were using Post-It notes, and everybody had a set of colors, and it was very apparent who was writing what. I loved the fact that the sticky-note style of working prompted others in the dojo to think about other aspects. Someone would write something on a Post-It note, and it would provoke thought process. I also liked that there was so much transparency and seeing how two different companies think, just laid out that way. Were we really different, or actually were we quite similar? It was really good to have that layout.

Michael Winslow: I agree. One thing that we added that I do not think I have ever used on a physical scrum board before was using these notes to also have heads-up on when people had to go away for meetings or take breaks. The pink ones were there. It was great to be able to go back constantly and say, "Oh, a break is coming up." I really enjoyed the MURAL interface as well.

We were product-focused. That was something that we thought was really important. One person in this picture, everybody who was in there the day of the dojo, is the product manager. Try to figure out which one of these people you think is the product manager, and we will see if you are right at the end of this short little game we are doing.

First, we know it is not Nisha. Nisha is the boss. She was not the product manager. This is Pete. His name is Peter, last initial C. He has the same exact name as a person who was sitting right next to him when he started working at Comcast. Imagine walking up to your cubicle and realizing the person sitting right next to you has your same exact name. Pete lovingly became PC2, but he is number one in our hearts. He is not the product manager.

Ryan: we are all convinced he is a robot. Ryan speaks computers. He speaks zeros and ones. He is amazing. He definitely is not the product manager. I am not the product manager. I am just happy that everybody let me stay in the room. We were having so much fun, and work needed to get done, and everybody kept reminding us that maybe Mike should stop making so many jokes.

It was not Dawn. Dawn, of course, by the T-shirt you could tell is at Sky over in London. We have healthy debates on what real football is, and I am sure she will figure out what real football is before I am done with our conversations. It was not Ollie, but being able to see screen shares of Ollie as he was handling DNS infrastructure and Azure, you could see that he had so much power as an individual over there. We loved watching the fact that we did not have to go to all kinds of different people. Having Ollie on the team allowed us to move so fast. It was great having him on the team.

That leaves two people. It is not Shiju. Shiju was on the Comcast side, AKA Tupperware, because he is always willing to sell you on containers. That leaves Paul. Paul was our product manager. If you guessed Paul, you were right. We made sure to have a product manager in the room. That was super important to us because one of the cards we put up had to do with making sure that Sky explained to us how DNS maintenance happened currently. Can you explain a little bit how it works currently, Nisha?

Nisha Parkash: Sure. It is exactly what Paul took us through. We currently have a setup where raising a DNS request requires certain levels of approval, specifically three types of groups. In our instance, we raise DNS entries often in our team, and actually we are one of those groups that technically approves our own DNS changes, or each other's DNS changes. That is the current process at the moment. There are SLAs involved and approval levels to allow a DNS entry.

Michael Winslow: That was really enlightening for us because we did not want to just come in and say, "We are just going to set up this application. We have no idea how you use DNS." It was great to get to know how you do it, and then also how we do it.

Of course, we were DevOps. How did that manifest itself? It was great to have people like Dawn and Nisha in the room, and Paul, who are self-proclaimed non-technical people. Then to have Ollie going onto Azure and creating Ubuntu instances and things like that, it gave a little bit of insight into what the technology people do when they are getting stuff up and running for you, Nisha, for your needs. Want to share anything about what it was like sitting there with all these techies?

Nisha Parkash: For someone who is non-tech dealing with the developers who were on this dojo, it was eye-opening. What struck us most was the fact that you as developers were extremely flexible, accommodating, and curious. I think that was really important. You were not just stuck in your own design. You were interested in how another business actually does what they do in order to accommodate what it is we were trying to achieve here. That was really nice for Dawn, myself, and Paul, who are non-developers, working with developers in quite an alien environment. You did not make us feel like we were working with aliens. That was really cool for us.

Michael Winslow: One thing I would mention is that even at times when there were long processing times and we had to wait five minutes for something to happen on the Azure side, people were so willing to jump in. To follow up on what Nisha was saying, at one point we were talking about Docker and containers, and Nisha and Dawn did not understand what we were talking about. During a break, Shiju decided to share his screen and go over a little presentation of what Docker and containers are. The dojo allows for ad hoc learning like this, to get everybody involved so we can all make great products together.

This was the big moment. This was the moment when we actually saw our application running for the first time on not-Comcast infrastructure. It was out there in Azure, available, and Ollie brought it up. You can see the adulation on everybody's face. We were excited to have something out there so that Nisha and Dawn and the others can actually explain to people what Vinyl is, and almost do a roadshow.

Are there any aspects about VinylDNS that get you excited to possibly work with it, Nisha?

Nisha Parkash: It is such a great tool. The fact that Vinyl allows for these precedents, with certain individuals being able to go in and make DNS changes without the approval levels that we currently have. Having availability to this code, even in a test environment, to see it is so exciting. Being able to make people feel empowered in their roles, to say that they actually are trusted, they do not need to go in and have a DNS entry vetted and then approved and approved and approved, and then it goes ahead. Vinyl definitely offers so much more flexibility and it avoids the SLAs, really. Like I said, it will really empower developers to see the benefits in what they are doing quicker. I think everybody can vouch for a developer wanting to get their work just done and out there, right? Vinyl does that.

Michael Winslow: Absolutely. It is important to note that once you get an infrastructure or footprint of DNS that is a certain size, this idea of filling out tickets becomes so cumbersome. You really need automation. Just to give you an idea, we usually take on about 400 new zone creations per month. Those are zones. Those are not even the records within those zones, just new zone creations every month: 400. Some of them get retired, but that is the kind of activity that you see in something this size.

You even get the ability to go into your zone, see all the recent things that have happened, and see the record-set types that people are adding or taking away from DNS. It is very important to be able to go back for audit purposes and to see how people are touching your internal zones and records.

Nisha, now that you have an instance up and running that you can get to, it is not behind our firewall, and you are not blocked, how do you plan on taking that and using it to try to get DNS implemented fully at Sky?

Nisha Parkash: I absolutely loved Vinyl from the day you demoed it less than a year ago. For me, when I saw Vinyl, I felt like it was very much Comcast. Now, just to see it sitting on a Sky-owned domain makes it so much more bespoke for us to go and show it.

For me, it is about being a champion of something that I really think is of value to our business, let alone any business. For the road ahead, it is about taking that test environment and championing it out to the right departments at Sky, getting the right people connected with the tool, and highlighting the benefits for how it will empower us to do our work a lot more efficiently and drop a lot of administration in our own working environment. Sky has great talent, so why do we need to add so many approval lines? Giving that talent the ability to say, "Yes, we trust you enough to go in and make this DNS change without that many levels of approval," will really change the dynamics of the way people think and do their roles. For me, the road ahead is championing it out to Sky, really getting it out there, and getting people to get the feel of it.

Michael Winslow: I agree. We went through a major migration of VinylDNS from a previous system here, and I got to see so many people see the power of VinylDNS. Once we got everybody migrated over, it is business as usual at Comcast now. I am super excited to see how, once people start adopting it at Sky, at NBCUniversal, at DreamWorks if possible, I am excited for that next level of excitement from the users.

Nisha Parkash: For sure. One thing that is really good to know is that Vinyl allows different types of levels. Whether that is giving one person access to the root domain or a subdomain, I think that is great. You can set that bespoke approval level based on the type of team that you are dealing with.

Michael Winslow: How can you help? This is an open source project, so it is pretty easy. You can go to VinylDNS, take a look at it, take a look at the README files, possibly bring it down for yourself, point to another teammate who might be able to use it, and give us a star. Those are all ways you can help with that.

Another way you can help is invite us back in another year. I would love to be able to say, "This was the beginning of our journey," and perhaps show a thriving group of people at Sky using VinylDNS a year from now. I am hearing rumblings already that there are people who have started to take advantage over there, and we want to make sure that we are there to support them in any way we can.

With that, we thank you all for your time. We will be ready to take any questions right after this, and we appreciate you listening.

Nisha Parkash: Thanks very much. Thank you.