You Can't Automate The Fire

00:01 — Jamie Parker
IT automation is kind of a big deal right now. Everyone's trying to figure out how and where they can make the best use of this technology. We tend to think of companies deciding on a system that works for them as a unit, but that's not always the case. Sometimes you get multiple teams trying separate solutions. That can be good for discovery, but eventually you have to make a decision and corporate politics can be a real stick in the wheel.

00:32 — Jamie Parker
In this episode Vincenzo Sposito, Director of Intelligent Automation and Transformation at Discover, shares how they consolidated their efforts to build their automation systems. It wasn't easy to harmonize so many independent voices.

00:47 — Vincenzo Sposito
To say that it's all systematized and beautiful, I guarantee some internal team members would not totally agree with that statement. I think it's pretty solid.

00:55 — Jamie Parker
While strong opinions and debate can be intimidating, they can also be vital elements of a company's development process.

01:03 — Vincenzo Sposito
We want people invested in this. They had a burning passion or reason for automating to begin with, and that fire you can't replicate.

01:10 — Jamie Parker
Passion, dedication, and determination help propel development and motivate teams to implement automation. They see a goal and develop a vision to accomplish it the best they can. Or as we've heard in previous episodes, companies have found that passion can be a hindrance to getting buy-in or changing culture. Consensus can be difficult to achieve, but when that argument is shaped into a productive debate, it can help lead to a better solution. That's exactly what they do at Discover.

01:40 — Vincenzo Sposito
I don't want people just checking the box and moving along. I don't think there's a single topic we have where you get unanimous agreement across the entire program that it's like, "Oh, we figured it out." Because if everyone unanimously agrees and there's no one contradicting, I get a little concerned like, "Okay, is there a devil's advocate here?"

01:58 — Jamie Parker
Without that debate, how do you know if you're considering all your options to their fullest potential? How do you know if you've thought through all the possible problems? Discover has fostered a culture that invites discussion and consideration. Vincenzo is confident that in the long run, it has helped the company make the right technical decisions and that spirit of debate doesn't happen on its own.

02:22 — Vincenzo Sposito
So we have a lot of different forums and conversations. There's a developers' forum where it's engineers from across the company that are putting automations in place and they discuss topics, ideas, thoughts. If we're trying to decide on an enterprise pattern, we'll actually shop it with that group to see, what are folks' thoughts? What are the opinions? Where do we get backlash? And it's a real consideration that if a large majority of the organization doesn't agree with the direction we're going in, are we doing the right thing? And again, it's an inflection point. Sometimes it's a minority opinion that's actually the right answer.

02:57 — Jamie Parker
There's the key that we're going to hear more about. Discover has a great mechanism for people to openly assess and share their opinions about the benefits and drawbacks of certain solutions. At the end of the day, a decision is made. Going against the majority opinion isn't popular, but it can be the right decision based on factors not considered in the technical discussion. Vincenzo explained that these discussions aren't just a place for people to debate the finer points of technical efficiencies. Discover's forums are a place for dialogue.

03:30 — Vincenzo Sposito
It's a genuine conversation, "We're thinking about this. This is our rationale. Here's why we think this is the right approach. Give us your arguments." And in some cases we wind up taking a different strategic direction because of some insight that we gained from one of those forums. If you didn't come in and you just kind of steamrolled the conversations as an organization, you wouldn't get value out of doing that. It has to be that it's a genuine and open conversation with the team members.

03:56 — Jamie Parker
This played out when Discover took its first steps in their automation journey. Vincenzo likened it to convergent evolution where organisms that aren't closely related end up independently developing similar traits. Think of bats and birds developing the ability to fly even though they're in very different evolutionary paths.

04:15 — Vincenzo Sposito
We had a bunch of different groups in the organization saying we need some form of process or task automation a few years ago, and maybe one team is going to go work with a vendor to try and stand that up. Maybe one team was trying to develop something themselves in-house to tackle the same problem, and that's great and the value though that you get as a conglomerated company organization is the economies of scale.

04:40 — Jamie Parker
Having all these teams try out different solutions was great for the discovery process. Here's what works for us, here's what doesn't, and here's why. Teams tapped into their passion to build and put some stakes behind the systems they were building, but at some point, a company needs to choose one solution. It was time to centralize. Some would see their solutions become the norm while others would have to adapt.

05:07 — Vincenzo Sposito
Centralizing really allowed everybody to take advantage of being a part of one bigger system. We didn't need 17 different groups all running the platform basically in their own disparate processes. It made more sense for one team to run the platform and everybody just leverage those licenses and build in their own delivery space, but it didn't necessarily make sense to centralize all development activity.

05:31 — Jamie Parker
Centralization doesn't mean taking complete control. That can backfire pretty spectacularly. We've covered why it's likely to fail in our other episodes. A single team can't automate everything on their own. They don't have the expertise.

05:46 — Vincenzo Sposito
You need to be close to the work. You need to understand what the business's problems are, your products' concerns, your customers' concerns in order for you to effectively solve their problems. So we centralize some activities, we leave some other activities federated, and what you get at the end here is this little hybrid approach of what stays centralized and what stays decentralized.

06:09 — Jamie Parker
What that centralization does allow is to more quickly roll out automation projects to the rest of the company. It allows for easier integration of those projects into the overall system and it eliminates the need to build separate support systems to maintain and monitor the new automations, freeing up the developers to move on to the next project much more quickly. These are difficult decisions to make. Discover put together a council to make informed decisions from a variety of viewpoints.

06:41 — Vincenzo Sposito
Discover had this moment of inflection for when to centralize, how to centralize, and actually getting that body put together, deciding what went into it. And that was with a lot of help, we had an advisory council at the time comprised of a cross-functional group. We had human resources, finance, corporate risk management, all of the technology leaders, a few of our change leaders thinking through, okay, as a company what's the best way to approach this problem?

07:10 — Jamie Parker
That advisory council had to think about the process of delivering automation. How do you get from point A, not having automation, to point B, where you do? What are the steps involved with each solution? How many people are involved in getting through those steps? How do you set up a process where a team that doesn't own the original process gets what they need to build the automation system? Discover's advisory council considered all these questions and many more when looking at the options. And as a group, they picked the most viable option to build a framework that everyone would then adopt.

07:48 — Jamie Parker
The next step was to go from a recommendation to a formal structure.

07:53 — Vincenzo Sposito
The first step was identify yourselves as a program. This is a company-wide initiative and we are going to collectively come up with the answers. If you view yourself as a bunch of disparate groups trying to work together, you're never going to come up with those common elements because it wouldn't even occur to you that those are common elements. You would view them as your own individual problems to solve.

08:12 — Vincenzo Sposito
So first is the vision. Then comes the pragmatic approach to breaking down the vision. Once you've got all the elements, then it's just a matter of figuring out what's the most effective and efficient way of handling this?

08:24 — Jamie Parker
Altogether now, we love collaboration. Discover created a formal structure from which to organize and connect their automation efforts. From there, they've worked to identify collective issues and systematize how to solve them.

08:44 — Jamie Parker
When we return, we hear all about the framework they've developed to consider and complete automation projects.

08:55 — Jamie Parker
Discover put a framework in place for how to assess an opportunity, consider the risks, and build a solution that addresses any issues. And that framework relies on what they like to call inflection points. These inflection points are a series of moments in the timeline of a project that the teams use to determine whether to continue with automation and what kind of resources the project is going to need. Centralization has helped Discover systematize, to a certain extent, a common set of questions that apply to most potential projects. Vincenzo describes a few of these common inflection points at the evaluation stage.

09:34 — Vincenzo Sposito
Great, fantastic. That is a great idea to automate, but we need to know a few more things about it. What exactly are these reports that you're reconciling? If it winds up being just an internal system output to make sure that a batch ran overnight, cool, not a huge amount of risk with that data, that's all internal stuff.

09:51 — Jamie Parker
So if it's a low-risk process, it's pretty safe to continue. But some projects touch data that's more sensitive—accounting statements, credit dispute management, and more. Before even thinking about making changes to those sorts of systems, Discover ironed out its workflow so they knew what questions to ask and how the answers would affect outcomes. That way when they do make any changes, they've considered and planned for any associated risks.

10:18 — Vincenzo Sposito
So even just in that first question of, what are you working with, what is this process, describe to us what you're doing, we've already found possibly three different levels of risk associated with how we're going to develop this thing and it'll lead you down different paths.

10:35 — Jamie Parker
There's a protocol for following each of these paths. It helps move things along rather than coming up with custom plans from scratch at the start of each potential project. It's about asking the right questions in the right moments and experience has helped develop that set of questions and the knowledge of when to ask them. But these inflection points don't just determine how each project will proceed.

10:58 — Vincenzo Sposito
There will be inflection points where it's just like, "Hey, not today." The environment we've got, the scrutiny that we're under. Also just, can we safely protect this data along the path? It's like, if we can't, then no, do not pass go, do not collect $200, please continue doing your manual process. But a lot of times it really just tells us how difficult is this problem to solve and how many folks are we going to have to bring in to figure this thing out?

11:25 — Jamie Parker
Discover wants to automate as many processes as they can and benefit from the efficiencies that brings, but they also have to consider the sensitivity of the data they're handling, and sometimes the risk is just too high for their comfort. Most of the time though, they're able to move forward with the help of the central hub team. They just need to figure out how much help they're going to need.

11:47 — Vincenzo Sposito
If it's a low risk from an information perspective, it's a pretty repeatable process, it scales pretty naturally. That's kind of our bread and butter. That's the easy stuff, we get a few engineers in place and we'll be able to put something down. But if it's like, okay, this is actually taking an image and we're going to be reading that image in an unstructured way and we need to transfer that into a set of metadata and then use that metadata to make a decision around what the right path is for escalation or for follow-up, and that happens a billion times a minute.

12:24 — Jamie Parker
That kind of complicated project needs a larger team and possibly even some help from vendors to cover specialized work and bring in tools Discover might not have in-house. And sometimes the answers to the questions lead to a different kind of project altogether.

12:40 — Vincenzo Sposito
What are you actually trying to get done though? What's the point to doing that? It's like, okay, you're actually just trying to gain trust between these two systems' reconciliation process, why don't these two systems talk naturally to each other? Why wasn't that a part of the architecture? Okay, well, it's because one of them's on a mainframe and one of them is cloud-based, and we run too many queries to be able to do that effectively. Great. The question might be to actually just modernize the other tool and build an API and that might be the best version of automation.

13:09 — Jamie Parker
Look at that, a team that considers solutions outside of their own area of expertise. If it's not a nail, they're not going to try to hammer it down when there's another more effective way of going about it. Having that kind of expertise and ability to analyze a problem from multiple angles doesn't happen on its own. The way Discover modeled their teams encourages mobility and actually makes use of their team members' distinct backgrounds. They call it the hub and spoke model.

13:41 — Vincenzo Sposito
The hub we view as like what some organizations might call a platform team, it runs the central stack that actually maintains a lot of the licenses we use for the automation. It's not just one tool, it's a combination of tools that we put in place for every step of the process, so the elements we need for actually designing the automations themselves, putting the automations in production, how do we monitor those activities. So they manage stuff that every engineer would need, and then when you actually get to the spoke teams, they're dedicated product teams within each of the core lines of business who go out and actually develop the automation.

14:22 — Jamie Parker
Remember earlier, when we were talking about not centralizing everything and leaving the implementations to those closer to the product? The hub and spoke model is how Discover achieves that. The spoke teams are able to specialize, work with specific business partners and product teams, and identify unique opportunities to automate that a generalist team might not have the expertise to identify.

14:48 — Jamie Parker
As specialized as the spoke teams are, they still share a common set of activities, processes, and trials. The solutions they build get reported back to the hub and in many cases, redistributed to other spoke teams.

15:02 — Vincenzo Sposito
It's this really fun relationship between, I'm trying to build something, this group needs this in the next three weeks and we've never had a human in the loop process before. It's like great, cool, I think other teams are going to need human in the loop, like other spoke teams, is this also something? They're like, oh, yeah, we turned away seven use cases because we didn't have this. That's kind of how the hub/spoke relationship works. We have the teams out there going and deploying, and we have the platform team that kind of maintains the structure.

15:29 — Jamie Parker
The hub team maintains the platform, helps develop the tools, and provides technical support to the spoke teams when needed. The spoke teams do the work of building out the solutions, following a playbook of how to assess and take on potential automation projects.

15:45 — Jamie Parker
There's a fair amount of training and mentorship to get people up to speed, and though the spoke teams may have deep knowledge of the products they're working with, signing onto a specific spoke team doesn't mean they need to stay there. It's quite the opposite. Some amount of mobility is encouraged.

16:02 — Vincenzo Sposito
Even at a talent perspective, conversations back and forth to make sure there's growth and opportunity between the groups. Like, okay, it looks like someone on the engineering team would really love to see how this applies to the entirety of the company. Great. Maybe that's a time for them to shift into the hub team for a little while, help us build out two or three cool new capabilities and functionality, and then maybe if they want to go back into the delivery world, they can. And it's a nice relationship between taking the enterprise view versus the individualized view, moving between spoke teams to get different areas of context from a product perspective.

16:35 — Vincenzo Sposito
So really, at a technology, a process, and a people level, the hub/spoke relationship has this dialogue of what should be centralized, what should be decentralized, and how do we both get the best efficiency of each situation.

16:51 — Jamie Parker
People move around, they learn from each other, they grow. According to Vincenzo, it's also a great way to keep people engaged with their work and it helps foster a closely knit community.

17:04 — Vincenzo Sposito
Also from a problem-solving perspective, that's probably one of the most untapped things that we're trying to add a little bit more structure to. You solve a problem in your situation, in your unique case, and you think that it is a unique case until you start looking around and seeing, oh no, actually everyone deals with that problem. I've actually solved something that 17 other teams would benefit from.

17:25 — Jamie Parker
People who have moved around to new teams have discovered that the problems they've had aren't unique and the solutions they've built can be quickly applied to their new situation. Having a number of spoke teams available helps spread the workload and keeps the work close to the product. That doesn't mean the spoke teams cover every aspect of the business, but the number of spoke teams isn't static either.

17:48 — Vincenzo Sposito
It definitely plateaued a little bit there. You kind of understood after the first two or three, okay, where are other processes like this happening within the organization and what really is the leftover opportunity? We kind of talked with a few of the leaders and we preempted where we thought spokes would go.

18:07 — Jamie Parker
In the places where they didn't plan to have a spoke team, the hub picked up any projects that might come up, especially when they were complex problems that span multiple products. When requests from a certain uncovered part of the business start to stack up, it's time to consider an addition.

18:23 — Vincenzo Sposito
They handle teams that don't have a spoke team but do need automation and it makes sense for them to automate. So, what we would wind up seeing is, actually we're getting a ton of requests coming over from this product group. It might make sense for them just to get a dedicated resource. Let's go talk with a director over there and see if they're ready to stand this up.

18:43 — Jamie Parker
In the first few years of the program, that scenario played out many times until it didn't anymore. They found a sort of equilibrium where the spokes were handling requests at a good pace and the number of teams hit a plateau. But nothing stands still forever.

19:01 — Vincenzo Sposito
I think we're about to—if I can stretch the metaphor—it's like we're about to add onto the wheel again because as we find more capabilities, it opens up new problems. And maybe groups that may not have had the more generalized task automation needs will have a consolidated set of problems with these more advanced technologies we're putting in place.

19:20 — Jamie Parker
As their expertise grows in tandem with the sophistication of the systems they're creating, they're finding they can handle more complicated challenges than before, maybe even some of those previously blocked because the risk was too high at the time. All of that was possible thanks to the judicious application of centralization—enough to get everyone working together and sharing their work, but not too much to lose out on the benefits of individual agency and discovery. That's not an easy balance to find, but thanks to their common set of tools, a standardized framework, and the freedom to move around, it sure seems like it's been worth the effort for Discover.

20:06 — Jamie Parker
That's it for Season 2 of Code Comments, but don't fret, Season 3 is coming soon. Stay tuned for more words of wisdom from me, your superlative host, Jamie Parker—and our guests too, of course.

20:22 — Jamie Parker
You can read more at redhat.com/codecommentspodcast, or visit redhat.com to find out more about our automation solutions. Many thanks to Vincenzo Sposito for being our guest, and thank you for joining us.

20:36 — Jamie Parker
This episode was produced by Johan Philippine, Kim Huang, Caroline Creaghead, and Brent Simoneaux. Our audio engineer is Robyn Edgar. The audio team includes Leigh Day, Stephanie Wonderlick, Mike Esser, Nick Burns, Aaron Williamson, Karen King, Jared Oates, Rachel Ertel, Carrie da Silva, Mira Cyril, Ocean Matthews, Paige Stroud, Alex Traboulsi, and Victoria Lawton. I'm Jamie Parker, and this has been Code Comments, an original podcast from Red Hat.