Post: #1

I work in IT for a large company. My role is to help deploy a new document management system company wide. We started rolling out the system to the UK first.

Yesterday, I noticed that for whatever reason regular users had complete access to a special "admin control" button. I flipped because we keep both internal and external vendors in this system. This is bad because there's sensitive financial data on the system and this admin control button could allow anyone to basically go in, read, and change it. Mostly relating to payroll data and vendor contracts stating pay.

It's not my fault the button appeared (thank god), but our entire dev is based in the states (it was MLK day) and i'm on my own in the UK. I opened a critical ticket with our development tracking system which is SOP, but then to inform my support team i opened a critical ticket in our regular incident system used to report general issues through a standard support portal.

Well, doing the second ticket ended up getting a lot more people involved who don't participate with my team. A few big wigs got attached to some email chains and my bosses boss who has been out for MLK day got some nastygrams about why this wasn't fixed considering what was affected and how many users were impacted. I'm pretty sure this went all the way up to the C level suite looking at the names involved.

This issue affected about 500ish people and was considered a high priority with this outside team for my developers to get resolved.

I'm at a crossroads however, I have a gut feeling I should just own this and say that this bug was sloppy and exposed sensitive data. This issue exposed a lot of structural problems (no on call engineer, lapse in QA procedures, everyone checking out for a holiday). These need to be fixed if we are to provide this service to the degree of perfection needed to avoid getting outsourced.

However at the same time, i think I may have shot myself in the foot professionally by calling so much attention to this and embarrassing a lot of people in the process. We all work in a ton of different areas globally, but i get the impression i'm fucked for the future and will be skipped over for a promotion for my vigilance.

I'm leaning towards number 1 and just own it considering what was at stake. How would you guys roll on such an issue?

Post: #2

I can't offer any direct advice, but I know that most people's first reaction to arduous/stressful issues is based on emotion, causing catastrophic thinking.

It may not have the impact that you believe it will, regarding future promotion etc.

You could just be being hard on yourself. Time will tell I guess. Good luck.

Post: #3

Hey man. Fellow IT guy here. It's hard to give any advice without knowing the particulars of your specific infrastructure but I'll go.

(01-19-2016 06:11 AM)The Beast1 Wrote:  It's not my fault the button appeared (thank god)

Then at the end all should be well, maybe you'll get a hit on your political standing within the firm but that's it.

(01-19-2016 06:11 AM)The Beast1 Wrote:  However at the same time, i think I may have shot myself in the foot professionally by calling so much attention to this and embarrassing a lot of people in the process.

How did the second ticket get escalated all the way to the C levels? It sounds like you took a shit and someone on your team placed a cherry on top by talking about it to his immediate superior, and since nobody has any idea what was compromised, the manager just heard the "payroll data and vendor contracts stating pay" part and shit just went downhill.

Since it's a big company your system probably keeps access logs, go in and find out if anything was compromised or if anyone changed anything, check the database backups and push people for a rollback if you're allowed to (and if it's necessary of course) Write a comprehensive report about what actually happened, how did it happen, what was at stake, what was compromised and what steps should you take/ have you taken in order to fix the issue. Write in detail yet simple enough so non technical people would read it, understand what its about and pass it higher up the chain.

At the end paint yourself as the guy who discovered and did his best to fix somebody else's shit.

Basic game principles really

Edit: Also what kind of admin control button is it? Does it expose some sort of API which someone tech savvy can use to gain backend access or does it take you to some sort of admin panel directly?

Post: #4

I'd man the fuck up, own the situation and do what needs to be done. If I'd already "caused" attention and embarrassment enough to devalue myself in their eyes then there's nothing much to lose. Perhaps doing so would redeem myself.

Post: #5

I don't see where you need to "own" anything or apologize. I really don't. I've been in a professional environment for years and see this every once in a while, albeit in a Finance environment rather than IT. Every time it happens, it's the result of a bigger issue. In other words, lots of steps gone wrong or not taken due to x, y or z. In this case, none of these seem to have come from you and the primary reason appears to be the MLK holiday and a lack of resources.

Look at it this way. If you hadn't exposed this, do you realize the amount of damage that could have been done? The fact that you involved more people doesn't change that. You're just one cog in that wheel.

Before jumping to conclusions about how bad it is FOR YOU, talk to your manager and ask a simple question, "what should I have done differently. " If he says that you shouldn't have involved so many people by opening the second ticket, let him know that it was you wanted to "fix the issue, given its magnitude." Notice his reaction. If he's a reasonable manager, you should feel a sense of understanding at this point. If you don't, there is a good chance blame is trickling down to him and then you. That's not a person you want to be around too much longer.

The positive from this situation is that you'll know the character of those involved in your organization.

Let us know how it goes. I'm looking forward to hearing back on how this is settled and happy to offer thoughts.

Post: #6

(01-19-2016 08:54 AM)the Thing Wrote:  Hey man. Fellow IT guy here. It's hard to give any advice without knowing the particulars of your specific infrastructure but I'll go.

(01-19-2016 06:11 AM)The Beast1 Wrote:  It's not my fault the button appeared (thank god)

Then at the end all should be well, maybe you'll get a hit on your political standing within the firm but that's it.

(01-19-2016 06:11 AM)The Beast1 Wrote:  However at the same time, i think I may have shot myself in the foot professionally by calling so much attention to this and embarrassing a lot of people in the process.

How did the second ticket get escalated all the way to the C levels? It sounds like you took a shit and someone on your team placed a cherry on top by talking about it to his immediate superior, and since nobody has any idea what was compromised, the manager just heard the "payroll data and vendor contracts stating pay" part and shit just went downhill.

Since it's a big company your system probably keeps access logs, go in and find out if anything was compromised or if anyone changed anything, check the database backups and push people for a rollback if you're allowed to (and if it's necessary of course) Write a comprehensive report about what actually happened, how did it happen, what was at stake, what was compromised and what steps should you take/ have you taken in order to fix the issue. Write in detail yet simple enough so non technical people would read it, understand what its about and pass it higher up the chain.

At the end paint yourself as the guy who discovered and did his best to fix somebody else's shit.

Basic game principles really

Edit: Also what kind of admin control button is it? Does it expose some sort of API which someone tech savvy can use to gain backend access or does it take you to some sort of admin panel directly?

Thanks for the reply, here's why it got kicked up so quickly.

We have two ticketing systems, one is set up for internal development of the company's applications. The second one is more of a general use system where users can open tickets. This is the same system where you'd logged a, "my computer doesn't work" type of issue.

The reason it got kicked up so quickly was because I listed the initial ticket as "critical" and high priority. This set off a bunch of alerts and was sent to a team specifically designed to handle such incidents. When I got the first email, I recognized some big names attached to it.

Since the summary of my ticket had pretty much all of the details related to what the admin panel can do, as well as what was at stake (financial data i mentioned above), and that they're all aware of the system since it's being rolled out company wide they considered it urgent.

For whatever reason, all of our production users were attached to the "Admin" group hence why everyone had access to it all at once. I'm not sure why this occurred.

We have logs and so far nothing has been compromised or accessed inappropriately which is reassuring. Thankfully it seems our users didn't care to try and play with it. The admin panel itself is a basic front end of a lot of database commands, anyone who can read english has enough knowledge to know what it does and how to use it. Basically keys to the castle type of stuff.

I have a call in 3 hours with my dev team and I know for a fact i'm going to get shat on for this, but considering how big of an egg this is for the team i'm grateful it got looked at quickly.

I didn't realize that tickets listed as critical set off a chain of events in the general ticketing system. There's an even bigger call tomorrow with this high priority incident manager to discuss why this happened and how it will be prevented in the future. Sadly this is above my pay grade at the moment.

Edit: I should add by "own" it I mean to not apologize to anyone for the decision process I did that lead to all of this. I opened the critical ticket on the general system which set off the chain of events. It was critical by everyone's standards (as well as my own) and could have led to some general embarrassment within each of the divisions as well as make negotiating pay trickier with outside contractors.

I guess i'm more worried about the collateral damage to my rep. I'll let you all know how it turns out!

Post: #7

(01-19-2016 09:19 AM)PAL Wrote:  I'd man the fuck up, own the situation and do what needs to be done. If I'd already "caused" attention and embarrassment enough to devalue myself in their eyes then there's nothing much to lose. Perhaps doing so would redeem myself.

I'd disagree. Manning up and doing "what needs to be done" doesn't mean defer to the frame of someone else and "redeeming yourself" doesn't mean redeem yourself based on someone else's standards especially when you don't agree with them. Again, figure out what you value first before you defer and succumb to what others value.

Post: #8

(01-19-2016 09:25 AM)Cobra Wrote:  

(01-19-2016 09:19 AM)PAL Wrote:  I'd man the fuck up, own the situation and do what needs to be done. If I'd already "caused" attention and embarrassment enough to devalue myself in their eyes then there's nothing much to lose. Perhaps doing so would redeem myself.

I'd disagree. Manning up and doing "what needs to be done" doesn't mean defer to the frame of someone else and "redeeming yourself" doesn't mean redeem yourself based on someone else's standards especially when you don't agree with them. Again, figure out what you value first before you defer and succumb to what others value.

It's not really a fuck up on my part at all. The only thing I guess i'm really concerned with is pissing some lazy people off. I thankfully don't deal a lot with a lot of office politics because we're spread all around the world. But I have the feeling I made some folks look bad and this may come to haunt me in the future. Chalk this up to one of those life lessons.

Post: #9

(01-19-2016 09:21 AM)The Beast1 Wrote:  

(01-19-2016 08:54 AM)the Thing Wrote:  

Thanks for the reply, here's why it got kicked up so quickly.

We have two ticketing systems, one is set up for internal development of the company's applications. The second one is more of a general use system where users can open tickets. This is the same system where you'd logged a, "my computer doesn't work" type of issue.

The reason it got kicked up so quickly was because I listed the initial ticket as "critical" and high priority. This set off a bunch of alerts and was sent to a team specifically designed to handle such incidents. When I got the first email, I recognized some big names attached to it.

Since the summary of my ticket had pretty much all of the details related to what the admin panel can do, as well as what was at stake (financial data i mentioned above), and that they're all aware of the system since it's being rolled out company wide they considered it urgent.

For whatever reason, all of our production users were attached to the "Admin" group hence why everyone had access to it all at once. I'm not sure why this occurred.

We have logs and so far nothing has been compromised or accessed inappropriately which is reassuring. Thankfully it seems our users didn't care to try and play with it. The admin panel itself is a basic front end of a lot of database commands, anyone who can read english has enough knowledge to know what it does and how to use it. Basically keys to the castle type of stuff.

I have a call in 3 hours with my dev team and I know for a fact i'm going to get shat on for this, but considering how big of an egg this is for the team i'm grateful it got looked at quickly.

I didn't realize that tickets listed as critical set off a chain of events in the general ticketing system. There's an even bigger call tomorrow with this high priority incident manager to discuss why this happened and how it will be prevented in the future. Sadly this is above my pay grade at the moment.

Edit: I should add by "own" it I mean to not apologize to anyone for the decision process I did that lead to all of this. I opened the critical ticket on the general system which set off the chain of events. It was critical by everyone's standards (as well as my own) and could have led to some general embarrassment within each of the divisions as well as make negotiating pay trickier with outside contractors.

I guess i'm more worried about the collateral damage to my rep. I'll let you all know how it turns out!

I see. Since nothing was compromised and it's fixed already (I assume you've revoked permissions to people who don't need them) this is reduced to a question of holding frame.

You said you help deploy this system. If it wasn't your responsibility to assign users to groups it's not your fuck up at all and don't let the dev team shit on you. If it's due to a bug in the system where all new users are created with admin rights you can even counter shit on the dev team.

Talk to your managers who know you as a hard working guy who swiftly gets the job done, and get them to hold your back on this, since you didn't do nothing wrong.

From many years of experience I've gained in working as an IT consultant, the most important thing I have learned is: most (if not all) people higher up the chain know absolutely jack shit about how these things actually work and if you hold a frame of authority they will say whatever you take at face value. Unless the person who was responsible for this, the person you made look bad, and the person who'll be promoting you aren't the same person, hold your frame and you should be golden.

Post: #10

(01-19-2016 09:21 AM)The Beast1 Wrote:  For whatever reason, all of our production users were attached to the "Admin" group hence why everyone had access to it all at once. I'm not sure why this occurred.

A lot will depend on where the lapse was.

As you mentioned, there are QA processes. People test the software before it goes live. Sometimes there's a test plan. Someone signs off that it's been tested and it's ready to go live. If there are any major screw ups, responsibility is not uniquely with the development team, but the testing team too.

Maybe the software was fine, and it was just that the person who set up permissions screwed it up. In that case, responsibility is much easier to pin down.

What are the lessons you could learn? When something looks highly visible, just palm off the decision to your manager. They make the big bucks, let them earn it. Tell them the situation and ask them what's the best approach. It's good that you inform them up front anyway, so they can see it coming and handle it.

Hopefully this would die down quickly, since it can be proven from the logs that nothing was unduly accessed or changed.

Post: #11

Thanks gents, I talked with a few colleagues about this and they all agree that what I did was proper considering it would have put the company in a bind if that financial data had been accessed.

I have that meeting in 10 minutes so I'll let you all know how it turns out!

Post: #12

(01-19-2016 09:39 AM)The Beast1 Wrote:  The only thing I guess i'm really concerned with is pissing some lazy people off. I thankfully don't deal a lot with a lot of office politics because we're spread all around the world. But I have the feeling I made some folks look bad and this may come to haunt me in the future. Chalk this up to one of those life lessons.

When I worked at a Fortune 500, I had a friend who got demoted for basically "being a snitch."

Sucked when it happened, but the upside is that we all then knew how corrupt the company was.

Post: #13

(01-19-2016 11:53 AM)The Beast1 Wrote:  Thanks gents, I talked with a few colleagues about this and they all agree that what I did was proper considering it would have put the company in a bind if that financial data had been accessed.

I have that meeting in 10 minutes so I'll let you all know how it turns out!

So how'd it go?

Post: #14

(01-19-2016 07:32 PM)AneroidOcean Wrote:  

(01-19-2016 11:53 AM)The Beast1 Wrote:  Thanks gents, I talked with a few colleagues about this and they all agree that what I did was proper considering it would have put the company in a bind if that financial data had been accessed.

I have that meeting in 10 minutes so I'll let you all know how it turns out!

So how'd it go?

Maybe they killed him.

Post: #15

(01-19-2016 07:32 PM)AneroidOcean Wrote:  

(01-19-2016 11:53 AM)The Beast1 Wrote:  Thanks gents, I talked with a few colleagues about this and they all agree that what I did was proper considering it would have put the company in a bind if that financial data had been accessed.

I have that meeting in 10 minutes so I'll let you all know how it turns out!

So how'd it go?

So far, nothing! I blew this out of proportion and it looks like the holes mentioned are being fixed. People appear to be owning up to their mistakes and fixing them.

Thanks for the re-assurance gents, there's a bigger meeting today about it but I can't attend as I have other tasks to take care of.

Post: #16

find a scape goat just in case.

Post: #17

Stalin is dying, and summons Comrade Khruschev to his bedside. Wheezing his last few words with difficulty, Stalin tells Khruschev, "Comrade, the reins of the country are now in your hands. But before I go, I want to give you some advice."
"Yes, yes, Great Leader, what is it?" says Khruschev.
Reaching under his pillow, Stalin produces two envelopes marked 1 and 2. "Take these letters," he tells Khruschev. "Keep them safely--don't open them. Only if the country is in turmoil and things start going badly, open the first one. That'll give you some advice on what to do. And, even after that, if things start going REALLY badly, open the second one." And with a gasp Stalin breathed his last.
Well, Khruschev succeeded him, and sure enough, within a few years things started going badly--unemployment increased, crops failed, people became restless. Nikita decided it was time to open the first letter. All it said was: "Blame everything on me!" So Khruschev launched a massive deStalinization campaign, and blamed Josef for all the excesses and purges and ills of the present system, and bought himself some time that way.
But things continued on the downslide--Kennedy successfully rebuffed Soviet missiles in Cuba, unemployment increased even more, crops failed even more, the Politburo was unhappy with Khruschev's leadership and upstarts like Brezhnev and Gromyko were threatening his credibility. So finally, after much deliberation, Nikita opened the second letter.
All it said was: "Write two letters."

Post: #18

(01-20-2016 06:35 AM)The Beast1 Wrote:  

(01-19-2016 07:32 PM)AneroidOcean Wrote:  

(01-19-2016 11:53 AM)The Beast1 Wrote:  Thanks gents, I talked with a few colleagues about this and they all agree that what I did was proper considering it would have put the company in a bind if that financial data had been accessed.

I have that meeting in 10 minutes so I'll let you all know how it turns out!

So how'd it go?

So far, nothing! I blew this out of proportion and it looks like the holes mentioned are being fixed. People appear to be owning up to their mistakes and fixing them.

Thanks for the re-assurance gents, there's a bigger meeting today about it but I can't attend as I have other tasks to take care of.

Cobra and The Thing's responses were spot on. In my experience, these situations usually resolve themselves exactly as you describe above. There's an initial "the sky is falling" response, followed by people getting in there and getting it fixed while downplaying the possible damage that was caused in order to help keep anyone from getting fired unless they criminally fucked up. So, strictly following the book in your initial response likely protected you from any negative fallout.

If you're working for an organization that tries to fuck you because you report something that's wrong, then that tells you that you need to get the heck out of there.