Join devRant
Do all the things like
++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatar
Sign Up
Pipeless API
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple API
Learn More
Search - "never touch a running system"
-
I couldn't sleep. I was staring at the blinking cursor. A slow, comforting blinking. Like everyone else, I had become a slave to the JavaScript ecosystem. If I saw something like a new build system, or a new framework, I had to have it.
My client changed the requirements again. I'm in pain.
- "You want to see pain?" my colleague said. Go read Apple support forums. That's pain.
I became addicted. Every time I died and every time I was born again. Resurrected.
During the night, I was crying in the Apple forums for an official answer that would never come. During the day, I was surfing StackOverflow to fix my problems. You get "single-serving" friends there. They help you, you help them, and then you never see them again.
- "Then you install Stack and boom, you're done. It's that easy to go functional."
That's how I met him.
- "You know why they make so many javascript frameworks?"
- "No, why?"
- "So that they can distract you while they put backdoors in them. So that you don't have time to check all of their code".
- "You are by far the most interesting "single-serving" friend I've ever met"
Then, my hard disk died. Of course, I didn't have backups: nobody has enough space for all those node_modules folders. All my addictions, lost.
Then I wrote him. If you asked me now, I couldn't tell you why I wrote him. We chatted a lot.
- "It's late, I should really go search another hdd on ebay"
- "Ebay? You called me so you could have my old hard disk."
- "No, I..."
- "Come on."
He sent me his old hard disk. It was a 256MB hard disk, but it was fine for running Arch. Then he asked me to rant about my problems in front of him.
- "I want you to rant as hard as you can"
- "Are you serious?"
We ranted all night about our bosses and clients and their fucked up requests. We kept in touch, and after a while more people were ranting with us. Every week, he gave the rules that he and I decided.
- "The first rule of devRant is -- you don't talk about devRant. The second rule of devRant is -- you don't talk about devRant."
I like to think this is how devRant started. This might also be the reason why we never see @trogus, only @dfox. A lot of shit still needs to happen.8 -
"Never touch a running system". I can't believe how many times I've seen this advice here. How the fuck are you supposes to upgrade, improve stuff? This is why stuff like WannaCry happens.8
-
When the CTO/CEO of your "startup" is always AFK and it takes weeks to get anything approved by them (or even secure a meeting with them) and they have almost-exclusive access to production and the admin account for all third party services.
Want to create a new messaging channel? Too bad! What about a new repository for that cool idea you had, or that new microservice you're expected to build. Expect to be blocked for at least a week.
When they also hold themselves solely responsible for security and operations, they've built their own proprietary framework that handles all the authentication, database models and microservice communications.
Speaking of which, there's more than six microservices per developer!
Oh there's a bug or limitation in the framework? Too bad. It's a black box that nobody else in the company can touch. Good luck with the two week lead time on getting anything changed there. Oh and there's no dedicated issue tracker. Have you heard of email?
When the systems and processes in place were designed for "consistency" and "scalability" in mind you can be certain that everything is consistently broken at scale. Each microservice offers:
1. Anemic & non-idempotent CRUD APIs (Can't believe it's not a Database Table™) because the consumer should do all the work.
2. Race Conditions, because transactions are "not portable" (but not to worry, all the code is written as if it were running single threaded on a single machine).
3. Fault Intolerance, just a single failure in a chain of layered microservice calls will leave the requested operation in a partially applied and corrupted state. Ger ready for manual intervention.
4. Completely Redundant Documentation, our web documentation is automatically generated and is always of the form //[FieldName] of the [ObjectName].
5. Happy Path Support, only the intended use cases and fields work, we added a bunch of others because YouAreGoingToNeedIt™ but it won't work when you do need it. The only record of this happy path is the code itself.
Consider this, you're been building a new microservice, you've carefully followed all the unwritten highly specific technical implementation standards enforced by the CTO/CEO (that your aware of). You've decided to write some unit tests, well um.. didn't you know? There's nothing scalable and consistent about running the system locally! That's not built-in to the framework. So just use curl to test your service whilst it is deployed or connected to the development environment. Then you can open a PR and once it has been approved it will be included in the next full deployment (at least a week later).
Most new 'services' feel like the are about one to five days of writing straightforward code followed by weeks to months of integration hell, testing and blocked dependencies.
When confronted/advised about these issues the response from the CTO/CEO
varies:
(A) "yes but it's an edge case, the cloud is highly available and reliable, our software doesn't crash frequently".
(B) "yes, that's why I'm thinking about adding [idempotency] to the framework to address that when I'm not so busy" two weeks go by...
(C) "yes, but we are still doing better than all of our competitors".
(D) "oh, but you can just [highly specific sequence of undocumented steps, that probably won't work when you try it].
(E) "yes, let's setup a meeting to go through this in more detail" *doesn't show up to the meeting*.
(F) "oh, but our customers are really happy with our level of [Documentation]".
Sometimes it can feel like a bit of a cult, as all of the project managers (and some of the developers) see the CTO/CEO as a sort of 'programming god' because they are never blocked on anything they work on, they're able to bypass all the limitations and obstacles they've placed in front of the 'ordinary' developers.
There's been several instances where the CTO/CEO will suddenly make widespread changes to the codebase (to enforce some 'standard') without having to go through the same review process as everybody else, these changes will usually break something like the automatic build process or something in the dev environment and its up to the developers to pick up the pieces. I think developers find it intimidating to identify issues in the CTO/CEO's code because it's implicitly defined due to their status as the "gold standard".
It's certainly frustrating but I hope this story serves as a bit of a foil to those who wish they had a more technical CTO/CEO in their organisation. Does anybody else have a similar experience or is this situation an absolute one of a kind?2 -
rent / question (there is a question at the end and I'd appreciate your opinion)
8 months ago, I agreed to help a not too distant relative of mine to do his master thesis at the company where I work. He was supposed to build something really MVP, but useful for us and I'd help him get some scientific questions out of it, and provide him with (computing) resources to test his theories / implementations under simulated and much heavier load.
Since then, he didn't get done anything even remotely useful, always just stuck on very rudimentary issues, claimed things are almost ready, I wrote a quick smoke test to prove that the whole application blows up when you touch it, in short - a disaster and went over to radio silence.
In the meanwhile, we didn't need it anymore, so 1.5 months ago, I got in touch with him again, with an even more technical proposal, something, at least I'd think, that's even cooler to do. He asked me some question about hypothetical load, the system should be able to handle eventually, to come up with alternative implementations to compare them against each other. He said that his exam period is going to be over soon and he'll get back to me with some initial version.
2 weeks ago, I got back in touch with him, trying to urge him, to get finally started and get something done. If he'd actually sit down and do it during the holidays as a "full time job", he'd be probably done in 2 weeks. Last week, he came back to me and said he has an initial PR ready to review.
I was excited about it, but basically froze when I realized what he did. He deleted all his previous work - some infrastructure stuff which took us basically 3 months of back and forth to get running - and as far as I could see, all the new code were only auto generated clients based on a swagger specification. In short - I could do it in less then an hour. If you really have no idea what you're doing, it might take you half a day, but definitely nowhere near to a week.
His brother, which a good friend of mine, thinks I'm being too hard on him. His argument was, that it's too hard, and he has to do it in C#, but he only knows Java (I gave him access to some of our repositories to copy paste code together, he didn't need to invent anything. I also prefer C# but wrote my master thesis in Java) Personally, I'm just pissed because he promises stuff that he never does. I totally understand him - I was like that as a student as well, I guess karma is a ... but still, he's wasting my time.
Right now I'm thinking how to get out of this, without having even more time wasted. I doubt he'd ever deliver anything useful. He got plenty of input from me about what he could consider for his scientific question, how to measure performance, ... He can keep his credentials to access our test environment with the test data, but I won't give him access to any additional computing resources, to compare how his solutions might scale on our company's cost. (mainly it's not the money, but I'd have to provide that stuff, and probably help him set it up)
does it sound like a fair deal (saying, I'm done with you. You can finish your topic on your own, but don't expect any help from me)? or am I being a dick about it and too demanding?1 -
This moment, you know ur movement mechanic in Unity works just as u want it to be. But its so damn hacky, u are afraid of refactoring, because u know.. u would break it...1