I will frame the question in terms of a specific C# objective that I am working on right now but I imagine the question is a pretty general one related to the Dunning-Kruger effect, in a way - how do you know how to build an application when you don’t know all the issues you are supposed to prevent?
There is a message hub and I am developing a consumer for it. The original plan was to just create a few background services that get initialized alongside the application, that have a loop to load new messages/events and then process them.
Some time has passed and it feels like I am knees deep in Wolverine, Quartz, Hangfire, MassTransit, transactional outbox and all manner of different related things now. The potential issues are dealing with downtime, preventing loss of messages by storing them in a separate table before processing them, and everyone on the planet has a different idea on how to prevent and solve them and none of them sound simple and straightforward.
Honestly at this point I don’t know enough about which problems are going to appear down the line and if I need to use third party libraries, but I am guessing they exist for a reason and people aren’t supposed to just manually create their own amateurish implementations of them instead? But how do you know where to draw a line when you don’t know exactly the problems that you are supposed to be solving?
What are the problems with having a table for the message queue over a whole 3rd party library for it, or what’s wrong with the MS BackgroundService class? How are you supposed to know this?
Thinking you can even prevent all unkowns is a foolish endeavor. Only work on solving the problem at hand. Why would I think about some future requirement and lose sleep over it? That’s the PM and sales team headache! Or think of it another way… you haven’t yet reached the bridge yet but you’re already thinking you need rope for the imagined bridge collapse.
All you can do with the present is make it relatively easier for your future self. Avoid complexity (thanks grug brained programmer!), don’t tie yourself into knots and back yourself in corners (keep the code readable, testable and simple. This means minimal external dependencies, lose coupling but good coherence, and avoiding reinventing solution to difficult to solve problem)
some of these goals seem contradictory but you have to apply them to a specific problem and the objective of solving the problem (why are you solving this problem?). For instance, for crypto the advice of avoiding reinventing the solution takes precedence to minimal dependencies one because your objective is security which is important to get right without tolerance for bugs.