Prioritizing goes hand in hand with efficiency and productivity whether it’s extreme (like triage) or the mundane (like paying bills), but how often do we actually stop and think about the process?
While at the bar yesterday my friend asked me a question about how I deal with my data. In particular the question was about how I prioritize whether to spend time making a reusable code verses a one-and-done hack job just to get it done. Since we are all familiar with the options, having taken one or the other on numerous occasions, why do we choose one over the other in any particular case? Also, is there a rule of thumb that can help simplify this decision?
An example from my own research would be figure generation from an ecosystem model. Since it is a task which I have to do once for each time I run my model, it is plainly simple to see that a script that automates the process is well worth the effort. At the other end of the spectrum there are analyses which I only plan on running once (for example those which I wanted to try out yet didn’t prove fruitful in the end). Should I automate those as well with scripts?
Not only would doing so be an extra investment in time, but without a guarantee of being used again the script I write would likely not be well adapted to future use cases. An analogous example would be in writing a tutorial for something new you’re learning at work. That first time you use the fancy new printer/copier at work–the one with the hole punch and three different paper sizes–would likely be a wait of time. Instead, once you’ve used it a few times, your notes will not only be clearer but also more efficient since you have experience to guide you. Writing scripts and building pipelines is much the same: experience is an essential guide throughout the process.
So personally I only consider making a pipeline (e.g. a script, a cheatsheet or instructions) after having already completed the task a couple times (typically 3+). This is my rule of thumb:
Do it a few times to get good at it then automate (if you’re going to do it at least a few more times).
Deadlines & Time Frames
Of course, when it comes to deadlines, all bets are off. These are the circumstances which are out of our control and that quite often necessitate an inefficient workflow for the sake of completion. Cramming to pump out a few half-decent figures and to get them into a concise and clear ppt is rarely a good use of time, objectively; yet it is often asked of us. It is in these situations when already-completed pipelines can really shine but simultaneously the time to build one is often inadequate.
It would be negligent to not point out an article written by the infamous Casey Muratori regarding his theory termed “Semantic Compression” that overlaps greatly with what I have discussed here. While my rule of thumb is meant to be applied generally, Casey’s is a programming-specific methodology that I hope you will all find as insightful and useful as I did.