Open Sourcing – Refactoring and Over-Engineering

The work I’ve been doing over the last week or so has centered around a refactor of our modules. We had an ambitious goal of upgrading the functionality of the Observer module so that it would accept a JSON config file that would allow them to expand the functionality. My previous blog posts detailed this. I set out to completely re-organize the Observer module to achieve a few goals:

  • Import functions from the library (it’s now an NPM module) as they’re needed, as specified by the config JSON file. This would allow our code to be selectively ignored for stuff the user didn’t want, and expanded based on new stuff they want, and it would mean a smaller JavaScript file to be sent to the browser since it wouldn’t include things it doesn’t need.
  • Break the individual “hooks” and “listeners” into their own files, to be imported together. This would make it clearer to maintain. No more 1000 line source code files.

I started to run into problems as I did this.

Here’s what went well:

  • Reorganizing the functions into their own files helped readability and maintainability a lot. It was suddenly easier to see how the structure of the project was laid out.
  • The dynamic importing of JavaScript worked, and I was able to use simple JSON to specify what custom behavior to be added. The user just had to write “click” and “user-profile” for example to start tracking clicking on user profile link.

But here’s what didn’t work well:

  • Reorganizing the functions into individual files meant that they had to be recombined. This is easy in most cases, you just nest imports. The fully loaded JS object you import comes from a source code file that imports the individual properties of the object. However, we use lots of middleware. We have error loggers and data loggers and caching and debugging… It ended up being really tricky to get all these pieces to play nice with each other. My team mate had already solved this technical problem with his creation. It may have monolithic source code files with over 1000 lines of code, but it Just Works, and that’s valuable at this later stage of the project.
  • Even though the functions could be dynamically imported, it didn’t reduce the JS file size because of the way compilation works. It needs to pull in all functions that *may* run. I could have wrote an additional tool to clean up the JS file to remove functions that aren’t needed anymore, but that would just add to complexity and our workload.
  • The JSON config is simple (way easier for a user to use than if they had to dive in and add JS functions to what we have in order to extend functionality) but it isn’t powerful enough. We need to consciously think about the ways the user might extend functionality. There’s JS events we haven’t even thought about. We’re too busy for that. We’re working on getting this thing published and documented! It may be a better idea right now to back off of this and just let the user be responsible for writing more code if they want more functionality, or perhaps allow the future open source users to contribute code to add to this.

After spending all this time looking at the source code and getting a better idea of how it works (my team mate did most of this module earlier in the project), I’ve come to the conclusion that the refactor I was attempting just isn’t worth it. I feel like I was falling into a trap where I wanted to over-engineer it. If I’m correct with this suspicion, then I’m happy I was able to see this before I sunk too much time into it. I feel like at this point, we should be more iterative. We should strive to release what we have, even if it’s kind of cookie cutter and tough to use. If we have time, we can clean it up carefully, knowing we’re not rushed, and work on our wish list items.

At this point, my efforts are redirected to solving the mission critical issues we have identified so that we can come up with a simple configuration method that doesn’t involve a lot of change to what we already have, and get this thing published. The sooner we have a cohesive system in our NPM modules, the sooner we can write up some killer documentation, and start to see the fruits of our labor.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s