golang gotchas #2 the curse of “import cycle not allowed”

“import cycle not allowed”

Welcome to the #2 post of the “golang gotchas” series. Today we will talk about the “import cycle not allowed”. This series is organised as follows:

• code samples,
• gotchas areas, and
• theories

We know developers are very efficient in reading blogs and sort; hence wouldn’t it be nice if we can get all the code samples and gotchas in the first 10 minutes? Whilst the boring (but important) theories are left at the very end of the reading. Now we do have a choice, to know more through the theories or simply just get moving on your code :)

code — how it works

The source code is available at: https://github.com/quoeamaster/golang_blogs/tree/master/2021-06-gg-cycle-imports

Assume we are going to develop a program to filter files for operations; the architecture could be as follows:

the Scrapper architecture — brainstorm

A “Scrapper” would be responsible for looping a target folder and filters out the matching files; the target folder and the matching pattern are configured within a JSON file and hence the “Config” package would provide the value extraction logics.

Everything looks fine and straightforward, so let’s code!

The Scrapper struct would be as follows:

Note the following:

• we have declared 2 constants referring to the config file’s key (KeyConfigFolder and KeyConfigPattern) and later on the config package would access these 2 constants when parsing the JSON file.
• inside the Config(), we would let the config package function to handle the parsing for us (a dependency on the config package is added here)
• the Scrap function handles the file filtering based on the pattern and the folder location parsed earlier.

Again, everything seems fine till now. Let’s take a look on the config package:

Note the following:

• we have imported the “scrapper” package. (a dependency on the scrapper package added here — created a “cycle import” at once)
• the Config function requires to parse a JSON file and then extract the corresponding values based on “scrapper” package’s constants (KeyConfigFolder and KeyConfigPattern)

By the first sight, all looks fine and logical, isn’t it? But after a while, VS Code starts complaining

VS Code complaining after a minute…

a warning message which makes no sense…

The warning message shown above is confusing~ We logically know that the package scrapper (in my code example “scrappercycleimport”) is MANDATORY for accessing the constant keys; so then why we end up with a message “could not import XXX (no required module provides package XXX)” ??

To reveal the mystery, let’s write a unit test and run it:

The test tries to read the local folder (containing only the test file + the JSON config file) and should only found 1 file matching the pattern “scrapper_test.go”. However when we try to run the test…

import cycle not allowed…

Alright… so it is a “cycle imports” issue. Hey~~~ When did we introduced a cycle import then?

Scrapper side dependencies

• we start with the struct “Scrapper” which defines 2 constants KeyConfigFolder and KeyConfigPattern.
• We then introduced our 1st dependency on the “config” package through Scrapper.Config(), inside the function, we called “config.Config()” to parse the JSON file for the values of the target-folder as well as the target-pattern.

Config side dependencies

• inside the config.Config(), it accesses the constants declared in the scrapper package. This introduces a bi-directional dependency between the “scrapper” and “config” packages at ONCE.

And that is what we called “cycle imports”. Yeah I know it is confusing at the very beginning, but if we look closely on the unit test warnings, clearly the “scrappercycleimport” package has been added to the build stack for TWICE. To some other programming language like Java, cycle imports are cleverly handled by the compiler; but for golang, this is not the case (at least for now). Hence the solution to avoid such bi-directional dependencies has to be developed.

Gotchas — solution for the above …

solution 1: everything back into 1 package

Cycle imports are introduced when a bi-directional dependency is applied. Hence the simplest solution is to “group” packages back into 1.

Woot~ That sounds easy! Exactly simple solution, but doesn’t fit for all use-cases~

Pros:

• we used to start our project in a single package from the beginning; hence not something “new” to us.
• if our project is a relatively small one (e.g. a utility program to handle specific tasks with minimal variations), having only 1 package is feasible and easier to maintain.

Cons:

• sometimes we need to re-use our code logics and hence it would be appropriate to extract them into separate reusable packages.
• also it is not 100% about code re-use; but logically we would like to separate specific logics into packages to achieve maintainability.
• if our project is getting big, having only 1 package would be super messy to maintain. (I could feel that hectic memory floating out in your mind now :)))
• if our project yields multiple binaries in the end (e.g. a “server” package produces a binary; whilst a “client” package also yields another binary). It would nice to have code related to each logic entity separated. PS. Maybe a better approach is to separate your project into multiples eventually.

So that’s why, we would need another solution…

solution 2: re-organise the dependency code blocks

From our previous architecture, the config package tries to access our scrapper package’s constants — this is where the bi-directional dependency established. Hence if we re-organise the code blocks, this dependency would be gone!

The only thing we would need to do is to:

• in the scrapper package, comment out the 2 constants — KeyConfigFolder and KeyConfigPattern.
• in the config package, add back these 2 constants.

And run our unit test again…

unit test passed~~~~

What~~~~~ It is just that simple?? Indeed! So remember, for certain use cases, just think carefully and re-organise the code blocks, then the bi-directional dependency would disappear~

All sounds good, right? But then, we would go through another variation of our scrapper project and this time, code re-organisation would NOT help.

A Scrapper Variation (with new problems)

The architecture is still consisting the “scrapper” and the “config” packages; with a new functionality added on the “scrapper” side for validating whether the config file’s value is a true string. Let’s take a look at the scrapper code:

Note on the following:

• everything seems identical to the original design; but a new “IsValueString()” function is introduced. Its goal is to cast the general interface{} value back to a string.
• the “Config()” function now passes a reference of “Scrapper” to the config package, since the “config” package would need to access our scrapper.IsValueString() later on.
• we also cleverly re-organise the code blocks by moving the constants (KeyConfigXXX) to the “config” package

Now the Config package:

Note the following:

• again most of the code are identical, except the “Config()” now requires an instance of “scrapper” so that the IsValueString() could be accessed.

If we run the unit test now…

our old friend “import cycle not allowed” ~~~

T_T … cycle import again. What causes it this time?

The bi-directional dependency was introduced when our scrapper.Config() passes a reference to the config package. Remember our goal is to avoid the config package referencing / depending on the scrapper package… hence our new design actually steps into this taboo again.

One of the solutions is again… try to re-organise code blocks. In this use case, we could simply move the IsValueString() to the config package and remove the dependency of “scrapper” package. Somehow in the real world, sometimes moving functions around might not logical work for the project’s architecture design goals, and thus we would need yet another solution.

solution: provide a minimal local interface for casting

If moving functions around is not a good move. Then try providing a minimal local interface for casting!

our new design — IValidator

The new code for “config” package:

Note the following:

• we have introduced a new interface named “IValidator” and declares only 1 method “IsValueString(v interface{}) (bool, string)}”
• this “IsValueString()” has the exact function syntax with the one declared in the Scrapper struct.
• now the Config() would be intaking an instance of “IValidator” instead of “Scrapper” (in my code example “scrapperbidependency.ScrapperBi”) and HENCE removed the scrapper package’s dependency~

before introducing IValidator, an import on the “scrapper” package is required

after introducing the IValidator, no more “scrapper” packge import~ Solved “cycle imports”~

Then what about our Scrapper struct? Surprisingly… no code change at all!

Run the unit test again… and

yay~ we did it~ Unit test passed

The unit test passed finally~ And… what’s the MAGIC behind? The truth is when we pass in the Scrapper reference to the config.Config(), the Scrapper would be casted into an instance of IValidator instead. Since IValidator is declared within the same config package; thus there is no hard dependency on the scrapper package anymore, so bi-directional dependency is removed once and for all.

PS. Even though this is a tutorial based project; however in the real world, it is quite often and obvious that we would end up having structs inter-depend on each other in certain use cases. (again if you were a Java developer before, inter or cross dependency between classes is quite common in client-server model architectures PLUS the Java compiler has ways to handle cycle imports for us automatically) It would be nice to know how interfaces could solve the “cycle imports” and not requiring numerous code block re-organisations; plus… re-organisation is not a solution for every use case as well.

Theories — why, what and how

Q. What is a “cycle import” or what causes “import cycle not allowed”?

• when packages have code depending on each other causing a bi-directional relationship; this is what we called “cycle import”
• Take an example: in a client-server system, a server might need to run push-notifications to the client and hence has a client “handle” referenced. On the other hand, a client needs to have a server “handle” so that requests could be sent to the server correctly (as well as receiving server responses). This is a bi-directional relationship. In some programming languages this is peacefully handled by the compiler, however in golang (until now) this type of relationship would create a “cycle import” if not handled with care.
• Another example: a multi-media program asking for jpeg files through a system API is uni-directional, as the system (OS in this case) will not need to access our multi-media program in return thus it is only our multi-media program interacting with the system in a 1-way direction. And no complains on “cycle imports”

Q. How do I solve the “cycle imports” then?

• various approaches and all with its pros and cons.
• solution 1: move everything back into 1 package. The GOOD point is no more extra packages within the project thus no chance to introduce “package dependencies”. The BAD point is sometimes logical separation of code into packages are encouraged for code re-use and maintainability.
• solution 2: code re-organisation. Examine carefully and move the code blocks around the packages to resolve dependency issues. The GOOD point is after minor moving of code blocks the dependency problem should be solved. The BAD point is sometimes it doesn’t make sense to move certain functions around packages as there might be several packages depending on these functions… thus moving around might solved ONLY certain packages’ dependency (e.g. package A, B and C depends on D; whilst moving the functions in D to package A might only solve package A’s dependency issues, for B and C, still suffering from “cycle imports”). A simple sentence is code re-organisation might not be the ultimate solution for every use case.
• solution 3: providing a minimal local interface for casting. Instead of providing the actual concrete struct, we would provide a local interface encapsulating only the minimal functionality; then cast the concrete struct into our local interface and thus removing the dependency. The GOOD point is we are providing a minimal PLUS local interface to cast over the actual concrete struct, thus no extra dependency is required. Since the local interface only provides a minimal set of functions required by the local package; thus even though the concrete struct might have added much more new functions in the future, as long as it still exposes this public function, the solution will work perfectly as expected. You can say that it is like a lean relationship instead of a strict one. The BAD point is we might end up declaring local interface in different affected packages. This would introduce maintenance issue eventually.
• So the suggested approach is always start with solution 2: re-organise code blocks, and if it doesn’t work, start to look at solution 3: interface casting.

Q. I am using VS Code, sometimes the warnings for “cycle imports” are not very clear, any suggestions on this?

• if vs code failed to provide a meaningful warning; use the good old unit test or run a “go build”
• sometimes we JUST need to get our hands dirty in the terminal / console :))))

Closings

Nice~ I hope you all had a good journey today as we just ride through the “cycle import” storm :))) What we have picked up are the followings:

• understanding what and why “cycle imports” happened
• found 3 solutions to resolve the nightmare

Till next time for more gotcha adventures :)

PS. if you need answers, do leave a private message when necessary~