Google Summer of Code 2015, reporting in. Everything goes well so far, tests are working. But testing takes quite a lot of time, especially if you need to test with different Android versions. Robolectric and Clojure startup times combined take their toll when you have to relaunch them several times. Can we do better? — I asked myself. Perhaps we can.
So I have an idea to adapt Skummet instead of Clojure for running tests. This is theoretically possible, but requires some work, so I decided to devote a few days to hacking Skummet. And to tell the truth I was quite pleased with the results.
Pumpet (Norwegian: pumped) is a swole brother of Skummet. He is like Skummet on steroids. While Skummet hits the treadmill, Pumpet lifts iron in the basement. He is also very athletic and fast, he never misses his legs day and squats like there is no tomorrow, but he can't outrun his bro Skummet (average startup time of hello-world with Pumpet is 350 ms). But boy oh boy can he lift. And by lifting I mean the dynamic compilation.
So the main difference between Skummet and Pumpet is that Pumpet can eval. Pumpet completely (I hope) supports dynamic compilation even though it is itself compiled in the lean mode. To be able to do that Pumpet needs to have some extra body weight, but trust me, it's all muscle.
For example, Skummet doesn't emit macros during compilation because after the compilation is done they are useless (if you don't plan to do eval). Pumpet on the other hand can't skip macros, he must keep all the carbs and protein to be able to lift (eval) whichever the weight (expression) you give him.
For the same reason, you can't use Proguard with Pumpet. Proguard is like fasting — you declare unused stuff as useless and remove it. Pumpet can't fast because otherwise he will lose his gains and won't be able to lift (eval).
For all remaining purposes Skummet and Pumpet are exactly the same. They might
probably become a single JAR again when I perform the
refactor. Still, the idea is the following: if you don't need
eval in your
program/application, you want to be able to reduce the binary size and memory
usage by removing unused functions, and you want every bit of fast startup —
go Skummet. If you need that dynamic compilation — choose Pumpet.
That's right. With Skummet version I just deployed, a hello world application finishes on average in 300 milliseconds on my half-decayed Thinkpad X220. For reference, Java hello-world finishes in 80 ms. I think that in absolute numbers this is a huge victory.
Although I pushed the boundaries even further. By replacing the vanilla
clojure.lang.Compiler class with the stubbed one in the final compiled
package I reduced the startup time to 250 ms. The only problem I have with
that is that I don't know how to package two Compiler classes (original and
stubbed) into the same jar. To be able to do that sensibly a significant
refactoring of Compiler class is required which I don't want to do just yet.
But it will be done at some point.
I made possible for
RT.var to find the lean Vars in runtime. It uses
reflection, but otherwise is pretty fast. This reduces the necessity to mark
Vars used like that as non-lean, and overall removes a lot of headache and
Once I started singletoning functions (removing references to them from
__init classes), I faced the problem of compiling recursive functions. The
problem got even worse when the function was multi-arity one, and had some
internal subfunctions that called the top-level one (see
Previously I had to resort to not-singletoning them but keeping their
instances in the namespace class. Now the code responsible for that is
rewritten, and even more functions are singletoned now (thus can be removed
by Proguard if unused).
core_print.clj contains a lot of calls to
prefer-method. Running them all takes some time. Specially for Android
where you usually don't start printing Clojure data structures right away,
this initialization can be delayed until
pr is called for the first time,
which also squeezes out some sweet milliseconds from the startup time.
I pushed the new versions of both compilers, versions below:
More information on Skummet and how to use it can be found here.
I haven't tested them on Android just yet but they should be working. Now that clojure 1.7 is out, I want to carefully polish them before releasing 1.7 versions myself. Meanwhile, please try them, report bugs and keep working out!
UPDATE: Just fixed a very nasty reflection bug that short-circuited
r4's should work now.