Translating Java to Kotlin at Scale

• Meta has been on a years-long endeavor to translate our whole Android codebase from Java to Kotlin.
• Immediately, regardless of having one of many largest Android codebases on the earth, we’re nicely previous the midway level and nonetheless going.
• We’re sharing among the tradeoffs we’ve made to help automating our transition to Kotlin, seemingly easy transformations which might be surprisingly difficult, and the way we’re collaborating with different corporations to seize tons of extra nook instances.

Android growth at Meta has been Kotlin-first since 2020, and builders have been saying they like Kotlin as a language for even longer.

However, adoption doesn’t essentially entail translation. We might merely determine to put in writing all new code in Kotlin and go away our present Java code as is, simply as many different corporations have. Or we might take it a little bit additional and translate simply a very powerful information. As an alternative, we determined that the one method to leverage the total worth of Kotlin was to go all in on conversion, even when it meant constructing our personal infrastructure to automate translation at scale. So, a number of years in the past, engineers at Meta determined to take roughly ten million lines of perfectly good Java code and rewrite them in Kotlin.

After all, we needed to clear up issues past translation, akin to sluggish construct speeds and inadequate linters. To be taught extra about Meta’s broader adoption effort, see Omer Strulovich’s 2022 weblog put up on our migration from Java to Kotlin or Lisa Watkin’s speak about Kotlin adoption at Instagram.

To maximise our positive factors in developer productiveness and null security, we’re aiming to translate just about all of our actively developed code, plus any code that’s central within the dependency graph. Not surprisingly, that’s most of our code, which provides as much as tens of hundreds of thousands of traces, together with among the most complicated information.

It’s fairly intuitive that if we need to maximize productiveness positive factors, we must always translate our actively developed code. It’s rather less apparent why translating past that gives incremental null-safety advantages. The quick reply is that any remaining Java code may be an agent of nullability chaos, particularly if it’s not null protected and much more so if it’s central to the dependency graph. (For a extra detailed rationalization, see the part under on null security.)

We additionally need to reduce the drawbacks of a blended codebase. So long as we’ve substantial quantities of Java, we have to proceed supporting parallel software chains. There’s additionally the much-lamented difficulty of slower construct speeds: Compiling Kotlin is slower than compiling Java, however compiling each collectively is the slowest of all. 

Like most people within the trade, we began migrating incrementally by repeatedly clicking a button within the Intellij IDE. This button would set off Intellij’s translation tool, generally referred to as J2K. It shortly turned clear that this method wasn’t going to scale for a codebase of our measurement: We must click on that button—after which wait the couple of minutes it takes to run—nearly 100,000 instances to translate our Android codebase. 

With this in thoughts, we got down to automate the conversion course of and reduce interference with our builders’ every day work. The outcome was a software we name the Kotlinator that we constructed round J2K. It’s now comprised of six phases:

1. “Deep” construct: Constructing the code we’re about to translate helps the IDE resolve all of the symbols, particularly when third-party dependencies or generated code are concerned.
2. Preprocessing: This section is constructed on high of our customized software, Editus. It incorporates about 50 steps for nullability, J2K workarounds, modifications to help our customized DI framework, and extra.
3. Headless J2K: The J2K we all know and love, however server-friendly!
4. Postprocessing: This section is comparable in structure to our preprocessing. It consists of about 150 steps for Android-specific modifications, in addition to extra nullability modifications, and tweaks to make the ensuing Kotlin extra idiomatic.
5. Linters: Working our linters with autofixes permits us to implement perennial fixes in a means that advantages each conversion diffs and common diffs going ahead.
6. Construct error-based fixes: Lastly, the Kotlinator makes much more fixes based mostly on construct errors. After a failed construct of the just-translated code, we parse the errors and apply additional fixes (e.g., including a lacking import or inserting a !!).

We’ll dive into extra element on essentially the most fascinating phases under.

Going headless with J2K

Step one was making a headless model of J2K that might run on a distant machine—not simple, given how tightly coupled J2K and the remainder of the Intellij IDE are. We thought of a number of approaches, together with working J2K utilizing a setup just like Intellij’s testing surroundings, however after speaking to JetBrains’ J2K knowledgeable, Ilya Kirillov, we ultimately settled on one thing extra like a headless inspection. To implement this method, we created an Intellij plugin that features a class extending ApplicationStarter and calling straight into the JavaToKotlinConverter class that’s additionally referenced by the IDE’s conversion button.

On high of not blocking builders’ native IDEs, the headless method allowed us to translate a number of information without delay, and it unblocked all kinds of useful however time-consuming steps, just like the “construct and repair errors” course of detailed under. Total conversion time grew longer (a typical distant conversion now takes about half-hour to run), however time spent by the builders decreased considerably.

After all, going headless presents one other conundrum: If builders aren’t clicking the button themselves, who decides what to translate, and the way does it get reviewed and shipped? The reply turned out to be fairly simple: Meta has an inner system that enables builders to arrange what is basically a cron job that produces a every day batch of diffs (our model of pull requests) based mostly on user-defined choice standards. This method additionally helps select related reviewers, ensures that assessments and different validations move, and ships the diff as soon as it’s accepted by a human. We additionally supply an online UI for builders to set off a distant conversion of a selected file or module; behind the scenes, it runs the identical course of because the cron job.

As for selecting what and when to translate, we don’t implement any specific order past prioritizing actively developed information. At this level, the Kotlinator is subtle sufficient to deal with most compatibility modifications required in exterior information (for instance, altering Kotlin dependents’ references of foo.getName() to foo.title), so there’s no must order our translations based mostly on the dependency graph. 

Including customized pre- and post-conversion steps

Because of the measurement of our codebase and the customized frameworks we use, the overwhelming majority of conversion diffs produced by the vanilla J2K wouldn’t construct. To handle this drawback, we added two customized phases to our conversion course of, preprocessing and postprocessing. Each phases comprise dozens of steps that take within the file being translated, analyze it (and generally its dependencies and dependents, too), and carry out a Java->Java or Kotlin->Kotlin transformation if wanted. A few of our postprocessing transformations have been open-sourced.

These customized translation steps are constructed on high of an inner metaprogramming software that leverages Jetbrains’ PSI libraries for each Java and Kotlin. In contrast to most metaprogramming instruments, it is rather a lot not a compiler plugin, so it might analyze damaged code throughout each languages, and does so in a short time. That is particularly useful for postprocessing as a result of it’s usually working on code with compilation errors, doing evaluation that requires kind data. Some postprocessing steps that take care of dependents could must resolve symbols throughout a number of thousand unbuildable Java and Kotlin information. For instance, considered one of our postprocessing steps helps translate interfaces by inspecting its Kotlin implementers and updating overridden getter features to as an alternative be overridden properties, like within the instance under.

interface JustConverted {
  val title: String // I was a way referred to as `getName`
}

class ConvertedAWhileAgo : JustConverted {
  override enjoyable getName(): String = "JustConvertedImpl"
}

class ConvertedAWhileAgo : JustConverted {
  override val title: String = "JustConvertedImpl"
}

The draw back to this software’s pace and adaptability is that it might’t at all times present solutions about kind data, particularly when symbols are outlined in third-party libraries. In these instances, it bails shortly and clearly, so we don’t execute a metamorphosis with false confidence. The ensuing Kotlin code may not construct, however the applicable repair is normally fairly apparent to a human (if a little bit tedious).

We initially added these customized phases to cut back developer effort, however over time we additionally leveraged them to cut back developer unreliability. Opposite to common perception, we’ve discovered it’s usually safer to go away essentially the most delicate transformations to bots. There are particular fixes we’ve automated as a part of postprocessing, despite the fact that they aren’t strictly essential, as a result of we need to reduce the temptation for human (i.e., error-prone) intervention. One instance is condensing lengthy chains of null checks: The ensuing Kotlin code isn’t extra right, but it surely’s much less prone to a well-meaning developer by accident dropping a negation. 

Leveraging construct errors

In the midst of doing our personal conversions, we observed that we spent plenty of time on the finish repeatedly constructing and fixing our code based mostly on the compiler’s error messages. In principle, we might repair many of those issues in our customized postprocessing, however doing so would require us to reimplement plenty of complicated logic that’s baked into the Kotlin compiler. 

As an alternative, we added a brand new, closing step within the Kotlinator that leverages the compiler’s error messages the identical means a human would. Like postprocessing, these fixes are carried out with a metaprogramming that may analyze unbuildable code.

The restrictions of customized tooling

Between the preprocessing, postprocessing, and post-build phases, the Kotlinator incorporates nicely over 200 customized steps. Sadly, some conversion points merely can’t be solved by including much more steps.

Initially we handled J2K as a black field—despite the fact that it was open sourced—as a result of its code was complicated and never actively developed; diving in and submitting PRs didn’t appear definitely worth the effort. That modified early in 2024, nevertheless, when JetBrains started work to make J2K appropriate with the brand new Kotlin compiler, K2. We took the chance to work with JetBrains to enhance J2K and handle issues that had been plaguing us for years, akin to disappearing override key phrases.

Collaborating with JetBrains additionally gave us the chance to insert hooks into J2K that might enable shoppers like Meta to run their very own customized steps straight within the IDE earlier than and after conversion. This may increasingly sound unusual, given the variety of customized processing steps we’ve already written, however there are a few main advantages:

1. Improved image decision. Our customized image decision is quick and versatile, but it surely’s much less exact than J2K’s, particularly in terms of resolving symbols outlined in third-party libraries. Porting a few of our preprocessing and postprocessing steps over to leverage J2K’s extension factors will make them extra correct, and permit us to make use of Intellij’s extra subtle static-analysis tooling.
2. Simpler open sourcing and collaboration. A few of our customized steps are too Android-specific to be included into J2K however may nonetheless be helpful to different corporations. Sadly, most of them depend upon our customized image decision. Porting these steps over to as an alternative depend on J2K’s image decision offers us the choice to open-source them and profit from the group’s pooled efforts.

In an effort to translate our code with out spewing null-pointer exceptions (NPEs) in all places, it first must be null protected (by “null protected” we imply code checked by a static analyzer akin to Nullsafe or NullAway). Null security nonetheless isn’t adequate to remove the potential for NPEs, but it surely’s a wonderful begin. Sadly, making code null protected is less complicated stated than accomplished.

Even null-safe Java throws NPEs generally

Anybody who has labored with null-safe Java code lengthy sufficient is aware of that whereas it’s extra dependable than vanilla Java code, it’s nonetheless susceptible to NPEs. Sadly static analysis is only 100% effective for 100% code coverage, which is solely not viable in any giant cellular codebase that interacts with the server and third-party libraries.

Right here’s a canonical instance of a seemingly innocuous change that may introduce an NPE:

MyNullsafeClass.java

@Nullsafe
public class MyNullsafeClass {

  void doThing(String s) {
    // can we safely add this dereference?
    // s.size;
  }
}

Say there are a dozen dependents that decision MyNullsafeJava::doThing. A single non-null-safe dependent might move in a null argument (for instance,  MyNullsafeJava().doThing(null)), which might result in an NPE if a dereference is inserted within the physique of doThing. 

After all, whereas we will’t remove NPEs in Java by way of null-safety protection, we will vastly cut back their frequency. Within the instance above, NPEs are attainable however pretty uncommon when there’s just one non-null-safe dependent. If a number of transitive dependents lacked null security, or if one of many extra central dependent nodes did, the NPE threat could be a lot greater.

What makes Kotlin completely different

The most important distinction between null-safe Java and Kotlin is the presence of runtime validation in Kotlin bytecode on the interlanguage boundary. This validation is invisible however highly effective as a result of it permits builders to belief the acknowledged nullability annotations in any code they’re modifying or calling.

If we return to our earlier instance, MyNullsafeClass.java, and translate it to Kotlin, we get one thing like:

MyNullsafeClass.kt

class MyNullsafeClass {

  enjoyable doThing(s: String) {
    // there's an invisible `checkNotNull(s)` right here within the bytecode
    // so including this dereference is now risk-free!
    // s.size
  }
}

Now there’s an invisible checkNotNull(s) within the bytecode initially of doThing’s physique, so we will safely add a dereference to s, as a result of if s have been nullable, this code would already be crashing. As you possibly can think about, this certainty makes for a lot smoother, safer growth.

There are additionally some variations on the static evaluation stage: The Kotlin compiler enforces a barely stricter set of null safety rules than Nullsafe does in terms of concurrency. Extra particularly, the Kotlin compiler throws an error for dereferences of class-level properties that might have been set to null in one other thread. This distinction isn’t terribly necessary to us, but it surely does result in extra !! than one may anticipate when translating null-safe code.

Nice, let’s translate all of it to Kotlin!

Not so quick. As is at all times the case, going from extra ambiguity to much less ambiguity doesn’t come at no cost. For a case like MyNullsafeClass, growth is far simpler after Kotlin translation, however somebody has to take that preliminary threat of successfully inserting a nonnull assertion for its hopefully-really-not-nullable parameter s. That “somebody” is whichever developer or bot finally ends up transport the Kotlin conversion.

We will take numerous steps to reduce the danger of introducing new NPEs throughout conversion, the best of which is erring on the aspect of “extra nullable” when translating parameters and return varieties. Within the case of MyNullsafeClass, the Kotlinator would have used context clues (on this case, the absence of any dereferences within the physique of doThing) to deduce that String s ought to be translated to s: String?.

One of many modifications we ask builders to scrutinize most when reviewing conversion diffs is the addition of !! outdoors of preexisting dereferences. Funnily sufficient, we’re not apprehensive about an expression like foo!!.title, as a result of it’s not any extra prone to crash in Kotlin than it was in Java. An expression akin to someMethodDefinedInJava(foo!!) is rather more regarding, nevertheless, as a result of it’s attainable that someMethodDefinedInJava is solely lacking a @Nullable on its parameter, and so including !! will introduce a really pointless NPE.

To keep away from issues like including pointless !! throughout conversion, we run over a dozen complementary codemods that comb by way of the codebase searching for parameters, return varieties, and member variables that could be lacking @Nullable. Extra correct nullability throughout the codebase—even in Java information that we could by no means translate—is just not solely safer, it’s additionally conducive to extra profitable conversions, particularly as we method the ultimate stretch on this venture.

After all, the final remaining null issues of safety in our Java code have normally caught round as a result of they’re very laborious to resolve. Earlier makes an attempt to resolve them relied totally on static evaluation, so we determined to borrow an thought from the Kotlin compiler and create a Java compiler plugin that helps us acquire runtime nullability knowledge. This plugin permits us to gather knowledge on all return varieties and parameters which might be receiving/returning a null worth and usually are not annotated as such. Whether or not these are from Java/Kotlin interop or lessons that have been annotated incorrectly at an area stage, we will decide final sources of fact and use codemods to lastly repair the annotations.

On high of the dangers of regressing null security, there are dozens of different methods to interrupt your code throughout conversion. In the midst of transport over 40,000 conversions, we’ve realized about many of those the laborious means and now have a number of layers of validation to forestall them. Listed here are a few our favorites:

Complicated initialization with getters

// Incorrect!
val title: String = getCurrentUser().title

// Right
val title: String
  get() = getCurrentUser().title

Nullable booleans

// Authentic
if (foo != null && !foo.isEnabled) println("Foo is just not null and disabled")

// Incorrect!
if (foo?.isEnabled != true) println("Foo is just not null and disabled")

// Right
if (foo?.isEnabled == false) println("Foo is just not null and disabled")

At this level, greater than half of Meta’s Android Java code has been translated to Kotlin (or, extra hardly ever, deleted). However that was the straightforward half! The actually enjoyable half lies forward of us, and it’s a doozy. There are nonetheless 1000’s of absolutely automated conversions we hope to unblock by including and refining customized steps and by contributing to J2K. And there are 1000’s extra semi-automated conversions we hope to ship easily and safely because of different Kotlinator enhancements.

Most of the issues we face additionally have an effect on different corporations translating their Android codebases. If this sounds such as you, we’d love so that you can leverage our fixes and share a few of your personal. Come chat with us and others within the #j2k channel of the Kotlinlang Slack.