10 votes

Swift: Google's bet on differentiable programming

7 comments

  1. [3]
    Wulfsta
    Link
    I'm not convinced that including this functionality in the compiler is any better than having a well defined library of types that support automatic differentiation. If you know you need automatic...

    I'm not convinced that including this functionality in the compiler is any better than having a well defined library of types that support automatic differentiation. If you know you need automatic differentiation then selectively using a type that supports these same features should be possible in any language I can think of. On top of that, it appears to only handle automatic differentiation of the first derivative. Being more general is old news for the Haskell community (though Haskell is limited in use due to its paradigm and compile times) and only handling the first derivative seems like a limitation in the scope of these projects. I can see an argument for this extension making the work syntactically nicer, but that's most of what seems to be gained?

    4 votes
    1. [2]
      pvik
      Link Parent
      I agree. Through most of this article I kep thinking "why?". Why does this have to be at the compiler level? I might be missing something obvious. And Ocaml and Lisp (even scheme) :) Given that...

      If you know you need automatic differentiation then selectively using a type that supports these same features should be possible in any language I can think of.

      I agree. Through most of this article I kep thinking "why?". Why does this have to be at the compiler level? I might be missing something obvious.

      Being more general is old news for the Haskell community

      And Ocaml and Lisp (even scheme) :)

      I can see an argument for this extension making the work syntactically nicer, but that's most of what seems to be gained?

      Given that one of their primary criteria in selecting Swift itself was to make it easy for the data-science community to work in, this might not be too far from the truth.

      2 votes
      1. Wulfsta
        Link Parent
        I hadn't thought about this when I wrote that comment, but I bet the second derivative could be constructed by gradient(of: gradient(of: func)). This can be done for the nth derivative. I can get...

        I hadn't thought about this when I wrote that comment, but I bet the second derivative could be constructed by gradient(of: gradient(of: func)). This can be done for the nth derivative. I can get behind this a little bit more with this realization, but I'm still not convinced. It would still require custom definitions of the gradient if a user wanted to call it on something like whatever the bindings for GMP are called in Swift.

        3 votes
  2. [3]
    Greg
    Link
    Am I missing something obvious, or is gradient(of: powerOf2)(3) // 5.545 actually supposed to be 12?

    Am I missing something obvious, or is gradient(of: powerOf2)(3) // 5.545 actually supposed to be 12?

    1 vote
    1. [2]
      pvik
      Link Parent
      They are defining powerOf2(x) -> 2^x d/dx(2^x) = log(2) * (2^x) With x = 3 , log(2) * (2^3) = 5.545

      They are defining powerOf2(x) -> 2^x

      d/dx(2^x) = log(2) * (2^x)

      With x = 3 , log(2) * (2^3) = 5.545

      3 votes
      1. Greg
        Link Parent
        D'oh! Thank you - I had their previous x^3 example in my head and somehow it didn't click that they were using a totally different function here.

        D'oh! Thank you - I had their previous x^3 example in my head and somehow it didn't click that they were using a totally different function here.

        2 votes
  3. pvik
    Link
    At the beginning of the article I was confused why a team from Google (Google's Swift 4 TensorFlow, S4FT) was working on Swift. The S4FT actually have a post here where they layout why they ended...

    At the beginning of the article I was confused why a team from Google (Google's Swift 4 TensorFlow, S4FT) was working on Swift. The S4FT actually have a post here where they layout why they ended up choosing Swift.

    Looks like they considered Go, C++, Rust, Julia, etc, or building a completely new language from ground up.

    I wish they would've considered Nim as well. It would've been easy for developers used to Python to pick up. It compiles down to native libraries and is pretty fast. Also, allows for decent debugging capabilities. And with a team like them considering Nim, they could've steered the language itself too to help out with ML use-cases!

    And also, I wish they would've considered CommonLisp (implementations like sbcl are very fast/performant and allow fine-tuned memory handling/management if needed too). However, a Lisp might've been a steep learning curve for the data-science community, maybe?

    1 vote