Deprecated: Function get_magic_quotes_gpc() is deprecated in /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php on line 99

Deprecated: The each() function is deprecated. This message will be suppressed on further calls in /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php on line 619

Warning: Cannot modify header information - headers already sent by (output started at /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php:99) in /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php on line 832

Warning: Cannot modify header information - headers already sent by (output started at /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php:99) in /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php on line 839

Warning: Cannot modify header information - headers already sent by (output started at /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php:99) in /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php on line 839

Warning: Cannot modify header information - headers already sent by (output started at /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php:99) in /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php on line 839

Warning: Cannot modify header information - headers already sent by (output started at /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php:99) in /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php on line 839

Warning: Cannot modify header information - headers already sent by (output started at /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php:99) in /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php on line 839

Warning: Cannot modify header information - headers already sent by (output started at /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php:99) in /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php on line 839

Warning: Cannot modify header information - headers already sent by (output started at /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php:99) in /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php on line 839

Warning: Cannot modify header information - headers already sent by (output started at /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php:99) in /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php on line 839

Warning: Cannot modify header information - headers already sent by (output started at /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php:99) in /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php on line 839

Warning: Cannot modify header information - headers already sent by (output started at /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php:99) in /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php on line 839

Warning: Cannot modify header information - headers already sent by (output started at /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php:99) in /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php on line 839

Warning: Cannot modify header information - headers already sent by (output started at /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php:99) in /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php on line 839

Warning: Cannot modify header information - headers already sent by (output started at /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php:99) in /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php on line 839

Warning: Cannot modify header information - headers already sent by (output started at /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php:99) in /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php on line 839

Warning: Cannot modify header information - headers already sent by (output started at /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php:99) in /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php on line 839

Warning: Cannot modify header information - headers already sent by (output started at /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php:99) in /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php on line 839

Warning: Cannot modify header information - headers already sent by (output started at /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php:99) in /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php on line 839

Warning: Cannot modify header information - headers already sent by (output started at /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php:99) in /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php on line 839

Warning: Cannot modify header information - headers already sent by (output started at /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php:99) in /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php on line 839

Warning: Cannot modify header information - headers already sent by (output started at /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php:99) in /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php on line 839

Warning: Cannot modify header information - headers already sent by (output started at /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php:99) in /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php on line 839
2025-10-20T18:29:46+00:00 http://elixir-lang.org José Valim 2025-10-16T00:00:00+00:00 /blog/2025/10/16/elixir-v1-19-0-released <p>Elixir v1.19 brings further improvements to the type system and compilation times, allowing us to find more bugs, faster.</p> <h2 id="type-system-improvements">Type system improvements</h2> <p>This release improves the type system around two key areas: type inference and type checking of anonymous functions and protocols. These enhancements seem simple on the surface but required us to go beyond existing literature by extending current theory and developing new techniques. We will outline the technical details in future articles. For now, let’s look at what’s new.</p> <h3 id="type-inference-of-all-constructs">Type inference of all constructs</h3> <p>Type inference (or reconstruction) is the ability of a type system to automatically deduce, either partially or fully, the type of an expression at compile time. Type inference may occur at different levels. For example, many programming languages can automatically infer the types of variables, also known “local type inference”, but not all can infer type signatures of functions.</p> <p>Originally, our plan with Elixir’s upcoming type system was to support type inference of patterns, guards, and return types. Therefore, if you wrote this simple function:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="n">even?</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="ow">when</span> <span class="n">is_integer</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="k">do</span> <span class="n">rem</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="mi">2</span><span class="p">)</span> <span class="o">==</span> <span class="mi">0</span> <span class="k">end</span> </code></pre></div></div> <p>Elixir would correctly infer the type to be <code class="language-plaintext highlighter-rouge">integer() -&gt; boolean()</code>. However, if you wrote this function:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="n">even?</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="k">do</span> <span class="n">rem</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="mi">2</span><span class="p">)</span> <span class="o">==</span> <span class="mi">0</span> <span class="k">end</span> </code></pre></div></div> <p>The type would be <code class="language-plaintext highlighter-rouge">dynamic() -&gt; boolean()</code>, since there are no guards, even though the functions behave virtually the same, as the <code class="language-plaintext highlighter-rouge">rem</code> operator expects both arguments to be integer (they just raise different exceptions for non-integer values).</p> <p>Inferring type signatures comes with a series of trade-offs:</p> <ul> <li> <p>Speed - type inference algorithms are often more computationally intensive than type checking algorithms.</p> </li> <li> <p>Expressiveness - in any given type system, the constructs that support inference are always a subset of those that can be type-checked. Therefore, if a programming language is restricted to only fully reconstructed types, it is less expressive than a solely type checked counterpart.</p> </li> <li> <p>Incremental compilation - type inference complicates incremental compilation. If module A depends on module B, which depends on module C, a change to C may require the type signature in B to be reconstructed, which may then require A to be recomputed (and so on). This dependency chain may require large projects to explicitly add type signatures for stability and compilation efficiency.</p> </li> <li> <p>Cascading errors - when a user accidentally makes type errors or the code has conflicting assumptions, type inference may lead to less clear error messages as the type system tries to reconcile diverging type assumptions across code paths.</p> </li> </ul> <p>On the other hand, type inference offers the benefit of enabling type checking for functions and codebases without requiring the user to add type annotations. To balance these trade-offs, we are exploring “module type inference”: our goal is to infer type signatures considering invocations of functions in the same module and of functions from <em>other applications</em> (such as Elixir itself and your dependencies). Once module types are inferred, your whole project is type checked considering all declared and inferred types.</p> <p>We have successfully implemented these features as part of Elixir v1.19, by performing inference of all constructs (except guards), taking into account the signatures from calls to functions within the same module and in Elixir’s standard library. This means the second function above, without the guard, will also infer the type <code class="language-plaintext highlighter-rouge">integer() -&gt; boolean()</code>.</p> <p>In future releases, we plan to perform type inference of guards (originally planned for v1.19) and also consider the type signatures of your dependencies during inference. Overall, these changes allow us to assess the impact of the trade-offs above as the type system evolves, which suits well our current goals of incrementally using types to find bugs in existing codebases, without changing them.</p> <p>Keep in mind this only applies to <em>type inference</em>. Once we introduce type signatures and you explicitly annotate your functions, type inference and the trade-offs above no longer play a role. Any function with an explicit type signature will be typed checked against the user-provided annotations, as in other statically typed languages.</p> <h3 id="type-checking-of-protocol-dispatch-and-implementations">Type checking of protocol dispatch and implementations</h3> <p>This release adds type checking when dispatching and implementing protocols.</p> <p>For example, string interpolation in Elixir uses the <code class="language-plaintext highlighter-rouge">String.Chars</code> protocol. If you pass a value that does not implement said protocol, Elixir will now emit a warning accordingly.</p> <p>Here is an example passing a range, which cannot be converted into a string, to an interpolation:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">Example</span> <span class="k">do</span> <span class="k">def</span> <span class="n">my_code</span><span class="p">(</span><span class="n">first</span><span class="o">..</span><span class="n">last</span><span class="o">//</span><span class="n">step</span> <span class="o">=</span> <span class="n">range</span><span class="p">)</span> <span class="k">do</span> <span class="s2">"hello </span><span class="si">#{</span><span class="n">range</span><span class="si">}</span><span class="s2">"</span> <span class="k">end</span> <span class="k">end</span> </code></pre></div></div> <p>the above emits the following warnings:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>warning: incompatible value given to string interpolation: data it has type: %Range{first: term(), last: term(), step: term()} but expected a type that implements the String.Chars protocol, it must be one of: dynamic( %Date{} or %DateTime{} or %NaiveDateTime{} or %Time{} or %URI{} or %Version{} or %Version.Requirement{} ) or atom() or binary() or float() or integer() or list(term()) </code></pre></div></div> <p>Warnings are also emitted if you pass a data type that does not implement the <code class="language-plaintext highlighter-rouge">Enumerable</code> protocol as a generator to for-comprehensions:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">Example</span> <span class="k">do</span> <span class="k">def</span> <span class="n">my_code</span><span class="p">(%</span><span class="no">Date</span><span class="p">{}</span> <span class="o">=</span> <span class="n">date</span><span class="p">)</span> <span class="k">do</span> <span class="n">for</span><span class="p">(</span><span class="n">x</span> <span class="o">&lt;-</span> <span class="n">date</span><span class="p">,</span> <span class="k">do</span><span class="p">:</span> <span class="n">x</span><span class="p">)</span> <span class="k">end</span> <span class="k">end</span> </code></pre></div></div> <p>will emit:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>warning: incompatible value given to for-comprehension: x &lt;- date it has type: %Date{year: term(), month: term(), day: term(), calendar: term()} but expected a type that implements the Enumerable protocol, it must be one of: dynamic( %Date.Range{} or %File.Stream{} or %GenEvent.Stream{} or %HashDict{} or %HashSet{} or %IO.Stream{} or %MapSet{} or %Range{} or %Stream{} ) or fun() or list(term()) or non_struct_map() </code></pre></div></div> <h3 id="type-checking-and-inference-of-anonymous-functions">Type checking and inference of anonymous functions</h3> <p>Elixir v1.19 can now type infer and type check anonymous functions. Here is a trivial example:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">Example</span> <span class="k">do</span> <span class="k">def</span> <span class="n">run</span> <span class="k">do</span> <span class="n">fun</span> <span class="o">=</span> <span class="k">fn</span> <span class="p">%{}</span> <span class="o">-&gt;</span> <span class="ss">:map</span> <span class="k">end</span> <span class="n">fun</span><span class="o">.</span><span class="p">(</span><span class="s2">"hello"</span><span class="p">)</span> <span class="k">end</span> <span class="k">end</span> </code></pre></div></div> <p>The example above has an obvious typing violation, as the anonymous function expects a map but a string is given. With Elixir v1.19, the following warning is now printed:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> warning: incompatible types given on function application: fun.("hello") given types: binary() but function has type: (dynamic(map()) -&gt; :map) typing violation found at: │ 6 │ fun.("hello") │ ~ │ └─ mod.exs:6:8: Example.run/0 </code></pre></div></div> <p>Function captures, such as <code class="language-plaintext highlighter-rouge">&amp;String.to_integer/1</code>, will also propagate the type as of Elixir v1.19, arising more opportunity for Elixir’s type system to catch bugs in our programs.</p> <h3 id="acknowledgements">Acknowledgements</h3> <p>The type system was made possible thanks to a partnership between <a href="https://www.cnrs.fr/">CNRS</a> and <a href="https://remote.com/">Remote</a>. The development work is currently sponsored by <a href="https://www.fresha.com/">Fresha</a>, <a href="https://starfish.team/">Starfish*</a>, and <a href="https://dashbit.co/">Dashbit</a>.</p> <h2 id="faster-compile-times-in-large-projects">Faster compile times in large projects</h2> <p>This release includes two compiler improvements that can lead up to 4x faster builds in large codebases.</p> <p>While Elixir has always compiled the given files in project or a dependency in parallel, the compiler would sometimes be unable to use all of the machine resources efficiently. This release addresses two common limitations, delivering performance improvements that scale with codebase size and available CPU cores.</p> <h3 id="code-loading-bottlenecks">Code loading bottlenecks</h3> <p>Prior to this release, Elixir would load modules as soon as they were defined. However, because the Erlang part of code loading happens within a single process (the code server), this would make it a bottleneck, reducing parallelization, especially on large projects.</p> <p>This release makes it so modules are loaded lazily. This reduces the pressure on the code server and the amount of work during compilation, with reports of more than two times faster compilation for large projects. The benefits depend on the codebase size and the number of CPU cores available.</p> <p>Implementation wise, <a href="https://elixir-lang.org/blog/2012/04/24/a-peek-inside-elixir-s-parallel-compiler/">the parallel compiler already acts as a mechanism to resolve modules during compilation</a>, so we built on that. By making sure the compiler controls both module compilation and module loading, it can also better guarantee deterministic builds.</p> <p>There are two potential regressions with this approach. The first one happens if you spawn processes during compilation which invoke other modules defined within the same project. For example:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">MyLib</span><span class="o">.</span><span class="no">SomeModule</span> <span class="k">do</span> <span class="n">list</span> <span class="o">=</span> <span class="p">[</span><span class="o">...</span><span class="p">]</span> <span class="no">Task</span><span class="o">.</span><span class="n">async_stream</span><span class="p">(</span><span class="n">list</span><span class="p">,</span> <span class="k">fn</span> <span class="n">item</span> <span class="o">-&gt;</span> <span class="no">MyLib</span><span class="o">.</span><span class="no">SomeOtherModule</span><span class="o">.</span><span class="n">do_something</span><span class="p">(</span><span class="n">item</span><span class="p">)</span> <span class="k">end</span><span class="p">)</span> <span class="k">end</span> </code></pre></div></div> <p>Because the spawned process is not visible to the compiler, it won’t be able to load <code class="language-plaintext highlighter-rouge">MyLib.SomeOtherModule</code>. You have two options, either use <a href="https://hexdocs.pm/elixir/Kernel.ParallelCompiler.html#pmap/2"><code class="language-plaintext highlighter-rouge">Kernel.ParallelCompiler.pmap/2</code></a> or explicitly call <a href="https://hexdocs.pm/elixir/Code.html#ensure_compiled!/1"><code class="language-plaintext highlighter-rouge">Code.ensure_compiled!(MyLib.SomeOtherModule)</code></a> before spawning the process that uses said module.</p> <p>The second one is related to <code class="language-plaintext highlighter-rouge">@on_load</code> callbacks (typically used for <a href="https://www.erlang.org/doc/system/nif.html">NIFs</a>) that invoke other modules defined within the same project. For example:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">MyLib</span><span class="o">.</span><span class="no">SomeModule</span> <span class="k">do</span> <span class="nv">@on_load</span> <span class="ss">:init</span> <span class="k">def</span> <span class="n">init</span> <span class="k">do</span> <span class="no">MyLib</span><span class="o">.</span><span class="no">AnotherModule</span><span class="o">.</span><span class="n">do_something</span><span class="p">()</span> <span class="k">end</span> <span class="k">def</span> <span class="n">something_else</span> <span class="k">do</span> <span class="o">...</span> <span class="k">end</span> <span class="k">end</span> <span class="no">MyLib</span><span class="o">.</span><span class="no">SomeModule</span><span class="o">.</span><span class="n">something_else</span><span class="p">()</span> </code></pre></div></div> <p>The reason this fails is because <code class="language-plaintext highlighter-rouge">@on_load</code> callbacks are invoked within the code server and therefore they have limited ability to load additional modules. It is generally advisable to limit invocation of external modules during <code class="language-plaintext highlighter-rouge">@on_load</code> callbacks but, in case it is strictly necessary, you can set <code class="language-plaintext highlighter-rouge">@compile {:autoload, true}</code> in the invoked module to address this issue in a forward and backwards compatible manner.</p> <p>Both snippets above could actually lead to non-deterministic compilation failures in the past, and as a result of these changes, compiling these cases are now deterministic.</p> <h3 id="parallel-compilation-of-dependencies">Parallel compilation of dependencies</h3> <p>This release introduces a variable called <code class="language-plaintext highlighter-rouge">MIX_OS_DEPS_COMPILE_PARTITION_COUNT</code>, which instructs <code class="language-plaintext highlighter-rouge">mix deps.compile</code> to compile dependencies in parallel.</p> <p>While fetching dependencies and compiling individual Elixir dependencies already happened in parallel, as outlined in the previous section, there were pathological cases where performance gains would be left on the table, such as when compiling dependencies with native code or dependencies where one or two large files would take most of the compilation time.</p> <p>By setting <code class="language-plaintext highlighter-rouge">MIX_OS_DEPS_COMPILE_PARTITION_COUNT</code> to a number greater than 1, Mix will now compile multiple dependencies at the same time, using separate OS processes. Empirical testing shows that setting it to half of the number of cores on your machine is enough to maximize resource usage. The exact speed up will depend on the number of dependencies and the number of machine cores and some users reported up to 4x faster compilation times when using our release candidates. If you plan to enable it on CI or build servers, keep in mind it will most likely have a direct impact on memory usage too.</p> <h2 id="erlangotp-28-support">Erlang/OTP 28 support</h2> <p>Elixir v1.19 officially supports Erlang/OTP 28.1+ and later. In order to support the new Erlang/OTP 28 representation for regular expressions, structs can now control how they are escaped into abstract syntax trees by defining a <code class="language-plaintext highlighter-rouge">__escape__/1</code> callback.</p> <p>On the other hand, the new representation for regular expressions in Erlang/OTP 28+ implies they can no longer be used as default values for struct fields. Therefore, this is not allowed:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">Foo</span> <span class="k">do</span> <span class="k">defstruct</span> <span class="ss">regex:</span> <span class="sr">~r/foo/</span> <span class="k">end</span> </code></pre></div></div> <p>You can, however, still use regexes when initializing the structs themselves:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">Foo</span> <span class="k">do</span> <span class="k">defstruct</span> <span class="p">[</span><span class="ss">:regex</span><span class="p">]</span> <span class="k">def</span> <span class="n">new</span> <span class="k">do</span> <span class="p">%</span><span class="no">Foo</span><span class="p">{</span><span class="ss">regex:</span> <span class="sr">~r/foo/</span><span class="p">}</span> <span class="k">end</span> <span class="k">end</span> </code></pre></div></div> <h2 id="openchain-certification">OpenChain certification</h2> <p>Elixir v1.19 is also our first release following OpenChain compliance, <a href="https://elixir-lang.org/blog/2025/02/26/elixir-openchain-certification/">as previously announced</a>. In a nutshell:</p> <ul> <li>Elixir releases now include a Source SBoM in CycloneDX 1.6 or later and SPDX 2.3 or later formats.</li> <li>Each release is attested along with the Source SBoM.</li> </ul> <p>These additions offer greater transparency into the components and licenses of each release, supporting more rigorous supply chain requirements.</p> <p>This work was performed by <a href="https://maennchen.dev">Jonatan Männchen</a> and sponsored by the <a href="https://erlef.org">Erlang Ecosystem Foundation</a>.</p> <h2 id="summary">Summary</h2> <p>There are many other goodies in this release, such as improved option parsing, better debuggability and performance in ExUnit, the addition of <code class="language-plaintext highlighter-rouge">mix help Mod</code>, <code class="language-plaintext highlighter-rouge">mix help Mod.fun</code>, <code class="language-plaintext highlighter-rouge">mix help Mod.fun/arity</code>, and <code class="language-plaintext highlighter-rouge">mix help app:package</code> to make documentation accessible via shell for humans and agents, and much more. <a href="https://hexdocs.pm/elixir/1.19/changelog.html">See the CHANGELOG</a> for the complete release notes.</p> <p>Happy coding!</p> Wojtek Mach José Valim 2025-08-18T00:00:00+00:00 /blog/2025/08/18/interop-and-portability <p>The Erlang Virtual Machine has, historically, provided three main options for interoperability with other languages and ecosystems, with different degrees of isolation:</p> <ul> <li> <p><a href="https://www.erlang.org/doc/apps/erts/erl_nif.html">NIFs (Native Implemented Functions)</a> integrate with third party code in the same memory space via C bindings. This translates to low overhead and best performance but it also means faulty code can bring the whole Virtual Machine down, bypassing some of Erlang’s fault-tolerance guarantees</p> </li> <li> <p><a href="https://www.erlang.org/doc/system/ports.html">Ports</a> start a separate Operating System process to communicate with other languages through STDIN/STDOUT, guaranteeing process isolation. In a typical Erlang fashion, ports are fully evented, concurrent, and distributed (i.e. you can pass and communicate with ports across nodes)</p> </li> <li> <p><a href="https://www.erlang.org/doc/apps/erts/erl_dist_protocol.html">Distributed nodes</a> rely on Erlang well-defined distribution and serialization protocol to communicate with other runtimes. Any language can implement said protocol and act as an Erlang node, giving you full node isolation between runtimes</p> </li> </ul> <p>Those mechanisms have led to multiple integrations between Elixir and other programming languages, such as Zig and Rust, and more recently C++, Python, and Swift, which we will explore here.</p> <p>Furthermore, alternative implementations of the Erlang VM and Elixir have brought a fourth category of <strong>interoperability through portability</strong>: where your Elixir program runs in a completely different environment to leverage its native capabilities, libraries, and ecosystem, while maintaining Elixir’s syntax and semantics (either partially or fully). This opens up some exciting new possibilities and since this approach is still relatively uncharted territory, let’s dive into it first.</p> <h2 id="portability">Portability</h2> <p>The <a href="https://atomvm.org">AtomVM</a> is a lightweight implementation of the Erlang VM that can run on constrained environments, such as microcontrollers with just a few hundred kilobytes of memory such as ESP32, STM32 or Pico. AtomVM supports a functional subset of Erlang VM and its standard library, all optimized to run on tiny microcontrollers.</p> <p>Given its low footprint, AtomVM can also target WebAssembly, paving the way to run Elixir in web browsers and alternative WASM runtimes in the future. The <a href="https://popcorn.swmansion.com">Popcorn</a> project, <a href="https://www.youtube.com/watch?v=ep--rQO1FRI">recently announced at ElixirConf EU 2025</a>, builds on those capabilities to provide better interoperability between Elixir and JavaScript.</p> <h3 id="popcorn">Popcorn</h3> <p><a href="https://popcorn.swmansion.com">Popcorn</a> is a library for running Elixir in web browsers, with JavaScript interoperability. Popcorn brings an extensive subset of Elixir semantics into the browser and, although it is in its infancy, it is already capable of <a href="https://popcorn.swmansion.com/demos/eval">running interactive Elixir code entirely client side</a>.</p> <p>And here is a quick example showing how to communicate with JavaScript from WASM:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">HelloPopcorn</span> <span class="k">do</span> <span class="kn">use</span> <span class="no">GenServer</span> <span class="nv">@process_name</span> <span class="ss">:main</span> <span class="k">def</span> <span class="n">start_link</span><span class="p">(</span><span class="n">args</span><span class="p">)</span> <span class="k">do</span> <span class="no">GenServer</span><span class="o">.</span><span class="n">start_link</span><span class="p">(</span><span class="bp">__MODULE__</span><span class="p">,</span> <span class="n">args</span><span class="p">,</span> <span class="ss">name:</span> <span class="nv">@process_name</span><span class="p">)</span> <span class="k">end</span> <span class="nv">@impl</span> <span class="no">true</span> <span class="k">def</span> <span class="n">init</span><span class="p">(</span><span class="n">_init_arg</span><span class="p">)</span> <span class="k">do</span> <span class="no">Popcorn</span><span class="o">.</span><span class="no">Wasm</span><span class="o">.</span><span class="n">register</span><span class="p">(</span><span class="nv">@process_name</span><span class="p">)</span> <span class="no">IO</span><span class="o">.</span><span class="n">puts</span><span class="p">(</span><span class="s2">"Hello console!"</span><span class="p">)</span> <span class="no">Popcorn</span><span class="o">.</span><span class="no">Wasm</span><span class="o">.</span><span class="n">run_js</span><span class="p">(</span><span class="sd">""" () =&gt; { document.body.innerHTML = "Hello from WASM!"; } """</span><span class="p">)</span> <span class="ss">:ignore</span> <span class="k">end</span> <span class="k">end</span> </code></pre></div></div> <p>Popcorn could help with Elixir adoption by making it really easy to create interactive guides with executable code right there in the browser. And once it’s production ready, it could enable offline, local-first applications, entirely in Elixir.</p> <h3 id="hologram">Hologram</h3> <p><a href="https://hologram.page">Hologram</a> is a full-stack isomorphic Elixir web framework that runs on top of Phoenix. It lets developers create dynamic, interactive web applications entirely in Elixir.</p> <p>Hologram transpiles Elixir code to JavaScript and provides a complete framework including templates, components, routing, and client-server communication for building rich web applications.</p> <p>Here is a snippet of a Hologram component that handles drawing events entirely client-side, taken from the official <a href="https://hologram.page/demos/svg-drawing">SVG Drawing Demo</a>:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">DrawingBoard</span> <span class="k">do</span> <span class="kn">use</span> <span class="no">Hologram</span><span class="o">.</span><span class="no">Component</span> <span class="k">def</span> <span class="n">init</span><span class="p">(</span><span class="n">_props</span><span class="p">,</span> <span class="n">component</span><span class="p">,</span> <span class="n">_server</span><span class="p">)</span> <span class="k">do</span> <span class="n">put_state</span><span class="p">(</span><span class="n">component</span><span class="p">,</span> <span class="ss">drawing?:</span> <span class="no">false</span><span class="p">,</span> <span class="ss">path:</span> <span class="s2">""</span><span class="p">)</span> <span class="k">end</span> <span class="k">def</span> <span class="n">template</span> <span class="k">do</span> <span class="err">~</span><span class="no">HOLO</span><span class="sd">""" &lt;svg class="cursor-crosshair touch-none bg-black w-[75vw] h-[75vh]" $pointer_down="start_drawing" $pointer_move="draw_move" $pointer_up="stop_drawing" $pointer_cancel="stop_drawing" &gt; &lt;path d={@path} stroke="white" stroke-width="2" fill="none" /&gt; &lt;/svg&gt; """</span> <span class="k">end</span> <span class="k">def</span> <span class="n">action</span><span class="p">(</span><span class="ss">:draw_move</span><span class="p">,</span> <span class="n">params</span><span class="p">,</span> <span class="n">component</span><span class="p">)</span> <span class="ow">when</span> <span class="n">component</span><span class="o">.</span><span class="n">state</span><span class="o">.</span><span class="n">drawing?</span> <span class="k">do</span> <span class="n">new_path</span> <span class="o">=</span> <span class="n">component</span><span class="o">.</span><span class="n">state</span><span class="o">.</span><span class="n">path</span> <span class="o">&lt;&gt;</span> <span class="s2">" L </span><span class="si">#{</span><span class="n">params</span><span class="o">.</span><span class="n">event</span><span class="o">.</span><span class="n">offset_x</span><span class="si">}</span><span class="s2"> </span><span class="si">#{</span><span class="n">params</span><span class="o">.</span><span class="n">event</span><span class="o">.</span><span class="n">offset_y</span><span class="si">}</span><span class="s2">"</span> <span class="n">put_state</span><span class="p">(</span><span class="n">component</span><span class="p">,</span> <span class="ss">:path</span><span class="p">,</span> <span class="n">new_path</span><span class="p">)</span> <span class="k">end</span> <span class="k">def</span> <span class="n">action</span><span class="p">(</span><span class="ss">:start_drawing</span><span class="p">,</span> <span class="n">params</span><span class="p">,</span> <span class="n">component</span><span class="p">)</span> <span class="k">do</span> <span class="n">new_path</span> <span class="o">=</span> <span class="n">component</span><span class="o">.</span><span class="n">state</span><span class="o">.</span><span class="n">path</span> <span class="o">&lt;&gt;</span> <span class="s2">" M </span><span class="si">#{</span><span class="n">params</span><span class="o">.</span><span class="n">event</span><span class="o">.</span><span class="n">offset_x</span><span class="si">}</span><span class="s2"> </span><span class="si">#{</span><span class="n">params</span><span class="o">.</span><span class="n">event</span><span class="o">.</span><span class="n">offset_y</span><span class="si">}</span><span class="s2">"</span> <span class="n">put_state</span><span class="p">(</span><span class="n">component</span><span class="p">,</span> <span class="ss">drawing?:</span> <span class="no">true</span><span class="p">,</span> <span class="ss">path:</span> <span class="n">new_path</span><span class="p">)</span> <span class="k">end</span> <span class="k">end</span> </code></pre></div></div> <p>While Popcorn runs on a lightweight implementation of the Erlang VM with all of its primitives, Hologram works directly on the Elixir syntax tree. They explore distinct paths for bringing Elixir to the browser and are both in active development.</p> <h2 id="native-implemented-functions-nifs">Native Implemented Functions (NIFs)</h2> <p>NIFs allow us to write performance-critical or system-level code and call it directly from Erlang and Elixir as if it were a regular function.</p> <p>NIFs solve practical problems like improving performance or using all Operating System capabilities. NIFs run in the same Operating System process as the VM, the same memory space. With them we can use third-party native libraries, execute syscalls, interface with the hardware, etc. On the other hand, using them can forgo some of Erlang’s stability and error handling guarantees.</p> <p>Originally, NIFs could never block and had to be written in a “yielding” fashion, which limited their applicability. Since Erlang/OTP 17, however, NIFs can be scheduled to run on separate OS threads called “dirty schedulers”, based on their workloads (IO or CPU). This has directly brought Elixir and the Erlang VM into new domains, such as <a href="https://github.com/elixir-nx">Numerical Elixir</a>, and to interop with new languages and ecosystems.</p> <h3 id="c">C</h3> <p>Erlang’s NIFs directly target the C programming language and is used to implement low-level functionality present in Erlang’s standard library:</p> <div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp">#include</span> <span class="cpf">&lt;erl_nif.h&gt;</span><span class="cp"> </span> <span class="k">static</span> <span class="n">ERL_NIF_TERM</span> <span class="nf">add_int64_nif</span><span class="p">(</span><span class="n">ErlNifEnv</span><span class="o">*</span> <span class="n">env</span><span class="p">,</span> <span class="kt">int</span> <span class="n">argc</span><span class="p">,</span> <span class="k">const</span> <span class="n">ERL_NIF_TERM</span> <span class="n">argv</span><span class="p">[])</span> <span class="p">{</span> <span class="kt">int64_t</span> <span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">;</span> <span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="n">enif_get_int64</span><span class="p">(</span><span class="n">env</span><span class="p">,</span> <span class="n">argv</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="o">&amp;</span><span class="n">a</span><span class="p">)</span> <span class="o">||</span> <span class="o">!</span><span class="n">enif_get_int64</span><span class="p">(</span><span class="n">env</span><span class="p">,</span> <span class="n">argv</span><span class="p">[</span><span class="mi">1</span><span class="p">],</span> <span class="o">&amp;</span><span class="n">b</span><span class="p">))</span> <span class="p">{</span> <span class="k">return</span> <span class="n">enif_make_badarg</span><span class="p">(</span><span class="n">env</span><span class="p">);</span> <span class="p">}</span> <span class="k">return</span> <span class="n">enif_make_int64</span><span class="p">(</span><span class="n">env</span><span class="p">,</span> <span class="n">a</span> <span class="o">+</span> <span class="n">b</span><span class="p">);</span> <span class="p">}</span> <span class="k">static</span> <span class="n">ErlNifFunc</span> <span class="n">nif_funcs</span><span class="p">[]</span> <span class="o">=</span> <span class="p">{</span> <span class="p">{</span><span class="s">"add"</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="n">add_int64_nif</span><span class="p">},</span> <span class="p">};</span> <span class="n">ERL_NIF_INIT</span><span class="p">(</span><span class="s">"Elixir.Example"</span><span class="p">,</span> <span class="n">nif_funcs</span><span class="p">,</span> <span class="nb">NULL</span><span class="p">,</span> <span class="nb">NULL</span><span class="p">,</span> <span class="nb">NULL</span><span class="p">,</span> <span class="nb">NULL</span><span class="p">)</span> </code></pre></div></div> <p>Writing NIFs in C can be verbose and error-prone. Fortunately, the Elixir ecosystem offers a number of high-quality libraries that make it possible to write NIFs in other languages, let’s check them out.</p> <h3 id="c-1">C++</h3> <p><a href="https://github.com/elixir-nx/fine">Fine</a> is a lightweight C++ library that wraps the NIF API with a modern interface. Given the widespread use of C++ in machine learning and data, Fine aims to reduce the friction of getting from Elixir to C++ and vice-versa.</p> <p>Here’s the same NIF that adds two numbers in C++, using Fine:</p> <div class="language-c++ highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp">#include</span> <span class="cpf">&lt;fine.hpp&gt;</span><span class="cp"> </span> <span class="kt">int64_t</span> <span class="nf">add</span><span class="p">(</span><span class="n">ErlNifEnv</span> <span class="o">*</span><span class="n">env</span><span class="p">,</span> <span class="kt">int64_t</span> <span class="n">a</span><span class="p">,</span> <span class="kt">int64_t</span> <span class="n">b</span><span class="p">)</span> <span class="p">{</span> <span class="k">return</span> <span class="n">a</span> <span class="o">+</span> <span class="n">b</span><span class="p">;</span> <span class="p">}</span> <span class="n">FINE_NIF</span><span class="p">(</span><span class="n">add</span><span class="p">,</span> <span class="mi">0</span><span class="p">);</span> <span class="n">FINE_INIT</span><span class="p">(</span><span class="s">"Elixir.Example"</span><span class="p">);</span> </code></pre></div></div> <p>Fine automatically encodes and decodes NIF arguments and return values based on the function signature, significantly reducing boilerplate code. It also has first-class support for Elixir structs, propagating C++ exceptions as Elixir exceptions, and more.</p> <h3 id="rust">Rust</h3> <p><a href="https://github.com/rusterlium/rustler">Rustler</a> is a library for writing NIFs in Rust. The goal is to make it impossible to crash the VM when using “safe” Rust code. Furthermore, Rustler makes it easy to encode/decode Rust values to and from Elixir terms while safely and ergonomically managing resources.</p> <p>Here’s an example NIF implemented with Rustler:</p> <div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">#[rustler::nif]</span> <span class="k">fn</span> <span class="nf">add</span><span class="p">(</span><span class="n">a</span><span class="p">:</span> <span class="nb">i64</span><span class="p">,</span> <span class="n">b</span><span class="p">:</span> <span class="nb">i64</span><span class="p">)</span> <span class="k">-&gt;</span> <span class="nb">i64</span> <span class="p">{</span> <span class="n">a</span> <span class="o">+</span> <span class="n">b</span> <span class="p">}</span> <span class="nn">rustler</span><span class="p">::</span><span class="nd">init!</span><span class="p">(</span><span class="s">"Elixir.Example"</span><span class="p">);</span> </code></pre></div></div> <h3 id="zig">Zig</h3> <p><a href="https://hexdocs.pm/zigler">Zigler</a> lets us write NIFs in Zig, a low-level programming language designed for maintaining robust, optimal, and reusable software. Zig removes hidden control flow, implicit memory allocation, and similar abstractions in favour of code that’s explicit and predictable.</p> <p>Zigler compiles Zig code at build time and exposes it directly to Elixir, without external build scripts or glue. It tightly integrates with Elixir tooling: Zig code is formatted via <code class="language-plaintext highlighter-rouge">mix format</code> and documentation written in Zig appears in IEx via the <code class="language-plaintext highlighter-rouge">h</code> helper.</p> <p>Here’s an example NIF in Zig:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">iex</span><span class="o">&gt;</span> <span class="no">Mix</span><span class="o">.</span><span class="n">install</span><span class="p">([</span><span class="ss">:zigler</span><span class="p">])</span> <span class="n">iex</span><span class="o">&gt;</span> <span class="k">defmodule</span> <span class="no">Example</span> <span class="k">do</span> <span class="kn">use</span> <span class="no">Zig</span><span class="p">,</span> <span class="ss">otp_app:</span> <span class="ss">:zigler</span> <span class="sx">~Z""</span><span class="s2">" pub fn add(a: i64, b: i64) i64 { return a + b; } """</span> <span class="k">end</span> <span class="n">iex</span><span class="o">&gt;</span> <span class="no">Example</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">)</span> <span class="mi">3</span> </code></pre></div></div> <p>We can write NIFs directly in IEx sessions, scripts, Livebook notebooks, and similar! And with Zig’s excellent interop with C, it’s really easy to experiment with native code on the Erlang VM.</p> <h3 id="python">Python</h3> <p><a href="https://github.com/livebook-dev/pythonx">Pythonx</a> runs a Python interpreter in the same OS process as your Elixir application, allowing you to evaluate Python code and conveniently convert between Python and Elixir data structures. Pythonx also integrates with the <a href="https://docs.astral.sh/uv/">uv</a> package manager, automating the management of Python and its dependencies.</p> <p>One caveat is that Python’s Global Interpreter Lock (GIL) prevents multiple threads from executing Python code at the same time so calling Pythonx from multiple Elixir processes does not provide concurrency we might expect and can become source of bottlenecks. However, GIL is a constraint for regular Python code only. Packages with CPU-intense functionality, such as <code class="language-plaintext highlighter-rouge">numpy</code>, have native implementation of many functions and invoking those releases the GIL (GIL is also released when waiting on I/O).</p> <p>Here’s an example of using <code class="language-plaintext highlighter-rouge">numpy</code> in Elixir:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">iex</span><span class="o">&gt;</span> <span class="no">Mix</span><span class="o">.</span><span class="n">install</span><span class="p">([{</span><span class="ss">:pythonx</span><span class="p">,</span> <span class="s2">"~&gt; 0.4.0"</span><span class="p">}])</span> <span class="n">iex</span><span class="o">&gt;</span> <span class="no">Pythonx</span><span class="o">.</span><span class="n">uv_init</span><span class="p">(</span><span class="sd">""" [project] name = "myapp" version = "0.0.0" requires-python = "==3.13.*" dependencies = [ "numpy==2.2.2" ] """</span><span class="p">)</span> <span class="n">iex</span><span class="o">&gt;</span> <span class="kn">import</span> <span class="no">Pythonx</span><span class="p">,</span> <span class="ss">only:</span> <span class="ss">:sigils</span> <span class="n">iex</span><span class="o">&gt;</span> <span class="n">x</span> <span class="o">=</span> <span class="mi">1</span> <span class="n">iex</span><span class="o">&gt;</span> <span class="err">~</span><span class="no">PY</span><span class="sd">""" import numpy as np a = np.int64(x) b = np.int64(2) a + b """</span> <span class="c1">#Pythonx.Object&lt;</span> <span class="n">np</span><span class="o">.</span><span class="n">int64</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span> <span class="o">&gt;</span> </code></pre></div></div> <p><a href="https://livebook.dev">Livebook</a> uses Pythonx to allow Elixir and Python code cells to co-exist in the same notebook (and in the same memory space), with low-overhead when transferring data between them.</p> <h2 id="distributed-nodes">Distributed nodes</h2> <p>Elixir, by way of Erlang, has built-in support for distributed systems. Multiple nodes can connect over a network and communicate using message passing, with the same primitives such as <code class="language-plaintext highlighter-rouge">send</code> and <code class="language-plaintext highlighter-rouge">receive</code> used for both local and remote processes.</p> <p>Nodes become discoverable in the cluster simply by starting them with names. Once we connect to a node, we can send messages, spawn remote processes, and more. Here’s an example:</p> <div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>iex <span class="nt">--name</span> a@127.0.0.1 <span class="nt">--cookie</span> secret <span class="nv">$ </span>iex <span class="nt">--name</span> b@127.0.0.1 <span class="nt">--cookie</span> secret iex<span class="o">(</span>a@127.0.0.1<span class="o">)&gt;</span> Node.connect<span class="o">(</span>:<span class="s2">"b@127.0.0.1"</span><span class="o">)</span> iex<span class="o">(</span>a@127.0.0.1<span class="o">)&gt;</span> node<span class="o">()</span> :<span class="s2">"a@127.0.0.1"</span> iex<span class="o">(</span>a@127.0.0.1<span class="o">)&gt;</span> :erpc.call<span class="o">(</span>:<span class="s2">"b@127.0.0.1"</span>, fn -&gt; node<span class="o">()</span> end<span class="o">)</span> :<span class="s2">"b@127.0.0.1"</span> </code></pre></div></div> <p>While Distributed Erlang is typically used for Erlang-Erlang communication, it can be also used for interacting with programs written in other programming languages. Erlang/OTP includes <a href="https://www.erlang.org/doc/apps/erl_interface">Erl_Interface</a>, a C library for writing programs that can participate in the Erlang cluster. Such programs are commonly called C nodes.</p> <p>Any language may implement these protocols from scratch or, alternatively, use <code class="language-plaintext highlighter-rouge">erl_interface</code> as its building block. For example, Erlang/OTP ships with <a href="https://www.erlang.org/doc/apps/jinterface">Jinterface</a> application, a Java library that lets JVM programs act as distributed Erlang nodes. Another recent example is the <a href="https://github.com/otp-interop/swift-erlang-actor-system">Swift Erlang Actor System</a>, for communicating between Swift and Erlang/Elixir programs.</p> <h2 id="ports">Ports</h2> <p>Last but not least, ports are the basic mechanism that Elixir/Erlang uses to communicate with the outside world. Ports are the most common of interoperability across programming languages, so we will only provide two brief examples.</p> <p>In Elixir, the <code class="language-plaintext highlighter-rouge">Port</code> module offers a low-level API to start separate programs. Here’s an example that runs <code class="language-plaintext highlighter-rouge">uname -s</code> to print the current operating system:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">iex</span><span class="o">&gt;</span> <span class="n">port</span> <span class="o">=</span> <span class="no">Port</span><span class="o">.</span><span class="n">open</span><span class="p">({</span><span class="ss">:spawn</span><span class="p">,</span> <span class="s2">"uname -s"</span><span class="p">},</span> <span class="p">[</span><span class="ss">:binary</span><span class="p">])</span> <span class="n">iex</span><span class="o">&gt;</span> <span class="n">flush</span><span class="p">()</span> <span class="p">{</span><span class="c1">#Port&lt;0.3&gt;, {:data, "Darwin\n"}}</span> <span class="n">iex</span><span class="o">&gt;</span> <span class="n">send</span><span class="p">(</span><span class="n">port</span><span class="p">,</span> <span class="p">{</span><span class="n">self</span><span class="p">(),</span> <span class="ss">:close</span><span class="p">})</span> <span class="n">iex</span><span class="o">&gt;</span> <span class="n">flush</span><span class="p">()</span> <span class="p">{</span><span class="c1">#Port&lt;0.3&gt;, :closed}</span> <span class="ss">:ok</span> </code></pre></div></div> <p>Most times, however, developers use <code class="language-plaintext highlighter-rouge">System.cmd/3</code> to invoke short-running programs:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">iex</span><span class="o">&gt;</span> <span class="no">System</span><span class="o">.</span><span class="n">cmd</span><span class="p">(</span><span class="s2">"uname"</span><span class="p">,</span> <span class="p">[</span><span class="s2">"-s"</span><span class="p">])</span> <span class="p">{</span><span class="s2">"Darwin</span><span class="se">\n</span><span class="s2">"</span><span class="p">,</span> <span class="mi">0</span><span class="p">}</span> </code></pre></div></div> <h2 id="summary">Summary</h2> <p>This article highlights many of the several options for interoperating with Elixir and the Erlang Virtual Machine. While it does not aim to be a complete reference, it covers integration across a range of languages, such as Rust, Zig, Python and Swift, as well as portability to different environments, including microcontrollers and web browsers.</p> José Valim 2025-08-05T00:00:00+00:00 /blog/2025/08/05/global-elixir-meetups <blockquote> <p>Update: Our first Global Elixir Meetups was a success with 46 meetups spread across six continents. Thanks to everyone who organized and attended. See you next time!</p> </blockquote> <p>We are launching <a href="https://globalelixirmeetups.com">Global Elixir Meetups (GEMs)</a> - a week where the Elixir community organizes meetups around the world to meet and learn from each other. Our goal is to spark local communities, old and new, to get together and discuss everything related to Elixir and the Erlang VM.</p> <p>Our first GEM will happen on 22-28 September and, if you were looking for an opportunity to organize a meetup, now is the perfect time: visit <a href="https://globalelixirmeetups.com">the Global Elixir Meetups website</a> to learn how to organize and attend. Organizers may also opt-in to live stream their meetups to the whole world directly from the website.</p> <p><img src="/images/contents/gem.jpeg" alt="Global Elixir Meetup banner" title="Global Elixir Meetup banner" /></p> <p>The Global Elixir Meetup is organized by <a href="https://swmansion.com">Software Mansion</a>, who brought their expertise as creators of <a href="https://membrane.stream">Membrane</a> and <a href="https://elixir-webrtc.org">Elixir WebRTC</a> to make it all possible. At launch, we are already counting with 7 meetups across Europe, South America, and North America, with hopefully more continents joining us soon.</p> <p>Go ahead and find your closest GEM or run your own!</p> José Valim 2025-06-02T00:00:00+00:00 /blog/2025/06/02/elixir-outreach-stipend-for-speakers <p><a href="https://dashbit.co">Dashbit</a>, <a href="https://oban.pro">Oban</a>, and the <a href="https://erlef.org">Erlang Ecosystem Foundation (EEF)</a> are glad to announce a new program, which we will trial over the next 12 months, called “Elixir Outreach”. Our goal is to provide funds to community members who want to present Elixir and Erlang to other ecosystems and communities, while respecting our joint values.</p> <p>In a nutshell:</p> <ul> <li> <p>We will provide funds to community members to speak <strong>in-person</strong> about anything related to Elixir and the Erlang VM.</p> </li> <li> <p>We will cover hotel and transportation costs for up to $700 USD. Please reach out, even if you expect to exceed that limit. This is our first time running the program and we’re refining the budget.</p> </li> <li> <p>The event must expect at least 150 attendees and happen outside of the Elixir, overall BEAM, and functional programming communities. In other words, we won’t cover costs for attending Erlang, Elixir, or other BEAM/FP conferences nor meetups. Consider it as an opportunity to learn and bring external knowledge and experiences to the BEAM community.</p> </li> <li> <p>You will be expected to send a report about your experience. The format and duration is up to you. We’d prefer that you write a blog post or an article sharing your overall experience with the involved communities. However, if you would prefer to only send it privately to us, that’s fine too!</p> </li> </ul> <p>The event should take place within your area. Our overall goal is to support multiple individuals, rather than drain our budget on a few long-distance flights (such as across coasts or continents). We are flexible on event location, distance, or type. If in doubt, reach out to <a href="mailto:elixir_outreach@erlef.org">elixir_outreach at erlef dot org</a></p> <p>Our initial budget of $7000 was donated by Dashbit ($5000) and Oban ($2000) to the Erlang Ecosystem Foundation (EEF), specifically for this program. The EEF will oversee the distribution of the funds.</p> <h2 id="requesting-a-stipend">Requesting a stipend</h2> <p>To request a stipend, visit the <a href="https://erlef.org/stipends/form?type=elixir-outreach">Erlang Ecosystem Foundation website and choose “Elixir Outreach” as the stipend type</a>.</p> <p>Given we have limited funds, we cannot guarantee they will be available when you request them. We recommend reaching out to us before submitting or acceptance your talk. Therefore, by contacting us early, we can validate if the event matches the criteria above, ask questions, and earmark the funds. Once your talk is accepted, send us any itemized travel and accommodation costs so we can transfer the stipend to you, (not in excess of $700 USD).</p> <p>You can also request a stipend after your talk has already been accepted, but then there are no guarantees a stipend will be available.</p> <p>Our goal is to make this process simple and as straight-forward as possible. Although, we reserve the right to refuse a request for any reason. If in doubt, reach out to <a href="mailto:elixir_outreach@erlef.org">elixir_outreach at erlef dot org</a>.</p> <h2 id="acknowledgements">Acknowledgements</h2> <p>This is a new effort for all involved! Please be patient while we figure out the details.</p> <p>If you are looking for conferences to speak at, <a href="https://www.codosaur.us/speaking/cfps-ending-soon">Dave Aronson keeps a list of CFPs closing soon</a> and there are likely others available. Note, we don’t necessarily endorse all of the conferences listed nor guarantee they meet the requirements above, but the list may help you get the ball rolling.</p> <p>Thanks to Parker Selbert, Shannon Selbert, Brian Cardarella, Alistair Woodman, and Lee Barney for feedback and helping make this a reality.</p> Lars Wikman José Valim 2025-03-25T00:00:00+00:00 /blog/2025/03/25/cyanview-elixir-case <p>How do you coordinate visual fidelity across two hundred cameras for a live event like the Super Bowl?</p> <p>The answer is: by using the craft of camera shading, which involves adjusting each camera to ensure they match up in color, exposure and various other visual aspects. The goal is to turn the live event broadcast into a cohesive and consistent experience. For every angle used, you want the same green grass and the same skin tones. Everything needs to be very closely tuned across a diverse set of products and brands. From large broadcast cameras, drone cameras, and PTZ cameras to gimbal-held mirrorless cameras and more. This is what Cyanview does. Cyanview is a small Belgian company that sells products for the live video broadcast industry, and its primary focus is shading.</p> <p>Broadcast is a business where you only get one chance to prove that your tool is up to the task. Reliability is king. There can be no hard failures.</p> <p>A small team of three built a product so powerful and effective that it spread across the industry purely on the strength of its functionality. Without any marketing, it earned a reputation among seasoned professionals and became a staple at the world’s top live events. Cyanview’s Remote Control Panel (RCP) is now used by specialist video operators on the Olympics, Super Bowl, NFL, NBA, ESPN, Amazon and many more. Even most fashion shows in Paris use Cyanview’s devices.</p> <p>These devices put Elixir right in the critical path for serious broadcast operations. By choosing Elixir, Cyanview gained best-in-class networking features, state-of-the-art resilience and an ecosystem that allowed fast iteration on product features.</p> <p><img src="/images/cases/bg/cyanview-4.jpg" alt="Operating many displays with Cyanview products" title="Operating many displays with Cyanview products" /></p> <h2 id="why-elixir">Why Elixir?</h2> <p>The founding team of Cyanview primarily had experience with embedded development, and the devices they produce involve a lot of low-level C code and plenty of FPGA. This is due to the low-level details of color science and the really tight timing requirements.</p> <p>If you’ve ever worked with camera software, you know it can be a mixed bag. Even after going fully digital, much of it remained tied to analog systems or relied on proprietary connectivity solutions. Cyanview has been targeting IP (as in Internet Protocol) from early on. This means Cyanview’s software can operate on commodity networks that work in well-known and well-understood ways. This has aligned well with an increase in remote production, partially due to the pandemic, where production crews operate from a central location with minimal crew on location. Custom radio frequency or serial wire protocols have a hard time scaling to cross-continent distances.</p> <p>This also paved the way for Elixir, as the Erlang VM was designed to communicate and coordinate millions of devices, reliably, over the network.</p> <p>Elixir was brought in by the developer Ghislain, who needed to build integrations with cameras and interact with the other bits of required video gear, with many different protocols over the network. The language comes with a lot of practical features for encoding and decoding binary data down to the individual bits. Elixir gave them a strong foundation and the tools to iterate fast.</p> <p>Ghislain has been building the core intellectual property of Cyanview ever since. While the physical device naturally has to be solid, reliable, and of high quality, a lot of the secret sauce ultimately lies in the massive number of integrations and huge amounts of reverse engineering. Thus, the product is able to work with as many professional camera systems and related equipment as possible. It is designed to be compatible with everything and anything a customer is using. Plus, it offers an API to ensure smooth integration with other devices.</p> <p>David Bourgeois, the founder of Cyanview, told us a story how these technical decisions alongside Elixir helped them tackle real-world challenges:</p> <p>“During the Olympics in China, a studio in Beijing relied on a large number of Panasonic PTZ cameras. Most of their team, however, was based in Paris and needed to control the cameras remotely to run various shows throughout the day. The problem? Panasonic’s camera protocols were never designed for internet use — they require precise timing and multiple messages for every adjustment. With network latency, that leads to timeouts, disconnects, and system failures… So they ended up placing our devices next to the cameras in Beijing and controlled them over IP from Paris — just as designed.”</p> <p><img src="/images/cases/bg/cyanview-2.jpg" alt="Cyanview RIO device mounted on a camera at a sports field" title="Cyanview RIO device mounted on a camera at a sports field" /></p> <p>The devices in a given location communicate and coordinate on the network over a custom MQTT protocol. Over a hundred cameras without issue on a single Remote Control Panel (RCP), implemented on top of Elixir’s network stack.</p> <h2 id="technical-composition">Technical composition</h2> <p>The system as a whole consists of RCP devices running a Yocto Linux system, with most of the logic built in Elixir and C. While Python is still used for scripting and tooling, its role has gradually diminished. The setup also includes multiple microcontrollers and the on-camera device, all communicating over MQTT. Additionally, cloud relays facilitate connectivity, while dashboards and controller UIs provide oversight and control. The two critical devices are the RCP offering control on the production end and the RIO handling low-latency manipulation of the camera. Both run Elixir.</p> <p>The configuration UI is currently built in Elm, but - depending on priorities - it might be converted to <a href="https://phoenixframework.org/">Phoenix LiveView</a> over time to reduce the number of languages in use. The controller web UI is already in LiveView, and it is performing quite well on a very low-spec embedded Linux machine.</p> <p>The cloud part of the system is very limited today, which is unusual in a world of SaaS. There are cloud relays for distributing and sharing camera control as well as forwarding network ports between locations and some related features, also built in Elixir, but cloud is not at the core of the business. The devices running Elixir on location form a cluster over IP using a custom MQTT-based protocol suited to the task and are talking to hundreds of cameras and other video devices.</p> <p>It goes without saying that integration with so much proprietary equipment comes with challenges. Some integrations are more reliable than others. Some devices are more common, and their quirks are well-known through hard-won experience. A few even have good documentation that you can reference while others offer mystery and constant surprises. In this context, David emphasizes the importance of Elixir’s mechanisms for recovering from failures:</p> <p>“If one camera connection has a blip, a buggy protocol or the physical connection to a device breaks it is incredibly important that everything else keeps working. And this is where Elixir’s supervision trees provide a critical advantage.”</p> <h2 id="growth--team-composition">Growth &amp; team composition</h2> <p>The team has grown over the 9 years that the company has been operating, but it did so at a slow and steady pace. On average, the company has added just one person per year. With nine employees at the time of writing, Cyanview supports some of the biggest broadcast events in the world.</p> <p>There are two Elixir developers on board: Daniil who is focusing on revising some of the UI as well as charting a course into more cloud functionality, and Ghislain, who works on cameras and integration. Both LiveView and Elm are used to power device UIs and dashboards.</p> <p>What’s interesting is that, overall, the other embedded developers say that they don’t know much about Elixir and they don’t use it in their day-to-day work. Nonetheless, they are very comfortable implementing protocols and encodings in Elixir. The main reason they haven’t fully learned the language is simply time — they have plenty of other work to focus on, and deep Elixir expertise hasn’t been necessary. After all, there’s much more to their work beyond Elixir: designing PCBs, selecting electronic components, reverse engineering protocols, interfacing with displays, implementing FPGAs, managing production tests, real productions and releasing firmware updates.</p> <h2 id="innovation-and-customer-focus">Innovation and customer focus</h2> <p><img src="/images/cases/bg/cyanview-3.jpg" alt="Operator using Cyanview RCP for a massive crowd in an arena" title="Operator using Cyanview RCP for a massive crowd in an arena" /></p> <p>Whether it’s providing onboard cameras in 40+ cars during the 24 hours of Le Mans, covering Ninja Warrior, the Australian Open, and the US Open, operating a studio in the Louvre, being installed in NFL pylons, or connecting over 200 cameras simultaneously – the product speaks for itself. Cyanview built a device for a world that runs on top of IP, using Elixir, a language with networking and protocols deep in its bones. This choice enabled them to do both: implement support for all the equipment and provide features no one else had.</p> <p>By shifting from conventional local-area radio frequency, serial connections, and inflexible proprietary protocols to IP networking, Cyanview’s devices redefined how camera systems operate. Their feature set is unheard of in the industry: Unlimited multicam. Tally lights. Pan &amp; Tilt control. Integration with color correctors. World-spanning remote production.</p> <p>The ease and safety of shipping new functionality have allowed the company to support new features very quickly. One example is the increasing use of mirrorless cameras on gimbals to capture crowd shots. Cyanview were able to prototype gimbal control, test it with a customer and validate that it worked in a very short amount of time. This quick prototyping and validation of features is made possible by a flexible architecture that ensures that critical fundamentals don’t break.</p> <p>Camera companies that don’t produce broadcast shading remotes, such as Canon or RED, recommend Cyanview to their customers. Rather than competing with most broadcast hardware companies, Cyanview considers itself a partner. The power of a small team, a quality product and powerful tools can be surprising. Rather than focusing on marketing, Cyanview works very closely with its customers by supporting the success of their events and providing in-depth customer service.</p> <h2 id="looking-back-and-forward">Looking back and forward</h2> <p>When asked if he would choose Elixir again, David responded:</p> <p>“Yes. We’ve seen what the Erlang VM can do, and it has been very well-suited to our needs. You don’t appreciate all the things Elixir offers out of the box until you have to try to implement them yourself. It was not pure luck that we picked it, but we were still lucky. Elixir turned out to bring a lot that we did not know would be valuable to us. And we see those parts clearly now.”</p> <p>Cyanview hopes to grow the team more, but plans to do so responsibly over time. Currently there is a lot more to do than the small team can manage.</p> <p>Development is highly active, with complementary products already in place alongside the main RCP device, and the future holds even more in that regard. Cloud offerings are on the horizon, along with exciting hardware projects that build on the lessons learned so far. As these developments unfold, we’ll see Elixir play an increasingly critical role in some of the world’s largest live broadcasts.</p> <p><img src="/images/cases/bg/cyanview-1.jpg" alt="Cyanview Remote Control Panels in a control room" title="Cyanview Remote Control Panels in a control room" /></p> <h2 id="in-summary">In summary</h2> <p>A high-quality product delivering the right innovation at the right time in an industry that’s been underserved in terms of good integration. Elixir provided serious leverage for developing a lot of integrations with high confidence and consistent reliability. In an era where productivity and lean, efficient teams are everything, Cyanview is a prime example of how Elixir empowers small teams to achieve an outsized impact.</p> Jonatan Männchen José Valim 2025-02-26T00:00:00+00:00 /blog/2025/02/26/elixir-openchain-certification <p>We are pleased to share that the Elixir project now complies with <a href="https://openchainproject.org/">OpenChain</a> (<a href="https://www.iso.org/standard/81039.html">ISO/IEC 5230</a>), an international standard for open source license compliance. This step aligns with broader efforts to meet industry standards for supply chain and cybersecurity best practices.</p> <p>“Today’s announcement around Elixir’s conformance represents another significant example of community maturity,” says Shane Coughlan, OpenChain General Manager. “With projects - the final upstream - using ISO standards for compliance and security with increasing frequency, we are seeing a shift to longer-term improvements to trust in the supply chain.”</p> <h2 id="why-openchain-compliance-helps">Why OpenChain Compliance Helps</h2> <p>By following OpenChain (ISO/IEC 5230), we demonstrate clear processes around license compliance. This benefits commercial and community users alike, making Elixir easier to adopt and integrate with confidence.</p> <h2 id="changes-for-elixir-users">Changes for Elixir Users</h2> <p>Elixir has an automated release process where its artifacts are signed. This change strengthens this process by:</p> <ul> <li>All future Elixir releases will include a Source SBoM in <a href="https://cyclonedx.org/specification/overview/">CycloneDX 1.6 or later</a> and <a href="https://spdx.org/rdf/terms/">SPDX 2.3 or later</a> formats.</li> <li>Each release will be attested along with the Source SBoM.</li> </ul> <p>These additions offer greater transparency into the components and licenses of each release, supporting more rigorous supply chain requirements.</p> <h2 id="changes-for-contributors">Changes for Contributors</h2> <p>Contributing to Elixir remains largely the same, we have added more clarity and guidelines around it:</p> <ul> <li>Contributions remain under the Apache-2.0 License. Other licenses cannot be accepted.</li> <li>The project now enforces the <a href="https://developercertificate.org/">Developer Certificate of Origin (DCO)</a>, ensuring clarity around contribution ownership.</li> </ul> <p>Contributors will notice minimal procedural changes, as standard practices around licensing remain in place.</p> <p>For more details, see the <a href="https://github.com/elixir-lang/elixir/blob/main/CONTRIBUTING.md">CONTRIBUTING guidelines</a>.</p> <h2 id="commitment">Commitment</h2> <p>These updates were made in collaboration with the <a href="https://erlef.org/">Erlang Ecosystem Foundation</a>, reflecting a shared commitment to robust compliance and secure development practices. Thank you to everyone who supported this milestone. We appreciate the community’s ongoing contributions and look forward to continuing the growth of Elixir under these established guidelines.</p> Hugo Baraúna 2025-01-21T00:00:00+00:00 /blog/2025/01/21/remote-elixir-case <p><em>Welcome to our series of <a href="/cases.html">case studies about companies using Elixir in production</a>.</em></p> <p>Remote is the everywhere employment platform enabling companies to find, hire, manage, and pay people anywhere across the world.</p> <p>Founded in 2019, they reached unicorn status in just over two years and have continued their rapid growth trajectory since.</p> <p>Since day zero, Elixir has been their primary technology. Currently, their engineering organization as a whole consists of nearly 300 individuals.</p> <p>This case study focuses on their experience using Elixir in a high-growth environment.</p> <p><img src="/images/cases/bg/remote.png" alt="Remote website screenshot" /></p> <h2 id="why-elixir">Why Elixir?</h2> <p>Marcelo Lebre, co-founder and president of Remote, had worked with many languages and frameworks throughout his career, often encountering the same trade-off: easy-to-code versus easy-to-scale.</p> <p>In 2015, while searching for alternatives, he discovered Elixir. Intrigued, Marcelo decided to give it a try and immediately saw its potential. At the time, Elixir was still in its early days, but he noticed how fast the community was growing, with support for packages and frameworks starting to show up aggressively.</p> <p>In December 2018, when Marcelo and his co-founder decided to start the company, they had to make a decision about the technology that would support their vision. Marcelo wanted to prioritize building a great product quickly without worrying about scalability issues from the start. He found Elixir to be the perfect match:</p> <blockquote> <p>I wanted to focus on building a great product fast and not really worry about its scalability. Elixir was the perfect match—reliable performance, easy-to-read syntax, strong community, and a learning curve that made it accessible to new hires.</p> <p>- Marcelo Lebre, Co-founder and President</p> </blockquote> <p>The biggest trade-off Marcelo identified was the smaller pool of Elixir developers compared to languages like Ruby or Python. However, he quickly realized that the quality of candidates more than made up for it:</p> <blockquote> <p>The signal-to-noise ratio in the quality of Elixir candidates was much higher, which made the trade-off worthwhile.</p> <p>- Marcelo Lebre, Co-founder and President</p> </blockquote> <h2 id="growing-with-a-monolithic-architecture">Growing with a monolithic architecture</h2> <p>Remote operates primarily with a monolith, with Elixir in the backend and React in the front-end.</p> <p>The monolith enabled speed and simplicity, allowing the team to iterate quickly and focus on building features. However, as the company grew, they needed to invest in tools and practices to manage almost 180 engineers working in the same codebase.</p> <p>One practice was boundary enforcement. They used the <a href="https://github.com/sasa1977/boundary">Boundary library</a> to maintain strict boundaries between modules and domains inside the codebase.</p> <p>Another key investment was optimizing their compilation time in the CI pipeline. Since their project has around 15,000 files, compiling it in every build would take too long. So, they implemented incremental builds in their CI pipeline, recompiling only the files affected by changes instead of the entire codebase.</p> <blockquote> <p>I feel confident making significant changes in the codebase. The combination of using a functional language and our robust test suite allows us to keep moving forward without too much worry.</p> <p>- André Albuquerque, Staff Engineer</p> </blockquote> <p>Additionally, as their codebase grew, the Elixir language continued to evolve, introducing better abstractions for developers working with large codebases. For example, with the release of Elixir v1.11, the <a href="/blog/2020/10/06/elixir-v1-11-0-released/">introduction of config/runtime.exs</a> provided the Remote team with a better foundation for managing configuration. This enabled them to move many configurations from compile-time to runtime, significantly reducing unnecessary recompilations caused by configuration updates.</p> <h2 id="infra-structure-and-operations">Infra-structure and operations</h2> <p>One might expect Remote’s infrastructure to be highly complex, given their global scale and the size of their engineering team. Surprisingly, their setup remains relatively simple, reflecting a thoughtful balance between scalability and operational efficiency.</p> <p>Remote runs on AWS, using EKS (Elastic Kubernetes Service). The main backend (the monolith) operates in only five pods, each with 10 GB of memory. They use <a href="https://www.erlang.org/doc/system/distributed.html">Distributed Erlang</a> to connect the nodes in their cluster, enabling seamless communication between processes running on different pods.</p> <p>For job processing, they rely on <a href="https://github.com/oban-bg/oban">Oban</a>, which runs alongside the monolith in the same pods.</p> <p>Remote also offers a public API for partners. While this API server runs separately from the monolith, it is the same application, configured to start a different part of its supervision tree. The separation was deliberate, as the team anticipated different load patterns for the API and wanted the flexibility to scale it independently.</p> <p>The database setup includes a primary PostgreSQL instance on AWS RDS, complemented by a read-replica for enhanced performance and scalability. Additionally, a separate Aurora PostgreSQL instance is dedicated to storing Oban jobs. Over time, the team has leveraged tools like PG Analyze to optimize performance, addressing bottlenecks such as long queries and missing indexes.</p> <p>This streamlined setup has proven resilient, even during unexpected spikes in workload. The team shared an episode where a worker’s job count unexpectedly grew by two orders of magnitude. Remarkably, the system handled the increase seamlessly, continuing to run as usual without requiring any design changes or manual intervention.</p> <blockquote> <p>We once noticed two weeks later that a worker’s load had skyrocketed. But the scheduler worked fine, and everything kept running smoothly. That was fun.</p> <p>- Alex Naser, Staff Engineer</p> </blockquote> <h2 id="team-organization-and-responsibilities">Team organization and responsibilities</h2> <p>Around 90% of their backend team works in the monolith, while the rest work in a few satellite services, also written in Elixir.</p> <p>Within the monolith, teams are organized around domains such as onboarding, payroll, and billing. Each team owns one or multiple domains.</p> <p>To streamline accountability in a huge monolith architecture, Remote invested heavily in team assignment mechanisms.</p> <p>They implemented a tagging system that assigns ownership down to the function level. This means any trace—whether sent to tools like Sentry or Datadog—carries a tag identifying the responsible team. This tagging also extends to endpoints, allowing teams to monitor their areas effectively and even set up dashboards for alerts, such as query times specific to their domain.</p> <p>The tagging system also simplifies CI workflows. When a test breaks, it’s automatically linked to the responsible team based on the Git commit. This ensures fast issue identification and resolution, removing the need for manual triaging.</p> <h2 id="hiring-and-training">Hiring and training</h2> <p>Remote’s hiring approach prioritizes senior engineers, regardless of their experience with Elixir.</p> <p>During the hiring process, all candidates are required to complete a coding exercise in Elixir. For those unfamiliar with the language, a tailored version of the exercise is provided, designed to introduce them to Elixir while reflecting the challenges they would face if hired.</p> <p>Once hired, new engineers are assigned an engineering buddy to guide them through the onboarding process.</p> <p>For hires without prior Elixir experience, Remote developed an internal Elixir training camp, a curated collection of best practices, tutorials, and other resources to introduce new hires to the language and ecosystem. This training typically spans two to four weeks.</p> <p>After completing the training, engineers are assigned their first tasks—carefully selected tickets designed to build confidence and familiarity with the codebase.</p> <h2 id="summing-up">Summing up</h2> <p>Remote’s journey highlights how thoughtful technology, infrastructure, and team organization decisions can support rapid growth.</p> <p>By leveraging Elixir’s strengths, they built a monolithic architecture that balanced simplicity with scalability. This approach allowed their engineers to iterate quickly in the early stages while effectively managing the complexities of a growing codebase.</p> <p>Investments in tools like the Boundary library and incremental builds ensured their monolith remained efficient and maintainable even as the team and codebase scaled dramatically.</p> <p>Remote’s relatively simple infrastructure demonstrates that scaling doesn’t always require complexity. Their ability to easily handle unexpected workload spikes reflects the robustness of their architecture and operational practices.</p> <p>Finally, their focus on team accountability and streamlined onboarding allowed them to maintain high productivity while integrating engineers from diverse technical backgrounds, regardless of their prior experience with Elixir.</p> José Valim 2024-12-19T00:00:00+00:00 /blog/2024/12/19/elixir-v1-18-0-released <p>Elixir v1.18 is an impressive release with improvements across the two main efforts happening within the Elixir ecosystem right now: set-theoretic types and language servers. It also comes with built-in JSON support and adds new capabilities to its unit testing library. Let’s go over each of those in detail.</p> <h2 id="type-inference-of-patterns-and-return-types">Type inference of patterns and return types</h2> <p>There are several updates in the typing department, so let’s break them down.</p> <h4 id="a-type-system-in-my-elixir">A type system? In my Elixir?</h4> <p>There is an on-going <a href="https://elixir-lang.org/blog/2023/06/22/type-system-updates-research-dev/">research and development</a> effort to bring static types to Elixir. Elixir’s type system is:</p> <ul> <li> <p><strong>sound</strong> - the types inferred and assigned by the type system align with the behaviour of the program</p> </li> <li> <p><strong>gradual</strong> - Elixir’s type system includes the <code class="language-plaintext highlighter-rouge">dynamic()</code> type, which can be used when the type of a variable or expression is checked at runtime. In the absence of <code class="language-plaintext highlighter-rouge">dynamic()</code>, Elixir’s type system behaves as a static one</p> </li> <li> <p><strong>developer friendly</strong> - the types are described, implemented, and composed using basic set operations: unions, intersections, and negation (hence it is a set-theoretic type system)</p> </li> </ul> <p>More interestingly, you can compose <code class="language-plaintext highlighter-rouge">dynamic()</code> with any type. For example, <code class="language-plaintext highlighter-rouge">dynamic(integer() or float())</code> means the type is either <code class="language-plaintext highlighter-rouge">integer()</code> or <code class="language-plaintext highlighter-rouge">float()</code> at runtime. This allows the type system to emit warnings if none of the types are satisfied, even in the presence of dynamism.</p> <h4 id="what-has-already-been-done">What has already been done?</h4> <p><a href="https://elixir-lang.org/blog/2024/06/12/elixir-v1-17-0-released/">Elixir v1.17 was the first release to incorporate the type system in the compiler</a>. In particular, we have added support for primitive types (integer, float, binary, pids, references, ports), atoms, and maps. We also added type checking to a handful of operations related to those types, such as accessing fields in maps, as in <code class="language-plaintext highlighter-rouge">user.adress</code> (mind the typo), performing structural comparisons between structs, as in <code class="language-plaintext highlighter-rouge">my_date &lt; ~D[2010-04-17]</code>, etc.</p> <h4 id="what-is-new-in-v118">What is new in v1.18?</h4> <p>The most exciting change in Elixir v1.18 is type checking of function calls, alongside gradual inference of patterns and return types. To understand how this will impact your programs, consider the following code defined in <code class="language-plaintext highlighter-rouge">lib/user.ex</code>:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">User</span> <span class="k">do</span> <span class="k">defstruct</span> <span class="p">[</span><span class="ss">:age</span><span class="p">,</span> <span class="ss">:car_choice</span><span class="p">]</span> <span class="k">def</span> <span class="n">drive</span><span class="p">(%</span><span class="no">User</span><span class="p">{</span><span class="ss">age:</span> <span class="n">age</span><span class="p">,</span> <span class="ss">car_choice:</span> <span class="n">car</span><span class="p">},</span> <span class="n">car_choices</span><span class="p">)</span> <span class="ow">when</span> <span class="n">age</span> <span class="o">&gt;=</span> <span class="mi">18</span> <span class="k">do</span> <span class="k">if</span> <span class="n">car</span> <span class="ow">in</span> <span class="n">car_choices</span> <span class="k">do</span> <span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">car</span><span class="p">}</span> <span class="k">else</span> <span class="p">{</span><span class="ss">:error</span><span class="p">,</span> <span class="ss">:no_choice</span><span class="p">}</span> <span class="k">end</span> <span class="k">end</span> <span class="k">def</span> <span class="n">drive</span><span class="p">(%</span><span class="no">User</span><span class="p">{},</span> <span class="n">_car_choices</span><span class="p">)</span> <span class="k">do</span> <span class="p">{</span><span class="ss">:error</span><span class="p">,</span> <span class="ss">:not_allowed</span><span class="p">}</span> <span class="k">end</span> <span class="k">end</span> </code></pre></div></div> <p>Elixir’s type system will infer the <code class="language-plaintext highlighter-rouge">drive</code> function expects a <code class="language-plaintext highlighter-rouge">User</code> struct as input and returns either <code class="language-plaintext highlighter-rouge">{:ok, dynamic()}</code> or <code class="language-plaintext highlighter-rouge">{:error, :no_choice}</code> or <code class="language-plaintext highlighter-rouge">{:error, :not_allowed}</code>. Therefore, the following code</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="no">User</span><span class="o">.</span><span class="n">drive</span><span class="p">({</span><span class="ss">:ok</span><span class="p">,</span> <span class="p">%</span><span class="no">User</span><span class="p">{}},</span> <span class="n">car_choices</span><span class="p">)</span> </code></pre></div></div> <p>will emit a warning stating that we are passing an invalid argument:</p> <p><img src="/images/contents/type-warning-function-clause.png" alt="Example of a warning when passing wrong argument to a function" /></p> <p>Now consider the expression below. We are expecting the <code class="language-plaintext highlighter-rouge">User.drive/2</code> call to return <code class="language-plaintext highlighter-rouge">:error</code>, which cannot possibly be true:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">case</span> <span class="no">User</span><span class="o">.</span><span class="n">drive</span><span class="p">(</span><span class="n">user</span><span class="p">,</span> <span class="n">car_choices</span><span class="p">)</span> <span class="k">do</span> <span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">car</span><span class="p">}</span> <span class="o">-&gt;</span> <span class="n">car</span> <span class="ss">:error</span> <span class="o">-&gt;</span> <span class="no">Logger</span><span class="o">.</span><span class="n">error</span><span class="p">(</span><span class="s2">"User cannot drive"</span><span class="p">)</span> <span class="k">end</span> </code></pre></div></div> <p>Therefore the code above would emit the following warning:</p> <p><img src="/images/contents/type-warning-case.png" alt="Example of a warning when a case clause won't ever match" /></p> <p>Our goal is for the warnings to provide enough contextual information that lead to clear reports and that’s an area we are actively looking for feedback. If you receive a warning that is unclear, please open up a bug report.</p> <p>Elixir v1.18 also augments the type system with support for tuples and lists, plus type checking of almost all Elixir language constructs, except <code class="language-plaintext highlighter-rouge">for</code>-comprehensions, <code class="language-plaintext highlighter-rouge">with</code>, and closures. Here is a non-exaustive list of the new violations that can be detected by the type system:</p> <ul> <li> <p>if you define a pattern that will never match any argument, such as <code class="language-plaintext highlighter-rouge">def function(x = y, x = :foo, y = :bar)</code></p> </li> <li> <p>matching or accessing tuples at an invalid index, such as <code class="language-plaintext highlighter-rouge">elem(two_element_tuple, 2)</code></p> </li> <li> <p>if you have a branch in a <code class="language-plaintext highlighter-rouge">try</code> that will never match the given expression</p> </li> <li> <p>if you have a branch in a <code class="language-plaintext highlighter-rouge">cond</code> that always passes (except the last one) or always fails</p> </li> <li> <p>if you attempt to use the return value of a call to <code class="language-plaintext highlighter-rouge">raise/2</code> (which by definition returns no value)</p> </li> </ul> <p>In summary, this release takes us further in our journey of providing type checking and type inference of existing Elixir programs, without requiring Elixir developers to explicitly add type annotations.</p> <p>For existing codebases with reasonable code coverage, most type system reports will come from uncovering dead code - code which won’t ever be executed - as seen in a <a href="https://github.com/phoenixframework/phoenix_live_view/commit/6c6e2aaf6a01957cc6bb8a27d2513bff273e8ca2">few</a> <a href="https://github.com/elixir-ecto/postgrex/commit/3308f277f455ec64f2d0d7be6263f77f295b1325">distinct</a> <a href="https://github.com/phoenixframework/flame/commit/0c0c2875e42952d2691cbdb7928fc32f4715e746">projects</a>. A notable example is the type system ability to track how private functions are used throughout a module and then point out which clauses are unused:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">Example</span> <span class="k">do</span> <span class="k">def</span> <span class="n">public</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="k">do</span> <span class="n">private</span><span class="p">(</span><span class="no">Integer</span><span class="o">.</span><span class="n">parse</span><span class="p">(</span><span class="n">x</span><span class="p">))</span> <span class="k">end</span> <span class="k">defp</span> <span class="n">private</span><span class="p">(</span><span class="no">nil</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="no">nil</span> <span class="k">defp</span> <span class="n">private</span><span class="p">(</span><span class="s2">"foo"</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="s2">"foo"</span> <span class="k">defp</span> <span class="n">private</span><span class="p">({</span><span class="n">int</span><span class="p">,</span> <span class="n">_rest</span><span class="p">}),</span> <span class="k">do</span><span class="p">:</span> <span class="n">int</span> <span class="k">defp</span> <span class="n">private</span><span class="p">(</span><span class="ss">:error</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="mi">0</span> <span class="k">defp</span> <span class="n">private</span><span class="p">(</span><span class="s2">"bar"</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="s2">"bar"</span> <span class="k">end</span> </code></pre></div></div> <p><img src="/images/contents/type-warning-private.png" alt="Example of a warning for unused private clauses" /></p> <p>Keep in mind the current implementation does not perform type inference of guards yet, which is an important source of typing information in programs. There is a lot the type system can learn about our codebases, that it does not yet. This brings us to the next topic.</p> <h4 id="future-work">Future work</h4> <p>The next Elixir release should improve the typing of maps, tuples, and closures, allowing us to type even more constructs. We also plan to fully type the <code class="language-plaintext highlighter-rouge">with</code> construct, <code class="language-plaintext highlighter-rouge">for</code>-comprehensions, as well as protocols.</p> <p>But more importantly, we want to focus on complete type inference of guards, which in turn will allow us to explore ideas such as redundant pattern matching clauses and exhaustiveness checks. Our goal with inference is to strike the right balance between developer experience, compilation times, and the ability of finding provable errors in existing codebases. You can learn more <a href="https://hexdocs.pm/elixir/1.18/gradual-set-theoretic-types.html#type-inference">about the trade-offs we made for inference in our documentation</a>.</p> <p>Future Elixir versions will introduce user-supplied type signatures, which should bring the benefits of a static type system without relying on inference. <a href="https://elixir-lang.org/blog/2023/06/22/type-system-updates-research-dev/">Check our previous article on the overall milestones for more information</a>.</p> <h4 id="sponsors">Sponsors</h4> <p>The type system was made possible thanks to a partnership between <a href="https://www.cnrs.fr/">CNRS</a> and <a href="https://remote.com/">Remote</a>. The development work is currently sponsored by <a href="https://www.fresha.com/">Fresha</a> (<a href="https://www.fresha.com/careers/openings?department=engineering">they are hiring!</a>), <a href="https://starfish.team/">Starfish*</a>, and <a href="https://dashbit.co/">Dashbit</a>.</p> <h2 id="language-server-listeners">Language server listeners</h2> <p>Three months ago, we welcomed <a href="https://elixir-lang.org/blog/2024/08/15/welcome-elixir-language-server-team/">the Official Language Server team</a>, with the goal of unifying the efforts behind code intelligence, tools, and editors in Elixir. Elixir v1.18 brings new features on this front by introducing locks and listeners to its compilation. Let’s understand what it means.</p> <p>At the moment, all language server implementations have their own compilation environment. This means that your project and dependencies during development are compiled once, for your own use, and then again for the language server. This duplicate effort could cause the language server experience to lag, when it could be relying on the already compiled artifacts of your project.</p> <p>This release addresses the issue by introducing a compiler lock, ensuring that only a single operating system running Elixir compiles your project at a given moment, and by providing the ability for one operating system process to listen to the compilation results of others. In other words, different Elixir instances can now communicate over the same compilation build, instead of racing each other.</p> <p>These enhancements do not only improve editor tooling, but they also directly benefit projects like IEx and Phoenix. Here is a quick snippet showing how to enable auto-reloading inside IEx, then running <code class="language-plaintext highlighter-rouge">mix compile</code> in one shell automatically reloads the module inside the IEx session:</p> <video controls=""> <source src="/images/contents/iex-auto-reload.mp4" type="video/mp4" /> Your browser does not support the video tag. </video> <h2 id="built-in-json">Built-in JSON</h2> <p><a href="https://www.erlang.org/doc/apps/stdlib/json.html">Erlang/OTP 27 added built-in support for JSON</a> and we are now bringing it to Elixir. A new module, called <a href="https://hexdocs.pm/elixir/1.18/JSON.html"><code class="language-plaintext highlighter-rouge">JSON</code></a>, has been added with functions to encode and decode JSON. Its most basic APIs reflect the ones <a href="https://hexdocs.pm/jason/Jason.html">from the Jason project</a> (the de-facto JSON library in the Elixir community up to this point).</p> <p>A new protocol, called <a href="https://hexdocs.pm/elixir/1.18/JSON.Encoder.html"><code class="language-plaintext highlighter-rouge">JSON.Encoder</code></a>, is also provided for those who want to customize how their own data types are encoded to JSON. You can also derive protocols for structs, with a single-line of code:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">@derive</span> <span class="p">{</span><span class="no">JSON</span><span class="o">.</span><span class="no">Encoder</span><span class="p">,</span> <span class="ss">only:</span> <span class="p">[</span><span class="ss">:id</span><span class="p">,</span> <span class="ss">:name</span><span class="p">]}</span> <span class="k">defstruct</span> <span class="p">[</span><span class="ss">:id</span><span class="p">,</span> <span class="ss">:name</span><span class="p">,</span> <span class="ss">:email</span><span class="p">]</span> </code></pre></div></div> <p>The deriving API mirrors the one from <code class="language-plaintext highlighter-rouge">Jason</code>, helping those who want to migrate to the new <code class="language-plaintext highlighter-rouge">JSON</code> module.</p> <h2 id="parameterized-tests-and-exunit-groups">Parameterized tests and ExUnit groups</h2> <p><a href="https://hexdocs.pm/ex_unit/1.18/ExUnit.Case.html#module-parameterized-tests">ExUnit now supports parameterized tests</a>. This allows your test modules to run multiple times under different parameters.</p> <p>For example, Elixir ships <a href="https://hexdocs.pm/elixir/Registry.html">a local, decentralized and scalable key-value process storage called <code class="language-plaintext highlighter-rouge">Registry</code></a>. The registry can be partitioned and its implementation differs depending if partitioning is enabled or not. Therefore, during tests, we want to ensure both modes are exercised. With Elixir v1.18, we can achieve this by writing:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">Registry</span><span class="o">.</span><span class="no">Test</span> <span class="k">do</span> <span class="kn">use</span> <span class="no">ExUnit</span><span class="o">.</span><span class="no">Case</span><span class="p">,</span> <span class="ss">async:</span> <span class="no">true</span><span class="p">,</span> <span class="ss">parameterize:</span> <span class="p">[</span> <span class="p">%{</span><span class="ss">partitions:</span> <span class="mi">1</span><span class="p">},</span> <span class="p">%{</span><span class="ss">partitions:</span> <span class="mi">8</span><span class="p">}</span> <span class="p">]</span> <span class="c1"># ... the actual tests ...</span> <span class="k">end</span> </code></pre></div></div> <p>Once specified, the number of partitions is available as part of the test configuration. For example, to start one registry per test with the correct number of partitions, you can write:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="n">setup</span> <span class="n">config</span> <span class="k">do</span> <span class="n">partitions</span> <span class="o">=</span> <span class="n">config</span><span class="o">.</span><span class="n">partitions</span> <span class="n">name</span> <span class="o">=</span> <span class="ss">:"</span><span class="si">#{</span><span class="n">config</span><span class="o">.</span><span class="n">test</span><span class="si">}</span><span class="ss">_</span><span class="si">#{</span><span class="n">partitions</span><span class="si">}</span><span class="ss">"</span> <span class="n">opts</span> <span class="o">=</span> <span class="p">[</span><span class="ss">keys:</span> <span class="ss">:unique</span><span class="p">,</span> <span class="ss">name:</span> <span class="n">name</span><span class="p">,</span> <span class="ss">partitions:</span> <span class="n">partitions</span><span class="p">]</span> <span class="n">start_supervised!</span><span class="p">({</span><span class="no">Registry</span><span class="p">,</span> <span class="n">opts</span><span class="p">})</span> <span class="n">opts</span> <span class="k">end</span> </code></pre></div></div> <p>Prior to parameterized tests, Elixir resorted on code generation, which increased compilation times. Furthermore, ExUnit parameterizes the whole test modules, which also allows the different parameters to run concurrently if the <code class="language-plaintext highlighter-rouge">async: true</code> option is given. Overall, this features allows you to compile and run multiple scenarios more efficiently.</p> <p>Finally, ExUnit also comes with the ability of specifying test groups. While ExUnit supports running tests concurrently, those tests must not have shared state between them. However, in large applications, it may be common for some tests to depend on some shared state, and other tests to depend on a completely separate state. For example, part of your tests may depend on Cassandra, while others depend on Redis. Prior to Elixir v1.18, these tests could not run concurrently, but in v1.18 they might as long as they are assigned to different groups:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">PGTest</span> <span class="k">do</span> <span class="kn">use</span> <span class="no">ExUnit</span><span class="o">.</span><span class="no">Case</span><span class="p">,</span> <span class="ss">async:</span> <span class="no">true</span><span class="p">,</span> <span class="ss">group:</span> <span class="ss">:pg</span> <span class="c1"># ...</span> <span class="k">end</span> </code></pre></div></div> <p>Tests modules within the same group do not run concurrently, but across groups, they might.</p> <p>With features like async tests, suite partitioning, and now grouping, Elixir developers have plenty of flexibility to make the most use of their machine resources, both in development and in CI.</p> <h2 id="mix-format---migrate"><code class="language-plaintext highlighter-rouge">mix format --migrate</code></h2> <p>The <code class="language-plaintext highlighter-rouge">mix format</code> command now supports an explicit <code class="language-plaintext highlighter-rouge">--migrate</code> flag, which will convert constructs that have been deprecated in Elixir to their latest version. Because this flag rewrites the AST, it is not guaranteed the migrated format will always be valid when used in combination with macros that also perform AST rewriting.</p> <p>As of this release, the following migrations are executed:</p> <ul> <li> <p>Normalize parens in bitstring modifiers - it removes unnecessary parentheses in known bitstring modifiers, for example <code class="language-plaintext highlighter-rouge">&lt;&lt;foo::binary()&gt;&gt;</code> becomes <code class="language-plaintext highlighter-rouge">&lt;&lt;foo::binary&gt;&gt;</code>, or adds parentheses for custom modifiers, where <code class="language-plaintext highlighter-rouge">&lt;&lt;foo::custom_type&gt;&gt;</code> becomes <code class="language-plaintext highlighter-rouge">&lt;&lt;foo::custom_type()&gt;&gt;</code>.</p> </li> <li> <p>Charlists as sigils - formats charlists as <code class="language-plaintext highlighter-rouge">~c</code> sigils, for example <code class="language-plaintext highlighter-rouge">'foo'</code> becomes <code class="language-plaintext highlighter-rouge">~c"foo"</code>.</p> </li> <li> <p><code class="language-plaintext highlighter-rouge">unless</code> as negated <code class="language-plaintext highlighter-rouge">if</code>s - rewrites <code class="language-plaintext highlighter-rouge">unless</code> expressions using <code class="language-plaintext highlighter-rouge">if</code> with a negated condition, for example <code class="language-plaintext highlighter-rouge">unless foo do</code> becomes <code class="language-plaintext highlighter-rouge">if !foo do</code>. We plan to deprecate <code class="language-plaintext highlighter-rouge">unless</code> in future releases.</p> </li> </ul> <p>More migrations will be added in future releases to help us push towards more consistent codebases.</p> <h2 id="summary">Summary</h2> <p>Other notable changes include <a href="https://hexdocs.pm/elixir/1.18/PartitionSupervisor.html#resize!/2"><code class="language-plaintext highlighter-rouge">PartitionSupervisor.resize!/2</code></a>, for resizing the number of partitions (aka processes) of a supervisor at runtime, <a href="https://hexdocs.pm/elixir/1.18/Registry.html#lock/3">Registry.lock/3</a> for simple in-process key locks, PowerShell versions of <code class="language-plaintext highlighter-rouge">elixir</code> and <code class="language-plaintext highlighter-rouge">elixirc</code> scripts for better DX on Windows, and more. <a href="https://hexdocs.pm/elixir/1.18/changelog.html">See the CHANGELOG</a> for the complete release notes.</p> <p>Happy coding!</p> José Valim 2024-08-28T00:00:00+00:00 /blog/2024/08/28/typing-lists-and-tuples <p>We have been working on <a href="https://elixir-lang.org/blog/2023/06/22/type-system-updates-research-dev/">a type system for the Elixir programming language</a>. The type system provides sound gradual typing: it can safely interface static and dynamic code, and if the program type checks, it will not produce type errors at runtime.</p> <p>It is important to emphasize <strong>type errors</strong>. The type systems used at scale today do not guarantee the absense of any runtime errors, but only typing ones. Many programming languages error when accessing the “head” of an empty list, most languages raise on division by zero or when computing the logarithm of negative numbers on a real domain, and others may fail to allocate memory or when a number overflows/underflows.</p> <p>Language designers and maintainers must outline the boundaries of what can be represented as typing errors and how that impacts the design of libraries. The goal of this article is to highlight some of these decisions in the context of lists and tuples in Elixir’s on-going type system work.</p> <blockquote> <p>In this article, the words “raise” and “exceptions” describe something unexpected happened, and not a mechanism for control-flow. Other programming languages may call them “panics” or “faults”.</p> </blockquote> <h2 id="the-head-of-a-list">The <code class="language-plaintext highlighter-rouge">head</code> of a list</h2> <p>Imagine you are designing a programming language and you want to provide a <code class="language-plaintext highlighter-rouge">head</code> function, which returns the head - the first element - of a list, you may consider three options.</p> <p>The first option, the one found in many programming languages, is to raise if an empty list is given. Its implementation in Elixir would be something akin to:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="err">$</span> <span class="n">list</span><span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">a</span> <span class="k">def</span> <span class="n">head</span><span class="p">([</span><span class="n">head</span> <span class="o">|</span> <span class="n">_</span><span class="p">]),</span> <span class="k">do</span><span class="p">:</span> <span class="n">head</span> <span class="k">def</span> <span class="n">head</span><span class="p">([]),</span> <span class="k">do</span><span class="p">:</span> <span class="k">raise</span> <span class="s2">"empty list"</span> </code></pre></div></div> <p>Because the type system cannot differentiate between an empty list and a non-empty list, you won’t find any typing violations at compile-time, but an error is raised at runtime for empty lists.</p> <p>An alternative would be to return an <code class="language-plaintext highlighter-rouge">option</code> type, properly encoding that the function may fail (or not):</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="err">$</span> <span class="n">list</span><span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">option</span><span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="k">def</span> <span class="n">head</span><span class="p">([</span><span class="n">head</span> <span class="o">|</span> <span class="n">_</span><span class="p">]),</span> <span class="k">do</span><span class="p">:</span> <span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">head</span><span class="p">}</span> <span class="k">def</span> <span class="n">head</span><span class="p">([]),</span> <span class="k">do</span><span class="p">:</span> <span class="ss">:none</span> </code></pre></div></div> <p>This approach may be a bit redundant. Returning an <code class="language-plaintext highlighter-rouge">option</code> type basically forces the caller to pattern match on the returned <code class="language-plaintext highlighter-rouge">option</code>. While many programming languages provide functions to compose <code class="language-plaintext highlighter-rouge">option</code> values, one may also get rid of the additional wrapping and directly pattern match on the list instead. So instead of:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">case</span> <span class="n">head</span><span class="p">(</span><span class="n">list</span><span class="p">)</span> <span class="k">do</span> <span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">head</span><span class="p">}</span> <span class="o">-&gt;</span> <span class="c1"># there is a head</span> <span class="ss">:none</span> <span class="o">-&gt;</span> <span class="c1"># do what you need to do</span> <span class="k">end</span> </code></pre></div></div> <p>You could just write:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">case</span> <span class="n">list</span> <span class="k">do</span> <span class="p">[</span><span class="n">head</span> <span class="o">|</span> <span class="n">_</span><span class="p">]</span> <span class="o">-&gt;</span> <span class="c1"># there is a head</span> <span class="p">[]</span> <span class="o">-&gt;</span> <span class="c1"># do what you need to do</span> <span class="k">end</span> </code></pre></div></div> <p>Both examples above are limited by the fact the type system cannot distinguish between empty and non-empty lists and therefore their handling must happen at runtime. If we get rid of this limitations, we could define <code class="language-plaintext highlighter-rouge">head</code> as follows:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="err">$</span> <span class="n">non_empty_list</span><span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">a</span> <span class="k">def</span> <span class="n">head</span><span class="p">([</span><span class="n">head</span> <span class="o">|</span> <span class="n">_</span><span class="p">]),</span> <span class="k">do</span><span class="p">:</span> <span class="n">head</span> </code></pre></div></div> <p>And now we get a typing violation at compile-time if an empty list is given as argument. There is no <code class="language-plaintext highlighter-rouge">option</code> tagging and no runtime exceptions. Win-win?</p> <p>The trouble with the above is that now it is responsibility of the language users to prove the list is not empty. For example, imagine this code:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">list</span> <span class="o">=</span> <span class="n">convert_json_array_to_elixir_list</span><span class="p">(</span><span class="n">json_array_as_string</span><span class="p">)</span> <span class="n">head</span><span class="p">(</span><span class="n">list</span><span class="p">)</span> </code></pre></div></div> <p>In the example above, since <code class="language-plaintext highlighter-rouge">convert_json_array_to_elixir_list</code> may return an empty list, there is a typing violation at compile-time. To resolve it, we need to prove the result of <code class="language-plaintext highlighter-rouge">convert_json_array_to_elixir_list</code> is not an empty list before calling <code class="language-plaintext highlighter-rouge">head</code>:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">list</span> <span class="o">=</span> <span class="n">convert_json_array_to_elixir_list</span><span class="p">(</span><span class="n">json_array_as_string</span><span class="p">)</span> <span class="k">if</span> <span class="n">list</span> <span class="o">==</span> <span class="p">[]</span> <span class="k">do</span> <span class="k">raise</span> <span class="s2">"empty list"</span> <span class="k">end</span> <span class="n">head</span><span class="p">(</span><span class="n">list</span><span class="p">)</span> </code></pre></div></div> <p>But, at this point, we might as well just use pattern matching and once again get rid of <code class="language-plaintext highlighter-rouge">head</code>:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">case</span> <span class="n">convert_json_array_to_elixir_list</span><span class="p">(</span><span class="n">json_array_as_string</span><span class="p">)</span> <span class="k">do</span> <span class="p">[</span><span class="n">head</span> <span class="o">|</span> <span class="n">_</span><span class="p">]</span> <span class="o">-&gt;</span> <span class="c1"># there is a head</span> <span class="p">[]</span> <span class="o">-&gt;</span> <span class="c1"># do what you need to do</span> <span class="k">end</span> </code></pre></div></div> <p>Most people would expect that encoding more information into the type system would bring only benefits but there is a tension here: the more you encode into types, the more you might have to prove in your programs.</p> <p>While different developers will prefer certain idioms over others, I am not convinced there is one clearly superior approach here. Having <code class="language-plaintext highlighter-rouge">head</code> raise a runtime error may be the most pragmatic approach <em>if</em> the developer expects the list to be non-empty in the first place. Returning <code class="language-plaintext highlighter-rouge">option</code> gets rid of the exception by forcing users to explicitly handle the result, but leads to more boilerplate compared to pattern matching, especially if the user does not expect empty lists. And, finally, adding precise types means there could be more for developers to prove.</p> <h3 id="what-about-elixir">What about Elixir?</h3> <p>Thanks to set-theoretic types, we will most likely distinguish between empty lists and non-empty lists in Elixir’s type system, since pattern matching on them is a common language idiom. Furthermore, several functions in Elixir, such as <code class="language-plaintext highlighter-rouge">String.split/2</code> are guaranteed to return non-empty lists, which can then be nicely encoded into a function’s return type.</p> <p>Elixir also has the functions <code class="language-plaintext highlighter-rouge">hd</code> (for head) and <code class="language-plaintext highlighter-rouge">tl</code> (for tail) inherited from Erlang, which are <a href="https://hexdocs.pm/elixir/patterns-and-guards.html">valid guards</a>. They only accept non-empty lists as arguments, which will now be enforced by the type system too.</p> <p>This covers almost all use cases but one: what happens if you want to access the first element of a list, which has not been proven to be empty? You could use pattern matching and conditionals for those cases, but as seen above, this can lead to common boilerplate such as:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">if</span> <span class="n">list</span> <span class="o">==</span> <span class="p">[]</span> <span class="k">do</span> <span class="k">raise</span> <span class="s2">"unexpected empty list"</span> <span class="k">end</span> </code></pre></div></div> <p>Luckily, it is common in Elixir to use the <code class="language-plaintext highlighter-rouge">!</code> suffix to encode the possibility of runtime errors for <em>valid</em> inputs. For these circumstances, we may introduce <code class="language-plaintext highlighter-rouge">List.first!</code> (and potentially <code class="language-plaintext highlighter-rouge">List.drop_first!</code> for the tail variant).</p> <h2 id="accessing-tuples">Accessing tuples</h2> <p>Now that we have discussed lists, we can talk about tuples. In a way, tuples are more challenging than lists for two reasons:</p> <ol> <li> <p>A list is a collection where all elements have the same type (be it a <code class="language-plaintext highlighter-rouge">list(integer())</code> or <code class="language-plaintext highlighter-rouge">list(integer() or float())</code>), while tuples carry the types of each element</p> </li> <li> <p>We natively access tuples by index, instead of its head and tail, such <code class="language-plaintext highlighter-rouge">elem(tuple, 0)</code></p> </li> </ol> <p>In the upcoming v1.18 release, Elixir’s new type system will support tuple types, and they are written between curly brackets. For example, the <a href="https://hexdocs.pm/elixir/File.html#read/1"><code class="language-plaintext highlighter-rouge">File.read/1</code> function</a> would have the return type <code class="language-plaintext highlighter-rouge">{:ok, binary()} or {:error, posix()}</code>, quite similar to today’s typespecs.</p> <p>The tuple type can also specify a minimum size, as you can also write: <code class="language-plaintext highlighter-rouge">{atom(), integer(), ...} </code>. This means the tuple has at least two elements, the first being an <code class="language-plaintext highlighter-rouge">atom()</code> and the second being an <code class="language-plaintext highlighter-rouge">integer()</code>. This definition is required for type inference in patterns and guards. After all, a guard <code class="language-plaintext highlighter-rouge">is_integer(elem(tuple, 1))</code> tells you the tuple has at least two elements, with the second one being an integer, but nothing about the other elements and the tuple overall size.</p> <p>With tuples support merged into main, we need to answer questions such as which kind of compile-time warnings and runtime exceptions tuple operations, such as <code class="language-plaintext highlighter-rouge">elem(tuple, index)</code> may emit. Today, we know that it raises if:</p> <ol> <li> <p>the index is out of bounds, as in <code class="language-plaintext highlighter-rouge">elem({:ok, "hello"}, 3)</code></p> </li> <li> <p>the index is negative, as in <code class="language-plaintext highlighter-rouge">elem({:ok, 123}, -1)</code></p> </li> </ol> <p>When typing <code class="language-plaintext highlighter-rouge">elem(tuple, index)</code>, one option is to use “avoid all runtime errors” as our guiding light and make <code class="language-plaintext highlighter-rouge">elem</code> return <code class="language-plaintext highlighter-rouge">option</code> types, such as: <code class="language-plaintext highlighter-rouge">{:ok, value}</code> or <code class="language-plaintext highlighter-rouge">:none</code>. This makes sense for an out of bounds error, but should it also return <code class="language-plaintext highlighter-rouge">:none</code> if the index is negative? One could argue that they are both out of bounds. On the other hand, a positive index may be correct depending on the tuple size but <strong>a negative index is always invalid</strong>. From this perspective, encoding an always invalid value as an <code class="language-plaintext highlighter-rouge">:none</code> can be detrimental to the developer experience, hiding logical bugs instead of (loudly) blowing up.</p> <p>Another option is to make these programs invalid. If we completely remove <code class="language-plaintext highlighter-rouge">elem/2</code> from the language and you can only access tuples via pattern matching (or by adding a literal notation such as <code class="language-plaintext highlighter-rouge">tuple.0</code>), then all possible bugs can be caught by the type checker. However, some data structures, such as <a href="https://www.erlang.org/doc/apps/stdlib/array.html">array in Erlang</a> rely on dynamic tuple access, and implementing those would be no longer possible.</p> <p>Yet another option is to encode integers themselves as values in the type system. In the same way that Elixir’s type system supports the values <code class="language-plaintext highlighter-rouge">:ok</code> and <code class="language-plaintext highlighter-rouge">:error</code> as types, we could support each integer, such as <code class="language-plaintext highlighter-rouge">13</code> and <code class="language-plaintext highlighter-rouge">-42</code> as types as well (or specific subsets, such as <code class="language-plaintext highlighter-rouge">neg_integer()</code>, <code class="language-plaintext highlighter-rouge">zero()</code> and <code class="language-plaintext highlighter-rouge">pos_integer()</code>). This way, the type system would know the possible values of <code class="language-plaintext highlighter-rouge">index</code> during type checking, allowing us to pass complex expressions to <code class="language-plaintext highlighter-rouge">elem(tuple, index)</code>, and emit typing errors if the indexes are invalid. However, remember that encoding more information into types may force developers to also prove that those indexes are within bounds in many other cases.</p> <p>Once again, there are different trade-offs, and we must select one that best fit into Elixir use and semantics today.</p> <h3 id="what-about-elixir-1">What about Elixir?</h3> <p>The approach we are taking in Elixir is two-fold:</p> <ul> <li> <p>If the index is a literal integer, it will perform an exact access on the tuple element. This means <code class="language-plaintext highlighter-rouge">elem(tuple, 1)</code> will work if we can prove the tuple has at least size 2, otherwise you will have a type error</p> </li> <li> <p>If the index is not a literal integer, the function will fallback to a dynamic type signature</p> </li> </ul> <p>Let’s expand on the second point.</p> <p>At a fundamental level, we could describe <code class="language-plaintext highlighter-rouge">elem</code> with the type signature of <code class="language-plaintext highlighter-rouge">tuple(a), integer() -&gt; a</code>. However, the trouble with this signature is that it does not tell the type system (nor users) the possibility of a runtime error. Luckily, because Elixir will offer a gradual type system, we could encode the type signature as <code class="language-plaintext highlighter-rouge">dynamic({...a}), integer() -&gt; dynamic(a)</code>. By encoding the argument and return type as dynamic, developers who want a fully static program will be notified of a typing error, while existing developers who rely on dynamic features of the language can continue to do so, and those choices are now encoded into the types.</p> <p>Overall,</p> <ul> <li> <p>For static programs (the ones that do not use the <code class="language-plaintext highlighter-rouge">dynamic()</code> type), <code class="language-plaintext highlighter-rouge">elem/2</code> will validate that the first argument is a tuple of known shape, and the second argument is a literal integer which is greater than or equal to zero and less than the tuple size. This guarantees no runtime exceptions.</p> </li> <li> <p>Gradual programs will have the same semantics (and runtime exceptions) as today.</p> </li> </ul> <h2 id="summary">Summary</h2> <p>I hope this article outlines some of the design decisions as we bring a gradual type system to Elixir. Although supporting tuples and lists is a “table stakes” feature in most type systems, bringing them to Elixir is an opportunity to understand how the type system will interact with several language idioms, as well as provide a foundation for future decisions. The most important take aways are:</p> <ol> <li> <p>Type safety is a commitment from both sides. If you want your type system to find even more bugs through more precise types, you will need to prove more frequently that your programs are free of certain typing violations.</p> </li> <li> <p>It is not a goal of the type system to avoid all runtime errors. This would require either a type system that is too precise (and require more proofs) or it would require functions to mask hard errors (such as a negative index) as error values. Exceptions still play an important role in typed Elixir and, given they are modelled as structs, they will also be typed in the future.</p> </li> <li> <p>Elixir’s convention of using the suffix <code class="language-plaintext highlighter-rouge">!</code> to provide variants that encode the possibility of runtime exceptions for a valid domain (the input types) nicely complements the type system, as it can help static programs avoid the boilerplate of converting <code class="language-plaintext highlighter-rouge">:none</code>/<code class="language-plaintext highlighter-rouge">:error</code> into exceptions for unexpected scenarios.</p> </li> <li> <p>Using <code class="language-plaintext highlighter-rouge">dynamic()</code> in function signatures is a mechanism available in Elixir’s type system to signal that a function has dynamic behaviour and may raise runtime errors, allowing violations to be reported on programs that wish to remain fully static. Similar to how other static languages provide dynamic behaviour via <code class="language-plaintext highlighter-rouge">Any</code> or <code class="language-plaintext highlighter-rouge">Dynamic</code> types.</p> </li> </ol> <p>The type system was made possible thanks to a partnership between <a href="https://www.cnrs.fr/">CNRS</a> and <a href="https://remote.com/">Remote</a>. The development work is currently sponsored by <a href="https://www.fresha.com/">Fresha</a> (<a href="https://www.fresha.com/careers/openings?department=engineering">they are hiring!</a>), <a href="https://starfish.team/">Starfish*</a>, and <a href="https://dashbit.co/">Dashbit</a>.</p> <p>Happy typing!</p> José Valim 2024-08-15T00:00:00+00:00 /blog/2024/08/15/welcome-elixir-language-server-team <blockquote> <p>Update: <a href="https://github.com/elixir-lang/expert">the new language server is called Expert and is now available as open source</a>.</p> </blockquote> <p>I am glad to welcome Elixir’s official Language Server team, formed by (in alphabetical order):</p> <ul> <li><a href="https://github.com/jonatanklosko">Jonatan Kłosko</a></li> <li><a href="https://github.com/lukaszsamson">Łukasz Samson</a></li> <li><a href="https://www.mitchellhanberg.com/">Mitch Hanberg</a></li> <li><a href="https://github.com/scohen">Steve Cohen</a></li> </ul> <p>The team will work on the code intelligence infrastructure to be used across tools and editors. These efforts are partially funded by <a href="https://fly.io/">Fly.io</a> and <a href="https://livebook.dev/">Livebook</a>.</p> <h2 id="a-brief-history">A brief history</h2> <p>The <a href="https://en.wikipedia.org/wiki/Language_Server_Protocol">Language Server Protocol (LSP)</a> was created by Microsoft as a protocol between IDEs and programming languages to provide language intelligence tools.</p> <p>The first implementation of said protocol for Elixir was started by <a href="https://github.com/JakeBecker/elixir-ls/">Jake Becker</a>, back in 2017, alongside an implementation for Visual Studio Code, and it relied on <a href="https://github.com/msaraiva/elixir_sense">the ElixirSense project from Marlus Saraiva</a> to extract and provide some of the language intelligence.</p> <p>As the Language Server Protocol adoption grew as a whole, so did the usage of Elixir’s implementation, which eventually became the main mechanism Elixir users interacted with the language from their editors.</p> <p>Eventually, Elixir’s language server implementation got its <a href="https://github.com/elixir-lsp/">own organization on GitHub</a>, and maintenance reins were given to Łukasz Samson and Jason Axelson.</p> <p>Over time, the Elixir Language Server has accrued technical debt. Some of it exists due to intrinsic complexities (for example, the Language Server Protocol uses UTF-16 for text encoding, instead of the more widely used UTF-8), while others are a consequence of working on codebase while both the programming language and the protocol themselves were still evolving.</p> <p>This led to Mitch Hanberg and Steve Cohen to create alternative language server implementations, exploring different trade-offs.</p> <p>For example, both <a href="https://github.com/elixir-tools/next-ls">Next LS</a> and <a href="https://github.com/lexical-lsp/lexical">Lexical</a> use Erlang Distribution to isolate the Language Server runtime from the user code.</p> <p>Next LS also focused on extracting the LSP protocol parts into <a href="https://github.com/elixir-tools/gen_lsp">GenLSP</a> (which can be used by anyone to easily create a language server), single binary distribution with <a href="https://github.com/burrito-elixir/burrito">Burrito</a>, and experimenting with SQLite for the symbol index.</p> <p><a href="https://github.com/lexical-lsp/lexical">Lexical</a> concerned itself with speed and abstractions to deal with documents, ranges, and more.</p> <p>This means the Elixir community had, for some time, three distinct language server implementations, each with their own strengths.</p> <h2 id="looking-forward">Looking forward</h2> <p>The current language server maintainers have agreed to move forward with a <em>single Language Server Protocol project</em>, relying on the strengths of each implementation:</p> <ul> <li>Lexical provides a stable foundation</li> <li>ElixirLS, through ElixirSense, provides the most complete implementation and wider functionality</li> <li>Next LS, through GenLSP, provides a general interface for LSP implementations and straight-forward packaging via <a href="https://github.com/burrito-elixir/burrito">Burrito</a></li> </ul> <p>The above is a rough outline, as the specific details of how the projects will move forward are still being discussed. While some of the team members also maintain direct integration with some editors, we will continue relying on the community’s help and efforts to get full coverage across all available editors.</p> <p>And there is still a lot more to do!</p> <p>Many underestimate the complexity behind implementing the Language Server Protocol. That’s not surprising: we mostly interact with it from an editor, allowing us to freely ignore what makes it tick.</p> <p>In practice, the Language Server needs, in many ways, to reimplement several parts of the language and its compiler.</p> <p>If the Elixir compiler sees the code <code class="language-plaintext highlighter-rouge">some_value +</code>, it can immediately warn and say: “this expression is incomplete”. However, the Language Server still needs to make sense of invalid code to provide features like completion. And that applies to everything: missing do-end blocks, invalid operators, invoking macros that do not exist, etc. Mitch has made <a href="https://github.com/elixir-tools/spitfire">Spitfire</a>, an error tolerant parser to tackle this particular problem.</p> <p>Some ecosystems have undertaken <a href="https://en.wikipedia.org/wiki/Roslyn_(compiler)">multi-year efforts to redesign their compilers and toolchains</a> to provide better tools for lexical and semantic code analysis (which most likely took a significant investment of time and resources to conclude). That’s to say some of the problems faced by Language Server implementations will be best tackled if they are also solved as part of Elixir itself.</p> <p>For example, every Language Server implementation compiles their own version of a project, making it so every application and its dependencies have to be compiled twice in development: once for Mix and once for the Language Server. Wouldn’t it be nice if Elixir and the Language Servers could all rely on the same compilation artifacts?</p> <p>This is not news to the Elixir team either: almost every Elixir release within the last 3 years has shipped new code analysis APIs, such as <a href="https://hexdocs.pm/elixir/Code.Fragment.html">Code.Fragment</a>, with the goal of removing duplication across Language Servers, <a href="https://hexdocs.pm/iex">IEx</a>, and <a href="https://livebook.dev/">Livebook</a>, as well as reduce their reliance on internal Elixir modules. Most recently, Elixir v1.17 shipped with <a href="https://hexdocs.pm/elixir/Macro.Env.html">new APIs to help developers emulate the compiler behaviour</a>. Our goal is to make these building blocks available for all Elixir developers, so their benefits are reaped beyond the language server tooling.</p> <p>Furthermore, as <a href="https://elixir-lang.org/blog/2024/06/12/elixir-v1-17-0-released/">set-theoretic types make their way into Elixir</a>, we also want to provide official APIs to integrate them into our tools.</p> <h2 id="sponsorships">Sponsorships</h2> <p>Currently, <a href="https://fly.io/">Fly.io</a> is sponsoring Łukasz Samson to work part-time on the Language Server and editor integration. The <a href="https://livebook.dev/">Livebook</a> project is donating development time from Jonatan Kłosko, creator of Livebook, to improve the Elixir compiler and its code intelligence APIs.</p> <p>We are grateful to both companies for investing into the community and you should check them out.</p> <p>As mentioned above, Language Server implementations are complex projects, and unifying efforts is an important step in the right direction. However, we also need community help, and one of the ways to do so is by sponsoring the developers making this possible:</p> <ul> <li><a href="https://github.com/sponsors/lukaszsamson">Łukasz Samson</a></li> <li><a href="https://github.com/sponsors/mhanberg">Mitch Hanberg</a></li> <li><a href="https://github.com/sponsors/scohen">Steve Cohen</a></li> </ul> <p>Companies who can afford to sponsor part-time development are welcome to reach out and help us achieve this important milestone.</p> <h2 id="progress-updates">Progress updates</h2> <p>A new project website and social media accounts will be created soon, and you can follow them to stay up to date with our progress and any interesting developments.</p> <p>The name of the new project is still in the works as well as many of the decisions we’ll need to make, so please have patience!</p> <p>In the meantime, you can continue to use the language server of your choice, and we’ll be sure to make the transition to the fourth and final project as smooth as possible.</p> <p>Thank you!</p> Andrea Leopardi 2024-06-12T00:00:00+00:00 /blog/2024/06/12/elixir-v1-17-0-released <p>Elixir v1.17 has just been released. 🎉</p> <p>This release introduces set-theoretic types into a handful of language constructs. While there are still <a href="https://elixir-lang.org/blog/2023/06/22/type-system-updates-research-dev/">many steps ahead of us</a>, this important milestone already brings benefits to developers in the form of new warnings for common mistakes. This new version also adds support for <a href="https://www.erlang.org/downloads/27">Erlang/OTP 27</a>, the latest and greatest Erlang release. You’ll also find a new calendar-related data type (<code class="language-plaintext highlighter-rouge">Duration</code>) and a <code class="language-plaintext highlighter-rouge">Date.shift/2</code> function.</p> <p>Let’s dive in.</p> <h2 id="warnings-from-gradual-set-theoretic-types">Warnings from gradual set-theoretic types</h2> <p>This release introduces gradual set-theoretic types to infer types from patterns and use them to type check programs, enabling the Elixir compiler to find faults and bugs in codebases without requiring changes to existing software. The underlying principles, theory, and roadmap of our work have been outlined in <a href="https://arxiv.org/abs/2306.06391">“The Design Principles of the Elixir Type System” by Giuseppe Castagna, Guillaume Duboc, José Valim</a>.</p> <p>At the moment, Elixir developers will interact with set-theoretic types only through <strong>warnings</strong> found by the type system. The current implementation models all data types in the language:</p> <ul> <li> <p><code class="language-plaintext highlighter-rouge">binary()</code>, <code class="language-plaintext highlighter-rouge">integer()</code>, <code class="language-plaintext highlighter-rouge">float()</code>, <code class="language-plaintext highlighter-rouge">pid()</code>, <code class="language-plaintext highlighter-rouge">port()</code>, <code class="language-plaintext highlighter-rouge">reference()</code> - these types are indivisible. This means both <code class="language-plaintext highlighter-rouge">1</code> and <code class="language-plaintext highlighter-rouge">13</code> get the same <code class="language-plaintext highlighter-rouge">integer()</code> type.</p> </li> <li> <p><code class="language-plaintext highlighter-rouge">atom()</code> - it represents all atoms and it is divisible. For instance, the atom <code class="language-plaintext highlighter-rouge">:foo</code> and <code class="language-plaintext highlighter-rouge">:hello_world</code> are also valid (distinct) types.</p> </li> <li> <p><code class="language-plaintext highlighter-rouge">map()</code> and structs - maps can be “closed” or “open”. Closed maps only allow the specified keys, such as <code class="language-plaintext highlighter-rouge">%{key: atom(), value: integer()}</code>. Open maps support any other keys in addition to the ones listed and their definition starts with <code class="language-plaintext highlighter-rouge">...</code>, such as <code class="language-plaintext highlighter-rouge">%{..., key: atom(), value: integer()}</code>. Structs are closed maps with the <code class="language-plaintext highlighter-rouge">__struct__</code> key.</p> </li> <li> <p><code class="language-plaintext highlighter-rouge">tuple()</code>, <code class="language-plaintext highlighter-rouge">list()</code>, and <code class="language-plaintext highlighter-rouge">function()</code> - currently they are modelled as indivisible types. The next Elixir versions will also introduce fine-grained support to them.</p> </li> </ul> <p>We focused on <em>atoms</em> and <em>maps</em> on this initial release as they are respectively the simplest and the most complex types representations, so we can stress the performance of the type system and quality of error messages. Modelling these types will also provide the most immediate benefits to Elixir developers. Assuming there is a variable named <code class="language-plaintext highlighter-rouge">user</code>, holding a <code class="language-plaintext highlighter-rouge">%User{}</code> struct with a <code class="language-plaintext highlighter-rouge">address</code> field, Elixir v1.17 will emit the following warnings at compile-time:</p> <ul> <li> <p>Pattern matching against a map or a struct that does not have the given key, such as <code class="language-plaintext highlighter-rouge">%{adress: ...} = user</code> (notice <code class="language-plaintext highlighter-rouge">address</code> vs <code class="language-plaintext highlighter-rouge">adress</code>).</p> </li> <li> <p>Accessing a key on a map or a struct that does not have the given key, such as <code class="language-plaintext highlighter-rouge">user.adress</code>.</p> </li> <li> <p>Invoking a function on non-modules, such as <code class="language-plaintext highlighter-rouge">user.address()</code>.</p> </li> <li> <p>Capturing a function on non-modules, such as <code class="language-plaintext highlighter-rouge">&amp;user.address/0</code>.</p> </li> <li> <p>Attempting to call an anonymous function without an actual function, such as <code class="language-plaintext highlighter-rouge">user.()</code>.</p> </li> <li> <p>Performing structural comparisons between structs, such as <code class="language-plaintext highlighter-rouge">my_date &lt; ~D[2010-04-17]</code>.</p> </li> <li> <p>Performing structural comparisons between non-overlapping types, such as <code class="language-plaintext highlighter-rouge">integer &gt;= string</code>.</p> </li> <li> <p>Building and pattern matching on binaries without the relevant specifiers, such as <code class="language-plaintext highlighter-rouge">&lt;&lt;name&gt;&gt;</code> (this warns because by default it expects an integer, it should have been <code class="language-plaintext highlighter-rouge">&lt;&lt;name::binary&gt;&gt;</code> instead).</p> </li> <li> <p>Attempting to rescue an undefined exception or a struct that is not an exception.</p> </li> <li> <p>Accessing a field that is not defined in a rescued exception.</p> </li> </ul> <p>Here’s an example of how the warning for accessing a misspelled field of a struct looks like:</p> <p><img src="/images/contents/type-warning-on-struct-field.png" alt="Example of a warning when accessing a mispelled struct field" /></p> <p>Another example, this time it’s a warning for structural comparison across two <code class="language-plaintext highlighter-rouge">Date</code> structs:</p> <p><img src="/images/contents/type-warning-on-date-comparison.png" alt="Example of a warning when comparing two structs with &quot;&gt;&quot;" /></p> <p>These warnings also work natively in text editors, as they are standard Elixir compiler warnings:</p> <p><img src="/images/contents/type-warning-in-editor.png" alt="Example of a type warning inline in an editor" /></p> <p>These new warnings will help Elixir developers find bugs earlier and give more confidence when refactoring code, especially around maps and structs. While Elixir already emitted some of these warnings in the past, those were discovered using syntax analysis. The new warnings are more reliable, precise, and with better error messages. Keep in mind, however, that the Elixir typechecker only infers types from patterns within the same function at the moment. Analysis from guards and across function boundaries will be added in future releases. For more details, see our new <a href="https://hexdocs.pm/elixir/gradual-set-theoretic-types.html">reference document on gradual set-theoretic types</a>.</p> <p>The type system was made possible thanks to a partnership between <a href="https://www.cnrs.fr/">CNRS</a> and <a href="https://remote.com/">Remote</a>. The development work is currently sponsored by <a href="https://www.fresha.com/">Fresha</a> (<a href="https://www.fresha.com/careers/openings?department=engineering">they are hiring!</a>), <a href="https://starfish.team/">Starfish*</a>, and <a href="https://dashbit.co/">Dashbit</a>.</p> <h2 id="erlangotp-support">Erlang/OTP support</h2> <p>This release adds support for Erlang/OTP 27 and drops support for Erlang/OTP 24. We recommend Elixir developers to migrate to Erlang/OTP 26 or later, especially on Windows. Support for WERL (a graphical user interface for the Erlang terminal on Windows) will be removed in Elixir v1.18.</p> <p>You can read more about Erlang/OTP 27 in <a href="https://www.erlang.org/downloads/27">their release announcement</a>. The bits that are particularly interesting for Elixir developers are the addition of a <a href="https://erlang.org/documentation/doc-15.0-rc3/lib/stdlib-6.0/doc/html/json.html"><code class="language-plaintext highlighter-rouge">json</code> module</a> and process labels (<code class="language-plaintext highlighter-rouge">proc_lib:set_label/1</code>). The latter will also be available in this Elixir release as <code class="language-plaintext highlighter-rouge">Process.set_label/1</code>.</p> <h2 id="new-duration-data-type-and-shifting-functions">New <code class="language-plaintext highlighter-rouge">Duration</code> data type and shifting functions</h2> <p>This Elixir version introduces the <code class="language-plaintext highlighter-rouge">Duration</code> data type and APIs to shift dates, times, and date times by a given duration, considering different calendars and time zones.</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">iex</span><span class="o">&gt;</span> <span class="no">Date</span><span class="o">.</span><span class="n">shift</span><span class="p">(</span><span class="sx">~D[2016-01-31]</span><span class="p">,</span> <span class="ss">month:</span> <span class="mi">2</span><span class="p">)</span> <span class="sx">~D[2016-03-31]</span> </code></pre></div></div> <p>We chose the name <em>“shift”</em> for this operation (instead of “add”) since working with durations does not obey properties such as <strong>associativity</strong>. For instance, adding one month and then one month does not give the same result as adding two months:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">iex</span><span class="o">&gt;</span> <span class="sx">~D[2016-01-31]</span> <span class="o">|&gt;</span> <span class="no">Date</span><span class="o">.</span><span class="n">shift</span><span class="p">(</span><span class="ss">month:</span> <span class="mi">1</span><span class="p">)</span> <span class="o">|&gt;</span> <span class="no">Date</span><span class="o">.</span><span class="n">shift</span><span class="p">(</span><span class="ss">month:</span> <span class="mi">1</span><span class="p">)</span> <span class="sx">~D[2016-03-29]</span> </code></pre></div></div> <p>Still, durations are essential for building intervals, recurring events, and modelling scheduling complexities found in the world around us. For <code class="language-plaintext highlighter-rouge">DateTime</code>s, Elixir will correctly deal with time zone changes (such as Daylight Saving Time). However, provisions are also available in case you want to surface conflicts, such as shifting to a wall clock that does not exist, because the clock has been moved forward by one hour. See <code class="language-plaintext highlighter-rouge">DateTime.shift/2</code> for examples.</p> <p>Finally, we added a new <code class="language-plaintext highlighter-rouge">Kernel.to_timeout/1</code> function, which helps developers normalize durations and integers to a timeout used by many APIs—like <code class="language-plaintext highlighter-rouge">Process</code>, <code class="language-plaintext highlighter-rouge">GenServer</code>, and more. For example, to send a message after one hour, you can now write:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="no">Process</span><span class="o">.</span><span class="n">send_after</span><span class="p">(</span><span class="n">pid</span><span class="p">,</span> <span class="ss">:wake_up</span><span class="p">,</span> <span class="n">to_timeout</span><span class="p">(</span><span class="ss">hour:</span> <span class="mi">1</span><span class="p">))</span> </code></pre></div></div> <h2 id="learn-more">Learn more</h2> <p>Here are other notable changes in this release:</p> <ul> <li> <p>There are new <code class="language-plaintext highlighter-rouge">Keyword.intersect/2,3</code> functions to mirror the equivalent in the <code class="language-plaintext highlighter-rouge">Map</code> module.</p> </li> <li> <p>A new Mix profiler was added, <code class="language-plaintext highlighter-rouge">mix profile.tprof</code>, which lets you use the new <a href="https://www.erlang.org/doc/apps/tools/tprof.html">tprof</a> profiler released with Erlang/OTP 27. This profiler leads to the soft-deprecation of <code class="language-plaintext highlighter-rouge">mix profile.cprof</code> and <code class="language-plaintext highlighter-rouge">mix profile.eprof</code>.</p> </li> <li> <p>We added <code class="language-plaintext highlighter-rouge">Kernel.is_non_struct_map/1</code>, a new guard to help with the common pitfall of matching on <code class="language-plaintext highlighter-rouge">%{}</code>, which also successfully matches structs (as they are maps underneath).</p> </li> <li> <p>Elixir’s Logger now formats <a href="https://www.erlang.org/doc/apps/stdlib/gen_statem.html"><code class="language-plaintext highlighter-rouge">gen_statem</code></a> reports and includes Erlang/OTP 27 <em>process labels</em> in logger events.</p> </li> </ul> <p>For a complete list of all changes, see the <a href="https://github.com/elixir-lang/elixir/releases/tag/v1.17.0">full release notes</a>.</p> <p>Check <a href="/install.html">the Install section</a> to get Elixir installed and read our <a href="https://hexdocs.pm/elixir/introduction.html">Getting Started guide</a> to learn more.</p> <p>Happy learning!</p> Hugo Baraúna 2024-03-05T00:00:00+00:00 /blog/2024/03/05/veeps-elixir-case <p><em>Welcome to our series of <a href="/cases.html">case studies about companies using Elixir in production</a>.</em></p> <p><a href="https://veeps.com/">Veeps</a> is a streaming service that offers direct access to live and on-demand events by award-winning artists at the most iconic venues. Founded in 2018, it became part of Live Nation Entertainment in 2021.</p> <p>Veeps have been named <a href="https://www.fastcompany.com/90848907/most-innovative-companies-music-2023">one of the ten most innovative companies in music</a> and nominated for an Emmy. They currently hold the <a href="https://www.guinnessworldrecords.com/world-records/650975-most-tickets-sold-for-a-livestreamed-concert-by-a-solo-male-artist-current-year">Guinness World Record</a> for the world’s largest ticketed livestream by a solo male artist—a performance where Elixir and Phoenix played an important role in the backend during the streaming.</p> <p>This case study examines how Elixir drove Veeps’ technical transformation, surpassing high-scale demands while keeping the development team engaged and productive.</p> <h2 id="the-challenge-scaling-to-hundreds-of-thousands-of-simultaneous-users">The challenge: scaling to hundreds of thousands of simultaneous users</h2> <p>Imagine you are tasked with building a system that can livestream a music concert to hundreds of thousands of viewers around the world at the same time.</p> <p>In some cases, users must purchase a ticket before the concert can be accessed. For a famous artist, it’s not uncommon to see thousands of fans continuously refreshing their browsers and attempting to buy tickets within the first few minutes of the announcement.</p> <p>The Veeps engineering team needed to handle both challenges.</p> <p>Early on, the Veeps backend was implemented in <a href="https://rubyonrails.org/">Ruby on Rails</a>. Its first version could handle a few thousand simultaneous users watching a concert without any impact to stream quality, which was fine when you have a handful of shows but would be insufficient with the expected show load and massive increase in concurrent viewership across streams. It was around that time that <a href="https://twitter.com/vinniefranco">Vincent Franco</a> joined Veeps as their CTO.</p> <p>Vincent had an extensive background in building and maintaining ticketing and event management software at scale. So, he used that experience to further improve the system to handle tens of thousands of concurrent users. However, it became clear that improving it to <em>hundreds</em> of thousands would be a difficult challenge, requiring substantial engineering efforts and increased operational costs. The team began evaluating other stacks that could provide the out-of-the-box tooling for scaling in order to reach both short and long-term goals.</p> <h2 id="adopting-elixir-hiring-and-rewriting-the-system">Adopting Elixir, hiring, and rewriting the system</h2> <p>Vincent, who had successfully deployed Elixir as part of high-volume systems in the past, believed Elixir was an excellent fit for Veeps’ requirements.</p> <p>Backed by his experience and several case studies from the Elixir community, such as <a href="/blog/2020/10/08/real-time-communication-at-scale-with-elixir-at-discord/">the one from Discord</a>, Vincent convinced management that Elixir could address their immediate scaling needs and become a reliable foundation on which the company could build.</p> <p>With buy-in from management, the plan was set in motion. They had two outstanding goals:</p> <ul> <li>Prepare the platform to welcome the most famous artists in the world.</li> <li>Build their own team of engineers to help innovate and evolve the product.</li> </ul> <p>Vincent knew that hiring right-fit technical people can take time and he didn’t want to rush the process. Hence, he hired <a href="https://dockyard.com/">DockYard</a> to rebuild the system while simultaneously searching for the right candidates to build out the team.</p> <p>Eight months later, the system had been entirely rewritten in Elixir and Phoenix. Phoenix Channels were used to enrich the live concert experience, while Phoenix LiveView empowered the ticket shopping journey.</p> <p>The rewrite was put to the test shortly after with a livestream that remains one of Veeps’ biggest, still to this day. Before the rewrite, 20 Rails nodes were used during big events, whereas now, the same service requires only 2 Elixir nodes. And the new platform was able to handle 83x more concurrent users than the previous system.</p> <p>The increase in infrastructure efficiency significantly reduced the need for complex auto-scaling solutions while providing ample capacity to handle high-traffic spikes.</p> <blockquote> <p>The rewrite marked the most extensive and intricate system migration in my career, and yet, it was also the smoothest.</p> <p>- Vincent Franco, CTO</p> </blockquote> <p>This was a big testament to Elixir and Phoenix’s scalability and gave the team confidence that they made the right choice.</p> <p>By the time the migration was completed, Veeps had also assembled an incredibly talented team of two backend and two frontend engineers, which continued to expand and grow the product.</p> <h2 id="perceived-benefits-of-using-elixir-and-its-ecosystem">Perceived benefits of using Elixir and its ecosystem</h2> <p>After using Elixir for more than two years, Veeps has experienced significant benefits. Here are a few of them.</p> <h3 id="architectural-simplicity">Architectural simplicity</h3> <p>Different parts of the Veeps system have different scalability requirements. For instance, when streaming a show, the backend receives metadata from users’ devices every 30 seconds to track viewership. This is the so-called <em>Beaconing service</em>.</p> <p>Say you have 250,000 people watching a concert: the Beaconing service needs to handle thousands of requests per second for a few hours at a time. As a result, it needs to scale differently from other parts of the system, such as the merchandise e-commerce or backstage management.</p> <p>To tackle this issue, they built a distributed system. They packaged each subsystem as an <a href="https://hexdocs.pm/elixir/config-and-releases.html#releases">Elixir release</a>, totaling five releases. For the communication layer, they used distributed Erlang, which is built into Erlang/OTP, allowing seamless inter-process communication across networked nodes.</p> <p>In a nutshell, each node contains several processes with specific responsibilities. Each of these processes belongs to their respective <a href="https://www.erlang.org/doc/man/pg.html">distributed process group</a>. If node A needs billing information, it will reach out to any process within the “billing process group”, which may be anywhere in the cluster.</p> <p>When deploying a new version of the system, they deploy a new cluster altogether, with all five subsystems at once. Given Elixir’s scalability, the whole system uses 9 nodes, making a simple deployment strategy affordable and practical. As we will see, this approach is well-supported during development too, thanks to the use of Umbrella Projects.</p> <h3 id="service-oriented-architecture-within-a-monorepo">Service-oriented architecture within a monorepo</h3> <p>Although they run a distributed system, they organize the code in only one repository, following the monorepo approach. To do that, they use the <a href="https://hexdocs.pm/elixir/dependencies-and-umbrella-projects.html#content">Umbrella Project feature</a> from Mix, the build tool that ships with Elixir.</p> <p>Their umbrella project consists of 16 applications (at the time of writing), which they <a href="https://hexdocs.pm/mix/Mix.Tasks.Release.html#module-umbrellas">sliced into five OTP releases</a>. The remaining applications contain code that needs to be shared between multiple applications. For example, one of the shared applications defines all the structs sent as messages across the subsystems, guaranteeing that all subsystems use the same schemas for that exchanged data.</p> <blockquote> <p>With umbrella projects, you can have the developer experience benefits of a single code repository, while being able to build a service-oriented architecture.</p> <p>- Andrea Leopardi, Principal Engineer</p> </blockquote> <h3 id="reducing-complexity-with-the-erlangelixir-toolbox">Reducing complexity with the Erlang/Elixir toolbox</h3> <p>Veeps has an e-commerce platform that allows concert viewers to purchase artist merchandise. In e-commerce, a common concept is a shopping cart. Veeps associates each shopping cart as a <a href="https://hexdocs.pm/elixir/GenServer.html">GenServer</a>, which is a lightweight process managed by the Erlang VM.</p> <p>This decision made it easier for them to implement other business requirements, such as locking the cart during payments and shopping cart expiration. Since each cart is a process, the expiration is as simple as sending a message to a cart process based on a timer, which is easy to do using GenServers.</p> <p>For caching, the team relies on <a href="https://www.erlang.org/doc/man/ets.html">ETS (Erlang Term Storage)</a>, a high-performing key-value store part of the Erlang standard library. For cache busting between multiple parts of the distributed system, they use <a href="https://github.com/phoenixframework/phoenix_pubsub">Phoenix PubSub</a>, a real-time publisher/subscriber library that comes with <a href="https://phoenixframework.org/">Phoenix</a>.</p> <p>Before the rewrite, the Beaconing service used Google’s Firebase. Now, the system uses <a href="https://elixir-broadway.org/">Broadway</a> to ingest data from hundreds of thousands of HTTP requests from concurrent users. Broadway is an Elixir library for building concurrent data ingestion and processing pipelines. They utilized the library’s capabilities to efficiently send requests to AWS services, regulating batch sizes to comply with AWS limits. They also used it to handle rate limiting to adhere to AWS service constraints. All of this was achieved with Broadway’s built-in functionality.</p> <p>Finally, they use <a href="https://getoban.pro/">Oban</a>, an Elixir library for background jobs, for all sorts of background-work use cases.</p> <p>Throughout the development journey, Veeps consistently found that Elixir and its ecosystem had built-in solutions for their technical challenges. Here’s what Vincent, CTO of Veeps, had to say about that:</p> <blockquote> <p>Throughout my career, I’ve worked with large-scale systems at several companies. However, at Veeps, it’s unique because we achieve this scale with minimal reliance on external tools. It’s primarily just Elixir and its ecosystem that empower us.</p> <p>- Vincent Franco, CTO</p> </blockquote> <p>This operational simplicity benefitted not only the production environment but also the development side. The team could focus on learning Elixir and its ecosystem without the need to master additional technologies, resulting in increased productivity.</p> <h3 id="liveview-simplifying-the-interaction-between-front-end-and-back-end-developers">LiveView: simplifying the interaction between front-end and back-end developers</h3> <p>After the rewrite, <a href="https://github.com/phoenixframework/phoenix_live_view">LiveView</a>, a Phoenix library for building interactive, real-time web apps, was used for every part of the front-end except for the “Onstage” subsystem (responsible for the live stream itself).</p> <p>The two front-end developers, who came from a React background, also started writing LiveView. After this new experience, the team found the process of API negotiation between the front-end and back-end engineers much simpler compared to when using React. This was because they only had to use Elixir modules and functions instead of creating additional HTTP API endpoints and all the extra work that comes with them, such as API versioning.</p> <blockquote> <p>Our front-end team, originally proficient in React, has made a remarkable transition to LiveView. They’ve wholeheartedly embraced its user-friendly nature and its smooth integration into our system.</p> <p>- Vincent Franco, CTO</p> </blockquote> <h2 id="conclusion-insights-from-veeps-elixir-experience">Conclusion: insights from Veeps’ Elixir experience</h2> <p>The decision to use Elixir has paid dividends beyond just system scalability. The team, with varied backgrounds in Java, PHP, Ruby, Python, and Javascript, found Elixir’s ecosystem to be a harmonious balance of simplicity and power.</p> <p>By embracing Elixir’s ecosystem, including Erlang/OTP, Phoenix, LiveView, and Broadway, they built a robust system, eliminated the need for numerous external dependencies, and kept productively developing new features.</p> <blockquote> <p>Throughout my career, I’ve never encountered a developer experience as exceptional as this. Whether it’s about quantity or complexity, tasks seem to flow effortlessly. The team’s morale is soaring, everyone is satisfied, and there’s an unmistakable atmosphere of positivity. We’re all unequivocally enthusiastic about this language.</p> <p>- Vincent Franco, CTO</p> </blockquote> <p>Veeps’ case illustrates how Elixir effectively handles high-scale challenges while keeping the development process straightforward and developer-friendly.</p> José Valim 2023-12-22T00:00:00+00:00 /blog/2023/12/22/elixir-v1-16-0-released <p>Elixir v1.16 has just been released. 🎉</p> <p>The Elixir team continues improving the developer experience via tooling, documentation, and precise feedback, while keeping the language stable and compatible.</p> <p>The notable improvements in this release are the addition of compiler diagnostics and extensive improvements to our docs in the forms of guides, anti-patterns, diagrams and more.</p> <h2 id="code-snippets-in-diagnostics">Code snippets in diagnostics</h2> <p>Elixir v1.15 introduced a new compiler diagnostic format and the ability to print multiple error diagnostics per compilation (in addition to multiple warnings).</p> <p>With Elixir v1.16, we also include code snippets in exceptions and diagnostics raised by the compiler, including ANSI coloring on supported terminals. For example, a syntax error now includes a pointer to where the error happened:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>** (SyntaxError) invalid syntax found on lib/my_app.ex:1:17: error: syntax error before: '*' │ 1 │ [1, 2, 3, 4, 5, *] │ ^ │ └─ lib/my_app.ex:1:17 </code></pre></div></div> <p>For mismatched delimiters, it now shows both delimiters:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>** (MismatchedDelimiterError) mismatched delimiter found on lib/my_app.ex:1:18: error: unexpected token: ) │ 1 │ [1, 2, 3, 4, 5, 6) │ │ └ mismatched closing delimiter (expected "]") │ └ unclosed delimiter │ └─ lib/my_app.ex:1:18 </code></pre></div></div> <p>For unclosed delimiters, it now shows where the unclosed delimiter starts:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>** (TokenMissingError) token missing on lib/my_app:8:23: error: missing terminator: ) │ 1 │ my_numbers = (1, 2, 3, 4, 5, 6 │ └ unclosed delimiter ... 8 │ IO.inspect(my_numbers) │ └ missing closing delimiter (expected ")") │ └─ lib/my_app:8:23 </code></pre></div></div> <p>Errors and warnings diagnostics also include code snippets. When possible, we will show precise spans, such as on undefined variables:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> error: undefined variable "unknown_var" │ 5 │ a - unknown_var │ ^^^^^^^^^^^ │ └─ lib/sample.ex:5:9: Sample.foo/1 </code></pre></div></div> <p>Otherwise the whole line is underlined:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>error: function names should start with lowercase characters or underscore, invalid name CamelCase │ 3 │ def CamelCase do │ ^^^^^^^^^^^^^^^^ │ └─ lib/sample.ex:3 </code></pre></div></div> <p>A huge thank you to Vinícius Müller for working on the new diagnostics.</p> <h2 id="revamped-documentation">Revamped documentation</h2> <p>The <a href="https://github.com/elixir-lang/ex_doc">ExDoc</a> package provides Elixir developers with one of the most complete and robust documentation generator. It <a href="/blog/2022/12/22/cheatsheets-and-8-other-features-in-exdoc-that-improve-the-developer-experience/">supports API references, tutorials, cheatsheets, and more</a>.</p> <p>However, because many of the language tutorials and reference documentation were written before ExDoc, they were maintained separately as part of the official website, separate from the language source code. With Elixir v1.16, <a href="https://hexdocs.pm/elixir/introduction.html">we have moved our learning material to the language repository</a>. This provides several benefits:</p> <ol> <li> <p>Tutorials are versioned alongside their relevant Elixir version</p> </li> <li> <p>You get full-text search across all API reference and tutorials</p> </li> <li> <p>ExDoc will autolink module and function names in tutorials to their relevant API documentation</p> </li> </ol> <p>Another feature we have incorporated in this release is the addition of cheatsheets, starting with <a href="https://hexdocs.pm/elixir/main/enum-cheat.html">a cheatsheet for the Enum module</a>. If you would like to contribute future cheatsheets to Elixir itself, feel free to start a discussion and collect feedback on the <a href="https://elixirforum.com/">Elixir Forum</a>.</p> <p>Finally, we have started enriching our documentation with <a href="https://mermaid.js.org/">Mermaid.js</a> diagrams. You can find examples in the <a href="https://hexdocs.pm/elixir/GenServer.html">GenServer</a> and <a href="https://hexdocs.pm/elixir/Supervisor.html">Supervisor</a> docs.</p> <p>Elixir has always been praised by its excellent documentation and we are glad to continue to raise the bar for the whole ecosystem.</p> <h2 id="living-anti-patterns-reference">Living anti-patterns reference</h2> <p>Elixir v1.16 incorporates and extends the work on <a href="https://github.com/lucasvegi/Elixir-Code-Smells/blob/main/etc/2023-emse-code-smells-elixir.pdf">Understanding Code Smells in Elixir Functional Language</a>, by Lucas Vegi and Marco Tulio Valente, from <a href="http://aserg.labsoft.dcc.ufmg.br/">ASERG/DCC/UFMG</a>, into <a href="https://hexdocs.pm/elixir/what-anti-patterns.html">the official documention in the form of anti-patterns</a>. Our goal is to provide examples of potential pitfalls for library and application developers, with additional context and guidance on how to improve their codebases.</p> <p>In earlier versions, Elixir’s official reference for library authors included a list of anti-patterns for library developers. Lucas Vegi and Marco Tulio Valente extended and refined this list based on the existing literature, articles, and community input (including feedback based on their prevalence in actual codebases).</p> <p>To incorporate the anti-patterns into the language, we trimmed the list down to keep only anti-patterns which are unambiguous and actionable, and divided them into four categories: <a href="https://hexdocs.pm/elixir/code-anti-patterns.html">code-related</a>, <a href="https://hexdocs.pm/elixir/design-anti-patterns.html">design-related</a>, <a href="https://hexdocs.pm/elixir/process-anti-patterns.html">process-related</a>, and <a href="https://hexdocs.pm/elixir/macro-anti-patterns.html">meta-programming</a>. Then we collected more community feedback during the release candidate period, further refining and removing unclear guidance.</p> <p>We are quite happy with the current iteration of anti-patterns but this is just the beginning. As they become available to the whole community, we expect to receive more input, questions, and concerns. We will continue listening and improving, as our ultimate goal is to provide a live reference that reflects the practices of the ecosystem, rather than a document that is written in stone and ultimately gets out of date. A perfect example of this is <a href="https://github.com/elixir-lang/elixir/pull/13194">the recent addition of “Sending unnecessary data” anti-pattern</a>, which was contributed by the community and describes a pitfall that may happen across codebases.</p> <h2 id="type-system-updates">Type system updates</h2> <p>As we get Elixir v1.16 out of door, the Elixir team will focus on bringing the initial core for set-theoretic types into the Elixir compiler, with the goal of running automated analysis in patterns and guards. This is <a href="/blog/2023/06/22/type-system-updates-research-dev/">the first step outlined in a previous article</a> and is sponsored by <a href="https://www.fresha.com">Fresha</a> (<a href="https://www.fresha.com/careers/openings?department=engineering">they are hiring!</a>), <a href="https://starfish.team">Starfish*</a> (<a href="https://starfish.team/jobs/experienced-elixir-developer">they are hiring!</a>), and <a href="https://dashbit.co">Dashbit</a>.</p> <h2 id="learn-more">Learn more</h2> <p>Other notable changes in this release are:</p> <ul> <li> <p>the addition of <a href="https://hexdocs.pm/elixir/String.html#replace_invalid/2"><code class="language-plaintext highlighter-rouge">String.replace_invalid/2</code></a>, to help deal with invalid UTF-8 encoding</p> </li> <li> <p>the addition of the <code class="language-plaintext highlighter-rouge">:limit</code> option in <a href="https://hexdocs.pm/elixir/Task.html#yield_many/2"><code class="language-plaintext highlighter-rouge">Task.yield_many/2</code></a> that limits the maximum number of tasks to yield</p> </li> <li> <p>improved binary pattern matching by allowing prefix binary matches, such as <code class="language-plaintext highlighter-rouge">&lt;&lt;^prefix::binary, rest::binary&gt;&gt;</code></p> </li> </ul> <p>For a complete list of all changes, see the <a href="https://github.com/elixir-lang/elixir/releases/tag/v1.16.0">full release notes</a>.</p> <p>Check <a href="/install.html">the Install section</a> to get Elixir installed and read our <a href="https://hexdocs.pm/elixir/introduction.html">Getting Started guide</a> to learn more.</p> <p>Happy learning!</p> José Valim 2023-09-20T00:00:00+00:00 /blog/2023/09/20/strong-arrows-gradual-typing <p><em>This is article expands on the topic of gradual set-theoretic typing discussed during my keynote at <a href="https://www.youtube.com/watch?v=giYbq4HmfGA">ElixirConf US 2023</a>.</em></p> <p>There is an on-going effort <a href="https://elixir-lang.org/blog/2023/06/22/type-system-updates-research-dev/">to research and develop a type system for Elixir</a>, lead by <a href="https://www.irif.fr/~gc/">Giuseppe Castagna</a>, CNRS Senior Researcher, and taken by <a href="https://www.irif.fr/users/gduboc/index">Guillaume Duboc</a> as part of his PhD studies.</p> <p>In this article, we will discuss how the proposed type system will tackle gradual typing and how it relates to set-theoretic types, with the goal of providing an introduction to the ideas <a href="https://arxiv.org/abs/2306.06391">presented in our paper</a>.</p> <h2 id="set-theoretic-types">Set-theoretic types</h2> <p>The type system we are currently researching and developing for Elixir is based on set-theoretic types, which is to say its operations are based on the fundamental set operations of union, intersection, and negation.</p> <p>For example, the atom <code class="language-plaintext highlighter-rouge">:ok</code> is a value in Elixir, that can be represented by the type <code class="language-plaintext highlighter-rouge">:ok</code>. All atoms in Elixir are represented by themselves in the type system. A function that returns either <code class="language-plaintext highlighter-rouge">:ok</code> or <code class="language-plaintext highlighter-rouge">:error</code> is said to return <code class="language-plaintext highlighter-rouge">:ok or :error</code>, where the <code class="language-plaintext highlighter-rouge">or</code> operator represents the union.</p> <p>The types <code class="language-plaintext highlighter-rouge">:ok</code> and <code class="language-plaintext highlighter-rouge">:error</code> are contained by the type <code class="language-plaintext highlighter-rouge">atom()</code>, which is an infinite set representing all atoms. The union of the types <code class="language-plaintext highlighter-rouge">:ok</code> and <code class="language-plaintext highlighter-rouge">atom()</code> can be written as <code class="language-plaintext highlighter-rouge">:ok or atom()</code>, and is equivalent to <code class="language-plaintext highlighter-rouge">atom()</code> (as <code class="language-plaintext highlighter-rouge">:ok</code> is a subset of <code class="language-plaintext highlighter-rouge">atom()</code>). The intersection of the types <code class="language-plaintext highlighter-rouge">:ok</code> and <code class="language-plaintext highlighter-rouge">atom()</code> can be written as <code class="language-plaintext highlighter-rouge">:ok and atom()</code>, and is equivalent to <code class="language-plaintext highlighter-rouge">:ok</code>.</p> <p>Similarly, <code class="language-plaintext highlighter-rouge">integer()</code> is another infinite set representing all integers. <code class="language-plaintext highlighter-rouge">integer() or atom()</code> is the union of all integers and atoms. The intersection <code class="language-plaintext highlighter-rouge">integer() and atom()</code> is an empty set, which we call <code class="language-plaintext highlighter-rouge">none()</code>. The union of all types that exist in Elixir is called <code class="language-plaintext highlighter-rouge">term()</code>.</p> <p>The beauty of set-theoretic types is that we can model many interesting properties found in Elixir programs on top of those fundamental set operations, which in turn we hope to make typing in Elixir both more expressive and accessible. Let’s see an example of how a type system feature, called bounded quantification (or bounded polymorphism), can be implemented with set-theoretic types.</p> <h2 id="upper-and-lower-bounds">Upper and lower bounds</h2> <p>The <code class="language-plaintext highlighter-rouge">identity</code> function is a function that receives an argument and returns it as is. In Java, it would be written as follows:</p> <div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">static</span> <span class="o">&lt;</span><span class="no">T</span><span class="o">&gt;</span> <span class="no">T</span> <span class="nf">identity</span><span class="o">(</span><span class="no">T</span> <span class="n">arg</span><span class="o">)</span> <span class="o">{</span> <span class="k">return</span> <span class="n">arg</span><span class="o">;</span> <span class="o">}</span> </code></pre></div></div> <p>In TypeScript:</p> <div class="language-typescript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">function</span> <span class="nx">identity</span><span class="o">&lt;</span><span class="nx">T</span><span class="o">&gt;</span><span class="p">(</span><span class="nx">arg</span><span class="p">:</span> <span class="nx">T</span><span class="p">):</span> <span class="nx">T</span> <span class="p">{</span> <span class="k">return</span> <span class="nx">arg</span><span class="p">;</span> <span class="p">}</span> </code></pre></div></div> <p>Or in Haskell:</p> <div class="language-haskell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">id</span> <span class="o">::</span> <span class="n">a</span> <span class="o">-&gt;</span> <span class="n">a</span> <span class="n">id</span> <span class="n">arg</span> <span class="o">=</span> <span class="n">arg</span> </code></pre></div></div> <p>In all of the examples above, we say the function receives an argument of type variable <code class="language-plaintext highlighter-rouge">T</code> (or type variable <code class="language-plaintext highlighter-rouge">a</code> in Haskell’s case) and return a value of the same type <code class="language-plaintext highlighter-rouge">T</code>. We call this parametric polymorphism, because the function parameter - its argument - can take many (poly) shapes (morphs). In Elixir, we could then support:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="err">$</span> <span class="n">a</span> <span class="o">-&gt;</span> <span class="n">a</span> <span class="k">def</span> <span class="n">identity</span><span class="p">(</span><span class="n">arg</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="n">arg</span> </code></pre></div></div> <p>Sometimes we may want to further constrain those type variables. As example, let’s constraint the identity function in Java to numbers:</p> <div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">static</span> <span class="o">&lt;</span><span class="no">T</span> <span class="kd">extends</span> <span class="nc">Number</span><span class="o">&gt;</span> <span class="no">T</span> <span class="nf">identity</span><span class="o">(</span><span class="no">T</span> <span class="n">arg</span><span class="o">)</span> <span class="o">{</span> <span class="k">return</span> <span class="n">arg</span><span class="o">;</span> <span class="o">}</span> </code></pre></div></div> <p>Or in TypeScript:</p> <div class="language-typescript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">function</span> <span class="nx">identity</span><span class="o">&lt;</span><span class="nx">T</span> <span class="kd">extends</span> <span class="kr">number</span><span class="o">&gt;</span><span class="p">(</span><span class="nx">arg</span><span class="p">:</span> <span class="nx">T</span><span class="p">):</span> <span class="nx">T</span> <span class="p">{</span> <span class="k">return</span> <span class="nx">arg</span><span class="p">;</span> <span class="p">}</span> </code></pre></div></div> <p>In Haskell, we can constrain to a typeclass, such as <code class="language-plaintext highlighter-rouge">Ord</code>:</p> <div class="language-haskell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">id</span> <span class="o">::</span> <span class="kt">Ord</span> <span class="n">a</span> <span class="o">=&gt;</span> <span class="n">a</span> <span class="o">-&gt;</span> <span class="n">a</span> <span class="n">id</span> <span class="n">x</span> <span class="o">=</span> <span class="n">x</span> </code></pre></div></div> <p>In other words, these functions can accept any type as long as they fulfill a given constraint. This in turn is called bounded polymorphism, because we are putting bounds on the types we can receive.</p> <p>With all that said, how can we implement bounded polymorphism in set-theoretic types? Imagine we have a type variable <code class="language-plaintext highlighter-rouge">a</code>, how can we ensure it is bounded or constrained to another type?</p> <p>With set-theoretic types, this operation is an intersection. If you have <code class="language-plaintext highlighter-rouge">a and atom()</code>, <code class="language-plaintext highlighter-rouge">a</code> can be the type <code class="language-plaintext highlighter-rouge">:foo</code>. <code class="language-plaintext highlighter-rouge">a</code> can also be the type <code class="language-plaintext highlighter-rouge">atom()</code>, which represents all atom types, but <code class="language-plaintext highlighter-rouge">a</code> cannot be <code class="language-plaintext highlighter-rouge">integer()</code>, as <code class="language-plaintext highlighter-rouge">integer() and atom()</code> will return an empty set. In other words, there is no need to introduce a new semantic construct, as intersections can be used to place upper bounds in type variables! Therefore, we could restrict Elixir’s identity function to numbers like this:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="err">$</span> <span class="n">a</span> <span class="ow">and</span> <span class="n">number</span><span class="p">()</span> <span class="o">-&gt;</span> <span class="n">a</span> <span class="ow">and</span> <span class="n">number</span><span class="p">()</span> <span class="k">def</span> <span class="n">identity</span><span class="p">(</span><span class="n">arg</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="n">arg</span> </code></pre></div></div> <p>Of course, we can provide syntax sugar for those constraints:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="err">$</span> <span class="n">a</span> <span class="o">-&gt;</span> <span class="n">a</span> <span class="ow">when</span> <span class="ss">a:</span> <span class="n">number</span><span class="p">()</span> <span class="k">def</span> <span class="n">identity</span><span class="p">(</span><span class="n">arg</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="n">arg</span> </code></pre></div></div> <p>But at the end of the day it will simply expand to intersections. The important bit is that, at the semantic level, there is no need for additional constructs and representations.</p> <blockquote> <p>Note: for the type-curious readers, set-theoretic types implement <a href="http://lucacardelli.name/Papers/OnUnderstanding.pdf">a limited form of bounded quantification <em>à la</em> Kernel Fun</a>. In a nutshell, it means we can only compare functions if they have the same bounds. For example, our type system states <code class="language-plaintext highlighter-rouge">a -&gt; a when a: integer() or boolean()</code> is not a subtype of <code class="language-plaintext highlighter-rouge">a -&gt; a when a: integer()</code>.</p> </blockquote> <p>We also get lower bounds for free. If intersections allow us to place an upper bound on a type variable, a union is equivalent to a lower bound as it specifies the type variable will always be augmented by the union-ed type. For example, <code class="language-plaintext highlighter-rouge">a or atom()</code> says the result will always include atoms plus whatever else specified by <code class="language-plaintext highlighter-rouge">a</code> (which may be an atom, <code class="language-plaintext highlighter-rouge">atom()</code> itself, or a completely disjoint type such as <code class="language-plaintext highlighter-rouge">integer()</code>).</p> <p>Elixir protocols, which is an Elixir construct equivalent to Haskell Typeclasses or Java interfaces, is another example of functionality that can be modelled and composed with set-theoretic types without additional semantics. The exact mechanism to do so is left as an exercise to the reader (or the topic of a future blog post).</p> <h2 id="enter-gradual-typing">Enter gradual typing</h2> <p>Elixir is a functional dynamic programming language. Existing Elixir programs are untyped, which means that a type system needs mechanisms to interface existing Elixir code with future statically typed Elixir code. We can achieve this with gradual typing.</p> <p>A gradual type system is a type system that defines a <code class="language-plaintext highlighter-rouge">dynamic()</code> type. It is sometimes written as <code class="language-plaintext highlighter-rouge">?</code> and sometimes known as the <code class="language-plaintext highlighter-rouge">any</code> type (but I prefer to avoid <code class="language-plaintext highlighter-rouge">any</code> because it is too short and too lax in languages like TypeScript).</p> <p>In Elixir, the <code class="language-plaintext highlighter-rouge">dynamic()</code> type means the type is only known at runtime, effectively disabling static checks for that type. More interestingly, we can also place upper and lower bounds on the dynamic type using set operations. As we will soon learn, this will reveal interesting properties about our type system.</p> <p>It is often said that gradual typing is the best of both words. Perhaps ironically, that’s true and false at the same time. If you use a gradual type system but you never use the <code class="language-plaintext highlighter-rouge">dynamic()</code> type, then it behaves exactly like a static type system. However, the more you use the <code class="language-plaintext highlighter-rouge">dynamic()</code> type, the fewer guarantees the type system will give you, the more the <code class="language-plaintext highlighter-rouge">dynamic()</code> type propagates through the system. Therefore, it is in our interest to reduce the occurrences of the <code class="language-plaintext highlighter-rouge">dynamic()</code> type as much as possible, and that’s what we set out to do.</p> <h2 id="interfacing-static-and-dynamic-code-the-trouble-with-dynamic">Interfacing static and dynamic code: the trouble with <code class="language-plaintext highlighter-rouge">dynamic()</code></h2> <p>Let’s go back to our constrained identity function that accepts only numbers:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="err">$</span> <span class="n">a</span> <span class="o">-&gt;</span> <span class="n">a</span> <span class="ow">when</span> <span class="ss">a:</span> <span class="n">number</span><span class="p">()</span> <span class="k">def</span> <span class="n">identity</span><span class="p">(</span><span class="n">arg</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="n">arg</span> </code></pre></div></div> <p>Now imagine that we have some untyped code that calls this function:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="n">debug</span><span class="p">(</span><span class="n">arg</span><span class="p">)</span> <span class="k">do</span> <span class="s2">"we got: "</span> <span class="o">&lt;&gt;</span> <span class="n">identity</span><span class="p">(</span><span class="n">arg</span><span class="p">)</span> <span class="k">end</span> </code></pre></div></div> <p>Since <code class="language-plaintext highlighter-rouge">debug/1</code> is untyped, its argument will receive the type <code class="language-plaintext highlighter-rouge">dynamic()</code>.</p> <p><code class="language-plaintext highlighter-rouge">debug/1</code> proceeds to call <code class="language-plaintext highlighter-rouge">identity</code> with an argument and then uses the string concatenation operator (<code class="language-plaintext highlighter-rouge">&lt;&gt;</code>) to concatenate <code class="language-plaintext highlighter-rouge">"we got: "</code> to the result of <code class="language-plaintext highlighter-rouge">identity(arg)</code>. Since <code class="language-plaintext highlighter-rouge">identity/1</code> is meant to return a number and string concatenation requires two strings as operands, there is a typing error in this program. On the other hand, if you call <code class="language-plaintext highlighter-rouge">debug("hello")</code> at runtime, the code will work and won’t raise any exceptions.</p> <p>In other words, the static typing version of the program and its runtime execution do not match in behaviour. So how do we tackle this?</p> <p>One option is to say that’s all behaving as expected. If <code class="language-plaintext highlighter-rouge">debug/1</code> is untyped, its <code class="language-plaintext highlighter-rouge">arg</code> has the <code class="language-plaintext highlighter-rouge">dynamic()</code> type. To type check this program, we specify that <code class="language-plaintext highlighter-rouge">identity(dynamic())</code> returns the <code class="language-plaintext highlighter-rouge">dynamic()</code> type, the concatenation of a string with <code class="language-plaintext highlighter-rouge">dynamic()</code> also returns <code class="language-plaintext highlighter-rouge">dynamic()</code>, and consequently <code class="language-plaintext highlighter-rouge">debug/1</code> gets the type <code class="language-plaintext highlighter-rouge">dynamic() -&gt; dynamic()</code>, with no type errors emitted.</p> <p>The trouble is: this is not a very useful choice. Once <code class="language-plaintext highlighter-rouge">dynamic()</code> enters the system, it <em>spreads everywhere</em>, we perform fewer checks, effectively discarding the information that <code class="language-plaintext highlighter-rouge">identity/1</code> returns a number, and the overall type system becomes less useful.</p> <p>Another option would be for us to say: once we call a statically typed function with <code class="language-plaintext highlighter-rouge">dynamic()</code>, we will ignore the <code class="language-plaintext highlighter-rouge">dynamic()</code> type. If the function says it returns a <code class="language-plaintext highlighter-rouge">number()</code>, then it will surely be a number! In this version, <code class="language-plaintext highlighter-rouge">identity(dynamic())</code> returns <code class="language-plaintext highlighter-rouge">number()</code> and the type system will catch a type error when concatenating a string with a number.</p> <p>This is similar to the approach taken by TypeScript. This means we can perform further static checks, but it also means we can call <code class="language-plaintext highlighter-rouge">debug("foobar")</code> and that will return the string <code class="language-plaintext highlighter-rouge">"we got: foobar"</code>! But how can that be possible when the type system told us that <code class="language-plaintext highlighter-rouge">identity</code> returns a <code class="language-plaintext highlighter-rouge">number()</code>? This can lead to confusion and surprising results at runtime. We say this system is unsound, because the types at runtime do not match our compile-time types.</p> <p>None of our solutions so far attempted to match the static and runtime behaviors, but rather, they picked one in favor of the other.</p> <p>But don’t despair, there is yet another option. We could introduce runtime checks whenever we cross the “dynamic &lt;-&gt; static” boundaries. In this case, we could say <code class="language-plaintext highlighter-rouge">identity(dynamic())</code> returns a <code class="language-plaintext highlighter-rouge">number()</code>, but we will introduce a runtime check into the code to guarantee that’s the case. This means we get static checks, we ensure the value is correct at runtime, with the cost of introducing additional checks at runtime. Unfortunately, those checks may affect performance, depending on the complexity of the data structure and on how many times we cross the “dynamic &lt;-&gt; static” boundary.</p> <blockquote> <p>Note: there is <a href="https://arxiv.org/abs/2206.13831">recent research in using the runtime checks introduced by a gradual type system to provide compiler optimizations</a>. Some of these techniques are already leveraged by the Erlang VM to optimize code based on patterns and guards.</p> </blockquote> <p>To summarize, we have three options:</p> <ul> <li> <p>Calling static code from dynamic code returns <code class="language-plaintext highlighter-rouge">dynamic()</code>, dropping the opportunity of further static typing checks (this is sound)</p> </li> <li> <p>Calling static code from dynamic code returns the static types, potentially leading to mismatched types at runtime (this is unsound)</p> </li> <li> <p>Calling static code from dynamic code returns the static types with additional runtime checks, unifying both behaviours but potentially impacting performance (this is sound)</p> </li> </ul> <h2 id="introducing-strong-arrows">Introducing strong arrows</h2> <p>I have always said that Elixir, thanks to Erlang, is an assertive language. For example, if our identity function is restricted to only numbers, in practice we would most likely write it as:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="err">$</span> <span class="n">a</span> <span class="o">-&gt;</span> <span class="n">a</span> <span class="ow">when</span> <span class="ss">a:</span> <span class="n">number</span><span class="p">()</span> <span class="k">def</span> <span class="n">identity</span><span class="p">(</span><span class="n">arg</span><span class="p">)</span> <span class="ow">when</span> <span class="n">is_number</span><span class="p">(</span><span class="n">arg</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="n">arg</span> </code></pre></div></div> <p>In the example above, <code class="language-plaintext highlighter-rouge">identity</code> will fail if given any value that is not a number. We often rely on pattern matching and guards and, in turn, they helps us assert on the types we are working with. Not only that, Erlang’s JIT compiler already relies on this information to <a href="https://www.erlang.org/blog/type-based-optimizations-in-the-jit/">perform optimizations</a> whenever possible.</p> <p>We also say Elixir is strongly typed because its functions and operators avoid implicit type conversions. The following functions also fail when their input does not match their type:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="err">$</span> <span class="n">binary</span><span class="p">()</span> <span class="o">-&gt;</span> <span class="n">binary</span><span class="p">()</span> <span class="k">def</span> <span class="n">debug</span><span class="p">(</span><span class="n">string</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="s2">"we got: "</span> <span class="o">&lt;&gt;</span> <span class="n">string</span> <span class="err">$</span> <span class="p">(</span><span class="n">integer</span><span class="p">()</span> <span class="o">-&gt;</span> <span class="n">integer</span><span class="p">())</span> <span class="ow">and</span> <span class="p">(</span><span class="n">float</span><span class="p">()</span> <span class="o">-&gt;</span> <span class="n">float</span><span class="p">())</span> <span class="k">def</span> <span class="n">increment</span><span class="p">(</span><span class="n">number</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="n">number</span> <span class="o">+</span> <span class="mi">1</span> </code></pre></div></div> <p><code class="language-plaintext highlighter-rouge">&lt;&gt;</code> only accepts binaries as arguments and will raise otherwise. <code class="language-plaintext highlighter-rouge">+</code> only accepts numbers (integers or floats) and will raise otherwise. <code class="language-plaintext highlighter-rouge">+</code> does not perform implicit conversions of non-numeric types, such as strings to number, as we can see next:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">iex</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span><span class="o">&gt;</span> <span class="n">increment</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span> <span class="mi">2</span> <span class="n">iex</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span><span class="o">&gt;</span> <span class="n">increment</span><span class="p">(</span><span class="mf">13.0</span><span class="p">)</span> <span class="mf">14.0</span> <span class="n">iex</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span><span class="o">&gt;</span> <span class="n">increment</span><span class="p">(</span><span class="s2">"foobar"</span><span class="p">)</span> <span class="o">**</span> <span class="p">(</span><span class="no">ArithmeticError</span><span class="p">)</span> <span class="n">bad</span> <span class="n">argument</span> <span class="ow">in</span> <span class="n">arithmetic</span> <span class="ss">expression:</span> <span class="s2">"foobar"</span> <span class="o">+</span> <span class="mi">1</span> </code></pre></div></div> <p>In other words, Elixir’s runtime consistently checks the values and their types at runtime. If <code class="language-plaintext highlighter-rouge">increment</code> fails when given something else than a number, then it will fail when the <code class="language-plaintext highlighter-rouge">dynamic()</code> type does not match its input at runtime. This guarantees <code class="language-plaintext highlighter-rouge">increment</code> returns its declared type and therefore we do not need to introduce runtime type checks when calling said function from untyped code.</p> <p>When we look at the <code class="language-plaintext highlighter-rouge">identity</code>, <code class="language-plaintext highlighter-rouge">debug</code>, and <code class="language-plaintext highlighter-rouge">increment</code> functions above, we - as developers - can state that these functions raise when given a value that does not match their input. However, how can we generalize this property so it is computed by the type system itself? To do so, we introduce a new concept called <strong>strong arrows</strong>, which relies on set-theoretical types to derive this property.</p> <p>The idea goes as follows: a strong arrow is a function that can be statically proven that, when evaluated on values outside of its input types (i.e. its domain), it will error. For example, in our <code class="language-plaintext highlighter-rouge">increment</code> function, if we pass a <code class="language-plaintext highlighter-rouge">string()</code> as argument, it won’t type check, because <code class="language-plaintext highlighter-rouge">string() + integer()</code> is not a valid operation. Thanks to set-theoretic types, we can compute all values outside of the domain by computing the negation of a set. Given <code class="language-plaintext highlighter-rouge">increment/1</code> will fail for all types which are <code class="language-plaintext highlighter-rouge">not number()</code>, the function is strong.</p> <p>By applying this rule to all typed functions, we will know which functions are strong and which ones are not. If a function is strong, the type system knows that calling it with a <code class="language-plaintext highlighter-rouge">dynamic()</code> type will always evaluate to its return type! Therefore we say the return type of <code class="language-plaintext highlighter-rouge">increment(dynamic())</code> is <code class="language-plaintext highlighter-rouge">number()</code>, which is sound and does not need further runtime checks!</p> <p>Going back to our <code class="language-plaintext highlighter-rouge">debug</code> function, when used with a guarded identity, it will be able to emit warnings at compile-time, errors at runtime, without introducing any additional runtime check:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="err">$</span> <span class="n">a</span> <span class="o">-&gt;</span> <span class="n">a</span> <span class="ow">when</span> <span class="ss">a:</span> <span class="n">number</span><span class="p">()</span> <span class="k">def</span> <span class="n">identity</span><span class="p">(</span><span class="n">arg</span><span class="p">)</span> <span class="ow">when</span> <span class="n">is_number</span><span class="p">(</span><span class="n">arg</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="n">arg</span> <span class="k">def</span> <span class="n">debug</span><span class="p">(</span><span class="n">arg</span><span class="p">)</span> <span class="k">do</span> <span class="s2">"we got: "</span> <span class="o">&lt;&gt;</span> <span class="n">identity</span><span class="p">(</span><span class="n">arg</span><span class="p">)</span> <span class="k">end</span> </code></pre></div></div> <p>However, if the <code class="language-plaintext highlighter-rouge">identity</code> function is not strong, then we must fallback to one of the strategies in the previous section.</p> <p>Another powerful property of strong arrows is that they are composable. Let’s pick an example from the paper:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="err">$</span> <span class="n">number</span><span class="p">(),</span> <span class="n">number</span><span class="p">()</span> <span class="o">-&gt;</span> <span class="n">number</span><span class="p">()</span> <span class="k">def</span> <span class="n">subtract</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">)</span> <span class="k">do</span> <span class="n">a</span> <span class="o">+</span> <span class="n">negate</span><span class="p">(</span><span class="n">b</span><span class="p">)</span> <span class="k">end</span> <span class="err">$</span> <span class="n">number</span><span class="p">()</span> <span class="o">-&gt;</span> <span class="n">number</span><span class="p">()</span> <span class="k">def</span> <span class="n">negate</span><span class="p">(</span><span class="n">int</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="o">-</span><span class="n">int</span> </code></pre></div></div> <p>In the example above, <code class="language-plaintext highlighter-rouge">negate/1</code>’s type is a strong arrow, as it raises for any input outside of its domain. <code class="language-plaintext highlighter-rouge">subtract/2</code>’s type is also a strong arrow, because both <code class="language-plaintext highlighter-rouge">+</code> and our own <code class="language-plaintext highlighter-rouge">negate</code> are strong arrows too. This is an important capability as it limits how <code class="language-plaintext highlighter-rouge">dynamic()</code> types spread throughout the system.</p> <blockquote> <p>Errata: my presentation used the type <code class="language-plaintext highlighter-rouge">integer()</code> instead of <code class="language-plaintext highlighter-rouge">number()</code> for the example above. However, that was a mistake in the slide. Giving the type <code class="language-plaintext highlighter-rouge">integer(), integer() -&gt; integer()</code> to <code class="language-plaintext highlighter-rouge">subtract</code> and <code class="language-plaintext highlighter-rouge">integer() -&gt; integer()</code> to <code class="language-plaintext highlighter-rouge">negate</code> does not make <code class="language-plaintext highlighter-rouge">subtract</code> a strong arrow. Can you tell why?</p> </blockquote> <p>Luckily, other gradually typed languages can also leverage strong arrows. However, the more polymorphic a language and its functions are, the more unlikely it is to conclude that a given function is strong. For example, in other gradually typed languages such as Python or Ruby, the <code class="language-plaintext highlighter-rouge">+</code> operator is extensible and the user can define custom types where the operation is valid. In TypeScript, <code class="language-plaintext highlighter-rouge">"foobar" + 1</code> is also a valid operation, which expands the function domain. In both cases, an <code class="language-plaintext highlighter-rouge">increment</code> function restricted to numbers would not have a strong arrow type, as the operator won’t fail for all types outside of <code class="language-plaintext highlighter-rouge">number()</code>. Therefore, to remain sound, they must either restrict the operands with further runtime checks or return <code class="language-plaintext highlighter-rouge">dynamic()</code> (reducing the number of compile-time checks).</p> <p>There is one last scenario to consider, which I did not include during my keynote for brevity. Take this function:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="err">$</span> <span class="n">integer</span><span class="p">()</span> <span class="o">-&gt;</span> <span class="ss">:ok</span> <span class="k">def</span> <span class="n">receives_integer_and_returns_ok</span><span class="p">(</span><span class="n">_arg</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="ss">:ok</span> </code></pre></div></div> <p>The function above can receive any type and return <code class="language-plaintext highlighter-rouge">:ok</code>. Is its type a strong arrow? Well, according to our definition, it is not. If we negate its input, type checking does not fail, it returns <code class="language-plaintext highlighter-rouge">:ok</code>.</p> <p>However, given the return type is always the same, it should be a strong arrow! To do so, let’s amend and rephrase our definition of strong arrows: we negate the domain (i.e. the inputs) of a function and then type check it. If the function returns <code class="language-plaintext highlighter-rouge">none()</code> (i.e. it does not type check) or a type which is a subset of its codomain (i.e. its output), then it is a strong arrow.</p> <h2 id="gradual-typing-and-false-positives">Gradual typing and false positives</h2> <p>There is one last scenario we must take into consideration when interfacing dynamic and static code. Imagine the following code:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="n">increment_and_remainder</span><span class="p">(</span><span class="n">numerator</span><span class="p">,</span> <span class="n">denominator</span><span class="p">)</span> <span class="k">do</span> <span class="n">rem</span><span class="p">(</span><span class="n">numerator</span><span class="p">,</span> <span class="n">increment</span><span class="p">(</span><span class="n">denominator</span><span class="p">))</span> <span class="k">end</span> <span class="err">$</span> <span class="p">(</span><span class="n">integer</span><span class="p">()</span> <span class="o">-&gt;</span> <span class="n">integer</span><span class="p">())</span> <span class="ow">and</span> <span class="p">(</span><span class="n">float</span><span class="p">()</span> <span class="o">-&gt;</span> <span class="n">float</span><span class="p">())</span> <span class="k">def</span> <span class="n">increment</span><span class="p">(</span><span class="n">number</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="n">number</span> <span class="o">+</span> <span class="mi">1</span> </code></pre></div></div> <p>The <code class="language-plaintext highlighter-rouge">increment_and_remainder/2</code> function is untyped, therefore both of its arguments receive type <code class="language-plaintext highlighter-rouge">dynamic()</code>. The function then computes the remainder of the numerator by the denominator incremented by one. For this example, let’s assume all uses of <code class="language-plaintext highlighter-rouge">increment_and_remainder/2</code> in our program passes two integers as arguments.</p> <p>Given <code class="language-plaintext highlighter-rouge">increment/1</code> has a strong arrow type, according to our definition, <code class="language-plaintext highlighter-rouge">increment(dynamic())</code> will return <code class="language-plaintext highlighter-rouge">integer() or float()</code> (also known as <code class="language-plaintext highlighter-rouge">number()</code>). Here lies the issue: if <code class="language-plaintext highlighter-rouge">increment(dynamic())</code> returns <code class="language-plaintext highlighter-rouge">integer() or float()</code>, the program above won’t type check because <code class="language-plaintext highlighter-rouge">rem/2</code> does not accept floats.</p> <p>When faced with this problem, there are two possible reactions:</p> <ol> <li> <p>It is correct for the function to not type check given <code class="language-plaintext highlighter-rouge">increment</code> may return a float</p> </li> <li> <p>It is incorrect for the function to not type check because the error it describes never occurs in the codebase</p> </li> </ol> <p>Another interesting property of gradual set-theoretic types is that we can also place upper bounds on the <code class="language-plaintext highlighter-rouge">dynamic()</code> type. If a function returns <code class="language-plaintext highlighter-rouge">number()</code>, it means the caller needs to handle both <code class="language-plaintext highlighter-rouge">integer()</code> and <code class="language-plaintext highlighter-rouge">float()</code>. However, if a function returns <code class="language-plaintext highlighter-rouge">dynamic() and number()</code>, it means the type is defined at runtime, but it must still verify it is one of <code class="language-plaintext highlighter-rouge">integer()</code> or <code class="language-plaintext highlighter-rouge">float()</code> at compile time.</p> <p>Therefore, <code class="language-plaintext highlighter-rouge">rem/2</code> will type check if its second argument has the type <code class="language-plaintext highlighter-rouge">dynamic() and number()</code>, as there is one type at runtime (<code class="language-plaintext highlighter-rouge">integer()</code>) that satisfies type checking. On the other hand, if you attempt to use the string concatenation operator (<code class="language-plaintext highlighter-rouge">&lt;&gt;</code>) on <code class="language-plaintext highlighter-rouge">dynamic() and number()</code>, then there is no acceptable runtime type and you’d still get a typing violation!</p> <p>Going back to strong arrows, there are two possible return types from a strong arrow:</p> <ol> <li> <p>A strong arrow, when presented with a dynamic type, returns its codomain</p> </li> <li> <p>A strong arrow, when presented with a dynamic type, returns the intersection of the codomain with the <code class="language-plaintext highlighter-rouge">dynamic()</code> type</p> </li> </ol> <p>The second option opens up the possibility for existing codebases to gradually migrate to static types without dealing with false positives. Coming from a dynamic background, false positives can be seen as noisy or as an indication that static types are not worth the trouble. With strong arrows and gradual set-theoretic types, we will be able to explore different trade-offs on mixed codebases. Which of the two choices above we will adopt as a default and how to customize them is yet to be decided. It will depend on the community feedback as we experiment and integrate the type system.</p> <p>Erlang and Elixir developers who use Dialyzer will be familiar with these trade-offs, as the second option mirrors Dialyzer’s behaviour of no false positives. The difference here is that our semantics are integrated into a complete type system. If no type signature is present, the <code class="language-plaintext highlighter-rouge">dynamic()</code> type is used, and we will leverage the techniques described here to interface dynamic and static code. If a function has a type signature, and no <code class="language-plaintext highlighter-rouge">dynamic()</code> type is present, then it will behave as statically typed code when called with statically typed arguments. Migrating to static types will naturally reduce the interaction points between dynamic and static code, removing the reliance on the <code class="language-plaintext highlighter-rouge">dynamic()</code> type.</p> <h2 id="summary">Summary</h2> <p>Set-theoretic types allow us to express many typing features based on set operations of union, intersection, and negation.</p> <p>In particular, we have been exploring a gradual set-theoretic type system for Elixir, paying special attention to how the type system will integrate with existing codebases and how it can best leverage the semantics of the Erlang Virtual Machine. The type system will also perform limited inference based on patterns and guards (as described in the paper), which - in addition to strong arrows - we hope to bring some of the benefits of static typing to codebases without changing a single line of code.</p> <p>While our efforts have officially moved from research into development, and <a href="https://elixir-lang.org/blog/2023/06/22/type-system-updates-research-dev/">we have outlined an implementation plan</a>, we haven’t yet fully implemented nor assessed the usability of set-theoretic types in existing Elixir codebases, either large or small. There is much to implement and validate, and we don’t rule the possibility of finding unforeseen deal breakers that could send us back to square one. Yet we are pleased and cautiously excited with the new developments so far.</p> <p>The development of Elixir’s type system is sponsored by <a href="https://www.fresha.com">Fresha</a> (<a href="https://www.fresha.com/careers/openings?department=engineering">they are hiring!</a>), <a href="https://starfish.team">Starfish*</a> (<a href="https://starfish.team/jobs/experienced-elixir-developer">they are hiring!</a>), and <a href="https://dashbit.co">Dashbit</a>.</p> José Valim 2023-06-22T00:00:00+00:00 /blog/2023/06/22/type-system-updates-research-dev <p>A year ago, at ElixirConf EU 2022, we announced an effort to research and develop a type system for Elixir (<a href="https://www.youtube.com/watch?v=Jf5Hsa1KOc8">video presentation</a>) (<a href="/blog/2022/10/05/my-future-with-elixir-set-theoretic-types/">written report</a>).</p> <p>This work is happening under the lead of <a href="https://www.irif.fr/~gc/">Giuseppe Castagna</a>, CNRS Senior Researcher, and taken by <a href="https://www.irif.fr/users/gduboc/index">Guillaume Duboc</a> as part of his PhD studies, with further guidance from myself (José Valim).</p> <p>This article is a summary of where we are in our efforts and where we are going.</p> <h2 id="out-of-research">Out of research</h2> <p>Our main goal during research is to find a type system that can model most of Elixir’s functional semantics and develop brand new theory on the areas we found to be incompatible or lacking. We believe we were able to achieve this goal with a gradual set-theoretic type system and we are now ready to head towards development. Over the last 2 months, we have published plenty of resources on our results:</p> <ul> <li><a href="https://arxiv.org/abs/2306.06391">A technical report on the design principles of the Elixir type system</a></li> <li><a href="https://youtube.com/watch?v=gJJH7a2J9O8">A technical presentation by Guillaume Duboc at ElixirConf 2023 on the work above</a></li> <li><a href="https://smartlogic.io/podcast/elixir-wizards/s10-e12-jose-guillaume-giuseppe-types-elixir/">An informal discussion with Giuseppe Castagna, Guillaume Duboc, and José Valim on the SmartLogic podcast</a></li> <li><a href="https://www.twitch.tv/videos/1841707383">An informal Q&amp;A with Guillaume Duboc, José Valim, and the community on Twitch</a></li> </ul> <p>Our focus so far has been on the semantics. While we have introduced a new syntax capable of expressing the semantics of the new set-theoretic type system, the syntax is not final as there are still no concrete plans for user-facing changes to the language. Once we are confident those changes will happen, we will have plenty of discussion with the community about the type system interface and its syntax.</p> <p>The work so far has been made possible thanks to a partnership between the <a href="https://www.cnrs.fr/fr">CNRS</a> and <a href="https://remote.com">Remote</a>, with sponsorships from <a href="https://www.fresha.com">Fresha</a>, <a href="https://supabase.com">Supabase</a>, and <a href="https://dashbit.co">Dashbit</a>.</p> <h2 id="into-development">Into development</h2> <p>While there is still on-going research, our focus for the second semester of 2023 onwards is on development.</p> <p>Incorporating a type system into a language used at scale can be a daunting task. Our concerns range from how the community will interact and use the type system to how it will perform on large codebases. Therefore, our plan is to gradually introduce our gradual (pun intended) type system into the Elixir compiler.</p> <p>In the first release, types will be used just internally by the compiler. The type system will extract type information from patterns and guards to find the most obvious mistakes, such as typos in field names or type mismatches from attempting to add an integer to a string, without introducing any user-facing changes to the language. At this stage, our main goal is to assess the performance impact of the type system and the quality of the reports we can generate in case of typing violations. If we are unhappy with the results, we still have time to reassess our work or drop the initiative altogether.</p> <p>The second milestone is to introduce type annotations only in structs, which are named and statically-defined in Elixir codebases. Elixir programs frequently pattern match on structs, which reveals information about the struct fields, but it knows nothing about their respective types. By propagating types from structs and their fields throughout the program, we will increase the type system’s ability to find errors while further straining our type system implementation.</p> <p>The third milestone is to introduce the (most likely) <code class="language-plaintext highlighter-rouge">$</code>-prefixed type annotations for functions, with no or very limited type reconstruction: users can annotate their code with types, but any untyped parameter will be assumed to be of the <code class="language-plaintext highlighter-rouge">dynamic()</code> type. If successful, then we will effectively have introduced a type system into the language.</p> <p>This new exciting development stage is sponsored by <a href="https://www.fresha.com">Fresha</a> (<a href="https://www.fresha.com/careers/openings?department=engineering">they are hiring!</a>), <a href="https://starfish.team">Starfish*</a> (<a href="https://starfish.team/jobs/experienced-elixir-developer">they are hiring!</a>), and <a href="https://dashbit.co">Dashbit</a>.</p> José Valim 2023-06-19T00:00:00+00:00 /blog/2023/06/19/elixir-v1-15-0-released <p>Elixir v1.15 has just been released. 🎉</p> <p>Elixir v1.15 is a smaller release with focused improvements on compilation and boot times. This release also completes our integration process with Erlang/OTP logger, bringing new features such as log rotation and compression out of the box.</p> <p>You will also find additional convenience functions in <code class="language-plaintext highlighter-rouge">Code</code>, <code class="language-plaintext highlighter-rouge">Map</code>, <code class="language-plaintext highlighter-rouge">Keyword</code>, all Calendar modules, and others.</p> <p>Finally, we are glad to welcome <a href="https://github.com/sabiwara/">Jean Klingler</a> as a member of the Elixir Core team. Thank you for your contributions!</p> <h2 id="compile-and-boot-time-improvements">Compile and boot-time improvements</h2> <p>The last several releases brought improvements to compilation time and this version is no different. In particular, Elixir now caches and prunes load paths before compilation, ensuring your project (and dependencies!) compile faster and in an environment closer to production.</p> <p>In a nutshell, the Erlang VM loads modules from code paths. Each application that ships with Erlang and Elixir plus each dependency become an entry in your code path. The larger the code path, the more work Erlang has to do in order to find a module.</p> <p>In previous versions, Mix would only add entries to the load paths. Therefore, if you compiled 20 dependencies and you went to compile the 21st, the code path would have 21 entries (plus all Erlang and Elixir apps). This allowed modules from unrelated dependencies to be seen and made compilation slower the more dependencies you had. With this release, we will now prune the code paths to only the ones listed as dependencies, bringing the behaviour closer to <code class="language-plaintext highlighter-rouge">mix release</code>.</p> <p>Furthermore, Erlang/OTP 26 allows us to start applications concurrently and cache the code path lookups, decreasing the cost of booting applications. The combination of Elixir v1.15 and Erlang/OTP 26 should also reduce the boot time of applications, such as when starting <code class="language-plaintext highlighter-rouge">iex -S mix</code> or running a single test with <code class="language-plaintext highlighter-rouge">mix test</code>.</p> <p>As an example, I have benchmarked <a href="https://github.com/livebook-dev/livebook">the Livebook application</a> on a M1 Max MacStudio across different Elixir and Erlang/OTP versions. At the time of benchmarking, Livebook had ~200 source <code class="language-plaintext highlighter-rouge">.ex</code> files and ~35 dependencies. Compilation-times were improved by 16%:</p> <p><img src="/images/contents/livebook-compile-1.15.png" alt="Livebook compilation times" /></p> <p>Livebook saw an improvement of 30% on boot times:</p> <p><img src="/images/contents/livebook-boot-1.15.png" alt="Livebook boot times" /></p> <p>Different application will see different results. Our expectations are the gains will be more meaningful the more dependencies you have, the more files you have, and the more cores you have. We have even received reports of up to 40% faster compilation times, although it is yet unclear how generalizable this will be in practice. Note this work does not improve the time to compile slow individual files.</p> <p>The compiler is also smarter in several ways: <code class="language-plaintext highlighter-rouge">@behaviour</code> declarations no longer add compile-time dependencies and aliases in patterns and guards add no dependency whatsoever, as no dispatching happens. Furthermore, Mix now tracks the digests of <code class="language-plaintext highlighter-rouge">@external_resource</code> files, reducing the amount of recompilation when swapping branches. Finally, dependencies are automatically recompiled when their compile-time configuration changes, providing a smoother development experience.</p> <h5 id="potential-incompatibilities">Potential incompatibilities</h5> <p>Due to the code path pruning, if you have an application or dependency that does not specify its dependencies on Erlang/OTP and core Elixir applications, which has always been erroneus behaviour, it may no longer compile successfully in Elixir v1.15. You can temporarily disable code path pruning by setting <code class="language-plaintext highlighter-rouge">prune_code_paths: false</code> in your <code class="language-plaintext highlighter-rouge">mix.exs</code>, although doing so may lead to runtime bugs that are only manifested inside a <code class="language-plaintext highlighter-rouge">mix release</code>.</p> <h2 id="compiler-warnings-and-errors">Compiler warnings and errors</h2> <p>The Elixir compiler can now emit many errors for a single file, making sure more feedback is reported to developers before compilation is aborted.</p> <p>In Elixir v1.14, an undefined function would be reported as:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>** (CompileError) undefined function foo/0 (there is no such import) my_file.exs:1 </code></pre></div></div> <p>In Elixir v1.15, the new reports will look like:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>error: undefined function foo/0 (there is no such import) my_file.exs:1 ** (CompileError) my_file.exs: cannot compile file (errors have been logged) </code></pre></div></div> <p>A new function, called <code class="language-plaintext highlighter-rouge">Code.with_diagnostics/2</code>, has been added so this information can be leveraged by editors, allowing them to point to several errors at once. We have currently ongoing work and contribution to further improve the compiler diagnostics in future Elixir releases.</p> <h5 id="potential-incompatibilities-1">Potential incompatibilities</h5> <p>As part of this effort, the behaviour where undefined variables were transformed into nullary function calls, often leading to confusing error reports, has been disabled during project compilation. You can invoke <code class="language-plaintext highlighter-rouge">Code.compiler_options(on_undefined_variable: :warn)</code> at the top of your <code class="language-plaintext highlighter-rouge">mix.exs</code> to bring the old behaviour back.</p> <h2 id="integration-with-erlangotp-logger">Integration with Erlang/OTP logger</h2> <p>This release provides additional features such as global logger metadata and <a href="https://hexdocs.pm/logger/Logger.html#module-erlang-otp-handlers">file logging</a> (with rotation and compression) out of the box!</p> <p>This release also soft-deprecates Elixir’s Logger Backends in favor of Erlang’s Logger handlers. Elixir will automatically convert your <code class="language-plaintext highlighter-rouge">:console</code> backend configuration into the new configuration. Previously, you would set:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">config</span> <span class="ss">:logger</span><span class="p">,</span> <span class="ss">:console</span><span class="p">,</span> <span class="ss">level:</span> <span class="ss">:error</span><span class="p">,</span> <span class="ss">format:</span> <span class="s2">"$time $message $metadata"</span> </code></pre></div></div> <p>Which is now translated to the equivalent:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">config</span> <span class="ss">:logger</span><span class="p">,</span> <span class="ss">:default_handler</span><span class="p">,</span> <span class="ss">level:</span> <span class="ss">:error</span> <span class="n">config</span> <span class="ss">:logger</span><span class="p">,</span> <span class="ss">:default_formatter</span><span class="p">,</span> <span class="ss">format:</span> <span class="s2">"$time $message $metadata"</span> </code></pre></div></div> <p>To replace the default console handler by one that writes to disk, with log rotation and compression:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">config</span> <span class="ss">:logger</span><span class="p">,</span> <span class="ss">:default_handler</span><span class="p">,</span> <span class="ss">config:</span> <span class="p">[</span> <span class="ss">file:</span> <span class="sx">~c"system.log"</span><span class="p">,</span> <span class="ss">filesync_repeat_interval:</span> <span class="mi">5000</span><span class="p">,</span> <span class="ss">file_check:</span> <span class="mi">5000</span><span class="p">,</span> <span class="ss">max_no_bytes:</span> <span class="mi">10_000_000</span><span class="p">,</span> <span class="ss">max_no_files:</span> <span class="mi">5</span><span class="p">,</span> <span class="ss">compress_on_rotate:</span> <span class="no">true</span> <span class="p">]</span> </code></pre></div></div> <p>Finally, the previous Logger Backends API is now soft-deprecated. If you implement your own backends, you want to consider migrating to <a href="https://github.com/elixir-lang/logger_backends"><code class="language-plaintext highlighter-rouge">:logger_backends</code></a> in the long term. See the new <a href="https://hexdocs.pm/logger"><code class="language-plaintext highlighter-rouge">Logger</code></a> documentation for more information on the new features and compatibility.</p> <h2 id="learn-more">Learn more</h2> <p>For a complete list of all changes, see the <a href="https://github.com/elixir-lang/elixir/releases/tag/v1.15.0">full release notes</a>.</p> <p>Check <a href="/install.html">the Install section</a> to get Elixir installed and read our <a href="https://hexdocs.pm/elixir/introduction.html">Getting Started guide</a> to learn more.</p> <p>Happy compiling!</p> Hugo Baraúna 2023-03-09T00:00:00+00:00 /blog/2023/03/09/embedded-and-cloud-elixir-at-sparkmeter <p><em>Welcome to our series of case studies about companies using Elixir in production. <a href="https://elixir-lang.org/cases.html">See all</a> cases we have published so far.</em></p> <p><a href="https://www.sparkmeter.io/">SparkMeter</a> is a company on a mission to increase access to electricity. They offer grid-management solutions that enable utilities in emerging markets to run financially-sustainable efficient, and reliable systems.</p> <p>Elixir has played an important role in simplifying SparkMeter systems by providing a unified developer experience across their products. Elixir’s versatility in different domains, such as embedded software, data processing, and HTTP APIs, proved to be a valuable asset to a team who aims to release robust products quickly and confidently.</p> <p>Two of their products are smart electrical meters and grid-management software. These can be used to measure electricity usage, gather health information about an electrical grid, and manage billing.</p> <p>Here’s an overview of their architecture:</p> <p><img src="/images/cases/bg/sparkmeter-old-architecture.png" alt="SparkMeter architecture generation one" /></p> <p>The meters are embedded devices responsible for collecting measures such as electricity usage. They communicate with each other via a mesh network and also communicate with the grid edge management unit. The grid edge management unit is an embedded system that receives and processes data from up to thousands of meters. The grid edge management unit also communicates with servers running in the cloud. Those servers send and receive data to the grid edge management units and process it for use by internal systems and user-facing software.</p> <h2 id="the-challenge">The challenge</h2> <p>The infrastructure in which their embedded devices are deployed is not reliable. The cellular network used for communication between the ground and the cloud could fail, and the electricity supply to the embedded systems could go down. Therefore, their system needed to be fault-tolerant, and they needed to build equipment that didn’t require constant field maintenance.</p> <p>In light of these requirements, they identified areas for improvement in the first generation of their product. One of the things they needed to improve was the development of a new grid edge management unit. Additionally, their product was mission-critical, so they wanted a technology they could confidently put into production and one that would not take more than a year of development and QA before releasing a new generation of their product.</p> <p>That’s when they discovered Elixir and Nerves.</p> <h2 id="the-trade-offs-of-adopting-elixir-and-nerves">The trade-offs of adopting Elixir and Nerves</h2> <p>Nerves is an open-source platform that combines the Erlang virtual machine and Elixir ecosystem to build and deploy embedded systems.</p> <p>When considering the adoption of Elixir and <a href="https://nerves-project.org/">Nerves</a>, SparkMeter recognized many advantages the technologies offered.</p> <p>Elixir helped them meet the requirement of building a distributed and fault-tolerant system. That’s because Elixir leverages the power of the Erlang VM and the OTP framework, which were designed with that requirement in mind.</p> <p>Regarding Nerves, they saw it as an entire ecosystem for doing embedded development with many advantages. For example, it has a good story for doing local development and going from that to <a href="https://www.nerves-hub.org/">deploying</a> on an embedded device. It makes it easy to connect to an embedded device for iterative development. And it also enables fine-grained control of system boot, so they can handle scenarios when certain parts of the system won’t start.</p> <p>That said, they had two concerns, the growth of Nerves and finding talent with expertise in the Elixir/Nerves stack.</p> <p>They wanted to ensure that Nerves would continue to grow. But they realized that even if it didn’t, the benefits Nerves was already offering could give them a lot of leverage. Here are’s what their senior VP of engineering, Jon Thacker, had to say about that:</p> <blockquote> <p>Without Nerves, we would be on our own to figure out a lot. How to do distribution, the development environment, and how to support different architectures. So it really is a batteries-included framework for doing production-grade embedded systems.</p> <p>- <em>Jon Thacker, Senior VP of Engineering</em></p> </blockquote> <p>When we interviewed Jon for this case study, they had already been using Elixir and Nerves for more than two years. And with the benefit of hindsight, here’s what he said about adopting Nerves:</p> <blockquote> <p>Making sure that Nerves continued to grow was a concern. But it has done so and is showing a very positive trajectory. It was a calculated risk and, as it turns out, it was the correct choice.</p> <p>- <em>Jon Thacker, Senior VP of Engineering</em></p> </blockquote> <p>When it came to finding talent, they approached the problem in two ways. First, they started to build the pilot with a contractor to ensure that the staffing risk didn’t affect their timeline. But they also wanted to have an internal team to take ownership of the product in the long term. So, shortly after finishing the first version of the new system, they hired two engineers with experience in Elixir, Michael Waud and Benjamin Milde.</p> <p>Besides hiring people with previous experience in Elixir, Jon noticed that training their embedded engineers in Elixir was also a viable option. Here’s what he told us about that:</p> <blockquote> <p>I’m traditionally an embedded engineer, and I only learned Elixir as part of this project. However, transferring my mental model was so easy that I do believe that we would be capable of training other embedded engineers as well.</p> <p>- <em>Jon Thacker, Senior VP of Engineering</em></p> </blockquote> <h2 id="the-new-system">The new system</h2> <p>SparkMeter used Elixir for the ground (embedded) and cloud aspects of the new system they built. Here is an overview of the architecture:</p> <p><img src="/images/cases/bg/sparkmeter-new-architecture.png" alt="SparkMeter architecture generation two" /></p> <p>For the firmware of the grid edge management unit, they used Nerves. For the hardware, they built on top of a BeagleBone Black device.</p> <p>The communication between the grid edge management unit and the meters was via radio, using Rust to manage the radio hardware module inside the grid edge management unit. They used <a href="https://hexdocs.pm/elixir/1.13.4/Port.html">Elixir Ports</a> to communicate with Rust and process the data from the meters.</p> <p>Elixir was also used for communication with the cloud servers via 3G or Edge. This communication required bandwidth usage optimization due to the cost of sending large volumes of data through the cellular network. They evaluated various solutions like REST, CoAP, MQTT, Kafka, and Websockets. Still, none fit their specific needs, so they created a custom protocol tailored to their use case, which involved designing a binary protocol and implementing a TCP server. Mike Waud discussed this in more detail in his talks at <a href="https://www.youtube.com/watch?v=DJRL86mO4ks">ElixirConf 2021</a> and <a href="https://www.youtube.com/watch?v=BxTIUvyZHKw">2022</a>.</p> <p>The grid edge management unit also required a local web user interface that could be accessed on-site via Wi-Fi. For this, they used Phoenix and Liveview.</p> <p>The cloud aspect of the system is responsible for receiving data from the grid edge management units and sending control commands. It also runs a TCP server with their custom protocol, implemented in Elixir. The data received from the grid edge management units is stored in PostgreSQL and then consumed by a <a href="https://elixir-broadway.org/">Broadway-based</a> data pipeline.</p> <p>The cloud system also exposes an HTTP API implemented with Phoenix. This API is consumed by other internal systems to interact with their PostgreSQL database.</p> <h2 id="reaping-the-benefits">Reaping the benefits</h2> <p>During and after the development of the new generation of their system, SparkMeter observed many benefits.</p> <p>One of them was the reduction of the complexity of the grid edge management unit. The old version had more moving parts, using Ubuntu and Docker for the system level, Python/Celery and RabbitMQ for asynchronous processing, and Systemd for managing starting job processes.</p> <p>In the new version, they replaced all of that mainly with Elixir and Nerves. And for the parts where they needed tools that were not part of the BEAM stack, they could manage them like any other BEAM process by using <a href="https://hexdocs.pm/elixir/1.13.4/Port.html">Elixir Ports</a>. Here’s what they said about that experience:</p> <blockquote> <p>The new grid edge management unit has a very unified architecture. We can treat everything as an (Elixir) process. We have full control over the start and stop within a single ecosystem. It’s just a very coherent storyline.</p> <p>- <em>Jon Thacker, Senior VP Of Engineering</em></p> </blockquote> <p>Another aspect they liked about Nerves was that it included security best practices. For example, they used SSL certificates on the client and the server side for communication between the ground and the cloud. Nerves made this easy through the <a href="https://github.com/nerves-hub/nerves_key">NervesKey component</a>, which enables the use of a hardware security module to protect the private key. Nerves also made it easy to keep up with system security patches, as the firmware generated by Nerves is a single bundle containing a minimal Linux platform and their application packaged as a <a href="https://hexdocs.pm/mix/Mix.Tasks.Release.html">release</a>. Here’s what they said about security in Nerves:</p> <blockquote> <p>It’s easy enough to keep tracking upstream changes, so we’re not getting behind the latest security patches. Nerves made that easy. Nerves just pushed us towards a good security model.</p> <p>- <em>Jon Thacker, Senior VP Of Engineering</em></p> </blockquote> <p>The communication between the ground and the cloud involved implementing a custom TCP server running in both parts of the system. Network programming is not an everyday task for many application developers, but Elixir helped them a lot with that:</p> <blockquote> <p>I had never written a TCP client or a server before, it’s just not something you even think about. But doing it in Elixir, particularly on the protocol level of sending binaries, was a pleasure to work with! Something that would be super tedious in an imperative language, with Elixir and pattern matching, is so clear!</p> <p>- <em>Michael Waud, Senior Software Engineer</em></p> </blockquote> <p>Another benefit they received from using Elixir on the ground and in the cloud was code reuse. For example, the encoding and decoding of their custom protocol were reused for both the embedded and cloud parts.</p> <blockquote> <p>It would’ve been a much larger challenge if we hadn’t been running Elixir in the cloud and on the grid edge management unit because we could write it once. The encoding and decoding we wrote once, we gained a lot from being able to share code.</p> <p>- <em>Michael Waud, Senior Software Engineer</em></p> </blockquote> <p>Michael also pointed out that by controlling the complete connection from the grid edge management unit up to the cloud, they could reduce bandwidth usage and improve resiliency, which were essential requirements for them.</p> <p>Finally, the new generation of their system also enabled them to release more often. Before, they were releasing new versions every quarter, but with the new system, they could release weekly when needed.</p> <h2 id="summing-up">Summing up</h2> <p>In conclusion, SparkMeter’s adoption of Elixir and Nerves has led to many benefits for their mission-critical grid-management system.</p> <p>Elixir was used to design elegant solutions across data processing, HTTP APIs, and within the embedded space. This unified development model led to a more productive and robust environment, with less complexity and fewer moving parts.</p> <p>Additionally, the ability to control the entire connection from the ground to the cloud resulted in reduced bandwidth usage and improved resiliency. This fulfills essential requirements, given the diversity of conditions and locations the grid edge management unit may be deployed at.</p> <p>The new system also allowed for more frequent releases, enabling SparkMeter to respond quickly to their business needs.</p> Hugo Baraúna 2022-12-22T00:00:00+00:00 /blog/2022/12/22/cheatsheets-and-8-other-features-in-exdoc-that-improve-the-developer-experience <p>ExDoc has a cool new feature, <a href="https://hexdocs.pm/ex_doc/cheatsheet.html">cheatsheets</a>!</p> <p>In this blog post, we’ll explain what that new feature is and the motivation behind it. We’ll also take the opportunity to highlight other ExDoc features that show how it has been evolving to make the documentation experience in Elixir better and better.</p> <h2 id="what-are-exdoc-cheatsheets-and-how-they-improve-the-documentation-experience">What are ExDoc cheatsheets and how they improve the documentation experience</h2> <p>ExDoc’s cheatsheets are Markdown files with the <code class="language-plaintext highlighter-rouge">.cheatmd</code> extension. You can see <a href="https://hexdocs.pm/ecto/crud.html">an example</a> of how the Ecto project is using them.</p> <p>Writing and reading cheatsheets is not exactly new to developers. What ExDoc brings to the table is the possibility of integrating cheatsheets alongside the rest of the documentation of an Elixir project, instead of hosting them in a different place.</p> <p>Developers need different kinds of documentation at different times. When one is learning about a new library, a guide format is proper. When one needs to know if a library can solve a specific problem, an API reference can be more appropriate. When someone wants to remember a couple of functions they already used from that library, a cheatsheet could be more practical.</p> <p>Imagine if you had to go to a different place for every type of documentation you’re looking for. That would make a very fragmented experience, not only for readers of documentation but also for writers.</p> <p>ExDoc cheatsheets represent one step further in the direction of making documentation in Elixir an even more comprehensive and integrated experience.</p> <p>ExDoc cheatsheets are inspired by <a href="https://devhints.io">devhints.io</a> from <a href="https://twitter.com/rstacruz">Rico Sta. Cruz</a>, and were contributed by <a href="https://twitter.com/paulovalim">Paulo Valim</a> and <a href="https://twitter.com/alchemist_ubi">Yordis Prieto</a>.</p> <h2 id="eight-features-that-show-how-exdoc-has-improved-developer-experience-over-time">Eight features that show how ExDoc has improved developer experience over time</h2> <p>We added cheatsheets to ExDoc because we value developer experience and believe documentation is a core aspect of it.</p> <p>Since the beginning, one of Elixir’s principles is that documentation should be a first-class citizen. What this idea means to us is that documentation should be easy to write and easy to read. ExDoc has been continuously evolving over the years, guided by this principle.</p> <p>Here are some of the features added to ExDoc over the years that make reading and writing documentation in Elixir a joy.</p> <h3 id="beautiful-and-usable-design">Beautiful and usable design</h3> <p>As developers, we may not have the skill to make beautifully designed UIs. That doesn’t mean we don’t appreciate it. Here’s what documentation generated with ExDoc looked like almost ten years ago, with its original layout based on <a href="https://yardoc.org/">YARD</a>:</p> <p><img src="https://i.imgur.com/O9xKjR8.jpg" alt="Screenshot of the Phoenix v0.5.0 documentation generated with an early version of ExDoc" /></p> <p>Here’s what it looks like today:</p> <p><img src="https://i.imgur.com/ZKI1T23.png" alt="Screenshot of the Phoenix v1.6.15 documentation generated with current ExDoc" /></p> <p>The evolution of ExDoc’s design helped documentation be more visually appealing and easier to read and navigate.</p> <h3 id="links-to-source-code">Links to source code</h3> <p>Sometimes you’re reading the documentation of a library, and you want to know more about the implementation of a function. Or, you found something in the documentation that could be improved and wants to help. In those situations, it’s helpful to go from the documentation to the source code. ExDoc makes that dead easy. For every module, function, or page, ExDoc gives you a link that you can click to go directly to the project’s source code on GitHub:</p> <p><img src="https://i.imgur.com/PXvoeDk.gif" alt="Short screencast of a user clicking on the &quot;link to source code&quot; button on the documentation for a function" /></p> <h3 id="guides">Guides</h3> <p>One of the most common formats of library documentation is an API reference. But depending on your needs, that’s not the most approachable format. For example, it’s not optimal when you’re just getting started with a library or when you want to learn how to solve a specific problem using it. That’s why ExDoc allows writing other types of docs besides API references, like <em>“Getting started” guides</em> or <em>How-tos</em>.</p> <p>Look at how <a href="https://hexdocs.pm/ecto/getting-started.html">Ecto’s documentation</a> uses that, for example:</p> <p><img src="https://i.imgur.com/KInZb4x.gif" alt="Screencast of a user exploring the guides in the Ecto documentation" /></p> <h3 id="custom-grouping-of-modules-functions-and-pages-in-the-sidebar">Custom grouping of modules, functions, and pages in the sidebar</h3> <p>Sometimes your library has dozens of modules. Sometimes, one given module has a large API surface area. In those situations, showing the list of functions as a single large list may not be the most digestible way to be consumed. For those cases, ExDoc allows modules, functions, or extra pages to be grouped in the sidebar in a way that makes more sense semantically.</p> <p>Here’s an example of how Ecto use grouped functions for its <code class="language-plaintext highlighter-rouge">Repo</code> module:</p> <p><img src="https://i.imgur.com/ZE7N312.png" alt="Screenshot of the sidebar of the Ecto documentation, showing grouped functions in the `Ecto.Repo` module" /></p> <p>Instead of listing the ~40 functions of <code class="language-plaintext highlighter-rouge">Ecto.Repo</code> as a single extensive list, it presents them grouped by five cohesive topics:</p> <ul> <li>Query API</li> <li>Schema API</li> <li>Transaction API</li> <li>Runtime API</li> <li>User callbacks</li> </ul> <p>The same functionality is available for modules and pages (guides, how-tos, and so on). Phoenix is a <a href="https://hexdocs.pm/phoenix/overview.html">good example</a> of how that’s used.</p> <h3 id="full-text-search">Full-text search</h3> <p>Sometimes you don’t know or don’t remember the name of the function that you’re looking for. For example, let’s say you’re looking for a function for dealing with file system directories.</p> <p>Although there’s no function or module called “directory” in Elixir, when you type “directory” in <a href="https://hexdocs.pm/elixir/search.html?q=directory">Elixir’s documentation</a>, it will return all the entries that have the word “directory” inside the documentation. It will even return entries with variations of the word “directory”, like “directories”, doing a fuzzy search.</p> <p><img src="https://i.imgur.com/IHHuej8.png" alt="Screenshot of the result of searching for &quot;directory&quot; in the Elixir documentation" /></p> <p>The search bar also supports autocompletion for module and function names:</p> <p><img src="https://i.imgur.com/2cmsuDi.gif" alt="Screencast of a user typing the word &quot;Enum&quot; in the search bar of Elixir's documentation and letting it autocomplete the module. Then, the user types &quot;Range&quot; and both modules and functions show up." /></p> <p>The best part is that full-text search is fully implemented on the client-side, which means ExDoc pages can be fully hosted as static websites (for example on GitHub Pages).</p> <h3 id="keyboard-shortcuts-to-navigate-to-docs-of-other-hex-packages">Keyboard shortcuts to navigate to docs of other Hex packages</h3> <p>It’s common for an application to have dependencies. While coding, we usually need to read the documentation of more than one of those dependencies.</p> <p>One solution is to keep a window open for the documentation of each dependency. However, ExDoc offers another option: a keyboard shortcut to search and go to another package documentation within the same window.</p> <p>Here’s what it looks like:</p> <p><img src="https://i.imgur.com/I9uJxUF.gif" alt="Screencast of a user enabling the `g` shortcut to search through dependencies documentation and then using it to search for &quot;phoenix_live&quot; in the documentation for Nerves." /></p> <p>There are more keyboard shortcuts to help you navigate within and between documentation:</p> <p><img src="https://i.imgur.com/qdoNUx9.png" alt="Screenshot of the keyboard shortcuts that you can enable in ExDoc" /></p> <h3 id="a-version-dropdown-to-switch-to-other-versions">A version dropdown to switch to other versions</h3> <p>Keeping our application updated with the latest versions of all its dependencies can be challenging. So, it’s common to need to look at the documentation of an older version of a library we’re using. ExDoc makes it simple to do that.</p> <p>When you access the documentation of a project, there’s a dropdown that you can use to select the version you’re looking for:</p> <p><img src="https://i.imgur.com/1krcY5g.gif" alt="Screencast of a user typing the version dropdown under the application name in the &quot;timex&quot; documentation, revealing all the versions." /></p> <h3 id="livebook-integration">Livebook integration</h3> <p><a href="https://livebook.dev/">Livebook</a> is a web application for writing interactive and collaborative code notebooks in Elixir.</p> <p>One of the ways Elixir developers have been using Livebook is for documentation. Because of its interactivity capabilities, it enables the reader to play with the code right inside the documentation, which makes it great for tutorials and augmenting the user experience.</p> <p>With that in mind, ExDoc offers the possibility of integrating Livebook notebooks. That means one can host Livebook-based documentation together with the API reference.</p> <p>Here’s an <a href="https://hexdocs.pm/req_sandbox/usage.html">example of using Livebook inside ExDoc for writing a Usage Guide</a>:</p> <p><img src="https://i.imgur.com/FxOLs0Y.gif" alt="Screencast of a user navigating through the &quot;req_sandbox&quot; documentation, finding a Livebook, clicking &quot;Run in Livebook&quot;, and using the Livebook that opens up on their local machine." /></p> <h3 id="bonus-erlang-support">Bonus: Erlang support</h3> <p><a href="https://www.erlang.org/eeps/eep-0048">EEP 48</a> proposed a standardized way for how BEAM languages could store API documentation. This allows any BEAM language to read documentation generated by each other.</p> <p>By leveraging that work, ExDoc can generate documentation for an Erlang project. For example, Telemetry is a library written in Erlang that has <a href="https://hexdocs.pm/telemetry/readme.html">its documentation</a> generated with ExDoc.</p> <p><img src="https://i.imgur.com/C4Idbuh.png" alt="Screenshot of &quot;telemetry&quot; documentation generated with ExDoc" /></p> <p>By using ExDoc to also generate documentation for Erlang-based projects, we can have more consistency in the user experience along the BEAM ecosystem. See the great <a href="https://hexdocs.pm/rebar3_ex_doc/"><code class="language-plaintext highlighter-rouge">rebar3_ex_doc</code></a> plugin to get started.</p> <h3 id="bonus-doctests">Bonus: Doctests</h3> <p>When writing documentation, it’s helpful to offer code examples. For instance, here’s the documentation of the <code class="language-plaintext highlighter-rouge">Enum.any?/1</code> function from Elixir’s standard library:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">@doc</span> <span class="sd">""" Returns `true` if at least one element in `enumerable` is truthy. When an element has a truthy value (neither `false` nor `nil`) iteration stops immediately and `true` is returned. In all other cases `false` is returned. ## Examples iex&gt; Enum.any?([false, false, false]) false iex&gt; Enum.any?([false, true, false]) true iex&gt; Enum.any?([]) false """</span> </code></pre></div></div> <p>To ensure examples do not get out of date, Elixir’s test framework ExUnit provides a feature called <strong>doctests</strong>. This allows developers to test the examples in their documentation. Doctests work by parsing out code samples starting with <code class="language-plaintext highlighter-rouge">iex&gt;</code> from the documentation.</p> <p>Although this is not a feature of ExDoc, it is an essential part of Elixir’s developer and documentation experience.</p> <h2 id="wrap-up">Wrap up</h2> <p>As we saw, ExDoc has evolved a lot throughout the years! As it continues to evolve into a more and more comprehensive documentation tool, we want to enable developers to keep investing more time writing the documentation itself instead of needing to spend time building custom documentation tools and websites. The best part is that all you need to do to leverage many of those features is to simply document your code using the <code class="language-plaintext highlighter-rouge">@doc</code> attribute!</p> <p>Here’s to a continuously improving documentation experience for the next years.</p> José Valim 2022-10-05T00:00:00+00:00 /blog/2022/10/05/my-future-with-elixir-set-theoretic-types <p><em>This is article contains excerpts from my keynotes at <a href="https://www.youtube.com/watch?v=Jf5Hsa1KOc8">ElixirConf Europe 2022</a> and <a href="https://www.youtube.com/watch?v=KmLw58qEtuM">ElixirConf US 2022</a>.</em></p> <p>In May 2022, we have celebrated 10 years since Elixir v0.5, the first public release of Elixir, was announced.</p> <p>At such occasions, it may be tempting to try to predict how Elixir will look in 10 years from now. However, I believe that would be a futile effort, because, 10 years ago, I would never have guessed Elixir would have gone <a href="https://phoenixframework.org/">beyond excelling at web development</a>, but also into domains such as <a href="https://www.nerves-project.org/">embedded software</a> and making inroads into machine learning and data analysis with projects such as <a href="https://github.com/elixir-nx/nx">Nx (Numerical Elixir)</a>, <a href="https://github.com/elixir-nx/explorer">Explorer</a>, <a href="https://github.com/elixir-nx/axon">Axon</a>, and <a href="https://livebook.dev/">Livebook</a> (<a href="https://github.com/elixir-nx/">here is a summary of the main Numerical Elixir projects</a>. Elixir was designed to be extensible and how it will be extended has always been a community effort.</p> <p>For these reasons, I choose to focus on <em>My Future</em> with Elixir. Those are the projects I am personally excited about and working on alongside other community members. The topic of today’s article is type systems, as discussed in my ElixirConf EU presentation in May 2022.</p> <h2 id="the-elephant-in-the-room-types">The elephant in the room: types</h2> <p>Throughout the years, the Elixir Core Team has addressed the biggest needs of the community. <a href="https://elixir-lang.org/blog/2018/01/17/elixir-v1-6-0-released/">Elixir v1.6 introduced the Elixir code formatter</a>, as the growing community and large teams saw an increased need for style guides and conventions around large codebases.</p> <p><a href="https://elixir-lang.org/blog/2019/06/24/elixir-v1-9-0-released/">Elixir v1.9 shipped with built-in support for releases</a>: self-contained archives that consist of your application code, all of its dependencies, plus the whole Erlang Virtual Machine (VM) and runtime. The goal was to address the perceived difficulty in deploying Elixir projects, by bringing tried approaches from both Elixir and Erlang communities into the official tooling. This paved the way to future automation, such as <code class="language-plaintext highlighter-rouge">mix phx.gen.release</code>, which automatically generates a Dockerfile tailored to your Phoenix applications.</p> <p>Given our relationship with the community, it would be disingenuous to talk about my future with Elixir without addressing what seems to be the biggest community need nowadays: static typing. However, when the community asks for static typing, what are we effectively expecting? And what is the Elixir community to gain from it?</p> <h2 id="types-and-elixir">Types and Elixir</h2> <p>Different programming languages and platforms extract different values from types. These values may or may not apply to Elixir.</p> <p>For example, different languages can extract performance benefits from types. However, Elixir still runs on the Erlang VM, which is dynamically typed, so we should not expect any meaningful performance gain from typing Elixir code.</p> <p>Another benefit of types is to <em>aid</em> documentation (emphasis on the word <em>aid</em> as I don’t believe types replace textual documentation). Elixir already reaps similar benefits from <a href="https://hexdocs.pm/elixir/typespecs.html">typespecs</a> and I would expect an integrated type system to be even more valuable in this area.</p> <p>However, the upsides and downsides of static typing become fuzzier and prone to exaggerations once we discuss them in the context of code maintenance, in particular when comparing types with other software verification techniques, such as tests. In those situations, it is common to hear unrealistic claims such as “a static type system would catch 80% of my Elixir bugs” or that “you need to write fewer tests once you have static types”.</p> <p>While <a href="https://www.youtube.com/watch?v=Jf5Hsa1KOc8">I explore why I don’t believe those claims are true during the keynote</a>, saying a static type system helps catch bugs is not helpful unless we discuss exactly the type of bugs it is supposed to identify, and that’s what we should focus on.</p> <p>For example, Rust’s type system helps prevent bugs such as deallocating memory twice, dangling pointers, data races in threads, and more. But adding such type system to Elixir would be unproductive because those are not bugs that we run into in the first place, as those properties are guaranteed by the garbage collector and the Erlang runtime.</p> <p>This brings another discussion point: a type system naturally restricts the amount of code we can write because, in order to prove certain properties about our code, certain styles have to be rejected. However, I would prefer to avoid restricting the expressive power of Elixir, because I am honestly quite happy with the language semantics (which we mostly inherited from Erlang).</p> <p>For Elixir, the benefit of a type system would revolve mostly around contracts. If function <code class="language-plaintext highlighter-rouge">caller(arg)</code> calls a function named <code class="language-plaintext highlighter-rouge">callee(arg)</code>, we want to guarantee that, as both these functions change over time, that <code class="language-plaintext highlighter-rouge">caller</code> is passing valid arguments into <code class="language-plaintext highlighter-rouge">callee</code> and that the <code class="language-plaintext highlighter-rouge">caller</code> properly handles the return types from <code class="language-plaintext highlighter-rouge">callee</code>.</p> <p>This may seem like a simple guarantee to provide, but we’d run into tricky scenarios even on small code samples. For example, imagine that we define a <code class="language-plaintext highlighter-rouge">negate</code> function, that negates numbers. One may implement it like this:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="n">negate</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="ow">when</span> <span class="n">is_integer</span><span class="p">(</span><span class="n">x</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="o">-</span><span class="n">x</span> </code></pre></div></div> <p>We could then say <code class="language-plaintext highlighter-rouge">negate</code> has the type <code class="language-plaintext highlighter-rouge">integer() -&gt; integer()</code>.</p> <p>With our custom negation in hand, we can implement a custom subtraction:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="n">subtract</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">)</span> <span class="ow">when</span> <span class="n">is_integer</span><span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="ow">and</span> <span class="n">is_integer</span><span class="p">(</span><span class="n">b</span><span class="p">)</span> <span class="k">do</span> <span class="n">a</span> <span class="o">+</span> <span class="n">negate</span><span class="p">(</span><span class="n">b</span><span class="p">)</span> <span class="k">end</span> </code></pre></div></div> <p>This would all work and typecheck as expected, as we are only working with integers. However, imagine in the future someone decides to make <code class="language-plaintext highlighter-rouge">negate</code> polymorphic, so it also negates booleans:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="n">negate</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="ow">when</span> <span class="n">is_integer</span><span class="p">(</span><span class="n">x</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="o">-</span><span class="n">x</span> <span class="k">def</span> <span class="n">negate</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="ow">when</span> <span class="n">is_boolean</span><span class="p">(</span><span class="n">x</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="ow">not</span> <span class="n">x</span> </code></pre></div></div> <p>If we were to naively say that <code class="language-plaintext highlighter-rouge">negate</code> now has the type <code class="language-plaintext highlighter-rouge">integer() | boolean() -&gt; integer() | boolean()</code>, we would now get a false positive warning in our implementation of subtract:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="no">Type</span> <span class="ss">warning:</span> <span class="o">|</span> <span class="o">|</span> <span class="k">def</span> <span class="n">subtract</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">)</span> <span class="ow">when</span> <span class="n">is_integer</span><span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="ow">and</span> <span class="n">is_integer</span><span class="p">(</span><span class="n">b</span><span class="p">)</span> <span class="k">do</span> <span class="o">|</span> <span class="n">a</span> <span class="o">+</span> <span class="n">negate</span><span class="p">(</span><span class="n">b</span><span class="p">)</span> <span class="o">^</span> <span class="n">the</span> <span class="n">operator</span> <span class="o">+</span> <span class="n">expects</span> <span class="n">integer</span><span class="p">(),</span> <span class="n">integer</span><span class="p">()</span> <span class="n">as</span> <span class="n">arguments</span><span class="p">,</span> <span class="n">but</span> <span class="n">the</span> <span class="n">second</span> <span class="n">argument</span> <span class="n">can</span> <span class="n">be</span> <span class="n">integer</span><span class="p">()</span> <span class="o">|</span> <span class="n">boolean</span><span class="p">()</span> </code></pre></div></div> <p><strong>So we want a type system that can type contracts between functions but, at the same time, avoids false positives and does not restrict the Elixir language</strong>. Balancing those trade-offs is not only a technical challenge but also one that needs to consider the needs of the community. The <a href="https://www.erlang.org/doc/man/dialyzer.html">Dialyzer project</a>, implemented in Erlang and available for Elixir projects, chose to have no false positives. However, that implies certain bugs may not be caught.</p> <p>At this point in time, it seems the overall community would prefer a system that flags more potential bugs, even if it means more false positives. This may be particularly tricky in the context of Elixir and Erlang because I like to describe them as <a href="https://dashbit.co/blog/writing-assertive-code-with-elixir"><em>assertive languages</em></a>: we write code that will crash in face of unexpected scenarios because we rely on supervisors to restart parts of our application whenever that happens. This is the foundation of building self-healing and fault-tolerant systems in those languages.</p> <p>On the other hand, this is what makes a type system for Erlang/Elixir so exciting and unique: the ability to deal with failure modes both at compile-time and runtime elegantly. Because at the end of the day, regardless of the type system of your choice, you will run into unexpected scenarios, especially when interacting with external resources such as the filesystem, APIs, distributed nodes, etc.</p> <h2 id="the-big-announcement">The big announcement</h2> <p>This brings me to the big announcement from ElixirConf EU 2022: <strong>we have an on-going PhD scholarship to research and develop a type system for Elixir based on set-theoretic types</strong>. Guillaume Duboc (PhD student) is the recipient of the scholarship, lead by Giuseppe Castagna (Senior Resercher) with support from José Valim (that’s me).</p> <p>The scholarship is a partnership between the <a href="https://www.cnrs.fr/">CNRS</a> and <a href="https://remote.com/">Remote</a>. It is sponsored by Supabase (<a href="https://supabase.com/company">they are hiring!</a>), Fresha (<a href="https://www.fresha.com/careers/openings?department=engineering">they are hiring!</a>), and <a href="https://dashbit.co/">Dashbit</a>, all heavily invested in Elixir’s future.</p> <h2 id="why-set-theoretic-types">Why set-theoretic types?</h2> <p>We want a type system that can elegantly model all of Elixir idioms and, at a first glance, set-theoretic types were an excellent match. In set-theoretic types, we use set operations to define types and ensure that the types satisfy the associativity and distributivity properties of the corresponding set-theoretic operations.</p> <p>For example, numbers in Elixir can be integers <em>or</em> floats, therefore we can write them as the union <code class="language-plaintext highlighter-rouge">integer() | float()</code> (which is equivalent to <code class="language-plaintext highlighter-rouge">float() | integer()</code>).</p> <p>Remember the <code class="language-plaintext highlighter-rouge">negate</code> function we wrote above?</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="n">negate</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="ow">when</span> <span class="n">is_integer</span><span class="p">(</span><span class="n">x</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="o">-</span><span class="n">x</span> <span class="k">def</span> <span class="n">negate</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="ow">when</span> <span class="n">is_boolean</span><span class="p">(</span><span class="n">x</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="ow">not</span> <span class="n">x</span> </code></pre></div></div> <p>We could think of it as a function that has both types <code class="language-plaintext highlighter-rouge">(integer() -&gt; integer())</code> and <code class="language-plaintext highlighter-rouge">(boolean() -&gt; boolean())</code>, which is as an intersection. This would naturally solve the problem described in the previous section: when called with an integer, it can only return an integer.</p> <p>We also have a data-structure called atoms in Elixir. They uniquely represent a value which is given by their own name. Such as <code class="language-plaintext highlighter-rouge">:sunday</code> or <code class="language-plaintext highlighter-rouge">:banana</code>. You can think of the type <code class="language-plaintext highlighter-rouge">atom()</code> as the set of all atoms. In addition, we can think of the values <code class="language-plaintext highlighter-rouge">:sunday</code> and <code class="language-plaintext highlighter-rouge">:banana</code> as subtypes of <code class="language-plaintext highlighter-rouge">atom()</code>, as they are contained in the set of all atoms. <code class="language-plaintext highlighter-rouge">:sunday</code> and <code class="language-plaintext highlighter-rouge">:banana</code> are also known as singleton types (as they are made up of only one value).</p> <p>In fact, we could even consider each integer to be a singleton type that belongs to the <code class="language-plaintext highlighter-rouge">integer()</code> set. The choice of which values will become singletons in our type system will strongly depend on the trade-offs we defined in the previous sections. The type system also has to be gradual, as typed Elixir code may interact with untyped Elixir code and vice-versa.</p> <p>Personally, I find set-theoretical types an elegant and accessible approach to reason about types. At the end of the day, an Elixir developer won’t have to think about intersections when writing a function with multiple clauses, but the modelling is straight-forward if they are ever to look under the hood.</p> <p>Despite the initial fit between Elixir semantics and set-theoretic types, there are open questions and existing challenges in putting the two together. Here are some examples:</p> <ul> <li> <p>Elixir has <a href="https://hexdocs.pm/elixir/patterns-and-guards.html">an expressive collection of idioms used in pattern matching and guards</a>, can we map them all to set-theoretic types?</p> </li> <li> <p>Elixir associative data structures, <a href="https://hexdocs.pm/elixir/Map.html">called maps</a>, can be used both as records and as dictionaries. Would it be possible to also type them with a unified foundation?</p> </li> <li> <p>Gradual type systems must introduce runtime type checks in order to remain sound. However, those type checks will happen in addition to the checks already done by the Erlang VM, which can degrade performance. Therefore, is it possible to leverage the existing runtime checks done by the Erlang VM so the resulting type system is still sound?</p> </li> </ul> <p>Those challenges are precisely what makes me excited to work with Giuseppe Castagna and Guillaume Duboc, as we believe it is important to formalize those problems and their solutions, before we dig deep into the implementation. To get started with set-theoretic types, I recommend <a href="https://www.irif.fr/~gc/papers/set-theoretic-types-2022.pdf">Programming with union, intersection, and negation types by Giuseppe Castagna</a>.</p> <p>Finally, it is important to note there are areas we don’t plan to tackle at the moment, such as typing of messages between processes.</p> <h2 id="expectations-and-roadmap">Expectations and roadmap</h2> <p>At this point, you may be expecting that Elixir will certainly become a gradually typed language at some moment in its future. However, it is important to note this may not be the case, as there is a long road ahead of us.</p> <p>One of the challenges in implementing a type system - at least for someone who doesn’t have the relevant academic background like myself - is that it feels like a single indivisible step: you take a language without a type system and at the end you have one, without much insight or opportunity for feedback in the middle. Therefore we have been planning to incorporate the type system into Elixir in steps, which I have been referring to as “a gradual gradual type system”: one where we add gradual types to the language gradually.</p> <p>The first step, the one we are currently working on, is to leverage the existing type information found in Elixir programs. As previously mentioned, <a href="https://dashbit.co/blog/writing-assertive-code-with-elixir">we write assertive code</a> in Elixir, which means there is a lot of type information in patterns and guards. We want to lift this information and use it to type check existing codebases. The Erlang compiler already does so to improve performance within a single module and we want to eventually do so across modules and applications too.</p> <p>During this phase, Elixir developers won’t have to change a single line of code to leverage the benefits of the type system. Of course, we will catch only part of existing bugs, but this will allows us to stress test, benchmark, and collect feedback from developers, making improvements behind the scenes (or even revert the whole thing if we believe it won’t take us where we expect).</p> <p>The next step is to introduce typed structs into the language, allowing struct types to propagate throughout the system, as you pattern match on structs throughout the codebase. In this stage we will introduce a new API for defining structs, yet to be discussed, and developers will have to use the new API to reap its benefits.</p> <p>Finally, once we are happy with the improvements and the feedback collected, we can migrate to introduce a new syntax for typing function signatures in Elixir codebases, including support for more advanced features such as polymorphic types. Those will allow us to type complex constructs such as the ones found in the <code class="language-plaintext highlighter-rouge">Enum</code> module.</p> <p>The important point to keep in mind is that those features will be explored and developed in steps, with plenty of opportunity to gather community feedback. I also hope our experience may be useful to other ecosystems who wish to gradually introduce type systems into existing programming languages, in a way that feels granular and participative.</p> <p>Thank you for reading and see you in future updates.</p> Andrea Leopardi 2022-09-01T00:00:00+00:00 /blog/2022/09/01/elixir-v1-14-0-released <p>Elixir v1.14 has just been released. 🎉</p> <p>Let’s check out new features in this release. Like many of the past Elixir releases, this one has a strong focus on developer experience and developer happiness, through improvements to debugging, new debugging tools, and improvements to term inspection. Let’s take a quick tour.</p> <p>Note: this announcement contains <a href="https://asciinema.org">asciinema</a> snippets. You may need to enable 3rd-party JavaScript on this site in order to see them. If JavaScript is disabled, <code class="language-plaintext highlighter-rouge">noscript</code> tags with the proper links will be shown.</p> <h2 id="dbg"><code class="language-plaintext highlighter-rouge">dbg</code></h2> <p><a href="https://hexdocs.pm/elixir/Kernel.html#dbg/2"><code class="language-plaintext highlighter-rouge">Kernel.dbg/2</code></a> is a new macro that’s somewhat similar to <a href="https://hexdocs.pm/elixir/IO.html#inspect/2"><code class="language-plaintext highlighter-rouge">IO.inspect/2</code></a>, but specifically tailored for <strong>debugging</strong>.</p> <p>When called, it prints the value of whatever you pass to it, plus the debugged code itself as well as its location.</p> <script id="asciicast-510632" src="https://asciinema.org/a/510632.js" async=""></script> <noscript><p><a href="https://asciinema.org/a/510632">See the example in asciinema</a></p></noscript> <p><code class="language-plaintext highlighter-rouge">dbg/2</code> can do more. It’s a macro, so it <em>understands Elixir code</em>. You can see that when you pass a series of <code class="language-plaintext highlighter-rouge">|&gt;</code> pipes to it. <code class="language-plaintext highlighter-rouge">dbg/2</code> will print the value for every step of the pipeline.</p> <script id="asciicast-509506" src="https://asciinema.org/a/509506.js" async=""></script> <noscript><p><a href="https://asciinema.org/a/509506">See the example in asciinema</a></p></noscript> <h2 id="iex--dbg">IEx + dbg</h2> <p>Interactive Elixir (IEx) is Elixir’s shell (also known as REPL). In v1.14, we have improved IEx breakpoints to also allow line-by-line stepping:</p> <script id="asciicast-508048" src="https://asciinema.org/a/508048.js" async=""></script> <noscript><p><a href="https://asciinema.org/a/508048">See the example in asciinema</a></p></noscript> <p>We have also gone one step further and integrated this new functionality with <code class="language-plaintext highlighter-rouge">dbg/2</code>.</p> <p><code class="language-plaintext highlighter-rouge">dbg/2</code> supports <strong>configurable backends</strong>. IEx automatically replaces the default backend by one that halts the code execution with IEx:</p> <script id="asciicast-509507" src="https://asciinema.org/a/509507.js" async=""></script> <noscript><p><a href="https://asciinema.org/a/509507">See the example in asciinema</a></p></noscript> <p>We call this process “prying”, as you get access to variables and imports, but without the ability to change how the code actually executes.</p> <p>This also works with pipelines: if you pass a series of <code class="language-plaintext highlighter-rouge">|&gt;</code> pipe calls to <code class="language-plaintext highlighter-rouge">dbg</code> (or pipe into it at the end, like <code class="language-plaintext highlighter-rouge">|&gt; dbg()</code>), you’ll be able to step through every line in the pipeline.</p> <script id="asciicast-509509" src="https://asciinema.org/a/509509.js" async=""></script> <noscript><p><a href="https://asciinema.org/a/509509">See the example in asciinema</a></p></noscript> <p>You can keep the default behavior by passing the <code class="language-plaintext highlighter-rouge">--no-pry</code> option to IEx.</p> <h2 id="dbg-in-livebook">dbg in Livebook</h2> <p><a href="https://livebook.dev/">Livebook</a> brings the power of computation notebooks to Elixir.</p> <p>As an another example of the power behind <code class="language-plaintext highlighter-rouge">dbg</code>, the Livebook team has implemented a visual representation for <code class="language-plaintext highlighter-rouge">dbg</code> as a backend, where it prints each step of the pipeline as its distinct UI element. You can select an element to see its output or even re-order and disable sections of the pipeline on the fly:</p> <video src="https://user-images.githubusercontent.com/17034772/187455667-b166ce50-c440-444c-94dc-e2c0280a4924.webm" data-canonical-src="https://user-images.githubusercontent.com/17034772/187455667-b166ce50-c440-444c-94dc-e2c0280a4924.webm" controls="controls" muted="muted" style="max-height:640px;"></video> <h2 id="partitionsupervisor">PartitionSupervisor</h2> <p><a href="https://hexdocs.pm/elixir/PartitionSupervisor.html"><code class="language-plaintext highlighter-rouge">PartitionSupervisor</code></a> implements a new supervisor type. It is designed to help when you have a single supervised process that becomes a bottleneck. If that process’ state can be easily partitioned, then you can use <code class="language-plaintext highlighter-rouge">PartitionSupervisor</code> to supervise multiple isolated copies of that process running concurrently, each assigned its own partition.</p> <p>For example, imagine you have a <code class="language-plaintext highlighter-rouge">ErrorReporter</code> process that you use to report errors to a monitoring service.</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># Application supervisor:</span> <span class="n">children</span> <span class="o">=</span> <span class="p">[</span> <span class="c1"># ...,</span> <span class="no">ErrorReporter</span> <span class="p">]</span> <span class="no">Supervisor</span><span class="o">.</span><span class="n">start_link</span><span class="p">(</span><span class="n">children</span><span class="p">,</span> <span class="ss">strategy:</span> <span class="ss">:one_for_one</span><span class="p">)</span> </code></pre></div></div> <p>As the concurrency of your application goes up, the <code class="language-plaintext highlighter-rouge">ErrorReporter</code> process might receive requests from many other processes and eventually become a bottleneck. In a case like this, it could help to spin up multiple copies of the <code class="language-plaintext highlighter-rouge">ErrorReporter</code> process under a <code class="language-plaintext highlighter-rouge">PartitionSupervisor</code>.</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># Application supervisor</span> <span class="n">children</span> <span class="o">=</span> <span class="p">[</span> <span class="p">{</span><span class="no">PartitionSupervisor</span><span class="p">,</span> <span class="ss">child_spec:</span> <span class="no">ErrorReporter</span><span class="p">,</span> <span class="ss">name:</span> <span class="no">Reporters</span><span class="p">}</span> <span class="p">]</span> </code></pre></div></div> <p>The <code class="language-plaintext highlighter-rouge">PartitionSupervisor</code> will spin up a number of processes equal to <code class="language-plaintext highlighter-rouge">System.schedulers_online()</code> by default (most often one per core). Now, when routing requests to <code class="language-plaintext highlighter-rouge">ErrorReporter</code> processes we can use a <code class="language-plaintext highlighter-rouge">:via</code> tuple and route the requests through the partition supervisor.</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">partitioning_key</span> <span class="o">=</span> <span class="n">self</span><span class="p">()</span> <span class="no">ErrorReporter</span><span class="o">.</span><span class="n">report</span><span class="p">({</span><span class="ss">:via</span><span class="p">,</span> <span class="no">PartitionSupervisor</span><span class="p">,</span> <span class="p">{</span><span class="no">Reporters</span><span class="p">,</span> <span class="n">partitioning_key</span><span class="p">}},</span> <span class="n">error</span><span class="p">)</span> </code></pre></div></div> <p>Using <code class="language-plaintext highlighter-rouge">self()</code> as the partitioning key here means that the same process will always report errors to the same <code class="language-plaintext highlighter-rouge">ErrorReporter</code> process, ensuring a form of back-pressure. You can use any term as the partitioning key.</p> <h3 id="a-common-example">A common example</h3> <p>A common and practical example of a good use case for <code class="language-plaintext highlighter-rouge">PartitionSupervisor</code> is partitioning something like a <code class="language-plaintext highlighter-rouge">DynamicSupervisor</code>. When starting many processes under it, a dynamic supervisor can be a bottleneck, especially if said processes take a long time to initialize. Instead of starting a single <code class="language-plaintext highlighter-rouge">DynamicSupervisor</code>, you can start multiple:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">children</span> <span class="o">=</span> <span class="p">[</span> <span class="p">{</span><span class="no">PartitionSupervisor</span><span class="p">,</span> <span class="ss">child_spec:</span> <span class="no">DynamicSupervisor</span><span class="p">,</span> <span class="ss">name:</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">DynamicSupervisors</span><span class="p">}</span> <span class="p">]</span> <span class="no">Supervisor</span><span class="o">.</span><span class="n">start_link</span><span class="p">(</span><span class="n">children</span><span class="p">,</span> <span class="ss">strategy:</span> <span class="ss">:one_for_one</span><span class="p">)</span> </code></pre></div></div> <p>Now you start processes on the dynamic supervisor for the right partition. For instance, you can partition by PID, like in the previous example:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="no">DynamicSupervisor</span><span class="o">.</span><span class="n">start_child</span><span class="p">(</span> <span class="p">{</span><span class="ss">:via</span><span class="p">,</span> <span class="no">PartitionSupervisor</span><span class="p">,</span> <span class="p">{</span><span class="no">MyApp</span><span class="o">.</span><span class="no">DynamicSupervisors</span><span class="p">,</span> <span class="n">self</span><span class="p">()}},</span> <span class="n">my_child_specification</span> <span class="p">)</span> </code></pre></div></div> <h2 id="improved-errors-on-binaries-and-evaluation">Improved errors on binaries and evaluation</h2> <p>Erlang/OTP 25 improved errors on binary construction and evaluation. These improvements apply to Elixir as well. Before v1.14, errors when constructing binaries would often be hard-to-debug, generic “argument errors”. Erlang/OTP 25 and Elixir v1.14 provide more detail for easier debugging. This work is part of <a href="https://www.erlang.org/eeps/eep-0054">EEP 54</a>.</p> <p>Before:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">int</span> <span class="o">=</span> <span class="mi">1</span> <span class="n">bin</span> <span class="o">=</span> <span class="s2">"foo"</span> <span class="n">int</span> <span class="o">&lt;&gt;</span> <span class="n">bin</span> <span class="c1">#=&gt; ** (ArgumentError) argument error</span> </code></pre></div></div> <p>Now:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">int</span> <span class="o">=</span> <span class="mi">1</span> <span class="n">bin</span> <span class="o">=</span> <span class="s2">"foo"</span> <span class="n">int</span> <span class="o">&lt;&gt;</span> <span class="n">bin</span> <span class="c1">#=&gt; ** (ArgumentError) construction of binary failed:</span> <span class="c1">#=&gt; segment 1 of type 'binary':</span> <span class="c1">#=&gt; expected a binary but got: 1</span> </code></pre></div></div> <p>Code evaluation (in IEx and Livebook) has also been improved to provide better error reports and stacktraces.</p> <h2 id="slicing-with-steps">Slicing with Steps</h2> <p>Elixir v1.12 introduced <strong>stepped ranges</strong>, which are ranges where you can specify the “step”:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="no">Enum</span><span class="o">.</span><span class="n">to_list</span><span class="p">(</span><span class="mi">1</span><span class="o">..</span><span class="mi">10</span><span class="o">//</span><span class="mi">3</span><span class="p">)</span> <span class="c1">#=&gt; [1, 4, 7, 10]</span> </code></pre></div></div> <p>Stepped ranges are particularly useful for numerical operations involving vectors and matrices (see <a href="https://github.com/elixir-nx/nx">Nx</a>, for example). However, the Elixir standard library was not making use of stepped ranges in its APIs. Elixir v1.14 starts to take advantage of steps with support for stepped ranges in a couple of functions. One of them is <a href="https://hexdocs.pm/elixir/Enum.html#slice/2"><code class="language-plaintext highlighter-rouge">Enum.slice/2</code></a>:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">letters</span> <span class="o">=</span> <span class="p">[</span><span class="s2">"a"</span><span class="p">,</span> <span class="s2">"b"</span><span class="p">,</span> <span class="s2">"c"</span><span class="p">,</span> <span class="s2">"d"</span><span class="p">,</span> <span class="s2">"e"</span><span class="p">,</span> <span class="s2">"f"</span><span class="p">,</span> <span class="s2">"g"</span><span class="p">,</span> <span class="s2">"h"</span><span class="p">,</span> <span class="s2">"i"</span><span class="p">,</span> <span class="s2">"j"</span><span class="p">]</span> <span class="no">Enum</span><span class="o">.</span><span class="n">slice</span><span class="p">(</span><span class="n">letters</span><span class="p">,</span> <span class="mi">0</span><span class="o">..</span><span class="mi">5</span><span class="o">//</span><span class="mi">2</span><span class="p">)</span> <span class="c1">#=&gt; ["a", "c", "e"]</span> </code></pre></div></div> <p><a href="https://hexdocs.pm/elixir/Kernel.html#binary_slice/2"><code class="language-plaintext highlighter-rouge">binary_slice/2</code></a> (and <a href="https://hexdocs.pm/elixir/Kernel.html#binary_slice/3"><code class="language-plaintext highlighter-rouge">binary_slice/3</code></a> for completeness) has been added to the <code class="language-plaintext highlighter-rouge">Kernel</code> module, that works with bytes and also support stepped ranges:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">binary_slice</span><span class="p">(</span><span class="s2">"Elixir"</span><span class="p">,</span> <span class="mi">1</span><span class="o">..</span><span class="mi">5</span><span class="o">//</span><span class="mi">2</span><span class="p">)</span> <span class="c1">#=&gt; "lxr"</span> </code></pre></div></div> <h2 id="expression-based-inspection-and-inspect-improvements">Expression-based Inspection and <code class="language-plaintext highlighter-rouge">Inspect</code> Improvements</h2> <p>In Elixir, it’s conventional to implement the <code class="language-plaintext highlighter-rouge">Inspect</code> protocol for opaque structs so that they’re inspected with a special notation, resembling this:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="no">MapSet</span><span class="o">.</span><span class="n">new</span><span class="p">([</span><span class="ss">:apple</span><span class="p">,</span> <span class="ss">:banana</span><span class="p">])</span> <span class="c1">#MapSet&lt;[:apple, :banana]&gt;</span> </code></pre></div></div> <p>This is generally done when the struct content or part of it is private and the <code class="language-plaintext highlighter-rouge">%name{...}</code> representation would reveal fields that are not part of the public API.</p> <p>The downside of the <code class="language-plaintext highlighter-rouge">#name&lt;...&gt;</code> convention is that <em>the inspected output is not valid Elixir code</em>. For example, you cannot copy the inspected output and paste it into an IEx session.</p> <p>Elixir v1.14 changes the convention for some of the standard-library structs. The <code class="language-plaintext highlighter-rouge">Inspect</code> implementation for those structs now returns a string with a valid Elixir expression that recreates the struct when evaluated. In the <code class="language-plaintext highlighter-rouge">MapSet</code> example above, this is what we have now:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">fruits</span> <span class="o">=</span> <span class="no">MapSet</span><span class="o">.</span><span class="n">new</span><span class="p">([</span><span class="ss">:apple</span><span class="p">,</span> <span class="ss">:banana</span><span class="p">])</span> <span class="no">MapSet</span><span class="o">.</span><span class="n">put</span><span class="p">(</span><span class="n">fruits</span><span class="p">,</span> <span class="ss">:pear</span><span class="p">)</span> <span class="c1">#=&gt; MapSet.new([:apple, :banana, :pear])</span> </code></pre></div></div> <p>The <code class="language-plaintext highlighter-rouge">MapSet.new/1</code> expression evaluates to exactly the struct that we’re inspecting. This allows us to hide the internals of <code class="language-plaintext highlighter-rouge">MapSet</code>, while keeping it as valid Elixir code. This expression-based inspection has been implemented for <code class="language-plaintext highlighter-rouge">Version.Requirement</code>, <code class="language-plaintext highlighter-rouge">MapSet</code>, and <code class="language-plaintext highlighter-rouge">Date.Range</code>.</p> <p>Finally, we have improved the <code class="language-plaintext highlighter-rouge">Inspect</code> protocol for structs so that fields are inspected in the order they are declared in <code class="language-plaintext highlighter-rouge">defstruct</code>. The option <code class="language-plaintext highlighter-rouge">:optional</code> has also been added when deriving the <code class="language-plaintext highlighter-rouge">Inspect</code> protocol, giving developers more control over the struct representation. See <a href="https://hexdocs.pm/elixir/Inspect.html">the updated documentation for <code class="language-plaintext highlighter-rouge">Inspect</code></a> for a general rundown on the approaches and options available.</p> <h2 id="learn-more">Learn more</h2> <p>For a complete list of all changes, see the <a href="https://github.com/elixir-lang/elixir/releases/tag/v1.14.0">full release notes</a>.</p> <p>Check <a href="/install.html">the Install section</a> to get Elixir installed and read our <a href="https://hexdocs.pm/elixir/introduction.html">Getting Started guide</a> to learn more.</p> <p>Happy debugging!</p> José Valim 2021-12-03T00:00:00+00:00 /blog/2021/12/03/elixir-v1-13-0-released <p>Elixir v1.13 has just been released!</p> <p>Generally speaking, new Elixir versions include improvements to its primary API, the one Elixir developers use every day, and also to the foundation that powers its tooling. In this release, however, it coincided that most new functionality centers around Elixir tooling. The result is a series of quality of life improvements that will impact Elixir developers immediately as well as in the long term.</p> <p>Let’s check them out!</p> <p>Note: this announcement contains <a href="https://asciinema.org">asciinema</a> snippets. You may need to enable 3rd-party JavaScript on this site in order to see them. If JavaScript is disabled, <code class="language-plaintext highlighter-rouge">noscript</code> tags with the proper links will be shown.</p> <h2 id="semantic-recompilation">Semantic recompilation</h2> <p>The feature that will most and immediately benefit all Elixir developers is the series of improvements we have made to how the compiler tracks file contents.</p> <p>Generally speaking, once a file changes, it may lead to other files in your codebase to be recompiled. In previous versions, however, Elixir made no effort to understand which parts of the file changed. This meant the smallest of changes to certain files, such as configuration files, could trigger a full project recompilation.</p> <p>This release comes with a series of improvements that better understand how your files change. In particular:</p> <ul> <li> <p>An Elixir file is no longer considered as changed if its size and its digest stay the same. This avoids recompiling many files when switching or rebasing branches.</p> </li> <li> <p>Changing your <code class="language-plaintext highlighter-rouge">mix.exs</code> will no longer trigger a full project recompilation, unless you specifically change the configurations used by the Elixir compiler (<code class="language-plaintext highlighter-rouge">:elixirc_paths</code> and <code class="language-plaintext highlighter-rouge">:elixirc_options</code>).</p> </li> <li> <p>Changing compile-time configuration files (<code class="language-plaintext highlighter-rouge">config/config.exs</code> and any other files imported from it) now only recompiles the project files that depend on the reconfigured applications, instead of a full project recompilation. However, if you change the configuration of your application itself, the whole project is still recompiled.</p> </li> <li> <p>Adding, updating or removing a dependency now only recompiles the project files that depend on the modified dependency.</p> </li> <li> <p>If your project has both Erlang and Elixir files, changing an Erlang file will now recompile only the Elixir files that depend on it.</p> </li> </ul> <p>In a nutshell, Elixir went from triggering full recompilations whenever any of <code class="language-plaintext highlighter-rouge">mix.exs</code>, <code class="language-plaintext highlighter-rouge">config/config.exs</code>, <code class="language-plaintext highlighter-rouge">src/*</code>, and <code class="language-plaintext highlighter-rouge">mix.lock</code> changed on disk to semantic recompilations. Now it only fully recompiles when:</p> <ul> <li>you change the compilation options in <code class="language-plaintext highlighter-rouge">mix.exs</code></li> <li>you change the configuration for the current project in <code class="language-plaintext highlighter-rouge">config/config.exs</code></li> </ul> <p>To give a more practical example, take a regular <a href="https://phoenixframework.org/">Phoenix project</a>. It is most likely divided in two main directories: <code class="language-plaintext highlighter-rouge">my_app</code> and <code class="language-plaintext highlighter-rouge">my_app_web</code>. Most of your usage of Phoenix’ APIs will happen within the files in the <code class="language-plaintext highlighter-rouge">my_app_web</code> directory. However, if you bumped your Phoenix version or changed its configuration in previous Elixir versions, it would cause all files, in both directories, to be recompiled. With these changes, the recompilation should affect mostly the files in <code class="language-plaintext highlighter-rouge">my_app_web</code>.</p> <blockquote> <p>To further clarify, the Elixir compiler is not tracking directories. It is just a consequence of how Phoenix projects are organized that most dependencies to Phoenix are within <code class="language-plaintext highlighter-rouge">my_app_web</code>.</p> </blockquote> <h2 id="code-fragments">Code fragments</h2> <p>The <a href="https://hexdocs.pm/elixir/Code.html"><code class="language-plaintext highlighter-rouge">Code</code></a> module got a companion module called <a href="https://hexdocs.pm/elixir/Code.Fragment.html"><code class="language-plaintext highlighter-rouge">Code.Fragment</code></a>.</p> <p>The <code class="language-plaintext highlighter-rouge">Code</code> module works with complete code. For example, its functions will consider the snippet <code class="language-plaintext highlighter-rouge">123 +</code> as invalid, since the right-hand side of <code class="language-plaintext highlighter-rouge">+</code> is missing. However, our tooling, such as editors, REPLs, and code notebooks must still parse and understand such snippets, in order to provide code completion, argument suggestion, etc.</p> <p>That’s the goal of the <code class="language-plaintext highlighter-rouge">Code.Fragment</code> module. It contains different heuristics to analyze and return context informational of code fragments, which are code snippets that may be incomplete.</p> <p>To better show the benefits of said improvements, let’s talk about IEx, Elixir’s interactive shell. IEx has been rewritten to use <code class="language-plaintext highlighter-rouge">Code.Fragment</code> and, in the process, it gained new functionality as part of its autocompletion system (available by hitting TAB). For example, it can now autocomplete sigils, used to <a href="https://hexdocs.pm/elixir/Kernel.html#sigil_r/2">create regexes</a> or <a href="https://hexdocs.pm/elixir/Kernel.html#sigil_w/2">lists of words</a>, and their terminators:</p> <script id="asciicast-By0cGpu9xSUgflc24cVlLgPgY" src="https://asciinema.org/a/By0cGpu9xSUgflc24cVlLgPgY.js" async=""></script> <noscript><p><a href="https://asciinema.org/a/By0cGpu9xSUgflc24cVlLgPgY">See the example in asciinema</a></p></noscript> <p>Similarly, you can now autocomplete struct names and their fields:</p> <script id="asciicast-A44auZ00saSud3l7DbOL4IMYn" src="https://asciinema.org/a/A44auZ00saSud3l7DbOL4IMYn.js" async=""></script> <noscript><p><a href="https://asciinema.org/a/A44auZ00saSud3l7DbOL4IMYn">See the example in asciinema</a></p></noscript> <p>Overall, we hope the <code class="language-plaintext highlighter-rouge">Code.Fragment</code> module will become the shared foundation to power many of the tools in the ecosystem. We have also added new reflection APIs to <a href="https://hexdocs.pm/elixir/Module.html"><code class="language-plaintext highlighter-rouge">Module</code></a>, which can then be used to power code intelligence features.</p> <h2 id="mix-xref">mix xref</h2> <p><a href="https://hexdocs.pm/mix/Mix.Tasks.Xref.html"><code class="language-plaintext highlighter-rouge">mix xref</code></a> is a tool that analyzes relationships between files. By analyzing the compile-time and runtime dependencies between them, it allows developers to understand what has to be recompiled whenever a file changes.</p> <p>Elixir v1.13 comes with many improvements to <code class="language-plaintext highlighter-rouge">mix xref</code>, such as:</p> <ul> <li> <p><code class="language-plaintext highlighter-rouge">mix xref graph</code> now supports <code class="language-plaintext highlighter-rouge">--label</code> to be set to “compile-connected”, which returns all compile-time dependencies that lead to additional transitive dependencies.</p> </li> <li> <p>A new <code class="language-plaintext highlighter-rouge">mix xref trace FILE</code> subcommand receives a file and returns all dependencies in said file, including the line and what caused said dependency (a function/macro call, an alias, a struct, etc).</p> </li> <li> <p>All <code class="language-plaintext highlighter-rouge">mix xref</code> subcommands support the <code class="language-plaintext highlighter-rouge">--fail-above</code> flag, which allows you to enforce your project has at most a certain number of compile-time cycles, transitive compile-time dependencies, etc. This can be useful on Continuous Integration (CI) servers.</p> </li> <li> <p><code class="language-plaintext highlighter-rouge">mix xref graph</code> now supports multiple <code class="language-plaintext highlighter-rouge">--sink</code> and <code class="language-plaintext highlighter-rouge">--source</code> to be given.</p> </li> </ul> <p>If you haven’t used <code class="language-plaintext highlighter-rouge">mix xref</code> before, it may be hard to visualize what those changes mean. If you want to learn more, you can <a href="https://youtu.be/ydjx2kKHzrM?t=772">watch the relevant section of my ElixirConf 2021 keynote</a> that includes a short introduction to <code class="language-plaintext highlighter-rouge">mix xref</code>.</p> <p>Those improvements came from direct feedback from the community. A special shout out to Marc-André Lafortune for the contributions and testing.</p> <h2 id="extended-code-formatting">Extended code formatting</h2> <p>Thanks to its sigils, Elixir provides the ability of embedding snippets in other languages inside its source code. One could use it to embed XML:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>~X""" &lt;?xml version="1.0" encoding="UTF-8"?&gt; &lt;text&gt;&lt;![CDATA[Hello World]]&gt;&lt;/text&gt; """ </code></pre></div></div> <p>Or even <a href="https://ziglang.org/">Zig</a>, <a href="https://github.com/ityonemo/zigler">via the Zigler project</a>:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>~Z""" /// nif: example_fun/2 fn example_fun(value1: f64, value2: f64) bool { return value1 &gt; value2; } """ </code></pre></div></div> <p>However, while you can format Elixir source code with <a href="https://hexdocs.pm/mix/Mix.Tasks.Format.html"><code class="language-plaintext highlighter-rouge">mix format</code></a>, you could not format the code inside snippets.</p> <p>Elixir v1.13 solves this by adding plugins to <code class="language-plaintext highlighter-rouge">mix format</code>. Plugins can teach the formatter how to format new files and how to format sigils, via the <code class="language-plaintext highlighter-rouge">Mix.Tasks.Format</code> behaviour.</p> <p>For example, imagine that your project uses Markdown in two distinct ways: via a custom <code class="language-plaintext highlighter-rouge">~M</code> sigil and via files with the <code class="language-plaintext highlighter-rouge">.md</code> and <code class="language-plaintext highlighter-rouge">.markdown</code> extensions. A custom plugin would look like this:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">MixMarkdownFormatter</span> <span class="k">do</span> <span class="nv">@behaviour</span> <span class="no">Mix</span><span class="o">.</span><span class="no">Tasks</span><span class="o">.</span><span class="no">Format</span> <span class="k">def</span> <span class="n">features</span><span class="p">(</span><span class="n">_opts</span><span class="p">)</span> <span class="k">do</span> <span class="p">[</span><span class="ss">sigils:</span> <span class="p">[</span><span class="ss">:M</span><span class="p">],</span> <span class="ss">extensions:</span> <span class="p">[</span><span class="s2">".md"</span><span class="p">,</span> <span class="s2">".markdown"</span><span class="p">]]</span> <span class="k">end</span> <span class="k">def</span> <span class="n">format</span><span class="p">(</span><span class="n">contents</span><span class="p">,</span> <span class="n">opts</span><span class="p">)</span> <span class="k">do</span> <span class="c1"># logic that formats markdown</span> <span class="k">end</span> <span class="k">end</span> </code></pre></div></div> <p>Now any application can use your formatter as follows:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># .formatter.exs</span> <span class="p">[</span> <span class="c1"># Define the desired plugins</span> <span class="ss">plugins:</span> <span class="p">[</span><span class="no">MixMarkdownFormatter</span><span class="p">],</span> <span class="c1"># Remember to update the inputs list to include the new extensions</span> <span class="ss">inputs:</span> <span class="p">[</span><span class="s2">"{mix,.formatter}.exs"</span><span class="p">,</span> <span class="s2">"{config,lib,test}/**/*.{ex,exs}"</span><span class="p">,</span> <span class="s2">"posts/*.{md,markdown}"</span><span class="p">]</span> <span class="p">]</span> </code></pre></div></div> <p>We are looking forward to see how this new functionality will be used by community, especially projects like <a href="https://github.com/surface-ui/surface">Surface</a> and <a href="https://github.com/phoenixframework/phoenix_live_view">Phoenix LiveView</a>, which provide a templating language on top of the HTML markup.</p> <h2 id="other-bits-and-bytes">Other bits and bytes</h2> <p><code class="language-plaintext highlighter-rouge">SyntaxError</code> and <code class="language-plaintext highlighter-rouge">TokenMissingError</code> were improved to show a code snippet whenever possible:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ elixir -e "hello + * world" ** (SyntaxError) nofile:1:9: syntax error before: '*' | 1 | hello + * world | ^ </code></pre></div></div> <p>The <code class="language-plaintext highlighter-rouge">Code</code> module has also been augmented with two functions: <a href="https://hexdocs.pm/elixir/Code.html#string_to_quoted_with_comments/2"><code class="language-plaintext highlighter-rouge">Code.string_to_quoted_with_comments/2</code></a> and <a href="https://hexdocs.pm/elixir/Code.html#quoted_to_algebra/2"><code class="language-plaintext highlighter-rouge">Code.quoted_to_algebra/2</code></a>. Those functions allow someone to retrieve the Elixir AST with their original source code comments, and then convert this AST to formatted code. In other words, those functions provide a wrapper around the Elixir Code Formatter, supporting developers who wish to create tools that directly manipulate and custom format Elixir source code.</p> <p><code class="language-plaintext highlighter-rouge">elixir --short-version</code> has been added to quickly get the Elixir version, without booting the Erlang VM. The <code class="language-plaintext highlighter-rouge">Task</code> module includes performance optimizations and <a href="https://hexdocs.pm/elixir/Task.html#ignore/1">new</a> <a href="https://hexdocs.pm/elixir/Task.html#completed/1">functions</a>. Finally, <code class="language-plaintext highlighter-rouge">mix test --profile-require=time</code> has been added to debug loading times of test suites and the recently added <a href="https://hexdocs.pm/mix/Mix.html#install#2"><code class="language-plaintext highlighter-rouge">Mix.install/2</code></a> has been improved with new options and environment variables.</p> <h2 id="learn-more">Learn more</h2> <p>For a complete list of all changes, see the <a href="https://github.com/elixir-lang/elixir/releases/tag/v1.13.0">full release notes</a>. You can also <a href="https://youtu.be/ydjx2kKHzrM">watch my ElixirConf 2021 keynote about Elixir v1.13</a> to learn more.</p> <p>Check <a href="/install.html">the Install section</a> to get Elixir installed and read our <a href="https://hexdocs.pm/elixir/introduction.html">Getting Started guide</a> to learn more.</p> <p>Have fun!</p> José Valim 2021-11-10T00:00:00+00:00 /blog/2021/11/10/embracing-open-data-with-elixir-at-the-ministry-of-ecological-transition-in-france <p><em>Welcome to our series of case studies about companies using Elixir in production. <a href="/cases.html">See all cases</a> we have published so far.</em></p> <p>A group of initiatives towards innovation and open data has given the opportunity for Elixir to play a central role in exploring, validating, and visualizing transportation data across all of France. This article will show how Elixir came to power <a href="https://transport.data.gouv.fr/">the National Access Point for transport data in France</a> and explore why it continues to be an excellent fit thanks to its real-time libraries, educational tooling, and orchestration capabilities.</p> <p><img src="/images/cases/bg/met-france.png" alt="Sample map of transports" /></p> <h2 id="state-startups">State Startups</h2> <p>In 2013, the French Government launched a Digital Services incubator, called <a href="https://beta.gouv.fr">beta.gouv.fr</a>, to spread the culture of digital innovation throughout the administration. They do this through State Startups.</p> <p>State Startups are a match between a team and a mission. They help “intrapreneurs” - public servants who identify frictions and opportunities to improve the lives of their fellow citizens - tackle real-world challenges alongside a team of experts. This team of 2 to 4 people has six months to build a proof of concept (<a href="https://beta.gouv.fr/en/">see official website</a>).</p> <p>The insight is: instead of trying to convince other stakeholders to work towards specific goals, it is better to empower innovation and those who want to drive change. Those individuals are given a budget and the autonomy to choose the technology and assemble their team. In exchange, they must open-source and publicly document all code, costs, metrics, and statistics.</p> <p>The first State Startup was <a href="https://www.data.gouv.fr/en">data.gouv.fr</a>, which transformed France’s open data efforts from a catalog of spreadsheets into a social platform that closed the gap between the citizens consuming the data and the institutions providing them. The tool is fully <a href="https://github.com/opendatateam/udata">open-source</a>, allowing other countries to use it in production too.</p> <p>At the time of writing, <a href="https://beta.gouv.fr/startups/">261 State Startups</a> have been launched and are in various states of development.</p> <h2 id="elixir-drives-by">Elixir drives by</h2> <p>In 2017, a team was assembled to begin a new State Startup focused on transportation data. An incoming European delegated regulation would make it mandatory for institutions and corporations to make transportation data public. The State Startup aimed at preparing the ecosystem actors for this regulatory change.</p> <p>To address this, the team decided to build a web application to search and visualize the existing transportation data available in <a href="https://www.data.gouv.fr">data.gouv.fr</a>. They initially targeted public transportation information provided by cities about buses, subways, and trams, all available in a static format called General Transit Feed Specification (<a href="https://gtfs.org/">GTFS</a>) (<a href="https://transport.data.gouv.fr/resources/50471#visualization">live example</a>).</p> <p>The two developers of the team, Vincent Lara and Mauko Quiroga, had heard about Elixir and were interested in learning more. They understood it could provide a robust but flexible and fun platform to explore the problem space. So <a href="https://github.com/etalab/transport-site/commit/837a048c37ac31151b51ac09432dbcbff3917de5">they bootstrapped the application</a> with the <a href="https://phoenixframework.org/">Phoenix web framework</a>.</p> <p>As they developed the system, they spotted gaps and errors in the data available. So they began validating the data and reaching out to the institutions publishing them, providing direct feedback and value to the open data platform. The incubator saw the benefits from their contributions and, after a few months, they had a successful State Startup in their hands alongside the approval to continue working on their mission.</p> <p>Between 2017 and 2021, the multi-disciplinary team (one public servant, “business developers”, and technical staff) worked to increase the coverage of published transportation data and helped cities and operators to reach their technical and regulatory goals.</p> <h2 id="current-challenges">Current challenges</h2> <p>In 2021, the State Startup has “graduated” from its “beta gouv” incubator and is now part of France’s Ministry of Ecological Transition. Now composed by Francis Chabouis, Thibaut Barrère, and Antoine Augusti, the technical part of the team is tackling new use cases and challenges as the platform grows in terms of needs and versatility.</p> <p>Many of those are driven by the adoption of new data formats by governments and corporations. For example, <a href="https://github.com/google/transit/tree/master/gtfs/spec/en">GTFS</a> provides a static (theoretical) itinerary: if a bus is currently delayed, this information would not be available in the feed. Enter the <a href="https://github.com/google/transit/tree/master/gtfs-realtime/spec/en">GTFS-RT</a> format, where RT stands for real-time, to address those gaps. The General Bikeshare Feed Specification (<a href="https://nabsa.net/resources/gbfs/">GBFS</a>) (<a href="https://transport.data.gouv.fr/datasets/velos-libre-service-creteil-cristolib-disponibilite-en-temps-reel/">live example</a>) tracks bicycles, scooters, carpooling, etc. Plus the <a href="https://en.wikipedia.org/wiki/Service_Interface_for_Real_Time_Information">SIRI</a> (Service Interface for Real-time Information) and <a href="https://en.wikipedia.org/wiki/NeTEx">NeTEx</a> families of protocols.</p> <p>Some of those formats have supporting technologies (validators, converters) written in other languages (Java, Rust, etc), which would be beneficial to integrate with. The team then realized the way forward is to adapt their Elixir system to orchestrate and coordinate those subsystems. Luckily, Elixir has shown to be exceptionally well suited to this task, thanks to the underlying Erlang Virtual Machine, designed for communication systems. Francis Chabouis expands: “We currently need to integrate with internal and external services in a variety of formats. Some are static, some require periodic pulls, and others keep open connections to push data. Elixir allows us to experiment, iterate, and scale those ideas quickly”.</p> <p>Overall, the data catalog now includes:</p> <ul> <li>Timetables for buses, subways, and trains, including trips and operators, as real-time updates</li> <li>Bicycle lanes and carpooling areas</li> <li>Charging and refueling stations</li> <li>Private parking areas</li> <li>Location of rental cars, scooters, bicycles, and others</li> </ul> <p>Many of those formats also bring real-time concerns as they evolve the application to sample and show events as they happen. This is where the team is currently working at leveraging <a href="http://github.com/phoenixframework/phoenix_live_view">Phoenix LiveView</a> to build the interactivity and functionality they need while keeping their stack simple and productive.</p> <p>The technical team has also recently grown to three developers, totaling seven members altogether. To prepare themselves for the new team members, Thibaut Barrère was responsible for upgrading their dependencies, including Elixir and Erlang, which were largely unchanged from 2017. While performing such changes can often be daunting in other stacks, Thibaut shares a very positive experience: “we did not see any breaking changes after performing long-due updates. Overall, the language and libraries seem quite stable and careful to avoid breaking changes. This experience gives us the confidence we can continue focusing on the needs of our users as we move forward”.</p> <h2 id="open-data-and-education">Open data and education</h2> <p>As with any other team, some challenges go beyond the technical aspects. For example, they sometimes spot companies and cities that do not make their transportation data available, which is against European Law. The team often heard concerns about making parts of their systems public, which could lead to failures in critical infrastructure.</p> <p>To address this, the team built <a href="https://github.com/etalab/transport-site/tree/master/apps/unlock">a small Phoenix application</a> that works as a simple proxy to those systems. The proxy caches the data for specific intervals, helping those entities address the security and traffic concerns around their critical systems. The application uses <a href="https://github.com/whitfin/cachex">Cachex</a> and provides a real-time dashboard, built with LiveView, where they can configure the system, track application load, and see cache usage data.</p> <p>Another area the team is actively investigating is how to make the data itself more accessible to developers who want to consume it. A non-trivial amount of work is required between fetching the city data, parsing it, and displaying it on a map, which can discourage developers from beginning their open data journey. To this end, they plan to assemble a collection of <a href="http://github.com/livebook-dev/livebook">Livebooks</a>, a recently released tool for writing code notebooks in Elixir that allows developers to get up and running quickly and obtain immediate feedback on their code.</p> <p>Thibaut remarks how the growth of the language and its ecosystem supports their application and needs: “every time we faced a new challenge, a solution was readily available to us. When we needed to orchestrate multiple subsystems, the stack excelled at it. When we required real-time features, Phoenix and LiveView had first-class support for it. Now we need to promote education and access to open data, and Livebook is shaping to be precisely what we need”.</p> José Valim 2021-07-29T00:00:00+00:00 /blog/2021/07/29/bootstraping-a-multiplayer-server-with-elixir-at-x-plane <p><em>Welcome to our series of case studies about companies using Elixir in production. <a href="/cases.html">See all cases</a> we have published so far.</em></p> <p><a href="https://www.x-plane.com/">X-Plane 11</a> is the world’s most comprehensive and powerful flight simulator for personal computers and mobile devices. X-Plane is not a game but an engineering tool created by Laminar Research that can be used to predict the flying qualities of fixed- and rotary-wing aircraft with incredible accuracy. The X-Plane franchise comes in both consumer and FAA-certifiable professional versions.</p> <p>Recently, the X-Plane team took on the challenge of adding a multiplayer experience with the goal of hosting north of 10000 users in the same session. This article explores why they chose Elixir and how a team of one developer - without prior language experience - learned the language and deployed a well-received multiplayer experience in 6 months. The overall solution features a brand new open-source implementation of the RakNet communication protocol in Elixir and overperforms the original requirements when put under unexpected load.</p> <p><img src="/images/cases/bg/x-plane.jpg" alt="X-Plane" /></p> <h2 id="requirements">Requirements</h2> <p>The X-Plane team has offered peer-to-peer multiplayer in the simulator for a long time but never server-hosted multiplayer. This was a new journey for them and they had complete freedom to pick the technology stack. <a href="https://developer.x-plane.com/2021/01/have-you-heard-the-good-news-about-elixir/">According to their blog post</a>, their goals were:</p> <ol> <li> <p>To build a rock-solid server with error isolation. For example, an exception during a client update should not bring the whole server down.</p> </li> <li> <p>To implement a single shared world that can scale to tens of thousands of concurrent pilots.</p> </li> <li> <p>To iterate quickly: because this was the first time the Laminar Research team offered a hosted multiplayer environment, they wanted to move quickly to ship this system. This would allow users to begin flying in groups immediately and serve as a platform to gauge interest in further feature development.</p> </li> <li> <p>To be fast and consistent. Multiplayer has a “soft real-time” constraint, and they need to service <em>all</em> clients consistently and on time. Quantitatively, this means the 99th percentile response times matter a lot more than the mean or median.</p> </li> </ol> <p>From those requirements, the need for stability and fast iteration ruled out low-level languages, even the ones in which they had plenty of in-house experience.</p> <p>The need for speed and vertical scalability excluded many modern web languages, such as Ruby and Python, where the model for scaling up is generally throwing more servers at it. It was essential to avoid synchronizing state across multiple machines, which requires more development time and worsens the user experience due to the increased latency.</p> <p>They eventually settled on three top contenders: Rust, Go, and Elixir. Elixir took the edge thanks to two exclusive features: fault tolerance and predictable latency. Both are built into the very core of the Erlang Virtual Machine - the robust platform that Elixir runs on. Tyler Young, X-Plane’s engineer leading this implementation, highlights: “We wanted a stack that could max server capacity. We would rather run a 64-core machine than dozens of 4-core VMs. Saša Jurić’s talk, <a href="https://www.youtube.com/watch?v=JvBT4XBdoUE">the Soul of Erlang and Elixir</a>, showed us that the concurrency model, process isolation, and partial restarts provided by the platform were the abstractions we were looking for.”</p> <h2 id="modeling-multiplayer-with-elixir">Modeling multiplayer with Elixir</h2> <p>Ready to give Elixir a try, Tyler picked up a couple books but soon realized the language’s <a href="https://hexdocs.pm/elixir/introduction.html">Getting Started guide</a> provided the background he needed. He explains: “while the introductory guide covers the language constructs, the advanced guide on the website has you build an actual project with TCP connections, with the basic architectural patterns we would use in production.”</p> <p>However, instead of jumping headfirst into the multiplayer server, he decided to give Elixir a try on a smaller problem. He wrote a web proxy to the National Oceanic and Atmospheric Administration (NOAA) weather services and put it in production. This experience taught him the importance of leveraging all of the instrumentation and metrics provided by the Erlang VM. They chose <a href="https://www.appsignal.com/">AppSignal</a> to help consume and digest this information.</p> <p>Two weeks later, he started working on the server by implementing <a href="https://en.wikipedia.org/wiki/RakNet">the UDP-centric RakNet protocol in Elixir</a>. Unfortunately, there is little documentation, so they had to refer to the reference implementation in C++ most of the time. Luckily, thanks to its roots in telecommunication and network services, <a href="https://hexdocs.pm/elixir/Kernel.SpecialForms.html#%3C%3C%3E%3E/1">Elixir and Erlang have built-in support for parsing binary packets</a>, which made the task a joy. The team also mapped each UDP connection to distinct lightweight threads of execution in Elixir, which we call <em>processes</em>. Elixir processes are cheap, isolated, concurrent, and are fairly scheduled by the runtime. This design allowed the X-Plane team to fully leverage the properties of robustness and predictable latency that first attracted them to the platform. Their implementation is written on top of Erlang’s <a href="http://www.erlang.org/doc/man/gen_udp.html">gen_udp</a> and <a href="https://github.com/X-Plane/elixir-raknet">is open source</a>.</p> <p>Five months after choosing Elixir, they began welcoming beta testers into the server. The community’s reaction was overwhelmingly positive, and the new multiplayer experience led to a strong uptick in the number of subscriptions as it went live a month later.</p> <h2 id="deployment-and-keeping-it-simple">Deployment and keeping it simple</h2> <p>At the moment, X-Plane’s player base in North America is powered by a single server, running on 1 eight-core machine with 16GB of memory, although only 200MB or 300MB of memory is effectively used. Each connected player sends 10 updates a second.</p> <p>For deployments, they use a blue-green strategy, alternating between two servers of the same capacity. Tyler explains: “We are aware the Erlang VM provides hot code swapping and distribution, but we are taking the simplest route whenever possible. It is much easier for us to alternate between two servers during deployments, as the servers are stable and we don’t deploy frequently. Similarly, when it comes to distribution, we prefer to scale vertically or set up different servers in different regions for players across the globe.”</p> <p>Paul McCarty, who joined the project after launch, can attest to its simplicity: “even without prior Elixir experience, I was able to jump in and add new functionality to our HTTP APIs early on.” Those APIs are built on top of <a href="http://github.com/elixir-lang/plug">Plug</a> to power their chat services, provide information about connected users, and more. He concludes: “When adding new features, the server development is never the bottleneck.”</p> <p>Paul and Tyler finished our conversation with a curious anecdote: a couple months ago, they distributed an updated client version with debug code in it. This additional code caused each connected user to constantly ping the server every 100ms, even if not in multiplayer mode. This caused their traffic to increase 1000x! They only discovered this increase 2 weeks later when they saw the CPU usage in their Elixir server went from 5% to 21%. Once they found out the root cause and how the system handled it, they realized they didn’t have to rush a client update to remove the debug code and they chose to maintain their regular release cycle. At the end of the day, it was a perfect example of the confidence they gained and built around the language and platform.</p> José Valim 2021-06-02T00:00:00+00:00 /blog/2021/06/02/social-virtual-spaces-with-elixir-at-mozilla <p><em>Welcome to our series of case studies about companies using Elixir in production. <a href="/cases.html">See all cases</a> we have published so far.</em></p> <p><a href="https://hubs.mozilla.com/">Hubs</a> is Mozilla’s take on virtual social experiences. You build your own private spaces and share them with your friends, co-workers, and community. Avatars in this space can move freely in a 3D social environment and watch videos, exchange messages, and talk to other people nearby. All you need is a browser and a microphone!</p> <p>Hubs is <a href="https://github.com/mozilla/hubs">fully</a> <a href="https://github.com/mozilla/reticulum">open source</a> and you can host it on your infrastructure via <a href="https://hubs.mozilla.com/cloud">Hubs Cloud</a>. Community managers, educators, and event organizers have been using Hubs Cloud to run virtual events and online activities tailored to their specific brandings and needs. All it takes to run your own version of Hubs is one click away - which perhaps makes Hubs the most deployed Phoenix application ever!</p> <p><img src="/images/cases/bg/mozilla-hubs.jpg" alt="Mozilla Hubs" /></p> <h2 id="from-vr-to-elixir">From VR to Elixir</h2> <p>The Hubs team started at Mozilla as the Mixed Reality team about 3.5 years ago. Their main goal was to explore ways for online social interaction via avatars and mixed reality.</p> <p>They quickly focused on building their first proof of concept, where avatars could communicate, move around, and join different rooms, everything running directly in the browser. This was a significant departure from the state of the art of Virtual Reality everywhere, as the getting started experience up to this point was cumbersome and often required steep investment in the form of VR headsets.</p> <p>The initial prototype was a success and it pushed the team to build a product. However, all communication in the proof of concept was peer-to-peer, which limited the features and experiences they could provide. Therefore the Hubs team knew they needed a capable backend technology to provide fan-out communication and coordinate how all different avatars interact within the virtual spaces. John Shaughnessy, Staff Software Engineer at Mozilla, comments: “When you get a lot of people in the same space, be it virtual or in the real world, there is never only a single conversation going on. Once you get ten or twenty different people in a room, conference calls don’t work well. In Hubs, people transition between multiple simultaneous conversations simply by moving around”.</p> <p>With this bold vision in hand, they assessed their whole stack. They settled on using JavaScript with <a href="https://threejs.org/">Three.js</a> in the front-end and chose <a href="https://phoenixframework.org/">the Phoenix web framework</a> for the backend. Greg Fodor, who was an Engineering Manager at Mozilla at the time, explains the choice: “We first listed all of the features we had to implement, from trivial things like REST endpoints, to more complex use cases, such as chat messaging and tracking where avatars are in the virtual world. Once I started to learn Phoenix, I saw all of those features were already there! The application we were building has to manage a large number of connections with real-time low latencies, something we knew the Erlang VM was an excellent fit for”.</p> <h2 id="in-production">In production</h2> <p>Hubs went live in January 2018. Almost everything in Hubs goes through Phoenix. The only exception is the WebRTC voice channels, which are handled by designated voice servers, initially implemented with <a href="https://janus.conf.meetecho.com/">Janus</a> and later ported to <a href="https://mediasoup.org/">MediaSoup</a>. However, the Phoenix app still manages the voice servers and how connections are assigned to them.</p> <p>The deployment is orchestrated by <a href="https://www.chef.io/products/chef-habitat/">Habitat</a> and running on Amazon EC2. Habitat provides packaging and orchestration. When a voice server joins the Habitat ring, the Phoenix services receive a message and start assigning voices to voice servers. Overall they run on 4 Phoenix and 4 voice servers.</p> <p>The Elixir experience in production has been quite smooth. Dominick D’Aniello, Staff Software Engineer at Mozilla, points out some areas they discussed improving: “the Phoenix application works mostly as a proxy, so we avoid decoding and reencoding the data unless we really need to. But sometimes we have to peek at the payloads and JSON is not the most efficient format to do so.” They have also considered relying more on Elixir processes and the Erlang distribution. Dominick continues: “when a new client joins, it needs to ask all other clients what is their state in the world, what they own, and what they care about. One option is to use Elixir processes in a cluster to hold the state of the different entities and objects in the virtual world”.</p> <h2 id="beyond-hubs">Beyond Hubs</h2> <p>With many large companies investing in online communication, the Mozilla team saw the possibility of virtual spaces becoming walled-gardens inside existing social platforms. This led the Hubs team to work on Hubs Cloud, with the mission to commoditize virtual spaces by allowing anyone to run their own version of Hubs with a single click.</p> <p>Hubs Cloud launched in February 2020 and it has been a hit. <a href="https://twitter.com/nyuniversity/status/1258401916096315399">New York University did its graduation ceremony on a Hubs Cloud instance</a>. <a href="https://www.computer.org/conferences/organize-a-conference/organizer-resources/hosting-a-virtual-event/success-stories/IEEE-VR-2020">The IEEE Virtual Reality Conference embraced Hubs</a> for a more accessible and sustainable event with talks and poster sessions all happening in virtual rooms, while <a href="https://www.twincities.com/2021/02/09/twins-set-to-launch-new-virtual-fan-experience/">the Minnesota Twins baseball team launched a Virtual Hall of Fame</a> on top of the platform.</p> <p>Their cloud version uses Amazon CloudFormation to instantiate Hubs inside the user’s account. This approach brought different challenges to the Hubs team: “we want Hubs Cloud to be as affordable and straightforward as possible. The Phoenix app has already been a massive help on this front. We have also moved some features to Amazon Lambda and made them optional, such as image resizing and video conversion” - details John.</p> <p>Since Hubs is also open source, developers can run their own Hubs instance in whatever platform they choose or change it however they want. That’s the path Greg Fodor recently took when he announced <a href="https://jel.app/">Jel</a>: “Jel is the video game for work. It is a mashup of Minecraft and Discord, where everything is 3D. My goal is to spark new directions and ideas to get people excited about VR”.</p> <h2 id="summing-up">Summing up</h2> <p>Today, the Hubs team has 10 contributors, half of whom are developers. Their engineering team is quite general and learning Elixir happens organically: “you are motivated by the feature you are working on. If it requires changing the backend, you learn Elixir with the help of the team and then make your contribution”.</p> <p>Overall, the bet on Phoenix was a successful one. Greg Fodor highlights: “The most significant benefit of Phoenix is in using a stack that excels at solving a large variety of problems. Once onboarded to Phoenix, there is a huge surface area our engineers can touch. Any feature they come up with, they can run with it. And because Hubs is open source, our contributors will also have the same experience. Overall, Elixir and Phoenix reduce the effort needed to cause the largest impact possible across our whole product”.</p> <p>Lately, they have leaned even further into the ecosystem, as they have started exposing Hubs APIs over GraphQL with the help of Absinthe. They have also migrated to Phoenix v1.5 and are using the <a href="https://github.com/phoenixframework/phoenix_live_dashboard">Phoenix LiveDashboard</a> to provide metrics and instrumentation to Hubs Cloud users.</p> José Valim 2021-05-19T00:00:00+00:00 /blog/2021/05/19/elixir-v1-12-0-released <p>Elixir v1.12 is out with improvements to scripting, tighter Erlang/OTP 24 integration, stepped ranges, and dozen of new functions across the standard library. Overall this is a small release, which continues our tradition of bringing Elixir developers quality of life improvements every 6 months. Some of these improvements directly relates with the <a href="https://dashbit.co/blog/nx-numerical-elixir-is-now-publicly-available">recent efforts of bringing Numerical Computing to Elixir</a>.</p> <p>Elixir v1.12 requires Erlang/OTP 22+. We also recommend running <code class="language-plaintext highlighter-rouge">mix local.rebar</code> after installation to upgrade to the latest Rebar versions, which includes support for Erlang OTP/24+.</p> <p>Note: this announcement contains <a href="https://asciinema.org">asciinema</a> snippets. You may need to enable 3rd-party JavaScript on this site in order to see them. If JavaScript is disabled, <code class="language-plaintext highlighter-rouge">noscript</code> tags with the proper links will be shown.</p> <h2 id="scripting-improvements-mixinstall2-and-systemtrap_signal3">Scripting improvements: <code class="language-plaintext highlighter-rouge">Mix.install/2</code> and <code class="language-plaintext highlighter-rouge">System.trap_signal/3</code></h2> <p>Elixir v1.12 brings new conveniences for those using Elixir for scripting (via <code class="language-plaintext highlighter-rouge">.exs</code> files). Elixir has been capable of managing dependencies for a quite long time, but it could only be done within Mix projects. In particular, the Elixir team is wary of global dependencies as any scripts that rely on system packages are brittle and hard to reproduce whenever your system changes.</p> <p><code class="language-plaintext highlighter-rouge">Mix.install/2</code> is meant to be a sweet spot between single-file scripts and full-blown Mix projects. With <code class="language-plaintext highlighter-rouge">Mix.install/2</code>, you can list your dependencies at the top of your scripts. When you execute the script for the first time, Elixir will download, compile, and cache your dependencies before running your script. Future invocations of the script will simply read the compiled artifacts from the cache:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="no">Mix</span><span class="o">.</span><span class="n">install</span><span class="p">([</span><span class="ss">:jason</span><span class="p">])</span> <span class="no">IO</span><span class="o">.</span><span class="n">puts</span><span class="p">(</span><span class="no">Jason</span><span class="o">.</span><span class="n">encode!</span><span class="p">(%{</span><span class="ss">hello:</span> <span class="ss">:world</span><span class="p">}))</span> </code></pre></div></div> <p><code class="language-plaintext highlighter-rouge">Mix.install/2</code> also performs protocol consolidation, which gives script developers an option to execute their code in the most performant format possible. Note <code class="language-plaintext highlighter-rouge">Mix.install/2</code> is currently experimental and it may change in future releases.</p> <p>Furthermore, <code class="language-plaintext highlighter-rouge">Mix.install/2</code> pairs nicely with Livebook, a newly announced project that brings interactive and collaborative notebook projects to Elixir. With Livebook and <code class="language-plaintext highlighter-rouge">Mix.install/2</code>, you can bring dependencies into your code notebooks and ensure they are fully reproducible. <a href="https://www.youtube.com/watch?v=RKvqc-UEe34">Watch the Livebook announcement to learn more</a>.</p> <p>Another improvement to scripting is the ability to trap exit signals via <code class="language-plaintext highlighter-rouge">System.trap_signal/3</code>. All you need is the signal name and a callback that will be invoked when the signal triggers. For example, ExUnit leverages this functionality to print all currently running tests when you abort the test suite via SIGQUIT (<code class="language-plaintext highlighter-rouge">Ctrl+\\ </code>). You can see this in action when running tests in the Plug project below:</p> <script type="text/javascript" src="https://asciinema.org/a/qPOJ9Vd8DiEXttEv7olNJPUR0.js" id="asciicast-qPOJ9Vd8DiEXttEv7olNJPUR0" async=""></script> <noscript><p><a href="https://asciinema.org/a/qPOJ9Vd8DiEXttEv7olNJPUR0">See the example in asciinema</a></p></noscript> <p>This is particularly useful when your tests get stuck and you want to know which one is the culprit.</p> <p><strong>Important</strong>: Trapping signals may have strong implications on how a system shuts down and behaves in production and therefore it is extremely discouraged for libraries to set their own traps. Instead, they should redirect users to configure them themselves. The only cases where it is acceptable for libraries to set their own traps is when using Elixir in script mode, such as in <code class="language-plaintext highlighter-rouge">.exs</code> files and via Mix tasks.</p> <h2 id="tighter-erlangotp-24-integration">Tighter Erlang/OTP 24 integration</h2> <p><a href="https://blog.erlang.org/My-OTP-24-Highlights/">Erlang/OTP 24 ships with JIT compilation</a> and Elixir developers don’t have to do anything to reap its benefits. There are many other features in Erlang/OTP 24 to look forwards to and Elixir v1.12 provides integration with many of them: such as support for 16bit floats in bitstrings as well as performance improvements in the compiler and during code evaluation.</p> <p>Another excellent feature in Erlang/OTP 24 is the implementation of <a href="http://www.erlang.org/eeps/eep-0054.html">EEP 54</a>, which provides extended error information for many functions in Erlang’s stdlib. Elixir v1.12 fully leverages this feature to improve reporting for errors coming from Erlang. For example, in earlier OTP versions, inserting an invalid argument into an ETS table that no longer exists would simply error with <code class="language-plaintext highlighter-rouge">ArgumentError</code>:</p> <script type="text/javascript" src="https://asciinema.org/a/1s79Cwf2JvSLYihAahIobVyBm.js" id="asciicast-1s79Cwf2JvSLYihAahIobVyBm" async=""></script> <noscript><p><a href="https://asciinema.org/a/1s79Cwf2JvSLYihAahIobVyBm">See the example in asciinema</a></p></noscript> <p>However, in Elixir v1.12 with Erlang/OTP 24:</p> <script type="text/javascript" src="https://asciinema.org/a/4l1ORaVDVdHB7Gi5DccIYFgSL.js" id="asciicast-4l1ORaVDVdHB7Gi5DccIYFgSL" async=""></script> <noscript><p><a href="https://asciinema.org/a/4l1ORaVDVdHB7Gi5DccIYFgSL">See the example in asciinema</a></p></noscript> <p>Finally, note Rebar v2 no longer works on Erlang/OTP 24+. Mix defaults to Rebar v3 since Elixir v1.4, so no changes should be necessary by the vast majority of developers. However, if you are explicitly setting <code class="language-plaintext highlighter-rouge">manager: :rebar</code> in your dependency, you want to move to Rebar v3 by removing the <code class="language-plaintext highlighter-rouge">:manager</code> option. Compatibility with unsupported Rebar versions will be removed from Mix in the future.</p> <h2 id="stepped-ranges">Stepped ranges</h2> <p>Elixir has had support for ranges from before its v1.0 release. Ranges support only integers and are inclusive, using the mathematic notation <code class="language-plaintext highlighter-rouge">a..b</code>. Ranges in Elixir are either increasing <code class="language-plaintext highlighter-rouge">1..10</code> or decreasing <code class="language-plaintext highlighter-rouge">10..1</code> and the direction of the range was always inferred from the first and last positions. Ranges are always lazy as its values are emitted as they are enumerated rather than being computed upfront.</p> <p>Unfortunately, due to this inference, it is not possible to have empty ranges. For example, if you want to create a list of <code class="language-plaintext highlighter-rouge">n</code> elements, you cannot express it with a range from <code class="language-plaintext highlighter-rouge">1..n</code>, as <code class="language-plaintext highlighter-rouge">1..0</code> (for <code class="language-plaintext highlighter-rouge">n=0</code>) is a decreasing range with two elements.</p> <p>Elixir v1.12 supports stepped ranges via <a href="https://hexdocs.pm/elixir/1.12/Kernel.html#..///3">the <code class="language-plaintext highlighter-rouge">first..last//step</code> notation</a>. For example: <code class="language-plaintext highlighter-rouge">1..10//2</code> will emit the numbers <code class="language-plaintext highlighter-rouge">1</code>, <code class="language-plaintext highlighter-rouge">3</code>, <code class="language-plaintext highlighter-rouge">5</code>, <code class="language-plaintext highlighter-rouge">7</code>, and <code class="language-plaintext highlighter-rouge">9</code>. You can consider the <code class="language-plaintext highlighter-rouge">//</code> operator as an equivalent to “range division”, as it effectively divides the number of elements in the range by <code class="language-plaintext highlighter-rouge">step</code>, rounding up on inexact scenarios. Steps can be either positive (increasing ranges) or negative (decreasing ranges). Stepped ranges bring more expressive power to Elixir ranges and they elegantly solve the empty range problem, as they allow the direction of the steps to be explicitly declared instead of inferred.</p> <p>As of Elixir v1.12, implicitly decreasing ranges are soft-deprecated and warnings will be emitted in future Elixir versions based on our <a href="https://hexdocs.pm/elixir/compatibility-and-deprecations.html#deprecations">deprecation policy</a>.</p> <h2 id="then2-and-tap2"><code class="language-plaintext highlighter-rouge">then/2</code> and <code class="language-plaintext highlighter-rouge">tap/2</code></h2> <p>Two new functions have been added to <code class="language-plaintext highlighter-rouge">Kernel</code> module, in order to ease working with pipelines. <a href="https://hexdocs.pm/elixir/1.12/Kernel.html#tap/2"><code class="language-plaintext highlighter-rouge">tap/2</code></a> passes the given argument to an anonymous function, returning the argument itself. <a href="https://hexdocs.pm/elixir/1.12/Kernel.html#then/2"><code class="language-plaintext highlighter-rouge">then/2</code></a> passes the given argument to an anonymous function, returning the result. The following:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="s2">"hello world"</span> <span class="o">|&gt;</span> <span class="n">tap</span><span class="p">(</span><span class="o">&amp;</span><span class="no">IO</span><span class="o">.</span><span class="n">puts</span><span class="o">/</span><span class="mi">1</span><span class="p">)</span> <span class="o">|&gt;</span> <span class="n">then</span><span class="p">(</span><span class="o">&amp;</span><span class="no">Regex</span><span class="o">.</span><span class="n">scan</span><span class="p">(</span><span class="sr">~r/\w+/</span><span class="p">,</span> <span class="nv">&amp;1</span><span class="p">))</span> </code></pre></div></div> <p>Is equivalent to this:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="s2">"hello world"</span> <span class="o">|&gt;</span> <span class="p">(</span><span class="k">fn</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="no">IO</span><span class="o">.</span><span class="n">puts</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span> <span class="k">end</span><span class="p">)</span><span class="o">.</span><span class="p">()</span> <span class="o">|&gt;</span> <span class="p">(</span><span class="o">&amp;</span><span class="no">Regex</span><span class="o">.</span><span class="n">scan</span><span class="p">(</span><span class="sr">~r/\w+/</span><span class="p">,</span> <span class="nv">&amp;1</span><span class="p">))</span><span class="o">.</span><span class="p">()</span> </code></pre></div></div> <p>Both <code class="language-plaintext highlighter-rouge">tap/2</code> and <code class="language-plaintext highlighter-rouge">then/2</code> are implemented as macros, and compiler improvements available on Erlang/OTP 24 ensure the intermediate anonymous functions is optimized away, which guarantees the idioms above do not have any performance impact on your code.</p> <h2 id="iex-improvements">IEx improvements</h2> <p>IEx got two important quality of life improvements in this release. Hitting tab after a function invocation will show all of the arguments for said function and it is now possible to paste code with pipelines in the shell. See both features in action below:</p> <script type="text/javascript" src="https://asciinema.org/a/IMSAZUqLFlmGRsPk4gKuJ3tN0.js" id="asciicast-IMSAZUqLFlmGRsPk4gKuJ3tN0" async=""></script> <noscript><p><a href="https://asciinema.org/a/IMSAZUqLFlmGRsPk4gKuJ3tN0">See the example in asciinema</a></p></noscript> <h2 id="additional-functions">Additional functions</h2> <p>Elixir v1.12 has also added many functions across the standard library. The <code class="language-plaintext highlighter-rouge">Enum</code> module received additions such as <code class="language-plaintext highlighter-rouge">Enum.count_until/2</code>, <code class="language-plaintext highlighter-rouge">Enum.product/1</code>, <code class="language-plaintext highlighter-rouge">Enum.zip_with/2</code>, and more. The <code class="language-plaintext highlighter-rouge">Integer</code> module now includes <code class="language-plaintext highlighter-rouge">Integer.pow/2</code> and <code class="language-plaintext highlighter-rouge">Integer.extended_gcd/2</code>.</p> <p>The <code class="language-plaintext highlighter-rouge">Code</code> module got a <a href="https://hexdocs.pm/elixir/1.12/Code.html#cursor_context/2"><code class="language-plaintext highlighter-rouge">cursor_context/2</code></a> function, which is now used to power <code class="language-plaintext highlighter-rouge">IEx</code> autocompletion and it is <a href="https://user-images.githubusercontent.com/17034772/115117125-533b2900-9f9d-11eb-94a9-a2cf2ccb7388.mp4">used by projects such as Livebook to provide intellisense</a>.</p> <p>The EEx application has also been extended to provide metadata on text segments. This has enabled the Surface and Phoenix LiveView teams to implement <a href="https://github.com/phoenixframework/phoenix_live_view/pull/1440">a new template language called HEEx</a>, which validates both HTML and EEx. Finally, the <code class="language-plaintext highlighter-rouge">Registry</code> module supports the <code class="language-plaintext highlighter-rouge">:compressed</code> option, which is useful for GraphQL applications managing hundreds of thousands of subscriptions via <a href="http://absinthe-graphql.org/">Absinthe</a>.</p> <p>For a complete list of all changes, see the <a href="https://github.com/elixir-lang/elixir/releases/tag/v1.12.0">full release notes</a>. Check <a href="/install.html">the Install section</a> to get Elixir installed and read our <a href="https://hexdocs.pm/elixir/introduction.html">Getting Started guide</a> to learn more.</p> <p>Have fun!</p> José Valim 2021-04-02T00:00:00+00:00 /blog/2021/04/02/marketing-and-sales-intelligence-with-elixir-at-pepsico <p><em>Welcome to our series of case studies about companies using Elixir in production. <a href="/cases.html">See all cases</a> we have published so far.</em></p> <p>PepsiCo is one of the world’s leading food and beverage companies serving more than 200 countries and territories around the world. Today Elixir is used at varying capacities inside PepsiCo by six different teams. This article explores how the Search Marketing and Sales Intelligence Platform teams adopted and use Elixir to build internal tools.</p> <p>Although we will explore only two teams in this article, <a href="https://www.pepsicojobs.com/teams-ecommerce">PepsiCo is hiring Elixir engineers across multiple teams</a>. Let’s get started.</p> <p><img src="/images/cases/bg/pepsico.jpg" alt="PepsiCo" /></p> <h2 id="the-first-steps">The first steps</h2> <p>The project that would become the first Elixir project and open the door for future Elixir applications inside PepsiCo was started by Jason Fertel back in 2016.</p> <p>Initially, the application provided workflow automation for managing search marketing operations on multiple web platforms. The product was a success and ended up integrated into PepsiCo in 2018.</p> <p>Now, the Elixir application plays a central role in a data pipeline that empowers PepsiCo’s marketing and sales teams with tools to query, analyze, and integrate with several search marketing partners.</p> <p>The pipeline starts with the Data Engineering team, which collects and stores data into <a href="https://www.snowflake.com/">Snowflake Data Cloud</a>. The Elixir application reads data from Snowflake’s platform, pre-process, and stores it in two databases: <a href="https://www.postgresql.org/">PostgreSQL</a> or <a href="https://druid.apache.org/">Apache Druid</a>, according to the data characteristics. Finally, a Phoenix application serves this data to internal teams and communicates directly with third-party APIs.</p> <h2 id="why-elixir">Why Elixir?</h2> <p>Elixir helps PepsiCo eCommerce focus and get things done fast. “Elixir allows our team to develop quickly with confidence,” says David Antaramian, a Software Engineering Manager at PepsiCo. “In turn, that lets us deliver value to the business quickly, and it’s the reason we’ve stuck with the language. Whether it’s streaming changes to the front-end or orchestrating concurrent data operations across multiple storage systems, Elixir offers a robust developer experience that translates to a great consumer experience.”</p> <p>Different Elixir features came together to help the PepsiCo team build compelling development and user experiences. Thanks to its functional and extensible aspects, PepsiCo used Elixir to create a domain-specific language that translates business queries into data structures sent to different stores. This gave them a stable foundation where they can continually add new queries and integrations, even as they grow in complexity.</p> <p>Furthermore, the reports generated by PepsiCo’s marketing and sales teams often have to query different tables or even separate storages, all while juggling long-running connections to different third-party APIs. Elixir’s programming model, inherited from the Erlang Virtual Machine, makes it trivial to run all of these operations concurrently, leading to fast and rich user interactions while the development team remains focused on delivering features.</p> <h2 id="libraries-and-frameworks">Libraries and frameworks</h2> <p>David Antaramian is quick to praise the Erlang runtime and its standard library. He says: “Since we are working with large amounts of data, it is also essential to avoid hitting the database whenever possible. Thankfully, Erlang ships with an in-memory table storage called <a href="http://www.erlang.org/doc/man/ets.html">ETS</a>, which we use to store hundreds of thousands of rows”.</p> <p>The Erlang standard library was also handy when communicating to some data stores. In particular, the Snowflake platform requires ODBC connections. The PepsiCo team built a library called <a href="https://github.com/pepsico-ecommerce/snowflex">Snowflex</a>, designed on top of <a href="http://www.erlang.org/doc/man/odbc.html">Erlang’s built-in ODBC drivers</a>.</p> <p>The Elixir ecosystem nicely complements the Erlang one. The front-end, written in React, talks to the server via <a href="http://absinthe-graphql.org/">the Absinthe GraphQL toolkit</a> running on top of <a href="http://phoenixframework.org/">the Phoenix web framework</a>. The <a href="https://github.com/elixir-ecto/ecto">Ecto database library</a> manages the communication to PostgreSQL. They also use the <a href="https://github.com/handnot2/esaml">esaml</a> and <a href="https://github.com/handnot2/samly">Samly</a> libraries to provide authentication within PepsiCo’s organization - another example of leveraging the tools within both Erlang and Elixir communities.</p> <p>Finally, the team also recognizes the <a href="https://erlef.org/">Erlang Ecosystem Foundation</a>’s efforts, which PepsiCo are sponsors of, particularly the Observability Working Group. David remarks: “The adoption of Telemetry by the ecosystem has been a massive help in bringing monitoring visibility and metrics to our system. Now when we see spikes in one place, we can easily correlate them with other system parts”.</p> <h2 id="hiring">Hiring</h2> <p>Today there are approximately 40+ Elixir engineers within PepsiCo distributed across six teams. Eight of those engineers are part of the Search Marketing and Sales Intelligence Platform teams.</p> <p>While the team recognizes that there aren’t as many Elixir engineers compared to communities like JavaScript, they were effective in hiring qualified Elixir candidates. Chase Gilliam, a Software Engineering Manager at PepsiCo, explains: “We have met many engineers that, like us, found Elixir due to being burned out by previous experiences. So when it came to hiring, many Elixir candidates had a mindset similar to ours, which ultimately sped up the process.”</p> <p>This initial group of Elixir engineers paved the way for the language’s growth inside PepsiCo. David adds: “At first, we looked for engineers with Elixir experience to help us build a team that could guide other developers. Then we extended the pool to anyone who has a functional programming background and now to developers with either Ruby or Erlang experience. However, if someone is the right candidate, we onboard them even if they have no Elixir experience and train them”. He continues: “We also make extensive use of the learning resources available in the community, such as conferences, books, online courses, and others.”</p> <p>As the team grew, they adopted best practices and saw the quality of the codebase improve at the same time. Chase concludes: “At the beginning, there were some large modules in our application. Luckily, refactoring in a functional programming language is straightforward, thanks to immutability and limited side-effects. Adopting tools like Credo, ExDoc, and the code formatter was also essential to standardize how we use Elixir internally.” For those interested in learning more about the different use cases for Elixir inside PepsiCo and help continue its growth, <a href="https://www.pepsicojobs.com/teams-ecommerce">they are hiring</a>.</p> José Valim 2021-02-03T00:00:00+00:00 /blog/2021/02/03/social-messaging-with-elixir-at-community <p><em>Welcome to our series of case studies about companies using Elixir in production. <a href="/cases.html">See all cases</a> we have published so far.</em></p> <p><a href="https://community.com/">Community</a> is a platform that enables instant and direct communication with the people you want to reach, using the simplicity of text messaging. Used by names like Paul McCartney, Metallica, and Barack Obama, Community connects small businesses, stars, and high-profile individuals directly to their audiences.</p> <p>Community is powered by the Erlang Ecosystem, with Elixir and RabbitMQ playing central roles. This article gives an overview of the system and the tools used to handle spikes of million of users caused by events such as this tweet:</p> <blockquote class="twitter-tweet" data-cards="hidden"><p lang="en" dir="ltr">All right, let&#39;s try something new. If you’re in the United States, send me a text at 773-365-9687 — I want to hear how you&#39;re doing, what&#39;s on your mind, and how you&#39;re planning on voting this year. <br /><br />I&#39;ll be in touch from time to time to share what&#39;s on my mind, too. <a href="https://t.co/NX91bSqbtG">pic.twitter.com/NX91bSqbtG</a></p>&mdash; Barack Obama (@BarackObama) <a href="https://twitter.com/BarackObama/status/1308769164190941187?ref_src=twsrc%5Etfw">September 23, 2020</a></blockquote> <script async="" src="https://platform.twitter.com/widgets.js" charset="utf-8"></script> <h2 id="the-first-steps-with-elixir">The first steps with Elixir</h2> <p>Tomas Koci and Ustin Zarubin were the two engineers behind Community’s initial implementation. The company was pivoting from a product they had written in Go and they felt the language was not expressive enough for the products they were building. So when faced with the challenge of developing a social messaging platform on top of SMS, they were open to trying a different stack.</p> <p>Their first encounter with Elixir was a casual one. They were chatting about the challenges ahead of them when their roommate mentioned Elixir. Shortly after, things started to click. They both had a physics background, so they found the functional paradigm quite intuitive. The Erlang VM also has its origins in telecommunications, and they were building a telecom centric product, which gave them more confidence.</p> <p>Besides the technological aspect, they also began to be active in the Elixir community. Tomas recaps: “we started attending the Elixir meetups happening here in Chattanooga. We met many developers, heard about production cases, and learned how companies like Bleacher Report were using Elixir at scale”. From then on, they were sold on giving Elixir a try.</p> <p>They started their prototype in January 2018, with the intent of onboarding dozens of users. They were learning Elixir while developing the system and reaching out to potential users.</p> <p>Their first challenge was in May 2018, when one of their users announced his phone number, managed by Community, to millions of viewers. Tomas still remembers that day: “It was a Saturday night, around 11:00 pm when we saw an influx of users. It caught us by surprise and, after 10 hours, more than 400 thousand users had signed up”. This influx of users stressed the system in unexpected ways, especially when it came to their upstream integrations. They had to patch the system to ensure they would not overload external systems or run over API limits they were required to conform to.</p> <p>This event also gave them insights into the types of spikes and traffic patterns the system would have to handle at scale. Early engineering hire Jeffrey Matthias urged them to break their application into different services, making it easy to scale each service individually, and he and Tomas decided to have those services communicate via message queues.</p> <h2 id="the-next-millions-of-users">The next millions of users</h2> <p>By October 2018, the company received funding and the newly-hired engineering team of five people, began to split the original application into services that could handle sharp increases in demand and operate at scale. Shortly after, they had their next challenge in hand: Metallica had just signed up with the platform and they were going to do an early launch with their fans on Feb 1st, 2019.</p> <p>The team is glad to report the announcement was a success with no hiccups on their end. They were then five backend engineers who tackled everything from architectural design and development to setting up and operating the whole infrastructure.</p> <p>Community was officially unveiled in May 2019, <a href="https://www.billboard.com/amp/articles/business/8543190/why-hundreds-music-stars-giving-fans-phone-numbers-community-app">attracting hundreds of music stars shortly after</a>. Fourteen months later, <a href="https://twitter.com/barackobama/status/1308769164190941187">Barack Obama tweeted to millions his phone number powered by Community</a>.</p> <h2 id="the-current-architecture">The current architecture</h2> <p>Today, more than 60 services with distinct responsibilities power Community, such as:</p> <ul> <li>A message hub between community leaders and members</li> <li>User data management</li> <li>Media services (video, audio, images)</li> <li>Systems for Community’s internal team</li> <li>Data science and machine learning</li> <li>Billing, administration, etc</li> </ul> <p>The vast majority of those services run Elixir, with Python covering the data science and machine learning endpoints, and Go on the infrastructure side.</p> <p><a href="https://www.rabbitmq.com/">RabbitMQ</a> handles the communication between services. The Erlang-backed message queue is responsible for broadcasting messages and acting as <a href="https://andrealeopardi.com/posts/rpc-over-rabbitmq-with-elixir/">their RPC backbone</a>. Messages between services are encoded with Protocol Buffers via <a href="https://github.com/elixir-protobuf/protobuf">the protobuf-elixir library</a>.</p> <p>Initially, they used <a href="http://github.com/elixir-lang/gen_stage/">the GenStage library</a> to interface with RabbitMQ, but they have migrated to the higher level <a href="https://github.com/dashbitco/broadway">Broadway</a> library over the last year. Andrea Leopardi, one of their engineers, outlines their challenges: “Our system has to handle different traffic patterns when receiving and delivering data. Incoming data may arrive at any time and be prone to spikes caused by specific events powered by actions within Communities. On the other hand, we deliver SMSes in coordination with partners who impose different restrictions on volumes, rate limiting, etc.”</p> <p>He continues: “both GenStage and Broadway have been essential in providing abstractions to handle these requirements. They provide back-pressure, ensure that spikes never overload the system, and guarantee we never send more messages than the amount defined by our delivery partners”. As they implemented the same patterns over and over in different services, they found Broadway to provide the ideal abstraction level for them.</p> <p>Their most in-demand service, the message hub, is powered by only five machines. They use <a href="https://mesos.apache.org/">Apache Mesos</a> to coordinate deployments.</p> <h2 id="growing-the-team">Growing the team</h2> <p>Community’s engineering team has seen stable growth over the last two years. Today they are 25 backend engineers, the majority being Elixir devs, and the company extends beyond 120 employees.</p> <p>Karl Matthias, who joined early on, believes the challenges they face and the excitement for working on a new language has been positive for hiring talent. He details: “we try to hire the best production engineers we can, sometimes they know Elixir, sometimes they don’t. Our team has generally seen learning Elixir as a positive and exciting experience”.</p> <p>The team is also happy and confident about the stability Elixir provides. Karl adds: “Elixir supervisors have our back every time something goes wrong. They automatically reestablish connections to RabbitMQ, they handle dropped database connections, etc. The system has never gone wrong to the point our infrastructure layer had to kick-in, which has been quite refreshing.”</p> <p>The Community team ended our conversation with a curious remark. They had just shut down their first implementation of the system, the one that received a sudden spike of four hundred thousand users on a Saturday night. Tomas concludes: “it is pretty amazing that the service we implemented while learning Elixir has been running and operating in production just fine, even after all of these milestones. And that’s generally true for all of our services: once deployed, we can mostly forget about them”.</p> José Valim 2021-01-13T00:00:00+00:00 /blog/2021/01/13/orchestrating-computer-vision-with-elixir-at-v7 <p><em>Welcome to our series of case studies about companies using Elixir in production. <a href="/cases.html">See all cases</a> we have published so far.</em></p> <p><a href="https://www.v7labs.com">V7</a> is a web platform to create the sense of sight. A hub for machine learning and software engineers to develop their computer vision projects with data set management, image/video labeling, and one-click model training to automate any visual task.</p> <p>Founded in 2018 by Alberto Rizzoli and Simon Edwardsson, V7 uses Elixir, Phoenix, and Cowboy to power their web platform, responsible for managing large amounts of data and orchestrating dozens of Python nodes to carry out machine learning jobs. They have <a href="https://www.notion.so/V7-Labs-raises-3-million-to-empower-AI-teams-with-automated-training-data-workflows-2c9b36d2043e44f3b536efae0a204632">recently closed a $3M seed round</a>, and they are currently <a href="https://www.v7labs.com/working-at-v7">hiring backend engineers to augment their Elixir team</a>.</p> <p><img src="/images/cases/bg/v7.png" alt="V7" /></p> <h2 id="visual-tasks">Visual tasks</h2> <p>Throughout the years, we have been continuously automating visual tasks to speed up manual processes and reduce the rate of errors. For example:</p> <ul> <li> <p>Routine inspection of infrastructure: oil pipelines and offshore oil rigs require constant examination against corrosion. Once there is too much rust, it can damage the pipeline and cause leakage. Nowadays, you can use drones to take pictures and automate the detection of oxidated spots.</p> </li> <li> <p>Medical examination: there is a growing use of digital pathology to assist doctors in diagnosing diseases. For example, during a biopsy of possible liver cancer, doctors use a microscope to visualize human tissue and stitch together an image of the cells, which are then individually analyzed. AI can double-check these images and help speed up problematic cells in case of positives.</p> </li> <li> <p>Agriculture and farming: a wine producer may want to count grapes in a vineyard to estimate the wine production for a given season with higher precision. Farmers may use video to assess the health and the amount of exercise on free-range chickens and pigs.</p> </li> <li> <p>Visual automation also plays a growing role in quality assurance and robotics: a fast-food manufacturer can use cameras to identify fries with black spots, while harvesters may use robots to pick apples from trees.</p> </li> </ul> <p>Neural networks are at the heart of these tasks, and there is a growing need to automate the creation of the networks themselves.</p> <h2 id="automating-ai">Automating AI</h2> <p>Training a neural network for image and video classification often requires multiple steps. First, you annotate images and frames with bounded-boxes, polygons, skeletons, and many other formats. The annotations are then labeled and used to train computer vision models. Labeled annotations are also used to verify models against biases, outliers, and over/underfitting.</p> <p>For many AI companies, this process exists in a loop as they continuously refine datasets and models. V7 helps teams manage and automate these steps, accelerating the creation of high-quality training data by 10-100x. Users may then export this data or use it to create neural networks directly via the platform.</p> <iframe width="560" height="315" style="margin: 0 auto 30px; display: block" src="https://www.youtube.com/embed/SvihDSAY4TQ" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe> <p>V7 uses Elixir to orchestrate all of these tasks. The front-end is a Vue.js application that talks to a <a href="https://phoenixframework.org/">Phoenix-powered</a> API. The Phoenix application has to work with a large amount of data across a wide variety of formats. For example, a microscope outputs images in a different format, often proprietary, than a regular laboratory camera.</p> <p>To perform all the machine learning tasks, V7 has a cluster of Python nodes orchestrated by an Elixir application running the <a href="https://github.com/ninenines/cowboy/">Cowboy</a> webserver. Once a Python node comes up, it establishes a WebSocket connection with Cowboy and sends how much memory, CPU, GPU, and other relevant data it has available.</p> <p>The Phoenix-powered backend communicates with the orchestrator using another Erlang VM-based technology: <a href="https://www.rabbitmq.com/">RabbitMQ</a>. For example, when the user tasks to auto-annotate an image, the Vue.js front-end sends a REST request to Phoenix. Phoenix then enqueues a message on RabbitMQ with the image’s location, typically an Amazon S3 bucket. The orchestrator picks this message up, finds an available Python node, and delivers the relevant instructions via WebSockets.</p> <h2 id="ecosystem-and-infrastructure">Ecosystem and Infrastructure</h2> <p>Other tools used by the V7 team are <a href="https://github.com/dashbitco/broadway">Broadway</a> and the Erlang Distribution.</p> <p>V7 has to process and normalize images and videos. For these, they have a separate service that receives RabbitMQ messages and invokes <a href="https://imagemagick.org/">ImageMagick</a> or <a href="https://ffmpeg.org/">FFmpeg</a> accordingly. They use Broadway to receive RabbitMQ messages and to execute these tasks concurrently.</p> <p>The Erlang Distribution helps them broadcast information across nodes. Since they store their multimedia data on S3, they need to generate pre-signed URLs whenever the user wants to see an image or video. However, if users are routed to a different node, they would get a different URL, which would force them to download the asset again. To address this, they use the Erlang Distribution to communicate which URLs they have generated and for which purposes.</p> <p>Overall, their backend runs on Amazon ECS on about four nodes, which talk directly to PostgreSQL. The largest part of their infrastructure is the Python cluster, which takes up to two dozens of machines.</p> <h2 id="learning-and-hiring">Learning and Hiring</h2> <p>Elixir has been present inside the company since day one, back in August 2018. Andrea Azzini, the first engineer at V7, was the one responsible for introducing it. He believed the language would be a good fit for the challenges ahead of them based on his experience running Elixir in production.</p> <p>Simon Edwardsson, their CTO, had to learn the language as they developed the system, but he was able to get up and running quickly, thanks to his previous experiences with Python and Haskell. He remarks: “As a team, we were more familiar with Django, but we were concerned it would not handle well the amount of data and annotations that we manage - which could lead to rewrites or frustrations down the road. From this perspective, the investment in Elixir was worth it, as we never had to do major changes on our backend since we started.”</p> <p>Part of this is thanks to Phoenix’s ability to provide high-level abstractions while making its building blocks accessible to developers: “While there is magic happening inside Phoenix, it is straight-forward to peek under the hood and make sense of everything.”</p> <p>V7 has recently welcomed a new Elixir engineer to their team, making it a total of four, and they are looking for more developers interested in joining them. Historically, more engineers have applied to their machine learning positions, but they also believe many Elixir developers are prepared but don’t consider themselves ready. Simon finishes with an invitation: “We are primarily looking for backend engineers with either existing Elixir experience or willingness to learn on the job. If you are interested in automating computer vision across a large range of industries, <a href="https://www.v7labs.com/working-at-v7">we welcome you to get in touch</a>.”</p> José Valim 2020-12-10T00:00:00+00:00 /blog/2020/12/10/integrating-travel-with-elixir-at-duffel <p><em>Welcome to our series of case studies about companies using Elixir in production. <a href="/cases.html">See all cases</a> we have published so far.</em></p> <p><a href="https://duffel.com/">Duffel</a> is building the new software backbone of the travel industry. Their first product is the Flights API, a developer-friendly platform that enables any business to instantly search flights, make bookings, reserve seats, and sell paid extras like checked bags. Duffel is connected to more than 20 of the world’s largest airlines, including American Airlines, Emirates, and Lufthansa. The company was founded in November 2017 and in 2019 it opened a private beta of their API and has raised $56M total in funding. It now has 40 employees across its offices in London and New York. This article discusses how Duffel has used Elixir as their technology of choice to modernize an industry built on old standards and outdated stacks. If you are interested in shaping the future of travel, <a href="https://duffel.com/careers">Duffel is hiring</a>.</p> <p><img src="/images/cases/bg/duffel.png" alt="Duffel" /></p> <h2 id="why-elixir">Why Elixir?</h2> <p>Today, to access flights and prices from airlines, companies have to go through a Global Distribution System (GDS), often using decades-old systems such as Amadeus and Sabre. Steve Domin, Duffel’s founder / CEO, explains: “The airline industry runs on a legacy data exchange standard called <a href="https://en.wikipedia.org/wiki/EDIFACT">EDIFACT standard</a> and only recently moved to a ‘modern’ SOAP/XML world. Any integration work with a GDS or an airline is always scheduled to take months, and this creates a very high barrier to entry for new businesses.”</p> <p>At its heart, Duffel is building the new operating system for travel. A single request to Duffel’s API may translate into a chain of dozens of requests to different airlines. The response of each request needs to be parsed, normalized, and potentially be hydrated with more outgoing requests. All of this while managing slow responses, timeouts, large data payloads, and more. These challenges made it clear to Steve that Elixir would be a great fit: “We are building a highly concurrent platform with intensive data trafficking. From day one, it was clear the Erlang VM would be a great fit, as it was designed for telecommunication with similar requirements in mind.” They chose the Erlang VM, alongside <a href="https://phoenixframework.org/">the Phoenix web framework</a> and <a href="https://github.com/elixir-ecto/ecto">the Ecto database library</a> as their stack to launch their initial JSON API. They leverage Elixir’s standard library for most of their concurrent work and <a href="https://github.com/qcam/saxy">the Saxy library for XML parsing</a>.</p> <h2 id="growing-with-open-source">Growing with Open Source</h2> <p>When Steve co-founded the company in November 2017, he already had plenty of experience with Elixir. Steve started using the language before it reached 1.0, back in 2013. He started his journey by hacking on <a href="https://github.com/devinus/dynamo">Dynamo</a>, Phoenix’s ancestor, and eventually introduced Elixir at his previous company, by using it for an internal project. He also organized meet-ups in London and contributed to Open Source projects, including some of his own, such as <a href="https://github.com/swoosh/swoosh">Swoosh</a>.</p> <p>The founders joined <a href="https://www.ycombinator.com/">Y Combinator</a> in Summer 2018. Once they came back to London, they hired Alan Kennedy as their first engineer. Alan first heard about Elixir when he and Steve were colleagues at GoCardless. Alan kept an eye on it but never actively used it until he joined Duffel. Alan recalls struggling to jump from a language that promotes mutability to an immutable language like Elixir. Once everything clicked, he acknowledged the new programming model is conceptually much simpler.</p> <p>Since then, the company has grown with a mixture of fresh and experienced engineers, including nearly 70% of the engineering organisation programming in Elixir.</p> <p>Johanna Larsson is one of the most recent engineers to join Duffel. She had already spoken at Elixir Conferences and made meaningful contributions to the ecosystem, such as <a href="https://diff.hex.pm/">the HexDiff project</a>, before she was hired. In her opinion, one of Elixir’s biggest assets is the community, which she considers welcoming and supportive.</p> <p>Duffel has often been able to leverage the ecosystem and reach out to existing solutions. However, they don’t shy away from creating their own and open-sourcing them whenever it makes sense. Overall, the Duffel team has contributed to many areas of the ecosystem. Besides the previously mentioned Swoosh and HexDiff projects, their team members created <a href="https://github.com/msz/hammox">Hammox</a>, <a href="https://github.com/stevedomin/bigflake">Bigflake</a>, the company’s own <a href="https://github.com/duffelhq/paginator/">Paginator</a> library, and others.</p> <h2 id="upcoming-challenges">Upcoming challenges</h2> <p>Duffel engineers have many interesting and exciting challenges ahead of them. For example, as more developers start using the product, they will begin to hit some rate-limits imposed by airlines that they haven’t yet exercised. As one would expect, different airlines have different rules and various constraints, and providing a unified solution has its hurdles.</p> <p>Some of the upcoming improvements are related to their usage of umbrella projects. Duffel started as a monolith, but they eventually migrated to Elixir’s umbrella projects - a mono-repo implementation within Elixir’s tooling - as soon as Phoenix v1.4 was released. Their primary motivation was to separate the communication with different airlines into different services. In the beginning, the services were clear in Steve’s head, but as the team grew, they experienced friction enforcing those boundaries, which led to cyclic dependencies.</p> <p>Luckily, Elixir v1.11 started emitting warnings for cyclic and undeclared dependencies between applications, which forced the Duffel team to revisit the areas that were not strict in the past to increase the quality of the codebase in the long term.</p> <p>The team is also always exploring how to improve their APIs by bringing new approaches and technologies, such as streaming and GraphQL, as well as intelligent ways to optimize their integrations. If you are interested in tackling these and many other challenges while reshaping the travel industry, you can <a href="https://duffel.com/careers">learn more about Duffel’s engineering opportunities</a>.</p> José Valim 2020-11-17T00:00:00+00:00 /blog/2020/11/17/real-time-collaboration-with-elixir-at-slab <p><em>Welcome to our series of case studies about companies using Elixir in production. <a href="/cases.html">See all cases</a> we have published so far.</em></p> <p><a href="https://slab.com/">Slab</a> is a knowledge base and team wiki that democratizes knowledge. Jason Chen started Slab in August 2016, after picking Elixir and Phoenix as the best tools to build real-time collaborative applications. The company has grown to 6 engineers since then, distributed worldwide, and relied upon by more than 7000 companies and customers like Asana, Discord, and Glossier. If you are interested in helping companies become a source of learning and purpose, especially during these times where remote collaboration is essential, <a href="https://slab.com/jobs?ref=elixir">Slab is hiring</a>.</p> <p><img src="/images/cases/bg/slab.png" alt="Slab" /></p> <h2 id="why-elixir">Why Elixir?</h2> <p>Slab was not the first time Jason wrote a collaborative web app. He had previous practice building them in Rails and Node.js and he believed there was a lot to improve in the development experience, especially when it came to working with WebSockets. Both technologies were also troublesome in production, as the team faced issues scaling them vertically and horizontally.</p> <blockquote style="font-size: 24px; color: #444"> <p>I wanted a framework with the same developer experience as Django and Rails, but one that was designed for real-time applications.</p> <p style="font-size: 20px">— Jason Chen, CEO, on the Phoenix web framework</p> </blockquote> <p>Jason doesn’t consider himself a person who is always looking for new things, but he knew he would have to survey the options around him when starting Slab. During this period, he explored two main languages: Go and Elixir. In the end, Jason chose Elixir, thanks to <a href="https://phoenixframework.org/">the Phoenix web framework</a>: “I was looking for a framework that offered a complete toolset for building web apps. I was not interested in making low-level choices, such as which <acronym title="Object-relational mapping">ORM</acronym> to use, which library to pick for parsing requests, etc. I wanted a framework with the same developer experience as Django and Rails, but one that was designed for real-time applications”.</p> <p>Jason gave himself two weeks to build a proof of concept. He wrote a collaborative blog, where multiple users could write a post simultaneously, and comments were added in real-time — all while learning Elixir and the Phoenix framework.</p> <p>The trial went better than expected, and Jason’s journey with Slab had officially begun.</p> <h2 id="growing-with-the-platform">Growing with the platform</h2> <p>Shortly after, Slab was in a private beta with a handful of companies as users. For each major feature they had along the way, Elixir and Phoenix provided the building blocks for it. When they implemented real-time comments, they used Phoenix Channels and Phoenix PubSub. The pattern goes on: “for asynchronous processing, we simply use <a href="https://hexdocs.pm/elixir/Task.html">Elixir tasks</a>”. Later on, to track users editing a document and give each a different cursor color, they used <a href="https://hexdocs.pm/phoenix/Phoenix.Presence.html">Phoenix Presence</a>, a tool that no other web framework offers out-of-the-box.</p> <p>Another leap in Jason’s journey with Slab and Elixir was when he had to learn Erlang/OTP, a group of behaviors that ship as part of Erlang’s standard library for building distributed and fault-tolerant applications.</p> <p>To improve the real-time collaborative editor that is part of Slab, Jason implemented <a href="https://en.wikipedia.org/wiki/Operational_transformation">Operational Transformation</a>. The client runs in the browser, implemented with <a href="https://reactjs.org/">React</a>. As users make changes to the text, their diffs are sent to the server, which arbitrates these updates and synchronizes them across the various clients.</p> <p>Tackling the synchronization problem is not trivial, especially when the application is running on multiple nodes. Here is the challenge they faced. Imagine user Alice has a WebSocket connection to node X and user Bob is connected to node Y. Both Alice and Bob are working on the same text. How can Slab guarantee that changes from both users are applied, so both see the same document once done editing?</p> <p>One could try to solve this problem by keeping the server stateless. Every time the server receives a diff from the client, the server would read the document from the database, apply the changes, normalize the result, and broadcast the clients’ updates. With this approach, the issue is that loading the text from the database on every client update would quickly become expensive, especially as they grow in size. Response times would become higher and the user experience would degrade.</p> <p>When working with Node.js, Jason tried a different approach. If Alice and Bob were writing to the same document, a load balancer would guarantee that both would be routed to the same node. After trying out both Apache and Nginx, he implemented the balancer in Node.js. The overall solution was time-consuming to get right and introduced operational complexities.</p> <p>Luckily, these problems are the bread and butter of Erlang/OTP. Jason knew he needed a stateful abstraction to keep this state on the server. He had already heard about the options the platform provides, but he was unsure which one to pick. Jason recalls: “I remember asking the community if I should use an <a href="https://hexdocs.pm/elixir/Agent.html">Agent</a> or a <a href="https://hexdocs.pm/elixir/GenServer.html">GenServer</a> and everyone was really helpful in providing guidance.” They quickly landed on GenServer as their tool of choice.</p> <p>By default, both GenServer and Agents are local to each node. However, they also support the <code class="language-plaintext highlighter-rouge">:global</code> option, which registers a given name across the cluster. To use this option, they need the Erlang distribution, which they were already using for Phoenix PubSub and Presence, so this was a straight-forward change. This guarantees both Alice and Bob talk to the same GenServer, regardless if they joined node X or node Y.</p> <p>Later on, when running the system in production, the platform continued to impress him. Every time they increased the machine resources, they could see the runtime efficiently using everything it had available, without changes to the code.</p> <h2 id="learning-and-tools">Learning and tools</h2> <p>There are other few notable tools in Slab’s stack.</p> <p>Back in 2017, they migrated to GraphQL <a href="http://absinthe-graphql.org/">powered by Elixir’s Absinthe</a>. There were concerns about adopting the query language, as it was a relatively new technology. Still, they felt it would address a real issue: they had different components in the application needing distinct data, and managing all of these possible combinations was becoming complex. This was one of the main problems GraphQL was designed to solve.</p> <p>They are also running on Google Cloud with Kubernetes (K8s), and, as many Elixir engineers, they wondered <a href="https://dashbit.co/blog/kubernetes-and-the-erlang-vm-orchestration-on-the-large-and-the-small">how the Erlang VM fit in a world with Docker and K8s</a>. Today they run on 6 nodes, 5 of them running application code. The sixth one handles <a href="https://en.wikipedia.org/wiki/Cron">cron jobs</a> and stays on standby for new deployments. They use <a href="https://github.com/mrluc/peerage">the peerage library</a> to establish Distributed Erlang connections between the nodes.</p> <blockquote style="font-size: 24px; color: #444"> <p>We really value Elixir's ability to build complex systems using fewer moving parts. The code is simpler, and the system is easier to operate.</p> <p style="font-size: 20px">— Sheharyar Naseer, engineer</p> </blockquote> <p>Overall the Slab team aims to keep the number of dependencies low, something they believe is made possible by the platform and positively impacts onboarding new developers. Sheharyar Naseer, a member of their engineering team, explains: “We really value Elixir’s ability to build complex systems using fewer moving parts. The code is simpler, and the system is easier to operate, making both experienced and new engineers more productive. We ran in production for more than 3 years without resorting to Redis. We just recently added it because we wanted our caches to survive across deployments. Many other stacks impose technologies like Redis from day one.”</p> <p>This approach also yields benefits when updating libraries. Sheharyar continues: “For the most part, upgrading Erlang, Elixir, and Phoenix is straight-forward. We go through the CHANGELOG, which always emphasizes the main changes we need to perform, and we have a pull request ready after one or two hours. The only time we could not upgrade immediately was when Erlang/OTP removed old SSL ciphers, which broke our HTTP client and we caught it early on during development.”</p> <p>When onboarding engineers, Slab recommends them different books and video courses — many of which you can find <a href="/learning.html">in our learning resources page</a> — so they have the flexibility to choose a medium they are most productive with. New engineers also work on Slab itself and receive guidance through pull requests. They start with small tasks, usually in the client and GraphQL layers, and slowly tackle more complex problems around the database and Erlang/OTP. If you are interested in improving remote collaboration, <a href="https://slab.com/jobs?ref=elixir">learn more about their opportunities on their website</a>.</p> José Valim 2020-10-27T00:00:00+00:00 /blog/2020/10/27/delivering-social-change-with-elixir-at-change.org <p><em>Welcome to our series of case studies about companies using Elixir in production. <a href="/cases.html">See all cases</a> we have published so far.</em></p> <p><a href="https://change.org/">Change.org</a> is a social change platform, with over 400 million users worldwide. Two years ago, their engineering team faced a challenge to migrate their messaging system from an external vendor to an in-house solution, to reduce costs and gain flexibility.</p> <p>This article will discuss how they approached this problem, why they chose Elixir, and how their system grew to deliver more than 1 billion emails per month. Change.org is also <a href="https://www.change.org/careers">hiring Elixir engineers to join their team</a>.</p> <p><img src="/images/cases/bg/change.png" alt="Change.org" /></p> <h2 id="the-path-to-elixir">The path to Elixir</h2> <p>The first step for Change.org’s engineering team was to outline the requirements for their system. The system would receive millions of events, such as campaign updates, new petitions, and more, and it should send emails to all interested parties whenever appropriate. They were looking for an event-driven solution at its core, in which concurrency and fault-tolerance were strong requirements.</p> <p>The next stage was to build proofs-of-concept in different programming languages. Not many companies can afford this step, but Change.org’s team knew the new system was vital to their business and wanted to be thorough in their analysis.</p> <p>Around this time, John Mertens, Director of Engineering, was coming back from parental leave. He used this opportunity to catch up with different technologies whenever possible. That’s when he stumbled upon <a href="https://www.youtube.com/watch?v=XPlXNUXmcgE">José Valim’s presentation at Lambda Days</a>, which discussed two libraries in the Elixir ecosystem: <a href="https://github.com/elixir-lang/gen_stage">GenStage</a> and <a href="https://github.com/dashbitco/flow">Flow</a>.</p> <p>They developed prototypes in four technologies: JRuby, Akka Streams, Node.js, and Elixir. The goal was to evaluate performance, developer experience, and community support for their specific use cases. Each technology had to process 100k messages as fast as possible. John was responsible for the Elixir implementation and put his newly acquired knowledge to use.</p> <p>After two evaluation rounds, the team chose to go ahead with Elixir. Their team of 3 engineers had 18 months to replace the stack they had been using for the last several years with their own Elixir implementation.</p> <h2 id="learning-elixir">Learning Elixir</h2> <p>When they started the project, none of the original team members had prior experience with Elixir. Only Justin Almeida, who joined when the project had been running by six months, had used Elixir before.</p> <p>Luckily, the team felt supported by the different resources available in the community. John recalls: “We were in one of our early meetings discussing how to introduce Elixir into our stack when Pragmatic Programmers announced the <a href="https://pragprog.com/titles/tvmelixir/adopting-elixir/">Adopting Elixir</a> book, which was extremely helpful in answering many of our questions.”</p> <h2 id="the-new-system">The new system</h2> <p>The team developed three Elixir applications to replace the external vendor. The first application processes all incoming events to decide whether an email should go out and to whom.</p> <p>The next application is the one effectively responsible for dispatching the emails. For each message, it finds the appropriate template as well as the user locale and preferences. It then assembles the email and delivers it with the help of a Mail Transfer Agent (MTA).</p> <p>The last application is responsible for analytics. It receives webhook calls from the MTA with batches of different events, which are processed and funneled into their data warehouse for later use.</p> <p>After about four months, they put the new system in production. While Change.org has dozens of different email templates, the initial deployment handled a single and straight-forward case: password recovery.</p> <p>Once the new system was in production, they continued to migrate different use cases to the system, increasing the numbers of handled events and delivered emails day after day. After one year, they had completed the migration ahead of schedule.</p> <h2 id="handling-spikes-and-load-regulation">Handling spikes and load regulation</h2> <p>Today, those applications run on a relatively small number of nodes. The first two applications use 6 to 8 nodes, while the last one uses only two nodes.</p> <p>John explains they are over-provisioned because spikes are relatively frequent in the system: “for large campaigns, a single event may fan out to thousands or hundreds of thousands of emails.”</p> <p>The team was kind enough to share some of their internal graphs. In the example below, you can see a spike of over 10 million messages coming to the system:</p> <p><img src="/images/cases/bg/change-graph.png" alt="Usage at Change.org" /></p> <p>Once this burst happens, all nodes max their CPUs, emitting around 3000 emails per second until they drain the message queue. The whole time memory usage remains at 5%.</p> <p>The back-pressure provided by the <a href="https://github.com/elixir-lang/gen_stage">GenStage</a> library played a crucial role in the system’s performance. Since those applications fetch events from message queues, process them, and submit them into third-party services, they must avoid overloading any part of the stack. GenStage addresses this by allowing the different components, called stages in the library terminology, to communicate how much data they can handle right now. For example, if sending messages to the MTA is slower than usual, the system will naturally get fewer events from the queue.</p> <p>Another essential feature of the system is to work in batches. Receiving and sending data is more efficient and cost-effective if you can do it in groups instead of one-by-one. John has given <a href="https://www.youtube.com/watch?v=t46L9RKmlNo">a presentation at ElixirConf Europe sharing the lessons learned from their first trillion messages</a>.</p> <p>The activity on Change.org has grown considerably over the last year too. The systems have coped just fine. Justin remarks: “everything has been working so well that some of those services are not really on our minds.”</p> <h2 id="working-with-the-ecosystem">Working with the ecosystem</h2> <p>Change.org has relied on and contributed to the ecosystem whenever possible. During the migration, both old and new systems had to access many shared resources, such as <a href="https://haml.info/">HAML templates</a>, Ruby’s I18N configuration files, and <a href="https://sidekiq.org/">Sidekiq’s background queues</a>. Fortunately, they were able to find compatible libraries in the Elixir ecosystem, respectively <a href="https://github.com/nurugger07/calliope">calliope</a>, <a href="https://github.com/change/linguist">linguist</a>, and <a href="https://github.com/akira/exq">exq</a>.</p> <p>Nowadays, some of those libraries have fallen out of flavor. For example, the community has chosen gettext for internationalization, as it is a more widely accepted format. For this reason, Change.org has stepped in and taken ownership of the linguist library.</p> <p>As Change.org adopted Elixir, the ecosystem grew to better support their use cases too. One recent example <a href="https://github.com/dashbitco/broadway">is the Broadway library</a>, which makes it easy to assemble data pipelines. John explains: “Broadway builds on top of GenStage, so it provides the load regulation, concurrency, and fault-tolerance that we need. It also provides batching and partitioning, which we originally had to build ourselves. For new projects, Broadway is our first choice for data ingestion and data processing.”</p> <h2 id="elixir-as-the-default-stack">Elixir as the default stack</h2> <p>As projects migrate to Elixir, Elixir has informally become the default stack at Change.org for backend services. Today they have more than twenty projects. The engineering team has also converged on a common pattern for services in their event driven architecture, built with Broadway and Phoenix.</p> <p>In a nutshell, they use Broadway to ingest, aggregate, and store events in the database. Then they use Phoenix to expose this data, either through APIs, as analytics or as tooling for their internal teams.</p> <p>One recent example is <a href="https://medium.com/making-change-org/our-elixir-bandit-service-e2b6af6eebc4">Change.org’s Bandit service</a>. The service provides a Phoenix API that decides which copy to present to users in various parts of their product. As users interact with these copies, data is fed into the system and analyzed in batches with Broadway. They use this feedback to optimize and make better choices in the future.</p> <p>The team has also grown to ten Elixir developers thanks to the multiple training and communities of practice they have organized internally. Change.org is also looking for Elixir backend engineers, as they aim to bring experience and diversity to their group. Interested developers can <a href="https://www.change.org/careers">learn more about these opportunities on their website</a>.</p> José Valim 2020-10-08T00:00:00+00:00 /blog/2020/10/08/real-time-communication-at-scale-with-elixir-at-discord <p><em>Welcome to our series of case studies about companies using Elixir in production. <a href="/cases.html">See all cases</a> we have published so far.</em></p> <p>Founded in 2015 by Jason Citron and Stan Vishnevskiy, <a href="https://discord.com/">Discord</a> is a permanent, invite-only space for your communities and friends, where people can hop between voice, video, and text, depending on how they want to talk, letting them have conversations in a very natural or authentic way. Today, the service has over 100 million monthly active users from across the globe. Every day people spend 4 billion minutes in conversation on Discord servers, across 6.7 million active servers / communities.</p> <p>From day one, Discord has used Elixir as the backbone of its chat infrastructure. When Discord first adopted the language, they were still working on building a viable business, with many questions and challenges in front of them. Elixir played a crucial role in giving them the desired technological flexibility to grow the company and also became the building block that would allow their systems to run on a massive scale.</p> <p><img src="/images/cases/bg/discord.jpg" alt="Discord" /></p> <h2 id="starting-technologies">Starting technologies</h2> <p>Back in 2015, Discord chose two main languages to build their infrastructure: Elixir and Python. Elixir was initially picked to power the WebSocket gateway, responsible for relaying messages and real-time replication, while Python powered their API.</p> <p>Nowadays, the Python API is a monolith while the Elixir stack contains 20 or so different services. These architectural choices do not represent a dichotomy between the languages but rather a pragmatic decision. Mark Smith, from the Discord team, explains it succinctly: “given the Elixir services would handle much bigger traffic, we designed them in a way where we could scale each service individually.”</p> <p>Discord has also explored other technologies along the way, Go and Rust being two examples, with distinct outcomes. While Discord completely phased out Go after a short foray, Rust has proven to be an excellent addition to their toolbox, boosted by its ability to play well with Elixir and Python.</p> <h2 id="communication-at-scale">Communication at scale</h2> <p>Effective communication plays an essential role when handling millions of connected users concurrently. To put things into perspective, some of Discord’s most popular servers, such as those dedicated to Fortnite and Minecraft, are nearing six hundred thousand users. At a given moment, it is not unlikely to encounter more than two hundred thousand active users in those servers. If someone changes their username, Discord has to broadcast this change to all connected users.</p> <p>Overall, Discord’s communication runs at impressive numbers. They have crossed more than 12 million concurrent users across all servers, with more than 26 million WebSocket events to clients per second, and Elixir is powering all of this.</p> <blockquote style="font-size: 24px; color: #444"> <p>In terms of real time communication, the Erlang VM is the best tool for the job.</p> <p style="font-size: 20px">— Jake Heinz, Lead Software Engineer</p> </blockquote> <p>When we asked their team “Why Elixir?”, Jake Heinz gave a straight-forward answer: “In terms of real time communication, the Erlang VM is the best tool for the job. It is a very versatile runtime with excellent tooling and reasoning for building distributed systems”. Technologically speaking, the language was a natural fit. However, Elixir was still a bet back in 2015: “Elixir v1.0 had just come out, so we were unsure in which direction the language would go. Luckily for us, we have been pleased with how the language has evolved and how the community shaped up.”</p> <h2 id="the-chat-infrastructure-team">The chat infrastructure team</h2> <p>To power their chat messaging systems, Discord runs a cluster with 400-500 Elixir machines. Perhaps, the most impressive feat is that Discord’s chat infrastructure team comprises five engineers. That’s right: five engineers are responsible for 20+ Elixir services capable of handling millions of concurrent users and pushing dozens of millions of messages per second.</p> <p>Discord also uses Elixir as the control plane of their audio and video services, also known as signaling, which establishes communication between users. C++ is then responsible for media streaming, a combination that altogether runs on 1000+ nodes.</p> <p>The Elixir services communicate between them using Distributed Erlang, the communication protocol that ships as part of the Erlang Virtual Machine. By default, Distributed Erlang builds a fully meshed network, but you can also ask the Erlang VM to leave the job of outlining the topology up to you, by setting the aptly named <code class="language-plaintext highlighter-rouge">-connect_all false</code> flag. The Discord team sets this option to assemble a partially meshed network with <a href="https://etcd.io/">etcd</a> being responsible for service discovery and hosting shared configuration.</p> <p>The chat infrastructure developers are not the only ones touching the Elixir codebases. According to Mark Smith, this is an important part of Discord’s culture: “We don’t work in silos. So a Python developer may have to work on the Elixir services when building a new feature. We will spec out the feature together, figure out the scalability requirements, and then they will work on a pull request, which we will review and help them iterate on it.”</p> <h2 id="community-and-challenges">Community and challenges</h2> <p>To run at this scale, Discord learned how to leverage the Erlang VM’s power, its community, and when to recognize challenges that require them to reach for their own solutions.</p> <p>For example, Discord uses <a href="https://github.com/ninenines/cowboy/">Cowboy</a> for handling WebSocket connections and TCP servers. To manage data bursts and provide load regulation, such as back-pressure and load-shedding, they use <a href="https://github.com/elixir-lang/gen_stage">GenStage</a>, which they have <a href="https://discord.com/blog/how-discord-handles-push-request-bursts-of-over-a-million-per-minute-with-elixirs-genstage">discussed in detail in the past</a>.</p> <p>Other times, the efforts of the company and the community go hand in hand. That was the case when Discord used <a href="https://github.com/rusterlium/rustler">the Rustler project</a>, which provides a safe bridge between Elixir and Rust, to <a href="https://discord.com/blog/using-rust-to-scale-elixir-for-11-million-concurrent-users">scale to 11 million concurrent users</a>. They used the Rustler to hook a custom data structure built in Rust directly into their Elixir services.</p> <p>However, the team has made abundantly clear that the powerhouse is the Erlang platform. Every time they had to push their stack forward, they never felt cornered by the technology. Quite the opposite, their engineers could always build efficient solutions that run at Discord’s scale, often in a few hundred lines of code. Discord frequently gives these projects back to the community, as seen in <a href="https://github.com/discord/manifold">Manifold</a> and <a href="https://github.com/discord/zen_monitor">ZenMonitor</a>.</p> <p>The Discord team also adapted quickly when things went wrong. For instance, they attempted twice to use <a href="https://www.erlang.org/doc/man/mnesia.html">Mnesia</a> in production —a database that ships as part of Erlang’s standard library. They tried Mnesia in persistent and in-memory modes, and the database nodes would often fall behind in failure scenarios, sometimes being unable to ever catch up. Eventually they ditched Mnesia altogether and built the desired functionality with Erlang’s builtin constructs, such as GenServer and ETS. Nowadays, they resolve these same failure scenarios within 2-3 seconds.</p> <h2 id="mastering-elixir">Mastering Elixir</h2> <p>None of the chat infrastructure engineers had experience with Elixir before joining the company. They all learned it on the job. Team members Matt Nowack and Daisy Zhou report initially struggling to understand how all of their services communicate. Matt adds: “In the beginning, it was hard to accept all of the guarantees that Erlang VM provides. I’d worry about data races and concurrency issues that were impossible to happen”. Eventually, they took these guarantees to heart and found themselves more productive and more capable of relying on the platform and its tools. Matt continues: “The introspection tools the Erlang VM provides is the best in class. We can look at any VM process in the cluster and see its message queue. We can use the remote shell to connect to any node and debug a live system. All of this has helped us countless times.”</p> <p>Running at Discord’s scale adds its own dimension to mastering the language, as they need to familiarize with the abstractions for providing concurrency, distribution, and fault-tolerance. Nowadays, frameworks such as Nerves and Phoenix handle these concerns for developers, but the underlying building blocks are always available for engineers assembling their own stack, such as the Discord team.</p> <p>In the end, Jake summarized how crucial Elixir and the Erlang VM have been at Discord and how it affected him personally: “What we do in Discord would not be possible without Elixir. It wouldn’t be possible in Node or Python. We would not be able to build this with five engineers if it was a C++ codebase. Learning Elixir fundamentally changed the way I think and reason about software. It gave me new insights and new ways of tackling problems.”</p> José Valim 2020-10-06T00:00:00+00:00 /blog/2020/10/06/elixir-v1-11-0-released <p>Over the last releases, the Elixir team has been focusing on the compiler, both in terms of catching more mistakes at compilation time and making it faster. Elixir v1.11 has made excellent progress on both fronts. This release also includes many other goodies, such as tighter Erlang integration, support for more guard expressions, built-in datetime formatting, and other calendar enhancements.</p> <p>During this period, we have also started <a href="/cases.html">publishing a series of production cases</a> on our website, featuring Elixir’s usage at <a href="/blog/2020/08/20/embedded-elixir-at-farmbot/">Farmbot</a> and <a href="/blog/2020/09/24/paas-with-elixir-at-Heroku/">Heroku</a>, with many more cases coming soon.</p> <p>For now, let’s focus on what’s new in Elixir v1.11.</p> <h2 id="tighter-erlang-integration">Tighter Erlang integration</h2> <p>On the footsteps of v1.10, we have further integrated with Erlang’s new logger by adding four new log levels: <code class="language-plaintext highlighter-rouge">notice</code>, <code class="language-plaintext highlighter-rouge">critical</code>, <code class="language-plaintext highlighter-rouge">alert</code>, and <code class="language-plaintext highlighter-rouge">emergency</code>, matching all log levels found in the Syslog standard. The <code class="language-plaintext highlighter-rouge">Logger</code> module now supports structured logging by passing maps and keyword lists to its various functions. It is also possible to specify the log level per module, via the <a href="https://hexdocs.pm/logger/Logger.html#put_module_level/2"><code class="language-plaintext highlighter-rouge">Logger.put_module_level/2</code></a> function. Log levels per application will be added in future releases.</p> <p>IEx also has been improved to show the documentation for Erlang modules directly from your Elixir terminal. For example, here is a clip of me accessing the documentation for <a href="erlang.org/doc/man/gen_server.html">Erlang’s gen_server module</a>:</p> <script id="asciicast-1Kqwwkn0wMl0feePvWQwHe85G" src="https://asciinema.org/a/1Kqwwkn0wMl0feePvWQwHe85G.js" async=""></script> <noscript><p><a href="https://asciinema.org/a/iAOk0yaZtQDsuJqn2sXa1FRQW">See the example in asciinema</a></p></noscript> <p>This works with Erlang/OTP 23+ and requires Erlang modules to have been compiled with documentation chunks. A huge thank you to the Erlang/OTP team and the Documentation Working Group of the <a href="https://erlef.org/">Erlang Ecosystem Foundation</a> for making this possible.</p> <h2 id="compiler-checks-application-boundaries">Compiler checks: application boundaries</h2> <p>Elixir v1.11 builds on top of the recently added compilation tracers to track application boundaries. From this release, Elixir will warn if you invoke a function from an existing module but this module does not belong to any of your listed dependencies.</p> <p>These two conditions may seem contradictory. After all, if a module is available, it must have come from a dependency. This is not true in two scenarios:</p> <ul> <li> <p>Modules from Elixir and Erlang/OTP are always available - even if their applications are not listed as a dependency</p> </li> <li> <p>In an umbrella project, because all child applications are compiled within the same VM, you may have a module from a sibling project available, even if you don’t depend on said sibling</p> </li> </ul> <p>This new compiler check makes sure that all modules that you invoke are listed as part of your dependencies, emitting a warning like below otherwise:</p> <div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>:ssl.connect/2 defined in application :ssl is used by the current application but the current application does not directly depend on :ssl. To fix this, you must do one of: 1. If :ssl is part of Erlang/Elixir, you must include it under :extra_applications inside "def application" in your mix.exs 2. If :ssl is a dependency, make sure it is listed under "def deps" in your mix.exs 3. In case you don't want to add a requirement to :ssl, you may optionally skip this warning by adding [xref: [exclude: :ssl] to your "def project" in mix.exs </code></pre></div></div> <p>This comes with extra benefits in umbrella projects, as it requires applications to depend on the siblings they depend on, which will fail if there are any cyclic dependencies.</p> <h2 id="compiler-checks-data-constructors">Compiler checks: data constructors</h2> <p>In Elixir v1.11, the compiler also tracks structs and maps fields across a function body. For example, imagine you wanted to write this code:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="n">drive?</span><span class="p">(%</span><span class="no">User</span><span class="p">{</span><span class="ss">age:</span> <span class="n">age</span><span class="p">}),</span> <span class="k">do</span><span class="p">:</span> <span class="n">age</span> <span class="o">&gt;=</span> <span class="mi">18</span> </code></pre></div></div> <p>If there is either a typo on the <code class="language-plaintext highlighter-rouge">:age</code> field or the <code class="language-plaintext highlighter-rouge">:age</code> field was not yet defined, the compiler will fail accordingly. However, if you wrote this code:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="n">drive?</span><span class="p">(%</span><span class="no">User</span><span class="p">{}</span> <span class="o">=</span> <span class="n">user</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="n">user</span><span class="o">.</span><span class="n">age</span> <span class="o">&gt;=</span> <span class="mi">18</span> </code></pre></div></div> <p>The compiler would not catch the missing field and an error would only be raised at runtime. With v1.11, Elixir will track the usage of all maps and struct fields within the same function, emitting warnings for cases like above:</p> <div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>warning: undefined field `age` in expression: # example.exs:7 user.age expected one of the following fields: name, address where "user" was given the type %User{} in: # example.exs:7 %User{} = user Conflict found at example.exs:7: Check.drive?/1 </code></pre></div></div> <p>The compiler also checks binary constructors. Consider you have to send a string over the wire with length-based encoding, where the string is prefixed by its length, up to 4MBs. Your initial attempt may be this:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="n">run_length</span><span class="p">(</span><span class="n">string</span><span class="p">)</span> <span class="ow">when</span> <span class="n">is_binary</span><span class="p">(</span><span class="n">string</span><span class="p">)</span> <span class="k">do</span> <span class="o">&lt;&lt;</span><span class="n">byte_size</span><span class="p">(</span><span class="n">string</span><span class="p">)::</span><span class="mi">32</span><span class="p">,</span> <span class="n">string</span><span class="o">&gt;&gt;</span> <span class="k">end</span> </code></pre></div></div> <p>However, the code above has a bug. Each segment given between <code class="language-plaintext highlighter-rouge">&lt;&lt;&gt;&gt;</code> must be an integer, unless specified otherwise. With Elixir v1.11, the compiler will let you know so:</p> <div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>warning: incompatible types: binary() !~ integer() in expression: &lt;&lt;byte_size(string)::integer()-size(32), string&gt;&gt; where "string" was given the type integer() in: # foo.exs:4 &lt;&lt;byte_size(string)::integer()-size(32), string&gt;&gt; where "string" was given the type binary() in: # foo.exs:3 is_binary(string) HINT: all expressions given to binaries are assumed to be of type integer() unless said otherwise. For example, &lt;&lt;expr&gt;&gt; assumes "expr" is an integer. Pass a modifier, such as &lt;&lt;expr::float&gt;&gt; or &lt;&lt;expr::binary&gt;&gt;, to change the default behaviour. Conflict found at foo.exs:4: Check.run_length/1 </code></pre></div></div> <p>Which can be fixed by adding <code class="language-plaintext highlighter-rouge">::binary</code> to the second component:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="n">run_length</span><span class="p">(</span><span class="n">string</span><span class="p">)</span> <span class="ow">when</span> <span class="n">is_binary</span><span class="p">(</span><span class="n">string</span><span class="p">)</span> <span class="k">do</span> <span class="o">&lt;&lt;</span><span class="n">byte_size</span><span class="p">(</span><span class="n">string</span><span class="p">)::</span><span class="mi">32</span><span class="p">,</span> <span class="n">string</span><span class="p">::</span><span class="n">binary</span><span class="o">&gt;&gt;</span> <span class="k">end</span> </code></pre></div></div> <p>While some of those warnings could be automatically fixed by the compiler, future versions will also perform those checks across functions and potentially across modules, where automatic fixes wouldn’t be desired (nor possible).</p> <h2 id="compilation-time-improvements">Compilation time improvements</h2> <p>Elixir v1.11 features many improvements to how the compiler tracks file dependencies, such that touching one file causes less files to be recompiled. In previous versions, Elixir tracked three types of dependencies:</p> <ul> <li>compile time dependencies - if A depends on B at compile time, such as by using a macro, whenever B changes, A is recompiled</li> <li>struct dependencies - if A depends on B’s struct, whenever B’s struct definition changed, A is recompiled</li> <li>runtime dependencies - if A depends on B at runtime, A is never recompiled</li> </ul> <p>However, because dependencies are transitive, if A depends on B at compile time and B depends on C at runtime, A would depend on C at compile time. Therefore, it is very important to reduce the amount of compile time dependencies.</p> <p>Elixir v1.11 replaces “struct dependencies” by “exports dependencies”. In other words, if A depends on B, whenever B public’s interface changes, A is recompiled. B’s public interface is made by its struct definition and all of its public functions and macros.</p> <p>This change allows us to mark <code class="language-plaintext highlighter-rouge">import</code>s and <code class="language-plaintext highlighter-rouge">require</code>s as “exports dependencies” instead of “compile time” dependencies. This simplifies the dependency graph considerably. For example, <a href="https://github.com/hexpm/hexpm">in the Hex.pm project</a>, changing the <code class="language-plaintext highlighter-rouge">user.ex</code> file in Elixir v1.10 would emit this:</p> <div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ touch lib/hexpm/accounts/user.ex &amp;&amp; mix compile Compiling 90 files (.ex) </code></pre></div></div> <p>In Elixir v1.11, we now get:</p> <div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ touch lib/hexpm/accounts/user.ex &amp;&amp; mix compile Compiling 16 files (.ex) </code></pre></div></div> <p>To make things even better, Elixir v1.11 also introduces a more granular tracking for umbrella projects (and path dependencies in general). In previous versions, a module from a sibling application would always be treated as a compile time dependency. This often meant that changing an application would cause many modules in sibling applications to recompile. Elixir v1.11 will tag modules from dependencies as exports whenever possible, yielding dramatic improvements in those cases.</p> <p>To round up the list of compiler enhancements, the <code class="language-plaintext highlighter-rouge">--profile=time</code> option added in Elixir v1.10 now also includes the time to compile each individual file. For example, in the Plug project, one can now get:</p> <div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[profile] lib/plug/conn.ex compiled in 935ms [profile] lib/plug/ssl.ex compiled in 147ms (plus 744ms waiting) [profile] lib/plug/static.ex compiled in 238ms (plus 654ms waiting) [profile] lib/plug/csrf_protection.ex compiled in 237ms (plus 790ms waiting) [profile] lib/plug/debugger.ex compiled in 719ms (plus 947ms waiting) [profile] Finished compilation cycle of 60 modules in 1802ms [profile] Finished group pass check of 60 modules in 75ms </code></pre></div></div> <p>While implementing those features, we have also made the <code class="language-plaintext highlighter-rouge">--long-compilation-threshold</code> flag more precise. In previous versions, <code class="language-plaintext highlighter-rouge">--long-compilation-threshold</code> would consider both the time a file spent to compile and the time spent waiting on other files. In Elixir v1.11, it considers only the compilation time. This means less false positives and you can now effectively get all files that take longer than 2s to compile, in execution time, by passing <code class="language-plaintext highlighter-rouge">--long-compilation-threshold 2</code>.</p> <h2 id="configruntimeexs-and-mix-appconfig"><code class="language-plaintext highlighter-rouge">config/runtime.exs</code> and <code class="language-plaintext highlighter-rouge">mix app.config</code></h2> <p>Elixir v1.9 introduced a new configuration file called <code class="language-plaintext highlighter-rouge">config/releases.exs</code>. However, this new configuration file was executed only during releases. For those not familiar with releases, a release is a self-contained artifact with the Erlang VM, Elixir and your application, ready to run in production.</p> <p>This new configuration file was considered a very useful addition to releases. Therefore, we are also introducing <code class="language-plaintext highlighter-rouge">config/runtime.exs</code>, which is executed after the code compilation on all environments (dev, test, and prod) - for both Mix and releases. Our goal is to provide a better runtime configuration experience to developers, in contrast to our current configuration system which has been mostly compile-time centric.</p> <p><code class="language-plaintext highlighter-rouge">config/runtime.exs</code> works the same as any other configuration file in Elixir. However, given <code class="language-plaintext highlighter-rouge">config/runtime.exs</code> is meant to run in production systems, where our <code class="language-plaintext highlighter-rouge">Mix</code> build tool is not available, developers must not use <a href="https://hexdocs.pm/mix/Mix.html#env/0"><code class="language-plaintext highlighter-rouge">Mix.env()</code></a> or <a href="https://hexdocs.pm/mix/Mix.html#target/0"><code class="language-plaintext highlighter-rouge">Mix.target()</code></a> in <code class="language-plaintext highlighter-rouge">config/runtime.exs</code>. Instead, they must use the new <code class="language-plaintext highlighter-rouge">config_env()</code> and <code class="language-plaintext highlighter-rouge">config_target()</code>, which have been added to the <a href="https://hexdocs.pm/elixir/Config.html"><code class="language-plaintext highlighter-rouge">Config</code></a> module.</p> <p>While <code class="language-plaintext highlighter-rouge">config/releases.exs</code> will continue to be supported, developers can migrate to <code class="language-plaintext highlighter-rouge">config/runtime.exs</code> without loss of functionality. For example, a <code class="language-plaintext highlighter-rouge">config/releases.exs</code> file such as this one</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># config/releases.exs</span> <span class="kn">import</span> <span class="no">Config</span> <span class="n">config</span> <span class="ss">:foo</span><span class="p">,</span> <span class="o">...</span> <span class="n">config</span> <span class="ss">:bar</span><span class="p">,</span> <span class="o">...</span> </code></pre></div></div> <p>could run as is as <code class="language-plaintext highlighter-rouge">config/runtime.exs</code>. However, given <code class="language-plaintext highlighter-rouge">config/runtime.exs</code> runs in all environments, you may want to restrict part of your configuration to the <code class="language-plaintext highlighter-rouge">:prod</code> environment:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># config/runtime.exs</span> <span class="kn">import</span> <span class="no">Config</span> <span class="k">if</span> <span class="n">config_env</span><span class="p">()</span> <span class="o">==</span> <span class="ss">:prod</span> <span class="k">do</span> <span class="n">config</span> <span class="ss">:foo</span><span class="p">,</span> <span class="o">...</span> <span class="n">config</span> <span class="ss">:bar</span><span class="p">,</span> <span class="o">...</span> <span class="k">end</span> </code></pre></div></div> <p>If both files are available, releases will pick the now preferred <code class="language-plaintext highlighter-rouge">config/runtime.exs</code> instead of <code class="language-plaintext highlighter-rouge">config/releases.exs</code>.</p> <p>To wrap it all up, <code class="language-plaintext highlighter-rouge">Mix</code> also includes a new task called <a href="https://hexdocs.pm/mix/Mix.Tasks.App.Config.html"><code class="language-plaintext highlighter-rouge">mix app.config</code></a>. This task loads all applications and configures them, without starting them. Whenever you write your own Mix tasks, you will typically want to invoke either <code class="language-plaintext highlighter-rouge">mix app.start</code> or <code class="language-plaintext highlighter-rouge">mix app.config</code> before running your own code. Which one is better depends if you want your applications running or only configured.</p> <h2 id="other-improvements">Other improvements</h2> <p>Elixir v1.11 adds the <code class="language-plaintext highlighter-rouge">is_struct/2</code>, <code class="language-plaintext highlighter-rouge">is_exception/1</code>, and <code class="language-plaintext highlighter-rouge">is_exception/2</code> guards. It also adds support for the <code class="language-plaintext highlighter-rouge">map.field</code> syntax in guards.</p> <p>The Calendar module ships with a new <a href="https://hexdocs.pm/elixir/Calendar.html#strftime/3"><code class="language-plaintext highlighter-rouge">Calendar.strftime/3</code></a> function, which provides datetime formatting based on the <code class="language-plaintext highlighter-rouge">strftime</code> format. The <a href="https://hexdocs.pm/elixir/Date.html"><code class="language-plaintext highlighter-rouge">Date</code></a> module got new functions for working with weeks and months, such as <code class="language-plaintext highlighter-rouge">Date.beginning_of_month/1</code> and <code class="language-plaintext highlighter-rouge">Date.end_of_week/2</code>. Finally, all calendar types got conversion functions from and to gregorian timestamps, such as <code class="language-plaintext highlighter-rouge">Date.from_gregorian_days/2</code> and <code class="language-plaintext highlighter-rouge">NaiveDateTime.to_gregorian_seconds/1</code>.</p> <p>Finally, to bring visibility to the compiler tracking improvements described in previous sections, we have also added new features to <a href="https://hexdocs.pm/mix/Mix.Tasks.Xref.html"><code class="language-plaintext highlighter-rouge">mix xref</code></a>. <code class="language-plaintext highlighter-rouge">mix xref</code> is a task that describes cross-references between files in your projects and can be helpful to diagnose large compilation cycles in projects.</p> <p>For a complete list of all changes, see the <a href="https://github.com/elixir-lang/elixir/releases/tag/v1.11.0">full release notes</a>.</p> <p>Check <a href="/install.html">the Install section</a> to get Elixir installed and read our <a href="https://hexdocs.pm/elixir/introduction.html">Getting Started guide</a> to learn more.</p> <p>Have fun!</p> José Valim 2020-09-24T00:00:00+00:00 /blog/2020/09/24/paas-with-elixir-at-Heroku <p><em>Welcome to our series of case studies about companies using Elixir in production. <a href="/cases.html">See all cases</a> we have published so far.</em></p> <p><a href="https://www.heroku.com">Heroku</a> provides services and tools to build, run, and scale web applications. They enable developers and teams to focus on the design and craft of their apps. Heroku started development back in 2007, focused on the Ruby programming language, and since then, they have expanded to support multiple runtimes, either officially or via buildpacks.</p> <p>As the platform grew, their engineering teams also adopted different languages, one of them being Elixir. In this article, we will talk about how two distinct engineering teams at Heroku, the Front-end team and the Vault team, have adopted Elixir.</p> <p><img src="/images/cases/bg/heroku.png" alt="Heroku" /></p> <h2 id="first-steps-with-elixir">First steps with Elixir</h2> <p>The Vault team was first to use Elixir inside Heroku. Their team is responsible for licensing and financial services, such as invoicing, credit card payments, etc. Most of their services are used internally at Heroku.</p> <p>They had to rewrite one of their existing services and that was the perfect occasion to give Elixir a try, since the difficulties and risks with the service were mostly known. The experiment was a success: they deployed and ran their first Elixir application in production. This paved the way to use Elixir more and more.</p> <p>Later on, they had a new challenge: they had to audit a large amount of data, and they knew from experience that the Ruby implementation would take too long to finish. Given they were already ramping up their familiarity with Elixir, they chose to apply <a href="https://github.com/elixir-lang/gen_stage">Elixir’s GenStage</a> to the problem, which is a low-level library for data processing, and that took only a couple hours. From this moment on, they were sold on the language and the platform.</p> <h2 id="tackling-operational-complexity-with-elixir">Tackling operational complexity with Elixir</h2> <p>The Front-end team shares a similar story: they first used Elixir to solve a well-understood problem and took it forward from there.</p> <p>The Front-end engineers are responsible for maintaining all user interfaces: the CLI, the dashboard, and a bunch of backend services that work with data. One of the features they provide to Heroku customers is analytics.</p> <p>At first, they were sending their analytics to Mixpanel. However, they had some issues fetching the data, due to cross-domain concerns, and they decided to replace Mixpanel by an in-house Elixir service. The service used <a href="https://github.com/elixir-plug/plug">Plug</a>, a library for building web applications, and had a single endpoint.</p> <blockquote style="font-size: 24px; color: #444"> <p>We were having a lot of fun and a lot of luck with it, so we kept doing it.</p> <p style="font-size: 20px">— Micah Woods, Lead Engineer, on migrating to Elixir.</p> </blockquote> <p>They later spent most of a year focused on operational stability, and during this period, they started rewriting part of their Node.js microservices into Elixir. Today they have migrated their numerous Node.js microservices into one main Elixir application with one auxiliary service for authentication. The fact that Elixir was capable of handling everything they threw at it alongside their experience with Erlang’s stability - <a href="https://blog.heroku.com/erlang-in-anger">Heroku’s router uses Erlang</a> - allowed them to simplify their operations considerably.</p> <h2 id="productivity-and-scalability">Productivity and scalability</h2> <p>The Front-end team has been using Elixir for two years. The team has 21 engineers: about 4 of them doing Elixir full-time, and 8 engineers altogether doing Elixir here and there.</p> <p>The first service that they built with Elixir, the analytics services, receives requests and puts them into an in-memory queue to be processed within the same VM. It handles about 3k to 4k requests per second. 99% of the response times stay within 0-1ms, occasionally 4ms. They use 3 Heroku dynos for fault-tolerance - of course, Heroku uses Heroku for their own infrastructure.</p> <p>The main Elixir application uses <a href="https://phoenixframework.org/">the Phoenix web framework</a> to power the Heroku Dashboard, provide real-time functionality via WebSockets, and support other services. This application runs on 5 Heroku dynos - although their engineering team believes they could probably do with less. Memory consumption is also on the lower side: their biggest dyno uses 256MB.</p> <p>The Vault team doing Elixir is only three engineers. Most of their apps are used internally, so they are generally not worried about performance. They continue using Elixir because <strong>they feel productive and happy with it</strong>. They have also found it is an easier language to maintain compared to their previous experiences.</p> <h2 id="on-phoenix">On Phoenix</h2> <p>Both teams generally use Phoenix for web applications, unless they have a reason not to, which is rare. They acknowledge there is not a performance penalty for using Phoenix and you get a lot out of the box. Phoenix makes it easy to opt-in on the pieces that they need and remove the parts that they don’t want.</p> <p>They have also found it easier to understand how Phoenix itself works under the hood, especially compared to their previous experiences with other frameworks, such as Ruby on Rails. This knowledge is consistently helping them maintain and update their applications as time passes.</p> <h2 id="learning-elixir-and-growing-the-team">Learning Elixir and growing the team</h2> <p>The growth of both Elixir teams has been mostly organic. Given there are multiple languages in their stack, they often hire for one or another language in particular, and not specifically for Elixir. If the new team members gravitate towards Elixir, they are further encouraged to explore and learn the language. They are also active practitioners of pair programming, so there are many opportunities in their team to learn from each other, rotate pairs, swap projects, and so on.</p> <p>According to Matthew Peck, “the paradigm shift from Object-Oriented languages to Functional Programming was our biggest challenge when first learning Elixir”. However, the team agrees the investment was worth it: “Learning Elixir has made us better programmers. We have found that immutability made our code more readable, easier to test, and simpler to make concurrent. Now when we go back to an Object-Oriented language, we are thinking about how we can apply the same concepts there” - said Mike Hagerdon.</p> <p>Amanda Dolan added some remarks on Elixir’s capabilities for writing concurrent and fault-tolerant applications: “One other challenge when learning Elixir is fully grasping concurrency and the Erlang/OTP patterns”. Some of them felt it took longer to master those concepts than they first expected.</p> <p>Taylor Mock has his take on the challenges teams may face when adopting Elixir: “Another difference between Elixir and our previous stacks, Ruby and Node.js, is in the ecosystems”. They were initially concerned that the Elixir ecosystem would lack when it comes to third-party tools, but that was not what they saw. Taylor continues: “We found out that we can get really far with the concepts and mechanisms that the language itself provides. This shift can be scary, but we are now past it, and we find ourselves with leaner applications and fewer dependencies”.</p> <p>Overall, both teams found the language itself quite approachable. Given they started with a small proof of concept, they were able to tackle their concerns in regards to adoption, development, and deployment as they moved forward. Historically Heroku also has had much success with Erlang, and that has contributed to the success adopting Elixir has seen inside Heroku.</p> José Valim 2020-08-20T00:00:00+00:00 /blog/2020/08/20/embedded-elixir-at-farmbot <p><em>Welcome to our series of case studies about companies using Elixir in production. <a href="/cases.html">See all cases</a> we have published so far.</em></p> <p><a href="https://farm.bot/">FarmBot</a> is an open-source precision agriculture CNC farming project that includes a robot farming machine, software, and documentation including a farming data repository. FarmBot’s machines use IoT technology to make it easy for farmers to remotely manage their gardens.</p> <p>Farmbot is built with <a href="https://www.nerves-project.org/">Nerves</a>, an open-source platform and infrastructure to build, deploy, and securely manage your fleet of IoT devices at speed and scale.</p> <p>When Connor Rigby, former embedded systems engineer at Farmbot, initially joined the company, his first project was to build a Farmbot application using Ruby. After completing the proof-of-concept, he knew that he needed a way to make the embedded development process more efficient, reliable, and secure. Connor had used Nerves before at a previous company and was a regular open-source contributor to Nerves, so he knew the platform would help him accomplish these goals.</p> <p><img src="/images/cases/bg/farmbot.jpg" alt="Farmbot" /></p> <h2 id="how-nerves-helped">How Nerves helped</h2> <p>Connor brought Nerves to Farmbot by porting the entire proof-of-concept Ruby application he’d created over to Nerves, which he did in his free time over the course of a month, taking him about 20 hours total. He also continued to make open-source contributions to Nerves, helping to structure the networking functionality that is now part of <a href="https://www.nerves-hub.org/">NervesHub</a>, the extensible web service that enables over-the-air firmware update management.</p> <blockquote style="font-size: 24px; color: #444"> <p>The biggest benefit of using Nerves is definitely how fast you can get up and running.</p> <p style="font-size: 20px">— Connor Rigby, Embedded Systems Engineer</p> </blockquote> <p>Connor says that the Nerves Platform and what eventually became NervesHub was a great choice for Farmbot because:</p> <h3 id="1-nerves-supports-lean-systems-and-operates-well-in-low-bandwidth-areas">1. Nerves supports lean systems and operates well in low-bandwidth areas</h3> <p>Because Nerves bundles entire applications into relatively small archives in terms of firmware images for full Linux systems, Farmbot can use NervesHub to send over-the-air updates more quickly and users can download them faster. For comparison, an Android update generally clocks in at around 4 GB, but a Nerves update can be packed into as little as 12 MB.</p> <p>This is especially helpful for Farmbot users who operate in more remote locations with lower bandwidth and less reliable access to Wi-Fi. When an internet connection is available, NervesHub will connect and check if there’s an update, and then prompt the user to install the update.</p> <h3 id="2-nerves-adds-convenience-with-low-overhead">2. Nerves adds convenience with low overhead</h3> <p>For devices that are already connected to the internet, connecting to Nerves requires no additional configuration because NervesHub is compatible with the current public key infrastructure for device-to-cloud communication. Since Farmbot already had internet-connected devices when they brought Nerves onboard, they were able to use the same “key” to sign in to NervesHub that they use for their cloud service.</p> <h3 id="3-nerves-has-all-the-benefits-of-elixir-and-erlang">3. Nerves has all the benefits of Elixir and Erlang</h3> <p>Because it’s written in Elixir and built within the Erlang runtime system, Nerves retains the qualities of that language and framework — notably that they are distributed, fault-tolerant, soft real-time, and highly available. Connor also says that with Nerves, it’s easy to reason about the things you build with Nerves because you only input what you need into a Nerves application, helping you to avoid unnecessary complexities or unforeseen security vulnerabilities. You can check up on devices as they’re running and debug them without disruption to the user experience.</p> <h2 id="the-result">The result</h2> <p>FarmBot now has around 300 devices live in NervesHub, with a different deployment for each of their device models. Nerves is built to scale, so as Farmbot continues to grow its user base and expand their product capabilities, they’ll be able to continue developing and releasing reliable firmware updates using Nerves.</p> <p><em>This case study has first been published on <a href="https://nerves-project.org/cases/farmbot">Nerves’ website</a></em>.</p> José Valim 2020-01-27T00:00:00+00:00 /blog/2020/01/27/elixir-v1-10-0-released <p>Elixir v1.10 comes with improvements to the standard library, the compiler, as well as several additions to the <a href="/blog/2019/06/24/elixir-v1-9-0-released/"><code class="language-plaintext highlighter-rouge">mix release</code> feature added in v1.9</a>. In particular, this version adds a lot of polish to existing features, such as our configuration system and our sorting APIs.</p> <p>Also note Elixir v1.10 requires Erlang/OTP 21+. This allows us to provide tighter integration with Erlang/OTP’s new logger. This means that the logger level, logger metadata, as well as all log messages are now shared between Erlang and Elixir APIs.</p> <p>Let’s take a look at what else is new.</p> <h2 id="releases-improvements">Releases improvements</h2> <p>Elixir v1.9 introduced releases as a mechanism to package self-contained applications. Elixir v1.10 further improves releases with bug fixes and new enhancements based on feedback we got from the community. The highlights are:</p> <ul> <li> <p>Allow the dual boot system of releases to be disabled on environments that are boot-time sensitive, such as embedded devices</p> </li> <li> <p>Track and raise if compile-time configuration is set or changes at runtime (more in the next section)</p> </li> <li> <p>Support overlays to easily add extra files to a packaged releases</p> </li> <li> <p>Allow <code class="language-plaintext highlighter-rouge">RELEASE_DISTRIBUTION</code> to be set to <code class="language-plaintext highlighter-rouge">none</code> in order to fully disable distribution</p> </li> <li> <p>Add a built-in <code class="language-plaintext highlighter-rouge">:tar</code> step that automatically packages releases</p> </li> </ul> <p>See the <a href="https://github.com/elixir-lang/elixir/releases/tag/v1.10.0">full release notes for more improvements</a>.</p> <h2 id="improvements-to-sort-based-apis-in-enum">Improvements to sort-based APIs in Enum</h2> <p><a href="https://hexdocs.pm/elixir/Enum.html#sort/1"><code class="language-plaintext highlighter-rouge">Enum.sort/1</code></a> in Elixir by default sorts from lowest to highest:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">iex</span><span class="o">&gt;</span> <span class="no">Enum</span><span class="o">.</span><span class="n">sort</span><span class="p">([</span><span class="s2">"banana"</span><span class="p">,</span> <span class="s2">"apple"</span><span class="p">,</span> <span class="s2">"pineapple"</span><span class="p">])</span> <span class="p">[</span><span class="s2">"apple"</span><span class="p">,</span> <span class="s2">"banana"</span><span class="p">,</span> <span class="s2">"pineapple"</span><span class="p">]</span> </code></pre></div></div> <p>If you want to sort from highest to lowest, you need to call <code class="language-plaintext highlighter-rouge">Enum.sort/2</code> with a custom sorting function, such as <code class="language-plaintext highlighter-rouge">Enum.sort(collection, &amp;&gt;=/2)</code>, which is not immediately obvious to someone reading the code:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">iex</span><span class="o">&gt;</span> <span class="no">Enum</span><span class="o">.</span><span class="n">sort</span><span class="p">([</span><span class="s2">"banana"</span><span class="p">,</span> <span class="s2">"apple"</span><span class="p">,</span> <span class="s2">"pineapple"</span><span class="p">],</span> <span class="o">&amp;&gt;=/</span><span class="mi">2</span><span class="p">)</span> <span class="p">[</span><span class="s2">"pineapple"</span><span class="p">,</span> <span class="s2">"banana"</span><span class="p">,</span> <span class="s2">"apple"</span><span class="p">]</span> </code></pre></div></div> <p>Furthermore, comparison operators, such as <code class="language-plaintext highlighter-rouge">&lt;=</code> and <code class="language-plaintext highlighter-rouge">&gt;=</code>, perform structural sorting, instead of a semantic one. For example, using <code class="language-plaintext highlighter-rouge">&gt;=</code> to sort dates descendingly won’t yield the correct result:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">iex</span><span class="o">&gt;</span> <span class="no">Enum</span><span class="o">.</span><span class="n">sort</span><span class="p">([</span><span class="sx">~D[2019-12-31]</span><span class="p">,</span> <span class="sx">~D[2020-01-01]</span><span class="p">])</span> <span class="p">[</span><span class="sx">~D[2020-01-01]</span><span class="p">,</span> <span class="sx">~D[2019-12-31]</span><span class="p">]</span> </code></pre></div></div> <p>To perform proper semantic comparison for dates, one would also need to pass a custom sorting function:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">iex</span><span class="o">&gt;</span> <span class="no">Enum</span><span class="o">.</span><span class="n">sort</span><span class="p">([</span><span class="sx">~D[2019-12-31]</span><span class="p">,</span> <span class="sx">~D[2020-01-01]</span><span class="p">],</span> <span class="o">&amp;</span><span class="p">(</span><span class="no">Date</span><span class="o">.</span><span class="n">compare</span><span class="p">(</span><span class="nv">&amp;1</span><span class="p">,</span> <span class="nv">&amp;2</span><span class="p">)</span> <span class="o">!=</span> <span class="ss">:lt</span><span class="p">))</span> <span class="p">[</span><span class="sx">~D[2019-12-31]</span><span class="p">,</span> <span class="sx">~D[2020-01-01]</span><span class="p">]</span> </code></pre></div></div> <p>Elixir v1.10 streamlines the sorting functions by introducing both <code class="language-plaintext highlighter-rouge">:asc</code> and <code class="language-plaintext highlighter-rouge">:desc</code> shortcuts:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">iex</span><span class="o">&gt;</span> <span class="no">Enum</span><span class="o">.</span><span class="n">sort</span><span class="p">([</span><span class="s2">"banana"</span><span class="p">,</span> <span class="s2">"apple"</span><span class="p">,</span> <span class="s2">"pineapple"</span><span class="p">],</span> <span class="ss">:asc</span><span class="p">)</span> <span class="p">[</span><span class="s2">"apple"</span><span class="p">,</span> <span class="s2">"banana"</span><span class="p">,</span> <span class="s2">"pineapple"</span><span class="p">]</span> <span class="n">iex</span><span class="o">&gt;</span> <span class="no">Enum</span><span class="o">.</span><span class="n">sort</span><span class="p">([</span><span class="s2">"banana"</span><span class="p">,</span> <span class="s2">"apple"</span><span class="p">,</span> <span class="s2">"pineapple"</span><span class="p">],</span> <span class="ss">:desc</span><span class="p">)</span> <span class="p">[</span><span class="s2">"pineapple"</span><span class="p">,</span> <span class="s2">"banana"</span><span class="p">,</span> <span class="s2">"apple"</span><span class="p">]</span> </code></pre></div></div> <p>As well as adding the possibility to pass a module to perform semantic comparisons. For example, to sort dates, one now only needs to pass the <code class="language-plaintext highlighter-rouge">Date</code> module or even <code class="language-plaintext highlighter-rouge">{:desc, Date}</code> for descending semantical sort:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">iex</span><span class="o">&gt;</span> <span class="no">Enum</span><span class="o">.</span><span class="n">sort</span><span class="p">([</span><span class="sx">~D[2019-12-31]</span><span class="p">,</span> <span class="sx">~D[2020-01-01]</span><span class="p">],</span> <span class="no">Date</span><span class="p">)</span> <span class="p">[</span><span class="sx">~D[2019-12-31]</span><span class="p">,</span> <span class="sx">~D[2020-01-01]</span><span class="p">]</span> <span class="n">iex</span><span class="o">&gt;</span> <span class="no">Enum</span><span class="o">.</span><span class="n">sort</span><span class="p">([</span><span class="sx">~D[2019-12-31]</span><span class="p">,</span> <span class="sx">~D[2020-01-01]</span><span class="p">],</span> <span class="p">{</span><span class="ss">:desc</span><span class="p">,</span> <span class="no">Date</span><span class="p">})</span> <span class="p">[</span><span class="sx">~D[2020-01-01]</span><span class="p">,</span> <span class="sx">~D[2019-12-31]</span><span class="p">]</span> </code></pre></div></div> <p>These API improvements make the code more concise and readable and they have also been added to <code class="language-plaintext highlighter-rouge">Enum.sort_by</code>, <code class="language-plaintext highlighter-rouge">Enum.min_by</code>, <code class="language-plaintext highlighter-rouge">Enum.max_by</code>, and friends.</p> <h3 id="tracking-of-compile-time-configuration">Tracking of compile-time configuration</h3> <p>In Elixir, we organize our code in applications. Libraries, your dependencies, and your own project are all separate applications. All applications in Elixir also come with an application environment.</p> <p>The application environment is a key-value store that allows us to configure said application. While reading the application environment at runtime is the preferred approach, in some rare occasions you may want to use the application environment to configure the compilation of a certain project. This is often done by calling <code class="language-plaintext highlighter-rouge">Application.get_env/3</code> outside of a function:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">DBClient</span> <span class="k">do</span> <span class="nv">@db_host</span> <span class="no">Application</span><span class="o">.</span><span class="n">get_env</span><span class="p">(</span><span class="ss">:my_app</span><span class="p">,</span> <span class="ss">:db_host</span><span class="p">,</span> <span class="s2">"db.local"</span><span class="p">)</span> <span class="k">def</span> <span class="n">start_link</span><span class="p">()</span> <span class="k">do</span> <span class="no">SomeLib</span><span class="o">.</span><span class="no">DBClient</span><span class="o">.</span><span class="n">start_link</span><span class="p">(</span><span class="ss">host:</span> <span class="nv">@db_host</span><span class="p">)</span> <span class="k">end</span> <span class="k">end</span> </code></pre></div></div> <p>This approach has one big limitation: if you change the value of the application environment after the code is compiled, the value used at runtime is not going to change! For example, if you are using <code class="language-plaintext highlighter-rouge">mix release</code> and your <code class="language-plaintext highlighter-rouge">config/releases.exs</code> has:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>config :my_app, :db_host, "db.production" </code></pre></div></div> <p>Because <code class="language-plaintext highlighter-rouge">config/releases.exs</code> is read after the code is compiled, the new value will have no effect as the code was compiled to connect to “db.local”.</p> <p>Of course, the obvious solution to this mismatch is to not read the application environment at compilation time in the first place, and instead move the code to inside a function:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">DBClient</span> <span class="k">do</span> <span class="k">def</span> <span class="n">start_link</span><span class="p">()</span> <span class="k">do</span> <span class="no">SomeLib</span><span class="o">.</span><span class="no">DBClient</span><span class="o">.</span><span class="n">start_link</span><span class="p">(</span><span class="ss">host:</span> <span class="n">db_host</span><span class="p">())</span> <span class="k">end</span> <span class="k">defp</span> <span class="n">db_host</span><span class="p">()</span> <span class="k">do</span> <span class="no">Application</span><span class="o">.</span><span class="n">get_env</span><span class="p">(</span><span class="ss">:my_app</span><span class="p">,</span> <span class="ss">:db_host</span><span class="p">,</span> <span class="s2">"db.local"</span><span class="p">)</span> <span class="k">end</span> <span class="k">end</span> </code></pre></div></div> <p>While this is the preferred approach, there are still two scenarios we need to address:</p> <ol> <li> <p>Not everyone may be aware of this pitfall, so they will mistakenly read the application environment at compile-time, until they are bitten by this behaviour</p> </li> <li> <p>In rare occasions, you truly need to read the application environment at compile-time, and you want to be warned when you try to configure at runtime something that is valid only at compilation time</p> </li> </ol> <p>Elixir v1.10 aims to solve these two scenarios by introducing a <code class="language-plaintext highlighter-rouge">Application.compile_env/3</code> function. For example, to read the value at compile time, you can now do:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">@db_host</span> <span class="no">Application</span><span class="o">.</span><span class="n">compile_env</span><span class="p">(</span><span class="ss">:my_app</span><span class="p">,</span> <span class="ss">:db_host</span><span class="p">,</span> <span class="s2">"db.local"</span><span class="p">)</span> </code></pre></div></div> <p>By using <code class="language-plaintext highlighter-rouge">compile_env/3</code>, Elixir will store the values used during compilation and compare them with the runtime values whenever your system starts, raising an error in case they differ. This helps developers ensure they are running their production systems with the configuration they intend to.</p> <p>In future versions, we will deprecate the use <code class="language-plaintext highlighter-rouge">Application.get_env/3</code> at compile-time with a clear message pointing users to configuration best practices, effectively addressing the scenario where users read from the application environment at compile time unaware of its pitfalls.</p> <h3 id="compiler-tracing">Compiler tracing</h3> <p>This release brings enhancements to the Elixir compiler and adds new capabilities for developers to listen to compilation events.</p> <p>In previous Elixir versions, Elixir would compile a database of cross references between modules (such as function calls, references, structs, etc) for each project in order to perform all kinds of checks, such as deprecations and undefined functions.</p> <p>Although this database was not public, developers would still use it to run their own checks against their projects. With time, developers would request more data to be included in the database, which was problematic as Elixir itself did not have a use for the additional data, and the database was not meant to be used externally in the first place.</p> <p>In Elixir v1.10, we have addressed these problems by <a href="https://hexdocs.pm/elixir/Code.html#module-compilation-tracers">introducing compiler tracing</a>. The compiler tracing allows developers to listen to events as they are emitted by the compiler, so they can store all of the information they need - and only the information they need.</p> <p>Elixir itself is using the new compiler tracing to provide new functionality. One advantage of this approach is that developers can now disable undefined function warnings directly on the callsite. For example, imagine you have an optional dependency which may not be available in some cases. You can tell the compiler to skip warning on calls to optional modules with:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>@compile {:no_warn_undefined, OptionalDependency} defdelegate my_function_call(arg), to: OptionalDependency </code></pre></div></div> <p>Previously, this information had to be added to the overall project configuration, which was far away from where the optional call effectively happened.</p> <h3 id="other-enhancements">Other enhancements</h3> <p>Elixir’s calendar data types got many improvements, such as sigil support for third-party calendars, as well as the additions of <a href="https://hexdocs.pm/elixir/DateTime.html#now!/2"><code class="language-plaintext highlighter-rouge">DateTime.now!/2</code></a>, <a href="https://hexdocs.pm/elixir/DateTime.html#shift_zone!/3"><code class="language-plaintext highlighter-rouge">DateTime.shift_zone!/3</code></a>, and <a href="https://hexdocs.pm/elixir/NaiveDateTime.html#local_now/0"><code class="language-plaintext highlighter-rouge">NaiveDateTime.local_now/0</code></a>.</p> <p>There are many improvements related to Elixir’s AST in this release too. <a href="https://hexdocs.pm/elixir/Code.html#string_to_quoted/2"><code class="language-plaintext highlighter-rouge">Code.string_to_quoted/2</code></a> has two new options, <code class="language-plaintext highlighter-rouge">:token_metadata</code> and <code class="language-plaintext highlighter-rouge">:literal_encoder</code>, that give more control over Elixir’s parser. This information was already available to the Elixir code formatter and has now been made public. These changes alongside compiler tracing means tools like <a href="https://github.com/rrrene/credo">Credo</a>, <a href="https://github.com/sasa1977/boundary">Boundary</a>, and IDE integrations have an even better foundation to analyze the source code.</p> <p><a href="https://hexdocs.pm/ex_unit">ExUnit</a>, our test framework, ships two small but important improvements: <code class="language-plaintext highlighter-rouge">ExUnit.CaptureIO</code> can now be used by tests that run concurrently and we have added “pattern-matching diffing”. To understand the last feature, take this code:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">assert</span> <span class="p">%{</span><span class="s2">"status"</span> <span class="o">=&gt;</span> <span class="mi">200</span><span class="p">,</span> <span class="s2">"body"</span> <span class="o">=&gt;</span> <span class="p">%{</span><span class="s2">"key"</span> <span class="o">=&gt;</span> <span class="s2">"foo"</span><span class="p">}}</span> <span class="o">=</span> <span class="n">json_payload</span> </code></pre></div></div> <p>Now imagine that <code class="language-plaintext highlighter-rouge">json_payload</code> is a large JSON blob and the <code class="language-plaintext highlighter-rouge">"key"</code> inside the <code class="language-plaintext highlighter-rouge">"body"</code> did not have value of <code class="language-plaintext highlighter-rouge">"foo"</code>. In previous Elixir versions, if the assertion failed, Elixir would print the right side and let you up to your own devices to figure out what went wrong. In Elixir v1.10, we diff the data structure against the pattern so you can see exactly which parts of the data matched the pattern and which ones did not. Note ExUnit already performed diffing when comparing data types, this new version adds diffing when matching data against a pattern.</p> <p>Finally, this release also adds two new guards, <code class="language-plaintext highlighter-rouge">is_struct/1</code> and <code class="language-plaintext highlighter-rouge">is_map_key/2</code>, thanks to the strict requirement on Erlang/OTP 21+.</p> <p>To learn what else is new, you can read the <a href="https://github.com/elixir-lang/elixir/releases/tag/v1.10.0">full release notes</a>.</p> <p>Check <a href="/install.html">the Install section</a> to get Elixir installed and read our <a href="https://hexdocs.pm/elixir/introduction.html">Getting Started guide</a> to learn more.</p> <p>Have fun!</p> José Valim 2019-06-24T00:00:00+00:00 /blog/2019/06/24/elixir-v1-9-0-released <p>Elixir v1.9 is out with releases support, improved configuration, and more.</p> <p>We are also glad to announce <a href="https://github.com/fertapric">Fernando Tapia Rico</a> has joined the Elixir Core Team. Fernando has been extremely helpful in keeping the issues tracker tidy, by fixing bugs and improving Elixir in many different areas, such as the code formatter, IEx, the compiler, and others.</p> <p>Now let’s take a look at what’s new in this new version.</p> <h2 id="releases">Releases</h2> <p>The main feature in Elixir v1.9 is the addition of releases. A release is a self-contained directory that consists of your application code, all of its dependencies, plus the whole Erlang Virtual Machine (VM) and runtime. Once a release is assembled, it can be packaged and deployed to a target as long as the target runs on the same operating system (OS) distribution and version as the machine running the <a href="https://hexdocs.pm/mix/Mix.Tasks.Release.html"><code class="language-plaintext highlighter-rouge">mix release</code></a> command.</p> <p>Releases have always been part of the Elixir community thanks to Paul Schoenfelder’s work on <a href="https://github.com/bitwalker/distillery">Distillery</a> (and EXRM before that). Distillery was announced in July 2016. Then in 2017, <a href="https://dockyard.com/">DockYard</a> hired Paul to work on improving deployments, an effort that would lead to <a href="https://dockyard.com/blog/2018/08/23/announcing-distillery-2-0">Distillery 2.0</a>. Distillery 2.0 provided important answers in areas where the community was struggling to establish conventions and best practices, such as configuration.</p> <p>At the beginning of this year, thanks to <a href="http://plataformatec.com.br/">Plataformatec</a>, I was able to prioritize the work on bringing releases directly into Elixir. Paul was aware that we wanted to have releases in Elixir itself and during <a href="https://elixirconf.com">ElixirConf 2018</a> I announced that releases was the last planned feature for Elixir.</p> <p>The goal of Elixir releases was to double down on the most important concepts provided by Distillery and provide extensions points for the other bits the community may find important. <a href="http://github.com/bitwalker/">Paul</a> and <a href="https://github.com/tsloughter">Tristan</a> (who maintains <a href="https://github.com/erlware/relx">Erlang’s relx</a>) provided excellent feedback on Elixir’s implementation, which we are very thankful for. <a href="https://dashbit.co/blog/updating-hex-pm-to-use-elixir-releases">The Hex package manager is already using releases in production</a> and we also got feedback from other companies doing the same.</p> <p>Enough background, let’s see why you would want to use releases and how to assemble one.</p> <h3 id="why-releases">Why releases?</h3> <p>Releases allow developers to precompile and package all of their code and the runtime into a single unit. The benefits of releases are:</p> <ul> <li> <p>Code preloading. The VM has two mechanisms for loading code: interactive and embedded. By default, it runs in the interactive mode which dynamically loads modules when they are used for the first time. The first time your application calls <code class="language-plaintext highlighter-rouge">Enum.map/2</code>, the VM will find the <code class="language-plaintext highlighter-rouge">Enum</code> module and load it. There’s a downside. When you start a new server in production, it may need to load many other modules, causing the first requests to have an unusual spike in response time. Releases run in embedded mode, which loads all available modules upfront, guaranteeing your system is ready to handle requests after booting.</p> </li> <li> <p>Configuration and customization. Releases give developers fine grained control over system configuration and the VM flags used to start the system.</p> </li> <li> <p>Self-contained. A release does not require the source code to be included in your production artifacts. All of the code is precompiled and packaged. Releases do not even require Erlang or Elixir in your servers, as they include the Erlang VM and its runtime by default. Furthermore, both Erlang and Elixir standard libraries are stripped to bring only the parts you are actually using.</p> </li> <li> <p>Multiple releases. You can assemble different releases with different configuration per application or even with different applications altogether.</p> </li> <li> <p>Management scripts. Releases come with scripts to start, restart, connect to the running system remotely, execute RPC calls, run as daemon, run as a Windows service, and more.</p> </li> </ul> <h3 id="1-2-3-released-assembled">1, 2, 3: released assembled!</h3> <p>You can start a new project and assemble a release for it in three easy steps:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ mix new my_app $ cd my_app $ MIX_ENV=prod mix release </code></pre></div></div> <p>A release will be assembled in <code class="language-plaintext highlighter-rouge">_build/prod/rel/my_app</code>. Inside the release, there will be a <code class="language-plaintext highlighter-rouge">bin/my_app</code> file which is the entry point to your system. It supports multiple commands, such as:</p> <ul> <li> <p><code class="language-plaintext highlighter-rouge">bin/my_app start</code>, <code class="language-plaintext highlighter-rouge">bin/my_app start_iex</code>, <code class="language-plaintext highlighter-rouge">bin/my_app restart</code>, and <code class="language-plaintext highlighter-rouge">bin/my_app stop</code> - for general management of the release</p> </li> <li> <p><code class="language-plaintext highlighter-rouge">bin/my_app rpc COMMAND</code> and <code class="language-plaintext highlighter-rouge">bin/my_app remote</code> - for running commands on the running system or to connect to the running system</p> </li> <li> <p><code class="language-plaintext highlighter-rouge">bin/my_app eval COMMAND</code> - to start a fresh system that runs a single command and then shuts down</p> </li> <li> <p><code class="language-plaintext highlighter-rouge">bin/my_app daemon</code> and <code class="language-plaintext highlighter-rouge">bin/my_app daemon_iex</code> - to start the system as a daemon on Unix-like systems</p> </li> <li> <p><code class="language-plaintext highlighter-rouge">bin/my_app install</code> - to install the system as a service on Windows machines</p> </li> </ul> <h3 id="hooks-and-configuration">Hooks and Configuration</h3> <p>Releases also provide built-in hooks for configuring almost every need of the production system:</p> <ul> <li> <p><code class="language-plaintext highlighter-rouge">config/config.exs</code> (and <code class="language-plaintext highlighter-rouge">config/prod.exs</code>) - provides build-time application configuration, which is executed when the release is assembled</p> </li> <li> <p><code class="language-plaintext highlighter-rouge">config/releases.exs</code> - provides runtime application configuration. It is executed every time the release boots and is further extensible via config providers</p> </li> <li> <p><code class="language-plaintext highlighter-rouge">rel/vm.args.eex</code> - a template file that is copied into every release and provides static configuration of the Erlang Virtual Machine and other runtime flags</p> </li> <li> <p><code class="language-plaintext highlighter-rouge">rel/env.sh.eex</code> and <code class="language-plaintext highlighter-rouge">rel/env.bat.eex</code> - template files that are copied into every release and executed on every command to set up environment variables, including ones specific to the VM, and the general environment</p> </li> </ul> <p>We have written <a href="https://hexdocs.pm/mix/Mix.Tasks.Release.html">extensive documentation on releases</a>, so we recommend checking it out for more information.</p> <h2 id="configuration">Configuration</h2> <p>We also use the work on releases to streamline Elixir’s configuration API. A new <code class="language-plaintext highlighter-rouge">Config</code> module has been added to Elixir. The previous configuration API, <code class="language-plaintext highlighter-rouge">Mix.Config</code>, was part of the Mix build tool. However, since releases provide runtime configuration and Mix is not included in releases, we ported the <code class="language-plaintext highlighter-rouge">Mix.Config</code> API to Elixir. In other words, <code class="language-plaintext highlighter-rouge">use Mix.Config</code> has been soft-deprecated in favor of <code class="language-plaintext highlighter-rouge">import Config</code>.</p> <p>Another important change related to configuration is that <code class="language-plaintext highlighter-rouge">mix new</code> will no longer generate a <code class="language-plaintext highlighter-rouge">config/config.exs</code> file. <a href="https://hexdocs.pm/elixir/library-guidelines.html#avoid-application-configuration">Relying on configuration is undesired for most libraries</a> and the generated config files pushed library authors in the wrong direction. Furthermore, <code class="language-plaintext highlighter-rouge">mix new --umbrella</code> will no longer generate a configuration for each child app, instead all configuration should be declared in the umbrella root. That’s how it has always behaved, we are now making it explicit.</p> <h2 id="other-improvements">Other improvements</h2> <p>There are many other enhancements in Elixir v1.9. The Elixir CLI got a handful of new options in order to best support releases. <code class="language-plaintext highlighter-rouge">Logger</code> now computes its sync/async/discard thresholds in a decentralized fashion, reducing contention. <code class="language-plaintext highlighter-rouge">EEx</code> (Embedded Elixir) templates support more complex expressions than before. Finally, there is a new <code class="language-plaintext highlighter-rouge">~U</code> sigil for working with UTC DateTimes as well as new functions in the <code class="language-plaintext highlighter-rouge">File</code>, <code class="language-plaintext highlighter-rouge">Registry</code>, and <code class="language-plaintext highlighter-rouge">System</code> modules.</p> <h2 id="whats-next">What’s next?</h2> <p>As mentioned earlier, releases was the last planned feature for Elixir. We don’t have any major user-facing feature in the works nor planned. I know for certain some will consider this fact the most excing part of this announcement!</p> <p>Of course, it does not mean that v1.9 is the last Elixir version. We will continue shipping new releases every 6 months with enhancements, bug fixes and improvements. You can see the <a href="http://github.com/elixir-lang/elixir/issues">Issues Tracker</a> for more details.</p> <p>We also are working on some structural changes. One of them is move the <code class="language-plaintext highlighter-rouge">mix xref</code> pass straight into the compiler, which would allow us to emit undefined function and deprecation warnings in more places. We are also considering a move to <a href="https://cirrus-ci.org/">Cirrus-CI</a>, so we can test Elixir on Windows, Unix, and FreeBSD through a single service.</p> <p>It is also important to highlight that there are two main reasons why we can afford to have an empty backlog.</p> <p>First of all, Elixir is built on top of Erlang/OTP and we simply leverage all of the work done by Ericsson and the OTP team on the runtime and Virtual Machine. The Elixir team has always aimed to contribute back as much as possible and those contributions have increased in the last years.</p> <p>Second, Elixir was designed to be an extensible language. The same tools and abstractions we used to create and enhance the language are also available to libraries and frameworks. This means the community can continue to improve the ecosystem without a need to change the language itself, which would effectively become a bottleneck for progress.</p> <p>Check <a href="/install.html">the Install section</a> to get Elixir installed and read our <a href="https://hexdocs.pm/elixir/introduction.html">Getting Started guide</a> to learn more. We have also updated our <a href="https://hexdocs.pm/elixir/introduction-to-mix.html">advanced Mix &amp; OTP</a> to talk about releases. If you are looking for a more fast paced introduction to the language, see the <a href="http://howistart.org/posts/elixir/1/index.html">How I Start: Elixir</a> tutorial, which has also been brought to the latest and greatest.</p> <p>Have fun!</p> Eric Meadows-Jönsson 2019-02-25T00:00:00+00:00 /blog/2019/02/25/mint-a-new-http-library-for-elixir <p><a href="https://github.com/ericmj/mint">Mint</a> is a new low-level HTTP client that aims to provide a small and functional core that others can build on top. Mint is connection based: each connection is a single struct with an associated socket belonging to the process that started the connection. Since no extra processes are started for the connection, you can choose the process architecture that better fits your application.</p> <p>To validate this we built out the library with a common API supporting both HTTP/1 and HTTP/2 with automatic version negotiation. In addition, Mint comes with a <a href="https://github.com/ericmj/castore">CA certificate store</a> to do safe by default HTTPS connections.</p> <h2 id="connections-without-processes">Connections without processes</h2> <p>Mint HTTP connections are managed directly in the process that starts the connection, which means no connection pool is used nor new processes spawned when a connection is opened. This allows the user of the library to build their own process structure that fits their application.</p> <p>With Mint each connection has a single immutable data structure that the user needs to manage. Mint uses <a href="http://www.erlang.org/doc/man/inet.html#setopts-2">“active mode”</a> sockets. This means data and events from the socket are sent as messages to the process that started the connection. The user passes the messages to the <code class="language-plaintext highlighter-rouge">stream/2</code> function that returns the updated connection and a list of “responses”. Responses are streamed back which means you won’t receive a single full HTTP response back from <code class="language-plaintext highlighter-rouge">stream/2</code>, instead the response is returned in partial response chunks. A chunk can be the status line, HTTP headers, or part of the response body.</p> <p>Let’s look at an example of sending a request with Mint:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">iex</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span><span class="o">&gt;</span> <span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">conn</span><span class="p">}</span> <span class="o">=</span> <span class="no">Mint</span><span class="o">.</span><span class="no">HTTP</span><span class="o">.</span><span class="n">connect</span><span class="p">(</span><span class="ss">:http</span><span class="p">,</span> <span class="s2">"httpbin.org"</span><span class="p">,</span> <span class="mi">80</span><span class="p">)</span> <span class="n">iex</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span><span class="o">&gt;</span> <span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">conn</span><span class="p">,</span> <span class="n">request_ref</span><span class="p">}</span> <span class="o">=</span> <span class="no">Mint</span><span class="o">.</span><span class="no">HTTP</span><span class="o">.</span><span class="n">request</span><span class="p">(</span><span class="n">conn</span><span class="p">,</span> <span class="s2">"GET"</span><span class="p">,</span> <span class="s2">"/"</span><span class="p">,</span> <span class="p">[],</span> <span class="s2">""</span><span class="p">)</span> <span class="n">iex</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span><span class="o">&gt;</span> <span class="k">receive</span> <span class="k">do</span> <span class="o">...</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span><span class="o">&gt;</span> <span class="n">message</span> <span class="o">-&gt;</span> <span class="o">...</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span><span class="o">&gt;</span> <span class="no">IO</span><span class="o">.</span><span class="n">inspect</span><span class="p">(</span><span class="n">message</span><span class="p">,</span> <span class="ss">label:</span> <span class="ss">:message</span><span class="p">)</span> <span class="o">...</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span><span class="o">&gt;</span> <span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">conn</span><span class="p">,</span> <span class="n">responses</span><span class="p">}</span> <span class="o">=</span> <span class="no">Mint</span><span class="o">.</span><span class="no">HTTP</span><span class="o">.</span><span class="n">stream</span><span class="p">(</span><span class="n">conn</span><span class="p">,</span> <span class="n">message</span><span class="p">)</span> <span class="o">...</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span><span class="o">&gt;</span> <span class="no">IO</span><span class="o">.</span><span class="n">inspect</span><span class="p">(</span><span class="n">responses</span><span class="p">,</span> <span class="ss">label:</span> <span class="ss">:responses</span><span class="p">)</span> <span class="o">...</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span><span class="o">&gt;</span> <span class="k">end</span> <span class="ss">message:</span> <span class="p">{</span><span class="ss">:tcp</span><span class="p">,</span> <span class="c1">#Port&lt;0.8&gt;, "HTTP/1.1 200 OK\r\n" &lt;&gt; ...}</span> <span class="ss">responses:</span> <span class="p">[</span> <span class="p">{</span><span class="ss">:status</span><span class="p">,</span> <span class="c1">#Reference&lt;...&gt;, 200},</span> <span class="p">{</span><span class="ss">:headers</span><span class="p">,</span> <span class="c1">#Reference&lt;...&gt;, [{"connection", "keep-alive"}, ...},</span> <span class="p">{</span><span class="ss">:data</span><span class="p">,</span> <span class="c1">#Reference&lt;...&gt;, "&lt;!DOCTYPE html&gt;" &lt;&gt; ...},</span> <span class="p">{</span><span class="ss">:done</span><span class="p">,</span> <span class="c1">#Reference&lt;...&gt;}</span> <span class="p">]</span> </code></pre></div></div> <p>As we can see all calls to <code class="language-plaintext highlighter-rouge">Mint.HTTP</code> functions return an updated <code class="language-plaintext highlighter-rouge">conn</code> which holds the state for the connection. It is important to carry on the <code class="language-plaintext highlighter-rouge">conn</code> to the next function call or the state will be corrupted.</p> <p>On line 2 we send a request to the server. A reference to the request is returned: this reference is useful when sending concurrent requests, either with HTTP/1 pipelining or with HTTP/2 multiplexed streams.</p> <p>Next we start a receive block waiting for a TCP active mode message and pass it to <code class="language-plaintext highlighter-rouge">stream/2</code>. The message is parsed and the response to the request is returned. As you can see the response is split over multiple tuples: <code class="language-plaintext highlighter-rouge">:status</code>, <code class="language-plaintext highlighter-rouge">:headers</code>, <code class="language-plaintext highlighter-rouge">:data</code>, and <code class="language-plaintext highlighter-rouge">:done</code>. This is because Mint was built from the ground with streaming in mind. The parts of the response will be returned continuously as TCP messages are passed to <code class="language-plaintext highlighter-rouge">stream/2</code> so that we don’t have to wait for the full response to complete before starting to process it.</p> <p>If the response body is larger than a single packet <code class="language-plaintext highlighter-rouge">stream/2</code> may return multiple <code class="language-plaintext highlighter-rouge">:data</code> tuples and if the response includes trailing headers multiple <code class="language-plaintext highlighter-rouge">:headers</code> will be returned. When the response is complete <code class="language-plaintext highlighter-rouge">:done</code> will be returned.</p> <p>Note that if you send concurrent requests on a HTTP/2 connection responses can be returned interleaved from the requests using HTTP/2’s stream multiplexing. Additionally, responses can be spread over multiple messages so we may need to continually receive messages and pass them to <code class="language-plaintext highlighter-rouge">stream/2</code>.</p> <p>See more examples on how to use Mint in the <a href="https://hexdocs.pm/mint">documentation</a>.</p> <h2 id="why-process-less">Why process-less?</h2> <p>Mint may seem more cumbersome to use than most other HTTP libraries you have used and that is true in many ways. But by providing a low-level API without a predetermined process architecture it gives more flexibility to the user of the library.</p> <p>Many times you do not need a general purpose connection pool and can avoid the additional complexity, single point of failure, and potential performance bottlenecks that it brings. For example, if you are building quick CLI scripts, you most likely don’t need a pool and performing a single one-off request with Mint is good enough.</p> <p>Another good use case for Mint is <a href="https://github.com/elixir-lang/gen_stage">GenStage</a>. If you write GenStage pipelines, it is most likely that you have a pool of producers that fetch data from external sources via HTTP. If you are using a high-level HTTP library, that comes with its own pool, now you have two pools, one of GenStage producers and another from the HTTP library. With Mint, you can have each GenStage producer manage its own connection, reducing overhead and simplifying the code.</p> <p>Of course, none of this stops you from building a connection pool on top of Mint. The point is exactly that Mint won’t impose an architecture onto you. At the end of the day, we hope Mint will be a useful building block for more complex scenario and use cases.</p> <h2 id="http1-and-http2">HTTP/1 and HTTP/2</h2> <p>The <code class="language-plaintext highlighter-rouge">Mint.HTTP</code> module has a single interface for both HTTP/1 and HTTP/2 connections and performs version negotiation on HTTPS connections, HTTP connections default to HTTP/1. You can specify which HTTP version you want to use or use the <code class="language-plaintext highlighter-rouge">Mint.HTTP1</code> or <code class="language-plaintext highlighter-rouge">Mint.HTTP2</code> modules directly if you want to use version-specific features.</p> <h2 id="safe-by-default-https">Safe-by-default HTTPS</h2> <p>When connecting over HTTPS, Mint will perform certificate verification by default. We believe it’s crucial that an HTTP library defaults to be secure out of the box.</p> <p>Mint uses an optional dependency on <a href="https://github.com/ericmj/castore">CAStore</a> to provide certificates from <a href="https://www.mozilla.org/en-US/about/governance/policies/security-group/certs/">Mozilla’s CA Certificate Store</a>.</p> <p>You can of course tweak specific SSL settings without re-building the safe defaults yourself.</p> <h2 id="current-state-of-the-library">Current state of the library</h2> <p>The first version of Mint has just been released. It is an experimental library trying a new approach to building HTTP libraries so don’t expect a fully stable API yet.</p> <p>Use Mint to explore new ideas for HTTP connection management and building higher level clients on top of Mint. In the future connection pooling and a higher level API may be added to supplement the current low level API, either directly to Mint or via different libraries.</p> <p><em>Note:</em> Mint is being announced in the official Elixir blog because it was originally being considered for inclusion in Elixir itself. However, at some point the Elixir team decided it doesn’t make sense to include an HTTP client in Elixir itself, at least as long as Erlang/OTP ships with a client too. Mint is not maintained by the Elixir team, although it is maintained by Eric and Andrea, who are part of the team.</p> José Valim 2019-01-14T00:00:00+00:00 /blog/2019/01/14/elixir-v1-8-0-released <p>Elixir v1.8 comes with many improvements at the infrastructure level, improving compilation time, speeding up common patterns, and adding features around introspection of the system.</p> <h2 id="custom-struct-inspections">Custom struct inspections</h2> <p>Elixir now provides a derivable implementation of the <code class="language-plaintext highlighter-rouge">Inspect</code> protocol. In a nutshell, this means it is really easy to filter data from your data structures whenever they are inspected. For example, imagine you have a user struct with security and privacy sensitive information:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">User</span> <span class="k">do</span> <span class="k">defstruct</span> <span class="p">[</span><span class="ss">:id</span><span class="p">,</span> <span class="ss">:name</span><span class="p">,</span> <span class="ss">:age</span><span class="p">,</span> <span class="ss">:email</span><span class="p">,</span> <span class="ss">:encrypted_password</span><span class="p">]</span> <span class="k">end</span> </code></pre></div></div> <p>By default, if you inspect a user via <code class="language-plaintext highlighter-rouge">inspect(user)</code>, it will include all fields. This can cause fields such as <code class="language-plaintext highlighter-rouge">:email</code> and <code class="language-plaintext highlighter-rouge">:encrypted_password</code> to appear in logs, error reports, etc. You could always define a custom implementation of the <code class="language-plaintext highlighter-rouge">Inspect</code> protocol for such cases but Elixir v1.8 makes it simpler by allowing you to derive the <code class="language-plaintext highlighter-rouge">Inspect</code> protocol:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">User</span> <span class="k">do</span> <span class="nv">@derive</span> <span class="p">{</span><span class="no">Inspect</span><span class="p">,</span> <span class="ss">only:</span> <span class="p">[</span><span class="ss">:id</span><span class="p">,</span> <span class="ss">:name</span><span class="p">,</span> <span class="ss">:age</span><span class="p">]}</span> <span class="k">defstruct</span> <span class="p">[</span><span class="ss">:id</span><span class="p">,</span> <span class="ss">:name</span><span class="p">,</span> <span class="ss">:age</span><span class="p">,</span> <span class="ss">:email</span><span class="p">,</span> <span class="ss">:encrypted_password</span><span class="p">]</span> <span class="k">end</span> </code></pre></div></div> <p>Now all user structs will be printed with all remaining fields collapsed:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>#User&lt;id: 1, name: "Jane", age: 33, ...&gt; </code></pre></div></div> <p>You can also pass <code class="language-plaintext highlighter-rouge">@derive {Inspect, except: [...]}</code> in case you want to keep all fields by default and exclude only some.</p> <h2 id="time-zone-database-support">Time zone database support</h2> <p>In Elixir v1.3, Elixir added four types, known as Calendar types, to work with dates and times: <code class="language-plaintext highlighter-rouge">Time</code>, <code class="language-plaintext highlighter-rouge">Date</code>, <code class="language-plaintext highlighter-rouge">NaiveDateTime</code> (without time zone), and <code class="language-plaintext highlighter-rouge">DateTime</code> (with time zone). Over the last versions we have added many enhancements to the Calendar types but the <code class="language-plaintext highlighter-rouge">DateTime</code> module always evolved at a slower pace since Elixir did not provide an API for time zone databases.</p> <p>Elixir v1.8 now defines a <code class="language-plaintext highlighter-rouge">Calendar.TimeZoneDatabase</code> behaviour, allowing developers to bring in their own time zone databases. By defining an explicit contract for time zone behaviours, Elixir can now extend the <code class="language-plaintext highlighter-rouge">DateTime</code> API, adding functions such as <code class="language-plaintext highlighter-rouge">DateTime.shift_zone/3</code>. By default, Elixir ships with a time zone database called <code class="language-plaintext highlighter-rouge">Calendar.UTCOnlyTimeZoneDatabase</code> that only handles UTC.</p> <p>Other Calendar related improvements include the addition of <code class="language-plaintext highlighter-rouge">Date.day_of_year/1</code>, <code class="language-plaintext highlighter-rouge">Date.quarter_of_year/1</code>, <code class="language-plaintext highlighter-rouge">Date.year_of_era/1</code>, and <code class="language-plaintext highlighter-rouge">Date.day_of_era/1</code>.</p> <h2 id="faster-compilation-and-other-performance-improvements">Faster compilation and other performance improvements</h2> <p>Due to improvements to the compiler made over the last year, Elixir v1.8 should compile code about 5% faster on average. This is yet another release where we have been able to reduce compilation times and provide a more joyful development experience to everyone.</p> <p>The compiler also emits more efficient code for range checks in guards (such as <code class="language-plaintext highlighter-rouge">x in y..z</code>), for charlists with interpolation (such as <code class="language-plaintext highlighter-rouge">'foo #{bar} baz'</code>), and when working with records via the <code class="language-plaintext highlighter-rouge">Record</code> module.</p> <p>Finally, EEx templates got their own share of optimizations, emitting more compact code that runs faster.</p> <h2 id="improved-instrumentation-and-ownership-with-callers">Improved instrumentation and ownership with <code class="language-plaintext highlighter-rouge">$callers</code></h2> <p>The <code class="language-plaintext highlighter-rouge">Task</code> module is one of the most common ways to spawn light-weight processes to perform work concurrently. Whenever you spawn a new process, Elixir annotates the parent of that process through the <code class="language-plaintext highlighter-rouge">$ancestors</code> key. This information can be used by instrumentation tools to track the relationship between events occurring within multiple processes. However, many times, tracking only the <code class="language-plaintext highlighter-rouge">$ancestors</code> is not enough.</p> <p>For example, we recommend developers to always start tasks under a supervisor. This provides more visibility and allows us to control how those tasks are terminated when a node shuts down. In your code, this can be done by invoking something like: <code class="language-plaintext highlighter-rouge">Task.Supervisor.start_child(MySupervisor, task_specification)</code>. This means that, although your code is the one who invokes the task, the actual parent of the task would be the supervisor, as the supervisor is the one spawning it. We would list the supervisor as one of the <code class="language-plaintext highlighter-rouge">$ancestors</code> for the task, but the relationship between your code and the task is lost.</p> <p>In Elixir v1.8, we now track the relationship between your code and the task via the <code class="language-plaintext highlighter-rouge">$callers</code> key in the process dictionary, which aligns well with the existing <code class="language-plaintext highlighter-rouge">$ancestors</code> key. Therefore, assuming the <code class="language-plaintext highlighter-rouge">Task.Supervisor</code> call above, we have:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[your code] -- calls --&gt; [supervisor] ---- spawns --&gt; [task] </code></pre></div></div> <p>which means we store the following relationships:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[your code] [supervisor] &lt;-- ancestor -- [task] ^ | |--------------------- caller ---------------------| </code></pre></div></div> <p>When a task is spawned directly from your code, without a supervisor, then the process running your code will be listed under both <code class="language-plaintext highlighter-rouge">$ancestors</code> and <code class="language-plaintext highlighter-rouge">$callers</code>.</p> <p>This small feature is very powerful. It allows instrumentation and monitoring tools to better track and relate the events happening in your system. This feature can also be used by tools like the “Ecto Sandbox”. The “Ecto Sandbox” allows developers to run tests concurrently against the database, by using transactions and an ownership mechanism where each process explicitly gets a connection assigned to it. Without <code class="language-plaintext highlighter-rouge">$callers</code>, every time you spawned a task that queries the database, the task would not know its caller, and therefore it would be unable to know which connection was assigned to it. This often meant features that rely on tasks could not be tested concurrently. With <code class="language-plaintext highlighter-rouge">$callers</code>, figuring out this relationship is trivial and you have more tests using the full power of your machine.</p> <h2 id="summing-up">Summing up</h2> <p>We are really proud of this release (as usual!) which brings many improvements at the infrastructure level. Those improvements were designed with feedback from the community and from the many different companies using Elixir in production. The full list of changes is available in our <a href="https://github.com/elixir-lang/elixir/releases/tag/v1.8.0">release notes</a>.</p> <p>There is only one last major feature planned for upcoming Elixir versions, which is the addition of <code class="language-plaintext highlighter-rouge">mix release</code> to Elixir itself, streamlining the experience provided by packages like <a href="https://github.com/bitwalker/distillery">distillery</a>. With <code class="language-plaintext highlighter-rouge">mix release</code>, a developer can bundle the VM and all compiled code in a single directory, which can then be packaged and sent to production. We are glad to say the <a href="https://github.com/elixir-lang/elixir/issues/8612">work on this feature has already started</a>.</p> <p>During <a href="https://www.youtube.com/watch?v=suOzNeMJXl0">my keynote at ElixirConf 2018 US</a>, I talked about the next five years for Elixir and much of the emphasis is put on the community. Elixir was designed to be an extensible language and therefore the work on the language itself is meant to reduce with time, which we have seen in the last two releases. We trust the community to continue building on this solid foundation, bringing new challenges to the ecosystem and taking the language to new domains.</p> <p>Check <a href="/install.html">the Install section</a> to get Elixir installed and read our <a href="https://hexdocs.pm/elixir/introduction.html">Getting Started guide</a> to learn more.</p> <p>Your turn. :)</p> José Valim 2018-07-25T00:00:00+00:00 /blog/2018/07/25/elixir-v1-7-0-released <p>A new semester has started, which means it is time for a new Elixir release! This release brings quality of life improvements to the documentation, to error handling, to logger reporting, and to ExUnit, Elixir’s testing library.</p> <p>We are also glad to welcome Michał Muskała to the Elixir Core team. Prior to joining the team, he was <a href="https://github.com/elixir-ecto/ecto">a member of the Ecto team</a>, he has made <a href="https://github.com/elixir-lang/elixir/pulls?utf8=%E2%9C%93&amp;q=is%3Apr+author%3Amichalmuskala">plenty of contributions to Elixir</a>, often to improve performance, and <a href="https://github.com/erlang/otp/pulls?utf8=%E2%9C%93&amp;q=is%3Apr+author%3Amichalmuskala">is a frequent to contribute to Erlang/OTP too</a>!</p> <h2 id="documentation-metadata">Documentation metadata</h2> <p>Elixir v1.7 implements <a href="http://www.erlang.org/eep/eeps/eep-0048.html">EEP 48</a>. EEP 48 aims to bring documentation interoperability across all languages running on the Erlang VM.</p> <p>Furthermore, EEP 48 introduces the ability to annotate documentation with metadata, which we have made possible to leverage from Elixir:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">@moduledoc</span> <span class="s2">"A brand new module"</span> <span class="nv">@moduledoc</span> <span class="ss">authors:</span> <span class="p">[</span><span class="s2">"Jane"</span><span class="p">,</span> <span class="s2">"Mary"</span><span class="p">],</span> <span class="ss">since:</span> <span class="s2">"1.4.0"</span> </code></pre></div></div> <p>Metadata can be given to <code class="language-plaintext highlighter-rouge">@moduledoc</code>, <code class="language-plaintext highlighter-rouge">@typedoc</code> and <code class="language-plaintext highlighter-rouge">@doc</code>.</p> <p>We have updated <a href="https://github.com/elixir-lang/ex_doc">the ExDoc tool</a> to start leveraging metadata in order to provide better documentation for developers. Some of the improvements include:</p> <ul> <li> <p>Deprecated modules, functions, callbacks and types have a warning automatically attached to them. <a href="https://hexdocs.pm/elixir/Behaviour.html">See the deprecated <code class="language-plaintext highlighter-rouge">Behaviour</code> module as an example</a></p> </li> <li> <p>Functions, macros, callbacks and types now include the version in which they were added. For example, <a href="https://hexdocs.pm/elixir/Kernel.html#defguard/1">see the top right corner of the <code class="language-plaintext highlighter-rouge">defguard</code> docs</a></p> </li> <li> <p>Future Elixir versions will <a href="https://hexdocs.pm/elixir/main/Kernel.html#guards">include its own section for guards in the documentation and in the sidebar</a>. We are currently exploring ways to <a href="https://github.com/elixir-lang/ex_doc/issues/876">generalize this feature in ExDoc itself</a></p> </li> </ul> <p>Elixir’s interactive shell, IEx, has also been updated to print metadata:</p> <p><img src="/images/contents/iex-metadata.png" alt="IEx metadata" /></p> <p>While Elixir allows any metadata to be given, those tools currently exhibit only <code class="language-plaintext highlighter-rouge">:deprecated</code> and <code class="language-plaintext highlighter-rouge">:since</code>. Other keys may be shown in the future.</p> <p>Those improvements are not exclusive to the standard library, they are available to every Elixir library and application. We hope one day they will be available to all applications running on the Erlang VM too.</p> <p>To access the new documentation format, developers should use <a href="https://hexdocs.pm/elixir/Code.html#fetch_docs/1"><code class="language-plaintext highlighter-rouge">Code.fetch_docs/1</code></a>. We have always been proud of treating documentation as a first-class citizen and the ability to add structured information to the documentation is a further step in this direction.</p> <h2 id="the-__stacktrace__-construct">The <code class="language-plaintext highlighter-rouge">__STACKTRACE__</code> construct</h2> <p>Erlang/OTP 21.0 introduces a new way to retrieve the stacktrace that is lexically scoped and no longer relies on side-effects like <code class="language-plaintext highlighter-rouge">System.stacktrace/0</code> does. Before one would write:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">try</span> <span class="k">do</span> <span class="o">...</span> <span class="n">something</span> <span class="n">that</span> <span class="n">may</span> <span class="n">fail</span> <span class="o">...</span> <span class="k">rescue</span> <span class="n">exception</span> <span class="o">-&gt;</span> <span class="n">log</span><span class="p">(</span><span class="n">exception</span><span class="p">,</span> <span class="no">System</span><span class="o">.</span><span class="n">stacktrace</span><span class="p">())</span> <span class="n">reraise</span><span class="p">(</span><span class="n">exception</span><span class="p">,</span> <span class="no">System</span><span class="o">.</span><span class="n">stacktrace</span><span class="p">())</span> <span class="k">end</span> </code></pre></div></div> <p>In Elixir v1.7, this can be written as:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">try</span> <span class="k">do</span> <span class="o">...</span> <span class="n">something</span> <span class="n">that</span> <span class="n">may</span> <span class="n">fail</span> <span class="o">...</span> <span class="k">rescue</span> <span class="n">exception</span> <span class="o">-&gt;</span> <span class="n">log</span><span class="p">(</span><span class="n">exception</span><span class="p">,</span> <span class="n">__STACKTRACE__</span><span class="p">)</span> <span class="n">reraise</span><span class="p">(</span><span class="n">exception</span><span class="p">,</span> <span class="n">__STACKTRACE__</span><span class="p">)</span> <span class="k">end</span> </code></pre></div></div> <p>This change may also yield performance improvements in the future, since the lexical scope allows us to track precisely when a stacktrace is used and we no longer need to keep references to stacktrace entries after the <code class="language-plaintext highlighter-rouge">try</code> construct finishes.</p> <p>Other parts of the exception system have also been improved. For example, more information is provided in certain occurrences of <code class="language-plaintext highlighter-rouge">ArgumentError</code>, <code class="language-plaintext highlighter-rouge">ArithmeticError</code> and <code class="language-plaintext highlighter-rouge">KeyError</code> messages.</p> <h2 id="erlangotp-logger-integration">Erlang/OTP logger integration</h2> <p>Erlang/OTP 21 includes a new <code class="language-plaintext highlighter-rouge">:logger</code> module. Elixir v1.7 fully integrates with the new <code class="language-plaintext highlighter-rouge">:logger</code> and leverages its metadata system. The <code class="language-plaintext highlighter-rouge">Logger.Translator</code> mechanism has also been improved to export metadata, allowing custom Logger backends to leverage information such as:</p> <ul> <li> <p><code class="language-plaintext highlighter-rouge">:crash_reason</code> - a two-element tuple with the throw/error/exit reason as the first argument and the stacktrace as the second</p> </li> <li> <p><code class="language-plaintext highlighter-rouge">:initial_call</code> - the initial call that started the process</p> </li> <li> <p><code class="language-plaintext highlighter-rouge">:registered_name</code> - the process’ registered name as an atom</p> </li> </ul> <p>We recommend Elixir libraries that previously hooked into Erlang’s <code class="language-plaintext highlighter-rouge">:error_logger</code> to hook into <code class="language-plaintext highlighter-rouge">Logger</code> instead, in order to support all current and future Erlang/OTP versions.</p> <h2 id="logger-compile-time-purging">Logger compile-time purging</h2> <p>Previously, Logger macros such as <code class="language-plaintext highlighter-rouge">debug</code>, <code class="language-plaintext highlighter-rouge">info</code>, and so on would always evaluate their arguments, even when nothing would be logged. From Elixir v1.7 the arguments are only evaluated when the message is logged.</p> <p>The Logger configuration system also accepts a new option called <code class="language-plaintext highlighter-rouge">:compile_time_purge_matching</code> that allows you to remove log calls with specific compile-time metadata. For example, to remove all logger calls from application <code class="language-plaintext highlighter-rouge">:foo</code> with level lower than <code class="language-plaintext highlighter-rouge">:info</code>, as well as remove all logger calls from <code class="language-plaintext highlighter-rouge">Bar.foo/3</code>, you can use the following configuration:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">config</span> <span class="ss">:logger</span><span class="p">,</span> <span class="ss">compile_time_purge_matching:</span> <span class="p">[</span> <span class="p">[</span><span class="ss">application:</span> <span class="ss">:foo</span><span class="p">,</span> <span class="ss">level_lower_than:</span> <span class="ss">:info</span><span class="p">],</span> <span class="p">[</span><span class="ss">module:</span> <span class="no">Bar</span><span class="p">,</span> <span class="ss">function:</span> <span class="s2">"foo/3"</span><span class="p">]</span> <span class="p">]</span> </code></pre></div></div> <h2 id="exunit-improvements">ExUnit improvements</h2> <p><a href="https://hexdocs.pm/ex_unit/">ExUnit</a> is Elixir’s unit testing library. ExUnit has always leveraged Elixir macros to provide excellent error reports when a failure happens. For example, the following code:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">assert</span> <span class="s2">"fox jumps over the lazy dog"</span> <span class="o">==</span> <span class="s2">"brown fox jumps over the dog"</span> </code></pre></div></div> <p>will fail with the following report:</p> <p><img src="/images/contents/exunit-diff.png" alt="ExUnit Diff" /></p> <p>The <code class="language-plaintext highlighter-rouge">assert</code> macro is able to look at the code, extract the current file, the line, extract the operands and show a diff between the data structures alongside the stacktrace when the assertion fails.</p> <p>However, sometimes we need to write assertions such as <code class="language-plaintext highlighter-rouge">assert some_function(expr1, var2)</code>. When such assertion fails, we usually have to re-run the tests, now debugging or printing the values of <code class="language-plaintext highlighter-rouge">expr1</code> and <code class="language-plaintext highlighter-rouge">var2</code>. In Elixir v1.7, when a “bare” assertion fails, we will print the value of each argument individually. For a simple example such as <code class="language-plaintext highlighter-rouge">assert some_vars(1 + 2, 3 + 4)</code>, we get this report:</p> <p><img src="/images/contents/exunit-bare-assertion-diff.png" alt="ExUnit Bare Assertion Diff" /></p> <p>We have also <a href="https://hexdocs.pm/ex_unit/ExUnit.DocTest.html#content">added coloring and diffing to doctests</a>.</p> <p>While ExUnit is our test framework, Mix is our build tool. Developers typically run their tests by calling <code class="language-plaintext highlighter-rouge">mix test</code>.</p> <p>On the <code class="language-plaintext highlighter-rouge">mix test</code> side of things, there is a new <code class="language-plaintext highlighter-rouge">--failed</code> flag that runs all tests that failed the last time they ran. Finally, coverage reports generated with <code class="language-plaintext highlighter-rouge">mix test --cover</code> include a summary out of the box:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Generating cover results ... Percentage | Module -----------|-------------------------- 100.00% | Plug.Exception.Any 100.00% | Plug.Adapters.Cowboy2.Stream 100.00% | Collectable.Plug.Conn 100.00% | Plug.Crypto.KeyGenerator 100.00% | Plug.Parsers 100.00% | Plug.Head 100.00% | Plug.Router.Utils 100.00% | Plug.RequestId ... | ... -----------|-------------------------- 77.19% | Total </code></pre></div></div> <h2 id="summing-up">Summing up</h2> <p>We are really proud of this release, as it focuses mostly on quality of life improvements, instead of flashy new features. As Elixir continues to mature, expect more releases like this one. The full list of changes is available in our <a href="https://github.com/elixir-lang/elixir/releases/tag/v1.7.0">release notes</a>.</p> <p>We have also seen important developments in other areas not directly related to the Elixir codebase:</p> <ul> <li> <p>We have added <a href="https://elixir-lang.org/development.html">a “Development” section</a> to the website, that outlines the Elixir team structure and goals</p> </li> <li> <p><a href="https://www.youtube.com/watch?v=lxYFOM3UJzo">Elixir now has its own mini-documentary</a> by Honeypot</p> </li> <li> <p>We have already highlighted some of the improvements in the ExDoc tool. Another improvement worth mentioning is the syntax highlighting is now done in Elixir itself, via the <a href="https://github.com/tmbb/makeup">Makeup</a> library. This gives us more control over the grammar, the style, and improves load times. If you would like to add support for other languages, <a href="https://github.com/tmbb/makeup">reach out</a>!</p> </li> </ul> <p>Finally, don’t forget <a href="https://elixirconf.com/">ElixirConf US</a> is coming soon, in Bellevue, WA, September 4-7. Last year my keynote focused on the last 5 years with Elixir. This year we are ready to look into the 5 years ahead.</p> <p>Check <a href="/install.html">the Install section</a> to get Elixir installed and read our <a href="https://hexdocs.pm/elixir/introduction.html">Getting Started guide</a> to learn more.</p> Andrea Leopardi 2018-06-28T00:00:00+00:00 /blog/2018/06/28/gsoc-projects <p>Like previous years, the Elixir community is happy to participate in <a href="https://summerofcode.withgoogle.com">Google Summer of Code</a> 2018. We are currently working on four different projects. Let’s have a look at them.</p> <h2 id="streamdata-integration-with-typespecs">StreamData integration with typespecs</h2> <p><em>Student: Nikola Jichev</em></p> <p><a href="https://github.com/whatyouhide/stream_data">StreamData</a> is a data-generation and property-based testing library for Elixir. The goal of this GSoC project is to integrate StreamData with typespecs.</p> <p>The data-generation side of StreamData provides tools to generate random data through composable generators. For example, you could generate keyword lists like this:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="no">StreamData</span> <span class="n">keywords_generator</span> <span class="o">=</span> <span class="n">list_of</span><span class="p">({</span><span class="n">atom</span><span class="p">(</span><span class="ss">:alphanumeric</span><span class="p">),</span> <span class="n">term</span><span class="p">()})</span> <span class="no">Enum</span><span class="o">.</span><span class="n">take</span><span class="p">(</span><span class="n">keywords_generator</span><span class="p">,</span> <span class="mi">2</span><span class="p">)</span> <span class="c1">#=&gt; [[_: [true]], [tm: 2, h: %{}]]</span> </code></pre></div></div> <p>In many cases, it would be useful to be able to generate such random data starting from already existing or user-defined types. For example, Elixir already provides a built-in <code class="language-plaintext highlighter-rouge">keyword/0</code> type for keyword lists defined roughly as:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">@type</span> <span class="n">keyword</span><span class="p">()</span> <span class="p">::</span> <span class="p">[{</span><span class="n">atom</span><span class="p">(),</span> <span class="n">any</span><span class="p">()}]</span> </code></pre></div></div> <p>The goal of the first part of this GSoC project is to provide StreamData with the ability to create data generators from type definitions. The API is not yet defined, but in this case, it could look something like the following:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="no">StreamData</span> <span class="n">keywords_generator</span> <span class="o">=</span> <span class="n">from_type</span><span class="p">(</span><span class="n">keyword</span><span class="o">/</span><span class="mi">0</span><span class="p">)</span> <span class="no">Enum</span><span class="o">.</span><span class="n">take</span><span class="p">(</span><span class="n">keywords_generator</span><span class="p">,</span> <span class="mi">2</span><span class="p">)</span> <span class="c1">#=&gt; [[_: [true]], [tm: 2, h: %{}]]</span> </code></pre></div></div> <p>In the second part of the GSoC project, the aim is to be able to property-test functions with specs automatically.</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">@spec</span> <span class="n">has_key?</span><span class="p">(</span><span class="n">keyword</span><span class="p">(),</span> <span class="n">atom</span><span class="p">())</span> <span class="p">::</span> <span class="n">boolean</span><span class="p">()</span> <span class="k">def</span> <span class="n">has_key?</span><span class="p">(</span><span class="n">keyword</span><span class="p">,</span> <span class="n">key</span><span class="p">)</span> <span class="k">do</span> <span class="c1"># ...</span> <span class="k">end</span> </code></pre></div></div> <p>The first part of the project focuses on generating data from types, so we know how to generate function arguments. The missing piece is <strong>validating</strong> that a given term <em>belongs to</em> a given type. For example, in the snippet above, we want to be able to check if a term is a <code class="language-plaintext highlighter-rouge">boolean()</code>. Once we’re able to do this, automatic spec validation will be straightforward: it will be a matter of generating random arguments for the given function, calling the function with those arguments, and asserting that the returned value belongs to the return type defined in the spec.</p> <p>This kind of property-based testing doesn’t test for <em>correctness</em>. In the snippet above, <code class="language-plaintext highlighter-rouge">has_key?/2</code> could be implemented to ignore arguments always return <code class="language-plaintext highlighter-rouge">false</code> and the automatic spec validation would pass since <code class="language-plaintext highlighter-rouge">false</code> is always a boolean. However, this is a kind of <strong>smoke testing</strong> useful for discovering inconsistencies in the arguments and return values of functions.</p> <h2 id="tensorflex-tensorflow-bindings-for-the-elixir-programming-language">Tensorflex: Tensorflow bindings for the Elixir programming language</h2> <p><em>Student: Anshuman Chhabra</em></p> <p>Currently, there is a lack of machine learning tools and frameworks for Elixir. With the number of programmers learning/using machine learning only set to grow, supporting machine learning capabilities is essential for any programming language. Moreover, there are discussions on <a href="https://elixirforum.com">ElixirForum</a> regarding this and recent talks given at ElixirConf that reflect the need for Elixir to provide machine learning capabilities.</p> <p>This project’s goal is Tensorflex, an Elixir machine learning framework similar to <a href="https://keras.io">Keras for Python</a>. Keras uses Tensorflow as a backend for doing all the machine learning. Tensorflex will use Using Native Implemented Functions (NIFs) and the Tensorflow C API as a backend to provide a low-level API. This low-level API will then be used to write a Keras-like framework in the form of a high-level API. This will allow Elixir developers to write expedient and efficient machine learning code in Elixir.</p> <h2 id="dialyzer-task-for-elixir">Dialyzer task for Elixir</h2> <p><em>Student: Gabriel Gatu</em></p> <p>Dialyzer is a discrepancy analyzer that ships as part of Erlang/OTP. Currently, there are two projects that add Dialyzer support to Elixir applications: <a href="https://github.com/jeremyjh/dialyxir">dialyxir</a> and <a href="https://github.com/Comcast/dialyzex">dialyzex</a>. The goal of this project is to bring ideas from both projects into Elixir itself in order to make using Dialyzer in Elixir projects easier. The task we aim to add to Elixir will focus on two main features: better user experience (in particular, better error messages and formatting) and the ability to analyze projects incrementally.</p> <h2 id="elixirbench">ElixirBench</h2> <p><em>Student: Tallys Martins</em></p> <p>ElixirBench aims to be a service to monitor performance of Elixir projects. The goal of the GSoC project is to bring ElixirBench up and have it run nightly performance monitoring of significant Elixir projects (including Elixir itself). The end goal is to have a platform that, given a project from GitHub, will monitor the performance of new releases of that project and look for performance regressions. The benchmarking process will be controlled through a configuration file that will specify the benchmark scripts to run.</p> <p>We have high hopes for this tool as we see value in it for the whole community and for core Elixir projects alike.</p> José Valim 2018-01-17T00:00:00+00:00 /blog/2018/01/17/elixir-v1-6-0-released <p>Elixir v1.6 includes new features, enhancements, and bug fixes. The main feature in this release is a code formatter. Important improvements can also be found in the standard library and in the Mix build tool.</p> <h2 id="code-formatter">Code formatter</h2> <p>The big feature in Elixir v1.6 is the addition of <a href="https://hexdocs.pm/elixir/Code.html#format_string!/2">a code formatter</a> and an accompanying <code class="language-plaintext highlighter-rouge">mix format</code> task that adds automatic formatting to your projects.</p> <p>The goal of the formatter is to automate the styling of codebases into a unique and consistent layout used across teams and the whole community. Code is now easier to write, as you no longer need to concern yourself with formatting rules. Code is also easier to read, as you no longer need to convert the styles of other developers in your mind.</p> <p>The formatter also helps new developers to learn the language by giving immediate feedback on code structure, and eases code reviews by allowing teams to focus on business rules and code quality rather than code style.</p> <p>To automatically format your codebase, you can run the <a href="https://hexdocs.pm/mix/Mix.Tasks.Format.html">new <code class="language-plaintext highlighter-rouge">mix format</code> task</a>. A <code class="language-plaintext highlighter-rouge">.formatter.exs</code> file may be added to your project root for rudimentary formatter configuration. The mix task also supports flags for CI integration. For instance, you can make your build or a Pull Request fail if the code is not formatted accordingly by using the <code class="language-plaintext highlighter-rouge">--check-formatted</code> flag. We also recommend developers to check their favorite editor and see if it already provides key bindings for <code class="language-plaintext highlighter-rouge">mix format</code>, allowing a file or a code snippet to be formatted without hassle.</p> <p>The Elixir codebase itself <a href="https://github.com/elixir-lang/elixir/issues/6643">has already been fully formatted</a> and all further contributions are expected to contain formatted code. We recommend existing codebases to be formatted in steps. While the formatter will correctly handle long lines and complex expressions, refactoring the code by breaking those into variables or smaller functions as you format them will lead to overall cleaner and more readable codebases.</p> <h2 id="dynamic-supervisor">Dynamic Supervisor</h2> <p>Supervisors in Elixir are responsible for starting, shutting down, and restarting child processes when things go wrong. Most of the interaction with supervisors happens through <a href="https://hexdocs.pm/elixir/Supervisor.html">the <code class="language-plaintext highlighter-rouge">Supervisor</code> module</a> and it provides three main strategies: <code class="language-plaintext highlighter-rouge">:one_for_one</code>, <code class="language-plaintext highlighter-rouge">:rest_for_one</code> and <code class="language-plaintext highlighter-rouge">:one_for_all</code>.</p> <p>However, sometimes the children of a supervisor are not known upfront and are rather started dynamically. For example, if you are building a web server, you have each request being handled by a separate supervised process. Those cases were handled in the Supervisor module under a special strategy called <code class="language-plaintext highlighter-rouge">:simple_one_for_one</code>.</p> <p>Unfortunately, this special strategy changed the semantics of the supervisor in regards to initialization and shutdown. Plus some APIs expected different inputs or would be completely unavailable depending on the supervision strategy.</p> <p>Elixir v1.6 addresses this issue by introducing <a href="https://hexdocs.pm/elixir/DynamicSupervisor.html">a new <code class="language-plaintext highlighter-rouge">DynamicSupervisor</code> module</a>, which encapsulates the old <code class="language-plaintext highlighter-rouge">:simple_one_for_one</code> strategy and APIs in a proper module while allowing the documentation and API of the <code class="language-plaintext highlighter-rouge">Supervisor</code> module to focus on its main use cases. Having a separate <code class="language-plaintext highlighter-rouge">DynamicSupervisor</code> module also makes it simpler to add new features to the dynamic supervisor, such as the new <code class="language-plaintext highlighter-rouge">:max_children</code> option that limits the maximum number of children supervised dynamically.</p> <h2 id="deprecated-and-since-attributes"><code class="language-plaintext highlighter-rouge">@deprecated</code> and <code class="language-plaintext highlighter-rouge">@since</code> attributes</h2> <p>This release also introduces two new attributes associated with function definitions: <code class="language-plaintext highlighter-rouge">@deprecated</code> and <code class="language-plaintext highlighter-rouge">@since</code>. The former marks if a function or macro is deprecated, the latter annotates the version the API was introduced:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">@doc</span> <span class="s2">"Breaks a collection into chunks"</span> <span class="nv">@since</span> <span class="s2">"1.0.0"</span> <span class="nv">@deprecated</span> <span class="s2">"Use chunk_every/2 instead"</span> <span class="k">def</span> <span class="n">chunk</span><span class="p">(</span><span class="n">collection</span><span class="p">,</span> <span class="n">chunk_size</span><span class="p">)</span> <span class="k">do</span> <span class="n">chunk_every</span><span class="p">(</span><span class="n">collection</span><span class="p">,</span> <span class="n">chunk_size</span><span class="p">)</span> <span class="k">end</span> </code></pre></div></div> <p>The <code class="language-plaintext highlighter-rouge">mix xref</code> task was also updated to warn if your project calls deprecated code. So if a definition is marked as <code class="language-plaintext highlighter-rouge">@deprecated</code> and a module invokes it, a warning will be emitted during compilation. This effectively provides libraries and frameworks a mechanism to deprecate code without causing multiple warnings to be printed in runtime and without impacting performance.</p> <p>Note those attributes are not yet available to tools that generate documentation. Such functionality will be added in Elixir v1.7 once <a href="https://github.com/elixir-lang/elixir/issues/7198">Elixir adopts EEP-48</a>. We still recommend developers to start annotating their APIs so the information is already available when the tooling is updated.</p> <h2 id="defguard-and-defguardp"><code class="language-plaintext highlighter-rouge">defguard</code> and <code class="language-plaintext highlighter-rouge">defguardp</code></h2> <p>Elixir provides the concepts of guards: expressions used alongside pattern matching to select a matching clause. Let’s see an example straight from <a href="https://elixir-lang.org">Elixir’s home page</a>:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="n">drive</span><span class="p">(%</span><span class="no">User</span><span class="p">{</span><span class="ss">age:</span> <span class="n">age</span><span class="p">})</span> <span class="ow">when</span> <span class="n">age</span> <span class="o">&gt;=</span> <span class="mi">16</span> <span class="k">do</span> <span class="c1"># Code that drives a car</span> <span class="k">end</span> </code></pre></div></div> <p><code class="language-plaintext highlighter-rouge">%User{age: age}</code> is matching on a <code class="language-plaintext highlighter-rouge">User</code> struct with an age field and <code class="language-plaintext highlighter-rouge">when age &gt;= 16</code> is the guard.</p> <p>Since only a handful of constructs are <a href="https://hexdocs.pm/elixir/guards.html#content">allowed in guards</a>, if you were in a situation where you had to check the age to be more than or equal to 16 in multiple places, extracting the guard to a separate function would be <a href="https://github.com/elixir-lang/elixir/issues/2469">less than obvious and error prone</a>. To address those issues, <a href="https://hexdocs.pm/elixir/Kernel.html#defguard/1">this release introduces <code class="language-plaintext highlighter-rouge">defguard/1</code> and <code class="language-plaintext highlighter-rouge">defguardp/1</code></a>:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defguard</span> <span class="n">is_old_to_drive</span><span class="p">(</span><span class="n">age</span><span class="p">)</span> <span class="ow">when</span> <span class="n">age</span> <span class="o">&gt;=</span> <span class="mi">16</span> <span class="k">def</span> <span class="n">drive</span><span class="p">(%</span><span class="no">User</span><span class="p">{</span><span class="ss">age:</span> <span class="n">age</span><span class="p">})</span> <span class="ow">when</span> <span class="n">is_old_to_drive</span><span class="p">(</span><span class="n">age</span><span class="p">)</span> <span class="k">do</span> <span class="c1"># Code that drives a car</span> <span class="k">end</span> </code></pre></div></div> <h2 id="iex-improvements">IEx improvements</h2> <p>IEx also got its share of improvements. The new code formatter allows us to pretty print code snippets, types and specifications, improving the overall experience when exploring code through the terminal.</p> <p>The autocomplete mechanism also got smarter, being able to provide context autocompletion. For example, typing <code class="language-plaintext highlighter-rouge">t Enum.</code> and hitting TAB will autocomplete only the types in Enum (in contrast to all functions). Typing <code class="language-plaintext highlighter-rouge">b GenServer.</code> and hitting TAB will autocomplete only the behaviour callbacks.</p> <p>Finally, the breakpoint functionality added <a href="https://elixir-lang.org/blog/2017/07/25/elixir-v1-5-0-released/">in Elixir v1.5</a> has been improved to support pattern matching and guards. For example, to pattern match on a function call when the first argument is the atom <code class="language-plaintext highlighter-rouge">:foo</code>, you may do:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>iex&gt; break! SomeFunction.call(:foo, _, _) </code></pre></div></div> <p>For more information, see <a href="https://hexdocs.pm/iex/IEx.html#break!/4"><code class="language-plaintext highlighter-rouge">IEx.break!/4</code></a>.</p> <h2 id="mix-xref">mix xref</h2> <p><a href="https://hexdocs.pm/mix/Mix.Tasks.Xref.html"><code class="language-plaintext highlighter-rouge">mix xref</code></a> is a task added in Elixir v1.3 which provides general information about how modules and files in an application depend on each other. This release brings many improvements to <code class="language-plaintext highlighter-rouge">xref</code>, extending the reach of the analysis and helping developers digest the vast amount of data it produces.</p> <p>One of such additions is the <code class="language-plaintext highlighter-rouge">--include-siblings</code> option that can be given to all <code class="language-plaintext highlighter-rouge">xref</code> commands inside umbrella projects. For example, to find all of the callers of a given module or function of an application in an umbrella:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ mix xref callers SomeModule --include-siblings </code></pre></div></div> <p>The <code class="language-plaintext highlighter-rouge">graph</code> command in <code class="language-plaintext highlighter-rouge">mix xref</code> now can also output general statistics about the graph. In <a href="https://github.com/hexpm/hexpm">the hexpm project</a>, you would get:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ mix xref graph --format stats Tracked files: 129 (nodes) Compile dependencies: 256 (edges) Structs dependencies: 46 (edges) Runtime dependencies: 266 (edges) Top 10 files with most outgoing dependencies: * test/support/factory.ex (18) * lib/hexpm/accounts/user.ex (13) * lib/hexpm/accounts/audit_log.ex (12) * lib/hexpm/web/controllers/dashboard_controller.ex (12) * lib/hexpm/repository/package.ex (12) * lib/hexpm/repository/releases.ex (11) * lib/hexpm/repository/release.ex (10) * lib/hexpm/web/controllers/package_controller.ex (10) * lib/mix/tasks/hexpm.stats.ex (9) * lib/hexpm/repository/registry_builder.ex (9) Top 10 files with most incoming dependencies: * lib/hexpm/web/web.ex (84) * lib/hexpm/web/router.ex (29) * lib/hexpm/web/controllers/controller_helpers.ex (29) * lib/hexpm/web/controllers/auth_helpers.ex (28) * lib/hexpm/web/views/view_helpers.ex (27) * lib/hexpm/web/views/icons.ex (27) * lib/hexpm/web/endpoint.ex (23) * lib/hexpm/ecto/changeset.ex (22) * lib/hexpm/accounts/user.ex (19) * lib/hexpm/repo.ex (19) </code></pre></div></div> <p><code class="language-plaintext highlighter-rouge">mix xref graph</code> also got the <code class="language-plaintext highlighter-rouge">--only-nodes</code> and <code class="language-plaintext highlighter-rouge">--label</code> options. The former asks Mix to only output file names (nodes) without the edges. The latter allows you to focus on certain relationships:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code># To get all files that depend on lib/foo.ex mix xref graph --sink lib/foo.ex --only-nodes # To get all files that depend on lib/foo.ex at compile time mix xref graph --label compile --sink lib/foo.ex --only-nodes # To get all files lib/foo.ex depends on mix xref graph --source lib/foo.ex --only-nodes # To limit statistics only to compile time dependencies mix xref graph --format stats --label compile </code></pre></div></div> <p>Those improvements will help developers better understand the relationship between files and reveal potentially complex parts of their systems.</p> <p>Other improvements in Mix include <a href="https://hexdocs.pm/mix/Mix.Task.Compiler.html">better compiler diagnostics</a> for editor integration, support for <a href="https://hexdocs.pm/mix/Mix.Tasks.Test.html">the <code class="language-plaintext highlighter-rouge">--slowest N</code> flag in <code class="language-plaintext highlighter-rouge">mix test</code></a> that shows the slowest tests in your suite, and a new <a href="https://hexdocs.pm/mix/Mix.Tasks.Eprof.html"><code class="language-plaintext highlighter-rouge">mix profile.eprof</code> task</a> that provides time based profiling, complementing the existing <a href="https://hexdocs.pm/mix/Mix.Tasks.Profile.Cprof.html"><code class="language-plaintext highlighter-rouge">mix profile.cprof</code> (count based)</a> and <a href="https://hexdocs.pm/mix/Mix.Tasks.Profile.Fprof.html"><code class="language-plaintext highlighter-rouge">mix profile.fprof</code> (flame based)</a>.</p> <h2 id="summing-up">Summing up</h2> <p>The full list of changes is available in our <a href="https://github.com/elixir-lang/elixir/releases/tag/v1.6.0">release notes</a>. There are many other exciting changes, such as compiler enhancements to better track dependencies, leading to less files being recompiled whenever there are changes in project, and overall faster compilation.</p> <p>Work on Elixir v1.7 has already started. We still welcome developers to try out the <a href="https://elixir-lang.org/blog/2017/10/31/stream-data-property-based-testing-and-data-generation-for-elixir/">previously announced StreamData library</a>, that aims to bring data generation and property-based testing to Elixir. The other <a href="https://github.com/elixir-lang/elixir/issues">features scheduled for v1.7 can be found in the issues tracker</a>.</p> <p>Don’t forget to check <a href="/install.html">the Install section</a> to get Elixir installed and our <a href="https://hexdocs.pm/elixir/introduction.html">Getting Started guide</a> to learn more.</p> Andrea Leopardi 2017-10-31T00:00:00+00:00 /blog/2017/10/31/stream-data-property-based-testing-and-data-generation-for-elixir <p>In this blog post, we’ll talk about property-based testing and sample data generation. We’ll cover what these are, why we want them in Elixir, and what are are plans for the future. If you want to use the features discussed here or you want to read more formal documentation, head over to <a href="https://github.com/whatyouhide/stream_data">stream_data</a>, which is a library that currently provides both features (albeit in beta form) and which is where we are focusing our efforts.</p> <h2 id="sample-data-generation">Sample data generation</h2> <p>The core of the <a href="https://github.com/whatyouhide/stream_data">stream_data</a> library is <code class="language-plaintext highlighter-rouge">StreamData</code>: this module provides all the functionalities related to generating sample data of many kinds. It includes both data generators for data types (like integers or booleans) as well as tools to combine other generators (such as <code class="language-plaintext highlighter-rouge">one_of(list_of_generators)</code>).</p> <p>Developers are not supposed to create generators from scratch, but use the provided generators and the provided combinator functions to compose them. An example of a generator is the one returned by <code class="language-plaintext highlighter-rouge">StreamData.integer()</code>: this function returns a generator that generates integers. Generators are infinite streams of terms that implement the <code class="language-plaintext highlighter-rouge">Enumerable</code> protocol. This means we can do operations such as taking terms out of a generator through functions from <code class="language-plaintext highlighter-rouge">Enum</code> and <code class="language-plaintext highlighter-rouge">Stream</code>:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="no">Enum</span><span class="o">.</span><span class="n">take</span><span class="p">(</span><span class="no">StreamData</span><span class="o">.</span><span class="n">integer</span><span class="p">(),</span> <span class="mi">5</span><span class="p">)</span> <span class="c1">#=&gt; [1, -1, 3, 4, 3]</span> </code></pre></div></div> <p><code class="language-plaintext highlighter-rouge">StreamData</code> contains some functions to modify generators. For example, you can build a generator of positive integers on top of <code class="language-plaintext highlighter-rouge">StreamData.integer()</code> and <code class="language-plaintext highlighter-rouge">StreamData.map/2</code>:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">generator</span> <span class="o">=</span> <span class="no">StreamData</span><span class="o">.</span><span class="n">map</span><span class="p">(</span><span class="no">StreamData</span><span class="o">.</span><span class="n">integer</span><span class="p">(),</span> <span class="o">&amp;</span><span class="n">abs</span><span class="o">/</span><span class="mi">1</span><span class="p">)</span> <span class="no">Enum</span><span class="o">.</span><span class="n">take</span><span class="p">(</span><span class="n">generator</span><span class="p">,</span> <span class="mi">5</span><span class="p">)</span> <span class="c1">#=&gt; [0, 1, 3, 3, 2]</span> </code></pre></div></div> <p><code class="language-plaintext highlighter-rouge">StreamData.map/2</code> is encouraged over <code class="language-plaintext highlighter-rouge">Stream.map/2</code> because generators return values that can shrink, which is something property-based testing takes advantage of as we’ll see later on. When treated as enumerables, generators return normal values that cannot be shrunk.</p> <p>We decided to separate data-generation from property-based testing because it’s something that developers can take advantage of in situations outside of property-based testing. For example, data streams can be used to seed a database or to have randomly generated data available during regular tests.</p> <h2 id="property-based-testing">Property-based testing</h2> <p>We often write tests like this:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">test</span> <span class="s2">"length/1 calculates the length of a list"</span> <span class="k">do</span> <span class="n">assert</span> <span class="n">length</span><span class="p">([])</span> <span class="o">==</span> <span class="mi">0</span> <span class="n">assert</span> <span class="n">length</span><span class="p">([</span><span class="ss">:one</span><span class="p">])</span> <span class="o">==</span> <span class="mi">1</span> <span class="n">assert</span> <span class="n">length</span><span class="p">([</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">])</span> <span class="o">==</span> <span class="mi">3</span> <span class="k">end</span> </code></pre></div></div> <p>This test is written using an <em>example-based approach</em>. We are writing both the input to the piece of software we are testing as well as the expected output, and the testing tool is verifying that running the software on the given input results in the expected output. This style of testing is common and useful because it lets you get up and running easily and also lets you test known corner cases in an explicit way. However, it’s hard to test many cases this way and even harder to uncover <em>unknown</em> corner cases that may reveal bugs in your code.</p> <p>Property-based testing is an intuitive way to fix some of the problems mentioned above.</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">property</span> <span class="s2">"length/1 is always &gt;= 0"</span> <span class="k">do</span> <span class="n">check</span> <span class="n">all</span> <span class="n">list</span> <span class="o">&lt;-</span> <span class="n">list_of</span><span class="p">(</span><span class="n">term</span><span class="p">())</span> <span class="k">do</span> <span class="n">assert</span> <span class="n">length</span><span class="p">(</span><span class="n">list</span><span class="p">)</span> <span class="o">&gt;=</span> <span class="mi">0</span> <span class="k">end</span> <span class="k">end</span> </code></pre></div></div> <p>With property-based testing, you specify a set of valid inputs (lists in the example above) for your code and verify that your code holds some property for values taken at random from the valid inputs. In the example above, the test takes many (usually around 100) values at random from the <code class="language-plaintext highlighter-rouge">list_of(term())</code> <em>generator</em> and verifies a property of <code class="language-plaintext highlighter-rouge">length/1</code>, that is, that <code class="language-plaintext highlighter-rouge">length/1</code> always returns a non-negative integer. A generator is just a <code class="language-plaintext highlighter-rouge">StreamData</code> generator, as we discussed in the previous section.</p> <h3 id="shrinking">Shrinking</h3> <p>Since we’re generating lots of random inputs to test, inputs that cause failures are often complex and convoluted. Take this trivial example of a property:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">property</span> <span class="s2">"list does not contain multiples of 4"</span> <span class="k">do</span> <span class="n">check</span> <span class="n">all</span> <span class="n">list</span> <span class="o">&lt;-</span> <span class="n">list_of</span><span class="p">(</span><span class="n">positive_integer</span><span class="p">())</span> <span class="k">do</span> <span class="n">refute</span> <span class="no">Enum</span><span class="o">.</span><span class="n">any?</span><span class="p">(</span><span class="n">list</span><span class="p">,</span> <span class="o">&amp;</span><span class="p">(</span><span class="n">rem</span><span class="p">(</span><span class="nv">&amp;1</span><span class="p">,</span> <span class="mi">4</span><span class="p">)</span> <span class="o">==</span> <span class="mi">0</span><span class="p">))</span> <span class="k">end</span> <span class="k">end</span> </code></pre></div></div> <p>When running this property, the failure might trigger for a list like this:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">[</span><span class="mi">19</span><span class="p">,</span> <span class="mi">12</span><span class="p">,</span> <span class="mi">6</span><span class="p">,</span> <span class="mi">11</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">20</span><span class="p">,</span> <span class="mi">10</span><span class="p">]</span> </code></pre></div></div> <p>From this list, it’s not easy to see why the test is failing (well, we know why because we wrote a doctored test that’s supposed to fail). When running the property though, the failure that will be reported will look like this:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>1) property list does not contain multiples of 7 (MyPropertyTest) my_property_test.exs:6 Failed with generated values (after 4 attempt(s)): list &lt;- list_of(positive_integer()) #=&gt; [4] Expected false or nil, got true code: refute Enum.any?(list, &amp;(rem(&amp;1, 4) == 0)) </code></pre></div></div> <p>This error shows the minimal generated value that triggers the failure, that is, <code class="language-plaintext highlighter-rouge">[4]</code>. The process of finding the minimal generated value that triggers a failure is called <em>shrinking</em>. All generators that come with <code class="language-plaintext highlighter-rouge">StreamData</code> generate values that “bundle” a way to shrink them so that property-based testing can use this to provide the shrinking functionality. Shrinking is a fundamental part of property-based testing as it takes out the <em>noise</em> of random-generated data to reduce the failing data to focused and easier-to-understand terms.</p> <h3 id="using-property-based-testing-in-stream_data">Using property-based testing in stream_data</h3> <p>The core of property-based testing in stream_data is the <code class="language-plaintext highlighter-rouge">check all</code> macro. In this macro, you list a bunch of generators and filters (very similarly to how you would in <code class="language-plaintext highlighter-rouge">for</code> comprehensions) and then pass a body where you can verify that a property holds for the generated data.</p> <p>To make the <code class="language-plaintext highlighter-rouge">check all</code> macro available in your test, alongside importing all functions from <code class="language-plaintext highlighter-rouge">StreamData</code>, you can <code class="language-plaintext highlighter-rouge">use ExUnitProperties</code>:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">MyPropertyTest</span> <span class="k">do</span> <span class="kn">use</span> <span class="no">ExUnit</span><span class="o">.</span><span class="no">Case</span><span class="p">,</span> <span class="ss">async:</span> <span class="no">true</span> <span class="kn">use</span> <span class="no">ExUnitProperties</span> <span class="n">test</span> <span class="s2">"the in/2 operator works with lists"</span> <span class="k">do</span> <span class="n">check</span> <span class="n">all</span> <span class="n">list</span> <span class="o">&lt;-</span> <span class="n">list_of</span><span class="p">(</span><span class="n">term</span><span class="p">()),</span> <span class="n">list</span> <span class="o">!=</span> <span class="p">[],</span> <span class="n">elem</span> <span class="o">&lt;-</span> <span class="n">member_of</span><span class="p">(</span><span class="n">list</span><span class="p">)</span> <span class="k">do</span> <span class="n">assert</span> <span class="n">elem</span> <span class="ow">in</span> <span class="n">list</span> <span class="k">end</span> <span class="k">end</span> <span class="k">end</span> </code></pre></div></div> <p>As you can see, we can filter generated data (<code class="language-plaintext highlighter-rouge">list != []</code>) directly in the <code class="language-plaintext highlighter-rouge">check all</code> macro. We can also do simple assignments. The example above uses the <code class="language-plaintext highlighter-rouge">check all</code> macro inside a regular <code class="language-plaintext highlighter-rouge">test</code>. If you want that your properties are reported as “property” at the end of an ExUnit test run, you can use the <code class="language-plaintext highlighter-rouge">property</code> macro instead:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">MyPropertyTest</span> <span class="k">do</span> <span class="kn">use</span> <span class="no">ExUnit</span><span class="o">.</span><span class="no">Case</span><span class="p">,</span> <span class="ss">async:</span> <span class="no">true</span> <span class="kn">use</span> <span class="no">ExUnitProperties</span> <span class="n">property</span> <span class="s2">"the in/2 operator works with lists"</span> <span class="k">do</span> <span class="n">check</span> <span class="n">all</span> <span class="n">list</span> <span class="o">&lt;-</span> <span class="n">list_of</span><span class="p">(</span><span class="n">term</span><span class="p">()),</span> <span class="n">list</span> <span class="o">!=</span> <span class="p">[],</span> <span class="n">elem</span> <span class="o">&lt;-</span> <span class="n">member_of</span><span class="p">(</span><span class="n">list</span><span class="p">)</span> <span class="k">do</span> <span class="n">assert</span> <span class="n">elem</span> <span class="ow">in</span> <span class="n">list</span> <span class="k">end</span> <span class="k">end</span> <span class="k">end</span> </code></pre></div></div> <p>By doing this your properties will also be tagged with the <code class="language-plaintext highlighter-rouge">:property</code> tag, which means you will be able to do things like:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mix <span class="nb">test</span> <span class="nt">--only</span> property </code></pre></div></div> <p>to run only properties.</p> <p>There’s not much more to the mechanics of stream_data. Most of the work you will have to do revolves around finding good properties to test for your code and writing good generators for the data over which you want to test. Head over to <a href="https://github.com/whatyouhide/stream_data">stream_data</a>’s documentation for detailed documentation.</p> <h3 id="advantages-of-property-based-testing">Advantages of property-based testing</h3> <p>Using property-based testing has some advantages. First of all, it lets you test properties of your code over many more values than you otherwise would with example-based testing. While it’s true that random data generation can’t cover all the possible values that a piece of code can deal with, the confidence in your codebase can still increase over time because the property-based tests will likely generate different values on each run. Example-based testing means your test data will not change over time.</p> <p>Property-based testing however can also have a more powerful impact on the way you design software. When you start writing property-based tests, you will start thinking about what guarantees your code provides and what properties it satisfies. If you write properties before writing code, this can easily influence the way you write that code.</p> <h3 id="learning-resources">Learning resources</h3> <p>Property-based testing is not something specific to Elixir. While having its roots in Haskell (check out the <a href="http://www.cs.tufts.edu/~nr/cs257/archive/john-hughes/quick.pdf">original QuickCheck paper</a> if you’re interested), nowadays many languages have stable and usable implementations of it: Clojure has <a href="https://github.com/clojure/test.check">test.check</a>, Python has <a href="https://github.com/HypothesisWorks/hypothesis-python">Hypothesis</a>, and many more. One of the most famous and complete tools for property-based testing exists for Erlang itself: <a href="http://quviq.com">QuickCheck</a> by Quviq is a complete commercial solution for property-based testing in Erlang of both stateless as well as stateful systems, and Quviq even provides a custom Erlang scheduler to test race conditions in your concurrent programs.</p> <p>A young but awesome book about property-based testing written by Fred Hebert is also available at <a href="http://propertesting.com">propertesting.com</a>. This book is a <em>proper</em> (pun intended) guide to property-based testing and uses an Erlang library called <a href="https://github.com/manopapad/proper">PropEr</a>. However, the concepts and techniques perfectly apply to Elixir and stream_data as well.</p> <h2 id="why-include-property-based-testing-in-elixir-and-rewriting-from-scratch">Why include property-based testing in Elixir (and rewriting from scratch)</h2> <p>The community has expressed some concern regarding two main things: why do we want to include a property-based testing tool in Elixir’s standard library? And why write such a tool from scratch instead of using one of the existing Erlang or Elixir solutions?</p> <p>The answer to the first question is that we believe providing such a tool in the standard library will encourage developers to use property-based testing and ultimately improve their software and the way they write it. At the same time, we want to be able to use property-based testing to test the Elixir codebase itself (which already turned out great <a href="https://github.com/elixir-lang/elixir/pull/5022#issuecomment-233195478">in the past</a>).</p> <p>The reasons for writing a new property-based testing library from scratch are best explained by José in <a href="https://elixirforum.com/t/questions-about-property-testing-stream-data/9445/47">this ElixirForum post</a>:</p> <blockquote> <ul> <li>Since we want to bundle it as part of Elixir, the code should be open source with an appropriate license.</li> <li>We wanted to add both data generation and property testing to Elixir. That’s why the library is called stream_data instead of something named after property tests. The goal is to reduce the learning curve behind property testing by exposing the data generation aspect as streams, which is a known construct to most Elixir developers. We had this approach in mind for a while and the first library we saw leveraging this in practice was <a href="https://github.com/pragdave/pollution">@pragdave’s pollution</a>.</li> <li>Finally, since the core team are taking the responsibility of maintaining property testing as part of Elixir for potentially the rest of our lives, we want to have full understanding of every single line of code. This is non-negotiable as it guarantees we can continue to consistently improve the code as we move forward.</li> </ul> <p>We understand rolling our own implementation has its downsides, especially since it lacks maturity compared to alternatives, but we balance it by actively seeking input from knowledgeable folks and by listening to the feedback that comes from the community, which we are very thankful for.</p> </blockquote> <h2 id="roadmap">Roadmap</h2> <p><code class="language-plaintext highlighter-rouge">stream_data</code> and the functionalities it includes are scheduled to be included in one of the next two Elixir releases, likely 1.6 but possibly 1.7. We have used the names <code class="language-plaintext highlighter-rouge">StreamData</code> and <code class="language-plaintext highlighter-rouge">ExUnitProperties</code> to avoid conflicts when those modules are eventually merged into Elixir. When merged, they will be renamed to the proper <code class="language-plaintext highlighter-rouge">Stream.Data</code> and <code class="language-plaintext highlighter-rouge">ExUnit.Properties</code> modules. Right now, all development is happening in the <a href="https://github.com/whatyouhide/stream_data">stream_data</a> repository, where we are discussing features and giving users a chance to try out the functionalities early on. We’d love for anyone to get involved in trying stream_data and we’d love feedback!</p> <p><strong>Update Jun/2020:</strong> after careful consideration, the Elixir team decided to not include <code class="language-plaintext highlighter-rouge">StreamData</code> in Elixir itself, and keep it as package, as it is able to provide all of the necessary features without a need for direct integration with the language.</p> José Valim 2017-07-25T00:00:00+00:00 /blog/2017/07/25/elixir-v1-5-0-released <p>Elixir v1.5 includes new features, enhancements, and bug fixes. While <a href="/blog/2017/01/05/elixir-v1-4-0-released/">Elixir v1.4</a> focused on tools for concurrency and scalability, Elixir v1.5 brings many improvements to the developer experience and quality of life. As we will see, many of those are powered by the latest Erlang/OTP 20. This is also the last Elixir release that supports Erlang/OTP 18.</p> <p>Note: this announcement contains <a href="https://asciinema.org">asciinema</a> snippets. You may need to enable 3rd-party JavaScript on this site in order to see them. If JavaScript is disabled, noscript tags with the proper links will be shown.</p> <h2 id="utf-8-atoms-function-names-and-variables">UTF-8 atoms, function names and variables</h2> <p>Elixir v1.5 supports non-quoted atoms and variables to be in UTF-8 when using Erlang/OTP 20. For example:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">test</span> <span class="s2">"こんにちは世界"</span> <span class="k">do</span> <span class="n">assert</span> <span class="p">:</span><span class="err">こんにちは世界</span> <span class="k">end</span> </code></pre></div></div> <p>Or:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">sauda</span><span class="err">çã</span><span class="n">o</span> <span class="o">=</span> <span class="s2">"Bom dia!"</span> </code></pre></div></div> <p>Elixir follows the recommendations in <a href="http://unicode.org/reports/tr31/">Unicode Annex #31</a> to make Elixir more accessible to other languages and communities. Identifiers must still be a sequence of letters, followed by digits and combining marks. This means symbols, such as mathematical notations and emoji, are not allowed in identifiers.</p> <p>For a complete reference on Elixir syntax, see the <a href="https://hexdocs.pm/elixir/1.5/syntax-reference.html">Syntax Reference</a>. For technical details on Unicode support, see <a href="https://hexdocs.pm/elixir/1.5/unicode-syntax.html">Unicode Syntax</a>.</p> <h2 id="iex-helpers-and-breakpoints">IEx helpers and breakpoints</h2> <p>IEx got many enhancements to the developer experience.</p> <p>First of all, the autocompletion system is now capable of autocompleting variables and user imports:</p> <script type="text/javascript" src="https://asciinema.org/a/iAOk0yaZtQDsuJqn2sXa1FRQW.js" id="asciicast-iAOk0yaZtQDsuJqn2sXa1FRQW" async=""></script> <noscript><p><a href="https://asciinema.org/a/iAOk0yaZtQDsuJqn2sXa1FRQW">See the example in asciinema</a></p></noscript> <p>IEx also got new functions, such as <code class="language-plaintext highlighter-rouge">exports/1</code>, for listing all functions and macros in a module, and the new <code class="language-plaintext highlighter-rouge">runtime_info/0</code>:</p> <script type="text/javascript" src="https://asciinema.org/a/NT3xvSaB8f1vv7yaTvzaoJxBD.js" id="asciicast-NT3xvSaB8f1vv7yaTvzaoJxBD" async=""></script> <noscript><p><a href="https://asciinema.org/a/NT3xvSaB8f1vv7yaTvzaoJxBD">See the example in asciinema</a></p></noscript> <p>Finally, IEx also features a breakpoint system for code debugging when running on Erlang/OTP 20. The following functions have been added to aid debugging:</p> <ul> <li><code class="language-plaintext highlighter-rouge">break!/2</code> - sets up a breakpoint for a given <code class="language-plaintext highlighter-rouge">Mod.fun/arity</code></li> <li><code class="language-plaintext highlighter-rouge">break!/4</code> - sets up a breakpoint for the given module, function, arity</li> <li><code class="language-plaintext highlighter-rouge">breaks/0</code> - prints all breakpoints and their ids</li> <li><code class="language-plaintext highlighter-rouge">continue/0</code> - continues until the next breakpoint in the same process</li> <li><code class="language-plaintext highlighter-rouge">open/0</code> - opens editor on the current breakpoint</li> <li><code class="language-plaintext highlighter-rouge">remove_breaks/0</code> - removes all breakpoints in all modules</li> <li><code class="language-plaintext highlighter-rouge">remove_breaks/1</code> - removes all breakpoints in a given module</li> <li><code class="language-plaintext highlighter-rouge">reset_break/1</code> - sets the number of stops on the given id to zero</li> <li><code class="language-plaintext highlighter-rouge">reset_break/3</code> - sets the number of stops on the given module, function, arity to zero</li> <li><code class="language-plaintext highlighter-rouge">respawn/0</code> - starts a new shell (breakpoints will ask for permission once more)</li> <li><code class="language-plaintext highlighter-rouge">whereami/1</code> - shows the current location</li> </ul> <p>Let’s see an example:</p> <script type="text/javascript" src="https://asciinema.org/a/0h3po0AmTcBAorc5GBNU97nrs.js" id="asciicast-0h3po0AmTcBAorc5GBNU97nrs" async=""></script> <noscript><p><a href="https://asciinema.org/a/0h3po0AmTcBAorc5GBNU97nrs">See the example in asciinema</a></p></noscript> <p>In the snippet above we set a breakpoint in the <code class="language-plaintext highlighter-rouge">URI.decode_query/2</code> function, which is then triggered when invoked the function. We used <code class="language-plaintext highlighter-rouge">whereami/1</code> to get more information about the surrounded code and we were also able to access the variables at place of debugging. From there, we can either set more breakpoints, remove existing breakpoints and continue execution. The session ended by calling <code class="language-plaintext highlighter-rouge">open</code>, which will open your editor at the file and line under debugging. <code class="language-plaintext highlighter-rouge">open/1</code> can also be invoked by passing any module or function, and IEx will open your editor at that place.</p> <p>The debugging functions improve the experience both within IEx and during testing. For example, if you are debugging a Phoenix application, you can start <code class="language-plaintext highlighter-rouge">IEx</code> while running your test suite with <code class="language-plaintext highlighter-rouge">iex -S mix test --trace</code> and then call <code class="language-plaintext highlighter-rouge">IEx.break!(MyAppWeb.UserController.index/2)</code> to debug the <code class="language-plaintext highlighter-rouge">index</code> action of the <code class="language-plaintext highlighter-rouge">UserController</code>. Note we gave the <code class="language-plaintext highlighter-rouge">--trace</code> flag to <code class="language-plaintext highlighter-rouge">mix test</code>, which ensures only one test runs at a time and removes any timeouts from the suite.</p> <h2 id="exceptionblame">Exception.blame</h2> <p><code class="language-plaintext highlighter-rouge">Exception.blame/3</code> is a new function in Elixir that is capable of attaching debug information to certain exceptions. Currently this is used to augment <code class="language-plaintext highlighter-rouge">FunctionClauseError</code>s with a summary of all clauses and which parts of clause match and which ones didn’t. Let’s try it out:</p> <script type="text/javascript" src="https://asciinema.org/a/EgQUdDe1CIz90EYYeipiS8jo8.js" id="asciicast-EgQUdDe1CIz90EYYeipiS8jo8" async=""></script> <noscript><p><a href="https://asciinema.org/a/EgQUdDe1CIz90EYYeipiS8jo8">See the example in asciinema</a></p></noscript> <p>In the example above, an argument that did not match or guard that did not evaluate to true are shown between in red. If the terminal does not support ANSI coloring, they are wrapped in <code class="language-plaintext highlighter-rouge">-</code> instead of shown in red.</p> <p>Since blaming an exception can be expensive, <code class="language-plaintext highlighter-rouge">Exception.blame/3</code> must be used exclusively in debugging situations. It is not advised to apply it to production components such as a Logger. This feature has been integrated into the compiler, the command line, ExUnit and IEx.</p> <p>This feature also requires Erlang/OTP 20.</p> <h2 id="streamlined-child-specs">Streamlined child specs</h2> <p>Elixir v1.5 streamlines how supervisors are defined and used in Elixir. Elixir now allows child specifications, which specify how a child process is supervised, to be defined in modules. In previous versions, a project using Phoenix would write:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="no">Supervisor</span><span class="o">.</span><span class="no">Spec</span> <span class="n">children</span> <span class="o">=</span> <span class="p">[</span> <span class="n">supervisor</span><span class="p">(</span><span class="no">MyApp</span><span class="o">.</span><span class="no">Repo</span><span class="p">,</span> <span class="p">[]),</span> <span class="n">supervisor</span><span class="p">(</span><span class="no">MyApp</span><span class="o">.</span><span class="no">Endpoint</span><span class="p">,</span> <span class="p">[])</span> <span class="p">]</span> <span class="no">Supervisor</span><span class="o">.</span><span class="n">start_link</span><span class="p">(</span><span class="n">children</span><span class="p">,</span> <span class="ss">strategy:</span> <span class="ss">:one_for_one</span><span class="p">)</span> </code></pre></div></div> <p>In Elixir v1.5, one might do:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">children</span> <span class="o">=</span> <span class="p">[</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">Repo</span><span class="p">,</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">Endpoint</span> <span class="p">]</span> <span class="no">Supervisor</span><span class="o">.</span><span class="n">start_link</span><span class="p">(</span><span class="n">children</span><span class="p">,</span> <span class="ss">strategy:</span> <span class="ss">:one_for_one</span><span class="p">)</span> </code></pre></div></div> <p>The above works by calling the <code class="language-plaintext highlighter-rouge">child_spec/1</code> function on the given modules.</p> <p>This new approach allows <code class="language-plaintext highlighter-rouge">MyApp.Repo</code> and <code class="language-plaintext highlighter-rouge">MyApp.Endpoint</code> to control how they run under a supervisor. This reduces the chances of mistakes being made, such as starting an Ecto repository as a worker or forgetting to declare that tasks are temporary in a supervision tree.</p> <p>If it is necessary to configure any of the children, such can be done by passing a tuple instead of an atom:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">children</span> <span class="o">=</span> <span class="p">[</span> <span class="p">{</span><span class="no">MyApp</span><span class="o">.</span><span class="no">Repo</span><span class="p">,</span> <span class="ss">url:</span> <span class="s2">"ecto://localhost:4567/my_dev"</span><span class="p">},</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">Endpoint</span> <span class="p">]</span> </code></pre></div></div> <p>The modules <code class="language-plaintext highlighter-rouge">Agent</code>, <code class="language-plaintext highlighter-rouge">Registry</code>, <code class="language-plaintext highlighter-rouge">Task</code>, and <code class="language-plaintext highlighter-rouge">Task.Supervisor</code> have been updated to include a <code class="language-plaintext highlighter-rouge">child_spec/1</code> function, allowing them to be used directly in a supervision tree similar to the examples above. <code class="language-plaintext highlighter-rouge">use Agent</code>, <code class="language-plaintext highlighter-rouge">use GenServer</code>, <code class="language-plaintext highlighter-rouge">use Supervisor</code>, and <code class="language-plaintext highlighter-rouge">use Task</code> have also been updated to automatically define an overridable <code class="language-plaintext highlighter-rouge">child_spec/1</code> function.</p> <p>Finally, child specifications are now provided as maps (data-structures) instead of the previous <code class="language-plaintext highlighter-rouge">Supervisor.Spec.worker/3</code> and <code class="language-plaintext highlighter-rouge">Supervisor.Spec.supervisor/3</code> APIs. This behaviour also aligns with how supervisors are configured in Erlang/OTP 18+. See the updated <a href="https://hexdocs.pm/elixir/1.5/Supervisor.html"><code class="language-plaintext highlighter-rouge">Supervisor</code></a> docs for more information, as well as the new <code class="language-plaintext highlighter-rouge">Supervisor.init/2</code> and <code class="language-plaintext highlighter-rouge">Supervisor.child_spec/2</code> functions.</p> <h2 id="impl">@impl</h2> <p>This release also allows developers to mark which functions in a given module are an implementation of a callback. For example, when using the <a href="https://github.com/elixir-lang/plug">Plug</a> project, one needs to implement both <code class="language-plaintext highlighter-rouge">init/1</code> and <code class="language-plaintext highlighter-rouge">call/2</code> when writing a Plug:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">MyApp</span> <span class="k">do</span> <span class="nv">@behaviour</span> <span class="no">Plug</span> <span class="k">def</span> <span class="n">init</span><span class="p">(</span><span class="n">_opts</span><span class="p">)</span> <span class="k">do</span> <span class="n">opts</span> <span class="k">end</span> <span class="k">def</span> <span class="n">call</span><span class="p">(</span><span class="n">conn</span><span class="p">,</span> <span class="n">_opts</span><span class="p">)</span> <span class="k">do</span> <span class="no">Plug</span><span class="o">.</span><span class="no">Conn</span><span class="o">.</span><span class="n">send_resp</span><span class="p">(</span><span class="n">conn</span><span class="p">,</span> <span class="mi">200</span><span class="p">,</span> <span class="s2">"hello world"</span><span class="p">)</span> <span class="k">end</span> <span class="k">end</span> </code></pre></div></div> <p>The problem with the approach above is that, once more and more functions are added to the <code class="language-plaintext highlighter-rouge">MyApp</code> module, it becomes increasingly harder to know the purposes of the <code class="language-plaintext highlighter-rouge">init/1</code> and <code class="language-plaintext highlighter-rouge">call/2</code> functions. For example, for a developer unfamiliar with Plug, are those functions part of the <code class="language-plaintext highlighter-rouge">MyApp</code> API or are they implementations of a given callback?</p> <p>Elixir v1.5 introduces the <code class="language-plaintext highlighter-rouge">@impl</code> attribute, which allows us to mark that certain functions are implementation of callbacks:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">MyApp</span> <span class="k">do</span> <span class="nv">@behaviour</span> <span class="no">Plug</span> <span class="nv">@impl</span> <span class="no">true</span> <span class="k">def</span> <span class="n">init</span><span class="p">(</span><span class="n">_opts</span><span class="p">)</span> <span class="k">do</span> <span class="n">opts</span> <span class="k">end</span> <span class="nv">@impl</span> <span class="no">true</span> <span class="k">def</span> <span class="n">call</span><span class="p">(</span><span class="n">conn</span><span class="p">,</span> <span class="n">_opts</span><span class="p">)</span> <span class="k">do</span> <span class="no">Plug</span><span class="o">.</span><span class="no">Conn</span><span class="o">.</span><span class="n">send_resp</span><span class="p">(</span><span class="n">conn</span><span class="p">,</span> <span class="mi">200</span><span class="p">,</span> <span class="s2">"hello world"</span><span class="p">)</span> <span class="k">end</span> <span class="k">end</span> </code></pre></div></div> <p>You may even use <code class="language-plaintext highlighter-rouge">@impl Plug</code> if you want to explicitly document which behaviour defines the callback you are implementing.</p> <p>Overall, using <code class="language-plaintext highlighter-rouge">@impl</code> has the following advantages:</p> <ul> <li> <p>Readability of the code is increased, as it is now clear which functions are part of your API and which ones are callback implementations. To reinforce this idea, <code class="language-plaintext highlighter-rouge">@impl true</code> automatically marks the function as <code class="language-plaintext highlighter-rouge">@doc false</code>, disabling documentation unless <code class="language-plaintext highlighter-rouge">@doc</code> is explicitly set</p> </li> <li> <p>If you define <code class="language-plaintext highlighter-rouge">@impl</code> before a function that is not a callback, Elixir will error. This is useful in case of typos or in case the behaviour definition changes (such as a new major version of a library you depend on is released)</p> </li> <li> <p>If you use <code class="language-plaintext highlighter-rouge">@impl</code> in one implementation, Elixir will force you to declare <code class="language-plaintext highlighter-rouge">@impl</code> for all other implementations in the same module, keeping your modules consistent</p> </li> </ul> <h2 id="calendar-improvements">Calendar improvements</h2> <p><a href="https://elixir-lang.org/blog/2016/06/21/elixir-v1-3-0-released/">Elixir v1.3</a> introduced the Calendar module with the underlying <code class="language-plaintext highlighter-rouge">Time</code>, <code class="language-plaintext highlighter-rouge">Date</code>, <code class="language-plaintext highlighter-rouge">NaiveDateTime</code> and <code class="language-plaintext highlighter-rouge">Datetime</code> data types. We are glad to announce we consider the base Calendar API to be finished in Elixir v1.5. This release includes many enhancements, such as <code class="language-plaintext highlighter-rouge">Date.range/2</code> and the ability to convert between different calendars.</p> <h2 id="summing-up">Summing up</h2> <p>The full list of changes is available in our <a href="https://github.com/elixir-lang/elixir/releases/tag/v1.5.0">release notes</a>. There are many other exciting changes, such as compiler enhancements that reduces compilation times by 10%-15% on averages. When taken into account with the compiler improvements in Erlang/OTP 20 itself, some applications have seen gains up to 30% in compilation times.</p> <p>Don’t forget to check <a href="/install.html">the Install section</a> to get Elixir installed and our <a href="https://hexdocs.pm/elixir/introduction.html">Getting Started guide</a> to learn more.</p> José Valim 2017-01-05T00:00:00+00:00 /blog/2017/01/05/elixir-v1-4-0-released <p>Elixir v1.4 brings new features, enhancements and bug fixes. The most notable changes are the addition of the <code class="language-plaintext highlighter-rouge">Registry</code> module, the <code class="language-plaintext highlighter-rouge">Task.async_stream/3</code> and <code class="language-plaintext highlighter-rouge">Task.async_stream/5</code> function which aid developers in writing concurrent software, and the new application inference and commands added to Mix.</p> <p>In this post we will cover the main additions. The complete <a href="https://github.com/elixir-lang/elixir/releases/tag/v1.4.0">release notes</a> are also available.</p> <h2 id="registry">Registry</h2> <p>The <a href="https://hexdocs.pm/elixir/Registry.html"><code class="language-plaintext highlighter-rouge">Registry</code></a> is a new module in Elixir’s standard library that allows Elixir developers to implement patterns such as name lookups, code dispatching or even a pubsub system in a simple and scalable way.</p> <p>Broadly speaking, the Registry is a local, decentralized and scalable key-value process storage. Let’s break this in parts:</p> <ul> <li>Local because keys and values are only accessible to the current node (opposite to distributed)</li> <li>Decentralized because there is no single entity responsible for managing the registry</li> <li>Scalable because performance scales linearly with the addition of more cores upon partitioning</li> </ul> <p>A registry may have unique or duplicate keys. Every key-value pair is associated to the process registering the key. Keys are automatically removed once the owner process terminates. Starting, registering and looking up keys is quite straight-forward:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">iex</span><span class="o">&gt;</span> <span class="no">Registry</span><span class="o">.</span><span class="n">start_link</span><span class="p">(</span><span class="ss">:unique</span><span class="p">,</span> <span class="no">MyRegistry</span><span class="p">)</span> <span class="n">iex</span><span class="o">&gt;</span> <span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">_</span><span class="p">}</span> <span class="o">=</span> <span class="no">Registry</span><span class="o">.</span><span class="n">register</span><span class="p">(</span><span class="no">MyRegistry</span><span class="p">,</span> <span class="s2">"hello"</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span> <span class="n">iex</span><span class="o">&gt;</span> <span class="no">Registry</span><span class="o">.</span><span class="n">lookup</span><span class="p">(</span><span class="no">MyRegistry</span><span class="p">,</span> <span class="s2">"hello"</span><span class="p">)</span> <span class="p">[{</span><span class="n">self</span><span class="p">(),</span> <span class="mi">1</span><span class="p">}]</span> </code></pre></div></div> <p>Finally, huge thanks to <a href="https://twitter.com/voltonez">Bram Verburg</a> who has performed <a href="https://docs.google.com/spreadsheets/d/1MByRZJMCnZ1wPiLhBEnSRRSuy1QXp8kr27PIOXO3qqg/edit#gid=0">extensive benchmarks</a> on the registry to show it scales linearly with the number of cores by increasing the number of partitions.</p> <h2 id="syntax-coloring">Syntax coloring</h2> <p>Elixir v1.4 introduces the ability to syntax color inspected data structures and IEx automatically relies on this feature to provide syntax coloring for evaluated shell results:</p> <p><img src="/images/contents/iex-coloring.png" alt="IEx coloring" /></p> <p>This behaviour can be configured via the <code class="language-plaintext highlighter-rouge">:syntax_colors</code> coloring option:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="no">IEx</span><span class="o">.</span><span class="n">configure</span> <span class="p">[</span><span class="ss">colors:</span> <span class="p">[</span><span class="ss">syntax_colors:</span> <span class="p">[</span><span class="ss">atom:</span> <span class="ss">:cyan</span><span class="p">,</span> <span class="ss">string:</span> <span class="ss">:green</span><span class="p">]]]</span> </code></pre></div></div> <p>To disable coloring altogether, simply pass an empty list to <code class="language-plaintext highlighter-rouge">:syntax_colors</code>.</p> <h2 id="taskasync_stream">Task.async_stream</h2> <p>When there is a need to traverse a collection of items concurrently, Elixir developers often resort to tasks:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">collection</span> <span class="o">|&gt;</span> <span class="no">Enum</span><span class="o">.</span><span class="n">map</span><span class="p">(</span><span class="o">&amp;</span><span class="no">Task</span><span class="o">.</span><span class="n">async</span><span class="p">(</span><span class="no">SomeMod</span><span class="p">,</span> <span class="ss">:function</span><span class="p">,</span> <span class="p">[</span><span class="nv">&amp;1</span><span class="p">]))</span> <span class="o">|&gt;</span> <span class="no">Enum</span><span class="o">.</span><span class="n">map</span><span class="p">(</span><span class="o">&amp;</span><span class="no">Task</span><span class="o">.</span><span class="n">await</span><span class="o">/</span><span class="mi">1</span><span class="p">)</span> </code></pre></div></div> <p>The snippet above will spawn a new task by invoking <code class="language-plaintext highlighter-rouge">SomeMod.function(element)</code> for every element in the collection and then await for the task results.</p> <p>However, the snippet above will spawn and run concurrently as many tasks as there are items in the collection. While this may be fine in many occasions, including small collections, sometimes it is necessary to restrict amount of tasks running concurrently, specially when shared resources are involved.</p> <p>Elixir v1.4 adds <code class="language-plaintext highlighter-rouge">Task.async_stream/3</code> and <code class="language-plaintext highlighter-rouge">Task.async_stream/5</code> which brings some of the lessons we learned from <a href="/blog/2016/07/14/announcing-genstage/">the GenStage project</a> directly into Elixir:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">collection</span> <span class="o">|&gt;</span> <span class="no">Task</span><span class="o">.</span><span class="n">async_stream</span><span class="p">(</span><span class="no">SomeMod</span><span class="p">,</span> <span class="ss">:function</span><span class="p">,</span> <span class="p">[],</span> <span class="ss">max_concurrency:</span> <span class="mi">8</span><span class="p">)</span> <span class="o">|&gt;</span> <span class="no">Enum</span><span class="o">.</span><span class="n">to_list</span><span class="p">()</span> </code></pre></div></div> <p>The code above will also start the same <code class="language-plaintext highlighter-rouge">SomeMod.function(element)</code> task for every element in the collection except it will also guarantee we have at most 8 tasks being processed at the same time. You can use <code class="language-plaintext highlighter-rouge">System.schedulers_online</code> to retrieve the number of cores and balance the processing based on the amount of cores available.</p> <p>The <code class="language-plaintext highlighter-rouge">Task.async_stream</code> functions are also lazy, allowing developers to partially consume the stream until a condition is reached. Furthermore, <code class="language-plaintext highlighter-rouge">Task.Supervisor.async_stream/4</code> and <code class="language-plaintext highlighter-rouge">Task.Supervisor.async_stream/6</code> can be used to ensure the concurrent tasks are spawned under a given supervisor.</p> <h2 id="application-inference">Application inference</h2> <p>In previous Mix versions, most of your dependencies had to be added both to your dependencies list and applications list. Here is how a <code class="language-plaintext highlighter-rouge">mix.exs</code> would look like:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="n">application</span> <span class="k">do</span> <span class="p">[</span><span class="ss">applications:</span> <span class="p">[</span><span class="ss">:logger</span><span class="p">,</span> <span class="ss">:plug</span><span class="p">,</span> <span class="ss">:postgrex</span><span class="p">]]</span> <span class="k">end</span> <span class="k">def</span> <span class="n">deps</span> <span class="k">do</span> <span class="p">[{</span><span class="ss">:plug</span><span class="p">,</span> <span class="s2">"~&gt; 1.2"</span><span class="p">},</span> <span class="p">{</span><span class="ss">:postgrex</span><span class="p">,</span> <span class="s2">"~&gt; 1.0"</span><span class="p">}]</span> <span class="k">end</span> </code></pre></div></div> <p>This was a common source of confusion and quite error prone as many developers would not list their dependencies in the applications list.</p> <p>Mix v1.4 now automatically infers your applications list as long as you leave the <code class="language-plaintext highlighter-rouge">:applications</code> key empty. The <code class="language-plaintext highlighter-rouge">mix.exs</code> above can be rewritten to:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="n">application</span> <span class="k">do</span> <span class="p">[</span><span class="ss">extra_applications:</span> <span class="p">[</span><span class="ss">:logger</span><span class="p">]]</span> <span class="k">end</span> <span class="k">def</span> <span class="n">deps</span> <span class="k">do</span> <span class="p">[{</span><span class="ss">:plug</span><span class="p">,</span> <span class="s2">"~&gt; 1.2"</span><span class="p">},</span> <span class="p">{</span><span class="ss">:postgrex</span><span class="p">,</span> <span class="s2">"~&gt; 1.0"</span><span class="p">}]</span> <span class="k">end</span> </code></pre></div></div> <p>With the above, Mix will automatically build your application list based on your dependencies. Developers now only need to specify which applications shipped as part of Erlang or Elixir that they require, such as <code class="language-plaintext highlighter-rouge">:logger</code>.</p> <p>Finally, if there is a dependency you don’t want to include in the application runtime list, you can do so by specifying the <code class="language-plaintext highlighter-rouge">runtime: false</code> option:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>{:distillery, "&gt; 0.0.0", runtime: false} </code></pre></div></div> <p>We hope this feature provides a more streamlined workflow for developers who are building releases for their Elixir projects.</p> <h2 id="mix-install-from-scm">Mix install from SCM</h2> <p>Mix v1.4 can now install escripts and archives from both Git and Hex, providing you with even more options for distributing Elixir code.</p> <p>This makes it possible to distribute CLI applications written in Elixir by publishing a package which builds an escript to Hex. <a href="https://hex.pm/packages/ex_doc"><code class="language-plaintext highlighter-rouge">ex_doc</code></a> has been updated to serve as an example of how to use this new functionality.</p> <p>Simply running:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mix escript.install hex ex_doc </code></pre></div></div> <p>will fetch <code class="language-plaintext highlighter-rouge">ex_doc</code> and its dependencies, build them, and then install <code class="language-plaintext highlighter-rouge">ex_doc</code> to <code class="language-plaintext highlighter-rouge">~/.mix/escripts</code> (by default). After adding <code class="language-plaintext highlighter-rouge">~/.mix/escripts</code> to your <code class="language-plaintext highlighter-rouge">PATH</code>, running <code class="language-plaintext highlighter-rouge">ex_doc</code> is as simple as:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ex_doc </code></pre></div></div> <p>You can now also install archives from Hex in this way. Since they are fetched and built on the user’s machine, they do not have the same limitations as pre-built archives. However, keep in mind archives are loaded on every Mix command and may conflict with modules or dependencies in your projects. For this reason, escripts is the preferred format for sharing executables.</p> <p>It is also possible to install escripts and archives by providing a Git/GitHub repo. See <code class="language-plaintext highlighter-rouge">mix help escript.install</code> and <code class="language-plaintext highlighter-rouge">mix help archive.install</code> for more details.</p> <h2 id="summing-up">Summing up</h2> <p>The full list of changes is available in our <a href="https://github.com/elixir-lang/elixir/releases/tag/v1.4.0">release notes</a>. Don’t forget to check <a href="/install.html">the Install section</a> to get Elixir installed and our <a href="https://hexdocs.pm/elixir/introduction.html">Getting Started guide</a> to learn more.</p> <p>Happy coding!</p> José Valim 2016-07-14T00:00:00+00:00 /blog/2016/07/14/announcing-genstage <p>Today we are glad to announce the official release of GenStage. GenStage is a new Elixir behaviour for exchanging events with back-pressure between Elixir processes. In the short-term, we expect GenStage to replace the use cases for GenEvent as well as providing a composable abstraction for consuming data from third-party systems.</p> <p>In this blog post we will cover the background that led us to GenStage, some example use cases, and what we are exploring for future releases. If instead you are looking for a quick reference, <a href="https://github.com/elixir-lang/gen_stage">check the project source code</a> and <a href="https://hexdocs.pm/gen_stage/Experimental.GenStage.html">access its documentation</a>.</p> <h2 id="background">Background</h2> <p>One of the original motivations for <a href="https://www.youtube.com/watch?v=Lqo9-pQuRKE">creating and designing Elixir was to introduce better abstractions for working with collections</a>. Not only that, we want to provide developers interested in manipulating collections with a path to take their code from eager to lazy, to concurrent and then distributed.</p> <p>Let’s discuss a simple but actual example: word counting. The idea of word counting is to receive one file and count how many times each word appears in the document. Using the <code class="language-plaintext highlighter-rouge">Enum</code> module it could be implemented as follows:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="no">File</span><span class="o">.</span><span class="n">read!</span><span class="p">(</span><span class="s2">"path/to/some/file"</span><span class="p">)</span> <span class="o">|&gt;</span> <span class="no">String</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s2">"</span><span class="se">\n</span><span class="s2">"</span><span class="p">)</span> <span class="o">|&gt;</span> <span class="no">Enum</span><span class="o">.</span><span class="n">flat_map</span><span class="p">(</span><span class="k">fn</span> <span class="n">line</span> <span class="o">-&gt;</span> <span class="no">String</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="n">line</span><span class="p">,</span> <span class="s2">" "</span><span class="p">)</span> <span class="k">end</span><span class="p">)</span> <span class="o">|&gt;</span> <span class="no">Enum</span><span class="o">.</span><span class="n">reduce</span><span class="p">(%{},</span> <span class="k">fn</span> <span class="n">word</span><span class="p">,</span> <span class="n">acc</span> <span class="o">-&gt;</span> <span class="no">Map</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="n">acc</span><span class="p">,</span> <span class="n">word</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="o">&amp;</span> <span class="nv">&amp;1</span> <span class="o">+</span> <span class="mi">1</span><span class="p">)</span> <span class="k">end</span><span class="p">)</span> <span class="o">|&gt;</span> <span class="no">Enum</span><span class="o">.</span><span class="n">to_list</span><span class="p">()</span> </code></pre></div></div> <p>While the solution above works fine and is efficient for small files, it is quite restrictive for large inputs as it loads the whole file into memory.</p> <p>Another issue with the solution above is that the <code class="language-plaintext highlighter-rouge">Enum.flat_map/2</code> step will build a huge list, with all the words in the file, before we effectively start counting them. Again, for a large document, this means more memory usage and a waste of processing time in building a list that will be traversed right after.</p> <p>Luckily, Elixir provides a solution to this problem (and has provided it for quite some time): streams. One of the advantage of streams is they are lazy, allowing us to traverse collections item by item, in this case, line by line, instead of loading the whole data set into memory. Let’s rewrite the example above to use streams:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="no">File</span><span class="o">.</span><span class="n">stream!</span><span class="p">(</span><span class="s2">"path/to/some/file"</span><span class="p">)</span> <span class="o">|&gt;</span> <span class="no">Stream</span><span class="o">.</span><span class="n">flat_map</span><span class="p">(</span><span class="k">fn</span> <span class="n">line</span> <span class="o">-&gt;</span> <span class="no">String</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="n">line</span><span class="p">,</span> <span class="s2">" "</span><span class="p">)</span> <span class="k">end</span><span class="p">)</span> <span class="o">|&gt;</span> <span class="no">Enum</span><span class="o">.</span><span class="n">reduce</span><span class="p">(%{},</span> <span class="k">fn</span> <span class="n">word</span><span class="p">,</span> <span class="n">acc</span> <span class="o">-&gt;</span> <span class="no">Map</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="n">acc</span><span class="p">,</span> <span class="n">word</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="o">&amp;</span> <span class="nv">&amp;1</span> <span class="o">+</span> <span class="mi">1</span><span class="p">)</span> <span class="k">end</span><span class="p">)</span> <span class="o">|&gt;</span> <span class="no">Enum</span><span class="o">.</span><span class="n">to_list</span><span class="p">()</span> </code></pre></div></div> <p>By using <code class="language-plaintext highlighter-rouge">File.stream!</code> and <code class="language-plaintext highlighter-rouge">Stream.flat_map</code>, we build a lazy computation that will emit a single line, break that line into words, and emit such words one by one without building huge lists in memory when enumerated. The functions in the <a href="https://hexdocs.pm/elixir/Stream.html">Stream module</a> just express the computation we want to perform. The computation itself, like traversing the file or breaking into words in <code class="language-plaintext highlighter-rouge">flat_map</code>, only happens when we call a function in the <code class="language-plaintext highlighter-rouge">Enum</code> module. We have covered <a href="https://dashbit.co/blog/introducing-reducees">the foundation for Enum and Streams</a> in another article.</p> <p>The solution above allows us to work with large datasets without loading them all into memory. For large files, it is going to provide much better performance than the eager version. However, the solution above still does not leverage concurrency. For a machine with more than one core, which is the huge majority of machines we have available today, it is a suboptimal solution.</p> <p>That said, how could we leverage concurrency in the example above?</p> <p>During my ElixirConf 2015 keynote, <a href="http://confreaks.tv/videos/elixirconf2015-keynote">I discussed one of the most immediate solutions to this problem</a> which was to convert parts of your pipeline to separate processes:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="no">File</span><span class="o">.</span><span class="n">stream!</span><span class="p">(</span><span class="s2">"path/to/some/file"</span><span class="p">)</span> <span class="o">|&gt;</span> <span class="no">Stream</span><span class="o">.</span><span class="n">flat_map</span><span class="p">(</span><span class="k">fn</span> <span class="n">line</span> <span class="o">-&gt;</span> <span class="no">String</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="n">line</span><span class="p">,</span> <span class="s2">" "</span><span class="p">)</span> <span class="k">end</span><span class="p">)</span> <span class="o">|&gt;</span> <span class="no">Stream</span><span class="o">.</span><span class="n">async</span><span class="p">()</span> <span class="c1"># NEW!</span> <span class="o">|&gt;</span> <span class="no">Enum</span><span class="o">.</span><span class="n">reduce</span><span class="p">(%{},</span> <span class="k">fn</span> <span class="n">word</span><span class="p">,</span> <span class="n">acc</span> <span class="o">-&gt;</span> <span class="no">Map</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="n">acc</span><span class="p">,</span> <span class="n">word</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="o">&amp;</span> <span class="nv">&amp;1</span> <span class="o">+</span> <span class="mi">1</span><span class="p">)</span> <span class="k">end</span><span class="p">)</span> <span class="o">|&gt;</span> <span class="no">Enum</span><span class="o">.</span><span class="n">to_list</span><span class="p">()</span> </code></pre></div></div> <p>The idea is that <code class="language-plaintext highlighter-rouge">Stream.async</code> would run the previous computations in a separate process that would stream its messages to the process that called <code class="language-plaintext highlighter-rouge">Enum.reduce</code>. Unfortunately, the solution above is less than ideal.</p> <p>First of all, we want to avoid moving data between processes as much as possible. Instead, we want to start multiple processes that perform the same computation in parallel. Not only that, if we are requiring developers to place <code class="language-plaintext highlighter-rouge">Stream.async</code> manually, it may lead to inefficient and error prone solutions.</p> <p>Although the solution above has many flaws, it has helped us ask the right questions:</p> <ul> <li> <p>If <code class="language-plaintext highlighter-rouge">Stream.async</code> is introducing new processes, how can we guarantee those processes are supervised?</p> </li> <li> <p>Since we are exchanging messages between processes, how do we prevent a process from receiving too many messages? We need a back-pressure mechanism that allows the receiving process to specify how much it can handle from the sending process.</p> </li> </ul> <p>We have jumped through different abstractions trying to answer those questions until we have finally settled on GenStage.</p> <h2 id="genstage">GenStage</h2> <p>GenStage is a new Elixir behaviour for exchanging events with back-pressure between Elixir processes. Developers who use GenStage only need to worry about how the data is produced, manipulated and consumed. The act of dispatching the data and providing back-pressure is completely abstracted away from the developers.</p> <p>As a quick example, let’s write a simple pipeline that will produce events as increasing numbers, multiply those numbers by two, and then print them to the terminal. We will do so by implementing three stages, the <code class="language-plaintext highlighter-rouge">:producer</code>, the <code class="language-plaintext highlighter-rouge">:producer_consumer</code> and the <code class="language-plaintext highlighter-rouge">:consumer</code>, which we will call <code class="language-plaintext highlighter-rouge">A</code>, <code class="language-plaintext highlighter-rouge">B</code> and <code class="language-plaintext highlighter-rouge">C</code> respectively. We will go back to the word counting example at the end of this post.</p> <p>Let’s start with the producer that we will call <code class="language-plaintext highlighter-rouge">A</code>. Since <code class="language-plaintext highlighter-rouge">A</code> is a producer, its main responsibility is to receive demand, which is the number of events the consumer is willing to handle, and generate events. Those events may be in memory or an external data source. For now let’s implement a simple counter starting from a given value of <code class="language-plaintext highlighter-rouge">counter</code> received on <code class="language-plaintext highlighter-rouge">init/1</code>:</p> <p>Note: all of the modules in the <code class="language-plaintext highlighter-rouge">GenStage</code> project are prefixed with the <code class="language-plaintext highlighter-rouge">Experimental</code> namespace. That’s why the examples below and your code should <code class="language-plaintext highlighter-rouge">alias Experimental.GenStage</code> at the top of your files.</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">alias</span> <span class="no">Experimental</span><span class="o">.</span><span class="no">GenStage</span> <span class="k">defmodule</span> <span class="no">A</span> <span class="k">do</span> <span class="kn">use</span> <span class="no">GenStage</span> <span class="k">def</span> <span class="n">init</span><span class="p">(</span><span class="n">counter</span><span class="p">)</span> <span class="k">do</span> <span class="p">{</span><span class="ss">:producer</span><span class="p">,</span> <span class="n">counter</span><span class="p">}</span> <span class="k">end</span> <span class="k">def</span> <span class="n">handle_demand</span><span class="p">(</span><span class="n">demand</span><span class="p">,</span> <span class="n">counter</span><span class="p">)</span> <span class="ow">when</span> <span class="n">demand</span> <span class="o">&gt;</span> <span class="mi">0</span> <span class="k">do</span> <span class="c1"># If the counter is 3 and we ask for 2 items, we will</span> <span class="c1"># emit the items 3 and 4, and set the state to 5.</span> <span class="n">events</span> <span class="o">=</span> <span class="no">Enum</span><span class="o">.</span><span class="n">to_list</span><span class="p">(</span><span class="n">counter</span><span class="o">..</span><span class="n">counter</span><span class="o">+</span><span class="n">demand</span><span class="o">-</span><span class="mi">1</span><span class="p">)</span> <span class="c1"># The events to emit is the second element of the tuple,</span> <span class="c1"># the third being the state.</span> <span class="p">{</span><span class="ss">:noreply</span><span class="p">,</span> <span class="n">events</span><span class="p">,</span> <span class="n">counter</span> <span class="o">+</span> <span class="n">demand</span><span class="p">}</span> <span class="k">end</span> <span class="k">end</span> </code></pre></div></div> <p><code class="language-plaintext highlighter-rouge">B</code> is a producer-consumer. This means it does not explicitly handle the demand because the demand is always forwarded to its producers. Once <code class="language-plaintext highlighter-rouge">A</code> receives the demand from <code class="language-plaintext highlighter-rouge">B</code>, it will send events to <code class="language-plaintext highlighter-rouge">B</code> which will be transformed by <code class="language-plaintext highlighter-rouge">B</code> as desired and then sent to <code class="language-plaintext highlighter-rouge">C</code>. In our case, B will receive events and multiply them by a number given on initialization and stored as the state:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">alias</span> <span class="no">Experimental</span><span class="o">.</span><span class="no">GenStage</span> <span class="k">defmodule</span> <span class="no">B</span> <span class="k">do</span> <span class="kn">use</span> <span class="no">GenStage</span> <span class="k">def</span> <span class="n">init</span><span class="p">(</span><span class="n">number</span><span class="p">)</span> <span class="k">do</span> <span class="p">{</span><span class="ss">:producer_consumer</span><span class="p">,</span> <span class="n">number</span><span class="p">}</span> <span class="k">end</span> <span class="k">def</span> <span class="n">handle_events</span><span class="p">(</span><span class="n">events</span><span class="p">,</span> <span class="n">_from</span><span class="p">,</span> <span class="n">number</span><span class="p">)</span> <span class="k">do</span> <span class="n">events</span> <span class="o">=</span> <span class="no">Enum</span><span class="o">.</span><span class="n">map</span><span class="p">(</span><span class="n">events</span><span class="p">,</span> <span class="o">&amp;</span> <span class="nv">&amp;1</span> <span class="o">*</span> <span class="n">number</span><span class="p">)</span> <span class="p">{</span><span class="ss">:noreply</span><span class="p">,</span> <span class="n">events</span><span class="p">,</span> <span class="n">number</span><span class="p">}</span> <span class="k">end</span> <span class="k">end</span> </code></pre></div></div> <p><code class="language-plaintext highlighter-rouge">C</code> is the consumer which will finally receive those events and print them every second to the terminal:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">alias</span> <span class="no">Experimental</span><span class="o">.</span><span class="no">GenStage</span> <span class="k">defmodule</span> <span class="no">C</span> <span class="k">do</span> <span class="kn">use</span> <span class="no">GenStage</span> <span class="k">def</span> <span class="n">init</span><span class="p">(</span><span class="n">sleeping_time</span><span class="p">)</span> <span class="k">do</span> <span class="p">{</span><span class="ss">:consumer</span><span class="p">,</span> <span class="n">sleeping_time</span><span class="p">}</span> <span class="k">end</span> <span class="k">def</span> <span class="n">handle_events</span><span class="p">(</span><span class="n">events</span><span class="p">,</span> <span class="n">_from</span><span class="p">,</span> <span class="n">sleeping_time</span><span class="p">)</span> <span class="k">do</span> <span class="c1"># Print events to terminal.</span> <span class="no">IO</span><span class="o">.</span><span class="n">inspect</span><span class="p">(</span><span class="n">events</span><span class="p">)</span> <span class="c1"># Sleep the configured time.</span> <span class="no">Process</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="n">sleeping_time</span><span class="p">)</span> <span class="c1"># We are a consumer, so we never emit events.</span> <span class="p">{</span><span class="ss">:noreply</span><span class="p">,</span> <span class="p">[],</span> <span class="n">sleeping_time</span><span class="p">}</span> <span class="k">end</span> <span class="k">end</span> </code></pre></div></div> <p>With the stages defined, we can start and connect them:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">a</span><span class="p">}</span> <span class="o">=</span> <span class="no">GenStage</span><span class="o">.</span><span class="n">start_link</span><span class="p">(</span><span class="no">A</span><span class="p">,</span> <span class="mi">0</span><span class="p">)</span> <span class="c1"># starting from zero</span> <span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">b</span><span class="p">}</span> <span class="o">=</span> <span class="no">GenStage</span><span class="o">.</span><span class="n">start_link</span><span class="p">(</span><span class="no">B</span><span class="p">,</span> <span class="mi">2</span><span class="p">)</span> <span class="c1"># multiply by 2</span> <span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">c</span><span class="p">}</span> <span class="o">=</span> <span class="no">GenStage</span><span class="o">.</span><span class="n">start_link</span><span class="p">(</span><span class="no">C</span><span class="p">,</span> <span class="mi">1000</span><span class="p">)</span> <span class="c1"># sleep for a second</span> <span class="no">GenStage</span><span class="o">.</span><span class="n">sync_subscribe</span><span class="p">(</span><span class="n">c</span><span class="p">,</span> <span class="ss">to:</span> <span class="n">b</span><span class="p">)</span> <span class="no">GenStage</span><span class="o">.</span><span class="n">sync_subscribe</span><span class="p">(</span><span class="n">b</span><span class="p">,</span> <span class="ss">to:</span> <span class="n">a</span><span class="p">)</span> <span class="c1"># Sleep so we see events printed.</span> <span class="no">Process</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="ss">:infinity</span><span class="p">)</span> </code></pre></div></div> <p>As soon as we subscribe the stages, we should see items being printed to the terminal. Notice that, even though we have introduced a sleep command to the consumer, the producers will never overflow the consumer with data. That’s because the communication between stages is demand-driven. The producer can only send items to consumers after the consumers have sent demand upstream. The producer must never send more items than the consumer has specified.</p> <p>One consequence of this design decision is that parallelizing stateless stages like the consumer above is really straightforward:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">a</span><span class="p">}</span> <span class="o">=</span> <span class="no">GenStage</span><span class="o">.</span><span class="n">start_link</span><span class="p">(</span><span class="no">A</span><span class="p">,</span> <span class="mi">0</span><span class="p">)</span> <span class="c1"># starting from zero</span> <span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">b</span><span class="p">}</span> <span class="o">=</span> <span class="no">GenStage</span><span class="o">.</span><span class="n">start_link</span><span class="p">(</span><span class="no">B</span><span class="p">,</span> <span class="mi">2</span><span class="p">)</span> <span class="c1"># multiply by 2</span> <span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">c1</span><span class="p">}</span> <span class="o">=</span> <span class="no">GenStage</span><span class="o">.</span><span class="n">start_link</span><span class="p">(</span><span class="no">C</span><span class="p">,</span> <span class="mi">1000</span><span class="p">)</span> <span class="c1"># sleep for a second</span> <span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">c2</span><span class="p">}</span> <span class="o">=</span> <span class="no">GenStage</span><span class="o">.</span><span class="n">start_link</span><span class="p">(</span><span class="no">C</span><span class="p">,</span> <span class="mi">1000</span><span class="p">)</span> <span class="c1"># sleep for a second</span> <span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">c3</span><span class="p">}</span> <span class="o">=</span> <span class="no">GenStage</span><span class="o">.</span><span class="n">start_link</span><span class="p">(</span><span class="no">C</span><span class="p">,</span> <span class="mi">1000</span><span class="p">)</span> <span class="c1"># sleep for a second</span> <span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">c4</span><span class="p">}</span> <span class="o">=</span> <span class="no">GenStage</span><span class="o">.</span><span class="n">start_link</span><span class="p">(</span><span class="no">C</span><span class="p">,</span> <span class="mi">1000</span><span class="p">)</span> <span class="c1"># sleep for a second</span> <span class="no">GenStage</span><span class="o">.</span><span class="n">sync_subscribe</span><span class="p">(</span><span class="n">c1</span><span class="p">,</span> <span class="ss">to:</span> <span class="n">b</span><span class="p">)</span> <span class="no">GenStage</span><span class="o">.</span><span class="n">sync_subscribe</span><span class="p">(</span><span class="n">c2</span><span class="p">,</span> <span class="ss">to:</span> <span class="n">b</span><span class="p">)</span> <span class="no">GenStage</span><span class="o">.</span><span class="n">sync_subscribe</span><span class="p">(</span><span class="n">c3</span><span class="p">,</span> <span class="ss">to:</span> <span class="n">b</span><span class="p">)</span> <span class="no">GenStage</span><span class="o">.</span><span class="n">sync_subscribe</span><span class="p">(</span><span class="n">c4</span><span class="p">,</span> <span class="ss">to:</span> <span class="n">b</span><span class="p">)</span> <span class="no">GenStage</span><span class="o">.</span><span class="n">sync_subscribe</span><span class="p">(</span><span class="n">b</span><span class="p">,</span> <span class="ss">to:</span> <span class="n">a</span><span class="p">)</span> <span class="c1"># Sleep so we see events printed.</span> <span class="no">Process</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="ss">:infinity</span><span class="p">)</span> </code></pre></div></div> <p>By simply starting multiple consumers, the stage <code class="language-plaintext highlighter-rouge">B</code> will now receive demand from multiple stages and dispatch events to those stages which are now running concurrently, always picking the stage that is able to process more items. We can also leverage concurrency from the opposite direction: if the producer is the slow stage in a pipeline, you can start multiple producers and have each consumer subscribe to them.</p> <p>In order to know which consumer should receive a particular event, producer stages depend on a behaviour called <a href="https://hexdocs.pm/gen_stage/Experimental.GenStage.Dispatcher.html"><code class="language-plaintext highlighter-rouge">GenStage.Dispatcher</code></a>. The default dispatcher is the <code class="language-plaintext highlighter-rouge">GenStage.DemandDispatcher</code> we have briefly described above: it will collect the demand from different consumers and dispatch to the one with highest demand. This means if one consumer is slow, maybe because we increased its sleeping time to 10 seconds, it will receive less items.</p> <h3 id="genstage-for-data-ingestion">GenStage for data-ingestion</h3> <p>One of the use cases for GenStage is to consume data from third-party systems. The demand system with back-pressure guarantees we won’t import more data than we can effectively handle. The demand dispatcher allows us to easily leverage concurrency when processing the data by simply adding more consumers.</p> <p>During the Elixir London Meetup, I have live-coded a short example that shows how to use <code class="language-plaintext highlighter-rouge">GenStage</code> to concurrently process data stored in a PostgreSQL database as a queue:</p> <iframe width="560" height="315" src="https://www.youtube.com/embed/aZuY5-2lwW4" class="video" allowfullscreen="" title="Elixir London June 2016 w/ José Valim"></iframe> <h3 id="genstage-for-event-dispatching">GenStage for event dispatching</h3> <p>Another scenario where GenStage can be useful today is to replace cases where developers would have used <a href="https://hexdocs.pm/elixir/GenEvent.html">GenEvent</a> in the past. For those unfamiliar with GenEvent, it is a behaviour where events are sent to an “event manager” which then proceeds to invoke “event handlers” for each event. GenEvent, however, has one big flaw: the event manager and all event handlers run in the same process. This means GenEvent handlers cannot easily leverage concurrency without forcing developers to implement those mechanisms themselves. Furthermore, GenEvent handlers have very awkward error semantics. Because event handlers are not separate processes, we cannot simply rely on supervisors restarting them.</p> <p>GenStage solves those problems by having a producer as the event manager. The producer itself should be configured to use <a href="https://hexdocs.pm/gen_stage/Experimental.GenStage.BroadcastDispatcher.html"><code class="language-plaintext highlighter-rouge">GenStage.BroadcastDispatcher</code></a> as its dispatcher. The broadcast dispatcher will guarantee events are dispatched to all consumers in a way that does not exceed the demand of any of the consumers. This allows us to leverage concurrency and having the “event manager” as a producer gives us much more flexibility in terms of buffering and reacting to failures.</p> <p>Let’s see an example of building an event manager as a producer:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">alias</span> <span class="no">Experimental</span><span class="o">.</span><span class="no">GenStage</span> <span class="k">defmodule</span> <span class="no">EventManager</span> <span class="k">do</span> <span class="kn">use</span> <span class="no">GenStage</span> <span class="nv">@doc</span> <span class="sd">""" Starts the manager. """</span> <span class="k">def</span> <span class="n">start_link</span><span class="p">()</span> <span class="k">do</span> <span class="no">GenStage</span><span class="o">.</span><span class="n">start_link</span><span class="p">(</span><span class="bp">__MODULE__</span><span class="p">,</span> <span class="ss">:ok</span><span class="p">,</span> <span class="ss">name:</span> <span class="bp">__MODULE__</span><span class="p">)</span> <span class="k">end</span> <span class="nv">@doc</span> <span class="sd">""" Sends an event and returns only after the event is dispatched. """</span> <span class="k">def</span> <span class="n">sync_notify</span><span class="p">(</span><span class="n">event</span><span class="p">,</span> <span class="n">timeout</span> <span class="p">\\</span> <span class="mi">5000</span><span class="p">)</span> <span class="k">do</span> <span class="no">GenStage</span><span class="o">.</span><span class="n">call</span><span class="p">(</span><span class="bp">__MODULE__</span><span class="p">,</span> <span class="p">{</span><span class="ss">:notify</span><span class="p">,</span> <span class="n">event</span><span class="p">},</span> <span class="n">timeout</span><span class="p">)</span> <span class="k">end</span> <span class="c1">## Callbacks</span> <span class="k">def</span> <span class="n">init</span><span class="p">(</span><span class="ss">:ok</span><span class="p">)</span> <span class="k">do</span> <span class="p">{</span><span class="ss">:producer</span><span class="p">,</span> <span class="p">{</span><span class="ss">:queue</span><span class="o">.</span><span class="n">new</span><span class="p">,</span> <span class="mi">0</span><span class="p">},</span> <span class="ss">dispatcher:</span> <span class="no">GenStage</span><span class="o">.</span><span class="no">BroadcastDispatcher</span><span class="p">}</span> <span class="k">end</span> <span class="k">def</span> <span class="n">handle_call</span><span class="p">({</span><span class="ss">:notify</span><span class="p">,</span> <span class="n">event</span><span class="p">},</span> <span class="n">from</span><span class="p">,</span> <span class="p">{</span><span class="n">queue</span><span class="p">,</span> <span class="n">demand</span><span class="p">})</span> <span class="k">do</span> <span class="n">dispatch_events</span><span class="p">(</span><span class="ss">:queue</span><span class="o">.</span><span class="ow">in</span><span class="p">({</span><span class="n">from</span><span class="p">,</span> <span class="n">event</span><span class="p">},</span> <span class="n">queue</span><span class="p">),</span> <span class="n">demand</span><span class="p">,</span> <span class="p">[])</span> <span class="k">end</span> <span class="k">def</span> <span class="n">handle_demand</span><span class="p">(</span><span class="n">incoming_demand</span><span class="p">,</span> <span class="p">{</span><span class="n">queue</span><span class="p">,</span> <span class="n">demand</span><span class="p">})</span> <span class="k">do</span> <span class="n">dispatch_events</span><span class="p">(</span><span class="n">queue</span><span class="p">,</span> <span class="n">incoming_demand</span> <span class="o">+</span> <span class="n">demand</span><span class="p">,</span> <span class="p">[])</span> <span class="k">end</span> <span class="k">defp</span> <span class="n">dispatch_events</span><span class="p">(</span><span class="n">queue</span><span class="p">,</span> <span class="n">demand</span><span class="p">,</span> <span class="n">events</span><span class="p">)</span> <span class="k">do</span> <span class="n">with</span> <span class="n">d</span> <span class="ow">when</span> <span class="n">d</span> <span class="o">&gt;</span> <span class="mi">0</span> <span class="o">&lt;-</span> <span class="n">demand</span><span class="p">,</span> <span class="p">{</span><span class="n">item</span><span class="p">,</span> <span class="n">queue</span><span class="p">}</span> <span class="o">=</span> <span class="ss">:queue</span><span class="o">.</span><span class="n">out</span><span class="p">(</span><span class="n">queue</span><span class="p">),</span> <span class="p">{</span><span class="ss">:value</span><span class="p">,</span> <span class="p">{</span><span class="n">from</span><span class="p">,</span> <span class="n">event</span><span class="p">}}</span> <span class="o">&lt;-</span> <span class="n">item</span> <span class="k">do</span> <span class="no">GenStage</span><span class="o">.</span><span class="n">reply</span><span class="p">(</span><span class="n">from</span><span class="p">,</span> <span class="ss">:ok</span><span class="p">)</span> <span class="n">dispatch_events</span><span class="p">(</span><span class="n">queue</span><span class="p">,</span> <span class="n">demand</span> <span class="o">-</span> <span class="mi">1</span><span class="p">,</span> <span class="p">[</span><span class="n">event</span> <span class="o">|</span> <span class="n">events</span><span class="p">])</span> <span class="k">else</span> <span class="n">_</span> <span class="o">-&gt;</span> <span class="p">{</span><span class="ss">:noreply</span><span class="p">,</span> <span class="no">Enum</span><span class="o">.</span><span class="n">reverse</span><span class="p">(</span><span class="n">events</span><span class="p">),</span> <span class="p">{</span><span class="n">queue</span><span class="p">,</span> <span class="n">demand</span><span class="p">}}</span> <span class="k">end</span> <span class="k">end</span> <span class="k">end</span> </code></pre></div></div> <p>The <code class="language-plaintext highlighter-rouge">EventManager</code> works as a buffer. If there is demand but not events to be sent, we store such demand. If there are events but no demand, we store such events in a queue. If a client tries to broadcast an event, the <code class="language-plaintext highlighter-rouge">sync_notify</code> call will block until the event is effectively broadcasted. The bulk of the logic is in the <code class="language-plaintext highlighter-rouge">dispatch_events/3</code> function that takes events from the queue while there is demand.</p> <p>By implementing the event manager as a producer, we can configure all sorts of behaviours that are simply not possible with <code class="language-plaintext highlighter-rouge">GenEvent</code>, such as how much data we want to queue (or for how long) and if events should be buffered or not when there are no consumers (via the <code class="language-plaintext highlighter-rouge">handle_subscribe/4</code> and <code class="language-plaintext highlighter-rouge">handle_cancel/3</code> callbacks).</p> <p>Implementing event handlers is as straightforward as writing any other consumer. We could in fact use the <code class="language-plaintext highlighter-rouge">C</code> consumer implemented earlier. However, given event managers are often defined before the handlers, it is recommended for handlers to subscribe to managers when they start:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">alias</span> <span class="no">Experimental</span><span class="o">.</span><span class="no">GenStage</span> <span class="k">defmodule</span> <span class="no">EventHandler</span> <span class="k">do</span> <span class="kn">use</span> <span class="no">GenStage</span> <span class="k">def</span> <span class="n">start_link</span><span class="p">()</span> <span class="k">do</span> <span class="no">GenStage</span><span class="o">.</span><span class="n">start_link</span><span class="p">(</span><span class="bp">__MODULE__</span><span class="p">,</span> <span class="ss">:ok</span><span class="p">)</span> <span class="k">end</span> <span class="c1"># Callbacks</span> <span class="k">def</span> <span class="n">init</span><span class="p">(</span><span class="ss">:ok</span><span class="p">)</span> <span class="k">do</span> <span class="c1"># Starts a permanent subscription to the broadcaster</span> <span class="c1"># which will automatically start requesting items.</span> <span class="p">{</span><span class="ss">:consumer</span><span class="p">,</span> <span class="ss">:ok</span><span class="p">,</span> <span class="ss">subscribe_to:</span> <span class="p">[</span><span class="no">EventManager</span><span class="p">]}</span> <span class="k">end</span> <span class="k">def</span> <span class="n">handle_events</span><span class="p">(</span><span class="n">events</span><span class="p">,</span> <span class="n">_from</span><span class="p">,</span> <span class="n">state</span><span class="p">)</span> <span class="k">do</span> <span class="no">IO</span><span class="o">.</span><span class="n">inspect</span> <span class="n">events</span> <span class="p">{</span><span class="ss">:noreply</span><span class="p">,</span> <span class="p">[],</span> <span class="n">state</span><span class="p">}</span> <span class="k">end</span> <span class="k">end</span> </code></pre></div></div> <p>Such guarantees that, if a supervised <code class="language-plaintext highlighter-rouge">EventHandler</code> crashes, the supervisor will start a new event handler which will promptly subscribe to the same manager, solving the awkward error handling semantics we have seen with <code class="language-plaintext highlighter-rouge">GenEvent</code>.</p> <h2 id="the-path-forward">The path forward</h2> <p>With the release of GenStage v0.3.0, we have reached an important milestone as <code class="language-plaintext highlighter-rouge">GenStage</code> can be used as both event managers and a way to exchange events between processes, often external data sources, with back-pressure.</p> <p>The v0.3.0 release also includes the <a href="https://hexdocs.pm/gen_stage/Experimental.GenStage.html#stream/1"><code class="language-plaintext highlighter-rouge">GenStage.stream</code></a> function, which allows us to consume data from a GenStage as a stream, and <a href="https://hexdocs.pm/gen_stage/Experimental.GenStage.html#from_enumerable/2"><code class="language-plaintext highlighter-rouge">GenStage.from_enumerable</code></a> which allows us to use an enumerable or a stream, like <code class="language-plaintext highlighter-rouge">File.stream!</code>, as a producer. Closing the gap between stages and streams.</p> <p>However, we are far from done!</p> <p>First of all, now is the moment for the community to step in and try GenStage out. If you have used GenEvent in the past, can it be replaced by a GenStage? Similarly, if you were planning to implement an event handling system, give GenStage a try.</p> <p>Developers who maintain libraries that integrate with external data sources, be it a RabbitMQ, Redis or Apacha Kafka, can explore GenStage as an abstraction for consuming data from those sources. Library developers must implement producers and leave it up for their users to configure the consumer stages.</p> <p>Once we get enough feedback, <code class="language-plaintext highlighter-rouge">GenStage</code> will be included in some shape as part of the standard library. The goal is to introduce <code class="language-plaintext highlighter-rouge">GenStage</code> and phase <code class="language-plaintext highlighter-rouge">GenEvent</code> out in the long term.</p> <p>We, on the Elixir team, have just got started too. The next milestone for GenStage is to revisit the original problem and provide developers a clear path to take their collection processing code from eager, to lazy, to concurrent (and then distributed).</p> <p>As seen earlier, today we allow developers to transform eager code into lazy by introducing streams.</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="no">File</span><span class="o">.</span><span class="n">stream!</span><span class="p">(</span><span class="s2">"path/to/some/file"</span><span class="p">)</span> <span class="o">|&gt;</span> <span class="no">Stream</span><span class="o">.</span><span class="n">flat_map</span><span class="p">(</span><span class="k">fn</span> <span class="n">line</span> <span class="o">-&gt;</span> <span class="no">String</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="n">line</span><span class="p">,</span> <span class="s2">" "</span><span class="p">)</span> <span class="k">end</span><span class="p">)</span> <span class="o">|&gt;</span> <span class="no">Enum</span><span class="o">.</span><span class="n">reduce</span><span class="p">(%{},</span> <span class="k">fn</span> <span class="n">word</span><span class="p">,</span> <span class="n">acc</span> <span class="o">-&gt;</span> <span class="no">Map</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="n">acc</span><span class="p">,</span> <span class="n">word</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="o">&amp;</span> <span class="nv">&amp;1</span> <span class="o">+</span> <span class="mi">1</span><span class="p">)</span> <span class="k">end</span><span class="p">)</span> <span class="o">|&gt;</span> <span class="no">Enum</span><span class="o">.</span><span class="n">to_list</span><span class="p">()</span> </code></pre></div></div> <p>While the above is helpful when working with large or infinite collections, it still does not leverage concurrency. To address that, we are currently exploring a solution named <a href="https://hexdocs.pm/gen_stage/Experimental.Flow.html"><code class="language-plaintext highlighter-rouge">GenStage.Flow</code></a>, that allows us to express our computations similarly to streams, except they will run across multiple stages instead of a single process:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">alias</span> <span class="no">Experimental</span><span class="o">.</span><span class="no">GenStage</span><span class="o">.</span><span class="no">Flow</span> <span class="no">File</span><span class="o">.</span><span class="n">stream!</span><span class="p">(</span><span class="s2">"path/to/some/file"</span><span class="p">)</span> <span class="o">|&gt;</span> <span class="no">Flow</span><span class="o">.</span><span class="n">from_enumerable</span><span class="p">()</span> <span class="o">|&gt;</span> <span class="no">Flow</span><span class="o">.</span><span class="n">flat_map</span><span class="p">(</span><span class="k">fn</span> <span class="n">line</span> <span class="o">-&gt;</span> <span class="n">for</span> <span class="n">word</span> <span class="o">&lt;-</span> <span class="no">String</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s2">" "</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="p">{</span><span class="n">word</span><span class="p">,</span> <span class="mi">1</span><span class="p">}</span> <span class="k">end</span><span class="p">)</span> <span class="o">|&gt;</span> <span class="no">Flow</span><span class="o">.</span><span class="n">reduce_by_key</span><span class="p">(</span><span class="o">&amp;</span> <span class="nv">&amp;1</span> <span class="o">+</span> <span class="nv">&amp;2</span><span class="p">)</span> <span class="o">|&gt;</span> <span class="no">Enum</span><span class="o">.</span><span class="n">to_list</span><span class="p">()</span> </code></pre></div></div> <p>And the highly optimized version:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">alias</span> <span class="no">Experimental</span><span class="o">.</span><span class="no">GenStage</span><span class="o">.</span><span class="no">Flow</span> <span class="c1"># Let's compile common patterns for performance</span> <span class="n">empty_space</span> <span class="o">=</span> <span class="ss">:binary</span><span class="o">.</span><span class="n">compile_pattern</span><span class="p">(</span><span class="s2">" "</span><span class="p">)</span> <span class="c1"># NEW!</span> <span class="no">File</span><span class="o">.</span><span class="n">stream!</span><span class="p">(</span><span class="s2">"path/to/some/file"</span><span class="p">,</span> <span class="ss">read_ahead:</span> <span class="mi">100_000</span><span class="p">)</span> <span class="c1"># NEW!</span> <span class="o">|&gt;</span> <span class="no">Flow</span><span class="o">.</span><span class="n">from_enumerable</span><span class="p">()</span> <span class="o">|&gt;</span> <span class="no">Flow</span><span class="o">.</span><span class="n">flat_map</span><span class="p">(</span><span class="k">fn</span> <span class="n">line</span> <span class="o">-&gt;</span> <span class="n">for</span> <span class="n">word</span> <span class="o">&lt;-</span> <span class="no">String</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="n">empty_space</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="p">{</span><span class="n">word</span><span class="p">,</span> <span class="mi">1</span><span class="p">}</span> <span class="k">end</span><span class="p">)</span> <span class="o">|&gt;</span> <span class="no">Flow</span><span class="o">.</span><span class="n">partition_with</span><span class="p">(</span><span class="ss">storage:</span> <span class="ss">:ets</span><span class="p">)</span> <span class="c1"># NEW!</span> <span class="o">|&gt;</span> <span class="no">Flow</span><span class="o">.</span><span class="n">reduce_by_key</span><span class="p">(</span><span class="o">&amp;</span> <span class="nv">&amp;1</span> <span class="o">+</span> <span class="nv">&amp;2</span><span class="p">)</span> <span class="o">|&gt;</span> <span class="no">Enum</span><span class="o">.</span><span class="n">to_list</span><span class="p">()</span> </code></pre></div></div> <p>Flow will look at the computations we want to perform and start a series of stages to execute our code while keeping the amount of data being transferred between processes to a minimum. If you are interested in <code class="language-plaintext highlighter-rouge">GenStage.Flow</code> and how the computations above are spread across multiple stages, <a href="https://hexdocs.pm/gen_stage/Experimental.Flow.html">we have written some documentation based on the prototypes we have built so far</a>. The code itself is coming in future GenStage releases. We will also have to consider how the <code class="language-plaintext highlighter-rouge">GenStage.Flow</code> API mirrors the functions in <code class="language-plaintext highlighter-rouge">Enum</code> and <code class="language-plaintext highlighter-rouge">Stream</code> to make the path from eager to concurrent clearer.</p> <p>For the word counting problem with a fixed data, early experiments show a linear increase in performance with a fixed overhead of 20%. In other words, a dataset that takes 60s with a single core, takes 36s on a machine with 2 cores and 18s in one with 4 cores. All of those gains by simply moving your computations from streams to Flow. We plan to benchmark on machines with over 40 cores soon.</p> <p>We are very excited with the possibilities GenStage brings to developers and all new paths it allows us to explore and research. So give it a try and let us know! <a href="http://www.elixirconf.com/">GenStage, Flows, and more will also be the topic of my keynote at ElixirConf 2016</a> and we hope to see you there.</p> <p>Finally, we want to thank the <a href="http://reactive-streams.io">akka-streams and reactive-streams projects</a> which provided us guidance in implementing the demand-driven exchange between stages as well as the <a href="http://spark.apache.org/">Apache Spark</a> and <a href="http://beam.incubator.apache.org/">Apache Beam</a> initiatives that inspire the work behind <code class="language-plaintext highlighter-rouge">GenStage.Flow</code>.</p> <p>Happy coding!</p> José Valim 2016-06-21T00:00:00+00:00 /blog/2016/06/21/elixir-v1-3-0-released <p>Elixir v1.3 brings many improvements to the language, the compiler and its tooling, specially Mix (Elixir’s build tool) and ExUnit (Elixir’s test framework). The most notable additions are the new Calendar types, the new cross-reference checker in Mix, and the assertion diffing in ExUnit. We will explore all of them and a couple more enhancements below.</p> <p>With this release, we also welcome <a href="http://github.com/whatyouhide">Andrea Leopardi</a> to Elixir Core Team. He has contributed greatly to this release and maintains important packages in the community, like <a href="https://github.com/elixir-lang/gettext">Gettext</a> and <a href="https://github.com/whatyouhide/redix">Redix</a>.</p> <h2 id="language-improvements">Language improvements</h2> <p>The language has been improved semantically and includes new types and APIs. Let’s see the three major features.</p> <h3 id="deprecation-of-imperative-assignment">Deprecation of imperative assignment</h3> <p>Elixir will now warn if constructs like <code class="language-plaintext highlighter-rouge">if</code>, <code class="language-plaintext highlighter-rouge">case</code> and friends assign to a variable that is accessed in an outer scope. As an example, imagine a function called <code class="language-plaintext highlighter-rouge">format</code> that receives a message and some options and it must return a path alongside the message:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="n">format</span><span class="p">(</span><span class="n">message</span><span class="p">,</span> <span class="n">opts</span><span class="p">)</span> <span class="k">do</span> <span class="n">path</span> <span class="o">=</span> <span class="k">if</span> <span class="p">(</span><span class="n">file</span> <span class="o">=</span> <span class="n">opts</span><span class="p">[</span><span class="ss">:file</span><span class="p">])</span> <span class="o">&amp;&amp;</span> <span class="p">(</span><span class="n">line</span> <span class="o">=</span> <span class="n">opts</span><span class="p">[</span><span class="ss">:line</span><span class="p">])</span> <span class="k">do</span> <span class="n">relative</span> <span class="o">=</span> <span class="no">Path</span><span class="o">.</span><span class="n">relative_to_cwd</span><span class="p">(</span><span class="n">file</span><span class="p">)</span> <span class="n">message</span> <span class="o">=</span> <span class="no">Exception</span><span class="o">.</span><span class="n">format_file_line</span><span class="p">(</span><span class="n">relative</span><span class="p">,</span> <span class="n">line</span><span class="p">)</span> <span class="o">&lt;&gt;</span> <span class="s2">" "</span> <span class="o">&lt;&gt;</span> <span class="n">message</span> <span class="n">relative</span> <span class="k">end</span> <span class="p">{</span><span class="n">path</span><span class="p">,</span> <span class="n">message</span><span class="p">}</span> <span class="k">end</span> </code></pre></div></div> <p>The <code class="language-plaintext highlighter-rouge">if</code> block above is implicitly changing the value in <code class="language-plaintext highlighter-rouge">message</code>. Now imagine we want to move the <code class="language-plaintext highlighter-rouge">if</code> block to its own function to clean up the implementation:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="n">format</span><span class="p">(</span><span class="n">message</span><span class="p">,</span> <span class="n">opts</span><span class="p">)</span> <span class="k">do</span> <span class="n">path</span> <span class="o">=</span> <span class="n">with_file_and_line</span><span class="p">(</span><span class="n">message</span><span class="p">,</span> <span class="n">opts</span><span class="p">)</span> <span class="p">{</span><span class="n">path</span><span class="p">,</span> <span class="n">message</span><span class="p">}</span> <span class="k">end</span> <span class="k">defp</span> <span class="n">with_file_and_line</span><span class="p">(</span><span class="n">message</span><span class="p">,</span> <span class="n">opts</span><span class="p">)</span> <span class="k">do</span> <span class="k">if</span> <span class="p">(</span><span class="n">file</span> <span class="o">=</span> <span class="n">opts</span><span class="p">[</span><span class="ss">:file</span><span class="p">])</span> <span class="o">&amp;&amp;</span> <span class="p">(</span><span class="n">line</span> <span class="o">=</span> <span class="n">opts</span><span class="p">[</span><span class="ss">:line</span><span class="p">])</span> <span class="k">do</span> <span class="n">relative</span> <span class="o">=</span> <span class="no">Path</span><span class="o">.</span><span class="n">relative_to_cwd</span><span class="p">(</span><span class="n">file</span><span class="p">)</span> <span class="n">message</span> <span class="o">=</span> <span class="no">Exception</span><span class="o">.</span><span class="n">format_file_line</span><span class="p">(</span><span class="n">relative</span><span class="p">,</span> <span class="n">line</span><span class="p">)</span> <span class="o">&lt;&gt;</span> <span class="s2">" "</span> <span class="o">&lt;&gt;</span> <span class="n">message</span> <span class="n">relative</span> <span class="k">end</span> <span class="k">end</span> </code></pre></div></div> <p>The refactored version is broken because the <code class="language-plaintext highlighter-rouge">if</code> block was actually returning two values, the relative path <em>and</em> the new message. Elixir v1.3 will warn on such cases, forcing both variables to be explicitly returned from <code class="language-plaintext highlighter-rouge">if</code>, <code class="language-plaintext highlighter-rouge">case</code> and other constructs. Furthermore, this change gives us the opportunity to unify the language scoping rules in future releases.</p> <h3 id="calendar-types-and-sigils">Calendar types and sigils</h3> <p>Elixir v1.3 introduces the <code class="language-plaintext highlighter-rouge">Calendar</code> module as well as 4 new calendar types:</p> <ul> <li><code class="language-plaintext highlighter-rouge">Date</code> - used to store dates (year, month, day) in a given calendar</li> <li><code class="language-plaintext highlighter-rouge">Time</code> - used to store time (hour, minute, second, microseconds)</li> <li><code class="language-plaintext highlighter-rouge">NaiveDateTime</code> - used to store datetimes without a timezone (year, month, day, hour, minute, second, microseconds) in a given calendar. It is called naïve because without a timezone, the datetime may not actually exist. For example, when there are daylight savings changes, a whole hour may not exist (when the clock moves forward) or a particular instant may happen twice (when the clock moves backwards)</li> <li><code class="language-plaintext highlighter-rouge">DateTime</code> - used to store datetimes with timezone (year, month, day, hour, minute, second, microsecond and time zone, with abbreviation, UTC and standard offset)</li> </ul> <p>The aim of the current implementation of the Calendar modules and its types is to provide a base for interoperatibility in the ecosystem instead of full-featured datetime API. This release includes basic functionality for building new types and converting them from and back strings.</p> <p>Elixir v1.3 also introduces 3 new sigils related to the types above:</p> <ul> <li><code class="language-plaintext highlighter-rouge">~D[2016-05-29]</code> - builds a new date</li> <li><code class="language-plaintext highlighter-rouge">~T[08:00:00]</code> and <code class="language-plaintext highlighter-rouge">~T[08:00:00.285]</code> - builds a new time (with different precisions)</li> <li><code class="language-plaintext highlighter-rouge">~N[2016-05-29 08:00:00]</code> - builds a naive date time</li> </ul> <h3 id="access-selectors">Access selectors</h3> <p>This release introduces new accessors to make it simpler for developers to traverse nested data structures, traversing and updating data in different ways. For instance, given a user with a list of languages, here is how to deeply traverse the map and convert all language names to uppercase:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">iex</span><span class="o">&gt;</span> <span class="n">user</span> <span class="o">=</span> <span class="p">%{</span><span class="ss">name:</span> <span class="s2">"john"</span><span class="p">,</span> <span class="o">...&gt;</span> <span class="ss">languages:</span> <span class="p">[%{</span><span class="ss">name:</span> <span class="s2">"elixir"</span><span class="p">,</span> <span class="ss">type:</span> <span class="ss">:functional</span><span class="p">},</span> <span class="o">...&gt;</span> <span class="p">%{</span><span class="ss">name:</span> <span class="s2">"c"</span><span class="p">,</span> <span class="ss">type:</span> <span class="ss">:procedural</span><span class="p">}]}</span> <span class="n">iex</span><span class="o">&gt;</span> <span class="n">update_in</span> <span class="n">user</span><span class="p">,</span> <span class="p">[</span><span class="ss">:languages</span><span class="p">,</span> <span class="no">Access</span><span class="o">.</span><span class="n">all</span><span class="p">(),</span> <span class="ss">:name</span><span class="p">],</span> <span class="o">&amp;</span><span class="no">String</span><span class="o">.</span><span class="n">upcase</span><span class="o">/</span><span class="mi">1</span> <span class="p">%{</span><span class="ss">name:</span> <span class="s2">"john"</span><span class="p">,</span> <span class="ss">languages:</span> <span class="p">[%{</span><span class="ss">name:</span> <span class="s2">"ELIXIR"</span><span class="p">,</span> <span class="ss">type:</span> <span class="ss">:functional</span><span class="p">},</span> <span class="p">%{</span><span class="ss">name:</span> <span class="s2">"C"</span><span class="p">,</span> <span class="ss">type:</span> <span class="ss">:procedural</span><span class="p">}]}</span> </code></pre></div></div> <p>You can see the new accessors in the <code class="language-plaintext highlighter-rouge">Access</code> module.</p> <h2 id="mix">Mix</h2> <p>Mix includes new tasks to improve your everyday workflow. Some of those tasks relies on many compiler improvements to know more about your code, providing static analysis to find possible bugs in your code and faster compilation cycles.</p> <h3 id="compiling-n-files">Compiling n files</h3> <p>Mix no longer announces every file it compiles. Instead it outputs how many files there is to compile per compilers. Here is the output for a project like <a href="https://github.com/elixir-lang/gettext"><code class="language-plaintext highlighter-rouge">gettext</code></a>:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Compiling 1 file (.yrl) Compiling 1 file (.erl) Compiling 19 files (.ex) Generated gettext app </code></pre></div></div> <p>In case a file is taking too long to compile, Mix will announce such, for example:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Compiling lib/gettext.ex (it's taking more than 10s) </code></pre></div></div> <p>The goal of these changes is to put an increased focus on the “warnings” emitted by the compiler.</p> <p>In any case, the previous behaviour can be brought back with the <code class="language-plaintext highlighter-rouge">--verbose</code> flag and the compilation threshold for files that are taking long can be set via the <code class="language-plaintext highlighter-rouge">--long-compilation-threshold</code> option.</p> <h3 id="mix-xref">mix xref</h3> <p>Speaking about warnings, Mix v1.3 includes a new task called <code class="language-plaintext highlighter-rouge">xref</code> that performs cross reference checks in your code. One of such checks is the ability to find calls to modules and functions that do not exist. For example, if in your library code you call <code class="language-plaintext highlighter-rouge">ThisModuleDoesNotExist.foo(1, 2, 3)</code>, <code class="language-plaintext highlighter-rouge">mix xref unreachable</code> will be able to find such code and let you know about it.</p> <p>Since such checks can discover possible bugs in your codebase, a new compiler called <code class="language-plaintext highlighter-rouge">xref</code> has been added to <code class="language-plaintext highlighter-rouge">Mix.compilers/0</code>, so it runs by default every time you compile your code. <a href="https://pragtob.wordpress.com/2016/06/02/elixir-1-3s-mix-xref-working-its-magic-in-a-real-world-example/">PragTob has written an article exploring how this new compiler has found bugs in existing projects</a>.</p> <p>We have included other modes in <code class="language-plaintext highlighter-rouge">xref</code>, such as:</p> <ul> <li> <p><code class="language-plaintext highlighter-rouge">mix xref callers Foo</code> - used to find all places in your code that calls a function from the module <code class="language-plaintext highlighter-rouge">Foo</code></p> </li> <li> <p><code class="language-plaintext highlighter-rouge">mix xref graph</code> - generates a graph with dependencies between source files</p> </li> </ul> <p>You can find documentation for all modes by running <code class="language-plaintext highlighter-rouge">mix help xref</code>. We hope tools and text editors can leverage such features to provide useful functionality for their users, helping developers understand code complexity and finding bugs early on.</p> <h3 id="better-dependency-tracking">Better dependency tracking</h3> <p>Besides <code class="language-plaintext highlighter-rouge">xref</code>, Elixir v1.3 provides better module tracking generally. For example, in previous versions, if you changed a <code class="language-plaintext highlighter-rouge">:path</code> dependency, Elixir would always fully recompile the current project. In this release, we have improved the tracking algorithms such that, if you change a <code class="language-plaintext highlighter-rouge">:path</code> dependency, only the files that depend on such dependency are recompiled.</p> <p>Such improvements do not only make compilation faster but they also make working with umbrella applications much more productive. Previously, changing a sibling application triggered a full project recompilation, now Elixir can track between sibling applications and recompile only what is needed.</p> <h3 id="mix-apptree-and-depstree">mix app.tree and deps.tree</h3> <p>Mix also includes both <code class="language-plaintext highlighter-rouge">mix app.tree</code> and <code class="language-plaintext highlighter-rouge">mix deps.tree</code>. The first will list all applications your current project needs to start in order to boot (i.e. the ones listed in <code class="language-plaintext highlighter-rouge">application/0</code> in your <code class="language-plaintext highlighter-rouge">mix.exs</code>) while the second will lists all of your dependencies and so on recursively.</p> <p>Here is a quick example from <a href="https://github.com/elixir-lang/plug">Plug</a>:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="err">$</span> <span class="n">mix</span> <span class="n">app</span><span class="o">.</span><span class="n">tree</span> <span class="n">plug</span> <span class="err">├──</span> <span class="n">elixir</span> <span class="err">├──</span> <span class="n">crypto</span> <span class="err">├──</span> <span class="n">logger</span> <span class="err">│</span> <span class="err">└──</span> <span class="n">elixir</span> <span class="err">└──</span> <span class="n">mime</span> <span class="err">└──</span> <span class="n">elixir</span> </code></pre></div></div> <p>The <code class="language-plaintext highlighter-rouge">--format dot</code> option can also be given to generate graph files to be opened by <a href="http://www.graphviz.org">GraphViz</a>. For example, here is the output of running <code class="language-plaintext highlighter-rouge">mix deps.tree --format dot --only prod</code> in the <a href="http://phoenixframework.org">Phoenix web framework</a>:</p> <p style="text-align: center"> <img src="/images/contents/deps-tree-phoenix.svg" alt="mix deps.tree for Phoenix in production" /> </p> <h3 id="mix-escriptinstall">mix escript.install</h3> <p>Mix also includes <code class="language-plaintext highlighter-rouge">mix escript.install</code> and <code class="language-plaintext highlighter-rouge">mix escript.uninstall</code> tasks for managing escripts. The tasks was designed in a way to mimic the existing <code class="language-plaintext highlighter-rouge">mix archive</code> functionality except that:</p> <ul> <li>Archives must be used sparingly because every new archive installed affects Mix performance, as every new archive is loaded when Mix boots. Escripts solve this by being managed apart from your Elixir/Mix installed</li> <li>Archives depends on the current Elixir version. Therefore, updating your Elixir version may break an archive. Fortunately, escripts include Elixir inside themselves, and therefore do not depend on your Elixir system version</li> </ul> <p>Escripts will be installed at <code class="language-plaintext highlighter-rouge">~/.mix/escripts</code> which must be added to your <a href="https://en.wikipedia.org/wiki/PATH_(variable)"><code class="language-plaintext highlighter-rouge">PATH</code> environment variable</a>.</p> <h3 id="option-parser-integration">Option parser integration</h3> <p>Elixir v1.3 includes improvements to the option parser, including <code class="language-plaintext highlighter-rouge">OptionParser.parse!/2</code> and <code class="language-plaintext highlighter-rouge">OptionParser.parse_head!/2</code> functions that will raise in case of invalid or unknown switches. Mix builds on top of this functionality to provide automatic error reporting solving a common complaint where invalid options were not reported by Mix tasks.</p> <p>For example, invoking <code class="language-plaintext highlighter-rouge">mix test --unknown</code> in earlier Elixir versions would silently discard the <code class="language-plaintext highlighter-rouge">--unknown</code> option. Now <code class="language-plaintext highlighter-rouge">mix test</code> correctly reports such errors:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ mix test --unknown ** (Mix) Could not invoke task "test": 1 error found! --unknown : Unknown option </code></pre></div></div> <p>Note not all tasks have been updated to use strict option parsing. Some tasks, like <code class="language-plaintext highlighter-rouge">mix compile</code>, are actually a front-end to many other tasks, and as such, it cannot effectively assert which options are valid.</p> <h2 id="exunit">ExUnit</h2> <p>ExUnit packs many improvements on the tooling side, better integration with external tools, as well as mechanisms to improve the readability of your tests.</p> <h3 id="mix-test---stale">mix test --stale</h3> <p>ExUnit builds on top of <code class="language-plaintext highlighter-rouge">mix xref</code> to provide the <code class="language-plaintext highlighter-rouge">mix test --stale</code> functionality. When the <code class="language-plaintext highlighter-rouge">--stale</code> flag is given, <code class="language-plaintext highlighter-rouge">mix</code> will only run the tests that may have changed since the last time you ran <code class="language-plaintext highlighter-rouge">mix test --stale</code>. For example:</p> <ul> <li>If you saved a test file on disk, Mix will run that file and ignore the ones that have not changed</li> <li>If you changed a library file, for example, <code class="language-plaintext highlighter-rouge">lib/foo.ex</code> that defines <code class="language-plaintext highlighter-rouge">Foo</code>, any test that invokes a function in <code class="language-plaintext highlighter-rouge">Foo</code> directly or indirectly will also run</li> <li>If you modify your <code class="language-plaintext highlighter-rouge">mix.exs</code> or your <code class="language-plaintext highlighter-rouge">test/test_helper.exs</code>, Mix will run the whole test suite</li> </ul> <p>This feature provides a great workflow for developers, allowing them to effortlessly focus on parts of the codebase when developing new features.</p> <h3 id="diffing">Diffing</h3> <p>ExUnit will now include diff-ing output every time a developer asserts <code class="language-plaintext highlighter-rouge">assert left == right</code> in their tests. For example, the assertion:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">assert</span> <span class="s2">"fox jumps over the lazy dog"</span> <span class="o">==</span> <span class="s2">"brown fox jumps over the dog"</span> </code></pre></div></div> <p>will fail with</p> <p><img src="/images/contents/exunit-diff.png" alt="ExUnit diff" /></p> <p>such that “lazy” in “lhs” will be shown in red to denote it has been removed from “rhs” while “brown” in “rhs” will be shown in green to denote it has been added to the “rhs”.</p> <p>When working with large or nested data structures, the diffing algorithm makes it fast and convenient to spot the actual differences in the asserted values.</p> <h3 id="test-types">Test types</h3> <p>ExUnit v1.3 includes the ability to register different test types. This means libraries like QuickCheck can now provide functionality such as:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">StringTest</span> <span class="k">do</span> <span class="kn">use</span> <span class="no">ExUnit</span><span class="o">.</span><span class="no">Case</span><span class="p">,</span> <span class="ss">async:</span> <span class="no">true</span> <span class="kn">use</span> <span class="no">PropertyTestingLibrary</span> <span class="n">property</span> <span class="s2">"starts_with?"</span> <span class="k">do</span> <span class="n">forall</span><span class="p">({</span><span class="n">s1</span><span class="p">,</span> <span class="n">s2</span><span class="p">}</span> <span class="o">&lt;-</span> <span class="p">{</span><span class="n">utf8</span><span class="p">,</span> <span class="n">utf8</span><span class="p">})</span> <span class="k">do</span> <span class="no">String</span><span class="o">.</span><span class="n">starts_with?</span><span class="p">(</span><span class="n">s1</span> <span class="o">&lt;&gt;</span> <span class="n">s2</span><span class="p">,</span> <span class="n">s1</span><span class="p">)</span> <span class="k">end</span> <span class="k">end</span> <span class="k">end</span> </code></pre></div></div> <p>At the end of the run, ExUnit will also report it as a property, including both the amount of tests and properties:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>1 property, 10 tests, 0 failures </code></pre></div></div> <h3 id="named-setups-and-describes">Named setups and describes</h3> <p>Finally, ExUnit v1.3 includes the ability to organize tests together in describe blocks:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">StringTest</span> <span class="k">do</span> <span class="kn">use</span> <span class="no">ExUnit</span><span class="o">.</span><span class="no">Case</span><span class="p">,</span> <span class="ss">async:</span> <span class="no">true</span> <span class="n">describe</span> <span class="s2">"String.capitalize/2"</span> <span class="k">do</span> <span class="n">test</span> <span class="s2">"uppercases the first grapheme"</span> <span class="k">do</span> <span class="n">assert</span> <span class="s2">"T"</span> <span class="o">&lt;&gt;</span> <span class="n">_</span> <span class="o">=</span> <span class="no">String</span><span class="o">.</span><span class="n">capitalize</span><span class="p">(</span><span class="s2">"test"</span><span class="p">)</span> <span class="k">end</span> <span class="n">test</span> <span class="s2">"lowercases the remaining graphemes"</span> <span class="k">do</span> <span class="n">assert</span> <span class="s2">"Test"</span> <span class="o">=</span> <span class="no">String</span><span class="o">.</span><span class="n">capitalize</span><span class="p">(</span><span class="s2">"TEST"</span><span class="p">)</span> <span class="k">end</span> <span class="k">end</span> <span class="k">end</span> </code></pre></div></div> <p>Every test inside a describe block will be tagged with the describe block name. This allows developers to run tests that belong to particular blocks, be them in the same file or across many files:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ mix test --only describe:"String.capitalize/2" </code></pre></div></div> <p>Note describe blocks cannot be nested. Instead of relying on hierarchy for composition, we want developers to build on top of named setups. For example:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">UserManagementTest</span> <span class="k">do</span> <span class="kn">use</span> <span class="no">ExUnit</span><span class="o">.</span><span class="no">Case</span><span class="p">,</span> <span class="ss">async:</span> <span class="no">true</span> <span class="n">describe</span> <span class="s2">"when user is logged in and is an admin"</span> <span class="k">do</span> <span class="n">setup</span> <span class="p">[</span><span class="ss">:log_user_in</span><span class="p">,</span> <span class="ss">:set_type_to_admin</span><span class="p">]</span> <span class="n">test</span> <span class="o">...</span> <span class="k">end</span> <span class="n">describe</span> <span class="s2">"when user is logged in and is a manager"</span> <span class="k">do</span> <span class="n">setup</span> <span class="p">[</span><span class="ss">:log_user_in</span><span class="p">,</span> <span class="ss">:set_type_to_manager</span><span class="p">]</span> <span class="n">test</span> <span class="o">...</span> <span class="k">end</span> <span class="k">defp</span> <span class="n">log_user_in</span><span class="p">(</span><span class="n">context</span><span class="p">)</span> <span class="k">do</span> <span class="c1"># ...</span> <span class="k">end</span> <span class="k">end</span> </code></pre></div></div> <p>By restricting hierarchies in favor of named setups, it is straight-forward for the developer to glance at each describe block and know exactly the setup steps involved.</p> <h2 id="summing-up">Summing up</h2> <p>The full list of changes is available in our <a href="https://github.com/elixir-lang/elixir/releases/tag/v1.3.0">release notes</a>. Don’t forget to check <a href="/install.html">the Install section</a> to get Elixir installed and our <a href="https://hexdocs.pm/elixir/introduction.html">Getting Started guide</a> to learn more.</p> <p>Happy coding!</p> José Valim 2016-01-03T00:00:00+00:00 /blog/2016/01/03/elixir-v1-2-0-released <p>v1.2 brings enhancements, bug fixes, performance improvements and more into Elixir. Elixir v1.2 relies on many features in Erlang 18, requiring at least Erlang 18+. Upgrading to Erlang 18 is therefore necessary before upgrading Elixir.</p> <p>To celebrate this release and the new year, we have also reviewed both the <a href="https://hexdocs.pm/elixir/introduction.html">Getting Started</a> and <a href="https://hexdocs.pm/elixir/introduction-to-mix.html">Mix &amp; OTP</a> guides, bringing it up to date and exploring new functionalities added since Elixir v1.0.</p> <h2 id="erlang-18-support">Erlang 18 support</h2> <p>We have brought many features specific to Erlang 18. Here are the highlights:</p> <ul> <li>Maps can now scale from dozens to millions of keys. Therefore, usage of the modules <code class="language-plaintext highlighter-rouge">Dict</code> and <code class="language-plaintext highlighter-rouge">HashDict</code> is now discouraged and will be deprecated in future releases, instead use <code class="language-plaintext highlighter-rouge">Map</code>. Similarly, <code class="language-plaintext highlighter-rouge">Set</code> and <code class="language-plaintext highlighter-rouge">HashSet</code> will be deprecated in favor of <code class="language-plaintext highlighter-rouge">MapSet</code></li> <li>Compilation times are ~15% faster on average due to improvements in both the Elixir and Erlang compilers and by using maps more extensively in the compilation stack</li> <li>Dialyzer now emits less false negative warnings thanks to new annotations available in the Erlang compiler</li> </ul> <h2 id="language-improvements">Language improvements</h2> <p>This release includes many notable language improvements.</p> <p>The first of them was the addition of multi aliases/imports/require. Often developers would write:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">alias</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">Foo</span> <span class="n">alias</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">Bar</span> <span class="n">alias</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">Baz</span> </code></pre></div></div> <p>Now it can be written in one line by using the new multi syntax:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">alias</span> <span class="no">MyApp</span><span class="o">.</span><span class="p">{</span><span class="no">Foo</span><span class="p">,</span> <span class="no">Bar</span><span class="p">,</span> <span class="no">Baz</span><span class="p">}</span> </code></pre></div></div> <p>We have also added support for variables in map keys. Now you can write:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">iex</span><span class="o">&gt;</span> <span class="n">key</span> <span class="o">=</span> <span class="ss">:hello</span> <span class="n">iex</span><span class="o">&gt;</span> <span class="n">value</span> <span class="o">=</span> <span class="s2">"world"</span> <span class="n">iex</span><span class="o">&gt;</span> <span class="p">%{</span><span class="n">key</span> <span class="o">=&gt;</span> <span class="n">value</span><span class="p">}</span> <span class="p">%{</span><span class="ss">:hello</span> <span class="o">=&gt;</span> <span class="s2">"world"</span><span class="p">}</span> </code></pre></div></div> <p>Furthermore, variables can also be used on pattern matching along-side the pin operator:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">iex</span><span class="o">&gt;</span> <span class="n">key</span> <span class="o">=</span> <span class="ss">:hello</span> <span class="n">iex</span><span class="o">&gt;</span> <span class="p">%{</span><span class="o">^</span><span class="n">key</span> <span class="o">=&gt;</span> <span class="n">value</span><span class="p">}</span> <span class="o">=</span> <span class="p">%{</span><span class="ss">:hello</span> <span class="o">=&gt;</span> <span class="s2">"another world"</span><span class="p">}</span> <span class="n">iex</span><span class="o">&gt;</span> <span class="n">value</span> <span class="s2">"another world"</span> </code></pre></div></div> <p>Finally, Elixir v1.2 introduces the <code class="language-plaintext highlighter-rouge">with</code> special form that allows developers to match on multiple expressions concisely. Previously, one would write</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">case</span> <span class="no">File</span><span class="o">.</span><span class="n">read</span><span class="p">(</span><span class="s2">"my_file.ex"</span><span class="p">)</span> <span class="k">do</span> <span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">contents</span><span class="p">}</span> <span class="o">-&gt;</span> <span class="k">case</span> <span class="no">Code</span><span class="o">.</span><span class="n">eval_string</span><span class="p">(</span><span class="n">contents</span><span class="p">)</span> <span class="k">do</span> <span class="p">{</span><span class="n">res</span><span class="p">,</span> <span class="n">_binding</span><span class="p">}</span> <span class="o">-&gt;</span> <span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">res</span><span class="p">}</span> <span class="n">error</span> <span class="o">-&gt;</span> <span class="n">error</span> <span class="n">error</span> <span class="o">-&gt;</span> <span class="n">error</span> <span class="n">error</span> <span class="k">end</span> </code></pre></div></div> <p>such can now be rewritten as</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">with</span> <span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">contents</span><span class="p">}</span> <span class="o">&lt;-</span> <span class="no">File</span><span class="o">.</span><span class="n">read</span><span class="p">(</span><span class="s2">"my_file.ex"</span><span class="p">),</span> <span class="p">{</span><span class="n">res</span><span class="p">,</span> <span class="n">binding</span><span class="p">}</span> <span class="o">&lt;-</span> <span class="no">Code</span><span class="o">.</span><span class="n">eval_string</span><span class="p">(</span><span class="n">contents</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">res</span><span class="p">}</span> </code></pre></div></div> <p><code class="language-plaintext highlighter-rouge">with</code> will match each left side of <code class="language-plaintext highlighter-rouge">&lt;-</code> against the right side, executing expressions until one of those match fails or until the <code class="language-plaintext highlighter-rouge">do: expression</code> is performed. In case a match fails, the non-matching result is returned.</p> <p>These improvements aim to make the language more consistent and expressive.</p> <h2 id="getting-started-experience">Getting started experience</h2> <p>We have also improved both the parser and compiler to be more aware of language constructs, emitting warnings on common pitfalls like when piping to expressions without parentheses or when defining unsafe variables. Such improvements will point developers to the more idiomatic way of writing Elixir code early on.</p> <p>Elixir v1.2 also introduces the <code class="language-plaintext highlighter-rouge">i/1</code> helper in IEx, which allows developers to retrieve information about any data type. This will help newcomers explore the language values while providing experienced developers with crucial information about the value they are introspecting. For example, giving a PID to <code class="language-plaintext highlighter-rouge">i/1</code> will show if it has a registered name, linked processes and more. Giving it a module, like <code class="language-plaintext highlighter-rouge">i(String)</code>, shows compile-time information and others.</p> <p>All of those improvements tie nicely with our updates to the Getting Started guide, ensuring learning Elixir is more fun and efficient than ever before.</p> <h2 id="workflow-improvements">Workflow improvements</h2> <p>One of Elixir goals is to build upon the abstractions provided by Erlang/OTP and make them more productive by focusing on the tooling aspect.</p> <p>One of such efforts resulted in “Umbrella Projects”, which allows developers to build multiple applications side-by-side, but still run and test them in isolation when desired. Because each application contains its own configuration, supervision tree and initialization cycle, this gives developers the proper mechanisms to break monolithic applications apart without introducing the complexity of managing multiple, different repositories.</p> <p>Up to this release, umbrella applications shared mostly dependencies, which meant each application still had their own build directory and their own compilation cycle. Elixir v1.2 allows developers to also share both build and configuration files. This change allows teams to drastically reduce compilation times in umbrella projects by adding the following configuration to each umbrella app’s <code class="language-plaintext highlighter-rouge">mix.exs</code> file:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="ss">build_path:</span> <span class="s2">"../../_build"</span><span class="p">,</span> <span class="ss">config_path:</span> <span class="s2">"../../config/config.exs"</span><span class="p">,</span> </code></pre></div></div> <p>Umbrella applications generated with Elixir v1.2 will by default include this configuration. The downside of this approach is that applications are a bit less isolated, since configuration is now shared across all projects, although developers can revert back to the previous behaviour by simply removing the flags above.</p> <p>Finally, Mix will now consolidate protocols by default as we are now able to consolidate in parallel and cache the consolidation results, providing the best performance across all environments without affecting compilation times.</p> <p>These are great additions on top of the faster compilation times we have achieved when migrating to Erlang 18.</p> <h2 id="rebar-3-support">Rebar 3 support</h2> <p>With Rebar 3 gaining more adoption in the Erlang community, Mix is now able to fetch and compile Rebar 3 dependencies. This feature is currently experimental and therefore opt-in: if you have a Rebar 3 dependency, you can ask Mix to use Rebar 3 to compile it by passing the <code class="language-plaintext highlighter-rouge">manager: :rebar3</code> option. Once configured, Mix will prompt you to install Rebar 3 if it is not yet available.</p> <p>The full list of changes is available in our <a href="https://github.com/elixir-lang/elixir/releases/tag/v1.2.0">release notes</a>. Don’t forget to check <a href="/install.html">the Install section</a> to get Elixir installed and our <a href="https://hexdocs.pm/elixir/introduction.html">Getting Started guide</a> to learn more.</p> <p>Happy coding!</p> José Valim 2015-09-28T00:00:00+00:00 /blog/2015/09/28/elixir-v1-1-0-released <p>Elixir v1.1 has been released and it brings enhancements, bug fixes, performance improvements and more into Elixir.</p> <p>Elixir v1.1 supports both Erlang 17 and Erlang 18. This is, however, the last release supporting Erlang 17, so upgrading to Erlang 18 is advised. Elixir v1.2 will introduce features that are Erlang 18 only.</p> <p>On the standard library side, about 40 new functions have been added to Elixir public APIs. For example, <a href="/docs/v1.1/elixir/Enum.html"><code class="language-plaintext highlighter-rouge">Enum</code></a> got <a href="/docs/v1.1/elixir/Enum.html#dedup/1"><code class="language-plaintext highlighter-rouge">dedup/1</code></a>, <a href="/docs/v1.1/elixir/Enum.html#random/1"><code class="language-plaintext highlighter-rouge">random/1</code></a>, and a couple more. The <a href="/docs/v1.1/elixir/String.html"><code class="language-plaintext highlighter-rouge">String</code></a> module can now <a href="/docs/v1.1/elixir/String.html#jaro_distance/2">calculate the distance between strings</a>. We use this feature to provide suggestions whenever an unknown task name is given when using Mix. You can also <a href="/docs/v1.1/elixir/Task.html#yield/2">yield to</a> and <a href="/docs/v1.1/elixir/Task.html#shutdown/2">shutdown</a> tasks in the <a href="/docs/v1.1/elixir/Task.html"><code class="language-plaintext highlighter-rouge">Task</code></a> module.</p> <p>The applications that ship with Elixir also has seen improvements and bug fixes. <a href="/docs/v1.1/ex_unit/ExUnit.html">ExUnit</a>, Elixir’s test framework, now has support for skipping tests via tags <code class="language-plaintext highlighter-rouge">@tag :skip</code>, as well as the ability to capture logs via <code class="language-plaintext highlighter-rouge">@tag :capture_log</code>, ensuring that all log messages during the tests are captured. Even better, in case of failures, all captured log messages are printed along-side the test error report.</p> <p>Mix ships with a <a href="/docs/v1.1/mix/Mix.Tasks.Profile.Fprof.html"><code class="language-plaintext highlighter-rouge">mix profile.fprof</code></a>, useful for profiling your application code. The <a href="/docs/v1.1/mix/Mix.Tasks.App.Start.html"><code class="language-plaintext highlighter-rouge">mix app.start</code></a> has also been publicly documented. Although you likely won’t invoke it directly through the command line, it is useful when writing your own tasks that require the current application to be up and running.</p> <p>Mix also provides faster re-compilation times. Every time you compile your Elixir code, Mix generates a graph of the dependencies between source files. For example, if <code class="language-plaintext highlighter-rouge">a.ex</code> depends on <code class="language-plaintext highlighter-rouge">b.ex</code>, every time <code class="language-plaintext highlighter-rouge">b.ex</code> changes, <code class="language-plaintext highlighter-rouge">a.ex</code> must be recompiled. Elixir v1.1 improves this tracking by separating compile-time dependencies from runtime ones, recompiling a file only if a compile-time dependency changed. In projects that have a main dispatch entity, like a web-app router, we have seen <code class="language-plaintext highlighter-rouge">mix compile</code> go from recompiling the whole project to one or two files per run.</p> <p>We have also seen great progress on areas that go beyond the source code. In particular, we have added a <a href="https://github.com/elixir-lang/elixir/blob/v1.1/CODE_OF_CONDUCT.md">CODE_OF_CONDUCT.md</a> to guarantee our community continues to grow into a safe and welcoming place for everyone.</p> <p>We have also released a new ExDoc version. It provides a <a href="https://hexdocs.pm/elixir/1.1.0">beautiful, clean and readable way to navigate the Elixir documentation</a> and it is available to any Elixir project. The latest version includes initial support for User Guides and we have more features and improvements coming on the way.</p> <p>The full list of changes is available in our <a href="https://github.com/elixir-lang/elixir/releases/tag/v1.1.0">release notes</a>. Don’t forget to check <a href="/install.html">the Install section</a> to get Elixir installed.</p> <p>Happy coding!</p> José Valim 2014-09-18T00:00:00+00:00 /blog/2014/09/18/elixir-v1-0-0-released <p>We are glad to announce Elixir v1.0 is finally out. It has been 8005 commits <a href="https://github.com/elixir-lang/elixir/graphs/contributors?from=2011-01-12&amp;to=2014-09-10&amp;type=c">by 189 contributors</a>, including the initial commit on <a href="https://github.com/elixir-lang/elixir/commit/337c3f2d569a42ebd5fcab6fef18c5e012f9be5b">January 9th, 2011</a>!</p> <h2 id="whats-elixir">What’s Elixir?</h2> <p>Elixir is a dynamic, functional language designed for building scalable and maintainable applications.</p> <p>Elixir leverages the Erlang VM, known for running low-latency, distributed and fault-tolerant systems, while also being successfully used in web development and the embedded software domain.</p> <h2 id="whats-new">What’s new?</h2> <p>This release is the consolidation of all the work done through the years. With v1.0, we have reached a stable milestone for the growth of software and projects written in Elixir.</p> <p>Elixir will follow <a href="http://semver.org">semantic versioning</a>, which means code written for Elixir v1.0 will continue to compile and run correctly for all versions under the v1 branch (e.g. v1.0.1, v1.1.0, v1.2.0).</p> <p>Elixir is composed of 6 applications, all under the same versioning constraints:</p> <ul> <li><a href="https://hexdocs.pm/elixir/">Elixir</a> - the Elixir compiler, runtime and the standard library</li> <li><a href="https://hexdocs.pm/eex/">EEx</a> - Elixir’s templating library, useful for generating any kind of document dynamically</li> <li><a href="https://hexdocs.pm/ex_unit/">ExUnit</a> - Elixir’s unit test library, with support for concurrent testing, custom formatters, filters and much more</li> <li><a href="https://hexdocs.pm/iex/">IEx</a> - Elixir’s interactive shell with code reloading, auto-complete, and easy access to documentation, typespecs and more</li> <li><a href="https://hexdocs.pm/logger/">Logger</a> - the latest addition to the group, Logger provides reliable logging and configurable backends (with syslog, file and many other backends provided by the community)</li> <li><a href="https://hexdocs.pm/mix/">Mix</a> - Elixir’s build tool that generates, compiles and tests projects as well as manages your dependencies</li> </ul> <p>With v1.0, we are providing a stable platform for the community to leverage and extend, and we are extremely excited with the projects and possibilities that are ahead of us!</p> <p>We hope the <a href="https://hex.pm">Hex package manager</a> will be the home of many of those projects and remember the whole Erlang ecosystem is also available to Elixir developers.</p> <h2 id="expectations">Expectations</h2> <p>We would like to elaborate on the expectations regarding Elixir v1.0. Although we expect that the vast majority of programs will remain compatible over time, it is impossible to guarantee that no future change will break any program.</p> <p>Under some unlikely circumstances, we may introduce changes that break existing code:</p> <ul> <li> <p>Security: a security issue in the implementation may arise whose resolution requires backwards incompatible changes. We reserve the right to address such security issues.</p> </li> <li> <p>Bugs: if an application has undesired behaviour, a program that depends on the buggy behaviour may break if the bug is fixed. We reserve the right to fix such bugs.</p> </li> <li> <p>Compiler front-end: improvements may be done to the compiler, introducing new warnings for ambiguous modes and providing more detailed error messages. Those can lead to compilation errors (when running with <code class="language-plaintext highlighter-rouge">--warning-as-errors</code>) or tooling failures when asserting on specific error messages (although one should avoid such). We reserve the right to do such improvements.</p> </li> <li> <p>Imports: new functions may be added to the Kernel module, which is auto-imported. They may collide with local functions defined in your modules. Collisions can be resolved in a backwards compatible fashion using <code class="language-plaintext highlighter-rouge">import Kernel, except: [...]</code> with a list of all functions you don’t want imported from Kernel. We reserve the right to do such additions.</p> </li> </ul> <p>Elixir binaries are guaranteed to be backwards compatible within the same branch. Code compiled with v1.0 shall work with Elixir v1.0.1 runtime but code compiled with Elixir v1.0 should be recompiled to work with Elixir v1.1.0 onwards.</p> <p>These expectations also apply to future releases under the v1 branch, except for experimental features, which will be explicitly marked as such and not provide any compatibility guarantee until they are stabilized.</p> <h2 id="learn-more">Learn more</h2> <p>You can get started with Elixir via our <a href="https://hexdocs.pm/elixir/introduction.html">Getting Started guide</a>. There are quite some Elixir books out there too, now getting sent to the presses, quite a few can be found in the sidebar, which also includes screencasts and other resources.</p> <p>You can also learn more about Elixir by checking out <a href="http://www.confreaks.com/events/elixirconf2014">the videos from ElixirConf 2014</a>, the first (and so far the best) Elixir conference ever! You can learn more about <a href="http://www.confreaks.com/videos/4134-elixirconf2014-keynote-elixir">the language history</a>, <a href="http://www.confreaks.com/videos/4119-elixirconf2014-opening-keynote-think-different">how Elixir can change the way you code</a> or <a href="http://www.confreaks.com/videos/4131-elixirconf2014-otp-in-production-the-nitty-gritty-details-of-game-servers">even hear stories of how Elixir is being used in production</a>.</p> <p>Finally, by popular demand, we have <a href="http://www.stickermule.com/user/1070631438/stickers">released some Elixir stickers</a>, which are available with a discounted price to celebrate v1.0!</p> José Valim 2014-08-07T00:00:00+00:00 /blog/2014/08/07/elixir-v0-15-0-released <p>Hello everyone!</p> <p>We are glad to announce v0.15.0 has been released. We have spent the last 2 months tidying up the existing APIs, ensuring consistency, improving performance and more. As a result, v0.15.0 is the last minor branch before Elixir v1.0!</p> <p>There are also no more planned deprecations nor backward incompatible changes which means it is extremely likely that code that runs on v0.15.0 will run exactly the same on Elixir v1.0.</p> <p>If you are interested in the specific details for this release, <a href="https://github.com/elixir-lang/elixir/blob/v0.15.0/CHANGELOG.md">please check our CHANGELOG</a>. In this post, we will focus on three new features in this release: Logger, Mix aliases, the fresh Elixir Web Installer for Windows, and share some exciting news at the end!</p> <h2 id="logger">Logger</h2> <p>Elixir now ships with a new application called logger. This application provides <a href="https://hexdocs.pm/logger/Logger.html">the Logger module</a>, which is the main API developers will use for logging:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">require</span> <span class="no">Logger</span> <span class="no">Logger</span><span class="o">.</span><span class="n">debug</span> <span class="s2">"hello"</span> </code></pre></div></div> <p>By default, the code above will log the following message to your console:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>10:27:39.083 [debug] hello </code></pre></div></div> <p>Logger provides multiple backends to where messages are logged. For now Elixir ships only with a console backend but there are developers already working on file (with support to external log rotation) and <a href="https://en.wikipedia.org/wiki/Syslog">syslog</a> backends.</p> <p>When we started Logger, the main objective was to translate Erlang messages into Elixir, so terms are formatted in Elixir syntax. Before this release, the following code</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="no">Task</span><span class="o">.</span><span class="n">async</span> <span class="k">fn</span> <span class="o">-&gt;</span> <span class="k">raise</span> <span class="s2">"oops"</span> <span class="k">end</span> </code></pre></div></div> <p>logged the following report:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>=ERROR REPORT==== 7-Aug-2014::10:35:59 === ** Task &lt;0.59.0&gt; terminating ** Started from &lt;0.53.0&gt; ** When function == #Fun&lt;erl_eval.20.90072148&gt; ** arguments == [] ** Reason for termination == ** {#{'__exception__' =&gt; true,'__struct__' =&gt; 'Elixir.RuntimeError',message =&gt; &lt;&lt;"oops"&gt;&gt;}, [{'Elixir.Task.Supervised',do_apply,2, [{file,"lib/task/supervised.ex"},{line,70}]}, {'Elixir.Task.Supervised',async,3, [{file,"lib/task/supervised.ex"},{line,15}]}, {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]} </code></pre></div></div> <p>Now, with Logger, we get this report:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>10:37:22.457 [error] Task #PID&lt;0.72.0&gt; started from #PID&lt;0.60.0&gt; terminating Function: #Function&lt;20.90072148/0 in :erl_eval.expr/5&gt; Args: [] ** (exit) an exception was raised: ** (RuntimeError) oops (elixir) lib/task/supervised.ex:70: Task.Supervised.do_apply/2 (elixir) lib/task/supervised.ex:15: Task.Supervised.async/3 (stdlib) proc_lib.erl:239: :proc_lib.init_p_do_apply/3 </code></pre></div></div> <p>As soon as we started working on Logger, we realized we could go further than simply translating Erlang messages and provide a fully featured logger library. At this moment, Logger also supports:</p> <ul> <li>4 log levels: debug, info, warn and error</li> <li>Custom formatting: you can specify a format string that tells exactly how messages should be logged. The default string is: “$time $metadata[$level] $message\n” but <a href="https://hexdocs.pm/logger/Logger.Formatter.html">many attributes are supported</a></li> <li>Custom translators: so you can translate log messages coming from any Erlang application into Elixir syntax</li> <li>Metadata: metadata allows developers to store information in the current process that will be available to all logged messages. For example, a web application can generate a <code class="language-plaintext highlighter-rouge">request_id</code>, store it as metadata, and all messages logged during that request will be properly identified with <code class="language-plaintext highlighter-rouge">request_id=...</code> in the log</li> </ul> <p>We have also relied a lot on the <a href="https://www.youtube.com/watch?v=8BNpOHFvg_Q">research and work done by Andrew Thompson and the folks at Basho behind Lager</a> to ensure our logger is performant and robust. On this front, Logger</p> <ul> <li>alternates between sync and async modes when logging messages to keep it performant when required but also apply back-pressure when under stress</li> <li>formats and truncates messages on the client to avoid clogging the backends</li> <li>provide a highwater mark around Erlang’s error_logger to avoid it from overflowing</li> </ul> <p>We are looking forward to everyone’s feedback on using Logger more and more in production.</p> <h2 id="mix-aliases">Mix aliases</h2> <p>Mix is Elixir’s build tool. Mix knows how to manage your dependencies, compile &amp; test your projects and much more. We have designed Mix, since day one, to be extensible. Developers were always able to define new tasks by simply defining a module:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">Mix</span><span class="o">.</span><span class="no">Tasks</span><span class="o">.</span><span class="no">Hello</span> <span class="k">do</span> <span class="kn">use</span> <span class="no">Mix</span><span class="o">.</span><span class="no">Task</span> <span class="k">def</span> <span class="n">run</span><span class="p">(</span><span class="n">_</span><span class="p">)</span> <span class="k">do</span> <span class="no">IO</span><span class="o">.</span><span class="n">puts</span> <span class="s2">"Hello world"</span> <span class="k">end</span> <span class="k">end</span> </code></pre></div></div> <p>The task above can now be invoked as <code class="language-plaintext highlighter-rouge">mix hello</code>.</p> <p>Defining custom Mix tasks is useful for projects and libraries that want to better integrate with the standard development workflow for Elixir. However, it is a bit verbose for creating one-off tasks or tasks to be used only locally in a given project.</p> <p>Furthermore, so far Mix did not allow developers to extend existing tasks. For example, imagine you want to perform some extra work when <code class="language-plaintext highlighter-rouge">mix clean</code> is invoked. Up to this release, it was not possible.</p> <p>Mix aliases solve both problems by providing a simple API for defining and overriding aliases. All projects that use Mix contain a <code class="language-plaintext highlighter-rouge">mix.exs</code> file with the project configuration. In order to define an alias, you just need to add a new key to your project configuration:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">MyProject</span> <span class="k">do</span> <span class="kn">use</span> <span class="no">Mix</span><span class="o">.</span><span class="no">Project</span> <span class="k">def</span> <span class="n">project</span> <span class="k">do</span> <span class="p">[</span><span class="ss">app:</span> <span class="ss">:my_project</span><span class="p">,</span> <span class="ss">aliases:</span> <span class="n">aliases</span><span class="p">]</span> <span class="k">end</span> <span class="k">defp</span> <span class="n">aliases</span> <span class="k">do</span> <span class="p">[</span><span class="ss">c:</span> <span class="s2">"compile"</span><span class="p">,</span> <span class="ss">hello:</span> <span class="o">&amp;</span><span class="n">print_hello</span><span class="o">/</span><span class="mi">1</span><span class="p">,</span> <span class="ss">clean:</span> <span class="p">[</span><span class="s2">"clean"</span><span class="p">,</span> <span class="o">&amp;</span><span class="n">clean_vendor</span><span class="o">/</span><span class="mi">1</span><span class="p">]]</span> <span class="k">end</span> <span class="k">defp</span> <span class="n">print_hello</span><span class="p">(</span><span class="n">_</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="no">IO</span><span class="o">.</span><span class="n">puts</span> <span class="s2">"Hello world"</span> <span class="k">defp</span> <span class="n">clean_vendor</span><span class="p">(</span><span class="n">_</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="no">File</span><span class="o">.</span><span class="n">rm_rf!</span><span class="p">(</span><span class="s2">"vendor"</span><span class="p">)</span> <span class="k">end</span> </code></pre></div></div> <p>In the project above, we have defined three aliases:</p> <ol> <li><code class="language-plaintext highlighter-rouge">mix c</code> - is now a shortcut to <code class="language-plaintext highlighter-rouge">mix compile</code></li> <li><code class="language-plaintext highlighter-rouge">mix hello</code> - is equivalent to the <code class="language-plaintext highlighter-rouge">Mix.Tasks.Hello</code> we have defined earlier, although now it is more easily defined as an alias</li> <li><code class="language-plaintext highlighter-rouge">mix clean</code> - extends the existing <code class="language-plaintext highlighter-rouge">clean</code> task to ensure we invoke <code class="language-plaintext highlighter-rouge">clean_vendor/1</code> afterwards</li> </ol> <p>In other words, aliases can be three different structures:</p> <ol> <li>A string containing the task and arguments to be invoked</li> <li>An anonymous function (that is invoked passing the task arguments)</li> <li>A list containing strings or anonymous functions</li> </ol> <p>You can find more information about aliases by reading the <a href="https://hexdocs.pm/mix/">Mix documentation</a> (there is a section about Aliases around the middle).</p> <p>We also would like to thank <a href="https://github.com/raynes">Anthony Grimes</a> for the support and <a href="https://github.com/technomancy">Phil Halgelberg</a> for <a href="https://github.com/technomancy/leiningen">the work on Lein</a> which Mix borrows a lot from.</p> <h2 id="elixir-web-installer-for-windows">Elixir Web Installer for Windows</h2> <p>At the beginning of this summer, <a href="https://github.com/chyndman">Chris Hyndman</a> joined us as a Google Summer of Code student to help us improve the Elixir story on Windows. Chris has been essential in:</p> <ul> <li>Guaranteeing our test suite is green on Windows, fixing many bugs in the process;</li> <li><a href="https://github.com/elixir-lang/elixir/wiki/Windows">Documenting how to compile Elixir from source on Windows</a></li> <li>Ensuring important projects like our <a href="https://github.com/devinus/markdown/pull/7">C markdown processor</a> compiles on Windows</li> </ul> <p>Chris has also built an <a href="https://github.com/elixir-lang/elixir-windows-setup">Elixir Web Installer for Windows</a>. The web installer checks all available Elixir versions and allows you to pick which one to install. It will also fetch and install Erlang in your machine in case it has not been installed yet.</p> <p>If you want to give Elixir and the Web Installer a try, you can <a href="https://repo.hex.pm/elixir-websetup.exe">download the current version here</a>. And, if <a href="https://chocolatey.org/">Chocolatey</a> is your thing, remember you can also install Elixir on Windows by running <code class="language-plaintext highlighter-rouge">cinst elixir</code>.</p> <h2 id="welcome-alexei">Welcome Alexei!</h2> <p>With v0.15.0, we also would like to welcome <a href="https://github.com/alco">Alexei</a> to the team of Elixir Maintainers! Alexei was one of the first to actively participate in the Elixir community and he has done an excellent job in guaranteeing quality throughout it, be it in the code, documentation, wiki or website.</p> <p>Alexei is also interested in how we can extend our tooling to the Erlang ecosystem as a whole, bringing Mix and Hex (our package manager) to more developers as well as adding tasks that are specific to Erlang projects.</p> <h2 id="whats-next">What’s next?</h2> <p>We are very close to launch Elixir v1.0! All planned features are already in Elixir’s codebase and at the moment there are only <a href="https://github.com/elixir-lang/elixir/issues?q=is%3Aopen+is%3Aissue+milestone%3Av1.0">four open issues in our tracker tagged with the v1.0 milestone</a>.</p> <p>Our estimated date for the first release candidate for Elixir v1.0 is August 30th. This means there is no better time to learn Elixir than now! If you haven’t started yet, you can get started with Elixir by reading <a href="https://hexdocs.pm/elixir/introduction.html">our Getting Started guide</a> or by checking one of the many “Learning Resources” on the sidebar.</p> José Valim 2014-06-17T00:00:00+00:00 /blog/2014/06/17/elixir-v0-14-0-released <p>Hello everyone!</p> <p>We are glad to announce v0.14.0 has been released.</p> <p>Our previous release, <a href="/blog/2014/04/21/elixir-v0-13-0-released/">v0.13.0</a>, brought important changes to the language. Back then we have introduced a new associative data structure, called maps, and presented the concept of structs to replace Elixir records.</p> <p>With v0.14.0 we have expanded on the work started on v0.13.0, replacing records by structs and integrating OTP behaviours, applications, and configurations into Mix.</p> <h2 id="derive">@derive</h2> <p>Maps are Elixir’s new associative data structure:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">%{</span><span class="ss">key:</span> <span class="s2">"value"</span><span class="p">}</span> </code></pre></div></div> <p>With maps, Elixir also introduced structs, that are basically raw, named maps:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">User</span> <span class="k">do</span> <span class="k">defstruct</span> <span class="ss">name:</span> <span class="s2">""</span><span class="p">,</span> <span class="ss">age:</span> <span class="mi">0</span> <span class="k">end</span> <span class="p">%</span><span class="no">User</span><span class="p">{</span><span class="ss">name:</span> <span class="s2">"jose"</span><span class="p">}</span> <span class="c1">#=&gt; %User{name: "jose", age: 0}</span> </code></pre></div></div> <p>Structs allow us to provide default values for a map fields. Structs also validate, at compilation time, that all fields given during the struct construction are valid fields. The following, for example, would fail:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">%</span><span class="no">User</span><span class="p">{</span><span class="ss">unknown:</span> <span class="s2">"field"</span><span class="p">}</span> <span class="c1">#=&gt; ** (CompileError) iex:2: unknown key :unknown for struct User</span> </code></pre></div></div> <p>We say structs are <em>raw</em> because they do not implement any of the protocols that are implemented for maps by default. For instance, we can call <code class="language-plaintext highlighter-rouge">Enum.each/2</code> for a map, which uses the <a href="https://hexdocs.pm/elixir/Enumerable.html"><code class="language-plaintext highlighter-rouge">Enumerable</code></a> protocol:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="no">Enum</span><span class="o">.</span><span class="n">each</span> <span class="p">%{</span><span class="ss">foo:</span> <span class="ss">:bar</span><span class="p">},</span> <span class="k">fn</span> <span class="p">{</span><span class="n">k</span><span class="p">,</span> <span class="n">v</span><span class="p">}</span> <span class="o">-&gt;</span> <span class="no">IO</span><span class="o">.</span><span class="n">puts</span> <span class="s2">"Got </span><span class="si">#{</span><span class="n">k</span><span class="si">}</span><span class="s2">: </span><span class="si">#{</span><span class="n">v</span><span class="si">}</span><span class="s2">"</span> <span class="k">end</span> <span class="c1">#=&gt; Got foo: bar</span> </code></pre></div></div> <p>But such a call would fail for the User struct:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="no">Enum</span><span class="o">.</span><span class="n">each</span> <span class="p">%</span><span class="no">User</span><span class="p">{},</span> <span class="k">fn</span> <span class="p">{</span><span class="n">k</span><span class="p">,</span> <span class="n">v</span><span class="p">}</span> <span class="o">-&gt;</span> <span class="no">IO</span><span class="o">.</span><span class="n">puts</span> <span class="s2">"Got </span><span class="si">#{</span><span class="n">k</span><span class="si">}</span><span class="s2">: </span><span class="si">#{</span><span class="n">v</span><span class="si">}</span><span class="s2">"</span> <span class="k">end</span> <span class="c1">#=&gt; ** (Protocol.UndefinedError) protocol Enumerable not implemented for %User{age: 0, name: ""}</span> </code></pre></div></div> <p>However, in many situations, we may want to enumerate the keys and values in a struct. Before this release, it would require us to manually implement the <code class="language-plaintext highlighter-rouge">Enumerable</code> protocol for every struct.</p> <p>Elixir v0.14.0 solves this issue by introducing <code class="language-plaintext highlighter-rouge">@derive</code> which allows us to dynamically derive implementations for structs based on the implementation for maps:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">User</span> <span class="k">do</span> <span class="nv">@derive</span> <span class="p">[</span><span class="no">Enumerable</span><span class="p">]</span> <span class="k">defstruct</span> <span class="ss">name:</span> <span class="s2">""</span><span class="p">,</span> <span class="ss">age:</span> <span class="mi">0</span> <span class="k">end</span> <span class="no">Enum</span><span class="o">.</span><span class="n">each</span> <span class="p">%</span><span class="no">User</span><span class="p">{</span><span class="ss">name:</span> <span class="s2">"jose"</span><span class="p">},</span> <span class="k">fn</span> <span class="p">{</span><span class="n">k</span><span class="p">,</span> <span class="n">v</span><span class="p">}</span> <span class="o">-&gt;</span> <span class="no">IO</span><span class="o">.</span><span class="n">puts</span> <span class="s2">"Got </span><span class="si">#{</span><span class="n">k</span><span class="si">}</span><span class="s2">: </span><span class="si">#{</span><span class="n">v</span><span class="si">}</span><span class="s2">"</span> <span class="k">end</span> <span class="c1">#=&gt; Got __struct__: Elixir.User</span> <span class="c1">#=&gt; Got name: jose</span> <span class="c1">#=&gt; Got age: 0</span> </code></pre></div></div> <p>The deriving functionality can be customized by implementing <code class="language-plaintext highlighter-rouge">PROTOCOL.Map.__deriving__/3</code>. For example, a JSON protocol could define a <code class="language-plaintext highlighter-rouge">JSON.Map.__deriving__/3</code> function that derives specific implementations for every struct. Such implementations could access the struct fields and generate a JSON template at compilation time, avoiding work at runtime.</p> <h2 id="protocol-consolidation">Protocol consolidation</h2> <p>In the example above, when we called <code class="language-plaintext highlighter-rouge">Enum.each/2</code>, it invoked the <code class="language-plaintext highlighter-rouge">Enumerable</code> protocol internally, which then checks if there is an implementation available for the <code class="language-plaintext highlighter-rouge">User</code> struct. This means that dispatching a protocol incurs some overhead to check if the implementation is available or not.</p> <p>While this behaviour is useful for development, in production all implementations for all protocols are usually known up-front, so we could avoid this overhead by doing a direct dispatch. That’s exactly what protocol consolidation does.</p> <p>Protocol consolidation checks all code paths in your project, looking for all protocols and all implementions. When all implementations are found it will recompile all protocols to have quick dispatch rules.</p> <p>You can invoke <code class="language-plaintext highlighter-rouge">Protocol.consolidate/2</code> to manually consolidate protocols. However, if you are inside a Mix project, you can consolidate all protocols by simply invoking <code class="language-plaintext highlighter-rouge">mix compile.protocols</code>. Type <code class="language-plaintext highlighter-rouge">mix help compile.protocols</code> in your terminal to get more information. This task should be invoked when deploying to production and we have opened issues in the Elixir buildpack as well as in the existing build tools so they automatically consolidade protocols for you.</p> <h2 id="nested-access">Nested access</h2> <p>With v0.14.0, Elixir introduced functionality to traverse deeply nested data structures. To show this functionality let’s imagine we have a gaming application where users can be in different dungeons. Each dungeon may have many rooms and users can talk to each other only if they are in the same room.</p> <p>We can model the game’s dungeons with a Dungeon struct:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">%</span><span class="no">Dungeon</span><span class="p">{</span><span class="ss">name:</span> <span class="s2">""</span><span class="p">,</span> <span class="ss">rooms:</span> <span class="p">%{}}</span> </code></pre></div></div> <p>The Dungeon can have many rooms and we keep them in a map, with the room id as key, and the Room struct as value:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">%</span><span class="no">Room</span><span class="p">{</span><span class="ss">id:</span> <span class="mi">0</span><span class="p">,</span> <span class="ss">users:</span> <span class="no">HashDict</span><span class="o">.</span><span class="n">new</span><span class="p">}</span> </code></pre></div></div> <p>A room has users and since we can possibly have hundreds of them, we store them in a HashDict, with the user id as key. Finally, the user may participate from different devices, so we need to keep a set of device codes in each user:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">%</span><span class="no">User</span><span class="p">{</span><span class="ss">id:</span> <span class="mi">0</span><span class="p">,</span> <span class="ss">username:</span> <span class="s2">""</span><span class="p">,</span> <span class="ss">device_codes:</span> <span class="no">HashSet</span><span class="o">.</span><span class="n">new</span><span class="p">}</span> </code></pre></div></div> <p>The nested access functionality that ships with Elixir allows us to quickly access or update a nested value. For example, given a dungeon named <code class="language-plaintext highlighter-rouge">dungeon</code>, we can access all the device codes for a given user as follow:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">dungeon</span><span class="o">.</span><span class="n">rooms</span><span class="p">[</span><span class="n">room_id</span><span class="p">]</span><span class="o">.</span><span class="n">users</span><span class="p">[</span><span class="n">user_id</span><span class="p">]</span><span class="o">.</span><span class="n">device_codes</span> </code></pre></div></div> <p>With the nested access functionality, we can now update a nested path directly too. For example, if a user changes his name, we can change it directly with:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">put_in</span> <span class="n">dungeon</span><span class="o">.</span><span class="n">rooms</span><span class="p">[</span><span class="n">room_id</span><span class="p">]</span><span class="o">.</span><span class="n">users</span><span class="p">[</span><span class="n">user_id</span><span class="p">]</span><span class="o">.</span><span class="n">username</span><span class="p">,</span> <span class="s2">"new username"</span> </code></pre></div></div> <p>If the user signs out from one device, we can delete the code from that particular device:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">update_in</span> <span class="n">dungeon</span><span class="o">.</span><span class="n">rooms</span><span class="p">[</span><span class="n">room_id</span><span class="p">]</span><span class="o">.</span><span class="n">users</span><span class="p">[</span><span class="n">user_id</span><span class="p">]</span><span class="o">.</span><span class="n">device_codes</span><span class="p">,</span> <span class="o">&amp;</span><span class="no">Set</span><span class="o">.</span><span class="n">delete</span><span class="p">(</span><span class="nv">&amp;1</span><span class="p">,</span> <span class="n">code</span><span class="p">)</span> </code></pre></div></div> <p><code class="language-plaintext highlighter-rouge">put_in/2</code> and <code class="language-plaintext highlighter-rouge">update_in/2</code> are macros that work as syntax sugar for the lower-level <code class="language-plaintext highlighter-rouge">put_in/3</code> and <code class="language-plaintext highlighter-rouge">update_in/3</code>, that expects the whole path to be given as a list:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">put_in</span> <span class="n">dungeon</span><span class="p">,</span> <span class="p">[</span><span class="ss">:rooms</span><span class="p">,</span> <span class="n">room_id</span><span class="p">,</span> <span class="ss">:users</span><span class="p">,</span> <span class="n">user_id</span><span class="p">,</span> <span class="ss">:username</span><span class="p">],</span> <span class="s2">"new username"</span> </code></pre></div></div> <p>You can read more information about nested access in <a href="https://hexdocs.pm/elixir/Access.html">the Access protocol documentation</a> and in the docs for <a href="https://hexdocs.pm/elixir/Kernel.html#get_in/2"><code class="language-plaintext highlighter-rouge">get_in/2</code></a> and friends.</p> <h2 id="mix-and-otp">Mix and OTP</h2> <p>OTP is a set of libraries that ships with Erlang. Erlang developers use OTP to build robust, fault-tolerant applications.</p> <p>In v0.14.0, Elixir closely integrates with OTP by providing modules for building <a href="https://hexdocs.pm/elixir/GenServer.html">servers</a>, <a href="https://hexdocs.pm/elixir/Supervisor.html">supervisors</a> and <a href="https://hexdocs.pm/elixir/Application.html">applications</a>.</p> <p>We have also introduced the concepts of <a href="https://hexdocs.pm/elixir/Agent.html">agents</a> and the idea of <a href="https://hexdocs.pm/elixir/Task.html">tasks</a>, which can be supervised and distributed. Application configuration has been made first class in Mix, allowing developers to configure their dependencies, sometimes even using different configurations per environment (dev, test or prod by default).</p> <p>This functionality is at the core of building applications in Erlang and Elixir. For this reason we have published a new guide called <a href="https://hexdocs.pm/elixir/introduction-to-mix.html">Mix and OTP</a> where we build a distributed key-value store to help explore all concepts mentioned above. The guide is quite fresh, so please do submit pull requests for typos and mistakes. Feedback is also welcome!</p> <p>Note “Mix and OTP” is our most advanced guide so far and it expects you to have read our introductory guide. In case you haven’t yet, you can <a href="https://hexdocs.pm/elixir/introduction.html">get started here</a>.</p> <h2 id="whats-next">What’s next?</h2> <p>With v0.14.0 we have reached many of the milestones <a href="/blog/2014/04/21/elixir-v0-13-0-released/#the-next-steps">we have set in the previous release</a>. This brings us closer to Elixir v1.0 and only a handful of tasks are pending:</p> <ul> <li> <p>Provide an Elixir logger that knows how to print and format Elixir exceptions and stacktraces. Work has already started on this front as Elixir already prints errors coming from the application startup nicely;</p> </li> <li> <p>Continue the work of cleaning up the <a href="https://hexdocs.pm/elixir/Kernel.html">Kernel module</a>. In v0.14.0, we added alternatives for conversion functions, like <code class="language-plaintext highlighter-rouge">integer_to_binary/1</code> to <code class="language-plaintext highlighter-rouge">Integer.to_string/1</code>, now they must be properly deprecated and removed;</p> </li> <li> <p>Support mix aliases, allowing developers to easily define Mix shortcuts for their favorite tasks;</p> </li> <li> <p>Solve all remaining <a href="https://github.com/elixir-lang/elixir/issues?state=open">open issues</a>. We have always kept the issues tracker tidy and there is little work left to solve the existing issues. Note we have also listed all <a href="https://github.com/elixir-lang/elixir/issues?labels=Note%3ABackwards+incompatible&amp;page=1&amp;state=open">upcoming backwards incompatible changes</a>. Many of those changes will actually be deprecated first and developers should be able to follow along without breaking changes in minor releases, but they are breaking changes in the sense they work in v0.14.0 but will work differently by the time v1.0 is released;</p> </li> </ul> <p>That’s all for now! Elixir developers can see <a href="https://github.com/elixir-lang/elixir/releases/tag/v0.14.0">a summary of all changes in v0.14.0 in the release notes</a>. In case you are new around here, you can get started with Elixir by reading <a href="https://hexdocs.pm/elixir/introduction.html">our Getting Started guide</a>.</p> <p>We hope to see you all this July at <a href="http://elixirconf.com/">ElixirConf</a>!</p> José Valim 2014-04-21T00:00:00+00:00 /blog/2014/04/21/elixir-v0-13-0-released <p>Hello folks!</p> <p>Elixir v0.13.0 has been released. It contains changes that will effectively shape how developers will write Elixir code from now on, making it an important milestone towards v1.0! On this post we are going to cover some of those changes, the road to Elixir v1.0, as well as the announcement of <a href="https://hex.pm">hex.pm</a>.</p> <p>Before we go into the changes, let’s briefly talk about ElixirConf!</p> <h2 id="elixirconf">ElixirConf</h2> <p>We are excited to announce <a href="http://elixirconf.com">ElixirConf</a>, the first ever Elixir conference, happening July 25-26, 2014 in Austin, TX. The Call For Proposals is open and we are waiting for your talks!</p> <p>The registration is also open and we hope you will join us on this exciting event. We welcome Elixir developers and enthusiasts that are looking forward to be part of our thrilling community!</p> <h2 id="summary">Summary</h2> <p>In a nutshell, here is what new:</p> <ul> <li> <p>Elixir now runs on and requires Erlang R17;</p> </li> <li> <p>With Erlang R17, Elixir also adds support for maps, which are key-value data structures that supports pattern matching. We’ll explore maps, their features and limitations in this post;</p> </li> <li> <p>Elixir v0.13 also provides structs, an alternative to Elixir records. Structs are more flexible than records, provide faster polymorphic operations, and still provide the same compile-time guarantees many came to love in records;</p> </li> <li> <p>The <a href="https://hexdocs.pm/elixir/introduction.html">Getting Started guide</a> was rewritten from scratch. The previous guide was comprised of 7 chapters and was about to become 2 years old. The new guide features 20 chapters, it explores the new maps and structs (which are part of this release), and it goes deeper into topics like IO and File handling. It also includes an extra guide, still in development, about <a href="https://hexdocs.pm/elixir/quote-and-unquote.html">Meta-Programming in Elixir</a>;</p> </li> <li> <p>Elixir v0.13 provides a new comprehension syntax that not only works with lists, but with any <a href="https://hexdocs.pm/elixir/Enumerable.html"><code class="language-plaintext highlighter-rouge">Enumerable</code></a>. The output of a comprehension is also extensible via the <a href="https://hexdocs.pm/elixir/Collectable.html"><code class="language-plaintext highlighter-rouge">Collectable</code></a> protocol;</p> </li> <li> <p>Mix, Elixir’s build tool, has been improved in order to provide better workflows when compiling projects and working with dependencies;</p> </li> <li> <p>There are many other changes, like the addition of <a href="https://hexdocs.pm/elixir/StringIO.html">StringIO</a>, support for <a href="https://hexdocs.pm/ex_unit/ExUnit.Case.html">tags and filters in ExUnit</a> and more. Please check the <a href="https://github.com/elixir-lang/elixir/blob/v0.13.0/CHANGELOG.md">CHANGELOG</a> for the complete list.</p> </li> </ul> <p>Even with all those improvements, Elixir v0.13.0 is backwards compatible with Elixir v0.12.5 and upgrading should be a clean process.</p> <h2 id="maps">Maps</h2> <p>Maps are key-value data structures:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">iex</span><span class="o">&gt;</span> <span class="n">map</span> <span class="o">=</span> <span class="p">%{</span><span class="s2">"hello"</span> <span class="o">=&gt;</span> <span class="ss">:world</span><span class="p">}</span> <span class="p">%{</span><span class="s2">"hello"</span> <span class="o">=&gt;</span> <span class="ss">:world</span><span class="p">}</span> <span class="n">iex</span><span class="o">&gt;</span> <span class="n">map</span><span class="p">[</span><span class="s2">"hello"</span><span class="p">]</span> <span class="ss">:world</span> <span class="n">iex</span><span class="o">&gt;</span> <span class="n">map</span><span class="p">[</span><span class="ss">:other</span><span class="p">]</span> <span class="no">nil</span> </code></pre></div></div> <p>Maps do not have a explicit ordering and keys and values can be any term.</p> <p>Maps can be pattern matched on:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">iex</span><span class="o">&gt;</span> <span class="p">%{</span><span class="s2">"hello"</span> <span class="o">=&gt;</span> <span class="n">world</span><span class="p">}</span> <span class="o">=</span> <span class="n">map</span> <span class="p">%{</span><span class="s2">"hello"</span> <span class="o">=&gt;</span> <span class="ss">:world</span><span class="p">}</span> <span class="n">iex</span><span class="o">&gt;</span> <span class="n">world</span> <span class="ss">:world</span> <span class="n">iex</span><span class="o">&gt;</span> <span class="p">%{}</span> <span class="o">=</span> <span class="n">map</span> <span class="p">%{</span><span class="s2">"hello"</span> <span class="o">=&gt;</span> <span class="ss">:world</span><span class="p">}</span> <span class="n">iex</span><span class="o">&gt;</span> <span class="p">%{</span><span class="s2">"other"</span> <span class="o">=&gt;</span> <span class="n">value</span><span class="p">}</span> <span class="o">=</span> <span class="n">map</span> <span class="o">**</span> <span class="p">(</span><span class="no">MatchError</span><span class="p">)</span> <span class="n">no</span> <span class="n">match</span> <span class="n">of</span> <span class="n">right</span> <span class="n">hand</span> <span class="n">side</span> <span class="n">value</span> </code></pre></div></div> <p>A map pattern will match any map that has all the keys specified in the pattern. The values for the matching keys must also match. For example, <code class="language-plaintext highlighter-rouge">%{"hello" =&gt; world}</code> will match any map that has the key <code class="language-plaintext highlighter-rouge">"hello"</code> and assign the value to <code class="language-plaintext highlighter-rouge">world</code>, while <code class="language-plaintext highlighter-rouge">%{"hello" =&gt; "world"}</code> will match any map that has the key <code class="language-plaintext highlighter-rouge">"hello"</code> with value equals to <code class="language-plaintext highlighter-rouge">"world"</code>. An empty map pattern (<code class="language-plaintext highlighter-rouge">%{}</code>) will match all maps.</p> <p>Developers can use the functions in the <a href="https://hexdocs.pm/elixir/Map.html"><code class="language-plaintext highlighter-rouge">Map</code> module</a> to work with maps. For more information on maps and how they compare to other associative data structures in the language, please check the <a href="https://hexdocs.pm/elixir/keywords-and-maps.html">Maps chapter in our new Getting Started guide</a>. Elixir Sips has also released two episodes that cover maps (<a href="http://elixirsips.com/episodes/054_maps_part_1.html">part 1</a> and <a href="http://elixirsips.com/episodes/055_maps_part_2.html">part 2</a>).</p> <p>Maps also provide special syntax for creating, accessing and updating maps with atom keys:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">iex</span><span class="o">&gt;</span> <span class="n">user</span> <span class="o">=</span> <span class="p">%{</span><span class="ss">name:</span> <span class="s2">"john"</span><span class="p">,</span> <span class="ss">age:</span> <span class="mi">27</span><span class="p">}</span> <span class="p">%{</span><span class="ss">name:</span> <span class="s2">"john"</span><span class="p">,</span> <span class="ss">age:</span> <span class="mi">27</span><span class="p">}</span> <span class="n">iex</span><span class="o">&gt;</span> <span class="n">user</span><span class="o">.</span><span class="n">name</span> <span class="s2">"john"</span> <span class="n">iex</span><span class="o">&gt;</span> <span class="n">user</span> <span class="o">=</span> <span class="p">%{</span><span class="n">user</span> <span class="o">|</span> <span class="ss">name:</span> <span class="s2">"meg"</span><span class="p">}</span> <span class="p">%{</span><span class="ss">name:</span> <span class="s2">"meg"</span><span class="p">,</span> <span class="ss">age:</span> <span class="mi">27</span><span class="p">}</span> <span class="n">iex</span><span class="o">&gt;</span> <span class="n">user</span><span class="o">.</span><span class="n">name</span> <span class="s2">"meg"</span> </code></pre></div></div> <p>Both access and update syntax above expect the given keys to exist. Trying to access or update a key that does not exist raises an error:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">iex</span><span class="o">&gt;</span> <span class="p">%{</span> <span class="n">user</span> <span class="o">|</span> <span class="ss">address:</span> <span class="p">[]</span> <span class="p">}</span> <span class="o">**</span> <span class="p">(</span><span class="no">ArgumentError</span><span class="p">)</span> <span class="n">argument</span> <span class="n">error</span> <span class="ss">:maps</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="ss">:address</span><span class="p">,</span> <span class="p">[],</span> <span class="p">%{})</span> </code></pre></div></div> <p>As we will see, this functionality becomes very useful when working with structs.</p> <h2 id="structs">Structs</h2> <p>Structs are meant to replace Elixir records. Records in Elixir are simply tuples supported by modules which store record metadata:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defrecord</span> <span class="no">User</span><span class="p">,</span> <span class="ss">name:</span> <span class="no">nil</span><span class="p">,</span> <span class="ss">age:</span> <span class="mi">0</span> </code></pre></div></div> <p>Internally, this record is represented as the following tuple:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># {tag, name, age}</span> <span class="p">{</span><span class="no">User</span><span class="p">,</span> <span class="no">nil</span><span class="p">,</span> <span class="mi">0</span><span class="p">}</span> </code></pre></div></div> <p>Records can also be created and pattern matched on:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">iex</span><span class="o">&gt;</span> <span class="n">user</span> <span class="o">=</span> <span class="no">User</span><span class="p">[</span><span class="ss">name:</span> <span class="s2">"john"</span><span class="p">]</span> <span class="no">User</span><span class="p">[</span><span class="ss">name:</span> <span class="s2">"john"</span><span class="p">,</span> <span class="ss">age:</span> <span class="mi">0</span><span class="p">]</span> <span class="n">iex</span><span class="o">&gt;</span> <span class="n">user</span><span class="o">.</span><span class="n">name</span> <span class="s2">"john"</span> <span class="n">iex</span><span class="o">&gt;</span> <span class="no">User</span><span class="p">[</span><span class="ss">name:</span> <span class="n">name</span><span class="p">]</span> <span class="o">=</span> <span class="n">user</span> <span class="no">User</span><span class="p">[</span><span class="ss">name:</span> <span class="s2">"john"</span><span class="p">,</span> <span class="ss">age:</span> <span class="mi">0</span><span class="p">]</span> <span class="n">iex</span><span class="o">&gt;</span> <span class="n">name</span> <span class="s2">"john"</span> </code></pre></div></div> <p>Pattern matching works because the record meta-data is stored in the User module which can be accessed when building patterns.</p> <p>However, records came with their own issues. First of all, since records were made of data (the underlying tuple) and a module (functions/behaviour), they were frequently misused as an attempt to bundle data and behaviour together in Elixir, for example:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defrecord</span> <span class="no">User</span><span class="p">,</span> <span class="ss">name:</span> <span class="no">nil</span><span class="p">,</span> <span class="ss">age:</span> <span class="mi">0</span> <span class="k">do</span> <span class="k">def</span> <span class="n">first_name</span><span class="p">(</span><span class="n">self</span><span class="p">)</span> <span class="k">do</span> <span class="n">self</span><span class="o">.</span><span class="n">name</span> <span class="o">|&gt;</span> <span class="no">String</span><span class="o">.</span><span class="n">split</span> <span class="o">|&gt;</span> <span class="no">Enum</span><span class="o">.</span><span class="n">at</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="k">end</span> <span class="k">end</span> <span class="no">User</span><span class="p">[</span><span class="ss">name:</span> <span class="s2">"john doe"</span><span class="p">]</span><span class="o">.</span><span class="n">first_name</span> <span class="c1">#=&gt; "john"</span> </code></pre></div></div> <p>Not only that, records were often slow in protocol dispatches because every tuple can potentially be a record, sometimes leading to expensive checks at runtime.</p> <p>Since maps are meant to replace many cases of records in Erlang, we saw with the introduction of maps the perfect opportunity to revisit Elixir records as well. In order to understand the reasoning behind structs, let’s list the features we got from Elixir records:</p> <ol> <li>A way to organize data by fields</li> <li>Efficient in-memory representation and operations</li> <li>Compile-time structures with compile-time errors</li> <li>The basic foundation for polymorphism in Elixir</li> </ol> <p>Maps naturally solve issues <code class="language-plaintext highlighter-rouge">1.</code> and <code class="language-plaintext highlighter-rouge">2.</code> above. In particular, maps that have same keys share the same key-space in memory. That’s why the update operation <code class="language-plaintext highlighter-rouge">%{map | ...}</code> we have seen above is relevant: if we know we are updating an existing key, the new map created as result of the update operation can share the same key space as the old map without extra checks. For more details on why Maps are efficient, I would recommend <a href="http://joearms.github.io/2014/02/01/big-changes-to-erlang.html">reading Joe’s blog post on the matter</a>.</p> <p>Structs were added to address features <code class="language-plaintext highlighter-rouge">3.</code> and <code class="language-plaintext highlighter-rouge">4.</code>. A struct needs to be explicitly defined via <code class="language-plaintext highlighter-rouge">defstruct</code>:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">User</span> <span class="k">do</span> <span class="k">defstruct</span> <span class="ss">name:</span> <span class="no">nil</span><span class="p">,</span> <span class="ss">age:</span> <span class="mi">0</span> <span class="k">end</span> </code></pre></div></div> <p>Now a <code class="language-plaintext highlighter-rouge">User</code> struct can be created without a need to explicitly list all necessary fields:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">iex</span><span class="o">&gt;</span> <span class="n">user</span> <span class="o">=</span> <span class="p">%</span><span class="no">User</span><span class="p">{</span><span class="ss">name:</span> <span class="s2">"john"</span><span class="p">}</span> <span class="p">%</span><span class="no">User</span><span class="p">{</span><span class="ss">name:</span> <span class="s2">"john"</span><span class="p">,</span> <span class="ss">age:</span> <span class="mi">0</span><span class="p">}</span> </code></pre></div></div> <p>Trying to create a struct with an unknown key raises an error during compilation:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">iex</span><span class="o">&gt;</span> <span class="n">user</span> <span class="o">=</span> <span class="p">%</span><span class="no">User</span><span class="p">{</span><span class="ss">address:</span> <span class="p">[]}</span> <span class="o">**</span> <span class="p">(</span><span class="no">CompileError</span><span class="p">)</span> <span class="n">unknown</span> <span class="n">key</span> <span class="ss">:address</span> <span class="n">for</span> <span class="n">struct</span> <span class="no">User</span> </code></pre></div></div> <p>Furthermore, every struct has a <code class="language-plaintext highlighter-rouge">__struct__</code> field which contains the struct name:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">iex</span><span class="o">&gt;</span> <span class="n">user</span><span class="o">.</span><span class="n">__struct__</span> <span class="no">User</span> </code></pre></div></div> <p>The <code class="language-plaintext highlighter-rouge">__struct__</code> field is also used for polymorphic dispatch in protocols, addressing issue <code class="language-plaintext highlighter-rouge">4.</code>.</p> <p>It is interesting to note that structs solve both drawbacks we have earlier mentioned regarding records. Structs are purely data and polymorphic dispatch is now faster and more robust as it happens only for explicitly tagged structs.</p> <p>For more information on structs, check out the <a href="https://hexdocs.pm/elixir/structs.html">Structs chapter in the getting started guide</a> (you may also want to read the new <a href="https://hexdocs.pm/elixir/protocols.html">Protocols chapter</a> after it).</p> <h2 id="maps-structs-and-the-future">Maps, structs and the future</h2> <p>With the introduction of maps and structs, some deprecations will arrive on upcoming releases. First of all, the <code class="language-plaintext highlighter-rouge">ListDict</code> data structure is being deprecated and phased out. Records are also being deprecated from the language, although it is going to be a longer process, as many projects and Elixir itself still use records in diverse occasions.</p> <p>Note though only Elixir records are being deprecated. Erlang records, which are basically syntax sugar around tuples, will remain in the language for the rare cases Elixir developers need to interact with Erlang libraries that provide records. In particular, the <a href="https://hexdocs.pm/elixir/Record.html">Record</a> has been updated to provide the new Record API (while keeping the old one for backwards compatibility).</p> <p>Finally, structs are still in active development and new features, like <code class="language-plaintext highlighter-rouge">@derive</code>, should land in upcoming Elixir releases. For those interested, the <a href="https://gist.github.com/josevalim/b30c881df36801611d13">original maps and structs proposal is still available</a>.</p> <h2 id="comprehensions">Comprehensions</h2> <p>Erlang R17 also introduced recursion to anonymous functions. This feature, while still not available from Elixir, allows Elixir to provide a more flexible and extensible comprehension syntax.</p> <p>The most common use case of a comprehension are <a href="https://en.wikipedia.org/wiki/List_comprehension">list comprehensions</a>. For example, we can get all the square values of elements in a list as follows:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">iex</span><span class="o">&gt;</span> <span class="n">for</span> <span class="n">n</span> <span class="o">&lt;-</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">4</span><span class="p">],</span> <span class="k">do</span><span class="p">:</span> <span class="n">n</span> <span class="o">*</span> <span class="n">n</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">4</span><span class="p">,</span> <span class="mi">9</span><span class="p">,</span> <span class="mi">16</span><span class="p">]</span> </code></pre></div></div> <p>We say the <code class="language-plaintext highlighter-rouge">n &lt;- [1, 2, 3, 4]</code> part is a comprehension generator. In previous Elixir versions, Elixir supported only lists in generators. In Elixir v0.13.0, any Enumerable is supported (ranges, maps, etc):</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">iex</span><span class="o">&gt;</span> <span class="n">for</span> <span class="n">n</span> <span class="o">&lt;-</span> <span class="mi">1</span><span class="o">..</span><span class="mi">4</span><span class="p">,</span> <span class="k">do</span><span class="p">:</span> <span class="n">n</span> <span class="o">*</span> <span class="n">n</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">4</span><span class="p">,</span> <span class="mi">9</span><span class="p">,</span> <span class="mi">16</span><span class="p">]</span> </code></pre></div></div> <p>As in previous Elixir versions, there is also support for a bitstring generator. In the example below, we receive a stream of RGB pixels as a binary and break it down into triplets:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">iex</span><span class="o">&gt;</span> <span class="n">pixels</span> <span class="o">=</span> <span class="o">&lt;&lt;</span><span class="mi">213</span><span class="p">,</span> <span class="mi">45</span><span class="p">,</span> <span class="mi">132</span><span class="p">,</span> <span class="mi">64</span><span class="p">,</span> <span class="mi">76</span><span class="p">,</span> <span class="mi">32</span><span class="p">,</span> <span class="mi">76</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">234</span><span class="p">,</span> <span class="mi">32</span><span class="p">,</span> <span class="mi">15</span><span class="o">&gt;&gt;</span> <span class="n">iex</span><span class="o">&gt;</span> <span class="n">for</span> <span class="o">&lt;&lt;</span><span class="n">r</span><span class="p">::</span><span class="mi">8</span><span class="p">,</span> <span class="n">g</span><span class="p">::</span><span class="mi">8</span><span class="p">,</span> <span class="n">b</span><span class="p">::</span><span class="mi">8</span> <span class="o">&lt;-</span> <span class="n">pixels</span><span class="o">&gt;&gt;</span><span class="p">,</span> <span class="k">do</span><span class="p">:</span> <span class="p">{</span><span class="n">r</span><span class="p">,</span> <span class="n">g</span><span class="p">,</span> <span class="n">b</span><span class="p">}</span> <span class="p">[{</span><span class="mi">213</span><span class="p">,</span> <span class="mi">45</span><span class="p">,</span> <span class="mi">132</span><span class="p">},</span> <span class="p">{</span><span class="mi">64</span><span class="p">,</span> <span class="mi">76</span><span class="p">,</span> <span class="mi">32</span><span class="p">},</span> <span class="p">{</span><span class="mi">76</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">},</span> <span class="p">{</span><span class="mi">234</span><span class="p">,</span> <span class="mi">32</span><span class="p">,</span> <span class="mi">15</span><span class="p">}]</span> </code></pre></div></div> <p>By default, a comprehension returns a list as a result. However the result of a comprehension can be inserted into different data structures by passing the <code class="language-plaintext highlighter-rouge">:into</code> option. For example, we can use bitstring generators with the <code class="language-plaintext highlighter-rouge">:into</code> option to easily remove all spaces in a string:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">iex</span><span class="o">&gt;</span> <span class="n">for</span> <span class="o">&lt;&lt;</span><span class="n">c</span> <span class="o">&lt;-</span> <span class="s2">" hello world "</span><span class="o">&gt;&gt;</span><span class="p">,</span> <span class="n">c</span> <span class="o">!=</span> <span class="err">?</span><span class="p">\</span><span class="n">s</span><span class="p">,</span> <span class="ss">into:</span> <span class="s2">""</span><span class="p">,</span> <span class="k">do</span><span class="p">:</span> <span class="o">&lt;&lt;</span><span class="n">c</span><span class="o">&gt;&gt;</span> <span class="s2">"helloworld"</span> </code></pre></div></div> <p>Sets, maps and other dictionaries can also be given with the <code class="language-plaintext highlighter-rouge">:into</code> option. In general, the <code class="language-plaintext highlighter-rouge">:into</code> accepts any structure as long as it implements the <a href="https://hexdocs.pm/elixir/Collectable.html"><code class="language-plaintext highlighter-rouge">Collectable</code> protocol</a>.</p> <p>For example, the <code class="language-plaintext highlighter-rouge">IO</code> module provides streams, that are both <code class="language-plaintext highlighter-rouge">Enumerable</code> and <code class="language-plaintext highlighter-rouge">Collectable</code>. You can implement an echo terminal that returns whatever is typed into the shell, but in upcase, using comprehensions:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">iex</span><span class="o">&gt;</span> <span class="n">stream</span> <span class="o">=</span> <span class="no">IO</span><span class="o">.</span><span class="n">stream</span><span class="p">(</span><span class="ss">:stdio</span><span class="p">,</span> <span class="ss">:line</span><span class="p">)</span> <span class="n">iex</span><span class="o">&gt;</span> <span class="n">for</span> <span class="n">line</span> <span class="o">&lt;-</span> <span class="n">stream</span><span class="p">,</span> <span class="ss">into:</span> <span class="n">stream</span> <span class="k">do</span> <span class="o">...&gt;</span> <span class="no">String</span><span class="o">.</span><span class="n">upcase</span><span class="p">(</span><span class="n">line</span><span class="p">)</span> <span class="o">&lt;&gt;</span> <span class="s2">"</span><span class="se">\n</span><span class="s2">"</span> <span class="o">...&gt;</span> <span class="k">end</span> </code></pre></div></div> <p>This makes comprehensions useful not only for working with in-memory collections but also with files, io devices, and other sources. In future releases, we will continue exploring how to make comprehensions more expressive, following in the footsteps of other functional programming research on the topic (like Comprehensive Comprehensions and Parallel Comprehensions).</p> <h2 id="mix-workflows">Mix workflows</h2> <p>The last big change we want to discuss in this release are the improvements done to Mix, Elixir’s build tool. Mix is an essential tool to Elixir developers and helps developers to compile their projects, manage their dependencies, run tests and so on.</p> <p>In previous releases, Mix was used to download and compile dependencies per environment. That meant the usual workflow was less than ideal: every time a dependency was updated, developers had to explicitly fetch and compile the dependencies for each environment. The workflow would be something like:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>mix deps.get <span class="nv">$ </span>mix compile <span class="nv">$ MIX_ENV</span><span class="o">=</span><span class="nb">test </span>mix deps.get <span class="nv">$ </span>mix <span class="nb">test</span> </code></pre></div></div> <p>In Elixir v0.13, <code class="language-plaintext highlighter-rouge">mix deps.get</code> only fetches dependencies and it does so across all environments (unless an <code class="language-plaintext highlighter-rouge">--only</code> flag is specified). To support this new behaviour, dependencies now support the <code class="language-plaintext highlighter-rouge">:only</code> option:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="n">deps</span> <span class="k">do</span> <span class="p">[{</span><span class="ss">:ecto</span><span class="p">,</span> <span class="ss">github:</span> <span class="s2">"elixir-lang/ecto"</span><span class="p">},</span> <span class="p">{</span><span class="ss">:hackney</span><span class="p">,</span> <span class="ss">github:</span> <span class="s2">"benoitc/hackney"</span><span class="p">,</span> <span class="ss">only:</span> <span class="p">[</span><span class="ss">:test</span><span class="p">]}]</span> <span class="k">end</span> </code></pre></div></div> <p>Dependencies now are also automatically compiled before you run a command. For example, <code class="language-plaintext highlighter-rouge">mix compile</code> will automatically compile pending dependencies for the current environment. <code class="language-plaintext highlighter-rouge">mix test</code> will do the same for test dependencies and so on, interrupting less the developer workflow.</p> <h2 id="hexpm">hex.pm</h2> <p>This release also marks the announcement of <a href="https://hex.pm/">hex.pm</a>, a package manager for the Erlang VM. Hex allows you to package and publish your projects while fetching them and performing dependency resolution in your applications.</p> <p>Currently Hex only integrates with Mix and contributions to extend it to other tools and other languages in the Erlang VM are welcome!</p> <h2 id="the-next-steps">The next steps</h2> <p>As seen in this announcement, this release dictates many of the developments that will happen in Elixir and its community in the following weeks. All projects are recommended to start moving from records to structs, paving the way for the deprecation of records before 1.0.</p> <p>The next months will also focus on integrating Elixir more tightly to OTP. During the keynote at Erlang Factory, <a href="https://www.youtube.com/watch?v=Djv4C9H9yz4">Catalyse Change</a>, Dave Thomas and I argued that there are many useful patterns, re-implemented everyday by developers, that could make development more productive within the Erlang VM if exposed accordingly.</p> <p>That said, in the next months we plan to:</p> <ul> <li>Integrate applications configuration (provided by OTP) right into Mix;</li> <li>Provide an Elixir logger that knows how to print and format Elixir exceptions and stacktraces;</li> <li>Properly expose the functionality provided by Applications, Supervisors, GenServers and GenEvents and study how they can integrate with Elixir. For example, how to consume events from GenEvent as a <a href="https://hexdocs.pm/elixir/Stream.html">stream of data</a>?</li> <li>Study how patterns like tasks and agents can be integrated into the language, often picking up the lessons learned by libraries like <a href="http://e2project.org/erlang.html">e2</a> and <a href="http://www.erlang.org/doc/man/rpc.html">functionality exposed by OTP itself</a>;</li> <li>Rewrite the Mix and ExUnit guides to focus on applications and OTP as a whole, rebranding it to “Building Apps with Mix and OTP”;</li> </ul> <p>You can learn more about Elixir in our <a href="https://hexdocs.pm/elixir/introduction.html">Getting Started guide</a> and download this release in the <a href="https://github.com/elixir-lang/elixir/releases/tag/v0.13.0">v0.13 announcement</a>. We hope to see you at <a href="http://elixirconf.com/">ElixirConf</a> as well as pushing your packages to <a href="https://hex.pm/">hex.pm</a>.</p> José Valim 2013-12-15T00:00:00+00:00 /blog/2013/12/15/elixir-v0-12-0-released <p>Elixir v0.12.0 has been released with improved enumerables, build patterns and welcoming a new member to our team.</p> <h2 id="enumerables">Enumerables</h2> <p>In previous versions, the Enumerable protocol was based on reduce/fold, and while it is very efficient for operations like <code class="language-plaintext highlighter-rouge">map</code>, <code class="language-plaintext highlighter-rouge">reduce</code> and <code class="language-plaintext highlighter-rouge">filter</code>, it was sub-optimal for operations that need to halt, like <code class="language-plaintext highlighter-rouge">take</code> and <code class="language-plaintext highlighter-rouge">take_while</code>, and it made it impossible for operations like <code class="language-plaintext highlighter-rouge">zip</code> to be implemented.</p> <p>In v0.12.0, Elixir’s Enumerable protocol has been extended to allow suspension and halting mechanisms, making operations like <code class="language-plaintext highlighter-rouge">take</code> simpler and operations that require interleaving, like <code class="language-plaintext highlighter-rouge">zip</code>, possible.</p> <p>Although most users don’t need to concern with the implementation of the Enumerable protocol, the side-effect is that both <code class="language-plaintext highlighter-rouge">Enum</code> and <code class="language-plaintext highlighter-rouge">Stream</code> modules have been considerably extended and improved in this release, with more than 15 new functions added to the <code class="language-plaintext highlighter-rouge">Stream</code> module.</p> <h2 id="mix">Mix</h2> <p>The tool that received most improvements in this release was Mix. The biggest change is that Mix no longer compiles projects in place but to the <code class="language-plaintext highlighter-rouge">_build</code> directory. For example, take the <a href="https://github.com/elixir-ecto/ecto">Ecto project</a> that <a href="https://github.com/elixir-ecto/ecto/blob/v0.1.0/mix.exs#L25-L30">depends on <code class="language-plaintext highlighter-rouge">postgrex</code> and <code class="language-plaintext highlighter-rouge">poolboy</code></a>. When compiled, all the artifacts will be placed in the <code class="language-plaintext highlighter-rouge">_build</code> directory like this:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>_build └── shared └── lib ├── ecto │   └── ebin | └── priv ├── poolboy │   └── ebin └── postgrex └── ebin </code></pre></div></div> <p>For those familiar with Erlang’s OTP, this is similar to the structure used by OTP when releasing your software. So this new structure makes our software one step close to production and guarantee it is designed correctly since day 1.</p> <p>This approach comes with the <code class="language-plaintext highlighter-rouge">:build_per_environment</code> option which, when set to true, creates a different build per environment (<code class="language-plaintext highlighter-rouge">dev</code>, <code class="language-plaintext highlighter-rouge">test</code>, <code class="language-plaintext highlighter-rouge">production</code> or more). Extremely useful when a project compile different artifacts depending on the compilation environment.</p> <p>Mix has also added support to optional dependencies and improved common patterns, like the usage of umbrella apps.</p> <h2 id="welcome-eric">Welcome, Eric!</h2> <p>With this release, we also want to welcome <a href="https://github.com/ericmj">Eric MJ</a> to the Elixir Team. He has done fantastic work on Elixir, helping us maintain the codebase and working on many of the important features from previous releases and now many more to come.</p> <p>Eric is also maintainer of both <a href="https://github.com/elixir-ecto/ecto">Ecto</a> and <a href="https://github.com/elixir-ecto/postgrex">Postgrex</a> projects. Which are proving to be very useful to the Elixir community too!</p> <h2 id="tidying-up">Tidying up</h2> <p>There were other small changes, like additions to the <code class="language-plaintext highlighter-rouge">Float</code> module and improvements the to the typespec syntax. To see the full list, please <a href="https://github.com/elixir-lang/elixir/blob/v0.12.0/CHANGELOG.md">see the CHANGELOG</a>.</p> <p>Give Elixir a try! You can start with our <a href="https://hexdocs.pm/elixir/introduction.html">getting started guide</a>, or check out our sidebar for other learning resources.</p> Peter Minten 2013-12-11T00:00:00+00:00 /blog/2013/12/11/elixir-s-new-continuable-enumerators <p>As you may have heard in the upcoming 0.12.0 release Elixir’s enumerators gained some new features. In this blog post I’ll explain what’s new, what it enables and how it works.</p> <p>For those of you who use the development version of Elixir these changes are already available. For the exact differences in code you can look at the <a href="https://github.com/elixir-lang/elixir/pull/1922">relevant pull request</a>.</p> <h2 id="a-recap-of-enumerators-and-some-terminology">A recap of enumerators, and some terminology</h2> <p>The basic idea of enumerators is that you traverse some data structure or resource (lines from a file) by putting the thing that is traversed in control. That is if you’re reading from a file you have a loop that reads lines from a file and for each line calls a function. Just calling a function isn’t all that useful for most tasks as there’d be no way to remember previous lines (ugly hacks aside), so some accumulator value is passed to the function and a new accumulator is returned by it.</p> <p>For example here’s how you can count the total length of strings in a list.</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="no">Enumerable</span><span class="o">.</span><span class="n">reduce</span><span class="p">(</span><span class="n">l</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="k">fn</span> <span class="n">x</span><span class="p">,</span> <span class="n">acc</span> <span class="o">-&gt;</span> <span class="no">String</span><span class="o">.</span><span class="n">length</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="o">+</span> <span class="n">acc</span> <span class="k">end</span><span class="p">)</span> </code></pre></div></div> <p>Often the actual call to <code class="language-plaintext highlighter-rouge">Enumerable.reduce/3</code> is hidden inside another function. Say that we want to define a <code class="language-plaintext highlighter-rouge">sum</code> function. The usual way is to write it like this:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="n">sum</span><span class="p">(</span><span class="n">coll</span><span class="p">)</span> <span class="k">do</span> <span class="no">Enumerable</span><span class="o">.</span><span class="n">reduce</span><span class="p">(</span><span class="n">coll</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="k">fn</span> <span class="n">x</span><span class="p">,</span> <span class="n">acc</span> <span class="o">-&gt;</span> <span class="n">x</span> <span class="o">+</span> <span class="n">acc</span> <span class="k">end</span><span class="p">)</span> <span class="k">end</span> </code></pre></div></div> <p>This could get called as <code class="language-plaintext highlighter-rouge">Enum.map(1..10, &amp;(&amp;1 * &amp;1)) |&gt; sum()</code> to get the sum of squares. Desugaring this means <code class="language-plaintext highlighter-rouge">sum(Enum.map(1..10, &amp;(&amp;1 * &amp;1)))</code>.</p> <p>The general pattern is this:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="n">outer_function</span><span class="p">(</span><span class="n">coll</span><span class="p">,</span> <span class="o">...</span><span class="p">)</span> <span class="k">do</span> <span class="o">...</span> <span class="no">Enumerable</span><span class="o">.</span><span class="n">reduce</span><span class="p">(</span><span class="n">coll</span><span class="p">,</span> <span class="n">initial_consumer_acc</span><span class="p">,</span> <span class="n">consumer</span><span class="p">)</span> <span class="o">...</span> <span class="k">end</span> <span class="n">something_that_returns_an_enumerable</span><span class="p">(</span><span class="o">...</span><span class="p">)</span> <span class="o">|&gt;</span> <span class="n">outer_function</span><span class="p">(</span><span class="o">...</span><span class="p">)</span> </code></pre></div></div> <p>You’ll notice the slightly uncommon terminology of “outer function” and “consumer” (normally called an “iteratee”). That’s intentional, naming an iteratee a consumer better reflects that it consumes values.</p> <p>Along the same lines I call the reduce function for a specific enumerable a producer, it produces values which are given to a consumer.</p> <p>The outer function is the function to which the enumerable is passed. Syntactically it looks like this is the consumer, but it’s really a function that combines the producer and the consumer. For simple consumers (say <code class="language-plaintext highlighter-rouge">fn x, acc -&gt; length(x) + acc end</code>) the consumer will often be written directly in the source text of the outer function, but let’s try to keep those concepts distinguished.</p> <h2 id="two-issues-with-classic-elixir-enumerators">Two issues with classic Elixir enumerators</h2> <p>Enumerators are great, but they have their limitations. One issue is that it’s not possible to define a function that only returns at most 3 elements without traversing all elements or using ugly tricks such as <code class="language-plaintext highlighter-rouge">throw</code> (with a <code class="language-plaintext highlighter-rouge">try...catch</code> construct in the outer function). The <code class="language-plaintext highlighter-rouge">throw</code> trick is used in <code class="language-plaintext highlighter-rouge">Enum</code> and <code class="language-plaintext highlighter-rouge">Stream</code> to implement functions such as <code class="language-plaintext highlighter-rouge">Enum.take/2</code> and <code class="language-plaintext highlighter-rouge">Stream.take_while/2</code>. It works, but it’s not what I’d call stylish.</p> <p>A bigger problem, that doesn’t have a workaround, is that there’s no way to interleave two enumerables. That is, it’s not possible to define a function that for two enumerables <code class="language-plaintext highlighter-rouge">A</code> and <code class="language-plaintext highlighter-rouge">B</code> returns a list <code class="language-plaintext highlighter-rouge">[A1, B1, A2, B2, A3, ...]</code> (where <code class="language-plaintext highlighter-rouge">A1</code> is the first element of A) without first traversing both lists and then interleaving the collected values. Interleaving is important because it’s the basis of a zip function. Without interleaving you cannot implement <code class="language-plaintext highlighter-rouge">Stream.zip/2</code>.</p> <p>The underlying problem, in both cases, is that the producer is fully in control. The producer simply pushes out as many elements to the consumer as it wants and then says “I’m done”. There’s no way aside from <code class="language-plaintext highlighter-rouge">throw/raise</code> for a consumer to tell a producer “stop producing”. There is definitely no way to tell a producer “stop for now but be prepared to continue where you left off later”.</p> <h2 id="power-to-the-consumer">Power to the consumer!</h2> <p>At CodeMeshIO José Valim and Jessica Kerr sat down and discussed this problem. They came up with a solution inspired by a <a href="http://themonadreader.files.wordpress.com/2010/05/issue16.pdf">Monad.Reader article</a> (third article). It’s an elegant extension of the old system, based on a simple idea. Instead of returning only an accumulator at every step (for every produced value) the consumer returns a combination of an accumulator and an instruction to the producer. Three instructions are available:</p> <ul> <li><code class="language-plaintext highlighter-rouge">:cont</code> - Keep producing.</li> <li><code class="language-plaintext highlighter-rouge">:halt</code> - Stop producing.</li> <li><code class="language-plaintext highlighter-rouge">:suspend</code> - Temporarily stop producing.</li> </ul> <p>A consumer that always returns <code class="language-plaintext highlighter-rouge">:cont</code> makes the producer behave exactly the same as in the old system. A consumer may return <code class="language-plaintext highlighter-rouge">:halt</code> to have the producer terminate earlier than it normally would.</p> <p>The real magic is in <code class="language-plaintext highlighter-rouge">:suspend</code> though. It tells a producer to return the accumulator and a continuation function.</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span> <span class="ss">:suspended</span><span class="p">,</span> <span class="n">n_</span><span class="p">,</span> <span class="n">cont</span> <span class="p">}</span> <span class="o">=</span> <span class="no">Enumerable</span><span class="o">.</span><span class="n">reduce</span><span class="p">(</span><span class="mi">1</span><span class="o">..</span><span class="mi">5</span><span class="p">,</span> <span class="p">{</span> <span class="ss">:cont</span><span class="p">,</span> <span class="mi">0</span> <span class="p">},</span> <span class="k">fn</span> <span class="n">x</span><span class="p">,</span> <span class="n">n</span> <span class="o">-&gt;</span> <span class="k">if</span> <span class="n">x</span> <span class="o">==</span> <span class="mi">3</span> <span class="k">do</span> <span class="p">{</span> <span class="ss">:suspend</span><span class="p">,</span> <span class="n">n</span> <span class="p">}</span> <span class="k">else</span> <span class="p">{</span> <span class="ss">:cont</span><span class="p">,</span> <span class="n">n</span> <span class="o">+</span> <span class="n">x</span> <span class="p">}</span> <span class="k">end</span> <span class="k">end</span><span class="p">)</span> </code></pre></div></div> <p>After running this code <code class="language-plaintext highlighter-rouge">n_</code> will be <code class="language-plaintext highlighter-rouge">3</code> (1 + 2) and <code class="language-plaintext highlighter-rouge">cont</code> will be a function. We’ll get back to <code class="language-plaintext highlighter-rouge">cont</code> in a minute but first take a look at some of the new elements here. The initial accumulator has an instruction as well, so you could suspend or halt a producer immediately, if you really want to. The value passed to the consumer (<code class="language-plaintext highlighter-rouge">n</code>) does not contain the instruction. The return value of the producer also has a symbol in it. Like with the instructions of consumers there are three possible values:</p> <ul> <li><code class="language-plaintext highlighter-rouge">:done</code> - Completed normally.</li> <li><code class="language-plaintext highlighter-rouge">:halted</code> - Consumer returned a <code class="language-plaintext highlighter-rouge">:halt</code> instruction.</li> <li><code class="language-plaintext highlighter-rouge">:suspended</code> - Consumer return a <code class="language-plaintext highlighter-rouge">:suspend</code> instruction.</li> </ul> <p>Together with the other values returned the possible return values from a producer are <code class="language-plaintext highlighter-rouge">{ :done, acc } | { :halted, acc } | { :suspended, acc, continuation }</code>.</p> <p>Back to the continuation. A continuation is a function that given an accumulator returns a new producer result. In other words it’s a way to swap out the accumulator but keep the same producer in the same state.</p> <h2 id="implementing-interleave">Implementing <code class="language-plaintext highlighter-rouge">interleave</code></h2> <p>Using the power of suspension it is now possible to create an interleave function.</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">Interleave</span> <span class="k">do</span> <span class="k">def</span> <span class="n">interleave</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">)</span> <span class="k">do</span> <span class="n">step</span> <span class="o">=</span> <span class="k">fn</span> <span class="n">x</span><span class="p">,</span> <span class="n">acc</span> <span class="o">-&gt;</span> <span class="p">{</span> <span class="ss">:suspend</span><span class="p">,</span> <span class="p">[</span><span class="n">x</span> <span class="o">|</span> <span class="n">acc</span><span class="p">]</span> <span class="p">}</span> <span class="k">end</span> <span class="n">af</span> <span class="o">=</span> <span class="o">&amp;</span><span class="no">Enumerable</span><span class="o">.</span><span class="n">reduce</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="nv">&amp;1</span><span class="p">,</span> <span class="n">step</span><span class="p">)</span> <span class="n">bf</span> <span class="o">=</span> <span class="o">&amp;</span><span class="no">Enumerable</span><span class="o">.</span><span class="n">reduce</span><span class="p">(</span><span class="n">b</span><span class="p">,</span> <span class="nv">&amp;1</span><span class="p">,</span> <span class="n">step</span><span class="p">)</span> <span class="n">do_interleave</span><span class="p">(</span><span class="n">af</span><span class="p">,</span> <span class="n">bf</span><span class="p">,</span> <span class="p">[])</span> <span class="o">|&gt;</span> <span class="ss">:lists</span><span class="o">.</span><span class="n">reverse</span><span class="p">()</span> <span class="k">end</span> <span class="k">defp</span> <span class="n">do_interleave</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">,</span> <span class="n">acc</span><span class="p">)</span> <span class="k">do</span> <span class="k">case</span> <span class="n">a</span><span class="o">.</span><span class="p">({</span> <span class="ss">:cont</span><span class="p">,</span> <span class="n">acc</span> <span class="p">})</span> <span class="k">do</span> <span class="p">{</span> <span class="ss">:suspended</span><span class="p">,</span> <span class="n">acc</span><span class="p">,</span> <span class="n">a</span> <span class="p">}</span> <span class="o">-&gt;</span> <span class="k">case</span> <span class="n">b</span><span class="o">.</span><span class="p">({</span> <span class="ss">:cont</span><span class="p">,</span> <span class="n">acc</span> <span class="p">})</span> <span class="k">do</span> <span class="p">{</span> <span class="ss">:suspended</span><span class="p">,</span> <span class="n">acc</span><span class="p">,</span> <span class="n">b</span> <span class="p">}</span> <span class="o">-&gt;</span> <span class="n">do_interleave</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">,</span> <span class="n">acc</span><span class="p">)</span> <span class="p">{</span> <span class="ss">:halted</span><span class="p">,</span> <span class="n">acc</span> <span class="p">}</span> <span class="o">-&gt;</span> <span class="n">acc</span> <span class="p">{</span> <span class="ss">:done</span><span class="p">,</span> <span class="n">acc</span> <span class="p">}</span> <span class="o">-&gt;</span> <span class="n">finish_interleave</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">acc</span><span class="p">)</span> <span class="k">end</span> <span class="p">{</span> <span class="ss">:halted</span><span class="p">,</span> <span class="n">acc</span> <span class="p">}</span> <span class="o">-&gt;</span> <span class="n">acc</span> <span class="p">{</span> <span class="ss">:done</span><span class="p">,</span> <span class="n">acc</span> <span class="p">}</span> <span class="o">-&gt;</span> <span class="n">finish_interleave</span><span class="p">(</span><span class="n">b</span><span class="p">,</span> <span class="n">acc</span><span class="p">)</span> <span class="k">end</span> <span class="k">end</span> <span class="k">defp</span> <span class="n">finish_interleave</span><span class="p">(</span><span class="n">a_or_b</span><span class="p">,</span> <span class="n">acc</span><span class="p">)</span> <span class="k">do</span> <span class="k">case</span> <span class="n">a_or_b</span><span class="o">.</span><span class="p">({</span> <span class="ss">:cont</span><span class="p">,</span> <span class="n">acc</span> <span class="p">})</span> <span class="k">do</span> <span class="p">{</span> <span class="ss">:suspended</span><span class="p">,</span> <span class="n">acc</span><span class="p">,</span> <span class="n">a_or_b</span> <span class="p">}</span> <span class="o">-&gt;</span> <span class="n">finish_interleave</span><span class="p">(</span><span class="n">a_or_b</span><span class="p">,</span> <span class="n">acc</span><span class="p">)</span> <span class="p">{</span> <span class="n">_</span><span class="p">,</span> <span class="n">acc</span> <span class="p">}</span> <span class="o">-&gt;</span> <span class="n">acc</span> <span class="k">end</span> <span class="k">end</span> <span class="k">end</span> <span class="no">Interleave</span><span class="o">.</span><span class="n">interleave</span><span class="p">([</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">],</span> <span class="p">[</span><span class="ss">:a</span><span class="p">,</span> <span class="ss">:b</span><span class="p">,</span> <span class="ss">:c</span><span class="p">,</span> <span class="ss">:d</span><span class="p">])</span> <span class="c1">#=&gt; [1, :a, 2, :b, :c, :d]</span> </code></pre></div></div> <p>Lets go through this step by step. The main <code class="language-plaintext highlighter-rouge">interleave</code> function first partially applies <code class="language-plaintext highlighter-rouge">Enumerable.reduce/3</code> to get function values that work just like the continuations. This makes things easier for <code class="language-plaintext highlighter-rouge">do_interleave</code>.</p> <p>The <code class="language-plaintext highlighter-rouge">do_interleave</code> function first calls <code class="language-plaintext highlighter-rouge">a</code> (<code class="language-plaintext highlighter-rouge">af</code> from <code class="language-plaintext highlighter-rouge">interleave</code>) with the <code class="language-plaintext highlighter-rouge">step</code> function so that the available element of <code class="language-plaintext highlighter-rouge">a</code> gets added to the accumulator and <code class="language-plaintext highlighter-rouge">a</code> immediately suspends afterwards. Then the same is done for <code class="language-plaintext highlighter-rouge">b</code>. If either producer is done all the remaining elements of the other get added to the accumulator list.</p> <p>Note that <code class="language-plaintext highlighter-rouge">acc</code> is sometimes used to mean a tuple like <code class="language-plaintext highlighter-rouge">{ :cont, x }</code> and sometimes the accumulator value proper. It’s a bit confusing, yes.</p> <p>This example shows that through clever combination of an outer function (<code class="language-plaintext highlighter-rouge">do_interleave</code>) and an inner function <code class="language-plaintext highlighter-rouge">step</code> two producers can be interleaved.</p> <h2 id="conclusion">Conclusion</h2> <p>The new system of enumerators certainly makes things a bit more complicated but also adds power. I suspect many interesting and “interesting” functions can be built on top of it.</p> José Valim 2013-11-05T00:00:00+00:00 /blog/2013/11/05/elixir-v0-11-0-released <p>After 4 months, Elixir v0.11.0 has been released with 832 commits since the previous minor release, done by more than 40 contributors. Although we have some great features in this release, the major focus in those 4 months was improving the common patterns used by the community and streamlining the existing workflows.</p> <h2 id="iex">IEx</h2> <p>One of the tools that most received improvements in this release was our interactive shell. Such improvements come as no surprise as Elixir developers spend a good amount of time in the shell, which is great for development, allowing you to quickly try and load code, to production, where IEx can connect to remote nodes to inspect production information.</p> <p>The IEx helper <code class="language-plaintext highlighter-rouge">h</code>, responsible for showing documentation for existing modules and functions, has now been improved to rely on ANSI codes and nicely format the documentation. let’s take a look at the docs for the String module:</p> <p><img src="/images/contents/string-help.png" alt="String module docs" /></p> <p>This change goes in line with Elixir’s goal of providing first-class documentation, which makes documentation easily accessible at runtime, support to doctests and more.</p> <p>In this new release, IEx also supports a very simple debugging mechanism called <code class="language-plaintext highlighter-rouge">IEx.pry</code>. Let’s see an screenshot of it in action:</p> <p><img src="/images/contents/iex-pry.png" alt="IEx pry example" /></p> <p>In Elixir, your code runs in many processes that talk to each other and the Elixir shell is no different. <code class="language-plaintext highlighter-rouge">IEx.pry</code> allows another process to take over the shell, allowing the developer to inspect the binding and halt the execution of the process being “pried” (i.e. the one that invoked <code class="language-plaintext highlighter-rouge">IEx.pry</code>). We called this feature <code class="language-plaintext highlighter-rouge">pry</code> as a gentle reminder that you can only inspect existing information, you cannot change the binding over a pried process. For more information, check the docs for <a href="https://hexdocs.pm/iex/IEx.html#pry/1"><code class="language-plaintext highlighter-rouge">IEx.pry/1</code></a>.</p> <h2 id="exunit">ExUnit</h2> <p><a href="/blog/2013/07/13/elixir-v0-10-0-released/">In the previous release</a>, we introduced great changes to ExUnit, like the support for the <code class="language-plaintext highlighter-rouge">--trace</code> option. This time we continued pushing improvements, like adding profiling to test cases (times can be seen with the <code class="language-plaintext highlighter-rouge">--trace</code> option), paving the way for other features like emitting warnings for test cases that are too slow.</p> <p>Another simple but significant change in ExUnit was the change in the default formatter to print changes as they come, instead of waiting until the suite is done running:</p> <p><img src="/images/contents/fast-fail.png" alt="ExUnit Fast Fail" /></p> <p>This change allows developer to get faster feedback from their test suites.</p> <h2 id="mix">Mix</h2> <p>Since the early days, Elixir took ahold of the compilation process in order to provide a seamless compilation experience. <a href="/blog/2012/04/24/a-peek-inside-elixir-s-parallel-compiler/">Elixir’s ParallelCompiler</a> was introduced even before the first official release, allowing developers to harness all the cores in their computer to compile Elixir code. However, once the first release came out, every time you changed any file, the whole project had to be recompiled.</p> <p>In the past releases we have improved this process to only compile files that changed and their dependencies. For v0.11.0, we have improved this process to be faster and less conservative than the previous version.</p> <p>Mix has also improved support for umbrella projects, which are projects that contain multiple OTP applications, essential for building large projects. The current release allows sharing of dependencies between projects and faster and dependency resolution times.</p> <h2 id="other-changes">Other changes</h2> <p>This release also introduces the new capture operator, which provides a convenient syntax for retrieving functions so they can be passed as arguments:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="no">Enum</span><span class="o">.</span><span class="n">all?</span><span class="p">([</span><span class="ss">:foo</span><span class="p">,</span> <span class="ss">:bar</span><span class="p">,</span> <span class="ss">:baz</span><span class="p">],</span> <span class="o">&amp;</span><span class="n">is_atom</span><span class="o">/</span><span class="mi">1</span><span class="p">)</span> <span class="c1">#=&gt; true</span> </code></pre></div></div> <p>Which can also be used for partially applying functions and macros:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">fun</span> <span class="o">=</span> <span class="o">&amp;</span><span class="n">is_record</span><span class="p">(</span><span class="nv">&amp;1</span><span class="p">,</span> <span class="no">Range</span><span class="p">)</span> <span class="n">fun</span><span class="o">.</span><span class="p">(</span><span class="mi">1</span><span class="o">..</span><span class="mi">3</span><span class="p">)</span> <span class="c1">#=&gt; true</span> </code></pre></div></div> <p>You can learn more about the <a href="https://hexdocs.pm/elixir/Kernel.SpecialForms.html#&amp;/1">new capture operator in our docs</a>.</p> <p>We have also pushed improvements to <a href="https://hexdocs.pm/elixir/String.html">the String module</a>, including new APIs. In particular, in order to know that <code class="language-plaintext highlighter-rouge">String.length("josé")</code> has length 4 (even though it takes 5 bytes to be represented in UTF-8), we need to use some algorithms defined by the Unicode Standard. These have been implemented as specified in the <a href="http://www.unicode.org/reports/tr29/">extended grapheme cluster algorithm, defined in the version 6.3.0 of the Unicode Standard</a>.</p> <p>In the optimization front, we have pushed the first iteration of a <a href="https://groups.google.com/forum/#!topic/elixir-lang-core/RoXAUtoyjk4">feature called Protocol consolidation</a>, which speeds up the polymorphic dispatch done by protocols, sometimes reducing the dispatching time to 10% of the original time. We will continue working in upcoming releases to integrate protocol consolidation as a regular part of the developer workflow.</p> <p>And finally, a minor but frequently asked feature has finally arrived into Elixir: variables follow the same rules as other identifiers in the language, which means developers can now name their variables <code class="language-plaintext highlighter-rouge">is_atom?</code>. For a general overview, <a href="https://github.com/elixir-lang/elixir/blob/v0.11.0/CHANGELOG.md">check out the CHANGELOG</a>.</p> <p>Give Elixir a try! You can start with our <a href="https://hexdocs.pm/elixir/introduction.html">getting started guide</a>, or check out our sidebar for other learning resources.</p> <p><strong>PS:</strong> We have just released v0.11.1 which addresses a regression in Mix and improves the dependencies update process.</p> José Valim 2013-08-08T00:00:00+00:00 /blog/2013/08/08/elixir-design-goals <p>During the last year, we have spoken at many conferences spreading the word about Elixir. We <a href="https://vimeo.com/53221562">usually started with introducing the Erlang VM</a>, then went on to talk about Elixir goals, saving some time at the end to do a live demo, showing some goodies like exchanging information between remote nodes and even hot code swapping.</p> <p>This post is a summary of those talks, focusing on the language goals: compatibility, productivity and extensibility.</p> <h2 id="compatibility">Compatibility</h2> <p>Elixir is meant to be compatible with the Erlang VM and the existing ecosystem. When we talk about Erlang, we can break it into three parts:</p> <ul> <li>A functional programming language, called Erlang</li> <li>A set of design principles, called OTP</li> <li>The Erlang Virtual Machine, referred to as EVM or BEAM</li> </ul> <p>Elixir runs in the same virtual machine and is compatible with OTP. Not only that, all the tools and libraries available in the Erlang ecosystem are also available in Elixir, simply because there is no conversion cost from calling Erlang from Elixir and vice-versa.</p> <p>We frequently say that <strong>the Erlang VM is Elixir’s strongest asset</strong>.</p> <p>All Elixir code is executed inside light-weight processes (actors), each with its own state, that exchange messages between each other. The Erlang VM multiplexes those processes onto many cores, making it trivial to run code concurrently.</p> <p>In fact if you compile any Elixir code, including the Elixir source, you will see all cores on your machine being used out of the box. With <a href="http://www.parallella.org/board/">technologies like Parallella</a> becoming more accessible and affordable, it is hard to ignore the power you can get out of the Erlang VM.</p> <p>Finally, the Erlang VM was designed to build systems that run forever, self-heal and scale. Joe Armstrong, one of Erlang’s creators, has recently given an excellent talk <a href="http://www.infoq.com/presentations/self-heal-scalable-system">about the design decisions behind OTP and the VM</a>.</p> <p>Nothing that we are describing here is particularly new. Open source projects like CouchDB, Riak, RabbitMQ, Chef11 and companies like Ericsson, Heroku, Basho, Klarna and Wooga are already enjoying the benefits provided by the Erlang VM, some of them for quite a long time.</p> <h2 id="productivity">Productivity</h2> <blockquote> <p>Now we need to go meta. We should now think of a language design as being a pattern for language designs. A tool for making more tools of the same kind. […] A language design can no longer be a thing. It must be a pattern, a pattern for growth. A pattern for growing a pattern, for defining the patterns that programmers can use for their real work and main goals.</p> </blockquote> <ul> <li>Guy Steele, keynote at the 1998 ACM OOPSLA conference on “Growing a Language”</li> </ul> <p>Productivity is, in general, a hard goal to measure. A language productive for creating desktop applications may not be productive for mathematical computing. Productivity depends directly on the field in which you intend to use the language, the available tools in the ecosystem and how easy it is to create and extend those tools.</p> <p>For this reason, we have opted for a small language core. For example, while some languages have <code class="language-plaintext highlighter-rouge">if</code>, <code class="language-plaintext highlighter-rouge">case</code>, <code class="language-plaintext highlighter-rouge">try</code> and so on as language keywords, each with its own rules in the parser, <strong>in Elixir they are just macros</strong>. This allows us to implement most of Elixir in Elixir and also allows developers to extend the language using the same tools we used to build the language itself, often extending the language to the specific domains they are working on.</p> <p>Here is an example of how someone would implement <code class="language-plaintext highlighter-rouge">unless</code>, which is a keyword in many languages, in Elixir:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmacro</span> <span class="k">unless</span><span class="p">(</span><span class="n">expr</span><span class="p">,</span> <span class="n">opts</span><span class="p">)</span> <span class="k">do</span> <span class="kn">quote</span> <span class="k">do</span> <span class="k">if</span><span class="p">(</span><span class="n">!unquote</span><span class="p">(</span><span class="n">expr</span><span class="p">),</span> <span class="kn">unquote</span><span class="p">(</span><span class="n">opts</span><span class="p">))</span> <span class="k">end</span> <span class="k">end</span> <span class="k">unless</span> <span class="no">true</span> <span class="k">do</span> <span class="no">IO</span><span class="o">.</span><span class="n">puts</span> <span class="s2">"this will never be seen"</span> <span class="k">end</span> </code></pre></div></div> <p>Since a macro receives the code representation as arguments, we can simply convert an <code class="language-plaintext highlighter-rouge">unless</code> into an <code class="language-plaintext highlighter-rouge">if</code> at compile time.</p> <p>Macros are also the base construct for meta-programming in Elixir: the ability to write code that generates code. Meta-programming allows developers to easily get rid of boilerplate and create powerful tools. A common example mentioned in talks is how our test framework uses macros for expressiveness. Let’s see an example:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="no">ExUnit</span><span class="o">.</span><span class="n">start</span> <span class="k">defmodule</span> <span class="no">MathTest</span> <span class="k">do</span> <span class="kn">use</span> <span class="no">ExUnit</span><span class="o">.</span><span class="no">Case</span><span class="p">,</span> <span class="ss">async:</span> <span class="no">true</span> <span class="n">test</span> <span class="s2">"adding two numbers"</span> <span class="k">do</span> <span class="n">assert</span> <span class="mi">1</span> <span class="o">+</span> <span class="mi">2</span> <span class="o">==</span> <span class="mi">4</span> <span class="k">end</span> <span class="k">end</span> </code></pre></div></div> <p>The first thing to notice is the <code class="language-plaintext highlighter-rouge">async: true</code> option. When your tests do not have any side-effects, you can run them concurrently by passing the <code class="language-plaintext highlighter-rouge">async: true</code> option.</p> <p>Next we define a test case and we do an assertion with the <code class="language-plaintext highlighter-rouge">assert</code> macro. Simply calling <code class="language-plaintext highlighter-rouge">assert</code> would be a bad practice in many languages as it would provide a poor error report. In such languages, functions/methods like <code class="language-plaintext highlighter-rouge">assertEqual</code> or <code class="language-plaintext highlighter-rouge">assert_equal</code> would be the recommended way of performing such assertion.</p> <p>In Elixir, however, <code class="language-plaintext highlighter-rouge">assert</code> is a macro and as such it can look into the code being asserted and infer that a comparison is being made. This code is then transformed to provide a detailed error report when the test runs:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>1) test adding two numbers (MathTest) ** (ExUnit.ExpectationError) expected: 3 to be equal to (==): 4 at test.exs:7 </code></pre></div></div> <p>This simple example illustrates how a developer can leverage macros to provide a concise but powerful API. Macros have access to the whole compilation environment, being able to check the imported functions, macros, defined variables and more.</p> <p>Those examples are just scratching the surface of what can be achieved with macros in Elixir. For example, we are currently using macros to compile routes from a web application into a bunch of patterns that are highly optimizable by the VM, providing an expressive but heavily optimized routing algorithm.</p> <p>The macro system also caused a huge impact on the syntax, which we will discuss briefly before moving to the last goal.</p> <h3 id="syntax">Syntax</h3> <p>Although syntax is usually one of the first topics that comes up when Elixir is being discussed, it was never a goal to simply provide a different syntax. Since we wanted to provide a macro system, we knew that the macro system would only be sane if we could represent Elixir syntax in terms of Elixir’s own data structures in a straight-forward fashion. With this goal in mind, we set out to design the first Elixir version, which looked like this:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span><span class="p">(</span><span class="no">Hello</span><span class="p">,</span> <span class="k">do</span><span class="p">:</span> <span class="p">(</span> <span class="k">def</span><span class="p">(</span><span class="n">calculate</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">,</span> <span class="n">c</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="p">(</span> <span class="o">=</span><span class="p">(</span><span class="n">temp</span><span class="p">,</span> <span class="o">*</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">))</span> <span class="o">+</span><span class="p">(</span><span class="n">temp</span><span class="p">,</span> <span class="n">c</span><span class="p">)</span> <span class="p">))</span> <span class="p">))</span> </code></pre></div></div> <p>In the snippet above, we represent everything, except variables, as a function or a macro call. Notice keyword arguments like <code class="language-plaintext highlighter-rouge">do:</code> have been present since the first version. To this, we slowly added new syntax, making some common patterns more elegant while keeping the same underlying data representation. We soon added infix notation for operators:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span><span class="p">(</span><span class="no">Hello</span><span class="p">,</span> <span class="k">do</span><span class="p">:</span> <span class="p">(</span> <span class="k">def</span><span class="p">(</span><span class="n">calculate</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">,</span> <span class="n">c</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="p">(</span> <span class="n">temp</span> <span class="o">=</span> <span class="n">a</span> <span class="o">*</span> <span class="n">b</span> <span class="n">temp</span> <span class="o">+</span> <span class="n">c</span> <span class="p">))</span> <span class="p">))</span> </code></pre></div></div> <p>The next step was to make parentheses optional:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">Hello</span><span class="p">,</span> <span class="k">do</span><span class="p">:</span> <span class="p">(</span> <span class="k">def</span> <span class="n">calculate</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">,</span> <span class="n">c</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="p">(</span> <span class="n">temp</span> <span class="o">=</span> <span class="n">a</span> <span class="o">*</span> <span class="n">b</span> <span class="n">temp</span> <span class="o">+</span> <span class="n">c</span> <span class="p">)</span> <span class="p">)</span> </code></pre></div></div> <p>And finally we added <code class="language-plaintext highlighter-rouge">do</code>-<code class="language-plaintext highlighter-rouge">end</code> as convenience for the common <code class="language-plaintext highlighter-rouge">do: (...)</code> construct:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">Hello</span> <span class="k">do</span> <span class="k">def</span> <span class="n">calculate</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">,</span> <span class="n">c</span><span class="p">)</span> <span class="k">do</span> <span class="n">temp</span> <span class="o">=</span> <span class="n">a</span> <span class="o">*</span> <span class="n">b</span> <span class="n">temp</span> <span class="o">+</span> <span class="n">c</span> <span class="k">end</span> <span class="k">end</span> </code></pre></div></div> <p>Given my previous background in Ruby, it is natural that some of the constructs added were borrowed from Ruby. However, those additions were a by-product, and not a language goal.</p> <p>Many language constructs are also inspired by their Erlang counter-parts, like some of the control-flow macros, operators and containers. Notice how some Elixir code:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># A tuple</span> <span class="n">tuple</span> <span class="o">=</span> <span class="p">{</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span> <span class="p">}</span> <span class="c1"># Adding two lists</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">]</span> <span class="o">++</span> <span class="p">[</span><span class="mi">4</span><span class="p">,</span> <span class="mi">5</span><span class="p">,</span> <span class="mi">6</span><span class="p">]</span> <span class="c1"># Case</span> <span class="k">case</span> <span class="n">expr</span> <span class="k">do</span> <span class="p">{</span> <span class="n">x</span><span class="p">,</span> <span class="n">y</span> <span class="p">}</span> <span class="o">-&gt;</span> <span class="n">x</span> <span class="o">+</span> <span class="n">y</span> <span class="n">other</span> <span class="ow">when</span> <span class="n">is_integer</span><span class="p">(</span><span class="n">other</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">other</span> <span class="k">end</span> </code></pre></div></div> <p>maps to Erlang:</p> <div class="language-erlang highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">% A tuple </span><span class="nv">Tuple</span> <span class="o">=</span> <span class="p">{</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span> <span class="p">}.</span> <span class="c">% Adding two lists </span><span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">]</span> <span class="o">++</span> <span class="p">[</span><span class="mi">4</span><span class="p">,</span> <span class="mi">5</span><span class="p">,</span> <span class="mi">6</span><span class="p">].</span> <span class="c">% Case </span><span class="k">case</span> <span class="nv">Expr</span> <span class="k">of</span> <span class="p">{</span> <span class="nv">X</span><span class="p">,</span> <span class="nv">Y</span> <span class="p">}</span> <span class="o">-&gt;</span> <span class="nv">X</span> <span class="o">+</span> <span class="nv">Y</span><span class="p">;</span> <span class="nv">Other</span> <span class="k">when</span> <span class="nb">is_integer</span><span class="p">(</span><span class="nv">Other</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nv">Other</span> <span class="k">end</span><span class="p">.</span> </code></pre></div></div> <h2 id="extensibility">Extensibility</h2> <p>By building on top of a small core, most of the constructs in the language can be replaced and extended as required by developers to target specific domains. However, there is a particular domain that Elixir is inherently good at, which is building concurrent, distributed applications, thanks to OTP and the Erlang VM.</p> <p>Elixir complements this domain by providing a standard library with:</p> <ul> <li>Unicode strings and unicode operations</li> <li>A powerful unit test framework</li> <li>More data structures like ranges, including novel implementations for sets and dictionaries</li> <li>Polymorphic records (in contrast to Erlang’s compilation-time only records)</li> <li>Strict and lazy enumeration APIs</li> <li>Convenience functions for scripting, like working with paths and the filesystem</li> <li>A project management tool to compile and test Elixir code</li> </ul> <p>And much more.</p> <p>Most of the features above provide their own extensibility mechanisms, too. For example, take the <code class="language-plaintext highlighter-rouge">Enum</code> module. The <code class="language-plaintext highlighter-rouge">Enum</code> module allow us to enumerate the built-in ranges, lists, sets, etc:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">list</span> <span class="o">=</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">]</span> <span class="no">Enum</span><span class="o">.</span><span class="n">map</span> <span class="n">list</span><span class="p">,</span> <span class="k">fn</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">x</span> <span class="o">*</span> <span class="mi">2</span> <span class="k">end</span> <span class="c1">#=&gt; [2, 4, 6]</span> <span class="n">range</span> <span class="o">=</span> <span class="mi">1</span><span class="o">..</span><span class="mi">3</span> <span class="no">Enum</span><span class="o">.</span><span class="n">map</span> <span class="n">range</span><span class="p">,</span> <span class="k">fn</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">x</span> <span class="o">*</span> <span class="mi">2</span> <span class="k">end</span> <span class="c1">#=&gt; [2, 4, 6]</span> <span class="n">set</span> <span class="o">=</span> <span class="no">HashSet</span><span class="o">.</span><span class="n">new</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">]</span> <span class="no">Enum</span><span class="o">.</span><span class="n">map</span> <span class="n">set</span><span class="p">,</span> <span class="k">fn</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">x</span> <span class="o">*</span> <span class="mi">2</span> <span class="k">end</span> <span class="c1">#=&gt; [2, 4, 6]</span> </code></pre></div></div> <p>Not only that, any developer can <strong>extend</strong> the <code class="language-plaintext highlighter-rouge">Enum</code> module to work with any data type as long as the data type implements <a href="https://hexdocs.pm/elixir/Enumerable.html">the <code class="language-plaintext highlighter-rouge">Enumerable</code> protocol</a> (protocols in Elixir are based on Clojure’s protocol). This is extremely convenient because the developer needs to know only the <code class="language-plaintext highlighter-rouge">Enum</code> API for enumeration, instead of memorizing specific APIs for sets, lists, dicts, etc.</p> <p>There are many other protocols exposed by the language, like <a href="https://hexdocs.pm/elixir/Inspect.html">the <code class="language-plaintext highlighter-rouge">Inspect</code> protocol</a> for pretty printing data structures and <a href="https://hexdocs.pm/elixir/Access.html">the <code class="language-plaintext highlighter-rouge">Access</code> protocol</a> for accessing key-value data by key. By being extensible, Elixir ensures developers can work <strong>with</strong> the language, instead of <strong>against</strong> the language.</p> <h2 id="summing-up">Summing up</h2> <p>The goal of this post was to sumarize the language goals: compatibility, productivity and extensibility. By being compatible with the Erlang VM, we are providing developers another toolset for building concurrent, distributed and fault-tolerant systems.</p> <p>We also hope to have clarified what Elixir brings to the Erlang VM, in particular, meta-programming through macros, polymorphic constructs for extensibility and a data-focused standard library with extensible and consistent APIs for diverse types, including strict and lazy enumeration, unicode handling, a test framework and more.</p> <p>Give Elixir a try! You can start with our <a href="https://hexdocs.pm/elixir/introduction.html">getting started guide</a>, or check out our sidebar for other learning resources.</p> José Valim 2013-07-13T00:00:00+00:00 /blog/2013/07/13/elixir-v0-10-0-released <p>Elixir v0.10.0 is released with support for streams, sets and many improvements to the Mix and ExUnit applications.</p> <h2 id="streams">Streams</h2> <p>The default mechanism for working with collections in Elixir is the <code class="language-plaintext highlighter-rouge">Enum</code> module. With it, you can map over ranges, lists, sets, dictionaries and any other structure as long as it implements the <code class="language-plaintext highlighter-rouge">Enumerable</code> protocol:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="no">Enum</span><span class="o">.</span><span class="n">map</span><span class="p">([</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">],</span> <span class="k">fn</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">x</span> <span class="o">*</span> <span class="mi">2</span> <span class="k">end</span><span class="p">)</span> <span class="c1">#=&gt; [2, 4, 6]</span> </code></pre></div></div> <p>The <code class="language-plaintext highlighter-rouge">Enum</code> module performs eager evaluation. Consider the following example:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">]</span> <span class="o">|&gt;</span> <span class="no">Enum</span><span class="o">.</span><span class="n">take_while</span><span class="p">(</span><span class="k">fn</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">x</span> <span class="o">&lt;</span> <span class="mi">3</span> <span class="k">end</span><span class="p">)</span> <span class="o">|&gt;</span> <span class="no">Enum</span><span class="o">.</span><span class="n">map</span><span class="p">(</span><span class="k">fn</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">x</span> <span class="o">*</span> <span class="mi">2</span> <span class="k">end</span><span class="p">)</span> <span class="c1">#=&gt; [2, 4]</span> </code></pre></div></div> <p>In the example above, we enumerate the items in list once, taking all elements that are less than 3, and then we enumerate the remaining elements again, multiplying them by two. In order to retrieve the final result, we have created one intermediate list. As we add more operations, more intermediate lists will be generated.</p> <p>This approach is simple and efficient for the majority of the cases but, when working with large collections, we can generate many, possibly large, intermediate lists affecting performance. That’s one of the problems Streams solve. Let’s rewrite the example above using Streams:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">]</span> <span class="o">|&gt;</span> <span class="no">Stream</span><span class="o">.</span><span class="n">take_while</span><span class="p">(</span><span class="k">fn</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">x</span> <span class="o">&lt;</span> <span class="mi">3</span> <span class="k">end</span><span class="p">)</span> <span class="o">|&gt;</span> <span class="no">Stream</span><span class="o">.</span><span class="n">map</span><span class="p">(</span><span class="k">fn</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">x</span> <span class="o">*</span> <span class="mi">2</span> <span class="k">end</span><span class="p">)</span> <span class="c1">#=&gt; #Stream.Lazy&lt;...&gt;</span> </code></pre></div></div> <p>Now, instead of getting the result back, we got a Stream. The list elements are yet to be enumerated! We can realize the stream by calling any of the Enum functions, like <code class="language-plaintext highlighter-rouge">Enum.to_list/1</code>. By doing so the list will be iterated just once avoiding the intermediary representations.</p> <p>In a nutshell, Streams are composable, lazy enumerables. Streams are also useful when doing IO or expressing infinite computations. We can retrieve a file as a stream:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="no">File</span><span class="o">.</span><span class="n">stream!</span><span class="p">(</span><span class="s2">"README.md"</span><span class="p">)</span> </code></pre></div></div> <p>In the example above, we got a stream that will enumerate the lines in the file one by one when enumerated. We could further extend the stream above, for example, by rejecting blank lines, and the file will be opened just when its results are actually needed.</p> <p>Do you need a random number generator? We got your back:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="no">Stream</span><span class="o">.</span><span class="n">repeatedly</span><span class="p">(</span><span class="k">fn</span> <span class="o">-&gt;</span> <span class="ss">:random</span><span class="o">.</span><span class="n">uniform</span> <span class="k">end</span><span class="p">)</span> <span class="o">|&gt;</span> <span class="no">Enum</span><span class="o">.</span><span class="n">take</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span> <span class="c1">#=&gt; [0.4435846174457203, 0.7230402056221108, 0.94581636451987]</span> </code></pre></div></div> <p><code class="language-plaintext highlighter-rouge">Stream.repeatedly/1</code> returns an infinite stream but that’s ok we just need its first three elements. You can learn more about <a href="https://hexdocs.pm/elixir/Stream.html">stream and related functions in <code class="language-plaintext highlighter-rouge">Stream</code> module documentation</a>.</p> <h2 id="sets">Sets</h2> <p>This release also adds <a href="https://hexdocs.pm/elixir/Set.html">the Sets API</a> to Elixir and a HashSet implementation. The HashSet implementation follows <a href="/blog/2013/01/27/elixir-v0-8-0-released/">the same design goals as the HashDict implementation</a> released at the beginning of this year, starting with a compact representation and expanding and contracting as needed.</p> <p>This feature was a contribution from <a href="https://github.com/josephwilk">Joseph Wilk</a> and he talks about its implementation and provides some benchmarks <a href="http://blog.josephwilk.net/elixir/sets-in-elixir.html">on his blog</a>.</p> <h2 id="pretty-printing">Pretty printing</h2> <p>Another addition to this release is pretty printing. The pretty printing started as an implementation of the <a href="http://homepages.inf.ed.ac.uk/wadler/papers/prettier/prettier.pdf">Wadler paper</a> by <a href="https://github.com/manpages">Jonns Mostovoys</a> which was then improved by <a href="https://github.com/brunoro">Gustavo Brunoro</a> under his Google Summer of Code project as described in <a href="http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.34.2200">Lindig’s <em>Strictly Prettier</em> paper</a>.</p> <p>As soon as you upgrade to Elixir v0.10.0 and start IEx, you will get pretty printing for all data structures provided by Elixir. We have also added documentation to the <code class="language-plaintext highlighter-rouge">Inspect</code> module about <a href="https://hexdocs.pm/elixir/Inspect.html">adding pretty printing to your own structures</a> as well as <a href="https://hexdocs.pm/elixir/Inspect.Algebra.html">using the document algebra for any other kind of formatting</a>.</p> <h2 id="other-improvements">Other improvements</h2> <p>Other notable improvements are:</p> <ul> <li> <p>We have improved Mix to be able to fetch Rebar dependencies, making integration with the existing Erlang ecossystem much easier, thanks to <a href="https://github.com/ericmj">Eric Meadows-Jonsson</a>;</p> </li> <li> <p>ExUnit now supports the trace option, enabled via <code class="language-plaintext highlighter-rouge">mix test --trace</code>, which forces tests to run sequentially and print the test names and extra information as it goes;</p> </li> <li> <p>We are also working hard on Windows support, improving its command-line tools and working towards a green test suite, thanks to <a href="https://github.com/tojans">Tom Jansens</a>;</p> </li> <li> <p>Meta-programming in Elixir was also improved by the addition of the <code class="language-plaintext highlighter-rouge">binding/0</code> and <code class="language-plaintext highlighter-rouge">binding/1</code> macros plus the additions of <code class="language-plaintext highlighter-rouge">Macro.expand_once/2</code> and <code class="language-plaintext highlighter-rouge">Macro.expand_all/2</code> to the <a href="https://hexdocs.pm/elixir/Macro.html"><code class="language-plaintext highlighter-rouge">Macro</code> module</a>;</p> </li> </ul> <p>There are also improvements to typespecs, error messages, many bug fixes and some backwards incompatible changes. We have posted a detailed <a href="https://groups.google.com/forum/?fromgroups#!topic/elixir-lang-talk/ksrefrgK1eY">upgrade instructions on the mailing list</a>. For a general overview, <a href="https://github.com/elixir-lang/elixir/blob/v0.10.0/CHANGELOG.md">check out the CHANGELOG</a>.</p> <p>Give Elixir a try! You can start with our <a href="https://hexdocs.pm/elixir/introduction.html">getting started guide</a>, or check out our sidebar for other learning resources.</p> José Valim 2013-05-23T00:00:00+00:00 /blog/2013/05/23/elixir-v0-9-0-released <p>While <a href="https://pragprog.com/book/elixir/programming-elixir">Programming Elixir</a> was being announced, we have been working on Elixir v0.9.0 which is finally out. This release contains new features, important performance optimizations and bug fixes.</p> <p>Elixir v0.9.0 also removes support for Erlang R15 and earlier versions. In case you still need to run Elixir software on R15, we have also released Elixir v0.8.3, which contains many of the enhancements in v0.9.0. Check the <a href="https://github.com/elixir-lang/elixir/blob/v0.9.0/CHANGELOG.md">CHANGELOG for more details for both releases</a>.</p> <p>All this work was achieved by our very vibrant community! Over the last month, 17 authors have pushed more than 500 commits, where more than 60 pull requests were merged and more than 80 issues were closed.</p> <p>Let’s talk about the goodies!</p> <h2 id="compilation-time-improvements">Compilation time improvements</h2> <p>We have spent some time improving compilation time. The particular scenario we have worked on was the definition of records:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defrecord</span> <span class="no">User</span><span class="p">,</span> <span class="ss">name:</span> <span class="no">nil</span><span class="p">,</span> <span class="ss">age:</span> <span class="no">nil</span> </code></pre></div></div> <p>Records are a good scenario because they are implemented in Elixir, using Elixir macros, and they also define a module underneath, which exercises the Erlang VM compilation stack.</p> <p>We have used <a href="http://www.erlang.org/doc/man/fprof.html">fprof</a> to identify the bottlenecks and made the compilation stack 35% faster. We have also identified bottlenecks coming from Erlang and <a href="https://github.com/erlang/otp/commit/32b194495f353dde014b00008a630eeff2a71056">pushed some patches</a> that should benefit both Elixir and Erlang code.</p> <p>A special thanks to <a href="https://github.com/yrashk">Yurii Rashkovskii</a> for the data and profiling.</p> <h2 id="umbrella-projects">Umbrella projects</h2> <p>In Elixir, an application denotes a component implementing some specific functionality, that can be started and stopped as a unit, and which can be re-used in other systems as well.</p> <p>As a project grows, it is recommended to break it apart into smaller, isolated applications and bundle them together. The issue so far was that Elixir did not provide good support for working with many applications at once, and compiling and managing those applications became rather a tedious work.</p> <p>Elixir v0.9.0 now supports umbrella projects which can work with many applications at the same time. You can create a new umbrella project with:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>mix new my_project <span class="nt">--umbrella</span> </code></pre></div></div> <p>The generated project will have the following structure:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>apps/ mix.exs README.md </code></pre></div></div> <p>Now, inside the <code class="language-plaintext highlighter-rouge">apps</code> directory, you can create as many applications as you want and running <code class="language-plaintext highlighter-rouge">mix compile</code> inside the umbrella project will automatically compile all applications. The <a href="https://github.com/elixir-lang/elixir/issues/667">original discussion for this feature</a> contains more details about how it all works.</p> <p>A special thanks to <a href="https://github.com/ericmj">Eric Meadows-Jonsson</a> for implementing this feature and to <a href="https://github.com/yrashk">Yurii</a> for testing it against different edge cases.</p> <h2 id="reducers">Reducers</h2> <p>Elixir v0.9.0 changes its main abstraction for enumeration from iterators to reducers. Before Elixir v0.9.0, when you invoked:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="no">Enum</span><span class="o">.</span><span class="n">map</span><span class="p">([</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">],</span> <span class="k">fn</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">x</span> <span class="o">*</span> <span class="n">x</span> <span class="k">end</span><span class="p">)</span> <span class="c1">#=&gt; [1, 4, 9]</span> </code></pre></div></div> <p>It asked the <code class="language-plaintext highlighter-rouge">Enum.Iterator</code> protocol for instructions on how to iterate the list <code class="language-plaintext highlighter-rouge">[1, 2, 3]</code>. This iteration happened by retrieving each item in the list, one by one, until there were no items left.</p> <p>This approach posed many problems:</p> <ul> <li>Iterators are very hard to compose;</li> <li>Iterators contain state. You need to know, at each moment, what is the next element you have to iterate next. We use functions and their bindings to pass the iteration state around;</li> <li>Iterators have the “dangling open resource” problem. Consider that you want to iterate a file with <code class="language-plaintext highlighter-rouge">Enum.map/2</code> as above. If any step during the iteration fails, there is no easy way to notify the resource being iterated (in this case, the opened file) that iteration failed, so we can’t close the file automatically, leaving it to the user.</li> </ul> <p>Reducers solve all of those problems by using a more functional approach. Instead of asking a list to spill its elements out one by one and then working on each element, we now generate a recipe of computations and pass it down to the list which applies those computations on itself.</p> <p>Here is how we implement the <code class="language-plaintext highlighter-rouge">Enumerable</code> protocol for lists:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defimpl</span> <span class="no">Enumerable</span><span class="p">,</span> <span class="ss">for:</span> <span class="no">List</span> <span class="k">do</span> <span class="k">def</span> <span class="n">reduce</span><span class="p">(</span><span class="n">list</span><span class="p">,</span> <span class="n">acc</span><span class="p">,</span> <span class="n">fun</span><span class="p">)</span> <span class="k">do</span> <span class="n">do_reduce</span><span class="p">(</span><span class="n">list</span><span class="p">,</span> <span class="n">acc</span><span class="p">,</span> <span class="n">fun</span><span class="p">)</span> <span class="k">end</span> <span class="k">defp</span> <span class="n">do_reduce</span><span class="p">([</span><span class="n">head</span> <span class="o">|</span> <span class="n">tail</span><span class="p">],</span> <span class="n">acc</span><span class="p">,</span> <span class="n">fun</span><span class="p">)</span> <span class="k">do</span> <span class="n">do_reduce</span><span class="p">(</span><span class="n">tail</span><span class="p">,</span> <span class="n">fun</span><span class="o">.</span><span class="p">(</span><span class="n">head</span><span class="p">,</span> <span class="n">acc</span><span class="p">),</span> <span class="n">fun</span><span class="p">)</span> <span class="k">end</span> <span class="k">defp</span> <span class="n">do_reduce</span><span class="p">([],</span> <span class="n">acc</span><span class="p">,</span> <span class="n">fun</span><span class="p">)</span> <span class="k">do</span> <span class="n">acc</span> <span class="k">end</span> <span class="k">end</span> </code></pre></div></div> <p>The implementation above works as a simple <code class="language-plaintext highlighter-rouge">reduce</code> function (also called <code class="language-plaintext highlighter-rouge">fold</code>, <code class="language-plaintext highlighter-rouge">inject</code> or <code class="language-plaintext highlighter-rouge">foldl</code> in other languages). Here is how it works:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># Sum all elements in a list</span> <span class="no">Enumerable</span><span class="o">.</span><span class="n">reduce</span><span class="p">([</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">],</span> <span class="mi">0</span><span class="p">,</span> <span class="k">fn</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">acc</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">x</span> <span class="o">+</span> <span class="n">acc</span> <span class="k">end</span><span class="p">)</span> <span class="c1">#=&gt; 6</span> </code></pre></div></div> <p>The <code class="language-plaintext highlighter-rouge">Enum.map/2</code> we have used above is now implemented in terms of this reducing function:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">Enum</span> <span class="k">do</span> <span class="k">def</span> <span class="n">map</span><span class="p">(</span><span class="n">collection</span><span class="p">,</span> <span class="n">fun</span><span class="p">)</span> <span class="k">do</span> <span class="no">Enumerable</span><span class="o">.</span><span class="n">reduce</span><span class="p">(</span><span class="n">collection</span><span class="p">,</span> <span class="p">[],</span> <span class="k">fn</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">acc</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="p">[</span><span class="n">fun</span><span class="o">.</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">acc</span><span class="p">)</span> <span class="o">|</span> <span class="n">acc</span><span class="p">]</span> <span class="k">end</span><span class="p">)</span> <span class="o">|&gt;</span> <span class="n">reverse</span> <span class="k">end</span> <span class="k">end</span> </code></pre></div></div> <p>This approach solves all the problems above:</p> <ul> <li>Reducers are composable (notice how we have implemented map on top of reduce by composing functions);</li> <li>Reducers are self-contained: there is no need keep state around, which also solves the “dangling open resource” problem. The data type now knows exactly when the iteration starts and when it finishes;</li> <li>Reducers do not dictate how a type should be enumerated. This means types like <code class="language-plaintext highlighter-rouge">Range</code> and <code class="language-plaintext highlighter-rouge">HashDict</code> can provide a much faster implementation for Reducers;</li> <li>Furthermore, the end result is a cleaner implementation of most of <code class="language-plaintext highlighter-rouge">Enum</code> functions (the <a href="https://github.com/elixir-lang/elixir/pull/1102">reducers pull request</a> removes over 500LOC) and better performance!</li> </ul> <p>Reducers also opens up room for lazy and parallel enumeration, as <a href="http://clojure.com/blog/2012/05/08/reducers-a-library-and-model-for-collection-processing.html">the Clojure community has already proven</a> and something we are looking forward to explore on upcoming releases.</p> <p>A special thanks to <a href="https://github.com/ericmj">Eric Meadows-Jonsson</a> for implementing this feature!</p> <h2 id="other-bits">Other bits</h2> <p>We have also many other smaller improvements:</p> <ul> <li>Our CLI now supports <code class="language-plaintext highlighter-rouge">--hidden</code> and <code class="language-plaintext highlighter-rouge">--cookie</code> flags which are useful for distributed modes;</li> <li>Our test framework, ExUnit, is now able to capture all the communication that happens with a registered IO device, like <code class="language-plaintext highlighter-rouge">:stdio</code> and <code class="language-plaintext highlighter-rouge">:stderr</code>, via <a href="https://hexdocs.pm/ex_unit/ExUnit.CaptureIO.html"><code class="language-plaintext highlighter-rouge">ExUnit.CaptureIO</code></a>. This is very useful for testing how your software reacts to some inputs and what it prints to the terminal;</li> <li><code class="language-plaintext highlighter-rouge">IEx</code> now allows files to be imported into the shell with <code class="language-plaintext highlighter-rouge">import_file</code> and also loads <code class="language-plaintext highlighter-rouge">~/.iex</code> on startup for custom configuration;</li> <li>The <code class="language-plaintext highlighter-rouge">String</code>, <code class="language-plaintext highlighter-rouge">Enum</code> and <code class="language-plaintext highlighter-rouge">Dict</code> modules got more convenience functions that goes from checking unicode character validity to taking values out of a dictionary;</li> <li>And many, many more!</li> </ul> <p>A huge thank you to our community for sending bug reports, providing bug fixes and contributing all those amazing features. And when are <strong>you</strong> joining us? :)</p> <p>Give Elixir a try! You can start with our <a href="https://hexdocs.pm/elixir/introduction.html">getting started guide</a>, or <a href="https://www.youtube.com/watch?v=a-off4Vznjs&amp;feature=youtu.be">check this 30 minute video from PragProg</a> or buy the beta version of <a href="https://pragprog.com/book/elixir/programming-elixir">Programming Elixir</a>.</p> José Valim 2013-05-02T00:00:00+00:00 /blog/2013/05/02/elixir-on-xen <p>Elixir uses Erlang underneath, all the way down. Thanks to this, an Elixir project can run on the recently revealed “OS-less” Erlang VM called LING VM. LING VM is the core technology of <a href="http://erlangonxen.org">Erlang on Xen</a>.</p> <h2 id="why-xen">Why Xen?</h2> <p><a href="https://en.wikipedia.org/wiki/Xen">Xen</a> is an open-source baremetal hypervisor that allows many operating systems to run on the same hardware. Xen is frequently used for server virtualization, Infrastructure as a Service (IaaS) and security applications.</p> <p>Elixir on Xen runs on top of the Xen Hypervisor (via the LING VM) but with no traditional OS underneath it, taking away numerous administrative, scalability, and performance issues. This limits options of a malicious attacker, making it an excellent choice for high-security applications, and reduces startup latency, allowing developers to spawn new VMs in less than 100 milliseconds.</p> <p>You can learn more about Xen and the LING VM on the <a href="http://erlangonxen.org">Erlang on Xen website</a>.</p> <h2 id="getting-started">Getting started</h2> <p>In order to run Elixir on the LING VM, you need to produce a Xen image of your Elixir project. This can be done with the help of the <a href="https://github.com/maximk/lingex">lingex project</a>, created by the LING VM team.</p> <p>Producing an Elixir image using the free Erlang on Xen Build Service requires just a few steps:</p> <ol> <li> <p>Add a dependency on <code class="language-plaintext highlighter-rouge">lingex</code> to your <code class="language-plaintext highlighter-rouge">mix.exs</code> file:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> def deps do [ { :lingex, github: "maximk/lingex" } ] end </code></pre></div> </div> </li> <li> <p>Run <code class="language-plaintext highlighter-rouge">mix deps.get</code> to update your dependencies. This adds a few custom tasks to the mix tool (<code class="language-plaintext highlighter-rouge">lingex.build</code>, <code class="language-plaintext highlighter-rouge">lingex.image</code>, and <code class="language-plaintext highlighter-rouge">lingex.build_image</code>)</p> </li> <li> <p>Set <code class="language-plaintext highlighter-rouge">lingex</code> options. Add the following lines to your <code class="language-plaintext highlighter-rouge">mix.exs</code> file:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> def project do [ lingex_opts: [ build_host: "build.erlangonxen.org:8080", username: "test", password: "test" ] ] end </code></pre></div> </div> </li> <li> <p>Optionally, you may register with the build service <a href="http://build.erlangonxen.org/register">here</a> and update the credentials accordingly. For the complete list of recognized options see the build service documentation.</p> </li> <li> <p>Run <code class="language-plaintext highlighter-rouge">mix lingex.build_image</code>. This will archive all <code class="language-plaintext highlighter-rouge">*.beam</code> files of your project and submit them to the build service.</p> </li> <li> <p>The build process will complete in about 30s. An image file called ‘vmling’ will appear in the current directory, ready to boot as a Xen guest. The image file will contain LING VM and your project code.</p> </li> </ol> <p>And this is all. Erlang on Xen is going to boot the Erlang VM and the standard Erlang shell. You can access Elixir shell in a couple steps:</p> <ol> <li> <p>In the Erlang shell, first start IEx:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> 1&gt; application:start(iex). ok </code></pre></div> </div> </li> <li> <p>Then hit <code class="language-plaintext highlighter-rouge">Ctrl+G</code>. This will open up the user switch command interface from the Erlang shell.</p> </li> <li> <p>In the user switch interface, type:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> User switch command --&gt; s 'Elixir.IEx' --&gt; c </code></pre></div> </div> </li> </ol> <p>This will bring you to Interactive Elixir and you can execute Elixir expressions as usual!</p> <h2 id="summing-up">Summing up</h2> <p>Running Elixir on Xen opens up many possibilities to Elixir developers. We are very thankful for the work done by <a href="http://erlangonxen.org">Erlang on Xen team</a>, who added support for Elixir and the <code class="language-plaintext highlighter-rouge">lingex</code> build tool.</p> <p>Erlang on Xen (and consequently Elixir on Xen) is still in active development, so don’t forget to read more about its concepts, use cases and limitations on <a href="http://erlangonxen.org/">Erlang on Xen website</a>.</p> José Valim 2013-04-29T00:00:00+00:00 /blog/2013/04/29/elixir-v0-8-2-released <p>The past week we have released Elixir v0.8.2. It contains many bug fixes and better support for Erlang R16, including <a href="https://github.com/elixir-lang/elixir/commit/0fad1883df9da541628e8485d28372fd4b977b89">the new built-in functions <code class="language-plaintext highlighter-rouge">insert_elem/3</code> and <code class="language-plaintext highlighter-rouge">delete_elem/2</code></a>.</p> <p>We have also added extensive support to ANSI escape codes. For example, by simply upgrading to the latest Elixir you will get colored output from your test suites:</p> <p><img src="/images/contents/exunit-ansi.png" alt="ANSI escape with ExUnit" /></p> <p>We have also added colored output to Interactive Elixir (IEx) but it requires Erlang R16.</p> <p>Finally, Elixir has always given special attention to documentation. You can easily document functions with the <code class="language-plaintext highlighter-rouge">@doc</code> attribute:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">Math</span> <span class="k">do</span> <span class="nv">@doc</span> <span class="sd">""" Add two numbers together. ## Examples iex&gt; Math.add(1, 2) 3 """</span> <span class="k">def</span> <span class="n">add</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">)</span> <span class="k">do</span> <span class="n">a</span> <span class="o">+</span> <span class="n">b</span> <span class="k">end</span> <span class="k">end</span> </code></pre></div></div> <p>The documentation above is embedded into the module and can be easily retrieved at runtime. For example, by typing <code class="language-plaintext highlighter-rouge">h Math.add/2</code> into Interactive Elixir, we can access the documentation for that module.</p> <p>Elixir v0.8.2 takes this to the next level by adding support to doctests. Given the example above, you can configure Elixir to automatically run the code samples in your documentation by including a call to the <code class="language-plaintext highlighter-rouge">doctest</code> macro in your test suite:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">MathTest</span> <span class="k">do</span> <span class="kn">use</span> <span class="no">ExUnit</span><span class="o">.</span><span class="no">Case</span><span class="p">,</span> <span class="ss">async:</span> <span class="no">true</span> <span class="n">doctest</span> <span class="no">Math</span> <span class="k">end</span> </code></pre></div></div> <p>You can learn more about <a href="https://hexdocs.pm/ex_unit/ExUnit.DocTest.html">doctests on our documentation page</a> and get more information about our latest release <a href="https://github.com/elixir-lang/elixir/blob/ed27611f48ba150404c95fe15f1d6058a4287330/CHANGELOG.md">on the CHANGELOG</a>.</p> <p>If you are new to Elixir, <a href="https://hexdocs.pm/elixir/introduction.html">it’s easy to get started with</a>!</p> José Valim 2013-04-19T00:00:00+00:00 /blog/2013/04/19/google-summer-of-code-2013 <p>We are pleased to announce that Elixir is taking part in Google Summer of Code 2013 as member of the <a href="http://beamcommunity.github.io">BEAM Community</a>. This means that students all around the world can get paid to work on Elixir during the summer!</p> <p>The rules require students to be enrolled in college full or part-time, and to be at least 18 years by May 27, 2013. You can find more information on <a href="http://www.google-melange.com/gsoc/homepage/google/gsoc2013">Google Summer of Code 2013 website</a>.</p> <p>We have published <a href="https://github.com/beamcommunity/beamcommunity.github.com/wiki/Project:-Elixir">a list of ideas we would like to see and explore in Elixir</a> that students can use as a basis, but students are also free to send their own proposals. If your proposal gets accepted, Google will pay you $5000 over the course of three months to work on Elixir. Students can start submitting their proposals on April 22 and the deadline is May 23.</p> <p>Note the BEAM Community serves as a mentoring organization for many other projects that run on the Erlang VM, including Elixir. To see the full list, <a href="http://beamcommunity.github.io">visit the website</a>.</p> <p>We hope to work with you during this summer! If you have more questions, feel free to join the <a href="https://groups.google.com/d/forum/beam-community">BEAM Community mailing list</a> or talk to us on <code class="language-plaintext highlighter-rouge">#beam-community</code> on irc.freenode.net.</p> José Valim 2013-01-27T00:00:00+00:00 /blog/2013/01/27/elixir-v0-8-0-released <p>On the last 9th January, we celebrated <a href="https://github.com/elixir-lang/elixir/commit/337c3f2d569a42ebd5fcab6fef18c5e012f9be5b">two years since Elixir’s first commit</a> and to celebrate this occasion we have prepared a big release. Elixir v0.8 is out, with documentation, optimizations, bug fixes and shiny new features. Let’s take a look at them!</p> <h2 id="otp-applications">OTP applications</h2> <p>One of the goals for the v0.8 release was better integration with OTP applications. By passing the <code class="language-plaintext highlighter-rouge">--sup</code> option to Mix, you can start a new OTP Application containing application callbacks and a supervisor:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mix new my_app --sup </code></pre></div></div> <p>And applications can be started directly from the command line as well:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>elixir --app my_app </code></pre></div></div> <p>We have written a whole <a href="https://hexdocs.pm/elixir/supervisor-and-application.html">guide chapter about creating OTP applications, supervisors and servers</a>. Give it a try!</p> <h2 id="improved-unicode-support">Improved Unicode support</h2> <p>Elixir favors the use of UTF-8 binaries since its first release. In the latest releases, we took it up a notch by adding Unicode support, built upon the Unicode Standard 6.2.0. Elixir v0.8 takes this even further, adding more convenience functions and better support to named sequences:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="no">String</span><span class="o">.</span><span class="n">capitalize</span><span class="p">(</span><span class="s2">"fiN"</span><span class="p">)</span> <span class="c1">#=&gt; "Fin"</span> </code></pre></div></div> <p>The example above contains a string with only two codepoints, <a href="http://www.fileformat.info/info/unicode/char/FB01/index.htm">the codepoint fi</a> and <a href="http://www.fileformat.info/info/unicode/char/006E/index.htm">the codepoint n</a>. Look how Elixir properly capitalizes the string, returning a new string made of three codepoints (all ascii letters).</p> <p>Learn more about <a href="https://hexdocs.pm/elixir/String.html">Unicode support with the String module</a>.</p> <h2 id="ast-metadata">AST metadata</h2> <p>As per this release, Elixir AST nodes can contain metadata. This metadata is compilation time only but may allow macros to annotate important information in AST nodes, like line numbers, file and other library specific information. If you quote an Elixir expression, we can see the metadata slot:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">quote</span> <span class="k">do</span><span class="p">:</span> <span class="n">hello</span><span class="p">(</span><span class="s2">"world"</span><span class="p">)</span> <span class="p">{</span> <span class="ss">:hello</span><span class="p">,</span> <span class="p">[],</span> <span class="p">[</span><span class="s2">"world"</span><span class="p">]</span> <span class="p">}</span> </code></pre></div></div> <p>In the example above, we can see the AST representation of the expression <code class="language-plaintext highlighter-rouge">hello("world")</code>. It is made of a tuple of three elements, the first one is the function name represented by the atom <code class="language-plaintext highlighter-rouge">:hello</code>, the second one is a keyword list containing metadata (in this case, no metadata is available) and the third is a list of arguments, containing the string “world”.</p> <p>By default, <code class="language-plaintext highlighter-rouge">quote</code> does not annotate line numbers, but we can pass it as an option:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">quote</span> <span class="ss">line:</span> <span class="n">__ENV__</span><span class="o">.</span><span class="n">line</span><span class="p">,</span> <span class="k">do</span><span class="p">:</span> <span class="n">hello</span><span class="p">(</span><span class="s2">"world"</span><span class="p">)</span> <span class="p">{</span> <span class="ss">:hello</span><span class="p">,</span> <span class="p">[</span><span class="ss">line:</span> <span class="mi">9</span><span class="p">],</span> <span class="p">[</span><span class="s2">"world"</span><span class="p">]</span> <span class="p">}</span> </code></pre></div></div> <p>Now, we can see the metadata spot being used to annotate the line number. This change allowed us to take our macros one step further…</p> <h2 id="macros-expansion">Macros expansion</h2> <p>Prior to this release, Elixir had limited expansion of imports and aliases. We decided this would be an important issue to tackle in this release, as people are building more and more projects on top of Elixir.</p> <p>Imagine you manually implemented <code class="language-plaintext highlighter-rouge">unless</code> as a macro, that does the opposite of <code class="language-plaintext highlighter-rouge">if</code>:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmacro</span> <span class="k">unless</span><span class="p">(</span><span class="n">expr</span><span class="p">,</span> <span class="n">opts</span><span class="p">)</span> <span class="k">do</span> <span class="kn">quote</span> <span class="k">do</span> <span class="k">if</span><span class="p">(</span><span class="n">!unquote</span><span class="p">(</span><span class="n">expr</span><span class="p">),</span> <span class="kn">unquote</span><span class="p">(</span><span class="n">opts</span><span class="p">))</span> <span class="k">end</span> <span class="k">end</span> </code></pre></div></div> <p>When some code call the <code class="language-plaintext highlighter-rouge">unless</code> macro above, in previous Elixir versions, it would expect the <code class="language-plaintext highlighter-rouge">if</code> macro to be available at the caller. This may not be necessarily true and, even worse, another implementation of the <code class="language-plaintext highlighter-rouge">if</code> macro, not compatible to the one above, could be available.</p> <p>Elixir v0.8 ensures that the <code class="language-plaintext highlighter-rouge">unless</code> macro above will expand to the same <code class="language-plaintext highlighter-rouge">if</code> macro available when quoted, guaranteeing different libraries can integrate easily without imposing hidden requirements.</p> <p>You can read more about <a href="https://hexdocs.pm/elixir/case-cond-and-if.html">macros in the getting started guide</a> or <a href="https://hexdocs.pm/elixir/Kernel.SpecialForms.html#quote/2">go deep into the quote macro docs</a>.</p> <h2 id="a-new-way-to-manipulate-pathnames">A new way to manipulate pathnames</h2> <p>Elixir v0.8 contains a bit of house cleaning too. We have created <a href="https://hexdocs.pm/elixir/Path.html">the Path module</a> to accommodate functions used to manipulate filesystem paths and have also added functions like <a href="https://hexdocs.pm/elixir/System.html"><code class="language-plaintext highlighter-rouge">System.tmp_dir</code> and <code class="language-plaintext highlighter-rouge">System.user_home</code></a> which are meant to work across different operating systems and are very handy when scripting.</p> <h2 id="the-new-hashdict">The new HashDict</h2> <p>For last but not least, Elixir ships with a <a href="https://github.com/elixir-lang/elixir/blob/main/lib/elixir/lib/hash_dict.ex">new HashDict implementation</a>. In Erlang, there are different key-value store implementations and often you need to pick which one is the best for you based on the average size of the dictionary. Generally speaking, <a href="http://www.erlang.org/doc/man/orddict.html">orddicts</a> are efficient and fast when you want to hold a handful of items, otherwise you should consider <a href="http://www.erlang.org/doc/man/gb_trees.html">gb_trees</a> unless you want to hold thousands of items, when then <a href="http://www.erlang.org/doc/man/dict.html">dict</a> becomes your best option. The fact those implementations do not provide the same API, makes it harder to change your code when you realize another implementation would be better fit.</p> <p>For Elixir, we decided to have a single dictionary implementation that would scale as needed. It would start as a compact representation for a handful of items and expand and rehash accordingly as new items are added or removed, providing fast access and modification times on all ranges. We are glad to say our goals were reached and a new <code class="language-plaintext highlighter-rouge">HashDict</code> implementation ships with Elixir v0.8.</p> <p>Let’s take a look at some benchmarks:</p> <p><img src="/images/contents/hash-dict-fetch.png" alt="Comparison of fetch times with string keys" /></p> <p><img src="/images/contents/hash-dict-update.png" alt="Comparison of update times with string keys" /></p> <p>For each number of keys, we have measured and normalized those values against <code class="language-plaintext highlighter-rouge">HashDict</code> results. This way it is easy to see which implementation takes more or less time compared to Elixir’s implementation.</p> <p><code class="language-plaintext highlighter-rouge">orddict</code> is still the faster representation for small ranges since it is a simple list. However, <code class="language-plaintext highlighter-rouge">HashDict</code> is able to be relatively fast compared to <code class="language-plaintext highlighter-rouge">orddict</code> for those small ranges and the fastest solution once you have dozens of keys. <a href="https://gist.github.com/436a9d2bca5051a6dfab">Those results can be verified when using other types as keys as well</a>.</p> <p>Finally, given <code class="language-plaintext highlighter-rouge">HashDict</code> starts with a compact representation, it also takes less memory. Compared to the <code class="language-plaintext highlighter-rouge">dict</code> implementation, an empty <code class="language-plaintext highlighter-rouge">HashDict</code> takes only 5 words, while <code class="language-plaintext highlighter-rouge">dict</code> takes 47.</p> <h2 id="wrapping-up">Wrapping up</h2> <p>We continue actively working on Elixir and this release is the <a href="https://github.com/elixir-lang/elixir/blob/v0.8.0/CHANGELOG.md">result of our efforts on different areas</a>! We have exciting plans and newer possibilities to explore, as a new release of Erlang OTP also comes out in a couple weeks.</p> <p>Also, we previously announced Elixir is going to be released frequently, every 2 to 4 weeks. We have made a small detour to get v0.8.0 out of the door, but we are back to our regular schedule as of today!</p> <p><a href="https://hexdocs.pm/elixir/introduction.html">Celebrate with us and give Elixir a try</a>!</p> Yurii Rashkovskii 2012-12-04T00:00:00+00:00 /blog/2012/12/04/elixir-v0-7-2-released <p>Hot out of the oven! We just released Elixir v0.7.2 with a number of delicious improvements.</p> <p>One of the most important changes in this minor release is a partial rehaul of the type specification syntax.</p> <p>Here’s the gist:</p> <div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">@spec</span> <span class="n">myfun</span><span class="p">(</span><span class="n">integer</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="n">integer</span> <span class="c1"># becomes</span> <span class="nv">@spec</span> <span class="n">myfun</span><span class="p">(</span><span class="n">integer</span><span class="p">)</span> <span class="p">::</span> <span class="n">integer</span> <span class="nv">@type</span> <span class="n">a</span> <span class="p">::</span> <span class="n">fun</span> <span class="c1"># becomes</span> <span class="nv">@type</span> <span class="n">a</span> <span class="p">::</span> <span class="p">(</span><span class="o">...</span> <span class="o">-&gt;</span> <span class="n">any</span><span class="p">)</span> <span class="ow">or</span> <span class="p">((</span><span class="o">...</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">any</span><span class="p">)</span> <span class="ow">or</span> <span class="p">(</span><span class="n">fun</span><span class="p">(</span><span class="o">...</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">any</span><span class="p">)</span> <span class="nv">@type</span> <span class="n">a</span> <span class="p">::</span> <span class="n">fun</span><span class="p">(</span><span class="k">do</span><span class="p">:</span> <span class="n">integer</span><span class="p">)</span> <span class="c1"># becomes</span> <span class="nv">@type</span> <span class="n">a</span> <span class="p">::</span> <span class="p">(()</span> <span class="o">-&gt;</span> <span class="n">integer</span><span class="p">)</span> <span class="ow">or</span> <span class="p">(</span><span class="n">fun</span><span class="p">()</span> <span class="o">-&gt;</span> <span class="n">integer</span><span class="p">)</span> <span class="nv">@type</span> <span class="n">a</span> <span class="p">::</span> <span class="n">fun</span><span class="p">(</span><span class="n">integer</span><span class="p">,</span> <span class="k">do</span><span class="p">:</span> <span class="n">integer</span><span class="p">)</span> <span class="c1"># becomes</span> <span class="nv">@type</span> <span class="n">a</span> <span class="p">::</span> <span class="p">(</span><span class="n">integer</span> <span class="o">-&gt;</span> <span class="n">integer</span><span class="p">)</span> <span class="ow">or</span> <span class="p">((</span><span class="n">integer</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">integer</span><span class="p">)</span> <span class="ow">or</span> <span class="p">(</span><span class="n">fun</span><span class="p">(</span><span class="n">integer</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">integer</span><span class="p">)</span> <span class="nv">@type</span> <span class="n">a</span> <span class="p">::</span> <span class="n">fun</span><span class="p">(</span><span class="n">integer</span><span class="p">,</span> <span class="n">integer</span><span class="p">,</span> <span class="k">do</span><span class="p">:</span> <span class="n">integer</span><span class="p">)</span> <span class="c1"># becomes</span> <span class="nv">@type</span> <span class="n">a</span> <span class="p">::</span> <span class="p">(</span><span class="n">integer</span><span class="p">,</span> <span class="n">integer</span> <span class="o">-&gt;</span> <span class="n">integer</span><span class="p">)</span> <span class="ow">or</span> <span class="p">((</span><span class="n">integer</span><span class="p">,</span> <span class="n">integer</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">integer</span><span class="p">)</span> <span class="ow">or</span> <span class="p">(</span><span class="n">fun</span><span class="p">(</span><span class="n">integer</span><span class="p">,</span> <span class="n">integer</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">integer</span><span class="p">)</span> </code></pre></div></div> <p>Another change is that Mix now echoes the output of external tools such as git and rebar, and handles exit status correctly. This have previously led to some less-than-perfect workflows.</p> <p>We’ve also added a more compact and visual form of the <code class="language-plaintext highlighter-rouge">function</code> helper. Now, instead of <code class="language-plaintext highlighter-rouge">function(Enum, :all?, 2)</code> you can use <code class="language-plaintext highlighter-rouge">function(Enum.all?/2)</code>.</p> <p>We’ve also figured out how to achieve an up to 6x <a href="https://github.com/elixir-lang/elixir/blob/v0.7.2/lib/elixir/lib/kernel.ex#L1386-L1417">performance increase</a> under some circumstances when using records.</p> <p>…and <a href="https://github.com/elixir-lang/elixir/blob/v0.7.2/CHANGELOG.md">many other fixes &amp; improvements</a>.</p> <p>Lastly, but not least importantly, I’d like to mention that we’re very excited about how the community around Elixir is building up. Thank you all for being around and supporting us!</p> <p><a href="https://hexdocs.pm/elixir/introduction.html">Learn more about Elixir</a>!</p> José Valim 2012-11-18T00:00:00+00:00 /blog/2012/11/18/elixir-v0-7-1-released <p>Elixir v0.7.1 was released this weekend to celebrate the end of a two months journey traveling around Europe, United States and Brazil talking about and exposing Elixir to new developers.</p> <p>This is a minor release that contains a couple enhancements regarding UTF-8, <a href="http://www.erlang.org/doc/man/dialyzer.html">dialyzer</a> support and bug fixes.</p> <p>During this time traveling around, we have spoken at many conferences, as <a href="http://thestrangeloop.com/">Strange Loop</a>, <a href="http://oredev.org/">Øredev</a>, <a href="http://qconsp.com/">QCon SP</a> and <a href="http://rupy.eu/">Rupy</a> as well as at different companies. Developers from different backgrounds have shown interest in Elixir, <a href="http://spin.atomicobject.com/2012/10/31/elixir-erlang-and-the-dining-philosophers/">written about it</a>, and contributed to the language. As of today, Elixir is powered by 51 different contributors!</p> <p>In case you missed any of those conferences, <a href="https://vimeo.com/53221562">the talk I presented at Øredev is available and you can watch it now</a>. The slides are also available below.</p> <p>If you want to hear more about Elixir at a conference or an event, please let us know. Thank you and don’t forget to <a href="https://hexdocs.pm/elixir/introduction.html">give Elixir a try</a>!</p> <script async="" class="speakerdeck-embed" data-id="cf4727401449013077d112313d1a82a3" data-ratio="1.2994923857868" src="//speakerdeck.com/assets/embed.js"></script> José Valim 2012-10-20T00:00:00+00:00 /blog/2012/10/20/elixir-v0-7-0-released <p>Elixir v0.7.0 was released with bug fixes and many improvements, like a <code class="language-plaintext highlighter-rouge">String</code> module to handle UTF-8 binaries and support to environments and nested dependencies in Mix.</p> <p>We have also taken important steps into normalizing our APIs. In Erlang, accesses to tuple and lists are one-based and binaries are zero-based, but in Elixir we have normalized all of them to rely on zero-based access.</p> <p>This release also includes some backwards incompatible changes, but the majority of changes were first deprecated, meaning your code will run just fine but with warnings. Those warnings will be removed in the next release v0.7.1, which should happen in a 2 to 4 weeks time span.</p> <p>For more information, read out the <a href="https://github.com/elixir-lang/elixir/blob/v0.7.0/CHANGELOG.md">CHANGELOG</a>.</p> <p>Thank you and don’t forget to <a href="https://hexdocs.pm/elixir/introduction.html">give Elixir a try</a>!</p> José Valim 2012-08-01T00:00:00+00:00 /blog/2012/08/01/elixir-v0-6-0-released <p>We have finally released <a href="/">Elixir</a> v0.6.0! This release includes a build tool called Mix, support for Erlang typespecs, many improvements to IEx and improved IO, File and Macro support.</p> <h2 id="whats-new">What’s new</h2> <p>When <a href="/blog/2012/05/25/elixir-v0-5-0-released/">we released version v0.5.0</a>, we have set three major goals for release v0.6.0:</p> <ol> <li>Provide a build tool that makes it easy to create, compile and test Elixir projects;</li> <li>Support <a href="http://www.erlang.org/doc/reference_manual/typespec.html">Erlang typespecs</a>;</li> <li>Improve IO and File modules to be more robust and complete.</li> </ol> <p>Our interactive shell (IEx) also had many improvements, thanks to the Elixir developer community. We now have easy access to documentation, remote shells, autocomplete and much more. In order to show you a bit of what you can do in this release, we have prepared a short (~6 min) screencast:</p> <iframe src="https://player.vimeo.com/video/46709928" title="Elixir v0.6 quick tour - Mix and IEx" class="video" width="600" height="337" allowfullscreen=""></iframe> <p><a href="https://vimeo.com/46709928">Elixir v0.6 quick tour - Mix and IEx</a> from <a href="https://vimeo.com/user3182384">Plataformatec</a> on <a href="https://vimeo.com">Vimeo</a>.</p> <p>That’s it. For the next months, we will continue improving Elixir (you can see some ideas floating around in the <a href="https://github.com/elixir-lang/elixir/issues">issues tracker</a>) but we will start to focus on other tools and libraries for the community.</p> <p>Thank you and don’t forget to <a href="https://hexdocs.pm/elixir/introduction.html">give Elixir a try</a>!</p> José Valim 2012-05-25T00:00:00+00:00 /blog/2012/05/25/elixir-v0-5-0-released <p>We have finally released <a href="/">Elixir</a> v0.5.0! This marks the first release since the language was rewritten. In this blog post, we will discuss what we achieved during this time and what are the next steps!</p> <p>If you don’t care about any of these, you can go straight to our <a href="https://hexdocs.pm/elixir/introduction.html">Getting Started guide</a>. If you do, keep on reading!</p> <h2 id="looking-back">Looking back</h2> <p>I have started working in Elixir at the beginning of 2011. Around April that year, I had released the version v0.3.0 that was stable enough for me to start using in my own projects. However, after using it in a couple projects quickly reviewed that I was not happy with some of the design decisions taken early on.</p> <p>At that time, Elixir attempted to be a considerable departure from Erlang and that revealed very fast to a bad design decision because, in order to use any Erlang module, we first would have to provide an Elixir wrapper for it. Any new function or module in new Erlang releases would have to be wrapped first in Elixir, which means we would always play catch up with Erlang.</p> <p>After not feeling productive enough with that Elixir version, I have decided to take a break from Elixir to study old, new and emerging languages. The challenge was to not re-invent Erlang as a language, but how to provide the productivity and flexibility I expect from Elixir while staying a 100% compatible with Erlang.</p> <p>It was around October 2011, during a short stay in San Francisco, that I came up with what would be <a href="https://github.com/josevalim/lego-lang">the foundation of Elixir’s current version</a> with the help of Yehuda Katz. Development of the new Elixir version started a few days before 2012 and continued steady when the new year came in.</p> <p>Around February of that year, feeling confident enough about the direction the language was moving (and initial benchmarks I had made at that point), I have pitched Elixir to <a href="http://plataformatec.com.br/">my company, Plataformatec</a>, and they have accepted to sponsor Elixir. With their help, Elixir developed even faster and that’s what we are going to take a look next.</p> <h2 id="where-we-are">Where we are</h2> <p>One of the goals we have set was to have a good website and documentation before the next official release. With the help of the Plataformatec team, we created a logo for Elixir and put this website live.</p> <p>At the same time, <a href="https://bitbucket.org/birkenfeld/pygments-main/pull-request/57/add-elixir-and-elixir-console-lexers">we were working on pygments support</a>, a <a href="https://github.com/elixir-lang/ex_doc">documentation generation tool</a> and many others. Soon, GitHub was able to syntax highlight Elixir code and <a href="/">our API documentation was online</a>.</p> <p>At the same time, people started to gather around #elixir-lang channel on irc.freenode.net (now migrated to #elixir on irc.libera.chat) to <a href="https://github.com/elixir-lang/elixir/tree/main/lib/mix">play with Elixir</a>, <a href="https://github.com/guedes/exdate">start their</a> <a href="https://github.com/yrashk/validatex">own projects</a> and <a href="https://github.com/alco/elixir/wiki/Erlang-Syntax:-A-Crash-Course">tutorials</a>.</p> <p>Although the initial release was scheduled to April 2012, the feedback from such early developers forced us to review some design and syntax decisions and were extremely important to shape the language as it is today.</p> <p>With v0.5.0 finally out, we are committing to a stable syntax and a basic standard library. In the last couple days before the release, we have been working on streamlining the documentation and ensure Elixir works on Mac, Linux and Windows machines!</p> <h2 id="looking-forward">Looking forward</h2> <p>There are still many, many things to do! In the next months, we will continue working on growing our community, talks and other documentation material. A huge thanks to <a href="https://twitter.com/true_droid">Alexei Sholik</a> who is moving this area forward.</p> <p>We will also work on better integration and documentation on building Erlang systems. Erlang ships with the <a href="https://en.wikipedia.org/wiki/Open_Telecom_Platform">Open Telecom Platform</a> which provides many tools to build distributed applications. In v0.5.0, all these tools are already available but we want to make the build process even simpler.</p> <p>In parallel, we will improve our <a href="https://github.com/elixir-lang/ex_doc">documentation generation tool</a> and <a href="https://github.com/elixir-lang/elixir/tree/main/lib/mix">build tool</a> which will likely be merged into core when they are solid enough.</p> <p>Finally, we will continue improving the Standard Library. Although Elixir’s goal is to rely on Erlang the most as possible, we also want to provide a small Standard Library which makes better use of Elixir semantics. For the next weeks, we will focus on improving the IO and File manipulation modules. New data types may also appear, for example, ranges come to my mind.</p> <p>Check out our <a href="/">home page</a> and the <a href="https://hexdocs.pm/elixir/introduction.html">getting started guide</a> for more information. Welcome aboard and grab a cup of Elixir, because you are certainly going to enjoy the ride!</p> José Valim 2012-04-24T00:00:00+00:00 /blog/2012/04/24/a-peek-inside-elixir-s-parallel-compiler <p>Today, a parallel compiler just landed in Elixir main. The goal of the parallel compiler is to compile files in parallel, automatically detecting dependencies between files. In this blog post, we are going to take a peek into the parallel compiler internals and learn more about Erlang and Elixir in the process.</p> <h2 id="process-based-serial-compilation">Process-based serial compilation</h2> <p>The idea of the parallel compiler is very simple: for each file we want to compile, we will spawn a new process that will be responsible for its compilation. When compilation finishes, the process is going to send a message to the main process (the one responsible for coordinating compilation) that compilation finished so a new file can be compiled.</p> <p>In Elixir, we could write this code as follows:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>def spawn_compilers([current | files], output) do parent = Process.self() child = spawn_link(fn -&gt; :elixir_compiler.file_to_path(current, output) send parent, { :compiled, Process.self() } end) receive do { :compiled, ^child } -&gt; spawn_compilers(files, output) { :EXIT, ^child, { reason, where } } -&gt; :erlang.raise(:error, reason, where) end end def spawn_compilers([], _output) do :done end </code></pre></div></div> <p>In the first line, we define a function named <code class="language-plaintext highlighter-rouge">spawn_compilers</code> that receives two arguments, the first is a list of files to compile and the second is a string telling us where to write the compiled file. The first argument is represented as a list with head and tail (<code class="language-plaintext highlighter-rouge">[current | files]</code>) where the top of the list is assigned to <code class="language-plaintext highlighter-rouge">current</code> and the remaining items to <code class="language-plaintext highlighter-rouge">files</code>. If the list is empty, the first clause of <code class="language-plaintext highlighter-rouge">spawn_compilers</code> is not going to match, the clause <code class="language-plaintext highlighter-rouge">spawn_compilers([], _output)</code> defined at the end will instead.</p> <p>Inside <code class="language-plaintext highlighter-rouge">spawn_compilers</code>, we first retrieve the PID of the current process with <code class="language-plaintext highlighter-rouge">Process.self</code> (remember we are talking about Erlang processes/actors and not OS processes) and then proceed to spawn a new process to execute the given function in parallel. Spawning a new process is done with the <code class="language-plaintext highlighter-rouge">spawn_link</code> function.</p> <p>The <code class="language-plaintext highlighter-rouge">spawn_link</code> function starts a new process and automatically links the current (parent) process with the spawned (child) one, returning the child PID. By linking the process we ensure that, if the child process dies, a message will be sent to the parent process which then can act on it.</p> <p>The function given to <code class="language-plaintext highlighter-rouge">spawn_link</code> is quite straight-forward. It simply invokes an Erlang function as <code class="language-plaintext highlighter-rouge">:elixir_compiler.file_to_path</code> and then proceeds to send a message to the parent process notifying that compilation finished.</p> <p>After the child process is spawned, we invoke the <code class="language-plaintext highlighter-rouge">receive</code> macro and start waiting for messages. At this point, we are expecting two types of messages:</p> <ul> <li> <p><code class="language-plaintext highlighter-rouge">{ :compiled, ^child }</code> - a message sent by the child informing us that compilation finished. Note that use of <code class="language-plaintext highlighter-rouge">^</code> before the variable <code class="language-plaintext highlighter-rouge">child</code> to tell Elixir to match the current value of <code class="language-plaintext highlighter-rouge">child</code> with the one received in the message. If compilation succeeds, we move forward and spawn the next child by calling <code class="language-plaintext highlighter-rouge">spawn_compilers</code> recursively;</p> </li> <li> <p><code class="language-plaintext highlighter-rouge">{ :EXIT, ^child, { reason, where } }</code> - this is the message sent by the child process in case it dies. This message is only received if the child is started via <code class="language-plaintext highlighter-rouge">spawn_link</code>. In the message, we can find the reason why it failed and the stacktrace. We then proceed to call an Erlang internal function to re-raise the error in the main process, effectively stopping compilation.</p> </li> </ul> <p>With this code, we were able to compile each file inside a different process. However, notice that we are not yet compiling in parallel. Every time we spawn a child process, we wait until it succeeds (or fails) before moving to the next step. We are going to eventually compile files in parallel, but before we reach to this point, let’s understand the problem of dependencies between files.</p> <h2 id="dependency-between-files">Dependency between files</h2> <p>Imagine that we have two files, <code class="language-plaintext highlighter-rouge">a.ex</code> and <code class="language-plaintext highlighter-rouge">b.ex</code>, with the following contents:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code># a.ex defmodule A do B.define end # b.ex defmodule B do defmacro define do quote do def one, do: 1 end end end </code></pre></div></div> <p>In order to compile <code class="language-plaintext highlighter-rouge">A</code>, we need to ensure that <code class="language-plaintext highlighter-rouge">B</code> is already compiled and loaded so we can invoke the <code class="language-plaintext highlighter-rouge">define</code> macro. This means the file <code class="language-plaintext highlighter-rouge">a.ex</code> depends on the file <code class="language-plaintext highlighter-rouge">b.ex</code>. When compiling files in parallel, we want to be able to detect such cases and automatically handle them.</p> <p>The way we are going to handle this is by pausing compilation every time a module that was not yet defined is invoked. In this case, when compiling the file <code class="language-plaintext highlighter-rouge">a.ex</code> and <code class="language-plaintext highlighter-rouge">B.define</code> is invoked, the process responsible for compiling <code class="language-plaintext highlighter-rouge">a.ex</code> is going to pause and notify our main process. The main process will then start the compilation of other files. Whenever the module <code class="language-plaintext highlighter-rouge">B</code> is compiled, the main process is going to tell the process responsible for <code class="language-plaintext highlighter-rouge">a.ex</code> to resume compilation since its dependency <code class="language-plaintext highlighter-rouge">B</code> is now available.</p> <p>In order to customize this process, we are going to take a look at Erlang’s error handler.</p> <h2 id="custom-error-handler">Custom error handler</h2> <p>By default, Elixir (and Erlang) code is autoloaded. This means that, if we invoke <code class="language-plaintext highlighter-rouge">List.delete</code> and the module <code class="language-plaintext highlighter-rouge">List</code> was not loaded yet, the Erlang VM is going to look into the <code class="language-plaintext highlighter-rouge">ebin</code> directory (the directory where we put compiled files) and try to load it. This process is controlled by the <a href="http://www.erlang.org/doc/man/error_handler.html"><code class="language-plaintext highlighter-rouge">error_handler</code> module in Erlang</a> via two callback functions: <code class="language-plaintext highlighter-rouge">undefined_function</code> and <code class="language-plaintext highlighter-rouge">undefined_lambda</code>.</p> <p>As discussed in the previous section, we want to extend the error handler to actually stop the currently running process whenever a module is not found and resume the process only after we ensure the module is compiled. To do that, we can simply define our own error handler and ask Erlang to use it. Our custom error handler is defined as follows:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>defmodule Elixir.ErrorHandler do def undefined_function(module, fun, args) do ensure_loaded(module) :error_handler.undefined_function(module, fun, args) end def undefined_lambda(module, fun, args) do ensure_loaded(module) :error_handler.undefined_lambda(module, fun, args) end defp ensure_loaded(module) do case Code.ensure_loaded(module) do { :module, _ } -&gt; [] { :error, _ } -&gt; parent = Process.get(:elixir_parent_compiler) send parent, { :waiting, Process.self, module } receive do { :release, ^parent } -&gt; ensure_loaded(module) end end end end </code></pre></div></div> <p>Our error handler defines two public functions. Both those functions are callbacks required to be implemented by the error handler. They simply call <code class="language-plaintext highlighter-rouge">ensure_loaded(module)</code> and then delegate the remaining logic to Erlang’s original <code class="language-plaintext highlighter-rouge">error_handler</code>.</p> <p>The private <code class="language-plaintext highlighter-rouge">ensure_loaded</code> function calls <code class="language-plaintext highlighter-rouge">Code.ensure_loaded(module)</code> which checks if the given module is loaded and, if not, tries to load it. In case it succeeds, it returns <code class="language-plaintext highlighter-rouge">{ :module, _ }</code>, which means the module is available and we don’t need to stop the current process. However, if it returns <code class="language-plaintext highlighter-rouge">{ :error, _ }</code>, it means the module cannot be found and we need to wait until it is compiled. For that, we invoke <code class="language-plaintext highlighter-rouge">Process.get(:elixir_parent_compiler)</code> to get the PID of the main process so we can notify it that we are waiting on a given module. Then we invoke the macro <code class="language-plaintext highlighter-rouge">receive</code> as a way to stop the current process until we receive a message from the parent saying new modules are available, starting the flow again.</p> <p>With our error handler code in place, the first thing we need to do is to change the function given to <code class="language-plaintext highlighter-rouge">spawn_link</code> to use the new error handler:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>spawn_link(fn -&gt; Process.put(:elixir_parent_compiler, parent) Process.flag(:error_handler, Elixir.ErrorHandler) :elixir_compiler.file_to_path(current, output) send parent, { :compiled, Process.self() } end) </code></pre></div></div> <p>Notice that we have two small additions. First we store the <code class="language-plaintext highlighter-rouge">:elixir_parent_compiler</code> PID in the process dictionary so we are able to read it from the error handler and then we proceed to configure a flag in our process so our new error handler is invoked whenever a module or function cannot be found.</p> <p>Second, our main process can now receive a new <code class="language-plaintext highlighter-rouge">{ :waiting, child, module }</code> message, so we need to extend it to account for those messages. Not only that, we need to control which PIDs we have spawned so we can notify them whenever a new module is compiled, forcing us to add a new argument to the <code class="language-plaintext highlighter-rouge">spawn_compilers</code> function. <code class="language-plaintext highlighter-rouge">spawn_compilers</code> would then be rewritten as follows:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>def spawn_compilers([current | files], output, stack) do parent = Process.self() child = spawn_link(fn -&gt; :elixir_compiler.file_to_path(current, output) send parent, { :compiled, Process.self() } end) wait_for_messages(files, output, [child | stack]) end # No more files and stack is empty, we are done def spawn_compilers([], _output, []) do :done end # No more files and stack is not empty, wait for all messages def spawn_compilers([], output, stack) do wait_for_messages([], output, stack) end </code></pre></div></div> <p>Notice we added an extra clause to <code class="language-plaintext highlighter-rouge">spawn_compilers</code> so we can properly handle the case where we don’t have more files to spawn but we are still waiting for processes in the stack. We have also moved our <code class="language-plaintext highlighter-rouge">receive</code> logic to a new private function called <code class="language-plaintext highlighter-rouge">wait_for_messages</code>, implemented as follows:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>defp wait_for_messages(files, output, stack) do receive do { :compiled, child } -&gt; new_stack = List.delete(stack, child) Enum.each new_stack, fn(pid) -&gt; send pid, { :release, Process.self } end spawn_compilers(files, output, new_stack) { :waiting, _child, _module } -&gt; spawn_compilers(files, output, stack) { :EXIT, _child, { reason, where } } -&gt; :erlang.raise(:error, reason, where) after 10_000 -&gt; raise "dependency on nonexistent module or possible deadlock" end end </code></pre></div></div> <p>The implementation for <code class="language-plaintext highlighter-rouge">wait_for_messages</code> is now broken into 4 clauses:</p> <ul> <li> <p><code class="language-plaintext highlighter-rouge">{ :compiled, child }</code> - Similar as before, it is the notification a child processed finished compilation. Every time we receive such notifications, we remove the child PID from the stack and notify the remaining PIDs in the stack that new modules are available. Notice that we no longer match on a specific <code class="language-plaintext highlighter-rouge">^child</code> PID, since now we can receive messages from different children at the same time;</p> </li> <li> <p><code class="language-plaintext highlighter-rouge">{ :waiting, _child, _module }</code> - A message received every time a child process is waiting on a module to be compiled. In this scenario, all we do is spawn a new process to compile another file, ensuring compilation is never blocked;</p> </li> <li> <p><code class="language-plaintext highlighter-rouge">{ :EXIT, _child, { reason, where } }</code> - The same behaviour as before, it simply raises an error if any of the child processes fail;</p> </li> <li> <p><code class="language-plaintext highlighter-rouge">after: 10_000</code> - This clause is going to be invoked whenever the main process does not receive a message for 10 seconds. This means a file depends on a module that does not exist (and therefore waits forever) or there is a cyclic dependency;</p> </li> </ul> <p>And that’s all we need to have a basic version of our parallel compilation working. Notice we start compiling only one file at a time but, as soon as we depend on other files, the number of PIDs in the stack starts to grow. If we wanted, we could modify the code to make use of a head start and compile more than one file since the beginning.</p> <p>It is important to notice that this code has room for improvements. First, every time a new module is compiled, we notify all child process that new modules are available. This is a waste of resource if we consider that the child modules tells us explicitly on which modules they are waiting on. Therefore, the code could be modified to store a mapping from each child process to the module it is waiting for so that when a new module is compiled, only the children that depend on it are notified.</p> <p>Also, if we start storing which module each process is depending on, we are able to know whenever we have a deadlock or a dependency on an nonexistent file, allowing us to get rid of the timeout.</p> <p>All those improvements and other goodies like callbacks are implemented in Elixir source code and we recommend you take a look at both the <a href="https://github.com/elixir-lang/elixir/blob/6182602f1205e2d9fc54666e0721270a27226fbc/lib/elixir/parallel_compiler.ex">Elixir.ParallelCompiler</a> and <a href="https://github.com/elixir-lang/elixir/blob/6182602f1205e2d9fc54666e0721270a27226fbc/lib/elixir/error_handler.ex">Elixir.ErrorHandler</a> modules to see all the details firsthand.</p> <p>Happy coding!</p>