Erlang -- List handling

User's Guide
PDF
Top

Efficiency Guide
User's Guide
Version 5.8.3

Expand All
Contract All

Chapters

- Introduction
  - Top of chapter
  - Purpose
  - Prerequisites
- The Eight Myths of Erlang Performance
  - Top of chapter
  - Myth: Funs are slow
  - Myth: List comprehensions are slow
  - Myth: Tail-recursive functions are MUCH faster than recursive functions
  - Myth: '++' is always bad
  - Myth: Strings are slow
  - Myth: Repairing a Dets file is very slow
  - Myth: BEAM is a stack-based byte-code virtual machine (and therefore slow)
  - Myth: Use '_' to speed up your program when a variable is not used
- Common Caveats
  - Top of chapter
  - The regexp module
  - The timer module
  - list_to_atom/1
  - length/1
  - setelement/3
  - size/1
  - split_binary/2
  - The '--' operator
- Constructing and matching binaries
  - Top of chapter
  - How binaries are implemented
  - Constructing binaries
  - Matching binaries
- List handling
  - Top of chapter
  - Creating a list
  - List comprehensions
  - Deep and flat lists
  - Why you should not worry about recursive lists functions
- Functions
  - Top of chapter
  - Pattern matching
  - Function Calls
  - Memory usage in recursion
- Tables and databases
  - Top of chapter
  - Ets, Dets and Mnesia
  - Ets specific
  - Mnesia specific
- Processes
  - Top of chapter
  - Creation of an Erlang process
  - Process messages
  - The SMP emulator
- Drivers
  - Top of chapter
  - Drivers and concurrency
  - Avoiding copying of binaries when calling a driver
  - Returning small binaries from a driver
  - Returning big binaries without copying from a driver
- Advanced
  - Top of chapter
  - Memory
  - System limits
- Profiling
  - Top of chapter
  - Do not guess about performance - profile
  - Big systems
  - What to look for
  - Tools
  - Benchmarking

# 5 List handling

### 5.1 Creating a list

Lists can only be built starting from the end and attaching list elements at the beginning. If you use the ++ operator like this

    List1 ++ List2

you will create a new list which is copy of the elements in List1, followed by List2. Looking at how lists:append/1 or ++ would be implemented in plain Erlang, it can be seen clearly that the first list is copied:

    append([H|T], Tail) ->
        [H|append(T, Tail)];
    append([], Tail) ->
        Tail.

So the important thing when recursing and building a list is to make sure that you attach the new elements to the beginning of the list, so that you build a list, and not hundreds or thousands of copies of the growing result list.

Let us first look at how it should not be done:

DO NOT

    bad_fib(N) ->
        bad_fib(N, 0, 1, []).

    bad_fib(0, _Current, _Next, Fibs) ->
        Fibs;
    bad_fib(N, Current, Next, Fibs) ->
        bad_fib(N - 1, Next, Current + Next, Fibs ++ [Current]).

Here we are not a building a list; in each iteration step we create a new list that is one element longer than the new previous list.

To avoid copying the result in each iteration, we must build the list in reverse order and reverse the list when we are done:

DO

    tail_recursive_fib(N) ->
        tail_recursive_fib(N, 0, 1, []).

    tail_recursive_fib(0, _Current, _Next, Fibs) ->
        lists:reverse(Fibs);
    tail_recursive_fib(N, Current, Next, Fibs) ->
        tail_recursive_fib(N - 1, Next, Current + Next, [Current|Fibs]).

### 5.2 List comprehensions

Lists comprehensions still have a reputation for being slow. They used to be implemented using funs, which used to be slow.

In recent Erlang/OTP releases (including R12B), a list comprehension

    [Expr(E) || E <- List]

is basically translated to a local function

    'lc^0'([E|Tail], Expr) ->
        [Expr(E)|'lc^0'(Tail, Expr)];
    'lc^0'([], _Expr) -> [].

In R12B, if the result of the list comprehension will obviously not be used, a list will not be constructed. For instance, in this code

    [io:put_chars(E) || E <- List],
    ok.

or in this code

    .
    .
    .
    case Var of
        ... ->
            [io:put_chars(E) || E <- List];
        ... ->
    end,
    some_function(...),
    .
    .
    .

the value is neither assigned to a variable, nor passed to another function, nor returned, so there is no need to construct a list and the compiler will simplify the code for the list comprehension to

    'lc^0'([E|Tail], Expr) ->
        Expr(E),
        'lc^0'(Tail, Expr);
    'lc^0'([], _Expr) -> [].

### 5.3 Deep and flat lists

lists:flatten/1 builds an entirely new list. Therefore, it is expensive, and even more expensive than the ++ (which copies its left argument, but not its right argument).

In the following situations, you can easily avoid calling lists:flatten/1:

- When sending data to a port. Ports understand deep lists so there is no reason to flatten the list before sending it to the port.
- When calling BIFs that accept deep lists, such as list_to_binary/1 or iolist_to_binary/1.
- When you know that your list is only one level deep, you can can use lists:append/1.

Port example

DO

          ...
          port_command(Port, DeepList)
          ...

DO NOT

          ...
          port_command(Port, lists:flatten(DeepList))
          ...

A common way to send a zero-terminated string to a port is the following:

DO NOT

          ...
          TerminatedStr = String ++ [0], % String="foo" => [$f, $o, $o, 0]
          port_command(Port, TerminatedStr)
          ...

Instead do like this:

DO

          ...
          TerminatedStr = [String, 0], % String="foo" => [[$f, $o, $o], 0]
          port_command(Port, TerminatedStr)
          ...

Append example

DO

          > lists:append([[1], [2], [3]]).
          [1,2,3]
          >

DO NOT

          > lists:flatten([[1], [2], [3]]).
          [1,2,3]
          >

### 5.4 Why you should not worry about recursive lists functions

In the performance myth chapter, the following myth was exposed: Tail-recursive functions are MUCH faster than recursive functions.

To summarize, in R12B there is usually not much difference between a body-recursive list function and tail-recursive function that reverses the list at the end. Therefore, concentrate on writing beautiful code and forget about the performance of your list functions. In the time-critical parts of your code (and only there), measure before rewriting your code.

Important note: This section talks about lists functions that construct lists. A tail-recursive function that does not construct a list runs in constant space, while the corresponding body-recursive function uses stack space proportional to the length of the list. For instance, a function that sums a list of integers, should not be written like this

DO NOT

    recursive_sum([H|T]) -> H+recursive_sum(T);
    recursive_sum([])    -> 0.

but like this

DO

    sum(L) -> sum(L, 0).

    sum([H|T], Sum) -> sum(T, Sum + H);
    sum([], Sum)    -> Sum.