Jekyll2024-01-05T12:33:51+00:00https://ufechner7.github.io/feed.xmlJulia programming notesNotes on using the Julia programming languageUweInstalling Julia 1.8 and VSCode2022-08-18T00:00:00+00:002022-08-18T00:00:00+00:00https://ufechner7.github.io/2022/08/18/installing-julia<h2 id="introduction">Introduction</h2>
<p>Installing Julia is easy, but perhaps you also want to install an integrated development environment (IDE) or a version control system (e.g. git), therefore I give some hints how to do that in this blog post.</p>
<p>Furthermore there are different ways to install multiple Julia versions in parallel and to keep your version up-to-date which are also explained in this blog post.</p>
<p>Highlights of version 1.8 of Julia are explained <a href="https://julialang.org/blog/2022/08/julia-1.8-highlights/">here</a> .</p>
<h2 id="installation-of-julia">Installation of Julia</h2>
<h3 id="windows">Windows</h3>
<p>Please download and install Julia as explained <a href="https://julialang.org/downloads/">here</a> .
Choose the “64-bit (installer)”. Make sure to check the option “Add Julia to path” when running the installer.
For advanced users it is suggested also to install <a href="https://gitforwindows.org/">git for Windows</a> which also includes the <code class="language-plaintext highlighter-rouge">bash</code> command line interface, very useful for small automation tasks. The git version control system keeps track of the changes of your files and allows SW development in a team.</p>
<p>It is suggested to launch Julia from the command line, using either the “Windows command prompt” or “bash” by typing <code class="language-plaintext highlighter-rouge">julia</code> or <code class="language-plaintext highlighter-rouge">julia --project</code> (if you work with projects). If you never used a command prompt before, read the <a href="https://www.makeuseof.com/tag/a-beginners-guide-to-the-windows-command-line/">Beginners Guide</a>.</p>
<h4 id="juliaup">Juliaup</h4>
<p>An installer and version manager for Julia called <a href="https://github.com/JuliaLang/juliaup">juliaup</a> is available in the Microsoft Store. It can be used to install specific Julia versions or update to the latest release. This package handles all PATH related aspects of Julia, and alerts users when new Julia versions are released.</p>
<h4 id="uninstallation">Uninstallation</h4>
<p>Uninstallation is preferably performed by using the Windows uninstaller. The directory in %HOME%/.julia can then be deleted if you want to remove all traces of Julia (this includes user installed packages).</p>
<h3 id="linux">Linux</h3>
<p>Copy and past the following line to install the latest stable version of Julia:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>bash -ci "$(curl -fsSL https://raw.githubusercontent.com/abelsiqueira/jill/master/jill.sh)"
</code></pre></div></div>
<p>If you want to be able to easily switch between different versions of Julia consider to install
the Python version of <a href="https://github.com/johnnychen94/jill.py">jill</a> .</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pip <span class="nb">install </span>jill <span class="nt">--user</span> <span class="nt">-U</span>
jill <span class="nb">install </span>1.8
</code></pre></div></div>
<p>If you should later install version 1.9 with <code class="language-plaintext highlighter-rouge">jill install 1.9</code> you can then switch between the versions with <code class="language-plaintext highlighter-rouge">jill switch 1.8</code> etc.</p>
<p>It is suggested to add the following line to your <code class="language-plaintext highlighter-rouge">.bashrc</code> file:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>alias jl='./bin/run_julia'
</code></pre></div></div>
<p>This makes it possible to run julia with the shortcut <code class="language-plaintext highlighter-rouge">jl</code> later, if you have a <code class="language-plaintext highlighter-rouge">run_julia</code> script in the <code class="language-plaintext highlighter-rouge">bin</code> folder of your projects. I suggest to use such a script, the most simple version of it would just contain the line <code class="language-plaintext highlighter-rouge">julia --project</code> .</p>
<h3 id="mac">Mac</h3>
<p>Please download Julia <a href="https://julialang.org/downloads/">here</a> .</p>
<p>A julia-1.8.0-mac64.dmg file is provided, which contains Julia-1.8.app. Installation is the same as any other Mac software: drag the Julia-1.8.app to Applications Folder’s Shortcut. The Julia download runs on macOS 10.9 Mavericks and later releases.</p>
<p>You can launch Julia by opening the Julia app like any other application.</p>
<h4 id="add-julia-to-path">Add Julia to PATH</h4>
<p>If you want to launch Julia from the command line, first open a new terminal window, then run the following snippet from your shell (e.g., using the Terminal app, not inside the Julia prompt).</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo mkdir</span> <span class="nt">-p</span> /usr/local/bin
<span class="nb">sudo rm</span> <span class="nt">-f</span> /usr/local/bin/julia
<span class="nb">sudo ln</span> <span class="nt">-s</span> /Applications/Julia-1.8.app/Contents/Resources/julia/bin/julia /usr/local/bin/julia
</code></pre></div></div>
<p>This code creates a symlink to a Julia version (here 1.8) of your choosing. To launch Julia, simply type <code class="language-plaintext highlighter-rouge">julia</code> inside your shell and press return. If you are working with projects, use the command <code class="language-plaintext highlighter-rouge">julia --project</code>.</p>
<h4 id="installing-multiple-versions-in-parallel">Installing multiple versions in parallel</h4>
<p>The jill installers will most likely also work on Mac and allow easy switching of different Julia versions (see Linux section).</p>
<h4 id="uninstallation-1">Uninstallation</h4>
<p>You can uninstall Julia by deleting <code class="language-plaintext highlighter-rouge">Julia.app</code> and the packages directory in <code class="language-plaintext highlighter-rouge">~/.julia</code> . Multiple <code class="language-plaintext highlighter-rouge">Julia.app</code> binaries can co-exist without interfering with each other. If you would also like to remove your preferences files, remove <code class="language-plaintext highlighter-rouge">~/.julia/config/startup.jl</code> and <code class="language-plaintext highlighter-rouge">~/.julia/logs/repl_history.jl</code> .</p>
<h2 id="installation-of-the-ide-vscode">Installation of the IDE VSCode</h2>
<p>It is useful to install the integrated development environment VSCode, even though it is not
required. You can also use any editor of your choice.</p>
<p>VSCode provides syntax highlighting, but also the feature “goto definition” which can help to understand and explore the code.</p>
<p align="center"><img src="https://raw.githubusercontent.com/ufechner7/ufechner7.github.io/main/_posts/vscode.png" width="600" /></p>
<p>You can download and install VSCode for all operating systems <a href="https://code.visualstudio.com/">here</a> .</p>
<p>For Ubuntu Linux the following ppa can be used to install vscode and to keep it up-to-date: <a href="https://www.ubuntuupdates.org/ppa/vscode">https://www.ubuntuupdates.org/ppa/vscode</a> .</p>
<h3 id="installing-the-julia-extension">Installing the Julia extension</h3>
<ul>
<li>Start or open Visual Studio Code.</li>
<li>Select <strong>View</strong> and then click <strong>Extensions</strong> to open Extension View.</li>
<li>Enter the term <code class="language-plaintext highlighter-rouge">julia</code> in the marketplace search box. Click the green <strong>Install</strong> button to download the extension.</li>
</ul>
<p><img src="https://raw.githubusercontent.com/ufechner7/ufechner7.github.io/main/_posts/julia_vscode_extension.png" alt="Julia VSCode extension" /></p>
<p>You successfully downloaded the Julia extension for VS Code.</p>
<p><em>NOTE:</em> It is recommended that you restart VS Code after installation.</p>
<p>Julia development with VSCode is well documented here: <a href="https://www.julia-vscode.org/docs/stable/">Julia Visual Studio Code Documentation</a></p>
<p>I would NOT use all the advanced features of julia-vscode, I prefer to just use the vscode terminal and launch julia from the terminal. This makes it easy to launch Julia with any command line options and also to start and restart Julia quickly.</p>
<h3 id="other-useful-vscode-extensions">Other useful VSCode extensions</h3>
<ul>
<li>Project Manager</li>
<li>Better TOML</li>
<li>Code Spell Checker</li>
</ul>
<p>VScode supports git out-of-the box.</p>UweIntroduction Installing Julia is easy, but perhaps you also want to install an integrated development environment (IDE) or a version control system (e.g. git), therefore I give some hints how to do that in this blog post.Working with Julia projects2022-08-16T00:00:00+00:002022-08-16T00:00:00+00:00https://ufechner7.github.io/2022/08/16/julia-projects<h2 id="introduction">Introduction</h2>
<p>When you start to use <a href="https://julialang.org/">Julia</a> you might ask yourself: How shall I structure my code?
There are different approaches for different use cases.</p>
<h2 id="simple-scripts">Simple scripts</h2>
<p>If you just write short, simple scripts that are not using any packages you can just keep them in one file and put them in any folder, no special folder structure is needed. If you care about performance you should put everything in a function. Example:</p>
<div class="language-julia highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># constants</span>
<span class="kd">const</span> <span class="n">LANG</span><span class="o">=</span><span class="s">"DE"</span>
<span class="c"># functions</span>
<span class="k">function</span><span class="nf"> hello</span><span class="x">(</span><span class="n">name</span><span class="x">)</span>
<span class="k">if</span> <span class="n">LANG</span> <span class="o">==</span> <span class="s">"DE"</span>
<span class="n">println</span><span class="x">(</span><span class="s">"Hallo liebe/r </span><span class="si">$</span><span class="s">name !"</span><span class="x">)</span>
<span class="k">else</span>
<span class="n">println</span><span class="x">(</span><span class="s">"Hello dear </span><span class="si">$</span><span class="s">name !"</span><span class="x">)</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="c"># main program</span>
<span class="k">function</span><span class="nf"> main</span><span class="x">()</span>
<span class="n">hello</span><span class="x">(</span><span class="s">"Peter"</span><span class="x">)</span>
<span class="n">hello</span><span class="x">(</span><span class="s">"Jane"</span><span class="x">)</span>
<span class="k">end</span>
<span class="n">main</span><span class="x">()</span>
<span class="nb">nothing</span>
</code></pre></div></div>
<p>The structure is:</p>
<ul>
<li>constants</li>
<li>functions</li>
<li>main function</li>
<li>call the main function</li>
<li>return nothing</li>
</ul>
<p>Do not use global variables! That kills your performance [1].</p>
<p>If you store this code in a file with the name <code class="language-plaintext highlighter-rouge">hello.jl</code> you can execute it from the REPL with the command:</p>
<div class="language-julia highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">include</span><span class="x">(</span><span class="s">"hello.jl"</span><span class="x">)</span>
</code></pre></div></div>
<p>The advantage of having a main() function is that you can include some error checks and return an error code if they fail. Furthermore, you can measure the performance by running:</p>
<div class="language-julia highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">@time</span> <span class="n">include</span><span class="x">(</span><span class="s">"hello.jl"</span><span class="x">)</span>
<span class="nd">@time</span> <span class="n">main</span><span class="x">()</span>
</code></pre></div></div>
<p>The first timing you get includes the compilation time, and the second number shows the pure execution time.</p>
<p>It is a good habit to return <code class="language-plaintext highlighter-rouge">nothing</code>, unless you want to return your result, e.g. a plot (diagram) or a dataset.</p>
<h2 id="scripts-that-use-packages">Scripts that use packages</h2>
<p>If you are using any packages, you should create a proper project to keep track of your dependencies and their versions.</p>
<p>That is simple:
Creating a project does not mean to create a package. It is much simpler:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">mkdir </span>my_project
<span class="nb">cd </span>my_project
<span class="nb">mkdir </span>src
julia <span class="nt">--project</span><span class="o">=</span><span class="s2">"."</span>
</code></pre></div></div>
<p>Now add the packages you need:</p>
<div class="language-julia highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">using</span> <span class="n">Pkg</span>
<span class="n">pkg</span><span class="s">"add Plots"</span>
<span class="n">pkg</span><span class="s">"add DataFrames"</span>
</code></pre></div></div>
<p>Now put your code in a file in the src folder, for example like this:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">cd </span>src
gedit my_plot.jl
</code></pre></div></div>
<p>and put the following code into it:</p>
<div class="language-julia highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">using</span> <span class="n">Plots</span><span class="x">,</span> <span class="n">DataFrames</span>
<span class="k">function</span><span class="nf"> main</span><span class="x">()</span>
<span class="n">time</span> <span class="o">=</span> <span class="mi">0</span><span class="o">:</span><span class="mf">0.01</span><span class="o">:</span><span class="mi">10</span> <span class="c"># step range from 0 to 10 step 0.1</span>
<span class="n">u</span> <span class="o">=</span> <span class="n">sin</span><span class="o">.</span><span class="x">(</span><span class="n">time</span><span class="o">*</span><span class="mi">5</span><span class="x">)</span> <span class="c"># signal with a frequency of 5 rad/s</span>
<span class="n">step</span> <span class="o">=</span> <span class="mi">1</span> <span class="o">.-</span> <span class="mi">1</span> <span class="o">./</span> <span class="n">exp</span><span class="o">.</span><span class="x">(</span><span class="n">time</span><span class="x">)</span> <span class="c"># step response</span>
<span class="n">df</span> <span class="o">=</span> <span class="n">DataFrame</span><span class="x">(;</span><span class="n">time</span><span class="x">,</span> <span class="n">u</span><span class="x">,</span> <span class="n">step</span><span class="x">)</span>
<span class="n">plt</span> <span class="o">=</span> <span class="n">plot</span><span class="x">(</span><span class="n">df</span><span class="o">.</span><span class="n">time</span><span class="x">,</span> <span class="n">u</span><span class="x">,</span> <span class="n">legend</span><span class="o">=</span><span class="nb">false</span><span class="x">)</span>
<span class="n">plot!</span><span class="x">(</span><span class="n">df</span><span class="o">.</span><span class="n">time</span><span class="x">,</span> <span class="n">step</span><span class="x">)</span>
<span class="n">plt</span>
<span class="k">end</span>
<span class="n">plt</span> <span class="o">=</span> <span class="n">main</span><span class="x">()</span>
</code></pre></div></div>
<p>and save it.
If you want to run it, make sure you are in the <code class="language-plaintext highlighter-rouge">my_project</code> folder and then
start Julia with:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>julia <span class="nt">--project</span>
</code></pre></div></div>
<p>and execute your script with:</p>
<div class="language-julia highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">include</span><span class="x">(</span><span class="s">"src/my_plot.jl"</span><span class="x">)</span>
</code></pre></div></div>
<p>You should see the following plot:</p>
<p><img src="https://raw.githubusercontent.com/ufechner7/ufechner7.github.io/main/_posts/myplot.png" alt="myplot" /></p>
<p>When you are happy with your code and the packages you are using, make a backup copy
of your Manifest.toml file:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">cp </span>Manifest.toml Manifest.toml.bak
</code></pre></div></div>
<p>If you - half a year later - update your packages and your code stops working, just restore the Manifest file:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">cp </span>Manifest.toml.bak Manifest.toml
</code></pre></div></div>
<p>No need to create any module or Julia package…</p>
<h2 id="using-compat-to-improve-long-term-robustness">Using compat to improve long-term robustness</h2>
<p>If you add compat bounds to the versions of the packages you are using your project becomes more robust. This means,
if you are adding new packages in the future the currently used packages will not be unintentionally upgraded.</p>
<p>If you are using Julia 1.8 or newer you can use the following approach:</p>
<ol>
<li>launch Julia with <code class="language-plaintext highlighter-rouge">julia --project</code></li>
<li>enter the package manager mode by pressing the key <code class="language-plaintext highlighter-rouge">]</code></li>
<li>list the status of your project:
<div class="language-julia highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="x">(</span><span class="n">my_project</span><span class="x">)</span> <span class="n">pkg</span><span class="o">></span> <span class="n">st</span>
<span class="n">Status</span> <span class="sb">`~/repos/my_project/Project.toml`</span>
<span class="x">[</span><span class="n">a93c6f00</span><span class="x">]</span> <span class="n">DataFrames</span> <span class="n">v1</span><span class="o">.</span><span class="mf">3.4</span>
<span class="x">[</span><span class="mi">91</span><span class="n">a5bcdd</span><span class="x">]</span> <span class="n">Plots</span> <span class="n">v1</span><span class="o">.</span><span class="mf">31.7</span>
</code></pre></div> </div>
<p>If you are careful allow only bugfixes for the installed packages which means only the last number of the version string is allowed to be increased. To achieve that, type</p>
<div class="language-julia highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="x">(</span><span class="n">my_project</span><span class="x">)</span> <span class="n">pkg</span><span class="o">></span> <span class="n">compat</span>
<span class="n">Compat</span> <span class="sb">`~/repos/my_project/Project.toml`</span>
<span class="n">Select</span> <span class="n">an</span> <span class="n">entry</span> <span class="n">to</span> <span class="n">edit</span><span class="o">:</span>
<span class="o">></span> <span class="n">julia</span> <span class="n">none</span>
<span class="x">[</span><span class="n">a93c6f00</span><span class="x">]</span> <span class="n">DataFrames</span> <span class="n">none</span>
<span class="x">[</span><span class="mi">91</span><span class="n">a5bcdd</span><span class="x">]</span> <span class="n">Plots</span> <span class="n">none</span>
</code></pre></div> </div>
<p>Now select Julia and enter ~1.8, DataFrames and enter ~1.3 and Plots and enter ~1.31. Use the first two numbers of the version strings of the currently installed packages as shown by the command <code class="language-plaintext highlighter-rouge">st</code>.
If you now type <code class="language-plaintext highlighter-rouge">compat</code> again it should look like this:</p>
<div class="language-julia highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="x">(</span><span class="n">my_project</span><span class="x">)</span> <span class="n">pkg</span><span class="o">></span> <span class="n">compat</span>
<span class="n">Compat</span> <span class="sb">`~/repos/my_project/Project.toml`</span>
<span class="n">Select</span> <span class="n">an</span> <span class="n">entry</span> <span class="n">to</span> <span class="n">edit</span><span class="o">:</span>
<span class="o">></span> <span class="n">julia</span> <span class="o">~</span><span class="mf">1.8</span>
<span class="x">[</span><span class="n">a93c6f00</span><span class="x">]</span> <span class="n">DataFrames</span> <span class="o">~</span><span class="mf">1.3</span>
<span class="x">[</span><span class="mi">91</span><span class="n">a5bcdd</span><span class="x">]</span> <span class="n">Plots</span> <span class="o">~</span><span class="mf">1.31</span>
</code></pre></div> </div>
<p>Press <code class="language-plaintext highlighter-rouge">q</code> to quit and then <code class="language-plaintext highlighter-rouge">backspace</code> to quit the package manager mode.</p>
</li>
</ol>
<p><em>Congratulations!</em></p>
<p>You have now set limits for your package versions, and if you type <code class="language-plaintext highlighter-rouge">up</code> in the package manager the next time you will only get bugfixes for the installed packages, but nothing that could break your current code.</p>
<p>Your Project.toml file should now look like this:</p>
<div class="language-julia highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="n">ufechner</span><span class="nd">@desktop</span><span class="o">:~/</span><span class="n">repos</span><span class="o">/</span><span class="n">my_project</span><span class="o">$</span> <span class="n">cat</span> <span class="n">Project</span><span class="o">.</span><span class="n">toml</span>
<span class="x">[</span><span class="n">deps</span><span class="x">]</span>
<span class="n">DataFrames</span> <span class="o">=</span> <span class="s">"a93c6f00-e57d-5684-b7b6-d8193f3e46c0"</span>
<span class="n">Plots</span> <span class="o">=</span> <span class="s">"91a5bcdd-55d7-5caf-9e0b-520d859cae80"</span>
<span class="x">[</span><span class="n">compat</span><span class="x">]</span>
<span class="n">DataFrames</span> <span class="o">=</span> <span class="s">"~1.3"</span>
<span class="n">Plots</span> <span class="o">=</span> <span class="s">"~1.31"</span>
<span class="n">julia</span> <span class="o">=</span> <span class="s">"~1.8"</span>
</code></pre></div></div>
<p>If you have a Julia version older than 1.8 you can also just edit the file Project.toml manually with your preferred editor to add the compat section. If your code was tested with multiple packages or Julia versions, you can create a list, for example:</p>
<div class="language-julia highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="x">[</span><span class="n">compat</span><span class="x">]</span>
<span class="n">julia</span> <span class="o">=</span> <span class="s">"~1.6,~1.7,~1.8"</span>
</code></pre></div></div>
<h2 id="further-reading">Further reading</h2>
<p>If you want to understand the meaning of the <code class="language-plaintext highlighter-rouge">semantic versioning</code> that is used by Julia packages, please read the section <a href="https://pkgdocs.julialang.org/v1/compatibility/">Compatibility</a> in the documentation of the package manager.</p>
<h2 id="outlook">Outlook</h2>
<p>If you want to create reusable packages that you want to use in multiple programs/ projects consider creating real Julia packages. This is a little bit more complicated, but it has the advantage of automated unit tests and easy installation for yourself and others. I will talk about that in one of my future blog posts.</p>
<hr />
<p>[1] You can assign a value with a different type to global variables. As a result, Julia has to generate more generic code that is valid for potentially all types which is bad for the performance. From Julia 1.8 onwards you can annotate global variables with a concrete type, e.g. <code class="language-plaintext highlighter-rouge">NUMBER::Int64</code>. In this case, the performance is better but still not as good as it is for local variables.</p>UweIntroduction When you start to use Julia you might ask yourself: How shall I structure my code? There are different approaches for different use cases.Why am I using Julia?2022-08-13T00:00:00+00:002022-08-13T00:00:00+00:00https://ufechner7.github.io/2022/08/13/why-julia<h2 id="introduction">Introduction</h2>
<p>I started programming in 1976 using the KIM 1 and the 6502 machine language. Since then I learned and used many languages, Assembler, Pascal, Delphi, Basic, Foxpro, Java, C, C++, Bash, Matlab, Python and more…</p>
<table>
<thead>
<tr>
<th style="text-align: center"><img src="https://raw.githubusercontent.com/ufechner7/ufechner7.github.io/main/_posts/440px-MOS_KIM-1_IMG_4208.jpg" alt="KIM 1" /></th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: center"><em>KIM 1</em></td>
</tr>
</tbody>
</table>
<p>Why do I think that Julia is superior to all of them in many (not all) applications?</p>
<h2 id="what-is-so-great-about-julia">What is so great about <a href="https://julialang.org/">Julia</a>?</h2>
<h3 id="reason-no-one-speed">Reason no. one: speed</h3>
<p>When you need to process large amounts of data speed is important. Julia is a just-in-time compiled language using the mature LLVM compiler infrastructure, so you get fast machine code for different processors. In contrast to Python also multi-threading is well supported. I wrote a simulation package in Python+Numba, and re-wrote it in Julia.</p>
<table>
<thead>
<tr>
<th style="text-align: center"><img src="https://raw.githubusercontent.com/ufechner7/ufechner7.github.io/main/_posts/Julia-vs-Python.png" alt="Performance advantage of Julia vs Python+Numba" /></th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: center">Performance advantage of Julia vs Python+Numba</td>
</tr>
</tbody>
</table>
<p>Depending on the test case Julia was 7 to 70 times faster than Python+Numba. And writing the Julia code was much easier because mixing Python with Numba is a real pain because they look similar, but behave very differently.</p>
<p>If you only use Python with algorithms for which C++ or Fortran libraries are available, then you will not see much of an advantage from using Julia. But if you write new programs or libraries using new algorithms Julia is much faster (if you follow the <a href="https://docs.julialang.org/en/v1/manual/performance-tips/">Performance Tips</a>).</p>
<h3 id="reason-no-two-simplicity-and-elegance">Reason no. two: simplicity and elegance</h3>
<p>You can operate with vectors as you would do it in mathematical notation, you can use <code class="language-plaintext highlighter-rouge">×</code> for the cross product and <code class="language-plaintext highlighter-rouge">⋅</code> for the dot product. And functions can return tuples (multiple values), which is very useful.</p>
<div class="language-julia highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># pos: vector of tether particle positions</span>
<span class="c"># v_apparent: apparent wind speed vector</span>
<span class="k">function</span><span class="nf"> kite_ref_frame</span><span class="x">(</span><span class="n">pos</span><span class="x">,</span> <span class="n">v_apparent</span><span class="x">)</span>
<span class="n">c</span> <span class="o">=</span> <span class="n">pos</span><span class="x">[</span><span class="k">end</span><span class="o">-</span><span class="mi">1</span><span class="x">]</span> <span class="o">-</span> <span class="n">pos</span><span class="x">[</span><span class="k">end</span><span class="x">]</span>
<span class="n">z</span> <span class="o">=</span> <span class="n">normalize</span><span class="x">(</span><span class="n">c</span><span class="x">)</span>
<span class="n">y</span> <span class="o">=</span> <span class="n">normalize</span><span class="x">(</span><span class="n">v_apparent</span> <span class="n">×</span> <span class="n">c</span><span class="x">)</span>
<span class="n">x</span> <span class="o">=</span> <span class="n">normalize</span><span class="x">(</span><span class="n">y</span> <span class="n">×</span> <span class="n">c</span><span class="x">)</span>
<span class="n">x</span><span class="x">,</span> <span class="n">y</span><span class="x">,</span> <span class="n">z</span>
<span class="k">end</span>
</code></pre></div></div>
<p>And you can use Greek letters in the code:</p>
<div class="language-julia highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">using</span> <span class="n">Distributions</span><span class="x">,</span> <span class="n">Random</span>
<span class="n">T</span> <span class="o">=</span> <span class="x">[</span><span class="mf">84.5</span><span class="x">,</span> <span class="mf">83.6</span><span class="x">,</span> <span class="mf">83.6</span><span class="x">]</span> <span class="c"># temperature measurements</span>
<span class="n">μ</span> <span class="o">=</span> <span class="n">mean</span><span class="x">(</span><span class="n">T</span><span class="x">)</span> <span class="c"># average cpu temperature</span>
<span class="n">σ</span> <span class="o">=</span> <span class="mf">2.5</span> <span class="c"># 95.45 % of the measurements are within ± 5 deg</span>
<span class="n">d</span> <span class="o">=</span> <span class="n">Normal</span><span class="x">(</span><span class="n">μ</span><span class="x">,</span> <span class="n">σ</span><span class="x">)</span>
<span class="n">good</span> <span class="o">=</span> <span class="n">cdf</span><span class="x">(</span><span class="n">d</span><span class="x">,</span> <span class="mf">88.0</span><span class="x">)</span> <span class="c"># this percentage of the cpus will not start to throttle</span>
</code></pre></div></div>
<p>And you can write arrays cleanly:</p>
<div class="language-julia highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">julia</span><span class="o">></span> <span class="n">A</span> <span class="o">=</span> <span class="x">[</span><span class="mi">1</span> <span class="mi">2</span><span class="nb">π</span>
<span class="mi">3</span> <span class="mi">4</span><span class="nb">π</span><span class="x">]</span>
<span class="mi">2</span><span class="n">×2</span> <span class="kt">Matrix</span><span class="x">{</span><span class="kt">Float64</span><span class="x">}</span><span class="o">:</span>
<span class="mf">1.0</span> <span class="mf">6.28319</span>
<span class="mf">3.0</span> <span class="mf">12.5664</span>
</code></pre></div></div>
<p>And any scalar function incl. user-defined functions can be applied on a vector using the dot syntax:</p>
<div class="language-julia highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">julia</span><span class="o">></span> <span class="n">sin</span><span class="o">.</span><span class="x">([</span><span class="mi">1</span><span class="x">,</span><span class="mi">2</span><span class="x">,</span><span class="mi">3</span><span class="x">])</span>
<span class="mi">3</span><span class="o">-</span><span class="n">element</span> <span class="kt">Vector</span><span class="x">{</span><span class="kt">Float64</span><span class="x">}</span><span class="o">:</span>
<span class="mf">0.8414709848078965</span>
<span class="mf">0.9092974268256817</span>
<span class="mf">0.1411200080598672</span>
</code></pre></div></div>
<h3 id="reason-no-three-great-repl-read-evaluate-and-print-loop">Reason no. three: Great REPL (Read, Evaluate and Print Loop)</h3>
<p>Normally, compiled languages don’t have any REPL, but if you ever used it you do not want to miss it again:</p>
<table>
<thead>
<tr>
<th style="text-align: center"><img src="https://raw.githubusercontent.com/ufechner7/ufechner7.github.io/main/_posts/julia.png" alt="Julia REPL" /></th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: center"><em>Julia REPL</em></td>
</tr>
</tbody>
</table>
<p>You can do basic calculations, but also try out any function you want to use. You get help by typing <code class="language-plaintext highlighter-rouge">?</code> and press <Enter>, or help for a specific function by typing for example:</p>
<div class="language-julia highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">?</span><span class="n">sin</span>
</code></pre></div></div>
<p>You can run your scripts using the include function, for example:</p>
<div class="language-julia highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">include</span><span class="x">(</span><span class="s">"hello_world.jl"</span><span class="x">)</span>
</code></pre></div></div>
<p>It has four modes of operation, the julia mode for executing julia code, the help modus, the shell modus (which you reach by pressing <code class="language-plaintext highlighter-rouge">;</code>) and the package manager mode explained in the next section. I always develop code in the REPL and when a line of code (or a function) works I copy it into the script. If you wonder how to get the character <code class="language-plaintext highlighter-rouge">π</code>: just type <code class="language-plaintext highlighter-rouge">\pi</code> and then the TAB key.</p>
<h3 id="reason-no-four-built-in-package-manager">Reason no. four: Built-in package manager</h3>
<p>In Python it can be very difficult to install packages. There is <code class="language-plaintext highlighter-rouge">pip</code> that might work, but not good for binary dependencies. There is <code class="language-plaintext highlighter-rouge">easy_install</code>, there is <code class="language-plaintext highlighter-rouge">conda</code>… Many package managers, but all of them only work sometimes, for some packages on some operating systems… On Julia is only one package manager and it works fine with all packages on all supported operating systems (Windows, Mac, Linux…) on ARM and X86.</p>
<p>You can reach the prompt of the package manager by typing <code class="language-plaintext highlighter-rouge">]</code>. It looks like this:</p>
<div class="language-julia highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="x">(</span><span class="nd">@v1.8</span><span class="x">)</span> <span class="n">pkg</span><span class="o">></span> <span class="n">st</span>
<span class="n">Status</span> <span class="sb">`~/.julia/environments/v1.8/Project.toml`</span>
<span class="x">[</span><span class="n">e1d33f9a</span><span class="x">]</span> <span class="n">DLMReader</span> <span class="n">v0</span><span class="o">.</span><span class="mf">4.5</span>
<span class="x">[</span><span class="mi">31</span><span class="n">c24e10</span><span class="x">]</span> <span class="n">Distributions</span> <span class="n">v0</span><span class="o">.</span><span class="mf">25.66</span>
<span class="x">[</span><span class="mi">5</span><span class="n">c01b14b</span><span class="x">]</span> <span class="n">InMemoryDatasets</span> <span class="n">v0</span><span class="o">.</span><span class="mf">7.7</span>
<span class="n">⌃</span> <span class="x">[</span><span class="mi">14</span><span class="n">b8a8f1</span><span class="x">]</span> <span class="n">PkgTemplates</span> <span class="n">v0</span><span class="o">.</span><span class="mf">7.26</span>
<span class="n">⌃</span> <span class="x">[</span><span class="mi">91</span><span class="n">a5bcdd</span><span class="x">]</span> <span class="n">Plots</span> <span class="n">v1</span><span class="o">.</span><span class="mf">29.0</span>
<span class="x">[</span><span class="mi">0</span><span class="n">c614874</span><span class="x">]</span> <span class="n">TerminalPager</span> <span class="n">v0</span><span class="o">.</span><span class="mf">3.1</span>
<span class="n">Info</span> <span class="n">Packages</span> <span class="n">marked</span> <span class="n">with</span> <span class="n">⌃</span> <span class="n">have</span> <span class="n">new</span> <span class="n">versions</span> <span class="n">available</span>
</code></pre></div></div>
<p>The prompt tells you which environment is active, and you can add, update and delete packages, but also display statistics or download the source code of a package for development. Each Julia package has a clear definition of the versions of the dependencies it works within a file called Project.toml.</p>
<p>Therefore it is easy to write code that will never break in the future. From my point of view thats a killer feature. If required, the package manager also installs pre-compiled C++ or Fortran libraries which works very well.</p>
<p>If you are looking for a specific package, <a href="https://juliahub.com/ui/Packages">JuliaHub</a> will help you to find it.</p>
<h3 id="reason-no-five-the-composability-of-packages">Reason no. five: the composability of packages</h3>
<p>Julia packages that are written independently often (not always) work together surprisingly well. Example: We have designed a control system, but now we want to do a Monte Carlo analysis to find out how robust it is in the presence of parameter uncertainties.</p>
<div class="language-julia highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">using</span> <span class="n">ControlSystems</span><span class="x">,</span> <span class="n">MonteCarloMeasurements</span><span class="x">,</span> <span class="n">Plots</span>
<span class="n">ω</span> <span class="o">=</span> <span class="mi">1</span> <span class="n">±</span> <span class="mf">0.1</span> <span class="c"># create an uncertain Gaussian parameter</span>
<span class="n">ζ</span> <span class="o">=</span> <span class="mf">0.3</span><span class="o">..</span><span class="mf">0.4</span> <span class="c"># create an uncertain uniform parameter</span>
<span class="n">G</span> <span class="o">=</span> <span class="n">tf</span><span class="x">(</span><span class="n">ω</span><span class="o">^</span><span class="mi">2</span><span class="x">,</span> <span class="x">[</span><span class="mi">1</span><span class="x">,</span> <span class="mi">2</span><span class="n">ζ</span><span class="o">*</span><span class="n">ω</span><span class="x">,</span> <span class="n">ω</span><span class="o">^</span><span class="mi">2</span><span class="x">])</span> <span class="c"># systems accept uncertain parameters</span>
</code></pre></div></div>
<p>Output:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">TransferFunction</span><span class="p">{</span><span class="n">Continuous</span><span class="p">,</span> <span class="n">ControlSystems</span><span class="p">.</span><span class="n">SisoRational</span><span class="p">{</span><span class="n">Particles</span><span class="p">{</span><span class="n">Float64</span><span class="p">,</span> <span class="mi">2000</span><span class="p">}}}</span>
<span class="mf">1.01</span> <span class="err">±</span> <span class="mf">0.2</span>
<span class="o">----------------------------------</span>
<span class="mf">1.0</span><span class="n">s</span><span class="o">^</span><span class="mi">2</span> <span class="o">+</span> <span class="mf">0.7</span> <span class="err">±</span> <span class="mf">0.092</span><span class="n">s</span> <span class="o">+</span> <span class="mf">1.01</span> <span class="err">±</span> <span class="mf">0.2</span>
<span class="n">Continuous</span><span class="o">-</span><span class="n">time</span> <span class="n">transfer</span> <span class="n">function</span> <span class="n">model</span>
</code></pre></div></div>
<p>The function <code class="language-plaintext highlighter-rouge">tf</code> which was originally written for real parameters only, when supplied with uncertain parameters it calculates and prints the correct result. Magic! 😄</p>
<p>Now lets plot the result for the frequency range 0.01 to 100 rad/s on a logarithmic scale:</p>
<div class="language-julia highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">w</span> <span class="o">=</span> <span class="n">exp10</span><span class="o">.</span><span class="x">(</span><span class="o">-</span><span class="mi">2</span><span class="o">:</span><span class="mf">0.02</span><span class="o">:</span><span class="mi">2</span><span class="x">)</span>
<span class="n">bodeplot</span><span class="x">(</span><span class="n">G</span><span class="x">,</span> <span class="n">w</span><span class="x">)</span>
</code></pre></div></div>
<p><img src="https://raw.githubusercontent.com/ufechner7/ufechner7.github.io/main/_posts/bodeplot.png" alt="Bodeplot" /></p>
<p>Plotting of diagrams with uncertainty works, even though the packages Plot.jl and MonteCarloMeasurements.jl were developed independently by different authors. You can also combine calculations with <a href="https://github.com/PainterQubits/Unitful.jl">physical units</a> and get plots that show the correct physical units. Just magic! 😄</p>
<p>This is so much better than Python or C++ where every library comes with its own implementation of types like vectors that are not compatible with each other. In Julia, you write a generic library, and then it can work with <a href="https://juliaarrays.github.io/StaticArrays.jl/stable/">StaticArrays.jl</a> (stack allocated) or normal <a href="https://docs.julialang.org/en/v1/manual/arrays/">arrays</a> (heap allocated) or <a href="https://engaging-web.mit.edu/~alir/cnhlab004/julia-1.5.3/share/doc/julia/html/en/stdlib/SparseArrays.html">sparse arrays</a> or whatever and it just works.</p>
<h3 id="reason-no-six-an-awesome-community">Reason no. six: an awesome community</h3>
<p>If you ask a question on <a href="https://discourse.julialang.org/">discourse</a> you usually get an answer within 15 minutes. And half of the people that reply are scientists, they know what they are talking about. So you get very qualified answers quickly, often some extra information that might help you to improve your approach.</p>
<h3 id="state-of-the-art-solvers-for-differential-equations">State-of-the-art solvers for differential equations</h3>
<p>If you are creating models of physical, chemical or biological systems, you will often need to solve differential equations. The package <a href="https://diffeq.sciml.ai/stable/">DifferentialEquations.jl</a> provides the world’s best set of solvers for these kind of problems.</p>
<h3 id="state-of-the-art-optimizers">State-of-the-art optimizers</h3>
<p>If you want to optimize a system for example minimize the costs or the time or other properties of a system you need a mathematical optimizer. <a href="https://jump.dev/JuMP.jl/stable/">JuMP</a> provides a domain-specific modeling language for mathematical optimization embedded in Julia. This is easy to use, fast and powerful. For example here is a solution of the Rosenbrock function:</p>
<div class="language-julia highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">using</span> <span class="n">JuMP</span>
<span class="k">import</span> <span class="n">Ipopt</span>
<span class="n">model</span> <span class="o">=</span> <span class="n">Model</span><span class="x">(</span><span class="n">Ipopt</span><span class="o">.</span><span class="n">Optimizer</span><span class="x">)</span>
<span class="nd">@variable</span><span class="x">(</span><span class="n">model</span><span class="x">,</span> <span class="n">x</span><span class="x">)</span>
<span class="nd">@variable</span><span class="x">(</span><span class="n">model</span><span class="x">,</span> <span class="n">y</span><span class="x">)</span>
<span class="nd">@NLobjective</span><span class="x">(</span><span class="n">model</span><span class="x">,</span> <span class="n">Min</span><span class="x">,</span> <span class="x">(</span><span class="mi">1</span> <span class="o">-</span> <span class="n">x</span><span class="x">)</span><span class="o">^</span><span class="mi">2</span> <span class="o">+</span> <span class="mi">100</span> <span class="o">*</span> <span class="x">(</span><span class="n">y</span> <span class="o">-</span> <span class="n">x</span><span class="o">^</span><span class="mi">2</span><span class="x">)</span><span class="o">^</span><span class="mi">2</span><span class="x">)</span>
<span class="n">optimize!</span><span class="x">(</span><span class="n">model</span><span class="x">)</span>
<span class="n">println</span><span class="x">(</span><span class="s">"x: "</span><span class="x">,</span> <span class="n">value</span><span class="x">(</span><span class="n">x</span><span class="x">))</span>
<span class="n">println</span><span class="x">(</span><span class="s">"y: "</span><span class="x">,</span> <span class="n">value</span><span class="x">(</span><span class="n">y</span><span class="x">))</span>
</code></pre></div></div>
<h3 id="the-scientific-machine-learning-ecosystem">The Scientific Machine Learning ecosystem</h3>
<p>The <a href="https://docs.sciml.ai/dev/">SciML</a> ecosystem provides a unified interface to libraries for scientific computing and machine learning. So far I mainly used <a href="https://docs.sciml.ai/dev/modules/NonlinearSolve/">NonlinearSolve</a> and <a href="https://docs.sciml.ai/dev/modules/Optimization/">Optimization.jl</a> for root-finding and for optimization problems where JuMP is not the best choice. The unified interfaces make it easy to try different solvers to find the best one for your problem.</p>
<p>If you are modeling real-world problems like the Corona disease then combining differential equations and machine learning can give superior results.</p>
<h2 id="limitations">Limitations</h2>
<h3 id="limitations-of-language-compiler-and-tooling">Limitations of language/ compiler and tooling</h3>
<h4 id="compilation-and-load-times">Compilation and load times</h4>
<p>Loading packages takes some time, and also the just-in-time compilation kicks in when you make your first function call. This is known as the time-to-first-plot problem (TTFP). Simple example:</p>
<div class="language-julia highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">using</span> <span class="n">Plots</span>
<span class="n">p</span> <span class="o">=</span> <span class="n">plot</span><span class="x">(</span><span class="n">rand</span><span class="x">(</span><span class="mi">2</span><span class="x">,</span><span class="mi">2</span><span class="x">))</span>
<span class="n">display</span><span class="x">(</span><span class="n">p</span><span class="x">)</span>
</code></pre></div></div>
<p>If we now save this code in the file <code class="language-plaintext highlighter-rouge">ttfp.jl</code> run the program with the command:</p>
<div class="language-julia highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">@time</span> <span class="n">include</span><span class="x">(</span><span class="s">"ttfp.jl"</span><span class="x">)</span>
</code></pre></div></div>
<p>we will see that it takes about 9 seconds on a computer with an i7-7700K CPU. This is quite a lot of time for such a small program. There a different approaches to mitigate this issue. What I usually do is create a system image with all packages I use for a curtain project and do an ahead-of-time compilation of these packages. I will explain in a separate blog post how to do that. If you are curious already, have a look at <a href="https://julialang.github.io/PackageCompiler.jl/stable/">PackageCompiler.jl</a>.</p>
<p>There is an ongoing effort to use binary caches of the compiled code so that you need to recompile only when the source code has changed, but it will take some time until this will work without any user interaction.</p>
<p><strong>UPDATE 2024:</strong> With Julia 1.10.0 this time is down to about 2s. So PackageCompiler is now only needed if you use very large packages like <a href="https://github.com/SciML/ModelingToolkit.jl">ModelingToolkit.jl</a>.</p>
<h4 id="for-use-on-embedded-systems">For use on embedded systems</h4>
<p>Currently, with Julia 1.8 the first issue is the amount of RAM that is needed: Using Julia
with less than 4GB RAM is not very enjoyable. A cross-compiler is missing. For these
reasons writing code for smaller embedded systems is not feasible. The smallest system
I would use with Julia is a Raspberry Pi 4 and a 64-bit OS.</p>
<p>I know that the Julia developers work hard to mitigate these issues, so in a few years, Julia will hopefully also be usable for smaller embedded systems.</p>
<h4 id="working-with-modules">Working with modules</h4>
<p>While working with <a href="https://docs.julialang.org/en/v1/manual/modules/">modules</a> is supported by the language, using modules that are NOT packages is not well supported by the tooling. The VSCode Julia plugin will NOT find symbols of modules in the LOAD_PATH, see <a href="https://github.com/julia-vscode/julia-vscode/issues/307">this</a> bug report. Workarounds exist, but I find them ugly. As a consequence I use only one module per app/ package and split the program using includes. This is not so good because a file is no longer a stand-alone unit of code that you can read on its own and requires some adaptation of your habits if you are an experienced C++ or Python programmer.</p>
<h3 id="limitations-of-the-ecosystem">Limitations of the ecosystem</h3>
<ul>
<li>
<p>The first limitation I see is that there is no easy-to-use, well-integrated GUI library. Yes, you can use <a href="https://github.com/barche/QML.jl">QML.jl</a> or <a href="https://github.com/JuliaGraphics/Gtk.jl">GTK.jl</a> and many others, but they all have their issues, one of the issues is the lack of Julia specific documentation.</p>
</li>
<li>
<p>My preferred solver for stiff differential algebraic equations is the <a href="https://jmodelica.org/assimulo/DAE_Radau5DAE.html">Radau5DAE</a> solver, which is available in Fortran and Python, but not yet wrapped or re-written in Julia. There is an open <a href="https://github.com/SciML/OrdinaryDiffEq.jl/issues/1406">issue</a> for this.</p>
</li>
</ul>
<p>Luckily you can use most Python and R libraries from within Julia, so the lack of a specific library is usually no show-stopper. Have a look at <a href="https://github.com/cjdoris/PythonCall.jl">PythonCall.jl</a> and <a href="https://juliainterop.github.io/RCall.jl/stable/">RCall.jl</a>. Also Fortran and C code can be called directly from Julia, for C++ you need an interface library like <a href="https://github.com/JuliaInterop/CxxWrap.jl">CxxWrap</a>.</p>
<h2 id="conclusions">Conclusions</h2>
<p>If you want to analyze data, model physical systems, design control systems or create a web portal with dynamic data Julia is a good choice for you. Also if you do number crunching on graphic cards or clusters. So far it is not a good choice for mobile apps or small embedded systems. It can also be used for machine learning, but I am not qualified to say how well that works (I guess <em>it depends…</em>)</p>
<p>So give Julia a try if you are now using Python, Matlab, Fortran or C++ for one of the mentioned tasks. Your life will become happier…</p>
<h2 id="acknowledgments">Acknowledgments</h2>
<p>Thanks to Fredrik Bagge Carlson for providing the <a href="https://juliacontrol.github.io/RobustAndOptimalControl.jl/dev/uncertainty/">example for composability</a> .</p>UweIntroductionFiltering and browsing datasets2022-08-12T00:00:00+00:002022-08-12T00:00:00+00:00https://ufechner7.github.io/2022/08/12/filtering-and-browsing-datasets<h2 id="introduction">Introduction</h2>
<p>The basic work flow of data analysis is filtering a large data set to extract the interesting aspects and to browse and/or plot the result. In this post I will give a basic example for filtering and browsing of data, plotting deserves a blog post on its own.</p>
<h2 id="creating-a-test-project">Creating a test project</h2>
<p>For trying out a new package and/or example it is always good to create a new project first.
<em>When you followed my first post you only have to add the package TerminalPager</em>.</p>
<p>For example using the following commands:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">mkdir </span>can
<span class="nb">cd </span>can
julia <span class="nt">--project</span><span class="o">=</span><span class="s2">"."</span>
</code></pre></div></div>
<p>And then install the required packages:</p>
<div class="language-julia highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">julia</span><span class="o">></span> <span class="k">using</span> <span class="n">pkg</span>
<span class="n">julia</span><span class="o">></span> <span class="n">pkg</span><span class="s">"add InMemoryDatasets"</span>
<span class="n">julia</span><span class="o">></span> <span class="n">pkg</span><span class="s">"add TerminalPager"</span>
</code></pre></div></div>
<p>Now quit julia with <CTRL><D> and restart it with:</D></CTRL></p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>julia <span class="nt">--project</span> <span class="nt">-t</span> auto
</code></pre></div></div>
<p>This uses the set of packages we just installed and starts julia using all available threads. This is useful when handling large data sets (millions of messages).</p>
<h2 id="creating-a-sample-data-set">Creating a sample data set</h2>
<p>To get a realistic impression of the performance we now create a sample data set with 100k entries. We use again a <a href="https://en.wikipedia.org/wiki/CAN_bus">CAN bus</a> log file as example, this time with <code class="language-plaintext highlighter-rouge">m=8</code> data columns, one byte per column. For the address column, which uses two bytes we use a range of 46 different, randomly choosen values, starting with 0x101.</p>
<div class="language-julia highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">using</span> <span class="n">InMemoryDatasets</span><span class="x">,</span> <span class="n">DLMReader</span><span class="x">,</span> <span class="n">Printf</span>
<span class="k">function</span><span class="nf"> demo_data</span><span class="x">()</span>
<span class="n">n</span> <span class="o">=</span> <span class="mi">100000</span>
<span class="n">m</span> <span class="o">=</span> <span class="mi">8</span>
<span class="n">time</span> <span class="o">=</span> <span class="mf">0.1</span><span class="o">:</span><span class="mf">0.1</span><span class="o">:</span><span class="n">n</span><span class="o">*</span><span class="mf">0.1</span>
<span class="n">addr</span> <span class="o">=</span> <span class="n">rand</span><span class="x">(</span><span class="kt">Int16</span><span class="x">(</span><span class="mh">0x101</span><span class="x">)</span><span class="o">:</span><span class="kt">Int16</span><span class="x">(</span><span class="mh">0x12e</span><span class="x">),</span> <span class="n">n</span><span class="x">)</span>
<span class="n">data</span> <span class="o">=</span> <span class="x">(</span><span class="kt">Symbol</span><span class="x">(</span><span class="s">"d"</span><span class="x">,</span> <span class="n">i</span><span class="x">)</span> <span class="o">=></span> <span class="n">rand</span><span class="x">(</span><span class="kt">UInt8</span><span class="x">,</span> <span class="n">n</span><span class="x">)</span> <span class="k">for</span> <span class="n">i</span> <span class="k">in</span> <span class="mi">1</span><span class="o">:</span><span class="n">m</span><span class="x">)</span>
<span class="n">ds</span> <span class="o">=</span> <span class="n">Dataset</span><span class="x">(;</span><span class="n">time</span><span class="x">,</span> <span class="n">addr</span><span class="x">,</span> <span class="n">data</span><span class="o">...</span><span class="x">)</span>
<span class="n">ds</span><span class="o">.</span><span class="n">addr</span><span class="x">[</span><span class="mi">5</span><span class="x">]</span> <span class="o">=</span> <span class="nb">missing</span>
<span class="n">ds</span>
<span class="k">end</span>
<span class="n">ds</span> <span class="o">=</span> <span class="n">demo_data</span><span class="x">()</span>
</code></pre></div></div>
<p>If we now print the data set in the julia console, only the first and the last values are shown:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>julia> ds = demo_data()
100000×10 Dataset
Row │ time addr d1 d2 d3 d4 d5 d6 d7 d8
│ identity identity identity identity identity identity identity identity identity identity
│ Float64? Int16? UInt8? UInt8? UInt8? UInt8? UInt8? UInt8? UInt8? UInt8?
────────┼────────────────────────────────────────────────────────────────────────────────────────────────────
1 │ 0.1 264 127 5 247 185 212 216 171 147
2 │ 0.2 278 127 239 183 48 127 98 27 44
3 │ 0.3 301 6 16 159 201 225 95 196 51
⋮ │ ⋮ ⋮ ⋮ ⋮ ⋮ ⋮ ⋮ ⋮ ⋮ ⋮
99999 │ 9999.9 294 95 185 15 205 178 64 13 31
100000 │ 10000.0 285 86 224 79 214 248 39 34 147
99995 rows omitted
</code></pre></div></div>
<h2 id="formating-the-data">Formating the data</h2>
<p>For CAN bus messages usually the hexadecimal data format is used. To see the numbers in hex format we need to define two functions, one that is formating the data bytes with two hex digits, called <code class="language-plaintext highlighter-rouge">hex2(n)</code> and one that formats the address column with four hex digits, called <code class="language-plaintext highlighter-rouge">hex4(n)</code>. To save space we want to display missing values with the string “–”.</p>
<div class="language-julia highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">function</span><span class="nf"> hex2</span><span class="x">(</span><span class="n">n</span><span class="x">)</span>
<span class="k">if</span> <span class="n">ismissing</span><span class="x">(</span><span class="n">n</span><span class="x">)</span> <span class="k">return</span> <span class="s">"--"</span> <span class="k">end</span>
<span class="n">string</span><span class="x">(</span><span class="n">n</span><span class="x">,</span> <span class="n">base</span><span class="o">=</span><span class="mi">16</span><span class="x">,</span> <span class="n">pad</span><span class="o">=</span><span class="mi">2</span><span class="x">)</span>
<span class="k">end</span>
<span class="k">function</span><span class="nf"> hex4</span><span class="x">(</span><span class="n">n</span><span class="x">)</span>
<span class="k">if</span> <span class="n">ismissing</span><span class="x">(</span><span class="n">n</span><span class="x">)</span> <span class="k">return</span> <span class="s">"--"</span> <span class="k">end</span>
<span class="n">string</span><span class="x">(</span><span class="n">n</span><span class="x">,</span> <span class="n">base</span><span class="o">=</span><span class="mi">16</span><span class="x">,</span> <span class="n">pad</span><span class="o">=</span><span class="mi">4</span><span class="x">)</span>
<span class="k">end</span>
</code></pre></div></div>
<p>The <code class="language-plaintext highlighter-rouge">setformat!</code> function can be used to set the format of a column of a dataset. The exclamation mark indicates that this function modifies the first argument. The function <code class="language-plaintext highlighter-rouge">Symbol</code> can be used to create symbols, composed of a letter and a number to be used in a loop.</p>
<div class="language-julia highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">setformat!</span><span class="x">(</span><span class="n">ds</span><span class="x">,</span> <span class="o">:</span><span class="n">addr</span> <span class="o">=></span> <span class="n">hex4</span><span class="x">)</span>
<span class="k">for</span> <span class="n">i</span> <span class="k">in</span> <span class="mi">1</span><span class="o">:</span><span class="mi">8</span>
<span class="n">setformat!</span><span class="x">(</span><span class="n">ds</span><span class="x">,</span> <span class="kt">Symbol</span><span class="x">(</span><span class="s">"d"</span><span class="x">,</span> <span class="n">i</span><span class="x">)</span> <span class="o">=></span> <span class="n">hex2</span><span class="x">)</span>
<span class="k">end</span>
</code></pre></div></div>
<h2 id="filtering-the-data-on-one-column">Filtering the data on one column</h2>
<p>Let us assume that the devices with the address 0x103, 0x106 and 0x109 are output devices, and we want to filter all the messages sent to the output devices. For complex filter conditions a function that returns <code class="language-plaintext highlighter-rouge">true</code> or <code class="language-plaintext highlighter-rouge">false</code> is needed. Missing values must be handled correctly.</p>
<div class="language-julia highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># CAN bus devices with a given set of addresses are output devices</span>
<span class="k">function</span><span class="nf"> isoutput</span><span class="x">(</span><span class="n">addr</span><span class="x">)</span>
<span class="k">if</span> <span class="n">ismissing</span><span class="x">(</span><span class="n">addr</span><span class="x">)</span> <span class="k">return</span> <span class="nb">false</span> <span class="k">end</span>
<span class="n">addr</span> <span class="k">in</span> <span class="x">[</span><span class="mh">0x103</span><span class="x">,</span> <span class="mh">0x106</span><span class="x">,</span> <span class="mh">0x109</span><span class="x">]</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Finally we can apply the <code class="language-plaintext highlighter-rouge">filter</code> function, which requires three parameters, the input dataset, the column for applying the filter function and - as named parameter - the filter function.</p>
<div class="language-julia highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># create the dataset msg_out with all messages sent to output devices</span>
<span class="n">msg_out</span> <span class="o">=</span> <span class="n">filter</span><span class="x">(</span><span class="n">ds</span><span class="x">,</span> <span class="o">:</span><span class="n">addr</span><span class="x">,</span> <span class="n">by</span><span class="o">=</span><span class="n">isoutput</span><span class="x">)</span>
</code></pre></div></div>
<h2 id="filtering-the-data-on-two-columns">Filtering the data on two columns</h2>
<p>Lets assume that in addition to apply a filter on the column <code class="language-plaintext highlighter-rouge">addr</code> we are only interested in the data from <code class="language-plaintext highlighter-rouge">t=1000s</code> to <code class="language-plaintext highlighter-rouge">t=3000s</code>. Then we need a second filter function:</p>
<div class="language-julia highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">function</span><span class="nf"> ininterval</span><span class="x">(</span><span class="n">time</span><span class="x">)</span>
<span class="k">if</span> <span class="n">ismissing</span><span class="x">(</span><span class="n">time</span><span class="x">)</span> <span class="k">return</span> <span class="nb">false</span> <span class="k">end</span>
<span class="n">time</span> <span class="o">>=</span> <span class="mf">1000.0</span> <span class="o">&&</span> <span class="n">time</span> <span class="o"><=</span> <span class="mf">3000.0</span>
<span class="k">end</span>
</code></pre></div></div>
<p>and pass arrays of the column names and of the column filter functions to the dataset filter function.</p>
<div class="language-julia highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">msg_out_ininterval</span> <span class="o">=</span> <span class="n">filter</span><span class="x">(</span><span class="n">ds</span><span class="x">,</span> <span class="x">[</span><span class="o">:</span><span class="n">time</span><span class="x">,</span> <span class="o">:</span><span class="n">addr</span><span class="x">],</span> <span class="n">by</span><span class="o">=</span><span class="x">[</span><span class="n">ininterval</span><span class="x">,</span> <span class="n">isoutput</span><span class="x">])</span>
</code></pre></div></div>
<h2 id="browsing-through-the-output-data">Browsing through the output data</h2>
<p>The filtered data set <code class="language-plaintext highlighter-rouge">msg_out</code> is still too big to print it on one screen, so we need a way to browse through the result set. I am using the following function to do that:</p>
<div class="language-julia highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">function</span><span class="nf"> browse</span><span class="x">(</span><span class="n">ds</span><span class="x">)</span>
<span class="n">io</span> <span class="o">=</span> <span class="kt">IOContext</span><span class="x">(</span><span class="kt">IOBuffer</span><span class="x">(),</span> <span class="o">:</span><span class="n">color</span> <span class="o">=></span> <span class="nb">true</span><span class="x">);</span>
<span class="n">show</span><span class="x">(</span><span class="n">io</span><span class="x">,</span> <span class="n">ds</span><span class="x">,</span> <span class="n">show_row_number</span><span class="o">=</span><span class="nb">false</span><span class="x">,</span> <span class="n">eltypes</span><span class="o">=</span><span class="nb">false</span><span class="x">)</span>
<span class="n">pager</span><span class="x">(</span><span class="kt">String</span><span class="x">(</span><span class="n">take!</span><span class="x">(</span><span class="n">io</span><span class="o">.</span><span class="n">io</span><span class="x">)),</span> <span class="n">frozen_rows</span><span class="o">=</span><span class="mi">3</span><span class="x">)</span>
<span class="k">end</span>
</code></pre></div></div>
<p>The option <code class="language-plaintext highlighter-rouge">:color => true</code> allows the use of formatting, e.g. the header will be in bold.
The options <code class="language-plaintext highlighter-rouge">show_row_number=false</code> and <code class="language-plaintext highlighter-rouge">eltypes=false</code> suppress the row numbers and the row with the column types.</p>
<p>You can now browse through the output messages with the command:</p>
<div class="language-julia highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">browse</span><span class="x">(</span><span class="n">msg_out</span><span class="x">)</span>
</code></pre></div></div>
<p>It will look like this (well, with the header in bold):</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>6475×10 Dataset
time addr d1 d2 d3 d4 d5 d6 d7 d8
──────────────────────────────────────────────
8.2 0109 17 d6 13 e2 60 14 a4 8f
9.9 0103 45 14 61 a3 6f 18 e7 35
12.1 0109 7b 16 43 65 e9 f2 87 99
12.8 0106 87 c0 46 f1 4e 49 ad 74
13.8 0106 bc 88 f1 0a a9 c4 99 ef
14.1 0106 63 bb 83 96 8f 56 9a 99
14.8 0106 f7 94 2b 01 b3 45 12 3a
15.3 0109 3e 74 04 16 15 93 5f 0c
15.6 0103 7b e6 53 c7 0d 89 6b f3
15.7 0109 8e db d2 30 c1 b0 32 fa
16.0 0106 5e 52 da 97 74 81 e6 b8
:
</code></pre></div></div>
<p>You can scroll through the dataset with the cursor keys and the page up and page down keys. You can also search for a string with the slash key, then typing the search term and then <ENTER>. To finish browsing press <code class="language-plaintext highlighter-rouge">q</code>.</p>
<p>As you can see, all the data is nicely formatted as hexadecimal numbers.</p>
<h2 id="further-reading">Further reading</h2>
<p>To learn more about the filtering options, have a look at <a href="https://sl-solution.github.io/InMemoryDatasets.jl/stable/man/filter/">Filter observations</a>.</p>
<h2 id="acknowledgements">Acknowledgements</h2>
<p>Thanks to <a href="https://discourse.julialang.org/t/how-to-combine-prettytables-and-pager/85593/4?u=ufechner7">James D Foster</a> for providing the browse() function and the developers of <a href="https://sl-solution.github.io/InMemoryDatasets.jl/stable/">InMemoryDataframes</a> for their nice package.</p>UweIntroduction The basic work flow of data analysis is filtering a large data set to extract the interesting aspects and to browse and/or plot the result. In this post I will give a basic example for filtering and browsing of data, plotting deserves a blog post on its own.Exporting formatted datasets2022-08-07T00:00:00+00:002022-08-07T00:00:00+00:00https://ufechner7.github.io/2022/08/07/exporting-formatted-datasets<h2 id="introduction">Introduction</h2>
<p>For analysing <a href="https://en.wikipedia.org/wiki/CAN_bus">CAN bus</a> log files I am exporting the data sets with CAN messages to .csv and then import them in LibreOffice spread sheets. Finally I create Excel files that I can give to my collegues.</p>
<p>The CAN bus is used in cars, wind turbines, electric chargers, UAVs and many other industrial devices for the communication between intelligent sensors, actuators, controllers and user interface devices.</p>
<p>One log file can easily contain millions of messages, therefore Julia is a good choice for statistics, error analysis and graphical presentation of relevant aspects of the CAN bus traffic due to the simplicity, power and performance Julia provides.</p>
<p>But CAN messages are usually hex encoded. So how can we export a dataset with some hex
encoded columns?</p>
<h2 id="creating-a-test-project">Creating a test project</h2>
<p>For trying out a new package and/or example it is always good to create a new project first.
For example using the following commands:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">mkdir </span>can
<span class="nb">cd </span>can
julia <span class="nt">--project</span><span class="o">=</span><span class="s2">"."</span>
</code></pre></div></div>
<p>And then install the required packages:</p>
<div class="language-julia highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">julia</span><span class="o">></span> <span class="k">using</span> <span class="n">pkg</span>
<span class="n">julia</span><span class="o">></span> <span class="n">pkg</span><span class="s">"add InMemoryDatasets"</span>
<span class="n">julia</span><span class="o">></span> <span class="n">pkg</span><span class="s">"add DLMReader"</span>
<span class="n">julia</span><span class="o">></span> <span class="n">pkg</span><span class="s">"add Printf"</span>
</code></pre></div></div>
<p>Now quit julia with <CTRL><D> and restart it with:</D></CTRL></p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>julia <span class="nt">--project</span> <span class="nt">-t</span> auto
</code></pre></div></div>
<p>This uses the set of packages we just installed and starts julia using all available threads. This is useful when handling large data sets (millions of messages).</p>
<h2 id="creating-a-sample-data-set">Creating a sample data set</h2>
<div class="language-julia highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">using</span> <span class="n">InMemoryDatasets</span><span class="x">,</span> <span class="n">DLMReader</span><span class="x">,</span> <span class="n">Printf</span>
<span class="n">ds</span> <span class="o">=</span> <span class="n">Dataset</span><span class="x">(</span><span class="n">time</span><span class="o">=</span><span class="mf">0.0</span><span class="x">,</span> <span class="n">d1</span><span class="o">=</span><span class="mi">10</span><span class="x">,</span> <span class="n">d2</span><span class="o">=</span><span class="mi">20</span><span class="x">)</span>
<span class="n">time</span> <span class="o">=</span> <span class="mf">0.1</span>
<span class="k">for</span> <span class="n">i</span> <span class="k">in</span> <span class="mi">1</span><span class="o">:</span><span class="mi">9</span>
<span class="kd">global</span> <span class="n">time</span>
<span class="n">ds2</span> <span class="o">=</span> <span class="n">Dataset</span><span class="x">(</span><span class="n">time</span><span class="o">=</span><span class="n">time</span><span class="x">,</span> <span class="n">d1</span><span class="o">=</span><span class="mi">10</span><span class="o">+</span><span class="mi">1</span><span class="x">,</span> <span class="n">d2</span><span class="o">=</span><span class="mi">20</span><span class="o">+</span><span class="n">i</span><span class="x">)</span>
<span class="n">append!</span><span class="x">(</span><span class="n">ds</span><span class="x">,</span> <span class="n">ds2</span><span class="x">)</span>
<span class="n">time</span> <span class="o">+=</span> <span class="mf">0.1</span>
<span class="k">end</span>
<span class="n">ds</span><span class="o">.</span><span class="n">d1</span><span class="x">[</span><span class="mi">4</span><span class="x">]</span> <span class="o">=</span> <span class="nb">missing</span>
</code></pre></div></div>
<p>If you run this code the dataset should look like this:</p>
<div class="language-julia highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">julia</span><span class="o">></span> <span class="n">ds</span>
<span class="mi">10</span><span class="n">×3</span> <span class="n">Dataset</span>
<span class="n">Row</span> <span class="n">│</span> <span class="n">time</span> <span class="n">d1</span> <span class="n">d2</span>
<span class="n">│</span> <span class="n">identity</span> <span class="n">identity</span> <span class="n">identity</span>
<span class="n">│</span> <span class="kt">Float64</span><span class="o">?</span> <span class="kt">Int64</span><span class="o">?</span> <span class="kt">Int64</span><span class="o">?</span>
<span class="n">─────┼──────────────────────────────</span>
<span class="mi">1</span> <span class="n">│</span> <span class="mf">0.0</span> <span class="mi">10</span> <span class="mi">20</span>
<span class="mi">2</span> <span class="n">│</span> <span class="mf">0.1</span> <span class="mi">11</span> <span class="mi">21</span>
<span class="mi">3</span> <span class="n">│</span> <span class="mf">0.2</span> <span class="mi">12</span> <span class="mi">22</span>
<span class="mi">4</span> <span class="n">│</span> <span class="mf">0.3</span> <span class="mi">13</span> <span class="mi">23</span>
<span class="mi">5</span> <span class="n">│</span> <span class="mf">0.4</span> <span class="nb">missing</span> <span class="mi">24</span>
<span class="mi">6</span> <span class="n">│</span> <span class="mf">0.5</span> <span class="mi">15</span> <span class="mi">25</span>
<span class="mi">7</span> <span class="n">│</span> <span class="mf">0.6</span> <span class="mi">16</span> <span class="mi">26</span>
<span class="mi">8</span> <span class="n">│</span> <span class="mf">0.7</span> <span class="mi">17</span> <span class="mi">27</span>
<span class="mi">9</span> <span class="n">│</span> <span class="mf">0.8</span> <span class="mi">18</span> <span class="mi">28</span>
<span class="mi">10</span> <span class="n">│</span> <span class="mf">0.9</span> <span class="mi">19</span> <span class="mi">29</span>
</code></pre></div></div>
<h2 id="formatting-the-output">Formatting the output</h2>
<p>For formatting the columns d1 and d2 in hex we need the following lines of code:</p>
<div class="language-julia highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">function</span><span class="nf"> round6</span><span class="x">(</span><span class="n">value</span><span class="x">)</span>
<span class="nd">@sprintf</span><span class="x">(</span><span class="s">"%12.6f"</span><span class="x">,</span> <span class="n">value</span><span class="x">)</span>
<span class="k">end</span>
<span class="k">function</span><span class="nf"> hex</span><span class="x">(</span><span class="n">n</span><span class="x">)</span>
<span class="k">if</span> <span class="n">ismissing</span><span class="x">(</span><span class="n">n</span><span class="x">)</span> <span class="k">return</span> <span class="s">"--"</span> <span class="k">end</span>
<span class="n">string</span><span class="x">(</span><span class="n">n</span><span class="x">,</span> <span class="n">base</span><span class="o">=</span><span class="mi">16</span><span class="x">,</span> <span class="n">pad</span><span class="o">=</span><span class="mi">2</span><span class="x">)</span>
<span class="k">end</span>
<span class="n">setformat!</span><span class="x">(</span><span class="n">ds</span><span class="x">,</span> <span class="o">:</span><span class="n">time</span> <span class="o">=></span> <span class="n">round6</span><span class="x">)</span>
<span class="n">setformat!</span><span class="x">(</span><span class="n">ds</span><span class="x">,</span> <span class="o">:</span><span class="n">d1</span> <span class="o">=></span> <span class="n">hex</span><span class="x">)</span>
<span class="n">setformat!</span><span class="x">(</span><span class="n">ds</span><span class="x">,</span> <span class="o">:</span><span class="n">d2</span> <span class="o">=></span> <span class="n">hex</span><span class="x">)</span>
</code></pre></div></div>
<p>Because our dataset can contain missing values we need to handle the special case
that n is missing. The <code class="language-plaintext highlighter-rouge">round6</code> function is not strictly required, but for easy readability
of the csv output I wanted to have a fixed number of digits for the time stamp.</p>
<p>If we now print the dataset in the REPL we get:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>julia> show(ds, show_row_number=false, eltypes=false)
10×3 Dataset
time d1 d2
──────────────────────
0.000000 0a 14
0.100000 0b 15
0.200000 0c 16
0.300000 0d 17
0.400000 -- 18
0.500000 0f 19
0.600000 10 1a
0.700000 11 1b
0.800000 12 1c
0.900000 13 1d
</code></pre></div></div>
<p>Now all columns are nicely formatted. I am using here the keyword parameters <code class="language-plaintext highlighter-rouge">show_row_number=false</code>
and<code class="language-plaintext highlighter-rouge">eltypes=false</code> to suppress the output of the column types and the row numbers.</p>
<p>Saving this as .csv file is now easy:</p>
<div class="language-julia highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">filewriter</span><span class="x">(</span><span class="s">"output.csv"</span><span class="x">,</span> <span class="n">ds</span><span class="x">,</span> <span class="n">mapformats</span><span class="o">=</span><span class="nb">true</span><span class="x">)</span>
</code></pre></div></div>
<p>The trick is to use the named parameter <code class="language-plaintext highlighter-rouge">mapformat=true</code>, if you do that the formatting function
is applied on the .csv output. The resulting file looks like this:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>shell> cat output.csv
time,d1,d2
0.000000,0a,14
0.100000,0b,15
0.200000,0c,16
0.300000,0d,17
0.400000,--,18
0.500000,0f,19
0.600000,10,1a
0.700000,11,1b
0.800000,12,1c
0.900000,13,1d
</code></pre></div></div>
<h2 id="importing-this-with-libreoffice">Importing this with LibreOffice</h2>
<p>You can just double click on the file <code class="language-plaintext highlighter-rouge">output.csv</code> and a dialog box will open. Just make sure to select the column type <code class="language-plaintext highlighter-rouge">text</code> for colum d1 and d2.</p>
<p><img src="https://raw.githubusercontent.com/ufechner7/ufechner7.github.io/4c0e76b638ec38c7de19d684ee8c019e43805b4a/_posts/dialog.png" alt="Dialog" /></p>
<p>When you click on OK you have a well formatted table which you can save as .odf spreadsheet or in Excel format
for further analysis and distribution.</p>UweIntroduction For analysing CAN bus log files I am exporting the data sets with CAN messages to .csv and then import them in LibreOffice spread sheets. Finally I create Excel files that I can give to my collegues.