2017-04-08

In a Galaxy Far Far Away

As many of you already know, at the beginning of the year I joined Oak Ridge National Laboratory as a Research Associate. My job is to use my knowledge in HPC performance analysis to enable scientific codes (like these) to run better on Titan - the fastest open-science super computer of the western world and its successor Summit (POWER9+NVIDIA Volta). Needless to say that this is super-exciting work!

As a result, I now live in Knoxville/Tennessee and there is so much to talk about :). This is not a coherent post. More like a fact and opinion rapid-fire.

The flag of Tennessee, where it is on the map, and the ORNL Logo

Everything feels around 1.5x as expensive as in Germany. Around what I would expect if the exchange rate to Euro would have stayed where it initially was. Gas costs 53¢/l. Electricity costs 9.8¢/kWh. Let that sink in for a second - Electricity and gas are practically dirt-cheap. A full tank costs me $23. No wonder people drive these giant-ass pickup trucks.
Houses are generally cheaply made - starting at ~$50k. In my apartment the air conditioning system both cools and heats. This combination of cheap housing and super-inefficient heating resulted in my first two month's electricity bills being around $40 (~400kWh). And this has to be just heating and maybe the fridge (It's a biiiig fridge). Although electricity is cheaper, I managed to pay more than a single-person household would pay in Germany. Electricity is so cheap people don't even bother getting LED light bulbs.
Wow, just wow.

Meat, especially beef is a lot cheaper. You can get a decent steak for $15 per kg. In general beef tastes a lot lot better than what you can get in Germany. It's seriously amazing. Beans and rice are also cheap. Ben & Jerry's costs $4-5, which is a blessing :).

A gigantic T-bone steak

Dinner time <3

My phone bill is around $30 for 20 minutes, 30 messages and 550MB, and Comcast wants $70 for 25/5MBit. Comcast has the worst customer service I have ever seen. I spent many attempts and a total of 2-3 hours to get two things sorted out. It's actually ridiculous how unhelpful their customer support is.

The typical rent in Oak Ridge/Knoxville ranges between $500 and $1000 for a normal sized flat.

Spatially, everything here is very very spread out. Knoxville has a population of 185k on roughly Dresden's area, and Knox County has 430k on five times that area. Oak Ridge has 30k people on 3/4ths the area of Dresden. In order to go downtown I need to drive about 8km. To the next mall I need somewhere between 2 and 8km depending on which. The only thing where I can really walk to is the gas station down the road. Sidewalks are rare, almost non-existent. Only downtown is it possible to get anywhere without a car. That makes having a car essential. This also means you cannot run, bike or take a decent walk from your door step, but rather have to drive somewhere and then go from there. It's annoying having to drive, go for a run, and then drive home all sweaty.

So I had to buy a car. The best option turned out to be to just buy it from a friend who left the country at the time I arrived. It's a shiny 2015 Ford Focus Sedan 6-shift manual.



When I arrived in the USA, I used a rental car for while. During that time I came to like automatic transmission. Sadly this one is a manual. Well, at least no one will be able to steal it - although.
I think Focus sedans don't exist in Germany. Overall there are more big cars on the road here and most of them are sedans and pickup trucks. Hatchbacks are very rare. Almost no one drives small cars. Very rarely do you see a Smart, a Fiesta or a Golf.
I bought a dash cam to play around with and for insurance purposes. People here are bad drivers: No signaling, random lane switching, overtaking on the right. Something like the TÜV does not exist. There are cars driving around where you are surprised they are able to drive at all. This also means there is no emission checking. I saw couple of cars that emitted pitch-black fumes. What the hell. Getting a driver's license is very easy, and costs just $30. Many don't even have car insurance. Phoning while driving is allowed. Texting is not.
In Tennessee, you can film anywhere where people have no reasonable expectation of privacy. Which is (almost?) everywhere in public. Everywhere else you are allowed to record conversations of where you have at least the consent of one involved party, for example yourself.

When you go shopping, everything is put into bags for you by the cashier. Bags bags bags. One notable exception is Aldi. Here's a picture of the result of my first shopping tour in the US.


No one recycles. In order to recycle you need to drive to the next recycling center - 7km for me. It's just a simple place with big containers for each kind of waste. Before I began recycling, I created at least four times the amount of trash I generated in Germany, which is crazy.  Separating biodegradable waste is not a thing at all.

Many stores are open every day and for way longer than in Germany. For example at Walmart you can go shopping 24/7.

Tennessee sees extremely quick weather changes, I think because it's continental and the mountain ranges are vertical so that weather coming in from north and south passes through quickly. One day it's 0°C, the next day it's 10°C and a few days later it can be more than 20°C. And then suddenly a thunderstorm with 1cm hails lasting for just one hour shows up.  Ughh. I feel like I have seen more extreme weather in four months than my hole life.

Here's nine people blowing leafs around the apartment complex towards their truck. I would expect rakes, but no.


Another peculiarity is beer. The old opinion that American beer is bad does not hold anymore. Actually, I think it's overall better than in Germany. Mainly because the variety is so large. There are many small breweries creating all kinds of beers - IPAs, Ales, Stouts, Lagers, Pilseners. Their Pilseners are terrible in comparison to ours, though. One big downside is that I get headaches very quickly from their stuff. Not sure what exactly causes this. Perhaps it's the added sugar in some of them. The price for beer starts at around $1-1.5 for a 350ml bottle of industrial beer. A beer at a bar is usually between $4 and $6.

A bazillion beers on tap. Every time a barrel is emptied,
they add a different beer. Great!

They ship this to here!

Create your own six-pack for $10 at Kroger.

The German/Austrian beers at Bearden Beer Market.

Trivia is a big thing around here. Lots of bars do it on a weekly basis. Our moderator Will at Hops and Hollers does an awesome job at it. Trivia is awesome! You have to collectively find the right answer, and betting-rounds or be-as-close-as-you-can-to-the-correct-answer increase tension and fun. Since I'm here we managed to win twice, and perhaps we will make it to a second-tier tournament later this year. Our strong suits are science, music, movies, history, religion, alcohol, geography, everything German / European. We are reasonable good at sports and suck a lot at Celebrities, TV and commercials.

Imperial units. Some call them freedom units or retard units. Only a handful of countries still use them. I got used to them more quickly than I expected. You only convert when you need to be exact. Most of the time you don't convert at all. On the road 10 miles is far, a quarter mile is soon, 1000 feet is sooner and 100 feet is very soon. An ounce is 30 grams. A pound is half a kilogram. A quarter pound is 100g, 12 fluid ounces is a can, a freedom pint is 16floz which is around 500ml. Yes, an American pint is really smaller than an imperial pint. A gallon is four liters.

While I'm here I have the goal to try as much local things as possible. I tried Pop Tarts, Twinkies, steak, philly cheese steak sandwich, mac & cheese, burger, hot dog, pizza, salsa, tortillas, tacos, quesadillas, chicken wings, and all those food chains like Chipotle, Five Guys, Blaze Pizza, Wendy's etc. There is actually quite a few things that are super-tasty!

Twinkies do have an expiration date!

Among other things I also want to see a rocket launch, go to a republican rally and to the shooting range. That's going to be interesting!

Since the election of Donald Trump politics has been a big shitfest. While Germany wrangles over sharing passport data with the police and meta data retention, things like these happen here:
I hope that voters pick up on these things and react accordingly.

Oak Ridge National Laboratory, short the lab, is a very interesting place to work at. It is located on a 40km2 patch of land in the middle of nowhere surrounded by mountain ranges. To get to it you need to drive at least 10km. My commute from Knoxville is about 30km. The nearest airport is 45km away.
Historically it is one of the sites created for the Manhattan Project. More specifically, the lab was tasked with creating large amounts of weapons-grade plutonium.
During this time the city of Oak Ridge has been established.

Today the lab still has the old graphite reactor opened for tourists. Besides that it has a number of leading-edge science facilities. Some of them are:
The OLCF houses Titan, the #1 civil supercomputer of the western world. In the Top500 there are two Chinese machines that score higher, but that's it. Titan consists of 200 cabinets, 18,000 nodes, 18,000 NVIDIA Kepler cards and 300,000 AMD Opteron cores.
The successor to it is Summit, which will be built by IBM and will contain POWER9 processors, NVIDIA Volta cards and NVRAM. It will deliver north of 100 petaFLOPS. Before Summit starts arriving later this year, right now we have a test setup consisting of POWER8 + P100 cards and local NVMe storage to gain experience with. Shiny!


The lab has a staff of around 4000 people, plus 3000 guest researchers, and 500 interns annually from all around the world. It is a very diverse work place. I could not find numbers, but I would guess that at least a third are foreigners. I know people from Belgium, China, Egypt, England, France, Germany, India, Kyrgystan, South Africa and Spain. It feels like almost everyone has a PhD, and people are generally super-smart and relaxed. I love it.

One of the main tasks of the national labs in the USA is the maintenance of the nuclear arsenal. That includes for example simulating the decay of the weapons.

The nature of interactions at the lab is a bit different from what I am used to. Oftentimes people don't directly answer/admit something. There is a kind of double-speak depending on the subject. I can't really put a finger on it or want to give examples. It's just how it feels. I think this is a product of the type of work some people are doing, the things some people know/suspect, the fact that you should assume to be watched, and the fact that suing people is a national sport. The amount of ass-covering going on is substantial, not just at the lab, but in general. For example there is a banner in my building thanking people for millions of safe work hours. And packaging of food or hot beverages often tells you not to be an idiot.

The lab is used both by the Department of Energy (DOE) and Department of Defense (DOD). The DOD is the parent agency of the NSA.

I don't know how it exactly works, but I think the DOE is a rather civil branch whereas the DOD is the military. One of the tasks of the DOE is to maintain and control the nuclear stockpile together with the DOD, so that not only one department has control over it.

The median income per person in the USA is around $30k, the median household is $50k. Tennessee is a bit below that. With a master's degree I am at more than twice the per-person-median and in the top 20%. I feel the income distribution is a bit worse than in Germany, especially towards the lower end. I feel that lab employees are overall fairly rich. Rumor has it that better positions pay way way better than e.g. higher positions in German public service. It's run like a private company.

To conclude this post, here are a few photos for you to enjoy:

A panoramic view of a few houses and the Interstate 40

Lake Shore Park is nice for running.

On our way to Ober Gatlingburg to go skiing


There is a Disney store in West Town Mall

It consists mainly of Star Wars merchandise and princess costumes.

Pi Day!

Authentic German cuisine. Haven't been there, yet.

A muskrat minding its business

The lab geese are having babies! Update: The nest is gone. Investigation pending.

I wonder whether these two have names. If not I will name them.
Perhaps they should have badges like any other employee.

A bit of Saxony far from home. Thank you Aldi.

Freedom sushi: Deep-fried, mayo-based sauce, creem cheese.
A friend ordered tea, and the first thing offered was sweet tea, haha.

Someone at the Lab drives this bad boy


Here is a time lapse of a short drive in West Knoxville:


I hope you are having a great time at home. So far I'm having lots of fun here. Make sure to hit me up if you are nearby. I'll try to be at the Super Computing Conference later this year. Hopefully, I'll see you there. Bye!

    2017-02-13

    Intel SSD 600p (1TB) Luks+Ext4 Encrypted Disk Performance

    I got myself a new toy - The Intel SSD 600p 1TB M.2 NVMe disk. Before I bought it, I tried to figure out how big the performance penalty for using full disk encryption would approximately be. But I wasn't able to find much on this topic. Now that I set it up, I want to share some benchmarks of how an encrypted LUKS+Ext4 partition performs in comparison to just using Ext4.

    All benchmark runs were made with GNOME Disks on a Lenovo Thinkpad T460s with a Core i5-6200U CPU running Ubuntu 16.10 Yakkety Yak (Kernel 4.8.0-37-generic x86_64) and KDE. The encrypted partition was set up via GNOME Disks.

    LUKS+Ext4

     

    Intel 600p 1TB LUKS+Ext4 Performance at 1 MiB Block Size
    1 MiB Block Size

    Intel 600p 1TB LUKS+Ext4 Performance at 10 MiB Block Size
    10 MiB Block Size

    Intel 600p 1TB LUKS+Ext4 Performance at 100 MiB Block Size
    100 MiB Block Size

    Intel 600p 1TB LUKS+Ext4 Performance at 1000 MiB Block Size #1
    1000 MiB Block Size

    The 1000 MiB block size attempts were inconsistent. After a minute of waiting before doing another run, I got the above result. Many look like the two runs below. It is either a temperature or caching effect I guess.

    Intel 600p 1TB LUKS+Ext4 Performance at 1000 MiB Block Size #2
    1000 MiB #2

    Intel 600p 1TB LUKS+Ext4 Performance at 1000 MiB Block Size #3
    1000 MiB #3

    Ext4 Only

     

    Intel 600p 1TB Ext4 Performance at 1 MiB Block Size
    1 MiB Block Size

    Intel 600p 1TB Ext4 Performance at 10 MiB Block Size
    10 MiB Block Size

    Intel 600p 1TB Ext4 Performance at 100 MiB Block Size
    100 MiB Block Size

    Intel 600p 1TB Ext4 Performance at 1000 MiB Block Size
    1000 MiB Block Size

     Evaluation

    • Just reading/writing data at 1.8/0.6 GB per second takes CPU time, regardless of whether encryption is enabled or not
    • LUKS+Ext4 incurs signifcant extra CPU work over just using Ext4. Notice that the CPU uses all four threads for the encrypted partition, but just one or two threads for the unencrypted one.
    • Access time according to GNOME disks is similar. The difference is less than a factor of 2 in all cases.
    • 1 MiB Block Size: Encryption incurs a bandwidth penalty of factor 2 to 4.
    • 10 MiB: There is a read slow down factor of 2 with encryption. Writing is not much slower.
    • 100 MiB: The encrypted drive alternates between 1.7 GB/s and 1.0 GB/s read speed for some reason. Writing looks OK.
    • 1000 MiB: You get almost full speed.

    2016-05-30

    Race Report: 19ter Lauf in den Frühling

    Am 6. Mai war es soweit, der Lauf in den Frühling stand an. Laut Trainingsplan sollten die 44 Minuten für 10km möglich sein. Andererseits ist eine 4:24min/km pace ziemlich hart.


    Nachdem die Kleinen jeweils 800m und 2km gelaufen, sind gehen die Großen an den Start. Ich war schon ein bisschen nervös. Vorsichtshalber habe ich mich weiter vorn postiert damit ich dann nicht so viel überholen muss. Kurz vor dem Start erfuhr ich noch dass es keine Kilometermarken gibt. Das bedeutet ich werde die ganze Zeit nur eine wage Ahnung haben wie schnell oder langsam ich bin. Ärgerlich.

    Los geht's.

    Die ersten paar Kilometer sind sehr flott und gehen gut von der Hand. Das Höhenprofil in Kombination mit dem Tempo macht den Lauf anstrengend.


    Die ersten fünf Kilometer gehen zügig vorüber. Nach dem ersten Wendepunkt (Spitze des Berges) auf dem Weg zurück treffe ich die Kollegen recht spät. Ich bin wohl ziemlich ziemlich schnell.

    Ab Kilometer 7 geht mir dann langsam die Puste aus. Ein bisschen langsamer machen. Ab dann wird es stückweise immer anstrengender und ich werde gefühlt immer langsamer. Wie langsam? Keine Ahnung. Wohl immer noch schneller als 5:00min/km.

    Ungefähr bei Kilometer 8 kann ich in der Ferne das Ziel sehen. Aber dort muss jeder nochmal zum zweiten Wendepunkt abbiegen. Der Körper würde aber gern aufhören. Puh. Durchhalten. Auf den letzten drei Kilometern haben mich circa zehn Läufer überholt :(.

    500m vor dem Finish steht fest: Ich schaffe die 44:00. Entspannung setzt ein und das Tempo sinkt noch ein wenig. Im Ziel sind es dann 43:12 (4:19min/km, 13.9km/h) Wohoo \o/. Die Platzierung ist auch ordentlich, sodass ich als einer der ersten 100 im Ziel ein T-Shirt gewinne.


    Die drei Kollegen kommen bei 47 bis 50 Minuten ins Ziel. Leider haben nur zwei von uns ein T-Shirt gewonnen. Ist aber auch nicht schlimm, weil nur Werbung in textmarker-grün drauf ist. Nichtmal "Lauf in den Frühling" oder etwas ähnliches steht drauf.

    Danach geht's ins Hains noch eine Stunde planschen. Das war im Startpreis inbegriffen. Super! Zwei neue Rutschen gilt es auszuprobieren. Eine rutscht man in einem Luftring herunter. Die andere ist eine Paarrutsche mit Zeitnahme, wo zwischen den beiden Bahnen teilweise nur eine transparente Wand ist sodass man den Anderen sieht. Yeah! Vier 30+-Männer haben lauthals ihren Spaß.

    Alles ein allem ein Top-Lauf, von der Orga, den Bedingungen, und der eigenen Leistung her. Und die Startgebühr war so gering dass mit dem freien Eintritt und dem T-Shirt es sich sogar gelohnt hat ;).

    Nächstes Jahr gerne wieder!

    ----

    Dieses mal hat mich mein Puls und meine Geschwindigkeit auf der Strecke wirklich interessiert. Aber bisher laufe ich zum Wettkampf nur mit der Vivosmart HR.


    Die Kurve muss man mit Vorsicht genießen, weil der optische Heartrate-Sensor nicht so gut ist - vor allem bei Tempo- und Temperaturvariation. Die Zahlen bis Minute 9 sind zum Beispiel komplett falsch. Die maxHR von 188 bei ca 21 Minuten stimmt wohl und dass ich bis Kilometer 7 mit immer über 180 ordentlich gelitten habe bzw. zu schnell gelaufen bin klingt auch richtig.